Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-64979

Behat: Behat tests are not reliable on slow machines

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide
      1. If necessary, run the Behat init.
      2. Add the following line to /index.php - I put it just before $PAGE->set_pagetype('site-index'); - about line 107:
        • $PAGE->requires->js_init_code('M.util.js_pending("stupid"); setTimeout(function() { M.util.js_complete("stupid"); }

          , 15000);');

        • This will make it delay for 15 seconds before Behat thinks the page is ready. We are simulating what it feels like to use a slow machine.
      3. Run Behat with the following scenario: blocks/private_files/tests/behat/block_private_files_frontpage.feature, second scenario 'Upload a file to the private files block from the frontpage'. (You can do this by name, by adding a tag to the file and using the tag, or in your IDE, it doesn't matter.)
        • You'll see an error message about JavaScript being very slow and taking more than 10 seconds.
      4. Edit your config.php and add the following line:
        • $CFG->behat_increasetimeout = 3;
        • This will make all the timeouts 3 times longer, e.g. 30 seconds.
      5. Re-run the Behat test
        • It should now pass.
      Show
      If necessary, run the Behat init. Add the following line to /index.php - I put it just before $PAGE->set_pagetype('site-index'); - about line 107: $PAGE->requires->js_init_code('M.util.js_pending("stupid"); setTimeout(function() { M.util.js_complete("stupid"); } , 15000);'); This will make it delay for 15 seconds before Behat thinks the page is ready. We are simulating what it feels like to use a slow machine. Run Behat with the following scenario: blocks/private_files/tests/behat/block_private_files_frontpage.feature, second scenario 'Upload a file to the private files block from the frontpage'. (You can do this by name, by adding a tag to the file and using the tag, or in your IDE, it doesn't matter.) You'll see an error message about JavaScript being very slow and taking more than 10 seconds. Edit your config.php and add the following line: $CFG->behat_increasetimeout = 3; This will make all the timeouts 3 times longer, e.g. 30 seconds. Re-run the Behat test It should now pass.
    • Affected Branches:
      MOODLE_35_STABLE, MOODLE_36_STABLE, MOODLE_37_STABLE
    • Fixed Branches:
      MOODLE_35_STABLE, MOODLE_36_STABLE
    • Pull 3.5 Branch:
      MDL-64979-m35
    • Pull 3.6 Branch:
      MDL-64979-m36
    • Pull Master Branch:
      MDL-64979-master

      Description

      When running on slow machines (e.g. using Windows, or with hard disk instead of SDD, or insufficient RAM), Behat tests can be unreliable because some of the timeouts are not long enough.

      The timeouts in Behat tests are defined in behat_base.php. They are fixed at 2 seconds, 6 seconds, or 10 seconds.

      I propose adding a config.php option $CFG->behat_increasetimeout = 3;

      This would increase all three time limits by the specified factor (so setting it to 3 makes the timeouts 6/18/30 which is enough at least for our machines, maybe not if somebody tries to run it on a VIC-20).

       

       

       

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  13/May/19

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 hour, 20 minutes
                  1h 20m