Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-57030

Add option to behat run tool to automatically rerun failures

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      Test with parallel

      1. Initialize behat with 4
      2. Introduce failure is mod_book features for run 2 and run 4 (check behat.yml for run 2 and run 4 to find feature files.)
      3. Run behat and ensure you see failure and there is no rerun executed.
      4. Now add --auto-rerun to the run.php command above.
      5. Ensure only failed behat runs are executed again once.
      6. Now try with --auto-rerun=3 to the run.php command above
      7. Ensure only failed behat runs are executed again thrice.
      8. Drop behat site and Init behat with -a=clean and -j=4
      9. Run behat with following options and ensure auto rerun happens for clean suite 3 times.
        • --suite=clean
        • --tags=@mod_book
        • --auto-rerun=3
        • --verbose

      TEST SINGLE RUN

      1. Drop previous behat site
      2. Initialize behat site for single run
      3. Using run.php run behat
      4. Ensure there is no-rerun executed.
      5. Add --auto-rerun and ensure rerun is executed once.
      6. Try with --auto-rerun=3 and ensure rerun is executed thrice.
      Show
      Test with parallel Initialize behat with 4 Introduce failure is mod_book features for run 2 and run 4 (check behat.yml for run 2 and run 4 to find feature files.) Run behat and ensure you see failure and there is no rerun executed. Now add --auto-rerun to the run.php command above. Ensure only failed behat runs are executed again once. Now try with --auto-rerun=3 to the run.php command above Ensure only failed behat runs are executed again thrice. Drop behat site and Init behat with -a=clean and -j=4 Run behat with following options and ensure auto rerun happens for clean suite 3 times. --suite=clean --tags=@mod_book --auto-rerun=3 --verbose TEST SINGLE RUN Drop previous behat site Initialize behat site for single run Using run.php run behat Ensure there is no-rerun executed. Add --auto-rerun and ensure rerun is executed once. Try with --auto-rerun=3 and ensure rerun is executed thrice.
    • Affected Branches:
      MOODLE_31_STABLE, MOODLE_32_STABLE, MOODLE_33_STABLE
    • Fixed Branches:
      MOODLE_32_STABLE
    • Pull Master Branch:
      wip-mdl-57030

      Description

      I would be really really handy to have a --automatic-rerun option on the behat run.php that, after a run (parallel or single) would rerun the failures.

      For parallel runs it would rerun the failure in non-parallel fashion, say it would rerun run 1, then run 4. Then any remaining errors after the rerun would be reported in the exit code.

      Parallel runs will almost always have initial failures, because the browser doesn't have OS focus. Right now, for parallel runs, you either have to manually rerun the failed jobs, or you need to write a convoluted bash script to do it. This seems like something prime to be included in the core handler.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              rajeshtaneja Rajesh Taneja
              Reporter:
              emerrill Eric Merrill
              Peer reviewer:
              Eric Merrill
              Integrator:
              Eloy Lafuente (stronk7)
              Tester:
              Ryan Wyllie
              Participants:
              Component watchers:
              Andrew Nicols, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Fix Release Date:
                13/Mar/17