Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-57030

Add option to behat run tool to automatically rerun failures

    XMLWordPrintable

Details

    • MOODLE_31_STABLE, MOODLE_32_STABLE, MOODLE_33_STABLE
    • MOODLE_32_STABLE
    • wip-mdl-57030
    • Hide

      Test with parallel

      1. Initialize behat with 4
      2. Introduce failure is mod_book features for run 2 and run 4 (check behat.yml for run 2 and run 4 to find feature files.)
      3. Run behat and ensure you see failure and there is no rerun executed.
      4. Now add --auto-rerun to the run.php command above.
      5. Ensure only failed behat runs are executed again once.
      6. Now try with --auto-rerun=3 to the run.php command above
      7. Ensure only failed behat runs are executed again thrice.
      8. Drop behat site and Init behat with -a=clean and -j=4
      9. Run behat with following options and ensure auto rerun happens for clean suite 3 times.
        • --suite=clean
        • --tags=@mod_book
        • --auto-rerun=3
        • --verbose

      TEST SINGLE RUN

      1. Drop previous behat site
      2. Initialize behat site for single run
      3. Using run.php run behat
      4. Ensure there is no-rerun executed.
      5. Add --auto-rerun and ensure rerun is executed once.
      6. Try with --auto-rerun=3 and ensure rerun is executed thrice.
      Show
      Test with parallel Initialize behat with 4 Introduce failure is mod_book features for run 2 and run 4 (check behat.yml for run 2 and run 4 to find feature files.) Run behat and ensure you see failure and there is no rerun executed. Now add --auto-rerun to the run.php command above. Ensure only failed behat runs are executed again once. Now try with --auto-rerun=3 to the run.php command above Ensure only failed behat runs are executed again thrice. Drop behat site and Init behat with -a=clean and -j=4 Run behat with following options and ensure auto rerun happens for clean suite 3 times. --suite=clean --tags=@mod_book --auto-rerun=3 --verbose TEST SINGLE RUN Drop previous behat site Initialize behat site for single run Using run.php run behat Ensure there is no-rerun executed. Add --auto-rerun and ensure rerun is executed once. Try with --auto-rerun=3 and ensure rerun is executed thrice.

    Description

      I would be really really handy to have a --automatic-rerun option on the behat run.php that, after a run (parallel or single) would rerun the failures.

      For parallel runs it would rerun the failure in non-parallel fashion, say it would rerun run 1, then run 4. Then any remaining errors after the rerun would be reported in the exit code.

      Parallel runs will almost always have initial failures, because the browser doesn't have OS focus. Right now, for parallel runs, you either have to manually rerun the failed jobs, or you need to write a convoluted bash script to do it. This seems like something prime to be included in the core handler.

      Attachments

        Issue Links

          Activity

            People

              rajeshtaneja Rajesh Taneja
              emerrill Eric Merrill
              Eric Merrill Eric Merrill
              Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
              Ryan Wyllie Ryan Wyllie
              Andrew Lyons, Huong Nguyen, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze, Stevani Andolo
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                13/Mar/17