Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-48741

New behat formatter to get --rerun compatible list of scenarios

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      1) Init the behat installation

      2) run --tags='@core_course' (with Firefox)

      3) Note the number of scenarios.

      4) run the same tag than 2) adding --dry-run --format=moodle_list > list.txt

      5) verify that the list.txt file contains the same number of scenarios than 3)

      6) run behat with --rerun="list.txt"

      7) verify it only executes the scenarios in that file. And the number matches 5)

      8 Perform a complete behat run with --dry-run --format=moodle_list > list.txt

      9) Verify that list.txt contains (at least) 742 scenarios (as of January 15th 2015, for master).

      That's all, the new moodle_list formatter is doing its job by providing a rerun compatible file.

      Note: @core_course has been selected for the tests because it includes both normal scenarios and outlines+examples. The formatter must support both.

      Note2: Failed steps (real or random) are not related with this issue. Here we are only interested on getting the counts and verify all them are ok and matching as specified above.

      Show
      1) Init the behat installation 2) run --tags='@core_course' (with Firefox) 3) Note the number of scenarios. 4) run the same tag than 2) adding --dry-run --format=moodle_list > list.txt 5) verify that the list.txt file contains the same number of scenarios than 3) 6) run behat with --rerun="list.txt" 7) verify it only executes the scenarios in that file. And the number matches 5) 8 Perform a complete behat run with --dry-run --format=moodle_list > list.txt 9) Verify that list.txt contains (at least) 742 scenarios (as of January 15th 2015, for master). That's all, the new moodle_list formatter is doing its job by providing a rerun compatible file. Note: @core_course has been selected for the tests because it includes both normal scenarios and outlines+examples. The formatter must support both. Note2: Failed steps (real or random) are not related with this issue. Here we are only interested on getting the counts and verify all them are ok and matching as specified above.
    • Affected Branches:
      MOODLE_27_STABLE, MOODLE_28_STABLE, MOODLE_29_STABLE
    • Fixed Branches:
      MOODLE_27_STABLE, MOODLE_28_STABLE
    • Pull from Repository:
    • Pull Master Branch:

      Description

      Reading here and there about behat and its internals I found the --rerun feature really interesting to keep apart random failures and also as a potential way to split/dispatch behat runs over multiple testing instances.

      But there is a missing piece in order to be able to use it properly and it is that we need to be able to build such "rerun" files easily, so they can be used later.

      Digging into behat, it seems that the best way to achieve that properly is to create a new formatter (behat already uses that internally for "rerun") and combine it with --dry-run, so we can get the "rerun" file ready for any use.

      So this issue aims to:

      1) Create a new formatter, available via moodle-behat-extension.
      2) Make moodle aware of it, so can be used easily.

      Note: I'm only providing master (2.9) patches for this new formatter, but I'd vote to backport it to all supported versions coz it's 100% isolated and can bring us interesting possibilities to be applied @ nightly and CIs.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stronk7 Eloy Lafuente (stronk7)
                Reporter:
                stronk7 Eloy Lafuente (stronk7)
                Peer reviewer:
                Rajesh Taneja
                Integrator:
                Andrew Nicols
                Tester:
                Jetha Chan
                Participants:
                Component watchers:
                Andrew Nicols, Mathew May, Michael Hawkins, Shamim Rezaie, Simey Lameze
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  2/Feb/15