Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-48741

New behat formatter to get --rerun compatible list of scenarios

XMLWordPrintable

    • MOODLE_27_STABLE, MOODLE_28_STABLE, MOODLE_29_STABLE
    • MOODLE_27_STABLE, MOODLE_28_STABLE
    • Hide

      1) Init the behat installation

      2) run --tags='@core_course' (with Firefox)

      3) Note the number of scenarios.

      4) run the same tag than 2) adding --dry-run --format=moodle_list > list.txt

      5) verify that the list.txt file contains the same number of scenarios than 3)

      6) run behat with --rerun="list.txt"

      7) verify it only executes the scenarios in that file. And the number matches 5)

      8 Perform a complete behat run with --dry-run --format=moodle_list > list.txt

      9) Verify that list.txt contains (at least) 742 scenarios (as of January 15th 2015, for master).

      That's all, the new moodle_list formatter is doing its job by providing a rerun compatible file.

      Note: @core_course has been selected for the tests because it includes both normal scenarios and outlines+examples. The formatter must support both.

      Note2: Failed steps (real or random) are not related with this issue. Here we are only interested on getting the counts and verify all them are ok and matching as specified above.

      Show
      1) Init the behat installation 2) run --tags='@core_course' (with Firefox) 3) Note the number of scenarios. 4) run the same tag than 2) adding --dry-run --format=moodle_list > list.txt 5) verify that the list.txt file contains the same number of scenarios than 3) 6) run behat with --rerun="list.txt" 7) verify it only executes the scenarios in that file. And the number matches 5) 8 Perform a complete behat run with --dry-run --format=moodle_list > list.txt 9) Verify that list.txt contains (at least) 742 scenarios (as of January 15th 2015, for master). That's all, the new moodle_list formatter is doing its job by providing a rerun compatible file. Note: @core_course has been selected for the tests because it includes both normal scenarios and outlines+examples. The formatter must support both. Note2: Failed steps (real or random) are not related with this issue. Here we are only interested on getting the counts and verify all them are ok and matching as specified above.

      Reading here and there about behat and its internals I found the --rerun feature really interesting to keep apart random failures and also as a potential way to split/dispatch behat runs over multiple testing instances.

      But there is a missing piece in order to be able to use it properly and it is that we need to be able to build such "rerun" files easily, so they can be used later.

      Digging into behat, it seems that the best way to achieve that properly is to create a new formatter (behat already uses that internally for "rerun") and combine it with --dry-run, so we can get the "rerun" file ready for any use.

      So this issue aims to:

      1) Create a new formatter, available via moodle-behat-extension.
      2) Make moodle aware of it, so can be used easily.

      Note: I'm only providing master (2.9) patches for this new formatter, but I'd vote to backport it to all supported versions coz it's 100% isolated and can bring us interesting possibilities to be applied @ nightly and CIs.

            stronk7 Eloy Lafuente (stronk7)
            stronk7 Eloy Lafuente (stronk7)
            Rajesh Taneja Rajesh Taneja
            Andrew Lyons Andrew Lyons
            Jetha Chan Jetha Chan
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.