Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-65435

vendor/bin/behat reruns for parallel executions config dependency

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      Requirements

      1. Ability to run the testing instructions with and without the patch installed.
      2. Execution of commands via command line.

      With the patch not applied (or reverted)

      1. Init behat for 2 parallel runs

        php admin/tool/behat/cli/init.php --parallel=2

      2. Let's run (with run.php, for parallel) some choice features having some name (no selenium is needed)

        php admin/tool/behat/cli/run.php --tags='@mod_choice&&~@javascript' --name='a choice'
        

      3. Verify that 2 parallel runs happen and both pass (10 scenarios right now).
      4. From the test output, after the Running 2 parallel behat sites, copy the [behatrun1] vendor... and [behatrun2] vendor.... We'll need them later.
      5. Edit mod/choice/tests/behat/change_response.feature and change any of the I should see ... expectations to force a failure (with some text not matching, like, yay!.
      6. Let's run again (with run.php, for parallel) some choice features having some name (no selenium is needed)

        php admin/tool/behat/cli/run.php --tags='@mod_choice&&~@javascript' --name='a choice'
        

      7. Verify that 2 parallel runs happen and one of them fails now. Let's call FAILED to the one failing (1 or 2).
      8. Let's rerun the failed run using run.php, replacing FAILED in the command below by the one that failed:

        php admin/tool/behat/cli/run.php --tags='@mod_choice&&~@javascript' --name='a choice' --fromrun=FAILED --torun=FAILED --rerun
        

      9. Verify that the run continues failing as expected and that only one scenario is executed (and failed).
      10. Now let's rerun the failed one using vendor/bin/behat, replacing CONFIGPATH in the command below with the path to the behat.yml corresponding to the run FAILED that you copied in step #4 above.
        • Note that there is a whitespace between --config and CONFIGPATH!
        • Note that you may need to create a link manually in your moodle root directory, depending of your webserver configuration if the behat wwwroot is not accesible: ln -s . behatrunX.

          vendor/bin/behat --config CONFIGPATH --tags='@mod_choice&&~@javascript' --name='a choice' --rerun
          

      11. Verify that the run continues failing as expected and that only one scenario is executed (and failed).
      12. Now let's rerun the failed one, exactly the same than the previous step but, instead of using a space between --config and CONFIGPATH, we use an equal symbol:

        vendor/bin/behat --config=CONFIGPATH --tags='@mod_choice&&~@javascript' --name='a choice' --rerun
        

      13. Verify that the run is not executed now and instead this error is shown:

        Your behat test site is outdated, please run
         
            php admin/tool/behat/cli/init.php
         
        from your moodle dirroot to drop and install the behat test site again.
        

        (this is the bug that we are fixing here, using "=" with --config is broken)

      With the patch applied

      1. Repeat steps 1-12 from the previous section.
      2. Verify that the execution of step 12, using the equal symbol instead of space, does not lead to the failure in step 13.
      3. Verify that, instead, the run is executed and the scenario fails as expected.
      4. Undo the changes that were performed to mod/choice/tests/behat/change_response.feature (in step 5 from the previous section) so it will be a valid test again.
      5. Run again the command in step 12 from the previous section.
      6. Verify that this time the run ends ok and the scenario passes.

      That's all, now we can use both space and equal with the --config parameter, and the other rerun methods do continue working, thanks for testing!

      Notes

      If at step 13 you instead see the following, you will need to add symlinks to your dirroot:

      2031 im:master> ./vendor/bin/behat --config /Users/nicols/Sites/moodles/im/moodledata_behat/behatrun1/behat/behat.yml --tags='@mod_choice&&~@javascript' --name='a choice' --rerun
      Behat requirement not satisfied: http://127.0.0.1/im/behatrun1 is not available, ensure you specified correct url and that the server is set up and started.
       More info in https://docs.moodle.org/dev/Running_acceptance_test
      

      You need to add the following symlinks:

      ln -s . behatrun1
      ln -s . behatrun2
      

      Show
      Requirements Ability to run the testing instructions with and without the patch installed. Execution of commands via command line. With the patch not applied (or reverted) Init behat for 2 parallel runs php admin/tool/behat/cli/init.php --parallel=2 Let's run (with run.php, for parallel) some choice features having some name (no selenium is needed) php admin/tool/behat/cli/run.php --tags='@mod_choice&&~@javascript' --name='a choice' Verify that 2 parallel runs happen and both pass (10 scenarios right now). From the test output , after the Running 2 parallel behat sites , copy the [behatrun1] vendor... and [behatrun2] vendor... . We'll need them later. Edit mod/choice/tests/behat/change_response.feature and change any of the I should see ... expectations to force a failure (with some text not matching, like, yay! . Let's run again (with run.php, for parallel) some choice features having some name (no selenium is needed) php admin/tool/behat/cli/run.php --tags='@mod_choice&&~@javascript' --name='a choice' Verify that 2 parallel runs happen and one of them fails now. Let's call FAILED to the one failing (1 or 2). Let's rerun the failed run using run.php , replacing FAILED in the command below by the one that failed: php admin/tool/behat/cli/run.php --tags='@mod_choice&&~@javascript' --name='a choice' --fromrun=FAILED --torun=FAILED --rerun Verify that the run continues failing as expected and that only one scenario is executed (and failed). Now let's rerun the failed one using vendor/bin/behat , replacing CONFIGPATH in the command below with the path to the behat.yml corresponding to the run FAILED that you copied in step #4 above. Note that there is a whitespace between --config and CONFIGPATH! Note that you may need to create a link manually in your moodle root directory, depending of your webserver configuration if the behat wwwroot is not accesible: ln -s . behatrunX . vendor/bin/behat --config CONFIGPATH --tags='@mod_choice&&~@javascript' --name='a choice' --rerun Verify that the run continues failing as expected and that only one scenario is executed (and failed). Now let's rerun the failed one, exactly the same than the previous step but, instead of using a space between --config and CONFIGPATH, we use an equal symbol: vendor/bin/behat --config=CONFIGPATH --tags='@mod_choice&&~@javascript' --name='a choice' --rerun Verify that the run is not executed now and instead this error is shown: Your behat test site is outdated, please run   php admin/tool/behat/cli/init.php   from your moodle dirroot to drop and install the behat test site again. (this is the bug that we are fixing here, using "=" with --config is broken) With the patch applied Repeat steps 1-12 from the previous section. Verify that the execution of step 12, using the equal symbol instead of space, does not lead to the failure in step 13. Verify that, instead, the run is executed and the scenario fails as expected. Undo the changes that were performed to mod/choice/tests/behat/change_response.feature (in step 5 from the previous section) so it will be a valid test again. Run again the command in step 12 from the previous section. Verify that this time the run ends ok and the scenario passes. That's all, now we can use both space and equal with the --config parameter, and the other rerun methods do continue working, thanks for testing! Notes If at step 13 you instead see the following, you will need to add symlinks to your dirroot: 2031 im:master> ./vendor/bin/behat --config /Users/nicols/Sites/moodles/im/moodledata_behat/behatrun1/behat/behat.yml --tags='@mod_choice&&~@javascript' --name='a choice' --rerun Behat requirement not satisfied: http://127.0.0.1/im/behatrun1 is not available, ensure you specified correct url and that the server is set up and started. More info in https://docs.moodle.org/dev/Running_acceptance_test You need to add the following symlinks: ln -s . behatrun1 ln -s . behatrun2
    • Affected Branches:
      MOODLE_35_STABLE, MOODLE_36_STABLE, MOODLE_37_STABLE
    • Fixed Branches:
      MOODLE_35_STABLE, MOODLE_36_STABLE
    • Pull from Repository:
    • Pull 3.5 Branch:
    • Pull 3.6 Branch:
    • Pull Master Branch:

      Description

      When a parallel behat is executed using admin/tool/behat/cli/run.php and there are failures, trying to rerun them using vendor/bin/behat fails sometimes.

      Curiously reruns executed via the same admin/tool/behat/cli/run.php work fine. Also reruns when the initial execution was not parallel do their work perfectly.

      Tracing down the problem... there is a well hidden dependency about how the current rerun to perform is decided based in the content of the --config parameter. Basically:

      1. If vendor/bin/behat uses --config path/to/behat.yml, rerun works.
      2. If vendor/bin/behat uses --config=path/to/behat.yml, rerun fails with the weird/miss-leading error:

        Install Behat before enabling it, use:
         php admin/tool/behat/cli/init.php
        

      This issue is about to make reruns to support both 1 & 2 above, so the mysterious dependency is gone and there isn't that confusion anymore.

        Attachments

          Activity

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Fix Release Date:
                13/May/19

                Time Tracking

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 1 day, 30 minutes
                1d 30m