Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-69278

BeforeScenario hook failure leads to skipped test and rerun

    XMLWordPrintable

Details

    • MOODLE_35_STABLE, MOODLE_37_STABLE, MOODLE_38_STABLE, MOODLE_39_STABLE
    • MOODLE_35_STABLE, MOODLE_37_STABLE, MOODLE_38_STABLE, MOODLE_39_STABLE
    • MDL-69278-master
    • Hide

      Note: For each branch you will need to stop and restart the docker image.

      1. Start Selenium Chrome Docker image:

        docker run --shm-size=2G -p 4444:4444 --rm --name sel selenium/standalone-chrome:3.141.59-20200515
        

      2. Unzip the attached 69278.zip file in your moodle document root

        unzip 69278.zip
        

      3. Ensure that your behat is configured such that you can make use of the docker image for testing
        I typically set the behat_wwwroot to http://host.docker.internal/path/to/site
      4. Init behat

        php admin/tool/behat/cli/init.php
        

      5. Run the new behat feature included in the zip file using the command supplied in the init, for example:

        vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml mod/book/tests/behat/test.feature
        

        1. Confirm that all tests passed
      6. Run it again but after the first dot appears run the following command

        docker cp bad/google-chrome-stable sel:/usr/bin/
        

      7. Let the tests finish
        1. Confirm that the scenarios after you copied the script failed
        2. Confirm that none of the scenarios were skipped
      8. Run it again
        1. Confirm that the tests didi not start and exited immediately with a stack trace containing debugging information
      Show
      Note: For each branch you will need to stop and restart the docker image. Start Selenium Chrome Docker image: docker run --shm-size=2G -p 4444:4444 --rm --name sel selenium/standalone-chrome:3.141.59-20200515 Unzip the attached 69278.zip file in your moodle document root unzip 69278.zip Ensure that your behat is configured such that you can make use of the docker image for testing I typically set the behat_wwwroot to http://host.docker.internal/path/to/site Init behat php admin/tool/behat/cli/init.php Run the new behat feature included in the zip file using the command supplied in the init, for example: vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml mod/book/tests/behat/test.feature Confirm that all tests passed Run it again but after the first dot appears run the following command docker cp bad/google-chrome-stable sel:/usr/bin/ Let the tests finish Confirm that the scenarios after you copied the script failed Confirm that none of the scenarios were skipped Run it again Confirm that the tests didi not start and exited immediately with a stack trace containing debugging information
    • 1
    • International 4.0 - Sprint 2

    Description

      When Selenium does fail in the BeforeScenario step it means that the exit code is non-zero, causing a re-run, but it also causes the Scenario to be skipped rather than to fail.

      If no other scenario step fails then the rerun.txt is empty.

      Because the Suite returned non-zero, the Suite is rerun, but the lack of a rerun.txt means that the entire Suite reruns ad not just the failed test.

      You can see that in the following job:

      https://ci.moodle.org/job/S37.06.01%20-%20Behat%20-%20Chrome%20+%20Postgres%20+%20Classic/157/console

      This happens because an Exception in a BeforeScenario hook leads to the Skip rather than Fail.

      To resolve it we can apply the same approach as taken in the BeforeStep setup where a BeforeStep failure stores the exception in a statically defined var, and then the "I look for exceptions" step finds it and fails the test.
      This may require an additional change to the behat extension to look for stored exceptions before the step runs.

      Attachments

        Issue Links

          Activity

            People

              dobedobedoh Andrew Lyons
              dobedobedoh Andrew Lyons
              Simey Lameze Simey Lameze
              Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
              Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                14/Sep/20

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 4 hours, 45 minutes
                  4h 45m