Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-69278

BeforeScenario hook failure leads to skipped test and rerun

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      Note: For each branch you will need to stop and restart the docker image.

      1. Start Selenium Chrome Docker image:

        docker run --shm-size=2G -p 4444:4444 --rm --name sel selenium/standalone-chrome:3.141.59-20200515
        

      2. Unzip the attached 69278.zip file in your moodle document root

        unzip 69278.zip
        

      3. Ensure that your behat is configured such that you can make use of the docker image for testing
        I typically set the behat_wwwroot to http://host.docker.internal/path/to/site
      4. Init behat

        php admin/tool/behat/cli/init.php
        

      5. Run the new behat feature included in the zip file using the command supplied in the init, for example:

        vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml mod/book/tests/behat/test.feature
        

        1. Confirm that all tests passed
      6. Run it again but after the first dot appears run the following command

        docker cp bad/google-chrome-stable sel:/usr/bin/
        

      7. Let the tests finish
        1. Confirm that the scenarios after you copied the script failed
        2. Confirm that none of the scenarios were skipped
      8. Run it again
        1. Confirm that the tests didi not start and exited immediately with a stack trace containing debugging information
      Show
      Note: For each branch you will need to stop and restart the docker image. Start Selenium Chrome Docker image: docker run --shm-size=2G -p 4444:4444 --rm --name sel selenium/standalone-chrome:3.141.59-20200515 Unzip the attached 69278.zip file in your moodle document root unzip 69278.zip Ensure that your behat is configured such that you can make use of the docker image for testing I typically set the behat_wwwroot to http://host.docker.internal/path/to/site Init behat php admin/tool/behat/cli/init.php Run the new behat feature included in the zip file using the command supplied in the init, for example: vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml mod/book/tests/behat/test.feature Confirm that all tests passed Run it again but after the first dot appears run the following command docker cp bad/google-chrome-stable sel:/usr/bin/ Let the tests finish Confirm that the scenarios after you copied the script failed Confirm that none of the scenarios were skipped Run it again Confirm that the tests didi not start and exited immediately with a stack trace containing debugging information
    • Affected Branches:
      MOODLE_35_STABLE, MOODLE_37_STABLE, MOODLE_38_STABLE, MOODLE_39_STABLE
    • Fixed Branches:
      MOODLE_35_STABLE, MOODLE_37_STABLE, MOODLE_38_STABLE, MOODLE_39_STABLE
    • Pull 3.5 Branch:
    • Pull 3.8 Branch:
    • Pull 3.9 Branch:
    • Pull Master Branch:
      MDL-69278-master
    • Story Points:
      1
    • Sprint:
      International 4.0 - Sprint 2

      Description

      When Selenium does fail in the BeforeScenario step it means that the exit code is non-zero, causing a re-run, but it also causes the Scenario to be skipped rather than to fail.

      If no other scenario step fails then the rerun.txt is empty.

      Because the Suite returned non-zero, the Suite is rerun, but the lack of a rerun.txt means that the entire Suite reruns ad not just the failed test.

      You can see that in the following job:

      https://ci.moodle.org/job/S37.06.01%20-%20Behat%20-%20Chrome%20+%20Postgres%20+%20Classic/157/console

      This happens because an Exception in a BeforeScenario hook leads to the Skip rather than Fail.

      To resolve it we can apply the same approach as taken in the BeforeStep setup where a BeforeStep failure stores the exception in a statically defined var, and then the "I look for exceptions" step finds it and fails the test.
      This may require an additional change to the behat extension to look for stored exceptions before the step runs.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              dobedobedoh Andrew Nicols
              Reporter:
              dobedobedoh Andrew Nicols
              Peer reviewer:
              Simey Lameze
              Integrator:
              Eloy Lafuente (stronk7)
              Tester:
              Eloy Lafuente (stronk7)
              Participants:
              Component watchers:
              Andrew Nicols, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Fix Release Date:
                14/Sep/20

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 4 hours, 45 minutes
                  4h 45m