Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-69278

BeforeScenario hook failure leads to skipped test and rerun

XMLWordPrintable

    • MOODLE_35_STABLE, MOODLE_37_STABLE, MOODLE_38_STABLE, MOODLE_39_STABLE
    • MOODLE_35_STABLE, MOODLE_37_STABLE, MOODLE_38_STABLE, MOODLE_39_STABLE
    • MDL-69278-master
    • Hide

      Note: For each branch you will need to stop and restart the docker image.

      1. Start Selenium Chrome Docker image:

        docker run --shm-size=2G -p 4444:4444 --rm --name sel selenium/standalone-chrome:3.141.59-20200515
        

      2. Unzip the attached 69278.zip file in your moodle document root

        unzip 69278.zip
        

      3. Ensure that your behat is configured such that you can make use of the docker image for testing
        I typically set the behat_wwwroot to http://host.docker.internal/path/to/site
      4. Init behat

        php admin/tool/behat/cli/init.php
        

      5. Run the new behat feature included in the zip file using the command supplied in the init, for example:

        vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml mod/book/tests/behat/test.feature
        

        1. Confirm that all tests passed
      6. Run it again but after the first dot appears run the following command

        docker cp bad/google-chrome-stable sel:/usr/bin/
        

      7. Let the tests finish
        1. Confirm that the scenarios after you copied the script failed
        2. Confirm that none of the scenarios were skipped
      8. Run it again
        1. Confirm that the tests didi not start and exited immediately with a stack trace containing debugging information
      Show
      Note: For each branch you will need to stop and restart the docker image. Start Selenium Chrome Docker image: docker run --shm-size=2G -p 4444:4444 --rm --name sel selenium/standalone-chrome:3.141.59-20200515 Unzip the attached 69278.zip file in your moodle document root unzip 69278.zip Ensure that your behat is configured such that you can make use of the docker image for testing I typically set the behat_wwwroot to http://host.docker.internal/path/to/site Init behat php admin/tool/behat/cli/init.php Run the new behat feature included in the zip file using the command supplied in the init, for example: vendor/bin/behat --config /Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml mod/book/tests/behat/test.feature Confirm that all tests passed Run it again but after the first dot appears run the following command docker cp bad/google-chrome-stable sel:/usr/bin/ Let the tests finish Confirm that the scenarios after you copied the script failed Confirm that none of the scenarios were skipped Run it again Confirm that the tests didi not start and exited immediately with a stack trace containing debugging information
    • 1
    • International 4.0 - Sprint 2

      When Selenium does fail in the BeforeScenario step it means that the exit code is non-zero, causing a re-run, but it also causes the Scenario to be skipped rather than to fail.

      If no other scenario step fails then the rerun.txt is empty.

      Because the Suite returned non-zero, the Suite is rerun, but the lack of a rerun.txt means that the entire Suite reruns ad not just the failed test.

      You can see that in the following job:

      https://ci.moodle.org/job/S37.06.01%20-%20Behat%20-%20Chrome%20+%20Postgres%20+%20Classic/157/console

      This happens because an Exception in a BeforeScenario hook leads to the Skip rather than Fail.

      To resolve it we can apply the same approach as taken in the BeforeStep setup where a BeforeStep failure stores the exception in a statically defined var, and then the "I look for exceptions" step finds it and fails the test.
      This may require an additional change to the behat extension to look for stored exceptions before the step runs.

            dobedobedoh Andrew Lyons
            dobedobedoh Andrew Lyons
            Simey Lameze Simey Lameze
            Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
            Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 4 hours, 45 minutes
                4h 45m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.