Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-69075

Add more information to behat failures during run

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      CI is not sufficient to test this change on it's own

      1. Place the attached 69075.feature file into lib/tests/behat
      2. Initialise behat: 

        php admin/tool/behat/cli/ini.php

      3. Edit config.php in your preferred editor
      4. Ensure that the following is set (you may need to remove any existing use of the same parameter): 

        $CFG->behat_pause_on_fail = true;

      5. Copy the command that was output from the previous step and run it with the above feature file, for example: 

        vendor/bin/behat --config=/Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml lib/tests/behat/69075.feature

        1. Confirm that the output looks a bit like this: 

          Moodle 4.0dev (Build: 20200615), c1f2209b4f3cf9f459ae5ab6716cdc9a25de04dc
          		 
          Php: 7.2.30, pgsql: 9.6.17, OS: Darwin 19.5.0 x86_64
          		 
          Server OS "Darwin", Browser: "chrome"
          		 
          Browser specific fixes have been applied. See http://docs.moodle.org/dev/Acceptance_testing#Browser_specific_fixes
          		 
          Started at 17-06-2020, 10:04
          		 
          .......
          		 
          Scenario failed at line /Users/nicols/Sites/moodles/sm/moodle/lib/tests/behat/69075.feature:14
          		 
          Paused for inspection. Press Enter/Return to continue.
          		 
          Failure detail follows:
          		 
          "Course 1" text was not found in the page

        2. Confirm that the test paused on line 14
        3. Confirm that the full path to the feature file, including line number was shown
      6. Press [enter]
        1. Confirm that the test stops again
        2. Confirm that the test paused on line 20
      7. Press [enter]
        1. Confirm that the test ended okay
        2. Confirm that 3 scenarios were tested
        3. Confirm that 1 scenarios passed
        4. Confirm that 1 scenarios failed
      8. Edit config.php in your preferred editor
      9. Ensure that the following is set (you may need to remove any existing use of the same parameter): 

        $CFG->behat_pause_on_fail = false;

      10. Copy the command that was output from the previous step and run it with the above feature file, for example: 

        vendor/bin/behat --config=/Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml lib/tests/behat/69075.feature

        1. Confirm that the entire test completed
        2. Confirm that the output looks a bit like this: 

          Moodle 4.0dev (Build: 20200615), c1f2209b4f3cf9f459ae5ab6716cdc9a25de04dc
          		 
          Php: 7.2.30, pgsql: 9.6.17, OS: Darwin 19.5.0 x86_64
          		 
          Server OS "Darwin", Browser: "chrome"
          		 
          Browser specific fixes have been applied. See http://docs.moodle.org/dev/Acceptance_testing#Browser_specific_fixes
          		 
          Started at 17-06-2020, 12:04
          		 
          .......
          		 
          Scenario failed at line /Users/nicols/Sites/moodles/sm/moodle/lib/tests/behat/69075.feature:14
          		 
          Failure detail follows:
          		 
          "Course 1" text was not found in the page
          		 
                 F...
          		 
          Scenario failed at line /Users/nicols/Sites/moodles/sm/moodle/lib/tests/behat/69075.feature:20
          		 
          Failure detail follows:
          		 
          "Course 1" text was not found in the page
          		 
                     F
          		 
           
          		 
          --- Failed steps:
          		 
           
          		 
          001 Scenario: Test that fails 1   # /Users/nicols/Sites/moodles/sm/moodle/lib/tests/behat/69075.feature:10
          		 
                But I should see "Course 1" # /Users/nicols/Sites/moodles/sm/moodle/lib/tests/behat/69075.feature:14
          		 
                  "Course 1" text was not found in the page (Behat\Mink\Exception\ExpectationException)
          		 
           
          		 
          002 Scenario: Test that fails 2   # /Users/nicols/Sites/moodles/sm/moodle/lib/tests/behat/69075.feature:16
          		 
                But I should see "Course 1" # /Users/nicols/Sites/moodles/sm/moodle/lib/tests/behat/69075.feature:20
          		 
                  "Course 1" text was not found in the page (Behat\Mink\Exception\ExpectationException)
          		 
           
          		 
          3 scenarios (1 passed, 2 failed)
          		 
          12 steps (10 passed, 2 failed)
          		 
          0m12.39s (49.92Mb)

        3. Confirm that 3 scenarios were tested
        4. Confirm that 1 scenarios passed
        5. Confirm that 1 scenarios failed
      Show
      CI is not sufficient to test this change on it's own Place the attached 69075.feature file into lib/tests/behat Initialise behat: php admin/tool/behat/cli/ini.php Edit config.php in your preferred editor Ensure that the following is set (you may need to remove any existing use of the same parameter): $CFG->behat_pause_on_fail = true; Copy the command that was output from the previous step and run it with the above feature file, for example: vendor/bin/behat --config=/Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml lib/tests/behat/69075.feature Confirm that the output looks a bit like this: Moodle 4.0dev (Build: 20200615), c1f2209b4f3cf9f459ae5ab6716cdc9a25de04dc Php: 7.2.30, pgsql: 9.6.17, OS: Darwin 19.5.0 x86_64 Server OS "Darwin", Browser: "chrome" Browser specific fixes have been applied. See http://docs.moodle.org/dev/Acceptance_testing#Browser_specific_fixes Started at 17-06-2020, 10:04 ....... Scenario failed at line /Users/nicols/Sites/moodles/sm/moodle/lib/tests/behat/69075.feature:14 Paused for inspection. Press Enter/Return to continue. Failure detail follows: "Course 1" text was not found in the page Confirm that the test paused on line 14 Confirm that the full path to the feature file, including line number was shown Press [enter] Confirm that the test stops again Confirm that the test paused on line 20 Press [enter] Confirm that the test ended okay Confirm that 3 scenarios were tested Confirm that 1 scenarios passed Confirm that 1 scenarios failed Edit config.php in your preferred editor Ensure that the following is set (you may need to remove any existing use of the same parameter): $CFG->behat_pause_on_fail = false; Copy the command that was output from the previous step and run it with the above feature file, for example: vendor/bin/behat --config=/Users/nicols/Sites/moodles/sm/moodledata_behat/behatrun/behat/behat.yml lib/tests/behat/69075.feature Confirm that the entire test completed Confirm that the output looks a bit like this: Moodle 4.0dev (Build: 20200615), c1f2209b4f3cf9f459ae5ab6716cdc9a25de04dc Php: 7.2.30, pgsql: 9.6.17, OS: Darwin 19.5.0 x86_64 Server OS "Darwin", Browser: "chrome" Browser specific fixes have been applied. See http://docs.moodle.org/dev/Acceptance_testing#Browser_specific_fixes Started at 17-06-2020, 12:04 ....... Scenario failed at line /Users/nicols/Sites/moodles/sm/moodle/lib/tests/behat/69075.feature:14 Failure detail follows: "Course 1" text was not found in the page F... Scenario failed at line /Users/nicols/Sites/moodles/sm/moodle/lib/tests/behat/69075.feature:20 Failure detail follows: "Course 1" text was not found in the page F   --- Failed steps:   001 Scenario: Test that fails 1 # /Users/nicols/Sites/moodles/sm/moodle/lib/tests/behat/69075.feature:10 But I should see "Course 1" # /Users/nicols/Sites/moodles/sm/moodle/lib/tests/behat/69075.feature:14 "Course 1" text was not found in the page (Behat\Mink\Exception\ExpectationException)   002 Scenario: Test that fails 2 # /Users/nicols/Sites/moodles/sm/moodle/lib/tests/behat/69075.feature:16 But I should see "Course 1" # /Users/nicols/Sites/moodles/sm/moodle/lib/tests/behat/69075.feature:20 "Course 1" text was not found in the page (Behat\Mink\Exception\ExpectationException)   3 scenarios (1 passed, 2 failed) 12 steps (10 passed, 2 failed) 0m12.39s (49.92Mb) Confirm that 3 scenarios were tested Confirm that 1 scenarios passed Confirm that 1 scenarios failed
    • Affected Branches:
      MOODLE_40_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      MDL-69075-master
    • Pull Master Diff URL:
    • Sprint:
      International 4.0 - Sprint 1

      Description

      At the moment we print out some information when a failure occurs, but not very much.

      If $CFG->behat_pause_on_fail is set then we pause the test and print out a bit more information, but we don't actually print out the full detail until the very end of the test.

      There are often times where it is helpful to have more information while a test is running, in particular when a job is running on CI if a test fails then the only ways to find out what has failed are:
      a) wait until the end and see it printed out; or
      b) click on workspaces, navigate to the node -> directory structure, find the pretty.txt file and find the fail in there, or find the screenshot and work out what went wrong.

      It should be very easy to print out a bit more useful information as we run.

        Attachments

          Activity

            People

            Assignee:
            dobedobedoh Andrew Nicols
            Reporter:
            dobedobedoh Andrew Nicols
            Participants:
            Component watchers:
            Andrew Nicols, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 1 hour, 1 minute
                1h 1m