Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-67334

Behat not correctly checking column values with Selenium

    XMLWordPrintable

Details

    • MOODLE_38_STABLE

    Description

      It appears that when testing for values in a table when using Selenium, identifying columns by their title string rather than column number does not restrict the check to the column specified. The result is that it is possible to have a false positive result, if any other column on that row contains the value in the test.

      Proof of concept:

      1. Add the attached MDL_67334_poc.feature file to the /mod/forum/report/summary/tests/behat/ directory.
      2. Re-initialise your site's behat.
      3. Modify your /lib/tests/behat/behat_general.php file's "row_column_of_table_should_contain" method, so that the TESTCODE section below is inserted. This will print the HTML node that the test is matching against:

            // Looks for the requested node inside the container node.
            $coumnnode = $this->getSession()->getDriver()->find($columnvaluexpath);
         
            /////START TESTCODE/////
            error_log("Node HTML for value '{$value}':");
            foreach($coumnnode as $thing) {
                error_log(print_r($thing->getOuterHtml(), true));
            }
            /////END TESTCODE/////
         
            if (empty($coumnnode)) {
                $locatorexceptionmsg = $value . '" in "' . $row . '" row with column "' . $column;
                throw new ElementNotFoundException($this->getSession(), "\n$columnvaluexpath\n\n".'Column value', null, $locatorexceptionmsg);
            }
        

      4. Run the behat test from step 1, using the following command (update the path your behat config):

        vendor/bin/behat --config /path/to/your/moodles/stable_master/moodledata_behat/behatrun/behat/behat.yml --tags="@MDL67334test1" --stop-on-failure
        

      5. Observe that the test passes. It should have failed, as Teacher 1 only has 2 replies posted, but the test checks for 3.
      6. Look at the output from the behat test, and observe that the test is matching each of the table values against all columns in the row that contain the specified value, not only the one it should be checking.
      7. Re-run behat, changing the --tags to test 2, ie "@MDL67334test2". This is the same test, but run without the javascript tag (ie running via Goutte instead of Selenium).
      8. Observe that the test fails, and also observe that the test is only matching the values against a single node (the correct behaviour).
      9. Re-run behat, changing the --tags to test 3, ie "@MDL67334test3". This replaces the text column headings with their column numbers.
      10. Observe that this is also working correctly (failing on the last value, the same as seen in step 7). This confirms that the bug only occurs when using column names to identify the column to be tested.
      11. Re-run behat, changing the --tags to test 4, ie "@MDL67334test4". This runs the same test (using column numbers), but with the correct testing values. This confirms the test does work when valid values are used with this method.

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              michaelh Michael Hawkins
              Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 5 hours, 21 minutes
                  5h 21m