It appears that when testing for values in a table when using Selenium, identifying columns by their title string rather than column number does not restrict the check to the column specified. The result is that it is possible to have a false positive result, if any other column on that row contains the value in the test.
Proof of concept:
- Add the attached MDL_67334_poc.feature file to the /mod/forum/report/summary/tests/behat/ directory.
- Re-initialise your site's behat.
- Modify your /lib/tests/behat/behat_general.php file's "row_column_of_table_should_contain" method, so that the TESTCODE section below is inserted. This will print the HTML node that the test is matching against:
- Run the behat test from step 1, using the following command (update the path your behat config):
- Observe that the test passes. It should have failed, as Teacher 1 only has 2 replies posted, but the test checks for 3.
- Look at the output from the behat test, and observe that the test is matching each of the table values against all columns in the row that contain the specified value, not only the one it should be checking.
- Observe that the test fails, and also observe that the test is only matching the values against a single node (the correct behaviour).
- Re-run behat, changing the --tags to test 3, ie "@MDL67334test3". This replaces the text column headings with their column numbers.
- Observe that this is also working correctly (failing on the last value, the same as seen in step 7). This confirms that the bug only occurs when using column names to identify the column to be tested.
- Re-run behat, changing the --tags to test 4, ie "@MDL67334test4". This runs the same test (using column numbers), but with the correct testing values. This confirms the test does work when valid values are used with this method.