Behat does not recognise the fatal error

NOTE: tail php error and ensure behat should fail if any error occur. No error should be caught.

Test 1: Existing failure:

1. Remove last commit (exclude known exceptions)
2. Run mod/quiz/tests/behat/editing_section_headings.feature
   • It should fail.
   • Pass with the last commit applied.

Test 2: Test execption in AJAX call is caught.

1. Apply following patch:

   --- a/user/lib.php

   +++ b/user/lib.php

   @@ -512,7 +512,7 @@ function user_get_user_details($user, $course = null, array $userfields = array(

   // If groups are in use and enforced throughout the course, then make sure we can meet in at least one course level group.

   if (in_array('groups', $userfields) && !empty($course) && $canaccessallgroups) {

   - $usergroups = groups_get_all_groups($course->id, $user->id, $course->defaultgroupingid,

   + $usergroups = groups_get_all_groups($course->id, $user->id, $course->defaultgroupingid1,

   'g.id, g.name,g.description,g.descriptionformat');

   $userdetails['groups'] = array();

   foreach ($usergroups as $group) {

2. Run behat with mod/assign/feedback/editpdf/tests/behat/annotate_pdf.feature
3. Ensure you get the failure, stating Undefined property: stdClass::$defaultgroupingid1

Test 3: Test server exception is caught.

1. Apply following (keep the above modification.)

   diff --git a/lib/behat/lib.php b/lib/behat/lib.php

   index c598961..5c7293e 100644

   --- a/lib/behat/lib.php

   +++ b/lib/behat/lib.php

   @@ -142,14 +142,14 @@ function behat_error_handler($errno, $errstr, $errfile, $errline, $errcontext) {

   }

   // If ajax script then throw exception, so the calling api catch it and show it on web page.

   - if (defined('AJAX_SCRIPT')) {

   + /*if (defined('AJAX_SCRIPT')) {

   throw new Exception("$errnostr: $errstr in $errfile on line $errline");

   - } else {

   + } else {*/

   // Wrapping the output.

   echo '<div class="phpdebugmessage" data-rel="phpdebugmessage">' . PHP_EOL;

   echo "$errnostr: $errstr in $errfile on line $errline" . PHP_EOL;

   echo '</div>';

   - }

   + /*}*/

   // Also use the internal error handler so we keep the usual behaviour.

   return false;

2. Run behat with mod/assign/feedback/editpdf/tests/behat/annotate_pdf.feature
3. Ensure you get the failure, stating Undefined property: stdClass::$defaultgroupingid1

Test 4: Manual quiz test:

1. Repeat steps in "Turn shuffling on for a section" scenario and make sure you don't see any error. (mod/quiz/tests/behat/editing_section_headings.feature)

FYI: Without this fix, above will pass.