Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-46514

Behat does not recognise the fatal error

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      NOTE: tail php error and ensure behat should fail if any error occur. No error should be caught.

      Test 1: Existing failure:

      1. Remove last commit (exclude known exceptions)
      2. Run mod/quiz/tests/behat/editing_section_headings.feature
        • It should fail.
        • Pass with the last commit applied.

      Test 2: Test execption in AJAX call is caught.

      1. Apply following patch:

        --- a/user/lib.php
        +++ b/user/lib.php
        @@ -512,7 +512,7 @@ function user_get_user_details($user, $course = null, array $userfields = array(
         
             // If groups are in use and enforced throughout the course, then make sure we can meet in at least one course level group.
             if (in_array('groups', $userfields) && !empty($course) && $canaccessallgroups) {
        -        $usergroups = groups_get_all_groups($course->id, $user->id, $course->defaultgroupingid,
        +        $usergroups = groups_get_all_groups($course->id, $user->id, $course->defaultgroupingid1,
                         'g.id, g.name,g.description,g.descriptionformat');
                 $userdetails['groups'] = array();
                 foreach ($usergroups as $group) {
        

      2. Run behat with mod/assign/feedback/editpdf/tests/behat/annotate_pdf.feature
      3. Ensure you get the failure, stating Undefined property: stdClass::$defaultgroupingid1

      Test 3: Test server exception is caught.

      1. Apply following (keep the above modification.)

        diff --git a/lib/behat/lib.php b/lib/behat/lib.php
        index c598961..5c7293e 100644
        --- a/lib/behat/lib.php
        +++ b/lib/behat/lib.php
        @@ -142,14 +142,14 @@ function behat_error_handler($errno, $errstr, $errfile, $errline, $errcontext) {
             }
         
             // If ajax script then throw exception, so the calling api catch it and show it on web page.
        -    if (defined('AJAX_SCRIPT')) {
        +    /*if (defined('AJAX_SCRIPT')) {
                 throw new Exception("$errnostr: $errstr in $errfile on line $errline");
        -    } else {
        +    } else {*/
                 // Wrapping the output.
                 echo '<div class="phpdebugmessage" data-rel="phpdebugmessage">' . PHP_EOL;
                 echo "$errnostr: $errstr in $errfile on line $errline" . PHP_EOL;
                 echo '</div>';
        -    }
        +    /*}*/
         
             // Also use the internal error handler so we keep the usual behaviour.
             return false;
        

      2. Run behat with mod/assign/feedback/editpdf/tests/behat/annotate_pdf.feature
      3. Ensure you get the failure, stating Undefined property: stdClass::$defaultgroupingid1

      Test 4: Manual quiz test:

      1. Repeat steps in "Turn shuffling on for a section" scenario and make sure you don't see any error. (mod/quiz/tests/behat/editing_section_headings.feature)

      FYI: Without this fix, above will pass.

      Show
      NOTE: tail php error and ensure behat should fail if any error occur. No error should be caught. Test 1: Existing failure: Remove last commit (exclude known exceptions) Run mod/quiz/tests/behat/editing_section_headings.feature It should fail. Pass with the last commit applied. Test 2: Test execption in AJAX call is caught. Apply following patch: --- a/user/lib.php +++ b/user/lib.php @@ -512,7 +512,7 @@ function user_get_user_details($user, $course = null, array $userfields = array( // If groups are in use and enforced throughout the course, then make sure we can meet in at least one course level group. if (in_array('groups', $userfields) && !empty($course) && $canaccessallgroups) { - $usergroups = groups_get_all_groups($course->id, $user->id, $course->defaultgroupingid, + $usergroups = groups_get_all_groups($course->id, $user->id, $course->defaultgroupingid1, 'g.id, g.name,g.description,g.descriptionformat'); $userdetails['groups'] = array(); foreach ($usergroups as $group) { Run behat with mod/assign/feedback/editpdf/tests/behat/annotate_pdf.feature Ensure you get the failure, stating Undefined property: stdClass::$defaultgroupingid1 Test 3: Test server exception is caught. Apply following (keep the above modification.) diff --git a/lib/behat/lib.php b/lib/behat/lib.php index c598961..5c7293e 100644 --- a/lib/behat/lib.php +++ b/lib/behat/lib.php @@ -142,14 +142,14 @@ function behat_error_handler($errno, $errstr, $errfile, $errline, $errcontext) { } // If ajax script then throw exception, so the calling api catch it and show it on web page. - if (defined('AJAX_SCRIPT')) { + /*if (defined('AJAX_SCRIPT')) { throw new Exception("$errnostr: $errstr in $errfile on line $errline"); - } else { + } else {*/ // Wrapping the output. echo '<div class="phpdebugmessage" data-rel="phpdebugmessage">' . PHP_EOL; echo "$errnostr: $errstr in $errfile on line $errline" . PHP_EOL; echo '</div>'; - } + /*}*/ // Also use the internal error handler so we keep the usual behaviour. return false; Run behat with mod/assign/feedback/editpdf/tests/behat/annotate_pdf.feature Ensure you get the failure, stating Undefined property: stdClass::$defaultgroupingid1 Test 4: Manual quiz test: Repeat steps in "Turn shuffling on for a section" scenario and make sure you don't see any error. (mod/quiz/tests/behat/editing_section_headings.feature) FYI: Without this fix, above will pass.
    • Affected Branches:
      MOODLE_27_STABLE
    • Fixed Branches:
      MOODLE_30_STABLE, MOODLE_31_STABLE
    • Pull Master Branch:
      wip-mdl-46514

      Description

      Example is: code calls the non-existing function or tries to create an instance of non-existing class. PHP displays a fatal error. Behat test eventually fails on the next step when it can't find some text or button or link or whatever.

      The failure message says : Can not find link/button/text. At the same time if you look at the screenshot of the step you can see that the PHP fatal error orange table is displayed with the backtrace. In case of a php warning/debugging message behat prints the backtrace in the console.

      So there are two problems here:
      1. More steps for developer to trace where the error occured.
      2. If the error occurs on the last step of a scenario it is not caught by behat at all!

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                11 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  12/Sep/16