Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-80844

Ensure that only advanced & database driver tests are using the debugging sink for unit tests

    XMLWordPrintable

Details

    • MOODLE_401_STABLE, MOODLE_402_STABLE, MOODLE_403_STABLE, MOODLE_404_STABLE
    • MOODLE_401_STABLE, MOODLE_402_STABLE, MOODLE_403_STABLE
    • MDL-80844_401
    • MDL-80844_402
    • MDL-80844_403
    • Hide

      Regression testing (local).

      1. Copy the attached mdl80844_test.php to the lib/tests directory of Moodle.
      2. Init phpunit (php admin/tool/phpunit/cli/init.php)
      3. Verify that the first line (advanced_testcase) is enabled, and the other two (database_driver_testcase and basic_testcase) are commented.
      4. Run the test (vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?")
      5. Verify that there isn't any debugging along the output (in the middle of the dots).
      6. *Verify that the execution ends with: ok 0 in the last line.
      7. Edit the lib/tests/mdl80844_test.php file, comment the advanced_testcase line and uncomment the database_driver_testcase line.
      8. Run the test (vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?")
      9. Verify that there isn't any debugging along the output (in the middle of the dots).
      10. *Verify that the execution ends with: ok 0 in the last line.
      11. Edit the lib/tests/mdl80844_test.php file, comment the database_driver_testcase line and uncomment the basic_testcase line.
      12. Run the test (vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?")
      13. Verify that there isn't any debugging along the output (in the middle of the dots).
      14. Verify that the execution ends with: ok 0 in the last line.

      Debugging testing (advanced and database_driver testcases)

      1. Edit the lang/en/deprecated.txt and add one new line at the end with contents: "yes,core" (without the quotes).
      2. Verify that the first line (advanced_testcase) is enabled, and the other two (database_driver_testcase and basic_testcase) are commented.
      3. Run the test (vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?")
      4. Verify that there in one error and zero risky tests with message: "Unexpected debugging() call detected" and a complete stack trace.
      5. Verify that the execution ends with: err 2 in the last line.
      6. Edit the lib/tests/mdl80844_test.php file, comment the advanced_testcase line and uncomment the database_driver_testcase line.
      7. Run the test (vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?")
      8. Verify that there in one error and zero risky tests with message: "Unexpected debugging() call detected" and a complete stack trace.
      9. Verify that the execution ends with: err 2 in the last line.

      Debugging testing (basic testcase)

      1. Edit the lib/tests/mdl80844_test.php file, comment the database_driver_testcase line and uncomment the basic_testcase line.
      2. Run the test (vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?")
      3. Verify that there in one risky test and zero errors with message: "This test printed output: ++ String [yes,core] is deprecated..." and a complete stack trace.
      4. Verify that the execution ends with: ok 0 in the last line.
      5. Run the test (vendor/bin/phpunit lib/tests/mdl80844_test.php --fail-on-risky && echo "ok $?" || echo "err $?"
      6. Verify that there in one risky test and zero errors with message: "This test printed output: ++ String [yes,core] is deprecated..." and a complete stack trace.
      7. Verify that the execution ends with: err 1 in the last line.
      Show
      Regression testing (local). Copy the attached mdl80844_test.php to the lib/tests directory of Moodle. Init phpunit ( php admin/tool/phpunit/cli/init.php ) Verify that the first line (advanced_testcase) is enabled, and the other two (database_driver_testcase and basic_testcase) are commented. Run the test ( vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?" ) Verify that there isn't any debugging along the output (in the middle of the dots). *Verify that the execution ends with: ok 0 in the last line. Edit the lib/tests/mdl80844_test.php file, comment the advanced_testcase line and uncomment the database_driver_testcase line. Run the test ( vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?" ) Verify that there isn't any debugging along the output (in the middle of the dots). *Verify that the execution ends with: ok 0 in the last line. Edit the lib/tests/mdl80844_test.php file, comment the database_driver_testcase line and uncomment the basic_testcase line. Run the test ( vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?" ) Verify that there isn't any debugging along the output (in the middle of the dots). Verify that the execution ends with: ok 0 in the last line. Debugging testing (advanced and database_driver testcases) Edit the lang/en/deprecated.txt and add one new line at the end with contents: "yes,core" (without the quotes). Verify that the first line (advanced_testcase) is enabled, and the other two (database_driver_testcase and basic_testcase) are commented. Run the test ( vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?" ) Verify that there in one error and zero risky tests with message: "Unexpected debugging() call detected" and a complete stack trace. Verify that the execution ends with: err 2 in the last line. Edit the lib/tests/mdl80844_test.php file, comment the advanced_testcase line and uncomment the database_driver_testcase line. Run the test ( vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?" ) Verify that there in one error and zero risky tests with message: "Unexpected debugging() call detected" and a complete stack trace. Verify that the execution ends with: err 2 in the last line. Debugging testing (basic testcase) Edit the lib/tests/mdl80844_test.php file, comment the database_driver_testcase line and uncomment the basic_testcase line. Run the test ( vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?" ) Verify that there in one risky test and zero errors with message: "This test printed output: ++ String [yes,core] is deprecated..." and a complete stack trace. Verify that the execution ends with: ok 0 in the last line. Run the test ( vendor/bin/phpunit lib/tests/mdl80844_test.php --fail-on-risky && echo "ok $?" || echo "err $?" Verify that there in one risky test and zero errors with message: "This test printed output: ++ String [yes,core] is deprecated..." and a complete stack trace. Verify that the execution ends with: err 1 in the last line.

    Description

      It has been detected that, sometimes (MDL-80418 with a deprecate string use, to be fixed by MDL-80839), when debugging is being output as part of unit tests, the output is not detected and PHPUnit ends normally, without any risky test or warning allowing us to detect it.

      pholden detected that it only happens when the test is a basic_testcase and, when the test is an advanced_testcase then the PHPUnit execution fails properly (with "Unexpected debugging() call detected" message).

      Tracing down the problem... the is that we are *semi-using* the debugging sink for all tests, when we should use it only for advanced_testcase and database_driver_testcase (the ones supporting the sink). And, later, other tests, that don't support the sink, fail to report the error above.

      So, what we have to do is to ensure that the *semi-using* the debugging sink doesn't happen any more and:

      • advanced_testcase and database_driver_testcase: They both store and report what is found in the debugging sink (current behaviour, no change).
      • basic_testcase: Will stop storing anything in the debugging sink (change) because later it's not reporting (current behaviour).

      That way, any basic_testcase emitting any debugging message just will be detected as a risky tests by PHPUnit itself, because we run tests with beStrictAboutOutputDuringTests enabled. And, once detected by PHPUnit we can decide to fail the run or whatever.

      So, TL;DR; keep non advanced_testcase and non database_driver_testcase tests completely out from the debugging sink, because right now we are semi-using it and that's leading to problems (debugging) to be detected.

      Attachments

        Issue Links

          Activity

            People

              stronk7 Eloy Lafuente (stronk7)
              stronk7 Eloy Lafuente (stronk7)
              Andrew Lyons Andrew Lyons
              Simey Lameze Simey Lameze
              Kim Jared Lucas Kim Jared Lucas
              Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 day, 2 hours, 47 minutes
                  1d 2h 47m

                  Clockify

                    Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.