Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-80844

Ensure that only advanced & database driver tests are using the debugging sink for unit tests

XMLWordPrintable

    • MOODLE_401_STABLE, MOODLE_402_STABLE, MOODLE_403_STABLE, MOODLE_404_STABLE
    • MOODLE_401_STABLE, MOODLE_402_STABLE, MOODLE_403_STABLE
    • MDL-80844_401
    • MDL-80844_402
    • MDL-80844_403
    • Hide

      Regression testing (local).

      1. Copy the attached mdl80844_test.php to the lib/tests directory of Moodle.
      2. Init phpunit (php admin/tool/phpunit/cli/init.php)
      3. Verify that the first line (advanced_testcase) is enabled, and the other two (database_driver_testcase and basic_testcase) are commented.
      4. Run the test (vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?")
      5. Verify that there isn't any debugging along the output (in the middle of the dots).
      6. *Verify that the execution ends with: ok 0 in the last line.
      7. Edit the lib/tests/mdl80844_test.php file, comment the advanced_testcase line and uncomment the database_driver_testcase line.
      8. Run the test (vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?")
      9. Verify that there isn't any debugging along the output (in the middle of the dots).
      10. *Verify that the execution ends with: ok 0 in the last line.
      11. Edit the lib/tests/mdl80844_test.php file, comment the database_driver_testcase line and uncomment the basic_testcase line.
      12. Run the test (vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?")
      13. Verify that there isn't any debugging along the output (in the middle of the dots).
      14. Verify that the execution ends with: ok 0 in the last line.

      Debugging testing (advanced and database_driver testcases)

      1. Edit the lang/en/deprecated.txt and add one new line at the end with contents: "yes,core" (without the quotes).
      2. Verify that the first line (advanced_testcase) is enabled, and the other two (database_driver_testcase and basic_testcase) are commented.
      3. Run the test (vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?")
      4. Verify that there in one error and zero risky tests with message: "Unexpected debugging() call detected" and a complete stack trace.
      5. Verify that the execution ends with: err 2 in the last line.
      6. Edit the lib/tests/mdl80844_test.php file, comment the advanced_testcase line and uncomment the database_driver_testcase line.
      7. Run the test (vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?")
      8. Verify that there in one error and zero risky tests with message: "Unexpected debugging() call detected" and a complete stack trace.
      9. Verify that the execution ends with: err 2 in the last line.

      Debugging testing (basic testcase)

      1. Edit the lib/tests/mdl80844_test.php file, comment the database_driver_testcase line and uncomment the basic_testcase line.
      2. Run the test (vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?")
      3. Verify that there in one risky test and zero errors with message: "This test printed output: ++ String [yes,core] is deprecated..." and a complete stack trace.
      4. Verify that the execution ends with: ok 0 in the last line.
      5. Run the test (vendor/bin/phpunit lib/tests/mdl80844_test.php --fail-on-risky && echo "ok $?" || echo "err $?"
      6. Verify that there in one risky test and zero errors with message: "This test printed output: ++ String [yes,core] is deprecated..." and a complete stack trace.
      7. Verify that the execution ends with: err 1 in the last line.
      Show
      Regression testing (local). Copy the attached mdl80844_test.php to the lib/tests directory of Moodle. Init phpunit ( php admin/tool/phpunit/cli/init.php ) Verify that the first line (advanced_testcase) is enabled, and the other two (database_driver_testcase and basic_testcase) are commented. Run the test ( vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?" ) Verify that there isn't any debugging along the output (in the middle of the dots). *Verify that the execution ends with: ok 0 in the last line. Edit the lib/tests/mdl80844_test.php file, comment the advanced_testcase line and uncomment the database_driver_testcase line. Run the test ( vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?" ) Verify that there isn't any debugging along the output (in the middle of the dots). *Verify that the execution ends with: ok 0 in the last line. Edit the lib/tests/mdl80844_test.php file, comment the database_driver_testcase line and uncomment the basic_testcase line. Run the test ( vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?" ) Verify that there isn't any debugging along the output (in the middle of the dots). Verify that the execution ends with: ok 0 in the last line. Debugging testing (advanced and database_driver testcases) Edit the lang/en/deprecated.txt and add one new line at the end with contents: "yes,core" (without the quotes). Verify that the first line (advanced_testcase) is enabled, and the other two (database_driver_testcase and basic_testcase) are commented. Run the test ( vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?" ) Verify that there in one error and zero risky tests with message: "Unexpected debugging() call detected" and a complete stack trace. Verify that the execution ends with: err 2 in the last line. Edit the lib/tests/mdl80844_test.php file, comment the advanced_testcase line and uncomment the database_driver_testcase line. Run the test ( vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?" ) Verify that there in one error and zero risky tests with message: "Unexpected debugging() call detected" and a complete stack trace. Verify that the execution ends with: err 2 in the last line. Debugging testing (basic testcase) Edit the lib/tests/mdl80844_test.php file, comment the database_driver_testcase line and uncomment the basic_testcase line. Run the test ( vendor/bin/phpunit lib/tests/mdl80844_test.php && echo "ok $?" || echo "err $?" ) Verify that there in one risky test and zero errors with message: "This test printed output: ++ String [yes,core] is deprecated..." and a complete stack trace. Verify that the execution ends with: ok 0 in the last line. Run the test ( vendor/bin/phpunit lib/tests/mdl80844_test.php --fail-on-risky && echo "ok $?" || echo "err $?" Verify that there in one risky test and zero errors with message: "This test printed output: ++ String [yes,core] is deprecated..." and a complete stack trace. Verify that the execution ends with: err 1 in the last line.

      It has been detected that, sometimes (MDL-80418 with a deprecate string use, to be fixed by MDL-80839), when debugging is being output as part of unit tests, the output is not detected and PHPUnit ends normally, without any risky test or warning allowing us to detect it.

      pholden detected that it only happens when the test is a basic_testcase and, when the test is an advanced_testcase then the PHPUnit execution fails properly (with "Unexpected debugging() call detected" message).

      Tracing down the problem... the is that we are *semi-using* the debugging sink for all tests, when we should use it only for advanced_testcase and database_driver_testcase (the ones supporting the sink). And, later, other tests, that don't support the sink, fail to report the error above.

      So, what we have to do is to ensure that the *semi-using* the debugging sink doesn't happen any more and:

      • advanced_testcase and database_driver_testcase: They both store and report what is found in the debugging sink (current behaviour, no change).
      • basic_testcase: Will stop storing anything in the debugging sink (change) because later it's not reporting (current behaviour).

      That way, any basic_testcase emitting any debugging message just will be detected as a risky tests by PHPUnit itself, because we run tests with beStrictAboutOutputDuringTests enabled. And, once detected by PHPUnit we can decide to fail the run or whatever.

      So, TL;DR; keep non advanced_testcase and non database_driver_testcase tests completely out from the debugging sink, because right now we are semi-using it and that's leading to problems (debugging) to be detected.

        1. (1) 12 Passed -- (Main)MDL-80844.png
          (1) 12 Passed -- (Main)MDL-80844.png
          60 kB
        2. (1) 3 Passed -- (Main)MDL-80844.png
          (1) 3 Passed -- (Main)MDL-80844.png
          34 kB
        3. (1) 5 Passed -- (Main)MDL-80844.png
          (1) 5 Passed -- (Main)MDL-80844.png
          45 kB
        4. (1) 9 Passed -- (Main)MDL-80844.png
          (1) 9 Passed -- (Main)MDL-80844.png
          56 kB
        5. (2) 2 Passed -- (Main)MDL-80844.png
          (2) 2 Passed -- (Main)MDL-80844.png
          227 kB
        6. (2) 4 Passed -- (Main)MDL-80844.png
          (2) 4 Passed -- (Main)MDL-80844.png
          149 kB
        7. (2) 8 Passed -- (Main)MDL-80844.png
          (2) 8 Passed -- (Main)MDL-80844.png
          144 kB
        8. (3) 3 Passed -- (Main)MDL-80844.png
          (3) 3 Passed -- (Main)MDL-80844.png
          236 kB
        9. (3) 6 Passed -- (Main)MDL-80844.png
          (3) 6 Passed -- (Main)MDL-80844.png
          232 kB
        10. mdl80844_test.php
          0.3 kB

            stronk7 Eloy Lafuente (stronk7)
            stronk7 Eloy Lafuente (stronk7)
            Andrew Lyons Andrew Lyons
            Simey Lameze Simey Lameze
            Kim Jared Lucas Kim Jared Lucas
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 1 day, 2 hours, 47 minutes
                1d 2h 47m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.