Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-72775

Add a new status check for the cron task api to watch very long running tasks

XMLWordPrintable

    • MOODLE_401_STABLE
    • MOODLE_402_STABLE
    • MDL-72775-master
    • Hide

      Preparation install this plugin to help make some test tasks:

      https://github.com/catalyst/moodle-tool_testtasks

      1) Ensure you start with a clean slate:

      php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 

      2) Visit /admin/tool/task/runningtasks.php

      3) Confirm the page is empty

      4) Queue up an adhoc task

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=100000 

      5) Start running this task running but then hard Control-C twice kill the process so it's metadata will be left behind

      php admin/cli/adhoc_task.php -e

      6) Confirm timed_adhoc_task appears in the runningstasks.php table without any warnings or errors

      7) Do the same to queue up a scheduled task and then hard kill it:

      php admin/cli/scheduled_task.php --execute='\tool_testtasks\task\slow_task'

      8) Confirm slow_task appears in the runningstasks.php table without any warnings or errors

      9) Visit status report: /report/status/index.php?detail=tool_task_longrunningtasks in a new tab

      10) Confirm that the check is OK and green

      11) Add $CFG->taskruntimewarn = 10; to config.php

      12) Confirm the status report now shows a warning for the slow tasks

      13) Confirm the runningstasks table also shows the same warnings for the slow tasks

      14) Add $CFG->taskruntimeerror = 20; to config.php

      15) Confirm the status report now shows an error for the slow tasks

      16) Confirm the runningstasks table also shows the same error for the slow tasks

       

      Show
      Preparation install this plugin to help make some test tasks: https://github.com/catalyst/moodle-tool_testtasks 1) Ensure you start with a clean slate: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 2) Visit /admin/tool/task/runningtasks.php 3) Confirm the page is empty 4) Queue up an adhoc task php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 100000 5) Start running this task running but then hard Control-C twice kill the process so it's metadata will be left behind php admin/cli/adhoc_task.php -e 6) Confirm timed_adhoc_task appears in the runningstasks.php table without any warnings or errors 7) Do the same to queue up a scheduled task and then hard kill it: php admin/cli/scheduled_task.php --execute= '\tool_testtasks\task\slow_task' 8) Confirm slow_task appears in the runningstasks.php table without any warnings or errors 9) Visit status report: /report/status/index.php?detail=tool_task_longrunningtasks in a new tab 10) Confirm that the check is OK and green 11) Add $CFG->taskruntimewarn = 10; to config.php 12) Confirm the status report now shows a warning for the slow tasks 13) Confirm the runningstasks table also shows the same warnings for the slow tasks 14) Add $CFG->taskruntimeerror = 20; to config.php 15) Confirm the status report now shows an error for the slow tasks 16) Confirm the runningstasks table also shows the same error for the slow tasks  

      We check that cron is running, and that tasks haven't failed. But a task may still be running forever and not being caught by the status checks. I think a reasonable real world default might be warn at 24 hours and error at 48 hours.

      Now that we have this metadata since MDL-67211 we can properly detect this.

            qihuichan Qihui Chan
            brendanheywood Brendan Heywood
            Brendan Heywood Brendan Heywood
            Sara Arjona (@sarjona) Sara Arjona (@sarjona)
            Kim Jared Lucas Kim Jared Lucas
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 2 hours, 35 minutes
                2h 35m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.