Prerequisites Install tool_testtasks: git clone git@github.com:catalyst/moodle-tool_testtasks.git admin/tool/testtasks Modify this patch so tasks get cleaned up quicker - the example below makes it MINSECS (one minute) but you could make it higher (e.g., 5*MINSECS) if you have difficulty quickly switching between terminals and the web browser (which is required in later testing instructions): Open lib/classes/task/manager.php In the cleanup_metadata method, look for: if ($task->timestarted > time() - HOURSECS) { continue; } Modify it to: if ($task->timestarted > time() - MINSECS) { continue; } Testing (normal case) Browse to "Site administration" > "Server" > "Tasks running now" (you'll need to get back to this page quickly after the next steps) Queue up 2 1000 second tasks: php admin/tool/testtasks/cli/queue_multiple_adhoc_tasks.php -n=2 -c="tool_testtasks\task\one_thousand_second_task" In two terminals, start processing the two tasks (one in each terminal): php admin/cli/adhoc_task.php -e Kill one of the adhoc task runners in one of the terminals by pressing ctrl+z followed by the command: kill -9 %1 Start a slow scheduled task in another terminal: php admin/cli/scheduled_task.php --execute=\tool_testtasks\\task\\slow_task Refresh the "Tasks running now page" and verify you see two 1000 second tasks processing, and a slow scheduled task Run the cleanup task: php admin/cli/scheduled_task.php --execute=\\core\\task\\task_lock_cleanup_task Verify both adhoc tasks and a slow task (for a total of 3 rows) are still shown on the "Tasks running now" page (you'll have to be quick and check this before the slow task completes) Once the slow task has completed, refresh the "Tasks running now" page and Verify the two 1000 second tasks are still there Wait for 5 minutes (keep checking the "Tasks running now page" until all tasks have been running for more than 5 minutes - also verify that the slow task goes away once it completes) Run the cleanup task again: php admin/cli/scheduled_task.php --execute=\\core\\task\\task_lock_cleanup_task Verify only one 1000 second task is left on the "Tasks running now" page Check the mdl_task_adhoc table and verify there are two records there, and verify that one of them has a faildelay set Start another slow task: php admin/cli/scheduled_task.php --execute=\tool_testtasks\\task\\slow_task Kill it by pressing ctrl+z followed by the command: kill -9 %1 Check the "Tasks running now" page and verify you see the 1000 second task and the slow task Run the cleanup task: php admin/cli/scheduled_task.php --execute=\\core\\task\\task_lock_cleanup_task Check the "Tasks running now" page and verify both tasks are still there Keep refreshing the "Tasks running now" page until it shows the slow task has been running for more than 5 minutes Run the cleanup task: php admin/cli/scheduled_task.php --execute=\\core\\task\\task_lock_cleanup_task Check the "Tasks running now" page and verify only the 1000 second task remains Testing (race condition) There is a chance of a race condition. To test this a testing commit is provided that exposes the conditions required to race more easily. While the cleanup task is processing running tasks, it is possible for the data to become stale (if a task finishes while the cleanup task is in the process of cleaning another task). The testing commit adds a sleep in the main loop of the cleanup task, making it easy for a task to finish while processing is happening. Revert the changes applied in the prerequisites step and apply attached patch (it also adds some extra logging and reduces the time required for a task to be considered for cleanup) git apply /PATH/TOO/PATCH/MDL- 73293 .patch Stop any tasks that are still running (use ctrl+c ) and cleanup the queue: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php Queue up two 100 second tasks: php admin/tool/testtasks/cli/queue_multiple_adhoc_tasks.php -n=2 -c="tool_testtasks\task\one_hundred_second_task" In two terminals, start processing the two tasks (one in each terminal): php admin/cli/adhoc_task.php -e Wait for 5 seconds and then start the cleanup task: php admin/cli/scheduled_task.php --execute=\\core\\task\\task_lock_cleanup_task Watch the logging messages carefully and ensure both adhoc tasks finish before the cleanup task Verify the output of the cleanup tasks show that an adhoc task completed and there's no need to do anything Queue up a single 100 second task and start processing it: php admin/tool/testtasks/cli/queue_multiple_adhoc_tasks.php -n=1 -c="tool_testtasks\task\one_hundred_second_task" php admin/cli/adhoc_task.php -e Start a slow scheduled task: php admin/cli/scheduled_task.php --execute=\tool_testtasks\\task\\slow_task Wait 5 seconds and run the cleanup task: php admin/cli/scheduled_task.php --execute=\\core\\task\\task_lock_cleanup_task Verify the output of the cleanup tasks show that a scheduled task completed and there's no need to do anything Start a slow scheduled task: php admin/cli/scheduled_task.php --execute=\tool_testtasks\\task\\slow_task Queue up a single 100 second task and start processing it: php admin/tool/testtasks/cli/queue_multiple_adhoc_tasks.php -n=1 -c="tool_testtasks\task\one_hundred_second_task" php admin/cli/adhoc_task.php -e Wait for 5 seconds and then start the cleanup task: php admin/cli/scheduled_task.php --execute=\\core\\task\\task_lock_cleanup_task Verify the output of the cleanup tasks show that an adhoc task completed and there's no need to do anything