Details
-
Improvement
-
Status: Closed
-
Minor
-
Resolution: Fixed
-
3.9, 3.10
-
MOODLE_310_STABLE, MOODLE_39_STABLE
-
MOODLE_310_STABLE
-
master-
MDL-67211_v6 -
Hide
Installation:
1. Install this to make the testing easier:
git clone git@github.com:catalyst/moodle-tool_testtasks.git admin/tool/testtasks/
2. Clear adhoc task queue:
php admin/tool/testtasks/cli/clear_adhoc_task_queue.php
3. Set "Path to PHP CLI" admin setting according to your environment:
php admin/cli/cfg.php --name=pathtophp --set='/usr/bin/php'
4. Enable "Allow 'Run now' for scheduled tasks" admin setting:
php admin/cli/cfg.php --name=enablerunnow --set='1' --component=tool_task
Testing instructions:
1. Open terminal and run cron:
php admin/cli/cron.php
2. Confirm, that cron runs as normal.
3. Generate an adhoc task:
php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=1 -n=1
4. Run the CLI for adhoc tasks:
php admin/cli/adhoc_task.php --execute
5. Confirm, that output looks like:
root@c69335460f7f:/siteroot# php admin/cli/adhoc_task.php --execute
Server Time: Tue, 14 Apr 2020 16:16:22 +1000
Execute adhoc task: tool_testtasks\task\timed_adhoc_task
... started 16:16:22. Current memory use 14.2MB.
Starting adhoc task '1 of 1' wth duration: 1
adhoc task running: 1/1 seconds
Ending adhoc task '1 of 1' with duration: 1
... used 2 dbqueries
... used 1.0067901611328 seconds
Adhoc task complete: tool_testtasks\task\timed_adhoc_task
Ran 1 adhoc tasks found at Tue, 14 Apr 2020 16:16:22 +1000
6. Run the CLI for scheduled task:
php admin/cli/scheduled_task.php --execute='\tool_recyclebin\task\cleanup_course_bin'
7. Confirm, that output looks like:
Execute scheduled task: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin)
... used 10 dbqueries
... used 0.016839027404785 seconds
Scheduled task complete: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin)
8. Navigate to Site administration > Server > Tasks > Task processing
9. Confirm, that new setting "cron_enabled" is presented and enabled.
10. Disable "cron_enabled" setting.
11. Run SQL and confirm, that static caches were cleared after cron was disabled via GUI:
moodle=# select to_timestamp(value::int) as "Caches cleared at" from mdl_config where name = 'scheduledtaskreset';
Caches cleared at
------------------------
2020-04-14 16:19:01+10
(1 row)
12. Run cron:
php admin/cli/cron.php
13. Confirm, that output looks like:
Cron is disabled. Use --force to override.
14. Enable cron via CLI:
php admin/cli/cron.php --enable
15. Confirm, that output looks like:
Cron has been enabled for the site.
16. Navigate to Site administration > Server > Tasks > Task processing.
17. Confirm, that cron is enbaled.
18. Disable cron via CLI:
php admin/cli/cron.php --disable
19. Confirm, that output looks like:
Cron has been disabled for the site.
20. Run SQL and confirm, that static caches were cleared after cron was disabled via CLI:
moodle=# select to_timestamp(value::int) as "Caches cleared at" from mdl_config where name = 'scheduledtaskreset';
Caches cleared at
------------------------
2020-04-14 16:30:33+10
(1 row)
21. Run cron with --force:
php admin/cli/cron.php --force
22. Confirm, that cron runs as normal.
23. Run the CLI for adhoc tasks:
php admin/cli/adhoc_task.php --execute
24. Confirm, that output looks like:
Cron is disabled. Use --force to override.
25. Run the CLI for adhoc tasks with --force:
php admin/cli/adhoc_task.php --execute --force
26. Confirm, that output looks like:
Server Time: Tue, 14 Apr 2020 16:35:14 +1000
Ran 0 adhoc tasks found at Tue, 14 Apr 2020 16:35:14 +1000
27. Run the CLI for scheduled task:
php admin/cli/scheduled_task.php --execute='\tool_recyclebin\task\cleanup_course_bin'
28. Confirm, that output looks like:
Cron is disabled. Use --force to override.
29. Run the CLI for scheduled task with --force:
php admin/cli/scheduled_task.php --execute='\tool_recyclebin\task\cleanup_course_bin' --force
30. Confirm, that output looks like:
Execute scheduled task: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin)
... used 8 dbqueries
... used 0.018432140350342 seconds
Scheduled task complete: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin)
31. Navigate to Site administration > Server > Tasks > Scheduled tasks.
32. Confirm, that warning message is displayed:
Cron is disabled. No new tasks will be started. The system will not operate properly until it is enabled again.
33. Run "tool_recyclebin\task\cleanup_course_bin" task by clicking "Run now" link.
34. Confirm, that task runs successfully and output looks like:
Cleanup course recycle bin
Execute scheduled task: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin)
... started 13:10:35. Current memory use 18.4MB.
... used 4 dbqueries
... used 0.0070009231567383 seconds
Scheduled task complete: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin)
35. Navigate to Site administration > Server > Tasks > Tasks running now.
36. Confirm, that warning message is displayed:
Cron is disabled. No new tasks will be started. The system will not operate properly until it is enabled again.
37. Enable cron.
38. Generate adhoc tasks:
php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=10 -n=100
39. Open two terminals and run the CLI for adhoc tasks:
php admin/cli/adhoc_task.php --execute
40. Navigate to Site administration > Server > Tasks > Tasks running now.
41. Confirm, there are 2 adhoc tasks running now.
42. Confirm, that "Time" and "Started" show when the task was started.
43. Confirm, that "Hostname" shows server name where the task is running.
44. Confirm, that "PID" shows PHP process ID.
45. Confirm, that each task has a label "Ad-hoc task id: N", where N - is an id of the task being processed.
46. Open the third terminal and disable cron via CLI with --disable-wait:
php admin/cli/cron.php --disable-wait
47. Confirm, that it waits for running tasks to finish, displays the number of running task every second and then exits:
Cron has been disabled for the site.Allocating 10 mins for the tasks to finish.2 tasks currently running.....1 tasks currently running..All scheduled and adhoc tasks finished.
48. Confirm, that both adhoc runners stopped working.
49. Enable cron via CLI:
php admin/cli/cron.php --enable
50. Run the CLI for 2 adhoc runners again:
php admin/cli/adhoc_task.php --execute
51. Disable cron via CLI with --disable-wait with a small wait value:
php admin/cli/cron.php --disable-wait=1
52. Confirm, that it waits for couple of seconds and then exits:
Cron has been disabled for the site.Allocating 1 sec for the tasks to finish.2 tasks currently running.
Wait time (1 sec) elapsed, but 2 task(s) still running.
Exiting with code 1.
53. Confirm, that it exits with status 1:
echo $?
54. Navigate to Site administration > Server > Tasks > Tasks running now.
55. Confirm, that you can see "Nothing to display" message.
56. Clear adhoc task queue:
php admin/tool/testtasks/cli/clear_adhoc_task_queue.php
57. Generate 40 adhoc task:
php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=1 -n=40
58. Emulate the cron at the hight scale and update 20 scheduled and 40 adhoc tasks like they are running now:
update mdl_task_scheduled set timestarted = 1586721857, hostname = 'host1', pid = 1111 where id < 21;
update mdl_task_adhoc set timestarted = 1586761857, hostname = 'host2', pid = 2222;
59. Navigate to Site administration > Server > Tasks > Tasks running now.
60. Confirm, that you can see 20 scheduled and 40 adhoc running tasks on the page.
61. Confirm, that you can sort the table by any column.
62. Run cron script with --list:
php admin/cli/cron.php --list
63. Confirm, that output contains all running task and well-formatted:
The list of currently running tasks:
PID HOST TYPE TIME CLASSNAME
1111 host1 scheduled 6 days 12 hours \core\task\backup_cleanup_task
1111 host1 scheduled 6 days 12 hours \core\task\blog_cron_task
1111 host1 scheduled 6 days 12 hours \core\task\cache_cleanup_task
...
...
2222 host2 adhoc 6 days 1 hour \tool_testtasks\task\timed_adhoc_task
2222 host2 adhoc 6 days 1 hour \tool_testtasks\task\timed_adhoc_task
2222 host2 adhoc 6 days 1 hour \tool_testtasks\task\timed_adhoc_task
ShowInstallation: 1. Install this to make the testing easier: git clone git @github .com:catalyst/moodle-tool_testtasks.git admin/tool/testtasks/ 2. Clear adhoc task queue: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 3. Set "Path to PHP CLI" admin setting according to your environment: php admin/cli/cfg.php --name=pathtophp --set= '/usr/bin/php' 4. Enable "Allow 'Run now' for scheduled tasks" admin setting: php admin/cli/cfg.php --name=enablerunnow --set= '1' --component=tool_task Testing instructions: 1. Open terminal and run cron: php admin/cli/cron.php 2. Confirm , that cron runs as normal. 3. Generate an adhoc task: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 1 -n= 1 4. Run the CLI for adhoc tasks: php admin/cli/adhoc_task.php --execute 5. Confirm , that output looks like: root @c69335460f7f :/siteroot# php admin/cli/adhoc_task.php --execute Server Time: Tue, 14 Apr 2020 16 : 16 : 22 + 1000 Execute adhoc task: tool_testtasks\task\timed_adhoc_task ... started 16 : 16 : 22 . Current memory use 14 .2MB. Starting adhoc task '1 of 1' wth duration: 1 adhoc task running: 1 / 1 seconds Ending adhoc task '1 of 1' with duration: 1 ... used 2 dbqueries ... used 1.0067901611328 seconds Adhoc task complete: tool_testtasks\task\timed_adhoc_task Ran 1 adhoc tasks found at Tue, 14 Apr 2020 16 : 16 : 22 + 1000 6. Run the CLI for scheduled task: php admin/cli/scheduled_task.php --execute= '\tool_recyclebin\task\cleanup_course_bin' 7. Confirm , that output looks like: Execute scheduled task: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin) ... used 10 dbqueries ... used 0.016839027404785 seconds Scheduled task complete: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin) 8. Navigate to Site administration > Server > Tasks > Task processing 9. Confirm , that new setting "cron_enabled" is presented and enabled. 10. Disable "cron_enabled" setting. 11. Run SQL and confirm , that static caches were cleared after cron was disabled via GUI: moodle=# select to_timestamp(value:: int ) as "Caches cleared at" from mdl_config where name = 'scheduledtaskreset' ; Caches cleared at ------------------------ 2020 - 04 - 14 16 : 19 : 01 + 10 ( 1 row) 12. Run cron: php admin/cli/cron.php 13. Confirm , that output looks like: Cron is disabled. Use --force to override. 14. Enable cron via CLI: php admin/cli/cron.php --enable 15. Confirm , that output looks like: Cron has been enabled for the site. 16. Navigate to Site administration > Server > Tasks > Task processing. 17. Confirm , that cron is enbaled. 18. Disable cron via CLI: php admin/cli/cron.php --disable 19. Confirm , that output looks like: Cron has been disabled for the site. 20. Run SQL and confirm, that static caches were cleared after cron was disabled via CLI: moodle=# select to_timestamp(value:: int ) as "Caches cleared at" from mdl_config where name = 'scheduledtaskreset' ; Caches cleared at ------------------------ 2020 - 04 - 14 16 : 30 : 33 + 10 ( 1 row) 21. Run cron with --force: php admin/cli/cron.php --force 22. Confirm , that cron runs as normal. 23. Run the CLI for adhoc tasks: php admin/cli/adhoc_task.php --execute 24. Confirm , that output looks like: Cron is disabled. Use --force to override. 25. Run the CLI for adhoc tasks with --force: php admin/cli/adhoc_task.php --execute --force 26. Confirm , that output looks like: Server Time: Tue, 14 Apr 2020 16 : 35 : 14 + 1000 Ran 0 adhoc tasks found at Tue, 14 Apr 2020 16 : 35 : 14 + 1000 27. Run the CLI for scheduled task: php admin/cli/scheduled_task.php --execute= '\tool_recyclebin\task\cleanup_course_bin' 28. Confirm , that output looks like: Cron is disabled. Use --force to override. 29. Run the CLI for scheduled task with --force: php admin/cli/scheduled_task.php --execute= '\tool_recyclebin\task\cleanup_course_bin' --force 30. Confirm , that output looks like: Execute scheduled task: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin) ... used 8 dbqueries ... used 0.018432140350342 seconds Scheduled task complete: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin) 31. Navigate to Site administration > Server > Tasks > Scheduled tasks. 32. Confirm , that warning message is displayed: Cron is disabled. No new tasks will be started. The system will not operate properly until it is enabled again. 33. Run "tool_recyclebin\task\cleanup_course_bin" task by clicking "Run now" link. 34. Confirm , that task runs successfully and output looks like: Cleanup course recycle bin Execute scheduled task: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin) ... started 13 : 10 : 35 . Current memory use 18 .4MB. ... used 4 dbqueries ... used 0.0070009231567383 seconds Scheduled task complete: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin) 35. Navigate to Site administration > Server > Tasks > Tasks running now. 36. Confirm , that warning message is displayed: Cron is disabled. No new tasks will be started. The system will not operate properly until it is enabled again. 37. Enable cron. 38. Generate adhoc tasks: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 10 -n= 100 39. Open two terminals and run the CLI for adhoc tasks: php admin/cli/adhoc_task.php --execute 40. Navigate to Site administration > Server > Tasks > Tasks running now. 41. Confirm , there are 2 adhoc tasks running now. 42. Confirm , that "Time" and "Started" show when the task was started. 43. Confirm , that "Hostname" shows server name where the task is running. 44. Confirm , that "PID" shows PHP process ID. 45. Confirm , that each task has a label "Ad-hoc task id: N", where N - is an id of the task being processed. 46. Open the third terminal and disable cron via CLI with --disable-wait: php admin/cli/cron.php --disable-wait 47. Confirm , that it waits for running tasks to finish, displays the number of running task every second and then exits: Cron has been disabled for the site.Allocating 10 mins for the tasks to finish. 2 tasks currently running..... 1 tasks currently running..All scheduled and adhoc tasks finished. 48. Confirm , that both adhoc runners stopped working. 49. Enable cron via CLI: php admin/cli/cron.php --enable 50. Run the CLI for 2 adhoc runners again: php admin/cli/adhoc_task.php --execute 51. Disable cron via CLI with --disable-wait with a small wait value: php admin/cli/cron.php --disable-wait= 1 52. Confirm , that it waits for couple of seconds and then exits: Cron has been disabled for the site.Allocating 1 sec for the tasks to finish. 2 tasks currently running. Wait time ( 1 sec) elapsed, but 2 task(s) still running. Exiting with code 1 . 53. Confirm , that it exits with status 1: echo $? 54. Navigate to Site administration > Server > Tasks > Tasks running now. 55. Confirm , that you can see "Nothing to display" message. 56. Clear adhoc task queue: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 57. Generate 40 adhoc task: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 1 -n= 40 58. Emulate the cron at the hight scale and update 20 scheduled and 40 adhoc tasks like they are running now: update mdl_task_scheduled set timestarted = 1586721857 , hostname = 'host1' , pid = 1111 where id < 21 ; update mdl_task_adhoc set timestarted = 1586761857 , hostname = 'host2' , pid = 2222 ; 59. Navigate to Site administration > Server > Tasks > Tasks running now. 60. Confirm , that you can see 20 scheduled and 40 adhoc running tasks on the page. 61. Confirm , that you can sort the table by any column. 62. Run cron script with --list: php admin/cli/cron.php --list 63. Confirm , that output contains all running task and well-formatted: The list of currently running tasks: PID HOST TYPE TIME CLASSNAME 1111 host1 scheduled 6 days 12 hours \core\task\backup_cleanup_task 1111 host1 scheduled 6 days 12 hours \core\task\blog_cron_task 1111 host1 scheduled 6 days 12 hours \core\task\cache_cleanup_task ... ... 2222 host2 adhoc 6 days 1 hour \tool_testtasks\task\timed_adhoc_task 2222 host2 adhoc 6 days 1 hour \tool_testtasks\task\timed_adhoc_task 2222 host2 adhoc 6 days 1 hour \tool_testtasks\task\timed_adhoc_task
Description
The use case for this is a system upgrade or some other procedure where we wish to reboot servers, and we do not want to interrupt a running scheduled or ad-hoc task.
There are two parts to this:
- How do you stop it running new tasks?
- How do you know if the existing tasks have finished?
For the first part, you can already do this by setting the two admin settings task_scheduled_concurrency_limit and task_adhoc_concurrency_limit to zero. However this is not a perfect solution because when you want to enable it again, you have to remember what the previous values were for that setting.
To address these two requirements, this new feature does the following:
1. Adds a new admin setting on existing 'Task processing' setting page to 'Disable background tasks'.
- If you turn this on, no new tasks (scheduled or ad-hoc) will be started by cron.
- The CLI tasks that run an individual scheduled task, or all ad-hoc tasks, will also not start any tasks, unless you use the new --force command-line argument.
- When this setting is on, a warning appears at the top of the scheduled tasks page including a link back to the settings page so that you can turn it off.
2. Keeps track of running tasks (previously the system did not know) and displays this information at the top of the 'Scheduled tasks' screen.
- A table shows all current running tasks (ad-hoc or scheduled), and how long they have been running for, at time of page load.
- There is a nice AJAX Refresh button so you can update the task list without reloading the whole page.
Attachments
- 1.png
- 6 kB
- 2.png
- 9 kB
- 3.png
- 13 kB
- 4.png
- 8 kB
- 5.png
- 5 kB
- accessibility_fixes.png
- 164 kB
- accessibility_issues.png
- 452 kB
- Screenshot_1.png
- 344 kB
- Screenshot_2.png
- 353 kB
- Screenshot_3.png
- 147 kB
- Screenshot_4.png
- 362 kB
- Screenshot_5.png
- 475 kB
Issue Links
- blocks
-
MDL-57852 Disable cron at site config level via gui and cli with fast shutdown
-
- Closed
-
- has a non-specific relationship to
-
MDL-57852 Disable cron at site config level via gui and cli with fast shutdown
-
- Closed
-
- has been marked as being related by
-
MDL-69227 Have the deprecated task cli's just call the new cli's
-
- Closed
-
-
MDL-67648 Cron task manager quality of service (version 3)
-
- Closed
-
- Testing discovered
-
MDL-68714 When running cron or adhoc tasks cli's update the process name to see status on the fly
-
- Closed
-
- will help resolve
-
MDL-67597 Blocking cron tasks only effectively lock into the future not the past, ie race condition
-
- Closed
-