Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-67211

Tasks: Show information about running tasks, allow tasks to be disabled

    XMLWordPrintable

Details

    • MOODLE_310_STABLE, MOODLE_39_STABLE
    • MOODLE_310_STABLE
    • master-MDL-67211_v6
    • Hide

      Installation:

      1. Install this to make the testing easier:

      git clone git@github.com:catalyst/moodle-tool_testtasks.git admin/tool/testtasks/

      2. Clear adhoc task queue:

      php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 

      3. Set "Path to PHP CLI" admin setting according to your environment:

      php admin/cli/cfg.php --name=pathtophp --set='/usr/bin/php'

      4. Enable "Allow 'Run now' for scheduled tasks" admin setting:

      php admin/cli/cfg.php --name=enablerunnow --set='1' --component=tool_task

      Testing instructions:

       1. Open terminal and run cron:

      php admin/cli/cron.php

      2. Confirm, that cron runs as normal.

      3. Generate an adhoc task:

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=1 -n=1

      4. Run the CLI for adhoc tasks:

      php admin/cli/adhoc_task.php --execute

      5. Confirm, that output looks like:

      root@c69335460f7f:/siteroot# php admin/cli/adhoc_task.php --execute
      Server Time: Tue, 14 Apr 2020 16:16:22 +1000
      Execute adhoc task: tool_testtasks\task\timed_adhoc_task
      ... started 16:16:22. Current memory use 14.2MB.
      Starting adhoc task '1 of 1' wth duration: 1
      adhoc task running: 1/1 seconds
      Ending adhoc task '1 of 1' with duration: 1
      ... used 2 dbqueries
      ... used 1.0067901611328 seconds
      Adhoc task complete: tool_testtasks\task\timed_adhoc_task
      Ran 1 adhoc tasks found at Tue, 14 Apr 2020 16:16:22 +1000
      

      6. Run the CLI for scheduled task:

      php admin/cli/scheduled_task.php --execute='\tool_recyclebin\task\cleanup_course_bin'

      7. Confirm, that output looks like:

      Execute scheduled task: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin)
      ... used 10 dbqueries
      ... used 0.016839027404785 seconds
      Scheduled task complete: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin)

      8. Navigate to Site administration > Server > Tasks > Task processing

      9. Confirm, that new setting "cron_enabled" is presented and enabled.

      10. Disable "cron_enabled" setting.

      11. Run SQL and confirm, that static caches were cleared after cron was disabled via GUI:

      moodle=# select to_timestamp(value::int) as "Caches cleared at" from mdl_config where name = 'scheduledtaskreset';
       Caches cleared at 
      ------------------------
       2020-04-14 16:19:01+10
      (1 row)
      

      12. Run cron:

      php admin/cli/cron.php

      13. Confirm, that output looks like:

      Cron is disabled. Use --force to override.

      14. Enable cron via CLI:

      php admin/cli/cron.php --enable

      15. Confirm, that output looks like:

      Cron has been enabled for the site.

      16. Navigate to Site administration > Server > Tasks > Task processing.

      17. Confirm, that cron is enbaled.

      18. Disable cron via CLI:

      php admin/cli/cron.php --disable

      19. Confirm, that output looks like:

      Cron has been disabled for the site.
      

      20. Run SQL and confirm, that static caches were cleared after cron was disabled via CLI:

      moodle=# select to_timestamp(value::int) as "Caches cleared at" from mdl_config where name = 'scheduledtaskreset';
       Caches cleared at 
      ------------------------
       2020-04-14 16:30:33+10
      (1 row)
      

      21. Run cron with --force:

      php admin/cli/cron.php --force

      22. Confirm, that cron runs as normal.

      23. Run the CLI for adhoc tasks:

      php admin/cli/adhoc_task.php --execute 

      24. Confirm, that output looks like:

      Cron is disabled. Use --force to override.
      

      25. Run the CLI for adhoc tasks with --force:

      php admin/cli/adhoc_task.php --execute --force
      

      26. Confirm, that output looks like:

      Server Time: Tue, 14 Apr 2020 16:35:14 +1000
      Ran 0 adhoc tasks found at Tue, 14 Apr 2020 16:35:14 +1000
      

      27. Run the CLI for scheduled task:

      php admin/cli/scheduled_task.php --execute='\tool_recyclebin\task\cleanup_course_bin'
      

      28. Confirm, that output looks like:

      Cron is disabled. Use --force to override.
      

      29. Run the CLI for scheduled task with --force:

      php admin/cli/scheduled_task.php --execute='\tool_recyclebin\task\cleanup_course_bin' --force
      

      30. Confirm, that output looks like:

      Execute scheduled task: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin)
       ... used 8 dbqueries
       ... used 0.018432140350342 seconds
       Scheduled task complete: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin)
      

      31. Navigate to Site administration > Server > Tasks > Scheduled tasks.

      32. Confirm, that warning message is displayed:

      Cron is disabled. No new tasks will be started. The system will not operate properly until it is enabled again.
      

      33. Run "tool_recyclebin\task\cleanup_course_bin" task by clicking "Run now" link.

      34. Confirm, that task runs successfully and output looks like:

      Cleanup course recycle bin
      Execute scheduled task: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin)
      ... started 13:10:35. Current memory use 18.4MB.
      ... used 4 dbqueries
      ... used 0.0070009231567383 seconds
      Scheduled task complete: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin)
      

      35. Navigate to Site administration > Server > Tasks > Tasks running now.

      36. Confirm, that warning message is displayed:

      Cron is disabled. No new tasks will be started. The system will not operate properly until it is enabled again.
      

      37. Enable cron.

      38. Generate adhoc tasks:

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=10 -n=100
      

      39. Open two terminals and run the CLI for adhoc tasks:

      php admin/cli/adhoc_task.php --execute

      40. Navigate to Site administration > Server > Tasks > Tasks running now.

      41. Confirm, there are 2 adhoc tasks running now.

      42. Confirm, that "Time" and "Started" show when the task was started.

      43. Confirm, that "Hostname" shows server name where the task is running.

      44. Confirm, that "PID" shows PHP process ID.

      45. Confirm, that each task has a label "Ad-hoc task id: N", where N - is an id of the task being processed.

      46. Open the third terminal and disable cron via CLI with --disable-wait:

      php admin/cli/cron.php --disable-wait
      

      47. Confirm, that it waits for running tasks to finish, displays the number of running task every second and then exits:

      Cron has been disabled for the site.Allocating 10 mins for the tasks to finish.2 tasks currently running.....1 tasks currently running..All scheduled and adhoc tasks finished.
      

      48. Confirm, that both adhoc runners stopped working.

      49. Enable cron via CLI:

      php admin/cli/cron.php --enable

       50. Run the CLI for 2 adhoc runners again:

      php admin/cli/adhoc_task.php --execute

      51. Disable cron via CLI with --disable-wait with a small wait value:

      php admin/cli/cron.php --disable-wait=1
      

      52. Confirm, that it waits for couple of seconds and then exits:

      Cron has been disabled for the site.Allocating 1 sec for the tasks to finish.2 tasks currently running.
      Wait time (1 sec) elapsed, but 2 task(s) still running.
      Exiting with code 1.
      

       53. Confirm, that it exits with status 1:

      echo $?

      54. Navigate to Site administration > Server > Tasks > Tasks running now.

      55. Confirm, that you can see "Nothing to display" message.

      56. Clear adhoc task queue:

      php admin/tool/testtasks/cli/clear_adhoc_task_queue.php
      

      57. Generate 40 adhoc task:

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=1 -n=40
      

      58. Emulate the cron at the hight scale and update 20 scheduled and 40 adhoc tasks like they are running now:

      update mdl_task_scheduled set timestarted = 1586721857, hostname = 'host1', pid = 1111 where id < 21;
      update mdl_task_adhoc set timestarted = 1586761857, hostname = 'host2', pid = 2222;
      

      59. Navigate to Site administration > Server > Tasks > Tasks running now.

      60. Confirm, that you can see 20 scheduled and 40 adhoc running tasks on the page.

      61. Confirm, that you can sort the table by any column.

      62. Run cron script with --list:

      php admin/cli/cron.php --list
      

      63. Confirm, that output contains all running task and well-formatted:

      The list of currently running tasks:
          PID HOST         TYPE      TIME                 CLASSNAME                                           
         1111 host1        scheduled 6 days 12 hours      \core\task\backup_cleanup_task                      
         1111 host1        scheduled 6 days 12 hours      \core\task\blog_cron_task                           
         1111 host1        scheduled 6 days 12 hours      \core\task\cache_cleanup_task                       
      ...
      ...
         2222 host2        adhoc     6 days 1 hour        \tool_testtasks\task\timed_adhoc_task               
         2222 host2        adhoc     6 days 1 hour        \tool_testtasks\task\timed_adhoc_task               
         2222 host2        adhoc     6 days 1 hour        \tool_testtasks\task\timed_adhoc_task    
      

       

       

      Show
      Installation: 1. Install this to make the testing easier: git clone git @github .com:catalyst/moodle-tool_testtasks.git admin/tool/testtasks/ 2. Clear adhoc task queue: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php  3. Set "Path to PHP CLI" admin setting according to your environment: php admin/cli/cfg.php --name=pathtophp --set= '/usr/bin/php' 4. Enable "Allow 'Run now' for scheduled tasks" admin setting: php admin/cli/cfg.php --name=enablerunnow --set= '1' --component=tool_task Testing instructions:  1. Open terminal and run cron: php admin/cli/cron.php 2. Confirm , that cron runs as normal. 3. Generate an adhoc task: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 1 -n= 1 4. Run the CLI for adhoc tasks: php admin/cli/adhoc_task.php --execute 5. Confirm , that output looks like: root @c69335460f7f :/siteroot# php admin/cli/adhoc_task.php --execute Server Time: Tue, 14 Apr 2020 16 : 16 : 22 + 1000 Execute adhoc task: tool_testtasks\task\timed_adhoc_task ... started 16 : 16 : 22 . Current memory use 14 .2MB. Starting adhoc task '1 of 1' wth duration: 1 adhoc task running: 1 / 1 seconds Ending adhoc task '1 of 1' with duration: 1 ... used 2 dbqueries ... used 1.0067901611328 seconds Adhoc task complete: tool_testtasks\task\timed_adhoc_task Ran 1 adhoc tasks found at Tue, 14 Apr 2020 16 : 16 : 22 + 1000 6. Run the CLI for scheduled task: php admin/cli/scheduled_task.php --execute= '\tool_recyclebin\task\cleanup_course_bin' 7. Confirm , that output looks like: Execute scheduled task: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin) ... used 10 dbqueries ... used 0.016839027404785 seconds Scheduled task complete: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin) 8. Navigate to Site administration > Server > Tasks > Task processing 9. Confirm , that new setting "cron_enabled" is presented and enabled. 10. Disable "cron_enabled" setting. 11. Run SQL and confirm , that static caches were cleared after cron was disabled via GUI: moodle=# select to_timestamp(value:: int ) as "Caches cleared at" from mdl_config where name = 'scheduledtaskreset' ; Caches cleared at ------------------------ 2020 - 04 - 14 16 : 19 : 01 + 10 ( 1 row) 12. Run cron: php admin/cli/cron.php 13. Confirm , that output looks like: Cron is disabled. Use --force to override. 14. Enable cron via CLI: php admin/cli/cron.php --enable 15. Confirm , that output looks like: Cron has been enabled for the site. 16. Navigate to Site administration > Server > Tasks > Task processing. 17.  Confirm , that cron is enbaled. 18. Disable cron via CLI: php admin/cli/cron.php --disable 19. Confirm , that output looks like: Cron has been disabled for the site. 20. Run SQL and confirm, that static caches were cleared after cron was disabled via CLI: moodle=# select to_timestamp(value:: int ) as "Caches cleared at" from mdl_config where name = 'scheduledtaskreset' ; Caches cleared at ------------------------ 2020 - 04 - 14 16 : 30 : 33 + 10 ( 1 row) 21. Run cron with --force: php admin/cli/cron.php --force 22. Confirm , that cron runs as normal. 23. Run the CLI for adhoc tasks: php admin/cli/adhoc_task.php --execute  24. Confirm , that output looks like: Cron is disabled. Use --force to override. 25. Run the CLI for adhoc tasks with --force: php admin/cli/adhoc_task.php --execute --force 26. Confirm , that output looks like: Server Time: Tue, 14 Apr 2020 16 : 35 : 14 + 1000 Ran 0 adhoc tasks found at Tue, 14 Apr 2020 16 : 35 : 14 + 1000 27. Run the CLI for scheduled task: php admin/cli/scheduled_task.php --execute= '\tool_recyclebin\task\cleanup_course_bin' 28. Confirm , that output looks like: Cron is disabled. Use --force to override. 29. Run the CLI for scheduled task with --force: php admin/cli/scheduled_task.php --execute= '\tool_recyclebin\task\cleanup_course_bin' --force 30. Confirm , that output looks like: Execute scheduled task: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin) ... used 8 dbqueries ... used 0.018432140350342 seconds Scheduled task complete: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin) 31. Navigate to Site administration > Server > Tasks > Scheduled tasks. 32. Confirm , that warning message is displayed: Cron is disabled. No new tasks will be started. The system will not operate properly until it is enabled again. 33. Run "tool_recyclebin\task\cleanup_course_bin" task by clicking "Run now" link. 34. Confirm , that task runs successfully and output looks like: Cleanup course recycle bin Execute scheduled task: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin) ... started 13 : 10 : 35 . Current memory use 18 .4MB. ... used 4 dbqueries ... used 0.0070009231567383 seconds Scheduled task complete: Cleanup course recycle bin (tool_recyclebin\task\cleanup_course_bin) 35. Navigate to Site administration > Server > Tasks > Tasks running now. 36. Confirm , that warning message is displayed: Cron is disabled. No new tasks will be started. The system will not operate properly until it is enabled again. 37. Enable cron. 38. Generate adhoc tasks: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 10 -n= 100 39. Open two terminals and run the CLI for adhoc tasks: php admin/cli/adhoc_task.php --execute 40. Navigate to Site administration > Server > Tasks > Tasks running now. 41.  Confirm , there are 2 adhoc tasks running now. 42.  Confirm , that "Time" and "Started" show when the task was started. 43.  Confirm , that "Hostname" shows server name where the task is running. 44. Confirm , that "PID" shows PHP process ID. 45. Confirm , that each task has a label "Ad-hoc task id: N", where N - is an id of the task being processed. 46. Open the third terminal and disable cron via CLI with --disable-wait: php admin/cli/cron.php --disable-wait 47.  Confirm , that it waits for running tasks to finish, displays the number of running task every second and then exits: Cron has been disabled for the site.Allocating 10 mins for the tasks to finish. 2 tasks currently running..... 1 tasks currently running..All scheduled and adhoc tasks finished. 48.  Confirm , that both adhoc runners stopped working. 49. Enable cron via CLI: php admin/cli/cron.php --enable  50. Run the CLI for 2 adhoc runners again: php admin/cli/adhoc_task.php --execute 51. Disable cron via CLI with --disable-wait with a small wait value: php admin/cli/cron.php --disable-wait= 1 52.  Confirm , that it waits for couple of seconds and then exits: Cron has been disabled for the site.Allocating 1 sec for the tasks to finish. 2 tasks currently running. Wait time ( 1 sec) elapsed, but 2 task(s) still running. Exiting with code 1 .  53. Confirm , that it exits with status 1: echo $? 54. Navigate to Site administration > Server > Tasks > Tasks running now. 55.  Confirm , that you can see "Nothing to display" message. 56. Clear adhoc task queue: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 57. Generate 40 adhoc task: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 1 -n= 40 58. Emulate the cron at the hight scale and update 20 scheduled and 40 adhoc tasks like they are running now: update mdl_task_scheduled set timestarted = 1586721857 , hostname = 'host1' , pid = 1111 where id < 21 ; update mdl_task_adhoc set timestarted = 1586761857 , hostname = 'host2' , pid = 2222 ; 59. Navigate to Site administration > Server > Tasks > Tasks running now. 60.  Confirm , that you can see 20 scheduled and 40 adhoc running tasks on the page. 61.  Confirm , that you can sort the table by any column. 62. Run cron script with --list: php admin/cli/cron.php --list 63.  Confirm , that output contains all running task and well-formatted: The list of currently running tasks: PID HOST TYPE TIME CLASSNAME 1111 host1 scheduled 6 days 12 hours \core\task\backup_cleanup_task 1111 host1 scheduled 6 days 12 hours \core\task\blog_cron_task 1111 host1 scheduled 6 days 12 hours \core\task\cache_cleanup_task ... ... 2222 host2 adhoc 6 days 1 hour \tool_testtasks\task\timed_adhoc_task 2222 host2 adhoc 6 days 1 hour \tool_testtasks\task\timed_adhoc_task 2222 host2 adhoc 6 days 1 hour \tool_testtasks\task\timed_adhoc_task    

    Description

      The use case for this is a system upgrade or some other procedure where we wish to reboot servers, and we do not want to interrupt a running scheduled or ad-hoc task.

      There are two parts to this:

      1. How do you stop it running new tasks?
      2. How do you know if the existing tasks have finished?

      For the first part, you can already do this by setting the two admin settings task_scheduled_concurrency_limit and task_adhoc_concurrency_limit to zero. However this is not a perfect solution because when you want to enable it again, you have to remember what the previous values were for that setting.

      To address these two requirements, this new feature does the following:

      1. Adds a new admin setting on existing 'Task processing' setting page to 'Disable background tasks'.

      • If you turn this on, no new tasks (scheduled or ad-hoc) will be started by cron.
      • The CLI tasks that run an individual scheduled task, or all ad-hoc tasks, will also not start any tasks, unless you use the new --force command-line argument.
      • When this setting is on, a warning appears at the top of the scheduled tasks page including a link back to the settings page so that you can turn it off.

      2. Keeps track of running tasks (previously the system did not know) and displays this information at the top of the 'Scheduled tasks' screen.

      • A table shows all current running tasks (ad-hoc or scheduled), and how long they have been running for, at time of page load.
      • There is a nice AJAX Refresh button so you can update the task list without reloading the whole page.

      Attachments

        1. 1.png
          1.png
          6 kB
        2. 2.png
          2.png
          9 kB
        3. 3.png
          3.png
          13 kB
        4. 4.png
          4.png
          8 kB
        5. 5.png
          5.png
          5 kB
        6. accessibility_fixes.png
          accessibility_fixes.png
          164 kB
        7. accessibility_issues.png
          accessibility_issues.png
          452 kB
        8. Screenshot_1.png
          Screenshot_1.png
          344 kB
        9. Screenshot_2.png
          Screenshot_2.png
          353 kB
        10. Screenshot_3.png
          Screenshot_3.png
          147 kB
        11. Screenshot_4.png
          Screenshot_4.png
          362 kB
        12. Screenshot_5.png
          Screenshot_5.png
          475 kB

        Issue Links

          Activity

            People

              mikhailgolenkov Misha Golenkov
              quen Sam Marshall
              Ferran Recio Ferran Recio
              Andrew Lyons Andrew Lyons
              Janelle Barcega Janelle Barcega
              Votes:
              7 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                9/Nov/20

                Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 days, 6 hours
                  2d 6h