Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-64610

Add support for per-task concurrency limits

XMLWordPrintable

    • MOODLE_37_STABLE, MOODLE_39_STABLE
    • MOODLE_39_STABLE
    • master_MDL-64610
    • Hide

      Installation

      1. Install this to make the testing easier:

      https://github.com/catalyst/moodle-tool_testtasks

      Default settings

      1. Clear adhoc task queue:

      php admin/tool/testtasks/cli/clear_adhoc_task_queue.php

      2. Create 50 tasks

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=10 -n=50

      3. Open 3 terminals and run cron:

      php admin/cli/cron.php

      4. Confirm, that all 3 runners are running queue_adhoc_tasks. Output should look like:

      Execute adhoc task: tool_testtasks\task\timed_adhoc_task
      		 
       ... started 02:32:14. Current memory use 19.5MB.
      		 
       Starting adhoc task '1 of 50' wth duration: 10
      		 
       adhoc task running: 1/10 seconds
      		 
       adhoc task running: 2/10 seconds
      		 
       adhoc task running: 3/10 seconds
      		 
       adhoc task running: 4/10 seconds
      		 
       adhoc task running: 5/10 seconds

      5. Open another terminal and run cron:

      php admin/cli/cron.php

      6. Confirm, that while 3 runners are processing the tasks, one runner has exited without processing:

      root@5d15afd93e52:/siteroot# php admin/cli/cron.php
      		 
       Server Time: Mon, 16 Dec 2019 06:25:26 +0000
      		 
      Skipping processing of adhoc tasks. Concurrency limit reached.
      		 
       Cron script completed correctly
      		 
       Cron completed at 06:25:29. Memory used 18.3MB.
      		 
       Execution took 2.186963 seconds

      Limit via default

      1. Clear adhoc task queue:

      php admin/tool/testtasks/cli/clear_adhoc_task_queue.php

      2. Set task_concurrency_limit_default to 2:

      php admin/cli/cfg.php --name=task_concurrency_limit_default --set=2

      3. Create 50 tasks

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=10 -n=50

      4. Open 3 terminals and run cron:

      php admin/cli/cron.php

      5. Confirm, that while 2 runners are processing the tasks, one runner has exited without processing:

      root@b24142f9f846:/siteroot# php admin/cli/cron.php
      		 
      Server Time: Thu, 02 Jan 2020 04:29:19 +0000
      		 
      Skipping \tool_testtasks\task\timed_adhoc_task adhoc task class as the per-task limit is reached.
      		 
      Ran 0 adhoc tasks found at Thu, 02 Jan 2020 04:29:19 +0000
      		 
      Cron script completed correctly
      		 
      Cron completed at 04:29:21. Memory used 18.4MB.
      		 
      Execution took 1.482996 seconds

       6. Create 'another' type of adhoc task:

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=1 -n=100 --class='tool_testtasks\task\another_timed_adhoc_task'

      7. Open 3rd terminal (idling) and run cron:

      php admin/cli/cron.php

      8. Confirm, that while 2 runners are processing 'tool_testtasks\task\timed_adhoc_task' tasks, third runner is processing 'tool_testtasks\task\another_timed_adhoc_task' task:

       Starting ANOTHER adhoc task '17 of 100' wth duration: 1
      		 
       adhoc ANOTHER task running: 1/1 seconds
      		 
       Ending ANOTHER adhoc task '17 of 100' with duration: 1
      		 
       ... used 0 dbqueries
      		 
       ... used 1.0005919933319 seconds
      		 
       Adhoc task complete: tool_testtasks\task\another_timed_adhoc_task

      Limit via per-task setting

      1. Clear adhoc task queue:

      php admin/tool/testtasks/cli/clear_adhoc_task_queue.php

      2. Set task_concurrency_limit_default to 3:

      php admin/cli/cfg.php --name=task_concurrency_limit_default --set=3

      3. Set per-task concurrency limit for 'tool_testtasks\task\timed_adhoc_task' task to 1 by adding this to config.php:

      $CFG->task_concurrency_limit = array('tool_testtasks\task\timed_adhoc_task' => 1);

      4. Create 50 tasks

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=10 -n=50

      5. Open 2 terminals and run cron:

      php admin/cli/cron.php

      6. Confirm, that while the first runner is processing the task, the second one has exited without processing:

      root@b24142f9f846:/siteroot# php admin/cli/cron.php
      		 
      Server Time: Thu, 02 Jan 2020 04:31:44 +0000
      		 
      Skipping \tool_testtasks\task\timed_adhoc_task adhoc task class as the per-task limit is reached.
      		 
      Ran 0 adhoc tasks found at Thu, 02 Jan 2020 04:31:44 +0000
      		 
      Cron script completed correctly
      		 
      Cron completed at 04:31:45. Memory used 18.4MB.
      		 
      Execution took 1.056334 seconds

      Load testing:

      1. Clear adhoc task queue:

      php admin/tool/testtasks/cli/clear_adhoc_task_queue.php

      2. Set task_concurrency_limit_default to 3:

      php admin/cli/cfg.php --name=task_concurrency_limit_default --set=3

      3. Set per-task concurrency limit for 'tool_testtasks\task\timed_adhoc_task' task to 1 by adding this to config.php:

      $CFG->task_concurrency_limit = array('tool_testtasks\task\timed_adhoc_task' => 1);

      4. Create 50 000 tasks:

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=300 -n=50000

      5. Open 2 terminals and run cron:

      php admin/cli/cron.php

      6. Confirm, that while the first runner is processing the task, the second one has exited without processing:

      root@b24142f9f846:/siteroot# php admin/cli/cron.php
      		 
      Server Time: Thu, 02 Jan 2020 04:34:55 +0000
      		 
      Skipping \tool_testtasks\task\timed_adhoc_task adhoc task class as the per-task limit is reached.
      		 
      Ran 0 adhoc tasks found at Thu, 02 Jan 2020 04:34:55 +0000
      		 
      Cron script completed correctly
      		 
      Cron completed at 04:34:56. Memory used 18.9MB.
      		 
      Execution took 1.057899 seconds

       7. Confirm, that inspite the huge task queue the second runner completed quickly (within 2 seconds):

      Execution took 1.057899 seconds

      Note: cron might run the full list of scheduled tasks and execution time might be greater (2-3 seconds).
      Re-run it shortly to make sure that ad-hock queue processing works quickly and doesn't take more than 2 seconds.

      8. Create 'another' type of adhoc task:

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=1 -n=20 --class='tool_testtasks\task\another_timed_adhoc_task'

      9. Open 2nd terminal (idling) and run cron:

      php admin/cli/cron.php

      10. Confirm, that the second runner is processing 'another' task:

      Starting ANOTHER adhoc task '20 of 20' wth duration: 1
      		 
       adhoc ANOTHER task running: 1/1 seconds
      		 
       Ending ANOTHER adhoc task '20 of 20' with duration: 1
      		 
       ... used 0 dbqueries
      		 
       ... used 1.0005798339844 seconds
      		 
       Adhoc task complete: tool_testtasks\task\another_timed_adhoc_task
      		 
       Cron script completed correctly
      		 
       Cron completed at 07:10:34. Memory used 40.8MB.
      		 
       Execution took 32.387464 seconds

      Unlimited run with --ignorelimits

      1. Clear adhoc task queue:

      php admin/tool/testtasks/cli/clear_adhoc_task_queue.php

      2. Set task_concurrency_limit_default to 1:

      php admin/cli/cfg.php --name=task_concurrency_limit_default --set=1

      3. Create 50 tasks:

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=10 -n=50

      4. Run cron from the first terminal:

      php admin/cli/cron.php

      5. Execute adhoc_task.php without limits:

      php admin/tool/task/cli/adhoc_task.php --execute --ignorelimits

      6. Confirm, that adhoc_task.php is processing the tasks:

      root@b24142f9f846:/siteroot# php admin/tool/task/cli/adhoc_task.php --execute --ignorelimits
      		 
      Server Time: Thu, 02 Jan 2020 04:17:39 +0000
      		 
      Execute adhoc task: tool_testtasks\task\timed_adhoc_task
      		 
      ... started 04:17:39. Current memory use 14.1MB.
      		 
      Starting adhoc task '4 of 50' wth duration: 10
      		 
      adhoc task running: 1/10 seconds
      		 
      adhoc task running: 2/10 seconds
      		 
      adhoc task running: 3/10 seconds

       

       

      Show
      Installation 1. Install this to make the testing easier: https: //github.com/catalyst/moodle-tool_testtasks Default settings 1. Clear adhoc task queue: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 2. Create 50 tasks php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 10 -n= 50 3. Open 3 terminals and run cron: php admin/cli/cron.php 4. Confirm , that all 3 runners are running queue_adhoc_tasks. Output should look like: Execute adhoc task: tool_testtasks\task\timed_adhoc_task ... started 02 : 32 : 14 . Current memory use 19 .5MB. Starting adhoc task '1 of 50' wth duration: 10 adhoc task running: 1 / 10 seconds adhoc task running: 2 / 10 seconds adhoc task running: 3 / 10 seconds adhoc task running: 4 / 10 seconds adhoc task running: 5 / 10 seconds 5. Open another terminal and run cron: php admin/cli/cron.php 6. Confirm , that while 3 runners are processing the tasks, one runner has exited without processing: root @5d15afd93e52 :/siteroot# php admin/cli/cron.php Server Time: Mon, 16 Dec 2019 06 : 25 : 26 + 0000 Skipping processing of adhoc tasks. Concurrency limit reached. Cron script completed correctly Cron completed at 06 : 25 : 29 . Memory used 18 .3MB. Execution took 2.186963 seconds Limit via default 1. Clear adhoc task queue: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 2. Set task_concurrency_limit_default to 2: php admin/cli/cfg.php --name=task_concurrency_limit_default --set= 2 3. Create 50 tasks php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 10 -n= 50 4. Open 3 terminals and run cron: php admin/cli/cron.php 5. Confirm , that while 2 runners are processing the tasks, one runner has exited without processing: root @b24142f9f846 :/siteroot# php admin/cli/cron.php Server Time: Thu, 02 Jan 2020 04 : 29 : 19 + 0000 Skipping \tool_testtasks\task\timed_adhoc_task adhoc task class as the per-task limit is reached. Ran 0 adhoc tasks found at Thu, 02 Jan 2020 04 : 29 : 19 + 0000 Cron script completed correctly Cron completed at 04 : 29 : 21 . Memory used 18 .4MB. Execution took 1.482996 seconds  6. Create 'another' type of adhoc task: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 1 -n= 100 -- class = 'tool_testtasks\task\another_timed_adhoc_task' 7. Open 3rd terminal (idling) and run cron: php admin/cli/cron.php 8. Confirm , that while 2 runners are processing 'tool_testtasks\task\timed_adhoc_task' tasks, third runner is processing 'tool_testtasks\task\another_timed_adhoc_task' task: Starting ANOTHER adhoc task '17 of 100' wth duration: 1 adhoc ANOTHER task running: 1 / 1 seconds Ending ANOTHER adhoc task '17 of 100' with duration: 1 ... used 0 dbqueries ... used 1.0005919933319 seconds Adhoc task complete: tool_testtasks\task\another_timed_adhoc_task Limit via per-task setting 1. Clear adhoc task queue: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 2. Set task_concurrency_limit_default to 3: php admin/cli/cfg.php --name=task_concurrency_limit_default --set= 3 3. Set per-task concurrency limit for 'tool_testtasks\task\timed_adhoc_task' task to 1 by adding this to config.php: $CFG->task_concurrency_limit = array( 'tool_testtasks\task\timed_adhoc_task' => 1 ); 4. Create 50 tasks php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 10 -n= 50 5. Open 2 terminals and run cron: php admin/cli/cron.php 6. Confirm , that while the first runner is processing the task, the second one has exited without processing: root @b24142f9f846 :/siteroot# php admin/cli/cron.php Server Time: Thu, 02 Jan 2020 04 : 31 : 44 + 0000 Skipping \tool_testtasks\task\timed_adhoc_task adhoc task class as the per-task limit is reached. Ran 0 adhoc tasks found at Thu, 02 Jan 2020 04 : 31 : 44 + 0000 Cron script completed correctly Cron completed at 04 : 31 : 45 . Memory used 18 .4MB. Execution took 1.056334 seconds Load testing: 1. Clear adhoc task queue: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 2. Set task_concurrency_limit_default to 3: php admin/cli/cfg.php --name=task_concurrency_limit_default --set= 3 3. Set per-task concurrency limit for 'tool_testtasks\task\timed_adhoc_task' task to 1 by adding this to config.php: $CFG->task_concurrency_limit = array( 'tool_testtasks\task\timed_adhoc_task' => 1 ); 4. Create 50 000 tasks: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 300 -n= 50000 5. Open 2 terminals and run cron: php admin/cli/cron.php 6. Confirm , that while the first runner is processing the task, the second one has exited without processing: root @b24142f9f846 :/siteroot# php admin/cli/cron.php Server Time: Thu, 02 Jan 2020 04 : 34 : 55 + 0000 Skipping \tool_testtasks\task\timed_adhoc_task adhoc task class as the per-task limit is reached. Ran 0 adhoc tasks found at Thu, 02 Jan 2020 04 : 34 : 55 + 0000 Cron script completed correctly Cron completed at 04 : 34 : 56 . Memory used 18 .9MB. Execution took 1.057899 seconds  7. Confirm , that inspite the huge task queue the second runner completed quickly (within 2 seconds): Execution took 1.057899 seconds Note : cron might run the full list of scheduled tasks and execution time might be greater (2-3 seconds). Re-run it shortly to make sure that ad-hock queue processing works quickly and doesn't take more than 2 seconds. 8. Create 'another' type of adhoc task: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 1 -n= 20 -- class = 'tool_testtasks\task\another_timed_adhoc_task' 9. Open 2nd terminal (idling) and run cron: php admin/cli/cron.php 10. Confirm , that the second runner is processing 'another' task: Starting ANOTHER adhoc task '20 of 20' wth duration: 1 adhoc ANOTHER task running: 1 / 1 seconds Ending ANOTHER adhoc task '20 of 20' with duration: 1 ... used 0 dbqueries ... used 1.0005798339844 seconds Adhoc task complete: tool_testtasks\task\another_timed_adhoc_task Cron script completed correctly Cron completed at 07 : 10 : 34 . Memory used 40 .8MB. Execution took 32.387464 seconds Unlimited run with --ignorelimits 1. Clear adhoc task queue: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 2. Set task_concurrency_limit_default to 1: php admin/cli/cfg.php --name=task_concurrency_limit_default --set= 1 3. Create 50 tasks: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 10 -n= 50 4. Run cron from the first terminal: php admin/cli/cron.php 5. Execute adhoc_task.php without limits: php admin/tool/task/cli/adhoc_task.php --execute --ignorelimits 6. Confirm , that adhoc_task.php is processing the tasks: root @b24142f9f846 :/siteroot# php admin/tool/task/cli/adhoc_task.php --execute --ignorelimits Server Time: Thu, 02 Jan 2020 04 : 17 : 39 + 0000 Execute adhoc task: tool_testtasks\task\timed_adhoc_task ... started 04 : 17 : 39 . Current memory use 14 .1MB. Starting adhoc task '4 of 50' wth duration: 10 adhoc task running: 1 / 10 seconds adhoc task running: 2 / 10 seconds adhoc task running: 3 / 10 seconds    

      MDL-64347 added limits to the number of concurrent task runners, but the suggestion was made to add a per-task concurrency limit for adhoc tasks too.

      Essentially we want to provide a way to set a limit for the number of times that an adhoc task can run at once (in the case of multiple instances of a single task).

      This becomes increasingly important as we increase the use of adhoc tasks.
      We are currently using them for privacy exports, and MDL-46881 seeks to move forum cron processing to adhoc tasks too. Additionally there is a suggested plan to move backup and restore to adhoc tasks.

      Each of these tasks has different load patterns. Whilst privacy tends to have a smaller number of heavy tasks, the forum will have a larger number of very lightweight tasks.

      This issue proposes to allow different concurrency limits for the different tasks.

      We need to decide:

      1. is it okay to only allow setting in config.php
      2. do we want the ability for adhoc tasks to have a way to specify a default concurrency
      3. do we want a UI to configure these?

      I think that we definitely want items 1, and 2. We will probably want item 3 (though we could potentially defer this).

      I do feel that this issue is highly important as the load patterns of different adhoc tasks will significantly change as we increase their usage in different areas.
      Without this functionality we may find that privacy exports lock all potential adhoc task runners and block any forum tasks as a result.

        1. defaultsetting.png
          defaultsetting.png
          391 kB
        2. limit_default&tasksetting.png
          limit_default&tasksetting.png
          439 kB
        3. loadtesting.png
          loadtesting.png
          377 kB

            mikhailgolenkov Misha Golenkov
            dobedobedoh Andrew Lyons
            Dmitrii Metelkin Dmitrii Metelkin
            Jake Dallimore Jake Dallimore
            Janelle Barcega Janelle Barcega
            Votes:
            3 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 3 hours, 1 minute
                3h 1m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.