Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-64610

Add support for per-task concurrency limits

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      Installation

      1. Install this to make the testing easier:

      https://github.com/catalyst/moodle-tool_testtasks

      Default settings

      1. Clear adhoc task queue:

      php admin/tool/testtasks/cli/clear_adhoc_task_queue.php

      2. Create 50 tasks

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=10 -n=50

      3. Open 3 terminals and run cron:

      php admin/cli/cron.php

      4. Confirm, that all 3 runners are running queue_adhoc_tasks. Output should look like:

      Execute adhoc task: tool_testtasks\task\timed_adhoc_task
       ... started 02:32:14. Current memory use 19.5MB.
       Starting adhoc task '1 of 50' wth duration: 10
       adhoc task running: 1/10 seconds
       adhoc task running: 2/10 seconds
       adhoc task running: 3/10 seconds
       adhoc task running: 4/10 seconds
       adhoc task running: 5/10 seconds

      5. Open another terminal and run cron:

      php admin/cli/cron.php

      6. Confirm, that while 3 runners are processing the tasks, one runner has exited without processing:

      root@5d15afd93e52:/siteroot# php admin/cli/cron.php
       Server Time: Mon, 16 Dec 2019 06:25:26 +0000
      Skipping processing of adhoc tasks. Concurrency limit reached.
       Cron script completed correctly
       Cron completed at 06:25:29. Memory used 18.3MB.
       Execution took 2.186963 seconds
      

      Limit via default

      1. Clear adhoc task queue:

      php admin/tool/testtasks/cli/clear_adhoc_task_queue.php

      2. Set task_concurrency_limit_default to 2:

      php admin/cli/cfg.php --name=task_concurrency_limit_default --set=2

      3. Create 50 tasks

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=10 -n=50

      4. Open 3 terminals and run cron:

      php admin/cli/cron.php

      5. Confirm, that while 2 runners are processing the tasks, one runner has exited without processing:

      root@b24142f9f846:/siteroot# php admin/cli/cron.php
      Server Time: Thu, 02 Jan 2020 04:29:19 +0000
      Skipping \tool_testtasks\task\timed_adhoc_task adhoc task class as the per-task limit is reached.
      Ran 0 adhoc tasks found at Thu, 02 Jan 2020 04:29:19 +0000
      Cron script completed correctly
      Cron completed at 04:29:21. Memory used 18.4MB.
      Execution took 1.482996 seconds
      

       6. Create 'another' type of adhoc task:

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=1 -n=100 --class='tool_testtasks\task\another_timed_adhoc_task'

      7. Open 3rd terminal (idling) and run cron:

      php admin/cli/cron.php

      8. Confirm, that while 2 runners are processing 'tool_testtasks\task\timed_adhoc_task' tasks, third runner is processing 'tool_testtasks\task\another_timed_adhoc_task' task:

       Starting ANOTHER adhoc task '17 of 100' wth duration: 1
       adhoc ANOTHER task running: 1/1 seconds
       Ending ANOTHER adhoc task '17 of 100' with duration: 1
       ... used 0 dbqueries
       ... used 1.0005919933319 seconds
       Adhoc task complete: tool_testtasks\task\another_timed_adhoc_task

      Limit via per-task setting

      1. Clear adhoc task queue:

      php admin/tool/testtasks/cli/clear_adhoc_task_queue.php

      2. Set task_concurrency_limit_default to 3:

      php admin/cli/cfg.php --name=task_concurrency_limit_default --set=3

      3. Set per-task concurrency limit for 'tool_testtasks\task\timed_adhoc_task' task to 1 by adding this to config.php:

      $CFG->task_concurrency_limit = array('tool_testtasks\task\timed_adhoc_task' => 1);

      4. Create 50 tasks

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=10 -n=50

      5. Open 2 terminals and run cron:

      php admin/cli/cron.php

      6. Confirm, that while the first runner is processing the task, the second one has exited without processing:

      root@b24142f9f846:/siteroot# php admin/cli/cron.php
      Server Time: Thu, 02 Jan 2020 04:31:44 +0000
      Skipping \tool_testtasks\task\timed_adhoc_task adhoc task class as the per-task limit is reached.
      Ran 0 adhoc tasks found at Thu, 02 Jan 2020 04:31:44 +0000
      Cron script completed correctly
      Cron completed at 04:31:45. Memory used 18.4MB.
      Execution took 1.056334 seconds
      

      Load testing:

      1. Clear adhoc task queue:

      php admin/tool/testtasks/cli/clear_adhoc_task_queue.php

      2. Set task_concurrency_limit_default to 3:

      php admin/cli/cfg.php --name=task_concurrency_limit_default --set=3

      3. Set per-task concurrency limit for 'tool_testtasks\task\timed_adhoc_task' task to 1 by adding this to config.php:

      $CFG->task_concurrency_limit = array('tool_testtasks\task\timed_adhoc_task' => 1);

      4. Create 50 000 tasks:

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=300 -n=50000

      5. Open 2 terminals and run cron:

      php admin/cli/cron.php

      6. Confirm, that while the first runner is processing the task, the second one has exited without processing:

      root@b24142f9f846:/siteroot# php admin/cli/cron.php
      Server Time: Thu, 02 Jan 2020 04:34:55 +0000
      Skipping \tool_testtasks\task\timed_adhoc_task adhoc task class as the per-task limit is reached.
      Ran 0 adhoc tasks found at Thu, 02 Jan 2020 04:34:55 +0000
      Cron script completed correctly
      Cron completed at 04:34:56. Memory used 18.9MB.
      Execution took 1.057899 seconds
      

       7. Confirm, that inspite the huge task queue the second runner completed quickly (within 2 seconds):

      Execution took 1.057899 seconds

      Note: cron might run the full list of scheduled tasks and execution time might be greater (2-3 seconds).
      Re-run it shortly to make sure that ad-hock queue processing works quickly and doesn't take more than 2 seconds.

      8. Create 'another' type of adhoc task:

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=1 -n=20 --class='tool_testtasks\task\another_timed_adhoc_task'

      9. Open 2nd terminal (idling) and run cron:

      php admin/cli/cron.php

      10. Confirm, that the second runner is processing 'another' task:

      Starting ANOTHER adhoc task '20 of 20' wth duration: 1
       adhoc ANOTHER task running: 1/1 seconds
       Ending ANOTHER adhoc task '20 of 20' with duration: 1
       ... used 0 dbqueries
       ... used 1.0005798339844 seconds
       Adhoc task complete: tool_testtasks\task\another_timed_adhoc_task
       Cron script completed correctly
       Cron completed at 07:10:34. Memory used 40.8MB.
       Execution took 32.387464 seconds

      Unlimited run with --ignorelimits

      1. Clear adhoc task queue:

      php admin/tool/testtasks/cli/clear_adhoc_task_queue.php

      2. Set task_concurrency_limit_default to 1:

      php admin/cli/cfg.php --name=task_concurrency_limit_default --set=1

      3. Create 50 tasks:

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=10 -n=50

      4. Run cron from the first terminal:

      php admin/cli/cron.php

      5. Execute adhoc_task.php without limits:

      php admin/tool/task/cli/adhoc_task.php --execute --ignorelimits

      6. Confirm, that adhoc_task.php is processing the tasks:

      root@b24142f9f846:/siteroot# php admin/tool/task/cli/adhoc_task.php --execute --ignorelimits
      Server Time: Thu, 02 Jan 2020 04:17:39 +0000
      Execute adhoc task: tool_testtasks\task\timed_adhoc_task
      ... started 04:17:39. Current memory use 14.1MB.
      Starting adhoc task '4 of 50' wth duration: 10
      adhoc task running: 1/10 seconds
      adhoc task running: 2/10 seconds
      adhoc task running: 3/10 seconds
      

       

       

      Show
      Installation 1. Install this to make the testing easier: https: //github.com/catalyst/moodle-tool_testtasks Default settings 1. Clear adhoc task queue: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 2. Create 50 tasks php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 10 -n= 50 3. Open 3 terminals and run cron: php admin/cli/cron.php 4. Confirm , that all 3 runners are running queue_adhoc_tasks. Output should look like: Execute adhoc task: tool_testtasks\task\timed_adhoc_task ... started 02 : 32 : 14 . Current memory use 19 .5MB. Starting adhoc task '1 of 50' wth duration: 10 adhoc task running: 1 / 10 seconds adhoc task running: 2 / 10 seconds adhoc task running: 3 / 10 seconds adhoc task running: 4 / 10 seconds adhoc task running: 5 / 10 seconds 5. Open another terminal and run cron: php admin/cli/cron.php 6. Confirm , that while 3 runners are processing the tasks, one runner has exited without processing: root @5d15afd93e52 :/siteroot# php admin/cli/cron.php Server Time: Mon, 16 Dec 2019 06 : 25 : 26 + 0000 Skipping processing of adhoc tasks. Concurrency limit reached. Cron script completed correctly Cron completed at 06 : 25 : 29 . Memory used 18 .3MB. Execution took 2.186963 seconds Limit via default 1. Clear adhoc task queue: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 2. Set task_concurrency_limit_default to 2: php admin/cli/cfg.php --name=task_concurrency_limit_default --set= 2 3. Create 50 tasks php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 10 -n= 50 4. Open 3 terminals and run cron: php admin/cli/cron.php 5. Confirm , that while 2 runners are processing the tasks, one runner has exited without processing: root @b24142f9f846 :/siteroot# php admin/cli/cron.php Server Time: Thu, 02 Jan 2020 04 : 29 : 19 + 0000 Skipping \tool_testtasks\task\timed_adhoc_task adhoc task class as the per-task limit is reached. Ran 0 adhoc tasks found at Thu, 02 Jan 2020 04 : 29 : 19 + 0000 Cron script completed correctly Cron completed at 04 : 29 : 21 . Memory used 18 .4MB. Execution took 1.482996 seconds  6. Create 'another' type of adhoc task: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 1 -n= 100 -- class = 'tool_testtasks\task\another_timed_adhoc_task' 7. Open 3rd terminal (idling) and run cron: php admin/cli/cron.php 8. Confirm , that while 2 runners are processing 'tool_testtasks\task\timed_adhoc_task' tasks, third runner is processing 'tool_testtasks\task\another_timed_adhoc_task' task: Starting ANOTHER adhoc task '17 of 100' wth duration: 1 adhoc ANOTHER task running: 1 / 1 seconds Ending ANOTHER adhoc task '17 of 100' with duration: 1 ... used 0 dbqueries ... used 1.0005919933319 seconds Adhoc task complete: tool_testtasks\task\another_timed_adhoc_task Limit via per-task setting 1. Clear adhoc task queue: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 2. Set task_concurrency_limit_default to 3: php admin/cli/cfg.php --name=task_concurrency_limit_default --set= 3 3. Set per-task concurrency limit for 'tool_testtasks\task\timed_adhoc_task' task to 1 by adding this to config.php: $CFG->task_concurrency_limit = array( 'tool_testtasks\task\timed_adhoc_task' => 1 ); 4. Create 50 tasks php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 10 -n= 50 5. Open 2 terminals and run cron: php admin/cli/cron.php 6. Confirm , that while the first runner is processing the task, the second one has exited without processing: root @b24142f9f846 :/siteroot# php admin/cli/cron.php Server Time: Thu, 02 Jan 2020 04 : 31 : 44 + 0000 Skipping \tool_testtasks\task\timed_adhoc_task adhoc task class as the per-task limit is reached. Ran 0 adhoc tasks found at Thu, 02 Jan 2020 04 : 31 : 44 + 0000 Cron script completed correctly Cron completed at 04 : 31 : 45 . Memory used 18 .4MB. Execution took 1.056334 seconds Load testing: 1. Clear adhoc task queue: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 2. Set task_concurrency_limit_default to 3: php admin/cli/cfg.php --name=task_concurrency_limit_default --set= 3 3. Set per-task concurrency limit for 'tool_testtasks\task\timed_adhoc_task' task to 1 by adding this to config.php: $CFG->task_concurrency_limit = array( 'tool_testtasks\task\timed_adhoc_task' => 1 ); 4. Create 50 000 tasks: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 300 -n= 50000 5. Open 2 terminals and run cron: php admin/cli/cron.php 6. Confirm , that while the first runner is processing the task, the second one has exited without processing: root @b24142f9f846 :/siteroot# php admin/cli/cron.php Server Time: Thu, 02 Jan 2020 04 : 34 : 55 + 0000 Skipping \tool_testtasks\task\timed_adhoc_task adhoc task class as the per-task limit is reached. Ran 0 adhoc tasks found at Thu, 02 Jan 2020 04 : 34 : 55 + 0000 Cron script completed correctly Cron completed at 04 : 34 : 56 . Memory used 18 .9MB. Execution took 1.057899 seconds  7. Confirm , that inspite the huge task queue the second runner completed quickly (within 2 seconds): Execution took 1.057899 seconds Note : cron might run the full list of scheduled tasks and execution time might be greater (2-3 seconds). Re-run it shortly to make sure that ad-hock queue processing works quickly and doesn't take more than 2 seconds. 8. Create 'another' type of adhoc task: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 1 -n= 20 -- class = 'tool_testtasks\task\another_timed_adhoc_task' 9. Open 2nd terminal (idling) and run cron: php admin/cli/cron.php 10. Confirm , that the second runner is processing 'another' task: Starting ANOTHER adhoc task '20 of 20' wth duration: 1 adhoc ANOTHER task running: 1 / 1 seconds Ending ANOTHER adhoc task '20 of 20' with duration: 1 ... used 0 dbqueries ... used 1.0005798339844 seconds Adhoc task complete: tool_testtasks\task\another_timed_adhoc_task Cron script completed correctly Cron completed at 07 : 10 : 34 . Memory used 40 .8MB. Execution took 32.387464 seconds Unlimited run with --ignorelimits 1. Clear adhoc task queue: php admin/tool/testtasks/cli/clear_adhoc_task_queue.php 2. Set task_concurrency_limit_default to 1: php admin/cli/cfg.php --name=task_concurrency_limit_default --set= 1 3. Create 50 tasks: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d= 10 -n= 50 4. Run cron from the first terminal: php admin/cli/cron.php 5. Execute adhoc_task.php without limits: php admin/tool/task/cli/adhoc_task.php --execute --ignorelimits 6. Confirm , that adhoc_task.php is processing the tasks: root @b24142f9f846 :/siteroot# php admin/tool/task/cli/adhoc_task.php --execute --ignorelimits Server Time: Thu, 02 Jan 2020 04 : 17 : 39 + 0000 Execute adhoc task: tool_testtasks\task\timed_adhoc_task ... started 04 : 17 : 39 . Current memory use 14 .1MB. Starting adhoc task '4 of 50' wth duration: 10 adhoc task running: 1 / 10 seconds adhoc task running: 2 / 10 seconds adhoc task running: 3 / 10 seconds    
    • Affected Branches:
      MOODLE_37_STABLE, MOODLE_39_STABLE
    • Fixed Branches:
      MOODLE_39_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      master_MDL-64610

      Description

      MDL-64347 added limits to the number of concurrent task runners, but the suggestion was made to add a per-task concurrency limit for adhoc tasks too.

      Essentially we want to provide a way to set a limit for the number of times that an adhoc task can run at once (in the case of multiple instances of a single task).

      This becomes increasingly important as we increase the use of adhoc tasks.
      We are currently using them for privacy exports, and MDL-46881 seeks to move forum cron processing to adhoc tasks too. Additionally there is a suggested plan to move backup and restore to adhoc tasks.

      Each of these tasks has different load patterns. Whilst privacy tends to have a smaller number of heavy tasks, the forum will have a larger number of very lightweight tasks.

      This issue proposes to allow different concurrency limits for the different tasks.

      We need to decide:

      1. is it okay to only allow setting in config.php
      2. do we want the ability for adhoc tasks to have a way to specify a default concurrency
      3. do we want a UI to configure these?

      I think that we definitely want items 1, and 2. We will probably want item 3 (though we could potentially defer this).

      I do feel that this issue is highly important as the load patterns of different adhoc tasks will significantly change as we increase their usage in different areas.
      Without this functionality we may find that privacy exports lock all potential adhoc task runners and block any forum tasks as a result.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              mikhailgolenkov Mikhail Golenkov
              Reporter:
              dobedobedoh Andrew Nicols
              Peer reviewer:
              Dmitrii Metelkin
              Integrator:
              Jake Dallimore
              Tester:
              Janelle Barcega
              Participants:
              Component watchers:
              Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Sara Arjona (@sarjona), Víctor Déniz Falcón
              Votes:
              3 Vote for this issue
              Watchers:
              14 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Fix Release Date:
                8/Jun/20

                  Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 3 hours, 1 minute
                  3h 1m