Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-67363

Add a Quality of Service layer to the processing of the ad-hoc task queue

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      0) Install this in admin/tool to make the testing easier:

      https://github.com/catalyst/moodle-tool_testtasks

       

      Testing scenario 1. Test the case with a single task runner

      1) Queue up 1000 one seconds tasks:

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=1 -n=1000
      

      2) Behind this queue up a single 'another' type of adhoc task:

      php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=1 -n=1 --class='tool_testtasks\task\another_timed_adhoc_task'

      3) Peek into the task queue by class:

      $ select count(*),classname from mdl_task_adhoc group by classname;
       count | classname 
      -------+-----------------------------------------------
       1000 | \tool_testtasks\task\timed_adhoc_task
       1 | \tool_testtasks\task\another_timed_adhoc_task
      (2 rows)

      4) Now start processing the queue:

      php admin/tool/task/cli/adhoc_task.php --execute 
      

      5) You should see that the first 2 tasks it picked off cycled through each type and very quickly you are left with a number lower than 1000 (i.e. 997):

      $ select count(*),classname from mdl_task_adhoc group by classname;
       count | classname 
      -------+---------------------------------------
       997 | \tool_testtasks\task\timed_adhoc_task
      (1 row)

       

      Testing scenario 2. A bunch of runners in parallel

      1) First, allow this to work in config.php:

      $CFG->task_adhoc_concurrency_limit = 1000;
      

      2) Queue up 1000 of type A and then another 1000 of type B behind it:

      $ php admin/tool/testtasks/cli/clear_adhoc_task_queue.php
      $ php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=1 -n=1000
      $ php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=1 -n=1000 --class='tool_testtasks\task\another_timed_adhoc_task' 
      

      3) Confirm the queues: 

      $ select count(*),classname from mdl_task_adhoc group by classname;
       count | classname 
      -------+-----------------------------------------------
       1000 | \tool_testtasks\task\timed_adhoc_task
       1000 | \tool_testtasks\task\another_timed_adhoc_task
      (2 rows)
      

      6) Now lets fire up several task runners, do this say 4 times (you may need to open several terminals in order to execute them):

      php admin/tool/task/cli/adhoc_task.php --execute &
      

      7) Recheck the queues and confirm they are being processed evenly (so the count column number is lower than in step #3):

      $ select count(*),classname from mdl_task_adhoc group by classname;
       count | classname 
      -------+-----------------------------------------------
       944 | \tool_testtasks\task\timed_adhoc_task
       944 | \tool_testtasks\task\another_timed_adhoc_task
      (2 rows) 
      

       

      Testing scenario 3. More cases [OPTIONAL]

      8) Lastly, and most importantly, keep throwing more and more runners at it and confirm that the overall system is still scaling up linearly the more processes you throw at it.

       
       

      Show
      0) Install this in admin/tool to make the testing easier: https://github.com/catalyst/moodle-tool_testtasks   Testing scenario 1. Test the case with a single task runner 1) Queue up 1000 one seconds tasks: php admin /tool/testtasks/cli/queue_adhoc_tasks .php -d=1 -n=1000 2) Behind this queue up a single 'another' type of adhoc task: php admin /tool/testtasks/cli/queue_adhoc_tasks .php -d=1 -n=1 --class= 'tool_testtasks\task\another_timed_adhoc_task' 3) Peek into the task queue by class: $ select count(*),classname from mdl_task_adhoc group by classname; count | classname -------+----------------------------------------------- 1000 | \tool_testtasks\task\timed_adhoc_task 1 | \tool_testtasks\task\another_timed_adhoc_task (2 rows) 4) Now start processing the queue: php admin /tool/task/cli/adhoc_task .php --execute  5) You should see that the first 2 tasks it picked off cycled through each type and very quickly you are left with a number lower than 1000 (i.e. 997): $ select count(*),classname from mdl_task_adhoc group by classname; count | classname -------+--------------------------------------- 997 | \tool_testtasks\task\timed_adhoc_task (1 row)   Testing scenario 2. A bunch of runners in parallel 1) First, allow this to work in config.php: $CFG ->task_adhoc_concurrency_limit = 1000; 2) Queue up 1000 of type A and then another 1000 of type B behind it: $ php admin /tool/testtasks/cli/clear_adhoc_task_queue .php $ php admin /tool/testtasks/cli/queue_adhoc_tasks .php -d=1 -n=1000 $ php admin /tool/testtasks/cli/queue_adhoc_tasks .php -d=1 -n=1000 --class= 'tool_testtasks\task\another_timed_adhoc_task'   3) Confirm the queues:  $ select count(*),classname from mdl_task_adhoc group by classname; count | classname -------+----------------------------------------------- 1000 | \tool_testtasks\task\timed_adhoc_task 1000 | \tool_testtasks\task\another_timed_adhoc_task ( 2 rows) 6) Now lets fire up several task runners, do this say 4 times (you may need to open several terminals in order to execute them): php admin /tool/task/cli/adhoc_task .php --execute & 7) Recheck the queues and confirm they are being processed evenly (so the count column number is lower than in step #3): $ select count(*),classname from mdl_task_adhoc group by classname; count | classname -------+----------------------------------------------- 944 | \tool_testtasks\task\timed_adhoc_task 944 | \tool_testtasks\task\another_timed_adhoc_task ( 2 rows)    Testing scenario 3. More cases [OPTIONAL] 8) Lastly, and most importantly, keep throwing more and more runners at it and confirm that the overall system is still scaling up linearly the more processes you throw at it.    
    • Affected Branches:
      MOODLE_39_STABLE
    • Fixed Branches:
      MOODLE_39_STABLE
    • Pull Master Branch:
      MDL-67363-adhoc-qos

      Description

      This is split from MDL-64610

      Suppose you have 10 ad hoc task runners and 1000 type A tasks in the queue, then we want to run 10 type A side by side.

      Suppose you have 10 ad hoc task runners and 1000 type B tasks in the queue, then we want to run 10 type B side by side.

      Suppose you have 10 ad hoc task runners and 500 A and then 500 B tasks afterwards in the queue, then we want to run 5 type A and 5 type B side by side regardless of the order they were queued in.

      Suppose you only have 1 ad hoc task runners and 500 A and then 500 B tasks afterwards in the queue, then we want to run 1 type A and then 1 type B and then 1 type A regardless of the order they were queued in.

       The intention behind MDL-64610 is more or less similar but the approach is quite different.

       

       

       

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              brendanheywood Brendan Heywood
              Reporter:
              brendanheywood Brendan Heywood
              Peer reviewer:
              Dmitrii Metelkin
              Integrator:
              Sara Arjona (@sarjona)
              Tester:
              Gladys Basiana
              Participants:
              Component watchers:
              Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Sara Arjona (@sarjona)
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Fix Release Date:
                15/Jun/20

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 hours, 40 minutes
                  2h 40m