Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-65072

All lock api implementations usleep even when the timeout is zero which limits ad hoc concurrency throughput

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      1) For ease of creating a massive amount of test ad hoc tasks install this admin tool

      https://github.com/catalyst/moodle-tool_testtasks

       

      2) For easy of visibility into seeing what tasks are running, install this admin tool:

      https://github.com/catalyst/moodle-tool_lockstats

      Then configure it:

      $CFG->lock_factory = "\\tool_lockstats\\proxy_lock_factory";
      $CFG->proxied_lock_factory = "\\core\\lock\\postgres_lock_factory";

       

      3) Remove the scaling throttle:

      $CFG->task_adhoc_concurrency_limit = 1000;
      $CFG->task_adhoc_max_runtime = 1000;

       

      3) Queue up a very large number of small ad hoc tasks, eg 1000 tasks which each take 1 second to run.

      /admin/tool/testtasks/cli $ php queue_adhoc_tasks.php -n=1000 -d=1

       

      4) Now fire up a single ad hoc task processor in the background:

      $ php admin/tool/task/cli/adhoc_task.php --execute &

       

      5) Note in the lock stats tool (/admin/tool/lockstats/) that there is usually 1 current task lock at any given time, reload the report a few times

       

      6) Repeat steps 4 & 5 adding more and more background processes. Without the patch in place you will naturally hit a limit of around 3-4 concurrent processors with an actual task lock. ie with 20 processes around 16-17 will be idle doing nothing in usleep.

      With the patch applied you should be able to scale up linearly as much as you want.

       

      7) Rinse and repeat with the 3 different lock implementations

       

      Show
      1) For ease of creating a massive amount of test ad hoc tasks install this admin tool https://github.com/catalyst/moodle-tool_testtasks   2) For easy of visibility into seeing what tasks are running, install this admin tool: https://github.com/catalyst/moodle-tool_lockstats Then configure it: $CFG->lock_factory = "\\tool_lockstats\\proxy_lock_factory" ; $CFG->proxied_lock_factory = "\\core\\lock\\postgres_lock_factory" ;   3) Remove the scaling throttle: $CFG->task_adhoc_concurrency_limit = 1000 ; $CFG->task_adhoc_max_runtime = 1000 ;   3) Queue up a very large number of small ad hoc tasks, eg 1000 tasks which each take 1 second to run. /admin/tool/testtasks/cli $ php queue_adhoc_tasks.php -n=1000 -d=1   4) Now fire up a single ad hoc task processor in the background: $ php admin/tool/task/cli/adhoc_task.php --execute &   5) Note in the lock stats tool (/admin/tool/lockstats/) that there is usually 1 current task lock at any given time, reload the report a few times   6) Repeat steps 4 & 5 adding more and more background processes. Without the patch in place you will naturally hit a limit of around 3-4 concurrent processors with an actual task lock. ie with 20 processes around 16-17 will be idle doing nothing in usleep. With the patch applied you should be able to scale up linearly as much as you want.   7) Rinse and repeat with the 3 different lock implementations  
    • Affected Branches:
      MOODLE_35_STABLE, MOODLE_36_STABLE, MOODLE_37_STABLE
    • Fixed Branches:
      MOODLE_37_STABLE
    • Pull Master Branch:
      MDL-65072-fast-locks

      Description

      All of the lock implementations that ship with core all wait for a random time 100-250ms even if the timeout is zero. This generally goes unnoticed but has a large impact on the throughput of the ad hoc task processor when run at scale, and effectively caps the level of concurrency that can be achieved.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                brendanheywood Brendan Heywood
                Reporter:
                brendanheywood Brendan Heywood
                Peer reviewer:
                Matt Porritt
                Integrator:
                Eloy Lafuente (stronk7)
                Tester:
                Eloy Lafuente (stronk7)
                Participants:
                Component watchers:
                Matteo Scaramuccia, Jake Dallimore, Jun Pataleta, Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Sara Arjona (@sarjona), Víctor Déniz Falcón
              • Votes:
                2 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  20/May/19

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 hour, 35 minutes
                  1h 35m