Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-67596

Cron / adhoc task runners ramp up slowly for no reason

    XMLWordPrintable

Details

    • MOODLE_39_STABLE
    • MOODLE_37_STABLE, MOODLE_38_STABLE
    • MDL-67596-cron-ramp-up
    • Hide
      1. Install testtasks to make testing easier:

        git clone https://github.com/catalyst/moodle-tool_testtasks admin/tool/testtasks
        

      2. Install lockstats to make testing easier:

        git clone https://github.com/catalyst/moodle-tool_lockstats admin/tool/lockstats
        

      3. Add this to config.php

        $CFG->lock_factory = '\tool_lockstats\proxy_lock_factory';
        $CFG->proxied_lock_factory = 'auto';
        

      4. In theory allow cron to scale right up

        $CFG->task_adhoc_concurrency_limit = 1000;
        $CFG->task_scheduled_concurrency_limit = 1000;
        $CFG->task_adhoc_max_runtime = 6000;
        

      5. In terminal 1 watch for open locks, should be empty:

        watch -n 1 php admin/tool/lockstats/cli/list_locks.php
        

      6. In terminal 2 queue up a ton of adhoc tasks:

        php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=5 -n=900 
        

      7. In terminal 2 spawn a new adhoc runner process:

        php admin/tool/task/cli/adhoc_task.php --execute &
        

        1. Confirm that you see runner and adhoc tasks in terminal 1 within a few seconds
      8. Terminal 2: Add a few more runners by running the following:

        php admin/tool/task/cli/adhoc_task.php --execute &
        php admin/tool/task/cli/adhoc_task.php --execute &
        php admin/tool/task/cli/adhoc_task.php --execute &
        

        1. Terminal 1: Confirm that they appear in the task list immediately
      9. Kill all the background processes:

        php admin/cli/cron.php --stop
        

      10. Now go crazy and ramp it right up

        for i in {1..30}; do php admin/tool/task/cli/adhoc_task.php --execute & done
        

        1. Confirm that you see runner and adhoc tasks in terminal 1 within a few seconds 

      Note

      There are some race conditions in the lockstats plugin. You may seem some PHP errors pop up in the output as it flies by your screen.

      Show
      Install testtasks to make testing easier: git clone https://github.com/catalyst/moodle-tool_testtasks admin/tool/testtasks Install lockstats to make testing easier: git clone https://github.com/catalyst/moodle-tool_lockstats admin/tool/lockstats Add this to config.php $CFG->lock_factory = '\tool_lockstats\proxy_lock_factory'; $CFG->proxied_lock_factory = 'auto'; In theory allow cron to scale right up $CFG->task_adhoc_concurrency_limit = 1000; $CFG->task_scheduled_concurrency_limit = 1000; $CFG->task_adhoc_max_runtime = 6000; In terminal 1 watch for open locks, should be empty: watch -n 1 php admin/tool/lockstats/cli/list_locks.php In terminal 2 queue up a ton of adhoc tasks: php admin/tool/testtasks/cli/queue_adhoc_tasks.php -d=5 -n=900  In terminal 2 spawn a new adhoc runner process: php admin/tool/task/cli/adhoc_task.php --execute & Confirm that you see runner and adhoc tasks in terminal 1 within a few seconds Terminal 2: Add a few more runners by running the following: php admin/tool/task/cli/adhoc_task.php --execute & php admin/tool/task/cli/adhoc_task.php --execute & php admin/tool/task/cli/adhoc_task.php --execute & Terminal 1: Confirm that they appear in the task list immediately Kill all the background processes: php admin/cli/cron.php --stop Now go crazy and ramp it right up for i in {1..30}; do php admin/tool/task/cli/adhoc_task.php --execute & done Confirm that you see runner and adhoc tasks in terminal 1 within a few seconds   Note There are some race conditions in the lockstats plugin. You may seem some PHP errors pop up in the output as it flies by your screen.

    Description

      If you spawn say 30 cron processes, each process tries to grab a runner lock and waits a second for each before moving on to the next. This means that the 30th process will sit there for at least 30 seconds doing essentially nothing. The more processes the worse it gets, which make it hard to tune for lots of short lived processes.

      Attachments

        Issue Links

          Activity

            People

              brendanheywood Brendan Heywood
              brendanheywood Brendan Heywood
              Matt Porritt Matt Porritt
              Andrew Lyons Andrew Lyons
              Janelle Barcega Janelle Barcega
              Matteo Scaramuccia, Andrew Lyons, Huong Nguyen, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze, Stevani Andolo, Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Ilya Tregubov, Laurent David, Raquel Ortega, Sara Arjona (@sarjona)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                13/Jan/20

                Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 3 hours, 43 minutes
                  3h 43m