Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-50719

Long-running scheduled task slows down all cron processing / task contention and re-proccessing bugs

    XMLWordPrintable

Details

    • MOODLE_28_STABLE, MOODLE_29_STABLE, MOODLE_31_STABLE, MOODLE_32_STABLE
    • MOODLE_31_STABLE, MOODLE_32_STABLE
    • MDL-50719-fast-locks
    • Hide

      A test script can be found here:

      https://github.com/catalyst/moodle-tool_testtasks/blob/master/tests/test9.sh

      A test run with 16 ad hoc tasks shoudl complete is just over 16 seconds with 1 cron process:

      time admin/tool/testtasks/tests/test9.sh 1

      Runs with multiple processes should NOT result in duplicated processing of ad hoc tasks (or scheduled but this script doesn't test that explicitely). Additionally the overall total should be reduce from 16 seconds to say 10

      time admin/tool/testtasks/tests/test9.sh 2

      As you throw more cron processes at it the overall processing time should asymptotically flatten out, eg

      time admin/tool/testtasks/tests/test9.sh 4

      The scheduled tasks also need to be hammered at scale. You should not see the same task being processed within the same cron tick because of different front ends reprocessing the same task after gaining a lock that had previously been locked elsewhere. You should also not ever get cron processes waiting a long time to try and run a scheduled task which is already running elsewhere

      Show
      A test script can be found here: https://github.com/catalyst/moodle-tool_testtasks/blob/master/tests/test9.sh A test run with 16 ad hoc tasks shoudl complete is just over 16 seconds with 1 cron process: time admin/tool/testtasks/tests/test9.sh 1 Runs with multiple processes should NOT result in duplicated processing of ad hoc tasks (or scheduled but this script doesn't test that explicitely). Additionally the overall total should be reduce from 16 seconds to say 10 time admin/tool/testtasks/tests/test9.sh 2 As you throw more cron processes at it the overall processing time should asymptotically flatten out, eg time admin/tool/testtasks/tests/test9.sh 4 The scheduled tasks also need to be hammered at scale. You should not see the same task being processed within the same cron tick because of different front ends reprocessing the same task after gaining a lock that had previously been locked elsewhere. You should also not ever get cron processes waiting a long time to try and run a scheduled task which is already running elsewhere

    Description

      A long running scheduled task slows down all cron processing in the following way.

      get_next_scheduled_task gets scheduled tasks that are due to run in order by lastruntime. A long running task that will repeatedly come up in the list and the get_next_scheduled_task will attempt to get its lock before trying the next task in the list. The result is that each iteration of the scheduled tasks gets an extra 10 seconds or so of time as it waits (over and over, again) for the long-running task to time out.

      This also greatly increases the occurence of the locktimeout exception because the task lock attempt occurs within a cron lock.

      Attachments

        Issue Links

          Activity

            People

              brendanheywood Brendan Heywood
              colin Colin Campbell
              Damyon Wiese Damyon Wiese
              Andrew Lyons Andrew Lyons
              Ankit Agarwal Ankit Agarwal
              Andrew Lyons, Huong Nguyen, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze
              Votes:
              9 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                13/Mar/17