Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-50719

Long-running scheduled task slows down all cron processing / task contention and re-proccessing bugs

XMLWordPrintable

    • MOODLE_28_STABLE, MOODLE_29_STABLE, MOODLE_31_STABLE, MOODLE_32_STABLE
    • MOODLE_31_STABLE, MOODLE_32_STABLE
    • MDL-50719-fast-locks
    • Hide

      A test script can be found here:

      https://github.com/catalyst/moodle-tool_testtasks/blob/master/tests/test9.sh

      A test run with 16 ad hoc tasks shoudl complete is just over 16 seconds with 1 cron process:

      time admin/tool/testtasks/tests/test9.sh 1

      Runs with multiple processes should NOT result in duplicated processing of ad hoc tasks (or scheduled but this script doesn't test that explicitely). Additionally the overall total should be reduce from 16 seconds to say 10

      time admin/tool/testtasks/tests/test9.sh 2

      As you throw more cron processes at it the overall processing time should asymptotically flatten out, eg

      time admin/tool/testtasks/tests/test9.sh 4

      The scheduled tasks also need to be hammered at scale. You should not see the same task being processed within the same cron tick because of different front ends reprocessing the same task after gaining a lock that had previously been locked elsewhere. You should also not ever get cron processes waiting a long time to try and run a scheduled task which is already running elsewhere

      Show
      A test script can be found here: https://github.com/catalyst/moodle-tool_testtasks/blob/master/tests/test9.sh A test run with 16 ad hoc tasks shoudl complete is just over 16 seconds with 1 cron process: time admin/tool/testtasks/tests/test9.sh 1 Runs with multiple processes should NOT result in duplicated processing of ad hoc tasks (or scheduled but this script doesn't test that explicitely). Additionally the overall total should be reduce from 16 seconds to say 10 time admin/tool/testtasks/tests/test9.sh 2 As you throw more cron processes at it the overall processing time should asymptotically flatten out, eg time admin/tool/testtasks/tests/test9.sh 4 The scheduled tasks also need to be hammered at scale. You should not see the same task being processed within the same cron tick because of different front ends reprocessing the same task after gaining a lock that had previously been locked elsewhere. You should also not ever get cron processes waiting a long time to try and run a scheduled task which is already running elsewhere

      A long running scheduled task slows down all cron processing in the following way.

      get_next_scheduled_task gets scheduled tasks that are due to run in order by lastruntime. A long running task that will repeatedly come up in the list and the get_next_scheduled_task will attempt to get its lock before trying the next task in the list. The result is that each iteration of the scheduled tasks gets an extra 10 seconds or so of time as it waits (over and over, again) for the long-running task to time out.

      This also greatly increases the occurence of the locktimeout exception because the task lock attempt occurs within a cron lock.

            brendanheywood Brendan Heywood
            colin Colin Campbell
            Damyon Wiese Damyon Wiese
            Andrew Lyons Andrew Lyons
            Ankit Agarwal Ankit Agarwal
            Votes:
            9 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated:
              Resolved:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.