Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-50719

Long-running scheduled task slows down all cron processing / task contention and re-proccessing bugs

    Details

    • Testing Instructions:
      Hide

      A test script can be found here:

      https://github.com/catalyst/moodle-tool_testtasks/blob/master/tests/test9.sh

      A test run with 16 ad hoc tasks shoudl complete is just over 16 seconds with 1 cron process:

      time admin/tool/testtasks/tests/test9.sh 1

      Runs with multiple processes should NOT result in duplicated processing of ad hoc tasks (or scheduled but this script doesn't test that explicitely). Additionally the overall total should be reduce from 16 seconds to say 10

      time admin/tool/testtasks/tests/test9.sh 2

      As you throw more cron processes at it the overall processing time should asymptotically flatten out, eg

      time admin/tool/testtasks/tests/test9.sh 4

      The scheduled tasks also need to be hammered at scale. You should not see the same task being processed within the same cron tick because of different front ends reprocessing the same task after gaining a lock that had previously been locked elsewhere. You should also not ever get cron processes waiting a long time to try and run a scheduled task which is already running elsewhere

      Show
      A test script can be found here: https://github.com/catalyst/moodle-tool_testtasks/blob/master/tests/test9.sh A test run with 16 ad hoc tasks shoudl complete is just over 16 seconds with 1 cron process: time admin/tool/testtasks/tests/test9.sh 1 Runs with multiple processes should NOT result in duplicated processing of ad hoc tasks (or scheduled but this script doesn't test that explicitely). Additionally the overall total should be reduce from 16 seconds to say 10 time admin/tool/testtasks/tests/test9.sh 2 As you throw more cron processes at it the overall processing time should asymptotically flatten out, eg time admin/tool/testtasks/tests/test9.sh 4 The scheduled tasks also need to be hammered at scale. You should not see the same task being processed within the same cron tick because of different front ends reprocessing the same task after gaining a lock that had previously been locked elsewhere. You should also not ever get cron processes waiting a long time to try and run a scheduled task which is already running elsewhere
    • Affected Branches:
      MOODLE_28_STABLE, MOODLE_29_STABLE, MOODLE_31_STABLE, MOODLE_32_STABLE
    • Fixed Branches:
      MOODLE_31_STABLE, MOODLE_32_STABLE
    • Pull 3.1 Branch:
      MDL-50719-fast-locks-MOODLE_31_STABLE
    • Pull Master Branch:
      MDL-50719-fast-locks

      Description

      A long running scheduled task slows down all cron processing in the following way.

      get_next_scheduled_task gets scheduled tasks that are due to run in order by lastruntime. A long running task that will repeatedly come up in the list and the get_next_scheduled_task will attempt to get its lock before trying the next task in the list. The result is that each iteration of the scheduled tasks gets an extra 10 seconds or so of time as it waits (over and over, again) for the long-running task to time out.

      This also greatly increases the occurence of the locktimeout exception because the task lock attempt occurs within a cron lock.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Votes:
                  9 Vote for this issue
                  Watchers:
                  18 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:
                    Fix Release Date:
                    13/Mar/17