Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-50009

Prevent scheduled tasks from leaving unfinished db transactions

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide
      1. Install the attached test plugin and go to /admin/tool/mdl50009 to try to break your task queue
      2. Run your cron job
      3. Confirm that the task causes an error message to appear in the cron log
      4. Confirm that the record in task_adhoc hasn't been removed after the cron job finishes
      5. Raise some other task and repeat
      6. Confirm that the second task hasn't been prevented from running by the broken first task.
      Show
      Install the attached test plugin and go to /admin/tool/mdl50009 to try to break your task queue Run your cron job Confirm that the task causes an error message to appear in the cron log Confirm that the record in task_adhoc hasn't been removed after the cron job finishes Raise some other task and repeat Confirm that the second task hasn't been prevented from running by the broken first task.
    • Affected Branches:
      MOODLE_28_STABLE, MOODLE_29_STABLE
    • Fixed Branches:
      MOODLE_30_STABLE
    • Pull Master Branch:

      Description

      It is possible for a badly-coded task class to cause havoc with the scheduled and ad-hoc task system. Specifically, if a task starts a transaction and doesn't allow commit on it, anything done by that run of the cron job will be rolled back when the script finishes running, so:

      • no tasks will ever be completed and removed successfully from the queue
      • any tasks which do anything outside the database will do it every time the cron runs (forever)

      At worst, the cron script should probably check for started transactions before and after running a task, and log an error message and roll back if one is found. Ideally, each task should have its own transaction so it is unable to break the task processing.

      I realise you could take the hard line that systems administrators should ensure that all plugins containing tasks always complete transactions correctly in all possible code paths, etc, (which is mostly true of course!) but I don't think this is reasonable in this case, for a few reasons:

      • the cron log shows no sign of the error so it appears that it has run correctly (there is a message in the error_log, but not the cron log)
      • there is no user interface for monitoring the ad-hoc task queue, so you may only notice the problem if you were actively monitoring the database table
      • many institutions are required to use a particular third-party plugin, and may not be able to change the code without invalidating a support contract. In these cases, you expect the problems caused by buggy code to be limited to the plugin's own functionality rather than affecting the whole system.

      This last issue is one we have encountered with an integration with a commercial service in the past. There were various bugs in its event handling which resulted in the (2.6) events queue becoming blocked, which luckily only affected that particular plugin. This would have been a nightmare if the event queue had become blocked for all plugins, which is broadly similar to what is happening with the task queue at the moment.

      (Note - filing this against "other" component as there is currently no component for the task API)

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  16/Nov/15