Moodle
  1. Moodle
  2. MDL-25499

Centralise management of all types of cron tasks with registration, scheduling, parallel task conflicts(blocking) and running once off tasks, all using an administration screen.

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1, 2.6
    • Fix Version/s: DEV backlog
    • Component/s: Administration
    • Labels:
    • Affected Branches:
      MOODLE_21_STABLE, MOODLE_26_STABLE
    • Rank:
      2807

      Description

      The document proposing this is http://docs.moodle.org/en/Development:Scheduled_Tasks_Proposal.

      related bugs

        Issue Links

          Activity

          Hide
          Penny Leach added a comment -

          you probably want to re-audit all the existing cronjobs, since that audit was done like in December....

          Show
          Penny Leach added a comment - you probably want to re-audit all the existing cronjobs, since that audit was done like in December....
          Hide
          Tim Hunt added a comment -
          Show
          Tim Hunt added a comment - Developer chat about this: http://moodle.org/mod/cvsadmin/view.php?conversationid=6219#c242871
          Hide
          Hubert Chathi added a comment -

          I mentioned this in the dev chat, but JFTR, we are implementing a scheduled tasks system based on this proposed system, and I'm planning on sharing our code when we're done. We won't implement all of it (e.g., we won't be converting the existing cron system to use the new system), but it should be a good start.

          Show
          Hubert Chathi added a comment - I mentioned this in the dev chat, but JFTR, we are implementing a scheduled tasks system based on this proposed system, and I'm planning on sharing our code when we're done. We won't implement all of it (e.g., we won't be converting the existing cron system to use the new system), but it should be a good start.
          Hide
          Hubert Chathi added a comment -

          Our implementation of part of this for 1.9:

          We have also patched /admin/cron.php to call our main cron function: https://github.com/remotelearner/elis.base/blob/master/patches/admin/cron.php.patch

          So far, we have only implemented the part that schedules tasks to run at certain times (MDL-25507). We have not yet done locking, and our version of "black magic" currently is just "sort by next run time".

          We will update when we release our 2.1 code, and when we implement locking (and more advanced black magic).

          Show
          Hubert Chathi added a comment - Our implementation of part of this for 1.9: The actual cron function: https://github.com/remotelearner/elis.base/blob/master/core/elis/core/cron.php Library functions (e.g. for loading [plugin]/db/tasks.db): https://github.com/remotelearner/elis.base/blob/master/core/elis/core/lib/tasklib.php Database schema: elis_scheduled_tasks table from https://github.com/remotelearner/elis.base/blob/master/core/elis/core/db/install.xml We have also patched /admin/cron.php to call our main cron function: https://github.com/remotelearner/elis.base/blob/master/patches/admin/cron.php.patch So far, we have only implemented the part that schedules tasks to run at certain times ( MDL-25507 ). We have not yet done locking, and our version of "black magic" currently is just "sort by next run time". We will update when we release our 2.1 code, and when we implement locking (and more advanced black magic).
          Hide
          Simon Coggins added a comment -

          In Totara we have some code that uses OS-level file locking to lock the cron file to prevent it running twice at the same time. The advantage of this approach is that the lock is automatically removed if the cron crashes part way through.

          We have also implemented a cron watcher script that can catch long running processes and inform the administrator, or kill the process if it exceeds a (configurable) time period.

          Happy to share the code if it would be of use.

          Simon

          Show
          Simon Coggins added a comment - In Totara we have some code that uses OS-level file locking to lock the cron file to prevent it running twice at the same time. The advantage of this approach is that the lock is automatically removed if the cron crashes part way through. We have also implemented a cron watcher script that can catch long running processes and inform the administrator, or kill the process if it exceeds a (configurable) time period. Happy to share the code if it would be of use. Simon
          Hide
          Michael Hughes added a comment -

          I've started another thread about this on the forums (https://moodle.org/mod/forum/discuss.php?d=229139) following on from the discussion around MDL-17783.

          I think this is something that really needs to be looked at as the simple approach of only permitting a single cron instance at a time really can cause bottlenecks when cron functions start taking longer than the initiation period (e.g. every 10 minutes).

          Show
          Michael Hughes added a comment - I've started another thread about this on the forums ( https://moodle.org/mod/forum/discuss.php?d=229139 ) following on from the discussion around MDL-17783 . I think this is something that really needs to be looked at as the simple approach of only permitting a single cron instance at a time really can cause bottlenecks when cron functions start taking longer than the initiation period (e.g. every 10 minutes).
          Hide
          Damyon Wiese added a comment -

          I need this implemented so I can start doing document conversions in cron for mod_assign. I've already added code for the locking subtask - and will continue working through these subtasks...

          Show
          Damyon Wiese added a comment - I need this implemented so I can start doing document conversions in cron for mod_assign. I've already added code for the locking subtask - and will continue working through these subtasks...
          Hide
          Petr Škoda added a comment -

          Do I read the spec and code right? It seems it moves away from periodic execution (every 5 minutes for example) to scheduled execution only (at 5th minute of every hour). This looks like a serious regression to me, I expected you would be able to do both.

          Also at preset we have dependency, for example auth should be always executed before enrol. How is that going nto be described in task definition and how is it going to be implemented?

          Maybe the task definition array could include word 'default' to clearly ondicate the valies can be later changed later in admin UI.

          Anyway thanks a lot for working on this long standing issue!

          Show
          Petr Škoda added a comment - Do I read the spec and code right? It seems it moves away from periodic execution (every 5 minutes for example) to scheduled execution only (at 5th minute of every hour). This looks like a serious regression to me, I expected you would be able to do both. Also at preset we have dependency, for example auth should be always executed before enrol. How is that going nto be described in task definition and how is it going to be implemented? Maybe the task definition array could include word 'default' to clearly ondicate the valies can be later changed later in admin UI. Anyway thanks a lot for working on this long standing issue!
          Hide
          Hubert Chathi added a comment -

          Petr: are you referring to this spec http://docs.moodle.org/dev/Scheduled_Tasks_Proposal ? It moves towards compatibility with the UNIX cron worldview which, you are right, does not have an exact equivalent to "run every 5 minutes", but if you use "*/5" in the minutes field, it will run at the 0th, 5th, 10th, 15th, 20th, etc. minute of every hour, which is a close approximation. I haven't read the code carefully, so I can't say whether it implements that correctly.

          Damyon, it looks like you wrote your own cron format-parsing functions. Is there any particular reason you chose to do that instead of using the functions written for Mahara? AFAICT, the Mahara functions have already been well tested, and I don't know about you, but I prefer using something that's already been written and tested when dealing with text parsing.

          Show
          Hubert Chathi added a comment - Petr: are you referring to this spec http://docs.moodle.org/dev/Scheduled_Tasks_Proposal ? It moves towards compatibility with the UNIX cron worldview which, you are right, does not have an exact equivalent to "run every 5 minutes", but if you use "*/5" in the minutes field, it will run at the 0th, 5th, 10th, 15th, 20th, etc. minute of every hour, which is a close approximation. I haven't read the code carefully, so I can't say whether it implements that correctly. Damyon, it looks like you wrote your own cron format-parsing functions. Is there any particular reason you chose to do that instead of using the functions written for Mahara? AFAICT, the Mahara functions have already been well tested, and I don't know about you, but I prefer using something that's already been written and tested when dealing with text parsing.
          Hide
          Petr Škoda added a comment -

          "/5" looks fine! How could I miss that? There is one thing that worries me, we should not align all cron task to specific time because that would kill servers that host multiple Moodle instances.

          Another thing I do not understand is how you are going to actually execute the tasks, PHP does not have threads, something would have to be calling admin/cron.php or executing admin/cli/cron.php at least once a minute to get some parallel execution.

          Another thing I miss in the proposal is task priority, on sites that for some reason cannot call cron every minute there will be a queue of missed tasks, I guess that doing it "the first scheduled is executed first" is not good enough, at present we have strict order/priorities.

          Show
          Petr Škoda added a comment - "/5" looks fine! How could I miss that? There is one thing that worries me, we should not align all cron task to specific time because that would kill servers that host multiple Moodle instances. Another thing I do not understand is how you are going to actually execute the tasks, PHP does not have threads, something would have to be calling admin/cron.php or executing admin/cli/cron.php at least once a minute to get some parallel execution. Another thing I miss in the proposal is task priority, on sites that for some reason cannot call cron every minute there will be a queue of missed tasks, I guess that doing it "the first scheduled is executed first" is not good enough, at present we have strict order/priorities.
          Hide
          Petr Škoda added a comment -

          Alternatively we could also hijack normal we requests and after the normal execution is finished we could close session, ignore user abort and continue executing stuff.

          Show
          Petr Škoda added a comment - Alternatively we could also hijack normal we requests and after the normal execution is finished we could close session, ignore user abort and continue executing stuff.
          Hide
          Hubert Chathi added a comment -

          Yes, when I wrote the ELIS code, the scheduled tasks code was called by the system cron every 5 minutes. It calculated the next run time for each task, and when it was called by the system cron, it would check whether the current time was past the next run time. I haven't looked into how Damyon's code works.

          I believe that we talked about task priority at the Czech hackfest, and it got lumped into the "black magic" in the spec. I think the big thing missing from the spec is hashing out exactly what's included in the "black magic" and how it works.

          Show
          Hubert Chathi added a comment - Yes, when I wrote the ELIS code, the scheduled tasks code was called by the system cron every 5 minutes. It calculated the next run time for each task, and when it was called by the system cron, it would check whether the current time was past the next run time. I haven't looked into how Damyon's code works. I believe that we talked about task priority at the Czech hackfest, and it got lumped into the "black magic" in the spec. I think the big thing missing from the spec is hashing out exactly what's included in the "black magic" and how it works.
          Hide
          Petr Škoda added a comment -

          Thanks for the info Hubert.

          Show
          Petr Škoda added a comment - Thanks for the info Hubert.
          Hide
          Tyler Bannister added a comment -

          There are also a number of "optional" cron scripts in Moodle, that could also be converted to using this system. A number of the core auth classes (cas, db, ldap) have sync scripts and a number of the core enrol plugins have sync scripts too (category, cohort, database, flatfile, ldap, manual, meta, paypal, and self). It seems like these occasionally cause problems for users who don't realize they need to add them to a cron configuration. How would these scripts fit into this issue, if at all?

          Should the screens to be developed by MDL-25505 list any CLI scripts belonging to enabled auth and enrol plugins with an option to schedule them to be run from the regular cron?

          Show
          Tyler Bannister added a comment - There are also a number of "optional" cron scripts in Moodle, that could also be converted to using this system. A number of the core auth classes (cas, db, ldap) have sync scripts and a number of the core enrol plugins have sync scripts too (category, cohort, database, flatfile, ldap, manual, meta, paypal, and self). It seems like these occasionally cause problems for users who don't realize they need to add them to a cron configuration. How would these scripts fit into this issue, if at all? Should the screens to be developed by MDL-25505 list any CLI scripts belonging to enabled auth and enrol plugins with an option to schedule them to be run from the regular cron?
          Hide
          Paul Nicholls added a comment -

          I like the idea of being able to configure the auth and enrol sync scripts from the web UI. At certain times of the year, we ramp up the frequency at which they run for a while - it'd be great to be able to do this without needing to manually edit the crontab.

          Show
          Paul Nicholls added a comment - I like the idea of being able to configure the auth and enrol sync scripts from the web UI. At certain times of the year, we ramp up the frequency at which they run for a while - it'd be great to be able to do this without needing to manually edit the crontab.
          Hide
          Damyon Wiese added a comment -

          Sorry for not responding - I just realised I hadn't added myself as a watcher to the meta issue.

          Petr: Re aligning to a specific time - this will only do anything when cron runs. For multiple sites - you could stagger when cron actually runs, or configure your tasks so in the admin interface so they don't all run at the same time.

          Re: parallel cron - yes about once per minute is how often I was thinking this would run. You could do it even more often if you like and the work will be shared.

          -1 for highjacking the end of user requests for cron - in a cluster you do not want cron running on the user facing nodes. Also - when a site is least busy (3am) is when you want to do lots of cron stuff.

          Hubert - this code does exactly what you described - each task calculates when it should run next and each time cron runs it only runs the tasks that are due. If multiple crons are running - they will not get handed the same tasks due to the locking.

          Re: Priority - I think requiring a defined order for cron tasks misses the point of allowing parallel cron. Even with the current system there is the chance e.g. for users signing up between the auth and enrol steps in cron - this is life. They will just have to wait 5 minutes! If cron is running much more often - this is less of an issue.

          Re: optional scripts - yes all "optional" scripts should be run by this new system and we should not have lots of different crons for different purposes. The scripts will no longer get blocked by long running crons and do not have to implement their own scheduling. There will be only one cli cron script.

          Show
          Damyon Wiese added a comment - Sorry for not responding - I just realised I hadn't added myself as a watcher to the meta issue. Petr: Re aligning to a specific time - this will only do anything when cron runs. For multiple sites - you could stagger when cron actually runs, or configure your tasks so in the admin interface so they don't all run at the same time. Re: parallel cron - yes about once per minute is how often I was thinking this would run. You could do it even more often if you like and the work will be shared. -1 for highjacking the end of user requests for cron - in a cluster you do not want cron running on the user facing nodes. Also - when a site is least busy (3am) is when you want to do lots of cron stuff. Hubert - this code does exactly what you described - each task calculates when it should run next and each time cron runs it only runs the tasks that are due. If multiple crons are running - they will not get handed the same tasks due to the locking. Re: Priority - I think requiring a defined order for cron tasks misses the point of allowing parallel cron. Even with the current system there is the chance e.g. for users signing up between the auth and enrol steps in cron - this is life. They will just have to wait 5 minutes! If cron is running much more often - this is less of an issue. Re: optional scripts - yes all "optional" scripts should be run by this new system and we should not have lots of different crons for different purposes. The scripts will no longer get blocked by long running crons and do not have to implement their own scheduling. There will be only one cli cron script.
          Hide
          Aparup Banerjee added a comment -

          linking a recent badge cron slowness issue. we had to hackily disable cron for just this.
          It would help if some more priority went towards this issue so that future cron issues in general to moodle sites can be more properly managed.

          Show
          Aparup Banerjee added a comment - linking a recent badge cron slowness issue. we had to hackily disable cron for just this. It would help if some more priority went towards this issue so that future cron issues in general to moodle sites can be more properly managed.
          Hide
          Aparup Banerjee added a comment -

          assigning this meta to Damyon as he's acting on the sub-tasks.

          Show
          Aparup Banerjee added a comment - assigning this meta to Damyon as he's acting on the sub-tasks.
          Hide
          David Mudrak added a comment -

          Hi Damyon. I just spotted strings like taskscheduleday_help that reads

          Day of month field for task schedule. The field uses the same format as unix cron. Some examples are:<br/><ul><li><strong>*</strong> Every day</li><li><strong>*/2</strong> Every 2nd day</li><li><strong>1</strong> The first of every month</li><li><strong>1,​15</strong> The first and fifteenth of every month</li></ul>
          

          Please note that all *_help and *_desc strings have native support for Markdown formatting to make the translation easier. You may wish to change them to something like

          Day of month field for task schedule. The field uses the same format as unix cron. Some examples are:
          
          * '''*''' Every day
          * '''*/2''' Every 2nd day
          * '''1''' The first of every month
          * '''1,​15''' The first and fifteenth of every month
          

          to achieve the same result with less HTML.

          Show
          David Mudrak added a comment - Hi Damyon. I just spotted strings like taskscheduleday_help that reads Day of month field for task schedule. The field uses the same format as unix cron. Some examples are:<br/><ul><li><strong>*</strong> Every day</li><li><strong>*/2</strong> Every 2nd day</li><li><strong>1</strong> The first of every month</li><li><strong>1,​15</strong> The first and fifteenth of every month</li></ul> Please note that all *_help and *_desc strings have native support for Markdown formatting to make the translation easier. You may wish to change them to something like Day of month field for task schedule. The field uses the same format as unix cron. Some examples are: * '''*''' Every day * '''*/2''' Every 2nd day * '''1''' The first of every month * '''1,​15''' The first and fifteenth of every month to achieve the same result with less HTML.

            People

            • Votes:
              12 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

              • Created:
                Updated: