Moodle
  1. Moodle
  2. MDL-25499

Centralise management of all types of cron tasks with registration, scheduling, parallel task conflicts(blocking) and running once off tasks, all using an administration screen.

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.1, 2.6
    • Fix Version/s: DEV backlog
    • Component/s: Administration
    • Labels:
    • Affected Branches:
      MOODLE_21_STABLE, MOODLE_26_STABLE

      Description

      The document proposing this is http://docs.moodle.org/en/Development:Scheduled_Tasks_Proposal.

      related bugs

        Gliffy Diagrams

          Issue Links

            Activity

            Hide
            Penny Leach added a comment -

            you probably want to re-audit all the existing cronjobs, since that audit was done like in December....

            Show
            Penny Leach added a comment - you probably want to re-audit all the existing cronjobs, since that audit was done like in December....
            Hide
            Tim Hunt added a comment -
            Show
            Tim Hunt added a comment - Developer chat about this: http://moodle.org/mod/cvsadmin/view.php?conversationid=6219#c242871
            Hide
            Hubert Chathi added a comment -

            I mentioned this in the dev chat, but JFTR, we are implementing a scheduled tasks system based on this proposed system, and I'm planning on sharing our code when we're done. We won't implement all of it (e.g., we won't be converting the existing cron system to use the new system), but it should be a good start.

            Show
            Hubert Chathi added a comment - I mentioned this in the dev chat, but JFTR, we are implementing a scheduled tasks system based on this proposed system, and I'm planning on sharing our code when we're done. We won't implement all of it (e.g., we won't be converting the existing cron system to use the new system), but it should be a good start.
            Hide
            Hubert Chathi added a comment -

            Our implementation of part of this for 1.9:

            We have also patched /admin/cron.php to call our main cron function: https://github.com/remotelearner/elis.base/blob/master/patches/admin/cron.php.patch

            So far, we have only implemented the part that schedules tasks to run at certain times (MDL-25507). We have not yet done locking, and our version of "black magic" currently is just "sort by next run time".

            We will update when we release our 2.1 code, and when we implement locking (and more advanced black magic).

            Show
            Hubert Chathi added a comment - Our implementation of part of this for 1.9: The actual cron function: https://github.com/remotelearner/elis.base/blob/master/core/elis/core/cron.php Library functions (e.g. for loading [plugin]/db/tasks.db): https://github.com/remotelearner/elis.base/blob/master/core/elis/core/lib/tasklib.php Database schema: elis_scheduled_tasks table from https://github.com/remotelearner/elis.base/blob/master/core/elis/core/db/install.xml We have also patched /admin/cron.php to call our main cron function: https://github.com/remotelearner/elis.base/blob/master/patches/admin/cron.php.patch So far, we have only implemented the part that schedules tasks to run at certain times ( MDL-25507 ). We have not yet done locking, and our version of "black magic" currently is just "sort by next run time". We will update when we release our 2.1 code, and when we implement locking (and more advanced black magic).
            Hide
            Simon Coggins added a comment -

            In Totara we have some code that uses OS-level file locking to lock the cron file to prevent it running twice at the same time. The advantage of this approach is that the lock is automatically removed if the cron crashes part way through.

            We have also implemented a cron watcher script that can catch long running processes and inform the administrator, or kill the process if it exceeds a (configurable) time period.

            Happy to share the code if it would be of use.

            Simon

            Show
            Simon Coggins added a comment - In Totara we have some code that uses OS-level file locking to lock the cron file to prevent it running twice at the same time. The advantage of this approach is that the lock is automatically removed if the cron crashes part way through. We have also implemented a cron watcher script that can catch long running processes and inform the administrator, or kill the process if it exceeds a (configurable) time period. Happy to share the code if it would be of use. Simon
            Hide
            Michael Hughes added a comment -

            I've started another thread about this on the forums (https://moodle.org/mod/forum/discuss.php?d=229139) following on from the discussion around MDL-17783.

            I think this is something that really needs to be looked at as the simple approach of only permitting a single cron instance at a time really can cause bottlenecks when cron functions start taking longer than the initiation period (e.g. every 10 minutes).

            Show
            Michael Hughes added a comment - I've started another thread about this on the forums ( https://moodle.org/mod/forum/discuss.php?d=229139 ) following on from the discussion around MDL-17783 . I think this is something that really needs to be looked at as the simple approach of only permitting a single cron instance at a time really can cause bottlenecks when cron functions start taking longer than the initiation period (e.g. every 10 minutes).
            Hide
            Damyon Wiese added a comment -

            I need this implemented so I can start doing document conversions in cron for mod_assign. I've already added code for the locking subtask - and will continue working through these subtasks...

            Show
            Damyon Wiese added a comment - I need this implemented so I can start doing document conversions in cron for mod_assign. I've already added code for the locking subtask - and will continue working through these subtasks...
            Hide
            Petr Skoda added a comment -

            Do I read the spec and code right? It seems it moves away from periodic execution (every 5 minutes for example) to scheduled execution only (at 5th minute of every hour). This looks like a serious regression to me, I expected you would be able to do both.

            Also at preset we have dependency, for example auth should be always executed before enrol. How is that going nto be described in task definition and how is it going to be implemented?

            Maybe the task definition array could include word 'default' to clearly ondicate the valies can be later changed later in admin UI.

            Anyway thanks a lot for working on this long standing issue!

            Show
            Petr Skoda added a comment - Do I read the spec and code right? It seems it moves away from periodic execution (every 5 minutes for example) to scheduled execution only (at 5th minute of every hour). This looks like a serious regression to me, I expected you would be able to do both. Also at preset we have dependency, for example auth should be always executed before enrol. How is that going nto be described in task definition and how is it going to be implemented? Maybe the task definition array could include word 'default' to clearly ondicate the valies can be later changed later in admin UI. Anyway thanks a lot for working on this long standing issue!
            Hide
            Hubert Chathi added a comment -

            Petr: are you referring to this spec http://docs.moodle.org/dev/Scheduled_Tasks_Proposal ? It moves towards compatibility with the UNIX cron worldview which, you are right, does not have an exact equivalent to "run every 5 minutes", but if you use "*/5" in the minutes field, it will run at the 0th, 5th, 10th, 15th, 20th, etc. minute of every hour, which is a close approximation. I haven't read the code carefully, so I can't say whether it implements that correctly.

            Damyon, it looks like you wrote your own cron format-parsing functions. Is there any particular reason you chose to do that instead of using the functions written for Mahara? AFAICT, the Mahara functions have already been well tested, and I don't know about you, but I prefer using something that's already been written and tested when dealing with text parsing.

            Show
            Hubert Chathi added a comment - Petr: are you referring to this spec http://docs.moodle.org/dev/Scheduled_Tasks_Proposal ? It moves towards compatibility with the UNIX cron worldview which, you are right, does not have an exact equivalent to "run every 5 minutes", but if you use "*/5" in the minutes field, it will run at the 0th, 5th, 10th, 15th, 20th, etc. minute of every hour, which is a close approximation. I haven't read the code carefully, so I can't say whether it implements that correctly. Damyon, it looks like you wrote your own cron format-parsing functions. Is there any particular reason you chose to do that instead of using the functions written for Mahara? AFAICT, the Mahara functions have already been well tested, and I don't know about you, but I prefer using something that's already been written and tested when dealing with text parsing.
            Hide
            Petr Skoda added a comment -

            "/5" looks fine! How could I miss that? There is one thing that worries me, we should not align all cron task to specific time because that would kill servers that host multiple Moodle instances.

            Another thing I do not understand is how you are going to actually execute the tasks, PHP does not have threads, something would have to be calling admin/cron.php or executing admin/cli/cron.php at least once a minute to get some parallel execution.

            Another thing I miss in the proposal is task priority, on sites that for some reason cannot call cron every minute there will be a queue of missed tasks, I guess that doing it "the first scheduled is executed first" is not good enough, at present we have strict order/priorities.

            Show
            Petr Skoda added a comment - "/5" looks fine! How could I miss that? There is one thing that worries me, we should not align all cron task to specific time because that would kill servers that host multiple Moodle instances. Another thing I do not understand is how you are going to actually execute the tasks, PHP does not have threads, something would have to be calling admin/cron.php or executing admin/cli/cron.php at least once a minute to get some parallel execution. Another thing I miss in the proposal is task priority, on sites that for some reason cannot call cron every minute there will be a queue of missed tasks, I guess that doing it "the first scheduled is executed first" is not good enough, at present we have strict order/priorities.
            Hide
            Petr Skoda added a comment -

            Alternatively we could also hijack normal we requests and after the normal execution is finished we could close session, ignore user abort and continue executing stuff.

            Show
            Petr Skoda added a comment - Alternatively we could also hijack normal we requests and after the normal execution is finished we could close session, ignore user abort and continue executing stuff.
            Hide
            Hubert Chathi added a comment -

            Yes, when I wrote the ELIS code, the scheduled tasks code was called by the system cron every 5 minutes. It calculated the next run time for each task, and when it was called by the system cron, it would check whether the current time was past the next run time. I haven't looked into how Damyon's code works.

            I believe that we talked about task priority at the Czech hackfest, and it got lumped into the "black magic" in the spec. I think the big thing missing from the spec is hashing out exactly what's included in the "black magic" and how it works.

            Show
            Hubert Chathi added a comment - Yes, when I wrote the ELIS code, the scheduled tasks code was called by the system cron every 5 minutes. It calculated the next run time for each task, and when it was called by the system cron, it would check whether the current time was past the next run time. I haven't looked into how Damyon's code works. I believe that we talked about task priority at the Czech hackfest, and it got lumped into the "black magic" in the spec. I think the big thing missing from the spec is hashing out exactly what's included in the "black magic" and how it works.
            Hide
            Petr Skoda added a comment -

            Thanks for the info Hubert.

            Show
            Petr Skoda added a comment - Thanks for the info Hubert.
            Hide
            Tyler Bannister added a comment -

            There are also a number of "optional" cron scripts in Moodle, that could also be converted to using this system. A number of the core auth classes (cas, db, ldap) have sync scripts and a number of the core enrol plugins have sync scripts too (category, cohort, database, flatfile, ldap, manual, meta, paypal, and self). It seems like these occasionally cause problems for users who don't realize they need to add them to a cron configuration. How would these scripts fit into this issue, if at all?

            Should the screens to be developed by MDL-25505 list any CLI scripts belonging to enabled auth and enrol plugins with an option to schedule them to be run from the regular cron?

            Show
            Tyler Bannister added a comment - There are also a number of "optional" cron scripts in Moodle, that could also be converted to using this system. A number of the core auth classes (cas, db, ldap) have sync scripts and a number of the core enrol plugins have sync scripts too (category, cohort, database, flatfile, ldap, manual, meta, paypal, and self). It seems like these occasionally cause problems for users who don't realize they need to add them to a cron configuration. How would these scripts fit into this issue, if at all? Should the screens to be developed by MDL-25505 list any CLI scripts belonging to enabled auth and enrol plugins with an option to schedule them to be run from the regular cron?
            Hide
            Paul Nicholls added a comment -

            I like the idea of being able to configure the auth and enrol sync scripts from the web UI. At certain times of the year, we ramp up the frequency at which they run for a while - it'd be great to be able to do this without needing to manually edit the crontab.

            Show
            Paul Nicholls added a comment - I like the idea of being able to configure the auth and enrol sync scripts from the web UI. At certain times of the year, we ramp up the frequency at which they run for a while - it'd be great to be able to do this without needing to manually edit the crontab.
            Hide
            Damyon Wiese added a comment -

            Sorry for not responding - I just realised I hadn't added myself as a watcher to the meta issue.

            Petr: Re aligning to a specific time - this will only do anything when cron runs. For multiple sites - you could stagger when cron actually runs, or configure your tasks so in the admin interface so they don't all run at the same time.

            Re: parallel cron - yes about once per minute is how often I was thinking this would run. You could do it even more often if you like and the work will be shared.

            -1 for highjacking the end of user requests for cron - in a cluster you do not want cron running on the user facing nodes. Also - when a site is least busy (3am) is when you want to do lots of cron stuff.

            Hubert - this code does exactly what you described - each task calculates when it should run next and each time cron runs it only runs the tasks that are due. If multiple crons are running - they will not get handed the same tasks due to the locking.

            Re: Priority - I think requiring a defined order for cron tasks misses the point of allowing parallel cron. Even with the current system there is the chance e.g. for users signing up between the auth and enrol steps in cron - this is life. They will just have to wait 5 minutes! If cron is running much more often - this is less of an issue.

            Re: optional scripts - yes all "optional" scripts should be run by this new system and we should not have lots of different crons for different purposes. The scripts will no longer get blocked by long running crons and do not have to implement their own scheduling. There will be only one cli cron script.

            Show
            Damyon Wiese added a comment - Sorry for not responding - I just realised I hadn't added myself as a watcher to the meta issue. Petr: Re aligning to a specific time - this will only do anything when cron runs. For multiple sites - you could stagger when cron actually runs, or configure your tasks so in the admin interface so they don't all run at the same time. Re: parallel cron - yes about once per minute is how often I was thinking this would run. You could do it even more often if you like and the work will be shared. -1 for highjacking the end of user requests for cron - in a cluster you do not want cron running on the user facing nodes. Also - when a site is least busy (3am) is when you want to do lots of cron stuff. Hubert - this code does exactly what you described - each task calculates when it should run next and each time cron runs it only runs the tasks that are due. If multiple crons are running - they will not get handed the same tasks due to the locking. Re: Priority - I think requiring a defined order for cron tasks misses the point of allowing parallel cron. Even with the current system there is the chance e.g. for users signing up between the auth and enrol steps in cron - this is life. They will just have to wait 5 minutes! If cron is running much more often - this is less of an issue. Re: optional scripts - yes all "optional" scripts should be run by this new system and we should not have lots of different crons for different purposes. The scripts will no longer get blocked by long running crons and do not have to implement their own scheduling. There will be only one cli cron script.
            Hide
            Aparup Banerjee added a comment -

            linking a recent badge cron slowness issue. we had to hackily disable cron for just this.
            It would help if some more priority went towards this issue so that future cron issues in general to moodle sites can be more properly managed.

            Show
            Aparup Banerjee added a comment - linking a recent badge cron slowness issue. we had to hackily disable cron for just this. It would help if some more priority went towards this issue so that future cron issues in general to moodle sites can be more properly managed.
            Hide
            Aparup Banerjee added a comment -

            assigning this meta to Damyon as he's acting on the sub-tasks.

            Show
            Aparup Banerjee added a comment - assigning this meta to Damyon as he's acting on the sub-tasks.
            Hide
            David Mudrak added a comment -

            Hi Damyon. I just spotted strings like taskscheduleday_help that reads

            Day of month field for task schedule. The field uses the same format as unix cron. Some examples are:<br/><ul><li><strong>*</strong> Every day</li><li><strong>*/2</strong> Every 2nd day</li><li><strong>1</strong> The first of every month</li><li><strong>1,​15</strong> The first and fifteenth of every month</li></ul>
            

            Please note that all *_help and *_desc strings have native support for Markdown formatting to make the translation easier. You may wish to change them to something like

            Day of month field for task schedule. The field uses the same format as unix cron. Some examples are:
             
            * '''*''' Every day
            * '''*/2''' Every 2nd day
            * '''1''' The first of every month
            * '''1,​15''' The first and fifteenth of every month
            

            to achieve the same result with less HTML.

            Show
            David Mudrak added a comment - Hi Damyon. I just spotted strings like taskscheduleday_help that reads Day of month field for task schedule. The field uses the same format as unix cron. Some examples are:<br/><ul><li><strong>*</strong> Every day</li><li><strong>*/2</strong> Every 2nd day</li><li><strong>1</strong> The first of every month</li><li><strong>1,​15</strong> The first and fifteenth of every month</li></ul> Please note that all *_help and *_desc strings have native support for Markdown formatting to make the translation easier. You may wish to change them to something like Day of month field for task schedule. The field uses the same format as unix cron. Some examples are:   * '''*''' Every day * '''*/2''' Every 2nd day * '''1''' The first of every month * '''1,​15''' The first and fifteenth of every month to achieve the same result with less HTML.
            Hide
            Mary Cooch added a comment -

            Just noting I started a page on Scheduled tasks here http://docs.moodle.org/27/en/Scheduled_tasks

            Show
            Mary Cooch added a comment - Just noting I started a page on Scheduled tasks here http://docs.moodle.org/27/en/Scheduled_tasks
            Hide
            Damyon Wiese added a comment -

            Closing this everything was done except portfolio - so I converted the portfolio changes to their own separate issue.

            Show
            Damyon Wiese added a comment - Closing this everything was done except portfolio - so I converted the portfolio changes to their own separate issue.

              People

              • Votes:
                13 Vote for this issue
                Watchers:
                28 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: