Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-25499

Centralise management of all types of cron tasks with registration, scheduling, parallel task conflicts(blocking) and running once off tasks, all using an administration screen.

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1, 2.6
    • Fix Version/s: 2.7
    • Component/s: Administration
    • Labels:
    • Affected Branches:
      MOODLE_21_STABLE, MOODLE_26_STABLE
    • Fixed Branches:
      MOODLE_27_STABLE

      Description

      The document proposing this is http://docs.moodle.org/en/Development:Scheduled_Tasks_Proposal.

      related bugs

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

              Hide
              mjollnir Penny Leach added a comment -

              you probably want to re-audit all the existing cronjobs, since that audit was done like in December....

              Show
              mjollnir Penny Leach added a comment - you probably want to re-audit all the existing cronjobs, since that audit was done like in December....
              Hide
              timhunt Tim Hunt added a comment -
              Show
              timhunt Tim Hunt added a comment - Developer chat about this: http://moodle.org/mod/cvsadmin/view.php?conversationid=6219#c242871
              Hide
              hchathi Hubert Chathi added a comment -

              I mentioned this in the dev chat, but JFTR, we are implementing a scheduled tasks system based on this proposed system, and I'm planning on sharing our code when we're done. We won't implement all of it (e.g., we won't be converting the existing cron system to use the new system), but it should be a good start.

              Show
              hchathi Hubert Chathi added a comment - I mentioned this in the dev chat, but JFTR, we are implementing a scheduled tasks system based on this proposed system, and I'm planning on sharing our code when we're done. We won't implement all of it (e.g., we won't be converting the existing cron system to use the new system), but it should be a good start.
              Hide
              hchathi Hubert Chathi added a comment -

              Our implementation of part of this for 1.9:

              We have also patched /admin/cron.php to call our main cron function: https://github.com/remotelearner/elis.base/blob/master/patches/admin/cron.php.patch

              So far, we have only implemented the part that schedules tasks to run at certain times (MDL-25507). We have not yet done locking, and our version of "black magic" currently is just "sort by next run time".

              We will update when we release our 2.1 code, and when we implement locking (and more advanced black magic).

              Show
              hchathi Hubert Chathi added a comment - Our implementation of part of this for 1.9: The actual cron function: https://github.com/remotelearner/elis.base/blob/master/core/elis/core/cron.php Library functions (e.g. for loading [plugin]/db/tasks.db): https://github.com/remotelearner/elis.base/blob/master/core/elis/core/lib/tasklib.php Database schema: elis_scheduled_tasks table from https://github.com/remotelearner/elis.base/blob/master/core/elis/core/db/install.xml We have also patched /admin/cron.php to call our main cron function: https://github.com/remotelearner/elis.base/blob/master/patches/admin/cron.php.patch So far, we have only implemented the part that schedules tasks to run at certain times ( MDL-25507 ). We have not yet done locking, and our version of "black magic" currently is just "sort by next run time". We will update when we release our 2.1 code, and when we implement locking (and more advanced black magic).
              Hide
              simoncoggins Simon Coggins added a comment -

              In Totara we have some code that uses OS-level file locking to lock the cron file to prevent it running twice at the same time. The advantage of this approach is that the lock is automatically removed if the cron crashes part way through.

              We have also implemented a cron watcher script that can catch long running processes and inform the administrator, or kill the process if it exceeds a (configurable) time period.

              Happy to share the code if it would be of use.

              Simon

              Show
              simoncoggins Simon Coggins added a comment - In Totara we have some code that uses OS-level file locking to lock the cron file to prevent it running twice at the same time. The advantage of this approach is that the lock is automatically removed if the cron crashes part way through. We have also implemented a cron watcher script that can catch long running processes and inform the administrator, or kill the process if it exceeds a (configurable) time period. Happy to share the code if it would be of use. Simon
              Hide
              mhughes2k Michael Hughes added a comment -

              I've started another thread about this on the forums (https://moodle.org/mod/forum/discuss.php?d=229139) following on from the discussion around MDL-17783.

              I think this is something that really needs to be looked at as the simple approach of only permitting a single cron instance at a time really can cause bottlenecks when cron functions start taking longer than the initiation period (e.g. every 10 minutes).

              Show
              mhughes2k Michael Hughes added a comment - I've started another thread about this on the forums ( https://moodle.org/mod/forum/discuss.php?d=229139 ) following on from the discussion around MDL-17783 . I think this is something that really needs to be looked at as the simple approach of only permitting a single cron instance at a time really can cause bottlenecks when cron functions start taking longer than the initiation period (e.g. every 10 minutes).
              Hide
              damyon Damyon Wiese added a comment -

              I need this implemented so I can start doing document conversions in cron for mod_assign. I've already added code for the locking subtask - and will continue working through these subtasks...

              Show
              damyon Damyon Wiese added a comment - I need this implemented so I can start doing document conversions in cron for mod_assign. I've already added code for the locking subtask - and will continue working through these subtasks...
              Hide
              skodak Petr Skoda added a comment -

              Do I read the spec and code right? It seems it moves away from periodic execution (every 5 minutes for example) to scheduled execution only (at 5th minute of every hour). This looks like a serious regression to me, I expected you would be able to do both.

              Also at preset we have dependency, for example auth should be always executed before enrol. How is that going nto be described in task definition and how is it going to be implemented?

              Maybe the task definition array could include word 'default' to clearly ondicate the valies can be later changed later in admin UI.

              Anyway thanks a lot for working on this long standing issue!

              Show
              skodak Petr Skoda added a comment - Do I read the spec and code right? It seems it moves away from periodic execution (every 5 minutes for example) to scheduled execution only (at 5th minute of every hour). This looks like a serious regression to me, I expected you would be able to do both. Also at preset we have dependency, for example auth should be always executed before enrol. How is that going nto be described in task definition and how is it going to be implemented? Maybe the task definition array could include word 'default' to clearly ondicate the valies can be later changed later in admin UI. Anyway thanks a lot for working on this long standing issue!
              Hide
              hchathi Hubert Chathi added a comment -

              Petr: are you referring to this spec http://docs.moodle.org/dev/Scheduled_Tasks_Proposal ? It moves towards compatibility with the UNIX cron worldview which, you are right, does not have an exact equivalent to "run every 5 minutes", but if you use "*/5" in the minutes field, it will run at the 0th, 5th, 10th, 15th, 20th, etc. minute of every hour, which is a close approximation. I haven't read the code carefully, so I can't say whether it implements that correctly.

              Damyon, it looks like you wrote your own cron format-parsing functions. Is there any particular reason you chose to do that instead of using the functions written for Mahara? AFAICT, the Mahara functions have already been well tested, and I don't know about you, but I prefer using something that's already been written and tested when dealing with text parsing.

              Show
              hchathi Hubert Chathi added a comment - Petr: are you referring to this spec http://docs.moodle.org/dev/Scheduled_Tasks_Proposal ? It moves towards compatibility with the UNIX cron worldview which, you are right, does not have an exact equivalent to "run every 5 minutes", but if you use "*/5" in the minutes field, it will run at the 0th, 5th, 10th, 15th, 20th, etc. minute of every hour, which is a close approximation. I haven't read the code carefully, so I can't say whether it implements that correctly. Damyon, it looks like you wrote your own cron format-parsing functions. Is there any particular reason you chose to do that instead of using the functions written for Mahara? AFAICT, the Mahara functions have already been well tested, and I don't know about you, but I prefer using something that's already been written and tested when dealing with text parsing.
              Hide
              skodak Petr Skoda added a comment -

              "/5" looks fine! How could I miss that? There is one thing that worries me, we should not align all cron task to specific time because that would kill servers that host multiple Moodle instances.

              Another thing I do not understand is how you are going to actually execute the tasks, PHP does not have threads, something would have to be calling admin/cron.php or executing admin/cli/cron.php at least once a minute to get some parallel execution.

              Another thing I miss in the proposal is task priority, on sites that for some reason cannot call cron every minute there will be a queue of missed tasks, I guess that doing it "the first scheduled is executed first" is not good enough, at present we have strict order/priorities.

              Show
              skodak Petr Skoda added a comment - "/5" looks fine! How could I miss that? There is one thing that worries me, we should not align all cron task to specific time because that would kill servers that host multiple Moodle instances. Another thing I do not understand is how you are going to actually execute the tasks, PHP does not have threads, something would have to be calling admin/cron.php or executing admin/cli/cron.php at least once a minute to get some parallel execution. Another thing I miss in the proposal is task priority, on sites that for some reason cannot call cron every minute there will be a queue of missed tasks, I guess that doing it "the first scheduled is executed first" is not good enough, at present we have strict order/priorities.
              Hide
              skodak Petr Skoda added a comment -

              Alternatively we could also hijack normal we requests and after the normal execution is finished we could close session, ignore user abort and continue executing stuff.

              Show
              skodak Petr Skoda added a comment - Alternatively we could also hijack normal we requests and after the normal execution is finished we could close session, ignore user abort and continue executing stuff.
              Hide
              hchathi Hubert Chathi added a comment -

              Yes, when I wrote the ELIS code, the scheduled tasks code was called by the system cron every 5 minutes. It calculated the next run time for each task, and when it was called by the system cron, it would check whether the current time was past the next run time. I haven't looked into how Damyon's code works.

              I believe that we talked about task priority at the Czech hackfest, and it got lumped into the "black magic" in the spec. I think the big thing missing from the spec is hashing out exactly what's included in the "black magic" and how it works.

              Show
              hchathi Hubert Chathi added a comment - Yes, when I wrote the ELIS code, the scheduled tasks code was called by the system cron every 5 minutes. It calculated the next run time for each task, and when it was called by the system cron, it would check whether the current time was past the next run time. I haven't looked into how Damyon's code works. I believe that we talked about task priority at the Czech hackfest, and it got lumped into the "black magic" in the spec. I think the big thing missing from the spec is hashing out exactly what's included in the "black magic" and how it works.
              Hide
              skodak Petr Skoda added a comment -

              Thanks for the info Hubert.

              Show
              skodak Petr Skoda added a comment - Thanks for the info Hubert.
              Hide
              tbannister Tyler Bannister added a comment -

              There are also a number of "optional" cron scripts in Moodle, that could also be converted to using this system. A number of the core auth classes (cas, db, ldap) have sync scripts and a number of the core enrol plugins have sync scripts too (category, cohort, database, flatfile, ldap, manual, meta, paypal, and self). It seems like these occasionally cause problems for users who don't realize they need to add them to a cron configuration. How would these scripts fit into this issue, if at all?

              Should the screens to be developed by MDL-25505 list any CLI scripts belonging to enabled auth and enrol plugins with an option to schedule them to be run from the regular cron?

              Show
              tbannister Tyler Bannister added a comment - There are also a number of "optional" cron scripts in Moodle, that could also be converted to using this system. A number of the core auth classes (cas, db, ldap) have sync scripts and a number of the core enrol plugins have sync scripts too (category, cohort, database, flatfile, ldap, manual, meta, paypal, and self). It seems like these occasionally cause problems for users who don't realize they need to add them to a cron configuration. How would these scripts fit into this issue, if at all? Should the screens to be developed by MDL-25505 list any CLI scripts belonging to enabled auth and enrol plugins with an option to schedule them to be run from the regular cron?
              Hide
              paul.n Paul Nicholls added a comment -

              I like the idea of being able to configure the auth and enrol sync scripts from the web UI. At certain times of the year, we ramp up the frequency at which they run for a while - it'd be great to be able to do this without needing to manually edit the crontab.

              Show
              paul.n Paul Nicholls added a comment - I like the idea of being able to configure the auth and enrol sync scripts from the web UI. At certain times of the year, we ramp up the frequency at which they run for a while - it'd be great to be able to do this without needing to manually edit the crontab.
              Hide
              damyon Damyon Wiese added a comment -

              Sorry for not responding - I just realised I hadn't added myself as a watcher to the meta issue.

              Petr: Re aligning to a specific time - this will only do anything when cron runs. For multiple sites - you could stagger when cron actually runs, or configure your tasks so in the admin interface so they don't all run at the same time.

              Re: parallel cron - yes about once per minute is how often I was thinking this would run. You could do it even more often if you like and the work will be shared.

              -1 for highjacking the end of user requests for cron - in a cluster you do not want cron running on the user facing nodes. Also - when a site is least busy (3am) is when you want to do lots of cron stuff.

              Hubert - this code does exactly what you described - each task calculates when it should run next and each time cron runs it only runs the tasks that are due. If multiple crons are running - they will not get handed the same tasks due to the locking.

              Re: Priority - I think requiring a defined order for cron tasks misses the point of allowing parallel cron. Even with the current system there is the chance e.g. for users signing up between the auth and enrol steps in cron - this is life. They will just have to wait 5 minutes! If cron is running much more often - this is less of an issue.

              Re: optional scripts - yes all "optional" scripts should be run by this new system and we should not have lots of different crons for different purposes. The scripts will no longer get blocked by long running crons and do not have to implement their own scheduling. There will be only one cli cron script.

              Show
              damyon Damyon Wiese added a comment - Sorry for not responding - I just realised I hadn't added myself as a watcher to the meta issue. Petr: Re aligning to a specific time - this will only do anything when cron runs. For multiple sites - you could stagger when cron actually runs, or configure your tasks so in the admin interface so they don't all run at the same time. Re: parallel cron - yes about once per minute is how often I was thinking this would run. You could do it even more often if you like and the work will be shared. -1 for highjacking the end of user requests for cron - in a cluster you do not want cron running on the user facing nodes. Also - when a site is least busy (3am) is when you want to do lots of cron stuff. Hubert - this code does exactly what you described - each task calculates when it should run next and each time cron runs it only runs the tasks that are due. If multiple crons are running - they will not get handed the same tasks due to the locking. Re: Priority - I think requiring a defined order for cron tasks misses the point of allowing parallel cron. Even with the current system there is the chance e.g. for users signing up between the auth and enrol steps in cron - this is life. They will just have to wait 5 minutes! If cron is running much more often - this is less of an issue. Re: optional scripts - yes all "optional" scripts should be run by this new system and we should not have lots of different crons for different purposes. The scripts will no longer get blocked by long running crons and do not have to implement their own scheduling. There will be only one cli cron script.
              Hide
              nebgor Aparup Banerjee added a comment -

              linking a recent badge cron slowness issue. we had to hackily disable cron for just this.
              It would help if some more priority went towards this issue so that future cron issues in general to moodle sites can be more properly managed.

              Show
              nebgor Aparup Banerjee added a comment - linking a recent badge cron slowness issue. we had to hackily disable cron for just this. It would help if some more priority went towards this issue so that future cron issues in general to moodle sites can be more properly managed.
              Hide
              nebgor Aparup Banerjee added a comment -

              assigning this meta to Damyon as he's acting on the sub-tasks.

              Show
              nebgor Aparup Banerjee added a comment - assigning this meta to Damyon as he's acting on the sub-tasks.
              Hide
              mudrd8mz David Mudrák added a comment -

              Hi Damyon. I just spotted strings like taskscheduleday_help that reads

              Day of month field for task schedule. The field uses the same format as unix cron. Some examples are:<br/><ul><li><strong>*</strong> Every day</li><li><strong>*/2</strong> Every 2nd day</li><li><strong>1</strong> The first of every month</li><li><strong>1,​15</strong> The first and fifteenth of every month</li></ul>

              Please note that all *_help and *_desc strings have native support for Markdown formatting to make the translation easier. You may wish to change them to something like

              Day of month field for task schedule. The field uses the same format as unix cron. Some examples are:
               
              * '''*''' Every day
              * '''*/2''' Every 2nd day
              * '''1''' The first of every month
              * '''1,​15''' The first and fifteenth of every month

              to achieve the same result with less HTML.

              Show
              mudrd8mz David Mudrák added a comment - Hi Damyon. I just spotted strings like taskscheduleday_help that reads Day of month field for task schedule. The field uses the same format as unix cron. Some examples are:<br/><ul><li><strong>*</strong> Every day</li><li><strong>*/2</strong> Every 2nd day</li><li><strong>1</strong> The first of every month</li><li><strong>1,​15</strong> The first and fifteenth of every month</li></ul> Please note that all *_help and *_desc strings have native support for Markdown formatting to make the translation easier. You may wish to change them to something like Day of month field for task schedule. The field uses the same format as unix cron. Some examples are:   * '''*''' Every day * '''*/2''' Every 2nd day * '''1''' The first of every month * '''1,​15''' The first and fifteenth of every month to achieve the same result with less HTML.
              Hide
              marycooch Mary Cooch added a comment -

              Just noting I started a page on Scheduled tasks here http://docs.moodle.org/27/en/Scheduled_tasks

              Show
              marycooch Mary Cooch added a comment - Just noting I started a page on Scheduled tasks here http://docs.moodle.org/27/en/Scheduled_tasks
              Hide
              damyon Damyon Wiese added a comment -

              Closing this everything was done except portfolio - so I converted the portfolio changes to their own separate issue.

              Show
              damyon Damyon Wiese added a comment - Closing this everything was done except portfolio - so I converted the portfolio changes to their own separate issue.
              Hide
              beichapman Bei Chapman added a comment -

              Hi Damyon,

              A quick question, is this fix (cron major update) in both 2.9.4 and 3.0.2 streams, or just 3.0.2, or none of them?

              I got this tracker number from Moodle org 3.0 upgrading document, but couldn't find/verify it from any 3.0.x release notes. I would like to confirm it with you.

              Thanks,
              Bei

              Show
              beichapman Bei Chapman added a comment - Hi Damyon, A quick question, is this fix (cron major update) in both 2.9.4 and 3.0.2 streams, or just 3.0.2, or none of them? I got this tracker number from Moodle org 3.0 upgrading document, but couldn't find/verify it from any 3.0.x release notes. I would like to confirm it with you. Thanks, Bei
              Hide
              rex Rex Lorenzo added a comment -
              Show
              rex Rex Lorenzo added a comment - Bei Chapman This was done for Moodle 2.7. See: https://docs.moodle.org/dev/Moodle_2.7_release_notes#Cron
              Hide
              damyon Damyon Wiese added a comment -

              This was in 2.7.0 - I just amended the fix version on this issue to the oldest version I can (I can't set it to 2.7 anymore because it is no longer a current version). Epic issues don't go through integration so the fields are sometimes not accurate (but the completed issues in the epic should all be accurate).

              Show
              damyon Damyon Wiese added a comment - This was in 2.7.0 - I just amended the fix version on this issue to the oldest version I can (I can't set it to 2.7 anymore because it is no longer a current version). Epic issues don't go through integration so the fields are sometimes not accurate (but the completed issues in the epic should all be accurate).
              Hide
              ghillenb Gisela Hillenbrand added a comment -

              I came here because I experienced the problem that no mail notifications are sent to students when a teacher gradesan assignment submission (as posted e.g. in this discussion: https://moodle.org/mod/forum/discuss.php?d=331910
              From the developers docs I learned that this was previously done by the cron job of the assignment module and now should be replaced by the new scheduled tasks API, as described here: https://docs.moodle.org/dev/Scheduled_Tasks_Proposal
              However, no schedulded task is defined for the assignment module (i.e. no file /mod/assign/db/task.php exists). In mod/assign/lib.php a function assign_cron() is defined but it seems that this function is never called (and so no notifications are sent to students about their assignment grades).
              So can you explain what is going on here? Is this a bug? Or is there a possibilty to define scheduled tasks for the assignment module like sending the notifications mentioned above (as a moodle admin)?

              Show
              ghillenb Gisela Hillenbrand added a comment - I came here because I experienced the problem that no mail notifications are sent to students when a teacher gradesan assignment submission (as posted e.g. in this discussion: https://moodle.org/mod/forum/discuss.php?d=331910 From the developers docs I learned that this was previously done by the cron job of the assignment module and now should be replaced by the new scheduled tasks API, as described here: https://docs.moodle.org/dev/Scheduled_Tasks_Proposal However, no schedulded task is defined for the assignment module (i.e. no file /mod/assign/db/task.php exists). In mod/assign/lib.php a function assign_cron() is defined but it seems that this function is never called (and so no notifications are sent to students about their assignment grades). So can you explain what is going on here? Is this a bug? Or is there a possibilty to define scheduled tasks for the assignment module like sending the notifications mentioned above (as a moodle admin)?
              Hide
              marina Marina Glancy added a comment -

              Hello Gisela, this issue has already been closed. Moodle forums https://moodle.org/forums is the best place to ask questions
              If you believe that you have found a bug please search existing issues on tracker, and, if it has not yet been reported, create a new one.
              TIA

              Show
              marina Marina Glancy added a comment - Hello Gisela, this issue has already been closed. Moodle forums https://moodle.org/forums is the best place to ask questions If you believe that you have found a bug please search existing issues on tracker, and, if it has not yet been reported, create a new one. TIA

                People

                • Votes:
                  13 Vote for this issue
                  Watchers:
                  30 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:
                    Fix Release Date:
                    12/May/14