Moodle

Single enrolment flatfile processed concurrently by 20 consecutive crons

Details

  • Type: Bug Bug
  • Status: Open Open
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: 1.9.2
  • Fix Version/s: STABLE backlog
  • Component/s: Enrolments
  • Labels:
    None
  • Environment:
    SUSE Linux Enterprise Server 10, Apache 2.2.3, MySQL 5.0.26, PHP 5.2.5
    2 x 2.5 GHz CPUs and 4 GB RAM
  • Database:
    MySQL
  • Affected Branches:
    MOODLE_19_STABLE

Description

Presumably this happens because it takes longer than 5 minutes for the first cron that encounters the file to process and delete it, so it's still there when the next cron starts - and so on for the next 18 crons, each one putting more load on MySQL and causing it to take even longer, exacerbating the problem.

Our enrolment file is currently 20,716 lines long and the term hasn't started yet so it's going to get a lot bigger. This gets processed by around 20 crons (I get a separate log file for every cron and they each process every line) and this causes a huge load on the MySQL server, slowing down the whole site. Eventually after about 4 hours the database activity dies down and the file is eventually deleted. Just before the end I get a load of emails telling me to check the permissions because the flafile can't be unlinked, but the permissions are correct (and indeed the last cron successfully deletes it) - I'm guessing it just can't be deleted while there's still another cron processing it.

Issue Links

This issue is a clone of:
MDL-17783 Multiple cron instance running parallely Major Open

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Eloy Lafuente (stronk7) added a comment -

Well, this has two solutions:

1) In the server side, duplicate execution of cron should be avoided using something like "dotlockfile"
2) In the cron process, we can create some "semaphore" record in DB say, "enrolflatfilerunning", with some ttl to prevent repeated executions of the process.

Assigning this to Jerôme to implement 2). Ciao

Show
Eloy Lafuente (stronk7) added a comment - Well, this has two solutions: 1) In the server side, duplicate execution of cron should be avoided using something like "dotlockfile" 2) In the cron process, we can create some "semaphore" record in DB say, "enrolflatfilerunning", with some ttl to prevent repeated executions of the process. Assigning this to Jerôme to implement 2). Ciao
Hide
Permalink
Tony Butler added a comment -

Thanks Eloy

Show
Tony Butler added a comment - Thanks Eloy
Hide
Permalink
jai gupta added a comment -

A very similar issue is with us.

Once a day cron send digest to 30,000 users which takes ~40 minutes. Rest of the time cron executes within a minute. We want to keep cron interval to the minimum but we are forced to keep it above 40 minutes.

If we keep it less then 40 minutes, say 10 minutes then because of multiple crons running many copies of same digest is sent to all users.

A new cron instance should not run if previous cron is still running.

Instead of inserting a record in DB cron may create a file in moodledata. I have no idea which method would give better performance.

Show
jai gupta added a comment - A very similar issue is with us. Once a day cron send digest to 30,000 users which takes ~40 minutes. Rest of the time cron executes within a minute. We want to keep cron interval to the minimum but we are forced to keep it above 40 minutes. If we keep it less then 40 minutes, say 10 minutes then because of multiple crons running many copies of same digest is sent to all users. A new cron instance should not run if previous cron is still running. Instead of inserting a record in DB cron may create a file in moodledata. I have no idea which method would give better performance.
Hide
Permalink
Patrick Pollet added a comment -

Hello,

Here is the code we are using here to prevent concurrent runs of Moodle's cron, especially when the global serach is enabled, it do take a huge amount of time, sometime diying when passing over the max script execution time... and breaking serach indexes.
BTW we have disabled assignments and user's profiles from global search since them do overload the mysql server ...

atop admin/cron.php we added just after require_once(dirname(_FILE_) . '/../config.php');

define ('LOCK_FILE','/cronrunning');

if (file_exists($CFG->dataroot.LOCK_FILE)) { $time=file_get_contents($CFG->dataroot.LOCK_FILE); $err="previous cron is still running started at ".userdate($time); mtrace ($err); add_to_log(SITEID, 'cron', 'cron', '', $err); $admin=get_record('user','username','ppollet'); <--- to be adjusted $subject = "execution cron moodle ".$CFG->wwwroot; email_to_user($admin,$admin,$subject,$err); exit; }
set_time_limit(0); //important otherwise may die when passing time_limit and lockfile still there ...
file_put_contents($CFG->dataroot.LOCK_FILE,time());

and at the very end

unlink($CFG->dataroot.LOCK_FILE);

It does prevent concurrent runs with a warning mail to one admin (me) and a line in Moodle log ...

Cheers.

Show
Patrick Pollet added a comment - Hello, Here is the code we are using here to prevent concurrent runs of Moodle's cron, especially when the global serach is enabled, it do take a huge amount of time, sometime diying when passing over the max script execution time... and breaking serach indexes. BTW we have disabled assignments and user's profiles from global search since them do overload the mysql server ... atop admin/cron.php we added just after require_once(dirname(_FILE_) . '/../config.php'); define ('LOCK_FILE','/cronrunning'); if (file_exists($CFG->dataroot.LOCK_FILE)) { $time=file_get_contents($CFG->dataroot.LOCK_FILE); $err="previous cron is still running started at ".userdate($time); mtrace ($err); add_to_log(SITEID, 'cron', 'cron', '', $err); $admin=get_record('user','username','ppollet'); <--- to be adjusted $subject = "execution cron moodle ".$CFG->wwwroot; email_to_user($admin,$admin,$subject,$err); exit; } set_time_limit(0); //important otherwise may die when passing time_limit and lockfile still there ... file_put_contents($CFG->dataroot.LOCK_FILE,time()); and at the very end unlink($CFG->dataroot.LOCK_FILE); It does prevent concurrent runs with a warning mail to one admin (me) and a line in Moodle log ... Cheers.

People

Vote (4)
Watch (6)

Dates

  • Created:
    Updated:
Atlassian JIRA (v4.4.3#663-r165197) | Report a problem
Powered by a free Atlassian JIRA open source license for Moodle. Try JIRA - bug tracking software for your team.