Moodle

Statistics is currently in catchup mode with TWO days pending

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Minor Minor
  • Resolution: Fixed
  • Affects Version/s: 1.6, 1.6.1, 1.6.2, 1.6.3, 1.7
  • Fix Version/s: 1.9
  • Component/s: Administration
  • Labels:
    None
  • Affected Branches:
    MOODLE_16_STABLE, MOODLE_17_STABLE
  • Fixed Branches:
    MOODLE_19_STABLE

Description

Bug discovered with Moodle 1.6.3 on Linux and PHP 5.2 .
I think this is a new bug which has nothing to do with other the registered bugs about statistics (in particular bug MDL-7385)

The script admin/report.php tell me that "Statistics is currently in catchup mode. So far 63 day(s) have been processed and 2 are pending".

  • 63 days is quite right: I started statistics for about two months. This number is incremented day after day: OK.
  • 2 days are pending. Why?

I debug and found that actually the last fully elapsed day (I mean "yesterday") has not be processed. And would never be processed, in my case.
For example:

  • suppose previous statistics have been processed until Monday (included); therefore last timeend in stats_daily table has been set to "Tuesday 00:00" (Tuesday early morning). OK.
  • on Wednesday, statistics is starting again and should be gather Tuesday's records, from Tuesday morning 00:00 till Wednesday morning 00:00 . But it does nothing. Because it believes that we are still on Tuesday.
    Indeed, in function stats_cron_daily of statslib.ph (line #101), days are processed one by one: "while ($midnight > $nextmidnight)". In other words: "if today midnight (Wednesday tonight 23:59) is after Tuesday midnight 00:00". But "today midnight" seems bad measured (with stats_getmidnight function) and is set to Tuesday midnight 00:00 (in place of Wednesday midnight 00:00). So the loop stops one step too early.

****************************
So I have patched function stats_cron_daily of statslib.ph, replacing line #67
$midnight = stats_getmidnight(time());
with
$midnight = stats_get_next_dayend(time());
****************************
And, for me, it works fine: the "yesterday" is always well processed.

See also some part of this discuss:
http://moodle.org/mod/forum/discuss.php?d=57242#274543

Issue Links

This issue will be resolved by:
MDL-13192 stats cleanup - meta Blocker Closed
This issue duplicates:
MDL-5973 statistics catchup doesn't Major Closed

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Anthony Borrow added a comment -

I believe that Penny Leach has been working on the stats - perhaps Martin would like to assign this patch to her for review.

Show
Anthony Borrow added a comment - I believe that Penny Leach has been working on the stats - perhaps Martin would like to assign this patch to her for review.
Hide
Permalink
Anthony Borrow added a comment -

Penny - The proposed patch makes sense to me. I think I will try it out on our production server and see what happens. In your opinion should the proposed patch be applied to CVS? Peace - Anthony

Show
Anthony Borrow added a comment - Penny - The proposed patch makes sense to me. I think I will try it out on our production server and see what happens. In your opinion should the proposed patch be applied to CVS? Peace - Anthony
Hide
Permalink
Martín Langhoff added a comment -

Hi Anthony - we have some affected users tracking this as MDL-5973 – and I'm asking them to try your change and report back. You might want to watch that bug too

Show
Martín Langhoff added a comment - Hi Anthony - we have some affected users tracking this as MDL-5973 – and I'm asking them to try your change and report back. You might want to watch that bug too
Hide
Permalink
Dan Marsden added a comment -

fixed part of the issue for me! - this fixed the 2 days pending issue, BUT - No stats were processed initially due to the problem I mention in the other bug - I had the "workshop" module "installed" although I had no "workshop" folder in my modules directory.... Maybe a good solution for this would be to get the admin page to run a check to see if any blocks/modules are installed that don't seem to exist and then throw a friendly error on the admin page to warn the users....

Dan

Show
Dan Marsden added a comment - fixed part of the issue for me! - this fixed the 2 days pending issue, BUT - No stats were processed initially due to the problem I mention in the other bug - I had the "workshop" module "installed" although I had no "workshop" folder in my modules directory.... Maybe a good solution for this would be to get the admin page to run a check to see if any blocks/modules are installed that don't seem to exist and then throw a friendly error on the admin page to warn the users.... Dan
Hide
Permalink
Penny Leach added a comment -

It should just do function_exists on the modules before trying to call a function.. or file_exists before including it or something.

At any rate, having workshop installed but not in the folder is bad (tm)

Show
Penny Leach added a comment - It should just do function_exists on the modules before trying to call a function.. or file_exists before including it or something. At any rate, having workshop installed but not in the folder is bad (tm)
Hide
Permalink
Anthony Borrow added a comment -

Dan - Thanks for encouraging those other users to try this patch although I cannot take credit for it as it was Arnaud's suggestion. I have applied the patch myself and will let you know if it resolves the issue. I suspect that it will and think that this patch should be put in CVS but I wanted it checked/verified by someone more familiar with the stats library.

MDL-5973 seems to be a related but separate issue. Penny's comment seems to be related to MDL-5973.

One quick question, does your graph and the tabular data match up on the statistics reports? Mine appear to be a day off. For example, my tabular data earliest date is January 22, 2007 but the first date listed in the graph is January 23rd. The last day listed on the graph is January 29th and the latest tabular date is January 28. From what I can tell, the data in the graph is correct and the dates listed in the table are off by a day. I'm a little tired so I just wanted to make sure that it was not my imagination. If others notice this I would be happy to create a separate issue in the tracker. Peace.

Show
Anthony Borrow added a comment - Dan - Thanks for encouraging those other users to try this patch although I cannot take credit for it as it was Arnaud's suggestion. I have applied the patch myself and will let you know if it resolves the issue. I suspect that it will and think that this patch should be put in CVS but I wanted it checked/verified by someone more familiar with the stats library. MDL-5973 seems to be a related but separate issue. Penny's comment seems to be related to MDL-5973. One quick question, does your graph and the tabular data match up on the statistics reports? Mine appear to be a day off. For example, my tabular data earliest date is January 22, 2007 but the first date listed in the graph is January 23rd. The last day listed on the graph is January 29th and the latest tabular date is January 28. From what I can tell, the data in the graph is correct and the dates listed in the table are off by a day. I'm a little tired so I just wanted to make sure that it was not my imagination. If others notice this I would be happy to create a separate issue in the tracker. Peace.
Hide
Permalink
Anthony Borrow added a comment -

In reference to my quick question about the graph, in order to be consistent with /course/report/stats/report.php line 134 (which does not have the standard version header information - hint, hint... - but is from Moodle 1.6.4+) that reads:

if (empty($CFG->loglifetime) || ($stat->timeend-(60*60*24)) >= (time()60*60*24*$CFG>loglifetime)) {

I believe that line 58 of the /course/reports/stats/graph.php (v1.2) should read:

$graph->x_data[] = userdate($stat->timeend-(60*60*24),get_string('strftimedate'),$CFG->timezone);

rather than its current:

$graph->x_data[] = userdate($stat->timeend,get_string('strftimedate'),$CFG->timezone);

Show
Anthony Borrow added a comment - In reference to my quick question about the graph, in order to be consistent with /course/report/stats/report.php line 134 (which does not have the standard version header information - hint, hint... - but is from Moodle 1.6.4+) that reads: if (empty($CFG->loglifetime) || ($stat->timeend-(60*60*24)) >= (time()60*60*24*$CFG>loglifetime)) { I believe that line 58 of the /course/reports/stats/graph.php (v1.2) should read: $graph->x_data[] = userdate($stat->timeend-(60*60*24),get_string('strftimedate'),$CFG->timezone); rather than its current: $graph->x_data[] = userdate($stat->timeend,get_string('strftimedate'),$CFG->timezone);
Hide
Permalink
Anthony Borrow added a comment -

In other words (in patch language):

58c58
< $graph->x_data[] = userdate($stat->timeend,get_string('strftimedate'),$CFG->timezone);
—
> $graph->x_data[] = userdate($stat->timeend-(60*60*24),get_string('strftimedate'),$CFG->timezone);

Show
Anthony Borrow added a comment - In other words (in patch language): 58c58 < $graph->x_data[] = userdate($stat->timeend,get_string('strftimedate'),$CFG->timezone); — > $graph->x_data[] = userdate($stat->timeend-(60*60*24),get_string('strftimedate'),$CFG->timezone);
Hide
Permalink
Martín Langhoff added a comment -

I think this is closely related to MDL-6795 where Mike figured out we were applying TZ twice. Which probably can account for the stats being a day or two off.

Re Dan's "missing module" issue, I'm treating that as MDL-7385 and I've fixed it on 17_STABLE, 18_STABLE and HEAD.

Re the graphic, I think that the graphic right now matches what we have in the DB. If the stats are off by a day or two, we have to fix that problem. Shifting the graph is only a coverup. Anthony, do you see the same offset in the data in the stats table?

Show
Martín Langhoff added a comment - I think this is closely related to MDL-6795 where Mike figured out we were applying TZ twice. Which probably can account for the stats being a day or two off. Re Dan's "missing module" issue, I'm treating that as MDL-7385 and I've fixed it on 17_STABLE, 18_STABLE and HEAD. Re the graphic, I think that the graphic right now matches what we have in the DB. If the stats are off by a day or two, we have to fix that problem. Shifting the graph is only a coverup. Anthony, do you see the same offset in the data in the stats table?
Hide
Permalink
Anthony Borrow added a comment -

Martin - the data in my stats table for March 7th
SELECT sum(studentreads)+sum(studentwrites) as StudActivity
FROM mdl_stats_daily
WHERE
timeend >= unix_timestamp('2007-03-07 00:00:00') AND
timeend <unix_timestamp('2007-03-08 00:00:00')
group by timeend;

produces 12537 which is what appear in the report as the Student Activity for March 6th.

Do you know the query that I could run on mdl_log to verify that what is in stats is what corresponds to the logs? I'm not sure what all is included as a studentread and studentwrite.

Show
Anthony Borrow added a comment - Martin - the data in my stats table for March 7th SELECT sum(studentreads)+sum(studentwrites) as StudActivity FROM mdl_stats_daily WHERE timeend >= unix_timestamp('2007-03-07 00:00:00') AND timeend <unix_timestamp('2007-03-08 00:00:00') group by timeend; produces 12537 which is what appear in the report as the Student Activity for March 6th. Do you know the query that I could run on mdl_log to verify that what is in stats is what corresponds to the logs? I'm not sure what all is included as a studentread and studentwrite.
Hide
Permalink
Anthony Borrow added a comment -

I am going to leave my /course/report/stats/graph.php modified with the day subtracted so at least what is in the table (whether it is accurate or not) is the same as that which reflected on the graph. I think that is at least a step in the right direction.

Show
Anthony Borrow added a comment - I am going to leave my /course/report/stats/graph.php modified with the day subtracted so at least what is in the table (whether it is accurate or not) is the same as that which reflected on the graph. I think that is at least a step in the right direction.
Hide
Permalink
Petr Škoda (skodak) added a comment -

should be fixed in latest 1.9dev, thanks for the report - see linked meta bug for more details

Show
Petr Škoda (skodak) added a comment - should be fixed in latest 1.9dev, thanks for the report - see linked meta bug for more details

Dates

  • Created:
    Updated:
    Resolved:
Atlassian JIRA (v4.4.3#663-r165197) | Report a problem
Powered by a free Atlassian JIRA open source license for Moodle. Try JIRA - bug tracking software for your team.