Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-18484

Statistics are impossibly slow

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.9.3, 1.9.4, 1.9.5
    • Fix Version/s: None
    • Component/s: Administration
    • Labels:
    • Environment:
      LAMP
    • Database:
      MySQL
    • Affected Branches:
      MOODLE_19_STABLE

      Description

      On 1.9.3+ (Build: 20090109) stats are completely unusable for us.

      My server tried to runs stats for ONE SINGLE DAY of logs on our site (a pilot of only 30-odd courses but with 30000 mostly-unused rows in the user table). After 8 HOURS, I had to restart MySQL in the morning as it was still locked up at 100% and moodle could not connect to the DB.

      MySQL struggled for 6 HOURS on a single query!!

      Here is the monstrous query. We showed it to our Computer Science SQL lecturer for a laugh who said 'it is definitely a very bad query. Using cross join on large tables is a definite NO'. Creating 1.2 billion rows from our little DB is insane.

      1. Time: 090306 6:30:42
      2. User@Host: moodle[moodle] @ localhost []
      3. Query_time: 21631 Lock_time: 0 Rows_sent: 0 Rows_examined: 1199857693
        UPDATE mdl_stats_daily
        SET stat2 = (SELECT COUNT(DISTINCT ra.userid)
        FROM mdl_role_assignments ra JOIN mdl_context ctx
        ON ctx.id = ra.contextid
        CROSS JOIN mdl_course c
        JOIN mdl_role_capabilities rc
        ON rc.roleid = ra.roleid LEFT OUTER JOIN mdl_course_categories cc1
        ON cc1.id = c.category
        WHERE ra.roleid = mdl_stats_daily.roleid AND
        c.id = mdl_stats_daily.courseid AND
        ((rc.capability = 'moodle/course:view' )
        AND rc.permission = 1 AND rc.contextid = 1
        AND (ctx.contextlevel = 10
        OR (c.id = ctx.instanceid AND ctx.contextlevel = 50) OR (cc1.id = ctx.instanceid AND ctx.contextlevel = 40))) AND
        EXISTS (SELECT 'x'
        FROM mdl_log l
        WHERE l.course = mdl_stats_daily.courseid AND
        l.userid = ra.userid AND l.time >= 1236164400 AND l.time < 1236250800))
        WHERE mdl_stats_daily.stattype = 'enrolments' AND
        mdl_stats_daily.timeend = 1236250800 AND
        mdl_stats_daily.courseid IN
        (SELECT DISTINCT l.course
        FROM mdl_log l
        WHERE l.time >= 1236164400 AND l.time < 1236250800);

        Attachments

        1. commit-5b5a23f
          14 kB
        2. statslib.php-rev01.diff
          17 kB
        3. statslib.php-rev02.diff
          18 kB
        4. statslib.php-rev03.diff
          28 kB
        5. statslib.php-rev04.diff
          31 kB
        6. statslib.php-rev05.diff
          34 kB
        7. statslib.php-rev06.diff
          36 kB
        8. statslib.php-rev07.diff
          36 kB
        9. statslib.php-rev08.diff
          38 kB
        10. statslib.php-rev09.diff
          38 kB

          Issue Links

            Activity

              People

              Assignee:
              moodle.com moodle.com
              Reporter:
              websterb Bruce Webster
              Participants:
              Component watchers:
              Andrew Nicols, Dongsheng Cai, Huong Nguyen, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze
              Votes:
              36 Vote for this issue
              Watchers:
              33 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: