Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-82384

New course cache purge severely slowing operations on large sites

XMLWordPrintable

    • MOODLE_404_STABLE
    • MOODLE_404_STABLE
    • MDL-82384-MOODLE_404_STABLE
    • MDL-82384-main
    • Hide

      Upgrade Test (Should be Faster)

      1. Take a snapshot of the database
      2. Run a CLI upgrade and time it

      `/usr/bin/time php admin/cli/upgrade.php --non-interactive`

      3. Apply the patch
      4. Reset the database to the same snapshot
      5. Re-run the CLI upgrade (time should be reduced)

      `/usr/bin/time php admin/cli/upgrade.php --non-interactive`

      Purge Course Caches Test (Should be Similar)

      1. View a series of courses to generate course caches
      2. Run a CLI course cache purge and time it

      `/usr/bin/time php admin/cli/purge_caches.php --courses`

      3. Apply the patch
      4. View the same series of courses to generate course aches
      5. Re-run the CLI course cache purge (the time should be similar)

      `/usr/bin/time php admin/cli/purge_caches.php --courses`

      Purge All Caches Test (Should be Faster)

      1. Browse the site across a range of pages to generate various caches.
      2. Snapshot the database, sitedata, caching, etc.
      3. Purge all caches and time it

      `/usr/bin/time php admin/cli/purge_caches.php`

      4. Reset the database, site data, caching, etc. to the snapshot
      5. Apply the patch.
      6. Purge all caches again and time it (the time should be less)

      `/usr/bin/time php admin/cli/purge_caches.php`

       

      Show
      Upgrade Test (Should be Faster) 1. Take a snapshot of the database 2. Run a CLI upgrade and time it `/usr/bin/time php admin/cli/upgrade.php --non-interactive` 3. Apply the patch 4. Reset the database to the same snapshot 5. Re-run the CLI upgrade (time should be reduced) `/usr/bin/time php admin/cli/upgrade.php --non-interactive` Purge Course Caches Test (Should be Similar) 1. View a series of courses to generate course caches 2. Run a CLI course cache purge and time it `/usr/bin/time php admin/cli/purge_caches.php --courses` 3. Apply the patch 4. View the same series of courses to generate course aches 5. Re-run the CLI course cache purge (the time should be similar) `/usr/bin/time php admin/cli/purge_caches.php --courses` Purge All Caches Test (Should be Faster) 1. Browse the site across a range of pages to generate various caches. 2. Snapshot the database, sitedata, caching, etc. 3. Purge all caches and time it `/usr/bin/time php admin/cli/purge_caches.php` 4. Reset the database, site data, caching, etc. to the snapshot 5. Apply the patch. 6. Purge all caches again and time it (the time should be less) `/usr/bin/time php admin/cli/purge_caches.php`  

      With the implementation of MDL-80985 Add --courses argument to purge_caches.php into Moodle 4.4 onward. I have found that this severely slows down many operations within Moodle (drastically so on larger implementations). Upgrades, uninstall of plugins and essentially anywhere in the code that implements a full purge caches (which now includes a full course cache purge).

      The course cache purge leverages the "increment_revision_number" function from /lib/datalib.php around line 1184. This ultimately means on any cache purge it's setting a new cacherev number for each mdl_course id in the database. I have found that this is delaying the start of Moodle upgrades and quite a few other operations in Moodle. 

      On further investigation, there's quite a few places in the Moodle code that call for the "purge_all_caches" function which is forcing a full cacherev overhaul each time and processes are waiting for this to finish. 

      Anywhere the 'purge_all_caches' function is called it's forcing the full purge and the difference between a purge caches with and without courses can be seen attached (2 seconds vs 14 minutes)

      Some scripts call the full purge twice too, so you can imaging the impact this is having on normal operations. 

            james_w James Williams
            connor.cosector Connor Giles
            Paul Holden Paul Holden
            Shamim Rezaie Shamim Rezaie
            Angelia Dela Cruz Angelia Dela Cruz
            Votes:
            10 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 1 hour, 49 minutes
                1h 49m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.