Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-69088

Make file cache store purges instant with a safe and async purge

    XMLWordPrintable

Details

    • MOODLE_400_STABLE
    • MOODLE_400_STABLE
    • Hide
      1. Go to Site Administration -> Plugins -> Caching -> Configuration
      2. Add new instance for File Cache - File Cache -> Add instance
      3. Confirm that you can see the Asynchronously purge directory setting
      4. Fill in all the details
        • Store name: Take note of this.
        • Cache path: A folder writable by the web server. e.g. in your moodledata folder, "/path/to/moodledata/testcache/filestorecaching"
        • Single directory store setting should NOT be checked
        • Tick 'Asynchronously purge directory'
      5. Click on Save changes
      6. Back on the Cache administration page, search for Config settings (core config) under "Known cache definitions" and click on Edit mappings
      7. Change this core/config store mappings Primary cache to the file cache store that you created earlier.
      8. Click on Save changes
      9. A new cache revision directory should get created in the file store cache instance's Cache path directory
        • The new cache revision will be CACHE REV DIRECTORY = /path/to/moodledata/testcache/filestorecaching/core_config/<YmdHis> where YmdHis = current year month day hour minutes seconds e.g. /path/to/moodledata/testcache/filestorecaching/core_config/20211208095836
      10. Confirm that core/config cache files are created in this CACHE REV DIRECTORY
      11. Confirm the word count of this directory e.g. find <CACHE REV DIRECTORY> -type f | xargs wc -w | tail -1
      12. Now let's change some core config settings.
        1. Go to Site Administration -> Server -> Maintenance mode
        2. Add in some two lines of text in Optional maintenance message (maintenance_message) setting
        3. Click on Save changes
      13. Confirm the word count of this directory e.g. find <CACHE REV DIRECTORY> -type f | xargs wc -w | tail -1
      14. The current word count should be higher than Step 6
      15. Make a note of this word count
      16. Now let's purge caches for core/config.
        1. Go to Site Administration -> Plugins -> Caching -> Configuration
        2. Search for Config settings (core/config) from the list
        3. Click on Purge
      17. A new cache revision directory will be created on successful purge in the Cache path directory
        • Cache files will be created in this new CACHE REV DIRECTORY
        • Old CACHE REV DIRECTORY should exist. Confirm the word count of this directory, it should be the same as noted in Step 17
        • Adhoc task \cachestore_file\task\asyncpurge will be created to delete the Old CACHE REV DIRECTORY (Check  mdl_task_adhoc table)
      18. Repeat Steps 16 and 17.
      19. Old CACHE REV DIRECTORY should exist. Confirm the word count of this directory, it should be the same as noted in Step 17
      20. Any core config changes made will update relevant cache files in the new CACHE REV DIRECTORY
      21. Execute the Adhoc task created in Step 9 via the CLI
        1. php admin/cli/adhoc_task.php --execute
      22. Confirm the Old CACHE REV DIRECTORY has been deleted.
      Show
      Go to Site Administration -> Plugins -> Caching -> Configuration Add new instance for File Cache - File Cache -> Add instance Confirm that you can see the Asynchronously purge directory setting Fill in all the details Store name: Take note of this. Cache path: A folder writable by the web server. e.g. in your moodledata folder, " /path/to/moodledata/testcache/filestorecaching " Single directory store setting should NOT be checked Tick 'Asynchronously purge directory' Click on Save changes Back on the Cache administration page, search for Config settings (core config) under " Known cache definitions " and click on Edit mappings Change this core/config store mappings Primary cache to the file cache store that you created earlier. Click on Save changes A new cache revision directory should get created in the file store cache instance's Cache path directory The new cache revision will be CACHE REV DIRECTORY = /path/to/moodledata/testcache/filestorecaching/core_config/<YmdHis> where YmdHis = current year month day hour minutes seconds e.g. /path/to/moodledata/testcache/filestorecaching/core_config/20211208095836 Confirm that core/config cache files are created in this CACHE REV DIRECTORY Confirm the word count of this directory e.g. find <CACHE REV DIRECTORY> -type f | xargs wc -w | tail -1 Now let's change some core config settings. Go to Site Administration -> Server -> Maintenance mode Add in some two lines of text in Optional maintenance message (maintenance_message) setting Click on Save changes Confirm the word count of this directory e.g. find <CACHE REV DIRECTORY> -type f | xargs wc -w | tail -1 The current word count should be higher than Step 6 Make a note of this word count Now let's purge caches for core/config . Go to Site Administration -> Plugins -> Caching -> Configuration Search for Config settings (core/config) from the list Click on Purge A new cache revision directory will be created on successful purge in the Cache path directory Cache files will be created in this new CACHE REV DIRECTORY Old CACHE REV DIRECTORY should exist. Confirm the word count of this directory, it should be the same as noted in Step 17 Adhoc task \cachestore_file\task\asyncpurge will be created to delete the Old CACHE REV DIRECTORY (Check  mdl_task_adhoc table) Repeat Steps 16 and 17. Old CACHE REV DIRECTORY should exist. Confirm the word count of this directory, it should be the same as noted in Step 17 Any core config changes made will update relevant cache files in the new CACHE REV DIRECTORY Execute the Adhoc task created in Step 9 via the CLI php admin/cli/adhoc_task.php --execute Confirm the Old CACHE REV DIRECTORY has been deleted.

    Description

      If you have a very large shared file store cache, and it is completely purged eg after a release, then it uses a file glob to iterate over the files and delete them which can be fairly slow with millions of files. The file deletion doesn't need to block the release from finishing.

      Proposing to add a new option to the file cache store which when set stores all cache keys under an extra version key:

      cache/cachestore_file/1592480344/default_application

      instead of

      cache/cachestore_file/default_application

      When a purge happens, it just bumps the key to invalidate, and then fires of an adhoc task to slowly delete the old set of data.

      The $CFG->localcachedirpurged key should be well suited, or something new so it could be 1 key per store instance.

      Or alternatively don't make an option and just make this the way it works for everyone.

      Attachments

        1. MDL-69088_1.png
          MDL-69088_1.png
          83 kB
        2. MDL-69088_2.png
          MDL-69088_2.png
          54 kB
        3. MDL-69088_3.png
          MDL-69088_3.png
          90 kB
        4. MDL-69088_4.png
          MDL-69088_4.png
          58 kB
        5. MDL-69088_5.png
          MDL-69088_5.png
          9 kB

        Issue Links

          Activity

            People

              jacdsouza Jackson D'souza
              brendanheywood Brendan Heywood
              Brendan Heywood Brendan Heywood
              Ilya Tregubov Ilya Tregubov
              Angelia Dela Cruz Angelia Dela Cruz
              Matteo Scaramuccia, Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Ilya Tregubov, Laurent David, Sara Arjona (@sarjona), Matteo Scaramuccia, Andrew Lyons, Huong Nguyen, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze
              Votes:
              4 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                19/Apr/22

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 hours, 31 minutes
                  2h 31m