Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-72328

Add TTL support for Redis caches

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 4.0
    • Fix Version/s: 4.0
    • Component/s: Caching
    • Labels:
    • Affected Branches:
      MOODLE_400_STABLE
    • Fixed Branches:
      MOODLE_400_STABLE
    • Pull Master Branch:
      MDL-72328-master
    • Testing Instructions:
      Hide

      Prerequisites:

      1. A Redis cache that you have access to. If running Moodle on your local machine, then running Redis (or, if using Windows, Memurai) on the same machine with default settings is probably fine.
      2. A tool to view the contents of the Redis cache to verify things are working. I used this one (Linux/Mac/Windows): https://github.com/qishibo/AnotherRedisDesktopManager - just connect to localhost:6379 with it.
      3. The phpredis PHP extension installed in your PHP.
      4. A Moodle setup that uses your Redis cache for some or all of its application-level cache stores. Configure this if necessary under Site administration / Plugins / Caching / Configuration. If you're doing it from scratch:
        1. Go to the Installed cache stores heading and use Add instance to create a new Redis cache instance with your favourite name and the correct address e.g. 'localhost'. You can probably leave other settings default, but make up a special key prefix if the Redis server is in use for other things.
        2. At the bottom of the page under Stores used when no mapping is present click the Edit link and change the Application dropdown to the name of your new cache instance, then save changes.
      5. The 'Run now' feature for scheduled tasks needs to be available. If you haven't already set this up, you need to complete the admin setting Path to PHP CLI under Site administration / Server / System paths.
      6. Activity completion must be enabled on the server under Site administration / Advanced features / Enable completion tracking.
      7. In addition to the admin account, you need two test accounts.

      Once you have all that set up you can do an actual test.

      1. Purge all caches using the button at Site administration / Developer / Purge caches.
      2. Create a new course with default settings (completion tracking should be enabled).
      3. Enrol both test accounts into this course.
      4. Make a note of the course page URL so you can go back to it later.
      5. Go to the scheduled tasks page (Site administration / Server / Tasks / Scheduled tasks) and find the task Free up memory used by expired entries in Redis caches.
      6. Edit settings for this task and make sure it is not scheduled to run during this test (e.g. set it only to run during a different hour of the day) - we need to run it only when the test says.
      7. In a different browser (so you don't lose your admin login), go to the course page URL and log in as the first test account. Make a note of the time, e.g. 15:00. You can now close this browser.
      8. Wait 10 minutes then repeat using the second test account, making a note of the time again, e.g. 15:10.
      9. In your Redis cache viewer, refresh if necessary and look at the top-level keys.
        • Expected: You should see a key like prefix_3c60e20af217ecb7bfc5732b71099888 and another like prefix_3c60e20af217ecb7bfc5732b71099888_ttl (where prefix is your specified prefix, if any).
        • Both keys should contain 2 values each.
      10. Back in Moodle logged in as admin on the scheduled tasks page, click Run now next to the task and confirm at the prompt.
        • Expected: The output should show a list of caches including core/completion: Deleted 0 key(s) (because the data has not yet expired).
      11. This is the tedious part - wait just over an hour from the time you viewed with the first account, e.g. 16:02.
      12. Click Run again to re-run the scheduled task.
        • Expected: The text should now include core/completion: Deleted 1 key(s)
      13. In the Redis viewer, refresh.
        • Expected: The keys from last time should both now contain just 1 value.
      14. Wait 10 minutes until it's more than an hour after the second user access.
      15. Click Run again to re-run the scheduled task.
        • Expected: The text should again include core/completion: Deleted 1 key(s)
      16. Refresh in the Redis viewer.
        • Expected: Both keys containing 3c60e20af217ecb7bfc5732b71099888 should have disappeared.
      Show
      Prerequisites: A Redis cache that you have access to. If running Moodle on your local machine, then running Redis (or, if using Windows, Memurai) on the same machine with default settings is probably fine. A tool to view the contents of the Redis cache to verify things are working. I used this one (Linux/Mac/Windows): https://github.com/qishibo/AnotherRedisDesktopManager - just connect to localhost:6379 with it. The phpredis PHP extension installed in your PHP. A Moodle setup that uses your Redis cache for some or all of its application-level cache stores. Configure this if necessary under Site administration / Plugins / Caching / Configuration . If you're doing it from scratch: Go to the Installed cache stores heading and use Add instance to create a new Redis cache instance with your favourite name and the correct address e.g. 'localhost'. You can probably leave other settings default, but make up a special key prefix if the Redis server is in use for other things. At the bottom of the page under Stores used when no mapping is present click the Edit link and change the Application dropdown to the name of your new cache instance, then save changes. The 'Run now' feature for scheduled tasks needs to be available. If you haven't already set this up, you need to complete the admin setting Path to PHP CLI under Site administration / Server / System paths . Activity completion must be enabled on the server under Site administration / Advanced features / Enable completion tracking . In addition to the admin account, you need two test accounts. Once you have all that set up you can do an actual test. Purge all caches using the button at Site administration / Developer / Purge caches . Create a new course with default settings (completion tracking should be enabled). Enrol both test accounts into this course. Make a note of the course page URL so you can go back to it later. Go to the scheduled tasks page ( Site administration / Server / Tasks / Scheduled tasks ) and find the task Free up memory used by expired entries in Redis caches . Edit settings for this task and make sure it is not scheduled to run during this test (e.g. set it only to run during a different hour of the day) - we need to run it only when the test says. In a different browser (so you don't lose your admin login), go to the course page URL and log in as the first test account. Make a note of the time, e.g. 15:00. You can now close this browser. Wait 10 minutes then repeat using the second test account, making a note of the time again, e.g. 15:10. In your Redis cache viewer, refresh if necessary and look at the top-level keys. Expected: You should see a key like prefix_ 3c60e20af217ecb7bfc5732b71099888 and another like prefix_ 3c60e20af217ecb7bfc5732b71099888_ttl (where prefix is your specified prefix, if any). Both keys should contain 2 values each. Back in Moodle logged in as admin on the scheduled tasks page, click Run now next to the task and confirm at the prompt. Expected: The output should show a list of caches including core/completion: Deleted 0 key(s) (because the data has not yet expired). This is the tedious part - wait just over an hour from the time you viewed with the first account, e.g. 16:02. Click Run again to re-run the scheduled task. Expected: The text should now include core/completion: Deleted 1 key(s) In the Redis viewer, refresh. Expected: The keys from last time should both now contain just 1 value. Wait 10 minutes until it's more than an hour after the second user access. Click Run again to re-run the scheduled task. Expected: The text should again include core/completion: Deleted 1 key(s) Refresh in the Redis viewer. Expected: Both keys containing 3c60e20af217ecb7bfc5732b71099888 should have disappeared.

      Description

      Redis caches do not have TTL support. Even if TTL is set for a cache (e.g. core/completion) then no items in that cache will ever be deleted.

      To take the example of core/completion (which is the one that showed up as a problem in our automated load test, although it is slightly less of a problem in reality):

      • There is one item in this cache for each user who visits a course with completion on (i.e. the number of items goes up to {users} * {courses that they visit}).
      • For a fairly large course with approx. 500 activities, that item will be about 80 KB, call it 100 KB including Redis overhead.
      • Supposing 10,000 different users visit the site each day, and only visit one course each, this would occupy approximately 100 * 10,000 KB = 1 GB of Redis cache per day.
      • Eventually this will stop increasing at that rate because some of tomorrow's users will be the same people as today, but it's still wasting a lot of memory on Redis for something that's only valid for an hour.

      The reason this is less of a problem in reality is that stupid stuff means that while staff are editing websites (which wasn't included in our load test), core/completion cache gets purged sometimes. Still, at least conceptually, it is a bit rubbish.

      Incidentally, Redis features such as automatically deleting keys when it runs low on memory would mitigate this problem, but they can't be used entirely satisfactorily on a Moodle setup because it can only delete hashes (i.e. purge entire caches) and not delete single items within a hash (individual keys).

      It would be relatively easy to implement TTL support for Redis caches within Moodle by using the Redis 'sorted list' data structure to hold expiry dates, and then a scheduled task to delete old items based on that structure. I have an implementation which I will contribute.

      This means that cache set and delete are 2x slower, only for caches using TTL, but they were very fast anyway and generally, get performance (which is unaffected) is much more significant.

        Attachments

        1. MDL-72328-retest-step-10.png
          MDL-72328-retest-step-10.png
          76 kB
        2. MDL-72328-retest-step-12.png
          MDL-72328-retest-step-12.png
          76 kB
        3. MDL-72328-retest-step-13.png
          MDL-72328-retest-step-13.png
          41 kB
        4. MDL-72328-retest-step-13-2.png
          MDL-72328-retest-step-13-2.png
          36 kB
        5. MDL-72328-retest-step-15.png
          MDL-72328-retest-step-15.png
          76 kB
        6. MDL-72328-retest-step-16.png
          MDL-72328-retest-step-16.png
          14 kB
        7. MDL-72328-retest-step-9.png
          MDL-72328-retest-step-9.png
          53 kB
        8. MDL-72328-retest-step-9-2.png
          MDL-72328-retest-step-9-2.png
          42 kB
        9. MDL-72328-step-10.png
          MDL-72328-step-10.png
          93 kB
        10. MDL-72328-step-12.png
          MDL-72328-step-12.png
          88 kB
        11. MDL-72328-step-13.png
          MDL-72328-step-13.png
          15 kB
        12. MDL-72328-step-9.png
          MDL-72328-step-9.png
          142 kB
        13. MDL-72328-step-9-2.png
          MDL-72328-step-9-2.png
          111 kB

          Activity

            People

            Assignee:
            quen Sam Marshall
            Reporter:
            quen Sam Marshall
            Peer reviewer:
            Tim Hunt Tim Hunt
            Integrator:
            Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
            Tester:
            Michael Hawkins Michael Hawkins
            Participants:
            Component watchers:
            Matteo Scaramuccia, Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Ilya Tregubov, Sara Arjona (@sarjona)
            Votes:
            1 Vote for this issue
            Watchers:
            13 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 3 hours, 30 minutes
                3h 30m