Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-72796

retry interval in milliseconds for redis session cache is far too high

XMLWordPrintable

    • MOODLE_310_STABLE, MOODLE_311_STABLE, MOODLE_39_STABLE
    • MOODLE_311_STABLE
    • MDL-72796_fix-retry-delay-for-redis-session-cache
    • Hide
      Setup
      1. The prerequisite is a working Redis installation available on localhost.
        1. Install the PHP Redis extension appropriate for your PHP version. e.g. for PHP 7.4 on Ubuntu 

          sudo apt install php7.4-redis

        2. Run a Redis server via Docker 

          docker run --name my-redis --network host -d redis

      2. Add the following to config.php 

        define('TEST_SESSION_REDIS_HOST', '127.0.0.1');
        		 
        $CFG->session_handler_class = '\core\session\redis';
        		 
        $CFG->session_redis_host = '127.0.0.1';
        		 
        $CFG->session_redis_acquire_lock_timeout = 120;
        		 
        $CFG->session_redis_lock_expire = 7200;

      Test
      1. Set up PHPUnit and run:
        vendor/bin/phpunit lib/tests/session_redis_test.php
        to verify that there are no regressions.
      2. Using the redis-cli set the connection timeout to 5 seconds.
        CONFIG SET timeout 5
      3. Then download the redis_sessions_timeout.php script from this issue and put it in the webroot and run it as the apache user:
        sudo -u www-data php path/to/moodle/redis_sessions_timeout.php

      Before the patch, this should wait a very long time, after applying the patch it should be rather quick to reconnect to the Redis server.

      Show
      Setup The prerequisite is a working Redis installation available on localhost. Install the PHP Redis extension appropriate for your PHP version. e.g. for PHP 7.4 on Ubuntu sudo apt install php7.4-redis Run a Redis server via Docker docker run --name my-redis --network host -d redis Add the following to config.php define( 'TEST_SESSION_REDIS_HOST' , '127.0.0.1' ); $CFG->session_handler_class = '\core\session\redis' ; $CFG->session_redis_host = '127.0.0.1' ; $CFG->session_redis_acquire_lock_timeout = 120 ; $CFG->session_redis_lock_expire = 7200 ; Test Set up PHPUnit and run: vendor/bin/phpunit lib/tests/session_redis_test.php to verify that there are no regressions. Using the redis-cli set the connection timeout to 5 seconds. CONFIG SET timeout 5 Then download the redis_sessions_timeout.php script from this issue and put it in the webroot and run it as the apache user: sudo -u www-data php path/to/moodle/redis_sessions_timeout.php Before the patch, this should wait a very long time, after applying the patch it should be rather quick to reconnect to the Redis server.

      In lib/classes/session/redis.php on Line 196, a random delay between 100000 and 500000 is selected. It is further used as the 5th parameter for the Redis->connect() call. That's the $retryInterval, a retry interval in milliseconds.

      This means we are effectively waiting with a retry delay of 100 seconds to 500 seconds. The indented wait should be 100 ms to 500 ms, see here for reference.
      That's off by a factor of a thousand! Using Redis as a session cache and when the connection hangs, you can get random cannot obtain session lock errors because it's waiting up to 500 seconds (or about 8.33 minutes) for a Redis connection.

        1. redis_sessions_timeout.php
          1 kB
        2. MDL-72796_master.webm
          37 kB
        3. MDL-72796_v311.webm
          42 kB

            Daniel Ziegenberg Daniel Ziegenberg
            Daniel Ziegenberg Daniel Ziegenberg
            Brendan Heywood Brendan Heywood
            Andrew Lyons Andrew Lyons
            Angelia Dela Cruz Angelia Dela Cruz
            Votes:
            3 Vote for this issue
            Watchers:
            15 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 1 hour
                1h

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.