Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-70687

Redis session lock expiration should default shorter than session timeout

XMLWordPrintable

    • MOODLE_39_STABLE
    • MOODLE_402_STABLE
    • MDL-70687_default-redis-session-lock-expiration-to-php-execution-time
    • Hide

      Setup

      1.  Checkout weekly 

         git checkout f3bf17cdfbee037

      1. Install your site
      2. Install the PHP Redis extension (replace "X" as your server's PHP version, e.g. 8.0) 

        sudo apt update && sudo apt install -y phpX-redis

      3. Restart your web server.
      4. Confirm the redis PHP extension is enabled: "php -i | grep redis" - You should see a bunch of lines related to redis.
      5. Install redis. e.g. using Docker 

        docker run --name redis -p 6379:6379 -d redis

      6. Setup redis as the session handler, e.g. copy the following to your config.php: 

        $CFG->session_handler_class = '\core\session\redis';
        		 
        $CFG->session_redis_host = '127.0.0.1';
        		 
        $CFG->session_redis_port = 6379;
        		 
        $CFG->session_redis_acquire_lock_timeout = 2;

      7. Configure session timeout in config.php 

        $CFG->sessiontimeout = 30;

      8. Configure the max_execution_time in your php.ini 

        max_execution_time = 60

      9. Install the session breaker plugin to help with testing: 

        git clone https://github.com/catalyst/moodle-tool_sessionbreaker/ admin/tool/sessionbreaker

      Before applying the patch

      Test 1
      1. Visit this page, which will lock the session for 2 minutes (which is way longer than the configured session timeout): /admin/tool/sessionbreaker/bad.php?wait=120
      2. In a second tab immediately load this page: /admin/tool/sessionbreaker/good.php
      3. Confirm after roughly 2, 15 and 25 seconds it always throws an exception. 

        Unable to obtain lock for session id [...]

      4. Confirm after 35 seconds it no longer throws an exception. It shows that the sessiontimeout is respected.
      Test 2
      1. Change the session timeout in config.php 

        $CFG->sessiontimeout = 60;

      2. Change the max_execution_time in your php.ini 

        max_execution_time = 30

      3. Restart your web server.
      4. Visit this page, which will lock the session for 2 minutes (which is way longer than the configured session timeout): /admin/tool/sessionbreaker/bad.php?wait=120
      5. In a second tab immediately load this page: /admin/tool/sessionbreaker/good.php
      6. Confirm after roughly 2, 15, 25 and 35 seconds it always throws an exception. 

        Unable to obtain lock for session id [...]

      7. Confirm after 60 seconds it no longer throws an exception. It shows that the sessiontimeout is still respected, and the max_execution_time has no effect.

      After applying the patch

      Test 3
      1. Checkout integration master:

      git checkout master

      1. Upgrade your site
      2. Change the session timeout in config.php 

        $CFG->sessiontimeout = 30;

      3. Change the max_execution_time in your php.ini 

        max_execution_time = 60

      4. Visit this page, which will lock the session for 2 minutes (which is way longer than the configured session timeout): /admin/tool/sessionbreaker/bad.php?wait=120
      5. In a second tab immediately load this page: /admin/tool/sessionbreaker/good.php
      6. Confirm after roughly 2, 15 and 25 seconds it always throws an exception. 

        Unable to obtain lock for session id [...]

      7. Confirm after 35 seconds it no longer throws an exception. It shows that the sessiontimeout is respected, which is lower than the max_execution_time.
      Test 4
      1. Change the session timeout in config.php 

        $CFG->sessiontimeout = 60;

      2. Change the max_execution_time in your php.ini 

        max_execution_time = 30

      3. Restart your web server.
      4. Visit this page, which will lock the session for 2 minutes (which is way longer than the configured session timeout): /admin/tool/sessionbreaker/bad.php?wait=120
      5. In a second tab immediately load this page: /admin/tool/sessionbreaker/good.php
      6. Confirm after roughly 2, 15 and 25 seconds it always throws an exception. 

        Unable to obtain lock for session id [...]

      7. Confirm after 35 seconds it no longer throws an exception. It shows that the max_execution_time is now respected, which is lower than the sessiontimeout.
      Show
      Setup  Checkout weekly git checkout f3bf17cdfbee037 Install your site Install the PHP Redis extension (replace " X " as your server's PHP version, e.g. 8.0) sudo apt update && sudo apt install -y phpX-redis Restart your web server. Confirm the redis PHP extension is enabled: " php -i | grep redis " - You should see a bunch of lines related to redis. Install redis. e.g. using Docker docker run --name redis -p 6379:6379 -d redis Setup redis as the session handler, e.g. copy the following to your config.php : $CFG ->session_handler_class = '\core\session\redis' ; $CFG ->session_redis_host = '127.0.0.1' ; $CFG ->session_redis_port = 6379; $CFG ->session_redis_acquire_lock_timeout = 2; Configure session timeout in config.php $CFG ->sessiontimeout = 30; Configure the max_execution_time in your php.ini max_execution_time = 60 Install the session breaker plugin to help with testing: git clone https://github.com/catalyst/moodle-tool_sessionbreaker/ admin/tool/sessionbreaker Before applying the patch Test 1 Visit this page, which will lock the session for 2 minutes (which is way longer than the configured session timeout): /admin/tool/sessionbreaker/bad.php?wait=120 In a second tab immediately load this page: /admin/tool/sessionbreaker/good.php Confirm after roughly 2, 15 and 25 seconds it always throws an exception. Unable to obtain lock for session id [...] Confirm after 35 seconds it no longer throws an exception. It shows that the sessiontimeout is respected. Test 2 Change the session timeout in config.php $CFG ->sessiontimeout = 60; Change the max_execution_time in your php.ini max_execution_time = 30 Restart your web server. Visit this page, which will lock the session for 2 minutes (which is way longer than the configured session timeout): /admin/tool/sessionbreaker/bad.php?wait=120 In a second tab immediately load this page: /admin/tool/sessionbreaker/good.php Confirm after roughly 2, 15, 25 and 35 seconds it always throws an exception. Unable to obtain lock for session id [...] Confirm after 60 seconds it no longer throws an exception. It shows that the sessiontimeout is still respected, and the max_execution_time has no effect. After applying the patch Test 3 Checkout integration master: git checkout master Upgrade your site Change the session timeout in config.php $CFG ->sessiontimeout = 30; Change the max_execution_time in your php.ini max_execution_time = 60 Visit this page, which will lock the session for 2 minutes (which is way longer than the configured session timeout): /admin/tool/sessionbreaker/bad.php?wait=120 In a second tab immediately load this page: /admin/tool/sessionbreaker/good.php Confirm after roughly 2, 15 and 25 seconds it always throws an exception. Unable to obtain lock for session id [...] Confirm after 35 seconds it no longer throws an exception. It shows that the sessiontimeout is respected, which is lower than the max_execution_time . Test 4 Change the session timeout in config.php $CFG ->sessiontimeout = 60; Change the max_execution_time in your php.ini max_execution_time = 30 Restart your web server. Visit this page, which will lock the session for 2 minutes (which is way longer than the configured session timeout): /admin/tool/sessionbreaker/bad.php?wait=120 In a second tab immediately load this page: /admin/tool/sessionbreaker/good.php Confirm after roughly 2, 15 and 25 seconds it always throws an exception. Unable to obtain lock for session id [...] Confirm after 35 seconds it no longer throws an exception. It shows that the max_execution_time is now respected, which is lower than the sessiontimeout .

      Currently the redis session lock expiration defaults to the session timeout. The session timeout is typically multiple hours.

      If a problem leads to a redis session lock not getting released properly, any subsequent request by a user in that session will wait to acquire the lock and eventually fail. The only way for the user to recover is to either delete their cookies or to wait several hours until the session times out.

      A reasonable session lock expiration is the maximum time a PHP request can be processed for. After this time expires without the lock being released it's clear that the php process must have ended without releasing it (there is potential for some system time not being counted, so to be on the safe side one could increase the default a bit over this minimum). Waiting any longer just causes more errors and problems for the users, so is not advisable.

      So this patch changes the default for session lock expiration in redis to the PHP max execution time. It can still be overwritten through $CFG->session_redis_lock_expire 

      I found the problem on 3.9.4 but it's still an issue on master.

        1. (I) 3 Passed -- (Master)MDL-70687.png
          (I) 3 Passed -- (Master)MDL-70687.png
          131 kB
        2. (I) 4 Passed -- (Master)MDL-70687.png
          (I) 4 Passed -- (Master)MDL-70687.png
          134 kB
        3. (II) 6 Passed -- (Master)MDL-70687.png
          (II) 6 Passed -- (Master)MDL-70687.png
          140 kB
        4. (II) 7 Passed -- (Master)MDL-70687.png
          (II) 7 Passed -- (Master)MDL-70687.png
          192 kB
        5. (III) 6 Passed -- (Master)MDL-70687.png
          (III) 6 Passed -- (Master)MDL-70687.png
          139 kB
        6. (III) 7 Passed -- (Master)MDL-70687.png
          (III) 7 Passed -- (Master)MDL-70687.png
          282 kB
        7. (IV) 6 Passed -- (Master)MDL-70687.png
          (IV) 6 Passed -- (Master)MDL-70687.png
          137 kB
        8. (IV) 7 Passed -- (Master)MDL-70687.png
          (IV) 7 Passed -- (Master)MDL-70687.png
          49 kB

            Daniel Ziegenberg Daniel Ziegenberg
            naderman Nils Adermann
            Jordi Pujol-Ahulló Jordi Pujol-Ahulló
            Jake Dallimore Jake Dallimore
            Kim Jared Lucas Kim Jared Lucas
            Votes:
            17 Vote for this issue
            Watchers:
            20 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 1 hour, 36 minutes
                1h 36m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.