Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-69684

It's possible to hold a redis session lock forever

    XMLWordPrintable

Details

    • Bug
    • Status: Waiting for peer review
    • Minor
    • Resolution: Unresolved
    • 3.9.3, 3.10, 3.11
    • None
    • Libraries
    • Any
    • MOODLE_310_STABLE, MOODLE_311_STABLE, MOODLE_39_STABLE
    • Hide

      Setup

      1. Install the PHP Redis extension (replace "X" as your server's PHP version, e.g. 8.0)

        sudo apt update && sudo apt install -y phpX-redis

      2. Restart your web server.
      3. Confirm the redis PHP extension is enabled: "php -i | grep redis" - You should see a bunch of lines related to redis.
      4. Install redis. e.g. using Docker

        docker run --name redis -p 6379:6379 -d redis

      5. Setup redis as the session handler, e.g. copy the following to your config.php:

        $CFG->session_handler_class = '\core\session\redis';
        $CFG->session_redis_host = '127.0.0.1';
        

      Before applying the patch

      Test
      1. To see if Redis is being used, in a prompt on the server type the following command

        redis-cli monitor

      2. Navigate around the site keeping an eye on the open terminal.
      3. Confirm that session locks are created with expiry time set to the same value as $CFG->session_redis_lock_expire set in your config.php.
      4. Confirm that you see a SETNX and a EXPIRE command for each created session lock, eg.

        1654812249.749869 [0 127.0.0.1:56392] "SETNX" "0h0cvmc680rjmriqpralti003j.lock" "s:50:\"[pid 274359] bons.ai.tuwien.ac.at:/login/index.php\";"
        1654812249.749937 [0 127.0.0.1:56392] "EXPIRE" "0h0cvmc680rjmriqpralti003j.lock" "30"
        

      After applying the patch

      Test
      1. To see if Redis is being used, in a prompt on the server type the following command

        redis-cli monitor

      2. Navigate around the site keeping an eye on the open terminal.
      3. Confirm that session locks are created with expiry time set to the same value as $CFG->session_redis_lock_expire set in your config.php.
      4. Confirm that you see only a SET command for each created session lock, eg.

        1654812196.987659 [0 127.0.0.1:46970] "SET" "1h43n2aq1ppcu7fvtsrskf6ef5.lock" "s:50:\"[pid 274359] bons.ai.tuwien.ac.at:/login/index.php\";" "ex" "30" "nx"
        

      Additional tests

      There is an attachment PHP file that can run additional tests:

      redis_test.php

      It is a slightly modified version of Nicholas Hoobin testscript from MDL-54606. Please place it in the root directory of your Moodle install. To see some runtime options try php redis_test.php --help

      This script uses the built in generators to create a list of accounts in Moodle. It then logs the users in to create individual session objects that are stored in Redis. It navigates to the user dashboard to retreive the session object and update it. When it navigates to this page it compares some of the session data with the generated data for a match.

      Test
      1. Setup the testing environment by changing the following values in your config.php:

        $CFG->disablelogintoken = true;
        $CFG->sessiontimeout = 3600;
        

      2. Generate initial test users.

        php redis_test.php --generate
         
        # Expected output.
        Created 25/200 users.
        Created 50/200 users.
        Created 75/200 users.
        Created 100/200 users.
        Created 125/200 users.
        Created 150/200 users.
        Created 175/200 users.
        Created 200/200 users.
        Created 200/200 users.
        

      3. Log the users in.

        php redis_test.php --login
         
        # Expected output.
        Current session timeout: 60 minutes
        Logged in 25/200 users.
        Logged in 50/200 users.
        Logged in 75/200 users.
        Logged in 100/200 users.
        Logged in 125/200 users.
        Logged in 150/200 users.
        Logged in 175/200 users.
        Logged in 200/200 users.
        

      4. Test after some time, 0 mins, 5 mins, 1 hour, or less than $CFG->sessiontimeout

        php redis_test.php --testall
         
        # Expected output.
        Updating sessions of 25/200 users.
        Updating sessions of 50/200 users.
        Updating sessions of 75/200 users.
        Updating sessions of 100/200 users.
        Updating sessions of 125/200 users.
        Updating sessions of 150/200 users.
        Updating sessions of 175/200 users.
        Updating sessions of 200/200 users.
        

      5. Cleanup accounts and temp files generated.

        php redis_test.php --delete
         
        # Expected output.
        Deleted 25/200 users.
        Deleted 50/200 users.
        Deleted 75/200 users.
        Deleted 100/200 users.
        Deleted 125/200 users.
        Deleted 150/200 users.
        Deleted 175/200 users.
        Deleted 200/200 users.
        

      Show
      Setup Install the PHP Redis extension (replace " X " as your server's PHP version, e.g. 8.0) sudo apt update && sudo apt install -y phpX-redis Restart your web server. Confirm the redis PHP extension is enabled: " php -i | grep redis " - You should see a bunch of lines related to redis. Install redis. e.g. using Docker docker run --name redis -p 6379:6379 -d redis Setup redis as the session handler, e.g. copy the following to your config.php : $CFG ->session_handler_class = '\core\session\redis' ; $CFG ->session_redis_host = '127.0.0.1' ; Before applying the patch Test To see if Redis is being used, in a prompt on the server type the following command redis-cli monitor Navigate around the site keeping an eye on the open terminal. Confirm that session locks are created with expiry time set to the same value as $CFG->session_redis_lock_expire set in your config.php . Confirm that you see a SETNX and a EXPIRE command for each created session lock, eg. 1654812249.749869 [0 127.0.0.1:56392] "SETNX" "0h0cvmc680rjmriqpralti003j.lock" "s:50:\"[pid 274359] bons.ai.tuwien.ac.at:/login/index.php\";" 1654812249.749937 [0 127.0.0.1:56392] "EXPIRE" "0h0cvmc680rjmriqpralti003j.lock" "30" After applying the patch Test To see if Redis is being used, in a prompt on the server type the following command redis-cli monitor Navigate around the site keeping an eye on the open terminal. Confirm that session locks are created with expiry time set to the same value as $CFG->session_redis_lock_expire set in your config.php . Confirm that you see only a SET command for each created session lock, eg. 1654812196.987659 [0 127.0.0.1:46970] "SET" "1h43n2aq1ppcu7fvtsrskf6ef5.lock" "s:50:\"[pid 274359] bons.ai.tuwien.ac.at:/login/index.php\";" "ex" "30" "nx" Additional tests There is an attachment PHP file that can run additional tests: redis_test.php It is a slightly modified version of Nicholas Hoobin testscript from MDL-54606 . Please place it in the root directory of your Moodle install. To see some runtime options try php redis_test.php --help This script uses the built in generators to create a list of accounts in Moodle. It then logs the users in to create individual session objects that are stored in Redis. It navigates to the user dashboard to retreive the session object and update it. When it navigates to this page it compares some of the session data with the generated data for a match. Test Setup the testing environment by changing the following values in your config.php : $CFG ->disablelogintoken = true; $CFG ->sessiontimeout = 3600; Generate initial test users. php redis_test.php --generate   # Expected output. Created 25/200 users. Created 50/200 users. Created 75/200 users. Created 100/200 users. Created 125/200 users. Created 150/200 users. Created 175/200 users. Created 200/200 users. Created 200/200 users. Log the users in. php redis_test.php --login   # Expected output. Current session timeout: 60 minutes Logged in 25/200 users. Logged in 50/200 users. Logged in 75/200 users. Logged in 100/200 users. Logged in 125/200 users. Logged in 150/200 users. Logged in 175/200 users. Logged in 200/200 users. Test after some time, 0 mins, 5 mins, 1 hour, or less than $CFG->sessiontimeout php redis_test.php --testall   # Expected output. Updating sessions of 25/200 users. Updating sessions of 50/200 users. Updating sessions of 75/200 users. Updating sessions of 100/200 users. Updating sessions of 125/200 users. Updating sessions of 150/200 users. Updating sessions of 175/200 users. Updating sessions of 200/200 users. Cleanup accounts and temp files generated. php redis_test.php --delete   # Expected output. Deleted 25/200 users. Deleted 50/200 users. Deleted 75/200 users. Deleted 100/200 users. Deleted 125/200 users. Deleted 150/200 users. Deleted 175/200 users. Deleted 200/200 users.

    Description

      in /lib/classes/session/redis.php
      a session lock is acquired from redis with the following code:

      $haslock = $this->connection->setnx($lockkey, $whoami);
      if ($haslock) {
          $this->locks[$id] = $this->time() + $this->lockexpire;
          $this->connection->expire($lockkey, $this->lockexpire);
          return true;
      }

      Since the lock is acquired and expired in two separate calls it's possible for it to lock out a user indefinitely if moodle is taken down.

      It should use the atomic call to set the expiry at the same time.

      $haslock = $this->connection->set($lockkey, $whoami, ['nx', 'ex'=>$this->lockexpire]);
      if ($haslock) {
          $this->locks[$id] = $this->time() + $this->lockexpire;
          return true;
      }
      

      We've seen this happen once in production. I'm more than happy to make a pr for this issue.

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              jamie@agilicus Jamie
              Ferran Recio Ferran Recio
              Adrian Greeve Adrian Greeve
              Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Ilya Tregubov, Laurent David, Raquel Ortega, Sara Arjona (@sarjona)
              Votes:
              6 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 55 minutes
                  55m