Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-53713

Sessions using memcached become corrupted after timeout

    XMLWordPrintable

    Details

    • Database:
      Any
    • Testing Instructions:
      Hide

      Configure your installation for memcached sessions (see config-dist.php).

      To make the effect manifest more quickly, adjust the lock timeout value to 10 seconds:
      $CFG->session_memcached_acquire_lock_timeout = 10;

      Simulate a long-running request by editing a repository plugin, in this case use repository/recent/lib.php, line 90, insert the statement "sleep(15);" to cause a delay in returning the result.

      Open your Moodle site, and navigate to a course where you have editing (teacher) capability, turn editing on for the course, and click "Add an activity or resource", select the File resource option, click Add. When the modedit interface (Add a new File) interface appears, click on the files display to cause the Filepicker dialog to appear.

      Access the developer tools extension for your browser (e.g. Firefox & Chrome, Ctrl + Shift + I), and select the tab to display network activity. From here you can click alternatively between 'Upload a file' and 'Recent files', allowing the request to finish between alternating clicks. When clicking on 'Recent files' you will see the two XHRs dispatched, one for setuserpref.php, and one for repository_ajax.php. When clicking on 'Upload files' expect to see only the call to setuserpref.php.

      Most often, you will see the setuserpref.php call completed almost immediately, and the subsequent call to repository_ajax.php will be pending until the simulated 15 sec. delay has elapsed. If the list of network calls gets too cluttered, you can clear the list. It may take several attempts toggling back and forth, but eventually you will see both requests pending, which means the repository_ajax.php call acquired the session key lock first. After 10 seconds (the session acquire lock timeout), the setuserpref.php will return, then 5 seconds afterward the repository_ajax call will return.

      Once this error condition has occurred you should notice the session cookie key value has changed--there was a new cookie value returned with the setuserpref.php operation. The mdl_sessions row corresponding to the original session cookie key value has been deleted. And, if using manual (Moodle internal) authentication, you should see that you now have to re-authenticate.

      An alternative testing method would be to toggle quickly, repeatedly, between 'Recent files', 'Upload files' and any of the other repositories. Those requests will be queued up, and while at least one of them is a 'Recent files' request where the simulated delay is in effect, the other requests will timeout waiting for a session key lock.

      Show
      Configure your installation for memcached sessions (see config-dist.php). To make the effect manifest more quickly, adjust the lock timeout value to 10 seconds: $CFG->session_memcached_acquire_lock_timeout = 10; Simulate a long-running request by editing a repository plugin, in this case use repository/recent/lib.php, line 90, insert the statement "sleep(15);" to cause a delay in returning the result. Open your Moodle site, and navigate to a course where you have editing (teacher) capability, turn editing on for the course, and click "Add an activity or resource", select the File resource option, click Add. When the modedit interface (Add a new File) interface appears, click on the files display to cause the Filepicker dialog to appear. Access the developer tools extension for your browser (e.g. Firefox & Chrome, Ctrl + Shift + I), and select the tab to display network activity. From here you can click alternatively between 'Upload a file' and 'Recent files', allowing the request to finish between alternating clicks. When clicking on 'Recent files' you will see the two XHRs dispatched, one for setuserpref.php, and one for repository_ajax.php. When clicking on 'Upload files' expect to see only the call to setuserpref.php. Most often, you will see the setuserpref.php call completed almost immediately, and the subsequent call to repository_ajax.php will be pending until the simulated 15 sec. delay has elapsed. If the list of network calls gets too cluttered, you can clear the list. It may take several attempts toggling back and forth, but eventually you will see both requests pending, which means the repository_ajax.php call acquired the session key lock first. After 10 seconds (the session acquire lock timeout), the setuserpref.php will return, then 5 seconds afterward the repository_ajax call will return. Once this error condition has occurred you should notice the session cookie key value has changed--there was a new cookie value returned with the setuserpref.php operation. The mdl_sessions row corresponding to the original session cookie key value has been deleted. And, if using manual (Moodle internal) authentication, you should see that you now have to re-authenticate. An alternative testing method would be to toggle quickly, repeatedly, between 'Recent files', 'Upload files' and any of the other repositories. Those requests will be queued up, and while at least one of them is a 'Recent files' request where the simulated delay is in effect, the other requests will timeout waiting for a session key lock.
    • Affected Branches:
      MOODLE_27_STABLE, MOODLE_28_STABLE, MOODLE_29_STABLE, MOODLE_30_STABLE
    • Fixed Branches:
      MOODLE_29_STABLE, MOODLE_30_STABLE
    • Pull Master Branch:

      Description

      Sessions using memcached for persistence become corrupted after a long running page request is followed up by a second request, such as an AJAX request from the filepicker. If user refreshes the page before the original request is completed, session state is lost, including authentication information. This then is sometimes compounded when using SSO (Shibboleth) authentication, and an authentication redirection loop occurs--though I've not been able to find the particular conditions (beyond the lost session) to reproduce this on demand.

      lib/classes/session/manager.php, line 77, the start() method, the boolean value returned from the handler's start method is ignored. Due to another bug, https://bugs.php.net/bug.php?id=71962, in php-memcached, when session_start is called, and the session key lock is already present, php-memcached will get the "value not stored" response, and reattempt until the configured timeout has expired (120 seconds). It then returns true, and presents an empty $_SESSION array, and still is referencing the same session key value, despite it having failed to legitimately acquire the lock.

      Because the USER element is now missing, when the initialise_user_session is called, the corresponding DB record for the original session will be removed (line 364), after a new session id is generated.

      Now, if the user attempts to refresh the page before the long-running operation has completed, or anything occurs before the original page request has an opportunity to call session_write_close, thus re-establishing the session state, then the original session state is lost--but even then, because the DB record was deleted by an AJAX request, on the next page request, the original session record will not be found and the original session will be discarded.

        Attachments

        1. MDL-53713-1.png
          MDL-53713-1.png
          111 kB
        2. MDL-53713-2.png
          MDL-53713-2.png
          332 kB
        3. MDL-53713-3.png
          MDL-53713-3.png
          403 kB

          Issue Links

            Activity

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  9/May/16