Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-58018

Reduce session lock contention with opt-in READ_ONLY_SESSION

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.3
    • Fix Version/s: None
    • Component/s: Performance
    • Labels:
      None
    • Testing Instructions:
      Hide

      Testing instructions:

      1. Fresh install of Moodle against PHP5 (PHP7 is fine too, and possibly better for showing the impact as the session locking will be far worse until the patch in MDL-57477 lands).

      2. After install, edit config.php to configure Moodle to talk to a memcached server for sessions and enable printing session acquisition times:

      // Always print how long session acquisition takes
      define('DEBUG_SESSION_TIMING_MIN_TIME', 0.0);
       
      $CFG->session_handler_class = '\core\session\memcached';
      $CFG->session_memcached_save_path = 'memcache:11211';
      $CFG->session_memcached_prefix = 'memc.sess.key.';
      $CFG->session_memcached_acquire_lock_timeout = 120;
      $CFG->session_memcached_lock_expire = 7200;
      

      3. While watching the php_error log, reload the dashboard page after logging in, and note similar to the following:

      [21-Feb-2017 00:57:32 ...<snip>... |duration:0.001s|session_url:/my/index.php
      [21-Feb-2017 00:57:34 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php
      [21-Feb-2017 00:57:34 ...<snip>... |duration:0.151s|session_url:/lib/ajax/service.php
      [21-Feb-2017 00:57:34 ...<snip>... |duration:0.307s|session_url:/lib/ajax/service.php
      

      Note the 3 concurrent AJAX requests that arrive immediately after the home page is loaded, and note that in this case the second one was blocked for one cycle (default memcached time between lock checking is 150ms) and the third had to wait for 300ms.

      4. Edit config.php to enable this patch:

      // Enable read only sessions, for pages/services that opt-in
      define('ENABLE_READ_ONLY_SESSIONS', true);
      

      5. Reload the home page:

      [21-Feb-2017 01:00:59 ...<snip>... |duration:0s|session_url:/my/index.php
      [21-Feb-2017 01:01:01 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php
      [21-Feb-2017 01:01:01 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php
      [21-Feb-2017 01:01:01 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php
      

      Note the 3 concurrent AJAX requests now waste no time with session acquisition. While one of them is configured (in the current version of the patch) to wait for a lock, the others don't need it, and as such not only load instantly, they also don't prevent the thread that need it from being blocked.

      Show
      Testing instructions: 1. Fresh install of Moodle against PHP5 (PHP7 is fine too, and possibly better for showing the impact as the session locking will be far worse until the patch in MDL-57477 lands). 2. After install, edit config.php to configure Moodle to talk to a memcached server for sessions and enable printing session acquisition times: // Always print how long session acquisition takes define( 'DEBUG_SESSION_TIMING_MIN_TIME' , 0.0);   $CFG ->session_handler_class = '\core\session\memcached' ; $CFG ->session_memcached_save_path = 'memcache:11211' ; $CFG ->session_memcached_prefix = 'memc.sess.key.' ; $CFG ->session_memcached_acquire_lock_timeout = 120; $CFG ->session_memcached_lock_expire = 7200; 3. While watching the php_error log, reload the dashboard page after logging in, and note similar to the following: [21-Feb-2017 00:57:32 ...<snip>... |duration:0.001s|session_url:/my/index.php [21-Feb-2017 00:57:34 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php [21-Feb-2017 00:57:34 ...<snip>... |duration:0.151s|session_url:/lib/ajax/service.php [21-Feb-2017 00:57:34 ...<snip>... |duration:0.307s|session_url:/lib/ajax/service.php Note the 3 concurrent AJAX requests that arrive immediately after the home page is loaded, and note that in this case the second one was blocked for one cycle (default memcached time between lock checking is 150ms) and the third had to wait for 300ms. 4. Edit config.php to enable this patch: // Enable read only sessions, for pages/services that opt-in define( 'ENABLE_READ_ONLY_SESSIONS' , true); 5. Reload the home page: [21-Feb-2017 01:00:59 ...<snip>... |duration:0s|session_url:/my/index.php [21-Feb-2017 01:01:01 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php [21-Feb-2017 01:01:01 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php [21-Feb-2017 01:01:01 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php Note the 3 concurrent AJAX requests now waste no time with session acquisition. While one of them is configured (in the current version of the patch) to wait for a lock, the others don't need it, and as such not only load instantly, they also don't prevent the thread that need it from being blocked.
    • Affected Branches:
      MOODLE_33_STABLE

      Description

      Background

      Many parts of Moodle now result in the browser making parallel requests to the Moodle server.

      For example, a home page load on a fresh Moodle installation makes no less than 3 AJAX requests in parallel. Similarly, calls to /pluginfile.php that return HTML, often result in multiple parallel requests for further resources such as JS, CSS, images etc.

      Problem Statement

      Currently when a request is processed by Moodle, a session is always started, and in order to start that session an exclusive lock is acquired on that session, which, in general, is not released until the server completes processing the request, where it is released by calling write_close() in the shutdown handler (although some pages explicitly release this early).

      The result is that even though the browser sends requests in parallel, once the requests hit the server they are in contention with each other, and effectively serialised by the session locking mechanism, resulting in busy spin locks and tying up server resources far longer than actually required. This is particularly apparent when the session lock code is buggy! (see MDL-57477, and PHP memcached issue: https://github.com/php-memcached-dev/php-memcached/issues/310)

      Observation

      In general, there is often no reason that a session lock is actually needed for cases where the processing of the request does not need any data to be persisted back to the session.

      For example, while retrieving the session is nearly always necessary (to check the user is authenticated and authorized to access the selected resource), for serving actual content, such as information about navigation items, where the context is fully specified in the request, or perhaps for serving static content, there is no need to acquire a session lock, where the underlying handler guarantees atomic operations (such as memcached, where a get / set is atomic).

      Proposed Solution

      We propose a patch that allows:
      1. Printing debug information on how long it takes to acquire a session.
      2. Ability for a page to declare that it needs only a READ_ONLY_SESSION, where, if supported by the session handler, sessions are acquired without waiting for a lock, sessions are not locked, and calls to write_close() are ignored.
      3. Opt-in of some key sources of contention to use this flag.

      Looking for feedback

      We would appreciate feedback on the validity of this approach, and the linked proposed patch.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                19 Vote for this issue
                Watchers:
                24 Start watching this issue

                Dates

                • Created:
                  Updated: