Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-58018

Reduce session lock contention with opt-in READ_ONLY_SESSION

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      Requirement:

      1. A server with the PHP memcached and Redis extensions installed (see memcached docs and Redis docs).
      2. A memcached daemon running on port 11211, eg memcached -d -m 24 -p 11211
      3. A Redis daemon running, eg redis-server /usr/local/etc/redis.conf

      Test 1 (memcached):

      1. Fresh install of Moodle.
      2. Add the attached test_async_requests_session_locking.php to your wwwroot.
      3. Edit config.php to configure Moodle to talk to the memcached server for sessions

        $CFG->session_handler_class = '\core\session\memcached';
        $CFG->session_memcached_save_path = '127.0.0.1:11211';
        $CFG->session_memcached_prefix = 'memc.sess.key.';
        $CFG->session_memcached_acquire_lock_timeout = 120;
        $CFG->session_memcached_lock_expire = 7200;
        

      4. Apply the changes in the commit https://github.com/markn86/moodle/commit/737691b4a0a2825af7d2bd6283949026bf5b7c91 to your code.
      5. Watch the php_error log (eg. tail -f /usr/local/var/log/httpd/error_log in your terminal) on another screen.
      6. Log in as the admin.
      7. Visit the page <yoursite>/test.php.
      8. Note similar to the following in your PHP error_log:

        [21-Feb-2017 00:57:32 ...<snip>... |duration:0.001s|session_url:/test.php
        [21-Feb-2017 00:57:34 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php
        [21-Feb-2017 00:57:34 ...<snip>... |duration:0.151s|session_url:/lib/ajax/service.php
        [21-Feb-2017 00:57:34 ...<snip>... |duration:0.307s|session_url:/lib/ajax/service.php
        

        Note the 3 concurrent AJAX requests that arrive immediately after the page is loaded, and note that in this case the second one was blocked for one cycle and the third had to wait for another cycle after that.

      9. Edit config.php and add the following line to enable this patch:

        $CFG->enable_read_only_sessions = true;
        

      10. Reload the page:
      11. Note similar to the following in your PHP error_log:

        [21-Feb-2017 01:00:59 ...<snip>... |duration:0s|session_url:/test.php
        [21-Feb-2017 01:01:01 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php
        [21-Feb-2017 01:01:01 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php
        [21-Feb-2017 01:01:01 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php
        

        Note the 3 concurrent AJAX requests now waste no time with session acquisition. While one of them is configured to wait for a lock, the others don't need it, and as such not only load instantly, they also don't prevent the thread that need it from being blocked.

      12. Edit config.php and edit the following setting to disable this patch:

        $CFG->enable_read_only_sessions = false;
        

      Test 2 (Redis):

      1. Same instructions as above except for step 3 configure Moodle to talk to the Redis server for sessions

        $CFG->session_handler_class = '\core\session\redis';
        $CFG->session_redis_host = '127.0.0.1';
        $CFG->session_redis_port = 6379;  // Optional.
        $CFG->session_redis_database = 0;  // Optional, default is db 0.
        $CFG->session_redis_auth = ''; // Optional, default is don't set one.
        $CFG->session_redis_prefix = ''; // Optional, default is don't set one.
        $CFG->session_redis_acquire_lock_timeout = 120;
        $CFG->session_redis_lock_expire = 7200;
        $CFG->session_redis_serializer_use_igbinary = false; // Optional, default is PHP builtin serializer.
        

      Test 3 (DB):

      1. Same instructions as above except for step 3 configure Moodle to talk to the DB for sessions

        $CFG->session_handler_class = '\core\session\database';
        $CFG->session_database_acquire_lock_timeout = 120;
        

      Test 4 (Read only requested but write performed):

      1. Fresh install of Moodle.
      2. Add the attached test_session_read_only_with_write.php to your wwwroot.
      3. Edit the file db/services.php and add 'requiresessionlock' => false to the core_user_update_user_preferences array.
      4. Watch the php_error log (eg. tail -f /usr/local/var/log/httpd/error_log in your terminal) on another screen.
      5. Log in as an admin.
      6. Visit <yoursite>/test_session_read_only_with_write.php
      7. Note in your PHP error_log you get a warning about a service saying it is read-only but isn't.
      Show
      Requirement: A server with the PHP memcached and Redis extensions installed (see memcached docs and Redis docs ). A memcached daemon running on port 11211, eg memcached -d -m 24 -p 11211 A Redis daemon running, eg redis-server /usr/local/etc/redis.conf Test 1 (memcached): Fresh install of Moodle. Add the attached test_async_requests_session_locking.php to your wwwroot. Edit config.php to configure Moodle to talk to the memcached server for sessions $CFG ->session_handler_class = '\core\session\memcached' ; $CFG ->session_memcached_save_path = '127.0.0.1:11211' ; $CFG ->session_memcached_prefix = 'memc.sess.key.' ; $CFG ->session_memcached_acquire_lock_timeout = 120; $CFG ->session_memcached_lock_expire = 7200; Apply the changes in the commit https://github.com/markn86/moodle/commit/737691b4a0a2825af7d2bd6283949026bf5b7c91 to your code. Watch the php_error log (eg. tail -f /usr/local/var/log/httpd/error_log in your terminal) on another screen. Log in as the admin. Visit the page <yoursite>/test.php. Note similar to the following in your PHP error_log: [21-Feb-2017 00:57:32 ...<snip>... |duration:0.001s|session_url:/test.php [21-Feb-2017 00:57:34 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php [21-Feb-2017 00:57:34 ...<snip>... |duration:0.151s|session_url:/lib/ajax/service.php [21-Feb-2017 00:57:34 ...<snip>... |duration:0.307s|session_url:/lib/ajax/service.php Note the 3 concurrent AJAX requests that arrive immediately after the page is loaded, and note that in this case the second one was blocked for one cycle and the third had to wait for another cycle after that. Edit config.php and add the following line to enable this patch: $CFG ->enable_read_only_sessions = true; Reload the page: Note similar to the following in your PHP error_log: [21-Feb-2017 01:00:59 ...<snip>... |duration:0s|session_url:/test.php [21-Feb-2017 01:01:01 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php [21-Feb-2017 01:01:01 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php [21-Feb-2017 01:01:01 ...<snip>... |duration:0.001s|session_url:/lib/ajax/service.php Note the 3 concurrent AJAX requests now waste no time with session acquisition. While one of them is configured to wait for a lock, the others don't need it, and as such not only load instantly, they also don't prevent the thread that need it from being blocked. Edit config.php and edit the following setting to disable this patch: $CFG ->enable_read_only_sessions = false; Test 2 (Redis): Same instructions as above except for step 3 configure Moodle to talk to the Redis server for sessions $CFG ->session_handler_class = '\core\session\redis' ; $CFG ->session_redis_host = '127.0.0.1' ; $CFG ->session_redis_port = 6379; // Optional. $CFG ->session_redis_database = 0; // Optional, default is db 0. $CFG ->session_redis_auth = '' ; // Optional, default is don't set one. $CFG ->session_redis_prefix = '' ; // Optional, default is don't set one. $CFG ->session_redis_acquire_lock_timeout = 120; $CFG ->session_redis_lock_expire = 7200; $CFG ->session_redis_serializer_use_igbinary = false; // Optional, default is PHP builtin serializer. Test 3 (DB): Same instructions as above except for step 3 configure Moodle to talk to the DB for sessions $CFG ->session_handler_class = '\core\session\database' ; $CFG ->session_database_acquire_lock_timeout = 120; Test 4 (Read only requested but write performed): Fresh install of Moodle. Add the attached test_session_read_only_with_write.php to your wwwroot. Edit the file db/services.php and add 'requiresessionlock' => false to the core_user_update_user_preferences array. Watch the php_error log (eg. tail -f /usr/local/var/log/httpd/error_log in your terminal) on another screen. Log in as an admin. Visit <yoursite>/test_session_read_only_with_write.php Note in your PHP error_log you get a warning about a service saying it is read-only but isn't.
    • Affected Branches:
      MOODLE_33_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      MDL-58018_master

      Description

      Background

      Many parts of Moodle now result in the browser making parallel requests to the Moodle server.

      For example, a home page load on a fresh Moodle installation makes no less than 3 AJAX requests in parallel. Similarly, calls to /pluginfile.php that return HTML, often result in multiple parallel requests for further resources such as JS, CSS, images etc.

      Problem Statement

      Currently when a request is processed by Moodle, a session is always started, and in order to start that session an exclusive lock is acquired on that session, which, in general, is not released until the server completes processing the request, where it is released by calling write_close() in the shutdown handler (although some pages explicitly release this early).

      The result is that even though the browser sends requests in parallel, once the requests hit the server they are in contention with each other, and effectively serialised by the session locking mechanism, resulting in busy spin locks and tying up server resources far longer than actually required. This is particularly apparent when the session lock code is buggy! (see MDL-57477, and PHP memcached issue: https://github.com/php-memcached-dev/php-memcached/issues/310)

      Observation

      In general, there is often no reason that a session lock is actually needed for cases where the processing of the request does not need any data to be persisted back to the session.

      For example, while retrieving the session is nearly always necessary (to check the user is authenticated and authorized to access the selected resource), for serving actual content, such as information about navigation items, where the context is fully specified in the request, or perhaps for serving static content, there is no need to acquire a session lock, where the underlying handler guarantees atomic operations (such as memcached, where a get / set is atomic).

      Proposed Solution

      We propose a patch that allows:
      1. Printing debug information on how long it takes to acquire a session.
      2. Ability for a page to declare that it needs only a READ_ONLY_SESSION, where, if supported by the session handler, sessions are acquired without waiting for a lock, sessions are not locked, and calls to write_close() are ignored.
      3. Opt-in of some key sources of contention to use this flag.

      Looking for feedback

      We would appreciate feedback on the validity of this approach, and the linked proposed patch.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                20 Vote for this issue
                Watchers:
                26 Start watching this issue

                Dates

                • Created:
                  Updated: