Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-81987

1-second Redis timeout causes exceptions instead of degraded performance

XMLWordPrintable

    • MOODLE_403_STABLE, MOODLE_404_STABLE
    • MOODLE_403_STABLE, MOODLE_404_STABLE
    • MDL-81987_403_STABLE
    • MDL-81987_404_STABLE
    • MDL-81987_main
    • Hide

      Testing for this is fairly involved as it requires you to simulate high latency between Moodle and Redis.

      SETUP

      1. Install redis server using docker by following the below commands:

        $ docker pull ubuntu/redis  
        $ docker run --cap-add=NET_ADMIN -d --name redis-container -e TZ=UTC -e DEBIAN_FRONTEND=noninteractive -p 30073:6379 -e REDIS_PASSWORD=mypassword ubuntu/redis:latest
        $ docker exec redis-container apt update
        $ docker exec -it redis-container /bin/bash
        $ apt install iproute2

      2. To begin with, you will need Moodle configured to use a redis server as the session handler. Add the following to your config.php:

        $CFG->session_handler_class = '\core\session\redis';
        $CFG->session_redis_host = '127.0.0.1';
        $CFG->session_redis_port = 30073;
        $CFG->session_redis_auth = 'mypassword';

      3. There are two ways run the "tc" command. Enter to docker shell by running the below command:

        $ docker exec -it redis-container /bin/bash
        $ tc qdisc add dev eth0 root netem delay 100ms

        Or

        docker exec redis-container tc qdisc add dev eth0 root netem delay 100ms

      TESTING

      1. In the terminal on your redis server, as root run `tc qdisc add dev eth0 root netem delay 100ms` where `eth0` is the name of the outbound network interface. This will add 100ms latency to all outgoing connections.
      2. Load your Moodle site in the browser.
        1. Confirm that the page loads.
      3. On the redis server, as root run `tc qdisc change dev eth0 root netem delay 2000ms`. This will add 2s of latency.
      4. Load the Moodle site in the browser
        1. Confirm that the page loads.
        2. Before fix: An exception is thrown with the message "Failed to connect (try 1 out of 5) to Redis at redis:6379, the error returned was: Connection timed out" or "read error on connection to redis:6379"
      5. On the redis server, as root run `tc qdisc change dev eth0 root netem delay 11000ms`. This will add 11s of latency.
      6. Load the Moodle site in the browser
        1. Confirm An exception is thrown with the message "Failed to connect (try 1 out of 5) to Redis at redis:6379, the error returned was: Connection timed out"
      7. On the redis server, as root run `tc qdisc del dev eth0 root netem delay 11000ms` to remove the latency.

       

       

      Show
      Testing for this is fairly involved as it requires you to simulate high latency between Moodle and Redis. SETUP Install redis server using docker by following the below commands: $ docker pull ubuntu/redis $ docker run --cap-add=NET_ADMIN -d --name redis-container -e TZ=UTC -e DEBIAN_FRONTEND=noninteractive -p 30073 : 6379 -e REDIS_PASSWORD=mypassword ubuntu/redis:latest $ docker exec redis-container apt update $ docker exec -it redis-container /bin/bash $ apt install iproute2 To begin with, you will need Moodle configured to use a redis server as the session handler. Add the following to your config.php: $CFG->session_handler_class = '\core\session\redis' ; $CFG->session_redis_host = '127.0.0.1' ; $CFG->session_redis_port = 30073 ; $CFG->session_redis_auth = 'mypassword' ; There are two ways run the " tc " command. Enter to docker shell by running the below command: $ docker exec -it redis-container /bin/bash $ tc qdisc add dev eth0 root netem delay 100ms Or docker exec redis-container tc qdisc add dev eth0 root netem delay 100ms TESTING In the terminal on your redis server, as root run `tc qdisc add dev eth0 root netem delay 100ms` where `eth0` is the name of the outbound network interface. This will add 100ms latency to all outgoing connections. Load your Moodle site in the browser. Confirm that the page loads. On the redis server, as root run `tc qdisc change dev eth0 root netem delay 2000ms`. This will add 2s of latency. Load the Moodle site in the browser Confirm that the page loads. Before fix: An exception is thrown with the message "Failed to connect (try 1 out of 5) to Redis at redis:6379, the error returned was: Connection timed out" or "read error on connection to redis:6379" On the redis server, as root run `tc qdisc change dev eth0 root netem delay 11000ms`. This will add 11s of latency. Load the Moodle site in the browser Confirm An exception is thrown with the message "Failed to connect (try 1 out of 5) to Redis at redis:6379, the error returned was: Connection timed out" On the redis server, as root run `tc qdisc del dev eth0 root netem delay 11000ms` to remove the latency.    

      MDL-72622 added TLS support for Redis connections, and in doing so hard-coded the Redis connection timeout to 1 second.

      In the event of degraded performance in the infrastructure which causes the connection to take longer than 1 second, which we have seen recently during several incidents, this results in an exception being shown to the user and them being unable to access the site. Previously, it would have resulted in slow response times, but the site still would have been available.

      Given that before MDL-72622 there was no timeout set, and it doesn't appear to be a requirement of TLS connections, I suggest we either remove the timeout or set it to something longer, perhaps 10 seconds, before an exception is thrown.

            marxjohnson Mark Johnson
            marxjohnson Mark Johnson
            Meirza Meirza
            Ilya Tregubov Ilya Tregubov
            Kim Jared Lucas Kim Jared Lucas
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 6 hours, 25 minutes
                6h 25m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.