Moodle
  1. Moodle
  2. MDL-38059

Must not put cachedir on local server disks in cluster

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4
    • Fix Version/s: 2.3.5, 2.4.2
    • Component/s: Caching
    • Labels:
    • Rank:
      47856

      Description

      Creating this at Matteo's suggest in https://moodle.org/mod/forum/discuss.php?d=222250.

      This is my scenario. We currently have Moodle 2.4 in dev and test environments and will be setting up a production environment over the next couple of months. (We have been running 2.2 for the past couple of years.) We just realized that with the new caching framework (MUC), we should not have $CFG->cachedir on a NFS mount. Instead, we would like to put it on local disk on each of the eight production servers we are running.

      Putting cachedir on local disk on each server seems straightforward except for one thing: Cache purging does not work in this scenario. That is the bug I am reporting.

      If it is not a bug, but simply the case that local disk cache is not compatible with a cluster, any recommendations for this scenario? Memcache?

        Activity

        Hide
        Petr Škoda added a comment -

        Hello,

        for now (2.3-2.4) both tempdir and cache dir must be on a shared disk, in the future the tempdir may be on a local disk (after we fix all remaining incorrect uses of tempdir - we must store data there during one request only).

        Sam: could we somehow detect separate cachedirs? I suppose it would need some version number there and $CFG flag, it looks like a big potential problem on all misconfigured sites...

        Show
        Petr Škoda added a comment - Hello, for now (2.3-2.4) both tempdir and cache dir must be on a shared disk, in the future the tempdir may be on a local disk (after we fix all remaining incorrect uses of tempdir - we must store data there during one request only). Sam: could we somehow detect separate cachedirs? I suppose it would need some version number there and $CFG flag, it looks like a big potential problem on all misconfigured sites...
        Hide
        Matteo Scaramuccia added a comment -

        Hi Petr&Sam,
        Moodle offers $CFG->cachedir to be overridden in config.php since 2.2 i.e. months before MUC has seen the light.
        I'm thinking at least at these options:

        • a. removing that possibility for $CFG->cachedir, to be always safe regardless the ability of the administrator to create a cluster. Potentially, she/he can use symlinks to properly distribute the $CFG->dataroot and keep on breaking things . Not my favorite option
        • b. promoting a memory based store to be used in such a scenario, e.g. a pool of memcached servers, already supported by MUC and to be configured in a post-install stage, always required to move an instance towards a LB cluster. A mention in the docs is required
        • c. improving the purging in order to be able to trigger the action per server by means of a file in $CFG->dataroot, required to be shared among the nodes. This option could be a great improvement towards scalability for those definition relying on a file store, the majority in a default installation but it requires attention to let the purging be successful on every node included the last node charged to remove the file used to trigger the purging. Something risky.
        Show
        Matteo Scaramuccia added a comment - Hi Petr&Sam, Moodle offers $CFG->cachedir to be overridden in config.php since 2.2 i.e. months before MUC has seen the light. I'm thinking at least at these options: a. removing that possibility for $CFG->cachedir , to be always safe regardless the ability of the administrator to create a cluster. Potentially, she/he can use symlinks to properly distribute the $CFG->dataroot and keep on breaking things . Not my favorite option b. promoting a memory based store to be used in such a scenario, e.g. a pool of memcached servers, already supported by MUC and to be configured in a post-install stage, always required to move an instance towards a LB cluster. A mention in the docs is required c. improving the purging in order to be able to trigger the action per server by means of a file in $CFG->dataroot , required to be shared among the nodes. This option could be a great improvement towards scalability for those definition relying on a file store, the majority in a default installation but it requires attention to let the purging be successful on every node included the last node charged to remove the file used to trigger the purging. Something risky.
        Hide
        Petr Škoda added a comment -

        a. no, if you have one server only you can put cache and temp on ramdisk - that is the intended usage, if you use clusters it is your responsibility to setup everything correctly

        b. sure, there is a lot of room for improvements in our docs; I believe that shared memory is not the silver bullet suitable for every scenario

        c. we should imho resolve other problems first: please use shared cachedir and report problems with separate tempdir for now

        Show
        Petr Škoda added a comment - a. no, if you have one server only you can put cache and temp on ramdisk - that is the intended usage, if you use clusters it is your responsibility to setup everything correctly b. sure, there is a lot of room for improvements in our docs; I believe that shared memory is not the silver bullet suitable for every scenario c. we should imho resolve other problems first: please use shared cachedir and report problems with separate tempdir for now
        Hide
        Petr Škoda added a comment -

        Please note that separate cachedir will be probably never supported because we would have to somehow distribute cache updates to all copies, that means pretty much reimplement file sharing in PHP which sounds very complicated and would be a waste of time imo.

        I am going to update the config-dist docs now to say "cachedir MUST be on a shared filesystem in cluster"

        Show
        Petr Škoda added a comment - Please note that separate cachedir will be probably never supported because we would have to somehow distribute cache updates to all copies, that means pretty much reimplement file sharing in PHP which sounds very complicated and would be a waste of time imo. I am going to update the config-dist docs now to say "cachedir MUST be on a shared filesystem in cluster"
        Hide
        Dan Poltawski added a comment -

        Integrated, thanks

        Show
        Dan Poltawski added a comment - Integrated, thanks
        Hide
        David Monllaó added a comment -

        Passing as "No testing necessary"

        Show
        David Monllaó added a comment - Passing as "No testing necessary"
        Hide
        Damyon Wiese added a comment -

        Thanks for your hard work - this issue has made it! Moodle is now a little bit better.

        Regards, Damyon

        Show
        Damyon Wiese added a comment - Thanks for your hard work - this issue has made it! Moodle is now a little bit better. Regards, Damyon

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: