Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-52674

Consider use of tempdir in file_storage API

    XMLWordPrintable

    Details

    • Affected Branches:
      MOODLE_31_STABLE

      Description

      A set of new functions were created in MDL-44874 to help improve the status quo for clustered installations.

      The new function, make_request_directory() creates a new directory which will persist for the lifetime of the request, and then will be destroyed by a shutdown handler.
      This directory is placed within the localcachedir instead of tempdir and so it is local to the server, rather than a shared directory.

      There are a number of uses of the Moodle tempdir within the file_storage API, and we should consider whether there is any merit to moving this to use of the request directory system instead.

      Note, it's not necessarily a clear-cut case here.

      At the moment, these tempdir instances are used for:

      1. file_storage::create_file_from_url();
      2. tgz_packer::archive_to_storage;
      3. tgz_packer::extract_to_storage;
      4. zip_packer::archive_to_storage; and
      5. zip_packer::extract_to_storage.

      In each of these cases a temporary file in tempdir is used to store file content before it is then added into the file pool via create_file_from_pathname().

      This then calls add_file_to_pool() which copies the file from the tempdir location to the shared filestorage.

      In this case, there is a potential performance win:

      • If not in a clustered environment the localcachedir is probably on the filesystem, so there is no difference in current behaviour; and
      • If there is a correctly configured clustered environment, then we move from a remote source and remote destination, to a local source with remote destination. AIR most network file systems will cause OTW traffic for such a copy, so we should see a performance win in this situation. If anyone has evidence either way on this, it would be great to hear them.

      So pending confirmation on the nature of remote to remote file copies on NFS (and others), I think that we can gain a performance win by switching from tempdir to a per-request directory.

      The added fringe benefits here would include automatic tidying of the relevant files.
      Although these are individually unlinked, there is a potential for this to fail (e.g. a fatal error part-way through).

      In the case of file_storage, and zip_packer, I believe that the change is easy.
      I suspect more work may be required for the tgz side of things though. Sam Marshall, are you able to advise there perhaps?

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              dobedobedoh Andrew Lyons
              Participants:
              Component watchers:
              Matteo Scaramuccia, Andrew Lyons, Dongsheng Cai, Huong Nguyen, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze, Matteo Scaramuccia, Andrew Lyons, Dongsheng Cai, Huong Nguyen, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze
              Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated: