Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-69057

Extend File system api to get multiple files in parallel

    XMLWordPrintable

    Details

      Description

      There are a couple code paths which are very file system latency bound, in particular h5p and the backup and restore processes which will potentially grab a very large number of files from the File API and copy them to somewhere and zip them or process them etc.

      A remote file system (eg objectfs / s3) has slightly worse latency which for a single file is not noticed but it all adds up and is multiplied.

      Proposing that a bunch of the methods like this:

      lib/filestorage/file_system.php

      85: public function get_local_path_from_storedfile(stored_file $file, $fetchifnotfound = false) {

      are expanded with signatures that take an array of stored_file's

      public function get_local_path_from_storedfile(array $files, $fetchifnotfound = false) 

      There are probably also touch points in stored_file and then also in the end code in h5p or backups which gathers up all the files and then gets them all at once.

      Then each file system api can implement whatever tricks that is can to speed this up, eg objectfs can do a parallel curl call and grab them in batches.

      lib/filestorage/stored_file.php would contain a base implementation which does a basic loops so everything gets it for free.

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            brendanheywood Brendan Heywood
            Participants:
            Component watchers:
            Matteo Scaramuccia, Andrew Nicols, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze, Matteo Scaramuccia, Andrew Nicols, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated: