Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-69057

Extend File system api to get / write multiple files in parallel

    XMLWordPrintable

Details

    Description

      There are a couple code paths which are very file system latency bound, in particular h5p and the backup and restore processes which will potentially grab a very large number of files from the File API and copy them to somewhere and zip them or process them etc.

      A remote file system (eg objectfs / s3) has slightly worse latency which for a single file is not noticed but it all adds up and is multiplied.

      Proposing that a bunch of the methods like this:

      lib/filestorage/file_system.php

      85: public function get_local_path_from_storedfile(stored_file $file, $fetchifnotfound = false) {

      are expanded with signatures that take an array of stored_file's

      public function get_local_path_from_storedfile(array $files, $fetchifnotfound = false) 

      There are probably also touch points in stored_file and then also in the end code in h5p or backups which gathers up all the files and then gets them all at once.

      Then each file system api can implement whatever tricks that is can to speed this up, eg objectfs can do a parallel curl call and grab them in batches.

      lib/filestorage/stored_file.php would contain a base implementation which does a basic loops so everything gets it for free.

       

      Candidates which would benefit from this:

      1. Assignment bulk download to zip:
        mod/assign/view.php?id=1233&action=downloadall
      2. Course Backup 
      3. HVP 
      4. Book export
      5. mod folder download

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            brendanheywood Brendan Heywood
            Matteo Scaramuccia, David Woloszyn, Huong Nguyen, Jake Dallimore, Meirza, Michael Hawkins, Raquel Ortega, Safat Shahin, Stevani Andolo, Matteo Scaramuccia, David Woloszyn, Huong Nguyen, Jake Dallimore, Meirza, Michael Hawkins, Raquel Ortega, Safat Shahin, Stevani Andolo
            Votes:
            3 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated: