Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-68481

mod/folder/download_folder.php should be a streaming zip download

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide
      Increase file upload size limit
      1. Edit your php.ini file. e.g.

        sudo xed /etc/php/7.4/apache2/php.ini
        

      2. Set the following config settings to the following values:
        • upload_max_filesize = 200M
        • post_max_size = 200M
      3. Save the php.ini file and restart apache

        sudo service apache2 restart
        

      4. Go to "Site administration / Security / Site security settings" and set "Maximum uploaded file size" to "Site upload limit (200MB)"
      5. Save the changes
      Test
      1. Log in as admin
      2. Create a course.
      3. Create a folder resource.
      4. View the folder resource.
      5. Click on edit.
      6. Create another folder using the file manager,
      7. Upload a dozen of files with total size > 100 MB, some of them going in the folder you created above.
      8. Click 'Save changes'
      9. Click on 'Download folder'.
      10. Confirm, the archive is downloading.
      11. While the archive is downloading open a new tab with any Moodle page and confirm it can be opened, which means the session is not locked during folder download.
      12. Confirm, that archive content is correct when finished downloading.
      Show
      Increase file upload size limit Edit your php.ini file. e.g. sudo xed /etc/php/7.4/apache2/php.ini Set the following config settings to the following values: upload_max_filesize = 200M post_max_size = 200M Save the php.ini file and restart apache sudo service apache2 restart Go to " Site administration / Security / Site security settings " and set " Maximum uploaded file size " to " Site upload limit (200MB) " Save the changes Test Log in as admin Create a course. Create a folder resource. View the folder resource. Click on edit. Create another folder using the file manager, Upload a dozen of files with total size > 100 MB, some of them going in the folder you created above. Click 'Save changes' Click on 'Download folder'. Confirm , the archive is downloading. While the archive is downloading open a new tab with any Moodle page and confirm it can be opened, which means the session is not locked during folder download. Confirm , that archive content is correct when finished downloading.
    • Fixed Branches:
      MOODLE_311_STABLE
    • Pull 3.11 Branch:
      MDL-68481_311
    • Pull Master Branch:
      MDL-68481_master

      Description

      mod/folder/download_folder.php is consistently one of the longer running outrider processes.

      There is a few things it could do better:

      1) Getting to this page should be a get not a post (also improves web analytics)

      2) write_close the session fairly early

      3) opt in a a candidate for read only sessions MDL-58018 (might be ruled out because of blocks)

      4) the zip is blindly compressing everything which is an expensive mistake. Most files like audio / video / images and many office docs, ie 95% of what gets put into the mod_folder, are already compressed and so compressing these binaries again churns a lot of cpu and time for very limited compression, typically 1% - 2%. These files should be just appended to the zip file as is.

      5) stream the zip downloading MDL-68533 so we never touch temp disk, and the TTFB is near instant

       

      Bonus points:

      6) there is a bunch of heavy zip processing which could be done async ahead of time, and then the file download itself is a normal plugin file urls and not a post. So this zip file cannot benefit from any file system / object storage serving, including download pausing / resuming.

      We'd need some mod_folder event stuff to rebuild the zip if the contents change, if someone tries to download the file in the mean time it could be rebuild on the file but that can be done using a progress bar for the zipping part, and then a normal plugin file url at the end once the progress bar is done, ie a button + reload / redirect to download.

      The rebuilding event / ad hoc task can be throttled so it doesn't run immediately:

       https://docs.moodle.org/dev/Task_API#Rescheduling_an_adhoc_task

      This will consume a lot of disk so is a trade off of speed vs storage. I'd only do this depending on much impact steps 4 and 5 have. 

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              markn Mark Nelson
              Reporter:
              brendanheywood Brendan Heywood
              Peer reviewer:
              Brendan Heywood
              Integrator:
              Jun Pataleta
              Tester:
              Janelle Barcega
              Participants:
              Component watchers:
              Matteo Scaramuccia, Andrew Nicols, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze, Matteo Scaramuccia, Andrew Nicols, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze
              Votes:
              4 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Fix Release Date:
                10/May/21

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 hour, 15 minutes
                  1h 15m