Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-68533

Allow mod_assign download all assignments to be streamed

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide
      Increase file upload size limit
      1. Edit your php.ini file. e.g.

        sudo xed /etc/php/7.4/apache2/php.ini
        

      2. Set the following config settings to the following values:
        • upload_max_filesize = 200M
        • post_max_size = 200M
      3. Save the php.ini file and restart apache

        sudo service apache2 restart
        

      4. Go to "Site administration / Security / Site security settings" and set "Maximum uploaded file size" to "Site upload limit (200MB)"
      5. Save the changes
      Test
      1. Log in as admin/course creator
      2. Create a course with a student.
      3. Create an assignment activity.
      4. On another browser session log in a student.
      5. Add a submission.
      6. Upload several files with a total size > 100 MB.
      7. On the admin's/course creator's browser session, go to the assignment and view all submissions.
      8. Select the student's submission and set "With selected..." to "Download selected submissions" then press "Go"
      9. Confirm the download of the submission.
      10. Confirm, the archive is downloading.
      11. While the archive is downloading open a new tab with any Moodle page and confirm it can be open, which means the session is not locked during assignment download.
      12. Confirm, that archive content is correct when finished downloading.
      Show
      Increase file upload size limit Edit your php.ini file. e.g. sudo xed /etc/php/7.4/apache2/php.ini Set the following config settings to the following values: upload_max_filesize = 200M post_max_size = 200M Save the php.ini file and restart apache sudo service apache2 restart Go to " Site administration / Security / Site security settings " and set " Maximum uploaded file size " to " Site upload limit (200MB) " Save the changes Test Log in as admin/course creator Create a course with a student. Create an assignment activity. On another browser session log in a student. Add a submission. Upload several files with a total size > 100 MB. On the admin's/course creator's browser session, go to the assignment and view all submissions. Select the student's submission and set " With selected... " to " Download selected submissions " then press " Go " Confirm the download of the submission. Confirm , the archive is downloading. While the archive is downloading open a new tab with any Moodle page and confirm it can be open, which means the session is not locked during assignment download. Confirm , that archive content is correct when finished downloading.
    • Fixed Branches:
      MOODLE_311_STABLE
    • Pull 3.11 Branch:
      MDL-68533_311-clean_param
    • Pull Master Branch:
      MDL-68533_master-clean_param

      Description

      This is almost the same pattern as MDL-68481, broadly the issue is this synchronously zips and download s massive chunk of files which can take ages and either hit or force various timeouts to be increased.

      I'm proposing:

      1) the zip preparation is done as a first step with a progress bar which then redirects to the second step which downloads. As the download is in a second http request it would need to copy the zip into shared disk and then clean it up

      2) both steps above can close the session

      3) bonus points (might be contentious): every time an assignment is uploaded then an ad hoc task can be rescheduled / queued for some healthy point into the future, say as 6 hours, which does the zip step ahead of time and stores it in the file api. By choosing 6 hours then many constant assignment uploads keep pushing back the final zip until it quiets down which should happen on the due date, lets say at midnight, and then the next day (assuming no late submissions) the zip is ready to go for the markers to download:

      https://docs.moodle.org/dev/Task_API#Rescheduling_an_adhoc_task

      If anyone still wants it in the mean time then you redo step 1 synchronously, and when it is done it would also de-queue the ad hoc task as it isn't needed any more. Most assignment files will not compress well because they are already compressed, so if this gets implemented maybe it could be behind a setting as it is trading off disk space vs download and generation time. If this setting is on then it would NOT need to cleanup the files in step 1, but we may want to clean up old zip files from old assignments so reduce disk.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              markn Mark Nelson
              Reporter:
              brendanheywood Brendan Heywood
              Peer reviewer:
              Brendan Heywood Brendan Heywood
              Integrator:
              Jun Pataleta Jun Pataleta
              Tester:
              Janelle Barcega Janelle Barcega
              Participants:
              Component watchers:
              Adrian Greeve, Jake Dallimore, Mathew May, Mihail Geshoski, Peter Dias, Sujith Haridasan, Matteo Scaramuccia, Andrew Lyons, Dongsheng Cai, Huong Nguyen, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze
              Votes:
              4 Vote for this issue
              Watchers:
              13 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Fix Release Date:
                17/May/21

                  Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 3 hours, 40 minutes
                  3h 40m