Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-35773

Include files option in backup settings

    XMLWordPrintable

    Details

    • Database:
      Any
    • Testing Instructions:
      Hide
      Initial setup
      1. Visit 'Site administration' > 'Development' > 'Make test course'.
      2. Create a small course.
      3. Visit the course.
      4. Backup the course with default values.
      5. Visit the restore area for this course.
      6. Check the size of the file in the backup area (should be approx ~9mb).
      Fileless UI backup
      1. Visit the course you created in the initial step.
      2. Backup the course via the UI but make sure to de-select “Include files” in the backup options.
      3. Check the backup completed successfully.
      4. Check the size of the backup and confirm it is much smaller than the initial backup file.
      5. Restore this backup into a new course.
      6. Check backup restore successfully and all files can be accessed in new course.
      Fileless Automated backup
      1. Visit 'Site administration' > 'Courses' > 'Backups' > 'Automated backup setup'.
      2. Set 'Active' to 'Manual'.
      3. Uncheck 'Include files' and save.
      4. Run php admin/cli/automated_backups.php.
      5. Visit the course you created in the initial step.
      6. Visit the restore area for this course.
      7. Scroll down to 'Automated backups'.
      8. Confirm the file size is smaller than the initial backup file.
      9. Restore this backup into a new course.
      10. Check backup restore successfully and all files can be accessed in new course.
      Adjustable trash cleanup
      1. Visit the course you created in the initial step.
      2. Visit the restore area for this course.
      3. Download a fileless backup of the initial course (one of the smaller sized backups).
      4. Delete all copies of the initial course (including restores made in previous steps).
      5. Visit 'Site administration' > 'Courses' > 'Restore course'.
      6. Use backup you downloaded earlier and restore as a new course.
      7. Check that all files exist in restored course.
      Adjustable trash cleanup fail
      1. Visit 'Site administration' > 'Server' > 'Cleanup'.
      2. Set 'Clean trash pool files' to 1 second.
      3. Wait a couple of seconds.
      4. Run php admin/tool/task/cli/schedule_task.php --execute='\core\task\file_trash_cleanup_task'.
      5. Visit the mdl_files table and delete any records that have the filearea set to draft.
      6. Visit your moodledata directory and delete the contents of the trashdir if it exists.
      7. Visit 'Site administration' > 'Courses' > 'Restore course'.
      8. Use backup you downloaded earlier and restore as a new course.
      9. Course should restore but will be missing files.
      Recycle bin
      1. Log in as the admin.
      2. Visit 'Site administration' > 'Courses' > 'Backups' > 'Automated backup setup'.
      3. Uncheck 'Include files'.
      4. Click 'Save changes'.
      5. Create a course.
      6. Enrol a user as a student.
      7. Create an assignment.
      8. Log in as the student.
      9. Visit the course.
      10. Add a submission with a file for the assignment.
      11. Log in as the admin.
      12. Visit the course.
      13. Delete the assignment.
      14. Run php admin/cli/cron.php
      15. Visit the mdl_files table and delete any records that have the filearea set to draft.
      16. Go to your moodledata directory and delete the trashdir if it exists.
      17. Visit the recycle bin by clicking on the course settings cog on the top-right (if you are using Boost).
      18. Click to restore the assignment.
      19. Visit the assignment.
      20. Confirm the file is still present.
      Show
      Initial setup Visit 'Site administration' > 'Development' > 'Make test course'. Create a small course. Visit the course. Backup the course with default values. Visit the restore area for this course. Check the size of the file in the backup area (should be approx ~9mb). Fileless UI backup Visit the course you created in the initial step. Backup the course via the UI but make sure to de-select “Include files” in the backup options. Check the backup completed successfully. Check the size of the backup and confirm it is much smaller than the initial backup file. Restore this backup into a new course. Check backup restore successfully and all files can be accessed in new course. Fileless Automated backup Visit 'Site administration' > 'Courses' > 'Backups' > 'Automated backup setup'. Set 'Active' to 'Manual'. Uncheck 'Include files' and save. Run php admin/cli/automated_backups.php . Visit the course you created in the initial step. Visit the restore area for this course. Scroll down to 'Automated backups'. Confirm the file size is smaller than the initial backup file. Restore this backup into a new course. Check backup restore successfully and all files can be accessed in new course. Adjustable trash cleanup Visit the course you created in the initial step. Visit the restore area for this course. Download a fileless backup of the initial course (one of the smaller sized backups). Delete all copies of the initial course (including restores made in previous steps). Visit 'Site administration' > 'Courses' > 'Restore course'. Use backup you downloaded earlier and restore as a new course. Check that all files exist in restored course. Adjustable trash cleanup fail Visit 'Site administration' > 'Server' > 'Cleanup'. Set 'Clean trash pool files' to 1 second. Wait a couple of seconds. Run php admin/tool/task/cli/schedule_task.php --execute='\core\task\file_trash_cleanup_task' . Visit the mdl_files table and delete any records that have the filearea set to draft . Visit your moodledata directory and delete the contents of the trashdir if it exists. Visit 'Site administration' > 'Courses' > 'Restore course'. Use backup you downloaded earlier and restore as a new course. Course should restore but will be missing files. Recycle bin Log in as the admin. Visit 'Site administration' > 'Courses' > 'Backups' > 'Automated backup setup'. Uncheck 'Include files'. Click 'Save changes'. Create a course. Enrol a user as a student. Create an assignment. Log in as the student. Visit the course. Add a submission with a file for the assignment. Log in as the admin. Visit the course. Delete the assignment. Run php admin/cli/cron.php Visit the mdl_files table and delete any records that have the filearea set to draft . Go to your moodledata directory and delete the trashdir if it exists. Visit the recycle bin by clicking on the course settings cog on the top-right (if you are using Boost). Click to restore the assignment. Visit the assignment. Confirm the file is still present.
    • Affected Branches:
      MOODLE_23_STABLE, MOODLE_31_STABLE, MOODLE_33_STABLE, MOODLE_34_STABLE, MOODLE_35_STABLE, MOODLE_36_STABLE, MOODLE_38_STABLE
    • Fixed Branches:
      MOODLE_38_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      MDL-35773_master

      Description

      When writing scripts which backup and then immediately restore to the same site, it would greatly improve performance if the system did not copy the files.

      In real usage it is easily possible for a course to be multiple gigabytes in size, and networked file systems are not necessarily all that fast, so it can add a minute or more to the time taken, potentially increasing the likelihood of timeouts. (Some front end load balancing webservers that direct requests time out after a few minutes if there is no output.)

      Using the import mode it is possible to prevent it from zipping the overall backup, but it is not currently possible (I think) to stop it from saving files. Also the restore does not work if the files are not present, even though it knows the content hash and therefore can check the file already exists.

      As well as custom external scripts, this would potentially also improve performance for the Moodle core 'duplicate' feature as I presume this also copies the file in the same manner. This might improve user experience a bit in that case, although with a single file it's obviously unlikely to be so huge it would actually time out.

      Eloy suggests the following setting:

      ->USE_AT_YOUR_OWN_RISK_im_gonna_restore_immediately_into_samesite_guy=true;

      Note we do not need this urgently or anything, it is just a nice-to-have that came up during development here.

        Attachments

        1. MDL-35773.jpg
          MDL-35773.jpg
          59 kB
        2. MDL-35773 (2).jpg
          MDL-35773 (2).jpg
          46 kB
        3. MDL-35773 (3).jpg
          MDL-35773 (3).jpg
          24 kB
        4. MDL-35773 (4).jpg
          MDL-35773 (4).jpg
          55 kB
        5. Screenshot from 2019-07-10 20-10-32.png
          Screenshot from 2019-07-10 20-10-32.png
          74 kB

          Issue Links

            Activity

              People

              Assignee:
              markn Mark Nelson
              Reporter:
              quen Sam Marshall
              Peer reviewer:
              Dmitrii Metelkin
              Integrator:
              Jake Dallimore
              Tester:
              Anna Carissa Sadia
              Participants:
              Component watchers:
              Adrian Greeve, Jake Dallimore, Mathew May, Mihail Geshoski, Peter Dias
              Votes:
              5 Vote for this issue
              Watchers:
              16 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Fix Release Date:
                18/Nov/19

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 days, 30 minutes
                  2d 30m