Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-38189

META Backup/restore issues with large courses

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Critical Critical
    • None
    • 2.3.4, 2.4.1
    • Backup
    • MOODLE_23_STABLE, MOODLE_24_STABLE

      I wrote a document describing a range of issues identified with backup/restore that prevent it functioning for large courses (either 'large' as in many activities, or 'large' as in large file sizes).

      Michael asked us to file these in the Moodle tracker. I've made this issue to track them, and filed subtasks for all the specific items.

      We don't intend to implement these ourselves at the present time. Also to note, I have set 'Affects version' as 2.3.x because that's where testing was done, but any changes on this scale clearly would not happen until a future Moodle major version.

      Here is some context about use of backup and restore at the Open University, which may explain why these issues cause us problems. We expect some of these factors will apply to other institutions too, now or in future.

      Many of our sites have the following characteristics which affect backup and restore performance:

      • A large number of activities. (Max: 1,770 activities. Median: 70. 39 courses with 500+.)
      • A large number of sections. (Max: 754 sections. Median: 100. 42 courses with 200+.)
      • A large number of files (due to the use of interactive activities and per-user data). (Max: 57,000 different files. Median: 70. 66 courses with 1,000+.)
      • Large files. Due to the provision of high-quality videos (and other files that include videos, such as EPUB3 ebooks), some of our sites contain a number of large files which add up to a large total size. (Max: 11GB in total. Median: 50MB. 129 courses have 1GB+.) (Note: These sizes include duplicate files with same contenthash; I wasn’t able to make a fast enough query that takes this into account.)
      • A large number of users (several thousand).

      We use backup and restore in two ways:

      • The standard Moodle backup and restore features for certain manual tasks.
      • A custom script to ‘roll forward’ a course to the next presentation (this uses the backup API and then the restore API, to automate the process).

        1. backup-issues.png
          backup-issues.png
          183 kB
        1.
        		 Backup and restore operations should display progress Sub-task Closed Sam Marshall
        2.
        		 Backup and restore should use large memory limit Sub-task Closed Sam Marshall
        3.
        		 Backup and restore: Allow selection by activity type Sub-task Closed Sam Marshall
        4.
        		 Backup and restore should be tested with large backup files Sub-task Closed David Monllaó
        5.
        		 Large backup files should be downloadable Sub-task Closed Sam Marshall
        6.
        		 Backup/restore API should have better option to skip 'zip' operatiorn Sub-task Closed Moodle HQ
        7.
        		 Backup and restore should display log Sub-task Closed Sam Marshall
        8.
        		 Automated large-course generation for testing Sub-task Closed Sam Marshall
        9.
        		 Large course won't backup due to POST limit Sub-task Closed Sam Marshall
        10.
        		 Backup progress: add progress tracking inside long-running steps Sub-task Closed Sam Marshall
        11.
        		 Backup and restore progress: include file copies Sub-task Closed Sam Marshall
        12.
        		 Backup and restore progress: include forum discussions Sub-task Closed Sam Marshall
        13.
        		 Large course restore fails due to time limit when unzipping Sub-task Closed Sam Marshall
        14.
        		 Large course restore fails due to time limit on schema page Sub-task Closed Sam Marshall
        15.
        		 Large course restore times out on Review page Sub-task Closed Sam Marshall
        16.
        		 Restore: Need a way to display progress during UI stages Sub-task Closed Sam Marshall
        17.
        		 Restore: Processing restore times out in precheck Sub-task Closed Michael Aherne
        18.
        		 Restore: Progress bar needs to include more tasks Sub-task Closed Sam Marshall
        19.
        		 Backup: Very large course times out on user interface pages Sub-task Closed Sam Marshall
        20.
        		 Restore: Very large course times out Sub-task Closed Sam Marshall
        21.
        		 Improve time limit handling (including configuration for front-end servers) Sub-task Closed Sam Marshall

            moodle.com Moodle HQ
            quen Sam Marshall
            Votes:
            7 Vote for this issue
            Watchers:
            14 Start watching this issue

              Created:
              Updated:
              Resolved:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.