Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-38189

META Backup/restore issues with large courses


    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.3.4, 2.4.1
    • Fix Version/s: None
    • Component/s: Backup
    • Labels:
    • Affected Branches:


      I wrote a document describing a range of issues identified with backup/restore that prevent it functioning for large courses (either 'large' as in many activities, or 'large' as in large file sizes).

      Michael asked us to file these in the Moodle tracker. I've made this issue to track them, and filed subtasks for all the specific items.

      We don't intend to implement these ourselves at the present time. Also to note, I have set 'Affects version' as 2.3.x because that's where testing was done, but any changes on this scale clearly would not happen until a future Moodle major version.

      Here is some context about use of backup and restore at the Open University, which may explain why these issues cause us problems. We expect some of these factors will apply to other institutions too, now or in future.

      Many of our sites have the following characteristics which affect backup and restore performance:

      • A large number of activities. (Max: 1,770 activities. Median: 70. 39 courses with 500+.)
      • A large number of sections. (Max: 754 sections. Median: 100. 42 courses with 200+.)
      • A large number of files (due to the use of interactive activities and per-user data). (Max: 57,000 different files. Median: 70. 66 courses with 1,000+.)
      • Large files. Due to the provision of high-quality videos (and other files that include videos, such as EPUB3 ebooks), some of our sites contain a number of large files which add up to a large total size. (Max: 11GB in total. Median: 50MB. 129 courses have 1GB+.) (Note: These sizes include duplicate files with same contenthash; I wasn’t able to make a fast enough query that takes this into account.)
      • A large number of users (several thousand).

      We use backup and restore in two ways:

      • The standard Moodle backup and restore features for certain manual tasks.
      • A custom script to ‘roll forward’ a course to the next presentation (this uses the backup API and then the restore API, to automate the process).

        Gliffy Diagrams


            Issue Links

            Backup and restore operations should display progress Sub-task Closed Sam Marshall
            Backup and restore should use large memory limit Sub-task Closed Sam Marshall
            Backup and restore: Allow selection by activity type Sub-task Closed Sam Marshall
            Backup and restore should be tested with large backup files Sub-task Closed David Monllaó
            Large backup files should be downloadable Sub-task Closed Sam Marshall
            Backup/restore API should have better option to skip 'zip' operatiorn Sub-task Closed moodle.com
            Backup and restore should display log Sub-task Closed Sam Marshall
            Automated large-course generation for testing Sub-task Closed Sam Marshall
            Large course won't backup due to POST limit Sub-task Closed Sam Marshall
            Backup progress: add progress tracking inside long-running steps Sub-task Closed Sam Marshall
            Backup and restore progress: include file copies Sub-task Closed Sam Marshall
            Backup and restore progress: include forum discussions Sub-task Closed Sam Marshall
            Large course restore fails due to time limit when unzipping Sub-task Closed Sam Marshall
            Large course restore fails due to time limit on schema page Sub-task Closed Sam Marshall
            Large course restore times out on Review page Sub-task Closed Sam Marshall
            Restore: Need a way to display progress during UI stages Sub-task Closed Sam Marshall
            Restore: Processing restore times out in precheck Sub-task Closed Michael Aherne
            Restore: Progress bar needs to include more tasks Sub-task Closed Sam Marshall
            Backup: Very large course times out on user interface pages Sub-task Closed Sam Marshall
            Restore: Very large course times out Sub-task Closed Sam Marshall
            Improve time limit handling (including configuration for front-end servers) Sub-task Closed Sam Marshall



                • Votes:
                  7 Vote for this issue
                  14 Start watching this issue


                  • Created: