Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-38189

META Backup/restore issues with large courses

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.3.4, 2.4.1
    • Fix Version/s: None
    • Component/s: Backup
    • Labels:
    • Affected Branches:
      MOODLE_23_STABLE, MOODLE_24_STABLE

      Description

      I wrote a document describing a range of issues identified with backup/restore that prevent it functioning for large courses (either 'large' as in many activities, or 'large' as in large file sizes).

      Michael asked us to file these in the Moodle tracker. I've made this issue to track them, and filed subtasks for all the specific items.

      We don't intend to implement these ourselves at the present time. Also to note, I have set 'Affects version' as 2.3.x because that's where testing was done, but any changes on this scale clearly would not happen until a future Moodle major version.

      Here is some context about use of backup and restore at the Open University, which may explain why these issues cause us problems. We expect some of these factors will apply to other institutions too, now or in future.

      Many of our sites have the following characteristics which affect backup and restore performance:

      • A large number of activities. (Max: 1,770 activities. Median: 70. 39 courses with 500+.)
      • A large number of sections. (Max: 754 sections. Median: 100. 42 courses with 200+.)
      • A large number of files (due to the use of interactive activities and per-user data). (Max: 57,000 different files. Median: 70. 66 courses with 1,000+.)
      • Large files. Due to the provision of high-quality videos (and other files that include videos, such as EPUB3 ebooks), some of our sites contain a number of large files which add up to a large total size. (Max: 11GB in total. Median: 50MB. 129 courses have 1GB+.) (Note: These sizes include duplicate files with same contenthash; I wasn’t able to make a fast enough query that takes this into account.)
      • A large number of users (several thousand).

      We use backup and restore in two ways:

      • The standard Moodle backup and restore features for certain manual tasks.
      • A custom script to ‘roll forward’ a course to the next presentation (this uses the backup API and then the restore API, to automate the process).

        Gliffy Diagrams

          Attachments

            Issue Links

            1.
            Backup and restore operations should display progress Sub-task Closed Sam Marshall
             
            2.
            Backup and restore should use large memory limit Sub-task Closed Sam Marshall
             
            3.
            Backup and restore: Allow selection by activity type Sub-task Closed Sam Marshall
             
            4.
            Backup and restore should be tested with large backup files Sub-task Closed David Monllaó
             
            5.
            Large backup files should be downloadable Sub-task Closed Sam Marshall
             
            6.
            Backup/restore API should have better option to skip 'zip' operatiorn Sub-task Closed moodle.com
             
            7.
            Backup and restore should display log Sub-task Closed Sam Marshall
             
            8.
            Automated large-course generation for testing Sub-task Closed Sam Marshall
             
            9.
            Large course won't backup due to POST limit Sub-task Closed Sam Marshall
             
            10.
            Backup progress: add progress tracking inside long-running steps Sub-task Closed Sam Marshall
             
            11.
            Backup and restore progress: include file copies Sub-task Closed Sam Marshall
             
            12.
            Backup and restore progress: include forum discussions Sub-task Closed Sam Marshall
             
            13.
            Large course restore fails due to time limit when unzipping Sub-task Closed Sam Marshall
             
            14.
            Large course restore fails due to time limit on schema page Sub-task Closed Sam Marshall
             
            15.
            Large course restore times out on Review page Sub-task Closed Sam Marshall
             
            16.
            Restore: Need a way to display progress during UI stages Sub-task Closed Sam Marshall
             
            17.
            Restore: Processing restore times out in precheck Sub-task Closed Michael Aherne
             
            18.
            Restore: Progress bar needs to include more tasks Sub-task Closed Sam Marshall
             
            19.
            Backup: Very large course times out on user interface pages Sub-task Closed Sam Marshall
             
            20.
            Restore: Very large course times out Sub-task Closed Sam Marshall
             
            21.
            Improve time limit handling (including configuration for front-end servers) Sub-task Closed Sam Marshall
             

              Activity

                People

                • Votes:
                  7 Vote for this issue
                  Watchers:
                  14 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: