Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-41838

Backup: Support different compression method to allow files >4GB, other improvements

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide

      1. Find the admin option (experimental settings) enabletgzbackups. It should default to off; turn it on.

      2. Go to a suitable test course (I used the 'S' test course) and make a backup of the course using default settings.

      EXPECTED: Backup completes normally.

      3. Change the admin option to turn it off again.

      4. Make a backup of the same course, also using default settings.

      EXPECTED: Backup completes normally.

      EXPECTED: You can tell that different internal formats were used because the files will be different sizes. Normally, the .tar.gz file (the older one) will be smaller.

      5. Restore both backups to new courses, using default settings. (Note that the experimental option is now turned off, and there is one backup of each format.)

      EXPECTED: Both backups restore correctly, with no visible difference to the user. The restored courses appear the same.

      6. Download the .tar.gz format backup (the older/smaller one) to your computer. Rename it to .tar.gz. Open it using an archive program that supports .tar.gz (WinZip or gunzip -c whatever.tar.gz | tar x).

      EXPECTED: The file unpacks successfully, demonstrating that the .tar.gz format is valid. The contents look like a normal Moodle backup.

      7. Configure and test that automated backups work correctly.

      Show
      1. Find the admin option (experimental settings) enabletgzbackups. It should default to off; turn it on. 2. Go to a suitable test course (I used the 'S' test course) and make a backup of the course using default settings. EXPECTED: Backup completes normally. 3. Change the admin option to turn it off again. 4. Make a backup of the same course, also using default settings. EXPECTED: Backup completes normally. EXPECTED: You can tell that different internal formats were used because the files will be different sizes. Normally, the .tar.gz file (the older one) will be smaller. 5. Restore both backups to new courses, using default settings. (Note that the experimental option is now turned off, and there is one backup of each format.) EXPECTED: Both backups restore correctly, with no visible difference to the user. The restored courses appear the same. 6. Download the .tar.gz format backup (the older/smaller one) to your computer. Rename it to .tar.gz. Open it using an archive program that supports .tar.gz (WinZip or gunzip -c whatever.tar.gz | tar x). EXPECTED: The file unpacks successfully, demonstrating that the .tar.gz format is valid. The contents look like a normal Moodle backup. 7. Configure and test that automated backups work correctly.
    • Affected Branches:
      MOODLE_26_STABLE
    • Fixed Branches:
      MOODLE_26_STABLE
    • Pull Master Branch:
      MDL-41838-master

      Description

      If you try to create a backup that is more than 4GB it will not work because the PHP zip system does not support the 'ZIP64' extension and, as a result, is limited to a maximum total zip size of 4GB.

      This support will apparently be included in PHP 5.6. (For comparison, it was introduced in Java 7.)

      After discussion with Eloy, we have agreed that one possible solution (that works in our supported PHP versions and platforms) would be to add a new packer format, such as .tar.gz (the .gz part is supported by php libraries and the .tar format is quite simple). Some other considerations:

      1. A custom format would be much simpler to implement than using a standard format, but not being able to open it outside Moodle would make troubleshooting more difficult, so Eloy has ruled it out.

      2. Eloy notes that backup/restore only uses fixed (shortish) paths and ASCII characters in filenames, so it would be OK for another packer to have restrictions on paths/filenames so long as this is documented and it isn't used for anything else.

      3. PHP includes built in .tar support, but you have to tar something first and then gzip it, which is likely to be inefficient and cause timeout problems. As a result it might be better to do custom (limited) support.

      4. Any proposed change of format used for backup would have to be interchangeable for restore (i.e. restore would work either with zip format or the new one, automatically detecting) and, initially, would be deployed as an experimental setting only.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                1 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  18/Nov/13