Affects Version/s: 2.3.1, 2.3.6, 2.4.1, 2.5
Fix Version/s: None
Environment:Windows Server 2008 R2, IIS 7.5, PHP 5.3.14
(This is also happening on 64-bit Linux deployments as well)
Create a course, and then test backing it up and restoring to a new course Add some large file(s) that will result in a course backup larger than 2G Install the externalzip local plugin from https://moodle.org/plugins/view.php?plugin=local_externalzip but do not follow patching instructions in the README file (these core modifications are already present in the branches above being submitted for review.) For each branch, navigate to the settings page for the externalzip plugin and ensure that "Zip file handler" is set to "External zip utility" For each course (each of which should now have >2G of content): test backing it up and confirm the resultant backup is >2G as expected test restoring that backup to a new course confirm the new course restores correctly and the large content is accessible
- Create a course, and then test backing it up and restoring to a new course
- Add some large file(s) that will result in a course backup larger than 2G
- Install the externalzip local plugin from https://moodle.org/plugins/view.php?plugin=local_externalzip but do not follow patching instructions in the README file (these core modifications are already present in the branches above being submitted for review.)
- For each branch, navigate to the settings page for the externalzip plugin and ensure that "Zip file handler" is set to "External zip utility"
- For each course (each of which should now have >2G of content):
- test backing it up and confirm the resultant backup is >2G as expected
- test restoring that backup to a new course
- confirm the new course restores correctly and the large content is accessible
Affected Branches:MOODLE_23_STABLE, MOODLE_24_STABLE, MOODLE_25_STABLE
Pull from Repository:
Pull Master Branch:wip_
We have been experiencing various issues with backup and restore of large courses (>2GB) during the course of our Moodle 1.9 -> 2.x upgrade, which I believe are the result of Moodle not being able to create / handle large ZIP files correctly.
We had a lot of problems doing a straight 1.9 -> 2.x upgrade with our large courses, so in the end we decided to remove the relevant courses from 1.9, perform the upgrade, and then re-import the courses afterwards.
The first issue we encountered when re-uploading the large courses is cosmetic - when selecting a course backup file larger than 2GB via Restore / Choose a file... (using a file repository), the file size is reported incorrectly as a negative number - e.g. 'Size: -950458409 bytes'. I belive this is probably a result of PHP using a signed integer to return the file size, as documented in the PHP man page (see http://php.net/manual/en/function.filesize.php). A number of solutions are suggested in the comments.
The second issue is that having selected the large course files, we experienced various errors when trying to extract them - one of them is returning 'Incorrect pool file content f162d6d1d025455940ff908c9af3e5116fa44788', and the other the following message (with debugging enabled):
I investigated to find out why this might be, and discovered that Moodle now uses the PHP ZipArchive class to unzip files before backup (http://php.net/manual/en/class.ziparchive.php). The >2GB ZIP file opens fine using other utilities (e.g. 7-Zip), but when I wrote a short PHP script to open the file, I discovered that the 'open' function was returning 19 - ZIPARCHIVE::ER_NOZIP.
I then looked into the Moodle code to see how this error was handled, but discovered that Moodle was not handling this error at all - see the following code from /moodle/lib/filestorage/zip_archive.php:
This may be related to MDL-31048 - either way, Moodle does not handle the ZipArchive class returning an error messsage at all, hence the 'unhelpful' error returned above.
With regards to the ZIPARCHIVE::ERR_NOZIP error itself, it seems as if this is a limitation of PHP itself at present, and specifically the 'libzip' library used by PHP - https://bugs.php.net/bug.php?id=55383, http://www.nih.at/listarchive/libzip-discuss/msg00078.html - although it may be partly fixed in the latest version of libzip - http://www.nih.at/listarchive/libzip-discuss/msg00190.html. Until PHP is updated it is always going to have problems with ZIP files greater than 2GB, and hence so will Moodle.
In addition to these problems, we have also been experiencing issues with backing up courses where the backup file size exceeds 2GB - I can see the course files being copied over, and a large (>2GB) ZIP file being created under moodledata\temp\backup\<random directory name>, but once this file is created it is immediately deleted, and the following error is returned (with debugging enabled):
I believe this is probably related to the same limitations as above, and possibly a similar lack of checking the return code when running $ziparch->close() in /moodle/lib/filestorage/zip_packer.php (archive_to_pathname function)
Either way, it looks like PHP is holding Moodle back in this area, so the only solution I can see (apart from handling the errors better) is to get PHP fixed. Alternatively the old Moodle 1.9 option of using external 'zip' and 'unzip' programs seemed to work fine - this is how we created these large backups in the first place - but I believe there are technical reasons against this?
If anyone has some leverage with the PHP developers, then getting the included libzip up to the latest version (0.10.1) might help with this and certain other issues (currenly PHP is complied with 0.9.1). However this has already been requested to resolve other issues (https://bugs.php.net/bug.php?id=59118, https://bugs.php.net/bug.php?id=51353) but the PHP developers seem to be dragging their heels a bit, so I'm not holding my breath about getting this issue resolved that way any time soon.
Thanks for your time, and any help with this issue would be greatly appreciated.