We have been experiencing various issues with backup and restore of large courses (>2GB) during the course of our Moodle 1.9 -> 2.x upgrade, which I believe are the result of Moodle not being able to create / handle large ZIP files correctly.

We had a lot of problems doing a straight 1.9 -> 2.x upgrade with our large courses, so in the end we decided to remove the relevant courses from 1.9, perform the upgrade, and then re-import the courses afterwards.

The first issue we encountered when re-uploading the large courses is cosmetic - when selecting a course backup file larger than 2GB via Restore / Choose a file... (using a file repository), the file size is reported incorrectly as a negative number - e.g. 'Size: -950458409 bytes'. I belive this is probably a result of PHP using a signed integer to return the file size, as documented in the PHP man page (see http://php.net/manual/en/function.filesize.php). A number of solutions are suggested in the comments.

The second issue is that having selected the large course files, we experienced various errors when trying to extract them - one of them is returning 'Incorrect pool file content f162d6d1d025455940ff908c9af3e5116fa44788', and the other the following message (with debugging enabled):

error/tmp_backup_directory_not_found

More information about this error

Debug info: 

Error code: tmp_backup_directory_not_found 

$a contents: D:\moodledata/temp/backup/9ff309fa9612b2cac45d39121f0b36de

Stack trace: 

     line 140 of \backup\util\helper\convert_helper.class.php: convert_helper_exception thrown

     line 250 of \backup\util\helper\backup_general_helper.class.php: call to convert_helper::detect_moodle2_format()

     line 188 of \backup\util\ui\restore_ui_stage.class.php: call to backup_general_helper::detect_backup_format()

     line 67 of \backup\restore.php: call to restore_ui_stage_confirm->display()

I investigated to find out why this might be, and discovered that Moodle now uses the PHP ZipArchive class to unzip files before backup (http://php.net/manual/en/class.ziparchive.php). The >2GB ZIP file opens fine using other utilities (e.g. 7-Zip), but when I wrote a short PHP script to open the file, I discovered that the 'open' function was returning 19 - ZIPARCHIVE::ER_NOZIP.

I then looked into the Moodle code to see how this error was handled, but discovered that Moodle was not handling this error at all - see the following code from /moodle/lib/filestorage/zip_archive.php:

...

        $result = $this->za->open($archivepathname, $flags);

        if ($result === true) {

          (yadda yadda)

        } else {

            $this->za = null;

            $this->archivepathname = null;

            $this->encooding       = 'utf-8';

            // TODO: maybe we should return some error info

            return false;

        }

This may be related to MDL-31048 - either way, Moodle does not handle the ZipArchive class returning an error messsage at all, hence the 'unhelpful' error returned above.

With regards to the ZIPARCHIVE::ERR_NOZIP error itself, it seems as if this is a limitation of PHP itself at present, and specifically the 'libzip' library used by PHP - https://bugs.php.net/bug.php?id=55383, http://www.nih.at/listarchive/libzip-discuss/msg00078.html - although it may be partly fixed in the latest version of libzip - http://www.nih.at/listarchive/libzip-discuss/msg00190.html. Until PHP is updated it is always going to have problems with ZIP files greater than 2GB, and hence so will Moodle.

In addition to these problems, we have also been experiencing issues with backing up courses where the backup file size exceeds 2GB - I can see the course files being copied over, and a large (>2GB) ZIP file being created under moodledata\temp\backup\<random directory name>, but once this file is created it is immediately deleted, and the following error is returned (with debugging enabled):

Coding error detected, it must be fixed by a programmer: backup_helper::store_backup_file() expects valid $filepath parameter

More information about this error

Debug info:

Error code: codingerror

Stack trace:

    line 218 of \backup\util\helper\backup_helper.class.php: coding_exception thrown

    line 1596 of \backup\moodle2\backup_stepslib.php: call to backup_helper::store_backup_file()

    line 34 of \backup\util\plan\backup_execution_step.class.php: call to backup_store_backup_file->define_execution()

    line 153 of \backup\util\plan\base_task.class.php: call to backup_execution_step->execute()

    line 163 of \backup\util\plan\base_plan.class.php: call to base_task->execute()

    line 110 of \backup\util\plan\backup_plan.class.php: call to base_plan->execute()

    line 309 of \backup\controller\backup_controller.class.php: call to backup_plan->execute()

    line 111 of \backup\util\ui\backup_ui.class.php: call to backup_controller->execute_plan()

    line 89 of \backup\backup.php: call to backup_ui->execute()

I believe this is probably related to the same limitations as above, and possibly a similar lack of checking the return code when running $ziparch->close() in /moodle/lib/filestorage/zip_packer.php (archive_to_pathname function)

Either way, it looks like PHP is holding Moodle back in this area, so the only solution I can see (apart from handling the errors better) is to get PHP fixed. Alternatively the old Moodle 1.9 option of using external 'zip' and 'unzip' programs seemed to work fine - this is how we created these large backups in the first place - but I believe there are technical reasons against this?

If anyone has some leverage with the PHP developers, then getting the included libzip up to the latest version (0.10.1) might help with this and certain other issues (currenly PHP is complied with 0.9.1). However this has already been requested to resolve other issues (https://bugs.php.net/bug.php?id=59118, https://bugs.php.net/bug.php?id=51353) but the PHP developers seem to be dragging their heels a bit, so I'm not holding my breath about getting this issue resolved that way any time soon.

Thanks for your time, and any help with this issue would be greatly appreciated.