Moodle
  1. Moodle
  2. MDL-34388

Issues with backup and restore of large courses / ZIP files - Moodle 2.3

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.3.1, 2.3.6, 2.4.1, 2.5
    • Fix Version/s: None
    • Component/s: Backup
    • Environment:
      Windows Server 2008 R2, IIS 7.5, PHP 5.3.14
      (This is also happening on 64-bit Linux deployments as well)
    • Testing Instructions:
      Hide
      1. Create a course, and then test backing it up and restoring to a new course
      2. Add some large file(s) that will result in a course backup larger than 2G
      3. Install the externalzip local plugin from https://moodle.org/plugins/view.php?plugin=local_externalzip but do not follow patching instructions in the README file (these core modifications are already present in the branches above being submitted for review.)
      4. For each branch, navigate to the settings page for the externalzip plugin and ensure that "Zip file handler" is set to "External zip utility"
      5. For each course (each of which should now have >2G of content):
        1. test backing it up and confirm the resultant backup is >2G as expected
        2. test restoring that backup to a new course
        3. confirm the new course restores correctly and the large content is accessible
      Show
      Create a course, and then test backing it up and restoring to a new course Add some large file(s) that will result in a course backup larger than 2G Install the externalzip local plugin from https://moodle.org/plugins/view.php?plugin=local_externalzip but do not follow patching instructions in the README file (these core modifications are already present in the branches above being submitted for review.) For each branch, navigate to the settings page for the externalzip plugin and ensure that "Zip file handler" is set to "External zip utility" For each course (each of which should now have >2G of content): test backing it up and confirm the resultant backup is >2G as expected test restoring that backup to a new course confirm the new course restores correctly and the large content is accessible
    • Affected Branches:
      MOODLE_23_STABLE, MOODLE_24_STABLE, MOODLE_25_STABLE
    • Pull 2.4 Branch:
      wip_MDL-34388_2.4_largebackuprestore
    • Pull Master Branch:
      wip_MDL-34388_2.5_largebackuprestore
    • Rank:
      42782

      Description

      We have been experiencing various issues with backup and restore of large courses (>2GB) during the course of our Moodle 1.9 -> 2.x upgrade, which I believe are the result of Moodle not being able to create / handle large ZIP files correctly.

      We had a lot of problems doing a straight 1.9 -> 2.x upgrade with our large courses, so in the end we decided to remove the relevant courses from 1.9, perform the upgrade, and then re-import the courses afterwards.

      The first issue we encountered when re-uploading the large courses is cosmetic - when selecting a course backup file larger than 2GB via Restore / Choose a file... (using a file repository), the file size is reported incorrectly as a negative number - e.g. 'Size: -950458409 bytes'. I belive this is probably a result of PHP using a signed integer to return the file size, as documented in the PHP man page (see http://php.net/manual/en/function.filesize.php). A number of solutions are suggested in the comments.

      The second issue is that having selected the large course files, we experienced various errors when trying to extract them - one of them is returning 'Incorrect pool file content f162d6d1d025455940ff908c9af3e5116fa44788', and the other the following message (with debugging enabled):

      error/tmp_backup_directory_not_found
      
      More information about this error
      
      Debug info: 
      Error code: tmp_backup_directory_not_found 
      $a contents: D:\moodledata/temp/backup/9ff309fa9612b2cac45d39121f0b36de
      Stack trace: 
           line 140 of \backup\util\helper\convert_helper.class.php: convert_helper_exception thrown
           line 250 of \backup\util\helper\backup_general_helper.class.php: call to convert_helper::detect_moodle2_format()
           line 188 of \backup\util\ui\restore_ui_stage.class.php: call to backup_general_helper::detect_backup_format()
           line 67 of \backup\restore.php: call to restore_ui_stage_confirm->display()
      

      I investigated to find out why this might be, and discovered that Moodle now uses the PHP ZipArchive class to unzip files before backup (http://php.net/manual/en/class.ziparchive.php). The >2GB ZIP file opens fine using other utilities (e.g. 7-Zip), but when I wrote a short PHP script to open the file, I discovered that the 'open' function was returning 19 - ZIPARCHIVE::ER_NOZIP.

      I then looked into the Moodle code to see how this error was handled, but discovered that Moodle was not handling this error at all - see the following code from /moodle/lib/filestorage/zip_archive.php:

      ...
              $result = $this->za->open($archivepathname, $flags);
      
              if ($result === true) {
                (yadda yadda)
              } else {
                  $this->za = null;
                  $this->archivepathname = null;
                  $this->encooding       = 'utf-8';
                  // TODO: maybe we should return some error info
                  return false;
              }
      

      This may be related to MDL-31048 - either way, Moodle does not handle the ZipArchive class returning an error messsage at all, hence the 'unhelpful' error returned above.

      With regards to the ZIPARCHIVE::ERR_NOZIP error itself, it seems as if this is a limitation of PHP itself at present, and specifically the 'libzip' library used by PHP - https://bugs.php.net/bug.php?id=55383, http://www.nih.at/listarchive/libzip-discuss/msg00078.html - although it may be partly fixed in the latest version of libzip - http://www.nih.at/listarchive/libzip-discuss/msg00190.html. Until PHP is updated it is always going to have problems with ZIP files greater than 2GB, and hence so will Moodle.

      In addition to these problems, we have also been experiencing issues with backing up courses where the backup file size exceeds 2GB - I can see the course files being copied over, and a large (>2GB) ZIP file being created under moodledata\temp\backup\<random directory name>, but once this file is created it is immediately deleted, and the following error is returned (with debugging enabled):

      Coding error detected, it must be fixed by a programmer: backup_helper::store_backup_file() expects valid $filepath parameter
      
      More information about this error
      Debug info:
      Error code: codingerror
      Stack trace:
      
          line 218 of \backup\util\helper\backup_helper.class.php: coding_exception thrown
          line 1596 of \backup\moodle2\backup_stepslib.php: call to backup_helper::store_backup_file()
          line 34 of \backup\util\plan\backup_execution_step.class.php: call to backup_store_backup_file->define_execution()
          line 153 of \backup\util\plan\base_task.class.php: call to backup_execution_step->execute()
          line 163 of \backup\util\plan\base_plan.class.php: call to base_task->execute()
          line 110 of \backup\util\plan\backup_plan.class.php: call to base_plan->execute()
          line 309 of \backup\controller\backup_controller.class.php: call to backup_plan->execute()
          line 111 of \backup\util\ui\backup_ui.class.php: call to backup_controller->execute_plan()
          line 89 of \backup\backup.php: call to backup_ui->execute()
      

      I believe this is probably related to the same limitations as above, and possibly a similar lack of checking the return code when running $ziparch->close() in /moodle/lib/filestorage/zip_packer.php (archive_to_pathname function)

      Either way, it looks like PHP is holding Moodle back in this area, so the only solution I can see (apart from handling the errors better) is to get PHP fixed. Alternatively the old Moodle 1.9 option of using external 'zip' and 'unzip' programs seemed to work fine - this is how we created these large backups in the first place - but I believe there are technical reasons against this?

      If anyone has some leverage with the PHP developers, then getting the included libzip up to the latest version (0.10.1) might help with this and certain other issues (currenly PHP is complied with 0.9.1). However this has already been requested to resolve other issues (https://bugs.php.net/bug.php?id=59118, https://bugs.php.net/bug.php?id=51353) but the PHP developers seem to be dragging their heels a bit, so I'm not holding my breath about getting this issue resolved that way any time soon.

      Thanks for your time, and any help with this issue would be greatly appreciated.

        Issue Links

          Activity

          Hide
          Christopher Hill added a comment -

          NB Other steps attempted to resolve this issue:

          Setting MySQL max_allowed_packet setting to 16MB
          Setting PHP memory_limit to -1
          Using 64-bit Windows build of PHP (http://www.anindya.com/)

          None of these made any difference.

          The reason why the courses are so large is that they contain SCORM and/or video files - we have a number of SCORM files which are 500MB+, and a few large (100MB) FLV files as well.

          Show
          Christopher Hill added a comment - NB Other steps attempted to resolve this issue: Setting MySQL max_allowed_packet setting to 16MB Setting PHP memory_limit to -1 Using 64-bit Windows build of PHP ( http://www.anindya.com/ ) None of these made any difference. The reason why the courses are so large is that they contain SCORM and/or video files - we have a number of SCORM files which are 500MB+, and a few large (100MB) FLV files as well.
          Hide
          Michael de Raadt added a comment -

          Thanks for investigating this. Please continue.

          If you are able to provide a patch or links to your Git repository branch, please add a patch label so we will spot it.

          Show
          Michael de Raadt added a comment - Thanks for investigating this. Please continue. If you are able to provide a patch or links to your Git repository branch, please add a patch label so we will spot it.
          Hide
          Richard Jones added a comment -

          This, or a very similar error is causing automated backups to fail after one particular course that contains very many videos. Using Moodle 2.3. "Coding error detected, it must be fixed by a programmer: backup_helper::store_backup_file() expects valid $filepath parameter"

          Subsequent courses at not backed up, regardless of size (partial report attached).

          Show
          Richard Jones added a comment - This, or a very similar error is causing automated backups to fail after one particular course that contains very many videos. Using Moodle 2.3. "Coding error detected, it must be fixed by a programmer: backup_helper::store_backup_file() expects valid $filepath parameter" Subsequent courses at not backed up, regardless of size (partial report attached).
          Hide
          Richard Jones added a comment -

          We have one course that contains a lot of video footage which generates the error: <Coding error detected, it must be fixed by a programmer: backup_helper::store_backup_file() expects valid $filepath parameter> when manually backed up.

          During automated backups it causes subsequent backups to fail (screenshot).

          Using Moodle 2.3, mysql, Debian Squeeze.

          Show
          Richard Jones added a comment - We have one course that contains a lot of video footage which generates the error: <Coding error detected, it must be fixed by a programmer: backup_helper::store_backup_file() expects valid $filepath parameter> when manually backed up. During automated backups it causes subsequent backups to fail (screenshot). Using Moodle 2.3, mysql, Debian Squeeze.
          Hide
          Richard Jones added a comment - - edited

          Comments weren't showing on Chrome - hence duplicate . Unable to attach screenshot, log reads:

          nanogong test 17 Aug, 02:55 - 17 Aug, 02:55 OK 21 Aug, 02:05
          Under 15A Rugby 17 Aug, 02:55 - 17 Aug, 02:55 OK 21 Aug, 02:05
          1. 1st XI Cricket Video Analysis 17 Aug, 02:55 - 17 Aug, 03:01 Error 21 Aug, 02:05
          Science Staff Training 17 Aug, 03:01 - 17 Aug, 03:01 Error 21 Aug, 02:05
          Year 6 Historical Narrative 17 Aug, 03:01 - 17 Aug, 03:01 Error 21 Aug, 02:05
          Yr 7 ICT SEM 2 17 Aug, 03:01 - 17 Aug, 03:01 Error 21 Aug, 02:05
          Year 10 IPT (Programming/Robotics) 17 Aug, 03:01 - 17 Aug, 03:01 Error 21 Aug, 02:05

          etc

          Show
          Richard Jones added a comment - - edited Comments weren't showing on Chrome - hence duplicate . Unable to attach screenshot, log reads: nanogong test 17 Aug, 02:55 - 17 Aug, 02:55 OK 21 Aug, 02:05 Under 15A Rugby 17 Aug, 02:55 - 17 Aug, 02:55 OK 21 Aug, 02:05 1. 1st XI Cricket Video Analysis 17 Aug, 02:55 - 17 Aug, 03:01 Error 21 Aug, 02:05 Science Staff Training 17 Aug, 03:01 - 17 Aug, 03:01 Error 21 Aug, 02:05 Year 6 Historical Narrative 17 Aug, 03:01 - 17 Aug, 03:01 Error 21 Aug, 02:05 Yr 7 ICT SEM 2 17 Aug, 03:01 - 17 Aug, 03:01 Error 21 Aug, 02:05 Year 10 IPT (Programming/Robotics) 17 Aug, 03:01 - 17 Aug, 03:01 Error 21 Aug, 02:05 etc
          Hide
          Christopher Hill added a comment -

          Just to say that we have the same issue with automated backups failing when a large course is reached.

          Show
          Christopher Hill added a comment - Just to say that we have the same issue with automated backups failing when a large course is reached.
          Hide
          Christopher Hill added a comment -

          FYI - PHP was updated to libzip 0.10.1 as at PHP 5.4.5 / PHP 5.3.15 (see https://bugs.php.net/bug.php?id=51353), but this has not resolved the problem.

          We are still waiting for an update to libzip to fully support ZIP64 - however, if Moodle could be fixed in the mean time to handle the PHP zip function errors properly, that would be great.

          I do not have the Moodle / PHP expertise to carry out this work myself.

          Show
          Christopher Hill added a comment - FYI - PHP was updated to libzip 0.10.1 as at PHP 5.4.5 / PHP 5.3.15 (see https://bugs.php.net/bug.php?id=51353 ), but this has not resolved the problem. We are still waiting for an update to libzip to fully support ZIP64 - however, if Moodle could be fixed in the mean time to handle the PHP zip function errors properly, that would be great. I do not have the Moodle / PHP expertise to carry out this work myself.
          Hide
          Sean Keogh added a comment -

          Hi,

          We have a client that is a also experiencing this. Same error: "codingerror backup_helper::store_backup_file() expects valid $filepath parameter" And as they are running on their own windows server they don't have any option of using external zip and unzip anyway. This is pretty urgent - the kind of organisations that we support (schools and colleges) often have large courses.

          Show
          Sean Keogh added a comment - Hi, We have a client that is a also experiencing this. Same error: "codingerror backup_helper::store_backup_file() expects valid $filepath parameter" And as they are running on their own windows server they don't have any option of using external zip and unzip anyway. This is pretty urgent - the kind of organisations that we support (schools and colleges) often have large courses.
          Hide
          Thomas van den Heuvel added a comment -

          What might solve your problem is, if you use file system repositories, re-upload the course ZIP-files into your repository (explicitly) in binary-mode. Although that was not directly the cause of my problem (still do not know why the original uploaded course seemed corrupt), re-uploading the courses seemed to solve the issue. Also see this topic http://moodle.org/mod/forum/discuss.php?d=207287.

          Hope this helps.

          Show
          Thomas van den Heuvel added a comment - What might solve your problem is, if you use file system repositories, re-upload the course ZIP-files into your repository (explicitly) in binary-mode. Although that was not directly the cause of my problem (still do not know why the original uploaded course seemed corrupt), re-uploading the courses seemed to solve the issue. Also see this topic http://moodle.org/mod/forum/discuss.php?d=207287 . Hope this helps.
          Hide
          Christopher Hill added a comment -

          That is a helpful suggestion to make, but we are running our own Windows server, copying the files directly into the file system, and hence it doesn't apply in our case.

          Good thing for others to check though.

          Show
          Christopher Hill added a comment - That is a helpful suggestion to make, but we are running our own Windows server, copying the files directly into the file system, and hence it doesn't apply in our case. Good thing for others to check though.
          Hide
          Anthony Borrow added a comment -

          Just to give a link to some of the history that led to external zip being used leading to the current implementation. Peace - Anthony

          Show
          Anthony Borrow added a comment - Just to give a link to some of the history that led to external zip being used leading to the current implementation. Peace - Anthony
          Hide
          Anthony Borrow added a comment -

          For those with access to http://moodle.org/plugins/view.php?plugin=local_externalzip - this local plugin was presented as a possible workaround but is *nix specific so not a good solution. Hopefully it can get fixed. Peace - Anthony

          Show
          Anthony Borrow added a comment - For those with access to http://moodle.org/plugins/view.php?plugin=local_externalzip - this local plugin was presented as a possible workaround but is *nix specific so not a good solution. Hopefully it can get fixed. Peace - Anthony
          Hide
          Christopher Hill added a comment -

          For everyone's information - current situation in all this is that latest libzip now does support Zip64 in Mercurial (http://nih.at/listarchive/libzip-discuss/msg00270.html), but has not yet been released as a 'formal' new version yet.

          We are waiting for this to be released as a 'formal' version so that it can be included in PHP.

          Once this is done we can test with Moodle again and see if it works.

          In the mean time Moodle could still be updated to check the return of za->open and $ziparch->close(), but at the moment this would just make the error messages a bit more comprehensible. The only other solution is to go back to using external zip/unzip commands, which it seems that the devs are reluctant to do due to other technical issues.

          Show
          Christopher Hill added a comment - For everyone's information - current situation in all this is that latest libzip now does support Zip64 in Mercurial ( http://nih.at/listarchive/libzip-discuss/msg00270.html ), but has not yet been released as a 'formal' new version yet. We are waiting for this to be released as a 'formal' version so that it can be included in PHP. Once this is done we can test with Moodle again and see if it works. In the mean time Moodle could still be updated to check the return of za->open and $ziparch->close(), but at the moment this would just make the error messages a bit more comprehensible. The only other solution is to go back to using external zip/unzip commands, which it seems that the devs are reluctant to do due to other technical issues.
          Hide
          Petr Škoda added a comment -

          Are you sure that 32bit PHP (or 64bit windows PHP) can handle these huge files? I would personally expect major problems even in 64bit PHP on linux.

          Show
          Petr Škoda added a comment - Are you sure that 32bit PHP (or 64bit windows PHP) can handle these huge files? I would personally expect major problems even in 64bit PHP on linux.
          Hide
          Christopher Hill added a comment -

          If it can't, then either it has to be fixed so that it can, or we need external zip/unzip support back.

          Moodle 1.9 worked with these files just fine.

          Show
          Christopher Hill added a comment - If it can't, then either it has to be fixed so that it can, or we need external zip/unzip support back. Moodle 1.9 worked with these files just fine.
          Hide
          Petr Škoda added a comment -

          "Note: Because PHP's integer type is signed and many platforms use 32bit integers, some filesystem functions may return unexpected results for files which are larger than 2GB." from http://php.net/manual/en/function.filesize.php

          That disqualifies the large files (over 2GB) from Moodle file API completely, the only things that could theoretically work is backup directly to native filesystem directory and restore from native filesystem directory.

          In backup archives there are no unicode characters so I suppose it would be technically possible to implement native zip/unzip in backup/restore only without changing the normal Moodle file API. There might be performance problems with copying of all the backed up files to a temporary location. Another trouble is selection of file to be restored - it would not survive the trip through our file picker.

          Show
          Petr Škoda added a comment - "Note: Because PHP's integer type is signed and many platforms use 32bit integers, some filesystem functions may return unexpected results for files which are larger than 2GB." from http://php.net/manual/en/function.filesize.php That disqualifies the large files (over 2GB) from Moodle file API completely, the only things that could theoretically work is backup directly to native filesystem directory and restore from native filesystem directory. In backup archives there are no unicode characters so I suppose it would be technically possible to implement native zip/unzip in backup/restore only without changing the normal Moodle file API. There might be performance problems with copying of all the backed up files to a temporary location. Another trouble is selection of file to be restored - it would not survive the trip through our file picker.
          Hide
          Christopher Hill added a comment -

          As far as I can see the only issue at the moment is with the ZIP extension. There is a cosmetic one with the file size being incorrectly detected (see my comments on this in the bug) but this doesn't seem to be a showstopper at present (and can be worked around, according to the PHP docs). Of course once PHP has zip64 support we might find other issues. But I don't think the problem of >2GB file sizes is going to go away.

          Show
          Christopher Hill added a comment - As far as I can see the only issue at the moment is with the ZIP extension. There is a cosmetic one with the file size being incorrectly detected (see my comments on this in the bug) but this doesn't seem to be a showstopper at present (and can be worked around, according to the PHP docs). Of course once PHP has zip64 support we might find other issues. But I don't think the problem of >2GB file sizes is going to go away.
          Hide
          Eric Merrill added a comment -

          PHP >= 5.3.15 and >= 5.4.5 supposedly includes zip64 support. We are working on getting our servers upgraded to see if this fixed our problems.

          Show
          Eric Merrill added a comment - PHP >= 5.3.15 and >= 5.4.5 supposedly includes zip64 support. We are working on getting our servers upgraded to see if this fixed our problems.
          Hide
          Aaron Wells added a comment -

          Just thought I'd mention, I'm the author of the plugin mentioned above, for using external zip & unzip: http://moodle.org/plugins/view.php?plugin=local_externalzip

          I've been using it in production for a couple of weeks for a client who backs up and restores a lot of >2GB backup files, and it has been working so far. The approach I took was basically a drop-in alternative for the zip_archive class (Moodle's wrapper around ZipArchive). On the downside, it unzips the entire archive into a temp directory, which does take up a lot of space. But then again, if you're dealing with huge backup files, you're already taking up a lot of space.

          I've only tested it on Ubuntu, but you could probably adapt it for Windows by just changing the externalzip_archive::exec() function, which is the part that actually invokes the external zip program.

          Show
          Aaron Wells added a comment - Just thought I'd mention, I'm the author of the plugin mentioned above, for using external zip & unzip: http://moodle.org/plugins/view.php?plugin=local_externalzip I've been using it in production for a couple of weeks for a client who backs up and restores a lot of >2GB backup files, and it has been working so far. The approach I took was basically a drop-in alternative for the zip_archive class (Moodle's wrapper around ZipArchive). On the downside, it unzips the entire archive into a temp directory, which does take up a lot of space. But then again, if you're dealing with huge backup files, you're already taking up a lot of space. I've only tested it on Ubuntu, but you could probably adapt it for Windows by just changing the externalzip_archive::exec() function, which is the part that actually invokes the external zip program.
          Hide
          Moodle Breaker added a comment -

          We're having this problem with restoring all 2.3 backups. 1.9 courses restore fine but native 2.3 backups give an 'incorrect pool file content' error.
          It doesn't appear to have anything to do with course size (none of the files I've been trying to restore are anywwhere close to 2GB, typically more like 10-30MB). Our site is running on 64bit Linux (Ubuntu). Apologies if this is the wrong place to post this but the error message appears to be the same so I thought I'd add it here in case it helps isolate that maybe it's not: Windows, 32bit, filesize...

          Show
          Moodle Breaker added a comment - We're having this problem with restoring all 2.3 backups. 1.9 courses restore fine but native 2.3 backups give an 'incorrect pool file content' error. It doesn't appear to have anything to do with course size (none of the files I've been trying to restore are anywwhere close to 2GB, typically more like 10-30MB). Our site is running on 64bit Linux (Ubuntu). Apologies if this is the wrong place to post this but the error message appears to be the same so I thought I'd add it here in case it helps isolate that maybe it's not: Windows, 32bit, filesize...
          Hide
          Christopher Hill added a comment -

          Hi,

          Just to say that we gave http://moodle.org/plugins/view.php?plugin=local_externalzip a try on our Windows server, using the GnuWin32 packages for Zip and Unzip (http://gnuwin32.sourceforge.net/), and although it all installed OK on our Windows server & we didn't get any errors, we found that the backup files which were being created just contained a copy of the moodle\backup folder from our Web root, rather than the course backup files as expected.

          This plugin does seem to hold some promise, but it would need to be updated to support Windows properly before it could be considered a true solution.

          Show
          Christopher Hill added a comment - Hi, Just to say that we gave http://moodle.org/plugins/view.php?plugin=local_externalzip a try on our Windows server, using the GnuWin32 packages for Zip and Unzip ( http://gnuwin32.sourceforge.net/ ), and although it all installed OK on our Windows server & we didn't get any errors, we found that the backup files which were being created just contained a copy of the moodle\backup folder from our Web root, rather than the course backup files as expected. This plugin does seem to hold some promise, but it would need to be updated to support Windows properly before it could be considered a true solution.
          Hide
          German Valero added a comment -

          Hi,
          I looked at the plugin written by Aaron Wells in https://moodle.org/plugins/view.php?plugin=local_externalzip , and while this seems a worthy addition to many Moodle sites on Linux, I could not find the language strings available for translating it to other languages in AMOS. This could be due to the language strings not properly managed (but they are indeed in the local_externalzip.php file). Would Aaron Wells please look at http://lang.moodle.org/mod/forum/discuss.php?d=2485 .
          Thanks

          Show
          German Valero added a comment - Hi, I looked at the plugin written by Aaron Wells in https://moodle.org/plugins/view.php?plugin=local_externalzip , and while this seems a worthy addition to many Moodle sites on Linux, I could not find the language strings available for translating it to other languages in AMOS. This could be due to the language strings not properly managed (but they are indeed in the local_externalzip.php file). Would Aaron Wells please look at http://lang.moodle.org/mod/forum/discuss.php?d=2485 . Thanks
          Hide
          Michael de Raadt added a comment -

          In MDL-30962 there is a reasonable suggestion to use volume splitting to resolve the problem of large backups. Of course that doesn't solve the problem of zips created without volume splitting.

          Show
          Michael de Raadt added a comment - In MDL-30962 there is a reasonable suggestion to use volume splitting to resolve the problem of large backups. Of course that doesn't solve the problem of zips created without volume splitting.
          Hide
          Derek Chirnside added a comment -

          I've just been trying to restore a 1GIG file and getting a similar error.

          Debug info:
          Error code: tmp_backup_directory_not_found
          $a contents: /home/moodledata/temp/backup/b173e1aea24850bbb99c1440f19b2bdf
          Stack trace:
          line 140 of /backup/util/helper/convert_helper.class.php: convert_helper_exception thrown
          line 284 of /backup/util/helper/backup_general_helper.class.php: call to convert_helper::detect_moodle2_format()
          line 188 of /backup/util/ui/restore_ui_stage.class.php: call to backup_general_helper::detect_backup_format()
          line 67 of /backup/restore.php: call to restore_ui_stage_confirm->display()

          I have managed a 490MEG file restore. I'm curious what the limit is. I'm going to try to delete say 200MEG from the 1GIG course and try to restore 800MEG.
          I realise this is a complex issue.

          If this is a known issue, why is it not more clearly spelled out? When in Moodle you take a backup of a course, then should it not say at the time "Caution, this is Moodle version 2.3 and it cannot be used to restore this backup file, please see tracker xxxx"
          I notice the comment "Moodle is not supposed to be used to store big files" occurring again. Point taken but <sigh> this sort of disclaimer should be a little more obvious.

          Well coders, I hope someone is able to make some progress. This does seem to be the key tracker number for this issue.

          As a final comment, https://tracker.moodle.org/browse/MDL-30962 does seem to hint at <2GIG as being possible. Should we be able to tweak the parameters to get Moodle 2.x to restore 1.9GIG?

          -Derek

          Show
          Derek Chirnside added a comment - I've just been trying to restore a 1GIG file and getting a similar error. Debug info: Error code: tmp_backup_directory_not_found $a contents: /home/moodledata/temp/backup/b173e1aea24850bbb99c1440f19b2bdf Stack trace: line 140 of /backup/util/helper/convert_helper.class.php: convert_helper_exception thrown line 284 of /backup/util/helper/backup_general_helper.class.php: call to convert_helper::detect_moodle2_format() line 188 of /backup/util/ui/restore_ui_stage.class.php: call to backup_general_helper::detect_backup_format() line 67 of /backup/restore.php: call to restore_ui_stage_confirm->display() I have managed a 490MEG file restore. I'm curious what the limit is. I'm going to try to delete say 200MEG from the 1GIG course and try to restore 800MEG. I realise this is a complex issue. If this is a known issue, why is it not more clearly spelled out? When in Moodle you take a backup of a course, then should it not say at the time "Caution, this is Moodle version 2.3 and it cannot be used to restore this backup file, please see tracker xxxx" I notice the comment "Moodle is not supposed to be used to store big files" occurring again. Point taken but <sigh> this sort of disclaimer should be a little more obvious. Well coders, I hope someone is able to make some progress. This does seem to be the key tracker number for this issue. As a final comment, https://tracker.moodle.org/browse/MDL-30962 does seem to hint at <2GIG as being possible. Should we be able to tweak the parameters to get Moodle 2.x to restore 1.9GIG? -Derek
          Hide
          Derek Chirnside added a comment -

          Just to restate the question, probably buried in verbiage in my last post: https://tracker.moodle.org/browse/MDL-30962 does seem to hint at <2GIG as being possible. Should we be able to tweak the parameters to get Moodle 2.x to restore 1.9GIG?
          -Derek

          Show
          Derek Chirnside added a comment - Just to restate the question, probably buried in verbiage in my last post: https://tracker.moodle.org/browse/MDL-30962 does seem to hint at <2GIG as being possible. Should we be able to tweak the parameters to get Moodle 2.x to restore 1.9GIG? -Derek
          Hide
          Derek Chirnside added a comment -

          @Michael: I've just been looking at MDL-30962.
          You say: "I should mention here that Moodle, particularly after 2.0, is not intended to be the place to store large files" Point taken. Does this hint at an official view is emerging that these large file problems are less of a priority?

          As a non-expert, this problem of 2GIG seems to hint at server limits, different for different servers. As David Tang said: "Since the PHP zip library does not support multi-volume, a quick way to achieve zip splitting is to use a multi-volume capable command line zip tool to zip/unzip, such as 7zip/p7zip. However this is a 3rd-party dependency and cannot guarantee cross-instance or cross-LMS compatibility. Implementing volume splitting manually in Moodle is an option, but probably less elegant" Does this mean a solution in a given instance may be server dependent, and need backend work to set up? And that we are really wasting our time to wait around for a generic solution built into Moodle 2.4.x? Should there just be a page somewhere that lists an option for Ubuntu, Win, Linux, Debian etc.

          David Tang mentioned a multi file option. I'm looking at this now, my home internet connection is doing a 5 hour upload as a trial of three backups: section 1-2, 3-4, 5-6 all under the limit I know I can restore. Is a solution to the 2GIG limit to automate this multi file option?

          This issue has been around for a few months now. I regard it as being significant enough that it should really be highlighted in the install/upgrade pages and docs.

          -Derek

          Show
          Derek Chirnside added a comment - @Michael: I've just been looking at MDL-30962 . You say: "I should mention here that Moodle, particularly after 2.0, is not intended to be the place to store large files" Point taken. Does this hint at an official view is emerging that these large file problems are less of a priority? As a non-expert, this problem of 2GIG seems to hint at server limits, different for different servers. As David Tang said: "Since the PHP zip library does not support multi-volume, a quick way to achieve zip splitting is to use a multi-volume capable command line zip tool to zip/unzip, such as 7zip/p7zip. However this is a 3rd-party dependency and cannot guarantee cross-instance or cross-LMS compatibility. Implementing volume splitting manually in Moodle is an option, but probably less elegant" Does this mean a solution in a given instance may be server dependent, and need backend work to set up? And that we are really wasting our time to wait around for a generic solution built into Moodle 2.4.x? Should there just be a page somewhere that lists an option for Ubuntu, Win, Linux, Debian etc. David Tang mentioned a multi file option. I'm looking at this now, my home internet connection is doing a 5 hour upload as a trial of three backups: section 1-2, 3-4, 5-6 all under the limit I know I can restore. Is a solution to the 2GIG limit to automate this multi file option? This issue has been around for a few months now. I regard it as being significant enough that it should really be highlighted in the install/upgrade pages and docs. -Derek
          Hide
          Derek Chirnside added a comment -

          I'd just like to report that (as many of you probably already knew) selectively backing up sections in a big course and restoring into the same course is a work around and works.
          -Derek

          Show
          Derek Chirnside added a comment - I'd just like to report that (as many of you probably already knew) selectively backing up sections in a big course and restoring into the same course is a work around and works. -Derek
          Hide
          Lael... added a comment -

          Is there any news / update on this problem?

          It would be nice at the very least to have some way of skipping larger courses so it doesn't cause all the other backups to fail.
          Is that a possible workaround until the problem is truly solved?

          Show
          Lael... added a comment - Is there any news / update on this problem? It would be nice at the very least to have some way of skipping larger courses so it doesn't cause all the other backups to fail. Is that a possible workaround until the problem is truly solved?
          Hide
          Lewis Carr added a comment -

          Strangely when running on Windows it fails, even a localhost XAMP/WAMP install. However, on my Mac, running MAMP backups can exceed 2GB and backup fine. I've even done a 5GB backup.

          I've matched PHP versons, Apache versions etc.. but it still fails.

          I have tried everything from the comments here and then some! But I still can't backup large Moodle 2 courses.

          Show
          Lewis Carr added a comment - Strangely when running on Windows it fails, even a localhost XAMP/WAMP install. However, on my Mac, running MAMP backups can exceed 2GB and backup fine. I've even done a 5GB backup. I've matched PHP versons, Apache versions etc.. but it still fails. I have tried everything from the comments here and then some! But I still can't backup large Moodle 2 courses.
          Hide
          Lewis Carr added a comment -

          OK, so I know what the problem is.
          PHP backups greater than 2GB only work on 64 Bit systems, hardware and 64bit compiled PHP and Apache.

          However, although 64 bit Windows versions are available they report back as 32 bit and the php_int_max will report back as 32 bit because of the way it's compiled with 32 bi sources.

          So basically, on Linux, use 64 bit LAMP stacks and you will be fine, same on Mac environments.
          On Windows we are currently in trouble because we cannot increase the integer size.

          Any thoughts anyone?

          Show
          Lewis Carr added a comment - OK, so I know what the problem is. PHP backups greater than 2GB only work on 64 Bit systems, hardware and 64bit compiled PHP and Apache. However, although 64 bit Windows versions are available they report back as 32 bit and the php_int_max will report back as 32 bit because of the way it's compiled with 32 bi sources. So basically, on Linux, use 64 bit LAMP stacks and you will be fine, same on Mac environments. On Windows we are currently in trouble because we cannot increase the integer size. Any thoughts anyone?
          Hide
          Michael OBrien added a comment - - edited

          When it comes to restoring is it possible to side step the limitations of the hosting platform regarding file uploads, php settings etc and have an Advanced restore option where elements of a full backup are restored in series?
          If moodle is provided the backup manifest file or similar index and allow bit by bit restoring from within the bounds of the hosting platform it would at lease help deal with limitations outside the control of moodle and give the end users a way to help themselves and get back some or all of their course up and available to use as soon as possible and would allow some recovery from corrupted backups in certain instances

          Example Use Case : User needs to restore a large course (over 2GB) but can't restore it in 1 go
          Process:
          User is requested to unzip the file on their own computer and upload the manifest file so moodle knows what components are in the backup.
          From a suitable gui the user is prompted to match the manifest entry to the file(s) from the backup (or repository) and indicate where its to be restored to. and moodle then starts to process the restore in series and rebuild the course bit by bit. This would also allow selective restoring of files from backups

          I know its not always possible to download 1 large backup file so maybe the reverse process could be applied and moodle would "break" or chunk a backup into sections which the end user could stitch back to form the whole course again using this restore GUI.

          The would I think kill 2 birds with 1 stone and remove the problem of restoring large backups in 1 go and also provide a way for users to restore just what they needed from a backup without having to restore a full course

          Attached image of very rough workup but at least if moodle knows
          a) what its environmental limitations are regarding file upload limits, execution timeouts etc
          b) the individual elements that are contained in a backup and their size
          It can provide a better user experience by alerting before errors are encountered and provide advice on workarounds and the user can iterate through the restore process until the entire process is complete

          Show
          Michael OBrien added a comment - - edited When it comes to restoring is it possible to side step the limitations of the hosting platform regarding file uploads, php settings etc and have an Advanced restore option where elements of a full backup are restored in series? If moodle is provided the backup manifest file or similar index and allow bit by bit restoring from within the bounds of the hosting platform it would at lease help deal with limitations outside the control of moodle and give the end users a way to help themselves and get back some or all of their course up and available to use as soon as possible and would allow some recovery from corrupted backups in certain instances Example Use Case : User needs to restore a large course (over 2GB) but can't restore it in 1 go Process: User is requested to unzip the file on their own computer and upload the manifest file so moodle knows what components are in the backup. From a suitable gui the user is prompted to match the manifest entry to the file(s) from the backup (or repository) and indicate where its to be restored to. and moodle then starts to process the restore in series and rebuild the course bit by bit. This would also allow selective restoring of files from backups I know its not always possible to download 1 large backup file so maybe the reverse process could be applied and moodle would "break" or chunk a backup into sections which the end user could stitch back to form the whole course again using this restore GUI. The would I think kill 2 birds with 1 stone and remove the problem of restoring large backups in 1 go and also provide a way for users to restore just what they needed from a backup without having to restore a full course Attached image of very rough workup but at least if moodle knows a) what its environmental limitations are regarding file upload limits, execution timeouts etc b) the individual elements that are contained in a backup and their size It can provide a better user experience by alerting before errors are encountered and provide advice on workarounds and the user can iterate through the restore process until the entire process is complete
          Hide
          Michael OBrien added a comment -

          Just a rough idea about an advanced restorer

          Show
          Michael OBrien added a comment - Just a rough idea about an advanced restorer
          Hide
          Christopher Hill added a comment -

          libzip 0.11 released yesterday, with Zip64 support (http://www.nih.at/listarchive/libzip-discuss/msg00327.html, http://www.nih.at/libzip/NEWS.html).

          Created a PHP bug today to get this updated (https://bugs.php.net/bug.php?id=64512) - apparently they are already working on it, although it will only be included in PHP 5.6.

          Just need to wait for this to happen - and see if it resolves the issue.

          Show
          Christopher Hill added a comment - libzip 0.11 released yesterday, with Zip64 support ( http://www.nih.at/listarchive/libzip-discuss/msg00327.html , http://www.nih.at/libzip/NEWS.html ). Created a PHP bug today to get this updated ( https://bugs.php.net/bug.php?id=64512 ) - apparently they are already working on it, although it will only be included in PHP 5.6. Just need to wait for this to happen - and see if it resolves the issue.
          Hide
          Derek Chirnside added a comment -

          There has been another thread here: https://moodle.org/mod/forum/discuss.php?d=226893 with more detail on workaround where you have ONLY a backup f1le that is too big.

          -Derek

          Show
          Derek Chirnside added a comment - There has been another thread here: https://moodle.org/mod/forum/discuss.php?d=226893 with more detail on workaround where you have ONLY a backup f1le that is too big. -Derek
          Hide
          Justin Filip added a comment -

          Updated this issue with testing instructions and a link to code in a Github repository. This solution implements the patch from the local_externalzip plugin with some improvements to it. (i.e. not breaking backup and restore if the plugin is not installed).

          Show
          Justin Filip added a comment - Updated this issue with testing instructions and a link to code in a Github repository. This solution implements the patch from the local_externalzip plugin with some improvements to it. (i.e. not breaking backup and restore if the plugin is not installed).
          Hide
          Justin Filip added a comment -

          Michael de Raadt Can we get someone from HQ to look this over for us? Thanks.

          Show
          Justin Filip added a comment - Michael de Raadt Can we get someone from HQ to look this over for us? Thanks.
          Hide
          Michael de Raadt added a comment -

          Thanks for working on this, guys.

          I can't see the solution at the moment as Github is down, but I've asked someone to check this out.

          Show
          Michael de Raadt added a comment - Thanks for working on this, guys. I can't see the solution at the moment as Github is down, but I've asked someone to check this out.
          Hide
          Michael de Raadt added a comment -

          Petr is unwell, but was able to have a look at this issue and provide some quick feedback.

          Replacing the current zip library unconditionally with an incomplete CLI zip tool is not acceptable, but we could add a special archive class for new Moodle backups that handles larger files. The oversized old backups, with sloppy zip extension, would require some bigger auto-detection magic.

          There are still issues with 32bit systems.

          Show
          Michael de Raadt added a comment - Petr is unwell, but was able to have a look at this issue and provide some quick feedback. Replacing the current zip library unconditionally with an incomplete CLI zip tool is not acceptable, but we could add a special archive class for new Moodle backups that handles larger files. The oversized old backups, with sloppy zip extension, would require some bigger auto-detection magic. There are still issues with 32bit systems.
          Hide
          Mike Churchward added a comment -

          Michael de Raadt "Replacing the current sip library unconditionally with an incomplete CLI zip tool is not acceptable"... Could you elaborate on this please? The solution offered is similar to what Moodle used to do - allow the system to 'optionally' configure an external zip tool that can be used if it exists. The code even checks to make sure the tool exists and defaults to the internal one if it doesn't. This is done by looking for a local plug-in to replace the default zip program.

          Is the problem that a local plug-in is not the desired solution, or something else? The discussion above was all about using a local plug-in.

          I presume the the special archive class that would be added is not something we'll see anytime soon? In the meantime, people cannot restore backup files.

          Show
          Mike Churchward added a comment - Michael de Raadt "Replacing the current sip library unconditionally with an incomplete CLI zip tool is not acceptable"... Could you elaborate on this please? The solution offered is similar to what Moodle used to do - allow the system to 'optionally' configure an external zip tool that can be used if it exists. The code even checks to make sure the tool exists and defaults to the internal one if it doesn't. This is done by looking for a local plug-in to replace the default zip program. Is the problem that a local plug-in is not the desired solution, or something else? The discussion above was all about using a local plug-in. I presume the the special archive class that would be added is not something we'll see anytime soon? In the meantime, people cannot restore backup files.
          Hide
          Michael de Raadt added a comment -

          Hi, Mike.

          As Petr was the person making the comment, I will ask him to respond.

          Show
          Michael de Raadt added a comment - Hi, Mike. As Petr was the person making the comment, I will ask him to respond.
          Hide
          Jorge Gonzalez Alonso added a comment -

          The same problem from Moodle 2.4 to 2.5. My backups are 1.5 KB to 16 000 KB

          Show
          Jorge Gonzalez Alonso added a comment - The same problem from Moodle 2.4 to 2.5. My backups are 1.5 KB to 16 000 KB
          Hide
          Roger Emery added a comment -

          I think this might possibly be a time out of some sort, maybe session.

          I've just attempted to upload 3000+ courses from a CSV to create to the new courses from a template course.
          It would be using most of the same routines for back-up/restore as described.

          It got to just over 2000 units and took a couple of hours to get that far then produced the same error as described.

          Re-running the rest, so see how it goes.

          Show
          Roger Emery added a comment - I think this might possibly be a time out of some sort, maybe session. I've just attempted to upload 3000+ courses from a CSV to create to the new courses from a template course. It would be using most of the same routines for back-up/restore as described. It got to just over 2000 units and took a couple of hours to get that far then produced the same error as described. Re-running the rest, so see how it goes.
          Hide
          Tim Atton added a comment -

          I am also finding that my automatic backups fail when they get to a large one. The log for the 1st one to fail reports Exception: codingerror backup_helper::store_backup_file() expects valid $filepath parameter

          None of the courses after that one will create backups (and they are the important new courses)

          If I try a manual backup i get
          Coding error detected, it must be fixed by a programmer: backup_helper::store_backup_file() expects valid $filepath parameter

          Show
          Tim Atton added a comment - I am also finding that my automatic backups fail when they get to a large one. The log for the 1st one to fail reports Exception: codingerror backup_helper::store_backup_file() expects valid $filepath parameter None of the courses after that one will create backups (and they are the important new courses) If I try a manual backup i get Coding error detected, it must be fixed by a programmer: backup_helper::store_backup_file() expects valid $filepath parameter
          Hide
          Ricardo Díaz added a comment -

          I have the same issue Tim, I can't make backups >1gb correctly. Backup process stops when has to rename and copy backup.mbz file. It's located at "moodledata/temp/backup/"course hash"/backup.mbz" . Maybe compress process takes too long and DB timeout conection.
          I have postgre db and moodle 2.4.6. and sorry for my english.

          Show
          Ricardo Díaz added a comment - I have the same issue Tim, I can't make backups >1gb correctly. Backup process stops when has to rename and copy backup.mbz file. It's located at "moodledata/temp/backup/"course hash"/backup.mbz" . Maybe compress process takes too long and DB timeout conection. I have postgre db and moodle 2.4.6. and sorry for my english.
          Hide
          Sam Marshall added a comment -

          For info, I have submitted MDL-41838 which takes a different approach without using external tools by optionally changing to .tar.gz compression format for backups. This also has other benefits and I am hoping it can become the new default.

          It may be complementary to this change.

          Show
          Sam Marshall added a comment - For info, I have submitted MDL-41838 which takes a different approach without using external tools by optionally changing to .tar.gz compression format for backups. This also has other benefits and I am hoping it can become the new default. It may be complementary to this change.
          Hide
          Eloy Lafuente (stronk7) added a comment -

          I really would close this as won't fix and put all the efforts into MDL-41838 (for Moodle 2.6 and upwards). The 4GB limit in zip files is a HARD one and it seems the new tar/gzip packer will bring way better results (both reliable and quick) soon.

          About zip… and tangentially related with this… in MDL-37877 some measures are being taken in order to verify the resulting zip file once it's written to avoid bad surprises when the restoration of an old backup file (supposedly correct) ends being a broken file.

          So +1 to close this, with MDL-41838 being the continuation/future of this. Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - I really would close this as won't fix and put all the efforts into MDL-41838 (for Moodle 2.6 and upwards). The 4GB limit in zip files is a HARD one and it seems the new tar/gzip packer will bring way better results (both reliable and quick) soon. About zip… and tangentially related with this… in MDL-37877 some measures are being taken in order to verify the resulting zip file once it's written to avoid bad surprises when the restoration of an old backup file (supposedly correct) ends being a broken file. So +1 to close this, with MDL-41838 being the continuation/future of this. Ciao
          Hide
          Christopher Hill added a comment -

          As the original submitter of this bug, if MDL-41838 is going to fix the issue another way (and without having to wait for PHP 5.6), I've no problem with closing the bug for now.

          Show
          Christopher Hill added a comment - As the original submitter of this bug, if MDL-41838 is going to fix the issue another way (and without having to wait for PHP 5.6), I've no problem with closing the bug for now.
          Hide
          Tim Atton added a comment -

          Im not sure this issue is just about large courses. I get this error on one particular course and I doubt it is anywhere near that big.

          Show
          Tim Atton added a comment - Im not sure this issue is just about large courses. I get this error on one particular course and I doubt it is anywhere near that big.
          Hide
          Roger Emery added a comment -

          my incarnation of the error wasn't related to file size, but number of courses it was trying to create from CSV, hence thinking it was a timeout, or maybe memory leak? So not sure.

          Show
          Roger Emery added a comment - my incarnation of the error wasn't related to file size, but number of courses it was trying to create from CSV, hence thinking it was a timeout, or maybe memory leak? So not sure.
          Hide
          Michael de Raadt added a comment -

          This issue has now been resolved by MDL-41838. If there are related issues that are still unresolved, they should be represented by other issues.

          Thanks to everyone involved in this issue.

          Show
          Michael de Raadt added a comment - This issue has now been resolved by MDL-41838 . If there are related issues that are still unresolved, they should be represented by other issues. Thanks to everyone involved in this issue.

            Dates

            • Created:
              Updated:
              Resolved: