Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-37877

Over-large backups fail transparently, then do not get deleted next time, so space fills up

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.4, 2.6
    • Fix Version/s: 2.4.7, 2.5.3
    • Component/s: Backup
    • Labels:
    • Testing Instructions:
      Hide
      1. Turn on full debug.
      2. Set time limit in your php ini something very large like a day.
      3. set time limit for a backup step by changing core_backup_progress::TIME_LIMIT_WITHOUT_PROGRESS to a few hours. (located in backup>util>progress>core_backup_progress.class.php)
      4. Generate a course with lots of activities/sections etc.
      5. Add atleast 8-10 GB of media content to the course spread over multiple activities.
      6. Set the following configs:-
        1. backup_auto_active - manual
        2. backup_auto_storage - specified directory
        3. backup_auto_destination - path to some writeable dir
      7. Run the script admin/cli/automated_backup.php
      8. Make sure backup was created for all courses except the large one. The large one would fail.
      9. Goto My home ► Site administration ► Reports ► Backups
      10. Make sure the status of all courses except the large course is "OK"
      11. Re run the backup script.
      12. Again, make sure backup was created for all courses except the large one.
      13. Goto My home ► Site administration ► Reports ► Backups
      14. Make sure the status of all courses except the large course is "OK"
      15. Status for the large course is "ERROR"
      16. Try backing up the courses using web interface. The large course backup should fail, generating an error. Any other course should go as expected.
      Show
      Turn on full debug. Set time limit in your php ini something very large like a day. set time limit for a backup step by changing core_backup_progress::TIME_LIMIT_WITHOUT_PROGRESS to a few hours. (located in backup>util>progress>core_backup_progress.class.php) Generate a course with lots of activities/sections etc. Add atleast 8-10 GB of media content to the course spread over multiple activities. Set the following configs:- backup_auto_active - manual backup_auto_storage - specified directory backup_auto_destination - path to some writeable dir Run the script admin/cli/automated_backup.php Make sure backup was created for all courses except the large one. The large one would fail. Goto My home ► Site administration ► Reports ► Backups Make sure the status of all courses except the large course is "OK" Re run the backup script. Again, make sure backup was created for all courses except the large one. Goto My home ► Site administration ► Reports ► Backups Make sure the status of all courses except the large course is "OK" Status for the large course is "ERROR" Try backing up the courses using web interface. The large course backup should fail, generating an error. Any other course should go as expected.
    • Affected Branches:
      MOODLE_24_STABLE, MOODLE_26_STABLE
    • Fixed Branches:
      MOODLE_24_STABLE, MOODLE_25_STABLE
    • Pull Master Branch:
      MDL-37877-master
    • Story Points (Obsolete):
      40
    • Sprint:
      BACKEND Sprint 6

      Description

      This refers to scheduled backups, where the number of versions kept is set to one. This customer site has about half a dozen very large courses, where the backup system generates vary large files (two examples from the test I ran this morning are 7.8GB and 9.2GB). These are obviously too large to be successfully zipped, and when the backup processes them, it eventually fails with the error like : "Error - backup-moodle2-course-1681-20130205-1034.mbz does not appear to be a valid backup (missing_moodle_backup_xml_file)"

      Then (in the log) it says "complete - next execution (the next scheduled backup time)"

      So it fails, presumably because it is too big - but then carries on. This in itself is good, as previously, failures like this were stopping the backup schedule completely.

      However, the next time that the scheduled course backup is run, all of the old versions of the backups are replaced (remember, we are only keeping one of each) EXCEPT the failed backups. Moodle does not seem to recognise them as valid backups, despite their filenames, so they are NOT deleted. Over the course of several days, the multiple copies of large failed backups that result from this eventually fill up the server disk, and cause teh site to fail.

      I know that this relates to the larger subject of handling very large course backups, but can anything be done with the checking process, so that these large failed backup files do not get left to fill up the server?

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Votes:
                  1 Vote for this issue
                  Watchers:
                  11 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:
                    Fix Release Date:
                    11/Nov/13