Moodle
  1. Moodle
  2. MDL-32547

Configure temporary file deletion, to avoid moodledata/temp/backup filling up the filesystem

    Details

    • Testing Instructions:
      Hide
      1. Create a course.
      2. Create a failed backup. I edited the backup_stepslib.php file and put a die(); in one of the functions.
      3. Check moodledata/temp/backup and ensure that there are directories listed there.
      4. Update the scheduled task "Delete stale temp files" [Site administration ► Server ► Scheduled tasks] and change all of the times to '*'.
        • You can either Wait an hour [Site administration ► Server ► Cleanup ] and set 'Clean up temporary data files older than' to '1 hours'.
        • Or change the scheduled task code [lib/classes/task/file_temp_cleanup_task.php]

          $time = time() - ($CFG->tempdatafoldercleanup * 3600);
          

          to

          $time = time() - 1;
          

      5. Run the cron.
      6. Have a look in the temp/backup directory and check that all of the directories have been removed.
      Show
      Create a course. Create a failed backup. I edited the backup_stepslib.php file and put a die(); in one of the functions. Check moodledata/temp/backup and ensure that there are directories listed there. Update the scheduled task "Delete stale temp files" [Site administration ► Server ► Scheduled tasks] and change all of the times to '*'. You can either Wait an hour [Site administration ► Server ► Cleanup ] and set 'Clean up temporary data files older than' to '1 hours'. Or change the scheduled task code [lib/classes/task/file_temp_cleanup_task.php] $time = time() - ($CFG->tempdatafoldercleanup * 3600); to $time = time() - 1; Run the cron. Have a look in the temp/backup directory and check that all of the directories have been removed.
    • Workaround:
      Hide

      Not a solution to the tempdir\backup filling with failed backups but to allow large backups to complete fully run automated backup via cli (be aware of this issue http://tracker.moodle.org/browse/MDL-32548 )

      Show
      Not a solution to the tempdir\backup filling with failed backups but to allow large backups to complete fully run automated backup via cli (be aware of this issue http://tracker.moodle.org/browse/MDL-32548 )
    • Affected Branches:
      MOODLE_22_STABLE, MOODLE_26_STABLE
    • Fixed Branches:
      MOODLE_29_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      wip-MDL-32547-master-v2
    • Sprint:
      BACKEND Sprint 18

      Description

      If backup tries to backup a large course and the backup fails to complete successfully a large amount of disk can be taken up in moodledata/temp/backup. Further attemps to complete the same course backups will not reuse the temp directory contents or clean out old contents but instead start again.

      This can cause disk space usage to grow to hit limit imposed by host or disk capacity with little or no notification to site admin or entry in server logs. Site functionality could also be affected if no space available for new courses/resources to be added. Unknowningly users with backup permissions could crash site by repeating command to backup large course and fill allocated disk space with incomplete course backups

      A side affect of this is non-moodle based backups of db and files take longer because contents of tempdir is larger than necessary

      To reproduce:
      Attempt to backup a large course either manually from backup UI or via automated backups configured via UL. Monitor the moodledata/filedir/temp/backup When the backup fails due to timeout of apache or php (e.g html error 500) the contents will remain along with an blank .log file with the same name as the temp folder which was created, in my case just under 10GB for 1 course
      rerun the backup again and a new folder will be created in the temp directory but without cleaning up the 1st attempt

      Possible Cures:
      1) Log start time of backup in logs, with progress updates entered and if backup is not completed in X hours delete the contents of temp\backup folder and notify admin. This would also help admins know how far backups get before they fail.

      2) Allow site admin/backup user to see list of tempdir on backup screen and clear as needed

      3) Possibly wider benefit from this but allow site admin to list temp dir size and contents and set "time to live" value for contents before cleaned up by cron process

      4) Emergency Option if files.api reports problems writing to disk due to insufficient space prompt admin to empty tempdir so users not affected.

      5) Impose quota on tempdir size set by site admin and include checking in cron process to warn admin if quota is neared

        Gliffy Diagrams

          Issue Links

            Activity

            Michael OBrien created issue -
            Michael OBrien made changes -
            Field Original Value New Value
            Description If backups tries to backup a large course and the backup fails to complete successfully a large amount of disk can be taken up in filedir/temp/backup of moodledata. Further attemps to complete the same course backups will not reuse the temp directory contents or clean out old contents but instead start again.

            This can cause disk space usage to grow to hit limit imposed by host or disk capacity with little notification to site admin or entry in server logs. Site functionality could also be affected if no space available for new courses/resources to be added. Unknowningly users with backup permissions could crash site by repeating command to backup large course

            A side affect of this is non-moodle based backups of db and file take longer because contents of tempdir is larger

            To reproduce:
            Attempt to backup a large course either manually or via automated backups. Monitor the moodledata/filedir/temp/backup When the backup fails due to timeout of apache or php (e.g html error 500) the contents will remain along with an blank .log file with the same name as the temp folder which was created, in my case just under 10GB
            rerun the backup again and a new folder will be created in the temp directory

            ToDo: Confirm behaviour is the same if backup auto/automated backup or auto/manual and backup command issued by cli

            Possible Cures:
            1) Log start time of backup in logs, with progress updates entered and if backup is not completed in X hours delete the temp folder and notify admin. This would also help admins know how far backups get before they fail

            2) Allow site admin/backup user to see list of tempdir on backup screen and clear as needed
            Include cleanup

            3) Possibly wider benefit from this but allow site admin to list temp dir size and contents and set "time to live" value for contents before cleaned up by cron process

            4) Emergency Option if files.api reports problems writing to disk due to insufficient space prompt admin to empty tempdir so users not affected.

            5) Impose quota on tempdir size set by site admin and include checking in cron process to warn admin if quota is neared
            If backup tries to backup a large course and the backup fails to complete successfully a large amount of disk can be taken up in filedir/temp/backup of moodledata. Further attemps to complete the same course backups will not reuse the temp directory contents or clean out old contents but instead start again.

            This can cause disk space usage to grow to hit limit imposed by host or disk capacity with little notification to site admin or entry in server logs. Site functionality could also be affected if no space available for new courses/resources to be added. Unknowningly users with backup permissions could crash site by repeating command to backup large course

            A side affect of this is non-moodle based backups of db and files take longer because contents of tempdir is larger

            To reproduce:
            Attempt to backup a large course either manually or via automated backups. Monitor the moodledata/filedir/temp/backup When the backup fails due to timeout of apache or php (e.g html error 500) the contents will remain along with an blank .log file with the same name as the temp folder which was created, in my case just under 10GB for 1 course
            rerun the backup again and a new folder will be created in the temp directory but without cleaning up the 1st attempt

            ToDo: Confirm behaviour is the same if backup auto/automated backup or auto/manual and backup command issued by cli

            Possible Cures:
            1) Log start time of backup in logs, with progress updates entered and if backup is not completed in X hours delete the temp folder and notify admin. This would also help admins know how far backups get before they fail

            2) Allow site admin/backup user to see list of tempdir on backup screen and clear as needed
            Include cleanup

            3) Possibly wider benefit from this but allow site admin to list temp dir size and contents and set "time to live" value for contents before cleaned up by cron process

            4) Emergency Option if files.api reports problems writing to disk due to insufficient space prompt admin to empty tempdir so users not affected.

            5) Impose quota on tempdir size set by site admin and include checking in cron process to warn admin if quota is neared
            Michael OBrien made changes -
            Labels administration backup capacity cron disk storage tempdir
            Michael OBrien made changes -
            Summary Failed Backups do not clean up contents of filedir/temp/backup after failing and future backups don't reuse the contents Failed Backups do not clean up contents of moodledata/temp/backup after failing and future backups don't reuse the contents
            Michael OBrien made changes -
            Description If backup tries to backup a large course and the backup fails to complete successfully a large amount of disk can be taken up in filedir/temp/backup of moodledata. Further attemps to complete the same course backups will not reuse the temp directory contents or clean out old contents but instead start again.

            This can cause disk space usage to grow to hit limit imposed by host or disk capacity with little notification to site admin or entry in server logs. Site functionality could also be affected if no space available for new courses/resources to be added. Unknowningly users with backup permissions could crash site by repeating command to backup large course

            A side affect of this is non-moodle based backups of db and files take longer because contents of tempdir is larger

            To reproduce:
            Attempt to backup a large course either manually or via automated backups. Monitor the moodledata/filedir/temp/backup When the backup fails due to timeout of apache or php (e.g html error 500) the contents will remain along with an blank .log file with the same name as the temp folder which was created, in my case just under 10GB for 1 course
            rerun the backup again and a new folder will be created in the temp directory but without cleaning up the 1st attempt

            ToDo: Confirm behaviour is the same if backup auto/automated backup or auto/manual and backup command issued by cli

            Possible Cures:
            1) Log start time of backup in logs, with progress updates entered and if backup is not completed in X hours delete the temp folder and notify admin. This would also help admins know how far backups get before they fail

            2) Allow site admin/backup user to see list of tempdir on backup screen and clear as needed
            Include cleanup

            3) Possibly wider benefit from this but allow site admin to list temp dir size and contents and set "time to live" value for contents before cleaned up by cron process

            4) Emergency Option if files.api reports problems writing to disk due to insufficient space prompt admin to empty tempdir so users not affected.

            5) Impose quota on tempdir size set by site admin and include checking in cron process to warn admin if quota is neared
            If backup tries to backup a large course and the backup fails to complete successfully a large amount of disk can be taken up in moodledata/temp/backup. Further attemps to complete the same course backups will not reuse the temp directory contents or clean out old contents but instead start again.

            This can cause disk space usage to grow to hit limit imposed by host or disk capacity with little or no notification to site admin or entry in server logs. Site functionality could also be affected if no space available for new courses/resources to be added. Unknowningly users with backup permissions could crash site by repeating command to backup large course and fill allocated disk space with incomplete course backups

            A side affect of this is non-moodle based backups of db and files take longer because contents of tempdir is larger than necessary

            To reproduce:
            Attempt to backup a large course either manually from backup UI or via automated backups configured via UL. Monitor the moodledata/filedir/temp/backup When the backup fails due to timeout of apache or php (e.g html error 500) the contents will remain along with an blank .log file with the same name as the temp folder which was created, in my case just under 10GB for 1 course
            rerun the backup again and a new folder will be created in the temp directory but without cleaning up the 1st attempt


            Possible Cures:
            1) Log start time of backup in logs, with progress updates entered and if backup is not completed in X hours delete the contents of temp\backup folder and notify admin. This would also help admins know how far backups get before they fail.

            2) Allow site admin/backup user to see list of tempdir on backup screen and clear as needed

            3) Possibly wider benefit from this but allow site admin to list temp dir size and contents and set "time to live" value for contents before cleaned up by cron process

            4) Emergency Option if files.api reports problems writing to disk due to insufficient space prompt admin to empty tempdir so users not affected.

            5) Impose quota on tempdir size set by site admin and include checking in cron process to warn admin if quota is neared
            Workaround Not a solution to the tempdir\backup filling with failed backups but to allow large backups to complete fully run automated backup via cli (be aware of this issue http://tracker.moodle.org/browse/MDL-32548 )
            Michael de Raadt made changes -
            Link This issue has a non-specific relationship to MDL-1520 [ MDL-1520 ]
            Michael de Raadt made changes -
            Link This issue has a non-specific relationship to MDL-27790 [ MDL-27790 ]
            Michael de Raadt made changes -
            Security Could be a security issue [ 10030 ]
            Fix Version/s STABLE backlog [ 10463 ]
            Labels administration backup capacity cron disk storage tempdir triaged
            Component/s Files API [ 10305 ]
            Michael de Raadt made changes -
            Link This issue has been marked as being related by MDL-33752 [ MDL-33752 ]
            Michael de Raadt made changes -
            Link This issue has been marked as being related by MDL-38189 [ MDL-38189 ]
            Rajesh Taneja made changes -
            Assignee Rajesh Taneja [ rajeshtaneja ]
            Marina Glancy made changes -
            Affects Version/s 2.6.4 [ 13551 ]
            Mark Nelson made changes -
            Sprint BACKEND Sprint 18 [ 34 ]
            Adrian Greeve made changes -
            Assignee Adrian Greeve [ abgreeve ]
            Adrian Greeve made changes -
            Status Open [ 1 ] Development in progress [ 3 ]
            Adrian Greeve made changes -
            Pull Master Diff URL https://github.com/abgreeve/moodle/compare/wip-MDL-32547-master
            Pull Master Branch wip-MDL-32547-master
            Pull from Repository git://github.com/abgreeve/moodle.git
            Adrian Greeve made changes -
            Testing Instructions # Create a course.
            # Create a failed backup. I edited the backup_stepslib.php file and put a die(); in one of the functions.
            # Check moodledata/temp/backup and ensure that there are directories listed there.
            #* Update the scheduled task "Clean up failed backup files from the temp directory" [Site administration ► Server ► Scheduled tasks] and change all of the times to '*'.
            #* You can either Wait an hour [Site administration ► Courses ► Backups ► General backup defaults] and set 'Failed backup cleanup' to '1 hours'
            #* Or change the scheduled task code [lib/classes/task/failed_backup_task.php]
            {code}
            $timedelay = intval(get_config('backup', 'failedbackupcheck')) * 3600;
            {code}
            to
            {code}
            $timedelay = 1;
            {code}
            # Run the cron.
            # Have a look in the temp/backup directory and check that all of the directories have been removed.
            Adrian Greeve made changes -
            Status Development in progress [ 3 ] Waiting for peer review [ 10012 ]
            CiBoT made changes -
            Labels triaged ci triaged
            Adrian Greeve made changes -
            Labels ci triaged triaged
            CiBoT made changes -
            Labels triaged ci triaged
            Adrian Greeve made changes -
            Labels ci triaged triaged
            CiBoT made changes -
            Labels triaged ci triaged
            Ankit Agarwal made changes -
            Original Estimate 0 minutes [ 0 ]
            Remaining Estimate 0 minutes [ 0 ]
            Status Waiting for peer review [ 10012 ] Peer review in progress [ 10013 ]
            Peer reviewer Ankit Agarwal [ ankit_frenz ]
            Ankit Agarwal made changes -
            Status Peer review in progress [ 10013 ] Development in progress [ 3 ]
            CiBoT made changes -
            Labels ci triaged triaged
            Adrian Greeve made changes -
            Link This issue is blocked by MDL-48252 [ MDL-48252 ]
            Adrian Greeve made changes -
            Testing Instructions # Create a course.
            # Create a failed backup. I edited the backup_stepslib.php file and put a die(); in one of the functions.
            # Check moodledata/temp/backup and ensure that there are directories listed there.
            #* Update the scheduled task "Clean up failed backup files from the temp directory" [Site administration ► Server ► Scheduled tasks] and change all of the times to '*'.
            #* You can either Wait an hour [Site administration ► Courses ► Backups ► General backup defaults] and set 'Failed backup cleanup' to '1 hours'
            #* Or change the scheduled task code [lib/classes/task/failed_backup_task.php]
            {code}
            $timedelay = intval(get_config('backup', 'failedbackupcheck')) * 3600;
            {code}
            to
            {code}
            $timedelay = 1;
            {code}
            # Run the cron.
            # Have a look in the temp/backup directory and check that all of the directories have been removed.
            # Create a course.
            # Create a failed backup. I edited the backup_stepslib.php file and put a die(); in one of the functions.
            # Check moodledata/temp/backup and ensure that there are directories listed there.
            # Update the scheduled task "Delete stale temp files" [Site administration ► Server ► Scheduled tasks] and change all of the times to '*'.
            #* You can either Wait an hour [Site administration ► Server ► Cleanup ] and set 'Clean up temporary data files older than' to '1 hours'.
            #* Or change the scheduled task code [lib/classes/task/file_temp_cleanup_task.php]
            {code}
            $timedelay = $CFG->tempdatafoldercleanup * 3600;
            {code}
            to
            {code}
            $timedelay = 1;
            {code}
            # Run the cron.
            # Have a look in the temp/backup directory and check that all of the directories have been removed.
            Adrian Greeve made changes -
            Status Development in progress [ 3 ] Waiting for peer review [ 10012 ]
            CiBoT made changes -
            Labels triaged ci triaged
            Ankit Agarwal made changes -
            Status Waiting for peer review [ 10012 ] Peer review in progress [ 10013 ]
            Ankit Agarwal made changes -
            Labels ci triaged ci docs_required triaged
            Ankit Agarwal made changes -
            Status Peer review in progress [ 10013 ] Development in progress [ 3 ]
            CiBoT made changes -
            Labels ci docs_required triaged docs_required triaged
            Adrian Greeve made changes -
            Status Development in progress [ 3 ] Waiting for integration review [ 10010 ]
            CiBoT made changes -
            Status Waiting for integration review [ 10010 ] Waiting for integration review [ 10010 ]
            Currently in integration Yes [ 10041 ]
            CiBoT made changes -
            Labels docs_required triaged ci docs_required triaged
            Dan Poltawski made changes -
            Status Waiting for integration review [ 10010 ] Integration review in progress [ 10004 ]
            Integrator Dan Poltawski [ poltawski ]
            Dan Poltawski made changes -
            Status Integration review in progress [ 10004 ] Waiting for testing [ 10005 ]
            Fix Version/s 2.9 [ 14251 ]
            Fix Version/s STABLE backlog [ 10463 ]
            Dan Poltawski made changes -
            Summary Failed Backups do not clean up contents of moodledata/temp/backup after failing and future backups don't reuse the contents Configure temporary file deletion, to avoid moodledata/temp/backup filling up the filesystem
            Rajesh Taneja made changes -
            Tester Rajesh Taneja [ rajeshtaneja ]
            Rajesh Taneja made changes -
            Status Waiting for testing [ 10005 ] Testing in progress [ 10011 ]
            Adrian Greeve made changes -
            Testing Instructions # Create a course.
            # Create a failed backup. I edited the backup_stepslib.php file and put a die(); in one of the functions.
            # Check moodledata/temp/backup and ensure that there are directories listed there.
            # Update the scheduled task "Delete stale temp files" [Site administration ► Server ► Scheduled tasks] and change all of the times to '*'.
            #* You can either Wait an hour [Site administration ► Server ► Cleanup ] and set 'Clean up temporary data files older than' to '1 hours'.
            #* Or change the scheduled task code [lib/classes/task/file_temp_cleanup_task.php]
            {code}
            $timedelay = $CFG->tempdatafoldercleanup * 3600;
            {code}
            to
            {code}
            $timedelay = 1;
            {code}
            # Run the cron.
            # Have a look in the temp/backup directory and check that all of the directories have been removed.
            # Create a course.
            # Create a failed backup. I edited the backup_stepslib.php file and put a die(); in one of the functions.
            # Check moodledata/temp/backup and ensure that there are directories listed there.
            # Update the scheduled task "Delete stale temp files" [Site administration ► Server ► Scheduled tasks] and change all of the times to '*'.
            #* You can either Wait an hour [Site administration ► Server ► Cleanup ] and set 'Clean up temporary data files older than' to '1 hours'.
            #* Or change the scheduled task code [lib/classes/task/file_temp_cleanup_task.php]
            {code}
            $time = time() - ($CFG->tempdatafoldercleanup * 3600);
            {code}
            to
            {code}
            $time = time() - 1;
            {code}
            # Run the cron.
            # Have a look in the temp/backup directory and check that all of the directories have been removed.
            Rajesh Taneja made changes -
            Status Testing in progress [ 10011 ] Tested [ 10006 ]
            Marina Glancy made changes -
            Labels ci docs_required triaged ci docs_required triaged ui_change
            Marina Glancy made changes -
            Issue Type Bug [ 1 ] Improvement [ 4 ]
            Eloy Lafuente (stronk7) made changes -
            Status Tested [ 10006 ] Closed [ 6 ]
            Resolution Fixed [ 1 ]
            Currently in integration Yes [ 10041 ]
            Integration date 05/Dec/14

              People

              • Votes:
                8 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Agile