Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-32547

Configure temporary file deletion, to avoid moodledata/temp/backup filling up the filesystem

    Details

    • Testing Instructions:
      Hide
      1. Create a course.
      2. Create a failed backup. I edited the backup_stepslib.php file and put a die(); in one of the functions.
      3. Check moodledata/temp/backup and ensure that there are directories listed there.
      4. Update the scheduled task "Delete stale temp files" [Site administration ► Server ► Scheduled tasks] and change all of the times to '*'.
        • You can either Wait an hour [Site administration ► Server ► Cleanup ] and set 'Clean up temporary data files older than' to '1 hours'.
        • Or change the scheduled task code [lib/classes/task/file_temp_cleanup_task.php]

          $time = time() - ($CFG->tempdatafoldercleanup * 3600);
          

          to

          $time = time() - 1;
          

      5. Run the cron.
      6. Have a look in the temp/backup directory and check that all of the directories have been removed.
      Show
      Create a course. Create a failed backup. I edited the backup_stepslib.php file and put a die(); in one of the functions. Check moodledata/temp/backup and ensure that there are directories listed there. Update the scheduled task "Delete stale temp files" [Site administration ► Server ► Scheduled tasks] and change all of the times to '*'. You can either Wait an hour [Site administration ► Server ► Cleanup ] and set 'Clean up temporary data files older than' to '1 hours'. Or change the scheduled task code [lib/classes/task/file_temp_cleanup_task.php] $time = time() - ($CFG->tempdatafoldercleanup * 3600); to $time = time() - 1; Run the cron. Have a look in the temp/backup directory and check that all of the directories have been removed.
    • Workaround:
      Hide

      Not a solution to the tempdir\backup filling with failed backups but to allow large backups to complete fully run automated backup via cli (be aware of this issue http://tracker.moodle.org/browse/MDL-32548 )

      Show
      Not a solution to the tempdir\backup filling with failed backups but to allow large backups to complete fully run automated backup via cli (be aware of this issue http://tracker.moodle.org/browse/MDL-32548 )
    • Affected Branches:
      MOODLE_22_STABLE, MOODLE_26_STABLE
    • Fixed Branches:
      MOODLE_29_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      wip-MDL-32547-master-v2
    • Sprint:
      BACKEND Sprint 18
    • Sprint:
      BACKEND Sprint 18

      Description

      If backup tries to backup a large course and the backup fails to complete successfully a large amount of disk can be taken up in moodledata/temp/backup. Further attemps to complete the same course backups will not reuse the temp directory contents or clean out old contents but instead start again.

      This can cause disk space usage to grow to hit limit imposed by host or disk capacity with little or no notification to site admin or entry in server logs. Site functionality could also be affected if no space available for new courses/resources to be added. Unknowningly users with backup permissions could crash site by repeating command to backup large course and fill allocated disk space with incomplete course backups

      A side affect of this is non-moodle based backups of db and files take longer because contents of tempdir is larger than necessary

      To reproduce:
      Attempt to backup a large course either manually from backup UI or via automated backups configured via UL. Monitor the moodledata/filedir/temp/backup When the backup fails due to timeout of apache or php (e.g html error 500) the contents will remain along with an blank .log file with the same name as the temp folder which was created, in my case just under 10GB for 1 course
      rerun the backup again and a new folder will be created in the temp directory but without cleaning up the 1st attempt

      Possible Cures:
      1) Log start time of backup in logs, with progress updates entered and if backup is not completed in X hours delete the contents of temp\backup folder and notify admin. This would also help admins know how far backups get before they fail.

      2) Allow site admin/backup user to see list of tempdir on backup screen and clear as needed

      3) Possibly wider benefit from this but allow site admin to list temp dir size and contents and set "time to live" value for contents before cleaned up by cron process

      4) Emergency Option if files.api reports problems writing to disk due to insufficient space prompt admin to empty tempdir so users not affected.

      5) Impose quota on tempdir size set by site admin and include checking in cron process to warn admin if quota is neared

        Gliffy Diagrams

          Issue Links

            Activity

            mobcdi Michael OBrien created issue -
            mobcdi Michael OBrien made changes -
            Field Original Value New Value
            Description If backups tries to backup a large course and the backup fails to complete successfully a large amount of disk can be taken up in filedir/temp/backup of moodledata. Further attemps to complete the same course backups will not reuse the temp directory contents or clean out old contents but instead start again.

            This can cause disk space usage to grow to hit limit imposed by host or disk capacity with little notification to site admin or entry in server logs. Site functionality could also be affected if no space available for new courses/resources to be added. Unknowningly users with backup permissions could crash site by repeating command to backup large course

            A side affect of this is non-moodle based backups of db and file take longer because contents of tempdir is larger

            To reproduce:
            Attempt to backup a large course either manually or via automated backups. Monitor the moodledata/filedir/temp/backup When the backup fails due to timeout of apache or php (e.g html error 500) the contents will remain along with an blank .log file with the same name as the temp folder which was created, in my case just under 10GB
            rerun the backup again and a new folder will be created in the temp directory

            ToDo: Confirm behaviour is the same if backup auto/automated backup or auto/manual and backup command issued by cli

            Possible Cures:
            1) Log start time of backup in logs, with progress updates entered and if backup is not completed in X hours delete the temp folder and notify admin. This would also help admins know how far backups get before they fail

            2) Allow site admin/backup user to see list of tempdir on backup screen and clear as needed
            Include cleanup

            3) Possibly wider benefit from this but allow site admin to list temp dir size and contents and set "time to live" value for contents before cleaned up by cron process

            4) Emergency Option if files.api reports problems writing to disk due to insufficient space prompt admin to empty tempdir so users not affected.

            5) Impose quota on tempdir size set by site admin and include checking in cron process to warn admin if quota is neared
            If backup tries to backup a large course and the backup fails to complete successfully a large amount of disk can be taken up in filedir/temp/backup of moodledata. Further attemps to complete the same course backups will not reuse the temp directory contents or clean out old contents but instead start again.

            This can cause disk space usage to grow to hit limit imposed by host or disk capacity with little notification to site admin or entry in server logs. Site functionality could also be affected if no space available for new courses/resources to be added. Unknowningly users with backup permissions could crash site by repeating command to backup large course

            A side affect of this is non-moodle based backups of db and files take longer because contents of tempdir is larger

            To reproduce:
            Attempt to backup a large course either manually or via automated backups. Monitor the moodledata/filedir/temp/backup When the backup fails due to timeout of apache or php (e.g html error 500) the contents will remain along with an blank .log file with the same name as the temp folder which was created, in my case just under 10GB for 1 course
            rerun the backup again and a new folder will be created in the temp directory but without cleaning up the 1st attempt

            ToDo: Confirm behaviour is the same if backup auto/automated backup or auto/manual and backup command issued by cli

            Possible Cures:
            1) Log start time of backup in logs, with progress updates entered and if backup is not completed in X hours delete the temp folder and notify admin. This would also help admins know how far backups get before they fail

            2) Allow site admin/backup user to see list of tempdir on backup screen and clear as needed
            Include cleanup

            3) Possibly wider benefit from this but allow site admin to list temp dir size and contents and set "time to live" value for contents before cleaned up by cron process

            4) Emergency Option if files.api reports problems writing to disk due to insufficient space prompt admin to empty tempdir so users not affected.

            5) Impose quota on tempdir size set by site admin and include checking in cron process to warn admin if quota is neared
            mobcdi Michael OBrien made changes -
            Labels administration backup capacity cron disk storage tempdir
            mobcdi Michael OBrien made changes -
            Summary Failed Backups do not clean up contents of filedir/temp/backup after failing and future backups don't reuse the contents Failed Backups do not clean up contents of moodledata/temp/backup after failing and future backups don't reuse the contents
            mobcdi Michael OBrien made changes -
            Description If backup tries to backup a large course and the backup fails to complete successfully a large amount of disk can be taken up in filedir/temp/backup of moodledata. Further attemps to complete the same course backups will not reuse the temp directory contents or clean out old contents but instead start again.

            This can cause disk space usage to grow to hit limit imposed by host or disk capacity with little notification to site admin or entry in server logs. Site functionality could also be affected if no space available for new courses/resources to be added. Unknowningly users with backup permissions could crash site by repeating command to backup large course

            A side affect of this is non-moodle based backups of db and files take longer because contents of tempdir is larger

            To reproduce:
            Attempt to backup a large course either manually or via automated backups. Monitor the moodledata/filedir/temp/backup When the backup fails due to timeout of apache or php (e.g html error 500) the contents will remain along with an blank .log file with the same name as the temp folder which was created, in my case just under 10GB for 1 course
            rerun the backup again and a new folder will be created in the temp directory but without cleaning up the 1st attempt

            ToDo: Confirm behaviour is the same if backup auto/automated backup or auto/manual and backup command issued by cli

            Possible Cures:
            1) Log start time of backup in logs, with progress updates entered and if backup is not completed in X hours delete the temp folder and notify admin. This would also help admins know how far backups get before they fail

            2) Allow site admin/backup user to see list of tempdir on backup screen and clear as needed
            Include cleanup

            3) Possibly wider benefit from this but allow site admin to list temp dir size and contents and set "time to live" value for contents before cleaned up by cron process

            4) Emergency Option if files.api reports problems writing to disk due to insufficient space prompt admin to empty tempdir so users not affected.

            5) Impose quota on tempdir size set by site admin and include checking in cron process to warn admin if quota is neared
            If backup tries to backup a large course and the backup fails to complete successfully a large amount of disk can be taken up in moodledata/temp/backup. Further attemps to complete the same course backups will not reuse the temp directory contents or clean out old contents but instead start again.

            This can cause disk space usage to grow to hit limit imposed by host or disk capacity with little or no notification to site admin or entry in server logs. Site functionality could also be affected if no space available for new courses/resources to be added. Unknowningly users with backup permissions could crash site by repeating command to backup large course and fill allocated disk space with incomplete course backups

            A side affect of this is non-moodle based backups of db and files take longer because contents of tempdir is larger than necessary

            To reproduce:
            Attempt to backup a large course either manually from backup UI or via automated backups configured via UL. Monitor the moodledata/filedir/temp/backup When the backup fails due to timeout of apache or php (e.g html error 500) the contents will remain along with an blank .log file with the same name as the temp folder which was created, in my case just under 10GB for 1 course
            rerun the backup again and a new folder will be created in the temp directory but without cleaning up the 1st attempt


            Possible Cures:
            1) Log start time of backup in logs, with progress updates entered and if backup is not completed in X hours delete the contents of temp\backup folder and notify admin. This would also help admins know how far backups get before they fail.

            2) Allow site admin/backup user to see list of tempdir on backup screen and clear as needed

            3) Possibly wider benefit from this but allow site admin to list temp dir size and contents and set "time to live" value for contents before cleaned up by cron process

            4) Emergency Option if files.api reports problems writing to disk due to insufficient space prompt admin to empty tempdir so users not affected.

            5) Impose quota on tempdir size set by site admin and include checking in cron process to warn admin if quota is neared
            Workaround Not a solution to the tempdir\backup filling with failed backups but to allow large backups to complete fully run automated backup via cli (be aware of this issue http://tracker.moodle.org/browse/MDL-32548 )
            salvetore Michael de Raadt made changes -
            Link This issue has a non-specific relationship to MDL-1520 [ MDL-1520 ]
            salvetore Michael de Raadt made changes -
            Link This issue has a non-specific relationship to MDL-27790 [ MDL-27790 ]
            salvetore Michael de Raadt made changes -
            Security Could be a security issue [ 10030 ]
            Fix Version/s STABLE backlog [ 10463 ]
            Labels administration backup capacity cron disk storage tempdir triaged
            Component/s Files API [ 10305 ]
            salvetore Michael de Raadt made changes -
            Link This issue has been marked as being related by MDL-33752 [ MDL-33752 ]
            salvetore Michael de Raadt made changes -
            Link This issue has been marked as being related by MDL-38189 [ MDL-38189 ]
            rajeshtaneja Rajesh Taneja made changes -
            Assignee Rajesh Taneja [ rajeshtaneja ]
            marina Marina Glancy made changes -
            Affects Version/s 2.6.4 [ 13551 ]
            markn Mark Nelson made changes -
            Sprint BACKEND Sprint 18 [ 34 ]
            abgreeve Adrian Greeve made changes -
            Assignee Adrian Greeve [ abgreeve ]
            abgreeve Adrian Greeve made changes -
            Status Open [ 1 ] Development in progress [ 3 ]
            abgreeve Adrian Greeve made changes -
            Pull Master Diff URL https://github.com/abgreeve/moodle/compare/wip-MDL-32547-master
            Pull Master Branch wip-MDL-32547-master
            Pull from Repository git://github.com/abgreeve/moodle.git
            abgreeve Adrian Greeve made changes -
            Testing Instructions # Create a course.
            # Create a failed backup. I edited the backup_stepslib.php file and put a die(); in one of the functions.
            # Check moodledata/temp/backup and ensure that there are directories listed there.
            #* Update the scheduled task "Clean up failed backup files from the temp directory" [Site administration ► Server ► Scheduled tasks] and change all of the times to '*'.
            #* You can either Wait an hour [Site administration ► Courses ► Backups ► General backup defaults] and set 'Failed backup cleanup' to '1 hours'
            #* Or change the scheduled task code [lib/classes/task/failed_backup_task.php]
            {code}
            $timedelay = intval(get_config('backup', 'failedbackupcheck')) * 3600;
            {code}
            to
            {code}
            $timedelay = 1;
            {code}
            # Run the cron.
            # Have a look in the temp/backup directory and check that all of the directories have been removed.
            abgreeve Adrian Greeve made changes -
            Status Development in progress [ 3 ] Waiting for peer review [ 10012 ]
            cibot CiBoT made changes -
            Labels triaged ci triaged
            abgreeve Adrian Greeve made changes -
            Labels ci triaged triaged
            cibot CiBoT made changes -
            Labels triaged ci triaged
            abgreeve Adrian Greeve made changes -
            Labels ci triaged triaged
            cibot CiBoT made changes -
            Labels triaged ci triaged
            ankit_frenz Ankit Agarwal made changes -
            Original Estimate 0 minutes [ 0 ]
            Remaining Estimate 0 minutes [ 0 ]
            Status Waiting for peer review [ 10012 ] Peer review in progress [ 10013 ]
            Peer reviewer Ankit Agarwal [ ankit_frenz ]
            ankit_frenz Ankit Agarwal made changes -
            Status Peer review in progress [ 10013 ] Development in progress [ 3 ]
            cibot CiBoT made changes -
            Labels ci triaged triaged
            abgreeve Adrian Greeve made changes -
            Link This issue is blocked by MDL-48252 [ MDL-48252 ]
            abgreeve Adrian Greeve made changes -
            Testing Instructions # Create a course.
            # Create a failed backup. I edited the backup_stepslib.php file and put a die(); in one of the functions.
            # Check moodledata/temp/backup and ensure that there are directories listed there.
            #* Update the scheduled task "Clean up failed backup files from the temp directory" [Site administration ► Server ► Scheduled tasks] and change all of the times to '*'.
            #* You can either Wait an hour [Site administration ► Courses ► Backups ► General backup defaults] and set 'Failed backup cleanup' to '1 hours'
            #* Or change the scheduled task code [lib/classes/task/failed_backup_task.php]
            {code}
            $timedelay = intval(get_config('backup', 'failedbackupcheck')) * 3600;
            {code}
            to
            {code}
            $timedelay = 1;
            {code}
            # Run the cron.
            # Have a look in the temp/backup directory and check that all of the directories have been removed.
            # Create a course.
            # Create a failed backup. I edited the backup_stepslib.php file and put a die(); in one of the functions.
            # Check moodledata/temp/backup and ensure that there are directories listed there.
            # Update the scheduled task "Delete stale temp files" [Site administration ► Server ► Scheduled tasks] and change all of the times to '*'.
            #* You can either Wait an hour [Site administration ► Server ► Cleanup ] and set 'Clean up temporary data files older than' to '1 hours'.
            #* Or change the scheduled task code [lib/classes/task/file_temp_cleanup_task.php]
            {code}
            $timedelay = $CFG->tempdatafoldercleanup * 3600;
            {code}
            to
            {code}
            $timedelay = 1;
            {code}
            # Run the cron.
            # Have a look in the temp/backup directory and check that all of the directories have been removed.
            abgreeve Adrian Greeve made changes -
            Status Development in progress [ 3 ] Waiting for peer review [ 10012 ]
            cibot CiBoT made changes -
            Labels triaged ci triaged
            ankit_frenz Ankit Agarwal made changes -
            Status Waiting for peer review [ 10012 ] Peer review in progress [ 10013 ]
            ankit_frenz Ankit Agarwal made changes -
            Labels ci triaged ci docs_required triaged
            ankit_frenz Ankit Agarwal made changes -
            Status Peer review in progress [ 10013 ] Development in progress [ 3 ]
            cibot CiBoT made changes -
            Labels ci docs_required triaged docs_required triaged
            abgreeve Adrian Greeve made changes -
            Status Development in progress [ 3 ] Waiting for integration review [ 10010 ]
            cibot CiBoT made changes -
            Status Waiting for integration review [ 10010 ] Waiting for integration review [ 10010 ]
            Currently in integration Yes [ 10041 ]
            cibot CiBoT made changes -
            Labels docs_required triaged ci docs_required triaged
            poltawski Dan Poltawski made changes -
            Status Waiting for integration review [ 10010 ] Integration review in progress [ 10004 ]
            Integrator Dan Poltawski [ poltawski ]
            poltawski Dan Poltawski made changes -
            Status Integration review in progress [ 10004 ] Waiting for testing [ 10005 ]
            Fix Version/s 2.9 [ 14251 ]
            Fix Version/s STABLE backlog [ 10463 ]
            poltawski Dan Poltawski made changes -
            Summary Failed Backups do not clean up contents of moodledata/temp/backup after failing and future backups don't reuse the contents Configure temporary file deletion, to avoid moodledata/temp/backup filling up the filesystem
            rajeshtaneja Rajesh Taneja made changes -
            Tester Rajesh Taneja [ rajeshtaneja ]
            rajeshtaneja Rajesh Taneja made changes -
            Status Waiting for testing [ 10005 ] Testing in progress [ 10011 ]
            abgreeve Adrian Greeve made changes -
            Testing Instructions # Create a course.
            # Create a failed backup. I edited the backup_stepslib.php file and put a die(); in one of the functions.
            # Check moodledata/temp/backup and ensure that there are directories listed there.
            # Update the scheduled task "Delete stale temp files" [Site administration ► Server ► Scheduled tasks] and change all of the times to '*'.
            #* You can either Wait an hour [Site administration ► Server ► Cleanup ] and set 'Clean up temporary data files older than' to '1 hours'.
            #* Or change the scheduled task code [lib/classes/task/file_temp_cleanup_task.php]
            {code}
            $timedelay = $CFG->tempdatafoldercleanup * 3600;
            {code}
            to
            {code}
            $timedelay = 1;
            {code}
            # Run the cron.
            # Have a look in the temp/backup directory and check that all of the directories have been removed.
            # Create a course.
            # Create a failed backup. I edited the backup_stepslib.php file and put a die(); in one of the functions.
            # Check moodledata/temp/backup and ensure that there are directories listed there.
            # Update the scheduled task "Delete stale temp files" [Site administration ► Server ► Scheduled tasks] and change all of the times to '*'.
            #* You can either Wait an hour [Site administration ► Server ► Cleanup ] and set 'Clean up temporary data files older than' to '1 hours'.
            #* Or change the scheduled task code [lib/classes/task/file_temp_cleanup_task.php]
            {code}
            $time = time() - ($CFG->tempdatafoldercleanup * 3600);
            {code}
            to
            {code}
            $time = time() - 1;
            {code}
            # Run the cron.
            # Have a look in the temp/backup directory and check that all of the directories have been removed.
            rajeshtaneja Rajesh Taneja made changes -
            Status Testing in progress [ 10011 ] Tested [ 10006 ]
            marina Marina Glancy made changes -
            Labels ci docs_required triaged ci docs_required triaged ui_change
            marina Marina Glancy made changes -
            Issue Type Bug [ 1 ] Improvement [ 4 ]
            stronk7 Eloy Lafuente (stronk7) made changes -
            Status Tested [ 10006 ] Closed [ 6 ]
            Resolution Fixed [ 1 ]
            Currently in integration Yes [ 10041 ]
            Integration date 05/Dec/14
            abgreeve Adrian Greeve made changes -
            Labels ci docs_required triaged ui_change ci triaged ui_change

              People

              • Votes:
                8 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  11/May/15

                  Agile