Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-71044

Extend the 'backup_cleanup_task' scheduled task to remove old files

    XMLWordPrintable

Details

    Description

      Investigating long running `\core_course\task\course_delete_modules` adhoc tasks, it's been found that recycle bin slows down the task:

      Execute adhoc task: core_course\task\course_delete_modules
      ... started 14:45:17. Current memory use 3.7MB.
      Wed, 03 Mar 2021 14:45:17 +1100 Deleting url
      Wed, 03 Mar 2021 14:45:17 +1100 Calling pre_course_module_delete callback
      Wed, 03 Mar 2021 14:45:17 +1100 Starting backup
      Wed, 03 Mar 2021 14:59:22 +1100 Backup size: 3.6KB
      Wed, 03 Mar 2021 14:59:22 +1100 Deleting questions and question categories
      Wed, 03 Mar 2021 14:59:22 +1100 Calling url_delete_instance function
      Wed, 03 Mar 2021 14:59:22 +1100 Removing all module files in case modules forget to do that
      Wed, 03 Mar 2021 14:59:22 +1100 Removing events from calendar
      Wed, 03 Mar 2021 14:59:22 +1100 Removing grade items
      Wed, 03 Mar 2021 14:59:22 +1100 Removing blogs
      Wed, 03 Mar 2021 14:59:22 +1100 Removing completion and availability data
      Wed, 03 Mar 2021 14:59:22 +1100 Removing all tags
      Wed, 03 Mar 2021 14:59:22 +1100 Calling hook_course_module_deleted
      Wed, 03 Mar 2021 14:59:22 +1100 Deleting the context
      Wed, 03 Mar 2021 14:59:22 +1100 Deleting the module from the course_modules table
      Wed, 03 Mar 2021 14:59:22 +1100 Deleting module from the course section
      Wed, 03 Mar 2021 14:59:22 +1100 Triggering course_module_deleted ervent
      Wed, 03 Mar 2021 14:59:22 +1100 Rebuilding course cache
      Wed, 03 Mar 2021 14:59:22 +1100 Time took to delete url: 845 seconds.
      ... used 484 dbqueries
      ... used 845.49422883987 seconds
      Adhoc task complete: core_course\task\course_delete_modules
      

      Profiling showed a bottleneck: `backup_helper::delete_old_backup_dirs` on `drop_and_clean_temp_stuff` step.

       

      This might happen in large sites with `temp/backup` dir located on NFS and with a big number or files in it (39k in this case). So total time to fetch files in the directory will be NFS latency * number of files.

      My proposal is to move `backup_helper::delete_old_backup_dirs` to a dedicated the existing 'backup_cleanup_task' scheduled task and run it once a day (or with any other schedule). I think we don't really need to full scan `temp/backup` directory after each backup/restore. Especially if we have thousands of these tasks per day.

      Steps to replicate the issue:

      Attachments

        Issue Links

          Activity

            People

              mikhailgolenkov Misha Golenkov
              mikhailgolenkov Misha Golenkov
              Brendan Heywood Brendan Heywood
              Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
              CiBoT CiBoT
              Adrian Greeve, Jake Dallimore, Mathew May, Mihail Geshoski, Sujith Haridasan
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                17/May/21

                Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 40 minutes
                  40m