Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-71044

Extend the 'backup_cleanup_task' scheduled task to remove old files

    XMLWordPrintable

    Details

      Description

      Investigating long running `\core_course\task\course_delete_modules` adhoc tasks, it's been found that recycle bin slows down the task:

      Execute adhoc task: core_course\task\course_delete_modules
      ... started 14:45:17. Current memory use 3.7MB.
      Wed, 03 Mar 2021 14:45:17 +1100 Deleting url
      Wed, 03 Mar 2021 14:45:17 +1100 Calling pre_course_module_delete callback
      Wed, 03 Mar 2021 14:45:17 +1100 Starting backup
      Wed, 03 Mar 2021 14:59:22 +1100 Backup size: 3.6KB
      Wed, 03 Mar 2021 14:59:22 +1100 Deleting questions and question categories
      Wed, 03 Mar 2021 14:59:22 +1100 Calling url_delete_instance function
      Wed, 03 Mar 2021 14:59:22 +1100 Removing all module files in case modules forget to do that
      Wed, 03 Mar 2021 14:59:22 +1100 Removing events from calendar
      Wed, 03 Mar 2021 14:59:22 +1100 Removing grade items
      Wed, 03 Mar 2021 14:59:22 +1100 Removing blogs
      Wed, 03 Mar 2021 14:59:22 +1100 Removing completion and availability data
      Wed, 03 Mar 2021 14:59:22 +1100 Removing all tags
      Wed, 03 Mar 2021 14:59:22 +1100 Calling hook_course_module_deleted
      Wed, 03 Mar 2021 14:59:22 +1100 Deleting the context
      Wed, 03 Mar 2021 14:59:22 +1100 Deleting the module from the course_modules table
      Wed, 03 Mar 2021 14:59:22 +1100 Deleting module from the course section
      Wed, 03 Mar 2021 14:59:22 +1100 Triggering course_module_deleted ervent
      Wed, 03 Mar 2021 14:59:22 +1100 Rebuilding course cache
      Wed, 03 Mar 2021 14:59:22 +1100 Time took to delete url: 845 seconds.
      ... used 484 dbqueries
      ... used 845.49422883987 seconds
      Adhoc task complete: core_course\task\course_delete_modules
      

      Profiling showed a bottleneck: `backup_helper::delete_old_backup_dirs` on `drop_and_clean_temp_stuff` step.

       

      This might happen in large sites with `temp/backup` dir located on NFS and with a big number or files in it (39k in this case). So total time to fetch files in the directory will be NFS latency * number of files.

      My proposal is to move `backup_helper::delete_old_backup_dirs` to a dedicated the existing 'backup_cleanup_task' scheduled task and run it once a day (or with any other schedule). I think we don't really need to full scan `temp/backup` directory after each backup/restore. Especially if we have thousands of these tasks per day.

      Steps to replicate the issue:

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              mikhailgolenkov Mikhail Golenkov
              Reporter:
              mikhailgolenkov Mikhail Golenkov
              Peer reviewer:
              Brendan Heywood Brendan Heywood
              Integrator:
              Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
              Tester:
              CiBoT CiBoT
              Participants:
              Component watchers:
              Adrian Greeve, Jake Dallimore, Mathew May, Mihail Geshoski, Peter Dias, Sujith Haridasan
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Fix Release Date:
                17/May/21

                  Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 40 minutes
                  40m