Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-71044

Extend the 'backup_cleanup_task' scheduled task to remove old files

    XMLWordPrintable

Details

    Description

      Investigating long running `\core_course\task\course_delete_modules` adhoc tasks, it's been found that recycle bin slows down the task:

      Execute adhoc task: core_course\task\course_delete_modules
      ... started 14:45:17. Current memory use 3.7MB.
      Wed, 03 Mar 2021 14:45:17 +1100 Deleting url
      Wed, 03 Mar 2021 14:45:17 +1100 Calling pre_course_module_delete callback
      Wed, 03 Mar 2021 14:45:17 +1100 Starting backup
      Wed, 03 Mar 2021 14:59:22 +1100 Backup size: 3.6KB
      Wed, 03 Mar 2021 14:59:22 +1100 Deleting questions and question categories
      Wed, 03 Mar 2021 14:59:22 +1100 Calling url_delete_instance function
      Wed, 03 Mar 2021 14:59:22 +1100 Removing all module files in case modules forget to do that
      Wed, 03 Mar 2021 14:59:22 +1100 Removing events from calendar
      Wed, 03 Mar 2021 14:59:22 +1100 Removing grade items
      Wed, 03 Mar 2021 14:59:22 +1100 Removing blogs
      Wed, 03 Mar 2021 14:59:22 +1100 Removing completion and availability data
      Wed, 03 Mar 2021 14:59:22 +1100 Removing all tags
      Wed, 03 Mar 2021 14:59:22 +1100 Calling hook_course_module_deleted
      Wed, 03 Mar 2021 14:59:22 +1100 Deleting the context
      Wed, 03 Mar 2021 14:59:22 +1100 Deleting the module from the course_modules table
      Wed, 03 Mar 2021 14:59:22 +1100 Deleting module from the course section
      Wed, 03 Mar 2021 14:59:22 +1100 Triggering course_module_deleted ervent
      Wed, 03 Mar 2021 14:59:22 +1100 Rebuilding course cache
      Wed, 03 Mar 2021 14:59:22 +1100 Time took to delete url: 845 seconds.
      ... used 484 dbqueries
      ... used 845.49422883987 seconds
      Adhoc task complete: core_course\task\course_delete_modules
      

      Profiling showed a bottleneck: `backup_helper::delete_old_backup_dirs` on `drop_and_clean_temp_stuff` step.

       

      This might happen in large sites with `temp/backup` dir located on NFS and with a big number or files in it (39k in this case). So total time to fetch files in the directory will be NFS latency * number of files.

      My proposal is to move `backup_helper::delete_old_backup_dirs` to a dedicated the existing 'backup_cleanup_task' scheduled task and run it once a day (or with any other schedule). I think we don't really need to full scan `temp/backup` directory after each backup/restore. Especially if we have thousands of these tasks per day.

      Steps to replicate the issue:

      Attachments

        Issue Links

          Activity

            People

              mikhailgolenkov Misha Golenkov
              mikhailgolenkov Misha Golenkov
              Brendan Heywood Brendan Heywood
              Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
              CiBoT CiBoT
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                17/May/21

                Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 40 minutes
                  40m