Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-65729

Ad-hoc task processing for automated backups

    XMLWordPrintable

Details

    • MOODLE_37_STABLE, MOODLE_38_STABLE
    • MOODLE_39_STABLE
    • master-MDL-65729
    • Hide
      1. Create a few courses
      2. Purge caches
      3. Go to Administration > Site administration > Courses > Backups > Automated backup setup
      4. Set backup_auto_active to enabled
      5. Choose all days of the week
      6. run 

        php admin/tool/task/cli/schedule_task.php --execute=\\core\\task\\automated_backup_task

        from the command line.

      7. Go to "Site administration > Reports > Backups" and check that the status is set to "Automated backup pending"
      8. run the following sql command on you database (requires direct access to the database) 

        UPDATE "mdl_backup_courses" SET nextstarttime = '1575350530'

        This will set the next start time into the past.

      9. run 

        php admin/tool/task/cli/schedule_task.php --execute=\\core\\task\\automated_backup_task

        from the command line.

      10. Go to "Site administration > Reports > Backups" and check that the status is set to "Queued"
      11. run 

        php admin/cli/cron.php

        to execute the adhoc tasks.

      12. Go to "Site administration > Reports > Backups" and check that the status is set to "OK"
      13. Head to the restore section of one of the courses (Course administration > Restore) and check that there is a backup file located in the "Automated backups" section.
      Show
      Create a few courses Purge caches Go to Administration > Site administration > Courses > Backups > Automated backup setup Set backup_auto_active to enabled Choose all days of the week run php admin/tool/task/cli/schedule_task.php --execute=\\core\\task\\automated_backup_task from the command line. Go to "Site administration > Reports > Backups" and check that the status is set to "Automated backup pending" run the following sql command on you database (requires direct access to the database) UPDATE "mdl_backup_courses" SET nextstarttime = '1575350530' This will set the next start time into the past. run php admin/tool/task/cli/schedule_task.php --execute=\\core\\task\\automated_backup_task from the command line. Go to "Site administration > Reports > Backups" and check that the status is set to "Queued" run php admin/cli/cron.php to execute the adhoc tasks. Go to "Site administration > Reports > Backups" and check that the status is set to "OK" Head to the restore section of one of the courses (Course administration > Restore) and check that there is a backup file located in the "Automated backups" section.

    Description

      Currently the processing of automated backups runs a single long running cron task with its own bespoke locking.
      One problem here is in a Moodle site that has a lot of courses that change often, the rate of change of the courses can be to much for the current way of processing automated backups. This means that automated course backups "get behind" and backups don't happen in a timely manner.
      MDL-65201 is a first step to making this better by improving how courses are ordered for automated backups.
      This request is to further refactor the automated backup code to:

      • Split individual backups into ad-hoc tasks. This gives the ability to run several automated backups in parallel and increase the overall throughput of the processing.
      • Remove the custom locking that is in the automated backup code and make automated backups use the regular scheduled task locking.

      Attachments

        Issue Links

          caused a regression

          Bug - A problem which impairs or prevents Moodle from functioning correctly. MDL-69615 Automated backup reports email should be send after all backups are done (all "Ad-hoc task to save pending course")

          • Minor - Minor loss of function where workaround is possible
          • Open
          has been marked as being related by

          Bug - A problem which impairs or prevents Moodle from functioning correctly. MDL-68739 Course not backed up with CLI script

          • Minor - Minor loss of function where workaround is possible
          • Closed
          is parent of

          Bug - A problem which impairs or prevents Moodle from functioning correctly. MDL-67502 moodle_exception can't be caught in course_backup_task class

          • Minor - Minor loss of function where workaround is possible
          • Closed
          will be (partly) resolved by

          Improvement - An enhancement to an existing Moodle feature. MDL-66308 Create an admin_setting_configcrontab object

          • Minor - Minor loss of function where workaround is possible
          • Open

          Improvement - An enhancement to an existing Moodle feature. MDL-64610 Add support for per-task concurrency limits

          • Minor - Minor loss of function where workaround is possible
          • Closed

          Improvement - An enhancement to an existing Moodle feature. MDL-65201 Automated backup course ordering

          • Minor - Minor loss of function where workaround is possible
          • Closed

          Improvement - An enhancement to an existing Moodle feature. MDL-67363 Add a Quality of Service layer to the processing of the ad-hoc task queue

          • Minor - Minor loss of function where workaround is possible
          • Closed

          Activity

            People

              yao9394 John Yao
              mattp@catalyst-au.net Matt Porritt
              Matt Porritt Matt Porritt
              Adrian Greeve Adrian Greeve
              Janelle Barcega Janelle Barcega
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                15/Jun/20

                Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 days, 3 hours, 26 minutes
                  2d 3h 26m