Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-65729

Ad-hoc task processing for automated backups

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.7.1, 3.8
    • Fix Version/s: 3.9
    • Component/s: Backup
    • Testing Instructions:
      Hide
      1. Create a few courses
      2. Purge caches
      3. Go to Administration > Site administration > Courses > Backups > Automated backup setup
      4. Set backup_auto_active to enabled
      5. Choose all days of the week
      6. run

        php admin/tool/task/cli/schedule_task.php --execute=\\core\\task\\automated_backup_task

        from the command line.

      7. Go to "Site administration > Reports > Backups" and check that the status is set to "Automated backup pending"
      8. run the following sql command on you database (requires direct access to the database)

        UPDATE "mdl_backup_courses" SET nextstarttime = '1575350530'

        This will set the next start time into the past.

      9. run

        php admin/tool/task/cli/schedule_task.php --execute=\\core\\task\\automated_backup_task

        from the command line.

      10. Go to "Site administration > Reports > Backups" and check that the status is set to "Queued"
      11. run

        php admin/cli/cron.php

        to execute the adhoc tasks.

      12. Go to "Site administration > Reports > Backups" and check that the status is set to "OK"
      13. Head to the restore section of one of the courses (Course administration > Restore) and check that there is a backup file located in the "Automated backups" section.
      Show
      Create a few courses Purge caches Go to Administration > Site administration > Courses > Backups > Automated backup setup Set backup_auto_active to enabled Choose all days of the week run php admin/tool/task/cli/schedule_task.php --execute=\\core\\task\\automated_backup_task from the command line. Go to "Site administration > Reports > Backups" and check that the status is set to "Automated backup pending" run the following sql command on you database (requires direct access to the database) UPDATE "mdl_backup_courses" SET nextstarttime = '1575350530' This will set the next start time into the past. run php admin/tool/task/cli/schedule_task.php --execute=\\core\\task\\automated_backup_task from the command line. Go to "Site administration > Reports > Backups" and check that the status is set to "Queued" run php admin/cli/cron.php to execute the adhoc tasks. Go to "Site administration > Reports > Backups" and check that the status is set to "OK" Head to the restore section of one of the courses (Course administration > Restore) and check that there is a backup file located in the "Automated backups" section.
    • Affected Branches:
      MOODLE_37_STABLE, MOODLE_38_STABLE
    • Fixed Branches:
      MOODLE_39_STABLE
    • Pull from Repository:
    • Pull 3.8 Branch:
      MOODLE_38_STABLE-MDL-65729
    • Pull Master Branch:
      master-MDL-65729

      Description

      Currently the processing of automated backups runs a single long running cron task with its own bespoke locking.
      One problem here is in a Moodle site that has a lot of courses that change often, the rate of change of the courses can be to much for the current way of processing automated backups. This means that automated course backups "get behind" and backups don't happen in a timely manner.
      MDL-65201 is a first step to making this better by improving how courses are ordered for automated backups.
      This request is to further refactor the automated backup code to:

      • Split individual backups into ad-hoc tasks. This gives the ability to run several automated backups in parallel and increase the overall throughput of the processing.
      • Remove the custom locking that is in the automated backup code and make automated backups use the regular scheduled task locking.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              yao9394 John Yao
              Reporter:
              mattp@catalyst-au.net Matt Porritt
              Peer reviewer:
              Matt Porritt
              Integrator:
              Adrian Greeve
              Tester:
              Janelle Barcega
              Participants:
              Component watchers:
              Adrian Greeve, Jake Dallimore, Mathew May, Mihail Geshoski, Peter Dias
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Fix Release Date:
                8/Jun/20

                  Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 days, 3 hours, 26 minutes
                  2d 3h 26m