[MDL-65729] Ad-hoc task processing for automated backups - Moodle Tracker

XML

Word

Printable

Details

Type: Improvement
Status: Closed
Priority: Minor
Resolution: Fixed
Affects Version/s: 3.7.1, 3.8
Fix Version/s: 3.9
Component/s: Backup
Labels:
- CatalystIT
- Partner
- ci
- release_notes
- triaged

Affected Branches:
MOODLE_37_STABLE, MOODLE_38_STABLE
Fixed Branches:
MOODLE_39_STABLE
Pull from Repository:
https://github.com/yao9394/moodle
Pull Master Branch:
master-~~MDL-65729~~
Pull Master Diff URL:
https://github.com/moodle/moodle/compare/master...yao9394:master-MDL-65729
Testing Instructions:
Hide

Create a few courses

Purge caches

Go to Administration > Site administration > Courses > Backups > Automated backup setup

Set backup_auto_active to enabled

Choose all days of the week

run

php admin/tool/task/cli/schedule_task.php --execute=\\core\\task\\automated_backup_task

from the command line.

Go to "Site administration > Reports > Backups" and check that the status is set to "Automated backup pending"

run the following sql command on you database (requires direct access to the database)

UPDATE "mdl_backup_courses" SET nextstarttime = '1575350530'

This will set the next start time into the past.

run

php admin/tool/task/cli/schedule_task.php --execute=\\core\\task\\automated_backup_task

from the command line.

Go to "Site administration > Reports > Backups" and check that the status is set to "Queued"

run

php admin/cli/cron.php

to execute the adhoc tasks.

Go to "Site administration > Reports > Backups" and check that the status is set to "OK"

Head to the restore section of one of the courses (Course administration > Restore) and check that there is a backup file located in the "Automated backups" section.
Show
Create a few courses Purge caches Go to Administration > Site administration > Courses > Backups > Automated backup setup Set backup_auto_active to enabled Choose all days of the week run php admin/tool/task/cli/schedule_task.php --execute=\\core\\task\\automated_backup_task from the command line. Go to "Site administration > Reports > Backups" and check that the status is set to "Automated backup pending" run the following sql command on you database (requires direct access to the database) UPDATE "mdl_backup_courses" SET nextstarttime = '1575350530' This will set the next start time into the past. run php admin/tool/task/cli/schedule_task.php --execute=\\core\\task\\automated_backup_task from the command line. Go to "Site administration > Reports > Backups" and check that the status is set to "Queued" run php admin/cli/cron.php to execute the adhoc tasks. Go to "Site administration > Reports > Backups" and check that the status is set to "OK" Head to the restore section of one of the courses (Course administration > Restore) and check that there is a backup file located in the "Automated backups" section.

Description

Currently the processing of automated backups runs a single long running cron task with its own bespoke locking.
One problem here is in a Moodle site that has a lot of courses that change often, the rate of change of the courses can be to much for the current way of processing automated backups. This means that automated course backups "get behind" and backups don't happen in a timely manner.
~~MDL-65201~~ is a first step to making this better by improving how courses are ordered for automated backups.
This request is to further refactor the automated backup code to:

Split individual backups into ad-hoc tasks. This gives the ability to run several automated backups in parallel and increase the overall throughput of the processing.
Remove the custom locking that is in the automated backup code and make automated backups use the regular scheduled task locking.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List
- Download All

screenshot-1.png
285 kB
04/Dec/19 2:53 PM

Issue Links

caused a regression

MDL-69615 Automated backup reports email should be send after all backups are done (all "Ad-hoc task to save pending course")

Open

has been marked as being related by

MDL-68739 Course not backed up with CLI script

Closed

is parent of

MDL-67502 moodle_exception can't be caught in course_backup_task class

Closed

will be (partly) resolved by

MDL-66308 Create an admin_setting_configcrontab object

Open

MDL-64610 Add support for per-task concurrency limits

Closed

MDL-65201 Automated backup course ordering

Closed

MDL-67363 Add a Quality of Service layer to the processing of the ad-hoc task queue

Closed

(2 will be (partly) resolved by)

Activity

People

Assignee:: John Yao

Reporter:: Matt Porritt

Peer reviewer:: Matt Porritt

Integrator:: Adrian Greeve

Tester:: Janelle Barcega

Participants:: Adrian Greeve, Brendan Heywood, CiBoT, Dmitrii Metelkin, Eloy Lafuente (stronk7), Janelle Barcega, John Yao, Matt Porritt

Component watchers:: Adrian Greeve, David Woloszyn, Huong Nguyen, Jake Dallimore, Meirza, Michael Hawkins, Raquel Ortega, Safat Shahin, Stevani Andolo

Votes:: 2 Vote for this issue

Watchers:: 8 Start watching this issue

Dates

Created:: 28/May/19 9:49 AM

Updated:: 03/Jun/22 12:10 PM

Resolved:: 05/Dec/19 8:45 PM

Fix Release Date:: 15/Jun/20

Time Tracking

Estimated:

Remaining:

Logged:

2d 3h 26m