Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-28505

Asynchronous course backup / restore

XMLWordPrintable

    • MOODLE_20_STABLE, MOODLE_23_STABLE, MOODLE_34_STABLE, MOODLE_37_STABLE
    • MOODLE_37_STABLE
    • master_MDL-28505_Asynchronous_backup_and_restore
    • Hide

      Setup:

      • Set up a Moodle with a test course.  To be test the progress bar functionality, create a large course with lots of content. Using the core test course creator development tool is a good way to do this.
      • Be able to run cron on demand (e.g. via CLI).  I recommend triggering cron via CLI as it gives greater control during testing. But if cron runs automatically that will work too.
      • Make sure site can send emails (to test messaging functionality)
        • Setup mailcatcher.
        • Visit ‘Site administration’ > ‘Server’ > ‘Outgoing mail configuration’.
        • Set the 'SMTP hosts' field to '127.0.0.1:1025'.
      • Change notification settings to accept notifications about asynchronous backups.
        • Go the the user menu (top right) and select preferences.
        • In User account follow "Notification preferences".
        • Under system - Asynchronous backup / restore notifications - enable email.
      • Enable asynchronous backup and restore: Site Administration > Advanced Features > enable asynchronous backups

       

      Backup course

      1. Access Moodle test course
      2. Choose backup from the course context menu
      3. Navigate through backup options as per normal (or select jump to final step if you’re lazy)
      4. You should be presented with the the backup process pending indication and text.
      5. Trigger cron from the command line (sudo -u  www-data php admin/cli/cron.php)
      6. The ajax progress should update while the backup is in progress, the UI text will also update.
      7. While the backup is in progress navigate to other parts of the Moodle site.  This should be possible and session not be locked
      8. While the backup is in progress navigate to the course “restore” area for the course that is being backed up. There should be an “in progress” backup at the top of the “Course backup area” table.
      9. When the backup completes the back up should appear as per normal in the  “Course backup area” table.
      10. You should receive an email detailing the backup was successful

       

      Multiple course backup prevention

      1. Access Moodle test course
      2. Choose backup from the course context menu
      3. Navigate through backup options as per normal (or select jump to final step if you’re lazy)
      4. You should be presented with the the backup process pending indication and text.
      5. Navigate back to the  Moodle test course
      6. Choose backup from the course context menu
      7. You should not be able to perform a backup, instead you should get a warning message
      8. Run run cron to perform the backup
      9. Navigate back to the original course page
      10. Choose backup from the course context menu
      11. You should now be able to perform the backup operation as normal

       

      Course restore

      1. Access Moodle test course
      2. Choose restore from the course context menu.
      3. From the course restore area choose one of the backups made in previous tests, and click the restore link.
      4. Navigate through restore options as per normal
      5. You should be presented with the the restore process pending indication and text.
      6. Trigger cron from the command line (sudo -u  www-data php admin/cli/cron.php)
      7. The ajax progress should update while the restore is in progress, the UI text will also update.
      8. While the restore is in progress navigate to other parts of the Moodle site.  This should be possible and session not be locked
      9. While the restore is in progress navigate to the course “restore” area for the course that is being backed up. There should be an “in progress” restore at the top of the new “Restores in progress” table.
      10. When the restore completes the restored item will be able to be accessed from the  table.
      11. You should receive an email detailing the restore was successful

       

      Backup activity

      1. Access Moodle test course
      2. Click on an activity within the course
      3. Choose backup from the activity context menu
      4. Navigate through backup options as per normal (or select jump to final step if you’re lazy)
      5. You should be presented with the the backup process pending indication and text.
      6. Trigger cron from the command line (sudo -u  www-data php admin/cli/cron.php)
      7. The ajax progress should update while the backup is in progress, the UI text will also update.
      8. While the backup is in progress navigate to other parts of the Moodle site.  This should be possible and session not be locked
      9. While the backup is in progress navigate to the course “restore” area for the course that is being backed up. There should be an “in progress” backup at the top of the “Activity backup area” table.
      10. When the backup completes the back up should appear as per normal in the  “Activity backup area” table.
      11. You should receive an email detailing the backup was successful

       

      Restore activity

      1. Access Moodle test course
      2. Click on an activity within the course
      3. Choose restore from the activity context menu
      4. From the activity restore area choose one of the backups made in previous tests, and click the restore link.
      5. Navigate through restore options as per normal
      6. You should be presented with the the restore process pending indication and text.
      7. Trigger cron from the command line (sudo -u  www-data php admin/cli/cron.php)
      8. The ajax progress should update while the restore is in progress, the UI text will also update.
      9. While the restore is in progress navigate to other parts of the Moodle site.  This should be possible and session not be locked
      10. While the restore is in progress navigate to the course “restore” area for the course that is being backed up. There should be an “in progress” restore at the top of the new “Restores in progress” table.
      11. When the restored completes the restored item will be able to be accessed from the  table.
      12. You should receive an email detailing the restore was successful

       

      Course backup edit warning

      1. Access Moodle test course
      2. Choose backup from the course context menu
      3. Navigate through backup options as per normal (or select jump to final step if you’re lazy)
      4. You should be presented with the the backup process pending indication and text.
      5. Navigate back to the landing page for the course
      6. Turn editing on
      7. You should be presented with a warning message that there is a backup pending for this course.
      8. Once the backup has completed, refresh the course page with editing on. The warning message should not appear
      Show
      Setup: Set up a Moodle with a test course.  To be test the progress bar functionality, create a large course with lots of content. Using the core test course creator development tool is a good way to do this. Be able to run cron on demand (e.g. via CLI).  I recommend triggering cron via CLI as it gives greater control during testing. But if cron runs automatically that will work too. Make sure site can send emails (to test messaging functionality) Setup mailcatcher. Visit ‘Site administration’ > ‘Server’ > ‘Outgoing mail configuration’. Set the 'SMTP hosts' field to '127.0.0.1:1025'. Change notification settings to accept notifications about asynchronous backups. Go the the user menu (top right) and select preferences. In User account follow "Notification preferences". Under system - Asynchronous backup / restore notifications - enable email. Enable asynchronous backup and restore: Site Administration > Advanced Features > enable asynchronous backups   Backup course Access Moodle test course Choose backup from the course context menu Navigate through backup options as per normal (or select jump to final step if you’re lazy) You should be presented with the the backup process pending indication and text. Trigger cron from the command line (sudo -u  www-data php admin/cli/cron.php) The ajax progress should update while the backup is in progress, the UI text will also update. While the backup is in progress navigate to other parts of the Moodle site.  This should be possible and session not be locked While the backup is in progress navigate to the course “restore” area for the course that is being backed up. There should be an “in progress” backup at the top of the “Course backup area” table. When the backup completes the back up should appear as per normal in the  “Course backup area” table. You should receive an email detailing the backup was successful   Multiple course backup prevention Access Moodle test course Choose backup from the course context menu Navigate through backup options as per normal (or select jump to final step if you’re lazy) You should be presented with the the backup process pending indication and text. Navigate back to the  Moodle test course Choose backup from the course context menu You should not be able to perform a backup, instead you should get a warning message Run run cron to perform the backup Navigate back to the original course page Choose backup from the course context menu You should now be able to perform the backup operation as normal   Course restore Access Moodle test course Choose restore from the course context menu. From the course restore area choose one of the backups made in previous tests, and click the restore link. Navigate through restore options as per normal You should be presented with the the restore process pending indication and text. Trigger cron from the command line (sudo -u  www-data php admin/cli/cron.php) The ajax progress should update while the restore is in progress, the UI text will also update. While the restore is in progress navigate to other parts of the Moodle site.  This should be possible and session not be locked While the restore is in progress navigate to the course “restore” area for the course that is being backed up. There should be an “in progress” restore at the top of the new “Restores in progress” table. When the restore completes the restored item will be able to be accessed from the  table. You should receive an email detailing the restore was successful   Backup activity Access Moodle test course Click on an activity within the course Choose backup from the activity context menu Navigate through backup options as per normal (or select jump to final step if you’re lazy) You should be presented with the the backup process pending indication and text. Trigger cron from the command line (sudo -u  www-data php admin/cli/cron.php) The ajax progress should update while the backup is in progress, the UI text will also update. While the backup is in progress navigate to other parts of the Moodle site.  This should be possible and session not be locked While the backup is in progress navigate to the course “restore” area for the course that is being backed up. There should be an “in progress” backup at the top of the “Activity backup area” table. When the backup completes the back up should appear as per normal in the  “Activity backup area” table. You should receive an email detailing the backup was successful   Restore activity Access Moodle test course Click on an activity within the course Choose restore from the activity context menu From the activity restore area choose one of the backups made in previous tests, and click the restore link. Navigate through restore options as per normal You should be presented with the the restore process pending indication and text. Trigger cron from the command line (sudo -u  www-data php admin/cli/cron.php) The ajax progress should update while the restore is in progress, the UI text will also update. While the restore is in progress navigate to other parts of the Moodle site.  This should be possible and session not be locked While the restore is in progress navigate to the course “restore” area for the course that is being backed up. There should be an “in progress” restore at the top of the new “Restores in progress” table. When the restored completes the restored item will be able to be accessed from the  table. You should receive an email detailing the restore was successful   Course backup edit warning Access Moodle test course Choose backup from the course context menu Navigate through backup options as per normal (or select jump to final step if you’re lazy) You should be presented with the the backup process pending indication and text. Navigate back to the landing page for the course Turn editing on You should be presented with a warning message that there is a backup pending for this course. Once the backup has completed, refresh the course page with editing on. The warning message should not appear

      It would be great if user-performed course backups could be asynchronous, as in, the request returns instantly with a message that the backup is "queued". The user can then return to the backups page later to check if the backup is complete. There could be an AJAX interface to prevent them from needing to hit refresh.

      The back-end would be implemented in a process-queue. The advantage being:

      • Long-running backup processes do not hold up apache threads and use excessive memory/cpu on page-serving nodes which can effect other page requests.
      • The user's session is not locked during the time that the backup is running
      • It prevents all issues relating to max execution time, load balancer idle timeouts, partially completed backups, that can make course backups/restores fail.
      • It would allow the process queue to be scaled by adding more servers that pull from the queue.

      It is pretty common amongst large-scale web applications to not 'block' on long-running operations - but rather to return instantly and send a job to a back-end process queue.

      Features/ Functionality:

      • Site option to enable Asynchronous backup/restores. Defaults to off to not break anyone's current workflows.
      • Course and activity backups and restores are now asynchronous. Processing of asynchronous backups are processed by ad-hoc scheduled tasks. (Each backup or restore action queues a new ad-hoc task)
      • Backup / Restore progress is updated in the user interface via Ajax.
      • User interface is not session locked during backup and restores.
      • A user cannot queue up multiple copies of a backup of the same resource (course or activity). Only one backup of a resource can be done by a user at a time.
      • When a course that either has a backup pending or in process is put into edit mode, a warning notification is displayed to the user; about the risks of editing while a backup is in progress (This is an issue in Moodle currently)
      • An option to send an message to users when a back/restore to complete.  Uses Moodle message API
      • Notification message template can be customised and can use placholders e.g. {user_firstname}

        1. admin_advanced_features.png
          admin_advanced_features.png
          28 kB
        2. async_backup_in_progress_overview.png
          async_backup_in_progress_overview.png
          38 kB
        3. async_message_settings.png
          async_message_settings.png
          68 kB
        4. async_restore_complete.png
          async_restore_complete.png
          18 kB
        5. async_restore_in_progress.png
          async_restore_in_progress.png
          21 kB
        6. backup_menu_category.png
          backup_menu_category.png
          31 kB
        7. backup_overview_pending.png
          backup_overview_pending.png
          46 kB
        8. backup_process_pending.png
          backup_process_pending.png
          34 kB
        9. backup_warning.png
          backup_warning.png
          24 kB

            Votes:
            38 Vote for this issue
            Watchers:
            37 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 3 days, 5 hours, 30 minutes
                3d 5h 30m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.