Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-38189 META Backup/restore issues with large courses
  3. MDL-38190

Backup and restore operations should display progress

XMLWordPrintable

    • MOODLE_23_STABLE, MOODLE_24_STABLE, MOODLE_25_STABLE, MOODLE_26_STABLE
    • MOODLE_26_STABLE
    • MDL-38190-master
    • Hide

      0. You will need a sample course to backup and restore. I used the 'M' test course created by the tool from MDL-38197.

      1. Go to the course. Do a backup with default settings. The screens should be unchanged except when you get to the end of step 3 and click to actually start the backup...

      EXPECTED:

      • Step 4 should display fairly soon (may be a bit of a pause while it loads the backup controller).
      • You should see a progress bar that moves across from left to right (albeit with some jerky parts) as the backup progresses.
      • There is no text next to the progress bar (I decided not to display the backup task names because they are not user-friendly) but there is a time remaining estimate which should update, as standard Moodle progress bar behaviour.
      • A thin bar below the progress bar should gradually change colour (from black to white and back) as the backup progresses. (This bar is intended to indicate that something is happening even when we can't necessarily measure progress numerically.)
      • Once the progress bar completes, the step 4 content should immediately disappear and step 5 - completed, with the continue button - should appear. (These two steps are actually on the same HTML page.)

      2. Continue to the list of backups. Click Restore against the backup you just made, and go through the steps with default options to restore it to a new course. This should be unchanged until you get to the point where it actually processes the restore.

      EXPECTED: The restore progress should be displayed in the same manner as for backup.

      3. Within a course, use the 'duplicate' feature to duplicate an activity.

      EXPECTED: The 'duplicate' feature should work exactly as before, without a progress bar. (Note: This test is because duplicating uses the backup/restore API, so we want to check it still works OK when there is no progress display.)

      4. You'll need another, probably smaller, course; I used the 'XS' test course. In your new course, choose the 'Import' option and select the smaller course. Accept default options and import everything.

      EXPECTED: When you click 'Process import', you should see a screen with a suitable stage heading and a progress bar that moves across as the import is carried out. As soon as it finishes, the progress bar should disappear and be replaced with the 'Continue' button (as was there before this change).

      Show
      0. You will need a sample course to backup and restore. I used the 'M' test course created by the tool from MDL-38197 . 1. Go to the course. Do a backup with default settings. The screens should be unchanged except when you get to the end of step 3 and click to actually start the backup... EXPECTED: Step 4 should display fairly soon (may be a bit of a pause while it loads the backup controller). You should see a progress bar that moves across from left to right (albeit with some jerky parts) as the backup progresses. There is no text next to the progress bar (I decided not to display the backup task names because they are not user-friendly) but there is a time remaining estimate which should update, as standard Moodle progress bar behaviour. A thin bar below the progress bar should gradually change colour (from black to white and back) as the backup progresses. (This bar is intended to indicate that something is happening even when we can't necessarily measure progress numerically.) Once the progress bar completes, the step 4 content should immediately disappear and step 5 - completed, with the continue button - should appear. (These two steps are actually on the same HTML page.) 2. Continue to the list of backups. Click Restore against the backup you just made, and go through the steps with default options to restore it to a new course. This should be unchanged until you get to the point where it actually processes the restore. EXPECTED: The restore progress should be displayed in the same manner as for backup. 3. Within a course, use the 'duplicate' feature to duplicate an activity. EXPECTED: The 'duplicate' feature should work exactly as before, without a progress bar. (Note: This test is because duplicating uses the backup/restore API, so we want to check it still works OK when there is no progress display.) 4. You'll need another, probably smaller, course; I used the 'XS' test course. In your new course, choose the 'Import' option and select the smaller course. Accept default options and import everything. EXPECTED: When you click 'Process import', you should see a screen with a suitable stage heading and a progress bar that moves across as the import is carried out. As soon as it finishes, the progress bar should disappear and be replaced with the 'Continue' button (as was there before this change).

      We would like backup and restore operations to display progress.

      • Backup and restore operations can take a very long time.
      • Nothing at all is displayed during the operation.

      For all Moodle users, not just us, this is unsatisfactory as it can cause users to become unsure about whether the operation is continuing or has failed.

      • Our front-end server (a NetScaler) has a time limit on requests. If nothing is output from the server, requests are cancelled after 6 minutes. As a result, backup and restore operations will fail if they take longer than 6 minutes (they nearly always do).
      • Other insititutions are likely to use similar front-end servers.

      Progress information could consist of any of the following:

      • A progress bar.
      • Dots printed periodically.
      • Anything else that causes a visible update via data sent as part of the HTTP response.

      Detail 1: The progress information must be output as part of the main HTTP request (the one that is taking a long time to complete) and not via any associated request such as an AJAX request.

      • This is because if there is no output as part of the main request for 6 minutes, then that request will fail on our systems, regardless of what other requests may be going on.
      • Note: This may mean it is not possible to see progress information on Internet Explorer 9 until it finishes, due to technical limitations on displaying partially-loaded web pages in that browser. This would be acceptable to us and probably most people...

      Detail 2: Progress information should be provided frequently so that there are no long pauses without progress information.

      • For example, we achieve some of this in internal code using a potential_dot function which displays a dot if it has been at least 1 second since the last dot was displayed, and then calling it in lots of places. After displaying the dot, it also resets the PHP time limit to a suitable time.
      • Implementing this will require potentially displaying progress in the following locations:
        • When each backup/restore task starts.
        • Before or after copying each file, when copying files.
        • Before or after zipping / unzipping each file (or a certain number of files).

      Note: We have previously achieved this by using the command-line zip and unzip tool, set to verbose mode, and reading its output buffer, displaying progress at each chunk of data read.

      Detail 3: As part of the backup/restore API it should be possible for plugins such as modules to themselves indicate that progress has occurred.

      • For example, when backing up user data from a forum, this could take a very long time if there are many discussions. The forum should be able to indicate progress between each discussion or each post.

      Detail 4: When an instance of the resource module contains a large number of files (e.g. 10,000) progress should be indicated between files.

      • Without looking at the code, I’m not sure whether this should happen in a generic way (like there is some ‘add file’ method that should be trapped) or it has to be specific to the resource module.
      • These large resources can take a very long time (> 100s) even in their standard backup task.

      Detail 5: The Moodle time limit used during backup/restore should be configurable in admin settings (similar to extramemorylimit), or fixed to a reasonable (low) value e.g. 2 minutes. The limit should apply from the last progress update.

      • Currently the time limit is fixed to 1 hour. This is not very useful in our system where the NetScaler cancels the request well before a PHP timeout occurs. It would be more helpful to see the timeout error.
      • Resetting the time limit each time progress is updated means that as long as progress is continually updated, the backup and restore can take hours if required. However the backup/restore will time out if there is a long time without a progress report, which indicates a system bug (the system is not reporting progress frequently enough).

      Detail 6: When using the code API to do backup and restore, it should be possible to hook into this progress system and control progress display, for example via a callback function which is called regularly with progress information.

      • Our customised roll forward system uses the code API and may need to display progress in a different way.
      • This will also be necessary for other non-OU users of the backup/restore API, such as the standard duplicate feature.

            quen Sam Marshall
            quen Sam Marshall
            Dan Poltawski Dan Poltawski
            Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
            David Monllaó David Monllaó
            Votes:
            6 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.