Moodle
  1. Moodle
  2. MDL-38734

Automated backup failure causing all courses to fail

    Details

    • Rank:
      48789

      Description

      During automated backups, if a course fails with an exception, the cleanup task is not run. This leaves the backup_ids_temp table in place, as well as files etc.

      The problem is that next backup cannot start because creation of the temp table will fail (throws exception ddltableexists).

      Currently, this appears to affect Moodle 2.5 and 2.4, but not the current master (pre-2.6).

      Replication steps:

      1. Set time limit in your php.ini something very large like a day.
      2. Log in as admin
      3. Ensure there are multiple courses created on your site.
      4. Navigate to a course in the middle (not the first or last)
      5. Add large files to that course so that it contains at least 6GB of media content spread over multiple activities.
      6. Set the following site configuration settings...
        • backup_auto_active - manual
        • backup_auto_storage - specified directory
        • backup_auto_destination - path to some writeable dir
      7. Run the script php admin/cli/automated_backup.php

      Expected result: Backup of the large course should fail, but others should succeed.

      Actual result: Backup of the large course fails, but so do courses that are backed up following it. They fail quickly.

      The log of the course backup supports what Tony has described.

      [17/10/2013 11:31:49 PM] [info] instantiating backup controller fce438d11f81798ed74c7369b0a08294
      [17/10/2013 11:31:49 PM] [debug] setting controller status to 100
      [17/10/2013 11:31:49 PM] [debug] loading controller plan
      [17/10/2013 11:31:49 PM] [debug] setting controller status to 300
      [17/10/2013 11:31:49 PM] [debug] applying plan defaults
      [17/10/2013 11:31:49 PM] [debug] setting controller status to 400
      [17/10/2013 11:31:49 PM] [info] checking plan security
      [17/10/2013 11:31:49 PM] [debug] setting controller status to 700
      [17/10/2013 11:31:49 PM] [debug] saving controller to db
      [17/10/2013 11:31:49 PM] [debug] calculating controller checksum 759ed72c7ceeb724a2d28bfc85903b6c
      [17/10/2013 11:31:49 PM] [debug] loading controller from db
      [17/10/2013 11:31:49 PM] [debug] setting controller status to 700
      [17/10/2013 11:31:49 PM] [debug] saving controller to db
      [17/10/2013 11:31:49 PM] [debug] calculating controller checksum 0a8a3b98cebaf467af8fc6c6710650db
      [17/10/2013 11:31:49 PM] [debug] loading controller from db
      [17/10/2013 11:31:49 PM] [debug] setting controller status to 800
      [17/10/2013 11:31:49 PM] [error] backup_auto_failed_on_course 24 Integration (MySQL) 103
      [17/10/2013 11:31:49 PM] [error]   Exception: ddltablealreadyexists backup_ids_temp
      [17/10/2013 11:31:49 PM] [debug]   Debug: 
      

        Issue Links

          Activity

          Hide
          Tony Levi added a comment -

          Possible patch: https://github.com/tlevi/moodle/compare/MOODLE_23_STABLE...mdl38734_23

          I'm not sure this is the correct/full solution as ideally one should somehow just run failed backup cleanup task.

          Show
          Tony Levi added a comment - Possible patch: https://github.com/tlevi/moodle/compare/MOODLE_23_STABLE...mdl38734_23 I'm not sure this is the correct/full solution as ideally one should somehow just run failed backup cleanup task.
          Hide
          Michael de Raadt added a comment -

          Thanks for reporting that and adding a patch.

          Fred: I've assigned this to you as you have worked on related issues most recently.

          Show
          Michael de Raadt added a comment - Thanks for reporting that and adding a patch. Fred: I've assigned this to you as you have worked on related issues most recently.
          Hide
          Dan Poltawski added a comment -

          Thanks for the Patch Tony!

          Just trying to recreate the issue here, whats the sort of issue which an exception in the backup?

          Show
          Dan Poltawski added a comment - Thanks for the Patch Tony! Just trying to recreate the issue here, whats the sort of issue which an exception in the backup?
          Hide
          Dan Poltawski added a comment -

          Hi Tony,

          Do you have any examples of exceptions which would cause this problem so that I could reproduce it?

          Show
          Dan Poltawski added a comment - Hi Tony, Do you have any examples of exceptions which would cause this problem so that I could reproduce it?
          Hide
          Tony Levi added a comment -

          No, sorry, this was a while ago now. I think you can fake it by throwing from launch_automated_backup though, perhaps while it is constructing the controller (IIRC there is a try/catch for the actual work).

          Show
          Tony Levi added a comment - No, sorry, this was a while ago now. I think you can fake it by throwing from launch_automated_backup though, perhaps while it is constructing the controller (IIRC there is a try/catch for the actual work).
          Hide
          Michael de Raadt added a comment -

          I encountered this while testing another issue. I've added some detail to the description.

          Show
          Michael de Raadt added a comment - I encountered this while testing another issue. I've added some detail to the description.
          Hide
          Dan Poltawski added a comment -

          I've rebased tonys fix onto the supported branches.

          Just trying to setup a test with Michaels replication instructions now.

          Show
          Dan Poltawski added a comment - I've rebased tonys fix onto the supported branches. Just trying to setup a test with Michaels replication instructions now.
          Hide
          Dan Poltawski added a comment -

          We've now had a number of people verify that this fix works in these failing scenarios.

          You can reproduce it on stable branches with an oversize backup. On master, that problem has been fixed, so it'll need to be tested with a manual exception.

          Show
          Dan Poltawski added a comment - We've now had a number of people verify that this fix works in these failing scenarios. You can reproduce it on stable branches with an oversize backup. On master, that problem has been fixed, so it'll need to be tested with a manual exception.
          Hide
          Eloy Lafuente (stronk7) added a comment -

          The main moodle.git repository has just been updated with latest weekly modifications. You may wish to rebase your PULL branches to simplify history and avoid any possible merge conflicts. This would also make integrator's life easier next week.

          TIA and ciao

          Show
          Eloy Lafuente (stronk7) added a comment - The main moodle.git repository has just been updated with latest weekly modifications. You may wish to rebase your PULL branches to simplify history and avoid any possible merge conflicts. This would also make integrator's life easier next week. TIA and ciao
          Hide
          Damyon Wiese added a comment -

          Attaching patch that causes the first backup to throw an exception before cleanup.

          Show
          Damyon Wiese added a comment - Attaching patch that causes the first backup to throw an exception before cleanup.
          Hide
          Damyon Wiese added a comment -

          Thanks Dan + Tony.

          Integrated to 24, 25 and master.

          I attached a patch to make testing easier and tested this on 24, 25 and master already.

          Show
          Damyon Wiese added a comment - Thanks Dan + Tony. Integrated to 24, 25 and master. I attached a patch to make testing easier and tested this on 24, 25 and master already.
          Hide
          Damyon Wiese added a comment -

          Testing passed on all branches.

          Show
          Damyon Wiese added a comment - Testing passed on all branches.
          Hide
          Dan Poltawski added a comment -

          Congratulations - this issue has been included in Moodle and is now available on our git mirrors and shortly will become available on the download servers shortly.

          Show
          Dan Poltawski added a comment - Congratulations - this issue has been included in Moodle and is now available on our git mirrors and shortly will become available on the download servers shortly.

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: