Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-23754 Performance improvements META
  3. MDL-27120

Backup / Restore / Import performance optimisations

    XMLWordPrintable

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.2, 2.3
    • Fix Version/s: 2.3
    • Component/s: Backup
    • Testing Instructions:
      Hide

      Test 1:

      1. Backup a course (Go to course and click backup), no error should be produced and course should be backed up.
      2. Restore same course and make sure course is restored properly.

      Test2:

      1. Backup multiple course. Open multiple/different browsers and backup different courses at the same time
      2. restore these courses at the same time in different browsers and make sure they get restored properly.

      Test 3:

      1. Set automated backup (Settings -> Site administrator -> Courses -> Backups -> Automated backup setup)
      2. Run cron (/admin/cron.php), no error should be produced.

      Test 4:
      Run phpunit and make sure you don't get any error (especially backup/converter/moodle1/tests/lib_test.php)

      Show
      Test 1: Backup a course (Go to course and click backup), no error should be produced and course should be backed up. Restore same course and make sure course is restored properly. Test2: Backup multiple course. Open multiple/different browsers and backup different courses at the same time restore these courses at the same time in different browsers and make sure they get restored properly. Test 3: Set automated backup (Settings -> Site administrator -> Courses -> Backups -> Automated backup setup) Run cron (/admin/cron.php), no error should be produced. Test 4: Run phpunit and make sure you don't get any error (especially backup/converter/moodle1/tests/lib_test.php)
    • Affected Branches:
      MOODLE_20_STABLE, MOODLE_23_STABLE
    • Fixed Branches:
      MOODLE_23_STABLE
    • Pull Master Branch:
      wip-mdl-27120

      Description

      Backup / restore is performing slowly which is causing many imports to fail for large courses.

      In addition to this, an interactive backup/restore/import does not increase max_execution_time, so max_execution_time will remain at PHP's default (which could be something like 60 secs). In Moodle 1.9, both backup and restore would set max execution time to 3000 (50 minutes).

      I have also identified 3 major bottlenecks so far in the backup code (see attached profiles for further evidence):
      1) base_plan::get_setting() function does a sequential scan lookup to find settings, instead of hash-based, which was very slow and called $setting->get_name() over 1 million times and taking ~30s (see get_setting.png)
      2) The various backup_ui_stage classes in backup/util/ui/backup_ui_stage.class.php each contain an initialise_stage_form() method which is using QuickForms in an inefficient way. It calls $form->add_setting() which calls to moodle_form->setDefault which calls QuickForm->setDefaults which fires an "updateValue" even to all quickform elements. So it has n*n complexity (see quickforms1.png and quickforms2.png)
      3) The backup_xml_transformer's constructor calls register_links_encoders() which causes an expensive call to get_plugins_list(). The problem is, the class can be constructed 1500+ times during a backup, so it needs a cache! (see getplugins1.png and getplugins2.png)

      I have patches for the above 3 issues which I will put on GitHub and link here.

        Attachments

        1. get_setting.png
          36 kB
          Ashley Holman
        2. getplugins1.png
          24 kB
          Ashley Holman
        3. getplugins2.png
          24 kB
          Ashley Holman
        4. quickforms1.png
          64 kB
          Ashley Holman
        5. quickforms2.png
          59 kB
          Ashley Holman

          Issue Links

            Activity

              People

              • Votes:
                10 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  25/Jun/12