Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-23754 Performance improvements META
  3. MDL-27120

Backup / Restore / Import performance optimisations

    XMLWordPrintable

Details

    • Sub-task
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.0.2, 2.3
    • 2.3
    • Backup
    • MOODLE_20_STABLE, MOODLE_23_STABLE
    • MOODLE_23_STABLE
    • wip-mdl-27120
    • Hide

      Test 1:

      1. Backup a course (Go to course and click backup), no error should be produced and course should be backed up.
      2. Restore same course and make sure course is restored properly.

      Test2:

      1. Backup multiple course. Open multiple/different browsers and backup different courses at the same time
      2. restore these courses at the same time in different browsers and make sure they get restored properly.

      Test 3:

      1. Set automated backup (Settings -> Site administrator -> Courses -> Backups -> Automated backup setup)
      2. Run cron (/admin/cron.php), no error should be produced.

      Test 4:
      Run phpunit and make sure you don't get any error (especially backup/converter/moodle1/tests/lib_test.php)

      Show
      Test 1: Backup a course (Go to course and click backup), no error should be produced and course should be backed up. Restore same course and make sure course is restored properly. Test2: Backup multiple course. Open multiple/different browsers and backup different courses at the same time restore these courses at the same time in different browsers and make sure they get restored properly. Test 3: Set automated backup (Settings -> Site administrator -> Courses -> Backups -> Automated backup setup) Run cron (/admin/cron.php), no error should be produced. Test 4: Run phpunit and make sure you don't get any error (especially backup/converter/moodle1/tests/lib_test.php)

    Description

      Backup / restore is performing slowly which is causing many imports to fail for large courses.

      In addition to this, an interactive backup/restore/import does not increase max_execution_time, so max_execution_time will remain at PHP's default (which could be something like 60 secs). In Moodle 1.9, both backup and restore would set max execution time to 3000 (50 minutes).

      I have also identified 3 major bottlenecks so far in the backup code (see attached profiles for further evidence):
      1) base_plan::get_setting() function does a sequential scan lookup to find settings, instead of hash-based, which was very slow and called $setting->get_name() over 1 million times and taking ~30s (see get_setting.png)
      2) The various backup_ui_stage classes in backup/util/ui/backup_ui_stage.class.php each contain an initialise_stage_form() method which is using QuickForms in an inefficient way. It calls $form->add_setting() which calls to moodle_form->setDefault which calls QuickForm->setDefaults which fires an "updateValue" even to all quickform elements. So it has n*n complexity (see quickforms1.png and quickforms2.png)
      3) The backup_xml_transformer's constructor calls register_links_encoders() which causes an expensive call to get_plugins_list(). The problem is, the class can be constructed 1500+ times during a backup, so it needs a cache! (see getplugins1.png and getplugins2.png)

      I have patches for the above 3 issues which I will put on GitHub and link here.

      Attachments

        1. get_setting.png
          get_setting.png
          36 kB
        2. getplugins1.png
          getplugins1.png
          24 kB
        3. getplugins2.png
          getplugins2.png
          24 kB
        4. quickforms1.png
          quickforms1.png
          64 kB
        5. quickforms2.png
          quickforms2.png
          59 kB

        Issue Links

          Activity

            People

              rajeshtaneja Rajesh Taneja
              ashleyholman Ashley Holman
              Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
              Aparup Banerjee Aparup Banerjee
              Aparup Banerjee Aparup Banerjee
              Adrian Greeve, David Woloszyn, Huong Nguyen, Jake Dallimore, Michael Hawkins, Stevani Andolo
              Votes:
              10 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                25/Jun/12