Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-76129

Improve upgrade/install performance relating to capabilities and settings

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Minor Minor
    • 4.2
    • 4.0.4, 4.0.5
    • Libraries
    • MOODLE_400_STABLE
    • MOODLE_402_STABLE
    • MDL-76129-master
    • Hide

      It's not easy to test the performance improvement (although feel free to do before/after timing if you like, see comments for some things I timed) but I suggest the following test steps to make sure install/upgrade still works.

      This test assumes that your test server doesn't have any important data in the BigBlueButton module, and you don't mind deleting all of it. If that isn't the case, you might want to rework the test script to use another module. Any module that clones permissions (i.e. where the 'access.php' includes at least one capability, usually mod/whatever:addinstance, with a 'clonepermissionsfrom' entry) is OK.

      1. On a test course, go to Participants / Permissions, type manageactivities in the Filter box. Click the + icon next to Roles with permission and add the Student role.
      2. Uninstall the BigBlueButton module by going to Site administration / Plugins / Plugins overview and clicking the Uninstall link. Confirm the pop-up prompt, but stop (don't continue) when you get to the second prompt that asks Do you want to remove the plugin folder now?.
      3. Reinstall the module either by visiting the admin page in a web browser (you can click the Cancel button to get there), or using the CLI install script (it doesn't matter which). If you use the web browser, continue through all screens until the module is reinstalled. On the screen with new settings, you can just scroll to the bottom and click Save changes without changing anything.
        • EXPECTED: Install process should be successful.
      4. Go back to the Participants / Permissions page for the course, set Filter to bigbluebutton and look at the top capability Add a new BigBlueButton room.
        • EXPECTED: The capability should appear (i.e. it was added back successfully by the install).
        • EXPECTED: The Student role should have that capability (i.e. the required logic about cloning that value from the manage activities capability should have worked).
      Show
      It's not easy to test the performance improvement (although feel free to do before/after timing if you like, see comments for some things I timed) but I suggest the following test steps to make sure install/upgrade still works. This test assumes that your test server doesn't have any important data in the BigBlueButton module, and you don't mind deleting all of it. If that isn't the case, you might want to rework the test script to use another module. Any module that clones permissions (i.e. where the 'access.php' includes at least one capability, usually mod/whatever:addinstance, with a 'clonepermissionsfrom' entry) is OK. On a test course, go to Participants / Permissions , type manageactivities in the Filter box. Click the + icon next to Roles with permission and add the Student role. Uninstall the BigBlueButton module by going to Site administration / Plugins / Plugins overview and clicking the Uninstall link. Confirm the pop-up prompt, but stop (don't continue) when you get to the second prompt that asks Do you want to remove the plugin folder now? . Reinstall the module either by visiting the admin page in a web browser (you can click the Cancel button to get there), or using the CLI install script (it doesn't matter which). If you use the web browser, continue through all screens until the module is reinstalled. On the screen with new settings, you can just scroll to the bottom and click Save changes without changing anything. EXPECTED: Install process should be successful. Go back to the Participants / Permissions page for the course, set Filter to bigbluebutton and look at the top capability Add a new BigBlueButton room . EXPECTED: The capability should appear (i.e. it was added back successfully by the install). EXPECTED: The Student role should have that capability (i.e. the required logic about cloning that value from the manage activities capability should have worked).

      The update_capabilities function in accesslib is called when installing a new plugin or updating an existing one. Performance seems quite poor. (I detected this as a result of install logging added in MDL-75751.)

      It is a bit slower than it ought to be in standard Moodle, but in OU Moodle we noticed it takes almost 17 minutes to run this function when installing mod_bigbluebuttonbn, which seems quite ridiculous (it doesn't even have a particularly large number of capabilities).

      In addition, the admin_apply_default_settings function (it's not timed in MDL-75751, but I noticed from getting annoyed having to wait for it at the end of each run when testing other changes) also seems slower than necessary.

      After investigation, I discovered there are three reasons for this, which are all addressed in separate commits within this issue:

      1) Adding new capabilities is inefficient when there is more than one capability because it clears the capability cache after adding each capability to the database, then rebuilds the cache each time. I have changed it to add all capabilities to the database and then clear it once only.

      2) Adding new capabilities that clone the value of an existing capability can be slow if there are lots of places where the existing capability is set/overridden. It's possible to roughly halve the number of queries required for this if we extend a database query and add a way to tell assign_capability that it doesn't have to bother checking some details.

      3) (Related to point 1) During install/upgrade, caching is disabled. This causes disastrously poor performance in update_capabilities and quite poor performance in admin_apply_default_settings, which both appear to heavily rely on caches. I made a way to temporarily enable in-memory caching.

      Together, changes 1 and 3 improve performance of a fresh Moodle install (phpunit init) on my Windows developer laptop by about 60% (13 minutes -> 5 minutes).

      I anticipate that these fixes will also significantly improve the performance of upgrades like the OU's Moodle 4 upgrade, removing that 17 minute delay (I can't test that scenario, but I can't see anything else wrong with it now).

      Note: We are are not actually relying on this change for our Moodle 4 upgrade, because it might not be ready in time even if we backport. Instead, we are going to manually insert capabilities into the database just before the upgrade, which should avoid the long delay. However, having done the investigation, it seems like this could be a big advantage for core Moodle.

      The third point above is probably most controversial. There are good reasons to disable caching during upgrade/install:

      • Upgrade/install can work even if cache infrastructure has problems.
      • There may be drastic database changes in areas that are normally stable, and using cached data could result in inconsisten behaviour.
      • If we did in-memory caching throughout large parts of upgrade/install then it would probably occupy a lot of memory.

      To deal with all these issues, my solution is to enable caching only very temporarily (during the run-time of one function) and discard caches afterwards. Caching is not enabled during any of the 'API' parts of upgrade/install, or the other parts that make database structure changes (upgrade.php/install.php/install.xml). I hope this is a safe solution.

            quen Sam Marshall
            quen Sam Marshall
            Tim Hunt Tim Hunt
            Ilya Tregubov Ilya Tregubov
            Kim Jared Lucas Kim Jared Lucas
            Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 2 hours, 45 minutes
                2h 45m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.