Moodle

function update_course_icon() does not scale

Details

  • Affected Branches:
    MOODLE_18_STABLE
  • Fixed Branches:
    MOODLE_18_STABLE, MOODLE_19_STABLE

Description

The function update_course_icon() in lib/weblib.php prints the course Turn editing on/off button. If a user does not have moodle/course:manageactivities or moodle/site:manageblocks at the course level, it then loops through every module, block and group instance in the course to check to see if the user has either of those capabilities in those contexts. This is where we run into performance issues because this section of code does not scale well. On top of that, after all that processing, the feature does not work. Example: give a student type user the admin role in the context of a block. The Turn editing on/off button will print, but you can never turn editing On.

Performance information:

Each child context that is checked generates roughly 11 database hits and this number increases if the course is nested below more than one category. Here is a breakdown:

1. update_course_icon() calls lib/accesslib.php:function get_child_contexts(). This method calls get_context_instance() for every block, module and group instance resulting in 1 db hit per instance. Instead of doing these calls individually, try to perform a batch process. Suggestion: new function get_context_instances(). This method could process an array and check the cache for any of the contexts and then make one query to get rest. It then caches result and returns an array of contexts. This would then do some looping and one db hit to get all child contexts instead of one db hit per child context.

2. in lib/weblib.php:function update_course_icon():
Replace:
$childcontext = get_record('context', 'id', $child);
With:
$childcontext = get_context_instance_by_id($child);

By using the API, we take advantage of the cache that was generated with the call to get_child_contexts(). This would reduce 1 db call per child context.

3. in lib/weblib.php:function update_course_icon():
Replace:
if (has_capability('moodle/course:manageactivities', $childcontext) ||
has_capability('moodle/site:manageblocks', $childcontext)) {
With:
if (has_capability('moodle/course:manageactivities', $childcontext, $USER->id, false) ||
has_capability('moodle/site:manageblocks', $childcontext, $USER->id, false)) {

If user had doanything, then s/he wouldn't be here in first place (I think). By turning off doanything, we reduce the db hits by 3 per child context.

4. in lib/accesslib.php:capability_search() - This method is called twice for every child context and generates 3 queries per call. This code should be evaluated for optimization and I don't have any suggestions at this time.

5. Lastly, the result of this (print edit button or not) should be saved in session so we don't have to do this again on the next page load.

Here are the db hit counts in the course I was testing this on:

  • With core code: 21,930 database hits to process all the child contexts
  • With code fixes 2 and 3 from above, database hits go down to 10,982.
  • With code fixes 1, 2 and 3 hits would go down to 9,418
  • Note: if you were to turn on db record cache at this point, the 9,418 would go down to approximately 1,581.

The majority of the remainder of the db hits (9,418) are coming from capability_search(). If this were to be optimized, then we could theoretically bring the db hits down to 100 or so total for the course page load (I think 60 or 80 db hits is normal for a course page load when not processing these child contexts).

That's all I have for now, good luck

  1. update_course_icon3.patch
    25/Aug/07 12:25 AM
    8 kB
    Petr Škoda (skodak)
  2. update_course_icon4.patch
    25/Aug/07 7:26 PM
    9 kB
    Petr Škoda (skodak)
  3. weblib_2.diff
    02/Aug/07 6:31 AM
    1 kB
    Mark Nielsen
  4. weblib.diff
    28/Jul/07 7:51 AM
    1 kB
    Mark Nielsen

Issue Links

Activity

Hide
Mark Nielsen added a comment -

Here is a diff off CVS MOODLE_18_STABLE with the 2 and 3 code fixes.

Show
Mark Nielsen added a comment - Here is a diff off CVS MOODLE_18_STABLE with the 2 and 3 code fixes.
Hide
Mark Nielsen added a comment -

Uploading a new diff of lib/weblib.php. I didn't copy/paste correctly with the first one, d'oh!

Show
Mark Nielsen added a comment - Uploading a new diff of lib/weblib.php. I didn't copy/paste correctly with the first one, d'oh!
Hide
Michael Penney added a comment -

It is going to be really hard on large sites with lots of content if this doesn't get fixed.

Show
Michael Penney added a comment - It is going to be really hard on large sites with lots of content if this doesn't get fixed.
Hide
Martin Dougiamas added a comment -

Thanks, looks like some very good ideas here.

Petr, can you please implement as much as possible in 1.9 and 1.8?

Show
Martin Dougiamas added a comment - Thanks, looks like some very good ideas here. Petr, can you please implement as much as possible in 1.9 and 1.8?
Hide
Petr Škoda (skodak) added a comment -

Sending a different patch:
1/ new function has_capability_in_child_contexts() that works mostly with already existing data in $USER->capabilities
2/ the result is cached in $USER->editenabled[$courseid] session variable; if loginas or switchrole actived the cache is not used

AFAIK Martín Langhoff is already working on major performance improvements for roles, I hope this will speed up everything substantially.

My patch is very small and I think it should be safe to commit into 1.8 - it should not collide with Martín's current work.

Show
Petr Škoda (skodak) added a comment - Sending a different patch: 1/ new function has_capability_in_child_contexts() that works mostly with already existing data in $USER->capabilities 2/ the result is cached in $USER->editenabled[$courseid] session variable; if loginas or switchrole actived the cache is not used AFAIK Martín Langhoff is already working on major performance improvements for roles, I hope this will speed up everything substantially. My patch is very small and I think it should be safe to commit into 1.8 - it should not collide with Martín's current work.
Hide
Petr Škoda (skodak) added a comment -

sending new patch, I have missed the code in pagelib.php - the reduced number of db queries should be noticeble finally

Show
Petr Škoda (skodak) added a comment - sending new patch, I have missed the code in pagelib.php - the reduced number of db queries should be noticeble finally
Hide
Petr Škoda (skodak) added a comment -

grrrr - removing the patch again - I have to fix some more places...

Show
Petr Škoda (skodak) added a comment - grrrr - removing the patch again - I have to fix some more places...
Hide
Petr Škoda (skodak) added a comment -

here it is

Show
Petr Škoda (skodak) added a comment - here it is
Hide
Petr Škoda (skodak) added a comment -

Could anybody please test it on some really large site?

Show
Petr Škoda (skodak) added a comment - Could anybody please test it on some really large site?
Hide
Mark Nielsen added a comment -

Attempting to test it now

Show
Mark Nielsen added a comment - Attempting to test it now
Hide
Mark Nielsen added a comment -

I had troubles applying the patch, so had to do it manually. I applied it to our 1.8.2+ checkout so the test was not performed on the latest 1.8 branch of code, but the results were nice.

Before Patch:

  • DB queries: 22,094
  • Page load time: 9.54 sec

After Patch:

  • DB queries: 1,763
  • Page load time: 1.11 sec

After Patch and after a refresh to test cache:

  • DB queries: 195
  • Page load time: .55 sec

So, the cache works wonderfully. 195 db hits for that page is normal. The admin role sees 217 db hits for the same page, so I think this patch works performance wise. I'm not entirely sure if it works functionally as I did minor testing with this, but it did work for those tests.

Thanks for the great work Petr!

Show
Mark Nielsen added a comment - I had troubles applying the patch, so had to do it manually. I applied it to our 1.8.2+ checkout so the test was not performed on the latest 1.8 branch of code, but the results were nice. Before Patch:
  • DB queries: 22,094
  • Page load time: 9.54 sec
After Patch:
  • DB queries: 1,763
  • Page load time: 1.11 sec
After Patch and after a refresh to test cache:
  • DB queries: 195
  • Page load time: .55 sec
So, the cache works wonderfully. 195 db hits for that page is normal. The admin role sees 217 db hits for the same page, so I think this patch works performance wise. I'm not entirely sure if it works functionally as I did minor testing with this, but it did work for those tests. Thanks for the great work Petr!
Hide
Petr Škoda (skodak) added a comment -

Thanks Mark!
I am going to commit it to HEAD and after some more testing backport it into STABLE next week. The functionality should be the same as before, I have tested also loginas and switchrole...

The problem with blocks mentioned above is different, I think this will be fixed properly in 1.9

Show
Petr Škoda (skodak) added a comment - Thanks Mark! I am going to commit it to HEAD and after some more testing backport it into STABLE next week. The functionality should be the same as before, I have tested also loginas and switchrole... The problem with blocks mentioned above is different, I think this will be fixed properly in 1.9
Hide
Petr Škoda (skodak) added a comment -

sending latest version (with minor refactoring) and committing into HEAD

Show
Petr Škoda (skodak) added a comment - sending latest version (with minor refactoring) and committing into HEAD
Hide
Petr Škoda (skodak) added a comment -

I fixed an incorrect capability value check in has_capability_including_child_contetxt() - the value is a sum of caps in all parent contexts
committed into cvs

Show
Petr Škoda (skodak) added a comment - I fixed an incorrect capability value check in has_capability_including_child_contetxt() - the value is a sum of caps in all parent contexts committed into cvs
Hide
Mark Pearson added a comment -

I checked the CHANGES file for the latest stable build of 1.8.2+ on Aug 30th, together with the resolved bug list, but could find no trace of this bug or the patch having been applied to 1.8.2 latest stable build. I am reluctant to overwrite my production site from CVS having been burned earlier this year with a malfunctioned CVS download so I'm wondering when the patch is likely to become available in the stable build for 1.8.2+. My faculty are getting desperate :-> Ta.

Show
Mark Pearson added a comment - I checked the CHANGES file for the latest stable build of 1.8.2+ on Aug 30th, together with the resolved bug list, but could find no trace of this bug or the patch having been applied to 1.8.2 latest stable build. I am reluctant to overwrite my production site from CVS having been burned earlier this year with a malfunctioned CVS download so I'm wondering when the patch is likely to become available in the stable build for 1.8.2+. My faculty are getting desperate :-> Ta.
Hide
Martin Dougiamas added a comment -

Petr can you make this a priority to get into MOODLE_18_STABLE please? Thanks.

Show
Martin Dougiamas added a comment - Petr can you make this a priority to get into MOODLE_18_STABLE please? Thanks.
Hide
Petr Škoda (skodak) added a comment -

already working on that

Show
Petr Škoda (skodak) added a comment - already working on that
Hide
Petr Škoda (skodak) added a comment -

patch is now committed into 1.8.x too, the performance git branch by Martín already contains different solution for 1.9 which should work much better

Show
Petr Škoda (skodak) added a comment - patch is now committed into 1.8.x too, the performance git branch by Martín already contains different solution for 1.9 which should work much better
Hide
Mark Pearson added a comment -

Totally dead brill . Thanks chaps. I shall get the updated 1.8.2+ installed quam celerrime

Show
Mark Pearson added a comment - Totally dead brill . Thanks chaps. I shall get the updated 1.8.2+ installed quam celerrime

People

Dates

  • Created:
    Updated:
    Resolved: