Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-82584

Workshop get_users_with_capability_sql method optimization

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • 4.3.5, 4.4, 4.5
    • Workshop
    • Any
    • MOODLE_403_STABLE, MOODLE_404_STABLE, MOODLE_405_STABLE
    • Hide
      1. Run the maketestcourse to make a XL course (50000 users):
            `php /admin/tool/generator/maketestcourse.php --size=XL --shortname=XL --fullname=XL`
      2. On the course, create a Grouping called "all"
      3. Auto-Create Groups, with 1000 users per group, and set them all to be assigned to Grouping "all"
      4. Create a workshop activity and under Common Module Settings, set Group Mode to "Separate Groups" with Grouping "all"
      5. Attempt to view the workshop activity or submissions allocation page.
      6. Compare page load times with and without patch

       

      Bench-marks:

      (Purged cache before each test)

      Pre-patch:

      • Workshop view page: (25s)
      • Workshop submissions allocation page [10 users per page]: (45s)

      With-patch:

      • Workshop view page: (9s)
      • Workshop submissions allocation page [10 users per page]: (16s)

       

      Screenshots:

      pre-patch-view-page

      pre-patch-allocation-page

      with-patch-view-page

      with-patch-allocation-page

       

      Show
      Run the maketestcourse to make a XL course (50000 users):     `php /admin/tool/generator/maketestcourse.php --size=XL --shortname=XL --fullname=XL` On the course, create a Grouping called "all" Auto-Create Groups, with 1000 users per group, and set them all to be assigned to Grouping "all" Create a workshop activity and under Common Module Settings, set Group Mode to "Separate Groups" with Grouping "all" Attempt to view the workshop activity or submissions allocation page. Compare page load times with and without patch   Bench-marks: (Purged cache before each test) Pre-patch: Workshop view page: (25s) Workshop submissions allocation page  [10 users per page] : (45s) With-patch: Workshop view page: (9s) Workshop submissions allocation page  [10 users per page] : (16s)   Screenshots: pre-patch-view-page pre-patch-allocation-page with-patch-view-page with-patch-allocation-page  

      The workshop activity module has a performance issue related to its method, get_users_with_capability_sql, which generates a poorly optimized SQL query. Currently, it uses a UNION to combine SQL queries, but this could be improved by implementing simple caching and avoiding complex UNION queries.

      For instance, when there are 1000 groups, loading content becomes significantly slow and may even result in a 504 error due to timeout. I have attached a text file containing a SQL query which spans 15,000 lines, along with an optimized version of the SQL. Initially, I encountered a different problem that I couldn't replicate locally, but this example is the closest scenario I've managed to recreate the issue.

      Edit (Conn Warwicker)

      I have taken this ticket from Vlad now as he had to work on other things.

      Here's what I've done:

      • Removed the calls to some of the core functions to build up the SQL, as this was creating huge SQL statements, e.g. 80,000+ lines long in some cases, with endless UNIONS and other stuff which simply wasn't needed.
      • Tried to streamline it all into 1 simple SQL query custom to the workshhop module
      • I did initially play around with doing the bare minimum in SQL and then running users through `has_capability()` but it causes memory errors on very large courses.

       

      Bench-marks:

      (Purged cache before each test)

      Pre-patch:

      • Workshop view page: (25s)
      • Workshop submissions allocation page [10 users per page]: (45s)

      With-patch:

      • Workshop view page: (9s)
      • Workshop submissions allocation page [10 users per page]: (16s)

       

      Screenshots:

      pre-patch-view-page

      pre-patch-allocation-page

      with-patch-view-page

      with-patch-allocation-page

       

       

        1. pre-patch-allocation-page
          111 kB
        2. pre-patch-view-page
          116 kB
        3. sql_after_improvement.txt
          113 kB
        4. sql_before_improvement.txt
          1.14 MB
        5. with-patch-allocation-page
          132 kB
        6. with-patch-view-page
          189 kB

            cwarwicker C Warwicker
            vladaskidanovas vladaskidanovas
            Mark Johnson Mark Johnson
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.