Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-77202

Query performance - bump_submission_for_stale_conversions

XMLWordPrintable

    • MOODLE_311_STABLE, MOODLE_400_STABLE, MOODLE_401_STABLE
    • MDL-77202-MOODLE_401_STABLE
    • MDL-77202-MOODLE_master
    • Hide

       

      Before Patch

      1. Upgrade a test Moodle site that is less than 3.11 to the latest version of 3.11, 4.0, 4.1 or the latest master release.
        1. This release should have a large mdl_files table for testing. 
      2. Evaluate the explain plan on the query to see that it matches closely to the explain plain reported in the initial issue.
      3. If using MySQL as a back end, the explain plan for the query can be generated as:

      EXPLAIN SELECT f3.id, f3.timemodified as fmodified, asu.id as submissionid
                            FROM mdl_files f1
                       LEFT JOIN mdl_files f2 ON f1.contenthash = f2.filename
                                 AND f2.component = 'core' AND f2.filearea = 'documentconversion'
                            JOIN mdl_assign_submission asu ON asu.id = f1.itemid
                            JOIN mdl_assign_grades asg ON asg.userid = asu.userid AND asg.assignment = asu.assignment
                            JOIN mdl_files f3 ON f3.itemid = asg.id
                           WHERE f1.filearea = 'submission_files'
                                 AND f3.timecreated >= '1533759980'
                                 AND (f1.filename LIKE '%.doc' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.docx' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.rtf' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.xls' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.xlsx' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.ppt' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.pptx' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.html' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.odt' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.ods' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.png' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.jpg' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.txt' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.gif' COLLATE utf8mb4_bin ESCAPE '\\')
                                 AND f2.filename IS NULL
                                 AND f3.component = 'assignfeedback_editpdf'
                                 AND f3.filearea = 'combined'
                                 AND f3.filename = 'combined.pdf'
                                 AND f3.timemodified >= asu.timemodified;

      # Run the query above to record the results and ensure documents are listed for processing.

      1. Process the adhoc_task.php file to have the `bump_submission_for_stale_conversions` task execute and note the time taken to complete the task.

      After Patch

      1. Upgrade a test Moodle site that is less than 3.11 to the latest version of 3.11, 4.0, 4.1 or the latest master release and apply the corresponding version patch.
        1. This release should have a large mdl_files table for testing. 
      2. Evaluate the explain plan on the query to see that it now has significantly less row evaluation.
      3. If using MySQL as a back end, the explain plan for the query can be generated as:

      EXPLAIN 
      SELECT f3.id, f3.timemodified as fmodified, asu.id as submissionid
        FROM mdl_files f1
       LEFT JOIN mdl_files f2 ON f1.contenthash = f2.filename
         AND f2.component = 'core' AND f2.filearea = 'documentconversion'
        JOIN mdl_assign_submission asu ON asu.id = f1.itemid
        JOIN mdl_assign_grades asg ON asg.userid = asu.userid AND asg.assignment = asu.assignment
        JOIN mdl_assign ma ON ma.id = asg.assignment
        JOIN mdl_context ctx ON ctx.instanceid = ma.id and ctx.contextlevel=70
        JOIN mdl_files f3 ON (f3.component = 'assignfeedback_editpdf' AND f3.filearea = 'combined' AND f3.contextid = ctx.id AND f3.itemid = asg.id) # Fix here: get the context id for the join
       WHERE (f1.filearea = 'submission_files' and f1.component = 'assignsubmission_file')
       AND f3.timecreated >= '1533759980'
             AND (f1.filename LIKE '%.doc' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.docx' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.rtf' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.xls' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.xlsx' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.ppt' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.pptx' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.html' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.odt' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.ods' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.png' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.jpg' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.txt' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.gif' COLLATE utf8mb4_bin ESCAPE '\\')
             AND f2.filename IS NULL
             AND f3.filename = 'combined.pdf'
             AND f3.timemodified >= asu.timemodified;
      

      # Run the query above to record the results and ensure documents are listed for processing.

      1. Process the adhoc_task.php file to have the `bump_submission_for_stale_conversions` task execute and note the time taken to complete the task.

      The explain plan should look similar to: 

      +------+-------------+-------+--------+------------------------------------------------------------------------------------------------------------+----------------------------+---------+-----------------------------------------------------+---------+------------------------------------+
      | id   | select_type | table | type   | possible_keys                                                                                              | key                        | key_len | ref                                                 | rows    | Extra                              |
      +------+-------------+-------+--------+------------------------------------------------------------------------------------------------------------+----------------------------+---------+-----------------------------------------------------+---------+------------------------------------+
      |    1 | SIMPLE      | f1    | ref    | mdl_file_comfilconite_ix                                                                                   | mdl_file_comfilconite_ix   | 604     | const,const                                         | 6740488 | Using index condition; Using where |
      |    1 | SIMPLE      | asu   | eq_ref | PRIMARY,mdl_assisubm_assusegroatt_uix,mdl_assisubm_use_ix,mdl_assisubm_ass_ix,mdl_assisubm_assusegrolat_ix | PRIMARY                    | 8       | moodle_customer.f1.itemid                              |       1 |                                    |
      |    1 | SIMPLE      | ctx   | eq_ref | PRIMARY,mdl_cont_conins_uix,mdl_cont_ins_ix                                                                | mdl_cont_conins_uix        | 16      | const,moodle_customer.asu.assignment                   |       1 | Using index                        |
      |    1 | SIMPLE      | asg   | ref    | PRIMARY,mdl_assigrad_assuseatt_uix,mdl_assigrad_use_ix,mdl_assigrad_ass_ix                                 | mdl_assigrad_assuseatt_uix | 16      | moodle_customer.asu.assignment,moodle_customer.asu.userid |       1 | Using index                        |
      |    1 | SIMPLE      | ma    | eq_ref | PRIMARY                                                                                                    | PRIMARY                    | 8       | moodle_customer.asu.assignment                         |       1 | Using index                        |
      |    1 | SIMPLE      | f3    | ref    | mdl_file_comfilconite_ix,mdl_file_con2_ix                                                                  | mdl_file_comfilconite_ix   | 620     | const,const,moodle_customer.ctx.id,moodle_customer.asg.id |       1 | Using index condition; Using where |
      |    1 | SIMPLE      | f2    | ref    | mdl_file_comfilconite_ix                                                                                   | mdl_file_comfilconite_ix   | 604     | const,const                                         |  899570 | Using where; Not exists            |
      +------+-------------+-------+--------+------------------------------------------------------------------------------------------------------------+----------------------------+---------+-----------------------------------------------------+---------+------------------------------------+

      Show
        Before Patch Upgrade a test Moodle site that is less than 3.11 to the latest version of 3.11, 4.0, 4.1 or the latest master release. This release should have a large mdl_files table for testing.  Evaluate the explain plan on the query to see that it matches closely to the explain plain reported in the initial issue. If using MySQL as a back end, the explain plan for the query can be generated as: EXPLAIN SELECT f3.id, f3.timemodified as fmodified, asu.id as submissionid                       FROM mdl_files f1                  LEFT JOIN mdl_files f2 ON f1.contenthash = f2.filename                            AND f2.component = 'core' AND f2.filearea = 'documentconversion'                       JOIN mdl_assign_submission asu ON asu.id = f1.itemid                       JOIN mdl_assign_grades asg ON asg.userid = asu.userid AND asg.assignment = asu.assignment                       JOIN mdl_files f3 ON f3.itemid = asg.id                      WHERE f1.filearea = 'submission_files'                            AND f3.timecreated >= '1533759980'                            AND (f1.filename LIKE '%.doc' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.docx' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.rtf' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.xls' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.xlsx' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.ppt' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.pptx' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.html' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.odt' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.ods' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.png' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.jpg' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.txt' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.gif' COLLATE utf8mb4_bin ESCAPE '\\' )                            AND f2.filename IS NULL                            AND f3.component = 'assignfeedback_editpdf'                            AND f3.filearea = 'combined'                            AND f3.filename = 'combined.pdf'                            AND f3.timemodified >= asu.timemodified; # Run the query above to record the results and ensure documents are listed for processing. Process the adhoc_task.php file to have the `bump_submission_for_stale_conversions` task execute and note the time taken to complete the task. After Patch Upgrade a test Moodle site that is less than 3.11 to the latest version of 3.11, 4.0, 4.1 or the latest master release and apply the corresponding version patch. This release should have a large mdl_files table for testing.  Evaluate the explain plan on the query to see that it now has significantly less row evaluation. If using MySQL as a back end, the explain plan for the query can be generated as: EXPLAIN SELECT f3.id, f3.timemodified as fmodified, asu.id as submissionid FROM mdl_files f1 LEFT JOIN mdl_files f2 ON f1.contenthash = f2.filename AND f2.component = 'core' AND f2.filearea = 'documentconversion' JOIN mdl_assign_submission asu ON asu.id = f1.itemid JOIN mdl_assign_grades asg ON asg.userid = asu.userid AND asg.assignment = asu.assignment JOIN mdl_assign ma ON ma.id = asg.assignment JOIN mdl_context ctx ON ctx.instanceid = ma.id and ctx.contextlevel= 70 JOIN mdl_files f3 ON (f3.component = 'assignfeedback_editpdf' AND f3.filearea = 'combined' AND f3.contextid = ctx.id AND f3.itemid = asg.id) # Fix here: get the context id for the join WHERE (f1.filearea = 'submission_files' and f1.component = 'assignsubmission_file' ) AND f3.timecreated >= '1533759980' AND (f1.filename LIKE '%.doc' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.docx' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.rtf' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.xls' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.xlsx' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.ppt' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.pptx' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.html' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.odt' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.ods' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.png' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.jpg' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.txt' COLLATE utf8mb4_bin ESCAPE '\\' OR f1.filename LIKE '%.gif' COLLATE utf8mb4_bin ESCAPE '\\' ) AND f2.filename IS NULL AND f3.filename = 'combined.pdf' AND f3.timemodified >= asu.timemodified; # Run the query above to record the results and ensure documents are listed for processing. Process the adhoc_task.php file to have the `bump_submission_for_stale_conversions` task execute and note the time taken to complete the task. The explain plan should look similar to:  +------+-------------+-------+--------+------------------------------------------------------------------------------------------------------------+----------------------------+---------+-----------------------------------------------------+---------+------------------------------------+ | id   | select_type | table | type   | possible_keys                                                                                              | key                        | key_len | ref                                                 | rows    | Extra                              | +------+-------------+-------+--------+------------------------------------------------------------------------------------------------------------+----------------------------+---------+-----------------------------------------------------+---------+------------------------------------+ |     1 | SIMPLE      | f1    | ref    | mdl_file_comfilconite_ix                                                                                   | mdl_file_comfilconite_ix   | 604     | const , const                                         | 6740488 | Using index condition; Using where | |     1 | SIMPLE      | asu   | eq_ref | PRIMARY,mdl_assisubm_assusegroatt_uix,mdl_assisubm_use_ix,mdl_assisubm_ass_ix,mdl_assisubm_assusegrolat_ix | PRIMARY                    | 8       | moodle_customer.f1.itemid                              |       1 |                                    | |     1 | SIMPLE      | ctx   | eq_ref | PRIMARY,mdl_cont_conins_uix,mdl_cont_ins_ix                                                                | mdl_cont_conins_uix        | 16      | const ,moodle_customer.asu.assignment                   |       1 | Using index                        | |     1 | SIMPLE      | asg   | ref    | PRIMARY,mdl_assigrad_assuseatt_uix,mdl_assigrad_use_ix,mdl_assigrad_ass_ix                                 | mdl_assigrad_assuseatt_uix | 16      | moodle_customer.asu.assignment,moodle_customer.asu.userid |       1 | Using index                        | |     1 | SIMPLE      | ma    | eq_ref | PRIMARY                                                                                                    | PRIMARY                    | 8       | moodle_customer.asu.assignment                         |       1 | Using index                        | |     1 | SIMPLE      | f3    | ref    | mdl_file_comfilconite_ix,mdl_file_con2_ix                                                                  | mdl_file_comfilconite_ix   | 620     | const , const ,moodle_customer.ctx.id,moodle_customer.asg.id |       1 | Using index condition; Using where | |     1 | SIMPLE      | f2    | ref    | mdl_file_comfilconite_ix                                                                                   | mdl_file_comfilconite_ix   | 604     | const , const                                         |   899570 | Using where; Not exists            | +------+-------------+-------+--------+------------------------------------------------------------------------------------------------------------+----------------------------+---------+-----------------------------------------------------+---------+------------------------------------+

      After upgrading a Moodle site >= 3.11.2 an adhoc task is created for the assignfeedback_editpdf plugin to bump stale submissions in the files table to ensure documents are converted. Several joins in the query used to gather these submissions are joined off index resulting in full table scans and joins resulting in huge search sets. 

      For one of our customers we can see the explain plan evaluating several million rows at each stage of the query:

      +------+-------------+-------+--------+------------------------------------------------------------------------------------------------------------+----------------------------+---------+-----------------------------------------------------+----------+------------------------------------+
      | id   | select_type | table | type   | possible_keys                                                                                              | key                        | key_len | ref                                                 | rows     | Extra                              |
      +------+-------------+-------+--------+------------------------------------------------------------------------------------------------------------+----------------------------+---------+-----------------------------------------------------+----------+------------------------------------+
      |    1 | SIMPLE      | f1    | ALL    | NULL                                                                                                       | NULL                       | NULL    | NULL                                                | 20923425 | Using where                        |
      |    1 | SIMPLE      | asu   | eq_ref | PRIMARY,mdl_assisubm_assusegroatt_uix,mdl_assisubm_use_ix,mdl_assisubm_ass_ix,mdl_assisubm_assusegrolat_ix | PRIMARY                    | 8       | moodle_customer.f1.itemid                              |        1 |                                    |
      |    1 | SIMPLE      | asg   | ref    | PRIMARY,mdl_assigrad_assuseatt_uix,mdl_assigrad_use_ix,mdl_assigrad_ass_ix                                 | mdl_assigrad_assuseatt_uix | 16      | moodle_customer.asu.assignment,moodle_customer.asu.userid |        1 | Using index                        |
      |    1 | SIMPLE      | f2    | ref    | mdl_file_comfilconite_ix                                                                                   | mdl_file_comfilconite_ix   | 604     | const,const                                         |   891968 | Using where; Not exists            |
      |    1 | SIMPLE      | f3    | ref    | mdl_file_comfilconite_ix                                                                                   | mdl_file_comfilconite_ix   | 604     | const,const                                         |  2960338 | Using index condition; Using where |
      +------+-------------+-------+--------+------------------------------------------------------------------------------------------------------------+----------------------------+---------+-----------------------------------------------------+----------+------------------------------------+

      We're finding that these tasks never complete, and just start running on top of each other due to the extreme processing time required by the database to gather this result set. 

      I believe I have an updated query that pulls some additional information from the context table allowing joins to be more performant. 

            Unassigned Unassigned
            inkjet2000 Justin Merrill
            Votes:
            25 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.