Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-63761

Failed fast regrading resulting in extremely long load times

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.5.2
    • Fix Version/s: 3.5.4, 3.6.2
    • Component/s: Gradebook
    • Labels:
    • Testing Instructions:
      Hide

       

      1. Make a small sized test course (admin -> development -> make test course)

      2. Add a new assignment to topic 1 and call it 'myassign'

      3. Add a manual grade item called 'custom grade item' in the gradebook (grades -> setup tab -> scroll to bottom -> add grade item)

      5. Find the newly created 'custom grade item' in the table and select 'edit calculation' from the 'edit' menu.

      6. On the subsequent page, find myassign in the bottom section 'ID numbers'. 

      7. Type 'myassign' into the corresponding text field and hit enter.

      8. Now, in the calculation box at the top of the page, enter:

       =[[myassign1]] + 1

      (where myassign is the idnumber of the assignment grade).

      9. Go to the assignment and assign a grade of 50 to ten students. 

      10. Go to the gradebook and verify all 10 students have a grade for the assignment and the manual grade item. 

      Show
        1. Make a small sized test course (admin -> development -> make test course) 2. Add a new assignment to topic 1 and call it 'myassign' 3. Add a manual grade item called 'custom grade item' in the gradebook (grades -> setup tab -> scroll to bottom -> add grade item) 5. Find the newly created 'custom grade item' in the table and select 'edit calculation' from the 'edit' menu. 6. On the subsequent page, find myassign in the bottom section 'ID numbers'.  7. Type 'myassign' into the corresponding text field and hit enter. 8. Now, in the calculation box at the top of the page, enter: =[[myassign1]] + 1 (where myassign is the idnumber of the assignment grade). 9. Go to the assignment and assign a grade of 50 to ten students.  10. Go to the gradebook and verify all 10 students have a grade for the assignment and the manual grade item. 
    • Affected Branches:
      MOODLE_35_STABLE
    • Fixed Branches:
      MOODLE_35_STABLE, MOODLE_36_STABLE
    • Pull from Repository:
    • Pull 3.5 Branch:
    • Pull 3.6 Branch:
    • Pull Master Branch:
      MDL-63761-master

      Description

      We are running larger moodle 3.5.2 site, 11000 users concentrated in about 10 courses. All active mostly before noon. We were experiencing long assignment loading times - 30 second up to 2 minutes and timeout error. Database write errors. Very high CPU load (even 60 on 16 vCPU machine), we even had 700 apache workers at spike. Adding more hardware resources did not help and problems occurred even off the traffic spike.

      What was going on:

      After saving grades by teacher, regrading for single user is triggered. But that fails and regrading for every course user is enforced. That regrading is done simultaneously on server by every course user accessing some assignment page until regrading is done and saved to database. Regrading takes at least 30 seconds. This also explains database write errors (duplicate keys), as regrading does not use locking.

      The bug:

      I have found problem in function grade_regrade_final_grades. At certain grade item order, grade processing iteration fails as no successful update ($count+) was marked event when item was added to $finalids. But at least one more iteration had to be done to correctly process all grade items. I have added $count+ at this place https://github.com/moodle/moodle/blob/76528a767ecf6fed2c155c0825979d4898bc0da2/lib/gradelib.php#L1249 . Grade saving and following form reload is instant (also took at least 30 seconds to regrade everything). I have run unit tests and behat tests (limited to @core_grade), they were all OK.

      We have this fix in production for over a week without complaints. Server load is below 2. We have typically 100 apache workers in spike.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                1 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  14/Jan/19

                  Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 hour, 30 minutes
                  1h 30m