We are running larger moodle 3.5.2 site, 11000 users concentrated in about 10 courses. All active mostly before noon. We were experiencing long assignment loading times - 30 second up to 2 minutes and timeout error. Database write errors. Very high CPU load (even 60 on 16 vCPU machine), we even had 700 apache workers at spike. Adding more hardware resources did not help and problems occurred even off the traffic spike.

What was going on:

After saving grades by teacher, regrading for single user is triggered. But that fails and regrading for every course user is enforced. That regrading is done simultaneously on server by every course user accessing some assignment page until regrading is done and saved to database. Regrading takes at least 30 seconds. This also explains database write errors (duplicate keys), as regrading does not use locking.

The bug:

I have found problem in function grade_regrade_final_grades. At certain grade item order, grade processing iteration fails as no successful update ($count+) was marked event when item was added to $finalids. But at least one more iteration had to be done to correctly process all grade items. I have added $count+ at this place https://github.com/moodle/moodle/blob/76528a767ecf6fed2c155c0825979d4898bc0da2/lib/gradelib.php#L1249 . Grade saving and following form reload is instant (also took at least 30 seconds to regrade everything). I have run unit tests and behat tests (limited to @core_grade), they were all OK.

We have this fix in production for over a week without complaints. Server load is below 2. We have typically 100 apache workers in spike.