We have a course archiving process that moves course that have finished running to an archive category (which restricts the ability of users to make changes to the courses) at the end of an academic year. This year it seems to have caused Moodle to become unusable while it was moving the courses, in previous years we have not noticed any slowdowns during the process.

The scheduled task doing this was moving the courses in batches of 2500 every even hour, each batch was taking about 20 minutes.

Looking at some of the slow traces we could see that there was a file lock in affect for over two minutes in most of them while the page was trying to get the category tree.

My conclusion is that it is caused by the changes in MDL-67674, because we are moving so many courses so quickly I think that the cache entries are constantly being invalided and since only one user can now rebuild the cache entry it meant that most users where constantly waiting for the lock so that they could have their go at generating the entry that would then be almost immediately invalid again.

The slowdowns lined up with when the task was scheduled to run:

Memecache writes were rather high:

A typical slow trace:

My guess is that in pervious years we would have had more activity on our database when doing this as users would have been able to regenerate cache entries in parallel (and possibly even more writing to our cache server; but end users were less affected by that.