Discovered while testing
When user is enrolled or unenrolled from the course the tag cache is not invalidated which causes tagged items to be not displayed or displayed. This lasts for up to 15 minutes (cache ttl). Can be resolved by logout and login faster.
tagindexbuilder is the session cache that should be invalidated when somebody is enrolled/unenrolled but I can't find any invalidation requests in core. They must have been accidentally removed at some point.
If cache access is removed, everything works without delay:
I will check which stable versions are affected and update