I just detected a pretty worrying problem with a file:

All the data in the database is consistent. There is a row in mdl_files saying this file exists, which relates to a particular quiz attempt which is also present and correct.

However, when you look up the contenthash in filedir, the file is not there, and (and this leads to errors being logged when people try to access the file, which is what lead me to this).

I konw this is a tricky distributed commit situation between the file system and the database, but I would have expected the code to make some effort to ensure consistency.

However, brendanheywood commented:

its not uncommon, most migrations we do from other hosting has some base error rate of missing files, and I think there are very small race conditions that lead to this. Error rate of around 2 per million

we've internally talked about swapping the write order to disk and then db, but then you might end up with the reverse situation, it probably needs to be something like a file journal, eg write a temp record, write the file, then update the record.

the error rate is high enough for us to notice but low enough for us to have not chased it much yet

The other odd thing is that I don't see any errors logged from the time when the file was written. That means the student had no idea that their file was not saved (and so no hint that they should re-try.) At the very least, we need to try to ensure that the file API throws an exception if the file is not writted to disc.