On very large sites there is some small base level of temporary file uploading failure.

I don't know the exact cause but I'd guess something like a file is uploaded to php tmp space, then it is moved into a File API .tmp and then by pure bad luck something borks (maybe a php timeout or a container gets reaped?) and the file is never fully copied into the correct place, somewhere around here:

https://github.com/moodle/moodle/blob/b88f1a84bfa944b5890728ea20c8ebd888fb9b3f/lib/filestorage/file_system_filedir.php#L380-L397

So thinking we want some cleanup task maybe once a week which just looks for all old .tmp files in filedir and deletes them.