This is a similar in philosophy to -
- which was done in Moodle 2.8. There, we randomised the starting value of DB sequences, to ensure that people did not write fragile unit tests because they had hard-coded expected ids in asserts. MDL-43835
There is another problem, where core unit tests assume that, for example, a new Moodle install only has one course category, but a third-party plugin my create a course category for its own use on install.
In order to prevent these test-breaking assumptions from getting in to Moodle core, perhaps we should change the PHPunit site installer, so that after a default Moodle install, it generates a small random number (0, 1 or 2) of some key entities (e.g. user, course, category, question). These things should be given very specific names, to minimise the change they break existing tests.
As a specific example, if you install qtype_coderunner, then core_question_privacy_provider_testcase starts failing.