Copying comments from MDL-59039 comments:

David Mo:
Today we got a random failure related to this issue:

1) search_manager_testcase::test_partial_indexing

Failed asserting that actual size 0 matches expected size 1.

/Users/Shared/Jenkins/Home/git_repositories/master/search/tests/manager_test.php:243

/Users/Shared/Jenkins/Home/git_repositories/master/lib/phpunit/classes/advanced_testcase.php:80

To re-run:

 vendor/bin/phpunit search_manager_testcase search/tests/manager_test.php

It is the first time it fails, I haven't checked how it all works but:
1. the label is created
2. we wait for the next second (waitForSecond)
3. we index contents for 1 second
4. ERROR 0 docs indexed
the only possible cause I can think of is that 2 full seconds pass from the second the label is created and the second we index, I imagine that, although not likely, it could be possible, any ideas?

Sam Marshall reply:
Sorry I was away on holiday but I've now had a look at this and I think I've figured it out. It doesn't matter how long time passes after the label being created (as long as it is at least the next second).
When we index contents with 1 second time limit, this part is supposed to test that the forum is done last (because it took longer last time). It is supposed to work as follows:
1 Index all the other search areas apart from forum
2 Most of these areas have no new data it should take less than 1 second to check all of them (ASSUMPTION!)
3 When it hits label, it will 'index' the label, and also do the fake 1.2 second delay that the mock code adds in to simulate indexing taking time
4 Because that took 1.2 seconds, indexing will now stop
It checks the time limit after processing each search area (even if no documents were added there), and after adding each document within a search area.
I think what is happening is that it's taking more than 1 second to check some other search area or areas (which contain no new documents but it still has to do the query) before it even gets to label. Therefore it exits before doing a single document. In other words the assumption is not always true.
I can think of three approaches to fix this:
1 In the test, increase all the times (throughout the test) a bit to reduce the chance of a failure. I tried to get the times fairly tight so this doesn't add ages to phpunit run time but we could make them a bit bigger...
or
2 In the real code, change the timeout logic so that it won't stop indexing before it indexes at least one document (i.e. if you set a stupid small limit and it takes that long to just look at all the search areas with nothing new in).
or
3 Change the real code so that there is a way to override the current time value, for use in unit tests, so that the timeout part is independent of real time (i.e. it would only 'advance' time when the mock search engine indexes a document, and nothing would really have to wait any more). This will probably mean using the PHPUNIT_TEST define and a global variable for the current fake time. Could be a little complicated and makes the tests fractionally less 'realistic', but maybe the most comprehensive solution.
Obviously there is also option 4 to do nothing if the failures are sufficiently rare.
I'm OK to code a fix if you want, but not certain which approach is best?