It started around 02 October 2022. And it's quite strange, be warned!

• It affects to ALL branches (39, 311, 400, master).
• It affects to ALL databases (though happen often with Oracle).
• Only Chrome
• Always happening on workers 11, 12 and 13, that are the ones already upgraded to Ubuntu 22.04 (jammy) and being tested before upgrading all.
• In Mac, I've been able to reproduce it, but with very varying results (0, 1, 19, 27 failures in 50 repetitions).

So it seems clear that, somehow (maybe because they are slower, or maybe because they are quicker...) only the workers running 22.04 are affected. Here there are some runs showing the failures:

• master + MySQL (boost) @ worker12: https://ci.moodle.org/view/B%20-%20master/job/W.03.02%20-%20Behat%20-%20Chrome%20+%20MySQLi/1331/
• 400 + Oracle (boost) @ worker13: https://ci.moodle.org/view/B%20-%20400/job/W400.03.05%20-%20Behat%20-%20Chrome%20+%20Oracle/105/
• 311 + PG (boost) @ worker12: https://ci.moodle.org/job/W311.03.01%20-%20Behat%20-%20Chrome%20+%20Postgres/455/
• 39 + PG (classic) @ worker12: https://ci.moodle.org/view/B%20-%20309/job/W39.06.01%20-%20Behat%20-%20Chrome%20+%20Postgres%20+%20Classic/367/

But, as said, results are varying, sometimes they pass without a fail, see:

• 400 + Pracle (boot) @ worker11: https://ci.moodle.org/job/W400.03.05%20-%20Behat%20-%20Chrome%20+%20Oracle/108/

Also, here there are some Mac runs (on iron, not using docker):

• 50 repetitions locally (mac iron headless): repeat finished: 50 executions, 49 ok, 1 failed.
• 50 repetitions locally (mac iron headless): repeat finished: 50 executions, 23 ok, 27 failed. !!!
• 50 repetitions locally (mac iron headless): rrepeat finished: 50 executions, 31 ok, 19 failed.
• 50 repetitions locally (mac iron headless): repeat finished: 50 executions, 50 ok, 0 failed.

And here there are some CI runs (on docker):

• Passed OK: 50 repetitions in worker08: https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/27668/
• 1 failure: 50 repetitions in worker02: https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/27673/
• lots of failures: 50 repetitions in worker11: https://ci.moodle.org/view/Testing/job/DEV.01%20-%20Developer-requested%20Behat/27674/

To reproduce the (random and mysterious) failures, just run the "Change grading options in an H5P activity" scenario a number of times and you will get 0..n failures.

Affected scenarios, all them within that feature are:

• Default grading is max attempt grade
• Change setting to average attempt
• Change maximum grade without rescaling grade
• Reescale existing grades changing the maximum grade

The theory is that, for some reason, the new workers are more sensible to those tests, maybe because they run quicker, or maybe because they run slower, that's to determine. And that makes that the 3 attempts the student does in the background of the feature file are not recorded on time. So, later, when it's checked in the scenarios, the averages, maximum and so on don't match the expectations, because the attempts (wrong, good, wrong / 0, 100, 0) are not there.

Also, getting so unconsistent results (0, 1, 19, 27..) failures makes it tricky to trace.

Let's see, here there are 2 proposed steps to try to advance:

1. Whenever an attempt is recorded, send something to webserver logs, so we can see how many of them are effectively saved.
2. Irrespectively of the previous point... we should create generators for the attempts of the h5p activity so all those "iframe" steps don't need to be executed anymore (but in some scenario exclusively checking them, we must ensure we have at very least one covering the manual attempts).

Ciao