Long-running scheduled task slows down all cron processing / task contention and re-proccessing bugs

A test script can be found here: https://github.com/catalyst/moodle-tool_testtasks/blob/master/tests/test9.sh A test run with 16 ad hoc tasks shoudl complete is just over 16 seconds with 1 cron process: time admin/tool/testtasks/tests/test9.sh 1 Runs with multiple processes should NOT result in duplicated processing of ad hoc tasks (or scheduled but this script doesn't test that explicitely). Additionally the overall total should be reduce from 16 seconds to say 10 time admin/tool/testtasks/tests/test9.sh 2 As you throw more cron processes at it the overall processing time should asymptotically flatten out, eg time admin/tool/testtasks/tests/test9.sh 4 The scheduled tasks also need to be hammered at scale. You should not see the same task being processed within the same cron tick because of different front ends reprocessing the same task after gaining a lock that had previously been locked elsewhere. You should also not ever get cron processes waiting a long time to try and run a scheduled task which is already running elsewhere