We have jobs which try and upgrade Moodle from earlier versions and compare to installed version - at the moment this only happens after an issue has been integrated.
We should add this to the checks done by cibot before an issue is integrated.
Guessing that we don't have this now because its complex and slow, some quick hacks we could do get round that:
- Only run when an upgrade.php/install.xml file has changed
- Only run it on master branch from earliest version (would find most problems)