Asynchronous restore keeps deleting course/student content

We have a site where a teacher setup a course restore with the 'delete course contents' setting enabled. It failed after the delete course content step, but before anything is meaningfully restored to the course. When it failed, the teacher went on to put their content in the course another way.

But because adhoc tasks are always re-tried if they fail, guess what? Once a day, it fires off the restore event, wiping the course content, including any student work from that day...

I would argue that something like restore should never retry - it tries, and if it fails, then it fails, and that's that. There is too much risk that you end up with a perpetually increasing amount of things in the course (if it partially completes) or something like this.

While of course we could just wrap the entire restore task in a try-catch(Throwable), so it never fails, I think that isn't a good solution, because then to logging, it appears successful.

I think we should add a method the adhoc_task abstract called something like allow_retry(): bool that is checked by the cron adhock runner if a adhock task threw an exception. That will allow tasks be reported as a failure, and to implement their own logic on if they should be resubmitted (including some that might just always return false, or some that might use fail delay, or something else to determine that they have failed a number of times, and to stop retrying).