Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-79130

Tasks: Ad-hoc tasks shouldn't retry after max fail delay

XMLWordPrintable

    • MOODLE_403_STABLE
    • MOODLE_404_STABLE
    • MDL-79130-master
    • Hide

      Covered by unittest

      Prerequisites

      1. Login as admin.
      2. Navigate to Site administration.
      3. Enable enableasyncbackup setting (Checked).
      4. Create a course called "Test restore course".
      5. Navigate to the created course.
      6. Create some activities (Assign, Quiz...).
      7. Back up that course.
      8. Run php admin/cli/adhoc_task.php --execute to complete the backup process.
      9. Navigate to Course Reuse and change the dropdown to Restore.
      10. Click the Download link in the Course Backup Area Table.

      Set PHP cli path

      1. Go to Site admin > Server > System path
      2. Set your PHP binary file path in "Path to PHP CLI"

      Testing

      1. Apply Patch_to_test_MDL-79130.patch
      2. Navigate to the created course.
      3. Click on More > Course reuse > Restore.
      4. Upload and restore the downloaded backup.
      5. In the Destination stage, make sure to select Delete the contents of this course and then restore option in "Restore into this course".
      6. Go to Site admin > Server > Adhoc tasks
      7. Verify that there is one record with Component / Class name = "asynchronous_restore_task".
      8. Note the id of that task.
      9. Click on "Run now" on that record
      10. Verify that you see the following message 

        Adhoc task failed: core\task\asynchronous_restore_task,error/test-79130-exception

      11. Run the following query in your Database tool: 

        select attemptsavailable from mdl_task_adhoc mta where id = [your_noted_task_id_in_step_8];

      12. Verify that the value is set to 8 because it has failed once so it's been decremented by 1.
      13. Repeat the steps from 6 to 9 for 8 more times and the do another run which will be the ninth times
      14. Go to Site admin > Server > Adhoc tasks
      15. Verify that there is one record with Component / Class name = "asynchronous_restore_task".
      16. Verify that the Next run value is: Never.
      17. Run the following query in your Database tool to make the task due: 

        update mdl_task_adhoc mta set nextruntime = 1700712048 where id = [your_noted_task_id_in_step_8];

      18. Go to Site admin > Server > Adhoc tasks
      19. Verify that there is one record with Component / Class name = "asynchronous_restore_task".
      20. Verify that the Ad hoc tasks due value is: 1.
      21. Run the following command in your terminal: 

        php admin/cli/adhoc_task.php --execute

      22. Verify that you don't see the following message: 

        Execute adhoc task: core\task\asynchronous_restore_task
        		 
        Adhoc task failed: core\task\asynchronous_restore_task,error/test-79130-exception

      Show
      Covered by unittest Prerequisites Login as admin. Navigate to Site administration. Enable enableasyncbackup setting (Checked). Create a course called "Test restore course". Navigate to the created course. Create some activities (Assign, Quiz...). Back up that course. Run php admin/cli/adhoc_task.php --execute to complete the backup process. Navigate to Course Reuse and change the dropdown to Restore. Click the Download link in the Course Backup Area Table. Set PHP cli path Go to Site admin > Server > System path Set your PHP binary file path in "Path to PHP CLI" Testing Apply Patch_to_test_MDL-79130.patch Navigate to the created course. Click on More > Course reuse > Restore. Upload and restore the downloaded backup. In the Destination stage, make sure to select  Delete the contents of this course and then restore  option in "Restore into this course". Go to Site admin > Server > Adhoc tasks Verify that there is one record with Component / Class name = "asynchronous_restore_task". Note the id of that task. Click on "Run now" on that record Verify that you see the following message Adhoc task failed: core\task\asynchronous_restore_task,error/test-79130-exception Run the following query in your Database tool: select attemptsavailable from mdl_task_adhoc mta where id = [your_noted_task_id_in_step_8]; Verify that the value is set to 8 because it has failed once so it's been decremented by 1. Repeat the steps from 6 to 9 for 8 more times and the do another run which will be the ninth times Go to Site admin > Server > Adhoc tasks Verify that there is one record with Component / Class name = "asynchronous_restore_task". Verify that the Next run value is: Never . Run the following query in your Database tool to make the task due: update mdl_task_adhoc mta set nextruntime = 1700712048 where id = [your_noted_task_id_in_step_8]; Go to Site admin > Server > Adhoc tasks Verify that there is one record with Component / Class name = "asynchronous_restore_task". Verify that the Ad hoc tasks due value is: 1 . Run the following command in your terminal: php admin/cli/adhoc_task.php --execute Verify that you don't see the following message: Execute adhoc task: core\task\asynchronous_restore_task Adhoc task failed: core\task\asynchronous_restore_task,error/test-79130-exception
    • 6
    • Team Hedgehog 2023 Sprint 4.1, Team Hedgehog 2023 Sprint 4.2, Team Hedgehog 2023 Sprint 4.3, Team Hedgehog 2023 Review 4

      Ad-hoc tasks are often used for asynchronous operations, and those that communicate with external systems. Because of this the existing task retry functionality with progressive backoff is good, and manages cases where there is a temporary issue to stop a task completing. 

      However, ad-hoc tasks retry indefinitely, this is not ideal.  If an adhoc task has failed 9 times over the course of 28 hours (current backoff behaviour), it is not going to succeed.

      Instead ad-hoc tasks should permanently fail once the max retry period has been exceeded.

      We should still keep the ability to re-run these tasks from the UI and CLI, as we can for existing ad-hoc tasks. This will help with some recovery and debugging ops for failed tasks.

      We should reuse the cleanup solution that will be introduced in MDL-79128 to manage cleaning up permanently failed tasks.

        1. MDL-79130.png
          1.05 MB
          Ron Carl Alfon Yu
        2. Patch_to_test_MDL-79130.patch
          1 kB
          Huong Nguyen

            stevani.andolo@moodle.com Stevani Andolo
            matt.porritt@moodle.com Matt Porritt
            Meirza Meirza
            Ilya Tregubov Ilya Tregubov
            Ron Carl Alfon Yu Ron Carl Alfon Yu
            Votes:
            3 Vote for this issue
            Watchers:
            17 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 2 days, 5 hours, 19 minutes
                2d 5h 19m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.