Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-68806

New quiz attempt states to support creating or grading them asynchronously

    • MOODLE_37_STABLE, MOODLE_38_STABLE, MOODLE_402_STABLE
    • MDL-68806_main
    • Hide

      Upgrade

      1. Before applying the changes, create a course and add a quiz
      2. Apply the changes and run the upgrade.
      3. During the upgrade, set the new "Precreate period" plugin setting to "5 hours".
      4. Go back to the quiz and access the settings page.
      5. Expand the "Timing" fieldset.
      6. Confirm that you do not see the "Pre-create attempts" setting.
      7. Go to Site Administration > Plugins > Activity modules > Quiz
      8. Uncheck the "Locked" checkbox for the "Precreate period" setting and save.
      9. Return to the quiz settings page.
      10. Expand the "Timing" fieldset.
      11. Confirm that you do not see the "Pre-create attempts" setting.
      12. Check "Enabled" on the "Open the quiz" setting
      13. Confirm that you do see the "Pre-create attempts" setting, and it is unchecked.
      14. Check "Pre-create attempts" and save settings
      15. Return to the quiz settings screen.
      16. Confirm that "Pre-create attempts" is checked.

      Pre-create attempts

      1. Create a course with some students and teacher enrolled.
      2. Log in as admin
      3. Go to Site Administration > Plugins > Activity modules > Quiz
      4. Set the new "Precreate period" plugin setting to "5 hours" and check "Locked", if not already, and save changes.
      5. Create a quiz called "Test Quiz" with Timing > Open the quiz set to 5 minutes from now.
      6. Add a question to the quiz (doesn't matter what).
      7. Go to Site Administration > Server > Tasks > Scheduled tasks, find \mod_quiz\task\precreate_attempts, and click "Run now"
        1. Confirm that you see the following output, where X is the number of enrolled students:
          Found 1 quizzes to create attempts for
          Creating attempts for Test Quiz
          Created X attempts for Test Quiz in Y seconds
          Created attempts for 1 quizzes
      8. Return to the quiz view page
        1. Confirm that you see an "Attempts: X"
      9. Follow the link to the list of attempts
        1. Confirm that the options at the top of the page are set to "Attempts from: enrolled users who have a quiz attempt" and "Attempts that are" has all 6 states selected.
        2. Confirm that the list of attempts contains an attempt for each enrolled student, in the state "Not started", with all subsequent columns containing "-".
      10. Set "Attempts from" to "Enrolled users who do not have a quiz attempt" and click "Show report"
        1. Confirm that no users are shown.
      11. Set "Attempts from" to "Enrolled users who have, or do not have a quiz attempt" and click "Show report"
        1. Confirm that the attempts are shown again
      12. Untick "Not started" from the list of attempt states and click "Show report"
        1. Confirm that no attempts are shown.
      13. Log out

      Start and submit a pre-created attempt

      1. Log in as a student
      2. If necessary, wait until the quiz open time is reached
      3. Go to the course and access the quiz
        1. Confirm you do not see any attempts listed on the page
        2. Confirm you see the "Attempt quiz" button
      4. Click "Attempt quiz"
        1. Confirm you are on this quiz page with the question displayed
      5. Click "Back"
        1. Confirm that the page lists 1 in-progress attempt
      6. Click "Continue attempt"
      7. Answer the question and click "Finish attempt..."
      8. Click "Submit all and finish"

      Review a submitted attempt

      1. Click "Submit all and finish" in the confirmation modal
        1. Confirm that you see the attempt review page
        2. Confirm that the attempt details table at the top of the page shows only the Started on, Completed on, Time taken and State rows, with the state "Submitted".
        3. Confirm that the message "This attempt has been submitted and is queued for grading. Please check back shortly to see grades and feedback." is displayed above the question.
        4. Confirm that the question is displayed with your answer in place, no marks or feedback, and your answer cannot be altered.
      2. Click "Finish review"
        1. Confirm that the quiz front page shows your attempt with state "Submitted", the submitted date, and "Not yet graded" for the Marks and Grade.

      Grade submitted attempts

      1. Run `php admin/cli/adhoc_task.php --classname='\mod_quiz\task\grade_submission'` to process the attempt.
      2. Click "Review" to return to the attempt review.
        1. Confirm that you now see the "Marks" and "Grade" rows in the summary table.
        2. Confirm that you no longer see the "queued for grading" message.
        3. Confirm that the question displays indicating whether the answer is correct, showing the marks awarded and any feedback is displayed.
      3. Log out.

      Re-create pre-created attempts when a question is edited

      1. Log in as admin
      2. Go back to "Test Quiz" and select the "Results" tab.
      3. Check all attempts in the list and select "Delete selected attempts"
      4. Go to the "Settings" tab.
      5. Set "Timing" > "Open the quiz" to 5 minutes from now and save changes.
      6. Go to Site Administration > Server > Tasks > Scheduled tasks, find \mod_quiz\task\precreate_attempts, and click "Run now"
      7. Return to the "Test Quiz" "Results" tab
        1. Confirm that you see a list of attempts in "Not started" state.
      8. Go to the "Question bank" tab.
      9. Pick one of the questions used in the quiz, select "Edit" > "Edit settings".
      10. Edit the question name and save changes.
      11. Go back to the "Results" tab.
        1. Confirm that there are no attempts listed as they have been deleted.
      12. Go to Site Administration > Server > Tasks > Scheduled tasks, find \mod_quiz\task\precreate_attempts, and click "Run now"
      13. Go back to the "Results" tab.
        1. Confirm that you see a list of attempts in "Not started" state.

       

       

      Show
      Upgrade Before applying the changes, create a course and add a quiz Apply the changes and run the upgrade. During the upgrade, set the new "Precreate period" plugin setting to "5 hours". Go back to the quiz and access the settings page. Expand the "Timing" fieldset. Confirm that you do not see the "Pre-create attempts" setting. Go to Site Administration > Plugins > Activity modules > Quiz Uncheck the "Locked" checkbox for the "Precreate period" setting and save. Return to the quiz settings page. Expand the "Timing" fieldset. Confirm that you do not see the "Pre-create attempts" setting. Check "Enabled" on the "Open the quiz" setting Confirm that you do see the "Pre-create attempts" setting, and it is unchecked. Check "Pre-create attempts" and save settings Return to the quiz settings screen. Confirm that "Pre-create attempts" is checked. Pre-create attempts Create a course with some students and teacher enrolled. Log in as admin Go to Site Administration > Plugins > Activity modules > Quiz Set the new "Precreate period" plugin setting to "5 hours" and check "Locked", if not already, and save changes. Create a quiz called "Test Quiz" with Timing > Open the quiz set to 5 minutes from now. Add a question to the quiz (doesn't matter what). Go to Site Administration > Server > Tasks > Scheduled tasks, find \mod_quiz\task\precreate_attempts, and click "Run now" Confirm that you see the following output, where X is the number of enrolled students: Found 1 quizzes to create attempts for Creating attempts for Test Quiz Created X attempts for Test Quiz in Y seconds Created attempts for 1 quizzes Return to the quiz view page Confirm that you see an "Attempts: X" Follow the link to the list of attempts Confirm that the options at the top of the page are set to "Attempts from: enrolled users who have a quiz attempt" and "Attempts that are" has all 6 states selected. Confirm that the list of attempts contains an attempt for each enrolled student, in the state "Not started", with all subsequent columns containing "-". Set "Attempts from" to "Enrolled users who do not have a quiz attempt" and click "Show report" Confirm that no users are shown. Set "Attempts from" to "Enrolled users who have, or do not have a quiz attempt" and click "Show report" Confirm that the attempts are shown again Untick "Not started" from the list of attempt states and click "Show report" Confirm that no attempts are shown. Log out Start and submit a pre-created attempt Log in as a student If necessary, wait until the quiz open time is reached Go to the course and access the quiz Confirm you do not see any attempts listed on the page Confirm you see the "Attempt quiz" button Click "Attempt quiz" Confirm you are on this quiz page with the question displayed Click "Back" Confirm that the page lists 1 in-progress attempt Click "Continue attempt" Answer the question and click "Finish attempt..." Click "Submit all and finish" Review a submitted attempt Click "Submit all and finish" in the confirmation modal Confirm that you see the attempt review page Confirm that the attempt details table at the top of the page shows only the Started on, Completed on, Time taken and State rows, with the state "Submitted". Confirm that the message "This attempt has been submitted and is queued for grading. Please check back shortly to see grades and feedback." is displayed above the question. Confirm that the question is displayed with your answer in place, no marks or feedback, and your answer cannot be altered. Click "Finish review" Confirm that the quiz front page shows your attempt with state "Submitted", the submitted date, and "Not yet graded" for the Marks and Grade. Grade submitted attempts Run `php admin/cli/adhoc_task.php --classname='\mod_quiz\task\grade_submission'` to process the attempt. Click "Review" to return to the attempt review. Confirm that you now see the "Marks" and "Grade" rows in the summary table. Confirm that you no longer see the "queued for grading" message. Confirm that the question displays indicating whether the answer is correct, showing the marks awarded and any feedback is displayed. Log out. Re-create pre-created attempts when a question is edited Log in as admin Go back to "Test Quiz" and select the "Results" tab. Check all attempts in the list and select "Delete selected attempts" Go to the "Settings" tab. Set "Timing" > "Open the quiz" to 5 minutes from now and save changes. Go to Site Administration > Server > Tasks > Scheduled tasks, find \mod_quiz\task\precreate_attempts, and click "Run now" Return to the "Test Quiz" "Results" tab Confirm that you see a list of attempts in "Not started" state. Go to the "Question bank" tab. Pick one of the questions used in the quiz, select "Edit" > "Edit settings". Edit the question name and save changes. Go back to the "Results" tab. Confirm that there are no attempts listed as they have been deleted. Go to Site Administration > Server > Tasks > Scheduled tasks, find \mod_quiz\task\precreate_attempts, and click "Run now" Go back to the "Results" tab. Confirm that you see a list of attempts in "Not started" state.    

      Currently, the state diagram for quiz attempts looks like this:

      This imposes two limitations which cause performance bottlenecks:

      1. Because a newly created quiz attempt goes straight into the 'in progress' state, we are required to go straight from the quiz attempt not existing, to the state where the quiz attempt is fully initialised. Therefore, all the work to create the attempt, randomly select question from the bank, and initialise them all, must be done synchronously when the students clicks 'Start attempt'.
      2. Similarly, becuase we have to do the transition In progress -> Finished, in a single step, we must do all the work of finishing all question when handling 'Submit all and finish'.

      The plan is two introduce two new states:

      • Created: This splits 1) above. It will be possible to do all the work of populating the quiz attempt and save that as a 'Created' attempt. Then, when the student clicks Start attempt, we only need to change quiz_attempt.state = 'inprogress' and quiz_attempt.timestart.
      • Submitted: This splits 2) above. When the student clicks Submit all and finished, then we will just set quiz_attempt.state ='submitted' and quiz_attempt.timefinish. Then only later (either on cron, or if someone tries to access the review page) will we do the processing to ensure that each question in the attempt is finished.

      To support those low-level changes, we will need to:

      • All the quiz UI will need to be reviewed to ensure attempts in these states are handled and displayed appropriately. (I don't think this is a big deal.)
      • Related to the Created state, we will want to add some UI (probably in quiz_overview report) to pre-create attempts for all or selected students.
      • There is a subtle detail. The start time is also stored in each question attempt, and it is stored when the quiz attempt is created. This is not really ideal anyway. E.g. for analytics, in a multipage quiz, that does not say when the student first saw that particual question. Therefore, I think the best thing to do here is to change the question system so that new question_attempst are created with first step timestamp null, and then that timestamp is set the first time the question is rendered.

      Doing this should help greatly smooth out the two biggest peaks in server load when students are making simultaneous quiz attempts.

      Note to self: these to docs pages need to be merged, and the obsolete information removed.
      https://docs.moodle.org/dev/States_of_a_quiz_attempt
      https://docs.moodle.org/dev/Better_handling_of_overdue_quiz_attempts

              marxjohnson Mark Johnson
              timhunt Tim Hunt
              Simon Adams Simon Adams
              Votes:
              14 Vote for this issue
              Watchers:
              25 Start watching this issue

                Created:
                Updated:

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 days, 1 hour, 30 minutes
                  2d 1h 30m

                    Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.