Moodle
  1. Moodle
  2. MDL-27843 META: Accessibility compliance for 2.x
  3. MDL-30885

Prevent dataloss when a student changes some answers, then navigates away from the quiz without clicking next.

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.1, 2.1.8, 2.2.5, 2.3.2
    • Fix Version/s: 2.3.3
    • Component/s: Accessibility, Quiz
    • Labels:
    • Testing Instructions:
      Hide

      Attempt a quiz.

      Change some answers, and navigate.

      If you navigate using part of the quiz UI, which will save the changes (Next button at the bottom, question navigation buttons, Finish attempt link, Start new preview) then you should not see a warning.

      If you navigate using some other link (e.g. nav bar, or nav or settings block) then you should get the warning.

      Show
      Attempt a quiz. Change some answers, and navigate. If you navigate using part of the quiz UI, which will save the changes (Next button at the bottom, question navigation buttons, Finish attempt link, Start new preview) then you should not see a warning. If you navigate using some other link (e.g. nav bar, or nav or settings block) then you should get the warning.
    • Affected Branches:
      MOODLE_21_STABLE, MOODLE_22_STABLE, MOODLE_23_STABLE
    • Fixed Branches:
      MOODLE_23_STABLE
    • Pull Master Branch:
    • Rank:
      33894

      Description

      In the question status window, questions tell users they are "not yet answered" even though an answer has been given. What it actually means is the user hasn't saved their answer yet. Once it is saved, by moving from page to page, then it will say it is answered. This is also the case for the text next to each question where it will also says "not yet answered" until the user moves from page to page.

      There should be some way to explicitly to either save a page, save an individual question, have pages automatically save, or else make the message clearer in the question status area. The biggest problem is screen reader users will read this text and won't necessarily know if they just missed a "save" button somewhere else on the page, which can lead to increased anxiety that they are going to mess something up.

      This is a tricky problem because one potential solution would be to reintroduce the "save" button that was removed in 2.x. I assume that was taken out for usability reasons, which I think I agree with. Here are some other potential solutions.

      1. Provide a save button/link next to each question that saves the question through an AJAX call

      2. Reintroduce some type of "save this page" button/link

      3. Have Moodle automatically save all of the questions on a page every 5 minutes through an AJAX call.

      4. Have some type of explanatory text as the quiz begins that explains that answers are saved as you move from page to page.

        Issue Links

          Activity

          Hide
          Tim Hunt added a comment -

          When you say 'quiz status window' do you mean the quiz navigation panel?

          The usual work-flow for a page of the quiz is to work down the page answering questions, and then click the Next button at the bottom.

          Or, if the quiz is using Adaptive or Interactive behaviour, the user will click a 'Check' button on each question as they do it.

          Creating the quiz with only one, or a few, questions per page is also much better for accessibility.

          I can't imagine anyone being confused by this. The same issue applies to sighted users. The question statuses don't update until they move on to the next page (in Deferred feedback mode). I have never heard any complaints. Who exactly reported this bug, was it a real user, or an accessibility 'export'?

          Show
          Tim Hunt added a comment - When you say 'quiz status window' do you mean the quiz navigation panel? The usual work-flow for a page of the quiz is to work down the page answering questions, and then click the Next button at the bottom. Or, if the quiz is using Adaptive or Interactive behaviour, the user will click a 'Check' button on each question as they do it. Creating the quiz with only one, or a few, questions per page is also much better for accessibility. I can't imagine anyone being confused by this. The same issue applies to sighted users. The question statuses don't update until they move on to the next page (in Deferred feedback mode). I have never heard any complaints. Who exactly reported this bug, was it a real user, or an accessibility 'export'?
          Hide
          Itamar Tzadok added a comment -

          A different but related scenario is with multiple attempts such that new attempts continue (build on) the previous ones. In subsequent attempts users want/need to change only incorrect answers from the previous attempt and make no changes to correct answers. In the summary, the status of those questions with correct answers which the user did not change will be 'Not yet answered'. Users are confused and frustrated by that.

          Show
          Itamar Tzadok added a comment - A different but related scenario is with multiple attempts such that new attempts continue (build on) the previous ones. In subsequent attempts users want/need to change only incorrect answers from the previous attempt and make no changes to correct answers. In the summary, the status of those questions with correct answers which the user did not change will be 'Not yet answered'. Users are confused and frustrated by that.
          Hide
          Greg Kraus added a comment -

          Sorry, I do mean the quiz navigation panel.

          That workflow makes sense, but when you think about how a screen reader user interacts with the page there could be some points of confusion. In a high stakes event like a quiz the user is going to want to make sure they have everything answered. As they scan through the page where they have not previously clicked next or previous, they will hear a lot of text about unanswered questions. One of the things screen reader users have grown accustomed to (out of necessity) is making sure they didn't miss something because of bad Web page coding or their assistive technology behaving weirdly. If they hear "not yet answered" they might wonder "did I really answer everything?" There are countless occasions when screen reader users encounter a Web page and are not able to fully read it and interact with it and they know they are missing parts of the page and they might wonder if they are just missing something here too. In a high stakes environment we want to minimize (or better yet, eliminate) any undue anxiety because the screen reader user is unsure of whether they answered everything or not.

          The confusion can also lead to screen reader users needing to spend more time than is necessary on a page to double check that they did in fact answer everything.

          Show
          Greg Kraus added a comment - Sorry, I do mean the quiz navigation panel. That workflow makes sense, but when you think about how a screen reader user interacts with the page there could be some points of confusion. In a high stakes event like a quiz the user is going to want to make sure they have everything answered. As they scan through the page where they have not previously clicked next or previous, they will hear a lot of text about unanswered questions. One of the things screen reader users have grown accustomed to (out of necessity) is making sure they didn't miss something because of bad Web page coding or their assistive technology behaving weirdly. If they hear "not yet answered" they might wonder "did I really answer everything?" There are countless occasions when screen reader users encounter a Web page and are not able to fully read it and interact with it and they know they are missing parts of the page and they might wonder if they are just missing something here too. In a high stakes environment we want to minimize (or better yet, eliminate) any undue anxiety because the screen reader user is unsure of whether they answered everything or not. The confusion can also lead to screen reader users needing to spend more time than is necessary on a page to double check that they did in fact answer everything.
          Hide
          Greg Kraus added a comment -

          On a non-accessibility note, we have had many problems in the past with students starting a quiz where all the questions are on one page, and the student simply closes the window when they are done instead of clicking save or submit. In this case all of the student's work is lost. This also happens in multi-page quizzes where the last page might not get saved.

          Yes, user education solves some of it. We've also told faculty to make multi-page tests so at least the student's answered get saved when they navigate page-to-page. It has been a serious problem for us despite user education efforts. This is partially what prompted the solution of implementing an asynchronous save that fires off every so often.

          Show
          Greg Kraus added a comment - On a non-accessibility note, we have had many problems in the past with students starting a quiz where all the questions are on one page, and the student simply closes the window when they are done instead of clicking save or submit. In this case all of the student's work is lost. This also happens in multi-page quizzes where the last page might not get saved. Yes, user education solves some of it. We've also told faculty to make multi-page tests so at least the student's answered get saved when they navigate page-to-page. It has been a serious problem for us despite user education efforts. This is partially what prompted the solution of implementing an asynchronous save that fires off every so often.
          Hide
          Tim Hunt added a comment -

          Your comments seem to suggest that students might encounter a Moodle quiz for the first time in a high-stakes scenario. That strikes me as very poor course design. I think every OU course with summative quizzes also has formative practice quizzes taht the students can use to familiarise themselves with what the quiz will be like before they do it for real.

          Show
          Tim Hunt added a comment - Your comments seem to suggest that students might encounter a Moodle quiz for the first time in a high-stakes scenario. That strikes me as very poor course design. I think every OU course with summative quizzes also has formative practice quizzes taht the students can use to familiarise themselves with what the quiz will be like before they do it for real.
          Hide
          Itamar Tzadok added a comment -

          In my scenario the label is just not right and quite annoying even if the student knows that it is not right. Stakes have nothing to do with that. It's probably a problem with the definition of a state. It seems that the states definition is local to an attempt and doesn't account for multiple attempts each attempt continues the previous one. In such mode the state of a question in a new attempt can't be 'not answered yet' if it was answered in the previous attempt because the answers are transferred to the new attempt.

          Show
          Itamar Tzadok added a comment - In my scenario the label is just not right and quite annoying even if the student knows that it is not right. Stakes have nothing to do with that. It's probably a problem with the definition of a state. It seems that the states definition is local to an attempt and doesn't account for multiple attempts each attempt continues the previous one. In such mode the state of a question in a new attempt can't be 'not answered yet' if it was answered in the previous attempt because the answers are transferred to the new attempt.
          Hide
          Tim Hunt added a comment -

          Itamar, as you said in your first comment, that is a different issue.

          Show
          Tim Hunt added a comment - Itamar, as you said in your first comment, that is a different issue.
          Hide
          Greg Kraus added a comment -

          This really isn't a comment on whether courses are designed well or not or whether students just forget sometimes. The fact is this problem occurs enough that it needs to be dealt with. When you have thousands of students taking numerous quizzes, sooner or later an accident is going to happen. This suggestion simply helps solve this problem. It is akin to this issue. http://tracker.moodle.org/browse/MDL-14963

          Show
          Greg Kraus added a comment - This really isn't a comment on whether courses are designed well or not or whether students just forget sometimes. The fact is this problem occurs enough that it needs to be dealt with. When you have thousands of students taking numerous quizzes, sooner or later an accident is going to happen. This suggestion simply helps solve this problem. It is akin to this issue. http://tracker.moodle.org/browse/MDL-14963
          Hide
          Glenn Ansley added a comment -

          I can see the argument for saved v. answered terminology. I can also appreciate the technical point that Tim makes.

          If there are multiple questions on a single page, the 'Not yet answered' label does become incorrect to the user as soon as an answer is given. If the student is relying on descriptive labels on a large quiz, then the importance of this incorrect description is not minimal.

          At the same time, the question is still 'not answered' as far as Moodle is concerned on a technical level. It will remain like this until the page is saved. We can't simply change the label from "Not answered" to "Not saved" though since there may be instances where a question was saved without being answered on purpose or by mistake.

          I wonder if a good solution would be to do the following:

          • Default all questions on new page load to "Not yet answered"
          • Once a question is answered on a page, but before it is saved, change it to "Not yet saved" via JS
          • When reloading a page that has been saved already, print "Answer saved" or "Not yet answered" depending on its status.

          This solution would not work for screen readers that have JS disabled, but as Tim point out - one really cant expect it to change before the page is submitted in those cases.

          Another observation - the final "Summary of attempt" view would seem to be a good fail safe (or even work around) for students relying on those descriptions. Clicking on "Finish Attempt" from any page will bring them to the summary which will give them a breakdown of statuses for each question.

          Other possible solutions / workarounds:

          • Changing "Finish attempt..." to "Review attempt"
          • Changing "Finish attempt..." to "Attempt summary"
          • Adding a JS popup warning before navigating away from the page when 'unanswered' questions remain.
          Show
          Glenn Ansley added a comment - I can see the argument for saved v. answered terminology. I can also appreciate the technical point that Tim makes. If there are multiple questions on a single page, the 'Not yet answered' label does become incorrect to the user as soon as an answer is given. If the student is relying on descriptive labels on a large quiz, then the importance of this incorrect description is not minimal. At the same time, the question is still 'not answered' as far as Moodle is concerned on a technical level. It will remain like this until the page is saved. We can't simply change the label from "Not answered" to "Not saved" though since there may be instances where a question was saved without being answered on purpose or by mistake. I wonder if a good solution would be to do the following: Default all questions on new page load to "Not yet answered" Once a question is answered on a page, but before it is saved, change it to "Not yet saved" via JS When reloading a page that has been saved already, print "Answer saved" or "Not yet answered" depending on its status. This solution would not work for screen readers that have JS disabled, but as Tim point out - one really cant expect it to change before the page is submitted in those cases. Another observation - the final "Summary of attempt" view would seem to be a good fail safe (or even work around) for students relying on those descriptions. Clicking on "Finish Attempt" from any page will bring them to the summary which will give them a breakdown of statuses for each question. Other possible solutions / workarounds: Changing "Finish attempt..." to "Review attempt" Changing "Finish attempt..." to "Attempt summary" Adding a JS popup warning before navigating away from the page when 'unanswered' questions remain.
          Hide
          Tim Hunt added a comment -

          Right, I see a way forwards here, in the last bullet above:

          Standard Moodleforms forms now have a warning before you navigate away from the page, if you have edited data and not saved it, and we should use the same in the quiz (which is not a standard moodleform).

          If we were to do anything more complex with JavaScript, it would make sense to implement autosave, but we already have MDL-14963 for that.

          Show
          Tim Hunt added a comment - Right, I see a way forwards here, in the last bullet above: Standard Moodleforms forms now have a warning before you navigate away from the page, if you have edited data and not saved it, and we should use the same in the quiz (which is not a standard moodleform). If we were to do anything more complex with JavaScript, it would make sense to implement autosave, but we already have MDL-14963 for that.
          Hide
          Tim Hunt added a comment -

          Right. formchangechecker implemented. That was surprisingly easy.

          Show
          Tim Hunt added a comment - Right. formchangechecker implemented. That was surprisingly easy.
          Hide
          Tim Hunt added a comment -

          Just updating the summary to better reflect the underlying problem and the solution.

          Show
          Tim Hunt added a comment - Just updating the summary to better reflect the underlying problem and the solution.
          Hide
          Dan Poltawski added a comment -

          The main moodle.git repository has just been updated with latest weekly modifications. You may wish to rebase your PULL branches to simplify history and avoid any possible merge conflicts. This would also make integrator's life easier next week.

          TIA and ciao

          Show
          Dan Poltawski added a comment - The main moodle.git repository has just been updated with latest weekly modifications. You may wish to rebase your PULL branches to simplify history and avoid any possible merge conflicts. This would also make integrator's life easier next week. TIA and ciao
          Hide
          Sam Hemelryk added a comment -

          Thanks Tim, that is a indeed a simple implementation! (has been integrated now btw)

          Show
          Sam Hemelryk added a comment - Thanks Tim, that is a indeed a simple implementation! (has been integrated now btw)
          Hide
          Jason Fowler added a comment -

          Also works as expected when navigating away when no changes have yet been made to the quiz answers. Great work Tim.

          Show
          Jason Fowler added a comment - Also works as expected when navigating away when no changes have yet been made to the quiz answers. Great work Tim.
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Closing as fixed, many thanks for your awesome collaboration.

          Show
          Eloy Lafuente (stronk7) added a comment - Closing as fixed, many thanks for your awesome collaboration.

            People

            • Votes:
              3 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: