Moodle
  1. Moodle
  2. MDL-8958

quiz problem with build on last attempt and random questions

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.4
    • Fix Version/s: 1.6.6, 1.7.2, 1.8, 1.9
    • Component/s: Quiz
    • Labels:
      None
    • Environment:
      Moodle 1.6.4, Postgres 8.2.0, PHP 4.3.9.
    • Database:
      Any
    • Affected Branches:
      MOODLE_16_STABLE
    • Fixed Branches:
      MOODLE_16_STABLE, MOODLE_17_STABLE, MOODLE_18_STABLE, MOODLE_19_STABLE
    • Rank:
      28967

      Description

      When we use "Each attempt builds on the last" feature, the 1st and 2nd attempts work nicely, but when the students attempts the quiz for the 3rd time, the questions are no longer the same. We tried this with our production site (Postgres) and test site (Mysql) and the problem seems to affect only Postgres. After some digging I found an old bug (see: http://tracker.moodle.org/browse/MDL-3684) with similar symptoms on Mysql. Could it be that the problem was solved only in the Mysql-spesific code?

      Steps to reproduce:
      1/ create a new quiz: "Each attempt builds on the last" enabled. Set the quiz so that it lets the student take the quiz e.g. 3 times.
      2/ add e.g. 10 questions to quiz
      3/ take the quiz: answer some of the questions and submit the answers.
      4) take a look at the submitted quiz: the quiz is 100% similar to the quiz you just answered - everything's OK here!
      5/ take the quiz again: you see the same set of questions with some of your questions answered - OK! Submit the quiz. The results page is also the same, so everyhting's OK.
      6/ now take the quiz for the 3rd time: the questions aren't the same anymore!

        Issue Links

          Activity

          Hide
          Olli Salo added a comment -

          I meant to write: somewhat similar symptoms. This case and the older http://tracker.moodle.org/browse/MDL-3684) -case aren't identical.

          Show
          Olli Salo added a comment - I meant to write: somewhat similar symptoms. This case and the older http://tracker.moodle.org/browse/MDL-3684 ) -case aren't identical.
          Hide
          Tim Hunt added a comment -

          What do you mean when you say 'the questions are no longer the same'? How have they changed? Are you using random questions?

          Show
          Tim Hunt added a comment - What do you mean when you say 'the questions are no longer the same'? How have they changed? Are you using random questions?
          Hide
          Olli Salo added a comment -

          Yep, we're using randomized questions. With "the questions are no longer the same" I mean that the question changes to a totally different question. So it's not that the question would have wrong answer alternatives or so, it is just that the whole question is totally different one in the 3rd attempt. An example:

          Attempt 1 might have e.g. the following questions: "What's your name?", What's your age? and "What's your weight?"
          Attempt 1 result page would have the same questions: "What's your name?", What's your age? and "What's your weight?"
          Attempt 2 would have the same questions: : "What's your name?", What's your age? and "What's your weight?"
          Attempt 2 result page would have the same questions: : "What's your name?", What's your age? and "What's your weight?"
          Attempt 3 would have totally different questions: "What is your favourite food?", "Where do you live?", "How old are you?"

          Show
          Olli Salo added a comment - Yep, we're using randomized questions. With "the questions are no longer the same" I mean that the question changes to a totally different question. So it's not that the question would have wrong answer alternatives or so, it is just that the whole question is totally different one in the 3rd attempt. An example: Attempt 1 might have e.g. the following questions: "What's your name?", What's your age? and "What's your weight?" Attempt 1 result page would have the same questions: "What's your name?", What's your age? and "What's your weight?" Attempt 2 would have the same questions: : "What's your name?", What's your age? and "What's your weight?" Attempt 2 result page would have the same questions: : "What's your name?", What's your age? and "What's your weight?" Attempt 3 would have totally different questions: "What is your favourite food?", "Where do you live?", "How old are you?"
          Hide
          Paulo Matos added a comment -

          Ok as far as I could understand this is not related to MDL3684, neither is Postgresql specific.
          The problem is located on attempt.php when calls get_question_states() of lib/questionlib.php, and this one creates a new randomized question.

          Show
          Paulo Matos added a comment - Ok as far as I could understand this is not related to MDL3684, neither is Postgresql specific. The problem is located on attempt.php when calls get_question_states() of lib/questionlib.php, and this one creates a new randomized question.
          Hide
          Paulo Matos added a comment -

          And after a lot of tests we got the reported problem after 2nd attempt submit, review page was already altered.

          Show
          Paulo Matos added a comment - And after a lot of tests we got the reported problem after 2nd attempt submit, review page was already altered.
          Hide
          Paulo Matos added a comment -

          Here goes the first release of the patch to solve this issue. It was done by moving the code
          responsible from build on last attempt from mod/quiz/attempt.php to lib/questionlib.php on
          get_question_states().

          lastattemptid was added as a new parameter which is set to null as default, this was done so quiz specific
          code didn't get into questionlib. Based on this value we decide if we get the last attempt or not.

          This block

          • // ... and save the new states
          • save_question_session($questions[$i], $states[$i]);
            was not moved because question session are saved after the call of get_question_states() if we are
            on a newattempt.

          Please test this and get back with your report.

          Show
          Paulo Matos added a comment - Here goes the first release of the patch to solve this issue. It was done by moving the code responsible from build on last attempt from mod/quiz/attempt.php to lib/questionlib.php on get_question_states(). lastattemptid was added as a new parameter which is set to null as default, this was done so quiz specific code didn't get into questionlib. Based on this value we decide if we get the last attempt or not. This block // ... and save the new states save_question_session($questions [$i] , $states [$i] ); was not moved because question session are saved after the call of get_question_states() if we are on a newattempt. Please test this and get back with your report.
          Hide
          Olli Salo added a comment -

          Hi Paulo,

          your code works! The "Each attempt builds on the last" feature now fetches the right questions each time the student takes the quiz. I tried one quiz (with 6 attempts) for a couple of times and it worked well.

          Thank you very much for solving this problem!

          Show
          Olli Salo added a comment - Hi Paulo, your code works! The "Each attempt builds on the last" feature now fetches the right questions each time the student takes the quiz. I tried one quiz (with 6 attempts) for a couple of times and it worked well. Thank you very much for solving this problem!
          Hide
          Paulo Matos added a comment -

          Hi Olli!
          Can you do some tests in adaptative mode and report if everything is ok!
          Regards,
          Paulo

          Show
          Paulo Matos added a comment - Hi Olli! Can you do some tests in adaptative mode and report if everything is ok! Regards, Paulo
          Hide
          Olli Salo added a comment -

          Hi Paulo,

          I did the following tests / checks:

          1) I set the quiz in adaptive mode and changed my answers in some questions during the 1st attempt: OK.

          2) changed my answers during the 2nd, 3rd, 4th and 5th attempt: OK.

          3) used checkbox, radiobutton and short answer -type of questions: all worked OK.

          At least to me it seems that the adaptive mode works well with your code.

          Regard,

          Olli

          Show
          Olli Salo added a comment - Hi Paulo, I did the following tests / checks: 1) I set the quiz in adaptive mode and changed my answers in some questions during the 1st attempt: OK. 2) changed my answers during the 2nd, 3rd, 4th and 5th attempt: OK. 3) used checkbox, radiobutton and short answer -type of questions: all worked OK. At least to me it seems that the adaptive mode works well with your code. Regard, Olli
          Hide
          Paulo Matos added a comment -

          This is somehow related.

          Show
          Paulo Matos added a comment - This is somehow related.
          Hide
          Paulo Matos added a comment -

          Forget comments get away from action
          What I meant on the previous comment was: This is somehow related with MDL-6269.

          Show
          Paulo Matos added a comment - Forget comments get away from action What I meant on the previous comment was: This is somehow related with MDL-6269 .
          Hide
          Paulo Matos added a comment -

          My tests with success, on a quiz as a set of 10 random questions.
          Available question types for random choose were: cloze; multichoice; multianswer; numerical; calculated (with/without datasets); and shortanswer.
          Multiple attempts (max 3, 4 and 5), using last attempt grading, mixed tests from completely empty until full answered
          attempts and reverse, with best score to worst score attempts and reverse, students with 1 attempt to max attempts allowed.

          Show
          Paulo Matos added a comment - My tests with success, on a quiz as a set of 10 random questions. Available question types for random choose were: cloze; multichoice; multianswer; numerical; calculated (with/without datasets); and shortanswer. Multiple attempts (max 3, 4 and 5), using last attempt grading, mixed tests from completely empty until full answered attempts and reverse, with best score to worst score attempts and reverse, students with 1 attempt to max attempts allowed.
          Hide
          Olli Salo added a comment -

          I couldn't reproduce MDL-6269 so I cannot say if the MDL-6269-problem still exists. I tried a new quiz with 3 random questions. I filled 2 questions with correct answers and got 2/3 correct. On my 2nd attempt I answered the 3rd question correctly and got 3/3 (100%) correct. I tried a 3rd attempt and still everything was just like it should be. So the grading seems to work - at least in our case.

          Show
          Olli Salo added a comment - I couldn't reproduce MDL-6269 so I cannot say if the MDL-6269 -problem still exists. I tried a new quiz with 3 random questions. I filled 2 questions with correct answers and got 2/3 correct. On my 2nd attempt I answered the 3rd question correctly and got 3/3 (100%) correct. I tried a 3rd attempt and still everything was just like it should be. So the grading seems to work - at least in our case.
          Hide
          Paulo Matos added a comment -

          Olli, as far as I could understand from the discussion referred on MDL-6269 it is a mix of this bug (MDL-8958) and MDL-8990.

          Show
          Paulo Matos added a comment - Olli, as far as I could understand from the discussion referred on MDL-6269 it is a mix of this bug ( MDL-8958 ) and MDL-8990 .
          Hide
          Olli Salo added a comment -

          Paulo, I took a look at MDL-8990 and tried reproducing it with your patched code. I came across the same error when I took the quiz (attempt builds enabled) so that I first answered questions 1, 2, 4 (out of 6 questions in my quiz) and got 3 out of 6 points. On my 2nd attempt I answered question 6, so my total score was 4/6. I then logged in as a teacher and hit the regrade-button. Suprisingly, the points I had (as a student) went down from 3 to 1. Like mentioned in MDL-8990, the correct answers were still visible for me as a student.

          As I took the quiz for the 3rd time, I didn't change my answers but I only saved and submitted the quiz. Then, as a teacher, I regraded the quiz again, and WOW: now the student's score was 0/6. Now, as a student, I took the quiz one more time and I saw all the right questions with right answers. As a student I submitted the quiz and again I got 4/6 points (correct value). Then, as a teacher, I once again regraded the quiz and the student's 4/6 point went back to 0.

          Apparently the MDL-8990 is still there - and hopefully the text above helps you solve it.

          Show
          Olli Salo added a comment - Paulo, I took a look at MDL-8990 and tried reproducing it with your patched code. I came across the same error when I took the quiz (attempt builds enabled) so that I first answered questions 1, 2, 4 (out of 6 questions in my quiz) and got 3 out of 6 points. On my 2nd attempt I answered question 6, so my total score was 4/6. I then logged in as a teacher and hit the regrade-button. Suprisingly, the points I had (as a student) went down from 3 to 1. Like mentioned in MDL-8990 , the correct answers were still visible for me as a student. As I took the quiz for the 3rd time, I didn't change my answers but I only saved and submitted the quiz. Then, as a teacher, I regraded the quiz again, and WOW: now the student's score was 0/6. Now, as a student, I took the quiz one more time and I saw all the right questions with right answers. As a student I submitted the quiz and again I got 4/6 points (correct value). Then, as a teacher, I once again regraded the quiz and the student's 4/6 point went back to 0. Apparently the MDL-8990 is still there - and hopefully the text above helps you solve it.
          Hide
          Olli Salo added a comment -

          Oops, a typo: "down from 3 to 1" was supposed to be "down from 4 to 1".

          Show
          Olli Salo added a comment - Oops, a typo: "down from 3 to 1" was supposed to be "down from 4 to 1".
          Hide
          Tim Hunt added a comment -

          The fix to MDL-8990 is a separate change that Paulo did not send you. I would like to get the fix to this bug, and MDL-8990 checked in soon. I think we have had enough testing. It is just a matter of finding time.

          Show
          Tim Hunt added a comment - The fix to MDL-8990 is a separate change that Paulo did not send you. I would like to get the fix to this bug, and MDL-8990 checked in soon. I think we have had enough testing. It is just a matter of finding time.
          Hide
          Olli Salo added a comment -

          Hi Tim,

          thanks for clarifying this. Hopefully both patches will be added to cvs soon.

          Yours, Olli

          Show
          Olli Salo added a comment - Hi Tim, thanks for clarifying this. Hopefully both patches will be added to cvs soon. Yours, Olli
          Hide
          Paulo Matos added a comment -

          Sorry for that Olli! If you want you can get the patch on MDL-8990 ticket page.

          Show
          Paulo Matos added a comment - Sorry for that Olli! If you want you can get the patch on MDL-8990 ticket page.
          Hide
          Olli Salo added a comment -

          No prob! Thx for the link, though.

          Show
          Olli Salo added a comment - No prob! Thx for the link, though.
          Hide
          Tim Hunt added a comment -

          I hope the check the fix into 1.6.4+ tomorrow, but it needs more testing.

          Show
          Tim Hunt added a comment - I hope the check the fix into 1.6.4+ tomorrow, but it needs more testing.
          Hide
          Olli Salo added a comment -

          Hi Tim,

          always nice to hear good news! I'm eagerly waiting for this batch to appear in the 1.6.4+ and so are the teachers who are using this feature!

          Thx for letting me know about the progress on this issue. Should you need any further help on testing this patch, please let me know.

          Yours, Olli

          Show
          Olli Salo added a comment - Hi Tim, always nice to hear good news! I'm eagerly waiting for this batch to appear in the 1.6.4+ and so are the teachers who are using this feature! Thx for letting me know about the progress on this issue. Should you need any further help on testing this patch, please let me know. Yours, Olli
          Hide
          Wen Hao Chuang added a comment -

          Just wondering if this is related to http://moodle.org/mod/forum/discuss.php?d=69134

          For our case, the instructor was using "grading method" set to "last attempt", and he was drawing randomized questions (10 questions) from a 30-40 questions test bank. In very rare occasion (not easy to reproduce), there is a discrepancy between the two different views (quiz/index.php and quiz/view.php) when log in as a student (and the gradebook will be using the lower grade based on what's displaying on quiz/index.php. Thanks!

          Show
          Wen Hao Chuang added a comment - Just wondering if this is related to http://moodle.org/mod/forum/discuss.php?d=69134 For our case, the instructor was using "grading method" set to "last attempt", and he was drawing randomized questions (10 questions) from a 30-40 questions test bank. In very rare occasion (not easy to reproduce), there is a discrepancy between the two different views (quiz/index.php and quiz/view.php) when log in as a student (and the gradebook will be using the lower grade based on what's displaying on quiz/index.php. Thanks!
          Hide
          Paulo Matos added a comment -

          As far as I could understand this is some miscalculation on best grade from quiz/index.php since the gradebook has the correct grades.
          Can you confirm this Wen?

          Regards, Paulo.

          Show
          Paulo Matos added a comment - As far as I could understand this is some miscalculation on best grade from quiz/index.php since the gradebook has the correct grades. Can you confirm this Wen? Regards, Paulo.
          Hide
          Wen Hao Chuang added a comment -

          Yes, Paulo, it was some miscalculation on best grade. Do you know if there is a patch or moodle ticket about this? Many thanks!

          Show
          Wen Hao Chuang added a comment - Yes, Paulo, it was some miscalculation on best grade. Do you know if there is a patch or moodle ticket about this? Many thanks!
          Hide
          Wen Hao Chuang added a comment -

          Wiat, Paulo, we are using gradebook plus v2.0 on our live site and it's showing 9/10 which is wrong. So it's not just a miscalculation on the best grade from quiz/index.php but the gradebook (from student's view) is also showing the wrong grade. Thanks!

          Show
          Wen Hao Chuang added a comment - Wiat, Paulo, we are using gradebook plus v2.0 on our live site and it's showing 9/10 which is wrong. So it's not just a miscalculation on the best grade from quiz/index.php but the gradebook (from student's view) is also showing the wrong grade. Thanks!
          Hide
          Olli Salo added a comment -

          Hi, I think this issue should be reopened. Please look at both of the following issues I've submitted (they're linked to this one, MDL-8958):

          http://tracker.moodle.org/browse/MDL-9375
          http://tracker.moodle.org/browse/MDL-9377

          Olli S.

          Show
          Olli Salo added a comment - Hi, I think this issue should be reopened. Please look at both of the following issues I've submitted (they're linked to this one, MDL-8958 ): http://tracker.moodle.org/browse/MDL-9375 http://tracker.moodle.org/browse/MDL-9377 Olli S.
          Hide
          Olli Salo added a comment -

          Tim / Paulo,

          could you give us an estimate whether this issue will be reopened? You see, this issue seems to be a bit more problematic than what I expected earlier: MDL-9377, which is closely linked to this, seems to have an issue when showing the grade: sometimes the grades are not shown to the student at all even after the 1st attempt. This of course is pretty annoying for the student because he instantly thinks that his grade is completely missing. And when there are 2000+ students taking the quiz, this becomes a problem for the sysadmin.

          Olli

          Show
          Olli Salo added a comment - Tim / Paulo, could you give us an estimate whether this issue will be reopened? You see, this issue seems to be a bit more problematic than what I expected earlier: MDL-9377 , which is closely linked to this, seems to have an issue when showing the grade: sometimes the grades are not shown to the student at all even after the 1st attempt. This of course is pretty annoying for the student because he instantly thinks that his grade is completely missing. And when there are 2000+ students taking the quiz, this becomes a problem for the sysadmin. Olli

            People

            • Votes:
              3 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: