Moodle
  1. Moodle
  2. MDL-27916

Incorrect feedback to students on adaptive questions in Moodle 2.1

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1
    • Fix Version/s: 2.1
    • Component/s: Quiz
    • Labels:
    • Testing Instructions:
      Hide

      Attempt quizzes in each of the adaptive and immediate feedback modes (four modes in total).

      (It may be simplest to try this with a one-question quiz. I suggest a single multiple choice question with choices correct, partially correct and incorrect, that do what they say on the tin.)

      Try various sequences of actions where you choose one of the options, and then either click check to have the response graded, or click next which just saves the response without grading it.

      Verify that the description of the question's current state, which appears to the left of the question under the question number, and also on the quiz summary page, makes sense whatever state the question is in.

      You may with to compare the way these these four modes respond with the the Interactive with multiple tries mode, which is used heavily at the OU, and so has been tested and scrutinised closely.

      Show
      Attempt quizzes in each of the adaptive and immediate feedback modes (four modes in total). (It may be simplest to try this with a one-question quiz. I suggest a single multiple choice question with choices correct, partially correct and incorrect, that do what they say on the tin.) Try various sequences of actions where you choose one of the options, and then either click check to have the response graded, or click next which just saves the response without grading it. Verify that the description of the question's current state, which appears to the left of the question under the question number, and also on the quiz summary page, makes sense whatever state the question is in. You may with to compare the way these these four modes respond with the the Interactive with multiple tries mode, which is used heavily at the OU, and so has been tested and scrutinised closely.
    • Workaround:
      Hide

      Ignore the messages. It works okay, it just tells the students the wrong status.

      Show
      Ignore the messages. It works okay, it just tells the students the wrong status.
    • URL:
      qa.moodle.net
    • Affected Branches:
      MOODLE_21_STABLE
    • Fixed Branches:
      MOODLE_21_STABLE
    • Pull from Repository:
    • Pull Master Branch:
    • Rank:
      17828

      Description

      If you set up an adaptive question and answer it correctly the first time (I used a multiple choice question), it marks it correct, but gives the student a message saying "this submission attracted a penalty of 0.3333." In fact the penalty was not actually assessed, but the student might think that was the case.

      If the student initially answered the question incorrectly, it correctly assesses the penalty and gives the right message. However, when the question is answered correctly on a subsequent try it still tells the student a penalty was attracted (even though it says the answer was correct). As in the previous case, it doesn't really extract the penalty, it only says it did.

      Another test I did was to submit the answer to a question in adaptive mode without first clicking on the check button. Moodle correctly proceeds to grade the question, but the Summary of attempt screen lists the question status as "Not yet answered." This could confuse and worry students as well. If the student proceeds to submit the quiz anyway, the question is appropriately graded.

      This final problem happens for a number of other choices in the "How questions behave" drop-down menu as well. I noticed it with Adaptive Mode (nopenalties) and Immediate feedback as well. It isn't a problem in deferred feedback mode. There is correctly reports the status "Answer saved."

        Issue Links

          Activity

          Hide
          Tim Hunt added a comment -

          Regarding the "this submission attracted a penalty of 0.3333." message. Please compare this with Moodle 2.0. You will find it has always been like that, and we do not want to change something people are familiar with.

          (Historical note: It was the OU's dissatisfaction with how Adaptive mode has always worked that lead us to create Interactive with multiple tries. And it was our desire to get that into Moodle core that lead to the whole new Question engine project.)

          The "Not yet answered" state is a definite issue. I will address that. However, it is not completely clear what the correct behaviour is. For example, the whole point of Immediate feedback mode is that the student clicks the Check button during the quiz attempt. Sure, we grade any response they have entered when they click Submit all and finish, but that is really just a fallback. Perhaps changing Not yet answered to Not complete is a good solution.

          Show
          Tim Hunt added a comment - Regarding the "this submission attracted a penalty of 0.3333." message. Please compare this with Moodle 2.0. You will find it has always been like that, and we do not want to change something people are familiar with. (Historical note: It was the OU's dissatisfaction with how Adaptive mode has always worked that lead us to create Interactive with multiple tries. And it was our desire to get that into Moodle core that lead to the whole new Question engine project.) The "Not yet answered" state is a definite issue. I will address that. However, it is not completely clear what the correct behaviour is. For example, the whole point of Immediate feedback mode is that the student clicks the Check button during the quiz attempt. Sure, we grade any response they have entered when they click Submit all and finish, but that is really just a fallback. Perhaps changing Not yet answered to Not complete is a good solution.
          Hide
          Tim Hunt added a comment -

          Just to note that Interactive with multiple tries was already using 'Not complete' to describe this state, and we reached that point at the OU after some discussion during testing, so I have made the adaptive and immediate modes work the same way.

          Show
          Tim Hunt added a comment - Just to note that Interactive with multiple tries was already using 'Not complete' to describe this state, and we reached that point at the OU after some discussion during testing, so I have made the adaptive and immediate modes work the same way.
          Hide
          Sam Hemelryk added a comment -

          Thanks Tim - integrated now.

          Show
          Sam Hemelryk added a comment - Thanks Tim - integrated now.
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Tim, was testing MDL-24594 and got problems in step 3 of its instructions:

          • Reviewing the question, if i pick the correct (100% grade) answer, I got: "Marks for this submission: 1.00/1.00. This submission attracted a penalty of 0.3333333.".
          • Reviewing the question, if I pick one partially-correct (5% grade) answer, I got: "Marks for this submission: 0.05/1.00. With previous penalties this gives 1.00/1.00. This submission attracted a penalty of 0.3333333."

          That seems to be exactly the sort of problem fixed by this, isn't it?

          Show
          Eloy Lafuente (stronk7) added a comment - Tim, was testing MDL-24594 and got problems in step 3 of its instructions: Reviewing the question, if i pick the correct (100% grade) answer, I got: "Marks for this submission: 1.00/1.00. This submission attracted a penalty of 0.3333333.". Reviewing the question, if I pick one partially-correct (5% grade) answer, I got: "Marks for this submission: 0.05/1.00. With previous penalties this gives 1.00/1.00. This submission attracted a penalty of 0.3333333." That seems to be exactly the sort of problem fixed by this, isn't it?
          Hide
          Eloy Lafuente (stronk7) added a comment - - edited

          It's really later here 02:06, so perhaps I'm just tired but I was looking to the code changes, just to see what changed, try to affine the testing and found this:

          public function get_state_string($showcorrectness) {
              $state = $this->qa->get_state()
              if ($state == question_state::$todo) {
              ....
              ....
          

          $todo ????

          both in:

          • question/behaviour/adaptive/behaviour.php
          • question/behaviour/immediatefeedback/behaviour.php
          Show
          Eloy Lafuente (stronk7) added a comment - - edited It's really later here 02:06, so perhaps I'm just tired but I was looking to the code changes, just to see what changed, try to affine the testing and found this: public function get_state_string($showcorrectness) { $state = $ this ->qa->get_state() if ($state == question_state::$todo) { .... .... $todo ???? both in: question/behaviour/adaptive/behaviour.php question/behaviour/immediatefeedback/behaviour.php
          Hide
          Eloy Lafuente (stronk7) added a comment -

          ah, crap, they are static properties containing question_state subclass instances for easier comparison, sorry! I should be sleeping, grrr.

          Show
          Eloy Lafuente (stronk7) added a comment - ah, crap, they are static properties containing question_state subclass instances for easier comparison, sorry! I should be sleeping, grrr.
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Bah forget all my comments in this issue.

          I've tried all the behaviours and seem to work as expected, so I'm passing this.

          BUT:

          • I don't like that "this submission attracted a penalty of 0.3333", because 1) is false, it has not attracted anything, 2) is completely confusing, surely not for used teacher, but for students and for me.
          • I don't like the "answered/not completed" thingies. Once again, if I'm doing one quiz, with all the stress and so on, I expect one nice screen at the end telling me: "Alright, you've already answered everything so don't worry". If I see "unanswered/not completed" phrases, I cannot do anything but getting nervous.

          Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - Bah forget all my comments in this issue. I've tried all the behaviours and seem to work as expected, so I'm passing this. BUT: I don't like that "this submission attracted a penalty of 0.3333", because 1) is false, it has not attracted anything, 2) is completely confusing, surely not for used teacher, but for students and for me. I don't like the "answered/not completed" thingies. Once again, if I'm doing one quiz, with all the stress and so on, I expect one nice screen at the end telling me: "Alright, you've already answered everything so don't worry". If I see "unanswered/not completed" phrases, I cannot do anything but getting nervous. Ciao
          Hide
          Eloy Lafuente (stronk7) added a comment -

          U.P.S.T.R.E.A.M

          Show
          Eloy Lafuente (stronk7) added a comment - U.P.S.T.R.E.A.M
          Hide
          Tim Hunt added a comment -

          Before complaining about any aspect of the Adaptive behaviour, please try the same sequence of actions in Moodle 2.0 or 1.9. The point of Adaptive behaviour in 2.1 is to be backwards compatible with older versions, however crazy they were.

          If you want non-crazy behaviour, use Interactive with multiple tries behaviour. At least that is the OU policy.

          Actually, I wonder if I should change the default order of the behaviours to be Deferred feedback, Interactive, then the rest? Alternatively, if I was Petr, I would rename Adaptive to Legacy Adaptive

          Show
          Tim Hunt added a comment - Before complaining about any aspect of the Adaptive behaviour, please try the same sequence of actions in Moodle 2.0 or 1.9. The point of Adaptive behaviour in 2.1 is to be backwards compatible with older versions, however crazy they were. If you want non-crazy behaviour, use Interactive with multiple tries behaviour. At least that is the OU policy. Actually, I wonder if I should change the default order of the behaviours to be Deferred feedback, Interactive, then the rest? Alternatively, if I was Petr, I would rename Adaptive to Legacy Adaptive
          Hide
          Michael de Raadt added a comment -

          I'm still not sure the feedback on adaptive questions is consistent and correct.

          I tried a three-question adaptive quiz to test MDLQA-1095. For the first two questions, after answering incorrectly then correctly, I got the response I expected.

          Marks for this submission: 1.00/1.00. With previous penalties this gives 0.90/1.00. This submission attracted a penalty of 0.1000000.

          On the third and final question (a true/false question) I received the following feedback.

          Marks for this submission: 1.00/1.00. With previous penalties this gives 0.00/1.00. This submission attracted a penalty of 1.0000000.

          And when I was shown the summary at the end of the quiz, all questions were shown as receiving 0.90 marks. So I believe it is calculating the marks correctly, but this final piece of question feedback was incorrect.

          This incorrect feedback was also shown during the quiz review.

          Show
          Michael de Raadt added a comment - I'm still not sure the feedback on adaptive questions is consistent and correct. I tried a three-question adaptive quiz to test MDLQA-1095 . For the first two questions, after answering incorrectly then correctly, I got the response I expected. Marks for this submission: 1.00/1.00. With previous penalties this gives 0.90/1.00. This submission attracted a penalty of 0.1000000. On the third and final question (a true/false question) I received the following feedback. Marks for this submission: 1.00/1.00. With previous penalties this gives 0.00/1.00. This submission attracted a penalty of 1.0000000. And when I was shown the summary at the end of the quiz, all questions were shown as receiving 0.90 marks. So I believe it is calculating the marks correctly, but this final piece of question feedback was incorrect. This incorrect feedback was also shown during the quiz review.
          Hide
          Tim Hunt added a comment -

          OK, please create a new bug report with your last comment, and I will take another look. Thanks.

          Show
          Tim Hunt added a comment - OK, please create a new bug report with your last comment, and I will take another look. Thanks.
          Hide
          Helen Foster added a comment -

          Michael's comment copied and pasted to MDL-28024.

          Show
          Helen Foster added a comment - Michael's comment copied and pasted to MDL-28024 .
          Hide
          Steve Turley added a comment -

          This still seems to be an unresolved issue. See MDL-28219.

          Show
          Steve Turley added a comment - This still seems to be an unresolved issue. See MDL-28219 .
          Hide
          Steve Turley added a comment -

          This still seems to be an unresolved issue. See MDL-28219.

          Show
          Steve Turley added a comment - This still seems to be an unresolved issue. See MDL-28219 .

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: