Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-27916

Incorrect feedback to students on adaptive questions in Moodle 2.1

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1
    • Fix Version/s: 2.1
    • Component/s: Quiz
    • Labels:
    • Testing Instructions:
      Hide

      Attempt quizzes in each of the adaptive and immediate feedback modes (four modes in total).

      (It may be simplest to try this with a one-question quiz. I suggest a single multiple choice question with choices correct, partially correct and incorrect, that do what they say on the tin.)

      Try various sequences of actions where you choose one of the options, and then either click check to have the response graded, or click next which just saves the response without grading it.

      Verify that the description of the question's current state, which appears to the left of the question under the question number, and also on the quiz summary page, makes sense whatever state the question is in.

      You may with to compare the way these these four modes respond with the the Interactive with multiple tries mode, which is used heavily at the OU, and so has been tested and scrutinised closely.

      Show
      Attempt quizzes in each of the adaptive and immediate feedback modes (four modes in total). (It may be simplest to try this with a one-question quiz. I suggest a single multiple choice question with choices correct, partially correct and incorrect, that do what they say on the tin.) Try various sequences of actions where you choose one of the options, and then either click check to have the response graded, or click next which just saves the response without grading it. Verify that the description of the question's current state, which appears to the left of the question under the question number, and also on the quiz summary page, makes sense whatever state the question is in. You may with to compare the way these these four modes respond with the the Interactive with multiple tries mode, which is used heavily at the OU, and so has been tested and scrutinised closely.
    • Workaround:
      Hide

      Ignore the messages. It works okay, it just tells the students the wrong status.

      Show
      Ignore the messages. It works okay, it just tells the students the wrong status.
    • URL:
      qa.moodle.net
    • Affected Branches:
      MOODLE_21_STABLE
    • Fixed Branches:
      MOODLE_21_STABLE
    • Pull from Repository:
    • Pull Master Branch:

      Description

      If you set up an adaptive question and answer it correctly the first time (I used a multiple choice question), it marks it correct, but gives the student a message saying "this submission attracted a penalty of 0.3333." In fact the penalty was not actually assessed, but the student might think that was the case.

      If the student initially answered the question incorrectly, it correctly assesses the penalty and gives the right message. However, when the question is answered correctly on a subsequent try it still tells the student a penalty was attracted (even though it says the answer was correct). As in the previous case, it doesn't really extract the penalty, it only says it did.

      Another test I did was to submit the answer to a question in adaptive mode without first clicking on the check button. Moodle correctly proceeds to grade the question, but the Summary of attempt screen lists the question status as "Not yet answered." This could confuse and worry students as well. If the student proceeds to submit the quiz anyway, the question is appropriately graded.

      This final problem happens for a number of other choices in the "How questions behave" drop-down menu as well. I noticed it with Adaptive Mode (nopenalties) and Immediate feedback as well. It isn't a problem in deferred feedback mode. There is correctly reports the status "Answer saved."

        Gliffy Diagrams

          Issue Links

            Activity

            Hide
            timhunt Tim Hunt added a comment -

            Regarding the "this submission attracted a penalty of 0.3333." message. Please compare this with Moodle 2.0. You will find it has always been like that, and we do not want to change something people are familiar with.

            (Historical note: It was the OU's dissatisfaction with how Adaptive mode has always worked that lead us to create Interactive with multiple tries. And it was our desire to get that into Moodle core that lead to the whole new Question engine project.)

            The "Not yet answered" state is a definite issue. I will address that. However, it is not completely clear what the correct behaviour is. For example, the whole point of Immediate feedback mode is that the student clicks the Check button during the quiz attempt. Sure, we grade any response they have entered when they click Submit all and finish, but that is really just a fallback. Perhaps changing Not yet answered to Not complete is a good solution.

            Show
            timhunt Tim Hunt added a comment - Regarding the "this submission attracted a penalty of 0.3333." message. Please compare this with Moodle 2.0. You will find it has always been like that, and we do not want to change something people are familiar with. (Historical note: It was the OU's dissatisfaction with how Adaptive mode has always worked that lead us to create Interactive with multiple tries. And it was our desire to get that into Moodle core that lead to the whole new Question engine project.) The "Not yet answered" state is a definite issue. I will address that. However, it is not completely clear what the correct behaviour is. For example, the whole point of Immediate feedback mode is that the student clicks the Check button during the quiz attempt. Sure, we grade any response they have entered when they click Submit all and finish, but that is really just a fallback. Perhaps changing Not yet answered to Not complete is a good solution.
            Hide
            timhunt Tim Hunt added a comment -

            Just to note that Interactive with multiple tries was already using 'Not complete' to describe this state, and we reached that point at the OU after some discussion during testing, so I have made the adaptive and immediate modes work the same way.

            Show
            timhunt Tim Hunt added a comment - Just to note that Interactive with multiple tries was already using 'Not complete' to describe this state, and we reached that point at the OU after some discussion during testing, so I have made the adaptive and immediate modes work the same way.
            Hide
            samhemelryk Sam Hemelryk added a comment -

            Thanks Tim - integrated now.

            Show
            samhemelryk Sam Hemelryk added a comment - Thanks Tim - integrated now.
            Hide
            stronk7 Eloy Lafuente (stronk7) added a comment -

            Tim, was testing MDL-24594 and got problems in step 3 of its instructions:

            • Reviewing the question, if i pick the correct (100% grade) answer, I got: "Marks for this submission: 1.00/1.00. This submission attracted a penalty of 0.3333333.".
            • Reviewing the question, if I pick one partially-correct (5% grade) answer, I got: "Marks for this submission: 0.05/1.00. With previous penalties this gives 1.00/1.00. This submission attracted a penalty of 0.3333333."

            That seems to be exactly the sort of problem fixed by this, isn't it?

            Show
            stronk7 Eloy Lafuente (stronk7) added a comment - Tim, was testing MDL-24594 and got problems in step 3 of its instructions: Reviewing the question, if i pick the correct (100% grade) answer, I got: "Marks for this submission: 1.00/1.00. This submission attracted a penalty of 0.3333333.". Reviewing the question, if I pick one partially-correct (5% grade) answer, I got: "Marks for this submission: 0.05/1.00. With previous penalties this gives 1.00/1.00. This submission attracted a penalty of 0.3333333." That seems to be exactly the sort of problem fixed by this, isn't it?
            Hide
            stronk7 Eloy Lafuente (stronk7) added a comment - - edited

            It's really later here 02:06, so perhaps I'm just tired but I was looking to the code changes, just to see what changed, try to affine the testing and found this:

            public function get_state_string($showcorrectness) {
                $state = $this->qa->get_state()
                if ($state == question_state::$todo) {
                ....
                ....

            $todo ????

            both in:

            • question/behaviour/adaptive/behaviour.php
            • question/behaviour/immediatefeedback/behaviour.php
            Show
            stronk7 Eloy Lafuente (stronk7) added a comment - - edited It's really later here 02:06, so perhaps I'm just tired but I was looking to the code changes, just to see what changed, try to affine the testing and found this: public function get_state_string($showcorrectness) { $state = $this->qa->get_state() if ($state == question_state::$todo) { .... .... $todo ???? both in: question/behaviour/adaptive/behaviour.php question/behaviour/immediatefeedback/behaviour.php
            Hide
            stronk7 Eloy Lafuente (stronk7) added a comment -

            ah, crap, they are static properties containing question_state subclass instances for easier comparison, sorry! I should be sleeping, grrr.

            Show
            stronk7 Eloy Lafuente (stronk7) added a comment - ah, crap, they are static properties containing question_state subclass instances for easier comparison, sorry! I should be sleeping, grrr.
            Hide
            stronk7 Eloy Lafuente (stronk7) added a comment -

            Bah forget all my comments in this issue.

            I've tried all the behaviours and seem to work as expected, so I'm passing this.

            BUT:

            • I don't like that "this submission attracted a penalty of 0.3333", because 1) is false, it has not attracted anything, 2) is completely confusing, surely not for used teacher, but for students and for me.
            • I don't like the "answered/not completed" thingies. Once again, if I'm doing one quiz, with all the stress and so on, I expect one nice screen at the end telling me: "Alright, you've already answered everything so don't worry". If I see "unanswered/not completed" phrases, I cannot do anything but getting nervous.

            Ciao

            Show
            stronk7 Eloy Lafuente (stronk7) added a comment - Bah forget all my comments in this issue. I've tried all the behaviours and seem to work as expected, so I'm passing this. BUT: I don't like that "this submission attracted a penalty of 0.3333", because 1) is false, it has not attracted anything, 2) is completely confusing, surely not for used teacher, but for students and for me. I don't like the "answered/not completed" thingies. Once again, if I'm doing one quiz, with all the stress and so on, I expect one nice screen at the end telling me: "Alright, you've already answered everything so don't worry". If I see "unanswered/not completed" phrases, I cannot do anything but getting nervous. Ciao
            Hide
            stronk7 Eloy Lafuente (stronk7) added a comment -

            U.P.S.T.R.E.A.M

            Show
            stronk7 Eloy Lafuente (stronk7) added a comment - U.P.S.T.R.E.A.M
            Hide
            timhunt Tim Hunt added a comment -

            Before complaining about any aspect of the Adaptive behaviour, please try the same sequence of actions in Moodle 2.0 or 1.9. The point of Adaptive behaviour in 2.1 is to be backwards compatible with older versions, however crazy they were.

            If you want non-crazy behaviour, use Interactive with multiple tries behaviour. At least that is the OU policy.

            Actually, I wonder if I should change the default order of the behaviours to be Deferred feedback, Interactive, then the rest? Alternatively, if I was Petr, I would rename Adaptive to Legacy Adaptive

            Show
            timhunt Tim Hunt added a comment - Before complaining about any aspect of the Adaptive behaviour, please try the same sequence of actions in Moodle 2.0 or 1.9. The point of Adaptive behaviour in 2.1 is to be backwards compatible with older versions, however crazy they were. If you want non-crazy behaviour, use Interactive with multiple tries behaviour. At least that is the OU policy. Actually, I wonder if I should change the default order of the behaviours to be Deferred feedback, Interactive, then the rest? Alternatively, if I was Petr, I would rename Adaptive to Legacy Adaptive
            Hide
            salvetore Michael de Raadt added a comment -

            I'm still not sure the feedback on adaptive questions is consistent and correct.

            I tried a three-question adaptive quiz to test MDLQA-1095. For the first two questions, after answering incorrectly then correctly, I got the response I expected.

            Marks for this submission: 1.00/1.00. With previous penalties this gives 0.90/1.00. This submission attracted a penalty of 0.1000000.

            On the third and final question (a true/false question) I received the following feedback.

            Marks for this submission: 1.00/1.00. With previous penalties this gives 0.00/1.00. This submission attracted a penalty of 1.0000000.

            And when I was shown the summary at the end of the quiz, all questions were shown as receiving 0.90 marks. So I believe it is calculating the marks correctly, but this final piece of question feedback was incorrect.

            This incorrect feedback was also shown during the quiz review.

            Show
            salvetore Michael de Raadt added a comment - I'm still not sure the feedback on adaptive questions is consistent and correct. I tried a three-question adaptive quiz to test MDLQA-1095 . For the first two questions, after answering incorrectly then correctly, I got the response I expected. Marks for this submission: 1.00/1.00. With previous penalties this gives 0.90/1.00. This submission attracted a penalty of 0.1000000. On the third and final question (a true/false question) I received the following feedback. Marks for this submission: 1.00/1.00. With previous penalties this gives 0.00/1.00. This submission attracted a penalty of 1.0000000. And when I was shown the summary at the end of the quiz, all questions were shown as receiving 0.90 marks. So I believe it is calculating the marks correctly, but this final piece of question feedback was incorrect. This incorrect feedback was also shown during the quiz review.
            Hide
            timhunt Tim Hunt added a comment -

            OK, please create a new bug report with your last comment, and I will take another look. Thanks.

            Show
            timhunt Tim Hunt added a comment - OK, please create a new bug report with your last comment, and I will take another look. Thanks.
            Hide
            tsala Helen Foster added a comment -

            Michael's comment copied and pasted to MDL-28024.

            Show
            tsala Helen Foster added a comment - Michael's comment copied and pasted to MDL-28024 .
            Hide
            rsturley Steve Turley added a comment -

            This still seems to be an unresolved issue. See MDL-28219.

            Show
            rsturley Steve Turley added a comment - This still seems to be an unresolved issue. See MDL-28219 .
            Hide
            rsturley Steve Turley added a comment -

            This still seems to be an unresolved issue. See MDL-28219.

            Show
            rsturley Steve Turley added a comment - This still seems to be an unresolved issue. See MDL-28219 .

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  1/Jul/11