Moodle
  1. Moodle
  2. MDL-1647

Allowing negative marks for questions

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.5
    • Fix Version/s: 2.1
    • Component/s: Quiz
    • Labels:
      None
    • Environment:
      All
    • Affected Branches:
      MOODLE_14_STABLE
    • Fixed Branches:
      MOODLE_21_STABLE
    • Rank:
      1514

      Description

      I needed a way to simulate Advanced Placement multiple choice tests in which incorrect answers deduct .25% from the total score of the test.

      I made two very minor changes to mod/quiz/lib.php to do this, and it seems to work.

      I only tested it on multiple choice (with both one and multiple answers), and Cloze questions (pull-down style only). For my purposes, I made the correct answer be 100%, and all incorrect choices be -25%. This with the changes in lib.php seem to work.

      This thread describes my changes, and Martin asked that I report it as a bug.

      http://moodle.org/mod/forum/discuss.php?d=9704

        Issue Links

          Activity

          Hide
          Martin Dougiamas added a comment -

          From Gustav Delius (gwd2 at york.ac.uk) Thursday, 30 December 2004, 07:08 PM:

          Hi Jill,

          if you still haven't observed any negative side effects of this hack then I would like to put it into Moodle 1.5.

          Gustav

          From Gustav Delius (gwd2 at york.ac.uk) Saturday, 7 May 2005, 01:39 AM:

          Ah, I almost forgot about this. Our new code again rules out negative marks for questions. I am not sure why, it just seemed natural to us. I will see if I find the time to fix that, it shouldn't be too difficult. I am upgrading the priority on this.

          From Gustav Delius (gwd2 at york.ac.uk) Thursday, 23 February 2006, 05:49 PM:

          This subject has come up again on the forums, see for example http://moodle.org/mod/forum/discuss.php?d=40310&parent=185678

          From Gustav Delius (gwd2 at york.ac.uk) Thursday, 23 February 2006, 05:51 PM:

          Also see http://moodle.org/mod/forum/discuss.php?d=40128.

          We could implement another quiz option to allow negative marks for questions.

          From John Isner (john.isner at gmail.com) Tuesday, 11 July 2006, 08:04 PM:

          Standardized tests such as the SAT deduct 1/4 point for each wrong answer to a MCQ. It is very difficult to simulate those tests in moodle without this feature.

          Show
          Martin Dougiamas added a comment - From Gustav Delius (gwd2 at york.ac.uk) Thursday, 30 December 2004, 07:08 PM: Hi Jill, if you still haven't observed any negative side effects of this hack then I would like to put it into Moodle 1.5. Gustav From Gustav Delius (gwd2 at york.ac.uk) Saturday, 7 May 2005, 01:39 AM: Ah, I almost forgot about this. Our new code again rules out negative marks for questions. I am not sure why, it just seemed natural to us. I will see if I find the time to fix that, it shouldn't be too difficult. I am upgrading the priority on this. From Gustav Delius (gwd2 at york.ac.uk) Thursday, 23 February 2006, 05:49 PM: This subject has come up again on the forums, see for example http://moodle.org/mod/forum/discuss.php?d=40310&parent=185678 From Gustav Delius (gwd2 at york.ac.uk) Thursday, 23 February 2006, 05:51 PM: Also see http://moodle.org/mod/forum/discuss.php?d=40128 . We could implement another quiz option to allow negative marks for questions. From John Isner (john.isner at gmail.com) Tuesday, 11 July 2006, 08:04 PM: Standardized tests such as the SAT deduct 1/4 point for each wrong answer to a MCQ. It is very difficult to simulate those tests in moodle without this feature.
          Hide
          Paulo Matos added a comment -

          This patch allows multichoice and multianswer questiontypes to have negative marks. I choose this two types because it seemd to me the most logical ones. The total sum of all questions on a quiz is set to zero if negative.

          Show
          Paulo Matos added a comment - This patch allows multichoice and multianswer questiontypes to have negative marks. I choose this two types because it seemd to me the most logical ones. The total sum of all questions on a quiz is set to zero if negative.
          Hide
          Tim Hunt added a comment -

          Some feedback on the patch:

          The reason this bug is non-trivial is that you have to consider what happens in adaptive mode as well as non-adaptive mode. Your patch does not consider that.

          Also, "//paulo.matos: ..." comments are not helpful. CVS tracks who changed what, so names in comments just clutter the code. So only use comments to describe what the code is doing, like "// Make sure sumgrades are non-negative."

          Comments like "// This should not be here, ..." do not inspire confidence (questionlib line ~1060). In fact, this comment confused me. It was clearer to me what was going on when I ignored the comment and worked out what the code is doing.

          I think that any patch that fixes this should treat all question types the same. This is a pain becuase the

          Show
          Tim Hunt added a comment - Some feedback on the patch: The reason this bug is non-trivial is that you have to consider what happens in adaptive mode as well as non-adaptive mode. Your patch does not consider that. Also, "//paulo.matos: ..." comments are not helpful. CVS tracks who changed what, so names in comments just clutter the code. So only use comments to describe what the code is doing, like "// Make sure sumgrades are non-negative." Comments like "// This should not be here, ..." do not inspire confidence (questionlib line ~1060). In fact, this comment confused me. It was clearer to me what was going on when I ignored the comment and worked out what the code is doing. I think that any patch that fixes this should treat all question types the same. This is a pain becuase the
          Hide
          Tim Hunt added a comment -

          (Dratted thing. I had not finished. Why did it suddenly submit my comment when I hit return, to continue ...)

          I think that any patch that fixes this should treat all question types the same. This is a pain because the

          $state->raw_grade = min(max((float) $state->raw_grade, 0.0), 1.0)

          • $question->maxgrade;

          line of code is copied and pasted into each question type. Probably we need to start by making a method in the base class that applies this logic, and then call that function from each question type. Then we can fix this bug in one place.

          So, a good attempt to fix this bug, but it needs more work before it can be committed to core.

          Also, I feel that this counts as a new feature, so I would be reluctant to add it to the 1.6 stable branch. I would rather put in into HEAD. By all means you can make a patch available to people who want to patch their own 1.6 installs, but for core, we need a patch against HEAD.

          Thanks for your help.

          Show
          Tim Hunt added a comment - (Dratted thing. I had not finished. Why did it suddenly submit my comment when I hit return, to continue ...) I think that any patch that fixes this should treat all question types the same. This is a pain because the $state->raw_grade = min(max((float) $state->raw_grade, 0.0), 1.0) $question->maxgrade; line of code is copied and pasted into each question type. Probably we need to start by making a method in the base class that applies this logic, and then call that function from each question type. Then we can fix this bug in one place. So, a good attempt to fix this bug, but it needs more work before it can be committed to core. Also, I feel that this counts as a new feature, so I would be reluctant to add it to the 1.6 stable branch. I would rather put in into HEAD. By all means you can make a patch available to people who want to patch their own 1.6 installs, but for core, we need a patch against HEAD. Thanks for your help.
          Hide
          Paulo Matos added a comment -

          > Also, "//paulo.matos: ..." comments are not helpful.
          Sorry for that I'll clean that on my next patches. I use it to track my changes while working on the patch.

          > Comments like "// This should not be here, ..." do not inspire confidence (questionlib line ~1060). In fact, this comment confused me. It was clearer to me what was going on when I ignored the comment and worked out what the code is doing.

          Ok, I might not expressed quite well on that comment! This comment was about the block code, that is runned every time, even if does not have penalties or timelimit constraints. It should be like a note to the merger. I'll post a new version of the patch with it more clear (hopefully).

          Show
          Paulo Matos added a comment - > Also, "//paulo.matos: ..." comments are not helpful. Sorry for that I'll clean that on my next patches. I use it to track my changes while working on the patch. > Comments like "// This should not be here, ..." do not inspire confidence (questionlib line ~1060). In fact, this comment confused me. It was clearer to me what was going on when I ignored the comment and worked out what the code is doing. Ok, I might not expressed quite well on that comment! This comment was about the block code, that is runned every time, even if does not have penalties or timelimit constraints. It should be like a note to the merger. I'll post a new version of the patch with it more clear (hopefully).
          Hide
          Paulo Matos added a comment -

          Ok, here's the new version of the patch.

          The comments on the previous one were doubts about the block:
          // Ensure that the grade does not go down
          $state->grade = max($state->grade, $state->last_graded->grade);

          I concluded that this block is specific to penalties/timelimit/closing time, and coded accordingly.

          Cheers,

          Paulo

          Show
          Paulo Matos added a comment - Ok, here's the new version of the patch. The comments on the previous one were doubts about the block: // Ensure that the grade does not go down $state->grade = max($state->grade, $state->last_graded->grade); I concluded that this block is specific to penalties/timelimit/closing time, and coded accordingly. Cheers, Paulo
          Hide
          Paulo Matos added a comment -

          Tim> I think that any patch that fixes this should treat all question types the same.
          I choosed multichoice and multianswer questiontypes to have negative marks because their were the single one's where you can set a negative mark for an answer through the UI.

          Show
          Paulo Matos added a comment - Tim> I think that any patch that fixes this should treat all question types the same. I choosed multichoice and multianswer questiontypes to have negative marks because their were the single one's where you can set a negative mark for an answer through the UI.
          Hide
          Paulo Matos added a comment -

          Here goes the updated patch for 1.8.x, only multichoice and multianswer questions are treated because those are the only ones that allow setting a negative value through UI.

          Show
          Paulo Matos added a comment - Here goes the updated patch for 1.8.x, only multichoice and multianswer questions are treated because those are the only ones that allow setting a negative value through UI.
          Hide
          Paulo Matos added a comment -

          1.8.6+ version

          Show
          Paulo Matos added a comment - 1.8.6+ version
          Hide
          Jitendra Patel added a comment -

          I am also facing the Problem but i dont know how to install patch in my site

          if anybody can help me

          Show
          Jitendra Patel added a comment - I am also facing the Problem but i dont know how to install patch in my site if anybody can help me
          Hide
          Tim Hunt added a comment -

          Note: a proper solution to this issue is part of this proposal: http://docs.moodle.org/en/Development:Changing_the_Moodle_Question_Engine (There is currently no schedule for implementing it.)

          Show
          Tim Hunt added a comment - Note: a proper solution to this issue is part of this proposal: http://docs.moodle.org/en/Development:Changing_the_Moodle_Question_Engine (There is currently no schedule for implementing it.)
          Hide
          Amanda Doughty added a comment -

          Tim> I have been testing various different patches and have found that Vikram's solution: http://moodle.org/mod/forum/discuss.php?d=46862#p501237 mostly works for 1.9.8.

          I did take into account your comments in http://tracker.moodle.org/browse/MDL-5324 regarding:

          1. Multiple-response multiple-choice questions.
          2. When the quiz allows multiple attempts at each question using the penalty-factor system.

          Using this patch for questions using the penalty-factor system as well as adaptive mode, I found that unchanged submitted answers (using the individual submit button) were regraded and therefore penalised again when 'submit all and finish' is clicked.

          I have therefore added to Vikram's patch to resolve this and initial tests look good. We will carry out further testing, but as I am a relative newbie to Moodle development, I wondered if you would have a quick look and let us know if you spot any immediate flaws. It is a very small patch. I will attach it as negativemarking.diff

          Thanks

          Show
          Amanda Doughty added a comment - Tim> I have been testing various different patches and have found that Vikram's solution: http://moodle.org/mod/forum/discuss.php?d=46862#p501237 mostly works for 1.9.8. I did take into account your comments in http://tracker.moodle.org/browse/MDL-5324 regarding: 1. Multiple-response multiple-choice questions. 2. When the quiz allows multiple attempts at each question using the penalty-factor system. Using this patch for questions using the penalty-factor system as well as adaptive mode, I found that unchanged submitted answers (using the individual submit button) were regraded and therefore penalised again when 'submit all and finish' is clicked. I have therefore added to Vikram's patch to resolve this and initial tests look good. We will carry out further testing, but as I am a relative newbie to Moodle development, I wondered if you would have a quick look and let us know if you spot any immediate flaws. It is a very small patch. I will attach it as negativemarking.diff Thanks
          Hide
          Tim Hunt added a comment -

          Any change that touches the function question_process_responses in lib/questionlib.php is very dangerous. I know from painful experience that change here need a lot of testing to ensure that the do not introduce a subtle bug. I cannot, from a quick look at your patch, say with any certainty whether they are correct or not. They look plausible, and if you have tested and they work in all situations that you care about, then it should be safe for you to use them.

          However, the proper solution to this problem is some work I am currently doing that is destined to go into Moodle 2.1. It is described on http://docs.moodle.org/en/Development:Question_Engine_2, and the code which is mostly working, is available from http://github.com/timhunt/Moodle-Question-Engine-2. Since this work is underway, I am not going to do anything about this issue in Moodle 1.9 or 2.0.

          Show
          Tim Hunt added a comment - Any change that touches the function question_process_responses in lib/questionlib.php is very dangerous. I know from painful experience that change here need a lot of testing to ensure that the do not introduce a subtle bug. I cannot, from a quick look at your patch, say with any certainty whether they are correct or not. They look plausible, and if you have tested and they work in all situations that you care about, then it should be safe for you to use them. However, the proper solution to this problem is some work I am currently doing that is destined to go into Moodle 2.1. It is described on http://docs.moodle.org/en/Development:Question_Engine_2 , and the code which is mostly working, is available from http://github.com/timhunt/Moodle-Question-Engine-2 . Since this work is underway, I am not going to do anything about this issue in Moodle 1.9 or 2.0.
          Hide
          Amanda Doughty added a comment -

          Ok Tim. Thanks for the advice.

          Show
          Amanda Doughty added a comment - Ok Tim. Thanks for the advice.
          Hide
          Tim Hunt added a comment -

          Fixed in 2.1 dev now that MDL-20636 has been integrated.

          That only too 6.5 years

          Show
          Tim Hunt added a comment - Fixed in 2.1 dev now that MDL-20636 has been integrated. That only too 6.5 years
          Hide
          Matt Jenner added a comment -

          Am I right in thinking this retrospectively makes questions which didn't have negative marking to now have negative marking if some of the possible answers are using the negative parts of the scoring dropdown menu?

          I ask because we have noticed in our 2.2 instance that some questions are not behaving as they used to.

          Show
          Matt Jenner added a comment - Am I right in thinking this retrospectively makes questions which didn't have negative marking to now have negative marking if some of the possible answers are using the negative parts of the scoring dropdown menu? I ask because we have noticed in our 2.2 instance that some questions are not behaving as they used to.
          Hide
          Tim Hunt added a comment -

          If your multiple-choice single-response question had -ve marks for incorrect answers, it is true that those -ve marks were treated as 0 in 1.9 and 2.0, and will be used as the actual -ve value in 2.1+

          If this is a problem for you, then probably the only solution is to change the -ve scores to 0 in question answer. Something like

          UPDATE mdl_question_answers
          SET fraction = 0
          WHERE fraction < 0 AND EXISTS (
              SELECT 1
              FROM mdl_question_multichoice
              WHERE mdl_question_multichoice.question = mdl_question_answers.question
                  AND mdl_question_multichoice.single = 1
          )
          

          might let you do that in bulk. WARNING, that SQL is completely untested. Use at your own risk.

          Show
          Tim Hunt added a comment - If your multiple-choice single-response question had -ve marks for incorrect answers, it is true that those -ve marks were treated as 0 in 1.9 and 2.0, and will be used as the actual -ve value in 2.1+ If this is a problem for you, then probably the only solution is to change the -ve scores to 0 in question answer. Something like UPDATE mdl_question_answers SET fraction = 0 WHERE fraction < 0 AND EXISTS ( SELECT 1 FROM mdl_question_multichoice WHERE mdl_question_multichoice.question = mdl_question_answers.question AND mdl_question_multichoice.single = 1 ) might let you do that in bulk. WARNING, that SQL is completely untested. Use at your own risk.
          Hide
          Matt Jenner added a comment -

          Thank you Tim, we're going to look at this SQL patch and evaluate the risk of running such a query.

          A concern is that we found this and consider it an important issue to resolve. We use Moodle for summative examinations and would be worried that other institutions/users have not noticed this trait as they upgrade and test. We are naturally happy to share our outcomes via MDL-1647 but I wonder now if this needs to be more widely disseminated?

          Will report back once we've looked into this further.

          Show
          Matt Jenner added a comment - Thank you Tim, we're going to look at this SQL patch and evaluate the risk of running such a query. A concern is that we found this and consider it an important issue to resolve. We use Moodle for summative examinations and would be worried that other institutions/users have not noticed this trait as they upgrade and test. We are naturally happy to share our outcomes via MDL-1647 but I wonder now if this needs to be more widely disseminated? Will report back once we've looked into this further.
          Hide
          Tim Hunt added a comment -

          On the other hand, this issue was introduced about a year ago in Moodle 2.1, and you are the first people to raise this, which is some indication that what you have identified is quite a rare issue.

          Probably the way to investigate what the above SQL will do is to run the equivalent SELECT to see what the update would change. Something like

          SELECT *
          FROM mdl_question_answers
          JOIN mdl_question ON mdl_question_answers.question = mdl_question.id
          WHERE fraction < 0 AND EXISTS (
              SELECT 1
              FROM mdl_question_multichoice
              WHERE mdl_question_multichoice.question = mdl_question_answers.question
                  AND mdl_question_multichoice.single = 1
          )
          
          Show
          Tim Hunt added a comment - On the other hand, this issue was introduced about a year ago in Moodle 2.1, and you are the first people to raise this, which is some indication that what you have identified is quite a rare issue. Probably the way to investigate what the above SQL will do is to run the equivalent SELECT to see what the update would change. Something like SELECT * FROM mdl_question_answers JOIN mdl_question ON mdl_question_answers.question = mdl_question.id WHERE fraction < 0 AND EXISTS ( SELECT 1 FROM mdl_question_multichoice WHERE mdl_question_multichoice.question = mdl_question_answers.question AND mdl_question_multichoice.single = 1 )
          Hide
          Jessica Gramp added a comment -

          Just to let everyone know. When we ran the above (UPDATE) code to set all negatively marked answers in single-choice question types back to 0 we encountered problems with the regrading now giving students 0 (even on questions that don't have negative marking available like matching) and showing their attempts as "not answered".

          Because of this we are probably going to have to roll-back our database changes and look at identifying quiz editors to speak to them about the issue. We may need to look at manually changing the negatively marked answers instead of changing them automatically.

          Show
          Jessica Gramp added a comment - Just to let everyone know. When we ran the above (UPDATE) code to set all negatively marked answers in single-choice question types back to 0 we encountered problems with the regrading now giving students 0 (even on questions that don't have negative marking available like matching) and showing their attempts as "not answered". Because of this we are probably going to have to roll-back our database changes and look at identifying quiz editors to speak to them about the issue. We may need to look at manually changing the negatively marked answers instead of changing them automatically.

            People

            • Votes:
              28 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: