Moodle
  1. Moodle
  2. MDL-15233

Calculated Questions: Tolerance is applied inconsistently to datasets and to student responses

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.8
    • Fix Version/s: None
    • Component/s: Questions, Quiz
    • Labels:
      None
    • Environment:
      Not sure (not an administrator)
    • Affected Branches:
      MOODLE_18_STABLE
    • Rank:
      4420

      Description

      Summary:
      For a dataset answer, the relative tolerance is applied to the correct answer after rounding it (as specified by "Correct answer shows"). For grading, however, the relative tolerance is applied to the correct answer before rounding. This is problem is most likely to be noticed when the tolerance is low.

      Description:
      As an example, suppose that a dataset item is 3.23/10.1 = 0.31980198, and you specify that the "correct answer shows" 3 significant figures, and that the relative tolerance is 0.00001. In the "edit the datasets" page, it is displayed as 0.320. The (min,max) = (0.3199968, 0.3200032). According to the datasets page, the tolerance is applied to the value 0.320. Therefore entering the value "0.320" would be correct, but 0.3198 wold be incorrect.
      What happens when you preview the questions? The tolerance is applied to the value 0.31980198, so (min,max) = (0.319798782, 0.319805178). As a result, entering 0.320 is incorrect.
      This is a bug because the user (teacher) cannot rely on the information in the datasets page to reflect how the question is graded. Personally, I prefer the way that the datasets page applies the tolerance, because in analytical chemistry, reporting extra decimals is incorrect. But I can see that other users would prefer the latter way of applying tolerances. Either way, the tolerances should be applied consistently.

      Reproduce:
      Define a new calculated question with the variable

      {a}. Set:
      "Correct Answer formula =" : {a}

      /10.1
      "Tolerance ± " : 0.00001
      "Tolerance Type" : relative
      "Correct answer shows" : 3
      "Format" : significant figures

      On the next page, select "will use the same existing private dataset as before"

      On the next page, add a new item:
      "Param

      {a}" : 3.23
      "Range of Values" : 1.01 - 8.01
      "Decimal Places" : 2
      "Distribution" : uniform

      Add this item, and you will get:
      Param {a}

      {a}

      /10.1 3.23/10.1 = 0.320
      Min: 0.3199968---Max: 0.3200032

      Actual Results:
      Save changes. Now preview the question. Do the following tests:
      1. Enter the answer "0.320" and click submit (I did this in adaptive mode). It will be marked incorrect. Click start over.
      2. Click Fill with correct. The answer box is filled with "0.320." Click submit. Again, it is marked incorrect. Click start over.
      3. Enter the answer "0.319802" and click submit. It will be marked correct.

      In doing this, did you notice another bug? When it is marked incorrect in tests 1 and 2, it gets a green check mark! The inattentive student thinks it was correct, but got no points! When it is marked correct in test 3, it gets a red x! Now do a final test:

      4. Enter the answer "1000" and click submit. It will be marked incorrect. In addition, it gets a red x, as expected.

      Evidently, the visual feedback (check and x) are decided by the datasets (min,max), not by the grading (min,max).

      Expected Results:
      I expected that any response between the min and max (as listed in the datasets page) would be marked correct. Someone else expected it, too, because the checkmark and x match this expectation. The reason I prefer this approach is that I am teaching analytical chemistry, so the datasets (min,max) marks students wrong if they report more digits than they should. (The current, grader's behavior rewards students who report too many digits). I understand if other users prefer the current behavior, but then the datasets page should reflect this behavior consistently.

      Kenyon College recently upgraded to Moodle 1.8, and I have only tried this out on 1.8, so I don't know if this was occurring on 1.7. Sorry I can't give any information about server environment, as I am a user not administrator. I am sending this information to the administrators as well – maybe they can chime in with the appropriate information.

      This issue is similar to this problem: MDL-6879. In this case, however, the problem is not a misunderstanding of tolerance. It is that there are two ways to apply tolerance, and they are not applied consistently by different parts of the software.

      1. qtype_datasetdependent.php
        3 kB
        Pierre Pichet
      2. test.xml
        2 kB
        Simon Garcia
      1. Dataset page.jpg
        22 kB
      2. Define question.jpg
        31 kB
      3. screenshot-1.jpg
        70 kB
      4. screenshot-2.jpg
        97 kB
      5. screenshot-3.jpg
        152 kB
      6. screenshot-4.jpg
        144 kB
      7. screenshot-5.jpg
        137 kB
      8. screenshot-6.jpg
        250 kB
      9. screenshot-7.jpg
        153 kB

        Issue Links

          Activity

          Hide
          Pierre Pichet added a comment -

          Your observations are correct and I am working on that these days.
          I am just completing the tests before CVS.
          I will comment more later.

          Show
          Pierre Pichet added a comment - Your observations are correct and I am working on that these days. I am just completing the tests before CVS. I will comment more later.
          Hide
          Pierre Pichet added a comment -

          You can see the actual results on
          http://132.208.141.198/moodle_head/
          log as user:moodle pw:moodle
          go to course TesHEAD and look at question "test 15233".
          Create your own questions if you want ot test further

          Show
          Pierre Pichet added a comment - You can see the actual results on http://132.208.141.198/moodle_head/ log as user:moodle pw:moodle go to course TesHEAD and look at question "test 15233". Create your own questions if you want ot test further
          Hide
          Pierre Pichet added a comment -

          With the significant figures and the tolerance limits setted, the Correct response is outside.
          The next thing to do is to modify the edit_datsetitems_form.php to detect these errors and to allow the teacher to modify them probably by using extended_details options of moodle forms.
          I will CVs thie actual modifications in the next two days but the edit_datsetitems_form.php will be done in august on return from my holidays...

          Show
          Pierre Pichet added a comment - With the significant figures and the tolerance limits setted, the Correct response is outside. The next thing to do is to modify the edit_datsetitems_form.php to detect these errors and to allow the teacher to modify them probably by using extended_details options of moodle forms. I will CVs thie actual modifications in the next two days but the edit_datsetitems_form.php will be done in august on return from my holidays...
          Hide
          Pierre Pichet added a comment -

          I propose that before edit_dataset_item_form is changed on Moodle version where it exists that the comparison between the Correct response and the true value is shown even when it is OK.
          This will help users diferentiate between the two parameters even if it give a more crowdy interface in the case of a large number of dataset items.

          Show
          Pierre Pichet added a comment - I propose that before edit_dataset_item_form is changed on Moodle version where it exists that the comparison between the Correct response and the true value is shown even when it is OK. This will help users diferentiate between the two parameters even if it give a more crowdy interface in the case of a large number of dataset items.
          Hide
          Simon Garcia added a comment -

          Hello Pierre,

          Thank you for looking at this issue. I looked at the test 15233 question. So now the tolerance is applied to the true value, and the "edit dataset items" form will generate a warning when the Correct Response is outside of the range. Then the teacher will realize the conflict between the tolerance and the significant figures, and will go back and change those parameters. Did I interpret it correctly?

          Cheers,
          Simon

          Show
          Simon Garcia added a comment - Hello Pierre, Thank you for looking at this issue. I looked at the test 15233 question. So now the tolerance is applied to the true value, and the "edit dataset items" form will generate a warning when the Correct Response is outside of the range. Then the teacher will realize the conflict between the tolerance and the significant figures, and will go back and change those parameters. Did I interpret it correctly? Cheers, Simon
          Hide
          Pierre Pichet added a comment -

          You interpret it correctly.

          Show
          Pierre Pichet added a comment - You interpret it correctly.
          Hide
          Pierre Pichet added a comment -

          Adding the answers tolerances so that the user can modify is there is a correct response outside the min-max.
          Will use the extended details process (not shown here)

          Show
          Pierre Pichet added a comment - Adding the answers tolerances so that the user can modify is there is a correct response outside the min-max. Will use the extended details process (not shown here)
          Hide
          Pierre Pichet added a comment -

          with advance on

          Show
          Pierre Pichet added a comment - with advance on
          Hide
          Pierre Pichet added a comment -

          advanced hide

          Show
          Pierre Pichet added a comment - advanced hide
          Hide
          Pierre Pichet added a comment -

          CVS a first version of adding tolerance edition in datasetitems)form.php.
          Need further work to handle well all the different combination between generating next item, tolerance modifications, adding and deleting.
          Contrary to other forms and following "historical" practice, most of the dataitems parameters are modified in the database before saving or should we say "ending edition".
          The regenerate and reuse previous value need to be clarify in the display.
          The "Item to add" should indicate what is the state of the value displayed i.e regenerate or reuse or reused modified...

          Show
          Pierre Pichet added a comment - CVS a first version of adding tolerance edition in datasetitems)form.php. Need further work to handle well all the different combination between generating next item, tolerance modifications, adding and deleting. Contrary to other forms and following "historical" practice, most of the dataitems parameters are modified in the database before saving or should we say "ending edition". The regenerate and reuse previous value need to be clarify in the display. The "Item to add" should indicate what is the state of the value displayed i.e regenerate or reuse or reused modified...
          Hide
          Pierre Pichet added a comment -

          Finally the "update datasets parameters" and "updatatolerance" are non submit.
          The first submit element put in a moodle form is the one that is send if the user type enter in any text field.
          The tolerance select elements do not generate a submit so we need the "updatatolerance".
          Updating the tolerance is also done whenever the user type enter.
          This is need to ensure coherence of the data all over the page.
          the add 1 item always add as the first the "item to add" independently of the reuse or generate setting.
          These setting work for the next item added i.e. when adding 10 or more.

          Will do more tests and try to have the 1.9 version for next week

          Show
          Pierre Pichet added a comment - Finally the "update datasets parameters" and "updatatolerance" are non submit. The first submit element put in a moodle form is the one that is send if the user type enter in any text field. The tolerance select elements do not generate a submit so we need the "updatatolerance". Updating the tolerance is also done whenever the user type enter. This is need to ensure coherence of the data all over the page. the add 1 item always add as the first the "item to add" independently of the reuse or generate setting. These setting work for the next item added i.e. when adding 10 or more. Will do more tests and try to have the 1.9 version for next week
          Hide
          Pierre Pichet added a comment -

          message when there is at least one correct answer that is not within true answer limit.
          The user has to correct the error if he want ot save the values or add or delete data items.

          Show
          Pierre Pichet added a comment - message when there is at least one correct answer that is not within true answer limit. The user has to correct the error if he want ot save the values or add or delete data items.
          Hide
          Pierre Pichet added a comment -

          Tim,
          Do you agree with the answers tolerance parameters interface texts so I can put everything in 1.9.

          Show
          Pierre Pichet added a comment - Tim, Do you agree with the answers tolerance parameters interface texts so I can put everything in 1.9.
          Hide
          Tim Hunt added a comment -

          That looks great, except that some of the error messages look like the were written by someone who is not a native English speaker . If you would like to attach the language file here, I'll have a go at rewording them.

          Show
          Tim Hunt added a comment - That looks great, except that some of the error messages look like the were written by someone who is not a native English speaker . If you would like to attach the language file here, I'll have a go at rewording them.
          Hide
          Pierre Pichet added a comment -

          Tim,
          I have done the necessary changes in 1.9 (and testing) and I am ready to CVs to 1.9 if this Ok for you.

          Show
          Pierre Pichet added a comment - Tim, I have done the necessary changes in 1.9 (and testing) and I am ready to CVs to 1.9 if this Ok for you.
          Hide
          Pierre Pichet added a comment -

          They have been CVS to HEAD
          RCS file: /cvsroot/moodle/moodle/lang/en_utf8/qtype_calculated.php,v
          retrieving revision 1.7
          retrieving revision 1.8
          diff -u -r1.7 -r1.8
          — qtype_calculated.php 20 Apr 2007 02:43:28 -0000 1.7
          +++ qtype_calculated.php 16 Jun 2008 03:15:55 -0000 1.8
          @@ -20,6 +20,8 @@
          $string['nosharedwildcard'] = 'No shared wild card in this category';
          $string['possiblehdr']='Possible wild cards present only in the question text';
          $string['tolerance'] = 'Tolerance ±';
          +$string['trueanswerinsidelimits'] = ' Correct answer : $a->correct inside limits of true value $a->true';
          +$string['trueansweroutsidelimits'] = '<span class=\"error\">ERROR Correct answer : $a->correct outside limits of true value $a->true</span>';
          $string['updatecategory'] = 'Update the category';
          $string['usedinquestion']='Used in Question';
          $string['youmustenteramultiplierhere'] = 'You must enter a multiplier here.';

          and

          RCS file: /cvsroot/moodle/moodle/lang/en_utf8/qtype_datasetdependent.php,v
          retrieving revision 1.4
          retrieving revision 1.5
          diff -u -r1.4 -r1.5
          — qtype_datasetdependent.php 22 May 2007 02:23:50 -0000 1.4
          +++ qtype_datasetdependent.php 25 Sep 2008 00:18:09 -0000 1.5
          @@ -1,5 +1,6 @@
          <?php
          $string['additem'] = 'Add Item';
          +$string['answerstoleranceparam'] = 'Answers tolerance parameters';
          $string['atleastonerealdataset']='There should be at least one real dataset in question text';
          $string['atleastonewildcard']='There should be at least one wild card in answer formula or question text';
          $string['calcdistribution'] = 'Distribution';
          @@ -34,12 +35,14 @@
          $string['newlocal2'] = 'a file from a new set of files that will only be used by this question';
          $string['newlocal3'] = 'a link from a new set of links that will only be used by this question';
          $string['nodataset'] = 'nothing - it is not a wild card';
          +$string['oneanswertrueansweroutsidelimits'] = 'At least one correct answer outside the true value limits.<br/>Modify the answers tolerance settings available as Advanced parameters';
          $string['param'] = 'Param

          {<strong>$a</strong>}

          ';
          $string['replacewithrandom'] = 'Replace with a random value';
          $string['reuseifpossible'] = 'reuse previous value if available';
          $string['sharedwildcard']='Shared wild card';
          $string['sharedwildcards']='Shared wild cards';
          $string['uniform'] = 'Uniform';
          +$string['updatetolerancesparam'] = 'Update the answers tolerance parameters';
          $string['updatedatasetparam'] = 'Update the datasets parameters';
          $string['youmustaddatleastoneitem'] = 'You must add at least one dataset item before you can save this question.';
          ?>
          so
          in qtype_calculated

          +$string['trueanswerinsidelimits'] = ' Correct answer : $a->correct inside limits of true value $a->true';
          +$string['trueansweroutsidelimits'] = '<span class=\"error\">ERROR Correct answer : $a->correct outside limits of true value $a->true</span>';

          and in qtype_datasetdependent.php

          +$string['answerstoleranceparam'] = 'Answers tolerance parameters';
          +$string['updatetolerancesparam'] = 'Update the answers tolerance parameters';
          +$string['oneanswertrueansweroutsidelimits'] = 'At least one correct answer outside the true value limits.<br/>Modify the answers tolerance settings available as Advanced parameters';

          I wrote Answers tolerance parameters as I want to translate the french
          Paramètres de tolérance des réponses
          In french there is no equivalent of answer and response.
          In fact the advanced parameters shown are the tolerance and the format.
          So should we name the two parameters i.e tolerance and format
          and should we use the singular answer or plural answers
          and is the specific english 's appropriate?

          Paramètres de la tolérance et du format des réponses

          or the shorter

          Tolérance et format des réponses

          Show
          Pierre Pichet added a comment - They have been CVS to HEAD RCS file: /cvsroot/moodle/moodle/lang/en_utf8/qtype_calculated.php,v retrieving revision 1.7 retrieving revision 1.8 diff -u -r1.7 -r1.8 — qtype_calculated.php 20 Apr 2007 02:43:28 -0000 1.7 +++ qtype_calculated.php 16 Jun 2008 03:15:55 -0000 1.8 @@ -20,6 +20,8 @@ $string ['nosharedwildcard'] = 'No shared wild card in this category'; $string ['possiblehdr'] ='Possible wild cards present only in the question text'; $string ['tolerance'] = 'Tolerance ±'; +$string ['trueanswerinsidelimits'] = ' Correct answer : $a->correct inside limits of true value $a->true'; +$string ['trueansweroutsidelimits'] = '<span class=\"error\">ERROR Correct answer : $a->correct outside limits of true value $a->true</span>'; $string ['updatecategory'] = 'Update the category'; $string ['usedinquestion'] ='Used in Question'; $string ['youmustenteramultiplierhere'] = 'You must enter a multiplier here.'; and RCS file: /cvsroot/moodle/moodle/lang/en_utf8/qtype_datasetdependent.php,v retrieving revision 1.4 retrieving revision 1.5 diff -u -r1.4 -r1.5 — qtype_datasetdependent.php 22 May 2007 02:23:50 -0000 1.4 +++ qtype_datasetdependent.php 25 Sep 2008 00:18:09 -0000 1.5 @@ -1,5 +1,6 @@ <?php $string ['additem'] = 'Add Item'; +$string ['answerstoleranceparam'] = 'Answers tolerance parameters'; $string ['atleastonerealdataset'] ='There should be at least one real dataset in question text'; $string ['atleastonewildcard'] ='There should be at least one wild card in answer formula or question text'; $string ['calcdistribution'] = 'Distribution'; @@ -34,12 +35,14 @@ $string ['newlocal2'] = 'a file from a new set of files that will only be used by this question'; $string ['newlocal3'] = 'a link from a new set of links that will only be used by this question'; $string ['nodataset'] = 'nothing - it is not a wild card'; +$string ['oneanswertrueansweroutsidelimits'] = 'At least one correct answer outside the true value limits.<br/>Modify the answers tolerance settings available as Advanced parameters'; $string ['param'] = 'Param {<strong>$a</strong>} '; $string ['replacewithrandom'] = 'Replace with a random value'; $string ['reuseifpossible'] = 'reuse previous value if available'; $string ['sharedwildcard'] ='Shared wild card'; $string ['sharedwildcards'] ='Shared wild cards'; $string ['uniform'] = 'Uniform'; +$string ['updatetolerancesparam'] = 'Update the answers tolerance parameters'; $string ['updatedatasetparam'] = 'Update the datasets parameters'; $string ['youmustaddatleastoneitem'] = 'You must add at least one dataset item before you can save this question.'; ?> so in qtype_calculated +$string ['trueanswerinsidelimits'] = ' Correct answer : $a->correct inside limits of true value $a->true'; +$string ['trueansweroutsidelimits'] = '<span class=\"error\">ERROR Correct answer : $a->correct outside limits of true value $a->true</span>'; and in qtype_datasetdependent.php +$string ['answerstoleranceparam'] = 'Answers tolerance parameters'; +$string ['updatetolerancesparam'] = 'Update the answers tolerance parameters'; +$string ['oneanswertrueansweroutsidelimits'] = 'At least one correct answer outside the true value limits.<br/>Modify the answers tolerance settings available as Advanced parameters'; I wrote Answers tolerance parameters as I want to translate the french Paramètres de tolérance des réponses In french there is no equivalent of answer and response. In fact the advanced parameters shown are the tolerance and the format. So should we name the two parameters i.e tolerance and format and should we use the singular answer or plural answers and is the specific english 's appropriate? Paramètres de la tolérance et du format des réponses or the shorter Tolérance et format des réponses
          Hide
          Samuli Karevaara added a comment -

          Please investigate a possibility to implement the Calculated question type without datasets alltogether, just having ranges and tolerances for the variables.

          Show
          Samuli Karevaara added a comment - Please investigate a possibility to implement the Calculated question type without datasets alltogether, just having ranges and tolerances for the variables.
          Hide
          Pierre Pichet added a comment -

          Tim,

          could you check the text for new lang strings
          thanks

          Show
          Pierre Pichet added a comment - Tim, could you check the text for new lang strings thanks
          Hide
          Pierre Pichet added a comment -

          Tim,
          Finally, wait for the lang strings checking as I will have new ones for synchronizing the calculated questions(MDL-17278).
          .

          Show
          Pierre Pichet added a comment - Tim, Finally, wait for the lang strings checking as I will have new ones for synchronizing the calculated questions( MDL-17278 ). .
          Hide
          Michael de Raadt added a comment -

          Thanks for reporting this issue.

          We have detected that this issue has been inactive for over a year has been recorded as affecting versions that are no longer supported.

          If you believe that this issue is still relevant to current versions (2.1 and beyond), please comment on the issue. Issues left inactive for a further month will be closed.

          Michael d;

          lqjjLKA0p6

          Show
          Michael de Raadt added a comment - Thanks for reporting this issue. We have detected that this issue has been inactive for over a year has been recorded as affecting versions that are no longer supported. If you believe that this issue is still relevant to current versions (2.1 and beyond), please comment on the issue. Issues left inactive for a further month will be closed. Michael d; lqjjLKA0p6
          Hide
          Pierre Pichet added a comment -

          This has been fixed in sept 2008
          https://github.com/moodle/moodle/commit/450f11274ffc14118d9a815a502299748789f23f

          forgot to close
          Thanks Micheal for the reminder

          Show
          Pierre Pichet added a comment - This has been fixed in sept 2008 https://github.com/moodle/moodle/commit/450f11274ffc14118d9a815a502299748789f23f forgot to close Thanks Micheal for the reminder

            People

            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: