Moodle
  1. Moodle
  2. MDL-41954

Unexpected grade for assessment calculated by workshop

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.2
    • Fix Version/s: 2.4.7, 2.5.3, 2.6
    • Component/s: Workshop
    • Labels:
    • Rank:
      53137

      Description

      This was observed by Helen Foster while she was evaluating the MOOC workshop today at learn.moodle.net site. The following description is slightly simplified from the real situation but it illustrates the situation well (and could be used as steps to reproduce).

      STEPS TO REPRODUCE

      • There is a workshop, using the grading strategy "Number of errors".
      • The assessment form has three assertions (dimensions), all with the same weight 1.
      • There was a submission made by a participant Alice that was peer-reviewed by three other participants Bob, Cindy and Dan.
      • Bob assessed the submission and filled the assessment form with No, No, No.
      • Both Cindy and Dan assessed the submission and filled the assessment form with Yes, Yes, Yes.
      • Helen (as the workshop facilitator/teacher) used the grading evaluation method "Comparison with the best assessment" during the grading evaluation phase to calculate the final grades.
      • Helen realized that Bob's assessment is wrong and that Cindy and Dan are right. So she set the weight of Bob's assessment to 0 (zero) and re-calculated grades.

      EXPECTED BEHAVIOUR

      Bob should get very low grade for this assessment and both Cindy and Dan should get full grade for their assessments.

      WHAT ACTUALLY HAPPENED

      Bob got the full grade for his assessment, too.

        Activity

        Hide
        David Mudrak added a comment -

        Good catch! There is a bug in the mod/workshop/eval/best/lib.php library. It affects the calculation of the grade for assessment in this specific situation only, luckily. What happened here is that, thanks to all the coincidences, the calculated variance of received assessments was zero in all three assertions (because Bob's assessment did not count and all others assessments were equal). And the situation of zero variance is not covered well in the code. The line

        if ($var > 0.01 and $agrade != $rgrade) {
        

        caused that the "distance" of Bob's assessment from the "best" (referential) one is not calculated if the variance is zero.

        I wrote a unit test that confirms this bug and applied a simple fix (that should had been here since ever).

        Show
        David Mudrak added a comment - Good catch! There is a bug in the mod/workshop/eval/best/lib.php library. It affects the calculation of the grade for assessment in this specific situation only, luckily. What happened here is that, thanks to all the coincidences, the calculated variance of received assessments was zero in all three assertions (because Bob's assessment did not count and all others assessments were equal). And the situation of zero variance is not covered well in the code. The line if ($ var > 0.01 and $agrade != $rgrade) { caused that the "distance" of Bob's assessment from the "best" (referential) one is not calculated if the variance is zero. I wrote a unit test that confirms this bug and applied a simple fix (that should had been here since ever).
        Hide
        David Mudrak added a comment -

        Submitting for integration. The submitted fix makes the code handle the calculation even in case of the dimension assessments variance lower than 0.01 by assuming it as the lowest reasonable value.

        Show
        David Mudrak added a comment - Submitting for integration. The submitted fix makes the code handle the calculation even in case of the dimension assessments variance lower than 0.01 by assuming it as the lowest reasonable value.
        Hide
        Marina Glancy added a comment -

        Thanks David, integrated in 2.4, 2.5 and master

        Show
        Marina Glancy added a comment - Thanks David, integrated in 2.4, 2.5 and master
        Hide
        Frédéric Massart added a comment -

        Passing, thanks!

        Show
        Frédéric Massart added a comment - Passing, thanks!
        Hide
        Dan Poltawski added a comment -

        You did it!

        Thanks for your contribution, this change has been merged upstream and is now available on your local git mirror and on download sites shortly.

        Show
        Dan Poltawski added a comment - You did it! Thanks for your contribution, this change has been merged upstream and is now available on your local git mirror and on download sites shortly.

          People

          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: