Moodle

Unexpected grade for assessment calculated by workshop

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 2.5.2
  • Fix Version/s: 2.4.7, 2.5.3, 2.6
  • Component/s: Workshop
  • Labels:

Description

This was observed by Helen Foster while she was evaluating the MOOC workshop today at learn.moodle.net site. The following description is slightly simplified from the real situation but it illustrates the situation well (and could be used as steps to reproduce).

STEPS TO REPRODUCE

  • There is a workshop, using the grading strategy "Number of errors".
  • The assessment form has three assertions (dimensions), all with the same weight 1.
  • There was a submission made by a participant Alice that was peer-reviewed by three other participants Bob, Cindy and Dan.
  • Bob assessed the submission and filled the assessment form with No, No, No.
  • Both Cindy and Dan assessed the submission and filled the assessment form with Yes, Yes, Yes.
  • Helen (as the workshop facilitator/teacher) used the grading evaluation method "Comparison with the best assessment" during the grading evaluation phase to calculate the final grades.
  • Helen realized that Bob's assessment is wrong and that Cindy and Dan are right. So she set the weight of Bob's assessment to 0 (zero) and re-calculated grades.

EXPECTED BEHAVIOUR

Bob should get very low grade for this assessment and both Cindy and Dan should get full grade for their assessments.

WHAT ACTUALLY HAPPENED

Bob got the full grade for his assessment, too.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
David Mudrak added a comment -

Good catch! There is a bug in the mod/workshop/eval/best/lib.php library. It affects the calculation of the grade for assessment in this specific situation only, luckily. What happened here is that, thanks to all the coincidences, the calculated variance of received assessments was zero in all three assertions (because Bob's assessment did not count and all others assessments were equal). And the situation of zero variance is not covered well in the code. The line

if ($var > 0.01 and $agrade != $rgrade) {

caused that the "distance" of Bob's assessment from the "best" (referential) one is not calculated if the variance is zero.

I wrote a unit test that confirms this bug and applied a simple fix (that should had been here since ever).

Show
David Mudrak added a comment - Good catch! There is a bug in the mod/workshop/eval/best/lib.php library. It affects the calculation of the grade for assessment in this specific situation only, luckily. What happened here is that, thanks to all the coincidences, the calculated variance of received assessments was zero in all three assertions (because Bob's assessment did not count and all others assessments were equal). And the situation of zero variance is not covered well in the code. The line if ($ var > 0.01 and $agrade != $rgrade) { caused that the "distance" of Bob's assessment from the "best" (referential) one is not calculated if the variance is zero. I wrote a unit test that confirms this bug and applied a simple fix (that should had been here since ever).
Hide
Permalink
David Mudrak added a comment -

Submitting for integration. The submitted fix makes the code handle the calculation even in case of the dimension assessments variance lower than 0.01 by assuming it as the lowest reasonable value.

Show
David Mudrak added a comment - Submitting for integration. The submitted fix makes the code handle the calculation even in case of the dimension assessments variance lower than 0.01 by assuming it as the lowest reasonable value.
Hide
Permalink
Marina Glancy added a comment -

Thanks David, integrated in 2.4, 2.5 and master

Show
Marina Glancy added a comment - Thanks David, integrated in 2.4, 2.5 and master
Hide
Permalink
Frédéric Massart added a comment -

Passing, thanks!

Show
Frédéric Massart added a comment - Passing, thanks!
Hide
Permalink
Dan Poltawski added a comment -

You did it!

Thanks for your contribution, this change has been merged upstream and is now available on your local git mirror and on download sites shortly.

Show
Dan Poltawski added a comment - You did it! Thanks for your contribution, this change has been merged upstream and is now available on your local git mirror and on download sites shortly.

People

Vote (1)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: