Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-41954

Unexpected grade for assessment calculated by workshop

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.5.2
    • Fix Version/s: 2.4.7, 2.5.3, 2.6
    • Component/s: Workshop
    • Labels:

      Description

      This was observed by Helen Foster while she was evaluating the MOOC workshop today at learn.moodle.net site. The following description is slightly simplified from the real situation but it illustrates the situation well (and could be used as steps to reproduce).

      STEPS TO REPRODUCE

      • There is a workshop, using the grading strategy "Number of errors".
      • The assessment form has three assertions (dimensions), all with the same weight 1.
      • There was a submission made by a participant Alice that was peer-reviewed by three other participants Bob, Cindy and Dan.
      • Bob assessed the submission and filled the assessment form with No, No, No.
      • Both Cindy and Dan assessed the submission and filled the assessment form with Yes, Yes, Yes.
      • Helen (as the workshop facilitator/teacher) used the grading evaluation method "Comparison with the best assessment" during the grading evaluation phase to calculate the final grades.
      • Helen realized that Bob's assessment is wrong and that Cindy and Dan are right. So she set the weight of Bob's assessment to 0 (zero) and re-calculated grades.

      EXPECTED BEHAVIOUR

      Bob should get very low grade for this assessment and both Cindy and Dan should get full grade for their assessments.

      WHAT ACTUALLY HAPPENED

      Bob got the full grade for his assessment, too.

        Gliffy Diagrams

          Attachments

            Activity

            Hide
            mudrd8mz David Mudrák added a comment -

            Good catch! There is a bug in the mod/workshop/eval/best/lib.php library. It affects the calculation of the grade for assessment in this specific situation only, luckily. What happened here is that, thanks to all the coincidences, the calculated variance of received assessments was zero in all three assertions (because Bob's assessment did not count and all others assessments were equal). And the situation of zero variance is not covered well in the code. The line

            if ($var > 0.01 and $agrade != $rgrade) {

            caused that the "distance" of Bob's assessment from the "best" (referential) one is not calculated if the variance is zero.

            I wrote a unit test that confirms this bug and applied a simple fix (that should had been here since ever).

            Show
            mudrd8mz David Mudrák added a comment - Good catch! There is a bug in the mod/workshop/eval/best/lib.php library. It affects the calculation of the grade for assessment in this specific situation only, luckily. What happened here is that, thanks to all the coincidences, the calculated variance of received assessments was zero in all three assertions (because Bob's assessment did not count and all others assessments were equal). And the situation of zero variance is not covered well in the code. The line if ($var > 0.01 and $agrade != $rgrade) { caused that the "distance" of Bob's assessment from the "best" (referential) one is not calculated if the variance is zero. I wrote a unit test that confirms this bug and applied a simple fix (that should had been here since ever).
            Hide
            mudrd8mz David Mudrák added a comment -

            Submitting for integration. The submitted fix makes the code handle the calculation even in case of the dimension assessments variance lower than 0.01 by assuming it as the lowest reasonable value.

            Show
            mudrd8mz David Mudrák added a comment - Submitting for integration. The submitted fix makes the code handle the calculation even in case of the dimension assessments variance lower than 0.01 by assuming it as the lowest reasonable value.
            Hide
            marina Marina Glancy added a comment -

            Thanks David, integrated in 2.4, 2.5 and master

            Show
            marina Marina Glancy added a comment - Thanks David, integrated in 2.4, 2.5 and master
            Hide
            fred Frédéric Massart added a comment -

            Passing, thanks!

            Show
            fred Frédéric Massart added a comment - Passing, thanks!
            Hide
            poltawski Dan Poltawski added a comment -

            You did it!

            Thanks for your contribution, this change has been merged upstream and is now available on your local git mirror and on download sites shortly.

            Show
            poltawski Dan Poltawski added a comment - You did it! Thanks for your contribution, this change has been merged upstream and is now available on your local git mirror and on download sites shortly.

              People

              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  11/Nov/13