Moodle
  1. Moodle
  2. MDL-28942

Wrong random guess score for mulitple-choice multiple-response questions

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1.1, 2.2
    • Fix Version/s: 2.1.2
    • Component/s: Questions
    • Labels:
    • Testing Instructions:
      Hide

      1. Create a quiz with two MC questions, one with radio buttons (One or multiple answers? = One answer only) and one with checkboxes (One or multiple answers? = Multiple answers allowed)
      2. Attempt the quiz as at least two different students.
      2.5 Run cron
      3. As teacher, go to the statistics report (Course -> Quiz -> Results -> Statistics)
      4. Verify that the correct random guess score is shown for the single-choice question, and no random guess score is shown for the multiple-response question.

      Show
      1. Create a quiz with two MC questions, one with radio buttons (One or multiple answers? = One answer only) and one with checkboxes (One or multiple answers? = Multiple answers allowed) 2. Attempt the quiz as at least two different students. 2.5 Run cron 3. As teacher, go to the statistics report (Course -> Quiz -> Results -> Statistics) 4. Verify that the correct random guess score is shown for the single-choice question, and no random guess score is shown for the multiple-response question.
    • Affected Branches:
      MOODLE_21_STABLE, MOODLE_22_STABLE
    • Fixed Branches:
      MOODLE_21_STABLE
    • Pull from Repository:
    • Pull Master Branch:
    • Rank:
      18613

      Description

      At the moment, it is using the same calculation for single-response and multiple-response questions which is definitely wrong.

      Unfortunately, I don't think there is any feasible way to calculate it for multiple-response questions, other than enumerating all possibilities, and even that depends on whether the question text says "Select all of the choices that ..." or "Select the N choices that ...". Moodle's scoring rules are just too flexible.

      (By comparison, https://github.com/timhunt/moodle-qtype_oumultiresponse has a fairly simple random guess score calculation.)

      I think the only solution is to not attempt to calculate the RGS in this situation. (This is an acceptable thing to do in the question type API.)

        Activity

        Hide
        Jean-Michel Vedrine added a comment -

        Hello Tim,
        The sentence : and even that depends on whether the question text says "Select all of the choices that ..." or "Select the N choices that ...". Moodle's scoring rules are just too flexible
        looks quite strange to me.
        Why are you suddenly changing the rules of the game ? In all other questions types you never look at the question text who may surely include also some clues changing the probability that a student choose at random some responses. Why are you looking at question text here ? it's irrelevant !
        For me random guess is random guess. Dot.
        For a shortanswer question it's the probability the "monkey typist" type a right answer so equal 0
        For a multichoice with one answer allowed (here the monkey don't type, he click !) the probability is clearly the one given by the code you ahev already done :
        $totalfraction = 0;
        foreach ($questiondata->options->answers as $answer)

        { $totalfraction += $answer->fraction; }

        return $totalfraction / count($questiondata->options->answers);
        In all following calculations let n be the numbers of answers = count($questiondata->options->answers)
        Here each radio button has a probability equal to 1/n to be choosen by the monkey so we calculate the expected value of a random variable and your formula is right.
        For a multichoice question where the monkey can check multiple checkboxes, each response is a sequence of n numbers who can be either 0 or 1 if we code 0 a non checked checkbox and 1 a checked checkbox.
        There is of course 2^n such sequences.
        For a given sequence it is quite doable to calculate the resulting marking (after all this is what is doing Moodle when he grade such questions)
        The grade is equal to the sum of all fractions corresponding to the 1 numbers in the sequence
        For instance 00...0 result in 0 in all cases !
        And 11...1 result in the sum of all stems fractions (not necessary a good choice if the teacher uses negatives fractions)
        This is where we see Moodle grading scheme of this kind of question is not the only one possible. On an other e-assesment sytem I have used the fraction associated with a stem is awarded if the state (checked or unchecked) is the same in the teacher answer and in the student response, and there is anothe value you can choose (usualy a negative value) wich is "awarded" if the student response (checked or unchecked) is opposite. You can also choose to award grades calculated on the number of "conformity" = number of checkboxes in the same state in the teacher answer and the student response. Very flexible system indeed.
        Back to Moodle. For a place i between 1 and n there is exactly 2^(n-1) sequence with a 1 at this place i (wich will be awarded the fraction for this stem) and 2^(n-1) sequence with a 0 at this place (wich will be awarded nothing for this stem). So the formula will be sum(fraction*2^(n-1))/(2^n) there is clearly a big simplification and we find sum(fraction)/2
        So random guess is simply the sum of all fractions divided by 2 !
        Here is your formula and it is quite doable to calculate it.
        But maybe I am simply too tired (I have been working at upgrading the formulas question type to Moodle 2.1 all day) and my calculation is wrong ! I will check tomorrow.

        Show
        Jean-Michel Vedrine added a comment - Hello Tim, The sentence : and even that depends on whether the question text says "Select all of the choices that ..." or "Select the N choices that ...". Moodle's scoring rules are just too flexible looks quite strange to me. Why are you suddenly changing the rules of the game ? In all other questions types you never look at the question text who may surely include also some clues changing the probability that a student choose at random some responses. Why are you looking at question text here ? it's irrelevant ! For me random guess is random guess. Dot. For a shortanswer question it's the probability the "monkey typist" type a right answer so equal 0 For a multichoice with one answer allowed (here the monkey don't type, he click !) the probability is clearly the one given by the code you ahev already done : $totalfraction = 0; foreach ($questiondata->options->answers as $answer) { $totalfraction += $answer->fraction; } return $totalfraction / count($questiondata->options->answers); In all following calculations let n be the numbers of answers = count($questiondata->options->answers) Here each radio button has a probability equal to 1/n to be choosen by the monkey so we calculate the expected value of a random variable and your formula is right. For a multichoice question where the monkey can check multiple checkboxes, each response is a sequence of n numbers who can be either 0 or 1 if we code 0 a non checked checkbox and 1 a checked checkbox. There is of course 2^n such sequences. For a given sequence it is quite doable to calculate the resulting marking (after all this is what is doing Moodle when he grade such questions) The grade is equal to the sum of all fractions corresponding to the 1 numbers in the sequence For instance 00...0 result in 0 in all cases ! And 11...1 result in the sum of all stems fractions (not necessary a good choice if the teacher uses negatives fractions) This is where we see Moodle grading scheme of this kind of question is not the only one possible. On an other e-assesment sytem I have used the fraction associated with a stem is awarded if the state (checked or unchecked) is the same in the teacher answer and in the student response, and there is anothe value you can choose (usualy a negative value) wich is "awarded" if the student response (checked or unchecked) is opposite. You can also choose to award grades calculated on the number of "conformity" = number of checkboxes in the same state in the teacher answer and the student response. Very flexible system indeed. Back to Moodle. For a place i between 1 and n there is exactly 2^(n-1) sequence with a 1 at this place i (wich will be awarded the fraction for this stem) and 2^(n-1) sequence with a 0 at this place (wich will be awarded nothing for this stem). So the formula will be sum(fraction*2^(n-1))/(2^n) there is clearly a big simplification and we find sum(fraction)/2 So random guess is simply the sum of all fractions divided by 2 ! Here is your formula and it is quite doable to calculate it. But maybe I am simply too tired (I have been working at upgrading the formulas question type to Moodle 2.1 all day) and my calculation is wrong ! I will check tomorrow.
        Hide
        Jean-Michel Vedrine added a comment -

        Oops I know I missed something, the line :
        $fraction = min(max(0, $fraction), 1.0);
        definitely makes my calcul wrong in some cases.

        Show
        Jean-Michel Vedrine added a comment - Oops I know I missed something, the line : $fraction = min(max(0, $fraction), 1.0); definitely makes my calcul wrong in some cases.
        Hide
        Jean-Michel Vedrine added a comment -

        In fact my calcul works in a ridiculous number of cases : if there is no negative fractions and if the sum of all positives fractions is no more than 1.0. Too bad. As I said I will look at it tomorrow

        Show
        Jean-Michel Vedrine added a comment - In fact my calcul works in a ridiculous number of cases : if there is no negative fractions and if the sum of all positives fractions is no more than 1.0. Too bad. As I said I will look at it tomorrow
        Hide
        Tim Hunt added a comment -

        Looks like you went thought exactly the same thought process as I did.

        Show
        Tim Hunt added a comment - Looks like you went thought exactly the same thought process as I did.
        Hide
        Tim Hunt added a comment -

        See http://moodle.org/mod/forum/discuss.php?d=183275 for more discussion of this.

        Show
        Tim Hunt added a comment - See http://moodle.org/mod/forum/discuss.php?d=183275 for more discussion of this.
        Hide
        Eloy Lafuente (stronk7) added a comment -

        The main moodle.git repository has just been updated with latest weekly modifications. You may wish to rebase your PULL branches to simplify history and avoid any possible merge conflicts. This would also make integrator's life easier next week.

        TIA and ciao

        Show
        Eloy Lafuente (stronk7) added a comment - The main moodle.git repository has just been updated with latest weekly modifications. You may wish to rebase your PULL branches to simplify history and avoid any possible merge conflicts. This would also make integrator's life easier next week. TIA and ciao
        Hide
        Eloy Lafuente (stronk7) added a comment -

        Integrated by cherry-picking & amend to fix one comment. Backported to 21_STABLE (the amended commit). So probably.. your history won't match.

        Ciao

        Show
        Eloy Lafuente (stronk7) added a comment - Integrated by cherry-picking & amend to fix one comment. Backported to 21_STABLE (the amended commit). So probably.. your history won't match. Ciao
        Hide
        Tim Hunt added a comment -

        What was wrong with the commit comment? I can't see anything! And I would like to know, so I can avoid making the same mistake in future.

        Show
        Tim Hunt added a comment - What was wrong with the commit comment? I can't see anything! And I would like to know, so I can avoid making the same mistake in future.
        Hide
        Michael de Raadt added a comment -

        Test result: Random guess score shown for single answer but not for multi-answer.

        Show
        Michael de Raadt added a comment - Test result: Random guess score shown for single answer but not for multi-answer.
        Hide
        Tim Hunt added a comment -

        Your testing instructions are wrong. There is no need to run cron.

        Show
        Tim Hunt added a comment - Your testing instructions are wrong. There is no need to run cron.
        Hide
        Eloy Lafuente (stronk7) added a comment -

        Many thanks for the hard work, this has been sent upstream and is available in all the git and cvs repositories.

        Show
        Eloy Lafuente (stronk7) added a comment - Many thanks for the hard work, this has been sent upstream and is available in all the git and cvs repositories.

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: