Moodle
  1. Moodle
  2. MDL-25526

Random Allocation does not properly balance workload when Manual Allocations already exist

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0
    • Fix Version/s: 2.0.2
    • Component/s: Workshop
    • Labels:
      None
    • Affected Branches:
      MOODLE_20_STABLE
    • Fixed Branches:
      MOODLE_20_STABLE
    • Rank:
      6673

      Description

      When some students have already been manually allocated to review other students' work, Random allocation fails to incorporate this information properly, resulting in an unbalanced workload amongst students.

      Steps to reproduce:

      Given a sample Workshop of five students in which all five students have submitted responses:
      1. Manually allocate some students. In my case, I allocated: two students to mark one submission, one student to mark two submissions; two submissions with one marker, one submission with two markers.
      2. Click Random Allocation.
      3. Allocate 3 reviews per submission, being sure not to remove current allocations.
      4. Check the allocations on the Manual Allocation page.

      Expected results:

      Each submission should have three reviewers, and each student should be reviewing three submissions (balanced workload).

      Actual results:

      Each submission has three reviewers, but one student has four reviews, and another has only two. This is not a balanced workload.

      Notes:

      Please note that due to the random element of the allocation alogrithm, this may not be reproducible every time. However, I have reproduced it three times in a row, each time with different manual allocations. I don't think reproducing it should be especially difficult.

        Activity

        Hide
        David Mudrak added a comment -

        Hi Morgan, thanks for the report. Is the Workshop in a group mode? If so, what groups are created and how are those students distributed in these groups?

        Show
        David Mudrak added a comment - Hi Morgan, thanks for the report. Is the Workshop in a group mode? If so, what groups are created and how are those students distributed in these groups?
        Hide
        Morgan Harris added a comment -

        The Workshop is in No Groups mode. I did file a separate bug pertaining to a Workshop in Visible Groups, where Random Allocation doesn't seem to work at all.

        Show
        Morgan Harris added a comment - The Workshop is in No Groups mode. I did file a separate bug pertaining to a Workshop in Visible Groups, where Random Allocation doesn't seem to work at all.
        Hide
        David Mudrak added a comment -

        Well, yes. This may happen. As it is described at http://moodle.org/mod/forum/discuss.php?d=128473
        the current algorithm just makes sure it satisfies your condition "X reviews per submission". If you can, try to it yourself: put five letters A, B, C, D and E into two rows as in

        A B C D E

        A B C D E

        Imagine the top row is the list of your students as reviewers and the bottom is the list of your students as authors. Now try to allocate the reviewers randomly by writing a line connecting a letter from the top row with a letter in the bottom row. Do not connect same letters (that would represent self-assessment which is separate). Start with some lines already pre-filled (to simulate the manual allocation). You may realize there are situations when random allocator is unable to actually fulfill the balance from both sides (number of reviews per submission and number of submissions per reviewer) so it just focus on the condition you have set.

        Show
        David Mudrak added a comment - Well, yes. This may happen. As it is described at http://moodle.org/mod/forum/discuss.php?d=128473 the current algorithm just makes sure it satisfies your condition "X reviews per submission". If you can, try to it yourself: put five letters A, B, C, D and E into two rows as in A B C D E A B C D E Imagine the top row is the list of your students as reviewers and the bottom is the list of your students as authors. Now try to allocate the reviewers randomly by writing a line connecting a letter from the top row with a letter in the bottom row. Do not connect same letters (that would represent self-assessment which is separate). Start with some lines already pre-filled (to simulate the manual allocation). You may realize there are situations when random allocator is unable to actually fulfill the balance from both sides (number of reviews per submission and number of submissions per reviewer) so it just focus on the condition you have set.
        Hide
        Morgan Harris added a comment -

        When you start allocating completely randomly, yes; that'll happen. But I've attached two screenshots here, the first is some manual allocations, and the second, the allocations after random allocation.

        (First let me say, I apologise for the names of our test students. I know they're a little hard to read. I didn't name them, and over time I've just gotten used to them.)

        It doesn't take long to figure out that if 14 reviewed 13's paper instead of 12, the workload would be balanced. Neither 12 nor 14 were involved in the original manual allocations; I'm perplexed as to how exactly they got singled out, but I guess it's a bit of a chaotic system.

        I faced a similar problem when I was writing the Team Builder plugin (http://moodle.org/mod/data/view.php?d=13&rid=4415). I got around it by working through the criteria first and figuring out if the student was going to be needed at a later date, and then prioritising the allocation of students who wouldn't be needed later. Obviously it's not the same setup, and I haven't looked through the algorithms for random allocation here, but maybe you could apply the same principle?

        Show
        Morgan Harris added a comment - When you start allocating completely randomly, yes; that'll happen. But I've attached two screenshots here, the first is some manual allocations, and the second, the allocations after random allocation. (First let me say, I apologise for the names of our test students. I know they're a little hard to read. I didn't name them, and over time I've just gotten used to them.) It doesn't take long to figure out that if 14 reviewed 13's paper instead of 12, the workload would be balanced. Neither 12 nor 14 were involved in the original manual allocations; I'm perplexed as to how exactly they got singled out, but I guess it's a bit of a chaotic system. I faced a similar problem when I was writing the Team Builder plugin ( http://moodle.org/mod/data/view.php?d=13&rid=4415 ). I got around it by working through the criteria first and figuring out if the student was going to be needed at a later date, and then prioritising the allocation of students who wouldn't be needed later. Obviously it's not the same setup, and I haven't looked through the algorithms for random allocation here, but maybe you could apply the same principle?
        Hide
        Morgan Harris added a comment -

        Before Random Allocation

        Show
        Morgan Harris added a comment - Before Random Allocation
        Hide
        Morgan Harris added a comment -

        After Random Allocation

        Show
        Morgan Harris added a comment - After Random Allocation
        Hide
        David Mudrak added a comment -

        Confirmed, this is really bug. I am going to try to modify the allocation algorithm to prevent this behaviour, if possible.

        Show
        David Mudrak added a comment - Confirmed, this is really bug. I am going to try to modify the allocation algorithm to prevent this behaviour, if possible.
        Hide
        David Mudrak added a comment -

        OK. So I reviewed the algorithm again and improved it a bit. I am going to commit a change that fixes randomization a changes the procedure a bit. It will allocate reviews in several iterations. If the user requires, say, 3 reviews per submission, it will allocate one review per each submission during the first iteration, second review during the second iteration etc. Right now, it first allocated all three reviews for the first submissions, all three reviews to the next submission etc. That might help the balancing a bit.

        However, the reported behaviour can be still reproduced, but I am not going to change it. It is not that clear bug as it might look. Let me demonstrate it.

        Imagine we have three students in the workshop in no-groups mode and there are no pre-allocations. We ask the random allocator to randomly allocate one review per submission. Let us pretend students are called A, B and C.

        For the submission by A, the allocator has to choose either B or C as the reviewer. It randomly picks B.
        For the submission by B, the allocator can choose from A or C. It randomly picks A.
        For the submission by C, we have to choose from A or B. Whatever we choose, we get unbalanced result because C does not review anything while either A or B have two submissions and the other has just one.

        The only way how to get balanced distribution here is if B was picked for A, C was picked for B and A was picked for C. But then there are less levels of freedom. This is a trade-off between randomness and balanced distribution. And I just decided that the randomness is preferred by this allocator.

        The similar situation happens in the reported case with 5 students and 3 reviews per submission. When the number of students minus number of reviews is greater than two, the chance of getting into this situation is much less probable. And random allocator is supposed to work well with large number of students.

        However, note that there is a way around if a teacher really needs it. Just firstly allocate more reviews than expected and then lower the number with ticking "Remove current allocations". For example, if you want 3 reviews per submission in a group of 5 students, just start with 4 reviews and then repeat allocating with setting 3 + remove current. That will prune those extra allocations.

        Show
        David Mudrak added a comment - OK. So I reviewed the algorithm again and improved it a bit. I am going to commit a change that fixes randomization a changes the procedure a bit. It will allocate reviews in several iterations. If the user requires, say, 3 reviews per submission, it will allocate one review per each submission during the first iteration, second review during the second iteration etc. Right now, it first allocated all three reviews for the first submissions, all three reviews to the next submission etc. That might help the balancing a bit. However, the reported behaviour can be still reproduced, but I am not going to change it. It is not that clear bug as it might look. Let me demonstrate it. Imagine we have three students in the workshop in no-groups mode and there are no pre-allocations. We ask the random allocator to randomly allocate one review per submission. Let us pretend students are called A, B and C. For the submission by A, the allocator has to choose either B or C as the reviewer. It randomly picks B. For the submission by B, the allocator can choose from A or C. It randomly picks A. For the submission by C, we have to choose from A or B. Whatever we choose, we get unbalanced result because C does not review anything while either A or B have two submissions and the other has just one. The only way how to get balanced distribution here is if B was picked for A, C was picked for B and A was picked for C. But then there are less levels of freedom. This is a trade-off between randomness and balanced distribution. And I just decided that the randomness is preferred by this allocator. The similar situation happens in the reported case with 5 students and 3 reviews per submission. When the number of students minus number of reviews is greater than two, the chance of getting into this situation is much less probable. And random allocator is supposed to work well with large number of students. However, note that there is a way around if a teacher really needs it. Just firstly allocate more reviews than expected and then lower the number with ticking "Remove current allocations". For example, if you want 3 reviews per submission in a group of 5 students, just start with 4 reviews and then repeat allocating with setting 3 + remove current. That will prune those extra allocations.
        Hide
        David Mudrak added a comment -

        Resolving. The improved procedure has been submitted for review.

        Show
        David Mudrak added a comment - Resolving. The improved procedure has been submitted for review.
        Hide
        Helen Foster added a comment -

        Reopening for further investigation as problem found when testing PULL-128.

        Show
        Helen Foster added a comment - Reopening for further investigation as problem found when testing PULL-128.
        Hide
        Helen Foster added a comment -

        Resolving again after reopening in error.

        Show
        Helen Foster added a comment - Resolving again after reopening in error.
        Hide
        Helen Foster added a comment -

        After following steps to reproduce, the result was as expected with each submission having 3 reviewers and each student reviewing 3 submissions.

        Morgan, thanks for your report and David, thanks for your fix.

        Show
        Helen Foster added a comment - After following steps to reproduce, the result was as expected with each submission having 3 reviewers and each student reviewing 3 submissions. Morgan, thanks for your report and David, thanks for your fix.

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: