Moodle
  1. Moodle
  2. MDL-31237

Create API for multiple graders in advanced grading

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: DEV backlog
    • Fix Version/s: DEV backlog
    • Component/s: Grading methods
    • Labels:
    • Rank:
      37698

      Description

      allow to use advanced grading in the situations when several graders (teachers or even other students?) grade students and final grade is calculated as for example average

      API should at least have a function (supposingly grading_manager::allow_multiple_graders()) that will tell if several active instances of grade are allowed (only for different graders)

        Activity

        Hide
        Elizabeth Dalton added a comment -

        This would be very helpful to us. Some thoughts:

        • Other Advanced Grading features should be supported, so multiple graders can use rubrics, marking guides, etc.
        • Different criteria might apply to who is allowed/assigned to grade a submission, and when. For example:

        – A student might be allowed (even required) to grade their own submission before anyone else is asked to grade it.
        – Other students might only be able to grade submissions after their own submission has received a satisfactory grade. This would support large-scale classes, e.g. MOOCs, with peer grading. Perhaps achievement of a satisfactory grade could be used to assign a user to a role automatically, with that role given permission to grade specific assignments.

        • Aggregation criteria might take roles into account, e.g. instructor grades might have more weight assigned than student peer grades. This might also affect the number of grades required for a submission to be considered "completely graded." One instructor grade might be worth 3 or 4 student peer grades.
        • Grading may be a gradable activity. Recipients of grades (or instructors, or other designated users) might be able to rate the quality of the feedback given, and these ratings might be tracked to identify the most helpful graders, e.g. for badges, compensation, extra credit, or other recognition. This could also be used for a system of random regrading by a senior instructor or grader.
        • Inter-rater reliability should be calculated, both at the assignment level and at the individual user level (as an aggregate of reliability within assignments). Inter-rater reliability might feed into grade ratings, in a similar way to the Workshop activity, but as an option.
        • A certain number of peer grading entries might be a required part of a course, i.e. a student might be required to grade an assignment n times in order to complete the course. Perhaps grading activity could have its own completion criteria? This would provide a connection to badges, as well.
        • Assignment to a grader should have automatic options based on criteria such as:
          • the number of graders required for each assignment
          • the number of submissions any individual grader has been assigned (load balancing)
          • the grouping (if any) of the assignment
          • the roles allowed to grade the assignment
          • cohort membership
        • Options for display to students might include:
          • Do not display grades to students until the required number of marks has been submitted
          • Display each mark as submitted
          • Display only the aggregated grade, not the individual marks
          • Display or hide identities of markers
          • Display or hide identities of assignment submitters

        What can I do to help support development? Will this be developed within the context of the Advanced Grading API, as a means of supporting multiple active grading form instances? Would additional changes to this API be needed?

        Show
        Elizabeth Dalton added a comment - This would be very helpful to us. Some thoughts: Other Advanced Grading features should be supported, so multiple graders can use rubrics, marking guides, etc. Different criteria might apply to who is allowed/assigned to grade a submission, and when. For example: – A student might be allowed (even required) to grade their own submission before anyone else is asked to grade it. – Other students might only be able to grade submissions after their own submission has received a satisfactory grade. This would support large-scale classes, e.g. MOOCs, with peer grading. Perhaps achievement of a satisfactory grade could be used to assign a user to a role automatically, with that role given permission to grade specific assignments. Aggregation criteria might take roles into account, e.g. instructor grades might have more weight assigned than student peer grades. This might also affect the number of grades required for a submission to be considered "completely graded." One instructor grade might be worth 3 or 4 student peer grades. Grading may be a gradable activity. Recipients of grades (or instructors, or other designated users) might be able to rate the quality of the feedback given, and these ratings might be tracked to identify the most helpful graders, e.g. for badges, compensation, extra credit, or other recognition. This could also be used for a system of random regrading by a senior instructor or grader. Inter-rater reliability should be calculated, both at the assignment level and at the individual user level (as an aggregate of reliability within assignments). Inter-rater reliability might feed into grade ratings, in a similar way to the Workshop activity, but as an option. A certain number of peer grading entries might be a required part of a course, i.e. a student might be required to grade an assignment n times in order to complete the course. Perhaps grading activity could have its own completion criteria? This would provide a connection to badges, as well. Assignment to a grader should have automatic options based on criteria such as: the number of graders required for each assignment the number of submissions any individual grader has been assigned (load balancing) the grouping (if any) of the assignment the roles allowed to grade the assignment cohort membership Options for display to students might include: Do not display grades to students until the required number of marks has been submitted Display each mark as submitted Display only the aggregated grade, not the individual marks Display or hide identities of markers Display or hide identities of assignment submitters What can I do to help support development? Will this be developed within the context of the Advanced Grading API, as a means of supporting multiple active grading form instances? Would additional changes to this API be needed?

          People

          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated: