This would be very helpful to us. Some thoughts:
- Other Advanced Grading features should be supported, so multiple graders can use rubrics, marking guides, etc.
- Different criteria might apply to who is allowed/assigned to grade a submission, and when. For example:
– A student might be allowed (even required) to grade their own submission before anyone else is asked to grade it.
– Other students might only be able to grade submissions after their own submission has received a satisfactory grade. This would support large-scale classes, e.g. MOOCs, with peer grading. Perhaps achievement of a satisfactory grade could be used to assign a user to a role automatically, with that role given permission to grade specific assignments.
- Aggregation criteria might take roles into account, e.g. instructor grades might have more weight assigned than student peer grades. This might also affect the number of grades required for a submission to be considered "completely graded." One instructor grade might be worth 3 or 4 student peer grades.
- Grading may be a gradable activity. Recipients of grades (or instructors, or other designated users) might be able to rate the quality of the feedback given, and these ratings might be tracked to identify the most helpful graders, e.g. for badges, compensation, extra credit, or other recognition. This could also be used for a system of random regrading by a senior instructor or grader.
- Inter-rater reliability should be calculated, both at the assignment level and at the individual user level (as an aggregate of reliability within assignments). Inter-rater reliability might feed into grade ratings, in a similar way to the Workshop activity, but as an option.
- A certain number of peer grading entries might be a required part of a course, i.e. a student might be required to grade an assignment n times in order to complete the course. Perhaps grading activity could have its own completion criteria? This would provide a connection to badges, as well.
- Assignment to a grader should have automatic options based on criteria such as:
- the number of graders required for each assignment
- the number of submissions any individual grader has been assigned (load balancing)
- the grouping (if any) of the assignment
- the roles allowed to grade the assignment
- cohort membership
- Options for display to students might include:
- Do not display grades to students until the required number of marks has been submitted
- Display each mark as submitted
- Display only the aggregated grade, not the individual marks
- Display or hide identities of markers
- Display or hide identities of assignment submitters
What can I do to help support development? Will this be developed within the context of the Advanced Grading API, as a means of supporting multiple active grading form instances? Would additional changes to this API be needed?