Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-65288

Follow up on inaccurate predictions

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.6
    • Fix Version/s: None
    • Component/s: Analytics
    • Labels:
      None
    • Affected Branches:
      MOODLE_36_STABLE

      Description

      As part of integrated continuous improvement in learning analytics, it is important to follow up when predictions by a model are inaccurate. Any recipient of an insight should be asked for input if the insight turns out to have been generated for a false prediction. A distinction must be made between false positive and false negative predictions (particularly because a model may be a "risk" prediction of a negative outcome or a prediction of a positive outcome).

      For each followup message (false positive and false negative), Provide the following:

      • Text of message (preferably using tokens as proposed for insight messages in MDL-62523)
      • Details of predictions per time slice, emphasizing predictors with highest deviance from expected values given the true outcome
      • Checkbox list of common reasons for false predictions, including:
        • The insight notification helped to change the outcome
        • One or more of the predictors are considered inappropriate or inaccurate by the reviewer (top n predictors listed in order of largest deviance residual)
        • Circumstances unrelated to the {sample} changed the outcome (e.g. the student suddenly had more or less time for the course than expected)
      • A free text field for response should be provided with the followup message.

       Any responses to this follow-up should be included in information about the accuracy of the model. Eventually this data might be factored into estimates of model accuracy.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              emdalton1 Elizabeth Dalton
              Participants:
              Component watchers:
              Elizabeth Dalton, Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Sara Arjona (@sarjona)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated: