Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-64787

Evaluate analytics models using the trained ML backend

XMLWordPrintable

    • MOODLE_37_STABLE
    • MOODLE_37_STABLE
    • MDL-64787_master
    • Hide

      Prerequisites

      1. Install the python ML backend as specified in https://docs.moodle.org/36/en/Analytics#Predictions_processor
        • If you already got it installed you will need to update it to the latest version (pip install -I moodlemlbackend==1.0.0)
      2. Install https://github.com/dmonllao/moodle-local_testanalytics
      3. Enable analytics and make sure everything is working, refer to the docs on how to set it up if you don't have it already.
      4. Go to site admin > analytics > analytics settings. Disable 'onlycli' option and ensure that 'Default predictions processor' is set to 'PHP machine learning backend'

      Test

      1. Log in as admin and go to Site admin > analytics > analytics models
      2. Click on Actions > Get predictions for Binary model
      3. Click on Actions > Evaluate for Binary model, you SHOULD NOT see a modal window to select the evaluation mode. Click Continue once the process finishes
      4. Export Binary model by clicking on Actions > Export configuration. You can delete this model now (Actions > Delete).
      5. Import the zip file you just downloaded by clicking on Import model button
      6. Open the Actions dropdown menu for the imported Binary model. There SHOULD NOT be an Evaluation log option
      7. Select Actions > Evaluation for the imported Binary model
      8. You SHOULD see an Evaluation mode modal window explaining the different options you have. Select one of them and click on Evaluate model, once the process is complete repeat this step using the other option. You SHOULD NOT see any error during the process
      9. Click on Actions > Evaluation log for the imported Binary model
      10. You SHOULD see one entry whose Evaluation mode column is Trained model and one of more entries whose Evaluation mode column is Configuration. Leave this browser window open.
      11. Go to a CLI window, cd to moodle's directory and php admin/tool/analytics/cli/evaluate_model.php
      12. You SHOULD see a list of options
      13. php admin/tool/analytics/cli/evaluate_model.php --list SHOULD return you a list of models. Remember Binary model id (it is likely to be the higher number)
      14. php admin/tool/analytics/cli/evaluate_model.php --modelid=BINARY_MODEL_ID_ABOVE --timesplitting='sataychicken' --mode=trainedmodel SHOULD return an error stating that 'timesplitting' and 'mode' options do not work together
      15. php admin/tool/analytics/cli/evaluate_model.php --modelid=BINARY_MODEL_ID_ABOVE --mode=trainedmodel SHOULD finish without errors
      16. php admin/tool/analytics/cli/evaluate_model.php --modelid=BINARY_MODEL_ID_ABOVE SHOULD finish without errors
      17. Return to Binary model Actions > Evaluation log browser window
      18. Copy the Evaluation log URL from the breadcrumb and open a new browser window, there SHOULD be new entries, including both Trained model and Configuration in the Evaluation mode column
      Show
      Prerequisites Install the python ML backend as specified in https://docs.moodle.org/36/en/Analytics#Predictions_processor If you already got it installed you will need to update it to the latest version (pip install -I moodlemlbackend==1.0.0) Install https://github.com/dmonllao/moodle-local_testanalytics Enable analytics and make sure everything is working, refer to the docs on how to set it up if you don't have it already. Go to site admin > analytics > analytics settings. Disable 'onlycli' option and ensure that 'Default predictions processor' is set to 'PHP machine learning backend' Test Log in as admin and go to Site admin > analytics > analytics models Click on Actions > Get predictions for Binary model Click on Actions > Evaluate for Binary model, you SHOULD NOT see a modal window to select the evaluation mode. Click Continue once the process finishes Export Binary model by clicking on Actions > Export configuration . You can delete this model now ( Actions > Delete ). Import the zip file you just downloaded by clicking on Import model button Open the Actions dropdown menu for the imported Binary model. There SHOULD NOT be an Evaluation log option Select Actions > Evaluation for the imported Binary model You SHOULD see an Evaluation mode modal window explaining the different options you have. Select one of them and click on Evaluate model , once the process is complete repeat this step using the other option. You SHOULD NOT see any error during the process Click on Actions > Evaluation log for the imported Binary model You SHOULD see one entry whose Evaluation mode column is Trained model and one of more entries whose Evaluation mode column is Configuration . Leave this browser window open. Go to a CLI window, cd to moodle's directory and php admin/tool/analytics/cli/evaluate_model.php You SHOULD see a list of options php admin/tool/analytics/cli/evaluate_model.php --list SHOULD return you a list of models. Remember Binary model id (it is likely to be the higher number) php admin/tool/analytics/cli/evaluate_model.php --modelid=BINARY_MODEL_ID_ABOVE --timesplitting='sataychicken' --mode=trainedmodel SHOULD return an error stating that 'timesplitting' and 'mode' options do not work together php admin/tool/analytics/cli/evaluate_model.php --modelid=BINARY_MODEL_ID_ABOVE --mode=trainedmodel SHOULD finish without errors php admin/tool/analytics/cli/evaluate_model.php --modelid=BINARY_MODEL_ID_ABOVE SHOULD finish without errors Return to Binary model Actions > Evaluation log browser window Copy the Evaluation log URL from the breadcrumb and open a new browser window, there SHOULD be new entries, including both Trained model and Configuration in the Evaluation mode column

      Training data can not be used as test data during models' evaluation. The purpose of the evaluation process is to check if the model, trained with data from the site, will generate accurate predictions. In MDL-60944 we add the option to export and import trained models so now a model can be trained, although not with data from the site where the model was imported. Sites using models that were trained with other sites' data should be able to evaluate how these trained models perform on their site.

      We need an extra option to admin/tool/analytics/cli/evaluate_model.php and to the models web interface so the performance of trained models can be evaluated using the data on the site as testing data.

        1. Screenshot from 2019-02-07 12-12-25.png
          Screenshot from 2019-02-07 12-12-25.png
          45 kB
        2. Screenshot from 2019-02-08 09-26-42.png
          Screenshot from 2019-02-08 09-26-42.png
          33 kB

            dmonllao David Monllaó
            dmonllao David Monllaó
            Victor Déniz Falcón Victor Déniz Falcón
            Adrian Greeve Adrian Greeve
            Amaia Anabitarte Amaia Anabitarte
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 4 hours
                4h

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.