Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-66476

Use the F1 value instead of the phi value as a final score for a model

XMLWordPrintable

    • Icon: Improvement Improvement
    • Resolution: Fixed
    • Icon: Minor Minor
    • 3.8
    • 3.8
    • Analytics
    • MOODLE_38_STABLE
    • MOODLE_38_STABLE
    • MDL-66476_master
    • Hide
      1. Install the new version of the python package 

        git clone git://github.com/dmonllao/moodle-mlbackend-python.git
        		 
        git checkout MDL-66476
        		 
        cd moodle-mlbackend-python
        		 
        pip3 install -e .

        1. Confirm that you did not see any error during the installation process
      2. Check the version: 

        python3 -m moodlemlbackend.version

        1. Confirm that the version reported was 2.3.0
      3. Set $CFG->pathtopython = '/full/path/to/python3'; in your config.php
      4. Run unit tests for the predictions: 

        vendor/bin/phpunit analytics/tests/prediction_test.php

        1. Confirm that there were no skipped tests
        2. Confirm that all tests passed
        3. Confirm that no Python warnings were shown during the tests
      Show
      Install the new version of the python package git clone git://github.com/dmonllao/moodle-mlbackend-python.git git checkout MDL-66476 cd moodle-mlbackend-python pip3 install -e . Confirm that you did not see any error during the installation process Check the version: python3 -m moodlemlbackend.version Confirm that the version reported was 2.3.0 Set $CFG->pathtopython = '/full/path/to/python3'; in your config.php Run unit tests for the predictions: vendor/bin/phpunit analytics/tests/prediction_test.php Confirm that there were no skipped tests Confirm that all tests passed Confirm that no Python warnings were shown during the tests

      The Phi (MCC) value can be misleading when there is no much data to analyse as its calculation requires the confusion matrix to be full (a != 0 value in all cells). If the confusion matrix is not full the phi value is 0 which is represented as a 50% score when evaluating models in moodle, which is wrong and confusing for users as it may seem that the model is not working. There is no significant difference for the final user if we use the F1 score instead. The fact that we will now (MDL-58992) support multi-class classification may make the confusion matrix issue worst.

            Apetrei Vlad Apetrei
            dmonllao David Monllaó
            David Monllaó David Monllaó
            Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
            Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 3 hours, 5 minutes
                3h 5m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.