Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-66476

Use the F1 value instead of the phi value as a final score for a model

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide
      1. Install the new version of the python package

        git clone git://github.com/dmonllao/moodle-mlbackend-python.git
        git checkout MDL-66476
        cd moodle-mlbackend-python
        pip3 install -e .
        

        1. Confirm that you did not see any error during the installation process
      2. Check the version:

        python3 -m moodlemlbackend.version
        

        1. Confirm that the version reported was 2.3.0
      3. Set $CFG->pathtopython = '/full/path/to/python3'; in your config.php
      4. Run unit tests for the predictions:

        vendor/bin/phpunit analytics/tests/prediction_test.php
        

        1. Confirm that there were no skipped tests
        2. Confirm that all tests passed
        3. Confirm that no Python warnings were shown during the tests
      Show
      Install the new version of the python package git clone git://github.com/dmonllao/moodle-mlbackend-python.git git checkout MDL-66476 cd moodle-mlbackend-python pip3 install -e . Confirm that you did not see any error during the installation process Check the version: python3 -m moodlemlbackend.version Confirm that the version reported was 2.3.0 Set $CFG->pathtopython = '/full/path/to/python3'; in your config.php Run unit tests for the predictions: vendor/bin/phpunit analytics/tests/prediction_test.php Confirm that there were no skipped tests Confirm that all tests passed Confirm that no Python warnings were shown during the tests
    • Affected Branches:
      MOODLE_38_STABLE
    • Fixed Branches:
      MOODLE_38_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      MDL-66476_master

      Description

      The Phi (MCC) value can be misleading when there is no much data to analyse as its calculation requires the confusion matrix to be full (a != 0 value in all cells). If the confusion matrix is not full the phi value is 0 which is represented as a 50% score when evaluating models in moodle, which is wrong and confusing for users as it may seem that the model is not working. There is no significant difference for the final user if we use the F1 score instead. The fact that we will now (MDL-58992) support multi-class classification may make the confusion matrix issue worst.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Apetrei Vlad Apetrei
                Reporter:
                dmonllao David Monllaó
                Peer reviewer:
                David Monllaó
                Integrator:
                Eloy Lafuente (stronk7)
                Tester:
                Eloy Lafuente (stronk7)
                Participants:
                Component watchers:
                Elizabeth Dalton, David Monllaó
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  18/Nov/19

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 3 hours, 5 minutes
                  3h 5m