Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-66476

Use the F1 value instead of the phi value as a final score for a model

    XMLWordPrintable

Details

    • MOODLE_38_STABLE
    • MOODLE_38_STABLE
    • MDL-66476_master
    • Hide
      1. Install the new version of the python package

        git clone git://github.com/dmonllao/moodle-mlbackend-python.git
        git checkout MDL-66476
        cd moodle-mlbackend-python
        pip3 install -e .
        

        1. Confirm that you did not see any error during the installation process
      2. Check the version:

        python3 -m moodlemlbackend.version
        

        1. Confirm that the version reported was 2.3.0
      3. Set $CFG->pathtopython = '/full/path/to/python3'; in your config.php
      4. Run unit tests for the predictions:

        vendor/bin/phpunit analytics/tests/prediction_test.php
        

        1. Confirm that there were no skipped tests
        2. Confirm that all tests passed
        3. Confirm that no Python warnings were shown during the tests
      Show
      Install the new version of the python package git clone git://github.com/dmonllao/moodle-mlbackend-python.git git checkout MDL-66476 cd moodle-mlbackend-python pip3 install -e . Confirm that you did not see any error during the installation process Check the version: python3 -m moodlemlbackend.version Confirm that the version reported was 2.3.0 Set $CFG->pathtopython = '/full/path/to/python3'; in your config.php Run unit tests for the predictions: vendor/bin/phpunit analytics/tests/prediction_test.php Confirm that there were no skipped tests Confirm that all tests passed Confirm that no Python warnings were shown during the tests

    Description

      The Phi (MCC) value can be misleading when there is no much data to analyse as its calculation requires the confusion matrix to be full (a != 0 value in all cells). If the confusion matrix is not full the phi value is 0 which is represented as a 50% score when evaluating models in moodle, which is wrong and confusing for users as it may seem that the model is not working. There is no significant difference for the final user if we use the F1 score instead. The fact that we will now (MDL-58992) support multi-class classification may make the confusion matrix issue worst.

      Attachments

        Issue Links

          Activity

            People

              Apetrei Vlad Apetrei
              dmonllao David Monllaó
              David Monllaó David Monllaó
              Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
              Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
              Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Ilya Tregubov, Laurent David, Raquel Ortega, Sara Arjona (@sarjona)
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                18/Nov/19

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 3 hours, 5 minutes
                  3h 5m