Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-67622

The evaluation metrics are not ready for multiclass in the python ML backend

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.8, 3.9
    • Fix Version/s: 3.8.2
    • Component/s: Analytics
    • Labels:
    • Testing Instructions:
      Hide

      Prerequisites:

      1. Need python3
      2. Need pip3 (apt-get install python3-pip)

      Testing

      1. Install the new version of the python package 

        git clone git://github.com/dmonllao/moodle-mlbackend-python.git
        		 
        cd moodle-mlbackend-python
        		 
        git checkout MDL-67622_master
        		 
        pip3 install setuptools wheel
        		 
        pip3 install -e .

        1. Confirm that you did not see any error during the installation process
      2. Check the version: 

        python3 -m moodlemlbackend.version

        1. Confirm that the version reported was 2.3.1
      3. Set $CFG->pathtopython = '/full/path/to/python3'; in your config.php (to just specify 'python3' is probably enough)
      4. Unit tests including PHPUNIT_LONGTEST should pass and no analytics tests should be skipped. You can do it running the command below: 

        vendor/bin/phpunit analytics/tests/prediction_test.php

      Show
      Prerequisites: Need python3 Need pip3 (apt-get install python3-pip) Testing Install the new version of the python package git clone git://github.com/dmonllao/moodle-mlbackend-python.git cd moodle-mlbackend-python git checkout MDL-67622_master pip3 install setuptools wheel pip3 install -e . Confirm that you did not see any error during the installation process Check the version: python3 -m moodlemlbackend.version Confirm that the version reported was 2.3.1 Set $CFG->pathtopython = '/full/path/to/python3'; in your config.php (to just specify 'python3' is probably enough) Unit tests including PHPUNIT_LONGTEST should pass and no analytics tests should be skipped. You can do it running the command below: vendor/bin/phpunit analytics/tests/prediction_test.php
    • Affected Branches:
      MOODLE_38_STABLE, MOODLE_39_STABLE
    • Fixed Branches:
      MOODLE_38_STABLE
    • Pull from Repository:
    • Pull 3.8 Branch:
    • Pull 3.8 Diff URL:
    • Pull Master Branch:
      MDL-67622_master
    • Pull Master Diff URL:

      Description

      In MDL-58992 we added multi-class capabilities to the ML backends in moodle core. We didn't update calculate_metrics function in the python ML backend. This function is responsible of returning the accuracy metrics that serve as a measure of the quality of the model. Also, the f1 score should be calculated using a weighted average like we do for the PHP ML backend.

        Attachments

          Issue Links

          blocks

          New Feature - A new Moodle feature which has yet to be developed. MDL-67040 Improve the performance of the Analytics ML layer when using small datasets

          • Minor - Minor loss of function where workaround is possible
          • Closed
          caused a regression

          Improvement - An enhancement to an existing Moodle feature. MDL-58992 Add multi-class capabilities to prediction processors

          • Minor - Minor loss of function where workaround is possible
          • Closed

            Activity

              People

              Assignee:
              dmonllao David Monllaó
              Reporter:
              dmonllao David Monllaó
              Peer reviewer:
              Víctor Déniz Falcón
              Integrator:
              Eloy Lafuente (stronk7)
              Tester:
              Eloy Lafuente (stronk7)
              Participants:
              Component watchers:
              Elizabeth Dalton, Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Sara Arjona (@sarjona)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Fix Release Date:
                9/Mar/20

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 days, 50 minutes
                  2d 50m