Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-59988

Discard datasets that are not ready for training / prediction and append them later to other datasets when ready

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.4
    • Fix Version/s: 3.4
    • Component/s: Analytics
    • Labels:
    • Testing Instructions:
      Hide

      Setup what you need to test this

      1. Clone local/testanalytics plugin (or get the latest code if you already had it) git clone git://github.com/dmonllao/moodle-local_testanalytics.git
      2. Setup a new site

      Test php backend

      1. Disable analytics/onlycli setting in admin/settings.php?section=analyticssettings and select the PHP machine learning processor
      2. Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions"
      3. Create a new user without picture which username does not contain 'prediction'
      4. Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions"
      5. You SHOULD see the message "There is no new data that can be used for training" along with other messages
      6. Create a new user without picture which username does not contain 'prediction'
      7. Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions"
      8. You SHOULD see the message "Training process finished"
      9. Create a new user without picture which username contains 'prediction'
      10. Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions"
      11. Back in admin/tool/analytics/index.php you SHOULD see that there is a new dropdown in "Users without pic" model
      12. Execute the following query select f.id, auf.action from mdl_files f join mdl_analytics_models am on component = 'analytics' and f.itemid = am.id left join mdl_analytics_used_files auf on auf.fileid = f.id where am.target = '\local_testanalytics\analytics\target\without_picture' and f.filepath like '/timesplitting%' and f.filename != '.' and f.filename != 'evaluation.csv' order by f.timecreated DESC
      13. All returned rows' action values SHOULD be 'trained' or 'predicted'

      Test python backend

      1. Install the python machine learning backend package as described in https://docs.moodle.org/dev/Analytics_models#Predictions_processor
      2. In "Site admin > Server > System paths" set the path to python (just 'python' should be enough)
      3. Go to admin/settings.php?section=analyticssettings and change your predictions processor to the python one (this will delete all previous trained algorithms and predictions)
      4. Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions"
      5. You SHOULD see "There is no new data that can be used for training"
      6. Create a new user with picture which username does not contain 'prediction'
      7. Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions"
      8. You SHOULD see "Training process finished" and "Prediction process finished"
      9. Execute the following query select f.id, auf.action from mdl_files f join mdl_analytics_models am on component = 'analytics' and f.itemid = am.id left join mdl_analytics_used_files auf on auf.fileid = f.id where am.target = '\local_testanalytics\analytics\target\without_picture' and f.filepath like '/timesplitting%' and f.filename != '.' and f.filename != 'evaluation.csv' order by f.timecreated DESC
      10. All returned rows' action values SHOULD be 'trained' or 'predicted'
      11. Create a new user without picture which username does not contain 'prediction'
      12. Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions"
      13. You SHOULD see "There is no new data that can be used for training"
      14. Execute the following query select f.id, auf.action from mdl_files f join mdl_analytics_models am on component = 'analytics' and f.itemid = am.id left join mdl_analytics_used_files auf on auf.fileid = f.id where am.target = '\local_testanalytics\analytics\target\without_picture' and f.filepath like '/timesplitting%' and f.filename != '.' and f.filename != 'evaluation.csv' order by f.timecreated DESC
      15. One of the returned rows SHOULD have an empty action column
      16. Create a new user with picture which username does not contain 'prediction'
      17. Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions"
      18. You SHOULD see "Training process finished"
      19. Execute the following query select f.id, auf.action from mdl_files f join mdl_analytics_models am on component = 'analytics' and f.itemid = am.id left join mdl_analytics_used_files auf on auf.fileid = f.id where am.target = '\local_testanalytics\analytics\target\without_picture' and f.filepath like '/timesplitting%' and f.filename != '.' and f.filename != 'evaluation.csv' order by f.timecreated DESC
      20. All returned rows' action values SHOULD be 'trained' or 'predicted'
      21. Create a new user with picture which username contains 'prediction'
      22. Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions"
      23. You SHOULD see "Prediction process finished"
      Show
      Setup what you need to test this Clone local/testanalytics plugin (or get the latest code if you already had it) git clone git://github.com/dmonllao/moodle-local_testanalytics.git Setup a new site Test php backend Disable analytics/onlycli setting in admin/settings.php?section=analyticssettings and select the PHP machine learning processor Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions" Create a new user without picture which username does not contain 'prediction' Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions" You SHOULD see the message "There is no new data that can be used for training" along with other messages Create a new user without picture which username does not contain 'prediction' Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions" You SHOULD see the message "Training process finished" Create a new user without picture which username contains 'prediction' Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions" Back in admin/tool/analytics/index.php you SHOULD see that there is a new dropdown in "Users without pic" model Execute the following query select f.id, auf.action from mdl_files f join mdl_analytics_models am on component = 'analytics' and f.itemid = am.id left join mdl_analytics_used_files auf on auf.fileid = f.id where am.target = '\local_testanalytics\analytics\target\without_picture' and f.filepath like '/timesplitting%' and f.filename != '.' and f.filename != 'evaluation.csv' order by f.timecreated DESC All returned rows' action values SHOULD be 'trained' or 'predicted' Test python backend Install the python machine learning backend package as described in https://docs.moodle.org/dev/Analytics_models#Predictions_processor In "Site admin > Server > System paths" set the path to python (just 'python' should be enough) Go to admin/settings.php?section=analyticssettings and change your predictions processor to the python one (this will delete all previous trained algorithms and predictions) Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions" You SHOULD see "There is no new data that can be used for training" Create a new user with picture which username does not contain 'prediction' Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions" You SHOULD see "Training process finished" and "Prediction process finished" Execute the following query select f.id, auf.action from mdl_files f join mdl_analytics_models am on component = 'analytics' and f.itemid = am.id left join mdl_analytics_used_files auf on auf.fileid = f.id where am.target = '\local_testanalytics\analytics\target\without_picture' and f.filepath like '/timesplitting%' and f.filename != '.' and f.filename != 'evaluation.csv' order by f.timecreated DESC All returned rows' action values SHOULD be 'trained' or 'predicted' Create a new user without picture which username does not contain 'prediction' Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions" You SHOULD see "There is no new data that can be used for training" Execute the following query select f.id, auf.action from mdl_files f join mdl_analytics_models am on component = 'analytics' and f.itemid = am.id left join mdl_analytics_used_files auf on auf.fileid = f.id where am.target = '\local_testanalytics\analytics\target\without_picture' and f.filepath like '/timesplitting%' and f.filename != '.' and f.filename != 'evaluation.csv' order by f.timecreated DESC One of the returned rows SHOULD have an empty action column Create a new user with picture which username does not contain 'prediction' Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions" You SHOULD see "Training process finished" Execute the following query select f.id, auf.action from mdl_files f join mdl_analytics_models am on component = 'analytics' and f.itemid = am.id left join mdl_analytics_used_files auf on auf.fileid = f.id where am.target = '\local_testanalytics\analytics\target\without_picture' and f.filepath like '/timesplitting%' and f.filename != '.' and f.filename != 'evaluation.csv' order by f.timecreated DESC All returned rows' action values SHOULD be 'trained' or 'predicted' Create a new user with picture which username contains 'prediction' Go to admin/tool/analytics/index.php, locate "Users without pic" model and press Actions > "Get predictions" You SHOULD see "Prediction process finished"
    • Affected Branches:
      MOODLE_34_STABLE
    • Fixed Branches:
      MOODLE_34_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      MDL-59988_master

      Description

      This issue is about having a way for machine learning backends to discard a dataset and wait until the dataset is "good enough" to be used for training or prediction.

      • The python backend needs samples for each class; I propose to wait until there are samples belonging to all classes to train the models for all different mlbackend plugins as it is not that strange what happens with python backend.
      • PHP ml backend requires at least 2 samples with at least 1 different value to work, fail gracefully if the provided dataset does not contain them.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              dmonllao David Monllaó
              Reporter:
              dmonllao David Monllaó
              Integrator:
              Andrew Nicols
              Tester:
              Ryan Wyllie
              Participants:
              Component watchers:
              Elizabeth Dalton, Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Sara Arjona (@sarjona), Víctor Déniz Falcón
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Fix Release Date:
                13/Nov/17