Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-66004

Allow the python machine learning backend to run from a separate server

XMLWordPrintable

    • MOODLE_38_STABLE
    • MOODLE_38_STABLE
    • MDL-66004_master
    • Hide
      1. Install the new version of the python package using the docker image (more info in moodle-mlbackend-python): 

        docker pull dmonllao/moodle-mlbackend-python:2.2.1-python3.7.4
        		 
        # Note that you need to add --network=moodledocker_default if your are using moodle-docker and you want this container to be visible from the web server.
        		 
        docker run -d -p 5000:5000 --name=mlbackendpython --rm --add-host=mlbackendpython:0.0.0.0 dmonllao/moodle-mlbackend-python:2.2.1-python3.7.4

      2. Set the following in your config.php 

        $CFG->pathtopython = 'python';
        		 
        define('TEST_MLBACKEND_PYTHON_HOST', 'mlbackendpython');
        		 
        define('TEST_MLBACKEND_PYTHON_PORT', 5000);
        		 
        define('TEST_MLBACKEND_PYTHON_USERNAME', 'default');
        		 
        define('TEST_MLBACKEND_PYTHON_PASSWORD', 'sshhhh');

      3. The following command SHOULD NOT return any skipped test and all tests SHOULD pass 

        vendor/bin/phpunit analytics/tests/prediction_test.php --verbose

      4. The following command SHOULD NOT list any other files nor directories than 'moodle 

        docker exec mlbackendpython ls -l /opt/mlbackenddata

      5. Comment out the TEST_MLBACKEND_PYTHON_ constants previously defined in your config.php
      6. Install the python package in your web server 

        git clone git://github.com/dmonllao/moodle-mlbackend-python.git
        		 
        cd moodle-mlbackend-python
        		 
        git checkout MDL-66004
        		 
        pip3 install -e .

      7. Replace python by python3 in your config.php 

        $CFG->pathtopython = 'python3';

      8. The following command SHOULD NOT return any skipped tests and all tests SHOULD pass 

        vendor/bin/phpunit analytics/tests/prediction_test.php --verbose

      9. Uncomment the TEST_MLBACKEND_PYTHON_ constants previously defined in your config.php
      10. Test the ML backend using AWS S3
        1. Create a AWS free account and setup S3 (as an alternative to going through these points below, ping David Monllao in telegram during Perth's office working hours for an AWS test account)
          1. Go to Services > S3 and create a new bucket, the default settings should be fine, save changes. Note the name of the bucket, you will need it later.
          2. Go to IAM service > add a testuser with "Programmatic access" > "Attach existing policies directly" > search for "AmazonS3FullAccess" permission and select it > No need for tags > Save changes
          3. Select the user you just created and click on "Security credentials" tab to create a new access key and secret (copy them, you will need them below)
      11. Stop the current mlbackendpython container and start a new one setting up the required environment variables for AWS S3 

        # Note that you need to add --network=moodledocker_default if your are using moodle-docker and you want this container to be visible from the web server.
        		 
        docker run -d -p 5000:5000 --name=mlbackendpython -e MOODLE_MLBACKEND_PYTHON_S3_BUCKET_NAME=i-am-a-bucket-name -e AWS_ACCESS_KEY_ID=TheKey -e AWS_SECRET_ACCESS_KEY=TheSecretKey --rm --add-host=mlbackendpython:0.0.0.0 dmonllao/moodle-mlbackend-python:2.2.1-python3.7.4

      12. The following command SHOULD NOT return any skipped test and all tests SHOULD pass 

        vendor/bin/phpunit analytics/tests/prediction_test.php --verbose

      13. Confirm that the files generated during the unit tests have been automatically removed from your S3 bucket
      Show
      Install the new version of the python package using the docker image (more info in moodle-mlbackend-python ): docker pull dmonllao/moodle-mlbackend-python:2.2.1-python3.7.4 # Note that you need to add --network=moodledocker_default if your are using moodle-docker and you want this container to be visible from the web server. docker run -d -p 5000:5000 --name=mlbackendpython --rm --add-host=mlbackendpython:0.0.0.0 dmonllao/moodle-mlbackend-python:2.2.1-python3.7.4 Set the following in your config.php $CFG->pathtopython = 'python'; define('TEST_MLBACKEND_PYTHON_HOST', 'mlbackendpython'); define('TEST_MLBACKEND_PYTHON_PORT', 5000); define('TEST_MLBACKEND_PYTHON_USERNAME', 'default'); define('TEST_MLBACKEND_PYTHON_PASSWORD', 'sshhhh'); The following command SHOULD NOT return any skipped test and all tests SHOULD pass vendor/bin/phpunit analytics/tests/prediction_test.php --verbose The following command SHOULD NOT list any other files nor directories than 'moodle docker exec mlbackendpython ls -l /opt/mlbackenddata Comment out the TEST_MLBACKEND_PYTHON_ constants previously defined in your config.php Install the python package in your web server git clone git://github.com/dmonllao/moodle-mlbackend-python.git cd moodle-mlbackend-python git checkout MDL-66004 pip3 install -e . Replace python by python3 in your config.php $CFG->pathtopython = 'python3'; The following command SHOULD NOT return any skipped tests and all tests SHOULD pass vendor/bin/phpunit analytics/tests/prediction_test.php --verbose Uncomment the TEST_MLBACKEND_PYTHON_ constants previously defined in your config.php Test the ML backend using AWS S3 Create a AWS free account and setup S3 (as an alternative to going through these points below, ping David Monllao in telegram during Perth's office working hours for an AWS test account) Go to Services > S3 and create a new bucket, the default settings should be fine, save changes. Note the name of the bucket, you will need it later. Go to IAM service > add a testuser with "Programmatic access" > "Attach existing policies directly" > search for "AmazonS3FullAccess" permission and select it > No need for tags > Save changes Select the user you just created and click on "Security credentials" tab to create a new access key and secret (copy them, you will need them below) Stop the current mlbackendpython container and start a new one setting up the required environment variables for AWS S3 # Note that you need to add --network=moodledocker_default if your are using moodle-docker and you want this container to be visible from the web server. docker run -d -p 5000:5000 --name=mlbackendpython -e MOODLE_MLBACKEND_PYTHON_S3_BUCKET_NAME=i-am-a-bucket-name -e AWS_ACCESS_KEY_ID=TheKey -e AWS_SECRET_ACCESS_KEY=TheSecretKey --rm --add-host=mlbackendpython:0.0.0.0 dmonllao/moodle-mlbackend-python:2.2.1-python3.7.4 The following command SHOULD NOT return any skipped test and all tests SHOULD pass vendor/bin/phpunit analytics/tests/prediction_test.php --verbose Confirm that the files generated during the unit tests have been automatically removed from your S3 bucket

      Moodle includes two machine learning backends, one written in PHP and a much superior option written in python. The python one is the recommended choice. However, we require sys admins to install the moodle-mlbackend-python package into the web server, which is not ideal as the ML training process affects the web server workload.

      The communication between Moodle and the python backend is through files (.csv with training data) and, at the moment, the python ML backend reads the input files from the web server filesystem. Sites with multiple frontend servers can use a directory shared among the web servers (https://docs.moodle.org/37/en/Analytics#Models_output_directory) but that is not enough to keep the processes separated. If we want the python backend to live in a separate server we should send the files through HTTP requests to it.

      We should discuss some points:

      • Do we want a separate mlbackend plugin? We can hide/show admin settings using JS since MDL-52167 but even with that having a separate plugin may be clearer.
      • What security measures should we implement? Or should the python ML service be only available in the local network? We should ideally provide security settings to allow people to connect to servers that are not part of the local network.
      • Python Flask to expose the python package functionalities through an API accessible through HTTP? It seems the best candidate.
      • Docker? Once we have the Flask service we can easily setup a docker image for people to quickly deploy a container with it.

        1. output_against_dockered_server.txt
          545 kB
          Eloy Lafuente (stronk7)
        2. s3storage.png
          85 kB
          Eloy Lafuente (stronk7)
        3. Screenshot from 2019-07-08 15-52-59.png
          24 kB
          David Monllaó
        4. Screenshot from 2019-07-08 15-53-11.png
          64 kB
          David Monllaó
        5. Screenshot from 2019-07-08 17-04-49.png
          22 kB
          David Monllaó
        6. Screenshot from 2019-07-08 17-04-58.png
          62 kB
          David Monllaó
        7. serverstorage.png
          169 kB
          Eloy Lafuente (stronk7)

            dmonllao David Monllaó
            dmonllao David Monllaó
            Peter Dias Peter Dias
            Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
            Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
            Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 2 weeks, 2 hours, 27 minutes
                2w 2h 27m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.