Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-66004

Allow the python machine learning backend to run from a separate server

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide
      1. Install the new version of the python package using the docker image (more info in moodle-mlbackend-python):

        docker pull dmonllao/moodle-mlbackend-python:2.1.0-python3.7.3
        # Note that you need to add --network=moodledocker_default if your are using moodle-docker and you want this container to be visible from the web server.
        docker run -d -p 5000:5000 --name=mlbackendpython --rm --add-host=mlbackendpython:0.0.0.0 dmonllao/moodle-mlbackend-python:2.1.0-python3.7.3
        

      2. Set the following in your config.php

        $CFG->pathtopython = 'python';
        define('TEST_MLBACKEND_PYTHON_HOST', 'mlbackendpython');
        define('TEST_MLBACKEND_PYTHON_PORT', 5000);
        define('TEST_MLBACKEND_PYTHON_BACKENDID', 'moodle');
        define('TEST_MLBACKEND_PYTHON_SECRET', 'sshhhh');
        

      3. The following command SHOULD NOT return any skipped test for any dataProvider case whose ending is -server (Example: "case-\mlbackend_python\processor-server" SHOULD NOT be skipped", "case-\mlbackend_python\processor" will be skipped unless you also installed the python package in your web server)

        vendor/bin/phpunit analytics/tests/prediction_test.php --verbose

      1. Install the python package in your web server

        git clone git://github.com/dmonllao/moodle-mlbackend-python.git
        git checkout MDL-66004
        cd moodle-mlbackend-python
        pip3 install -e .
        

      2. Replace python by python3 in your config.php

        $CFG->pathtopython = 'python3';
        

      3. The following command SHOULD NOT return any skipped test and all tests SHOULD pass

        vendor/bin/phpunit analytics/tests/prediction_test.php --verbose

      4. Test the ML backend using AWS S3
        1. Create a AWS free account and setup S3
          1. Go to Services > S3 and create a new bucket, the default settings should be fine, save changes. Note the name of the bucket, you will need it later.
          2. Go to IAM service > add a testuser with "Programmatic access" > "Attach existing policies directly" > search for "AmazonS3FullAccess" permission and select it > No need for tags > Save changes
          3. Select the user you just created and click on "Security credentials" tag to create a new access key and secret (copy them, you will need them below)
        2. Stop the current mlbackendpython container and start a new one setting up the required environment variables for AWS S3

          docker run -d -p 5000:5000 --name=mlbackendpython -e MOODLE_MLBACKEND_PYTHON_S3_BUCKET_NAME=i-am-a-bucket-name -e AWS_ACCESS_KEY_ID=TheKey -e AWS_SECRET_ACCESS_KEY=TheSecretKey --rm --add-host=mlbackendpython:0.0.0.0 dmonllao/moodle-mlbackend-python:2.1.0-python3.7.3
          

      5. The following command SHOULD NOT return any skipped test and all tests SHOULD pass

        vendor/bin/phpunit analytics/tests/prediction_test.php --verbose

      Show
      Install the new version of the python package using the docker image (more info in moodle-mlbackend-python ): docker pull dmonllao/moodle-mlbackend-python:2.1.0-python3.7.3 # Note that you need to add --network=moodledocker_default if your are using moodle-docker and you want this container to be visible from the web server. docker run -d -p 5000:5000 --name=mlbackendpython --rm --add-host=mlbackendpython:0.0.0.0 dmonllao/moodle-mlbackend-python:2.1.0-python3.7.3 Set the following in your config.php $CFG->pathtopython = 'python'; define('TEST_MLBACKEND_PYTHON_HOST', 'mlbackendpython'); define('TEST_MLBACKEND_PYTHON_PORT', 5000); define('TEST_MLBACKEND_PYTHON_BACKENDID', 'moodle'); define('TEST_MLBACKEND_PYTHON_SECRET', 'sshhhh'); The following command SHOULD NOT return any skipped test for any dataProvider case whose ending is -server (Example: "case-\mlbackend_python\processor-server" SHOULD NOT be skipped", "case-\mlbackend_python\processor" will be skipped unless you also installed the python package in your web server) vendor/bin/phpunit analytics/tests/prediction_test.php --verbose Install the python package in your web server git clone git://github.com/dmonllao/moodle-mlbackend-python.git git checkout MDL-66004 cd moodle-mlbackend-python pip3 install -e . Replace python by python3 in your config.php $CFG->pathtopython = 'python3'; The following command SHOULD NOT return any skipped test and all tests SHOULD pass vendor/bin/phpunit analytics/tests/prediction_test.php --verbose Test the ML backend using AWS S3 Create a AWS free account and setup S3 Go to Services > S3 and create a new bucket, the default settings should be fine, save changes. Note the name of the bucket, you will need it later. Go to IAM service > add a testuser with "Programmatic access" > "Attach existing policies directly" > search for "AmazonS3FullAccess" permission and select it > No need for tags > Save changes Select the user you just created and click on "Security credentials" tag to create a new access key and secret (copy them, you will need them below) Stop the current mlbackendpython container and start a new one setting up the required environment variables for AWS S3 docker run -d -p 5000:5000 --name=mlbackendpython -e MOODLE_MLBACKEND_PYTHON_S3_BUCKET_NAME=i-am-a-bucket-name -e AWS_ACCESS_KEY_ID=TheKey -e AWS_SECRET_ACCESS_KEY=TheSecretKey --rm --add-host=mlbackendpython:0.0.0.0 dmonllao/moodle-mlbackend-python:2.1.0-python3.7.3 The following command SHOULD NOT return any skipped test and all tests SHOULD pass vendor/bin/phpunit analytics/tests/prediction_test.php --verbose
    • Affected Branches:
      MOODLE_38_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      MDL-66004_master

      Description

      Moodle includes two machine learning backends, one written in PHP and a much superior option written in python. The python one is the recommended choice. However, we require sys admins to install the moodle-mlbackend-python package into the web server, which is not ideal as the ML training process affects the web server workload.

      The communication between Moodle and the python backend is through files (.csv with training data) and, at the moment, the python ML backend reads the input files from the web server filesystem. Sites with multiple frontend servers can use a directory shared among the web servers (https://docs.moodle.org/37/en/Analytics#Models_output_directory) but that is not enough to keep the processes separated. If we want the python backend to live in a separate server we should send the files through HTTP requests to it.

      We should discuss some points:

      • Do we want a separate mlbackend plugin? We can hide/show admin settings using JS since MDL-52167 but even with that having a separate plugin may be clearer.
      • What security measures should we implement? Or should the python ML service be only available in the local network? We should ideally provide security settings to allow people to connect to servers that are not part of the local network.
      • Python Flask to expose the python package functionalities through an API accessible through HTTP? It seems the best candidate.
      • Docker? Once we have the Flask service we can easily setup a docker image for people to quickly deploy a container with it.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                dmonllao David Monllaó
                Reporter:
                dmonllao David Monllaó
                Participants:
                Component watchers:
                Elizabeth Dalton, David Monllaó
              • Votes:
                2 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 week, 4 days
                  1w 4d