Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-73431

User enrolments sync on every web service call

    XMLWordPrintable

Details

    • MOODLE_310_STABLE, MOODLE_311_STABLE, MOODLE_39_STABLE, MOODLE_401_STABLE
    • MOODLE_401_STABLE
    • MDL-73431-master-v2
    • Hide

      Configuration 

      Configure integration

      1. On you moodle DB run following SQL to create a new table

        CREATE TABLE integration (moodle_user varchar(10), moodle_course varchar(10), moodle_role varchar(10));

         # Go to /admin/settings.php?section=manageenrols and enable External database method

      2. Go to /admin/settings.php?section=enrolsettingsdatabase and set following settings:

        dbtype = your DB type (e.g. if using PostgreSQL, choose "Postgres")
        dbhost = value of $CFG->dbhost from your config.php
        dbuser = value of $CFG->dbuser from your config.php
        dbpass = value of $CFG->dbpass from your config.php
        dbname = value of $CFG->dbname from your config.php
        localcoursefield = shortname
        localuserfield = username
        localrolefield = shortname
        remoteenroltable = integration
        remotecoursefield = moodle_course
        remoteuserfield = moodle_user
        remoterolefield = moodle_role
        

      3. Create user with username "student"
      4. Create 3 courses with short names "course1", "course2" and "course3".

      Configure WS

      1. Navigate to /admin/roles/define.php?action=edit&roleid=7 (Admin > Users > Define roles > Edit Authenticated users) and allow "webservice/rest:use"" permission
      2. Navigate to /admin/search.php?query=enablewebservices (Admin > Advanced features) and Enable web services 
      3. Navigate to /admin/settings.php?section=webserviceprotocols (Admin > Server > Web services > Manage protocols) and enable REST protocol
      4. Navigate to /admin/settings.php?section=externalservices (Admin > Server > Web services > External services), create a service "Test service" with shortname 'testservice' and enable it.
      5. On the next screen add block_recentlyaccesseditems_get_recent_items function to that service
      6. Navigate to /admin/webservice/tokens.php (Admin > Server > Web services > Manage tokens) and Create a token for Student user for Test service.
      7. Note that token as it will be used in tests later.

      Configure RSS

      1. Navigate to /admin/search.php?query=enablerssfeeds and enable RSS feeds globally and for Forum (You need to save to enable RSS feeds globally then after the reload, you can enable RSS fields for Forum)
      2. Navigate to course3 and Navigate to Announcements forum activity settings.
      3. Set RSS feed for this activity to Discussion
      4. Number of RSS recent articles = 50
      5. Save settings
      6. Add a new discussion
      7. Navigate to the student's profile page. We'll come back to this later.

      Testing

      Web login

      1. Run following SQL to inset an enrolment record

        INSERT INTO integration VALUES('student', 'course1', 'student');

      2. Login as student in a different browser (incognito session).
      3. Confirm that student was successfully enrolled into the course1 using External database method.
      4. View course1's "Announcements" forum
      5. Run following SQLs to update enrolments:

        INSERT INTO integration VALUES('student', 'course2', 'student');

        INSERT INTO integration VALUES('student', 'course3', 'student');

        DELETE FROM integration WHERE moodle_course = 'course1';

      6. Log out
      7. Log in as student again.
      8. Confirm that student is not enrolled into the course1.
      9. Confirm that student is now enrolled into the course2 and course3 using External database method.
      10. View course2's and course3's "Announcements" forums

      WS login

      1. Run following SQL 

        DELETE FROM integration;

      2. Open a terminal and run

         curl '<YOUR_MOODLE_URL>/webservice/rest/server.php?wstoken=<TOKEN>&wsfunction=block_recentlyaccesseditems_get_recent_items&moodlewsrestformat=json' | python -m json.tool
         

      3. Confirm that you get an empty array
      4. Run following SQL 

        INSERT INTO integration VALUES('student', 'course1', 'student');

      5. Run the curl command again
      6. Confirm that user you still get an empty array
      7. Add the following in config.php

        $CFG->enrolments_sync_interval = 10;

      8. Wait for at least 10 seconds.
      9. Run the curl command again
      10. Confirm that course1's announcements forum is returned, but not course2's and course3's.
      11. Run following SQL 

        DELETE FROM integration WHERE moodle_course = 'course1';
        INSERT INTO integration VALUES('student', 'course2', 'student');

      12. Make sure at least 10 seconds have passed
      13. Run the curl command again
      14. Confirm that course2's announcements forum is returned, but not course1's and course3's.
      15. Run following SQL 

        DELETE FROM integration WHERE moodle_course = 'course2';
        INSERT INTO integration VALUES('student', 'course3', 'student');

      16. Set $CFG->enrolments_sync_interval = 3600;
      17. Run the curl command again
      18. Confirm that course2's announcements forum is returned, but still not course1's and course3's.

      Login to get token

      1. Navigate to /login/token.php?username=student&password=<PASSWORD>&service=testservice
      2. Confirm that the student's web service token is returned.
      3. Back on the admin's browser window reload the student's profile page
      4. Confirm that user is enrolled only in course3

      RSS login

      1. Back on the student's browser window, navigate to Course 3 and Announcements forum activity
      2. Click on "RSS feed of discussions"" and note the URL (you will use it for testing).
      3. Logout as student
      4. Set $CFG->enrolments_sync_interval = 10;
      5. Run following SQL 

        DELETE FROM integration;

      6. Navigate to RSS URL from step 2
      7. Confirm that the XML contains an error like "Your RSS link does not contain a valid authentication token."
      8. On the admin's browser, reload the student's profile page and Confirm that the student is not enrolled in course1, course2 and course3
      9. Comment out $CFG->enrolments_sync_interval = 10; or set it to 3600
      10. Run following SQL 

        INSERT INTO integration VALUES('student', 'course1', 'student');

      11. Navigate to RSS URL from step 2
      12. On the admin's browser, reload the student's profile page and Confirm that the student is not enrolled in course1, course2 and course3
      13. Set $CFG->enrolments_sync_interval = 10;
      14. Navigate to RSS URL from step 2
      15. On the admin's browser, reload the student's profile page and Confirm that the student is now enrolled in course1, but not in course2 and course3
      16. Run following SQL 

        DELETE FROM integration WHERE moodle_course = 'course1';
        INSERT INTO integration VALUES('student', 'course2', 'student');

      17. Navigate to RSS URL from step 2
      18. Confirm that user is not enrolled in course1
      19. Confirm that user is enrolled in course2
      20. Run following SQL 

        DELETE FROM integration WHERE moodle_course = 'course2';
        INSERT INTO integration VALUES('student', 'course3', 'student');

      21. Set $CFG->enrolments_sync_interval = 3600;
      22. Navigate to RSS URL from step 2
      23. Confirm that user is still enrolled in course2
      24. Confirm that user is still not enrolled in course3

      Scheduled task

      1. Continuing from the previous step, run the curl command in the terminal.
      2. Confirm that course2's Announcements forum is returned.
      3. Run the database enrolment sync scheduled task:

        php admin/cli/scheduled_task.php --execute="\\enrol_database\\task\\sync_enrolments"
        

      4. Run the curl command again.
      5. Confirm that course3's Announcements forum is returned this time and not course2's.
      Show
      Configuration  Configure integration On you moodle DB run following SQL to create a new table CREATE TABLE integration (moodle_user varchar( 10 ), moodle_course varchar( 10 ), moodle_role varchar( 10 ));  # Go to /admin/settings.php?section=manageenrols and enable External database method Go to /admin/settings.php?section=enrolsettingsdatabase and set following settings: dbtype = your DB type (e.g. if using PostgreSQL, choose "Postgres" ) dbhost = value of $CFG->dbhost from your config.php dbuser = value of $CFG->dbuser from your config.php dbpass = value of $CFG->dbpass from your config.php dbname = value of $CFG->dbname from your config.php localcoursefield = shortname localuserfield = username localrolefield = shortname remoteenroltable = integration remotecoursefield = moodle_course remoteuserfield = moodle_user remoterolefield = moodle_role Create user with username "student" Create 3 courses with short names "course1", "course2" and "course3". Configure WS Navigate to /admin/roles/define.php?action=edit&roleid=7 (Admin > Users > Define roles > Edit Authenticated users) and allow " webservice/rest:use "" permission Navigate to /admin/search.php?query=enablewebservices (Admin > Advanced features) and Enable web services  Navigate to /admin/settings.php?section=webserviceprotocols (Admin > Server > Web services > Manage protocols) and enable REST protocol Navigate to /admin/settings.php?section=externalservices (Admin > Server > Web services > External services), create a service "Test service" with shortname 'testservice' and enable it. On the next screen add block_recentlyaccesseditems_get_recent_items function to that service Navigate to /admin/webservice/tokens.php (Admin > Server > Web services > Manage tokens) and Create a token for Student user for Test service. Note that token as it will be used in tests later. Configure RSS Navigate to /admin/search.php?query=enablerssfeeds and enable RSS feeds globally and for Forum (You need to save to enable RSS feeds globally then after the reload, you can enable RSS fields for Forum) Navigate to course3 and Navigate to Announcements forum activity settings. Set RSS feed for this activity to Discussion Number of RSS recent articles = 50 Save settings Add a new discussion Navigate to the student's profile page. We'll come back to this later. Testing Web login Run following SQL to inset an enrolment record INSERT INTO integration VALUES( 'student' , 'course1' , 'student' ); Login as student in a different browser (incognito session). Confirm that student was successfully enrolled into the course1 using External database method. View course1's "Announcements" forum Run following SQLs to update enrolments: INSERT INTO integration VALUES( 'student' , 'course2' , 'student' ); INSERT INTO integration VALUES( 'student' , 'course3' , 'student' ); DELETE FROM integration WHERE moodle_course = 'course1' ; Log out Log in as student again. Confirm that student is not enrolled into the course1. Confirm that student is now enrolled into the course2 and course3 using External database method. View course2's and course3's "Announcements" forums WS login Run following SQL  DELETE FROM integration; Open a terminal and run curl '<YOUR_MOODLE_URL>/webservice/rest/server.php?wstoken=<TOKEN>&wsfunction=block_recentlyaccesseditems_get_recent_items&moodlewsrestformat=json' | python -m json.tool Confirm that you get an empty array Run following SQL  INSERT INTO integration VALUES('student', 'course1', 'student'); Run the curl command again Confirm that user you still get an empty array Add the following in config.php $CFG->enrolments_sync_interval = 10; Wait for at least 10 seconds. Run the curl command again Confirm that course1's announcements forum is returned, but not course2's and course3's. Run following SQL  DELETE FROM integration WHERE moodle_course = 'course1'; INSERT INTO integration VALUES('student', 'course2', 'student'); Make sure at least 10 seconds have passed Run the curl command again Confirm that course2's announcements forum is returned, but not course1's and course3's. Run following SQL  DELETE FROM integration WHERE moodle_course = 'course2'; INSERT INTO integration VALUES('student', 'course3', 'student'); Set $CFG->enrolments_sync_interval = 3600; Run the curl command again Confirm that course2's announcements forum is returned, but still not course1's and course3's. Login to get token Navigate to /login/token.php?username=student&password=<PASSWORD>&service=testservice Confirm that the student's web service token is returned. Back on the admin's browser window reload the student's profile page Confirm that user is enrolled only in course3 RSS login Back on the student's browser window, navigate to Course 3 and Announcements forum activity Click on " RSS feed of discussions "" and note the URL (you will use it for testing). Logout as student Set $CFG->enrolments_sync_interval = 10; Run following SQL  DELETE FROM integration; Navigate to RSS URL from step 2 Confirm that the XML contains an error like "Your RSS link does not contain a valid authentication token." On the admin's browser, reload the student's profile page and Confirm that the student is not enrolled in course1, course2 and course3 Comment out $CFG->enrolments_sync_interval = 10; or set it to 3600 Run following SQL  INSERT INTO integration VALUES('student', 'course1', 'student'); Navigate to RSS URL from step 2 On the admin's browser, reload the student's profile page and Confirm that the student is not enrolled in course1, course2 and course3 Set $CFG->enrolments_sync_interval = 10; Navigate to RSS URL from step 2 On the admin's browser, reload the student's profile page and Confirm that the student is now enrolled in course1, but not in course2 and course3 Run following SQL  DELETE FROM integration WHERE moodle_course = 'course1'; INSERT INTO integration VALUES('student', 'course2', 'student'); Navigate to RSS URL from step 2 Confirm that user is not enrolled in course1 Confirm that user is enrolled in course2 Run following SQL  DELETE FROM integration WHERE moodle_course = 'course2'; INSERT INTO integration VALUES('student', 'course3', 'student'); Set $CFG->enrolments_sync_interval = 3600; Navigate to RSS URL from step 2 Confirm that user is still enrolled in course2 Confirm that user is still not enrolled in course3 Scheduled task Continuing from the previous step, run the curl command in the terminal. Confirm that course2's Announcements forum is returned. Run the database enrolment sync scheduled task: php admin/cli/scheduled_task.php --execute="\\enrol_database\\task\\sync_enrolments" Run the curl command again. Confirm that course3's Announcements forum is returned this time and not course2's.

    Description

      On each external web service call Moodle syncs user's enrolments which leads to a performance issue if using slow enrolment plugin that  requires users to be synced form an external resource( e.g. enrol_database).

      That could become a  performance issue when an external mobile application calls several moodle web services on a single page. This will trigger each of the calls to sync enrolments for the same user. In scale it could quickly become a big issue. 

      See code  https://github.com/moodle/moodle/blob/master/webservice/lib.php#L1134 

      I have attached a callgraph for a simple WS call from one of our instances.

       

      Attachments

        1. callgraph.png
          callgraph.png
          1.11 MB
        2. Web Login_Screenshot.png
          Web Login_Screenshot.png
          79 kB
        3. WS Login_Screenshot.png
          WS Login_Screenshot.png
          191 kB
        4. Login to get token_Screenshot.PNG
          Login to get token_Screenshot.PNG
          39 kB
        5. RSS Login_Screenshot.png
          RSS Login_Screenshot.png
          389 kB
        6. Scheduled task_Screenshot.png
          Scheduled task_Screenshot.png
          136 kB

        Activity

          People

            dmitriim Dmitrii Metelkin
            dmitriim Dmitrii Metelkin
            Brendan Heywood Brendan Heywood
            Jun Pataleta Jun Pataleta
            Gladys Basiana Gladys Basiana
            Matteo Scaramuccia, Andrew Lyons, Huong Nguyen, Jun Pataleta, Michael Hawkins, Shamim Rezaie, Simey Lameze, Stevani Andolo, Juan Leyva, Amaia Anabitarte, Carlos Escobedo, Ferran Recio, Ilya Tregubov, Laurent David, Raquel Ortega, Sara Arjona (@sarjona)
            Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:
              14/Nov/22

              Time Tracking

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 7 hours
                7h