Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-34867

Improve course log export resource consumption and reliability

    XMLWordPrintable

    Details

    • Testing Instructions:
      Hide
      1. Run all and individual unit tests:

        ./vendor/bin/phpunit admin/tool/log/store/standard/tests/store_test.php
        ./vendor/bin/phpunit admin/tool/log/store/legacy/tests/store_test.php
        ./vendor/bin/phpunit
        

      2. Behat tests should pass, ask integrators about failures in log reports related tests
      3. Next, test legacy and the new standard logs.
        1. Standard logs:
          • Navigate to Site administration > Reports > Logs and select “Standard log” from the dropdown and click “Get these logs”. If that option is not available then you are already viewing the standard log by default. If you do not see much data listed then you can generate it by performing different actions such as visiting a course page, logging in with invalid credentials, etc. Create another test user and also perform those action in order to get log data generated. That's so we can filter the logs by that user later.
          • For each of the formats available under the “Download table data as” label, select the format and click “Download”. Open the downloaded file and verify that the data in the file, including the headers, match the data on the page.
          • Filter the log table by some criteria such as course or participants and click “Get these logs”. For each of the formats available under the “Download table data as” label select the format and click “Download”. Open the downloaded file and verify that the data in the file, including the headers, match the data on the page.
        2. Legacy logging:
          • Since M27 log data is no longer stored in the table mdl_log by default but we will need some data generated in order to verify the downloaded data. You can enable legacy logging from Site administration > Plugins > Logging > Manage log stores. Select the option "Log legacy data" and save changes.
          • From the Logs page select the “Legacy log” option from the dropdown and select “Get these logs". Perform the same action as you did earlier for the standard log which includes downloading all the formats and filtering.
        3. Testing the live log:
          • Navigate to Site administration > Reports > Live logs and select "Legacy log" from the dropdrop. New events will be added to that page dynamically every 60 seconds. You can open another browser tab or window and perform some actions such as viewing log tables and vising courses. Verify that those actions on the Live Logs page appear once it updates.
          • Re-enable the "Standard log" and return to the Live logs page and similarly as above verify that log appears once it updates.
      Show
      Run all and individual unit tests: ./vendor/bin/phpunit admin/tool/log/store/standard/tests/store_test.php ./vendor/bin/phpunit admin/tool/log/store/legacy/tests/store_test.php ./vendor/bin/phpunit Behat tests should pass, ask integrators about failures in log reports related tests Next, test legacy and the new standard logs. Standard logs: Navigate to Site administration > Reports > Logs and select “Standard log” from the dropdown and click “Get these logs”. If that option is not available then you are already viewing the standard log by default. If you do not see much data listed then you can generate it by performing different actions such as visiting a course page, logging in with invalid credentials, etc. Create another test user and also perform those action in order to get log data generated. That's so we can filter the logs by that user later. For each of the formats available under the “Download table data as” label, select the format and click “Download”. Open the downloaded file and verify that the data in the file, including the headers, match the data on the page. Filter the log table by some criteria such as course or participants and click “Get these logs”. For each of the formats available under the “Download table data as” label select the format and click “Download”. Open the downloaded file and verify that the data in the file, including the headers, match the data on the page. Legacy logging: Since M27 log data is no longer stored in the table mdl_log by default but we will need some data generated in order to verify the downloaded data. You can enable legacy logging from Site administration > Plugins > Logging > Manage log stores. Select the option "Log legacy data" and save changes. From the Logs page select the “Legacy log” option from the dropdown and select “Get these logs". Perform the same action as you did earlier for the standard log which includes downloading all the formats and filtering. Testing the live log: Navigate to Site administration > Reports > Live logs and select "Legacy log" from the dropdrop. New events will be added to that page dynamically every 60 seconds. You can open another browser tab or window and perform some actions such as viewing log tables and vising courses. Verify that those actions on the Live Logs page appear once it updates. Re-enable the "Standard log" and return to the Live logs page and similarly as above verify that log appears once it updates.
    • Affected Branches:
      MOODLE_21_STABLE, MOODLE_22_STABLE, MOODLE_23_STABLE, MOODLE_26_STABLE, MOODLE_27_STABLE
    • Fixed Branches:
      MOODLE_29_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      MDL-34867-master
    • Sprint:
      BACKEND Sprint 18

      Description

      Exporting to various formats from the course log report is excessively resource intensive, often failing due to various limits:

      • The ODS exporter consumes all memory until hitting the limit
      • The Excel exporter in particular uses all CPU until time limit reached
      • All exporters utilise get_logs which will hit memory limit for many records
      • Main loop for all exporters makes 1 query for each log entry - slow and unnecessary DB load
      • Queries are made with strings for an id number field

      With some profile guidance I have reworked the critical paths somewhat to improve this situation.

      The main bottleneck remaining the Excel and ODS exporters which probably need a more through revisit to really solve this.

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                9 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  11/May/15