Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-61102

core_search: Improve performance by reducing schema checks

XMLWordPrintable

    • MOODLE_35_STABLE
    • MOODLE_35_STABLE
    • MDL-61102-master
    • Hide

      In order to run this test you need:

      • A Moodle setup with Moodle global search configured, using Solr.
      • Access to your Solr admin interface (in order to test this feature, we are going to on-purpose break the Moodle search schema).
      • A method to run SQL queries/updates on your Moodle database.
      1. Go to the search areas page (Site administration / Plugins / Search / Search areas).
        • EXPECTED (no change): There is no error, search is available.
      2. Do a search using the search box in the header.
        • EXPECTED (no change): There is no error, search works.
      3. To confirm that the search schema is not checked on every visit to the search page, we will now break the search schema in the Solr admin interface. These instructions are based on my Solr version 6.x and may be slightly different on other versions.
        • In the core selector dropdown, select the relevant core used for your Moodle index.
        • Select the Schema tab.
        • In the field select dropdown, select userid.
        • Press delete field button and confirm the Are you sure prompt.
      4. Back in Moodle, visit the search page /search/index.php without doing an actual search.
        • EXPECTED: The page appears without indication of any error.
      5. (Note: You can try an actual search at this point if you like - it probably works but there will be incorrect behaviour related to user id in the results.)
      6. Go back to the search areas page.
        • EXPECTED (no change): This page shows an error The search engine is not available. Please contact your administrator.
      7. Now go back to /search/index.php.
        • EXPECTED: This page now shows the same error too.
      8. Go to the manage global search page (Site administration / Plugins / Search / Manage global search).
        • EXPECTED (no change): The search engine shows Error validating Solr schema in step 3.
      9. Click the link in step 3 to reset the schema. (Note: In order to actually correct your search data, you would also need to reindex everything, but we'll skip that for this test.)
      10. Run this SQL query (change mdl_ to your prefix): SELECT * FROM mdl_config_plugins WHERE plugin='search_solr' AND name='lastschemacheck';
        • EXPECTED: No results.
      11. Visit /search/index.php
        • EXPECTED: The page now loads without error. (It should have done a schema check.)
      12. Run the SQL query again: SELECT * FROM mdl_config_plugins WHERE plugin='search_solr' AND name='lastschemacheck';
        • EXPECTED: The query should have a result, and the value field should correspond to the current time (in Unix time, seconds since 1970). Keep a note of the value.
      13. Reload the search page and also go to the search areas page, then repeat the SQL query.
        • EXPECTED: There should be no change in the query results (it doesn't update the config field every time it does an index, only once per ten minutes).
      14. Subtract 3600 (one hour) from the value, and update the field to have that value using the SQL update: UPDATE mdl_config_plugins SET value = MYNEWVALUE WHERE plugin='search_solr' AND name='lastschemacheck';
      15. Visit the purge caches page /admin/purgecaches.php and press the button to purge caches.
      16. Load the search areas page again (so that it checks the schema).
      17. Rerun the SQL query: SELECT * FROM mdl_config_plugins WHERE plugin='search_solr' AND name='lastschemacheck';
        • EXPECTED: The value field should have been updated to the current time again.
      18. Subtract more than 4 hours (e.g. 20000) from the value, and update the field using the SQL update: UPDATE mdl_config_plugins SET value = MYNEWVALUE WHERE plugin='search_solr' AND name='lastschemacheck';
      19. Visit the purge caches page /admin/purgecaches.php and press the button to purge caches.
      20. Go to the search page /search/index.php.
      21. Rerun the SQL query: SELECT * FROM mdl_config_plugins WHERE plugin='search_solr' AND name='lastschemacheck';
        • EXPECTED: The value field should have been updated to the current time again, indicating that this time it checked the schema even on the search page.
      Show
      In order to run this test you need: A Moodle setup with Moodle global search configured, using Solr. Access to your Solr admin interface (in order to test this feature, we are going to on-purpose break the Moodle search schema). A method to run SQL queries/updates on your Moodle database. Go to the search areas page ( Site administration / Plugins / Search / Search areas ). EXPECTED (no change): There is no error, search is available. Do a search using the search box in the header. EXPECTED (no change): There is no error, search works. To confirm that the search schema is not checked on every visit to the search page, we will now break the search schema in the Solr admin interface. These instructions are based on my Solr version 6.x and may be slightly different on other versions. In the core selector dropdown, select the relevant core used for your Moodle index. Select the Schema tab. In the field select dropdown, select userid . Press delete field button and confirm the Are you sure prompt. Back in Moodle, visit the search page /search/index.php without doing an actual search. EXPECTED: The page appears without indication of any error. (Note: You can try an actual search at this point if you like - it probably works but there will be incorrect behaviour related to user id in the results.) Go back to the search areas page. EXPECTED (no change): This page shows an error The search engine is not available. Please contact your administrator. Now go back to /search/index.php . EXPECTED: This page now shows the same error too. Go to the manage global search page ( Site administration / Plugins / Search / Manage global search ). EXPECTED (no change): The search engine shows Error validating Solr schema in step 3. Click the link in step 3 to reset the schema. (Note: In order to actually correct your search data, you would also need to reindex everything, but we'll skip that for this test.) Run this SQL query (change mdl_ to your prefix): SELECT * FROM mdl_config_plugins WHERE plugin='search_solr' AND name='lastschemacheck'; EXPECTED: No results. Visit /search/index.php EXPECTED: The page now loads without error. (It should have done a schema check.) Run the SQL query again: SELECT * FROM mdl_config_plugins WHERE plugin='search_solr' AND name='lastschemacheck'; EXPECTED: The query should have a result, and the value field should correspond to the current time (in Unix time, seconds since 1970). Keep a note of the value. Reload the search page and also go to the search areas page, then repeat the SQL query. EXPECTED: There should be no change in the query results (it doesn't update the config field every time it does an index, only once per ten minutes). Subtract 3600 (one hour) from the value, and update the field to have that value using the SQL update: UPDATE mdl_config_plugins SET value = MYNEWVALUE WHERE plugin='search_solr' AND name='lastschemacheck'; Visit the purge caches page /admin/purgecaches.php and press the button to purge caches. Load the search areas page again (so that it checks the schema). Rerun the SQL query: SELECT * FROM mdl_config_plugins WHERE plugin='search_solr' AND name='lastschemacheck'; EXPECTED: The value field should have been updated to the current time again. Subtract more than 4 hours (e.g. 20000) from the value, and update the field using the SQL update: UPDATE mdl_config_plugins SET value = MYNEWVALUE WHERE plugin='search_solr' AND name='lastschemacheck'; Visit the purge caches page /admin/purgecaches.php and press the button to purge caches. Go to the search page /search/index.php. Rerun the SQL query: SELECT * FROM mdl_config_plugins WHERE plugin='search_solr' AND name='lastschemacheck'; EXPECTED: The value field should have been updated to the current time again, indicating that this time it checked the schema even on the search page.

      Every time the search manager is instantiated (for example, when displaying a search query page), it checks the schema of the search engine. When using external hosting (rather than the search engine running on the same machine) this can take a significant amount of time. In our system (Moodle hosted on-site in Milton Keynes, search engine in AWS London, which is only about 50 miles geographically) it takes about 250ms to initialise the search engine in total if it checks the schema, compared to about 20ms if not.

      Since our pages generally load in about a second, this is a significant increase in the loading time for search pages and might cause perceived slowness for students. We want the 'new' search to feel nice and fast.

      It would be nice to reduce the stringency of this schema checking. My proposal is:

      • On the admin-type pages and in cron, continue checking schema on every request.
      • On the search page, check schema only if it has not been successfully checked within a certain time (I suggest 24 hours).

      The idea is that if somebody changes search engine settings they are pretty much bound to go to the search areas page or the main global search page. Also if there is an error with search and indexing stops working (for instance) then either somebody should notice it in the cron logs, or they'll go to the search areas page to see what it's doing and then get the error there. And if somebody has just turned search on or whatever, and they try to go search, then since it doesn't have a successful check yet, it'll still check that time as well.

        1. mdl-61102-localhost.xlsx
          11 kB

            quen Sam Marshall
            quen Sam Marshall
            Tim Hunt Tim Hunt
            David Monllaó David Monllaó
            David Monllaó David Monllaó
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated:
              Resolved:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.