XMLWordPrintable

Details

    • Sub-task
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 3.1
    • 3.1
    • Global search
    • MOODLE_31_STABLE
    • MOODLE_31_STABLE
    • MDL-53516-master
    • Hide

      Background items

      Indexing

      When the instructions call for indexing, that means to run this:

      php admin/tool/task/cli/schedule_task.php --execute="\\core\\task\\search_index_task"
      

      Performing Solr queries

      To test properly, you will need to perform some queries against the Solr core directly.
      To do that (assume you are running Solr locally), you will access localhost:8983 in your browser.

      Once there, on the left go to core selector, then select the core you are using for the test. The click query. You will enter the provided query in the q field.

      Test files

      Attached is a zip with some test files in it. Download and unzip it.
      Files File1-5 have the content of "Some FileContentX" in them (with X replaced with the file number). This gives us a unique token to search for.

      Setup

      For simplicity, you will want to do this on a clean Moodle install, and a new Solr core.

      1. Configure Solr search.
        • Make sure file indexing is enabled, and set the Max file size to 1.
      2. Install the schema
      3. Enable the Folder, Resource, Forum Post, and Assign search areas.
      4. Enable global search
      5. Create a course we will work in.
      6. In search/classes/manager.php, comment out the cache return return $results; on line ~413. This will prevent cached search returns.

      Main testing

      Basic folder test
      1. Create a folder resource, named Folder 1
      2. Give it a description of "Folder 1 Description"
      3. Attach the 5 FileX files
      4. Save
      5. Run the indexing command
      6. Search for "FileContent4"
      7. Confirm you get a result for Folder 1, and it lists a matched file of File4
      8. Search for "FileContent2"
      9. Confirm you get a result for Folder 1, and it lists a matched file of File2
      10. Search for "File2"
      11. Confirm you get a result for Folder 1, and it lists a matched file of File2
      12. Search for "Some"
      13. Confirm you get a result for Folder 1, and it lists 3 matched files
      14. Run the Solr query areaid:mod_folder-activity
      15. Confirm there are 6 records found (reported as numfound)
      16. Inspect the results, confirm you see 1 record that is type: 1 with the contents for the folder description and title
      17. Confirm there are 5 type: 2 records, one for each file. They should have a solr_fileindexedcontent field value of 1
      18. Note the version value of File1 and File2
      Re-indexing
      1. Edit the folder resource from above.
      2. Remove File4
      3. Click on File2, and rename it File2Edit
      4. Save
      5. Run the indexing command
      6. Search for "FileContents4"
      7. Confirm you do not get a result for Folder 1
      8. Search for "File2Edit"
      9. Confirm you get a result for Folder 1, and it lists a matched file of File2Edit
      10. Search for "FileContent2"
      11. Confirm you get a result for Folder 1, and it lists a matched file of File2Edit
      12. Run the Solr query areaid:mod_folder-activity
      13. Confirm there are 5 records found
      14. Confirm there is no record for File4
      15. Confirm that the version field for File1 has not changed
      16. Confirm that the version field for File2 has changed, and the title is now File2Edit
      Delete folder

      Note: We can't do this at this time because of MDL-53272.

      1. Delete the above activity
      2. Search for "FileContent3"
      3. Confirm no results are returned
      4. Run the Solr query areaid:mod_folder-activity
      5. Confirm there are 0 records found
      File resource
      1. Create a File resource, title "File Resource 1"
      2. Attach File1 and File2
      3. Click on File2 and click Set main file
      4. Save
      5. Run indexing command
      6. Search for "FileContent2"
      7. Confirm you get a result for File Resource 1, and it lists a matched file of File2
      8. Search for "FileContent1"
      9. Confirm you do not get a result for File Resource 1
      10. Run the Solr query areaid:mod_resource-activity
      11. Confirm there are 2 records found - one type: 1 for the resource itself, and one type: 2 for File2
      Assign Attachment
      1. Create an assignment, titled "Assign 1"
      2. Attach File4 and File5
      3. Save
      4. Run indexing command
      5. Search for "FileContent4"
      6. Confirm you get a result for Assign 1, and it lists a matched file of File4
      7. Search for "FileContent5"
      8. Confirm you get a result for Assign 1, and it lists a matched file of File5
      9. Search for "Assign"
      10. Confirm you get a result for Assign 1, with no matching files
      11. Run the Solr query areaid:mod_assign-activity
      12. Confirm there are 3 records found - one type: 1 for the assign itself, and 2 type: 2 for File4 and File5
      Forum Post
      1. Create an Forum, titled "Forum 1"
      2. Create a discussion post with the subject "DiscussionSubject" body "DiscussionPost"
      3. Attach File3 and File4
      4. Save
      5. Run indexing command
      6. Search for "FileContent3"
      7. Confirm you get a result for DiscussionSubject, and it lists a matched file of File3
      8. Search for "FileContent4"
      9. Confirm you get a result for DiscussionSubject, and it lists a matched file of File4
      10. Search for "DiscussionPost"
      11. Confirm you get a result for DiscussionSubject, with no matching files
      12. Run the Solr query areaid:mod_forum-post
      13. Confirm there are 3 records found - one type: 1 for the post itself, and 2 type: 2 for File3 and File4
      Oversized file
      1. Create a file resource with the title File Resource 2
      2. Attach the file Oversized
      3. Save
      4. Run indexing command
      5. Search for "OversizedContents"
      6. Confirm you no not get a result for File Resource 2
      7. Search for "Oversized"
      8. Confirm you get a result for File Resource 2, and it lists a matched file of Oversized
      9. Run the Solr query areaid:mod_resource-activity
      10. Find the record for the file Oversized.
      11. Confirm solr_fileindexedcontent field value of 0
      12. Change the max Solr filesize to 100
      13. Edit File Resource 2 and a content field
      14. Save
      15. Run the index command
      16. Search for "OversizedContents"
      17. Confirm you get a result for File Resource 2, and it lists a matched file of Oversized
      18. Search for "Oversized"
      19. Confirm you get a result for File Resource 2, and it lists a matched file of Oversized
      20. Run the Solr query areaid:mod_resource-activity
      21. Find the record for the file Oversized.
      22. Confirm solr_fileindexedcontent field value of 1

      Freeform testing

      1. Set the Max filesize to 0 (unlimited)
      2. Find some PDFs, Word docs, Power Points, etc
      3. Add them as file resources to a course
      4. Run the indexing command
        • Note: You may get indexing failures due to various problems in Tika (malformed file, heap memory, etc). If that happens, the contents of that file are not indexed, but the filename still will be. If you find the record in Solr, the solr_fileindexedcontent field should have a value of -1. This is not Moodle's fault, but we expect Moodle to gracefully carry on indexing.
      5. Search for contents of those files and see if you get expected results

      Unit tests

      1. Configure unit tests with lines similar to these in config.php

        define('TEST_SEARCH_SOLR_HOSTNAME', '127.0.0.1');
        define('TEST_SEARCH_SOLR_PORT', '8983');
        define('TEST_SEARCH_SOLR_INDEXNAME', 'unittest');
        

        The INDEXNAME is a Solr core you have created for unittesting. Make sure to use a fresh one, and schema changes can cause a failure

      Show
      Background items Indexing When the instructions call for indexing, that means to run this: php admin/tool/task/cli/schedule_task.php --execute="\\core\\task\\search_index_task" Performing Solr queries To test properly, you will need to perform some queries against the Solr core directly. To do that (assume you are running Solr locally), you will access localhost:8983 in your browser. Once there, on the left go to core selector, then select the core you are using for the test. The click query. You will enter the provided query in the q field. Test files Attached is a zip with some test files in it. Download and unzip it. Files File1-5 have the content of "Some FileContentX" in them (with X replaced with the file number). This gives us a unique token to search for. Setup For simplicity, you will want to do this on a clean Moodle install, and a new Solr core. Configure Solr search. Make sure file indexing is enabled, and set the Max file size to 1. Install the schema Enable the Folder, Resource, Forum Post, and Assign search areas. Enable global search Create a course we will work in. In search/classes/manager.php, comment out the cache return return $results; on line ~413. This will prevent cached search returns. Main testing Basic folder test Create a folder resource, named Folder 1 Give it a description of "Folder 1 Description" Attach the 5 FileX files Save Run the indexing command Search for "FileContent4" Confirm you get a result for Folder 1, and it lists a matched file of File4 Search for "FileContent2" Confirm you get a result for Folder 1, and it lists a matched file of File2 Search for "File2" Confirm you get a result for Folder 1, and it lists a matched file of File2 Search for "Some" Confirm you get a result for Folder 1, and it lists 3 matched files Run the Solr query areaid:mod_folder-activity Confirm there are 6 records found (reported as numfound) Inspect the results, confirm you see 1 record that is type: 1 with the contents for the folder description and title Confirm there are 5 type: 2 records, one for each file. They should have a solr_fileindexedcontent field value of 1 Note the version value of File1 and File2 Re-indexing Edit the folder resource from above. Remove File4 Click on File2, and rename it File2Edit Save Run the indexing command Search for "FileContents4" Confirm you do not get a result for Folder 1 Search for "File2Edit" Confirm you get a result for Folder 1, and it lists a matched file of File2Edit Search for "FileContent2" Confirm you get a result for Folder 1, and it lists a matched file of File2Edit Run the Solr query areaid:mod_folder-activity Confirm there are 5 records found Confirm there is no record for File4 Confirm that the version field for File1 has not changed Confirm that the version field for File2 has changed, and the title is now File2Edit Delete folder Note: We can't do this at this time because of MDL-53272 . Delete the above activity Search for "FileContent3" Confirm no results are returned Run the Solr query areaid:mod_folder-activity Confirm there are 0 records found File resource Create a File resource, title "File Resource 1" Attach File1 and File2 Click on File2 and click Set main file Save Run indexing command Search for "FileContent2" Confirm you get a result for File Resource 1, and it lists a matched file of File2 Search for "FileContent1" Confirm you do not get a result for File Resource 1 Run the Solr query areaid:mod_resource-activity Confirm there are 2 records found - one type: 1 for the resource itself, and one type: 2 for File2 Assign Attachment Create an assignment, titled "Assign 1" Attach File4 and File5 Save Run indexing command Search for "FileContent4" Confirm you get a result for Assign 1, and it lists a matched file of File4 Search for "FileContent5" Confirm you get a result for Assign 1, and it lists a matched file of File5 Search for "Assign" Confirm you get a result for Assign 1, with no matching files Run the Solr query areaid:mod_assign-activity Confirm there are 3 records found - one type: 1 for the assign itself, and 2 type: 2 for File4 and File5 Forum Post Create an Forum, titled "Forum 1" Create a discussion post with the subject "DiscussionSubject" body "DiscussionPost" Attach File3 and File4 Save Run indexing command Search for "FileContent3" Confirm you get a result for DiscussionSubject, and it lists a matched file of File3 Search for "FileContent4" Confirm you get a result for DiscussionSubject, and it lists a matched file of File4 Search for "DiscussionPost" Confirm you get a result for DiscussionSubject, with no matching files Run the Solr query areaid:mod_forum-post Confirm there are 3 records found - one type: 1 for the post itself, and 2 type: 2 for File3 and File4 Oversized file Create a file resource with the title File Resource 2 Attach the file Oversized Save Run indexing command Search for "OversizedContents" Confirm you no not get a result for File Resource 2 Search for "Oversized" Confirm you get a result for File Resource 2, and it lists a matched file of Oversized Run the Solr query areaid:mod_resource-activity Find the record for the file Oversized. Confirm solr_fileindexedcontent field value of 0 Change the max Solr filesize to 100 Edit File Resource 2 and a content field Save Run the index command Search for "OversizedContents" Confirm you get a result for File Resource 2, and it lists a matched file of Oversized Search for "Oversized" Confirm you get a result for File Resource 2, and it lists a matched file of Oversized Run the Solr query areaid:mod_resource-activity Find the record for the file Oversized. Confirm solr_fileindexedcontent field value of 1 Freeform testing Set the Max filesize to 0 (unlimited) Find some PDFs, Word docs, Power Points, etc Add them as file resources to a course Run the indexing command Note: You may get indexing failures due to various problems in Tika (malformed file, heap memory, etc). If that happens, the contents of that file are not indexed, but the filename still will be. If you find the record in Solr, the solr_fileindexedcontent field should have a value of -1. This is not Moodle's fault, but we expect Moodle to gracefully carry on indexing. Search for contents of those files and see if you get expected results Unit tests Configure unit tests with lines similar to these in config.php define('TEST_SEARCH_SOLR_HOSTNAME', '127.0.0.1'); define('TEST_SEARCH_SOLR_PORT', '8983'); define('TEST_SEARCH_SOLR_INDEXNAME', 'unittest'); The INDEXNAME is a Solr core you have created for unittesting. Make sure to use a fresh one, and schema changes can cause a failure

    Description

      Implement the file indexing API changes in Solr search.

      Attachments

        Issue Links

          Activity

            People

              emerrill Eric Merrill
              emerrill Eric Merrill
              David Monllaó David Monllaó
              Dan Poltawski Dan Poltawski
              Damyon Wiese Damyon Wiese
              David Woloszyn, Huong Nguyen, Jake Dallimore, Meirza, Michael Hawkins, Raquel Ortega, Safat Shahin, Stevani Andolo
              Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:
                23/May/16