Implement file indexing in Solr search

Background items Indexing When the instructions call for indexing, that means to run this: php admin/tool/task/cli/schedule_task.php --execute="\\core\\task\\search_index_task" Performing Solr queries To test properly, you will need to perform some queries against the Solr core directly. To do that (assume you are running Solr locally), you will access localhost:8983 in your browser. Once there, on the left go to core selector, then select the core you are using for the test. The click query. You will enter the provided query in the q field. Test files Attached is a zip with some test files in it. Download and unzip it. Files File1-5 have the content of "Some FileContentX" in them (with X replaced with the file number). This gives us a unique token to search for. Setup For simplicity, you will want to do this on a clean Moodle install, and a new Solr core. Configure Solr search. Make sure file indexing is enabled, and set the Max file size to 1. Install the schema Enable the Folder, Resource, Forum Post, and Assign search areas. Enable global search Create a course we will work in. In search/classes/manager.php, comment out the cache return return $results; on line ~413. This will prevent cached search returns. Main testing Basic folder test Create a folder resource, named Folder 1 Give it a description of "Folder 1 Description" Attach the 5 FileX files Save Run the indexing command Search for "FileContent4" Confirm you get a result for Folder 1, and it lists a matched file of File4 Search for "FileContent2" Confirm you get a result for Folder 1, and it lists a matched file of File2 Search for "File2" Confirm you get a result for Folder 1, and it lists a matched file of File2 Search for "Some" Confirm you get a result for Folder 1, and it lists 3 matched files Run the Solr query areaid:mod_folder-activity Confirm there are 6 records found (reported as numfound) Inspect the results, confirm you see 1 record that is type: 1 with the contents for the folder description and title Confirm there are 5 type: 2 records, one for each file. They should have a solr_fileindexedcontent field value of 1 Note the version value of File1 and File2 Re-indexing Edit the folder resource from above. Remove File4 Click on File2, and rename it File2Edit Save Run the indexing command Search for "FileContents4" Confirm you do not get a result for Folder 1 Search for "File2Edit" Confirm you get a result for Folder 1, and it lists a matched file of File2Edit Search for "FileContent2" Confirm you get a result for Folder 1, and it lists a matched file of File2Edit Run the Solr query areaid:mod_folder-activity Confirm there are 5 records found Confirm there is no record for File4 Confirm that the version field for File1 has not changed Confirm that the version field for File2 has changed, and the title is now File2Edit Delete folder Note: We can't do this at this time because of MDL-53272 . Delete the above activity Search for "FileContent3" Confirm no results are returned Run the Solr query areaid:mod_folder-activity Confirm there are 0 records found File resource Create a File resource, title "File Resource 1" Attach File1 and File2 Click on File2 and click Set main file Save Run indexing command Search for "FileContent2" Confirm you get a result for File Resource 1, and it lists a matched file of File2 Search for "FileContent1" Confirm you do not get a result for File Resource 1 Run the Solr query areaid:mod_resource-activity Confirm there are 2 records found - one type: 1 for the resource itself, and one type: 2 for File2 Assign Attachment Create an assignment, titled "Assign 1" Attach File4 and File5 Save Run indexing command Search for "FileContent4" Confirm you get a result for Assign 1, and it lists a matched file of File4 Search for "FileContent5" Confirm you get a result for Assign 1, and it lists a matched file of File5 Search for "Assign" Confirm you get a result for Assign 1, with no matching files Run the Solr query areaid:mod_assign-activity Confirm there are 3 records found - one type: 1 for the assign itself, and 2 type: 2 for File4 and File5 Forum Post Create an Forum, titled "Forum 1" Create a discussion post with the subject "DiscussionSubject" body "DiscussionPost" Attach File3 and File4 Save Run indexing command Search for "FileContent3" Confirm you get a result for DiscussionSubject, and it lists a matched file of File3 Search for "FileContent4" Confirm you get a result for DiscussionSubject, and it lists a matched file of File4 Search for "DiscussionPost" Confirm you get a result for DiscussionSubject, with no matching files Run the Solr query areaid:mod_forum-post Confirm there are 3 records found - one type: 1 for the post itself, and 2 type: 2 for File3 and File4 Oversized file Create a file resource with the title File Resource 2 Attach the file Oversized Save Run indexing command Search for "OversizedContents" Confirm you no not get a result for File Resource 2 Search for "Oversized" Confirm you get a result for File Resource 2, and it lists a matched file of Oversized Run the Solr query areaid:mod_resource-activity Find the record for the file Oversized. Confirm solr_fileindexedcontent field value of 0 Change the max Solr filesize to 100 Edit File Resource 2 and a content field Save Run the index command Search for "OversizedContents" Confirm you get a result for File Resource 2, and it lists a matched file of Oversized Search for "Oversized" Confirm you get a result for File Resource 2, and it lists a matched file of Oversized Run the Solr query areaid:mod_resource-activity Find the record for the file Oversized. Confirm solr_fileindexedcontent field value of 1 Freeform testing Set the Max filesize to 0 (unlimited) Find some PDFs, Word docs, Power Points, etc Add them as file resources to a course Run the indexing command Note: You may get indexing failures due to various problems in Tika (malformed file, heap memory, etc). If that happens, the contents of that file are not indexed, but the filename still will be. If you find the record in Solr, the solr_fileindexedcontent field should have a value of -1. This is not Moodle's fault, but we expect Moodle to gracefully carry on indexing. Search for contents of those files and see if you get expected results Unit tests Configure unit tests with lines similar to these in config.php define('TEST_SEARCH_SOLR_HOSTNAME', '127.0.0.1'); define('TEST_SEARCH_SOLR_PORT', '8983'); define('TEST_SEARCH_SOLR_INDEXNAME', 'unittest'); The INDEXNAME is a Solr core you have created for unittesting. Make sure to use a fresh one, and schema changes can cause a failure