Details

    • Story Points (Obsolete):
      20

      Description

      Global Search is currently removed from Moodle 2.2.

      As part of GSoC '13 project, I've re-written Tomasz Muras's Global Search implementation for the current 2.6 (dev) master branch.

      You may view the demo and play with it here: http://global-search.jmuras.com/
      ( Username:search_admin Password: Search_admin1 )

      The docs are here: http://docs.moodle.org/dev/Global_search

        Gliffy Diagrams

          Issue Links

            Issues in Epic

              Activity

              Hide
              nebgor Aparup Banerjee added a comment -

              Thanks for working on this Tomasz

              It might be useful to put up a git repository link here.
              (also because i'm wondering how its going )

              Show
              nebgor Aparup Banerjee added a comment - Thanks for working on this Tomasz It might be useful to put up a git repository link here. (also because i'm wondering how its going )
              Hide
              tmuras Tomasz Muras added a comment -

              I've added link to my git repo to the description.
              As I've promised in the forums, I am starting with the performance testing. I'm going to test:

              • performance of the indexing
              • performance of the search (split into Lucene performance and perfomance of security checking - that is, filtering down the search results depending on the permissions)
              • memory footprint of the above

              Although it's far from completion, I have now enough code to do that. I will post my finindings in the dev forum.

              Show
              tmuras Tomasz Muras added a comment - I've added link to my git repo to the description. As I've promised in the forums, I am starting with the performance testing. I'm going to test: performance of the indexing performance of the search (split into Lucene performance and perfomance of security checking - that is, filtering down the search results depending on the permissions) memory footprint of the above Although it's far from completion, I have now enough code to do that. I will post my finindings in the dev forum.
              Hide
              nebgor Aparup Banerjee added a comment -

              this isn't minor! --> major!

              Show
              nebgor Aparup Banerjee added a comment - this isn't minor! --> major!
              Hide
              tmuras Tomasz Muras added a comment -

              Checked indexing performance - moving on the the performance of the search. See http://moodle.org/mod/forum/discuss.php?d=199636 for discussion.

              Show
              tmuras Tomasz Muras added a comment - Checked indexing performance - moving on the the performance of the search. See http://moodle.org/mod/forum/discuss.php?d=199636 for discussion.
              Hide
              tmuras Tomasz Muras added a comment -

              Update on http://moodle.org/mod/forum/discuss.php?d=199636 on index optimization.

              Show
              tmuras Tomasz Muras added a comment - Update on http://moodle.org/mod/forum/discuss.php?d=199636 on index optimization.
              Hide
              nadavkav Nadav Kavalerchik added a comment -

              Hi Tomasz
              Is this ready for some testing? what is the timeline and when it is scheduled for upstream integration?

              Show
              nadavkav Nadav Kavalerchik added a comment - Hi Tomasz Is this ready for some testing? what is the timeline and when it is scheduled for upstream integration?
              Hide
              tmuras Tomasz Muras added a comment -

              Hi Nadav,

              Global Search 2 rewrite will not be ready before Moodle 2.4. As soon as I get some time, I'll continue my work on testing the search performance. I will be updating forum discussion with my findings.

              Show
              tmuras Tomasz Muras added a comment - Hi Nadav, Global Search 2 rewrite will not be ready before Moodle 2.4. As soon as I get some time, I'll continue my work on testing the search performance. I will be updating forum discussion with my findings.
              Hide
              nadavkav Nadav Kavalerchik added a comment -

              Hi Tomasz,
              Thank you for the clarification

              btw, I have already git-clone it from you and downloaded the latest Zend/Search library files
              I got the search_reset_index() to work, I guess. but not the search.php
              If you have any tips or suggestions... that would be nice

              Show
              nadavkav Nadav Kavalerchik added a comment - Hi Tomasz, Thank you for the clarification btw, I have already git-clone it from you and downloaded the latest Zend/Search library files I got the search_reset_index() to work, I guess. but not the search.php If you have any tips or suggestions... that would be nice
              Hide
              tmuras Tomasz Muras added a comment -

              The development is not finished there yet. While search should work already, the biggest work now (implementation-wise) is to do access control.
              Would you be interested in helping with the development/coding?

              Show
              tmuras Tomasz Muras added a comment - The development is not finished there yet. While search should work already, the biggest work now (implementation-wise) is to do access control. Would you be interested in helping with the development/coding?
              Hide
              nadavkav Nadav Kavalerchik added a comment -

              Like, you. I am very much busy with allot of work, which i already committed to.
              so I can not promises to help the development. but please feel free to ask anything, specific, when it arises.
              I can definitely help testing. ( I have set "watching" on the git repository )

              Show
              nadavkav Nadav Kavalerchik added a comment - Like, you. I am very much busy with allot of work, which i already committed to. so I can not promises to help the development. but please feel free to ask anything, specific, when it arises. I can definitely help testing. ( I have set "watching" on the git repository )
              Hide
              dougiamas Martin Dougiamas added a comment -

              I'm keeping an eye on this. Let us know where/how we can help.

              Show
              dougiamas Martin Dougiamas added a comment - I'm keeping an eye on this. Let us know where/how we can help.
              Hide
              tmuras Tomasz Muras added a comment -

              Just to update you: I'm very slow at the moment because of the lack of free time.
              My plan is to do some slow progress between now and end of September and then do most of the work until the end of October. That would leave November for the reviews and integration - so it can be included in Moodle 2.4 in December. I'd sill mark it as experimental feature and keep working on it in later releases.

              Show
              tmuras Tomasz Muras added a comment - Just to update you: I'm very slow at the moment because of the lack of free time. My plan is to do some slow progress between now and end of September and then do most of the work until the end of October. That would leave November for the reviews and integration - so it can be included in Moodle 2.4 in December. I'd sill mark it as experimental feature and keep working on it in later releases.
              Hide
              dougiamas Martin Dougiamas added a comment -

              Thanks, Tomasz!

              Code freeze for 2.4 is actually mid-October, and it really should already have had some HQ review by then too. it would be great if it gets that attention before then but otherwise it would have to wait for 2.5.

              Show
              dougiamas Martin Dougiamas added a comment - Thanks, Tomasz! Code freeze for 2.4 is actually mid-October, and it really should already have had some HQ review by then too. it would be great if it gets that attention before then but otherwise it would have to wait for 2.5.
              Hide
              tmuras Tomasz Muras added a comment -

              I will keep the ball rolling: http://moodle.org/mod/forum/discuss.php?d=208958 .
              I think it would be better to have even simper version available for 2.4 than to wait again until 2.5. I should get it done mid-October and I'll make sure that the implementation is not a surprise for any core dev.

              Show
              tmuras Tomasz Muras added a comment - I will keep the ball rolling: http://moodle.org/mod/forum/discuss.php?d=208958 . I think it would be better to have even simper version available for 2.4 than to wait again until 2.5. I should get it done mid-October and I'll make sure that the implementation is not a surprise for any core dev.
              Hide
              tmuras Tomasz Muras added a comment -

              Hi All,

              It's been another weekend when I planned to do some work on Global Search but couldn't. I hate to say it but the truth is that I don't have enough time to work on it and will not be able to deliver!
              At the very least I will dump all I've learnt so far onto the wiki page, to make it easier for another developer to start where I've stopped.

              Sorry about that, I've overestimated the resources I had at my disposal . If nobody picks it up, then it's very likely I will complete the coding at some point but absolutely don't count on it.

              cheers,
              Tomek

              Show
              tmuras Tomasz Muras added a comment - Hi All, It's been another weekend when I planned to do some work on Global Search but couldn't. I hate to say it but the truth is that I don't have enough time to work on it and will not be able to deliver! At the very least I will dump all I've learnt so far onto the wiki page, to make it easier for another developer to start where I've stopped. Sorry about that, I've overestimated the resources I had at my disposal . If nobody picks it up, then it's very likely I will complete the coding at some point but absolutely don't count on it. cheers, Tomek
              Hide
              libertymoodle Luis de Vasconcelos added a comment -

              Will this Global Search function be part of 2.4?

              Show
              libertymoodle Luis de Vasconcelos added a comment - Will this Global Search function be part of 2.4?
              Hide
              tmuras Tomasz Muras added a comment -

              No, it will not.

              Show
              tmuras Tomasz Muras added a comment - No, it will not.
              Hide
              libertymoodle Luis de Vasconcelos added a comment -

              wrensoft.com has a search engine that looks interesting. We plan to test it and see if it is a viable solution.

              How to create a search engine for your Moodle site
              http://www.wrensoft.com/zoom/support/tutorial_moodle.html

              How to index sites requiring authentication with Zoom
              http://www.wrensoft.com/zoom/support/auth.html

              Show
              libertymoodle Luis de Vasconcelos added a comment - wrensoft.com has a search engine that looks interesting. We plan to test it and see if it is a viable solution. How to create a search engine for your Moodle site http://www.wrensoft.com/zoom/support/tutorial_moodle.html How to index sites requiring authentication with Zoom http://www.wrensoft.com/zoom/support/auth.html
              Hide
              tsala Helen Foster added a comment -

              Hi Tomasz,

              I hope you don't mind me adding an infobox to http://docs.moodle.org/dev/Global_Search so people can easily see details of the project. I also removed the 2.3 template. Please amend as necessary.

              Show
              tsala Helen Foster added a comment - Hi Tomasz, I hope you don't mind me adding an infobox to http://docs.moodle.org/dev/Global_Search so people can easily see details of the project. I also removed the 2.3 template. Please amend as necessary.
              Hide
              aspark Alistair Spark added a comment -

              We have just finalised the implementation of the wrensoft search tool which works very well for us as a small institution with a limited number of long courses.

              The Version 7 alpha release is required - http://www.wrensoft.com/zoom/v7.html

              We have created student accounts for each course which index the appropriate Moodle courses which students will have access to.

              I wouldn't want to implement this on a site with 1500 courses. We essentially have 4 main courses which run for 3-5 years and have massive amounts of contents. For this, this software package is perfect.

              Show
              aspark Alistair Spark added a comment - We have just finalised the implementation of the wrensoft search tool which works very well for us as a small institution with a limited number of long courses. The Version 7 alpha release is required - http://www.wrensoft.com/zoom/v7.html We have created student accounts for each course which index the appropriate Moodle courses which students will have access to. I wouldn't want to implement this on a site with 1500 courses. We essentially have 4 main courses which run for 3-5 years and have massive amounts of contents. For this, this software package is perfect.
              Hide
              libertymoodle Luis de Vasconcelos added a comment -

              Alistair, why do you say you "wouldn't want to implement this on a site with 1500 courses"? Is that Zoom Search product not a good one for large Moodle installations? If not, can you describe why not? Thanks.

              Show
              libertymoodle Luis de Vasconcelos added a comment - Alistair, why do you say you "wouldn't want to implement this on a site with 1500 courses"? Is that Zoom Search product not a good one for large Moodle installations? If not, can you describe why not? Thanks.
              Hide
              aspark Alistair Spark added a comment -

              Hi Luis,

              The Zoom Search product is a great one and it can cope with 100 000s of files without any issues, their tech support is very good and ultra-responsive (free limited time trial, bug fix within 24 hours of emailing,...) - we were impressed by the cost&service.

              However, from an architectural standpoint we have gone through the route of creating student-like read-only accounts so that we have a search page for each student group (class/academic year combinations), those accounts have access to the exact same courses that the student group would have.

              Once you have created one working template for your site, creating more is just a question of changing the username&password combinations, scheduling the indexing and creating a new search box in the course (html block).

              Each of these search engines are then located in a different folder (eg http://domain.com/course1year1/search.php http://domain.com/course2year1/search.php) - this is automated by the FTP upload settings.

              However, we only needed to configure about 30 profiles in total. (ie Course 1 Year 1, Course 1 Year 2, Course 2 Year 1,....). If we had to have 1500 manual accounts in Moodle just for search, that would become a massive job to create accounts, configure all of the profiles, it becomes kind of a security risk, and maintaining this in the long run would be hard to keep track of.

              You could simply have a setup which would crawl your whole Moodle site and give you search results BUT if you have that many courses, it is more than likely that some will cover the same subject at various points, causing major user annoyance when they get the error message telling them they don't have permissions to view these results.

              Show
              aspark Alistair Spark added a comment - Hi Luis, The Zoom Search product is a great one and it can cope with 100 000s of files without any issues, their tech support is very good and ultra-responsive (free limited time trial, bug fix within 24 hours of emailing,...) - we were impressed by the cost&service. However, from an architectural standpoint we have gone through the route of creating student-like read-only accounts so that we have a search page for each student group (class/academic year combinations), those accounts have access to the exact same courses that the student group would have. Once you have created one working template for your site, creating more is just a question of changing the username&password combinations, scheduling the indexing and creating a new search box in the course (html block). Each of these search engines are then located in a different folder (eg http://domain.com/course1year1/search.php http://domain.com/course2year1/search.php ) - this is automated by the FTP upload settings. However, we only needed to configure about 30 profiles in total. (ie Course 1 Year 1, Course 1 Year 2, Course 2 Year 1,....). If we had to have 1500 manual accounts in Moodle just for search, that would become a massive job to create accounts, configure all of the profiles, it becomes kind of a security risk, and maintaining this in the long run would be hard to keep track of. You could simply have a setup which would crawl your whole Moodle site and give you search results BUT if you have that many courses, it is more than likely that some will cover the same subject at various points, causing major user annoyance when they get the error message telling them they don't have permissions to view these results.
              Hide
              dougiamas Martin Dougiamas added a comment -

              In the hackfest last week we came up with a pretty good design for completing this project.

              If this was a GSOC project, would you be available to mentor it Tomasz? I guess it would not be a lot of work for you (certainly much less than writing code!)

              Show
              dougiamas Martin Dougiamas added a comment - In the hackfest last week we came up with a pretty good design for completing this project. If this was a GSOC project, would you be available to mentor it Tomasz? I guess it would not be a lot of work for you (certainly much less than writing code!)
              Hide
              libertymoodle Luis de Vasconcelos added a comment -

              Thanks Martin. Is there any documentation for that design?

              Show
              libertymoodle Luis de Vasconcelos added a comment - Thanks Martin. Is there any documentation for that design?
              Hide
              nikunjness Nikunj Thakkar added a comment -

              Hello people,

              I am also interested in this project.One of our professor did a research for a similar kind of application and I got their research paper and currently I am working on prototype design for the project.I am first time applying for GSoC this year.So I have very little idea about approaching the project.But I am very much interested in this project.I will be updating the status of the application here.Any suggestions are appreciated.

              Show
              nikunjness Nikunj Thakkar added a comment - Hello people, I am also interested in this project.One of our professor did a research for a similar kind of application and I got their research paper and currently I am working on prototype design for the project.I am first time applying for GSoC this year.So I have very little idea about approaching the project.But I am very much interested in this project.I will be updating the status of the application here.Any suggestions are appreciated.
              Hide
              libertymoodle Luis de Vasconcelos added a comment -

              Martin, is that design that you created at the hackfest the same design described on: http://docs.moodle.org/dev/Global_search_%28GSoC2013%29

              Are you happy that that solution will only work with Moodle sites running on Apache? And other web servers, including IIS, are not supported?

              Show
              libertymoodle Luis de Vasconcelos added a comment - Martin, is that design that you created at the hackfest the same design described on: http://docs.moodle.org/dev/Global_search_%28GSoC2013%29 Are you happy that that solution will only work with Moodle sites running on Apache? And other web servers, including IIS, are not supported?
              Hide
              tmuras Tomasz Muras added a comment - - edited

              @Luis
              Is that because atm there is no DLL for solr extension? http://www.php.net/manual/en/solr.installation.php

              Show
              tmuras Tomasz Muras added a comment - - edited @Luis Is that because atm there is no DLL for solr extension? http://www.php.net/manual/en/solr.installation.php
              Hide
              xan Prateek Sachan added a comment -

              Here's a quick update of Moodle's Global Search feature. Currently, many of the things have been implemented and I would like to ask developers to try it and test it for any security leaks.

              The complete code is on Github here. The Global Search discussion is here.

              All the information on is given in the Global Search wiki:

              Steps on how to install and set-up Global Search prototype is given here: http://docs.moodle.org/dev/Global_search#Prototype:_version_1.0
              List of search features currently implemented: http://docs.moodle.org/dev/Global_search#Introduction
              Searchable content through Global Search: http://docs.moodle.org/dev/Global_search#Searchable_content_in_Global_Search

              I've also included a section Tests where there are many cases to test. Developers may test these cases or add their own. You may update the wiki with your own test cases and results of the test, which would help me in making Global Search better and prone to more security leaks.

              If you have any problem in setting-up Global Search, feel free to post here Testing Moodle's Global Search prototype or you may directly message me.

              Thanks.

              Show
              xan Prateek Sachan added a comment - Here's a quick update of Moodle's Global Search feature. Currently, many of the things have been implemented and I would like to ask developers to try it and test it for any security leaks. The complete code is on Github here . The Global Search discussion is here . All the information on is given in the Global Search wiki : Steps on how to install and set-up Global Search prototype is given here: http://docs.moodle.org/dev/Global_search#Prototype:_version_1.0 List of search features currently implemented: http://docs.moodle.org/dev/Global_search#Introduction Searchable content through Global Search: http://docs.moodle.org/dev/Global_search#Searchable_content_in_Global_Search I've also included a section Tests where there are many cases to test. Developers may test these cases or add their own. You may update the wiki with your own test cases and results of the test, which would help me in making Global Search better and prone to more security leaks. If you have any problem in setting-up Global Search, feel free to post here Testing Moodle's Global Search prototype or you may directly message me . Thanks.
              Show
              libertymoodle Luis de Vasconcelos added a comment - Tomasz, Will these compiled SOLR extensions not work on Windows? https://github.com/ecaron/php_solr.dll or http://downloads.php.net/pierre/php_solr-5.3-nts-svn20091122-vc9-x86.zip http://downloads.php.net/pierre/php_solr-5.3-svn20091122-vc9-x86.zip
              Hide
              nadavkav Nadav Kavalerchik added a comment -

              Installing php-solr-pecl extention on Debian/Ubuntu
              http://www.wijiti.com/help/37-jsolr/140-installing-apache-solr-php-extension

              Show
              nadavkav Nadav Kavalerchik added a comment - Installing php-solr-pecl extention on Debian/Ubuntu http://www.wijiti.com/help/37-jsolr/140-installing-apache-solr-php-extension
              Hide
              tmuras Tomasz Muras added a comment -

              Luis de Vasconcelos

              Luis,

              I'm using Debian and Ubuntu for the deployments, and this is what I'm testing. Tested instructions how to set it up there are on http://docs.moodle.org/dev/Global_search_%28GSoC2013%29 - I would recommend you to stick to LAMP environment. If you want to test it on Windows, then simply check those DLLs and let us know if they work for you.

              Tomek

              Show
              tmuras Tomasz Muras added a comment - Luis de Vasconcelos Luis, I'm using Debian and Ubuntu for the deployments, and this is what I'm testing. Tested instructions how to set it up there are on http://docs.moodle.org/dev/Global_search_%28GSoC2013%29 - I would recommend you to stick to LAMP environment. If you want to test it on Windows, then simply check those DLLs and let us know if they work for you. Tomek
              Hide
              xan Prateek Sachan added a comment - - edited

              Hi,
              Dan Poltawski, Eloy Lafuente (stronk7), Sam Hemelryk

              I wanted to add you all to this issue to help integrate Global Search into Moodle core.
              Aparup Banerjee advised me to add you.

              This is the original branch built on MOODLE_25 as base: https://github.com/prateeksachan/moodle/tree/gs2
              This is the rebased branch, (rebased off current master): https://github.com/prateeksachan/moodle/tree/gs2rebased

              You may find Global Search docs here: http://docs.moodle.org/dev/Global_search

              Thanks.

              Show
              xan Prateek Sachan added a comment - - edited Hi, Dan Poltawski , Eloy Lafuente (stronk7) , Sam Hemelryk I wanted to add you all to this issue to help integrate Global Search into Moodle core. Aparup Banerjee advised me to add you. This is the original branch built on MOODLE_25 as base: https://github.com/prateeksachan/moodle/tree/gs2 This is the rebased branch, (rebased off current master): https://github.com/prateeksachan/moodle/tree/gs2rebased You may find Global Search docs here: http://docs.moodle.org/dev/Global_search Thanks.
              Hide
              timhunt Tim Hunt added a comment -

              What about mod/quiz?

              How does this work? What do I need to do to enable Global search in a new module? I can't see anything on http://docs.moodle.org/dev/Global_search.

              Show
              timhunt Tim Hunt added a comment - What about mod/quiz? How does this work? What do I need to do to enable Global search in a new module? I can't see anything on http://docs.moodle.org/dev/Global_search .
              Hide
              xan Prateek Sachan added a comment - - edited

              Hi Tim.

              Thanks for the quick feedback. Actually the quiz module isn't covered under Global Search yet. I'll be writing the API for the remaining mods shortly. They have plenty of security issues and I restricted myself mainly to Resources and some smaller modules. These are the mods/resources for which I've written the GS APIs currently : http://docs.moodle.org/dev/Global_search#Searchable_content_in_Global_Search

              Edit: Every mod/resource has a different set of APIs to handle the content being indexed and its security which varies across different mods/resources.

              I've updated the Global Search docs. You may refer to http://docs.moodle.org/dev/Global_search_(GSoC2013)#Quick_setup_for_testing
              One needs to first download and install the pecl-php-solr extension, clone gs2rebased branch and set-up solr as mentioned in the docs.

              I've also included a readme file for the same purpose. Also, Tomasz installed installed GS on his own server (http://global-search.jmuras.com/) a few weeks back. (Currently, it hasn't been updated but you may view it. It works perfectly).

              Thanks.

              Show
              xan Prateek Sachan added a comment - - edited Hi Tim. Thanks for the quick feedback. Actually the quiz module isn't covered under Global Search yet. I'll be writing the API for the remaining mods shortly. They have plenty of security issues and I restricted myself mainly to Resources and some smaller modules. These are the mods/resources for which I've written the GS APIs currently : http://docs.moodle.org/dev/Global_search#Searchable_content_in_Global_Search Edit: Every mod/resource has a different set of APIs to handle the content being indexed and its security which varies across different mods/resources. I've updated the Global Search docs. You may refer to http://docs.moodle.org/dev/Global_search_(GSoC2013)#Quick_setup_for_testing One needs to first download and install the pecl-php-solr extension, clone gs2rebased branch and set-up solr as mentioned in the docs. I've also included a readme file for the same purpose. Also, Tomasz installed installed GS on his own server ( http://global-search.jmuras.com/ ) a few weeks back. (Currently, it hasn't been updated but you may view it. It works perfectly). Thanks.
              Hide
              drex Mark Drechsler added a comment -

              Noting that this doesn't have a fix version - anyone know when this is targeted for implementation?

              Show
              drex Mark Drechsler added a comment - Noting that this doesn't have a fix version - anyone know when this is targeted for implementation?
              Hide
              xan Prateek Sachan added a comment - - edited

              Hi Mark,

              The original branch was made on Moodle 2.5. I've now rebased it off the current master dev.

              My plan was to make it a part of core (2.6+) which would be released in November. I hope the feature gets integrated in the coming weeks. It is open for testing right now. You may see the forum entry: Moodle's Global Search

              Show
              xan Prateek Sachan added a comment - - edited Hi Mark, The original branch was made on Moodle 2.5. I've now rebased it off the current master dev. My plan was to make it a part of core (2.6+) which would be released in November. I hope the feature gets integrated in the coming weeks. It is open for testing right now. You may see the forum entry: Moodle's Global Search
              Hide
              dougiamas Martin Dougiamas added a comment -

              Hi Prateek,

              I don't want you to get your hopes up too much .... code freeze for 2.6 is two weeks or so away and there is a huge pile of work for us to deal with. I really doubt that there will be time for Global Search to get the review it needs. However, assuming the code is good then we should be able to land it early in the cycle for 2.7, probably January.

              Thanks for your work and I'm really keen to see it working!

              Cheers,
              Martin

              Show
              dougiamas Martin Dougiamas added a comment - Hi Prateek, I don't want you to get your hopes up too much .... code freeze for 2.6 is two weeks or so away and there is a huge pile of work for us to deal with. I really doubt that there will be time for Global Search to get the review it needs. However, assuming the code is good then we should be able to land it early in the cycle for 2.7, probably January. Thanks for your work and I'm really keen to see it working! Cheers, Martin
              Hide
              xan Prateek Sachan added a comment -

              Sure..maybe 2.7 next year then.

              Thanks.

              Show
              xan Prateek Sachan added a comment - Sure..maybe 2.7 next year then. Thanks.
              Hide
              tmuras Tomasz Muras added a comment -

              OK, So it looks to me, that the best way to go, is to wait for 2.6 release and new master for 2.7. As soon as we have a new branch, we should seize the opportunity and send the code for integration.

              Show
              tmuras Tomasz Muras added a comment - OK, So it looks to me, that the best way to go, is to wait for 2.6 release and new master for 2.7. As soon as we have a new branch, we should seize the opportunity and send the code for integration.
              Hide
              xan Prateek Sachan added a comment - - edited

              Yes. I hope it gets the review required by the next stable moodle (2.7) release. In the remaining months of this year, I'll try and write GS APIs for the remaining modules and then asking moodle integrators to review it.

              Thanks.

              Show
              xan Prateek Sachan added a comment - - edited Yes. I hope it gets the review required by the next stable moodle (2.7) release. In the remaining months of this year, I'll try and write GS APIs for the remaining modules and then asking moodle integrators to review it. Thanks.
              Hide
              dougiamas Martin Dougiamas added a comment - - edited

              The first reviewers at HQ will be the Moodle HQ BACKEND team.

              Of course anyone from the community is welcome to pitch in with reviews and comments here anytime.

              Show
              dougiamas Martin Dougiamas added a comment - - edited The first reviewers at HQ will be the Moodle HQ BACKEND team. Of course anyone from the community is welcome to pitch in with reviews and comments here anytime.
              Hide
              libertymoodle Luis de Vasconcelos added a comment -

              This new Global Search requires Apache Solr, so what will happen to Moodle sites that run under IIS on Windows? Will Global Search not work under IIS?

              Show
              libertymoodle Luis de Vasconcelos added a comment - This new Global Search requires Apache Solr, so what will happen to Moodle sites that run under IIS on Windows? Will Global Search not work under IIS?
              Hide
              xan Prateek Sachan added a comment -

              Hi Luis,

              As pointed out earlier, GS uses php solr extension. As long as you have php solr extension installed on your server, GS will work. It's just another php extension, so it won't affect Moodle sites. (I've just tested out GS on LAMP). You may find this link useful which underlines all the dependencies and their installation of php solr extension.

              Show
              xan Prateek Sachan added a comment - Hi Luis, As pointed out earlier, GS uses php solr extension. As long as you have php solr extension installed on your server, GS will work. It's just another php extension, so it won't affect Moodle sites. (I've just tested out GS on LAMP). You may find this link useful which underlines all the dependencies and their installation of php solr extension.
              Hide
              itamart Itamar Tzadok added a comment -

              Prateek, I haven't had the chance to examine thoroughly so off hand do you have any idea how easy or otherwise it would be to make this work with Solr php client? For windows installations it may not be straightforward to add a php_solr extension but the client may be a viable option.

              Show
              itamart Itamar Tzadok added a comment - Prateek, I haven't had the chance to examine thoroughly so off hand do you have any idea how easy or otherwise it would be to make this work with Solr php client? For windows installations it may not be straightforward to add a php_solr extension but the client may be a viable option.
              Hide
              xan Prateek Sachan added a comment - - edited

              Hi Itamar Tzadok,

              I researched a bit for choosing the client for interacting with solr. Making it work using the Solr Php client would be easy and do-able, though may require many modifications in the current implementation. I concentrated towards using the php_solr extension.

              There are many php_solr dlls available for Windows. I haven't checked them though. I used LAMP for GS implementation.

              https://github.com/ecaron/php_solr.dll
              http://downloads.php.net/pierre/php_solr-5.3-nts-svn20091122-vc9-x86.zip
              http://downloads.php.net/pierre/php_solr-5.3-svn20091122-vc9-x86.zip

              If you want, you may test it on Windows using the dlls, and notify me. It would be great.

              Edit: I installed php_solr extension using the github dll given above on Windows. The extension is working.

              Thanks.

              Show
              xan Prateek Sachan added a comment - - edited Hi Itamar Tzadok , I researched a bit for choosing the client for interacting with solr. Making it work using the Solr Php client would be easy and do-able, though may require many modifications in the current implementation. I concentrated towards using the php_solr extension. There are many php_solr dlls available for Windows. I haven't checked them though. I used LAMP for GS implementation. https://github.com/ecaron/php_solr.dll http://downloads.php.net/pierre/php_solr-5.3-nts-svn20091122-vc9-x86.zip http://downloads.php.net/pierre/php_solr-5.3-svn20091122-vc9-x86.zip If you want, you may test it on Windows using the dlls, and notify me. It would be great. Edit: I installed php_solr extension using the github dll given above on Windows. The extension is working. Thanks.
              Hide
              juerg.hoerner Juerg Hoerner added a comment -

              in your testmoodle I tried to find words in the attached pdf in the glossary. As example: "Introduction" . Maybe that this does not work yet.

              can I delete the index and reindexing all content, like in the old module?

              Thanks for your very important Global search, one of the most important module.

              Show
              juerg.hoerner Juerg Hoerner added a comment - in your testmoodle I tried to find words in the attached pdf in the glossary. As example: "Introduction" . Maybe that this does not work yet. can I delete the index and reindexing all content, like in the old module? Thanks for your very important Global search, one of the most important module.
              Hide
              xan Prateek Sachan added a comment -

              Hi Juerg Hoerner,

              Yes. You may delete your index and re-index content from cron. All specifications are giving in Global Search docs.

              Thanks.

              Show
              xan Prateek Sachan added a comment - Hi Juerg Hoerner , Yes. You may delete your index and re-index content from cron. All specifications are giving in Global Search docs. Thanks.
              Hide
              mandrews Mark Andrews added a comment -

              Hi Prateek,
              I've just had a play with this and it's great! - really pleased to hear it might make 2.7

              couple of questions/comments from testing though;

              • As it's a global search will it also be searching; people & courses? would help to make it truly global search. - Perhaps as an option that admin's could enable/disable
              • Also I wonder if there needs to be some tweaking done to the 'filter query' area as fields like 'From records having this author' could mean different things to different users; a participant looking for a particular forum post by a class mate to the up loader of a resource.

              Cheers
              Mark

              Show
              mandrews Mark Andrews added a comment - Hi Prateek, I've just had a play with this and it's great! - really pleased to hear it might make 2.7 couple of questions/comments from testing though; As it's a global search will it also be searching; people & courses? would help to make it truly global search. - Perhaps as an option that admin's could enable/disable Also I wonder if there needs to be some tweaking done to the 'filter query' area as fields like 'From records having this author' could mean different things to different users; a participant looking for a particular forum post by a class mate to the up loader of a resource. Cheers Mark
              Hide
              xan Prateek Sachan added a comment -

              Hi Mark,
              Thanks for the reviews.

              As it was the first version of GS, hence I restricted myself to only mods/resources. I still have some mods for which I would be writing GS APIs in next months. Only then, I could think about including users/courses.

              "From records having this author" means "all records that have an author as x". These are mainly uploaded PDFs/PPTs/etc. that have an "author" field in their metadata. I guess I should change the string a bit to suit the users. Thanks for offering your thoughts on this one.

              Show
              xan Prateek Sachan added a comment - Hi Mark, Thanks for the reviews. As it was the first version of GS, hence I restricted myself to only mods/resources. I still have some mods for which I would be writing GS APIs in next months. Only then, I could think about including users/courses. "From records having this author" means "all records that have an author as x". These are mainly uploaded PDFs/PPTs/etc. that have an "author" field in their metadata. I guess I should change the string a bit to suit the users. Thanks for offering your thoughts on this one.
              Hide
              dougiamas Martin Dougiamas added a comment -

              This is now in the backlog of the BACKEND team to review.

              Show
              dougiamas Martin Dougiamas added a comment - This is now in the backlog of the BACKEND team to review.
              Hide
              timdrescher Tim Drescher added a comment -

              We would really love a global search. It would greatly assist our staff/students in finding the course or information they need.

              Show
              timdrescher Tim Drescher added a comment - We would really love a global search. It would greatly assist our staff/students in finding the course or information they need.
              Hide
              nebgor Aparup Banerjee added a comment - - edited

              Please note there could be other plugins utilizing solr directly
              (such as https://moodle.org/plugins/view.php?plugin=tool_coursesearch)
              so steps could be made wrt allowing migration of those to use the global search api to access the installed search engine (the api in moodle core after this patch).

              or to just keep those separate (ie: self managed instance(s) of solr), but this would lead to more inefficient usage of solr.

              ps: it would be great if someone could lend even a glancing eye to this for a review..

              Show
              nebgor Aparup Banerjee added a comment - - edited Please note there could be other plugins utilizing solr directly (such as https://moodle.org/plugins/view.php?plugin=tool_coursesearch ) so steps could be made wrt allowing migration of those to use the global search api to access the installed search engine (the api in moodle core after this patch). or to just keep those separate (ie: self managed instance(s) of solr), but this would lead to more inefficient usage of solr. ps: it would be great if someone could lend even a glancing eye to this for a review..
              Hide
              xan Prateek Sachan added a comment -

              The Global Search implementation differs from that of the coursesearch plugin as they are using completely different Solr php-client-libraries for interacting with the Solr server through Moodle. Therefore, the API in moodle core after this patch wouldn't affect course search.

              Show
              xan Prateek Sachan added a comment - The Global Search implementation differs from that of the coursesearch plugin as they are using completely different Solr php-client-libraries for interacting with the Solr server through Moodle. Therefore, the API in moodle core after this patch wouldn't affect course search.
              Hide
              libertymoodle Luis de Vasconcelos added a comment - - edited

              Adam Morris and antriver have developed the moodle-block_search block. See https://github.com/antriver/moodle-block_search and https://moodle.org/mod/forum/discuss.php?d=239646.

              How does that search block compare with this Global Search 2 feature?

              Show
              libertymoodle Luis de Vasconcelos added a comment - - edited Adam Morris and antriver have developed the moodle-block_search block. See https://github.com/antriver/moodle-block_search and https://moodle.org/mod/forum/discuss.php?d=239646 . How does that search block compare with this Global Search 2 feature?
              Hide
              amirelion Amir Elion added a comment - - edited

              Luis, I dd some initial testing with the suggested block.
              it does look like a good start.
              It searches course and activity names and description, and can be configured to choose which ones to search.
              It also seems to handle permissions (with ability to show grayed out/not show users inaccessible search results), wildcards, "quoted" search terms, exclude words.
              Also classifies and allows quick navigation by types of search results and

              It lacks search inside activity content - e.g. page content, glossary terms, wiki pages, forum posts, etc.
              If the above can be added, I think it would be a good candidate for this.

              Show
              amirelion Amir Elion added a comment - - edited Luis, I dd some initial testing with the suggested block. it does look like a good start. It searches course and activity names and description, and can be configured to choose which ones to search. It also seems to handle permissions (with ability to show grayed out/not show users inaccessible search results), wildcards, "quoted" search terms, exclude words. Also classifies and allows quick navigation by types of search results and It lacks search inside activity content - e.g. page content, glossary terms, wiki pages, forum posts, etc. If the above can be added, I think it would be a good candidate for this.
              Hide
              tmuras Tomasz Muras added a comment - - edited

              As far as I can see this search implementation does not do any relevancy search. It will simply return all rows with a given keyword, not neceserily the relevant first. This will not work well for more than 10 results.

              As this is simply implemented as a DB query with like %keyword% clause it will not scale - this can only be used on smaller sites.

              You can also compare it with solr features: http://lucene.apache.org/solr/features.html - most of them will be missing.

              Show
              tmuras Tomasz Muras added a comment - - edited As far as I can see this search implementation does not do any relevancy search. It will simply return all rows with a given keyword, not neceserily the relevant first. This will not work well for more than 10 results. As this is simply implemented as a DB query with like %keyword% clause it will not scale - this can only be used on smaller sites. You can also compare it with solr features: http://lucene.apache.org/solr/features.html - most of them will be missing.
              Hide
              xan Prateek Sachan added a comment -

              As Moodle sites become big, it becomes very difficult (concerning speed and optimization) to do direct DB searches. The whole concept behind "Global Search" was to make the entire content within modules and resources searchable alongwith providing complex search queries. I highly doubt if these could be implemented via direct DB searches in an optimized way.

              That's why indexing was used which is the obvious solution for making huge chunks of data searchable.

              Also, the kinds of queries that one would use for searching will put a heavy load on your database server.

              Thanks.

              Show
              xan Prateek Sachan added a comment - As Moodle sites become big, it becomes very difficult (concerning speed and optimization) to do direct DB searches. The whole concept behind "Global Search" was to make the entire content within modules and resources searchable alongwith providing complex search queries . I highly doubt if these could be implemented via direct DB searches in an optimized way. That's why indexing was used which is the obvious solution for making huge chunks of data searchable. Also, the kinds of queries that one would use for searching will put a heavy load on your database server. Thanks.
              Hide
              binare Eugene Venter added a comment - - edited

              We're currently planning on using this implementation for one of our clients. The current state of the code seems to need more work and optimisation and we've started fixing up a few things (with a lot more to come).

              For a start, we'll be pushing our contributable updates here: https://github.com/catalyst/moodle

              Show
              binare Eugene Venter added a comment - - edited We're currently planning on using this implementation for one of our clients. The current state of the code seems to need more work and optimisation and we've started fixing up a few things (with a lot more to come). For a start, we'll be pushing our contributable updates here: https://github.com/catalyst/moodle
              Hide
              dougiamas Martin Dougiamas added a comment -

              Pushed back to BACKEND to look at in this sprint. Needs a basic review before going to integration.

              Show
              dougiamas Martin Dougiamas added a comment - Pushed back to BACKEND to look at in this sprint. Needs a basic review before going to integration.
              Hide
              jonathan Jonathan Harker added a comment - - edited

              Here is my 2¢

              1. I'd say that as it is, it is not sufficiently loosely coupled for use with anything other than Solr. A rewrite would be required for pluggable backends for Elastic et al.
              2. The Solr implementation should make use of dynamic fields, which requires much less dependence on Solr schema.xml configuration. Dynamic fields are in the default example configuration, and work fine with 1.4, 3.x, and 4.x
              3. The code needs a bunch of clean-up - e.g. everything in search/documents is unreachable.
              4. Pagination should be done via the Solr API rather than fetching a hundred documents and hoping for the best.
              5. We've done some work to display facets, which should be a part of this solution.

              Show
              jonathan Jonathan Harker added a comment - - edited Here is my 2¢ 1. I'd say that as it is, it is not sufficiently loosely coupled for use with anything other than Solr. A rewrite would be required for pluggable backends for Elastic et al. 2. The Solr implementation should make use of dynamic fields, which requires much less dependence on Solr schema.xml configuration. Dynamic fields are in the default example configuration, and work fine with 1.4, 3.x, and 4.x 3. The code needs a bunch of clean-up - e.g. everything in search/documents is unreachable. 4. Pagination should be done via the Solr API rather than fetching a hundred documents and hoping for the best. 5. We've done some work to display facets, which should be a part of this solution.
              Hide
              tmuras Tomasz Muras added a comment -

              1. I wouldn't worry about plug-ability too much. I think that chances for anyone to implement any other backend are minimal, let's worry when someone tries.
              2. Yeah, why not.
              3. It's most likely an old core - you're right, it should be cleaned up.
              4. It's going to be very difficult to replicate all Moodle logic (permissions check) on the solr side. I suggest first version with keeping it in Moodle, then building from there. First step could be to filter out obvious non-accessible documents and double check remaining ones with Moodle.
              5. Phase 2 maybe? Let's get at least minimal working version into Moodle core.

              Show
              tmuras Tomasz Muras added a comment - 1. I wouldn't worry about plug-ability too much. I think that chances for anyone to implement any other backend are minimal, let's worry when someone tries. 2. Yeah, why not. 3. It's most likely an old core - you're right, it should be cleaned up. 4. It's going to be very difficult to replicate all Moodle logic (permissions check) on the solr side. I suggest first version with keeping it in Moodle, then building from there. First step could be to filter out obvious non-accessible documents and double check remaining ones with Moodle. 5. Phase 2 maybe? Let's get at least minimal working version into Moodle core.
              Hide
              jonathan Jonathan Harker added a comment - - edited

              I think it should be possible to do the permissions thing. There appear to be two tricks to doing permissions right:

              1. make each module responsible at index time for ensuring that anything needed to filter on permissions is indexed in a field in the module document (e.g. course ID, hidden, visible, deleted, etc), and
              2. make each module responsible at query time for figuring out what $USER can and cannot see, and apply the right filter (using the above).

              Using dynamic fields, different document types (e.g. course, lesson, forum) can have fields in common, one of which should probably be course_id - this will at the very least enable us to restrict results to the courses $USER is enrolled in by using something like

               'course_id:(' . join(',', $enrolled_courses) . ')' 

              It may be a week or two before we get time to collapse the work we've done here into something that deserves a PULL request... there are some curly bespoke bits we'll have to remove first!

              Show
              jonathan Jonathan Harker added a comment - - edited I think it should be possible to do the permissions thing. There appear to be two tricks to doing permissions right: 1. make each module responsible at index time for ensuring that anything needed to filter on permissions is indexed in a field in the module document (e.g. course ID, hidden, visible, deleted, etc), and 2. make each module responsible at query time for figuring out what $USER can and cannot see, and apply the right filter (using the above). Using dynamic fields, different document types (e.g. course, lesson, forum) can have fields in common, one of which should probably be course_id - this will at the very least enable us to restrict results to the courses $USER is enrolled in by using something like 'course_id:(' . join(',', $enrolled_courses) . ')' It may be a week or two before we get time to collapse the work we've done here into something that deserves a PULL request... there are some curly bespoke bits we'll have to remove first!
              Hide
              tmuras Tomasz Muras added a comment -

              That makes sense. It will require a lot of testing but it would be a Holy Grail of Search if you can put that logic into solr side.

              Show
              tmuras Tomasz Muras added a comment - That makes sense. It will require a lot of testing but it would be a Holy Grail of Search if you can put that logic into solr side.
              Hide
              xan Prateek Sachan added a comment -

              Hi all,

              This is just an update. The official php-pecl-solr extensions for Solr 4.x[1] have been released. I'll be testing it out and updating the Docs in the coming week.

              Thanks.

              [1]: http://pecl.php.net/package/solr

              Show
              xan Prateek Sachan added a comment - Hi all, This is just an update. The official php-pecl-solr extensions for Solr 4.x [1] have been released. I'll be testing it out and updating the Docs in the coming week. Thanks. [1] : http://pecl.php.net/package/solr
              Hide
              danielneis Daniel Neis added a comment -

              Hello,

              i've managed to make it work on top of current master (2.7 beta):
              https://github.com/danielneis/moodle/tree/MDL-31989

              I've tested it creating a medium test course (with 500 forum posts + 200 pages) and it worked well.
              Also, the current php-pecl-solr extension version 2.0.0b works well with the Solr 4.8.0

              Kind regards,
              Daniel

              Show
              danielneis Daniel Neis added a comment - Hello, i've managed to make it work on top of current master (2.7 beta): https://github.com/danielneis/moodle/tree/MDL-31989 I've tested it creating a medium test course (with 500 forum posts + 200 pages) and it worked well. Also, the current php-pecl-solr extension version 2.0.0b works well with the Solr 4.8.0 Kind regards, Daniel
              Hide
              nebgor Aparup Banerjee added a comment -

              Thanks Daniel, good to know core hasn't changed that much for this.

              2.7 is due for release next week so;
              It'll be great if Jonathan Harker's excellent suggestions on index/query time permissions API were ready so that this could be incorporated early into 2.8's development. Scrutiny on security seems the major point on getting this integrated.

              Show
              nebgor Aparup Banerjee added a comment - Thanks Daniel, good to know core hasn't changed that much for this. 2.7 is due for release next week so; It'll be great if Jonathan Harker 's excellent suggestions on index/query time permissions API were ready so that this could be incorporated early into 2.8's development. Scrutiny on security seems the major point on getting this integrated.
              Hide
              martinlanghoff Martín Langhoff added a comment -

              Looking at using this (and helping polish it along the way. Jonathan Harker - you mention you have done some work on facets. Is that available anywhere ? Thanks!

              Show
              martinlanghoff Martín Langhoff added a comment - Looking at using this (and helping polish it along the way. Jonathan Harker - you mention you have done some work on facets. Is that available anywhere ? Thanks!
              Hide
              dmonllao David Monllaó added a comment -

              Reviewing this now

              Show
              dmonllao David Monllaó added a comment - Reviewing this now
              Hide
              dmonllao David Monllaó added a comment -

              Assigning back to Prateek as he expressed his will to continue helping here

              Show
              dmonllao David Monllaó added a comment - Assigning back to Prateek as he expressed his will to continue helping here
              Hide
              dmonllao David Monllaó added a comment -

              Thanks for working on this Prateek; it looks very good and works, although before submitting it to integration we would need to consider a few things.

              1. Multiple search plugins support: Even if supporting multiple plugins is deferred to a future release there are parts of the code that would be better to abstract now, before 3rd parties begins adding support for global search in their own plugins, otherwise they will have to update a big bunch of code (even abstracting now they will probably have to change something in future)
                1. To abstract before it gets integrated
                  1. We shouldn't allow hardcoded solr calls in plugins code, we should provide a common API a core_search\document for example each search plugin can extend it, but the modules will reference the generic document.
                  2. global_search_engine should be abstracted too, we can have a \core_search\engine + a \core_search\solr\engine (which would be moved to \search_solr\engine in future)
                2. Leaving to abstract in future:
                  1. Directory structure with new plugin type (search/PLUGINSDIR/solr, search/PLUGINSDIR/elastic...)
                  2. Administration settings pages reading a dynamic list of plugins from search/PLUGINSDIR
                  3. Settings values; move them to the future search_solr (example name) plugin space
              2. Modules (or plugins/components) implementing FEATURE_GLOBAL_SEARCH: You can easily abstract most of $MODULENAME_search_access(), including groups, activities visibility... You can abstract even the 'mod/$PLUGINNAME:view' capability name in most cases, also in this case is more appropriate to use has_capability() than require_capability + try & catch; all this would be easier using OOP, related with the next point
              3. I would move search-related stuff to plugin_or_component/db/search.php, which would be common for all types of plugins; IMO this patch adds too much logic into lib.php files and in future we might be interested in adding support for other plugin types or components like in https://github.com/catalyst/moodle/commit/5aba875be1ddc2690a6e2ffd67f766ccb51b8fc0#diff-3 search_get_iterators()
              4. In general, I think that we can move a few functions files to classes according to their behaviour, proposed changes below
              5. Again, related with support for other plugin types or components; I would rename module and modulelink fields to component and component link
              6. The iterators are using timemodified/modified fields intensively, some of the used fields are not indexed in the database, I'm not sure about the current policies to add new index to db
              7. Coding styles (in my dev branch I've fixed what git --check proposes me, but local_moodlecheck and local_codechecker should also pass with a 100%)
              8. Unit tests should use another collection, they are currently affecting the "production" one
              9. We will also need acceptance tests as this is a new feature http://docs.moodle.org/dev/Acceptance_testing but I don't think it will be easy to do...
              10. A renderer.php should be used for all display-related stuff
              11. Silly question, I can't see how will search/install.php be accessed, probably I am missing something
              12. Use fullname() rather than $user->firstname . ' ' . $user->lastname
              13. You could try using get_fast_modinfo($course)->instances[$moduletype][$instanceid] in $MODULENAME_search_get_documents() and $MODULENAME_search_access(), you can probably save some db queries
              14. The new MDL-31089 functions can be used in forum_search_access() to control the discussions groups.
              15. Silly question again, about gmdate('Y-m-d\TH:i:s\Z', $page->timemodified). Are you required to use this specific format?
              16. As a personal preference I try to avoid files like search/solr/connection.php, including files inside other files can overwrite other vars, a function would be an alternative.
              17. No hardcoded text in scripts please, all in lang strings
              18. Class files should only include 1 class
              19. Any reason for set_time_limit(576000);? Can't just be 0?
              20. If possible would also be good to address/comment the pagination and access points Jonathan Harker commented about, I'm not a solr expert so I guess you all can judge better than me

              Proposed code organization changes:

              • classes/document.php (representation of a document)
              • classes/engine.php (ABSTRACT)
                • search_get_modules() -> search_get_components() to include all plugin types + core
                • search_get_iterators()
                • search_reset_config()
                • search_get_config()
                • search_get_config_file()
                • Generic abstract methods declaration
              • classes/solr/engine.php (EXTENDS classes/engine.php)
                • Renaming methods to fit classes/engine.php common API methods
                • search/solr/connection.php
                • search/solr/lib.php
                • solr_check_server() (STATIC)
                • solr_installed() (STATIC)
                • solr_execute_query()
                • solr_set_query()
                • solr_set_highlight()
                • solr_prepare_filter()
                • solr_add_fields()
                • solr_add_highlight_content()
                • solr_merge_highlight_field_values()
                • solr_query_response()
                • solr_post_file()
              • classes/indexer.php (DEPENDS ON classes/engine.php, all specific solr logic should be in classes/solr/engine)
                • search_optimize_index()
                • search_index()
                • search_index_files()
                • search_delete_index()
                • search_delete_index_by_id()
              • classes/search_form.php
                • class core_search_search_form
              • renderer.php
                • search_display_results()
                • search_get_user_url()
                • //search_display_form() IMO that function is unnecessary

              I've created a dev/testing branch where I've:

              • Rebased and updated the current implementation on top of latest 2.8dev
              • Considered that the 2.0 PECL solr package was released and 2.x -> solr 4.x and 1.x -> solr 3.x
              • Included catalyst and Daniel fixes
              • Removed styles (ignore me completely)

              *https://github.com/dmonllao/moodle/compare/MDL-31989_master*

              Show
              dmonllao David Monllaó added a comment - Thanks for working on this Prateek; it looks very good and works, although before submitting it to integration we would need to consider a few things. Multiple search plugins support: Even if supporting multiple plugins is deferred to a future release there are parts of the code that would be better to abstract now, before 3rd parties begins adding support for global search in their own plugins, otherwise they will have to update a big bunch of code (even abstracting now they will probably have to change something in future) To abstract before it gets integrated We shouldn't allow hardcoded solr calls in plugins code, we should provide a common API a core_search\document for example each search plugin can extend it, but the modules will reference the generic document. global_search_engine should be abstracted too, we can have a \core_search\engine + a \core_search\solr\engine (which would be moved to \search_solr\engine in future) Leaving to abstract in future: Directory structure with new plugin type (search/PLUGINSDIR/solr, search/PLUGINSDIR/elastic...) Administration settings pages reading a dynamic list of plugins from search/PLUGINSDIR Settings values; move them to the future search_solr (example name) plugin space Modules (or plugins/components) implementing FEATURE_GLOBAL_SEARCH: You can easily abstract most of $MODULENAME_search_access(), including groups, activities visibility... You can abstract even the 'mod/$PLUGINNAME:view' capability name in most cases, also in this case is more appropriate to use has_capability() than require_capability + try & catch; all this would be easier using OOP, related with the next point I would move search-related stuff to plugin_or_component/db/search.php, which would be common for all types of plugins; IMO this patch adds too much logic into lib.php files and in future we might be interested in adding support for other plugin types or components like in https://github.com/catalyst/moodle/commit/5aba875be1ddc2690a6e2ffd67f766ccb51b8fc0#diff-3 search_get_iterators() In general, I think that we can move a few functions files to classes according to their behaviour, proposed changes below Again, related with support for other plugin types or components; I would rename module and modulelink fields to component and component link The iterators are using timemodified/modified fields intensively, some of the used fields are not indexed in the database, I'm not sure about the current policies to add new index to db Coding styles (in my dev branch I've fixed what git --check proposes me, but local_moodlecheck and local_codechecker should also pass with a 100%) Unit tests should use another collection, they are currently affecting the "production" one We will also need acceptance tests as this is a new feature http://docs.moodle.org/dev/Acceptance_testing but I don't think it will be easy to do... A renderer.php should be used for all display-related stuff Silly question, I can't see how will search/install.php be accessed, probably I am missing something Use fullname() rather than $user->firstname . ' ' . $user->lastname You could try using get_fast_modinfo($course)->instances [$moduletype] [$instanceid] in $MODULENAME_search_get_documents() and $MODULENAME_search_access(), you can probably save some db queries The new MDL-31089 functions can be used in forum_search_access() to control the discussions groups. Silly question again, about gmdate('Y-m-d\TH:i:s\Z', $page->timemodified) . Are you required to use this specific format? As a personal preference I try to avoid files like search/solr/connection.php, including files inside other files can overwrite other vars, a function would be an alternative. No hardcoded text in scripts please, all in lang strings Class files should only include 1 class Any reason for set_time_limit(576000); ? Can't just be 0? If possible would also be good to address/comment the pagination and access points Jonathan Harker commented about, I'm not a solr expert so I guess you all can judge better than me Proposed code organization changes: classes/document.php (representation of a document) classes/engine.php (ABSTRACT) search_get_modules() -> search_get_components() to include all plugin types + core search_get_iterators() search_reset_config() search_get_config() search_get_config_file() Generic abstract methods declaration classes/solr/engine.php (EXTENDS classes/engine.php) Renaming methods to fit classes/engine.php common API methods search/solr/connection.php search/solr/lib.php solr_check_server() (STATIC) solr_installed() (STATIC) solr_execute_query() solr_set_query() solr_set_highlight() solr_prepare_filter() solr_add_fields() solr_add_highlight_content() solr_merge_highlight_field_values() solr_query_response() solr_post_file() classes/indexer.php (DEPENDS ON classes/engine.php, all specific solr logic should be in classes/solr/engine) search_optimize_index() search_index() search_index_files() search_delete_index() search_delete_index_by_id() classes/search_form.php class core_search_search_form renderer.php search_display_results() search_get_user_url() //search_display_form() IMO that function is unnecessary I've created a dev/testing branch where I've: Rebased and updated the current implementation on top of latest 2.8dev Considered that the 2.0 PECL solr package was released and 2.x -> solr 4.x and 1.x -> solr 3.x Included catalyst and Daniel fixes Removed styles (ignore me completely) * https://github.com/dmonllao/moodle/compare/MDL-31989_master*
              Hide
              dmonllao David Monllaó added a comment -

              As commented in BACKEND meeting, as there is still a bit of work to do before this can be integrated we can move this issue to an epic and open subtasks, I will begin tomorrow.

              Show
              dmonllao David Monllaó added a comment - As commented in BACKEND meeting, as there is still a bit of work to do before this can be integrated we can move this issue to an epic and open subtasks, I will begin tomorrow.
              Hide
              xan Prateek Sachan added a comment -

              Hi all,

              I'm happy to work on this as suggested by David and have actually started the implementation. (It might take some time though)

              Thanks.

              Show
              xan Prateek Sachan added a comment - Hi all, I'm happy to work on this as suggested by David and have actually started the implementation. (It might take some time though) Thanks.
              Hide
              dmonllao David Monllaó added a comment -

              Hi Prateek,

              Good to hear that! As commented by private message, today I will create linked issues to this epic so we can split the work in reasonable pieces, feel free to assign issues to you.

              Show
              dmonllao David Monllaó added a comment - Hi Prateek, Good to hear that! As commented by private message, today I will create linked issues to this epic so we can split the work in reasonable pieces, feel free to assign issues to you.
              Hide
              salvetore Michael de Raadt added a comment -

              I've taken this out of development so that Prateek, or anyone else interested, can work on the tasks David has defined.

              Show
              salvetore Michael de Raadt added a comment - I've taken this out of development so that Prateek, or anyone else interested, can work on the tasks David has defined.
              Hide
              danielneis Daniel Neis added a comment -

              Hello, David

              thanks for your review and for integrating my changes!
              I've made some more on top of your work:

              https://github.com/danielneis/moodle/tree/MDL-31989_master

              Hope that you like,
              Daniel

              Show
              danielneis Daniel Neis added a comment - Hello, David thanks for your review and for integrating my changes! I've made some more on top of your work: https://github.com/danielneis/moodle/tree/MDL-31989_master Hope that you like, Daniel
              Hide
              danielneis Daniel Neis added a comment -

              Hello,

              i just added a commit that moves the forms to search/classes directory as David suggested.

              Kind regards,
              Daniel

              Show
              danielneis Daniel Neis added a comment - Hello, i just added a commit that moves the forms to search/classes directory as David suggested. Kind regards, Daniel
              Hide
              dmonllao David Monllaó added a comment -

              Hi Daniel,

              Thanks for your contribution, I will probably not work on this issue though, Prateek Sachan expressed his interest in working on it so probably he will take a look at them.

              Regards,
              David

              Show
              dmonllao David Monllaó added a comment - Hi Daniel, Thanks for your contribution, I will probably not work on this issue though, Prateek Sachan expressed his interest in working on it so probably he will take a look at them. Regards, David
              Hide
              salvetore Michael de Raadt added a comment -

              It's good that progress has been made on this issue. I'm taking it out of the sprint so we can close the sprint. It would be good if progress can continue now that some changes have been suggested. I'll leave that to Prateek in the short-term.

              Show
              salvetore Michael de Raadt added a comment - It's good that progress has been made on this issue. I'm taking it out of the sprint so we can close the sprint. It would be good if progress can continue now that some changes have been suggested. I'll leave that to Prateek in the short-term.
              Hide
              tuekorsgaard Tue Korsgaard added a comment -

              Any news on whether development will resume on t this issue?

              Show
              tuekorsgaard Tue Korsgaard added a comment - Any news on whether development will resume on t this issue?
              Hide
              nbruley Nathan Bruley added a comment -

              Where do I go to download the global search files solconfig.xml and schema.xml mentioned at (https://docs.moodle.org/dev/Global_search_%28GSoC2013%29#Installing_PHP_Solr_extension) as well as the plugin files if I don't use git?
              Thanks.

              Show
              nbruley Nathan Bruley added a comment - Where do I go to download the global search files solconfig.xml and schema.xml mentioned at ( https://docs.moodle.org/dev/Global_search_%28GSoC2013%29#Installing_PHP_Solr_extension ) as well as the plugin files if I don't use git? Thanks.
              Show
              tmuras Tomasz Muras added a comment - Nathan Bruley try those: https://github.com/danielneis/moodle/tree/MDL-31989_master/search/solr/solr_conf
              Hide
              td0s toby saunders added a comment -

              I was wondering if there was a timescale for merging this into the stable branch at the moment?

              Show
              td0s toby saunders added a comment - I was wondering if there was a timescale for merging this into the stable branch at the moment?
              Hide
              dmonllao David Monllaó added a comment -

              For reference, I've rebased the review branch on top of latest weekly master https://github.com/dmonllao/moodle/tree/MDL-31989_master

              Show
              dmonllao David Monllaó added a comment - For reference, I've rebased the review branch on top of latest weekly master https://github.com/dmonllao/moodle/tree/MDL-31989_master
              Hide
              danielneis Daniel Neis added a comment -

              Hello,

              I've started some code to support elastic search:
              https://github.com/danielneis/moodle/commit/d24d6e165e440c69240a3f2e465c65ba3e60a16a

              I've first tackled with changes needed to add the another search engine.
              Next step is implementing the functions to add documents to index.
              As elastic search has a nice restful interface, similar to the existing solr api, it should be easy to implement =)

              Show
              danielneis Daniel Neis added a comment - Hello, I've started some code to support elastic search: https://github.com/danielneis/moodle/commit/d24d6e165e440c69240a3f2e465c65ba3e60a16a I've first tackled with changes needed to add the another search engine. Next step is implementing the functions to add documents to index. As elastic search has a nice restful interface, similar to the existing solr api, it should be easy to implement =)
              Hide
              skodak Petr Skoda added a comment -

              The searching part is easy - the problem is to do the access control properly.

              Show
              skodak Petr Skoda added a comment - The searching part is easy - the problem is to do the access control properly.
              Hide
              mchaney2010 Mark Chaney added a comment -

              Hi all, is there an update regarding the search functionality and any perceived Moodle version when it may be released (even if its as an experimental setting).

              Show
              mchaney2010 Mark Chaney added a comment - Hi all, is there an update regarding the search functionality and any perceived Moodle version when it may be released (even if its as an experimental setting).
              Hide
              danielneis Daniel Neis added a comment - - edited

              Hello,

              I've got Global Search basic indexing and searching function working with elastic search:
              https://github.com/danielneis/moodle/tree/MDL-31989_master

              The searching part is easy, and that is why elastic search is cool =)
              Also, the creation of the index and the setup is easy too, as you can see at
              https://www.elastic.co/guide/en/elasticsearch/reference/1.5/_installation.html

              You just need a working JVM and do not need to bother with configuration XML,
              until now you also need to know nothing about tokenizing, stemming, common words and so.

              About the access control, it is already being done: may not be the best, but it may work as a function is
              called for each document to test if user can see it. Each module implements it's own "search_access" function,
              that at least tests capabilities like 'mod/book:read' in the context of the document.

              Any suggestions to improve this are welcome =)

              Kind regards,
              Daniel

              Show
              danielneis Daniel Neis added a comment - - edited Hello, I've got Global Search basic indexing and searching function working with elastic search: https://github.com/danielneis/moodle/tree/MDL-31989_master The searching part is easy, and that is why elastic search is cool =) Also, the creation of the index and the setup is easy too, as you can see at https://www.elastic.co/guide/en/elasticsearch/reference/1.5/_installation.html You just need a working JVM and do not need to bother with configuration XML, until now you also need to know nothing about tokenizing, stemming, common words and so. About the access control, it is already being done: may not be the best, but it may work as a function is called for each document to test if user can see it. Each module implements it's own "search_access" function, that at least tests capabilities like 'mod/book:read' in the context of the document. Any suggestions to improve this are welcome =) Kind regards, Daniel
              Hide
              danielneis Daniel Neis added a comment -

              Hello,

              here is an updated version of global search.
              Engines (solr and elasticsearch) are now plugins and there is a new class core_search that is used by the block and the renderers.
              Also, all configs are now under "Plugins - Global Search" sothere is no more top level Global Search.
              Still have to move some strings from lang/en/admin.php to lang/en/search.php and some other cleanups but things are getting better organized at least =)

              Kind regards,
              Daniel

              Show
              danielneis Daniel Neis added a comment - Hello, here is an updated version of global search. Engines (solr and elasticsearch) are now plugins and there is a new class core_search that is used by the block and the renderers. Also, all configs are now under "Plugins - Global Search" sothere is no more top level Global Search. Still have to move some strings from lang/en/admin.php to lang/en/search.php and some other cleanups but things are getting better organized at least =) Kind regards, Daniel
              Hide
              dmonllao David Monllaó added a comment -

              Thanks for working on this Daniel, we will try to review it properly as soon as possible.

              Show
              dmonllao David Monllaó added a comment - Thanks for working on this Daniel, we will try to review it properly as soon as possible.
              Hide
              jumb0eadg A B added a comment - - edited

              Thanks for the work Daniel. You mentioned https://github.com/danielneis/moodle/tree/MDL-31989_master,
              but MDL-31989_master branch has been deleted. Can you please verify. Does the branch MDL-31989 have the same code?
              I installed using MDL-31989, but when using solr, when I click on the statistics link, I get the following error:

              Show
              jumb0eadg A B added a comment - - edited Thanks for the work Daniel. You mentioned https://github.com/danielneis/moodle/tree/MDL-31989_master , but MDL-31989 _master branch has been deleted. Can you please verify. Does the branch MDL-31989 have the same code? I installed using MDL-31989 , but when using solr, when I click on the statistics link, I get the following error:
              Hide
              danielneis Daniel Neis added a comment -

              Hello, A B

              you are usig the right branch that is MDL-31989.
              I've refactored the code but have not tested that much with the Solr engine.
              Just yesterday someone made a pull request adding a missing } to the solr/classes/engine.php .
              Feel free to fix the code and submit a pull request.

              Kind regards,
              Daniel

              Show
              danielneis Daniel Neis added a comment - Hello, A B you are usig the right branch that is MDL-31989 . I've refactored the code but have not tested that much with the Solr engine. Just yesterday someone made a pull request adding a missing } to the solr/classes/engine.php . Feel free to fix the code and submit a pull request. Kind regards, Daniel
              Hide
              jumb0eadg A B added a comment -

              Thanks for the info. Actually I am the one who created the pull request adding a missing }. I am trying to make it work all I can.
              Thanks again.

              Show
              jumb0eadg A B added a comment - Thanks for the info. Actually I am the one who created the pull request adding a missing }. I am trying to make it work all I can. Thanks again.

                Dates

                • Created:
                  Updated: