Details

    • Rank:
      12

      Description

      Global Search is currently removed from Moodle 2.2.

      As part of GSoC '13 project, I've re-written Tomasz Muras's Global Search implementation for the current 2.6 (dev) master branch.

      You may view the demo and play with it here: http://global-search.jmuras.com/
      ( Username:search_admin Password: Search_admin1 )

      The docs are here: http://docs.moodle.org/dev/Global_search

        Activity

        Hide
        Aparup Banerjee added a comment -

        Thanks for working on this Tomasz

        It might be useful to put up a git repository link here.
        (also because i'm wondering how its going )

        Show
        Aparup Banerjee added a comment - Thanks for working on this Tomasz It might be useful to put up a git repository link here. (also because i'm wondering how its going )
        Hide
        Tomasz Muras added a comment -

        I've added link to my git repo to the description.
        As I've promised in the forums, I am starting with the performance testing. I'm going to test:

        • performance of the indexing
        • performance of the search (split into Lucene performance and perfomance of security checking - that is, filtering down the search results depending on the permissions)
        • memory footprint of the above

        Although it's far from completion, I have now enough code to do that. I will post my finindings in the dev forum.

        Show
        Tomasz Muras added a comment - I've added link to my git repo to the description. As I've promised in the forums, I am starting with the performance testing. I'm going to test: performance of the indexing performance of the search (split into Lucene performance and perfomance of security checking - that is, filtering down the search results depending on the permissions) memory footprint of the above Although it's far from completion, I have now enough code to do that. I will post my finindings in the dev forum.
        Hide
        Aparup Banerjee added a comment -

        this isn't minor! --> major!

        Show
        Aparup Banerjee added a comment - this isn't minor! --> major!
        Hide
        Tomasz Muras added a comment -

        Checked indexing performance - moving on the the performance of the search. See http://moodle.org/mod/forum/discuss.php?d=199636 for discussion.

        Show
        Tomasz Muras added a comment - Checked indexing performance - moving on the the performance of the search. See http://moodle.org/mod/forum/discuss.php?d=199636 for discussion.
        Hide
        Tomasz Muras added a comment -

        Update on http://moodle.org/mod/forum/discuss.php?d=199636 on index optimization.

        Show
        Tomasz Muras added a comment - Update on http://moodle.org/mod/forum/discuss.php?d=199636 on index optimization.
        Hide
        Nadav Kavalerchik added a comment -

        Hi Tomasz
        Is this ready for some testing? what is the timeline and when it is scheduled for upstream integration?

        Show
        Nadav Kavalerchik added a comment - Hi Tomasz Is this ready for some testing? what is the timeline and when it is scheduled for upstream integration?
        Hide
        Tomasz Muras added a comment -

        Hi Nadav,

        Global Search 2 rewrite will not be ready before Moodle 2.4. As soon as I get some time, I'll continue my work on testing the search performance. I will be updating forum discussion with my findings.

        Show
        Tomasz Muras added a comment - Hi Nadav, Global Search 2 rewrite will not be ready before Moodle 2.4. As soon as I get some time, I'll continue my work on testing the search performance. I will be updating forum discussion with my findings.
        Hide
        Nadav Kavalerchik added a comment -

        Hi Tomasz,
        Thank you for the clarification

        btw, I have already git-clone it from you and downloaded the latest Zend/Search library files
        I got the search_reset_index() to work, I guess. but not the search.php
        If you have any tips or suggestions... that would be nice

        Show
        Nadav Kavalerchik added a comment - Hi Tomasz, Thank you for the clarification btw, I have already git-clone it from you and downloaded the latest Zend/Search library files I got the search_reset_index() to work, I guess. but not the search.php If you have any tips or suggestions... that would be nice
        Hide
        Tomasz Muras added a comment -

        The development is not finished there yet. While search should work already, the biggest work now (implementation-wise) is to do access control.
        Would you be interested in helping with the development/coding?

        Show
        Tomasz Muras added a comment - The development is not finished there yet. While search should work already, the biggest work now (implementation-wise) is to do access control. Would you be interested in helping with the development/coding?
        Hide
        Nadav Kavalerchik added a comment -

        Like, you. I am very much busy with allot of work, which i already committed to.
        so I can not promises to help the development. but please feel free to ask anything, specific, when it arises.
        I can definitely help testing. ( I have set "watching" on the git repository )

        Show
        Nadav Kavalerchik added a comment - Like, you. I am very much busy with allot of work, which i already committed to. so I can not promises to help the development. but please feel free to ask anything, specific, when it arises. I can definitely help testing. ( I have set "watching" on the git repository )
        Hide
        Martin Dougiamas added a comment -

        I'm keeping an eye on this. Let us know where/how we can help.

        Show
        Martin Dougiamas added a comment - I'm keeping an eye on this. Let us know where/how we can help.
        Hide
        Tomasz Muras added a comment -

        Just to update you: I'm very slow at the moment because of the lack of free time.
        My plan is to do some slow progress between now and end of September and then do most of the work until the end of October. That would leave November for the reviews and integration - so it can be included in Moodle 2.4 in December. I'd sill mark it as experimental feature and keep working on it in later releases.

        Show
        Tomasz Muras added a comment - Just to update you: I'm very slow at the moment because of the lack of free time. My plan is to do some slow progress between now and end of September and then do most of the work until the end of October. That would leave November for the reviews and integration - so it can be included in Moodle 2.4 in December. I'd sill mark it as experimental feature and keep working on it in later releases.
        Hide
        Martin Dougiamas added a comment -

        Thanks, Tomasz!

        Code freeze for 2.4 is actually mid-October, and it really should already have had some HQ review by then too. it would be great if it gets that attention before then but otherwise it would have to wait for 2.5.

        Show
        Martin Dougiamas added a comment - Thanks, Tomasz! Code freeze for 2.4 is actually mid-October, and it really should already have had some HQ review by then too. it would be great if it gets that attention before then but otherwise it would have to wait for 2.5.
        Hide
        Tomasz Muras added a comment -

        I will keep the ball rolling: http://moodle.org/mod/forum/discuss.php?d=208958 .
        I think it would be better to have even simper version available for 2.4 than to wait again until 2.5. I should get it done mid-October and I'll make sure that the implementation is not a surprise for any core dev.

        Show
        Tomasz Muras added a comment - I will keep the ball rolling: http://moodle.org/mod/forum/discuss.php?d=208958 . I think it would be better to have even simper version available for 2.4 than to wait again until 2.5. I should get it done mid-October and I'll make sure that the implementation is not a surprise for any core dev.
        Hide
        Tomasz Muras added a comment -

        Hi All,

        It's been another weekend when I planned to do some work on Global Search but couldn't. I hate to say it but the truth is that I don't have enough time to work on it and will not be able to deliver!
        At the very least I will dump all I've learnt so far onto the wiki page, to make it easier for another developer to start where I've stopped.

        Sorry about that, I've overestimated the resources I had at my disposal . If nobody picks it up, then it's very likely I will complete the coding at some point but absolutely don't count on it.

        cheers,
        Tomek

        Show
        Tomasz Muras added a comment - Hi All, It's been another weekend when I planned to do some work on Global Search but couldn't. I hate to say it but the truth is that I don't have enough time to work on it and will not be able to deliver! At the very least I will dump all I've learnt so far onto the wiki page, to make it easier for another developer to start where I've stopped. Sorry about that, I've overestimated the resources I had at my disposal . If nobody picks it up, then it's very likely I will complete the coding at some point but absolutely don't count on it. cheers, Tomek
        Hide
        Luis de Vasconcelos added a comment -

        Will this Global Search function be part of 2.4?

        Show
        Luis de Vasconcelos added a comment - Will this Global Search function be part of 2.4?
        Hide
        Tomasz Muras added a comment -

        No, it will not.

        Show
        Tomasz Muras added a comment - No, it will not.
        Hide
        Luis de Vasconcelos added a comment -

        wrensoft.com has a search engine that looks interesting. We plan to test it and see if it is a viable solution.

        How to create a search engine for your Moodle site
        http://www.wrensoft.com/zoom/support/tutorial_moodle.html

        How to index sites requiring authentication with Zoom
        http://www.wrensoft.com/zoom/support/auth.html

        Show
        Luis de Vasconcelos added a comment - wrensoft.com has a search engine that looks interesting. We plan to test it and see if it is a viable solution. How to create a search engine for your Moodle site http://www.wrensoft.com/zoom/support/tutorial_moodle.html How to index sites requiring authentication with Zoom http://www.wrensoft.com/zoom/support/auth.html
        Hide
        Helen Foster added a comment -

        Hi Tomasz,

        I hope you don't mind me adding an infobox to http://docs.moodle.org/dev/Global_Search so people can easily see details of the project. I also removed the 2.3 template. Please amend as necessary.

        Show
        Helen Foster added a comment - Hi Tomasz, I hope you don't mind me adding an infobox to http://docs.moodle.org/dev/Global_Search so people can easily see details of the project. I also removed the 2.3 template. Please amend as necessary.
        Hide
        Alistair Spark added a comment -

        We have just finalised the implementation of the wrensoft search tool which works very well for us as a small institution with a limited number of long courses.

        The Version 7 alpha release is required - http://www.wrensoft.com/zoom/v7.html

        We have created student accounts for each course which index the appropriate Moodle courses which students will have access to.

        I wouldn't want to implement this on a site with 1500 courses. We essentially have 4 main courses which run for 3-5 years and have massive amounts of contents. For this, this software package is perfect.

        Show
        Alistair Spark added a comment - We have just finalised the implementation of the wrensoft search tool which works very well for us as a small institution with a limited number of long courses. The Version 7 alpha release is required - http://www.wrensoft.com/zoom/v7.html We have created student accounts for each course which index the appropriate Moodle courses which students will have access to. I wouldn't want to implement this on a site with 1500 courses. We essentially have 4 main courses which run for 3-5 years and have massive amounts of contents. For this, this software package is perfect.
        Hide
        Luis de Vasconcelos added a comment -

        Alistair, why do you say you "wouldn't want to implement this on a site with 1500 courses"? Is that Zoom Search product not a good one for large Moodle installations? If not, can you describe why not? Thanks.

        Show
        Luis de Vasconcelos added a comment - Alistair, why do you say you "wouldn't want to implement this on a site with 1500 courses"? Is that Zoom Search product not a good one for large Moodle installations? If not, can you describe why not? Thanks.
        Hide
        Alistair Spark added a comment -

        Hi Luis,

        The Zoom Search product is a great one and it can cope with 100 000s of files without any issues, their tech support is very good and ultra-responsive (free limited time trial, bug fix within 24 hours of emailing,...) - we were impressed by the cost&service.

        However, from an architectural standpoint we have gone through the route of creating student-like read-only accounts so that we have a search page for each student group (class/academic year combinations), those accounts have access to the exact same courses that the student group would have.

        Once you have created one working template for your site, creating more is just a question of changing the username&password combinations, scheduling the indexing and creating a new search box in the course (html block).

        Each of these search engines are then located in a different folder (eg http://domain.com/course1year1/search.php http://domain.com/course2year1/search.php) - this is automated by the FTP upload settings.

        However, we only needed to configure about 30 profiles in total. (ie Course 1 Year 1, Course 1 Year 2, Course 2 Year 1,....). If we had to have 1500 manual accounts in Moodle just for search, that would become a massive job to create accounts, configure all of the profiles, it becomes kind of a security risk, and maintaining this in the long run would be hard to keep track of.

        You could simply have a setup which would crawl your whole Moodle site and give you search results BUT if you have that many courses, it is more than likely that some will cover the same subject at various points, causing major user annoyance when they get the error message telling them they don't have permissions to view these results.

        Show
        Alistair Spark added a comment - Hi Luis, The Zoom Search product is a great one and it can cope with 100 000s of files without any issues, their tech support is very good and ultra-responsive (free limited time trial, bug fix within 24 hours of emailing,...) - we were impressed by the cost&service. However, from an architectural standpoint we have gone through the route of creating student-like read-only accounts so that we have a search page for each student group (class/academic year combinations), those accounts have access to the exact same courses that the student group would have. Once you have created one working template for your site, creating more is just a question of changing the username&password combinations, scheduling the indexing and creating a new search box in the course (html block). Each of these search engines are then located in a different folder (eg http://domain.com/course1year1/search.php http://domain.com/course2year1/search.php ) - this is automated by the FTP upload settings. However, we only needed to configure about 30 profiles in total. (ie Course 1 Year 1, Course 1 Year 2, Course 2 Year 1,....). If we had to have 1500 manual accounts in Moodle just for search, that would become a massive job to create accounts, configure all of the profiles, it becomes kind of a security risk, and maintaining this in the long run would be hard to keep track of. You could simply have a setup which would crawl your whole Moodle site and give you search results BUT if you have that many courses, it is more than likely that some will cover the same subject at various points, causing major user annoyance when they get the error message telling them they don't have permissions to view these results.
        Hide
        Martin Dougiamas added a comment -

        In the hackfest last week we came up with a pretty good design for completing this project.

        If this was a GSOC project, would you be available to mentor it Tomasz? I guess it would not be a lot of work for you (certainly much less than writing code!)

        Show
        Martin Dougiamas added a comment - In the hackfest last week we came up with a pretty good design for completing this project. If this was a GSOC project, would you be available to mentor it Tomasz? I guess it would not be a lot of work for you (certainly much less than writing code!)
        Hide
        Luis de Vasconcelos added a comment -

        Thanks Martin. Is there any documentation for that design?

        Show
        Luis de Vasconcelos added a comment - Thanks Martin. Is there any documentation for that design?
        Hide
        Nikunj Thakkar added a comment -

        Hello people,

        I am also interested in this project.One of our professor did a research for a similar kind of application and I got their research paper and currently I am working on prototype design for the project.I am first time applying for GSoC this year.So I have very little idea about approaching the project.But I am very much interested in this project.I will be updating the status of the application here.Any suggestions are appreciated.

        Show
        Nikunj Thakkar added a comment - Hello people, I am also interested in this project.One of our professor did a research for a similar kind of application and I got their research paper and currently I am working on prototype design for the project.I am first time applying for GSoC this year.So I have very little idea about approaching the project.But I am very much interested in this project.I will be updating the status of the application here.Any suggestions are appreciated.
        Hide
        Luis de Vasconcelos added a comment -

        Martin, is that design that you created at the hackfest the same design described on: http://docs.moodle.org/dev/Global_search_%28GSoC2013%29

        Are you happy that that solution will only work with Moodle sites running on Apache? And other web servers, including IIS, are not supported?

        Show
        Luis de Vasconcelos added a comment - Martin, is that design that you created at the hackfest the same design described on: http://docs.moodle.org/dev/Global_search_%28GSoC2013%29 Are you happy that that solution will only work with Moodle sites running on Apache? And other web servers, including IIS, are not supported?
        Hide
        Tomasz Muras added a comment - - edited

        @Luis
        Is that because atm there is no DLL for solr extension? http://www.php.net/manual/en/solr.installation.php

        Show
        Tomasz Muras added a comment - - edited @Luis Is that because atm there is no DLL for solr extension? http://www.php.net/manual/en/solr.installation.php
        Hide
        Prateek Sachan added a comment -

        Here's a quick update of Moodle's Global Search feature. Currently, many of the things have been implemented and I would like to ask developers to try it and test it for any security leaks.

        The complete code is on Github here. The Global Search discussion is here.

        All the information on is given in the Global Search wiki:

        Steps on how to install and set-up Global Search prototype is given here: http://docs.moodle.org/dev/Global_search#Prototype:_version_1.0
        List of search features currently implemented: http://docs.moodle.org/dev/Global_search#Introduction
        Searchable content through Global Search: http://docs.moodle.org/dev/Global_search#Searchable_content_in_Global_Search

        I've also included a section Tests where there are many cases to test. Developers may test these cases or add their own. You may update the wiki with your own test cases and results of the test, which would help me in making Global Search better and prone to more security leaks.

        If you have any problem in setting-up Global Search, feel free to post here Testing Moodle's Global Search prototype or you may directly message me.

        Thanks.

        Show
        Prateek Sachan added a comment - Here's a quick update of Moodle's Global Search feature. Currently, many of the things have been implemented and I would like to ask developers to try it and test it for any security leaks. The complete code is on Github here . The Global Search discussion is here . All the information on is given in the Global Search wiki : Steps on how to install and set-up Global Search prototype is given here: http://docs.moodle.org/dev/Global_search#Prototype:_version_1.0 List of search features currently implemented: http://docs.moodle.org/dev/Global_search#Introduction Searchable content through Global Search: http://docs.moodle.org/dev/Global_search#Searchable_content_in_Global_Search I've also included a section Tests where there are many cases to test. Developers may test these cases or add their own. You may update the wiki with your own test cases and results of the test, which would help me in making Global Search better and prone to more security leaks. If you have any problem in setting-up Global Search, feel free to post here Testing Moodle's Global Search prototype or you may directly message me . Thanks.
        Show
        Luis de Vasconcelos added a comment - Tomasz, Will these compiled SOLR extensions not work on Windows? https://github.com/ecaron/php_solr.dll or http://downloads.php.net/pierre/php_solr-5.3-nts-svn20091122-vc9-x86.zip http://downloads.php.net/pierre/php_solr-5.3-svn20091122-vc9-x86.zip
        Hide
        Nadav Kavalerchik added a comment -

        Installing php-solr-pecl extention on Debian/Ubuntu
        http://www.wijiti.com/help/37-jsolr/140-installing-apache-solr-php-extension

        Show
        Nadav Kavalerchik added a comment - Installing php-solr-pecl extention on Debian/Ubuntu http://www.wijiti.com/help/37-jsolr/140-installing-apache-solr-php-extension
        Hide
        Tomasz Muras added a comment -

        Luis de Vasconcelos

        Luis,

        I'm using Debian and Ubuntu for the deployments, and this is what I'm testing. Tested instructions how to set it up there are on http://docs.moodle.org/dev/Global_search_%28GSoC2013%29 - I would recommend you to stick to LAMP environment. If you want to test it on Windows, then simply check those DLLs and let us know if they work for you.

        Tomek

        Show
        Tomasz Muras added a comment - Luis de Vasconcelos Luis, I'm using Debian and Ubuntu for the deployments, and this is what I'm testing. Tested instructions how to set it up there are on http://docs.moodle.org/dev/Global_search_%28GSoC2013%29 - I would recommend you to stick to LAMP environment. If you want to test it on Windows, then simply check those DLLs and let us know if they work for you. Tomek
        Hide
        Prateek Sachan added a comment - - edited

        Hi,
        Dan Poltawski, Eloy Lafuente (stronk7), Sam Hemelryk

        I wanted to add you all to this issue to help integrate Global Search into Moodle core.
        Aparup Banerjee advised me to add you.

        This is the original branch built on MOODLE_25 as base: https://github.com/prateeksachan/moodle/tree/gs2
        This is the rebased branch, (rebased off current master): https://github.com/prateeksachan/moodle/tree/gs2rebased

        You may find Global Search docs here: http://docs.moodle.org/dev/Global_search

        Thanks.

        Show
        Prateek Sachan added a comment - - edited Hi, Dan Poltawski , Eloy Lafuente (stronk7) , Sam Hemelryk I wanted to add you all to this issue to help integrate Global Search into Moodle core. Aparup Banerjee advised me to add you. This is the original branch built on MOODLE_25 as base: https://github.com/prateeksachan/moodle/tree/gs2 This is the rebased branch, (rebased off current master): https://github.com/prateeksachan/moodle/tree/gs2rebased You may find Global Search docs here: http://docs.moodle.org/dev/Global_search Thanks.
        Hide
        Tim Hunt added a comment -

        What about mod/quiz?

        How does this work? What do I need to do to enable Global search in a new module? I can't see anything on http://docs.moodle.org/dev/Global_search.

        Show
        Tim Hunt added a comment - What about mod/quiz? How does this work? What do I need to do to enable Global search in a new module? I can't see anything on http://docs.moodle.org/dev/Global_search .
        Hide
        Prateek Sachan added a comment - - edited

        Hi Tim.

        Thanks for the quick feedback. Actually the quiz module isn't covered under Global Search yet. I'll be writing the API for the remaining mods shortly. They have plenty of security issues and I restricted myself mainly to Resources and some smaller modules. These are the mods/resources for which I've written the GS APIs currently : http://docs.moodle.org/dev/Global_search#Searchable_content_in_Global_Search

        Edit: Every mod/resource has a different set of APIs to handle the content being indexed and its security which varies across different mods/resources.

        I've updated the Global Search docs. You may refer to http://docs.moodle.org/dev/Global_search_(GSoC2013)#Quick_setup_for_testing
        One needs to first download and install the pecl-php-solr extension, clone gs2rebased branch and set-up solr as mentioned in the docs.

        I've also included a readme file for the same purpose. Also, Tomasz installed installed GS on his own server (http://global-search.jmuras.com/) a few weeks back. (Currently, it hasn't been updated but you may view it. It works perfectly).

        Thanks.

        Show
        Prateek Sachan added a comment - - edited Hi Tim. Thanks for the quick feedback. Actually the quiz module isn't covered under Global Search yet. I'll be writing the API for the remaining mods shortly. They have plenty of security issues and I restricted myself mainly to Resources and some smaller modules. These are the mods/resources for which I've written the GS APIs currently : http://docs.moodle.org/dev/Global_search#Searchable_content_in_Global_Search Edit: Every mod/resource has a different set of APIs to handle the content being indexed and its security which varies across different mods/resources. I've updated the Global Search docs. You may refer to http://docs.moodle.org/dev/Global_search_(GSoC2013)#Quick_setup_for_testing One needs to first download and install the pecl-php-solr extension, clone gs2rebased branch and set-up solr as mentioned in the docs. I've also included a readme file for the same purpose. Also, Tomasz installed installed GS on his own server ( http://global-search.jmuras.com/ ) a few weeks back. (Currently, it hasn't been updated but you may view it. It works perfectly). Thanks.
        Hide
        Mark Drechsler added a comment -

        Noting that this doesn't have a fix version - anyone know when this is targeted for implementation?

        Show
        Mark Drechsler added a comment - Noting that this doesn't have a fix version - anyone know when this is targeted for implementation?
        Hide
        Prateek Sachan added a comment - - edited

        Hi Mark,

        The original branch was made on Moodle 2.5. I've now rebased it off the current master dev.

        My plan was to make it a part of core (2.6+) which would be released in November. I hope the feature gets integrated in the coming weeks. It is open for testing right now. You may see the forum entry: Moodle's Global Search

        Show
        Prateek Sachan added a comment - - edited Hi Mark, The original branch was made on Moodle 2.5. I've now rebased it off the current master dev. My plan was to make it a part of core (2.6+) which would be released in November. I hope the feature gets integrated in the coming weeks. It is open for testing right now. You may see the forum entry: Moodle's Global Search
        Hide
        Martin Dougiamas added a comment -

        Hi Prateek,

        I don't want you to get your hopes up too much .... code freeze for 2.6 is two weeks or so away and there is a huge pile of work for us to deal with. I really doubt that there will be time for Global Search to get the review it needs. However, assuming the code is good then we should be able to land it early in the cycle for 2.7, probably January.

        Thanks for your work and I'm really keen to see it working!

        Cheers,
        Martin

        Show
        Martin Dougiamas added a comment - Hi Prateek, I don't want you to get your hopes up too much .... code freeze for 2.6 is two weeks or so away and there is a huge pile of work for us to deal with. I really doubt that there will be time for Global Search to get the review it needs. However, assuming the code is good then we should be able to land it early in the cycle for 2.7, probably January. Thanks for your work and I'm really keen to see it working! Cheers, Martin
        Hide
        Prateek Sachan added a comment -

        Sure..maybe 2.7 next year then.

        Thanks.

        Show
        Prateek Sachan added a comment - Sure..maybe 2.7 next year then. Thanks.
        Hide
        Tomasz Muras added a comment -

        OK, So it looks to me, that the best way to go, is to wait for 2.6 release and new master for 2.7. As soon as we have a new branch, we should seize the opportunity and send the code for integration.

        Show
        Tomasz Muras added a comment - OK, So it looks to me, that the best way to go, is to wait for 2.6 release and new master for 2.7. As soon as we have a new branch, we should seize the opportunity and send the code for integration.
        Hide
        Prateek Sachan added a comment - - edited

        Yes. I hope it gets the review required by the next stable moodle (2.7) release. In the remaining months of this year, I'll try and write GS APIs for the remaining modules and then asking moodle integrators to review it.

        Thanks.

        Show
        Prateek Sachan added a comment - - edited Yes. I hope it gets the review required by the next stable moodle (2.7) release. In the remaining months of this year, I'll try and write GS APIs for the remaining modules and then asking moodle integrators to review it. Thanks.
        Hide
        Martin Dougiamas added a comment - - edited

        The first reviewers at HQ will be the Moodle HQ BACKEND team.

        Of course anyone from the community is welcome to pitch in with reviews and comments here anytime.

        Show
        Martin Dougiamas added a comment - - edited The first reviewers at HQ will be the Moodle HQ BACKEND team. Of course anyone from the community is welcome to pitch in with reviews and comments here anytime.
        Hide
        Luis de Vasconcelos added a comment -

        This new Global Search requires Apache Solr, so what will happen to Moodle sites that run under IIS on Windows? Will Global Search not work under IIS?

        Show
        Luis de Vasconcelos added a comment - This new Global Search requires Apache Solr, so what will happen to Moodle sites that run under IIS on Windows? Will Global Search not work under IIS?
        Hide
        Prateek Sachan added a comment -

        Hi Luis,

        As pointed out earlier, GS uses php solr extension. As long as you have php solr extension installed on your server, GS will work. It's just another php extension, so it won't affect Moodle sites. (I've just tested out GS on LAMP). You may find this link useful which underlines all the dependencies and their installation of php solr extension.

        Show
        Prateek Sachan added a comment - Hi Luis, As pointed out earlier, GS uses php solr extension. As long as you have php solr extension installed on your server, GS will work. It's just another php extension, so it won't affect Moodle sites. (I've just tested out GS on LAMP). You may find this link useful which underlines all the dependencies and their installation of php solr extension.
        Hide
        Itamar Tzadok added a comment -

        Prateek, I haven't had the chance to examine thoroughly so off hand do you have any idea how easy or otherwise it would be to make this work with Solr php client? For windows installations it may not be straightforward to add a php_solr extension but the client may be a viable option.

        Show
        Itamar Tzadok added a comment - Prateek, I haven't had the chance to examine thoroughly so off hand do you have any idea how easy or otherwise it would be to make this work with Solr php client? For windows installations it may not be straightforward to add a php_solr extension but the client may be a viable option.
        Hide
        Prateek Sachan added a comment - - edited

        Hi Itamar Tzadok,

        I researched a bit for choosing the client for interacting with solr. Making it work using the Solr Php client would be easy and do-able, though may require many modifications in the current implementation. I concentrated towards using the php_solr extension.

        There are many php_solr dlls available for Windows. I haven't checked them though. I used LAMP for GS implementation.

        https://github.com/ecaron/php_solr.dll
        http://downloads.php.net/pierre/php_solr-5.3-nts-svn20091122-vc9-x86.zip
        http://downloads.php.net/pierre/php_solr-5.3-svn20091122-vc9-x86.zip

        If you want, you may test it on Windows using the dlls, and notify me. It would be great.

        Edit: I installed php_solr extension using the github dll given above on Windows. The extension is working.

        Thanks.

        Show
        Prateek Sachan added a comment - - edited Hi Itamar Tzadok , I researched a bit for choosing the client for interacting with solr. Making it work using the Solr Php client would be easy and do-able, though may require many modifications in the current implementation. I concentrated towards using the php_solr extension. There are many php_solr dlls available for Windows. I haven't checked them though. I used LAMP for GS implementation. https://github.com/ecaron/php_solr.dll http://downloads.php.net/pierre/php_solr-5.3-nts-svn20091122-vc9-x86.zip http://downloads.php.net/pierre/php_solr-5.3-svn20091122-vc9-x86.zip If you want, you may test it on Windows using the dlls, and notify me. It would be great. Edit: I installed php_solr extension using the github dll given above on Windows. The extension is working. Thanks.
        Hide
        Juerg Hoerner added a comment -

        in your testmoodle I tried to find words in the attached pdf in the glossary. As example: "Introduction" . Maybe that this does not work yet.

        can I delete the index and reindexing all content, like in the old module?

        Thanks for your very important Global search, one of the most important module.

        Show
        Juerg Hoerner added a comment - in your testmoodle I tried to find words in the attached pdf in the glossary. As example: "Introduction" . Maybe that this does not work yet. can I delete the index and reindexing all content, like in the old module? Thanks for your very important Global search, one of the most important module.
        Hide
        Prateek Sachan added a comment -

        Hi Juerg Hoerner,

        Yes. You may delete your index and re-index content from cron. All specifications are giving in Global Search docs.

        Thanks.

        Show
        Prateek Sachan added a comment - Hi Juerg Hoerner , Yes. You may delete your index and re-index content from cron. All specifications are giving in Global Search docs. Thanks.
        Hide
        Mark Andrews added a comment -

        Hi Prateek,
        I've just had a play with this and it's great! - really pleased to hear it might make 2.7

        couple of questions/comments from testing though;

        • As it's a global search will it also be searching; people & courses? would help to make it truly global search. - Perhaps as an option that admin's could enable/disable
        • Also I wonder if there needs to be some tweaking done to the 'filter query' area as fields like 'From records having this author' could mean different things to different users; a participant looking for a particular forum post by a class mate to the up loader of a resource.

        Cheers
        Mark

        Show
        Mark Andrews added a comment - Hi Prateek, I've just had a play with this and it's great! - really pleased to hear it might make 2.7 couple of questions/comments from testing though; As it's a global search will it also be searching; people & courses? would help to make it truly global search. - Perhaps as an option that admin's could enable/disable Also I wonder if there needs to be some tweaking done to the 'filter query' area as fields like 'From records having this author' could mean different things to different users; a participant looking for a particular forum post by a class mate to the up loader of a resource. Cheers Mark
        Hide
        Prateek Sachan added a comment -

        Hi Mark,
        Thanks for the reviews.

        As it was the first version of GS, hence I restricted myself to only mods/resources. I still have some mods for which I would be writing GS APIs in next months. Only then, I could think about including users/courses.

        "From records having this author" means "all records that have an author as x". These are mainly uploaded PDFs/PPTs/etc. that have an "author" field in their metadata. I guess I should change the string a bit to suit the users. Thanks for offering your thoughts on this one.

        Show
        Prateek Sachan added a comment - Hi Mark, Thanks for the reviews. As it was the first version of GS, hence I restricted myself to only mods/resources. I still have some mods for which I would be writing GS APIs in next months. Only then, I could think about including users/courses. "From records having this author" means "all records that have an author as x". These are mainly uploaded PDFs/PPTs/etc. that have an "author" field in their metadata. I guess I should change the string a bit to suit the users. Thanks for offering your thoughts on this one.
        Hide
        Martin Dougiamas added a comment -

        This is now in the backlog of the BACKEND team to review.

        Show
        Martin Dougiamas added a comment - This is now in the backlog of the BACKEND team to review.
        Hide
        Tim Drescher added a comment -

        We would really love a global search. It would greatly assist our staff/students in finding the course or information they need.

        Show
        Tim Drescher added a comment - We would really love a global search. It would greatly assist our staff/students in finding the course or information they need.
        Hide
        Aparup Banerjee added a comment - - edited

        Please note there could be other plugins utilizing solr directly
        (such as https://moodle.org/plugins/view.php?plugin=tool_coursesearch)
        so steps could be made wrt allowing migration of those to use the global search api to access the installed search engine (the api in moodle core after this patch).

        or to just keep those separate (ie: self managed instance(s) of solr), but this would lead to more inefficient usage of solr.

        ps: it would be great if someone could lend even a glancing eye to this for a review..

        Show
        Aparup Banerjee added a comment - - edited Please note there could be other plugins utilizing solr directly (such as https://moodle.org/plugins/view.php?plugin=tool_coursesearch ) so steps could be made wrt allowing migration of those to use the global search api to access the installed search engine (the api in moodle core after this patch). or to just keep those separate (ie: self managed instance(s) of solr), but this would lead to more inefficient usage of solr. ps: it would be great if someone could lend even a glancing eye to this for a review..
        Hide
        Prateek Sachan added a comment -

        The Global Search implementation differs from that of the coursesearch plugin as they are using completely different Solr php-client-libraries for interacting with the Solr server through Moodle. Therefore, the API in moodle core after this patch wouldn't affect course search.

        Show
        Prateek Sachan added a comment - The Global Search implementation differs from that of the coursesearch plugin as they are using completely different Solr php-client-libraries for interacting with the Solr server through Moodle. Therefore, the API in moodle core after this patch wouldn't affect course search.
        Hide
        Luis de Vasconcelos added a comment - - edited

        Adam Morris and antriver have developed the moodle-block_search block. See https://github.com/antriver/moodle-block_search and https://moodle.org/mod/forum/discuss.php?d=239646.

        How does that search block compare with this Global Search 2 feature?

        Show
        Luis de Vasconcelos added a comment - - edited Adam Morris and antriver have developed the moodle-block_search block. See https://github.com/antriver/moodle-block_search and https://moodle.org/mod/forum/discuss.php?d=239646 . How does that search block compare with this Global Search 2 feature?
        Hide
        Amir Elion added a comment - - edited

        Luis, I dd some initial testing with the suggested block.
        it does look like a good start.
        It searches course and activity names and description, and can be configured to choose which ones to search.
        It also seems to handle permissions (with ability to show grayed out/not show users inaccessible search results), wildcards, "quoted" search terms, exclude words.
        Also classifies and allows quick navigation by types of search results and

        It lacks search inside activity content - e.g. page content, glossary terms, wiki pages, forum posts, etc.
        If the above can be added, I think it would be a good candidate for this.

        Show
        Amir Elion added a comment - - edited Luis, I dd some initial testing with the suggested block. it does look like a good start. It searches course and activity names and description, and can be configured to choose which ones to search. It also seems to handle permissions (with ability to show grayed out/not show users inaccessible search results), wildcards, "quoted" search terms, exclude words. Also classifies and allows quick navigation by types of search results and It lacks search inside activity content - e.g. page content, glossary terms, wiki pages, forum posts, etc. If the above can be added, I think it would be a good candidate for this.
        Hide
        Tomasz Muras added a comment - - edited

        As far as I can see this search implementation does not do any relevancy search. It will simply return all rows with a given keyword, not neceserily the relevant first. This will not work well for more than 10 results.

        As this is simply implemented as a DB query with like %keyword% clause it will not scale - this can only be used on smaller sites.

        You can also compare it with solr features: http://lucene.apache.org/solr/features.html - most of them will be missing.

        Show
        Tomasz Muras added a comment - - edited As far as I can see this search implementation does not do any relevancy search. It will simply return all rows with a given keyword, not neceserily the relevant first. This will not work well for more than 10 results. As this is simply implemented as a DB query with like %keyword% clause it will not scale - this can only be used on smaller sites. You can also compare it with solr features: http://lucene.apache.org/solr/features.html - most of them will be missing.
        Hide
        Prateek Sachan added a comment -

        As Moodle sites become big, it becomes very difficult (concerning speed and optimization) to do direct DB searches. The whole concept behind "Global Search" was to make the entire content within modules and resources searchable alongwith providing complex search queries. I highly doubt if these could be implemented via direct DB searches in an optimized way.

        That's why indexing was used which is the obvious solution for making huge chunks of data searchable.

        Also, the kinds of queries that one would use for searching will put a heavy load on your database server.

        Thanks.

        Show
        Prateek Sachan added a comment - As Moodle sites become big, it becomes very difficult (concerning speed and optimization) to do direct DB searches. The whole concept behind "Global Search" was to make the entire content within modules and resources searchable alongwith providing complex search queries . I highly doubt if these could be implemented via direct DB searches in an optimized way. That's why indexing was used which is the obvious solution for making huge chunks of data searchable. Also, the kinds of queries that one would use for searching will put a heavy load on your database server. Thanks.
        Hide
        Eugene Venter added a comment - - edited

        We're currently planning on using this implementation for one of our clients. The current state of the code seems to need more work and optimisation and we've started fixing up a few things (with a lot more to come).

        For a start, we'll be pushing our contributable updates here: https://github.com/catalyst/moodle

        Show
        Eugene Venter added a comment - - edited We're currently planning on using this implementation for one of our clients. The current state of the code seems to need more work and optimisation and we've started fixing up a few things (with a lot more to come). For a start, we'll be pushing our contributable updates here: https://github.com/catalyst/moodle
        Hide
        Martin Dougiamas added a comment -

        Pushed back to BACKEND to look at in this sprint. Needs a basic review before going to integration.

        Show
        Martin Dougiamas added a comment - Pushed back to BACKEND to look at in this sprint. Needs a basic review before going to integration.
        Hide
        Jonathan Harker added a comment - - edited

        Here is my 2¢

        1. I'd say that as it is, it is not sufficiently loosely coupled for use with anything other than Solr. A rewrite would be required for pluggable backends for Elastic et al.
        2. The Solr implementation should make use of dynamic fields, which requires much less dependence on Solr schema.xml configuration. Dynamic fields are in the default example configuration, and work fine with 1.4, 3.x, and 4.x
        3. The code needs a bunch of clean-up - e.g. everything in search/documents is unreachable.
        4. Pagination should be done via the Solr API rather than fetching a hundred documents and hoping for the best.
        5. We've done some work to display facets, which should be a part of this solution.

        Show
        Jonathan Harker added a comment - - edited Here is my 2¢ 1. I'd say that as it is, it is not sufficiently loosely coupled for use with anything other than Solr. A rewrite would be required for pluggable backends for Elastic et al. 2. The Solr implementation should make use of dynamic fields, which requires much less dependence on Solr schema.xml configuration. Dynamic fields are in the default example configuration, and work fine with 1.4, 3.x, and 4.x 3. The code needs a bunch of clean-up - e.g. everything in search/documents is unreachable. 4. Pagination should be done via the Solr API rather than fetching a hundred documents and hoping for the best. 5. We've done some work to display facets, which should be a part of this solution.
        Hide
        Tomasz Muras added a comment -

        1. I wouldn't worry about plug-ability too much. I think that chances for anyone to implement any other backend are minimal, let's worry when someone tries.
        2. Yeah, why not.
        3. It's most likely an old core - you're right, it should be cleaned up.
        4. It's going to be very difficult to replicate all Moodle logic (permissions check) on the solr side. I suggest first version with keeping it in Moodle, then building from there. First step could be to filter out obvious non-accessible documents and double check remaining ones with Moodle.
        5. Phase 2 maybe? Let's get at least minimal working version into Moodle core.

        Show
        Tomasz Muras added a comment - 1. I wouldn't worry about plug-ability too much. I think that chances for anyone to implement any other backend are minimal, let's worry when someone tries. 2. Yeah, why not. 3. It's most likely an old core - you're right, it should be cleaned up. 4. It's going to be very difficult to replicate all Moodle logic (permissions check) on the solr side. I suggest first version with keeping it in Moodle, then building from there. First step could be to filter out obvious non-accessible documents and double check remaining ones with Moodle. 5. Phase 2 maybe? Let's get at least minimal working version into Moodle core.
        Hide
        Jonathan Harker added a comment - - edited

        I think it should be possible to do the permissions thing. There appear to be two tricks to doing permissions right:

        1. make each module responsible at index time for ensuring that anything needed to filter on permissions is indexed in a field in the module document (e.g. course ID, hidden, visible, deleted, etc), and
        2. make each module responsible at query time for figuring out what $USER can and cannot see, and apply the right filter (using the above).

        Using dynamic fields, different document types (e.g. course, lesson, forum) can have fields in common, one of which should probably be course_id - this will at the very least enable us to restrict results to the courses $USER is enrolled in by using something like

         'course_id:(' . join(',', $enrolled_courses) . ')' 

        It may be a week or two before we get time to collapse the work we've done here into something that deserves a PULL request... there are some curly bespoke bits we'll have to remove first!

        Show
        Jonathan Harker added a comment - - edited I think it should be possible to do the permissions thing. There appear to be two tricks to doing permissions right: 1. make each module responsible at index time for ensuring that anything needed to filter on permissions is indexed in a field in the module document (e.g. course ID, hidden, visible, deleted, etc), and 2. make each module responsible at query time for figuring out what $USER can and cannot see, and apply the right filter (using the above). Using dynamic fields, different document types (e.g. course, lesson, forum) can have fields in common, one of which should probably be course_id - this will at the very least enable us to restrict results to the courses $USER is enrolled in by using something like 'course_id:(' . join(',', $enrolled_courses) . ')' It may be a week or two before we get time to collapse the work we've done here into something that deserves a PULL request... there are some curly bespoke bits we'll have to remove first!
        Hide
        Tomasz Muras added a comment -

        That makes sense. It will require a lot of testing but it would be a Holy Grail of Search if you can put that logic into solr side.

        Show
        Tomasz Muras added a comment - That makes sense. It will require a lot of testing but it would be a Holy Grail of Search if you can put that logic into solr side.
        Hide
        Prateek Sachan added a comment -

        Hi all,

        This is just an update. The official php-pecl-solr extensions for Solr 4.x[1] have been released. I'll be testing it out and updating the Docs in the coming week.

        Thanks.

        [1]: http://pecl.php.net/package/solr

        Show
        Prateek Sachan added a comment - Hi all, This is just an update. The official php-pecl-solr extensions for Solr 4.x [1] have been released. I'll be testing it out and updating the Docs in the coming week. Thanks. [1] : http://pecl.php.net/package/solr

          People

          • Votes:
            52 Vote for this issue
            Watchers:
            62 Start watching this issue

            Dates

            • Created:
              Updated: