The Lucene engine is pretty powerful, allows consistent searching across all parts of Moodle, and most of the work is done (in moodle/search), however it's going to be tricky bringing roles into the picture (but possible I think).

Shouldn't we focus on that instead? Or should we change direction and use the native database for searching?

I thought that (don't ask why) that such project was semi-dead and we could start supporting our own implementation, specially if we want to index uploaded files (perhaps this is the KEY question) and if lucene works well with non-spaced languages (like japanese).

Those (upload files + non-spaced languages) are the two major drawbacks of implementing our own indexes. Does Lucene uses its own index format (file based) or does it use DB specific indexes... uhm.

Too much questions! Let's halt this until 1.8 release and then examine a bit more about how lucene is working.... oki?

Times for some searches I tried have dropped from 11 seconds to about 1 second !

+1000 for better handling of this in 1.9.

note that current experimental implementation is far from perfect because, to mimic current behaviour and semantics of relational searching we are executing all the searches by appending the "*" wildchar to all the terms. It reduces real benefit a lot (compared with the execution without wildchars).

That's because I proposed an alternative search form + help file to be showed describing the new semantics used when the documental search is enabled. While the search with wildchars can be a good idea in sites with fewer results (more records are found), sites like moodle.org (+250.000 posts) will return enough information just searching by "exact" words, that is, in fact, the default search mode worldwide!

Anyway, I'll start documenting about the lucene-php-mysql thing after release... that seems to be the preferred way (if we don't want to apply such changes in semantics for now).

http://www.moskalyuk.com/blog/effective-mysql-fulltext-searching/1380

http://epsilondelta.net/2006/05/05/does-the-mysql-fulltext-index-scale/

http://peter-zaitsev.livejournal.com/12809.html

http://www.sphinxsearch.com/

http://jayant7k.blogspot.com/2006/06/benchmarking-results-of-mysql-lucene.html

last week I've been playing with 300.000 posts and Zend Lucene. Some numbers:

• got 1000+ indexed documents/minute.
• most searches resolved in less than 1 sec, with 1-3 more secs needed to fetch and render one page of results
• absolutely easy to customize for our needs (UTF-8, keeping numbers, versions... in the index) all this working with < 100 lines code.
• easy to keep synchronised with one intemediate "lucene_documents" relational table (where all the modules will write the actions to be performed).
• relatively easy to support groups/users/allowed forums in the documental searches.

but I've found some problems with current Zend Ludene implementation:

• relies on old lucene 1.4 code.
• index format has been dramatically improved for Lucene 2.0-2.1
• tons of memory needed (> 120MB) to fully optimize the index (and this should be executed, at least, weekly, to avoid fragmentation and reduce number of segments).
• after looking bugs and activity... not sure if it's

Uhm... just to point to another alternative... could we "reinvent the wheel" ? Drupal seems to have one interesting "hand-made" documental index for all its contents... perhaps it would be enough for our "token based searches"....

Ciao

Will Lucene be the final solution...uhm... ciao

Link: http://framework.zend.com/download

Ciao