Linux
PHP 5.1.6
MySQL 5.0.24

The main issue, apart from a few bugs, is to make it work with Roles.

I'm targeting this for 2.0 because of the PHP5 requirement.

a temporary fix is downloadable as attached resource of the thread http://moodle.org/mod/forum/discuss.php?d=74975 in forum (sorry, it's a french thread) which does still not install datamodel.

May I, if Martin allows me, terminate some primary Roles integration to have the full text available sooner than 2.0. Martin, please could you send me a notice for Role minimal handling so I can spend effort on it ?

More info : I informed Martin of a quasi availability of PDF and other bynary encoded file indexing in a few weeks, using xpdf and other content extractors. This is actually in pretty work.

Server Time: Sat, 30 Jun 2007 14:26:33 +0100

Warning: Indexing was not successfully completed last time, restarting.

Using C:\MoodleWindowsInstaller-latest-16\moodle\moodledata/search as data directory.
Database error. Please check settings/files.

Using Moodle 1.8+. Any idea what I've missed?

Well, the issue about permissions is that each piece of data indexed needs to have enough context information stored with it so that when the search results are displayed, each item can be checked to make sure that item should be displayed to the current user (the one doing the search).

Some example scenarios that need to be coped with:

• user is not a member of the course
• user can access a course but not a forum (due to overrides in place on their role)
• user can access a course but not some forum messages (group mode is set to 'separate' and they are in a different group without accessallgroups capability)
• user can access course but not some forum messages because they are "timed" posts and not available yet

Here's a bit of a plan that I think will work:

Each bit of information must have these things attached to it:

• the $context of the information (context->id, context->contextlevel, context->instanceid)
• the $capability required to read the info (eg mod/forum:viewdiscussion)
• the $path to the information (eg mod/forum, mod/resource etc)
• the $item of information (eg identify record in a table $item->type = post, $item->id = 473)
• the $courseid (usually easy to determine and will help with checks later on)

And additionally if we know it:

• the groupid of the information (but only for information that is restricted to one group)
• the userid of the person who made the information (if we know it)

So the global search should return a big long list of all items that match items first in all parts of Moodle and return for further processing before display.

Admins of course need no checks and can see everything so that should be fast.

The first check will be on the $courseid, all items that do not match get_my_courses can be eliminated immediately.

The second check for other users would be a has_capability($capability, $context) ... that will be fast because the current user has all this cached in memory (no database access required) so we'll eliminate a lot of stuff at once. If false then we can ignore this text item and leave it out from the list. If true then we keep checking.

The third check would be to use a function from $path/lib.php (if one is defined) to do a further check on that text, passing all the info we have. eg xxxxx_check_text_access() This function would return true or false and would make the final definitive call. Most modules (like resources) wouldn't need such a function, it's only useful for complex access situations like forum.

It would be terrific if you could implement this, Valery!!!!! Let me know if I can help further (eg CVS access etc)

The message you get tells something was wrong with the table. The checkDB() method wich is called (in indexlib.php) tries to drop the table and to set it back as new. (Could you noticed the table disapeared ? it should). Note that this procedure is completely hidden to output, so yoi don't notice it's done. I noticed the SQL had some little errors on DEFAULTs. Should be :

in search/db/mysql.sql :

CREATE TABLE IF NOT EXISTS `prefix_search_documents` (
`id` int(11) NOT NULL auto_increment,
`docid` int(11) NOT NULL,
`doctype` varchar(12) NOT NULL default 'none',
`title` varchar(255) NOT NULL default '',
`url` varchar(255) NOT NULL default '',
`docdate` timestamp NOT NULL default 0,
`updated` timestamp NOT NULL default CURRENT_TIMESTAMP,
`courseid` int(11) NOT NULL default 0,
`groupid` int(11) NOT NULL default 0,
PRIMARY KEY (`id`)
) ENGINE=MyISAM AUTO_INCREMENT=1;

This could explain why this table was not created properly. I don't know if these defaults hang on all MySQL versions.

I anticipate raising size of title and url fields that appear being to shorts in some experimentations.

I just wrote a long technical discussion that I've just lost for login timeout in tracker !!. Sigh !! Try to make it shorter.

I checked the possibility to add any field we need in indexing records of Lucene. Seems fine.

• course_id is available
• group_id also
• user identification is not satisfactory as it may appear as full text user name in "author" field. We may add user_id field to Lucene records.
• information identification is available through 'doctype' and 'docid' fields that let not subclass information within a module. a field for subqualifying the 'itemtype' might be added.

Note that I just patched the way of indexing block dependant information either, for those blocks which manage some amount of local information.

One key question is about the necessity to record capabilityand context information in indexing records.

The alternative scenario is :

1. We get indexing information out of Lucene (generic querying loop)
2. We check out course_id that allows discarding a lot of information
3 We use the path (you mentionned... so straight analysis !!) to perform an indirect callback to the module access policy that needs only sufficiant information on the item identity (and item type) and the current $USER (which should be the same than who is actually querying). Module should locally be responsible of checking who and how (capabilities) is accessing to what (item identification).

This responsibility could be implemented in two locations :

• in the document type plugin (search/documents/module_document.php) among with indexing functions. I noticed that component's library is called for those modules whom central functions are used.
• in the module library itself wich may be closer to the original concerns for the developper, but far away from the "document indexing" strategy.

This way of doing should allow the indexor engine to ignore all about capabilities and context, (which is simpler) and reports all this knowledge to the module (which seems being more cohesive).

A rationale to this orientation is that if we agree in the view capability sapce should be kept as flat as possible, there is no way to prevent some developpers to use complex formula or combination of capabilities to access a piece of information. In which case pushing back into the module the task of checking for accesses "from anywhere" could let the architecture more versatile.

Could you enlight me about that point ?

I don't think we should do this because this task can be database-intensive, and would have to be repeated for EVERY search result.

I think it would be better to shift this load to the indexing phase (which is infrequent and allowed to be slow) so that searches (which happen often) are as fast and light as possible.

I'm sensibilized with load issues as I run my Moodle volume on an old 450MHtz Pentium III with 324Mo memory amount with an extra little machine as database (the only machines I can afford, as I have no political support in the institution I work, and being internal non-official competitor to the inner e-elarning home made project), and 1.8+ seems being much more time consuming.

So I'll try to make the 'cleverest' 'contextual access information' caching I can find out after resolving the item identification and callback chaîn design.

Just to give you good news of search engine :

• all framework OK
• added fields to index records ok : item_type, path, and userid when relevant. Ready for context and capabilities.
• handles blocks from now

Indexing works great !!

• callback chaîn OK :
• can_display() implemented in querylib.php.
• performs user vs. course matching handling guest situation.
• room left for cached context/capability scheme checks
• performs generic module access callbacks

Callback xxxx_check_text_access() is located in the document type description in /search/documents so that old code of modules and block should not be revised.

Filtering chaîn works well. Now we should implement local policies for all standard activities, and test, test, test, test, document, document, document, document.... it all.

Telegraphic style, but a bit tired today.

Regards.

Global Search now indexes :

• forums, indexing separately thread heads (discussions) and other posts
• resources (including files)
• wiki (untested)
• glossaries (without attachements)
• databases, indexing separately data recordsets and comments
• techproject, indexing separately requirements, specification entries, tasks, milestones and deliverables
• chat : indexing separate sessions as single documents (synchronized on what /chat/report.php calculates).

update.php delete.php add.php entirely reviewed, so were all xxxxx_document for complying to the information subclassing by 'itemtype', allowing complex modules to index more than one document type.

Test in progress : updating/add/delete integrity using 'itemtype' subclassing.
Test in progress : checking efficiency of results filtering (many tests to perform).

Internationalization integration completed.

Will check what is to do for XMLDB compliance.

Looking forward to it coming out of experimental! Thank you Valery for taking this on. Great!

Kind regards,

Hans (who has long ago learned to use Opera when typing long posts as it keep the form contents and fills them in when you press the back button after the timeout! )

Only the files that are published as "file" resources are indexed, leaving all stuff within file repositories unindexed (we could not really offer to users the way to see them cause the direct URL should not be used for these files).

I should work a bit further to handle files in folders when these folders are published (and so opened) within a course content.

I should also make some add-ons to index attachements in key modules such as WiKi or forums, or even data records where the record type is an uploaded file.

Maybe could I get from Martin a nice method to put these requirements for voting.

An experimental CVS version is in the HEAD repository, ready to download and test. Many french fellows have already agreed for testing regularily all the stuff.

Fatal error: Call to a member function MoveNext() on a non-object in /home/tbshsweb/public_html/moodle/search/documents/resource_document.php on line 55

this seems being older code, you may checkout back a renewed version of the search engine from the CVS, which is not published yet in distributions, even in 1.8.2. Note that this is still an experimental version that should be extensively tested by a community of french Moodlers in final August.

any calls to MoveNext() (old direct call within an ADOdb recordset) should have disappeared, prefering the use of standard get_records(),insert_record(),delete_records() primitives.

I'm on 1.8.2+ from CVS (updated yesterday) but I can't get any movement on this. Running the indexer from

http://mymoodlesite.org/search/indexer.php?areyousure=yes

leaves me with the same error about the database. I have tried pasting your SQL above into PHPMyAdmin (altering the prefix) but it still wipes the table every time as before:

Database error. Please check settings/files.

I'm on mysql 5.0.18 win2k3

Fix report :

bad default value format in search/db/mysql.sql for docdate - > must be 0 and not '0'
url field length risen to length 255 (max varchar length).
added postgress database model
fixed bad path in checkDB()
checked old cron issue when search disabled (may block cron process, now fixed)

presetting tables from search block is not necessary any more.

PHP error log gives:

[12-Sep-2007 11:15:12] PHP Notice: Undefined property: stdClass::$search_index_size in C:\MoodleWindowsInstaller-latest-16\moodle\moodle\search\query.php on line 236
[12-Sep-2007 11:15:20] PHP Warning: require_once(Zend/Search/Lucene/Exception.php) [<a href='function.require-once'>function.require-once</a>]: failed to open stream: No such file or directory in C:\MoodleWindowsInstaller-latest-16\moodle\moodle\search\Zend\Search\Lucene.php on line 23
[12-Sep-2007 11:15:20] PHP Fatal error: require_once() [<a href='function.require'>function.require</a>]: Failed opening required 'Zend/Search/Lucene/Exception.php' (include_path='.;C:\MoodleWindowsInstaller-latest-16\moodle\php\pear\:../') in C:\MoodleWindowsInstaller-latest-16\moodle\moodle\search\Zend\Search\Lucene.php on line 23
[12-Sep-2007 11:15:25] SQL Invalid default value for 'docdate' in C:\MoodleWindowsInstaller-latest-16\moodle\moodle\lib\dmllib.php on line 233. STATEMENT: CREATE TABLE IF NOT EXISTS `mdl_search_documents` ( `id` int(11) NOT NULL auto_increment, `docid` int(11) NOT NULL, `doctype` varchar(12) NOT NULL default 'none', `title` varchar(100) NOT NULL default '', `url` varchar(100) NOT NULL default '', `docdate` timestamp NOT NULL default '0', `updated` timestamp NOT NULL default CURRENT_TIMESTAMP, `courseid` int(11) NOT NULL default '0', `groupid` int(11) NOT NULL default '0', PRIMARY KEY (`id`)) ENGINE=MyISAM AUTO_INCREMENT=1
[12-Sep-2007 11:16:08] PHP Notice: MagpieRSS [debug] Got 304 for http://www.psychologytoday.com/articles/relationships.rss in C:\MoodleWindowsInstaller-latest-16\moodle\moodle\lib\magpie\rss_fetch.inc on line 255
[12-Sep-2007 11:16:08] PHP Notice: MagpieRSS [debug] Fetch successful in C:\MoodleWindowsInstaller-latest-16\moodle\moodle\lib\magpie\rss_fetch.inc on line 255
[12-Sep-2007 11:16:47] SQL Invalid default value for 'docdate' in C:\MoodleWindowsInstaller-latest-16\moodle\moodle\lib\dmllib.php on line 233. STATEMENT: CREATE TABLE IF NOT EXISTS `mdl_search_documents` ( `id` int(11) NOT NULL auto_increment, `docid` int(11) NOT NULL, `doctype` varchar(12) NOT NULL default 'none', `title` varchar(100) NOT NULL default '', `url` varchar(100) NOT NULL default '', `docdate` timestamp NOT NULL default '0', `updated` timestamp NOT NULL default CURRENT_TIMESTAMP, `courseid` int(11) NOT NULL default '0', `groupid` int(11) NOT NULL default '0', PRIMARY KEY (`id`)) ENGINE=MyISAM AUTO_INCREMENT=1
[12-Sep-2007 11:19:37] SQL Invalid default value for 'docdate' in C:\MoodleWindowsInstaller-latest-16\moodle\moodle\lib\dmllib.php on line 233. STATEMENT: CREATE TABLE IF NOT EXISTS `mdl_search_documents` ( `id` int(11) NOT NULL auto_increment, `docid` int(11) NOT NULL, `doctype` varchar(12) NOT NULL default 'none', `title` varchar(100) NOT NULL default '', `url` varchar(100) NOT NULL default '', `docdate` timestamp NOT NULL default '0', `updated` timestamp NOT NULL default CURRENT_TIMESTAMP, `courseid` int(11) NOT NULL default '0', `groupid` int(11) NOT NULL default '0', PRIMARY KEY (`id`)) ENGINE=MyISAM AUTO_INCREMENT=1
[12-Sep-2007 11:19:40] SQL Invalid default value for 'docdate' in C:\MoodleWindowsInstaller-latest-16\moodle\moodle\lib\dmllib.php on line 233. STATEMENT: CREATE TABLE IF NOT EXISTS `mdl_search_documents` ( `id` int(11) NOT NULL auto_increment, `docid` int(11) NOT NULL, `doctype` varchar(12) NOT NULL default 'none', `title` varchar(100) NOT NULL default '', `url` varchar(100) NOT NULL default '', `docdate` timestamp NOT NULL default '0', `updated` timestamp NOT NULL default CURRENT_TIMESTAMP, `courseid` int(11) NOT NULL default '0', `groupid` int(11) NOT NULL default '0', PRIMARY KEY (`id`)) ENGINE=MyISAM AUTO_INCREMENT=1

SQL file seems not being up to date : should show :

...
`docdate` TIMESTAMP DEFAULT 0
...

you should have an extra field :
...
`itemtype` VARCHAR(32)
...
in the "search_documents" table.

fix it in your local search/db/mysql.sql file in the meanwhile.

Some pathes for include seem strange:

[12-Sep-2007 11:15:20] PHP Warning: require_once(Zend/Search/Lucene/Exception.php) [<a href='function.require-once'>function.require-once</a>]: failed to open stream: No such file or directory in C:\MoodleWindowsInstaller-latest-16\moodle\moodle\search\Zend\Search\Lucene.php on line 23

line 23 should (and all relevant close to it) show :

require_once $CFG->dirroot.'/search/Zend/Search/Lucene/Exception.php';

so that absolute and reliable path is used.

If it is not the case, the Zend framework for global search has not be correctly updated.

WindowsInstaller-latest-16 ?? What is your real Moodle version ? Search engine review has been tested only for 1.8 version and upper (I can test it on 1.7 eventually if needed).

Could you check this ?
Thanks.

I made a last check of consistence in code. Every seems being OK now and will run from a clean install of the "block_search" module. There is not any more "drop and create back" procedure for the indexing table. I put it away as we agreed with Martin and Eloy. Regenerating index table only clears it out. We can ensure nicer upgrades from now on.

cron has been protected for php 4.3.0 environements so it will not be perturbated by Lucene exceptions catching in the cron hook. The worst that can happen is the search index update procedure aborting and letting other cron tasks be properly finished.

Will this functionality be backported to 1.8 you think?

Kind regards (and thank you for all your efforts...!),

Hans de Zwart

I'm available anyway for help, use likely moodle.org messaging.
Thank for support.

[17-Sep-2007 09:26:13] PHP Fatal error: Call to undefined function build_navigation() in C:\MoodleWindowsInstaller-latest-16\moodle\moodle\search\query.php on line 140

Have I miised something?

I am trying to understand the current status of this tracker issue. I would like to help with testing, but I am experiencing much of what Matt Gibson describes. It appears that much of the functionality exists in the 1.9 nightly builds, although the ability to add global search as a block has been removed.

I have make a test wiki page that does not have much more than the word wiki inside a wiki page.

When I run test/index.php I get the following:

Notice: /opt/lampp/htdocs/library0/search/documents/assignment_document.php does not exist, this module will not be indexed.
ERROR: chat_iterator() doesn't seem to be returning an object array.
Notice: /opt/lampp/htdocs/library0/search/documents/choice_document.php does not exist, this module will not be indexed.
ERROR: data_iterator() doesn't seem to be returning an object array.
Success: 'forum' module seems to be ready for indexing.
ERROR: glossary_iterator() doesn't seem to be returning an object array.
Notice: /opt/lampp/htdocs/library0/search/documents/hotpot_document.php does not exist, this module will not be indexed.
Notice: /opt/lampp/htdocs/library0/search/documents/journal_document.php does not exist, this module will not be indexed.
Notice: /opt/lampp/htdocs/library0/search/documents/label_document.php does not exist, this module will not be indexed.
Notice: /opt/lampp/htdocs/library0/search/documents/lams_document.php does not exist, this module will not be indexed.
Notice: /opt/lampp/htdocs/library0/search/documents/lesson_document.php does not exist, this module will not be indexed.
Notice: /opt/lampp/htdocs/library0/search/documents/quiz_document.php does not exist, this module will not be indexed.
Success: 'resource' module seems to be ready for indexing.
Notice: /opt/lampp/htdocs/library0/search/documents/scorm_document.php does not exist, this module will not be indexed.
Notice: /opt/lampp/htdocs/library0/search/documents/survey_document.php does not exist, this module will not be indexed.
Success: 'wiki' module seems to be ready for indexing.
Notice: /opt/lampp/htdocs/library0/search/documents/workshop_document.php does not exist, this module will not be indexed.
Notice: /opt/lampp/htdocs/library0/search/documents/exercise_document.php does not exist, this module will not be indexed.

Finished checking activity modules.

Since the wiki is the only thing that has content, and very little at that it makes sense that that is the only module that reports it is ready for indexing.

Performing the actual index gets me this:

Server Time: Fri, 05 Oct 2007 06:16:13 -0400

Using /opt/lampp/htdocs/moodledata/search as data directory.
Deleting old index entries.
Starting activity modules

18 modules found.
31 blocks found.
Processing module function chat_get_content_for_index ...
– 0 documents indexed
done.

Processing module function data_get_content_for_index ...
– 0 documents indexed
done.

Processing module function forum_get_content_for_index ...
– 0 documents indexed
done.

Processing module function glossary_get_content_for_index ...
– 0 documents indexed
done.

Processing module function resource_get_content_for_index ...
– 0 documents indexed
done.

Processing module function wiki_get_content_for_index ...
– 0 documents indexed
done.

Finished activity modules

The one wiki document does not appear to be indexed.

Can I use search in its current state in 1.9, or is it still necessary to pull the some of the fixes Valery mentions in http://tracker.moodle.org/browse/MDL-8074#action_31877 and apply them to the existing source tree.

I would be excited to help test the feature further.

Thanks.

We have a very satisfactory result of global search with Hans de Zwart, encouraging us to pursue Global Search finalizing.

I personally don't know much about 1.9 distribution strategy. The search engine reengineering was made and is still tested over 1.8.2. I will check that point. Anyway, it is mandatory the accessory search block being installed, as its global configuration parameters are needed to enable indexation.

Once installed, checking global configuration for that block will allow switching the indexing enabler, and configure pathes for the <format>totext converters.

Could you make me some tigher reporting on the search_block problem ?

Thanks.

[21-Nov-2007 10:28:49] SQL Column 'docid' cannot be null in C:\MoodleWindowsInstaller-latest-16\moodle\moodle\search\indexlib.php on line 232. STATEMENT: INSERT INTO mdl_block_search_documents ( DOCID, DOCTYPE, ITEMTYPE, TITLE, URL, DOCDATE, COURSEID, GROUPID ) VALUES ( null, 'forum', 'post', '', 'http://mymoodle.com/mod/forum/discuss.php?d=#', null, 66, null )
[21-Nov-2007 10:28:49] SQL Column 'docid' cannot be null in C:\MoodleWindowsInstaller-latest-16\moodle\moodle\search\indexlib.php on line 232. STATEMENT: INSERT INTO mdl_block_search_documents ( DOCID, DOCTYPE, ITEMTYPE, TITLE, URL, DOCDATE, COURSEID, GROUPID ) VALUES ( null, 'forum', 'post', '', 'http://mymoodle.com/mod/forum/discuss.php?d=#', null, 66, null )
[21-Nov-2007 10:28:49] SQL Column 'docid' cannot be null in C:\MoodleWindowsInstaller-latest-16\moodle\moodle\search\indexlib.php on line 232. STATEMENT: INSERT INTO mdl_block_search_documents ( DOCID, DOCTYPE, ITEMTYPE, TITLE, URL, DOCDATE, COURSEID, GROUPID ) VALUES ( null, 'forum', 'post', '', 'http://mymoodle.com/mod/forum/discuss.php?d=#', null, 66, null )
[21-Nov-2007 10:28:49] SQL Column 'docid' cannot be null in C:\MoodleWindowsInstaller-latest-16\moodle\moodle\search\indexlib.php on line 232. STATEMENT: INSERT INTO mdl_block_search_documents ( DOCID, DOCTYPE, ITEMTYPE, TITLE, URL, DOCDATE, COURSEID, GROUPID ) VALUES ( null, 'forum', 'post', '', 'http://mymoodle.com/mod/forum/discuss.php?d=#', null, 66, null )

I am getting forums indexing fine, so maybe you can provide a course backup file for testing?

Cheers,

Yu

If a new user enables via the experimental screen and then adds the block, is that sufficient with no more actions to get it working?

So it works "out-of-the-box", no need to import contrib code.