Moodle

solved

inside the file /moodle/search/Zend/search/Lucent.php
i set "$create=true" on line 204 (one time only !!!) and ran cron.php
then removed the "$create=true" and run cron.php again ( to make sure all works fine)

maybe it should have been wone of the tasks that moodle was suppose to do when
i upgraded from from php 4.x to php 5.x ?

Hi Nadav.

Well done on tracking down the cause of this bug.

A better fix (although still rough in my opinion) is to add the following code at line 204

// See tracker MDL-15824
if (!$this->_directory->fileExists('segments'){ $create=true; }

This is better because it will only create the segments file if necessary.

However, a better fix would be for a Moodle_Zend_Search_Lucene class to be created which extends Zend_Search_Lucene.
The __constructor of Moodle_Zend_Search_Lucene would then perform the checking of 'segments' and would create the file if necessary.
It would then call the parent __constructor.

Basically, the new Moodle_Zend_Search_Lucene class would completely replace the Zend_Search_Lucene class when called from Moodle code. This is better because it means the moodle dev team don't have to keep re-applying hacks to the Zend Lucence library every time a new version comes out.

P.S. I'm not on the dev team so I can't implement this idea.

Apologies, there is a bracket missing in my proposed code:

It should be:

// See tracker MDL-15824
if (!$this->_directory->fileExists('segments')){ $create=true; }

your code looks great

i will add it to my local install

and i second you suggestion about the __constructor.
i am not on the dev team so we can wait for some one to pick it up

Guy - Your code did correct my issue. Thanks!

Creating a class that extends Zend_Search_Lucene is definitely a better way to go.

I have added the suggested code to version 1.9.3 and I am still getting the following error in my CRON output...

___________________________________

Fatal exception from Lucene subsystem. Search engine may not have been updated.
exception 'Zend_Search_Lucene_Exception' with message 'fopen(/var/vle-data/student/search/index.lock):
failed to open stream: Permission denied' in /var/www/vle/search/Zend/Search/Lucene/Storage/File/Filesystem.php:63
Stack trace:
#0 /var/www/vle/search/Zend/Search/Lucene/Storage/Directory/Filesystem.php(154):
Zend_Search_Lucene_Storage_File_Filesystem->__construct('/var/vle-data/s...', 'w+b')
#1 /var/www/vle/search/Zend/Search/Lucene.php(201): Zend_Search_Lucene_Storage_Directory_Filesystem->createFile('index.lock')
#2 /var/www/vle/search/delete.php(44): Zend_Search_Lucene->__construct('/var/vle-data/s...')
#3 /var/www/vle/search/cron_php5.php(16): require_once('/var/www/vle/se...')
#4 /var/www/vle/search/cron.php(29): include('/var/www/vle/se...')
#5 /var/www/vle/blocks/search/block_search.php(83): include('/var/www/vle/se...')
#6 /var/www/vle/admin/cron.php(148): block_search->cron()
#7 {main}
done.

may be here is a solution:

http://framework.zend.com/issues/browse/ZF-3500

The problem is that it is using chmod() instead of umask(), which is probably a bad thing for a library to do. This is bad when the owner of the file is not the server user (e.g., publisher:apache or publisher:www) but you do not want to give the user (in this example, publisher) sudo rights so that it can change ownership to apache:apache.

Is it right that "search" needs to be updated with the 1.6.0 Zend code ?

I afraid I don't agree that this is "Minor". Cron failing stops nightly enrollment and backups !

We are on 1.9.4+ (2009Feb18) and have this issue as well.

I put in Guy's suggested solution (it's line 421 for my release) , but it did not fix the issue.

public function __construct($directory = null, $create = false)
{
...
// Escalate read lock to prevent current generation index files to be deleted while opening process is not done
Zend_Search_Lucene_LockManager::escalateReadLock($this->_directory);

$this->_generation = self::getActualGeneration($this->_directory);

// See tracker MDL-15824 - 2009Feb25 added by UVic
if (!$this->_directory->fileExists('segments')) { $create=true; }

if ($create) {
try {
Zend_Search_Lucene_LockManager::obtainWriteLock($this->_directory);
...

I replaced the current Zend Search packaged with 1.9.4+ with Zend Search from Zend framework 1.7.0 - it did nothing to resolve the issue.

I also think that this is not a minor issue, because it breaks the cron job.

As well as the fix to "search/Zend/Search/Lucene.php" (around line 421),

// NEW LINES START
if (!$this->_directory->fileExists('segments')){
    $create=true;
}
// NEW LINES STOP
if ($create) {

I also needed to comment out the "chmod" lines in "search/Zend/Search/Lucene/Storage/Directory/Filesystem.php" (around line 188), thus:

/* OLD LINES START **
if (!@chmod($this->_dirPath . '/' . $filename, self::$_defaultFilePermissions)) {
    ini_set('track_errors', $trackErrors);
    throw new Zend_Search_Lucene_Exception($php_errormsg);
}
** OLD LINES STOP ***/

Issue persists through version 1.9.5+ (Build: 20090722)...

<code>
Processing cron function for search....
-DELETE---
Fatal exception from Lucene subsystem. Search engine may not have been updated.
exception 'Zend_Search_Lucene_Exception' with message 'Index doesn't exists in the specified directory.' in /Volumes/Consul's Data HD/web/multi-learn.org/moodle/htdocs/search/Zend/Search/Lucene.php:451
Stack trace:
#0 /web/moodle/htdocs/search/delete.php(50): Zend_Search_Lucene->__construct('/Volumes/Consul...')
#1 /web/moodle/htdocs/search/cron_php5.php(18): require_once('/Volumes/Consul...')
#2 /web/moodle/htdocs/search/cron.php(25): include('/Volumes/Consul...')
#3 /web/moodle/htdocs/blocks/search/block_search.php(83): include('/Volumes/Consul...')
#4 /web/moodle/htdocs/admin/cron.php(148): block_search->cron()
#5 {main}
done.
</code>

Using PHP Version 5.3.0 (www.entropy.ch Release 3).

Likewise, issue is persisting through 1.9.6 (20091022) for us, as below.

<code>
Processing cron function for search....
-DELETE---
Fatal exception from Lucene subsystem. Search engine may not have been updated.
exception 'Zend_Search_Lucene_Exception' with message 'Index doesn't exists in the specified directory.' in /srv/moodle/search/Zend/Search/Lucene.php:453
Stack trace:
#0 /srv/moodle/search/delete.php(49): Zend_Search_Lucene->__construct('/srv/moodle_dat...')
#1 /srv/moodle/search/cron_php5.php(16): require_once('/srv/moodle/rel...')
#2 /srv/moodle/search/cron.php(25): include('/srv/moodle/rel...')
#3 /srv/moodle/blocks/search/block_search.php(83): include('/srv/moodle/rel...')
#4 /srv/moodle/admin/cron.php(148): block_search->cron()
#5 {main}
done.
</code>

PHP Version 5.2.4-2ubuntu5.4

Getting this problem in 1.9.7+ after trying Gordons fixes... Any ideas?

Processing cron function for search....
-DELETE---
Fatal exception from Lucene subsystem. Search engine may not have been updated.
exception 'Zend_Search_Lucene_Exception' with message 'Wrong segments.gen file format' in /home/vle/public_html/search/Zend/Search/Lucene.php:230
Stack trace:
#0 /home/vle/public_html/search/Zend/Search/Lucene.php(420): Zend_Search_Lucene::getActualGeneration(Object(Zend_Search_Lucene_Storage_Directory_Filesystem))
#1 /home/vle/public_html/search/delete.php(49): Zend_Search_Lucene->__construct('/home/vle/moodl...')
#2 /home/vle/public_html/search/cron_php5.php(16): require_once('/home/vle/publi...')
#3 /home/vle/public_html/search/cron.php(25): include('/home/vle/publi...')
#4 /home/vle/public_html/blocks/search/block_search.php(83): include('/home/vle/publi...')
#5 /home/vle/public_html/admin/cron.php(148): block_search->cron()
#6 {main}
done.

Fixed my problem.

I had to delete the search folder from the moodledata directory! Once it re-created everything has been fine!

woop woop

The quick way of solving this issue is as follows (for Moodle 1.9.6+ that is):

1. Go to line 421 in <MOODLE>/search/Zend/Search/Lucene.php and add this line:
// See tracker MDL-15824
$create = !$this->_directory->fileExists('segments');

2. Go to Moodledata folder and:
a) delete search folder if it exists
b) execute chmod -R 777 moodledata
c) execute chown -R <webserveruser>:<webservergroup> moodledata
d) recreate index from global search block <moodle>/search/indexer.php?areyousure=yes

3. In case you have cronjob running from cron system that calls cron task using command line php, after initial index creation do this:
a) execute chmod -R 777 moodledata/search

I'm suprised this is still not fixed.

I don't know if we have a reversion but in Moodle 1.9.11+ (Build: 20110303)
cron error is Fatal error</b>: Allowed memory size of 50331648 bytes exhausted (tried to allocate 77 bytes) in <b> moodle/search/Zend/Search/Lucene.php</b> on line <b>970
then re running cron got
<b>Fatal error</b>: Allowed memory size of 50331648 bytes exhausted (tried to allocate 78 bytes) in <b>moodle/search/Zend/Search/Lucene/Storage/File/Filesystem.php</b> on line <b>153</b><br />
then rerun brings back
When I turned off global search the error stopped.

I don't know if we have a reversion but in Moodle 1.9.11+ (Build: 20110303)
cron error is Fatal error</b>: Allowed memory size of 50331648 bytes exhausted (tried to allocate 77 bytes) in <b> moodle/search/Zend/Search/Lucene.php</b> on line <b>970
then re running cron got
<b>Fatal error</b>: Allowed memory size of 50331648 bytes exhausted (tried to allocate 78 bytes) in <b>moodle/search/Zend/Search/Lucene/Storage/File/Filesystem.php</b> on line <b>153</b><br />
then rerun brings back
When I turned off global search the error stopped.

This issue still persists in Moodle 2.1.1

Here is the error report from cron.php just after enabling "Global Search" experimental option and Block.

Processing cron function for search....
--DELETE----
Fatal exception from Lucene subsystem. Search engine may not have been updated.
exception 'Zend_Search_Lucene_Exception' with message 'Index doesn't exists in the specified directory.' in /var/www/html/moodle/search/Zend/Search/Lucene.php:448
Stack trace:
#0 /var/www/html/moodle/search/delete.php(52): Zend_Search_Lucene->__construct('/var/www/moodle...')
#1 /var/www/html/moodle/search/cron_php5.php(20): require_once('/var/www/html/m...')
#2 /var/www/html/moodle/search/cron.php(25): include('/var/www/html/m...')
#3 /var/www/html/moodle/blocks/search/block_search.php(68): include('/var/www/html/m...')
#4 /var/www/html/moodle/lib/cronlib.php(116): block_search->cron()
#5 /var/www/html/moodle/admin/cron.php(79): cron_run()
#6 {main}
done.
Finished blocks

It seems that the workaround i tried on Moodle 1.9.x works on Moodle 2.1.1 too

/var/www/html/moodle/search/Zend/Search/Lucene.php
I have added the line 417 "$create = true;"
Ran the cron.php script again. and got:

Processing cron function for search....
--DELETE----
Starting clean-up of removed records...
Index size before: 

9 modules to search in / 27 modules found.
0 blocks to search in / 43 blocks found.
1 additional to search in.

and then removed line 417.
and ran the cron.php script again to see that it works fine.

More...

running the Indexer (from: /moodle/search/stats.php ), i got the following error:
(/moodle/search/indexer.php?areyousure=yes)

Error writing to database

More information about this error
Debug info: Data too long for column 'title' at row 1
INSERT INTO mdl_block_search_documents (doctype,docid,itemtype,title,url,updated,docdate,courseid,groupid) VALUES(?,?,?,?,?,?,?,?,?)
[array (
0 => 'data',
1 => '64',
2 => 'record',
3 => '<p>Excel file with costs of travel to date - we have not had time to deal with staff costs yet and the equipment now requires a separate invoice, so that will take time and have to be reported in the next report. Also the books we ordered have not yet arrived, so we haven\'t paid for those yet either and they will appear in the next report.</p> 1 ',
4 => 'http://tempus-efa.proj.ac.il/moodle/mod/data/view.php?d=6&amp;rid=64',
5 => 1318364609,
6 => '1306934436',
7 => '2',
8 => '0',
)]
Stack trace:

    line 397 of /lib/dml/moodle_database.php: dml_write_exception thrown
    line 878 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
    line 920 of /lib/dml/mysqli_native_moodle_database.php: call to mysqli_native_moodle_database->insert_record_raw()
    line 244 of /search/indexlib.php: call to mysqli_native_moodle_database->insert_record()
    line 174 of /search/indexer.php: call to IndexDBControl->addDocument()

Changing (DB table) from VARCHAR 255 to TEXT, solved it.
(ALTER TABLE `mdl_block_search_documents` CHANGE `title` `title` TEXT CHARACTER SET utf8 COLLATE utf8_general_ci NOT NULL DEFAULT '')
I (we) should probably fix it in the code and not change the table structure.
Please review this issue and advise.

More...

sunning the indexer again, got some more issues:

Error writing to database

More information about this error
Debug info: Column 'docdate' cannot be null
INSERT INTO mdl_block_search_documents (doctype,docid,itemtype,title,url,updated,docdate,courseid,groupid) VALUES(?,?,?,?,?,?,?,?,?)
[array (
0 => 'data',
1 => '6',
2 => 'comment',
3 => 'Comment on ',
4 => 'http://tempus-efa.proj.ac.il/moodle/mod/data/view.php?d=6&rid=',
5 => 1318364924,
6 => NULL,
7 => '2',
8 => '0',
)]
Stack trace:

    line 397 of /lib/dml/moodle_database.php: dml_write_exception thrown
    line 878 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
    line 920 of /lib/dml/mysqli_native_moodle_database.php: call to mysqli_native_moodle_database->insert_record_raw()
    line 244 of /search/indexlib.php: call to mysqli_native_moodle_database->insert_record()
    line 174 of /search/indexer.php: call to IndexDBControl->addDocument()

Adding the following code to line 173 inside moodle/search/indexer.php

if ($document->date == NULL) continue;

worked around this issue.

More...

running the indexer.php again:

Error reading from database

More information about this error
Debug info: Unknown column 'r.summary' in 'field list'

SELECT
r.id as trueid,
cm.id as id,
r.course as course,
r.name as name,
r.summary as summary,
r.alltext as alltext,
r.reference as reference,
r.type as type,
r.timemodified as timemodified
FROM
mdl_resource_old as r,
mdl_course_modules as cm,
mdl_modules as m
WHERE
r.type = 'file' AND
cm.instance = r.id AND
cm.course = r.course AND
cm.module = m.id AND
m.name = 'resource'

[array (
)]
Stack trace:

    line 394 of /lib/dml/moodle_database.php: dml_read_exception thrown
    line 794 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
    line 149 of /search/documents/resource_document.php: call to mysqli_native_moodle_database->get_records_sql()
    line 162 of /search/indexer.php: call to resource_get_content_for_index()

changing line 133 in file 'moodle/search/documents/resource_document.php',
from:

r.summary as summary,

to:

r.intro as summary,

worked around this issue.
( I was guessing the field name from the table's structure )
please review this workaround.

Finally, the Indexer finished indexing our Moodle system successfully