Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-60357

core_search: Future modified times cause serious problems

XMLWordPrintable

    • MOODLE_32_STABLE, MOODLE_33_STABLE, MOODLE_34_STABLE
    • MOODLE_32_STABLE, MOODLE_33_STABLE
    • MDL-60357-master
    • Hide

      You will need a working global search setup, and the ability to directly edit your Moodle database.

      1 Go to a forum (or create a new one).

      2 Add a new forum discussion including the words phliggleblorp 1.

      3 Add a reply to the discussion, also including the words phliggleblorp 2.

      4 Find the forum post ID of the reply (e.g. look at the links about it)

      5 Run this SQL command in the database, replacing mdl_ with your database prefix and whatever with the forum post ID you just identified:

      UPDATE mdl_forum_posts SET modified=1751328000 WHERE id = whatever;

      6 Reload the discussion view and confirm that the forum post now appears to have been modified in the year 2025.

      7 Run search indexing, for example by clicking 'Run now' on the relevant scheduled task in the admin scheduled tasks screen.

      8 Look at the search areas screen and check the last indexed date shown against the forum posts search area.

      EXPECTED: This should show the current date. (Before fix: It would have shown 2025.)

      9 Do a search for phliggleblorp.

      EXPECTED: This should show only the normal post, phliggleblorp 1, not the one you messed with the date. (Before fix: It would show both 1 and 2.)

      10 Create another reply in the same discussion, including phliggleblorp 3.

      11 Do indexing again.

      12 Do another search for phliggleblorp.

      EXPECTED: This should show results 1 and 3. (Before fix: It would only ever show 1 and 2, however many more times you posted between now and 2025.)

      Show
      You will need a working global search setup, and the ability to directly edit your Moodle database. 1 Go to a forum (or create a new one). 2 Add a new forum discussion including the words phliggleblorp 1 . 3 Add a reply to the discussion, also including the words phliggleblorp 2 . 4 Find the forum post ID of the reply (e.g. look at the links about it) 5 Run this SQL command in the database, replacing mdl_ with your database prefix and whatever with the forum post ID you just identified: UPDATE mdl_forum_posts SET modified=1751328000 WHERE id = whatever; 6 Reload the discussion view and confirm that the forum post now appears to have been modified in the year 2025. 7 Run search indexing, for example by clicking 'Run now' on the relevant scheduled task in the admin scheduled tasks screen. 8 Look at the search areas screen and check the last indexed date shown against the forum posts search area. EXPECTED: This should show the current date. (Before fix: It would have shown 2025.) 9 Do a search for phliggleblorp. EXPECTED: This should show only the normal post, phliggleblorp 1, not the one you messed with the date. (Before fix: It would show both 1 and 2.) 10 Create another reply in the same discussion, including phliggleblorp 3 . 11 Do indexing again. 12 Do another search for phliggleblorp. EXPECTED: This should show results 1 and 3. (Before fix: It would only ever show 1 and 2, however many more times you posted between now and 2025.)

      (I am setting up search on some of our test systems and consequently finding problems!)

      The search indexing process works based on modified time (usually a 'timemodified' field), roughly like this:

      • Get all documents newer than the previous indexed time (in time order)
      • Index all those documents
      • Set the indexed time to the modified time of the last document

      Supposing there are some documents where timemodified is set in the future (e.g. 2020), this causes a serious problem, because the indexed time will be set to the future. Next time it tries to index, it will only be looking for documents since 2020, and it will never index all the new documents added (with correct time) between now and 2020.

      Incorrect timemodified values can happen in various search areas, usually due to incorrect backup/restore implementation. These may also exist in historic data that hasn't been touched for ages. As a safety measure, I propose the following solution:

      • Do not index any document with timemodified in the future.

      (The other option was to just prevent it setting indexed time in the future, but if we did that, it would repeatedly index a stack of 'future' documents every single time indexing is called, which is poor.)

      This might cause problems when somebody restores a site and it gets the dates wrong as then it would never index those items, but restore generally doesn't work with search indexing anyway, and that would be resolved by my fix to MDL-55356. It's better to have the occasional screwed-up document not being indexed than to prevent all indexing of documents until some distant future date as present...

      I will submit a fix for this bug soon. Given all the other changes in search it will probably be easiest if we fix only in 3.4+ although it would be possible to do a fix for earlier versions too if required.

       

        1. solrreindex.png
          solrreindex.png
          35 kB

            quen Sam Marshall
            quen Sam Marshall
            David Monllaó David Monllaó
            Eloy Lafuente (stronk7) Eloy Lafuente (stronk7)
            Marina Glancy Marina Glancy
            Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved:

                Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.