Moodle
  1. Moodle
  2. MDL-6336

After utf8-migration unserializing of old essay fails

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.1
    • Fix Version/s: 1.6.3
    • Component/s: Lesson, Unicode
    • Labels:
      None
    • Environment:
      MySQL 5.0.22, PHP 4.3.10, Apache 1.3.33, Debian Linux
    • Database:
      MySQL
    • Affected Branches:
      MOODLE_16_STABLE
    • Fixed Branches:
      MOODLE_16_STABLE
    • Rank:
      28397

      Description

      After upgrade from 1.5.2+ to 1.6.1 and utf8-migration, essays writen before upgrade and therefore migrated do not get show to users (inc. admin). Turning on debug show a message:
      Notice: unserialize(): Error at offset 909 of 1110 bytes in /var/www/moodle/mod/lesson/view.php on line 1672.

      All pre-migration essays throw that error. New essays seem to work. In the db the attempts look to be fine. Not truncated or anything obvious. Essays are writen in Finnish, so there are a lot of non-ascii characters in them.

      Seem fairly obvious that unserialize can handle "useranswer" strings that have gone through utf8-migration. However, the error must be rather subtle, because the "useranswer" strings do seem identical.

      Here's the "useranswer" field of lesson_attempt table for the same attempt at the error message above.
      ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
      O:8:"stdClass":5:

      {s:4:"sent";i:1;s:6:"graded";i:1;s:5:"score";i:0;s:6:"answer";s:823:"Ensimmäinen askel kohti sähköistä arkistoa on viitearkisto. Siinä on sähköisessä muodossa vain viitetiedot ja varsinaiset dokumentit ovat paperilla. Tämä soveltuu parhaiten, jos kaikkien tietojen siirto sähköiseen muotoon vaatisi liikaa resursseja. Esim. kirjastojen hakukoneet ovat viitearkistoja. Sähköinen tekstiarkisto sisältää dokumenttien tekstit sähköisessä muodossa. Kuvat ovat tallennettu erikseen kuvatietokannaksi. tämä sopii silloin, jos arkistoitavasta tiedosta suurin osa on tekstiä. esim. kirjanpidon aineistot ovat näitä tekstiarkistoja. Sähköinen dokumenttiaineisto perustu kokonaisten dokumenttien käsittelyjärjestelmään. Dokumenttiarkistojen tieto tallennetaan yleensä kuvamuodossa. Tiedonhaku perustuu näytöltä luettavassa muodossa olevaan hakemistoon, joka sisältää jäsentelyn arkiston tiedoista.";s:8:"response";s:130:"Ok, hyvä vastaus! Lisäksi olisit voinut vielä pohtia kunkin arkistotason soveltumista taloushallinnon aineistojen säilyttämiseen.";}

      ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

        Issue Links

          Activity

          Hide
          Jussi Hannunen added a comment -

          Doh! Always forgetting details.

          That specific issue/error is from https://moodle.tpu.fi/mod/lesson/view.php?id=21198&action=essaygrade&attemptid=83, a page which doesn't have a name, but is the "single essay view", accessible from Grade Essays tab (https://moodle.tpu.fi/mod/lesson/view.php?id=21198&action=essayview). Every essay is unserialized when viewing the Grade Essays tab itself and a couple of dozen error messages are show like this:
          Notice: unserialize(): Error at offset 769 of 832 bytes in /var/www/moodle/mod/lesson/view.php on line 1603

          In moodle/mod/lesson/view.php both the line 1603 and the line 1672 are:
          $essayinfo = unserialize($essay->useranswer);

          Default charset for db and table mdl_lesson_attempt is utf8.

          Show
          Jussi Hannunen added a comment - Doh! Always forgetting details. That specific issue/error is from https://moodle.tpu.fi/mod/lesson/view.php?id=21198&action=essaygrade&attemptid=83 , a page which doesn't have a name, but is the "single essay view", accessible from Grade Essays tab ( https://moodle.tpu.fi/mod/lesson/view.php?id=21198&action=essayview ). Every essay is unserialized when viewing the Grade Essays tab itself and a couple of dozen error messages are show like this: Notice: unserialize(): Error at offset 769 of 832 bytes in /var/www/moodle/mod/lesson/view.php on line 1603 In moodle/mod/lesson/view.php both the line 1603 and the line 1672 are: $essayinfo = unserialize($essay->useranswer); Default charset for db and table mdl_lesson_attempt is utf8.
          Hide
          Mark Nielsen added a comment -

          I'm not at all familiar with the migration process

          I'm guessing that some sort of procedure was used directly on the useranswer field for the LESSON_ESSAY question type before unserializing it. This would then break the serialized string if any of the values changed in length.

          More info:

          The useranswer field for LESSON_ESSAY is a serialized object with the following set:

          sent = 1 or 0
          graded = 1 or 0
          score = a number
          answer = text
          response = text

          I looked in the migrate2utf8 scripts in mod/lesson/db and found the only reference to useranswer was in migrate2utf8.xml at line 8, but I have no clue what it is doing

          If someone could point me to where I would need to modify the migration2utf8 script to fix this problem, I'll be happy to tackle it. But right now I feel like I would have to dig around for hours just to figure out how it works.

          Cheers,
          Mark

          Show
          Mark Nielsen added a comment - I'm not at all familiar with the migration process I'm guessing that some sort of procedure was used directly on the useranswer field for the LESSON_ESSAY question type before unserializing it. This would then break the serialized string if any of the values changed in length. More info: The useranswer field for LESSON_ESSAY is a serialized object with the following set: sent = 1 or 0 graded = 1 or 0 score = a number answer = text response = text I looked in the migrate2utf8 scripts in mod/lesson/db and found the only reference to useranswer was in migrate2utf8.xml at line 8, but I have no clue what it is doing If someone could point me to where I would need to modify the migration2utf8 script to fix this problem, I'll be happy to tackle it. But right now I feel like I would have to dig around for hours just to figure out how it works. Cheers, Mark
          Hide
          Mark Nielsen added a comment -

          Added Eloy as a Watcher so hopefully he will be emailed. Sorry Eloy if you do not handle utf8 migration bugs, but I'm sure you will know who would.

          I think the problem with this one lies in the migrate2utf8.xml file near the top. It needs to call a php function for lesson_attempts table, but I have no clue how to do this. We need this function because the useranswer field for essay questions has a serialized object and needs to be unserialized, converted and then serialized for storage. Least this is what I think is going wrong

          Thanks for any help in resolving this issue.

          Show
          Mark Nielsen added a comment - Added Eloy as a Watcher so hopefully he will be emailed. Sorry Eloy if you do not handle utf8 migration bugs, but I'm sure you will know who would. I think the problem with this one lies in the migrate2utf8.xml file near the top. It needs to call a php function for lesson_attempts table, but I have no clue how to do this. We need this function because the useranswer field for essay questions has a serialized object and needs to be unserialized, converted and then serialized for storage. Least this is what I think is going wrong Thanks for any help in resolving this issue.
          Hide
          Mark Nielsen added a comment -

          Added Petr as a watcher. Just trying to get some help

          Show
          Mark Nielsen added a comment - Added Petr as a watcher. Just trying to get some help
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Aha!

          1- So, the lesson_attempts->useranswer CAN contain:

          • One normal string
          • One serialised object

          2- Oki. And when it contains one serialised object the structure of such object is ALWAYS:

          • sent (number)
          • graded (number)
          • score (number)
          • answer (text)
          • response (text)

          3- And the answer is written by the student.

          4- And the response is writen by the teacher.

          5- And this is the only one field in the lesson module using serialised storage.

          Please, check and conffirm/fix all the statements above. Thats really important in order to be able to follow the behaviour below.

          The basic idea is that we have to change current lesson_attempts->useranswer conversion from the current simple SQL definition (done, because we assumed it was always a simple text field) to one spesialised function to be stored in mod/lesson/migrate2utf8.php called migrate2utf8_lesson_attempts_useranswer()

          This function should:

          • Detect the nature of the useranswer field for the recordid passed
          • If it's one plain string, just convert and return the converted field
          • If it's one serialised object, unserialise, convert attributes (answer, response), serialise and return the converted field

          (I've added Yu here too) Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - Aha! 1- So, the lesson_attempts->useranswer CAN contain: One normal string One serialised object 2- Oki. And when it contains one serialised object the structure of such object is ALWAYS: sent (number) graded (number) score (number) answer (text) response (text) 3- And the answer is written by the student. 4- And the response is writen by the teacher. 5- And this is the only one field in the lesson module using serialised storage. Please, check and conffirm/fix all the statements above. Thats really important in order to be able to follow the behaviour below. The basic idea is that we have to change current lesson_attempts->useranswer conversion from the current simple SQL definition (done, because we assumed it was always a simple text field) to one spesialised function to be stored in mod/lesson/migrate2utf8.php called migrate2utf8_lesson_attempts_useranswer() This function should: Detect the nature of the useranswer field for the recordid passed If it's one plain string, just convert and return the converted field If it's one serialised object, unserialise, convert attributes (answer, response), serialise and return the converted field (I've added Yu here too) Ciao
          Hide
          Mark Nielsen added a comment -

          Hi Eloy,

          Thanks for your comment!

          All 1 through 4 are correct. As for other serialized objects, there is one in the lesson and lesson_default tables. The field in both tables is called conditions. Its object contains three settings: timespent, completed, and gradebetterthan. All three are number values.

          Everything in your post sounds great as well. Let me know if I can help further.

          Show
          Mark Nielsen added a comment - Hi Eloy, Thanks for your comment! All 1 through 4 are correct. As for other serialized objects, there is one in the lesson and lesson_default tables. The field in both tables is called conditions. Its object contains three settings: timespent, completed, and gradebetterthan. All three are number values. Everything in your post sounds great as well. Let me know if I can help further.
          Hide
          Yu Zhang added a comment -

          Hi,

          I made some changes in 1.6 and HEAD regarding this issue. Basically checking if unserialization succeeds or not, and perform the right action accordingly. Please let me know if there are still problems with this, as I do not have a large number of data to play with.

          Are the other serialized objects giving you problems?

          Yu

          Show
          Yu Zhang added a comment - Hi, I made some changes in 1.6 and HEAD regarding this issue. Basically checking if unserialization succeeds or not, and perform the right action accordingly. Please let me know if there are still problems with this, as I do not have a large number of data to play with. Are the other serialized objects giving you problems? Yu
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Just testing MDL-6854 I got this:

          Processsing db table lesson_branch...

          Success

          Success

          Success

          Parse error: syntax error, unexpected T_OBJECT_OPERATOR in /mod/lesson/db/migrate2utf8.php on line 44

          So, perhaps, it's broken now...reopening...

          Show
          Eloy Lafuente (stronk7) added a comment - Just testing MDL-6854 I got this: Processsing db table lesson_branch... Success Success Success Parse error: syntax error, unexpected T_OBJECT_OPERATOR in /mod/lesson/db/migrate2utf8.php on line 44 So, perhaps, it's broken now...reopening...
          Hide
          Eloy Lafuente (stronk7) added a comment -

          resolving it again. It was one simple typo, preventing migration to continue.

          Show
          Eloy Lafuente (stronk7) added a comment - resolving it again. It was one simple typo, preventing migration to continue.

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: