Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-29442

There are many "json_encode(): Invalid UTF-8 sequence in argument" errors in Apache's error_log file

    Details

    • Database:
      Any
    • Testing Instructions:
      Hide

      Use some Chinese character in the course title, then visit these course from navigation menu, apache error_log will show this error.

      You need PHP < 5.4.0

      Show
      Use some Chinese character in the course title, then visit these course from navigation menu, apache error_log will show this error. You need PHP < 5.4.0
    • Affected Branches:
      MOODLE_21_STABLE, MOODLE_24_STABLE
    • Fixed Branches:
      MOODLE_22_STABLE, MOODLE_23_STABLE, MOODLE_24_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      w49_MDL-29442_m25_entityutf8

      Description

      If you visit other course from navigation menu, apache's error_log file will record something like "json_encode(): Invalid UTF-8 sequence in argument". It seems json_encode() function received none utf8 character. Navigation menu can't work sometimes. I found some problematic code in lib/navigationlib.php line 4000:

      lib/navigationlib.php line 4000

      if ($child->forcetitle || $child->title !== $child->text) {
          $attributes['title'] = htmlentities($child->title);
      }

      htmlentities use ISO-8859-1 character set as default, but my course title has Chinese character, after called htmlentities, these Chinese character will not be recognizable any more. I think the code should be changed like this:

      if ($child->forcetitle || $child->title !== $child->text) {
          $attributes['title'] = htmlentities($child->title,null,'UTF-8');
      }

      I found moodle called htmlentities in many places, if these callings use ISO-8859-1 as default charset, the Asia user of moodle will suffer inconvenience. I hope all these callings will be checked carefully.

        Gliffy Diagrams

          Issue Links

            Activity

            Hide
            salvetore Michael de Raadt added a comment -

            Thanks for reporting this.

            I've put it on our backlog and we'll try to get to it as soon as we can.

            In the meantime adding more information, such as screenshots, replication instructions, fix test instructions, a workaround or even a code solution, will help us and other users

            Show
            salvetore Michael de Raadt added a comment - Thanks for reporting this. I've put it on our backlog and we'll try to get to it as soon as we can. In the meantime adding more information, such as screenshots, replication instructions, fix test instructions, a workaround or even a code solution, will help us and other users
            Hide
            samhemelryk Sam Hemelryk added a comment -

            Hi Yanfie,

            I've just been attempting to replicate this now but have not had any luck.
            If possible could provide us with a little bit more information about your system that may aid us in replicating this issue and finding a global solution.
            It would be helpful to know:

            1. What version of PHP you have.
            2. What database engine you are using and what version that is.
            3. What operating system you are using.

            Cheers
            Sam

            Show
            samhemelryk Sam Hemelryk added a comment - Hi Yanfie, I've just been attempting to replicate this now but have not had any luck. If possible could provide us with a little bit more information about your system that may aid us in replicating this issue and finding a global solution. It would be helpful to know: What version of PHP you have. What database engine you are using and what version that is. What operating system you are using. Cheers Sam
            Hide
            luyanfei Yanfei Lu added a comment - - edited

            Hi Sam.
            My PHP version is 5.3.2, database engine is Oracle 11GR2 11.2.0.1.0, Operating System is CentOS 6.
            To replicate this problem, I list some steps:
            1. create some course using some Chinese character as title.
            2. visit one course.
            3. in this course, visit navigation bar, click other Chinese titled course( first click will fail, but next click will success).
            4. check the httpd's error_log.
            I hope these will help you.
            Thanks for your concern.

            Show
            luyanfei Yanfei Lu added a comment - - edited Hi Sam. My PHP version is 5.3.2, database engine is Oracle 11GR2 11.2.0.1.0, Operating System is CentOS 6. To replicate this problem, I list some steps: 1. create some course using some Chinese character as title. 2. visit one course. 3. in this course, visit navigation bar, click other Chinese titled course( first click will fail, but next click will success). 4. check the httpd's error_log. I hope these will help you. Thanks for your concern.
            Hide
            richardlynch Richard Lynch added a comment - - edited

            PHP Warning: json_encode(): Invalid UTF-8 sequence in argument in /var/www/html/lms/learn/lib/navigationlib.php on line 4068

            We are getting this because Microsoft users paste in funky curly quotes, mdashes, etc for the navigation text.

            For example:
            "Week 01 \u2013 Charismatic and Transformational Leadership"

            Unfortunately, the correct solution will vary by what charset/encoding is in play by the end user, as far as I know.

            I could provide a function that converts all kinds of Microsoft [bleep] into the moral equivalent ASCII, but it would make hash of this guy's Chinese characters, almost for sure.

            It's remotely possible that somebody who understands UTF-8 et al far better than I could have a magic bullet to correctly "guess" the input charset/encoding, but I've never found such a beast for the general case.

            The second best correct solution would be to attempt to json_encode() the input at the point of input, and if it fails, reject it, because it's not valid UTF-8, and provide Help links to make common suggestions like using "Save as..." in MS Word to let IT convert its whack characters to UTF-8.

            At least that way it's caught at the GUI / UI / UX level, and the data isn't allowed to get in and plague us for the rest of time.

            I'll try to figure out where my guys are pasting this stuff in, and then the backend part of Moodle that validates it, and then submit a patch, but I'd bet a core Moodle DEV could do this in about 5 minutes instead of the week it will take me... And if I don't get it by 12/23, it won't be before 1/9, as I'm on vacation

            Show
            richardlynch Richard Lynch added a comment - - edited PHP Warning: json_encode(): Invalid UTF-8 sequence in argument in /var/www/html/lms/learn/lib/navigationlib.php on line 4068 We are getting this because Microsoft users paste in funky curly quotes, mdashes, etc for the navigation text. For example: "Week 01 \u2013 Charismatic and Transformational Leadership" Unfortunately, the correct solution will vary by what charset/encoding is in play by the end user, as far as I know. I could provide a function that converts all kinds of Microsoft [bleep] into the moral equivalent ASCII, but it would make hash of this guy's Chinese characters, almost for sure. It's remotely possible that somebody who understands UTF-8 et al far better than I could have a magic bullet to correctly "guess" the input charset/encoding, but I've never found such a beast for the general case. The second best correct solution would be to attempt to json_encode() the input at the point of input, and if it fails, reject it, because it's not valid UTF-8, and provide Help links to make common suggestions like using "Save as..." in MS Word to let IT convert its whack characters to UTF-8. At least that way it's caught at the GUI / UI / UX level, and the data isn't allowed to get in and plague us for the rest of time. I'll try to figure out where my guys are pasting this stuff in, and then the backend part of Moodle that validates it, and then submit a patch, but I'd bet a core Moodle DEV could do this in about 5 minutes instead of the week it will take me... And if I don't get it by 12/23, it won't be before 1/9, as I'm on vacation
            Hide
            richardlynch Richard Lynch added a comment -

            Given that I haven't even come close to my self-imposed deadline of starting this, I'm officially notifying anybody who cares that I'm not going to have time to fix it.

            I just told my guys "Don't do that, or it will look bad." and moved on.

            Show
            richardlynch Richard Lynch added a comment - Given that I haven't even come close to my self-imposed deadline of starting this, I'm officially notifying anybody who cares that I'm not going to have time to fix it. I just told my guys "Don't do that, or it will look bad." and moved on.
            Hide
            poltawski Dan Poltawski added a comment -

            Seeing this in the moodle.org logs quite a lot too:

            PHP Warning:  json_encode(): Invalid UTF-8 sequence in argument in /var/www/vhosts/moodle.org/html/lib/navigationlib.php on line 
            4308, referer: https://moodle.org/forums/

            Show
            poltawski Dan Poltawski added a comment - Seeing this in the moodle.org logs quite a lot too: PHP Warning: json_encode(): Invalid UTF-8 sequence in argument in /var/www/vhosts/moodle.org/html/lib/navigationlib.php on line 4308, referer: https://moodle.org/forums/
            Hide
            poltawski Dan Poltawski added a comment -

            (If I can help try track it down)

            Show
            poltawski Dan Poltawski added a comment - (If I can help try track it down)
            Hide
            skodak Petr Skoda added a comment -

            Sorry for stealing this issue, I think I found the source of this problem, workaround should be to use PHP >= 5.4.0 which defaults to UTF-8 encoding in most functions...

            Show
            skodak Petr Skoda added a comment - Sorry for stealing this issue, I think I found the source of this problem, workaround should be to use PHP >= 5.4.0 which defaults to UTF-8 encoding in most functions...
            Hide
            samhemelryk Sam Hemelryk added a comment -

            Thanks Petr! this has been integrated now. Great to finally have this tracked down and sorted.

            Show
            samhemelryk Sam Hemelryk added a comment - Thanks Petr! this has been integrated now. Great to finally have this tracked down and sorted.
            Hide
            abgreeve Adrian Greeve added a comment -

            Tested on the master integration and master stable branches.
            I couldn't replicate the problem.
            Things I did:

            • Created a course with only Chinese characters.
            • Added the Mandarin language pack.
            • Double checked my version of php (running php 5.3.2)

            I didn't get any errors in the apache error logs (integration or stable).
            Test passed only because the new code doesn't generate any errors either.

            Show
            abgreeve Adrian Greeve added a comment - Tested on the master integration and master stable branches. I couldn't replicate the problem. Things I did: Created a course with only Chinese characters. Added the Mandarin language pack. Double checked my version of php (running php 5.3.2) I didn't get any errors in the apache error logs (integration or stable). Test passed only because the new code doesn't generate any errors either.
            Hide
            stronk7 Eloy Lafuente (stronk7) added a comment -

            Many thanks for your effort, the whole Moodle Community will be enjoying your great solutions starting now!

            Closing, ciao

            Show
            stronk7 Eloy Lafuente (stronk7) added a comment - Many thanks for your effort, the whole Moodle Community will be enjoying your great solutions starting now! Closing, ciao

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  14/Jan/13