Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Critical
-
Resolution: Fixed
-
Affects Version/s: 1.5, 1.8.9, 1.9.5
-
Component/s: General
-
Labels:None
-
Environment:Both PHP4 and PHP5
-
Affected Branches:MOODLE_15_STABLE, MOODLE_18_STABLE, MOODLE_19_STABLE
-
Fixed Branches:MOODLE_18_STABLE, MOODLE_19_STABLE
Description
html2text function called indirectly from format_text_email function is not compatible with utf-8 charset encoding. html2text replaces all chr(160) bytes with ' ' at end of the function, while chr(160) in utf-8 encoding does not mean a white space. This causes some characters in utf-8 encoding such as 'da' (U+3060) characters in ja_utf8 garbled in text formatted email.
— ../20050325/moodle/lib/html2text.php Sun Jan 23 11:18:50 2005
+++ html2text.php Sat Mar 26 16:56:06 2005
@@ -157,12 +157,12 @@
$goodStr = wordwrap( $goodStr, 78 );
//make sure there are no more than 3 linebreaks in a row and trim whitespace
- $goodStr = str_replace(chr(160), ' ', $goodStr );
+// $goodStr = str_replace(chr(160), ' ', $goodStr );
$goodStr = preg_replace(/\r\n?/\f/, \n, $goodStr);
$goodStr = preg_replace(/\n(\s*\n)
{2}/, \n\n\n, $goodStr);
$goodStr = preg_replace(/[ \t]+(\n/$)/, $1, $goodStr);
$goodStr = preg_replace(/^\n*/\n*$/, '', $goodStr);
- $goodStr = str_replace(chr(160), ' ', $goodStr );
+// $goodStr = str_replace(chr(160), ' ', $goodStr );
return $goodStr;
Gliffy Diagrams
Issue Links
Activity
assign to a valid user
I have just one question: what is the use of the html_to_text($html) function in moodle weblib ? That function uses the html2text.php library, which mangles utf8 text.
I found this by accident, because html_to_text() IS used in the Questionnaire plugin (where it causes a problem in non-ASCII languages), but nowhere else.
In MDL-17542, François Marier reported "I have committed an updated version of this file to CVS, along with a readme describing its origin." but again, what's the point of it all if that library is not being used anywhere in Moodle core files?
I remain puzzled,
Joseph
It's also used for example when emailing out forum posts.
François, sorry but in moodle 1.9 I cannot find a reference to the html_to_text($html) function being used anywhere. In which script is it actually used?
Hi 
html2text is not working correctly in 1.9.5. To reproduce the problem:
1. Write a forum email with non ASCII charaters
2. Send the email
Look at the text/plain part. Part of the body is incorrectly encoded.
Problem is in method _convert() in html2text.php. html_entity_decode() works in latin1 by default, so first text should be converted to latin1 or specify what's the input encoding.
The patch attached seems to solve the problem.
Thanks in advance
Joseph, if you look in lib/weblib.php, the html_to_text() function is used once within the format_text_email() function:
function format_text_email($text, $format) {
switch ($format)
{ ... case FORMAT_HTML: return html_to_text($text); break; ... }}
Alright, I've got a patch which seems to work both on PHP5 and PHP4.
Can people please test it and confirm whether or not it fixes their issues? I'm particularly interested to hear whether it works on non-latin locales (for example, Japanese).
Cheers,
Francois
Sorry for the noise guys, but I submitted a wrong patch the other day... bad day 
This solves our issues with plain text email...
I'll try to provide feedback for the other patch too.
bye 
Hi Juan,
I'm not sure about the conversion to latin1. What happens if there are characters (e.g. Japanese characters) which fall outside of that range?
This is why I'd like someone using a non-latin1 locale to confirm that the html_entity_decode_utf8 patch works.
Cheers,
Francois
Well done Francois
Thanks for the Fix Francois and Juan!
Works on m1.9.5+ (php 5.2.9)
Cheers,
Jens
Hi,
as commented in MDL-19499, +1 to use current textlib->entities_to_utf8(), plus tests in lib/simpletest/testweblib.php to see if it works ok. (-1 to add new function into html2text).
Ciao
Thanks for that Eloy, I'm going to have a look at textlib and test it on PHP4.
New patch based on Eloy's suggestion.
Fixed in 1.8 and 1.9 (HEAD was not affected by this problem).
I have updated the unit tests to match the output of this new library. They all pass now.
I've backported tests to 18_STABLE and they are passing ok under all branches.
So closing as reviewed. Thanks, Francois B)
Ciao
From Martin Dougiamas (martin at moodle.com) Sunday, 27 March 2005, 01:50 PM:
Thanks! Fixed in 1.5 CVS