Issue Details (XML | Word | Printable)

Key: MDL-14137
Type: Bug Bug
Status: Closed Closed
Resolution: Fixed
Priority: Critical Critical
Assignee: Dongsheng Cai
Reporter: Urs Hunkler
Votes: 1
Watchers: 2
Operations

Add/Edit UI Mockup to this issue
If you were logged in you would be able to see more operations.
Moodle

Databas field description with Umlauts get seaved as strange encodings -> preset can't be imported

Created: 02/Apr/08 12:28 AM   Updated: 29/Apr/08 08:33 AM
Return to search
Component/s: Database activity module
Affects Version/s: 1.9
Fix Version/s: 1.9.1

File Attachments: None
Image Attachments:

1. Insertion_of_new_field_failed.png
(9 kB)

Participants: Dongsheng Cai, Eloy Lafuente (stronk7), Petr Skoda, Robert Allerstorfer and Urs Hunkler
Security Level: None
QA Assignee: Eloy Lafuente (stronk7)
Resolved date: 18/Apr/08
Affected Branches: MOODLE_19_STABLE
Fixed Branches: MOODLE_19_STABLE


 Description  « Hide
Database field entries with Umlauts get saved as strange encodings -> preset can't be imported.
"können" gets "können" in the preset.xml

 All   Comments   Change History   Version Control      Sort Order: Ascending order - Click to sort in descending order
Robert Allerstorfer added a comment - 09/Apr/08 06:02 AM
You also must not use these characters: & (ampersand), ' (single quote).
Creating such fields within the web interface work without problems, exporting the resulting preset.zip works, too, however, re-imprting the .zip will fail!

So, for the moment take much care not to use any "forbidden" characters in the fields' names, descriptions or options.


Eloy Lafuente (stronk7) added a comment - 09/Apr/08 06:40 AM
The key here, if I'm not wrong is that we should be using htmlspecialchars() when exporting data to XML, instead of the full (and buggy) htmlentities().

Moving this to DC... can you please test the generated preset with some "strange" characters both with htmlentities() (current behaviour) and htmlspecialchars() (proposed behaviour).

Also, you'll need to test of that preset using htmlspecialchars() to check that "strange" characters are also imported properly.

I would suggest you to post here your research results before committing anything, just to decide if the proposed htmlspecialchars() is the correct one (I hope yes).

TIA and ciao


Dongsheng Cai added a comment - 09/Apr/08 02:04 PM
The differences between htmlspecialchars and htmlentities can be found in php.net:

http://au2.php.net/manual/en/function.htmlentities.php
http://au2.php.net/manual/en/function.htmlspecialchars.php

I think we should filter single and double quote here, so I used ENT_QUOTES parameter here. I test umlauts and Chinese characters in mod/data, it is alright now.


Robert Allerstorfer added a comment - 09/Apr/08 02:33 PM
Dongsheng, since you have closed this bug, I am wondering where your fix is available. Could you please attach the patch here?

Dongsheng Cai added a comment - 09/Apr/08 02:44 PM
Sorry, I forgot to attach the patch. You can find the changes here: http://cvs.moodle.org/moodle/mod/data/lib.php?r1=1.147&r2=1.148

Robert Allerstorfer added a comment - 09/Apr/08 03:14 PM
Thank you, now I see. So the fix was committed into the HEAD branch. Will it also be available in MOODLE_19_STABLE?

Dongsheng Cai added a comment - 09/Apr/08 03:23 PM
Yes, it is available in MOODLE_19_STABLE branch

Robert Allerstorfer added a comment - 09/Apr/08 04:50 PM
Very good news, thank you Dongsheng.

What about also committing the attached little beast (preset-import-menu.patch) which fixes the bug discussed at MDL-12439? I worked long to find out how to fix that and it works (maybe it just makes no change on PHP 5, but it solved the thing on PHP 4).


Robert Allerstorfer added a comment - 11/Apr/08 06:38 AM
Sorry for posting the "preset-import-menu.patch" here - the bug fixed by that patch is MDL-9534 (which is open since one year).

Eloy Lafuente (stronk7) added a comment - 11/Apr/08 10:33 AM
Verified it's working here too. Closing.

Robert, what it I delete your patch from here and we continue working MDL-9534 and MDL-12439 in their respective places?

Really I need to re-read both... doing that now. Thanks!


Robert Allerstorfer added a comment - 11/Apr/08 05:35 PM
Eloy, yes, please remove my attachment from here - the right place to discuss this issue should be MDL-9534. Thank you.

Eloy Lafuente (stronk7) added a comment - 12/Apr/08 12:24 AM
Deleted. Thanks Robert.

Robert Allerstorfer added a comment - 17/Apr/08 03:06 AM
Unfortunately, this bug is not yet completely solved - please reopen.

What is still not possible is using single quotes within the description of fields (not tested at other places yet). At exporting the preset.zip, single quotes get replaced by "'" in the preset.xml - for example, there will be a field definition like this:

<field>
<type>text</type>
<name>Title of material</name>
<description>The material's title</description>
</field>

When trying to reimport that preset.zip, all fields containing this escaped single quote (or a true single quote character) will NOT import


Dongsheng Cai added a comment - 17/Apr/08 10:50 AM
As I mentioned before, I used ENT_QUOTES argument to filter single and double quote in the field name(for security), now I realize single and double quote are very useful in the field name, so I should remove this argument, how do you think, Eloy?

Dongsheng Cai added a comment - 17/Apr/08 11:01 AM
Keep single and double quote in present.xml, Thanks Robert.

Eloy Lafuente (stronk7) added a comment - 17/Apr/08 11:42 AM
If I'm not wrong... XML data must be free, exclusively of these chars: < > & and they must be transformed by their corresponding entitied.

In the other side... XML attributes must be free of the chars above AND these chars: " '

Our XML doesn't use XML attributes at all, so it safe to use NOENT_QUOTES as you've done. Then, automatically, XML parsers, when reading that XML, will revert the entities to their original chars.

In fact, the problem with single quotes is that htmlspecialchars() doesn't encode it properly. It should be ' and not ' But that's not important here, as we aren't using XML attributes, as explained above.

I say all this because, for exampe, in backup, we use htmlspecialchars() without parameters. That means that apart of < > &, also " are being transformed to " and, when we restore those files they are transformed back to " automatically.

So:

1) Patch looks ok (although I guess that it's safe to, simply, don't use the 2nd parameter, like backups does).
2) Could you try some single quotes and double quotes using htmlspecialchars() without second parameter?

Ciao


Robert Allerstorfer added a comment - 17/Apr/08 06:52 PM
Still does not work. Please re-re-open.

Now the difference to the previous attempt is that a single quote gets exported literally, however a single quote within a preset.xml's field description still fails to import, no matter if it is there literally (as ') or as entity (').


Eloy Lafuente (stronk7) added a comment - 17/Apr/08 07:43 PM
Grrr, this is the never ending bug! Thanks for testing Robert.

Oki, to fix this, definitively, Dongsheng, can you, please:

1) Use htmlspecialchars() without parameter at all.
2) Review where data is inserted to DB, I guess there is some missing addslashes() somewhere (I think I suggested this in another database activity bug recently, but obviously, it hasn't been fixed).
3) Test it works before commit.

TIA! B-)


Eloy Lafuente (stronk7) added a comment - 17/Apr/08 07:44 PM
Reopening.... as requested... thanks again!

Dongsheng Cai added a comment - 18/Apr/08 10:09 AM
About htmlspecialchars(), I remove the second parameter now, it worked, but the double quote will be changed to entity.

function get_settings() doesn't use addslashes(), that is the reason why you cannot import files with single quote.


Dongsheng Cai added a comment - 18/Apr/08 10:15 AM
fixed and tested.
Double quote and single quote can be included in preset.xml now, double quote will be changed to entity, single quote doesn't change, please review, feel free to open it if you find any problems. (I hope it works well )

Robert Allerstorfer added a comment - 19/Apr/08 03:18 AM
Finally..... it W O R K S )

Thank you all.


Petr Skoda added a comment - 22/Apr/08 03:32 PM - edited
hmm, the use of magic quotes is not optional at all in data mod
fortunaletly it will be gone in 2.0

Eloy Lafuente (stronk7) added a comment - 29/Apr/08 08:33 AM
Verified. Closing...

BTW, Petr... I don't get your latest comment... anything important? Feel free to reopen if so.

Ciao