Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-33753

moodle unzipping is not utf-8 compatible

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2, 2.3
    • Fix Version/s: 2.4
    • Component/s: General
    • Labels:
    • Testing Instructions:
      Hide

      1/ run phpunit tests (it includes tests of the most common zip utils)
      2/ try to unzip random zip files with utf-8 chars - please note you may need to change moodle language to match Windows default non-unicode locale

      Show
      1/ run phpunit tests (it includes tests of the most common zip utils) 2/ try to unzip random zip files with utf-8 chars - please note you may need to change moodle language to match Windows default non-unicode locale
    • Affected Branches:
      MOODLE_22_STABLE, MOODLE_23_STABLE
    • Fixed Branches:
      MOODLE_24_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      w28_MDL-33753_m24_utfunzip

      Description

      At present we can only unzip archives created with unicode filenames - but the standard is to use unicode flag or extra attribute for utf-8 names, it is also expected to fall back to dos charset.

        Gliffy Diagrams

        1. downlodall-zip-heb-utf8-gebberish.png
          210 kB
        2. hebFile-gebbrish-fromAssignment-zip.png
          57 kB

          Issue Links

            Activity

            Hide
            poltawski Dan Poltawski added a comment -

            Taking integration held issues out of integration (whilst we are keeping master and 23_STABLE in sync).

            Show
            poltawski Dan Poltawski added a comment - Taking integration held issues out of integration (whilst we are keeping master and 23_STABLE in sync).
            Hide
            samhemelryk Sam Hemelryk added a comment -

            Again changes look good and appear to work perfectly thanks Petr. Has been integrated now.

            Show
            samhemelryk Sam Hemelryk added a comment - Again changes look good and appear to work perfectly thanks Petr. Has been integrated now.
            Hide
            skodak Petr Skoda added a comment -

            arrgggh, I have fixed a sloppy zipping regression caused by refactoring in this issue, please merge.

            Show
            skodak Petr Skoda added a comment - arrgggh, I have fixed a sloppy zipping regression caused by refactoring in this issue, please merge.
            Hide
            poltawski Dan Poltawski added a comment -

            Integrated, thanks

            Show
            poltawski Dan Poltawski added a comment - Integrated, thanks
            Hide
            rwijaya Rossiani Wijaya added a comment -

            This looks great.
            Thank for the quick fix.

            Test passed.

            Show
            rwijaya Rossiani Wijaya added a comment - This looks great. Thank for the quick fix. Test passed.
            Hide
            poltawski Dan Poltawski added a comment -

            Congratulations!

            You've made it into the weekly release!

            Thanks for your contribution - here are some random drummers to keep you inspired for the next week!
            http://www.youtube.com/watch?v=_QhpHUmVCmY

            Show
            poltawski Dan Poltawski added a comment - Congratulations! You've made it into the weekly release! Thanks for your contribution - here are some random drummers to keep you inspired for the next week! http://www.youtube.com/watch?v=_QhpHUmVCmY
            Show
            vadimon Vadim Dvorovenko added a comment - - edited 2.2 & 2.3 backport https://github.com/vadimonus/moodle/moodle.git Branches: MDL-33710 -33753-22 & MDL-33710 -33753-23 Diff: https://github.com/vadimonus/moodle/compare/4db36990ee032eef3b46d06179e85595fade3fbd...MDL-33710-33753-22 https://github.com/vadimonus/moodle/compare/32efb09e62b4a68299c6c9eaafe001cce517e5d0...MDL-33710-33753-23
            Hide
            mikidream miki Alliel added a comment -

            This is the zip file that I was trying to upload to moodle and to extract but the Hebrew filnames were disappeared from the filenames

            Show
            mikidream miki Alliel added a comment - This is the zip file that I was trying to upload to moodle and to extract but the Hebrew filnames were disappeared from the filenames
            Hide
            mikidream miki Alliel added a comment -

            Gebbrish filename instead of Hebrew filename after extracting zip file with assignments

            Show
            mikidream miki Alliel added a comment - Gebbrish filename instead of Hebrew filename after extracting zip file with assignments
            Hide
            mikidream miki Alliel added a comment -

            Hello
            I checked this on Moodle 2.3.2+ (Build: 20120927)
            I uploaded zip file (attached here) and extract it in moodle site the file contains 2 files which the name of the files are in Hebrew (utf8 lang) but after extracting the zip I saw that the names disappeared (infact one file includes hebrew name and a number (1) so only the number (1) stayed as the filename).

            Another problem as shown in the images attached , after clicking on "Download all assignments as a zip" the files with Hebrew names in the zip where Gibbrish!

            Hope there is solution
            Thanks

            Show
            mikidream miki Alliel added a comment - Hello I checked this on Moodle 2.3.2+ (Build: 20120927) I uploaded zip file (attached here) and extract it in moodle site the file contains 2 files which the name of the files are in Hebrew (utf8 lang) but after extracting the zip I saw that the names disappeared (infact one file includes hebrew name and a number (1) so only the number (1) stayed as the filename). Another problem as shown in the images attached , after clicking on "Download all assignments as a zip" the files with Hebrew names in the zip where Gibbrish! Hope there is solution Thanks
            Hide
            nadavkav Nadav Kavalerchik added a comment -

            I can confirm Miki's comments above when tested on an updated master branch (from 27-9-2012)

            Show
            nadavkav Nadav Kavalerchik added a comment - I can confirm Miki's comments above when tested on an updated master branch (from 27-9-2012)
            Hide
            skodak Petr Skoda added a comment -

            Hi, the attached file is not unicode compatible, sorry, you should use some different packer instead - such as 7zip. The built-in zipping in Windows OS is horrible.

            I am going to add Hebrew DOs charset fallback in MDL-35795, thanks for the sample zip file.

            Show
            skodak Petr Skoda added a comment - Hi, the attached file is not unicode compatible, sorry, you should use some different packer instead - such as 7zip. The built-in zipping in Windows OS is horrible. I am going to add Hebrew DOs charset fallback in MDL-35795 , thanks for the sample zip file.
            Hide
            skodak Petr Skoda added a comment -

            Could somebody please test my patch in MDL-35795? I have uploaded the attached file there too.

            Show
            skodak Petr Skoda added a comment - Could somebody please test my patch in MDL-35795 ? I have uploaded the attached file there too.
            Hide
            nadavkav Nadav Kavalerchik added a comment - - edited

            Petr,

            "the attached file is not unicode compatible"
            I know. that's way I attached it. so you can see what we have to deal with
            "The built-in zipping in Windows OS is horrible."
            Tell me about it

            Here, in the middle east, there is no way we can tell the teachers to use anything. (you can watch the News and see why)
            We are lucky if they use Chrome or Firefox and not IE.
            Most computer labs at School and Teacher Training Colleges are "managed" (locked down) and you can not install 7Zip.
            (I have tried. even wrote feature comparison document. but, nothing helps)

            We need to try to fix it on the server side. If we can.
            MDL-33068 Seems to do the trick. In addition, we should create a system setting for a default encoding. So the Teachers do not have to understand it and decide what to choose. And have the Administrator set it up for them. (In our, Hebrew, case it is WINDOWS-1255 or ISO-8859-8)

            Also, I have tested your beautiful solution on MDL-35795, which works great too!

            One of these solution should be implemented everywhere there is an option to download or upload a zip file ( File, Folder, Assignments... )

            Show
            nadavkav Nadav Kavalerchik added a comment - - edited Petr, "the attached file is not unicode compatible" I know. that's way I attached it. so you can see what we have to deal with "The built-in zipping in Windows OS is horrible." Tell me about it Here, in the middle east, there is no way we can tell the teachers to use anything. (you can watch the News and see why) We are lucky if they use Chrome or Firefox and not IE. Most computer labs at School and Teacher Training Colleges are "managed" (locked down) and you can not install 7Zip. (I have tried. even wrote feature comparison document. but, nothing helps) We need to try to fix it on the server side. If we can. MDL-33068 Seems to do the trick. In addition, we should create a system setting for a default encoding. So the Teachers do not have to understand it and decide what to choose. And have the Administrator set it up for them. (In our, Hebrew, case it is WINDOWS-1255 or ISO-8859-8) Also, I have tested your beautiful solution on MDL-35795 , which works great too! One of these solution should be implemented everywhere there is an option to download or upload a zip file ( File, Folder, Assignments... )
            Hide
            mikidream miki Alliel added a comment -

            downloadall in zip - Heb (utf8) file names are seen in Gibbrish after opening the zip file in windows

            Show
            mikidream miki Alliel added a comment - downloadall in zip - Heb (utf8) file names are seen in Gibbrish after opening the zip file in windows
            Hide
            mikidream miki Alliel added a comment -

            I'v just check this problem in this site
            http://qa.moodle.net

            which is moodle 2.4 as I understand.

            This problem still exists there!

            when downloading assignment in zip file (download all) the file names which were in Hebrew are in Gibberish when opening the zip file
            like the image I attach here
            thanks

            Show
            mikidream miki Alliel added a comment - I'v just check this problem in this site http://qa.moodle.net which is moodle 2.4 as I understand. This problem still exists there! when downloading assignment in zip file (download all) the file names which were in Hebrew are in Gibberish when opening the zip file like the image I attach here thanks
            Hide
            skodak Petr Skoda added a comment -

            Windows does not support Unicode in zip archives at all, it is no wonder it shows everything as Gibberish. Please use some real zip extractor, not the broken built-in unzipping. It is able to extract only borked archives which were created in Windows with the same DOS encoding, nothing else.

            Show
            skodak Petr Skoda added a comment - Windows does not support Unicode in zip archives at all, it is no wonder it shows everything as Gibberish. Please use some real zip extractor, not the broken built-in unzipping. It is able to extract only borked archives which were created in Windows with the same DOS encoding, nothing else.
            Hide
            mikidream miki Alliel added a comment -

            You are right , Im sorry,
            I just tried again to open this zip file with 7zip program ( on windows 7) and finally I saw the Hebrew file name letters.
            Thanks

            Show
            mikidream miki Alliel added a comment - You are right , Im sorry, I just tried again to open this zip file with 7zip program ( on windows 7) and finally I saw the Hebrew file name letters. Thanks

              People

              • Votes:
                3 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Fix Release Date:
                  3/Dec/12