Moodle
  1. Moodle
  2. MDL-22584

Upgrade from 1.9 with file path depth more then 255 characters

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0
    • Fix Version/s: 2.0.4, 2.1.1
    • Component/s: Resource
    • Labels:
    • Testing Instructions:
      Hide

      1/ install 1.9
      2/ create some course files in deep nested dirs that exceed 225 chars
      3/ create some files with more than 255 chars (OS X does not support this)
      4/ try upgrade

      Show
      1/ install 1.9 2/ create some course files in deep nested dirs that exceed 225 chars 3/ create some files with more than 255 chars (OS X does not support this) 4/ try upgrade
    • Affected Branches:
      MOODLE_20_STABLE
    • Fixed Branches:
      MOODLE_20_STABLE, MOODLE_21_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      w29_MDL-22584_m22_monsterpaths
    • Rank:
      18132

      Description

      With the possibility of have material many folders deep the filepath can be greater 255 characters. I suggest we need to increase this to 512 characters instead.

      alter table mdl_files alter column filepath type char(512);

      OR attached diff.

        Activity

        Hide
        Petr Škoda added a comment -

        I am afraid that char(512) is not officially supported in Moodle yet, I am going to add a warning into upgrade log and skip the files with too long path names, sorry

        Show
        Petr Škoda added a comment - I am afraid that char(512) is not officially supported in Moodle yet, I am going to add a warning into upgrade log and skip the files with too long path names, sorry
        Hide
        Petr Škoda added a comment -

        All files with filepath longer than 255 chars are now skipped during upgrade.
        Thank you for the report and patch.

        Petr Skoda

        Show
        Petr Škoda added a comment - All files with filepath longer than 255 chars are now skipped during upgrade. Thank you for the report and patch. Petr Skoda
        Hide
        Tim Atton added a comment -

        Just upgraded an got the skipped message for almost everything.
        e.g.
        "File path longer than 255 chars, skipping: C:\Program Files\Apache Software Foundation\Apache2.2\moodledata/1/Awardscheme/arrow.gif"

        clearly this is less than 255 characters (in fact its only 88)

        Tim Atton

        Show
        Tim Atton added a comment - Just upgraded an got the skipped message for almost everything. e.g. "File path longer than 255 chars, skipping: C:\Program Files\Apache Software Foundation\Apache2.2\moodledata/1/Awardscheme/arrow.gif" clearly this is less than 255 characters (in fact its only 88) Tim Atton
        Hide
        Petr Škoda added a comment -

        oops, sorry, should be fixed now, thanks for the report.

        Show
        Petr Škoda added a comment - oops, sorry, should be fixed now, thanks for the report.
        Hide
        Thomas Haines added a comment -

        I'm trying to upgrade from 1.9.11 to 2.0.2 and my CLI upgrade fails for this same reason. It appears as though these are not being skipped for me.

        Default exception handler: Error writing to database Debug: Data too long for column 'filepath' at row 1
        INSERT INTO mdl_files (contextid,component,filearea,itemid,filepath,filename,contenthash,filesize,timecreated,timemodified,mimetype,userid,pathnamehash) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?)
        [array (
        0 => '14903',
        1 => 'course',
        2 => 'legacy',
        3 => 0,
        4 => '/US_9/Quarter_1/Honors_Book_Project/Floyd ChainsPackage/Chains/Blog/Entries/2010/10/25_Isabel_6_Item_seventh__I_desire_that_my_executors_take_no_authority_over_the_negroes_Felix_and_Catharine,_that_I_do_not_wish_them_included_nor_counted_as_my_property._And_that_my_executors_are_not,_nor_shall_be_held_responsible_now_nor_h_files/',
        5 => '.',
        6 => 'da39a3ee5e6b4b0d3255bfef95601890afd80709',
        7 => 0,
        8 => 1301070314,
        9 => 1301070314,
        10 => NULL,
        11 => NULL,
        12 => '391dd00615c14dba2a61369f0bd84db171f16aa9',
        )]

        • line 394 of /lib/dml/moodle_database.php: dml_write_exception thrown
        • line 874 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
        • line 916 of /lib/dml/mysqli_native_moodle_database.php: call to mysqli_native_moodle_database->insert_record_raw()
        • line 542 of /lib/filestorage/file_storage.php: call to mysqli_native_moodle_database->insert_record()
        • line 356 of /lib/db/upgradelib.php: call to file_storage->create_directory()
        • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
        • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
        • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
        • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
        • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
        • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
        • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
        • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
        • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
        • line 98 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
        • line 593 of /lib/db/upgrade.php: call to upgrade_migrate_files_courses()
        • line 1383 of /lib/upgradelib.php: call to xmldb_main_upgrade()
        • line 141 of /admin/cli/upgrade.php: call to upgrade_core()
        Show
        Thomas Haines added a comment - I'm trying to upgrade from 1.9.11 to 2.0.2 and my CLI upgrade fails for this same reason. It appears as though these are not being skipped for me. Default exception handler: Error writing to database Debug: Data too long for column 'filepath' at row 1 INSERT INTO mdl_files (contextid,component,filearea,itemid,filepath,filename,contenthash,filesize,timecreated,timemodified,mimetype,userid,pathnamehash) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?) [array ( 0 => '14903', 1 => 'course', 2 => 'legacy', 3 => 0, 4 => '/US_9/Quarter_1/Honors_Book_Project/Floyd ChainsPackage/Chains/Blog/Entries/2010/10/25_Isabel_6_Item_seventh__I_desire_that_my_executors_take_no_authority_over_the_negroes_Felix_and_Catharine,_that_I_do_not_wish_them_included_nor_counted_as_my_property._And_that_my_executors_are_not,_nor_shall_be_held_responsible_now_nor_h_files/', 5 => '.', 6 => 'da39a3ee5e6b4b0d3255bfef95601890afd80709', 7 => 0, 8 => 1301070314, 9 => 1301070314, 10 => NULL, 11 => NULL, 12 => '391dd00615c14dba2a61369f0bd84db171f16aa9', )] line 394 of /lib/dml/moodle_database.php: dml_write_exception thrown line 874 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end() line 916 of /lib/dml/mysqli_native_moodle_database.php: call to mysqli_native_moodle_database->insert_record_raw() line 542 of /lib/filestorage/file_storage.php: call to mysqli_native_moodle_database->insert_record() line 356 of /lib/db/upgradelib.php: call to file_storage->create_directory() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 98 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 593 of /lib/db/upgrade.php: call to upgrade_migrate_files_courses() line 1383 of /lib/upgradelib.php: call to xmldb_main_upgrade() line 141 of /admin/cli/upgrade.php: call to upgrade_core()
        Hide
        Thomas Haines added a comment -

        In a test environment I've altered the table so the filepath column has a length of 512, and only then do I see the "!! File path longer than 255 chars, skipping: /moodledata/ecougar.kasd.org/261/US_9/Quarter_1/Honors_Book_Project/Floyd ChainsPackage/Chains/Blog/Entries/2010/10/25_Isabel_6_Item_seventh__I_desire_that_my_executors_take_no_authority_over_the_negroes_Felix_and_Catharine,_that_I_do_not_wish_them_included_nor_counted_as_my_property._And_that_my_executors_are_not,_nor_shall_be_held_responsible_now_nor_h_files/5.png !!" warning.

        Show
        Thomas Haines added a comment - In a test environment I've altered the table so the filepath column has a length of 512, and only then do I see the "!! File path longer than 255 chars, skipping: /moodledata/ecougar.kasd.org/261/US_9/Quarter_1/Honors_Book_Project/Floyd ChainsPackage/Chains/Blog/Entries/2010/10/25_Isabel_6_Item_seventh__I_desire_that_my_executors_take_no_authority_over_the_negroes_Felix_and_Catharine,_that_I_do_not_wish_them_included_nor_counted_as_my_property._And_that_my_executors_are_not,_nor_shall_be_held_responsible_now_nor_h_files/5.png !!" warning.
        Hide
        Chris Follin added a comment -

        We're having this issue with clients. Skipping the files is not really acceptable in the absence of some other workaround. Additionally, similar to Thomas' comment from March, the code is not skipping properly for an upgrade that is executed via CLI.

        Show
        Chris Follin added a comment - We're having this issue with clients. Skipping the files is not really acceptable in the absence of some other workaround. Additionally, similar to Thomas' comment from March, the code is not skipping properly for an upgrade that is executed via CLI.
        Hide
        Petr Škoda added a comment -

        What solution do you propose? The db field can not be extended because our db layer does not support varchars longer than 255, we can not use text field because we could not index them, it would be slower and take more space. I doubt we can shorten them automatically because we do not know if it actually is allowed. I suppose any workaround might have to be done before the upgrade.

        Show
        Petr Škoda added a comment - What solution do you propose? The db field can not be extended because our db layer does not support varchars longer than 255, we can not use text field because we could not index them, it would be slower and take more space. I doubt we can shorten them automatically because we do not know if it actually is allowed. I suppose any workaround might have to be done before the upgrade.
        Hide
        Chris Follin added a comment -

        Petr,
        If it can't be fixed then skipping the files is okay as long as there is a workaround to retrieve the files and move them manually. We're thinking that perhaps the skipped files could be copied into one folder per course within the server files or some other file area so that they are at least accessible. Numbers may need to be appended to filenames if consolidating down to a single folder causes name conflicts. Something based on a timestamp could probably be used for that. The upgrade process should send a notification to site admins to essentially say, "The following files were skipped by the upgrade because the paths were too long but here's where you'll find the files if you need them." The burden is then on the user to retrieve them but they have the option to do so rather than the files just being skipped and that's that.

        The second issue is that the CLI upgrade is still broken. The skipping is not working via CLI. Regardless of the final decision for what to do with skipped files, the CLI needs to be addressed.

        Show
        Chris Follin added a comment - Petr, If it can't be fixed then skipping the files is okay as long as there is a workaround to retrieve the files and move them manually. We're thinking that perhaps the skipped files could be copied into one folder per course within the server files or some other file area so that they are at least accessible. Numbers may need to be appended to filenames if consolidating down to a single folder causes name conflicts. Something based on a timestamp could probably be used for that. The upgrade process should send a notification to site admins to essentially say, "The following files were skipped by the upgrade because the paths were too long but here's where you'll find the files if you need them." The burden is then on the user to retrieve them but they have the option to do so rather than the files just being skipped and that's that. The second issue is that the CLI upgrade is still broken. The skipping is not working via CLI. Regardless of the final decision for what to do with skipped files, the CLI needs to be addressed.
        Hide
        Petr Škoda added a comment - - edited

        Thanks, reopening, it is a good idea to copy the files to special folder. Unfortunately you may not be able to update the html linking of the files because there is no file manager inside tinymce editor (but it is a separate issue). I am going to fix the CLI at the same time.

        Show
        Petr Škoda added a comment - - edited Thanks, reopening, it is a good idea to copy the files to special folder. Unfortunately you may not be able to update the html linking of the files because there is no file manager inside tinymce editor (but it is a separate issue). I am going to fix the CLI at the same time.
        Hide
        Petr Škoda added a comment -

        Hello,
        the problem was not in CLI, I have added support for eversized individual dir and file names. Hopefully it should never stop now during the upgrade.

        If this gets integrated please file new issues for any remaining problems.

        Thanks for the report and cooperation.

        Petr

        Show
        Petr Škoda added a comment - Hello, the problem was not in CLI, I have added support for eversized individual dir and file names. Hopefully it should never stop now during the upgrade. If this gets integrated please file new issues for any remaining problems. Thanks for the report and cooperation. Petr
        Hide
        Sam Hemelryk added a comment -

        Thanks Petr, this has been integrated now.
        The end solution looks pretty good, as requested by Petr please if you find any further bugs please create new issues.

        Cheers
        Sam

        Show
        Sam Hemelryk added a comment - Thanks Petr, this has been integrated now. The end solution looks pretty good, as requested by Petr please if you find any further bugs please create new issues. Cheers Sam
        Hide
        Aparup Banerjee added a comment - - edited

        yup this worked for me.

        results seen:
        File path longer than 255 chars '/home/aparup/mdata/19/mysql/2/1/a/b/c/d/e/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/10codes.pdf', file path truncated to '/directory_over_255_chars/8503e250e9ca1bb892659926798fabb1/10codes.pdf'

        Show
        Aparup Banerjee added a comment - - edited yup this worked for me. results seen: File path longer than 255 chars '/home/aparup/mdata/19/mysql/2/1/a/b/c/d/e/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/10codes.pdf', file path truncated to '/directory_over_255_chars/8503e250e9ca1bb892659926798fabb1/10codes.pdf'
        Hide
        Petr Škoda added a comment -

        Thanks everybody, this is now part of the weekly build.

        Show
        Petr Škoda added a comment - Thanks everybody, this is now part of the weekly build.

          People

          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: