Moodle
  1. Moodle
  2. MDL-22584

Upgrade from 1.9 with file path depth more then 255 characters

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0
    • Fix Version/s: 2.0.4, 2.1.1
    • Component/s: Resource
    • Labels:
    • Testing Instructions:
      Hide

      1/ install 1.9
      2/ create some course files in deep nested dirs that exceed 225 chars
      3/ create some files with more than 255 chars (OS X does not support this)
      4/ try upgrade

      Show
      1/ install 1.9 2/ create some course files in deep nested dirs that exceed 225 chars 3/ create some files with more than 255 chars (OS X does not support this) 4/ try upgrade
    • Affected Branches:
      MOODLE_20_STABLE
    • Fixed Branches:
      MOODLE_20_STABLE, MOODLE_21_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      w29_MDL-22584_m22_monsterpaths

      Description

      With the possibility of have material many folders deep the filepath can be greater 255 characters. I suggest we need to increase this to 512 characters instead.

      alter table mdl_files alter column filepath type char(512);

      OR attached diff.

        Gliffy Diagrams

          Activity

          Hide
          Petr Skoda added a comment -

          I am afraid that char(512) is not officially supported in Moodle yet, I am going to add a warning into upgrade log and skip the files with too long path names, sorry

          Show
          Petr Skoda added a comment - I am afraid that char(512) is not officially supported in Moodle yet, I am going to add a warning into upgrade log and skip the files with too long path names, sorry
          Hide
          Petr Skoda added a comment -

          All files with filepath longer than 255 chars are now skipped during upgrade.
          Thank you for the report and patch.

          Petr Skoda

          Show
          Petr Skoda added a comment - All files with filepath longer than 255 chars are now skipped during upgrade. Thank you for the report and patch. Petr Skoda
          Hide
          Tim Atton added a comment -

          Just upgraded an got the skipped message for almost everything.
          e.g.
          "File path longer than 255 chars, skipping: C:\Program Files\Apache Software Foundation\Apache2.2\moodledata/1/Awardscheme/arrow.gif"

          clearly this is less than 255 characters (in fact its only 88)

          Tim Atton

          Show
          Tim Atton added a comment - Just upgraded an got the skipped message for almost everything. e.g. "File path longer than 255 chars, skipping: C:\Program Files\Apache Software Foundation\Apache2.2\moodledata/1/Awardscheme/arrow.gif" clearly this is less than 255 characters (in fact its only 88) Tim Atton
          Hide
          Petr Skoda added a comment -

          oops, sorry, should be fixed now, thanks for the report.

          Show
          Petr Skoda added a comment - oops, sorry, should be fixed now, thanks for the report.
          Hide
          Thomas Haines added a comment -

          I'm trying to upgrade from 1.9.11 to 2.0.2 and my CLI upgrade fails for this same reason. It appears as though these are not being skipped for me.

          Default exception handler: Error writing to database Debug: Data too long for column 'filepath' at row 1
          INSERT INTO mdl_files (contextid,component,filearea,itemid,filepath,filename,contenthash,filesize,timecreated,timemodified,mimetype,userid,pathnamehash) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?)
          [array (
          0 => '14903',
          1 => 'course',
          2 => 'legacy',
          3 => 0,
          4 => '/US_9/Quarter_1/Honors_Book_Project/Floyd ChainsPackage/Chains/Blog/Entries/2010/10/25_Isabel_6_Item_seventh__I_desire_that_my_executors_take_no_authority_over_the_negroes_Felix_and_Catharine,_that_I_do_not_wish_them_included_nor_counted_as_my_property._And_that_my_executors_are_not,_nor_shall_be_held_responsible_now_nor_h_files/',
          5 => '.',
          6 => 'da39a3ee5e6b4b0d3255bfef95601890afd80709',
          7 => 0,
          8 => 1301070314,
          9 => 1301070314,
          10 => NULL,
          11 => NULL,
          12 => '391dd00615c14dba2a61369f0bd84db171f16aa9',
          )]

          • line 394 of /lib/dml/moodle_database.php: dml_write_exception thrown
          • line 874 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end()
          • line 916 of /lib/dml/mysqli_native_moodle_database.php: call to mysqli_native_moodle_database->insert_record_raw()
          • line 542 of /lib/filestorage/file_storage.php: call to mysqli_native_moodle_database->insert_record()
          • line 356 of /lib/db/upgradelib.php: call to file_storage->create_directory()
          • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
          • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
          • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
          • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
          • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
          • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
          • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
          • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
          • line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
          • line 98 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course()
          • line 593 of /lib/db/upgrade.php: call to upgrade_migrate_files_courses()
          • line 1383 of /lib/upgradelib.php: call to xmldb_main_upgrade()
          • line 141 of /admin/cli/upgrade.php: call to upgrade_core()
          Show
          Thomas Haines added a comment - I'm trying to upgrade from 1.9.11 to 2.0.2 and my CLI upgrade fails for this same reason. It appears as though these are not being skipped for me. Default exception handler: Error writing to database Debug: Data too long for column 'filepath' at row 1 INSERT INTO mdl_files (contextid,component,filearea,itemid,filepath,filename,contenthash,filesize,timecreated,timemodified,mimetype,userid,pathnamehash) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?) [array ( 0 => '14903', 1 => 'course', 2 => 'legacy', 3 => 0, 4 => '/US_9/Quarter_1/Honors_Book_Project/Floyd ChainsPackage/Chains/Blog/Entries/2010/10/25_Isabel_6_Item_seventh__I_desire_that_my_executors_take_no_authority_over_the_negroes_Felix_and_Catharine,_that_I_do_not_wish_them_included_nor_counted_as_my_property._And_that_my_executors_are_not,_nor_shall_be_held_responsible_now_nor_h_files/', 5 => '.', 6 => 'da39a3ee5e6b4b0d3255bfef95601890afd80709', 7 => 0, 8 => 1301070314, 9 => 1301070314, 10 => NULL, 11 => NULL, 12 => '391dd00615c14dba2a61369f0bd84db171f16aa9', )] line 394 of /lib/dml/moodle_database.php: dml_write_exception thrown line 874 of /lib/dml/mysqli_native_moodle_database.php: call to moodle_database->query_end() line 916 of /lib/dml/mysqli_native_moodle_database.php: call to mysqli_native_moodle_database->insert_record_raw() line 542 of /lib/filestorage/file_storage.php: call to mysqli_native_moodle_database->insert_record() line 356 of /lib/db/upgradelib.php: call to file_storage->create_directory() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 360 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 98 of /lib/db/upgradelib.php: call to upgrade_migrate_files_course() line 593 of /lib/db/upgrade.php: call to upgrade_migrate_files_courses() line 1383 of /lib/upgradelib.php: call to xmldb_main_upgrade() line 141 of /admin/cli/upgrade.php: call to upgrade_core()
          Hide
          Thomas Haines added a comment -

          In a test environment I've altered the table so the filepath column has a length of 512, and only then do I see the "!! File path longer than 255 chars, skipping: /moodledata/ecougar.kasd.org/261/US_9/Quarter_1/Honors_Book_Project/Floyd ChainsPackage/Chains/Blog/Entries/2010/10/25_Isabel_6_Item_seventh__I_desire_that_my_executors_take_no_authority_over_the_negroes_Felix_and_Catharine,_that_I_do_not_wish_them_included_nor_counted_as_my_property._And_that_my_executors_are_not,_nor_shall_be_held_responsible_now_nor_h_files/5.png !!" warning.

          Show
          Thomas Haines added a comment - In a test environment I've altered the table so the filepath column has a length of 512, and only then do I see the "!! File path longer than 255 chars, skipping: /moodledata/ecougar.kasd.org/261/US_9/Quarter_1/Honors_Book_Project/Floyd ChainsPackage/Chains/Blog/Entries/2010/10/25_Isabel_6_Item_seventh__I_desire_that_my_executors_take_no_authority_over_the_negroes_Felix_and_Catharine,_that_I_do_not_wish_them_included_nor_counted_as_my_property._And_that_my_executors_are_not,_nor_shall_be_held_responsible_now_nor_h_files/5.png !!" warning.
          Hide
          Chris Follin added a comment -

          We're having this issue with clients. Skipping the files is not really acceptable in the absence of some other workaround. Additionally, similar to Thomas' comment from March, the code is not skipping properly for an upgrade that is executed via CLI.

          Show
          Chris Follin added a comment - We're having this issue with clients. Skipping the files is not really acceptable in the absence of some other workaround. Additionally, similar to Thomas' comment from March, the code is not skipping properly for an upgrade that is executed via CLI.
          Hide
          Petr Skoda added a comment -

          What solution do you propose? The db field can not be extended because our db layer does not support varchars longer than 255, we can not use text field because we could not index them, it would be slower and take more space. I doubt we can shorten them automatically because we do not know if it actually is allowed. I suppose any workaround might have to be done before the upgrade.

          Show
          Petr Skoda added a comment - What solution do you propose? The db field can not be extended because our db layer does not support varchars longer than 255, we can not use text field because we could not index them, it would be slower and take more space. I doubt we can shorten them automatically because we do not know if it actually is allowed. I suppose any workaround might have to be done before the upgrade.
          Hide
          Chris Follin added a comment -

          Petr,
          If it can't be fixed then skipping the files is okay as long as there is a workaround to retrieve the files and move them manually. We're thinking that perhaps the skipped files could be copied into one folder per course within the server files or some other file area so that they are at least accessible. Numbers may need to be appended to filenames if consolidating down to a single folder causes name conflicts. Something based on a timestamp could probably be used for that. The upgrade process should send a notification to site admins to essentially say, "The following files were skipped by the upgrade because the paths were too long but here's where you'll find the files if you need them." The burden is then on the user to retrieve them but they have the option to do so rather than the files just being skipped and that's that.

          The second issue is that the CLI upgrade is still broken. The skipping is not working via CLI. Regardless of the final decision for what to do with skipped files, the CLI needs to be addressed.

          Show
          Chris Follin added a comment - Petr, If it can't be fixed then skipping the files is okay as long as there is a workaround to retrieve the files and move them manually. We're thinking that perhaps the skipped files could be copied into one folder per course within the server files or some other file area so that they are at least accessible. Numbers may need to be appended to filenames if consolidating down to a single folder causes name conflicts. Something based on a timestamp could probably be used for that. The upgrade process should send a notification to site admins to essentially say, "The following files were skipped by the upgrade because the paths were too long but here's where you'll find the files if you need them." The burden is then on the user to retrieve them but they have the option to do so rather than the files just being skipped and that's that. The second issue is that the CLI upgrade is still broken. The skipping is not working via CLI. Regardless of the final decision for what to do with skipped files, the CLI needs to be addressed.
          Hide
          Petr Skoda added a comment - - edited

          Thanks, reopening, it is a good idea to copy the files to special folder. Unfortunately you may not be able to update the html linking of the files because there is no file manager inside tinymce editor (but it is a separate issue). I am going to fix the CLI at the same time.

          Show
          Petr Skoda added a comment - - edited Thanks, reopening, it is a good idea to copy the files to special folder. Unfortunately you may not be able to update the html linking of the files because there is no file manager inside tinymce editor (but it is a separate issue). I am going to fix the CLI at the same time.
          Hide
          Petr Skoda added a comment -

          Hello,
          the problem was not in CLI, I have added support for eversized individual dir and file names. Hopefully it should never stop now during the upgrade.

          If this gets integrated please file new issues for any remaining problems.

          Thanks for the report and cooperation.

          Petr

          Show
          Petr Skoda added a comment - Hello, the problem was not in CLI, I have added support for eversized individual dir and file names. Hopefully it should never stop now during the upgrade. If this gets integrated please file new issues for any remaining problems. Thanks for the report and cooperation. Petr
          Hide
          Sam Hemelryk added a comment -

          Thanks Petr, this has been integrated now.
          The end solution looks pretty good, as requested by Petr please if you find any further bugs please create new issues.

          Cheers
          Sam

          Show
          Sam Hemelryk added a comment - Thanks Petr, this has been integrated now. The end solution looks pretty good, as requested by Petr please if you find any further bugs please create new issues. Cheers Sam
          Hide
          Aparup Banerjee added a comment - - edited

          yup this worked for me.

          results seen:
          File path longer than 255 chars '/home/aparup/mdata/19/mysql/2/1/a/b/c/d/e/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/10codes.pdf', file path truncated to '/directory_over_255_chars/8503e250e9ca1bb892659926798fabb1/10codes.pdf'

          Show
          Aparup Banerjee added a comment - - edited yup this worked for me. results seen: File path longer than 255 chars '/home/aparup/mdata/19/mysql/2/1/a/b/c/d/e/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/12345678901234567890123456789012345678901234567890/10codes.pdf', file path truncated to '/directory_over_255_chars/8503e250e9ca1bb892659926798fabb1/10codes.pdf'
          Hide
          Petr Skoda added a comment -

          Thanks everybody, this is now part of the weekly build.

          Show
          Petr Skoda added a comment - Thanks everybody, this is now part of the weekly build.

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: