Moodle
  1. Moodle
  2. MDL-18799

Backup/Restore problem. ID don´t change on resource or image links

    Details

    • Rank:
      31782

      Description

      I'm using a moodle instalation 1.9.4. from 3 weeks ago. I'm having the
      following problem:

      When restoring a course into a new course, the image and resource links
      are broken. Looking the links I can see that it's calling the old course
      id, not the new one. For example:

      a course with id= 35 have the image links like:
      http://avirtual.forempv.ccoo.es/file.php?file=%2F32%2Fimage.jpg, where 32
      is the id of the old course from wich I did the backup.

      I saw several posts with more or less the same problem, but I could'nt see
      any clear solution or could'nt see any explanation that there isn´t a
      solution now to this problem. I could'nt find this on the tracker.

      I would be glad to know if there is any solution to this, because i and
      I'm having to change by hand the id number of all the images and resources.

      Thank's in advance

        Issue Links

          Activity

          Hide
          Eloy Lafuente (stronk7) added a comment -

          Hi Daniel,

          thanks a lot for sharing your backup example!

          I've been analysing it and I've found that it includes links like this:

          http://yourserver/file.php?file=%2F260%2Fimage.jpg

          And, IMO, that's the cause of your problem, those %2F encoded chars that should be normal and simple slashes.

          The key is that backup uses to transform any link of the form:

          http://yourserver/file.php?file=/COURSEID to something called $@FILEPHP@$ and later, restore is able to transform back all those $@FILEPHP@$ to the new URL with the new courseid.

          In your case, those %2F are breaking the transformation on backup, hence restore hasn't anything to transform back, so your original URLs are left unmodified.

          While I think (good news) this is easy to fix (in backup process, by looking also to all those %2F chars... I'm very interested about to KNOW how those horrible encoded chars have ended there. Can you, please, explain how have you added those images?

          • Copying them from the file manager?
          • Using the HTML Editor? The "official" one or any alternative?
          • Using plain text editor and typing the HTML by hand?
          • ....

          As said, I think we can modify (and will do) the backup procedure to be able to detect those chars easily but we should prevent them to be there at origin.

          ANy info will be welcome, TIA! Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - Hi Daniel, thanks a lot for sharing your backup example! I've been analysing it and I've found that it includes links like this: http://yourserver/file.php?file=%2F260%2Fimage.jpg And, IMO, that's the cause of your problem, those %2F encoded chars that should be normal and simple slashes. The key is that backup uses to transform any link of the form: http://yourserver/file.php?file=/COURSEID to something called $@FILEPHP@$ and later, restore is able to transform back all those $@FILEPHP@$ to the new URL with the new courseid. In your case, those %2F are breaking the transformation on backup, hence restore hasn't anything to transform back, so your original URLs are left unmodified. While I think (good news) this is easy to fix (in backup process, by looking also to all those %2F chars... I'm very interested about to KNOW how those horrible encoded chars have ended there. Can you, please, explain how have you added those images? Copying them from the file manager? Using the HTML Editor? The "official" one or any alternative? Using plain text editor and typing the HTML by hand? .... As said, I think we can modify (and will do) the backup procedure to be able to detect those chars easily but we should prevent them to be there at origin. ANy info will be welcome, TIA! Ciao
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Oki, these are the possibilities we can find right now in Moodle 1.9, expanded to all the possible switches:

          • file.php/CID/path/to/document.html#someanchor?forcedownload=1
          • file.php?file=%2FCID%2Fpath%2Fto%2Fdocument.html#someanchor&forcedownload=1

          Where CID is the course id, someanchor is one optional anchor in the URL and forcedownload is one optional setting.

          To make this work in backup / restore we need to transform those URLs to something neutral, not dependent of the slash settings in the origin and target sites. So the proposal is to convert those links to:

          $@FILEPHP@$$@SLASH@$path$@SLASH@$to$@SLASH@$document.html#someanchor$@FORCEDOWNLOAD@$

          That format is independent of the slash-arguments setting and can be easily converted back to proper URLs by replacing:
          $@FILEPHP@$ => new wwwroot + file.php + courseid in the target server
          $@SLASH@$ => slashes or encoded slashes in the target server
          $@FORCEDOWNLOAD@$ => forcedownload parameter in the target server

          Going to try that approach. Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - Oki, these are the possibilities we can find right now in Moodle 1.9, expanded to all the possible switches: file.php/CID/path/to/document.html#someanchor?forcedownload=1 file.php?file=%2FCID%2Fpath%2Fto%2Fdocument.html#someanchor&forcedownload=1 Where CID is the course id, someanchor is one optional anchor in the URL and forcedownload is one optional setting. To make this work in backup / restore we need to transform those URLs to something neutral, not dependent of the slash settings in the origin and target sites. So the proposal is to convert those links to: $@FILEPHP@$$@SLASH@$path$@SLASH@$to$@SLASH@$document.html#someanchor$@FORCEDOWNLOAD@$ That format is independent of the slash-arguments setting and can be easily converted back to proper URLs by replacing: $@FILEPHP@$ => new wwwroot + file.php + courseid in the target server $@SLASH@$ => slashes or encoded slashes in the target server $@FORCEDOWNLOAD@$ => forcedownload parameter in the target server Going to try that approach. Ciao
          Hide
          Eloy Lafuente (stronk7) added a comment - - edited

          Annotating here this REGEXP (general URL extractor) to have it present in this bug.

          $pattern = '/\b(https|http):\/\/([\pL-:@.0-9])(\/(??:([-;:@#&=\pL0-9\$~_.+!*\',]?))|[-;:@#&=\pL0-9\$~_.!\',]|%[a-fA-F0-9]

          {2}|\/))?(?:?((??:([-;:@#&=\pL0-9\$_.+!*\',]?))|[-;:@#&=?\pL0-9\$_.+!*\',]|%[a-fA-F0-9]{2}

          |\/)*))?(?<![,.;])/';

          Edited to add allowed ~ and some changes in quotes.

          Show
          Eloy Lafuente (stronk7) added a comment - - edited Annotating here this REGEXP (general URL extractor) to have it present in this bug. $pattern = '/\b(https|http):\/\/( [\pL-:@.0-9] )(\/(? ?:( [-;:@#&=\pL0-9\$~_.+!*\',] ?))|[-;:@#&=\pL0-9\$~_. ! \',]|% [a-fA-F0-9] {2}|\/) )?(?:?((? ?:( [-;:@#&=\pL0-9\$_.+!*\',] ?))|[-;:@#&=?\pL0-9\$_.+!*\',]|% [a-fA-F0-9] {2} |\/)*))?(?<! [,.;] )/'; Edited to add allowed ~ and some changes in quotes.
          Hide
          Allen Cole added a comment -

          Is this a "fix" yet? We have been dealing with this issue for 8 months - every time we restore a course - even to the same server. I am find this same issue. It looks lke you are still considering this way of fixing this issue. Love to have this resolved. Will this be in Moodle 1.9.5 (by any chance)?

          Thanks for looking into this.

          Show
          Allen Cole added a comment - Is this a "fix" yet? We have been dealing with this issue for 8 months - every time we restore a course - even to the same server. I am find this same issue. It looks lke you are still considering this way of fixing this issue. Love to have this resolved. Will this be in Moodle 1.9.5 (by any chance)? Thanks for looking into this.
          Hide
          Eloy Lafuente (stronk7) added a comment - - edited

          Fix implemented in 19_STABLE and HEAD. Files changed in CVS for Moodle 1.9 are these:

          backup/backuplib.php: http://cvs.moodle.org/moodle/backup/backuplib.php?view=markup&pathrev=MOODLE_19_STABLE
          backup/restorelib.php: http://cvs.moodle.org/moodle/backup/restorelib.php?view=markup&pathrev=MOODLE_19_STABLE

          They are available in CVS and will be included in next Moodle 1.9 weekly release (next Wednesday, 22th April 2009).

          The approach followed consists into saving those links in a neutral format in backup, so restore will be able to construct them back to correct links (pointing to new course files).

          Note the bug "only" affected sites running with slasharguments off, and also note it's very interesting to have that setting enabled in order to get relative links (resources, scorm...) working properly.

          Please test, by performing new backup and restore. Any feedback will be really welcome.

          I'll leave this open some days... ciao

          Show
          Eloy Lafuente (stronk7) added a comment - - edited Fix implemented in 19_STABLE and HEAD. Files changed in CVS for Moodle 1.9 are these: backup/backuplib.php: http://cvs.moodle.org/moodle/backup/backuplib.php?view=markup&pathrev=MOODLE_19_STABLE backup/restorelib.php: http://cvs.moodle.org/moodle/backup/restorelib.php?view=markup&pathrev=MOODLE_19_STABLE They are available in CVS and will be included in next Moodle 1.9 weekly release (next Wednesday, 22th April 2009). The approach followed consists into saving those links in a neutral format in backup, so restore will be able to construct them back to correct links (pointing to new course files). Note the bug "only" affected sites running with slasharguments off, and also note it's very interesting to have that setting enabled in order to get relative links (resources, scorm...) working properly. Please test, by performing new backup and restore. Any feedback will be really welcome. I'll leave this open some days... ciao
          Hide
          Daniel Costa added a comment -

          Hi Eloy,

          I downloaded the files:

          backup/backuplib.php: http://cvs.moodle.org/moodle/backup/backuplib.php?view=markup&pathrev=MOODLE_19_STABLE
          backup/restorelib.php: http://cvs.moodle.org/moodle/backup/restorelib.php?view=markup&pathrev=MOODLE_19_STABLE

          uploaded them and tried to make a backup and a restore of a course ... and it didn't work. It looks like something is missing because when I started the restore procedure all the course information was missing. It created an empty course in a weekly format. The source course was a course in topic format. All content was missing.

          I'm looking forward to see this fix

          Thank's for your work and attention.

          Greetings

          Daniel

          Show
          Daniel Costa added a comment - Hi Eloy, I downloaded the files: backup/backuplib.php: http://cvs.moodle.org/moodle/backup/backuplib.php?view=markup&pathrev=MOODLE_19_STABLE backup/restorelib.php: http://cvs.moodle.org/moodle/backup/restorelib.php?view=markup&pathrev=MOODLE_19_STABLE uploaded them and tried to make a backup and a restore of a course ... and it didn't work. It looks like something is missing because when I started the restore procedure all the course information was missing. It created an empty course in a weekly format. The source course was a course in topic format. All content was missing. I'm looking forward to see this fix Thank's for your work and attention. Greetings Daniel
          Hide
          Eloy Lafuente (stronk7) added a comment - - edited

          Hi Daniel,

          I assume you are running Moodle 1.9.4+ (recent build) correct?

          Using those two files:

          backup/backuplib.php: http://cvs.moodle.org/moodle/backup/backuplib.php?view=markup&pathrev=MOODLE_19_STABLE
          backup/restorelib.php: http://cvs.moodle.org/moodle/backup/restorelib.php?view=markup&pathrev=MOODLE_19_STABLE

          within older Moodle builds can lead to "strange" problems, like the one commented by you above, that I think it's 100% unrelated with the fix applied to solve the links problem.

          Perhaps the best way to be 100% sure is:

          • For now, revert to your previous backuplib and restorelib files.
          • In 14 hours from now... there will be newer downloads available at: http://download.moodle.org , pick the 19_WEEKLY one and perform a complete update of your site. That way you'll have the whole moodle running correct versions.
          • Test the original links problem, should be fixed.
          • If you find any problem, please enable debug to developer level and execute the backup/restore again, attaching complete output here.

          Apologises by putting there those 2 links, as it's obvious can cause problems if running non up-to-date Moodle versions.

          Please, always upgrade Moodle as a whole (straight from CVS or from downloads.moodle.org), but as a whole.

          Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - - edited Hi Daniel, I assume you are running Moodle 1.9.4+ (recent build) correct? Using those two files: backup/backuplib.php: http://cvs.moodle.org/moodle/backup/backuplib.php?view=markup&pathrev=MOODLE_19_STABLE backup/restorelib.php: http://cvs.moodle.org/moodle/backup/restorelib.php?view=markup&pathrev=MOODLE_19_STABLE within older Moodle builds can lead to "strange" problems, like the one commented by you above, that I think it's 100% unrelated with the fix applied to solve the links problem. Perhaps the best way to be 100% sure is: For now, revert to your previous backuplib and restorelib files. In 14 hours from now... there will be newer downloads available at: http://download.moodle.org , pick the 19_WEEKLY one and perform a complete update of your site. That way you'll have the whole moodle running correct versions. Test the original links problem, should be fixed. If you find any problem, please enable debug to developer level and execute the backup/restore again, attaching complete output here. Apologises by putting there those 2 links, as it's obvious can cause problems if running non up-to-date Moodle versions. Please, always upgrade Moodle as a whole (straight from CVS or from downloads.moodle.org), but as a whole. Ciao
          Hide
          Daniel Costa added a comment -

          Hi Eloy,

          Ok, thank's for the quick answer

          greetings

          Daniel

          Show
          Daniel Costa added a comment - Hi Eloy, Ok, thank's for the quick answer greetings Daniel
          Hide
          Eloy Lafuente (stronk7) added a comment - - edited

          Changed the regexp expression above to be PHP 4.3.x compatible (we were using some PHP 4.4 features and Moodle 1.9 must work with PHP 4.3). Keeping HEAD without modifications.

          Ciao

          Edited: Ref URL about this problem: http://moodle.org/mod/forum/discuss.php?d=121946

          Show
          Eloy Lafuente (stronk7) added a comment - - edited Changed the regexp expression above to be PHP 4.3.x compatible (we were using some PHP 4.4 features and Moodle 1.9 must work with PHP 4.3). Keeping HEAD without modifications. Ciao Edited: Ref URL about this problem: http://moodle.org/mod/forum/discuss.php?d=121946
          Hide
          Eloy Lafuente (stronk7) added a comment - - edited

          It seems that some PHP 5.2.x server already have the same problem, with unicode modifiers not being allowed in regular expressions.

          Looking for info, it seems that it's a problem in the underlying OS regexp libraries, being compiled without unicode support.

          So I've modified both 19_STABLE and HEAD to do some checking of the unicode modifiers to decide what regular expression to use. Code looks now like this:

          $unicoderegexp = @preg_match('/\pL/u', 'a'); // This will fail silenty, returning false ,if regexp libraries don't support unicode
          if ($unicoderegexp) { // We can use unicode modifiers
              $regexp = 'whatever_using_unicode_modifiers';
          } else { // We cannot ue unicode modifiers
              $regexp = 'whatever_NOT_using_unicode_modifiers';
          }
          
          Show
          Eloy Lafuente (stronk7) added a comment - - edited It seems that some PHP 5.2.x server already have the same problem, with unicode modifiers not being allowed in regular expressions. Looking for info, it seems that it's a problem in the underlying OS regexp libraries, being compiled without unicode support. So I've modified both 19_STABLE and HEAD to do some checking of the unicode modifiers to decide what regular expression to use. Code looks now like this: $unicoderegexp = @preg_match('/\pL/u', 'a'); // This will fail silenty, returning false , if regexp libraries don't support unicode if ($unicoderegexp) { // We can use unicode modifiers $regexp = 'whatever_using_unicode_modifiers'; } else { // We cannot ue unicode modifiers $regexp = 'whatever_NOT_using_unicode_modifiers'; }
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Done, current code should work ok under any PHP/OS libraries combination. Resolving as fixed.

          Show
          Eloy Lafuente (stronk7) added a comment - Done, current code should work ok under any PHP/OS libraries combination. Resolving as fixed.
          Hide
          Ger Tielemans added a comment -

          Same problem in 1.9.3+ (just for the record)

          Solution: move the link to the image to the wysiwyg editor and backup is fine again.

          Show
          Ger Tielemans added a comment - Same problem in 1.9.3+ (just for the record) Solution: move the link to the image to the wysiwyg editor and backup is fine again.

            People

            • Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: