Backup/Restore problem. ID don´t change on resource or image links

Hi Daniel,

thanks a lot for sharing your backup example!

I've been analysing it and I've found that it includes links like this:

http://yourserver/file.php?file=%2F260%2Fimage.jpg

And, IMO, that's the cause of your problem, those %2F encoded chars that should be normal and simple slashes.

The key is that backup uses to transform any link of the form:

http://yourserver/file.php?file=/COURSEID to something called $@FILEPHP@$ and later, restore is able to transform back all those $@FILEPHP@$ to the new URL with the new courseid.

In your case, those %2F are breaking the transformation on backup, hence restore hasn't anything to transform back, so your original URLs are left unmodified.

While I think (good news) this is easy to fix (in backup process, by looking also to all those %2F chars... I'm very interested about to KNOW how those horrible encoded chars have ended there. Can you, please, explain how have you added those images?

• Copying them from the file manager?
• Using the HTML Editor? The "official" one or any alternative?
• Using plain text editor and typing the HTML by hand?
• ....

As said, I think we can modify (and will do) the backup procedure to be able to detect those chars easily but we should prevent them to be there at origin.

ANy info will be welcome, TIA! Ciao

Oki, these are the possibilities we can find right now in Moodle 1.9, expanded to all the possible switches:

• file.php/CID/path/to/document.html#someanchor?forcedownload=1
• file.php?file=%2FCID%2Fpath%2Fto%2Fdocument.html#someanchor&forcedownload=1

Where CID is the course id, someanchor is one optional anchor in the URL and forcedownload is one optional setting.

To make this work in backup / restore we need to transform those URLs to something neutral, not dependent of the slash settings in the origin and target sites. So the proposal is to convert those links to:

$@FILEPHP@$$@SLASH@$path$@SLASH@$to$@SLASH@$document.html#someanchor$@FORCEDOWNLOAD@$

That format is independent of the slash-arguments setting and can be easily converted back to proper URLs by replacing:
$@FILEPHP@$ => new wwwroot + file.php + courseid in the target server
$@SLASH@$ => slashes or encoded slashes in the target server
$@FORCEDOWNLOAD@$ => forcedownload parameter in the target server

Going to try that approach. Ciao

Annotating here this REGEXP (general URL extractor) to have it present in this bug.

$pattern = '/\b(https|http):\/\/([\pL-:@.0-9]+)(\/(??:([-;:@#&=\pL0-9\$~_.+!*\',]*?)|[-;:@#&=\pL0-9\$~_.+!*\',]|%[a-fA-F0-9]

)?(?:?((??:([-;:@#&=\pL0-9\$_.+!*\',]

|\/)*))?(?<![,.;])/';

Edited to add allowed ~ and some changes in quotes.

Is this a "fix" yet? We have been dealing with this issue for 8 months - every time we restore a course - even to the same server. I am find this same issue. It looks lke you are still considering this way of fixing this issue. Love to have this resolved. Will this be in Moodle 1.9.5 (by any chance)?

Thanks for looking into this.

Fix implemented in 19_STABLE and HEAD. Files changed in CVS for Moodle 1.9 are these:

backup/backuplib.php: http://cvs.moodle.org/moodle/backup/backuplib.php?view=markup&pathrev=MOODLE_19_STABLE
backup/restorelib.php: http://cvs.moodle.org/moodle/backup/restorelib.php?view=markup&pathrev=MOODLE_19_STABLE

They are available in CVS and will be included in next Moodle 1.9 weekly release (next Wednesday, 22th April 2009).

The approach followed consists into saving those links in a neutral format in backup, so restore will be able to construct them back to correct links (pointing to new course files).

Note the bug "only" affected sites running with slasharguments off, and also note it's very interesting to have that setting enabled in order to get relative links (resources, scorm...) working properly.

Please test, by performing new backup and restore. Any feedback will be really welcome.

I'll leave this open some days... ciao

Hi Eloy,

I downloaded the files:

backup/backuplib.php: http://cvs.moodle.org/moodle/backup/backuplib.php?view=markup&pathrev=MOODLE_19_STABLE
backup/restorelib.php: http://cvs.moodle.org/moodle/backup/restorelib.php?view=markup&pathrev=MOODLE_19_STABLE

uploaded them and tried to make a backup and a restore of a course ... and it didn't work. It looks like something is missing because when I started the restore procedure all the course information was missing. It created an empty course in a weekly format. The source course was a course in topic format. All content was missing.

I'm looking forward to see this fix

Thank's for your work and attention.

Greetings

Daniel

Hi Daniel,

I assume you are running Moodle 1.9.4+ (recent build) correct?

Using those two files:

backup/backuplib.php: http://cvs.moodle.org/moodle/backup/backuplib.php?view=markup&pathrev=MOODLE_19_STABLE
backup/restorelib.php: http://cvs.moodle.org/moodle/backup/restorelib.php?view=markup&pathrev=MOODLE_19_STABLE

within older Moodle builds can lead to "strange" problems, like the one commented by you above, that I think it's 100% unrelated with the fix applied to solve the links problem.

Perhaps the best way to be 100% sure is:

• For now, revert to your previous backuplib and restorelib files.
• In 14 hours from now... there will be newer downloads available at: http://download.moodle.org , pick the 19_WEEKLY one and perform a complete update of your site. That way you'll have the whole moodle running correct versions.
• Test the original links problem, should be fixed.
• If you find any problem, please enable debug to developer level and execute the backup/restore again, attaching complete output here.

Apologises by putting there those 2 links, as it's obvious can cause problems if running non up-to-date Moodle versions.

Please, always upgrade Moodle as a whole (straight from CVS or from downloads.moodle.org), but as a whole.

Ciao

Hi Eloy,

Ok, thank's for the quick answer

greetings

Daniel

Changed the regexp expression above to be PHP 4.3.x compatible (we were using some PHP 4.4 features and Moodle 1.9 must work with PHP 4.3). Keeping HEAD without modifications.

Ciao

Edited: Ref URL about this problem: http://moodle.org/mod/forum/discuss.php?d=121946

It seems that some PHP 5.2.x server already have the same problem, with unicode modifiers not being allowed in regular expressions.

Looking for info, it seems that it's a problem in the underlying OS regexp libraries, being compiled without unicode support.

So I've modified both 19_STABLE and HEAD to do some checking of the unicode modifiers to decide what regular expression to use. Code looks now like this:

$unicoderegexp = @preg_match('/\pL/u', 'a'); // This will fail silenty, returning false ,if regexp libraries don't support unicode
if ($unicoderegexp) { // We can use unicode modifiers
    $regexp = 'whatever_using_unicode_modifiers';
} else { // We cannot ue unicode modifiers
    $regexp = 'whatever_NOT_using_unicode_modifiers';
}

Done, current code should work ok under any PHP/OS libraries combination. Resolving as fixed.

Same problem in 1.9.3+ (just for the record)

Solution: move the link to the image to the wysiwyg editor and backup is fine again.