Moodle

Backup/restore should respect symbolic links

Details

  • Type: Improvement Improvement
  • Status: Closed Closed
  • Priority: Minor Minor
  • Resolution: Won't Fix
  • Affects Version/s: 1.6.3
  • Fix Version/s: None
  • Component/s: Backup
  • Labels:
    None
  • Affected Branches:
    MOODLE_16_STABLE

Description

When backing up courses, symbolically linked folders are included as native folders, so that each course backup includes the whole symbolically linked folder with the actual files, duplicating the same files for each course. This defeats a technique used by some for sharing files among courses. In this technique, the administrator creates a symbolic link to a subdirectory of Site files within the Files area of each course (see discussion http://moodle.org/mod/forum/discuss.php?d=65049). Can the default behavior of Backup be changed to preserve symbolic links?

Activity

Hide
Eloy Lafuente (stronk7) added a comment -

Sorry John,

but I really think it's really difficult to implement, because, to guarantee integrity the backup must be auto-contained. I cannot see "preserving links" like a suitable alternative.

Anyway, re-creating such link shouldn't be really complex (one simple script could iterate over all courses, deleting real dirs and buiilding links as needed....

Uhm... marking this as won't fix. Sorry.

Show
Eloy Lafuente (stronk7) added a comment - Sorry John, but I really think it's really difficult to implement, because, to guarantee integrity the backup must be auto-contained. I cannot see "preserving links" like a suitable alternative. Anyway, re-creating such link shouldn't be really complex (one simple script could iterate over all courses, deleting real dirs and buiilding links as needed.... Uhm... marking this as won't fix. Sorry.
Hide
Petr Škoda (skodak) added a comment -

Proper solution is IMO to create shared resources, but it would need major changes in file handling in core and resource mod itself - maybe a "Resource II" would be a better than fixing the old one that has several other design problems.

My +1 for will not fix links too

Show
Petr Škoda (skodak) added a comment - Proper solution is IMO to create shared resources, but it would need major changes in file handling in core and resource mod itself - maybe a "Resource II" would be a better than fixing the old one that has several other design problems. My +1 for will not fix links too
Hide
John Isner added a comment -

The symbolic links used for file sharing are to other directories in moodledata (not outside), so backups WOULD be auto-contained. Oh well, I will wait for Resource II.

Show
John Isner added a comment - The symbolic links used for file sharing are to other directories in moodledata (not outside), so backups WOULD be auto-contained. Oh well, I will wait for Resource II.
Hide
Timothy Allen added a comment -

To avoid this problem, I used to put the symbolic link in the "backupdata" folder. This worked because nothing in that folder is included in a backup.

Some time ago, this was no longer useful because students could not access any files included in a course via a symbolic link - I guess someone changed the code to block this, probably for good security reasons. However it made the symbolic links no longer work for the student. For this reason, even if the above issue was fixed it would probably only be useful if the symbolic links were to the site files area, which I think are accessible to students.

By the way, I am wondering whether Eloy possibly misunderstood the original request. What was suggested is that during backup symbolic links are IGNORED, not that they are included (perhaps the original word "preserved" used was a little unclear). At the present time they are included, we want them to be ignored (for the purposes of backup only) instead.

I am curious how large institutions like the Open University with, I guess, many courses, use Moodle without a way to share files between courses. Over a lot of courses, having to use mulitple copies of identical files in many courses would be almost unworkable I would have thought. I wonder is anyone has a suggestion to solve the exasperating problems discussed in http://moodle.org/mod/forum/discuss.php?d=65049?

Show
Timothy Allen added a comment - To avoid this problem, I used to put the symbolic link in the "backupdata" folder. This worked because nothing in that folder is included in a backup. Some time ago, this was no longer useful because students could not access any files included in a course via a symbolic link - I guess someone changed the code to block this, probably for good security reasons. However it made the symbolic links no longer work for the student. For this reason, even if the above issue was fixed it would probably only be useful if the symbolic links were to the site files area, which I think are accessible to students. By the way, I am wondering whether Eloy possibly misunderstood the original request. What was suggested is that during backup symbolic links are IGNORED, not that they are included (perhaps the original word "preserved" used was a little unclear). At the present time they are included, we want them to be ignored (for the purposes of backup only) instead. I am curious how large institutions like the Open University with, I guess, many courses, use Moodle without a way to share files between courses. Over a lot of courses, having to use mulitple copies of identical files in many courses would be almost unworkable I would have thought. I wonder is anyone has a suggestion to solve the exasperating problems discussed in http://moodle.org/mod/forum/discuss.php?d=65049?
Hide
John Isner added a comment -

A very good solution! Putting the symbolic link to a site file subdirectory in course files backupdata neatly solves the problem of backup file replication.

Quoting Joseph Rezeau in the discussion that Tim referenced, "Unfortunately there is at the moment no single user-friendly solution for sharing resources (not just files) across courses in Moodle. This is for me the most serious flaw in Moodle. I am satisfied with none of the solutions that are proposed whenever this "sharing of resources" question crops up (which is quite frequently)." So how do the big institutions cope with this flaw? For them, the problem is multiplied by 200,000.

Show
John Isner added a comment - A very good solution! Putting the symbolic link to a site file subdirectory in course files backupdata neatly solves the problem of backup file replication. Quoting Joseph Rezeau in the discussion that Tim referenced, "Unfortunately there is at the moment no single user-friendly solution for sharing resources (not just files) across courses in Moodle. This is for me the most serious flaw in Moodle. I am satisfied with none of the solutions that are proposed whenever this "sharing of resources" question crops up (which is quite frequently)." So how do the big institutions cope with this flaw? For them, the problem is multiplied by 200,000.
Hide
Martin Dougiamas added a comment -

Can't we just add something like -y to the zip command?

Show
Martin Dougiamas added a comment - Can't we just add something like -y to the zip command?
Hide
Eloy Lafuente (stronk7) added a comment -

Uhm...

it isn't so easy, because all the files are copied to temp/backup storage area before zipping all together. Links should be detected while copying them to that area and annotated in some way (creating new links?) to be able to do that consistently.

Show
Eloy Lafuente (stronk7) added a comment - Uhm... it isn't so easy, because all the files are copied to temp/backup storage area before zipping all together. Links should be detected while copying them to that area and annotated in some way (creating new links?) to be able to do that consistently.
Hide
Timothy Allen added a comment -

Do you mean that some it is necessary to follow some kinds of symbolic links and not others, so that it is more complicated?..

If you copy first, then zip, you could use a similar switch for cp - I think it is --no-dereference.

Maybe I am not understanding the issue - I am no programmer and would defer to Eloy and co any day!

Show
Timothy Allen added a comment - Do you mean that some it is necessary to follow some kinds of symbolic links and not others, so that it is more complicated?.. If you copy first, then zip, you could use a similar switch for cp - I think it is --no-dereference. Maybe I am not understanding the issue - I am no programmer and would defer to Eloy and co any day!
Hide
Petr Škoda (skodak) added a comment -

What would happen if somebody tried to restore a symbolic link to config.php? I guess we would have to deal with more security problems.
And then there is the problem with Win32 - all standard features should be supported on all platforms, right?

Show
Petr Škoda (skodak) added a comment - What would happen if somebody tried to restore a symbolic link to config.php? I guess we would have to deal with more security problems. And then there is the problem with Win32 - all standard features should be supported on all platforms, right?
Hide
Timothy Allen added a comment -

Umm...how could you ( and thus would you want to) restore a symbolic link to config.php (or anything else) if it has been ignored during the backup operation and thus is not there in the first place? Isn't this the problem NOW, and wouldn't it actually be solved by doing what is suggested in this bug? Why would anyone want to use such a symbolic link to a config file?

I take your point about Win32 though.

Show
Timothy Allen added a comment - Umm...how could you ( and thus would you want to) restore a symbolic link to config.php (or anything else) if it has been ignored during the backup operation and thus is not there in the first place? Isn't this the problem NOW, and wouldn't it actually be solved by doing what is suggested in this bug? Why would anyone want to use such a symbolic link to a config file? I take your point about Win32 though.
Hide
Petr Škoda (skodak) added a comment -

that was just argument against -y zip parameter

Show
Petr Škoda (skodak) added a comment - that was just argument against -y zip parameter
Hide
Eloy Lafuente (stronk7) added a comment -

Hi,

I'm not trying to say the it's necessary to follow some kind of links and no others. I just was explaining that, if finally implemented, such detection of links in the course area must be performed when copying them to the temporal area used in backup, not when zipping. Was just one explanation of how the backup does its job (using the temp area) to confirm that the zip switch isn't a solution at all.

So, the "detection" of links must be performed when copying them from the course area to the temp area. And that's isn't either a matter of adding one switch to the "cp" command, maily because we copy all those files from php, using our own moodle functions, so we'll need to extend such functions to support links, allowing to use BOTH alternatives (the current, default, following links and the new proposed one, not following links).

I'm only saying that it's more complex than simply adding one switch to a zip or copy operation. And also, it must be handled carefully, because it will "break" one of the basic backup concepts: Being 100% auto-contained (i.e, such backup file only will be fully restoreable in the SAME server it has been generated, because it depends of some external files).

I really think we need to "normalise" the way Moodle access to shared resources (repos...) to be able to implement them in a robust way, but hacking the backup to use solutions like the "links" one....uhm, not sure if that's the correct way.

So, yes, we could implement it (if cross-os compatible). Just wondering if it's the way to solve the "shared resources access" problem properly.

Ciao

Show
Eloy Lafuente (stronk7) added a comment - Hi, I'm not trying to say the it's necessary to follow some kind of links and no others. I just was explaining that, if finally implemented, such detection of links in the course area must be performed when copying them to the temporal area used in backup, not when zipping. Was just one explanation of how the backup does its job (using the temp area) to confirm that the zip switch isn't a solution at all. So, the "detection" of links must be performed when copying them from the course area to the temp area. And that's isn't either a matter of adding one switch to the "cp" command, maily because we copy all those files from php, using our own moodle functions, so we'll need to extend such functions to support links, allowing to use BOTH alternatives (the current, default, following links and the new proposed one, not following links). I'm only saying that it's more complex than simply adding one switch to a zip or copy operation. And also, it must be handled carefully, because it will "break" one of the basic backup concepts: Being 100% auto-contained (i.e, such backup file only will be fully restoreable in the SAME server it has been generated, because it depends of some external files). I really think we need to "normalise" the way Moodle access to shared resources (repos...) to be able to implement them in a robust way, but hacking the backup to use solutions like the "links" one....uhm, not sure if that's the correct way. So, yes, we could implement it (if cross-os compatible). Just wondering if it's the way to solve the "shared resources access" problem properly. Ciao
Hide
Timothy Allen added a comment -

Thank you very much for explaining this Eloy, I understand the issues much better now.

Show
Timothy Allen added a comment - Thank you very much for explaining this Eloy, I understand the issues much better now.
Hide
Eloy Lafuente (stronk7) added a comment -

Thanks Tim!

I would vote to won't fix this for now, waiting for the arrival of a proper "shared resources" solution. I think that using "tricks" like storing to the "backupdata" or similar can help all them to "survive" until the correct implementation is ready.

And thought about this? yes, no...? TIA!

Show
Eloy Lafuente (stronk7) added a comment - Thanks Tim! I would vote to won't fix this for now, waiting for the arrival of a proper "shared resources" solution. I think that using "tricks" like storing to the "backupdata" or similar can help all them to "survive" until the correct implementation is ready. And thought about this? yes, no...? TIA!
Hide
Eloy Lafuente (stronk7) added a comment -

Closing (Won't fix) in some hours if nobody has anything against it....

Show
Eloy Lafuente (stronk7) added a comment - Closing (Won't fix) in some hours if nobody has anything against it....
Hide
Eloy Lafuente (stronk7) added a comment -

as promised some months ago... closing this as won't fix.

Show
Eloy Lafuente (stronk7) added a comment - as promised some months ago... closing this as won't fix.

Dates

  • Created:
    Updated:
    Resolved: