Issue Details (XML | Word | Printable)

Key: MDLSITE-411
Type: New Feature New Feature
Status: Resolved Resolved
Resolution: Fixed
Priority: Trivial Trivial
Assignee: Eloy Lafuente (stronk7)
Reporter: Anthony Borrow
Votes: 3
Watchers: 4
Operations

Add/Edit UI Mockup to this issue
If you were logged in you would be able to see more operations.
Moodle Community Sites

Download: Ability to package several directories in CVS into a package

Created: 08/Apr/08 07:46 AM   Updated: 29/Jul/09 09:47 AM
Component/s: download.moodle.org

Participants: A. T. Wyatt, Anthony Borrow, Dmitry Pupinin, Eloy Lafuente (stronk7), Marc Grober and Martin Dougiamas
Security Level: None
Resolved date: 06/Jan/09


 Description  « Hide
I have this idea of having patches, plugins, and packages in CONTRIB. The idea behind a package is that it is several patches, plugins, etc. combined - in other words a package. It would allow us to pull together into a single zip file several plugins. So how can this be helpful or useful. Consider the feedback package (which although it will hopefully be in Moodle 2.0) serves as a current good example. It consists of a block and a module (activity). New users have to know to download and install both; however, it would be nice to be able to offer them a single zip file. In particular this comes up as we get things like the ILP package which consists of two blocks and two modules.

Here are some ideas of how I envision implementing it. If we have a package.txt file for example for the feedback example we would create a file contrib/packages/feedback/package.txt with the following content:

contrib/plugins/mod/feedback
contrib/plugins/block/feedback

we could have a script to package both of those plugins into a single feedback.zip file. As with other plugins and patches,

http://download.moodle.org/package/feedback.zip - would give the HEAD branch of feedback block and module
http://download.moodle.org/package18/feedback.zip - would give the 18STABLE branch of feedback block and module

Does that make sense? Seem doable?

 All   Comments   Change History      Sort Order: Ascending order - Click to sort in descending order
Eloy Lafuente (stronk7) added a comment - 08/Apr/08 08:00 AM
Sounds like a brilliant idea IMO, as far as we keep those package.txt packages under control (perhaps it would be a good idea to keep them out from contrib to restrict write access on them?).

+1 to advance in this area.


Marc Grober added a comment - 02/May/08 08:44 AM
Please consider the posts here http://moodle.org/mod/forum/discuss.php?d=95666&parent=423792
http://moodle.org/mod/forum/discuss.php?d=95780&parent=423571

Are we talking about two different kinds of packages?

I don't think so as it looks to me like this is what took place with dragmath.... and there are I think substantial issues with non-static sets of files.

I think we need to some additional discussion in the forum regarding nomenclature....


Martin Dougiamas added a comment - 05/May/08 02:19 PM
What exactly is the dragmath issue?

Martin Dougiamas added a comment - 07/May/08 03:06 PM
I'm OK with this kind of idea as long as we're ONLY talking about modules ... as soon as we includes patched versions of files that are in core libraries etc then it becomes dangerous.

Anthony Borrow added a comment - 10/May/08 06:32 AM
Martin - I agree, after speaking about patches I really see them more as temporary solutions. Packages would still be useful for more complex plugins that have a module and block. Peace - Anthony

Anthony Borrow added a comment - 25/Jun/08 09:03 PM
FYI - GSOC contains an example of a package. Akshit Sharma created a customizable theme which has a block for dealing with user information so the custom_theme package would contain:

contrib/plugins/blocks/customtheme
contrib/plugins/theme/custom_theme

Peace - Anthony


Anthony Borrow added a comment - 03/Jul/08 05:10 AM
FYI - I am using the tracker to keep a list of the various pieces of contributed code that would make good packages. At this point, I see a package as being defined as the grouping together of two or more plugins that function in a unified way. The flexpage course format contains mod, theme, course format, block, etc. In CVS each would go into its respective directory and the package will pull together those zip files. Peace - Anthony

Anthony Borrow added a comment - 11/Nov/08 05:59 AM
I'm thinking of creating sub-tasks of packages that need to be attended to once this is resolved. I'll add the attendance block and attforblock module to this list. Peace - Anthony

Eloy Lafuente (stronk7) added a comment - 11/Nov/08 06:38 AM
Oki,

so... it should support the aggregation of (exclusively) anything under http://cvs.moodle.org/contrib/plugins/ correct?

The point I don't see (after reading this bug again) is the one defining those package.txt files within contrib/packages/feedback. It can be done, of course but won't be easier to have only one packages.txt file (under contrib/tools/packager or so) controlling all the packages to be built? That way everything will be centralised in one file, easier to track and control.

Just one idea. Ciao


Anthony Borrow added a comment - 11/Nov/08 06:53 AM
Eloy - Initially i was thinking that this would be something the maintainer would provide with the code; however, in order to keep control over it I think it would be better as you suggest to have a single package list. So my +1 for a single file. Peace - Anthony

Eloy Lafuente (stronk7) added a comment - 11/Nov/08 08:48 AM
And the idea is to unzip it in moodle root folder to get everything installed in place, correct?

Also... could packages.txt format be something like:

packagename:oneplugin;anotherplugin;.....;lastplugin
otherpackagename:oneplugin;anotherplugin;.....;lastplugin
....

that with example data will look like this:

#packagename:oneplugin;anotherplugin;.....;lastplugin
feedback_package:mod/feedback;blocks/feedback
customtheme_package:blocks/customtheme;theme/custom_theme
...

  1. = comment line
    : = separator between package name and list of components
    ; = separator between components

(note I've added the _package suffix to all packages in example, perhaps it's a good idea to name all them that way to avoid possible confusions with individual plugins with the same name. This can be automated of course, leaving "_package" out from the packages.txt file but adding it automatically on creation).


Anthony Borrow added a comment - 11/Nov/08 09:58 AM
Eloy - I think all of what you say above looks perfectly fine. In terms of the download and generation of the zip files I would want to have the ability to download from:

http://download.moodle.org/download.php/contrib/packages/feedback.zip (HEAD)
http://download.moodle.org/download.php/contrib/packages19/feedback.zip (MOODLE_19_STABLE)
http://download.moodle.org/download.php/contrib/packages18/feedback.zip (MOODLE_18_STABLE)

As for the package suffix, that would be fine if you think it would help. Would you envision that changing things so that the download URL would include it or would that be automatically added as needed? I'm OK with having the suffix if it actually provides benefit; otherwise, since those who will have access to the packages file and those working with it will just be developers (and mostly me, and in rare situations) we might be able to avoid it. But I don't have strong feelings either way so if it is beneficial or makes things easier for you by all means feel free to use it.

Peace - Anthony


Eloy Lafuente (stronk7) added a comment - 11/Nov/08 10:07 AM
yup, yup. Packages will be available in that sort of URLs, agree 100%.

About the names, I thought about to append "_package" to all them because, that way we avoid conflicts between feedback.zip (the module) and feedback.zip (the package). I can imagine people downloading them and then trying to uncompress in the wrong place (the package within "mod", for example).

http://download.moodle.org/download.php/contrib/packages/feedback_package.zip (HEAD)
http://download.moodle.org/download.php/contrib/packages19/feedback_package.zip (MOODLE_19_STABLE)
http://download.moodle.org/download.php/contrib/packages18/feedback_package.zip (MOODLE_18_STABLE)

I know it's a bit repetitive to have "package" both in the path and in the name of the file but I think it'll help people downloading and installing a lot of things. It 100% the same for me (when creating the packager). But I feel having unique names will help moodlers in general.

Ciao


Anthony Borrow added a comment - 11/Nov/08 12:26 PM
Sounds good to me or as Picard (TNG) would say "make it so". Thanks for working on this. While there are not that many examples, it will be very helpful when it is needed. Peace - Anthony

A. T. Wyatt added a comment - 12/Nov/08 01:41 AM
My plus one for this too. My attendance mod/block people would like a single download, but we do need to make sure they can tell them apart so as to avoid mis-installation issues.

Eloy Lafuente (stronk7) added a comment - 04/Jan/09 09:05 AM
Done.

All the source code for packages generation is here: http://cvs.moodle.org/contrib/tools/package_builder/

build.sh - the script in charge of generating everything
README.txt - contains some info about the builder itself.
packages.txt - packages description file
readme - dir containing files that will be used to build one README.txt file for each package (global.inc is added to all packages and other files can be added for specific packages)

Code is only in HEAD (and only will be there). I've added one example package:

feedback_package:mod/feedback;blocks/feedback

and you can see how it's available for download here:

Right now packages are built daily. Just add new packages definitions to packages.txt all they will be built in some hours. Both Anthony and me are receiving mails about the package generation daily (useful to find errors and so on). Please, use and watch the packages.txt file carefully.

I'll leave this open some days...ciao


Anthony Borrow added a comment - 04/Jan/09 03:00 PM
Eloy rocks! What a great present and just in time for the celebration of Los Tres Magos Peace - Anthony

Eloy Lafuente (stronk7) added a comment - 06/Jan/09 12:26 AM
Resolving as fixed (implemented). Daily generation working ok. Feel free to reopen if necessary.

Perhaps we'll need a page in download showing all the available packages (or whatever we name them)? Once we have a bunch of them... I think i'll be useful. (Use new issue for that, plz).

Ciao


Dmitry Pupinin added a comment - 06/Jan/09 01:15 AM
Anthony, please add next package:
attendance_package:mod/attforblock;blocks/attendance

Eloy Lafuente (stronk7) added a comment - 06/Jan/09 05:12 AM
Hi,

I've added your package as requested (if I'm not wrong Anthony is going to be out some days).

It should be available tomorrow (if nothing fails) at:

Ciao (and thanks)


Anthony Borrow added a comment - 30/Jan/09 11:24 PM
Thanks Eloy for following up on the attendance package! One less thing to do as I try to catch up Peace - Anthony

Anthony Borrow added a comment - 14/Feb/09 06:52 AM
Eloy - Before re-opening, I wanted to check and see how you had handled the various branches. For example, with feedback there should be a 1.7 and 1.8 branch in CVS; however, it did not seem to have created : http://download.moodle.org/download.php/packages17/feedback_package.zip and : http://download.moodle.org/download.php/packages18/feePeacdback_package.zip. Ideally, I think it should create one for each branch available. Peace - Anthony

Eloy Lafuente (stronk7) added a comment - 15/Feb/09 01:09 AM
Ah, yes, Anthony,

right now... we are only building 1.9 and HEAD packages (I think you should be receiving 2 emails daily about that generation, yup?).

But that's simply, because I limited that, because of 1.7 and 1.8 becoming older and older. Do you want me to start generating 1.7 & 1.8 packages? It's only a matter of enabling that in the server.

Ciao


Eloy Lafuente (stronk7) added a comment - 15/Feb/09 01:19 AM
Done.

You should have received two emails about 1.7 and 1.8 generation and packages should be available at their respective directories (packages17 and packages18).

Will be regenerated daily, like the rest. Ciao


Anthony Borrow added a comment - 15/Feb/09 01:23 AM
Eloy - Giving it some more thought, I do not think that we need 1.7 since we generally discourage folks from using that version in favor of at least 1.8 I think there are enough 1.8 folks out there yet that it may be helpful to go ahead and enable it. I'm glad to hear that it would be pretty simple to do so. AFAIK, I have not been receiving any daily emails. Which email address is currently being used? I would suggest using contrib@moodle.org but you can also use anthony@moodle.org since they both go to the same place (arborrow@jesuits.net). Although I do recall seeing something when a commit change is made but nothing like a status or completion report. That would be good to have (but not essential). Peace - Anthony

Anthony Borrow added a comment - 15/Feb/09 01:27 AM
Excellent - thanks, it does not hurt to have 1.7. Although I do not think I am receiving the emails. Peace - Anthony

Eloy Lafuente (stronk7) added a comment - 15/Feb/09 01:45 AM
Ah, my fault. I was sending emails to: aborrow@jesuits.net (missing "r"). Just switching to contrib@moodle.org ...

Ciao


Anthony Borrow added a comment - 15/Feb/09 01:53 AM
np, I just received the emails - mil gracias

Anthony Borrow added a comment - 18/Mar/09 11:44 AM
Eloy - I wanted to get your take on a possible use of package. It involves something that would technically be a patch. For example, I'm looking at CONTRIB-728 which is mostly a module that gets plugged in and then requires some really small hacks. My thinking is that the plugin module could go in contrib/plugins/mod/includecode and the patch file for the small hack in contrib/patches/includecode and then create a package that consists of the patch and the plugin. The advantage from my perspective is that we are not creating bloated patch files that are predominantly plugins. The disadvantage is that someone might try to manually download the plugin, not read the README.txt or INSTALL.txt files and wonder why it is not working. From an organizational perspective, it allows for the possibility of taking something that currently is a patch and working toward making it a plugin and we really would not have to move things around. The purist in me says that if its a patch, it is a patch; however, I like the idea of being able to distribute the plugin part of it as a plugin and keeping a small patch. The quiz report analysis open learner patch is a good example of something that is mostly a plugin, with binary files like images, that is a big complicated patch file that could really be simplified by this splitting approach (http://cvs.moodle.org/contrib/patches/quiz_reportanalysis_openlearner/) which was CONTRIB-621. Thanks for your feedback and guidance. My gut says that this is a reasonable use of package but I figured I would run it by you and see what you think. I don't want to abuse packages. So my question is does this sound like a good idea or an abuse of packages? Peace - Anthony

Anthony Borrow added a comment - 24/Mar/09 03:13 AM
Eloy - I spoke with MD about CONTRIB-870 (adding a pages block that would provide some CMS like behavior). Here is an example of a package I would like to be able to create:

page_package:page_package:plugins/blocks/page,patches/block_page

Notice that I added the plugins at the beginning. The easiest way I can see implementing this by changing the SOURCE_PATH to contrib rather than contrib/plugins.

That way it would only require adding /plugins to the beginning of each of the VALID_PLUGINS and adding patches to the list.

I can make all the changes with the exception of the cron job on the server that provides the initial SOURCE_PATH.

How does this sound to you?

Peace - Anthony


Eloy Lafuente (stronk7) added a comment - 25/Mar/09 07:53 AM
Hi, hehe, I was sure your were going to comment about that in less than 2 hours (btw I missed your previous comment 1 week ago, sorry).

I'm going to play a bit with that... to confirm that only that change is necessary (because I think that will cause one "plugins" dir to be created in the root of the package... and I'm not sure if that is ok, because unzipping in moodle root dir won't work any more).

Assuming that we need to leave all the plugins/whatever stuff into the root level of the zip file.... can you imagine a place to extract those patches? Perhaps one simple patches dir will be enough? With automatic instructions about how to apply it?

Ciao


Anthony Borrow added a comment - 30/Mar/09 09:35 PM
Eloy - I was so happy to have packages that I didn't even consider patches. In terms of time, no worries as I've been pretty busy myself. I would say that we definitely do want the unzipping of plugins to continue to work so that they can be unzipped directly to the user's moodle root. If we need to wait and make some changes to the script, I would rather wait and do it right than rush so please take whatever time you need. I see no reason why we could not have a simple patches directory in the zip file and instructions for the users to check out http://docs.moodle.org/en/Development:How_to_apply_a_patch. Ideally this should be able to hold multiple patch files. Thinking of the case where a package might be defined as containing multiple patches I wonder if each patch should create its own subdirectory under the patches folder. I'm thinking of the case where two patches might both have README.txt files. Does that make sense? Peace - Anthony

Anthony Borrow added a comment - 29/Jul/09 09:47 AM
Eloy - I am just now finding opportunity to take advantage of generating a good README.txt file for a package. Nice job anticipating the need for this. Simple but slick! Peace - Anthony