Moodle
  1. Moodle
  2. MDL-13114

Include bulk course create and remove with Moodle through the use of CSV files

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.8.4, 1.9.10, 2.0.1, 2.3.2, 2.4, 2.5
    • Fix Version/s: 2.6
    • Component/s: Administration
    • Environment:
      Redhat Enterprise 4AS running Apache 2.2.x, MySQL 5, and Moodle 1.8.x and Moodle 1.9.xbeta
    • Database:
      MySQL
    • Testing Instructions:
      Hide
      1. Create a CSV to import data from
      2. Navigate to Home / Site administration / Courses / Upload courses
      3. TEST: Different combinations of settings
      4. TEST: Different combinations of CSV data (https://tracker.moodle.org/browse/MDL-13114?focusedCommentId=231131&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-231131)
      5. Make sure
        • Expected errors occurred
        • Update/Create/Delete/Rename, etc... worked when no errors occurred

      Repeat using the command line tool.

      Show
      Create a CSV to import data from Navigate to Home / Site administration / Courses / Upload courses TEST : Different combinations of settings TEST : Different combinations of CSV data ( https://tracker.moodle.org/browse/MDL-13114?focusedCommentId=231131&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-231131 ) Make sure Expected errors occurred Update/Create/Delete/Rename, etc... worked when no errors occurred Repeat using the command line tool.
    • Affected Branches:
      MOODLE_18_STABLE, MOODLE_19_STABLE, MOODLE_20_STABLE, MOODLE_23_STABLE, MOODLE_24_STABLE, MOODLE_25_STABLE
    • Fixed Branches:
      MOODLE_26_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      MDL-13114-master
    • Story Points:
      100
    • Rank:
      164
    • Sprint:
      BACKEND Sprint 2

      Description

      We're looking to Moodle for a replacement to Blackboard Enterprise. In order for Moodle to be a viable replacement we need a way to bulk create and delete courses based on CSV files. The bulk create courses is handled by use of a 3rd party module (uploadcourse.php created by Rory Allford). However once we import say 700+ courses there's no way to them bulk remove them at the end of a semester (removing 1 by 1 would be too time consuming). We would really prefer that this functionality be included within Moodle, as relying on 3rd party modules (no matter how good they are), inherits a risk if the module stops getting updated.

        Issue Links

          Activity

          Hide
          John Isner added a comment -

          http://moodle.org/mod/forum/discuss.php?d=27333

          As you can see from the above nearly three-year discussion, there is tremendous interest in bulk course creation. I was pursuaded of its importance by a presentation by A.T. Wyatt at the Oklahoma City Moodle Moot in June 2008. At Wyatt's university, a course is created for each teacher training student to serve as a preservice teaching porfolio. FYI Wyatt uses a script written by Mark Pearson and posted in http://moodle.org/mod/forum/discuss.php?d=30126. This script is no longer being maintained.

          The Modules and Plugins entry http://moodle.org/mod/data/view.php?d=13&rid=395 ("Bulk Course and Category Creation Tool (Upload CSV)") which seems to be most popular is also no longer actively maintained by its author, Rory Alford, and zip files have once again begun to appear in the forum. Rory's tool can create categories (and subcategories) as well as courses.

          Is there any way that this feature can be made standard? For example, Site administration -> Courses -> Upload courses.

          Show
          John Isner added a comment - http://moodle.org/mod/forum/discuss.php?d=27333 As you can see from the above nearly three-year discussion, there is tremendous interest in bulk course creation. I was pursuaded of its importance by a presentation by A.T. Wyatt at the Oklahoma City Moodle Moot in June 2008. At Wyatt's university, a course is created for each teacher training student to serve as a preservice teaching porfolio. FYI Wyatt uses a script written by Mark Pearson and posted in http://moodle.org/mod/forum/discuss.php?d=30126 . This script is no longer being maintained. The Modules and Plugins entry http://moodle.org/mod/data/view.php?d=13&rid=395 ("Bulk Course and Category Creation Tool (Upload CSV)") which seems to be most popular is also no longer actively maintained by its author, Rory Alford, and zip files have once again begun to appear in the forum. Rory's tool can create categories (and subcategories) as well as courses. Is there any way that this feature can be made standard? For example, Site administration -> Courses -> Upload courses.
          Hide
          A. T. Wyatt added a comment -

          I have to change the attribution to Jeff Church.
          http://moodle.org/mod/forum/discuss.php?d=96622

          Mark Pearson and I did work on this, and Mark has some good documentation on it. But Mark did not write the script.

          But I think this is sorely needed in the standard instance. There are a lot of people who have the need to do bulk course creation (and deletions). Mike Churchward had some good ideas on this also; I visited with him about it last September.

          Show
          A. T. Wyatt added a comment - I have to change the attribution to Jeff Church. http://moodle.org/mod/forum/discuss.php?d=96622 Mark Pearson and I did work on this, and Mark has some good documentation on it. But Mark did not write the script. But I think this is sorely needed in the standard instance. There are a lot of people who have the need to do bulk course creation (and deletions). Mike Churchward had some good ideas on this also; I visited with him about it last September.
          Hide
          Mark Pearson added a comment -

          I will 'third' this as an important issue to address and would support this being bumped from minor to major priority. Besides it's inherent value, having a bulk course upload script and procedure formalized and incorporated into the Moodle core would also be a first step towards formalizing integration with institutional ERP systems. I would like to see some discussion about whether the data format should be simplified and some definition of what the expected behaviour of the script should be (eg not duplicating courses already present). It seems to me that this process should not be an onerous task since the script is already there and working (though I have not tested it with 1.9).

          Show
          Mark Pearson added a comment - I will 'third' this as an important issue to address and would support this being bumped from minor to major priority. Besides it's inherent value, having a bulk course upload script and procedure formalized and incorporated into the Moodle core would also be a first step towards formalizing integration with institutional ERP systems. I would like to see some discussion about whether the data format should be simplified and some definition of what the expected behaviour of the script should be (eg not duplicating courses already present). It seems to me that this process should not be an onerous task since the script is already there and working (though I have not tested it with 1.9).
          Hide
          Greg Rodenhiser added a comment -

          The bulk course creation tool still works in 1.9. I was not aware that the script is no longer maintained. This is bad news for us as we require it's use. I really hope that this makes it into the default Moodle install. Why not? All the work is basicaly done, it just now needs to be maintained for future Moodle releases. This has been a major frustration for us in the Moodle project. Fairly complex features like Moodle Networking are developed, but simple things like batch course create/delete are not. This seems like such a basic feature....

          Show
          Greg Rodenhiser added a comment - The bulk course creation tool still works in 1.9. I was not aware that the script is no longer maintained. This is bad news for us as we require it's use. I really hope that this makes it into the default Moodle install. Why not? All the work is basicaly done, it just now needs to be maintained for future Moodle releases. This has been a major frustration for us in the Moodle project. Fairly complex features like Moodle Networking are developed, but simple things like batch course create/delete are not. This seems like such a basic feature....
          Hide
          Marc Grober added a comment -

          I have tried to review the discussion in the forum from time to time on the bulk course add and delete admin scripts as there has been a good deal of confusion from time to time as to what's what.

          As I understand the situation now, there is an upload utility in M&P that came via Jeff in M&P and a delete utility that was also posted by Jeff and then massaged by Chad posted to the forum, but not added to M&P, and both add and delete utilities now work in 1.8 and 1.9. As I understand it, these utilities have been functioning appropriately and I don;t think I have seen anything in the way of problems lately.

          I created a tracker issue to add the delete utility as a new M&P page or as part of an add/delete utility M&P page pending addition of the utility to Moodle core.

          Show
          Marc Grober added a comment - I have tried to review the discussion in the forum from time to time on the bulk course add and delete admin scripts as there has been a good deal of confusion from time to time as to what's what. As I understand the situation now, there is an upload utility in M&P that came via Jeff in M&P and a delete utility that was also posted by Jeff and then massaged by Chad posted to the forum, but not added to M&P, and both add and delete utilities now work in 1.8 and 1.9. As I understand it, these utilities have been functioning appropriately and I don;t think I have seen anything in the way of problems lately. I created a tracker issue to add the delete utility as a new M&P page or as part of an add/delete utility M&P page pending addition of the utility to Moodle core.
          Hide
          Marc Grober added a comment -

          The discussion still seems to get convoluted and I would like to see perhaps the following steps taken:

          1) clearly document the source of the bulk add and delete course utilities that were offered by Jeff and then tweaked for 1.9

          2) clearly identify both of the bulk course utilities in M&P, preferably packaging them together and offering them from the same M&P page

          3) review same to determine what needs to be done to incorporate these utilities into core

          4) incoporate them into core.

          Show
          Marc Grober added a comment - The discussion still seems to get convoluted and I would like to see perhaps the following steps taken: 1) clearly document the source of the bulk add and delete course utilities that were offered by Jeff and then tweaked for 1.9 2) clearly identify both of the bulk course utilities in M&P, preferably packaging them together and offering them from the same M&P page 3) review same to determine what needs to be done to incorporate these utilities into core 4) incoporate them into core.
          Hide
          Anthony Borrow added a comment -

          Marc - OK, I think I am ready to start making some progress on this. I think this is where we stand.

          1) The course for the bulk course upload done by Jeff is available at http://moodle.org/file.php/5/moddata/forum/33/427997/bulk_course_tools.zip and the M&P entry which should have its download link updated is http://moodle.org/mod/data/view.php?d=13&rid=1536&filter=1. As for backup and delete, Moodlerooms has a course management block that has these functions available at http://development.moodlerooms.com/file.php/1/Moodlerooms_Releases/Course_Management_Block_for_1.9/coursemanagementblock19.zip and one provided by Robert Russo in CONTRIB-605 is available at http://download.moodle.org/download.php/plugins/blocks/backadel.zip. I've asked for Justin Hardin, Robert Russo and Ashley Gooding to help me understand the relationship (if any) between these two similar blocks. In speaking with Dan Marsden, I think the goal we are looking for is to have it as consistent as possible with other areas of import and functionality in Moodle so it would be integrated under Administration rather than a block.

          2) Once we have identified the preferred backup and delete tool we can package them together into a single patch.

          3) Once we have a patch we can then move forward on incorporating these tools into core.

          Peace - Anthony

          Show
          Anthony Borrow added a comment - Marc - OK, I think I am ready to start making some progress on this. I think this is where we stand. 1) The course for the bulk course upload done by Jeff is available at http://moodle.org/file.php/5/moddata/forum/33/427997/bulk_course_tools.zip and the M&P entry which should have its download link updated is http://moodle.org/mod/data/view.php?d=13&rid=1536&filter=1 . As for backup and delete, Moodlerooms has a course management block that has these functions available at http://development.moodlerooms.com/file.php/1/Moodlerooms_Releases/Course_Management_Block_for_1.9/coursemanagementblock19.zip and one provided by Robert Russo in CONTRIB-605 is available at http://download.moodle.org/download.php/plugins/blocks/backadel.zip . I've asked for Justin Hardin, Robert Russo and Ashley Gooding to help me understand the relationship (if any) between these two similar blocks. In speaking with Dan Marsden, I think the goal we are looking for is to have it as consistent as possible with other areas of import and functionality in Moodle so it would be integrated under Administration rather than a block. 2) Once we have identified the preferred backup and delete tool we can package them together into a single patch. 3) Once we have a patch we can then move forward on incorporating these tools into core. Peace - Anthony
          Hide
          Anthony Borrow added a comment -

          Should we change the affects version to 1.9? I do not anticipate doing this for 1.8 but perhaps we can backport it once we have it working.

          Show
          Anthony Borrow added a comment - Should we change the affects version to 1.9? I do not anticipate doing this for 1.8 but perhaps we can backport it once we have it working.
          Hide
          Seba Espinosa added a comment -

          Hi, yesterday (26-4-2009) we have just release a free tool for realizing bulk actions on moodle. This tool allows you realize bacth process for deleting courses, creating courses, assaign teachers to courses and deleting users.

          We develop this applicarion as a result of our work Universidad Adolfo Ibañez. They needed a tool the perform the change of semester (delete oldo courses, create new courses, assaign teacher to the new courses and delete old users) and this application is the result. We tested duringa the past year deleting more than 700 courses each semester and creating over 800 courses each semester without problems.

          This is our first release of this applicarion and we hope to continue improving it in time. Please take a look at it at www.moodleadminpro.com

          Thanks
          Sebastián Espinosa
          MoodleAdmin Team

          Show
          Seba Espinosa added a comment - Hi, yesterday (26-4-2009) we have just release a free tool for realizing bulk actions on moodle. This tool allows you realize bacth process for deleting courses, creating courses, assaign teachers to courses and deleting users. We develop this applicarion as a result of our work Universidad Adolfo Ibañez. They needed a tool the perform the change of semester (delete oldo courses, create new courses, assaign teacher to the new courses and delete old users) and this application is the result. We tested duringa the past year deleting more than 700 courses each semester and creating over 800 courses each semester without problems. This is our first release of this applicarion and we hope to continue improving it in time. Please take a look at it at www.moodleadminpro.com Thanks Sebastián Espinosa MoodleAdmin Team
          Hide
          Geoffrey Rowland added a comment -

          Here is a minor fix of uploadcourse.php that restores the ability to add multiple teachers to a course

          http://moodle.org/mod/forum/discuss.php?d=91850#p572909

          Show
          Geoffrey Rowland added a comment - Here is a minor fix of uploadcourse.php that restores the ability to add multiple teachers to a course http://moodle.org/mod/forum/discuss.php?d=91850#p572909
          Hide
          Jason Hollowell added a comment -

          It seems that this has been dormant for a short time. I would really like to see the Bulk Course Upload tool for 1.9 (one version with all of the features of Rory Alford & Jeff Church's contributions) added to and maintained in the Modules and Plugins page. I am not sure if this capability is being added to 2.0 or not but nevertheless it needs to be maintained for users of 1.9 who are not going to make the jump to 2.0 for some time.

          I have incorporated the following fixes into the tool and am willing to volunteer to maintain it in the Modules and Plugins area for 1.9 as best I can until it is added to the 1.9 core. I am not a true programmer but am capable of making adjustments as people on the forums contribute as well as at making small additions myself and am willing to do the legwork to keep this updated for 1.9.

          Updates to the tool I originally obtained from Jeff Church here (http://moodle.org/mod/forum/discuss.php?d=91850)

          1. Small changes to the text strings to make the explanation of the fields easier to understand and/or to update information as field possibilities have changed.

          2. Added the ability to use the 'enrol' field in the CSV file to set the enrollment setting for courses.

          3. Made adjustments explained here (http://moodle.org/mod/forum/discuss.php?d=91850#p572025) to enable the tool to enroll multiple teachers.

          And will also update the code to remove the bug explained by Jeff Sherk here (http://moodle.org/mod/forum/discuss.php?d=91850#p405748) as well as some of the field 'expansion' fixes explained by Fernando here (http://moodle.org/mod/forum/discuss.php?d=27333#p405593).

          Is there any way that we can move forward a little on this and get one entry in the Modules and Plugins area that points to a file instead of to a forum? Again, I'm willing to help out by maintaining this in the M & P area.

          Jason

          Show
          Jason Hollowell added a comment - It seems that this has been dormant for a short time. I would really like to see the Bulk Course Upload tool for 1.9 (one version with all of the features of Rory Alford & Jeff Church's contributions) added to and maintained in the Modules and Plugins page. I am not sure if this capability is being added to 2.0 or not but nevertheless it needs to be maintained for users of 1.9 who are not going to make the jump to 2.0 for some time. I have incorporated the following fixes into the tool and am willing to volunteer to maintain it in the Modules and Plugins area for 1.9 as best I can until it is added to the 1.9 core. I am not a true programmer but am capable of making adjustments as people on the forums contribute as well as at making small additions myself and am willing to do the legwork to keep this updated for 1.9. Updates to the tool I originally obtained from Jeff Church here ( http://moodle.org/mod/forum/discuss.php?d=91850 ) 1. Small changes to the text strings to make the explanation of the fields easier to understand and/or to update information as field possibilities have changed. 2. Added the ability to use the 'enrol' field in the CSV file to set the enrollment setting for courses. 3. Made adjustments explained here ( http://moodle.org/mod/forum/discuss.php?d=91850#p572025 ) to enable the tool to enroll multiple teachers. And will also update the code to remove the bug explained by Jeff Sherk here ( http://moodle.org/mod/forum/discuss.php?d=91850#p405748 ) as well as some of the field 'expansion' fixes explained by Fernando here ( http://moodle.org/mod/forum/discuss.php?d=27333#p405593 ). Is there any way that we can move forward a little on this and get one entry in the Modules and Plugins area that points to a file instead of to a forum? Again, I'm willing to help out by maintaining this in the M & P area. Jason
          Hide
          Jason Hollowell added a comment -

          This version includes the fix to allow for multiple teacher enrollment and also the addition of the enrol field so that the course enrolment plugin can be set via the tool.

          Show
          Jason Hollowell added a comment - This version includes the fix to allow for multiple teacher enrollment and also the addition of the enrol field so that the course enrolment plugin can be set via the tool.
          Hide
          Greg Rodenhiser added a comment -

          *Bump... Any news on if this functionality will be added/avaialble for Moodle 2.0? Currently evaluating Moodle 2.0 RC1 and sadly there appears no way to bulk create courses.

          Show
          Greg Rodenhiser added a comment - *Bump... Any news on if this functionality will be added/avaialble for Moodle 2.0? Currently evaluating Moodle 2.0 RC1 and sadly there appears no way to bulk create courses.
          Hide
          Jason Hollowell added a comment -

          Greg,

          Sorry, I'm not sure. I haven't experimented with 2.0 enough yet to even know that the capability doesn't exist. I'm a bit surprised that it wasn't added though....!

          I'll try and work with it a little and see how far off this tool is in terms of being usable for 2.0. If it requires serious modification I'm afraid it will be beyond me but I'll see what I can do.

          Anyone else know what the plan is for bulk creation of courses in 2.0 onward?

          Jason

          Show
          Jason Hollowell added a comment - Greg, Sorry, I'm not sure. I haven't experimented with 2.0 enough yet to even know that the capability doesn't exist. I'm a bit surprised that it wasn't added though....! I'll try and work with it a little and see how far off this tool is in terms of being usable for 2.0. If it requires serious modification I'm afraid it will be beyond me but I'll see what I can do. Anyone else know what the plan is for bulk creation of courses in 2.0 onward? Jason
          Hide
          Anthony Borrow added a comment -

          I'm adding 1.9 and 2.0 affected branches so that this does not fall through the cracks. I'm also adding the Fix Version of DEV backlog to hopefully get work on this moving forward. Peace - Anthony

          Show
          Anthony Borrow added a comment - I'm adding 1.9 and 2.0 affected branches so that this does not fall through the cracks. I'm also adding the Fix Version of DEV backlog to hopefully get work on this moving forward. Peace - Anthony
          Hide
          Jason Hardy added a comment -

          This is going to be a key component in getting my organization from switching from BB to Moodle. I know that there was a patch in 1.9 backwards but there is nothing yet for 2.0 (which is what they would want to switch to). Unless someone can fix the old file to work in 2.0.

          Show
          Jason Hardy added a comment - This is going to be a key component in getting my organization from switching from BB to Moodle. I know that there was a patch in 1.9 backwards but there is nothing yet for 2.0 (which is what they would want to switch to). Unless someone can fix the old file to work in 2.0.
          Hide
          Derek Chirnside added a comment -

          Anthony: any progress?
          I notice there is some code in this thread: http://moodle.org/mod/forum/discuss.php?d=91850&parent=744931

          -Derek

          Show
          Derek Chirnside added a comment - Anthony: any progress? I notice there is some code in this thread: http://moodle.org/mod/forum/discuss.php?d=91850&parent=744931 -Derek
          Hide
          Paul Fynn added a comment -

          Oh how I wish......

          Show
          Paul Fynn added a comment - Oh how I wish......
          Hide
          Martin Dougiamas added a comment -

          I had a quick look at the script in the forums and unfortunately there's no way it can make core in the current condition. It's not even remotely following our coding guidelines or existing Moodle code. Just some quick examples:

          direct access to MySQL at the top (not using our DB layer).
          doesn't use core API to do the work
          no string translations
          no phpdocs

          Is there some kind of spec on exactly what this function should do? Can somebody start one?

          Show
          Martin Dougiamas added a comment - I had a quick look at the script in the forums and unfortunately there's no way it can make core in the current condition. It's not even remotely following our coding guidelines or existing Moodle code. Just some quick examples: direct access to MySQL at the top (not using our DB layer). doesn't use core API to do the work no string translations no phpdocs Is there some kind of spec on exactly what this function should do? Can somebody start one?
          Hide
          Patrick Dubois added a comment -

          Just a quick note to mention this would be very useful for myself as well. If I can help in any way that would be great!

          Show
          Patrick Dubois added a comment - Just a quick note to mention this would be very useful for myself as well. If I can help in any way that would be great!
          Hide
          Michael de Raadt added a comment -

          Elevating the priority on this so it is more likely to get some love post-2.1.

          Show
          Michael de Raadt added a comment - Elevating the priority on this so it is more likely to get some love post-2.1.
          Hide
          Peter Stacey added a comment -

          As someone who's moved over from being a Blackboard admin I'm missing this functionality badly.

          Show
          Peter Stacey added a comment - As someone who's moved over from being a Blackboard admin I'm missing this functionality badly.
          Hide
          Derek Chirnside added a comment -

          Well thanks for the vote Peter, that makes 99.
          -Derek

          Show
          Derek Chirnside added a comment - Well thanks for the vote Peter, that makes 99. -Derek
          Hide
          David Mudrak added a comment -

          As Martin mentions above, some spec of the functionality would be nice to have before we start work on this. Let me suggest to follow "do one thing and do it well" approach instead of trying to combine course creation with enrolments etc via one CVS.

          Show
          David Mudrak added a comment - As Martin mentions above, some spec of the functionality would be nice to have before we start work on this. Let me suggest to follow "do one thing and do it well" approach instead of trying to combine course creation with enrolments etc via one CVS.
          Hide
          Paul Fynn added a comment -

          We seem to be stalled here on something that
          1) a lot of users agree is an essential item of functionality
          2) we had a running version of for 1.9
          3) is fundamental to upscaling Moodle to large organisations

          Martin's question was "Is there some kind of spec on exactly what this function should do? Can somebody start one?". Clearly the currently available fixes are not meeting the tech spec that is required for 'growed up' Moodle - however these can act as functional models of what is required. Hence the 'what it should do' is addressed by existing fixes and patches.

          For my money I simply need

          • to create (or draw down from our local registry) a list of courses
          • to manipulate the list to add/edit shortname, course code etc.
          • to be able to set core settings for courses (format, number of topics, start and end date etc.)
          • to be able to add at least one editing teacher via the list
          • to upload to the moodle site and cross check for duplicates

          Can we flesh this out towards a fuller spec ?

          Here's an example of a 'quick'n dirty .csv upload that I was using for 1.9 using one of the patches

          category,enrol,enrollable,expirynotify,format,fullname,groupmode,groupmodeforce,guest,metacourse,newsitems,notifystudents,numsections,password,shortname,showgrades,showreports
          2011.1 PG Courses,manual,0,0,topics,011.1 719 Cost Price and Revenue Management (PG),0,0,0,0,0,0,5,locked,011.1 719 Cost ,0,0

          In my view the current urgency is to establish the bulk course creation, for those about to join Moodle, and for those of us trying to start a new teaching year on 2.0/2.1 - bulk course removal isn't going to be an issue for some time.....

          Show
          Paul Fynn added a comment - We seem to be stalled here on something that 1) a lot of users agree is an essential item of functionality 2) we had a running version of for 1.9 3) is fundamental to upscaling Moodle to large organisations Martin's question was "Is there some kind of spec on exactly what this function should do? Can somebody start one?". Clearly the currently available fixes are not meeting the tech spec that is required for 'growed up' Moodle - however these can act as functional models of what is required. Hence the 'what it should do' is addressed by existing fixes and patches. For my money I simply need to create (or draw down from our local registry) a list of courses to manipulate the list to add/edit shortname, course code etc. to be able to set core settings for courses (format, number of topics, start and end date etc.) to be able to add at least one editing teacher via the list to upload to the moodle site and cross check for duplicates Can we flesh this out towards a fuller spec ? Here's an example of a 'quick'n dirty .csv upload that I was using for 1.9 using one of the patches category,enrol,enrollable,expirynotify,format,fullname,groupmode,groupmodeforce,guest,metacourse,newsitems,notifystudents,numsections,password,shortname,showgrades,showreports 2011.1 PG Courses,manual,0,0,topics,011.1 719 Cost Price and Revenue Management (PG),0,0,0,0,0,0,5,locked,011.1 719 Cost ,0,0 In my view the current urgency is to establish the bulk course creation, for those about to join Moodle, and for those of us trying to start a new teaching year on 2.0/2.1 - bulk course removal isn't going to be an issue for some time.....
          Hide
          Eduardo Domínguez added a comment -

          Do you need a CSV file?
          We're doing this for years in XML with an IMS Enterprise file.

          Maybe this can help to someone: http://www.stylusstudio.com/csv_to_xml.html or http://csv2xml.sourceforge.net/
          you can convert the CSV file to IMS file

          Show
          Eduardo Domínguez added a comment - Do you need a CSV file? We're doing this for years in XML with an IMS Enterprise file . Maybe this can help to someone: http://www.stylusstudio.com/csv_to_xml.html or http://csv2xml.sourceforge.net/ you can convert the CSV file to IMS file
          Hide
          Petr Škoda added a comment -

          If this gets implemented could we please first create real CSV import/export library that supports different separators and standard quoting?

          Show
          Petr Škoda added a comment - If this gets implemented could we please first create real CSV import/export library that supports different separators and standard quoting?
          Hide
          Derek Chirnside added a comment -

          "If this gets implemented could we please first create real CSV import/export library that supports different separators and standard quoting?" Petr, please translate.

          "Current implementation for CVS upload to create accounts is inadequate"
          "Old methods need to be fixed to new methods"
          "If we fix some things now it will improve things and make other CSV importe easy"
          "We now use only one separator and non standard quotes"
          "This means a further delay since 2008 when this was first proposed"
          "What is currently used is not a real import/export/library" ????????????????

          I'm not a PHP coder. But it just seems that this should be not a big deal. (is it?) What saddens me is that people are creating tons of hacks (which uses time) but no-one is putting in the effort to co-ordinate and focus efforst to get this into standard Moodle.

          What is involved in doing what you ask Petr? A day of coding? Does MoodleHQ want it or see the need for it?
          </ramble> </rant>
          -Derek

          Show
          Derek Chirnside added a comment - "If this gets implemented could we please first create real CSV import/export library that supports different separators and standard quoting?" Petr, please translate. "Current implementation for CVS upload to create accounts is inadequate" "Old methods need to be fixed to new methods" "If we fix some things now it will improve things and make other CSV importe easy" "We now use only one separator and non standard quotes" "This means a further delay since 2008 when this was first proposed" "What is currently used is not a real import/export/library" ???????????????? I'm not a PHP coder. But it just seems that this should be not a big deal. (is it?) What saddens me is that people are creating tons of hacks (which uses time) but no-one is putting in the effort to co-ordinate and focus efforst to get this into standard Moodle. What is involved in doing what you ask Petr? A day of coding? Does MoodleHQ want it or see the need for it? </ramble> </rant> -Derek
          Hide
          Charles Fulton added a comment -

          I've written a small proof-of-concept patch for 2.1 which borrows very liberally from the bulk user upload page. This patch is deliberately incomplete. It's missing most forms of error-checking and validation. It doesn't display results unless something blows up. It does, however, use a standard interface and standard internal functions, which I believe addresses a number of the basic concerns raised by Martin and others.

          Is it worth taking this patch further? If not, I don't mind the wasted morning.

          Patch here on github: https://github.com/mackensen/moodle/compare/mdl-13114.

          Show
          Charles Fulton added a comment - I've written a small proof-of-concept patch for 2.1 which borrows very liberally from the bulk user upload page. This patch is deliberately incomplete. It's missing most forms of error-checking and validation. It doesn't display results unless something blows up. It does, however, use a standard interface and standard internal functions, which I believe addresses a number of the basic concerns raised by Martin and others. Is it worth taking this patch further? If not, I don't mind the wasted morning. Patch here on github: https://github.com/mackensen/moodle/compare/mdl-13114 .
          Hide
          Rosario Carcò added a comment - - edited

          As far as I have followed this question, I realize that course creation and user enrolment depend on very singular situations in the different schools and universities. That is why I started to program my own import script with a GUI AND an unattended mode version. See here: http://moodle.org/mod/forum/discuss.php?d=128317

          With Anthnoy Borrow I considered to modify my script so as to:

          • import/create categories (adding the function into the GUI, like upload users)
          • import/create courses (adding the function into teh GUI, like upload users)
          • import/create/enrol users (as already in the GUI)
          • delete courses (adding the function into the GUI, like upload users)
          • all-in-one script/function to do all this with one and the same .csv File and script (adding this function to the GUI, like upload users)
          • all-in-one script that can be executed in unattended mode, during night, through a cron-job (already working for me and my needs. Our Workflow:
          • courses maintained in our school administration system are exported nightly to a windows-share in .csv UTF-8 format. The csv file contains all the columns you can see in my template in the above thread, short everything needed to create new courses and enroll the given users in their respective roles
          • the Moodle cron-job executes my uploadusersandcourses.php script every night writing back to the share a text file with error-codes like "course xyz already exists", or "category xyz does not exist", or "course xyz created", etc. so that we know exactly what the script did during night)

          My script only lacks deletion of courses as at our university we have to consider also archiving of old courses. So we could define what archiving means: copying only the backup-zip-File to a certain backup-server oder backup-archive-directory on the same server, and so on. Archiving could also mean MOVING the whole course to an archive category in the same Moodle server. Actually I move the courses through the GUI to an archive or toDelete category where I leave them hidden and without teachers for another 6 months before deleting them definitively or letting them in the archive category forever.

          But my script has also some more features (realized and planned):

          • course copy/creation from a given template (backup-zip-file) of an existing course (working fine)
          • rename course from old name to new name (because we have courses that change only their name before starting over for a next semester or academic year, not programmed yet, but feasible, scheduled for next release)
          • archive course (has to be defined, not programmed yet)

          So we just had to join our efforts to get existing code of different programmers into the GUI and core of Moodle. I hope to find enough time before winter to realize the above listed functions separatedly and in an all-in-one script.

          And this gui already programmed a course upload script for Moodle 2.x: http://moodle.org/mod/forum/discuss.php?d=173705

          Rosario

          Show
          Rosario Carcò added a comment - - edited As far as I have followed this question, I realize that course creation and user enrolment depend on very singular situations in the different schools and universities. That is why I started to program my own import script with a GUI AND an unattended mode version. See here: http://moodle.org/mod/forum/discuss.php?d=128317 With Anthnoy Borrow I considered to modify my script so as to: import/create categories (adding the function into the GUI, like upload users) import/create courses (adding the function into teh GUI, like upload users) import/create/enrol users (as already in the GUI) delete courses (adding the function into the GUI, like upload users) all-in-one script/function to do all this with one and the same .csv File and script (adding this function to the GUI, like upload users) all-in-one script that can be executed in unattended mode, during night, through a cron-job (already working for me and my needs. Our Workflow: courses maintained in our school administration system are exported nightly to a windows-share in .csv UTF-8 format. The csv file contains all the columns you can see in my template in the above thread, short everything needed to create new courses and enroll the given users in their respective roles the Moodle cron-job executes my uploadusersandcourses.php script every night writing back to the share a text file with error-codes like "course xyz already exists", or "category xyz does not exist", or "course xyz created", etc. so that we know exactly what the script did during night) My script only lacks deletion of courses as at our university we have to consider also archiving of old courses. So we could define what archiving means: copying only the backup-zip-File to a certain backup-server oder backup-archive-directory on the same server, and so on. Archiving could also mean MOVING the whole course to an archive category in the same Moodle server. Actually I move the courses through the GUI to an archive or toDelete category where I leave them hidden and without teachers for another 6 months before deleting them definitively or letting them in the archive category forever. But my script has also some more features (realized and planned): course copy/creation from a given template (backup-zip-file) of an existing course (working fine) rename course from old name to new name (because we have courses that change only their name before starting over for a next semester or academic year, not programmed yet, but feasible, scheduled for next release) archive course (has to be defined, not programmed yet) So we just had to join our efforts to get existing code of different programmers into the GUI and core of Moodle. I hope to find enough time before winter to realize the above listed functions separatedly and in an all-in-one script. And this gui already programmed a course upload script for Moodle 2.x: http://moodle.org/mod/forum/discuss.php?d=173705 Rosario
          Hide
          Eduardo Domínguez added a comment -

          Hi Rosario,

          as I wrote at some comments above, we've been doing all this kind of stuff since 2005 with XML (not CSV) with an "IMS Enterprise" file that is built-in in moodle (you don't need to maintain it) and it's an standard (we didn't want to reinvent the wheel and yes, we know some other people feels more comfortable with csv).

          The "IMS Enterprise" lacks the course creation from a given template (but you can rename a course, move it from one category to an other...), so I'll study the way to do this with the "IMS Enterprise" file, this will save lot of work to our staff.

          Ciao.

          Show
          Eduardo Domínguez added a comment - Hi Rosario, as I wrote at some comments above, we've been doing all this kind of stuff since 2005 with XML (not CSV) with an "IMS Enterprise" file that is built-in in moodle (you don't need to maintain it) and it's an standard (we didn't want to reinvent the wheel and yes, we know some other people feels more comfortable with csv). The "IMS Enterprise" lacks the course creation from a given template (but you can rename a course, move it from one category to an other...), so I'll study the way to do this with the "IMS Enterprise" file, this will save lot of work to our staff. Ciao.
          Hide
          Alejandro Gomez added a comment -

          Eduardo, I think the main reason behind csv is that we usually take the courses list from another system. Some of these may be made with old languages and can be a PITA to export data to xml. Ie: Fox and Clipper. It's easy to get a .dbf and convert it to csv and when the original programmer or source code from that system is not available it's a problem.

          Show
          Alejandro Gomez added a comment - Eduardo, I think the main reason behind csv is that we usually take the courses list from another system. Some of these may be made with old languages and can be a PITA to export data to xml. Ie: Fox and Clipper. It's easy to get a .dbf and convert it to csv and when the original programmer or source code from that system is not available it's a problem.
          Hide
          Eduardo Domínguez added a comment -

          To transform the csv to xml is a one-step procedure in excell using the concat() function. This is what I do when some teachers want me to create 50 courses, they give me a csv file and I add the last column (this is a short example and nobody needs a programmer to do this, but if you have one, you can automate this step in a couple of hours):

          id shortname longname Category Content of file.xml
          curs01 asdf1234 Maths for dummies Phisycs =CONCAT("<group><sourcedid><source>AGORA</source><id>";A2;"</id></sourcedid><description><short>";B2;"</short><long>";C2;"</long></description><org><orgunit>";D2;"</orgunit></org></group>")
          curs02 asdf1235 Physics for dummies Phisycs =CONCAT("<group><sourcedid><source>AGORA</source><id>";A3;"</id></sourcedid><description><short>";B3;"</short><long>";C3;"</long></description><org><orgunit>";D3;"</orgunit></org></group>")
          ... ... ... ... ...

          and so on.

          What I want to said is that lot of people have been waiting for an utility that is already made. I know that the people is busy and maybe they don't have time to read the IMS Enterprise specification, what I'm telling is that the solution to their needs has been there for years and that someone, like me, can tell them where it is.

          Show
          Eduardo Domínguez added a comment - To transform the csv to xml is a one-step procedure in excell using the concat() function. This is what I do when some teachers want me to create 50 courses, they give me a csv file and I add the last column (this is a short example and nobody needs a programmer to do this, but if you have one, you can automate this step in a couple of hours): id shortname longname Category Content of file.xml curs01 asdf1234 Maths for dummies Phisycs =CONCAT("<group><sourcedid><source>AGORA</source><id>";A2;"</id></sourcedid><description><short>";B2;"</short><long>";C2;"</long></description><org><orgunit>";D2;"</orgunit></org></group>") curs02 asdf1235 Physics for dummies Phisycs =CONCAT("<group><sourcedid><source>AGORA</source><id>";A3;"</id></sourcedid><description><short>";B3;"</short><long>";C3;"</long></description><org><orgunit>";D3;"</orgunit></org></group>") ... ... ... ... ... and so on. What I want to said is that lot of people have been waiting for an utility that is already made. I know that the people is busy and maybe they don't have time to read the IMS Enterprise specification, what I'm telling is that the solution to their needs has been there for years and that someone, like me, can tell them where it is.
          Hide
          Rosario Carcò added a comment - - edited

          Alejandro, Eduardo, I DID consider IMS Enterprise Import before I decided to program my own uploadusersandcourses.php and I made some posts at that time in the moodle forums. The problem is exactly as Alejandro says, other systems used for course and user management are all able to export into simple text or csv files, but not to IMS Enterprise format. So even if we/I could automate a csv into IMS transformation, this would be an extra step to control and maintain over years. And my intention was to offer the same GUI interface Moodle offers in upload users which uses csv files and to separately combine all the code into one all-in-one script you can schedule during night.

          Show
          Rosario Carcò added a comment - - edited Alejandro, Eduardo, I DID consider IMS Enterprise Import before I decided to program my own uploadusersandcourses.php and I made some posts at that time in the moodle forums. The problem is exactly as Alejandro says, other systems used for course and user management are all able to export into simple text or csv files, but not to IMS Enterprise format. So even if we/I could automate a csv into IMS transformation, this would be an extra step to control and maintain over years. And my intention was to offer the same GUI interface Moodle offers in upload users which uses csv files and to separately combine all the code into one all-in-one script you can schedule during night.
          Hide
          Paul Fynn added a comment -

          108 votes - how many does it take?
          :-~

          Show
          Paul Fynn added a comment - 108 votes - how many does it take? :-~
          Hide
          Steven Parker added a comment -

          Awesome thanks for the tip Eduardo, legend.

          Show
          Steven Parker added a comment - Awesome thanks for the tip Eduardo, legend.
          Hide
          Steve Brydges added a comment -

          Eduardo, I found your guidance on using the IMS Enterprise import to be very helpful, so I'd also like to add my thanks.
          However I think we still need a csv upload/delete.

          I'm just starting Moodle 2.x , and trying to visualise how it might be scaled to Enterprise level. I seem to be stuck at these issues:

          IMS Enterprise does not delete courses
          IMS Enterprise requires a unique course before creating a category (At the start I only want an empty structure)
          IMS Enterprise does not create sub-categories
          IMS Enterprise does not support course properties (number of sections etc.)

          It seems to be a fundamental restriction that IMS only knows about generic IMS things, whereas a CSV import could be targetted at Moodle concepts.

          Show
          Steve Brydges added a comment - Eduardo, I found your guidance on using the IMS Enterprise import to be very helpful, so I'd also like to add my thanks. However I think we still need a csv upload/delete. I'm just starting Moodle 2.x , and trying to visualise how it might be scaled to Enterprise level. I seem to be stuck at these issues: IMS Enterprise does not delete courses IMS Enterprise requires a unique course before creating a category (At the start I only want an empty structure) IMS Enterprise does not create sub-categories IMS Enterprise does not support course properties (number of sections etc.) It seems to be a fundamental restriction that IMS only knows about generic IMS things, whereas a CSV import could be targetted at Moodle concepts.
          Hide
          Eduardo Domínguez added a comment -

          Thanx to you two too,

          I missed some functionality to IMS Enterprise too (groups and grouping for instance) but IMS can be extended in any way you need. We'd extended the use to support WebDAV and SSO from our intranet.

          I'll post a suggestion to the developers to do the same job to IMS and CSV import. This makes no sense to have different functionality between two import systems from a file (csv, ims, ods...)

          Show
          Eduardo Domínguez added a comment - Thanx to you two too, I missed some functionality to IMS Enterprise too (groups and grouping for instance) but IMS can be extended in any way you need. We'd extended the use to support WebDAV and SSO from our intranet. I'll post a suggestion to the developers to do the same job to IMS and CSV import. This makes no sense to have different functionality between two import systems from a file (csv, ims, ods...)
          Hide
          Piers Harding added a comment -

          Hi -

          This may or may not be of any use to people on this thread, but I create a basic course uploader for 2.x modelled on the user uploader, as I needed something for NZ schools - it's available here https://gitorious.org/moodle-tool_uploadcourse/moodle-tool_uploadcourse for the curious.

          Cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - Hi - This may or may not be of any use to people on this thread, but I create a basic course uploader for 2.x modelled on the user uploader, as I needed something for NZ schools - it's available here https://gitorious.org/moodle-tool_uploadcourse/moodle-tool_uploadcourse for the curious. Cheers, Piers Harding.
          Hide
          Sean Keogh added a comment -

          Piers, I'm trying to use your course upload tool with Moodle 2.1.2, but I am unable to get Moodle to recognise it. Does it work with 2.1.x? I put the files in the relevant place, but nothing I do kicks off an installation so it never shows up on the menu.

          Sean K

          Show
          Sean Keogh added a comment - Piers, I'm trying to use your course upload tool with Moodle 2.1.2, but I am unable to get Moodle to recognise it. Does it work with 2.1.x? I put the files in the relevant place, but nothing I do kicks off an installation so it never shows up on the menu. Sean K
          Hide
          Piers Harding added a comment -

          Hi Sean - sorry, but I have only seen this just now. It is an admin tool plugin and these are only picked up from 2.2 onwards. I've recently updated the plugin to use templating from other courses, or a Moodle backup.

          Cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - Hi Sean - sorry, but I have only seen this just now. It is an admin tool plugin and these are only picked up from 2.2 onwards. I've recently updated the plugin to use templating from other courses, or a Moodle backup. Cheers, Piers Harding.
          Hide
          John Smith added a comment -

          Absolutely unbelievable that a tool used by universities across the world does not come with state-of-the-art mass import / export / handling tools... hopefully this is just a misunderstanding or I have just a bad dream.

          Show
          John Smith added a comment - Absolutely unbelievable that a tool used by universities across the world does not come with state-of-the-art mass import / export / handling tools... hopefully this is just a misunderstanding or I have just a bad dream.
          Hide
          Paul Fynn added a comment -

          Hi John,
          You get used to it after a while - Moodle still seems to be dogged by 'small system' perspectives.
          However, if you've moved to 2.02 plus you can try Piers Hardings contribution - this got us out of trouble last semester, and I believe that Piers has improved his plugin further have a look her: https://github.com/piersharding/moodle-tool_uploadcourse
          cheers
          Paul

          Show
          Paul Fynn added a comment - Hi John, You get used to it after a while - Moodle still seems to be dogged by 'small system' perspectives. However, if you've moved to 2.02 plus you can try Piers Hardings contribution - this got us out of trouble last semester, and I believe that Piers has improved his plugin further have a look her: https://github.com/piersharding/moodle-tool_uploadcourse cheers Paul
          Hide
          Piers Harding added a comment -

          John - I wouldn't be too harsh about it. Two things have happened in 2.x that make it possible to build better tools for managing courses -

          • the back up/restore engine has been improved immensely (I believe, thanks to Eloy)
          • there is now a great admin/tools plugin architecture

          These two things have made it possible for the creation of https://github.com/piersharding/moodle-tool_uploadcourse , and it is probably only a matter of time before something like this is available in core.

          Have a go, and let me know how you get on.

          Cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - John - I wouldn't be too harsh about it. Two things have happened in 2.x that make it possible to build better tools for managing courses - the back up/restore engine has been improved immensely (I believe, thanks to Eloy) there is now a great admin/tools plugin architecture These two things have made it possible for the creation of https://github.com/piersharding/moodle-tool_uploadcourse , and it is probably only a matter of time before something like this is available in core. Have a go, and let me know how you get on. Cheers, Piers Harding.
          Hide
          Susan Mangan added a comment -

          Excellent tool Piers, thanks so much for developing this! This is going to help us immensely. Quick question - do you know of a way to bulk create course categories as well? If not, no biggie, much easier to just create the categories first and then generate the courses. Thanks again!!

          Show
          Susan Mangan added a comment - Excellent tool Piers, thanks so much for developing this! This is going to help us immensely. Quick question - do you know of a way to bulk create course categories as well? If not, no biggie, much easier to just create the categories first and then generate the courses. Thanks again!!
          Hide
          Piers Harding added a comment -

          Hi Susan -
          No I don't know of a facility for bulk creating course categories - something to think about developing as a complementary tool.

          Cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - Hi Susan - No I don't know of a facility for bulk creating course categories - something to think about developing as a complementary tool. Cheers, Piers Harding.
          Hide
          Susan Mangan added a comment - - edited

          thx. just trying the delete function and it doesn't seem to be working. I am using deleted with a value of 1

          any ideas?

          .. oh never mind I figured it out!!

          Show
          Susan Mangan added a comment - - edited thx. just trying the delete function and it doesn't seem to be working. I am using deleted with a value of 1 any ideas? .. oh never mind I figured it out!!
          Hide
          Susan Mangan added a comment - - edited

          Does adding the teacher or editingteacher field allow you to actually assign that role to someone via the csv file?

          • I see you can do that using upload users. I should probably stop commenting here ... sorry all - last one
          Show
          Susan Mangan added a comment - - edited Does adding the teacher or editingteacher field allow you to actually assign that role to someone via the csv file? I see you can do that using upload users. I should probably stop commenting here ... sorry all - last one
          Hide
          Piers Harding added a comment -

          Hi -

          I just built a course category uploader here - https://github.com/piersharding/moodle-tool_uploadcoursecategory

          cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - Hi - I just built a course category uploader here - https://github.com/piersharding/moodle-tool_uploadcoursecategory cheers, Piers Harding.
          Hide
          Susan Mangan added a comment -

          Yay! thanks!

          Show
          Susan Mangan added a comment - Yay! thanks!
          Hide
          Susan Mangan added a comment -

          Hi Piers,

          I am using your tool to upload mass amt of courses but every once in a while it hiccups right after I upload the file and gives me an error:

          [Error occur during loading CSV file! More information about this error]

          which leads me here: http://docs.moodle.org/22/en/error/moodle/csvloaderror

          I can't make heads or tails of it. It seems if I do smaller chunks it works ok.. but I can't find a common denominator here. The original data it hiccupped on uploaded fine in smaller chunks.

          Any thoughts on why it would be tripping up?

          thanks

          Show
          Susan Mangan added a comment - Hi Piers, I am using your tool to upload mass amt of courses but every once in a while it hiccups right after I upload the file and gives me an error: [Error occur during loading CSV file! More information about this error] which leads me here: http://docs.moodle.org/22/en/error/moodle/csvloaderror I can't make heads or tails of it. It seems if I do smaller chunks it works ok.. but I can't find a common denominator here. The original data it hiccupped on uploaded fine in smaller chunks. Any thoughts on why it would be tripping up? thanks
          Hide
          Piers Harding added a comment -

          Hi Susan - It might be an idea to take this off this tracker as what I've developed isn't an official Moodle effort (of course, everyone is welcome to it), so you can contact me direct - piers at ompka dot net. Do you think it could be a timeout (or memory) related problem? How long does it run for before it dies? If you take the ones that are missing and continue processing them, do they now process ok?

          Cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - Hi Susan - It might be an idea to take this off this tracker as what I've developed isn't an official Moodle effort (of course, everyone is welcome to it), so you can contact me direct - piers at ompka dot net. Do you think it could be a timeout (or memory) related problem? How long does it run for before it dies? If you take the ones that are missing and continue processing them, do they now process ok? Cheers, Piers Harding.
          Hide
          Ray Lawrence added a comment -

          It's quite useful to have this here so that others who are interested can benefit. We can always opt out if the issue becomes too busy.

          Show
          Ray Lawrence added a comment - It's quite useful to have this here so that others who are interested can benefit. We can always opt out if the issue becomes too busy.
          Hide
          Paul Fynn added a comment -

          Susan,
          I've been tripped up by content length before.
          If content is too long for the receiving field .csv uploads sometimes seem to crash, but with no errror message.
          looking forwards to reports on the status of the program
          best Paul

          Show
          Paul Fynn added a comment - Susan, I've been tripped up by content length before. If content is too long for the receiving field .csv uploads sometimes seem to crash, but with no errror message. looking forwards to reports on the status of the program best Paul
          Hide
          Susan Mangan added a comment -

          Thanks for your feedback! Would it make sense to move this discussion to the forum?

          How difficult would it be to add enrollment options to this? For example, I have created 500 new courses and it would be beneficial if I could pre-load each course with the self-enrollment option and an initial password. (I'm not asking for much am I )

          Show
          Susan Mangan added a comment - Thanks for your feedback! Would it make sense to move this discussion to the forum? How difficult would it be to add enrollment options to this? For example, I have created 500 new courses and it would be beneficial if I could pre-load each course with the self-enrollment option and an initial password. (I'm not asking for much am I )
          Hide
          Robert Puffer added a comment -

          Relatively few users deal directly with the tracker so I strongly recommend taking prolonged, non-programatic discussions to the forums.

          Show
          Robert Puffer added a comment - Relatively few users deal directly with the tracker so I strongly recommend taking prolonged, non-programatic discussions to the forums.
          Hide
          Piers Harding added a comment -

          Hi Susan - support for creating enrolment options (not enrolling users) is something that I would like to add, but it is a reasonably complex piece of functionality, so it is not likely to be done for at least a few weeks. It would need to be flexible, as courses can have multiple enrolment methods, with each enrolment method having configurable options.

          Cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - Hi Susan - support for creating enrolment options (not enrolling users) is something that I would like to add, but it is a reasonably complex piece of functionality, so it is not likely to be done for at least a few weeks. It would need to be flexible, as courses can have multiple enrolment methods, with each enrolment method having configurable options. Cheers, Piers Harding.
          Hide
          Piers Harding added a comment -

          Hi -

          I've pushed a set of changes to https://github.com/piersharding/moodle-tool_uploadcourse to handle population of enrolment methods - please see the README for details.

          Cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - Hi - I've pushed a set of changes to https://github.com/piersharding/moodle-tool_uploadcourse to handle population of enrolment methods - please see the README for details. Cheers, Piers Harding.
          Hide
          Susan Mangan added a comment -

          Thanks!!!

          Show
          Susan Mangan added a comment - Thanks!!!
          Hide
          Patrick Landry added a comment -

          Has anyone modified this code so that it can be invoked from the command line, say from a cron job, without having to use the Moodle interface? If not, can anyone more knowledgeable than me comment on how much work that might be? Thanks for a great piece of code!

          Show
          Patrick Landry added a comment - Has anyone modified this code so that it can be invoked from the command line, say from a cron job, without having to use the Moodle interface? If not, can anyone more knowledgeable than me comment on how much work that might be? Thanks for a great piece of code!
          Hide
          Piers Harding added a comment -

          Hi Patrick - It hasn't been done, and it is probably a day or twos work to do. The majority of the logic could be pulled out and remain the same, but the front-end code that deals with input parameters for defaults, switches, and file source would have to be hauled out and remodelled.
          The other major piece of work would be coming up with a sane output format for all the information that gets reported back to the user on job completion.

          I wrote a framework for this kind of thing for Moodle 1.9 called moodlectl, so some of this could be pulled out, but it's still more work than I can do in the short term - I'll put it on the to do list. The very next thing that will be asked for is the same for User upload .

          Cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - Hi Patrick - It hasn't been done, and it is probably a day or twos work to do. The majority of the logic could be pulled out and remain the same, but the front-end code that deals with input parameters for defaults, switches, and file source would have to be hauled out and remodelled. The other major piece of work would be coming up with a sane output format for all the information that gets reported back to the user on job completion. I wrote a framework for this kind of thing for Moodle 1.9 called moodlectl, so some of this could be pulled out, but it's still more work than I can do in the short term - I'll put it on the to do list. The very next thing that will be asked for is the same for User upload . Cheers, Piers Harding.
          Hide
          Derek Chirnside added a comment -

          Just about to work on something like 47 new courses, and in a slightly grumpy mood throught I would check in to see if this 130 vote issue dating back over four years and lots of posts and hacks over the years had had any traction for being integrated. Not today. I wonder if it is quicker to create a bunch of these courses in a local install and then restore them one by one in the real destination?

          -Derek

          Show
          Derek Chirnside added a comment - Just about to work on something like 47 new courses, and in a slightly grumpy mood throught I would check in to see if this 130 vote issue dating back over four years and lots of posts and hacks over the years had had any traction for being integrated. Not today. I wonder if it is quicker to create a bunch of these courses in a local install and then restore them one by one in the real destination? -Derek
          Hide
          Paul Fynn added a comment -

          Just pre-testing this ahead of uploading. Piers - am I right in saying that the create course via existing course template does NOT automatically pick up the number of sections from the old course?
          many thanks
          Paul

          Show
          Paul Fynn added a comment - Just pre-testing this ahead of uploading. Piers - am I right in saying that the create course via existing course template does NOT automatically pick up the number of sections from the old course? many thanks Paul
          Hide
          Piers Harding added a comment -

          Hi Paul - that is what I would expect (NOT auto pickup) - are you experiencing something different? Can you give me an example?
          Cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - Hi Paul - that is what I would expect (NOT auto pickup) - are you experiencing something different? Can you give me an example? Cheers, Piers Harding.
          Hide
          Paul Fynn added a comment -

          Hi Piers,
          This may get a bit detailed - could we switch to http://moodle.org/mod/forum/discuss.php?d=173705&parent=854771 ?
          Thanks
          Paul

          Show
          Paul Fynn added a comment - Hi Piers, This may get a bit detailed - could we switch to http://moodle.org/mod/forum/discuss.php?d=173705&parent=854771 ? Thanks Paul
          Hide
          James Henestofel added a comment -

          Is there any movement on adding Piers Harding's code to core? There are a few things that I've noticed that need to be fixed to make it even possible to add to the core.

          For some reason I decided to build basically the same thing as Piers but very basic. Its built off the same setup as the Upload Users. It gives all the options that you get when creating courses manually.

          Repos is here on github https://github.com/henesnarfel/Moodle-Upload-Courses

          I feel that the work Piers has done was very good but just needs some updates, modifications, and cleanup to maybe be added to core someday. I think that to at least start getting the functionality into the core we should keep it simple. Then add to it as the community asks. That's why I created my very basic version.

          Let me know your thoughts

          Show
          James Henestofel added a comment - Is there any movement on adding Piers Harding's code to core? There are a few things that I've noticed that need to be fixed to make it even possible to add to the core. For some reason I decided to build basically the same thing as Piers but very basic. Its built off the same setup as the Upload Users. It gives all the options that you get when creating courses manually. Repos is here on github https://github.com/henesnarfel/Moodle-Upload-Courses I feel that the work Piers has done was very good but just needs some updates, modifications, and cleanup to maybe be added to core someday. I think that to at least start getting the functionality into the core we should keep it simple. Then add to it as the community asks. That's why I created my very basic version. Let me know your thoughts
          Hide
          Mary Evans added a comment -

          The problem with this tracker issue is that, although the initial idea is still valid, it wont be going anywhere at all as long as the Moodle versions, listed at the top of this page, stay as they are. You may not be aware of the fact that those version are no longer supported other than the odd security fix for Moodle 2.0.

          Show
          Mary Evans added a comment - The problem with this tracker issue is that, although the initial idea is still valid, it wont be going anywhere at all as long as the Moodle versions, listed at the top of this page, stay as they are. You may not be aware of the fact that those version are no longer supported other than the odd security fix for Moodle 2.0.
          Hide
          Mary Evans added a comment - - edited

          @Michael,

          I've just added you as a watcher, as I feel MDL-13114 will slip off the radar if not updated and triaged.

          Thanks
          Mary

          Show
          Mary Evans added a comment - - edited @Michael, I've just added you as a watcher, as I feel MDL-13114 will slip off the radar if not updated and triaged. Thanks Mary
          Hide
          Michael de Raadt added a comment -

          Thanks to the people who have contributed to this issue. It looks to be a popular improvement worth considering.

          Show
          Michael de Raadt added a comment - Thanks to the people who have contributed to this issue. It looks to be a popular improvement worth considering.
          Hide
          Derek Chirnside added a comment -

          @Michael, do you think anything will happen with this? Has anyone had a look at the code James has worked on?
          Just curious, before I go ahead and manually create 92 courses.

          I've checked out https://moodle.org/mod/forum/discuss.php?d=173705 but the latest activity is in August.

          -Derek

          Show
          Derek Chirnside added a comment - @Michael, do you think anything will happen with this? Has anyone had a look at the code James has worked on? Just curious, before I go ahead and manually create 92 courses. I've checked out https://moodle.org/mod/forum/discuss.php?d=173705 but the latest activity is in August. -Derek
          Hide
          Patrick Landry added a comment -

          @Derek, Just FYI I am using the latest code from Piers Harding referenced above to bulk upload courses to Moodle 2.2. I am also using uploadusersandcourses_silently.php by Rosario Carcò referenced above for Moodle 1.9.

          Show
          Patrick Landry added a comment - @Derek, Just FYI I am using the latest code from Piers Harding referenced above to bulk upload courses to Moodle 2.2. I am also using uploadusersandcourses_silently.php by Rosario Carcò referenced above for Moodle 1.9.
          Hide
          Derek Chirnside added a comment -

          @Patrick. Thanks. The reason I was re-visiting this issue is that I had some work to do in Moodle 2.3.2. Do you know if there is anything for this version yet?

          Show
          Derek Chirnside added a comment - @Patrick. Thanks. The reason I was re-visiting this issue is that I had some work to do in Moodle 2.3.2. Do you know if there is anything for this version yet?
          Hide
          Patrick Landry added a comment -

          @Derek, I have not used it but Rosario did post a version of his code for 2.3 in this thread:

          https://moodle.org/mod/forum/discuss.php?d=128317

          He calls it a beta version.

          Show
          Patrick Landry added a comment - @Derek, I have not used it but Rosario did post a version of his code for 2.3 in this thread: https://moodle.org/mod/forum/discuss.php?d=128317 He calls it a beta version.
          Hide
          Michael de Raadt added a comment -

          I can say that no one at HQ is working on this currently and it won't be in 2.4.

          With the popularity of the issue, the fact that it has a patch, and the fact that it doesn't replace existing functionality, I think it is a good candidate for the roadmap. I've recommended it.

          Show
          Michael de Raadt added a comment - I can say that no one at HQ is working on this currently and it won't be in 2.4. With the popularity of the issue, the fact that it has a patch, and the fact that it doesn't replace existing functionality, I think it is a good candidate for the roadmap. I've recommended it.
          Hide
          Piers Harding added a comment -

          Hi Michael -

          Which way is this leaning - is it towards the CSV upload admin tool that I've written or something else? If it is a matter of working through an acceptance process then I'm happy to work with a core dev on it.

          Cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - Hi Michael - Which way is this leaning - is it towards the CSV upload admin tool that I've written or something else? If it is a matter of working through an acceptance process then I'm happy to work with a core dev on it. Cheers, Piers Harding.
          Hide
          Dan Marsden added a comment -

          talked with David/Michael and have re-assigned this to myself - I'll pick this up after 2.4 has been released/branched and will see if we can get this into core.

          Show
          Dan Marsden added a comment - talked with David/Michael and have re-assigned this to myself - I'll pick this up after 2.4 has been released/branched and will see if we can get this into core.
          Hide
          Rosario Carcò added a comment - - edited

          Dear all, my offer is still valid. I reengineered my uploadusersandcourses.php and uploaduserandcourses_silently.php code to be used on Moodle 2.3

          It is working in production for us since August/September.

          As adding third party code/modules has become even easier in Moodle 2, there is no need to ask to incorporate such code into core, unless HQ wants to do it. We can have those pieces of code neatly added as I already did with my uploadusersandcourses.php - See my thread here, download and test it at your will. The GUI-version works almost out of the box, but read the according readme-file first:

          http://moodle.org/mod/forum/discuss.php?d=128317

          Show
          Rosario Carcò added a comment - - edited Dear all, my offer is still valid. I reengineered my uploadusersandcourses.php and uploaduserandcourses_silently.php code to be used on Moodle 2.3 It is working in production for us since August/September. As adding third party code/modules has become even easier in Moodle 2, there is no need to ask to incorporate such code into core, unless HQ wants to do it. We can have those pieces of code neatly added as I already did with my uploadusersandcourses.php - See my thread here, download and test it at your will. The GUI-version works almost out of the box, but read the according readme-file first: http://moodle.org/mod/forum/discuss.php?d=128317
          Hide
          Dan Marsden added a comment -

          Thanks Rosario - I think there's general consensus that it would be good to include a tool that performs this functionality into Moodle core but this will require a final +1 from Martin D before anything lands in core.

          I'll review your code and Piers code after 2.4 is branched and will check with Martin (the number of votes on this issue make this an easy case to make). At this stage I think adding this as an admin tool is the best approach but I haven't reviewed any of the code yet. (will probably do this sometime in Jan/Feb)

          Show
          Dan Marsden added a comment - Thanks Rosario - I think there's general consensus that it would be good to include a tool that performs this functionality into Moodle core but this will require a final +1 from Martin D before anything lands in core. I'll review your code and Piers code after 2.4 is branched and will check with Martin (the number of votes on this issue make this an easy case to make). At this stage I think adding this as an admin tool is the best approach but I haven't reviewed any of the code yet. (will probably do this sometime in Jan/Feb)
          Hide
          Rosario Carcò added a comment - - edited

          Dan, thanks for the great news. As you plan for Jan/Feb, this could also be my own pace, as I still have to reengineer also myCourses and siteNavigation blocks of mine to be used on Moodle 2.3 first. As I decided to use an all-in-one strategy mainly to use the _silently.php version as batch/cron job, we would have to decide whether to leave/put the logic of WHAT HAPPENS WITH THE COURSE either in the csv-file, as I actually implemented it with my ACTION-Column, where you write NEW (to create a new course), COPY (to copy an existing course), RENAME (to rename a course for the next semester/term, not implemented yet), ARCHIVE (to archive an existing course, but not defined yet, and not implemented yet), DELETE (to delete a course, not implemented yet) and so on. OR whether we wanted to extract/move this logic out of the csv-file into some Web-Form-Options in the GUI-Version, which means we would have to implement something like:

          BULK COURSE ACTIONS:

          • CREATE NEW COURSES
          • COPY EXISTING COURSES
          • RENAME EXISTING COURSES
          • ARCHIVE EXISTING COURSES (whatever and however this will be defined, eg. move only the mbk-zip-file into an ARCHIVE-Course or Category)
          • DELETE EXISTING COURSES (which could also be defined as archive first, then delete or move from the category it is in)
          • and so on

          uploaduser.php itself is a hybrid thing, containing columns with ACTIONS and offering a lot of GUI-options at the same time. So I think that deciding what way to go (complex csv file and lean GUI or simple/lean csv file and complex GUI options) is even more difficult than to reengineer our (the above mentioned) codes. If we want to stick to the idea of ONE csv file for everything, so that it can be used also in unattended batch/cron mode, AND in the GUI, we would have to proceed on my way I think, i.e. complex csv-file an lean GUI.

          Show
          Rosario Carcò added a comment - - edited Dan, thanks for the great news. As you plan for Jan/Feb, this could also be my own pace, as I still have to reengineer also myCourses and siteNavigation blocks of mine to be used on Moodle 2.3 first. As I decided to use an all-in-one strategy mainly to use the _silently.php version as batch/cron job, we would have to decide whether to leave/put the logic of WHAT HAPPENS WITH THE COURSE either in the csv-file, as I actually implemented it with my ACTION-Column, where you write NEW (to create a new course), COPY (to copy an existing course), RENAME (to rename a course for the next semester/term, not implemented yet), ARCHIVE (to archive an existing course, but not defined yet, and not implemented yet), DELETE (to delete a course, not implemented yet) and so on. OR whether we wanted to extract/move this logic out of the csv-file into some Web-Form-Options in the GUI-Version, which means we would have to implement something like: BULK COURSE ACTIONS: CREATE NEW COURSES COPY EXISTING COURSES RENAME EXISTING COURSES ARCHIVE EXISTING COURSES (whatever and however this will be defined, eg. move only the mbk-zip-file into an ARCHIVE-Course or Category) DELETE EXISTING COURSES (which could also be defined as archive first, then delete or move from the category it is in) and so on uploaduser.php itself is a hybrid thing, containing columns with ACTIONS and offering a lot of GUI-options at the same time. So I think that deciding what way to go (complex csv file and lean GUI or simple/lean csv file and complex GUI options) is even more difficult than to reengineer our (the above mentioned) codes. If we want to stick to the idea of ONE csv file for everything, so that it can be used also in unattended batch/cron mode, AND in the GUI, we would have to proceed on my way I think, i.e. complex csv-file an lean GUI.
          Hide
          Piers Harding added a comment -

          Hi -
          Just split the code out so that there is a command line version as well as the admin/tool - thanks to Moshe Golden for help - https://github.com/piersharding/moodle-tool_uploadcourse

          Cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - Hi - Just split the code out so that there is a command line version as well as the admin/tool - thanks to Moshe Golden for help - https://github.com/piersharding/moodle-tool_uploadcourse Cheers, Piers Harding.
          Hide
          Martin Dougiamas added a comment -

          Looks good (in principle) to me - needs a peer review from DEV for 2.5!

          Show
          Martin Dougiamas added a comment - Looks good (in principle) to me - needs a peer review from DEV for 2.5!
          Hide
          Dan Marsden added a comment -

          great - thanks Martin, updating this with Piers latest version - I'm pretty busy over Feb/March so it would be good if someone from HQ has time to peer review - if not I'll try to squeeze something in.

          Show
          Dan Marsden added a comment - great - thanks Martin, updating this with Piers latest version - I'm pretty busy over Feb/March so it would be good if someone from HQ has time to peer review - if not I'll try to squeeze something in.
          Hide
          Piers Harding added a comment -

          Just pushed a commit to pass local/codechecker. Will do the same for local/moodlecheck, and comment back here.

          Show
          Piers Harding added a comment - Just pushed a commit to pass local/codechecker. Will do the same for local/moodlecheck, and comment back here.
          Hide
          Mike Wilday added a comment -

          Does this ability include updating the course start date and corresponding dates in activities? Looking forward to seeing what this tool has to offer!

          Show
          Mike Wilday added a comment - Does this ability include updating the course start date and corresponding dates in activities? Looking forward to seeing what this tool has to offer!
          Hide
          Piers Harding added a comment -

          At the moment, you can process updates for the course startdate. It is a little uncertain what this whole feature will end up looking like, as it still needs to be reviewed/evaluated for acceptance.

          Cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - At the moment, you can process updates for the course startdate. It is a little uncertain what this whole feature will end up looking like, as it still needs to be reviewed/evaluated for acceptance. Cheers, Piers Harding.
          Hide
          Piers Harding added a comment -

          Just pushed another commit to ensure that it passes the local/moodlecheck as well to https://github.com/piersharding/moodle-tool_uploadcourse

          Show
          Piers Harding added a comment - Just pushed another commit to ensure that it passes the local/moodlecheck as well to https://github.com/piersharding/moodle-tool_uploadcourse
          Hide
          Dan Poltawski added a comment -

          Hi,

          I just skimmed the code (I'm afraid I don't have time to do a proper peer review now, but thought it better I comment about these things I spotted in case nobody else can get to it quickly):

          1. The cc_ prefix for the various functions would need to change to something a bit more verbsoe for core inclusion. I'd advocate moving to something OO like tool_uploadcourse and static methods on that. But i've not looked deeply.
          2. settings.php uses moodle/site:uploadusers, thats not right - its probably best to introduce a tool sepcific capability
          3. cc_process_course_upload should be split up - its 1000 lines long!
          4. Be good to see some unit tests
          Show
          Dan Poltawski added a comment - Hi, I just skimmed the code (I'm afraid I don't have time to do a proper peer review now, but thought it better I comment about these things I spotted in case nobody else can get to it quickly): The cc_ prefix for the various functions would need to change to something a bit more verbsoe for core inclusion. I'd advocate moving to something OO like tool_uploadcourse and static methods on that. But i've not looked deeply. settings.php uses moodle/site:uploadusers, thats not right - its probably best to introduce a tool sepcific capability cc_process_course_upload should be split up - its 1000 lines long! Be good to see some unit tests
          Hide
          Piers Harding added a comment -

          Hi -

          1 and 2 are easy to fix.

          3 presents a bit more of a challenge as uploadcourse is modeled on uploaduser - cc_process_course_upload() is just the equivalent of index.php for uploaduser - wrapped so that it can be called either interactively or via the command line.

          4 unit tests can be built with no difficulty when the decision is made whether to pursue incorporating uploadcourse. It will be easy to pump test data into cc_process_course_upload() for this purpose.

          Cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - Hi - 1 and 2 are easy to fix. 3 presents a bit more of a challenge as uploadcourse is modeled on uploaduser - cc_process_course_upload() is just the equivalent of index.php for uploaduser - wrapped so that it can be called either interactively or via the command line. 4 unit tests can be built with no difficulty when the decision is made whether to pursue incorporating uploadcourse. It will be easy to pump test data into cc_process_course_upload() for this purpose. Cheers, Piers Harding.
          Hide
          Richard van Iwaarden added a comment -

          @Piers,
          I tried to upload the example from http://docs.moodle.org/24/en/Bulk_course_upload#Example_2st_record_in_csv_file

          This is what I tried to upload:
          fullname;shortname;category;sortorder;idnumber;summary;format;showgrades;newsitems;teacher;teachers;student;students;startdate;numsections;maxbytes;visible;groupmode;timecreated;timemodified;password;enrolperiod;groupmodeforce;metacourse;lang;theme;cost;showreports;guest;enrollable;enrolstartdate;enrolenddate;notifystudents;expirynotify;expirythreshold;teacher1_role;teacher1_account
          Greatest Course;GC101;Education Portfolios;1;;University Portfolio;topics;0;0;Owner;Owners;Visitor;Visitors;1/14/2008;10;15728640;0;0;1/12/2008;1/12/2008;portfolio;0;0;0;;;;0;2;1;1/14/2008;5/10/2008;0;0;10;editingteacher;lastname1.firstname@email.edu

          This is the error I got:
          "sortorder" is geen geldige veldnaam [translation: sortorder is not a valid fieldname]

          Are there translation issues? Can you supply an example .csv file for me to test?

          Show
          Richard van Iwaarden added a comment - @Piers, I tried to upload the example from http://docs.moodle.org/24/en/Bulk_course_upload#Example_2st_record_in_csv_file This is what I tried to upload: fullname;shortname;category;sortorder;idnumber;summary;format;showgrades;newsitems;teacher;teachers;student;students;startdate;numsections;maxbytes;visible;groupmode;timecreated;timemodified;password;enrolperiod;groupmodeforce;metacourse;lang;theme;cost;showreports;guest;enrollable;enrolstartdate;enrolenddate;notifystudents;expirynotify;expirythreshold;teacher1_role;teacher1_account Greatest Course;GC101;Education Portfolios;1;;University Portfolio;topics;0;0;Owner;Owners;Visitor;Visitors;1/14/2008;10;15728640;0;0;1/12/2008;1/12/2008;portfolio;0;0;0;;;;0;2;1;1/14/2008;5/10/2008;0;0;10;editingteacher;lastname1.firstname@email.edu This is the error I got: "sortorder" is geen geldige veldnaam [translation: sortorder is not a valid fieldname] Are there translation issues? Can you supply an example .csv file for me to test?
          Hide
          Piers Harding added a comment -

          Hi Richard - the example you refer to is for a different course uploader, so I would not expect it to work. You can start with the documentation here https://github.com/piersharding/moodle-tool_uploadcourse and a simple file example is:
          fullname,shortname,category,idnumber,summary
          Students,Students,,Students,Students
          Staff,Staff,,Staff,Staff

          Cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - Hi Richard - the example you refer to is for a different course uploader, so I would not expect it to work. You can start with the documentation here https://github.com/piersharding/moodle-tool_uploadcourse and a simple file example is: fullname,shortname,category,idnumber,summary Students,Students,,Students,Students Staff,Staff,,Staff,Staff Cheers, Piers Harding.
          Hide
          Richard van Iwaarden added a comment -

          Thanks Piers, that explains it. Hope you can get this in the core 2.5 as there are a couple of versions going around now, hard to keep track

          Show
          Richard van Iwaarden added a comment - Thanks Piers, that explains it. Hope you can get this in the core 2.5 as there are a couple of versions going around now, hard to keep track
          Hide
          Rosario Carcò added a comment - - edited

          I am very busy these times. Can you quickly tell me whether the actual code will also create missing categories like "english-courses/beginners/first-semester" and the like, i.e. the whole category path where to put in the course we are creating, if that category-path does not exist yet. Marco Cassisa has implemented that code in my uploadusersandcourses.php and I will review his code asap. In a certain sense I am still advocating for my split up functionality at GUI level, as I proposed higher up already, sort of

          BULK COURSE ACTIONS:

          CREATE NEW COURSES
          COPY EXISTING COURSES
          RENAME EXISTING COURSES
          ARCHIVE EXISTING COURSES (whatever and however this will be defined, eg. move only the mbk-zip-file into an ARCHIVE-Course or Category)
          DELETE EXISTING COURSES (which could also be defined as archive first, then delete or move from the category it is in)
          CREATE NEW CATEGORIES (whole Path if it does not exist)
          COPY EXISTING CATEGORIES (including every subcategory/whole category path, including every course in cats and subcats
          and so on

          While still integrating everything into one silently uploadusersandcourses which becomes bulk_course_actions for unattended command-line/cron-jobs. Apologies if this is already the idea but I did not have time to read the docs on Pears' link.

          Show
          Rosario Carcò added a comment - - edited I am very busy these times. Can you quickly tell me whether the actual code will also create missing categories like "english-courses/beginners/first-semester" and the like, i.e. the whole category path where to put in the course we are creating, if that category-path does not exist yet. Marco Cassisa has implemented that code in my uploadusersandcourses.php and I will review his code asap. In a certain sense I am still advocating for my split up functionality at GUI level, as I proposed higher up already, sort of BULK COURSE ACTIONS: CREATE NEW COURSES COPY EXISTING COURSES RENAME EXISTING COURSES ARCHIVE EXISTING COURSES (whatever and however this will be defined, eg. move only the mbk-zip-file into an ARCHIVE-Course or Category) DELETE EXISTING COURSES (which could also be defined as archive first, then delete or move from the category it is in) CREATE NEW CATEGORIES (whole Path if it does not exist) COPY EXISTING CATEGORIES (including every subcategory/whole category path, including every course in cats and subcats and so on While still integrating everything into one silently uploadusersandcourses which becomes bulk_course_actions for unattended command-line/cron-jobs. Apologies if this is already the idea but I did not have time to read the docs on Pears' link.
          Hide
          Séverin Terrier added a comment -

          Talking about bulk course actions, i've created MDL-38501.

          Show
          Séverin Terrier added a comment - Talking about bulk course actions, i've created MDL-38501 .
          Hide
          Séverin Terrier added a comment -

          Piers : I've tested your tools to automatically create courses (or course categories), and they are useful !

          To be more usable, i think that some things should be changed : in README, you show which fields can be specified, but it would be really useful to have the list of required and optional fields (like in Upload user documentation : http://docs.moodle.org/en/Upload_users ).
          It seems that actually several fields are required by your tools (like idnumber), but are not (required) in Moodle. I'm not sure it's really a good idea. Would better let only Moodle required fields to be required.

          Show
          Séverin Terrier added a comment - Piers : I've tested your tools to automatically create courses (or course categories), and they are useful ! To be more usable, i think that some things should be changed : in README, you show which fields can be specified, but it would be really useful to have the list of required and optional fields (like in Upload user documentation : http://docs.moodle.org/en/Upload_users ). It seems that actually several fields are required by your tools (like idnumber), but are not (required) in Moodle. I'm not sure it's really a good idea. Would better let only Moodle required fields to be required.
          Hide
          Andrew Davis added a comment -

          Hello. This issue is missing testing instructions. Please add a description of how it should be tested remembering to include, where appropriate, verification that no existing functionality was broken.

          Show
          Andrew Davis added a comment - Hello. This issue is missing testing instructions. Please add a description of how it should be tested remembering to include, where appropriate, verification that no existing functionality was broken.
          Hide
          Frédéric Massart added a comment -

          Hi everyone,

          I spend some time reading through the code and trying out the plugin, it's a great job, thanks everyone!

          My main concerns about the code lead more towards the maintainability and robustness of the code. Currently it is really procedural and there are bits and pieces of code duplicated in different areas (lib, index.php, CLI), they could certainly be merged into one single place. Also, the CLI and the UI differ a bit in their implementation, and would be hard to maintain as both side should benefit any further improvement.

          I would highly recommend to wrap all the functionality in a class, which would include its own constants, different methods for each different operations (detect enrolment settings, restore a course, reset, ...), keep track of the amount of courses updated/deleted/created, the upload mode, etc.... It would also be easier to perform the checks like "If this course exists, then update the shortname and consider that it does not exist", and easier to perform the capability checks for specific fields (moodle/course:changecategory, moodle/course:changeidnumber, moodle/course:changeshortname) if need be.

          But most importantly, there is currently no Unit Test coverage, and while it would surely be possible to write some basic tests with the existing code, having a class would make it a lot easier and granular.

          In regard with the current functionalities, I wonder if we should really allow for course deletion as part of uploadcourses tool. Perhaps deletion should be a sub part of the plugin and completely distinct from the upload part. But then, maybe the module should be renamed 'Bulk course operations' or something... food for thoughts.

          About the current patch

          Here are some comments based on a use in the latest 2.6-dev.

          Need unit testing

          We do need a serious unit testing to prevent horrible things happening on existing courses!

          Use of deprecated objects

          • make_categories_list() is deprecated.
          • get_context_instance() is deprecated.
          • PARAM_MULTILANG is deprecated.

          Capability checks

          I am not sure about the current policy in regard with course restore and the capabilities associated to it, but should there be more granular checks when updating a current course? Some examples are:

          • moodle/course:visibility
          • moodle/course:reset
          • moodle/course:changesummary
          • moodle/course:changeshortname
          • ...

          Also, at the beginning of the script, the user requires course:create, course:update and course:delete but that might be too restrictive if the user is not trying to delete or update courses.

          More importantly, the tool needs to define its own capabilities and stop using 'moodle/site:uploadusers'. Also in settings.php, we should definitely not make use of 'moodle/restore:restorecourse', but of the soon-to-be-introduced capability.

          Needs clarification

          1. What is $bulk? It does not seem to be ever set to something else than 0.
          2. standardshortnames is quite ambiguous. What does it do? Shouldn't we ALWAYS clean the shortname?
          3. When is $backupfile not empty? And what is $roles in the logic associated to it?
          4. Is $CFG->keeptempdirectoriesonbackup relevant here, as far as I'm concerned, we are not really backuping a course, we're using this method to restore some content.
          5. The case where $optype === CC_COURSE_ADDNEW then if ($existingcourse) should never happen. Around line 500.
          6. In form::get_data() what is this description field?

          To consider

          Here are some comments on things that I think we should consider. Also, as this script could eventually process a lot of courses, keeping in mind performances is important.

          1. I am not a fan of oldshortname, could we consider keeping the existing shortname in its column and having another column rename_shortname_to or new_shortname?
          2. templatename considers none as empty, this should be avoided and replaced with a proper empty value.
          3. It seems quite expensive to extract a backup just to make sure it's valid. Shouldn't we extract it once and for all as it's going to be used by each course?
          4. I dislike enrolment_1 then using enroloption_1, I think it's more clear to use enrolment_[0-9]<option>. That would also allow for more same settings of the sort _anotherthing_1_option later on.
          5. Role renaming should be applied to all existing roles, not only the default ones.
          6. Deleting the course could happen right after checking if the course exists, rather than later.
          7. Support multilang filter. Ie: <span lang="en" class="multilang">English</span><span lang="fr" class="multilang">French</span> Cat. Detecting the course category does not support it.
          8. Speaking of categories... even though category IDs are harder to work with, they would solve any potential issue in resolving the path. Also we could have an option to fallback on the default one if the ID is not found. (Or category ID number, or one or the other)
          9. In the same was than 'Allow deletes' or 'Allow renames', there should be an 'Allow resets'.
          10. We should clearly define a list of mandatory fields for each upload type.
          11. There is a die() in the middle of the process when a backup could not be restored for some reason. I don't think we should halt the whole process, but perhaps just skip the current course.
          12. Do we really want to restore from another course, and then restore from a course file, and then reset eventually? Those 3 actions can be executed in a row.
          13. When restoring from a file, do we ensure that we do not restore user data? If not, should we?

          Minor comments

          1. We usually only have one @copyright per file, the person who created the file writes his name there and it does not change: http://docs.moodle.org/dev/Coding_style#.40copyright
          2. Could use _DIR_ with ../ instead of dirname().
          3. We do not use underscores in variable names.
          4. in if (in_array($field, $stdfields) or in_array($lcfield, $stdfields)) the later is enough, our fields should never use caps.
          5. Adding support for Summary files (new in 2.5)
          6. Form headers addElement('header') should have a name in order to be collapsible.
          7. ccshortname help button mentions to view the 'help' for examples, what help?
          8. There should not be any default 'grouping' as they are defined in the course itself.
          9. Do we need to flush some cache after/during the process?
          10. The dropdown to select a course to use as a template could potentially contain hundreds of courses, it would make it unusable. Perhaps, we should not support this and rely on the uploaded restore file only.
          11. The default course settings should reflect the new tree/structure/sections/labels/values introduced in Moodle 2.5
          12. The use of file as a reference to the CSV file is rather confusing, I often asked myself if we were referring to the restore file or the CSV. Rewording my help here (applies to UI and CLI).
          13. Strings in the lang file should ideally be alphabetically ordered
          14. It might be safer for later use to place the defaults values in the form in their respective array instead of mixing them with the options: $formdata['defaults']['foo'] = 'bar'; And if so, perhaps the function tool_uploadcourse_process_course_upload() could get the defaults as an argument.
          15. CC* variables and constants should be renamed to match the frankestyle name of this tool.
          16. The form should use setType() on 'ccshortname' and 'contextid'
          17. In tool_uploadcourse_process_template(), the letters flu do not match sfi in the callback function tool_uploadcourse_process_template_callback().

          I am willing to take over the work to be done from here, but we first need to decide whether we want to change the structure of this code or not?

          Thanks everyone for your involvement in this issue!

          Cheers,
          Fred

          Show
          Frédéric Massart added a comment - Hi everyone, I spend some time reading through the code and trying out the plugin, it's a great job, thanks everyone! My main concerns about the code lead more towards the maintainability and robustness of the code. Currently it is really procedural and there are bits and pieces of code duplicated in different areas (lib, index.php, CLI), they could certainly be merged into one single place. Also, the CLI and the UI differ a bit in their implementation, and would be hard to maintain as both side should benefit any further improvement. I would highly recommend to wrap all the functionality in a class, which would include its own constants, different methods for each different operations (detect enrolment settings, restore a course, reset, ...), keep track of the amount of courses updated/deleted/created, the upload mode, etc.... It would also be easier to perform the checks like "If this course exists, then update the shortname and consider that it does not exist", and easier to perform the capability checks for specific fields (moodle/course:changecategory, moodle/course:changeidnumber, moodle/course:changeshortname) if need be. But most importantly, there is currently no Unit Test coverage, and while it would surely be possible to write some basic tests with the existing code, having a class would make it a lot easier and granular. In regard with the current functionalities, I wonder if we should really allow for course deletion as part of uploadcourses tool. Perhaps deletion should be a sub part of the plugin and completely distinct from the upload part. But then, maybe the module should be renamed 'Bulk course operations' or something... food for thoughts. About the current patch Here are some comments based on a use in the latest 2.6-dev. Need unit testing We do need a serious unit testing to prevent horrible things happening on existing courses! Use of deprecated objects make_categories_list() is deprecated. get_context_instance() is deprecated. PARAM_MULTILANG is deprecated. Capability checks I am not sure about the current policy in regard with course restore and the capabilities associated to it, but should there be more granular checks when updating a current course? Some examples are: moodle/course:visibility moodle/course:reset moodle/course:changesummary moodle/course:changeshortname ... Also, at the beginning of the script, the user requires course:create, course:update and course:delete but that might be too restrictive if the user is not trying to delete or update courses. More importantly, the tool needs to define its own capabilities and stop using 'moodle/site:uploadusers'. Also in settings.php, we should definitely not make use of 'moodle/restore:restorecourse', but of the soon-to-be-introduced capability. Needs clarification What is $bulk? It does not seem to be ever set to something else than 0. standardshortnames is quite ambiguous. What does it do? Shouldn't we ALWAYS clean the shortname? When is $backupfile not empty? And what is $roles in the logic associated to it? Is $CFG->keeptempdirectoriesonbackup relevant here, as far as I'm concerned, we are not really backuping a course, we're using this method to restore some content. The case where $optype === CC_COURSE_ADDNEW then if ($existingcourse) should never happen. Around line 500. In form::get_data() what is this description field? To consider Here are some comments on things that I think we should consider. Also, as this script could eventually process a lot of courses, keeping in mind performances is important. I am not a fan of oldshortname , could we consider keeping the existing shortname in its column and having another column rename_shortname_to or new_shortname ? templatename considers none as empty , this should be avoided and replaced with a proper empty value. It seems quite expensive to extract a backup just to make sure it's valid. Shouldn't we extract it once and for all as it's going to be used by each course? I dislike enrolment_1 then using enroloption_1 , I think it's more clear to use enrolment_ [0-9] <option> . That would also allow for more same settings of the sort _anotherthing_1_option later on. Role renaming should be applied to all existing roles, not only the default ones. Deleting the course could happen right after checking if the course exists, rather than later. Support multilang filter. Ie: <span lang="en" class="multilang">English</span><span lang="fr" class="multilang">French</span> Cat . Detecting the course category does not support it. Speaking of categories... even though category IDs are harder to work with, they would solve any potential issue in resolving the path. Also we could have an option to fallback on the default one if the ID is not found. (Or category ID number, or one or the other) In the same was than 'Allow deletes' or 'Allow renames', there should be an 'Allow resets'. We should clearly define a list of mandatory fields for each upload type. There is a die() in the middle of the process when a backup could not be restored for some reason. I don't think we should halt the whole process, but perhaps just skip the current course. Do we really want to restore from another course, and then restore from a course file, and then reset eventually? Those 3 actions can be executed in a row. When restoring from a file, do we ensure that we do not restore user data? If not, should we? Minor comments We usually only have one @copyright per file, the person who created the file writes his name there and it does not change: http://docs.moodle.org/dev/Coding_style#.40copyright Could use _ DIR _ with ../ instead of dirname(). We do not use underscores in variable names. in if (in_array($field, $stdfields) or in_array($lcfield, $stdfields)) the later is enough, our fields should never use caps. Adding support for Summary files (new in 2.5) Form headers addElement('header') should have a name in order to be collapsible. ccshortname help button mentions to view the 'help' for examples, what help? There should not be any default 'grouping' as they are defined in the course itself. Do we need to flush some cache after/during the process? The dropdown to select a course to use as a template could potentially contain hundreds of courses, it would make it unusable. Perhaps, we should not support this and rely on the uploaded restore file only. The default course settings should reflect the new tree/structure/sections/labels/values introduced in Moodle 2.5 The use of file as a reference to the CSV file is rather confusing, I often asked myself if we were referring to the restore file or the CSV. Rewording my help here (applies to UI and CLI). Strings in the lang file should ideally be alphabetically ordered It might be safer for later use to place the defaults values in the form in their respective array instead of mixing them with the options: $formdata ['defaults'] ['foo'] = 'bar'; And if so, perhaps the function tool_uploadcourse_process_course_upload() could get the defaults as an argument. CC* variables and constants should be renamed to match the frankestyle name of this tool. The form should use setType() on 'ccshortname' and 'contextid' In tool_uploadcourse_process_template(), the letters flu do not match sfi in the callback function tool_uploadcourse_process_template_callback(). I am willing to take over the work to be done from here, but we first need to decide whether we want to change the structure of this code or not? Thanks everyone for your involvement in this issue! Cheers, Fred
          Hide
          Piers Harding added a comment -

          Hi Fred -

          Thanks for taking a look at this for me - it's much appreciated. I think all that you have pointed out, and your outlined approach for dealing with it is fine apart from the following:

          • I think there should be an option to select a course as template should remain - if the length of the list is the problem then a user should just type in a valid shortname
          • the oldshortname renaming process is modelled on the user uploader admin tool - infact the entire flavour and coding style is modelled on this deliberately as it seems like a supported core functional approach. However - the extended procedural nature of the code is now an obstacle, so it does need proper encapsulation.
          • the use of multiple copyrights was done to indicate where the ideas came from to create the code - irrespective of how little/great the contribution. If we were to strip it back down to who actually created the files then that would be me.

          All in all - I would be very happy if you were to take it forward from here - the main thing is that this functionality is long overdue in core Moodle, so if you can do that I'll be cheering you on.

          Thanks,
          Piers Harding.

          Show
          Piers Harding added a comment - Hi Fred - Thanks for taking a look at this for me - it's much appreciated. I think all that you have pointed out, and your outlined approach for dealing with it is fine apart from the following: I think there should be an option to select a course as template should remain - if the length of the list is the problem then a user should just type in a valid shortname the oldshortname renaming process is modelled on the user uploader admin tool - infact the entire flavour and coding style is modelled on this deliberately as it seems like a supported core functional approach. However - the extended procedural nature of the code is now an obstacle, so it does need proper encapsulation. the use of multiple copyrights was done to indicate where the ideas came from to create the code - irrespective of how little/great the contribution. If we were to strip it back down to who actually created the files then that would be me. All in all - I would be very happy if you were to take it forward from here - the main thing is that this functionality is long overdue in core Moodle, so if you can do that I'll be cheering you on. Thanks, Piers Harding.
          Hide
          Dan Marsden added a comment -

          Thanks Frédéric - great to hear you're able to pick this up and work on it (re-assigning this to you) - is this something you can finish for inclusion in 2.6?
          btw - when Piers first did this work Unit Tests weren't happening much in core and there wasn't a lot of point adding them until someone had managed to review the code and provide feedback on structure etc - great that you have managed to do this - thanks!

          Show
          Dan Marsden added a comment - Thanks Frédéric - great to hear you're able to pick this up and work on it (re-assigning this to you) - is this something you can finish for inclusion in 2.6? btw - when Piers first did this work Unit Tests weren't happening much in core and there wasn't a lot of point adding them until someone had managed to review the code and provide feedback on structure etc - great that you have managed to do this - thanks!
          Hide
          Derek Chirnside added a comment -

          How are you going Frédéric?
          And @Michael: what are the implications of taking this out of Sprint 1? Slight lowering of priority? Progress too slow?

          -Derek

          Show
          Derek Chirnside added a comment - How are you going Frédéric? And @Michael: what are the implications of taking this out of Sprint 1? Slight lowering of priority? Progress too slow? -Derek
          Hide
          Frédéric Massart added a comment -

          Hi Derek,

          I have been actively working on this issue, but I need to rewrite quite a lot of it to allow for proper Unit Testing. It has not really been taken out of the sprint, it has been pushed in the Sprint 2 as I could not finish it in the previous sprint. I am hoping to push it for peer review in the next couple of days.

          Cheers,
          Fred

          Show
          Frédéric Massart added a comment - Hi Derek, I have been actively working on this issue, but I need to rewrite quite a lot of it to allow for proper Unit Testing. It has not really been taken out of the sprint, it has been pushed in the Sprint 2 as I could not finish it in the previous sprint. I am hoping to push it for peer review in the next couple of days. Cheers, Fred
          Hide
          Dan Poltawski added a comment -

          Hi Fred,

          Just skim through at the end of the day with a big disclaimer that i've not fully considered it at all, but to write it down whilst its fresh in my head:

          1. The handling of the numeric shortname caught my attention
            1. I think we allow all numeric shortnames, so it could be problematic, if you are assuming that is a template placeholder.
            2. In a few places you are doing is_numeric checks for a placeholder. I wondered if you could instead do the conversion to a template on 'input'
          2. In the processor, rather than using in_array() with a list of options, I think that you could make the code easier to understand/modify with a switch statement or default (but that might be my personal preference)
          3. get_final_update_data/get_final_create_data phpdoc needs to clarify difference between them
          4. The OUTCOME_ xxx names are a bit confusing with moodle terminonology of outcomes being related to a course. I would suggest considering something different
          Show
          Dan Poltawski added a comment - Hi Fred, Just skim through at the end of the day with a big disclaimer that i've not fully considered it at all, but to write it down whilst its fresh in my head: The handling of the numeric shortname caught my attention I think we allow all numeric shortnames, so it could be problematic, if you are assuming that is a template placeholder. In a few places you are doing is_numeric checks for a placeholder. I wondered if you could instead do the conversion to a template on 'input' In the processor, rather than using in_array() with a list of options, I think that you could make the code easier to understand/modify with a switch statement or default (but that might be my personal preference) get_final_update_data/get_final_create_data phpdoc needs to clarify difference between them The OUTCOME_ xxx names are a bit confusing with moodle terminonology of outcomes being related to a course. I would suggest considering something different
          Hide
          Frédéric Massart added a comment - - edited

          Here is my first patch pushed for peer review.

          Dan, the shortname can be 0 according to course/edit.php, and so I implemented that check. Also, I do not want to clean the data as it could potentially lead to random behaviours. I also improved the readability following your comments.

          For further reference, at this stage, the CSV accepts the following data:

          Course data

          Field Description
          fullname
          shortname
          idnumber
          category Category ID
          category_idnumber
          category_path Each node separated by ' / '
          visible
          startdate Passed to strtotime()
          summary
          format
          theme
          lang
          newsitems
          showgrades
          showreports
          legacyfiles
          maxbytes
          groupmode
          groupmodeforce
          groupmodeforce
          enablecompletion

          I intentionally removed numsections and other course format related options because they should be handled properly for any course format possible.

          Required fields:

          • shortname (unless shortname template is set and mode does not allow for updates)
          • fullname (on create)
          • category (on create)

          Enrolment data

          Field Value Description
          enrolment_1 manual The enrolment 1 method
          enrolment_1_anyoption value The value for any option for enrolment 1
          enrolment_1_disable 1 Disables the enrolment method
          enrolment_1_delete 1 Deletes the enrolment method
          enrolment_1_enrolperiod 5 days Timestamp or passed to strtotime()
          enrolment_1_startdate 1st July 2012 Passed to strtotime()
          enrolment_1_enddate 1st July 2012 Passed to strtotime()
          enrolment_1_role teacher Role shortname

          Role renaming

          Use role_shortname.

          Field Value
          role_teacher Super Teacher
          role_student Great Student

          Extra options

          Field Description
          rename Shortname to rename the course with
          reset Reset the course after import
          delete Delete the course before any other update
          backupfile Backup file of a course to restore over the created/updated course
          templatecourse Shortname of a course to restore over the created/updated course

          There are some possible cache improvement, but I believe they could be part of subtask of this issue once we have a solid base. Same for the course format options, they could be delayed to another issue.

          Not sure what more information I can give so far, please ping me for any more information on how and why it works.

          Cheers!
          Fred

          Edit: Removed summary from the required fields.

          Show
          Frédéric Massart added a comment - - edited Here is my first patch pushed for peer review. Dan, the shortname can be 0 according to course/edit.php, and so I implemented that check. Also, I do not want to clean the data as it could potentially lead to random behaviours. I also improved the readability following your comments. For further reference, at this stage, the CSV accepts the following data: Course data Field Description fullname shortname idnumber category Category ID category_idnumber category_path Each node separated by ' / ' visible startdate Passed to strtotime() summary format theme lang newsitems showgrades showreports legacyfiles maxbytes groupmode groupmodeforce groupmodeforce enablecompletion I intentionally removed numsections and other course format related options because they should be handled properly for any course format possible. Required fields : shortname (unless shortname template is set and mode does not allow for updates) fullname (on create) category (on create) Enrolment data Field Value Description enrolment_1 manual The enrolment 1 method enrolment_1_anyoption value The value for any option for enrolment 1 enrolment_1_disable 1 Disables the enrolment method enrolment_1_delete 1 Deletes the enrolment method enrolment_1_enrolperiod 5 days Timestamp or passed to strtotime() enrolment_1_startdate 1st July 2012 Passed to strtotime() enrolment_1_enddate 1st July 2012 Passed to strtotime() enrolment_1_role teacher Role shortname Role renaming Use role_shortname . Field Value role_teacher Super Teacher role_student Great Student Extra options Field Description rename Shortname to rename the course with reset Reset the course after import delete Delete the course before any other update backupfile Backup file of a course to restore over the created/updated course templatecourse Shortname of a course to restore over the created/updated course There are some possible cache improvement, but I believe they could be part of subtask of this issue once we have a solid base. Same for the course format options, they could be delayed to another issue. Not sure what more information I can give so far, please ping me for any more information on how and why it works. Cheers! Fred Edit: Removed summary from the required fields.
          Hide
          Dan Poltawski added a comment -

          Hi Fred,

          Great work, I really like the look of this!

          Some (mostly small) suggestions:

          admin/tool/uploadcourse/settings.php

          1. You don't need to do $CFG->admin, it'll be auto-translated

          admin/tool/uploadcourse/index.php

          2. I think you've got some leftover includes in there (at least, I can't see why you need clilib there)
          3. I dislike the variable/param name iid, how about importid? "Variable names should always be easy-to-read, meaningful lower-case English words."
          4. In the interface, i'm not especially convinced by the modes - I think it would be good to ask Mary Cooch if she can try it out, and she might able to help us come up with a simple way to explain it. I'm wondering if it should be something much more explicitly like "How to handle new courses". Its tricky - hoping Mary can help!

          admin/tool/uploadcourse/classes/course.php

          5. I continue to think you could split up prepare() into more manageable chunks, but I suppose you disagree
          6. Line 642 looks unfinished "// We cannot"
          7. Line 248: Phpdocs for function tool_uploadcourse_course::exists has incomplete parameters list
          8. Line 835: Phpdocs for function tool_uploadcourse_course::reset has incomplete parameters list

          admin/tool/uploadcourse/classes/helper.php

          9. Well firstly, I always think that using a static class property is slightly strange. Doesn't seem like there is need for it to be visible across the whole instance at the moment. But really quite soon you're gonna run into probably and need to add a reset method for unit testing, which would then make it make sense - but..
          10. Ideally you'd be using an add-hoc MUC cache and you'd get reseting for unit tests for free here. Its partly why we added the MUC framework, to ensure everything is using consistent reset. The code is trivial:

          $cache = cache::make_from_params(cache_store::MODE_SESSION, 'repository_skydrive', 'foldernamelist');
          $cache->set('key', $value);
          $cache->get('key');
          

          11. Line 30: Class tool_uploadcourse_helper does not have @copyright tag
          12. Line 30: Class tool_uploadcourse_helper does not have @license tag

          admin/tool/uploadcourse/classes/processor.php

          13. Line 260: Phpdocs for function tool_uploadcourse_processor::get_restore_content_dir has incomplete parameters list

          With the tests..

          15. When I run the whole of phpunit tests I got (maybe related to other reseting problems I mentioned)

          There was 1 error:
          
          1) tool_uploadcourse_helper_testcase::test_get_restore_content_dir
          backup_helper_exception: Backup is missing XML file: /Users/danp/moodles/pm/moodledata_phpu/temp/backup/86729a23efa6e5b12e37451c68a4c66d/moodle_backup.xml
          
          /Users/danp/moodles/pm/moodle/backup/util/helper/backup_general_helper.class.php:123
          /Users/danp/moodles/pm/moodle/admin/tool/uploadcourse/tests/helper_test.php:150
          /Users/danp/moodles/pm/moodle/lib/phpunit/classes/advanced_testcase.php:76
          
          To re-run:
           vendor/bin/phpunit tool_uploadcourse_helper_testcase admin/tool/uploadcourse/tests/helper_test.php
          
          FAILURES!
          Tests: 1967, Assertions: 36191, Errors: 1, Skipped: 5.
          

          16. Its great to see all this really well unit testable code. Can I suggest you run the code coverage report on the tests and try and get better coverage of some of the bits that are missing at the moment. Looks like this plugin can be a great example that we can show people of the sort of testing we want on new plugins

          17. For bonus points, you could add a few behat tests.

          18. There are a few trivial issues picked up by the code checker, it would be good to fix them. (Again to make this a shining example plugin )

          Once again, great work!

          Show
          Dan Poltawski added a comment - Hi Fred, Great work, I really like the look of this! Some (mostly small) suggestions: admin/tool/uploadcourse/settings.php 1. You don't need to do $CFG->admin, it'll be auto-translated admin/tool/uploadcourse/index.php 2. I think you've got some leftover includes in there (at least, I can't see why you need clilib there) 3. I dislike the variable/param name iid, how about importid? "Variable names should always be easy-to-read, meaningful lower-case English words." 4. In the interface, i'm not especially convinced by the modes - I think it would be good to ask Mary Cooch if she can try it out, and she might able to help us come up with a simple way to explain it. I'm wondering if it should be something much more explicitly like "How to handle new courses". Its tricky - hoping Mary can help! admin/tool/uploadcourse/classes/course.php 5. I continue to think you could split up prepare() into more manageable chunks, but I suppose you disagree 6. Line 642 looks unfinished "// We cannot" 7. Line 248: Phpdocs for function tool_uploadcourse_course::exists has incomplete parameters list 8. Line 835: Phpdocs for function tool_uploadcourse_course::reset has incomplete parameters list admin/tool/uploadcourse/classes/helper.php 9. Well firstly, I always think that using a static class property is slightly strange. Doesn't seem like there is need for it to be visible across the whole instance at the moment. But really quite soon you're gonna run into probably and need to add a reset method for unit testing, which would then make it make sense - but.. 10. Ideally you'd be using an add-hoc MUC cache and you'd get reseting for unit tests for free here. Its partly why we added the MUC framework, to ensure everything is using consistent reset. The code is trivial: $cache = cache::make_from_params(cache_store::MODE_SESSION, 'repository_skydrive', 'foldernamelist'); $cache->set('key', $value); $cache->get('key'); 11. Line 30: Class tool_uploadcourse_helper does not have @copyright tag 12. Line 30: Class tool_uploadcourse_helper does not have @license tag admin/tool/uploadcourse/classes/processor.php 13. Line 260: Phpdocs for function tool_uploadcourse_processor::get_restore_content_dir has incomplete parameters list With the tests.. 15. When I run the whole of phpunit tests I got (maybe related to other reseting problems I mentioned) There was 1 error: 1) tool_uploadcourse_helper_testcase::test_get_restore_content_dir backup_helper_exception: Backup is missing XML file: /Users/danp/moodles/pm/moodledata_phpu/temp/backup/86729a23efa6e5b12e37451c68a4c66d/moodle_backup.xml /Users/danp/moodles/pm/moodle/backup/util/helper/backup_general_helper.class.php:123 /Users/danp/moodles/pm/moodle/admin/tool/uploadcourse/tests/helper_test.php:150 /Users/danp/moodles/pm/moodle/lib/phpunit/classes/advanced_testcase.php:76 To re-run: vendor/bin/phpunit tool_uploadcourse_helper_testcase admin/tool/uploadcourse/tests/helper_test.php FAILURES! Tests: 1967, Assertions: 36191, Errors: 1, Skipped: 5. 16. Its great to see all this really well unit testable code. Can I suggest you run the code coverage report on the tests and try and get better coverage of some of the bits that are missing at the moment. Looks like this plugin can be a great example that we can show people of the sort of testing we want on new plugins 17. For bonus points, you could add a few behat tests. 18. There are a few trivial issues picked up by the code checker, it would be good to fix them. (Again to make this a shining example plugin ) Once again, great work!
          Hide
          Frédéric Massart added a comment - - edited

          Thanks a lot for your review Dan.

          I have taken car of all the comments you made above. The unit testing coverage has been improved though it does not answer 100% from PHP_CodeCoverage, but this is pretty minor. I have also added 2 basic Behat tests for bonus points, and MUC for caching. You were right, the failed tests were caused by the use of class variables.

          Would you mind having a second look at the Processor/Tracker classes as those are the ones I'm more concerned about in terms of design.

          I forgot to mention that I agree on the wording problem, but I think that would be easier to fix up once this is integrated, so that it can be reviewed in place by Mary, Helen, etc....

          Cheers,
          Fred

          Show
          Frédéric Massart added a comment - - edited Thanks a lot for your review Dan. I have taken car of all the comments you made above. The unit testing coverage has been improved though it does not answer 100% from PHP_CodeCoverage, but this is pretty minor. I have also added 2 basic Behat tests for bonus points, and MUC for caching. You were right, the failed tests were caused by the use of class variables. Would you mind having a second look at the Processor/Tracker classes as those are the ones I'm more concerned about in terms of design. I forgot to mention that I agree on the wording problem, but I think that would be easier to fix up once this is integrated, so that it can be reviewed in place by Mary, Helen, etc.... Cheers, Fred
          Hide
          Frédéric Massart added a comment -

          Pushing this for integration after chatting a bit with Dan about it.

          Further improvements that we should keep in mind:

          • Handling of course format options
          • Handling of course summary files
          • Caching when checking if a course exists (tricky as it needs proper very accurate invalidation)
          Show
          Frédéric Massart added a comment - Pushing this for integration after chatting a bit with Dan about it. Further improvements that we should keep in mind: Handling of course format options Handling of course summary files Caching when checking if a course exists (tricky as it needs proper very accurate invalidation)
          Hide
          Dan Poltawski added a comment -

          +1, sorry I forgot to press the button myself. I've played with it quite a bit and really like it

          Show
          Dan Poltawski added a comment - +1, sorry I forgot to press the button myself. I've played with it quite a bit and really like it
          Hide
          Elizabeth Dalton added a comment - - edited

          I'm sorry to have to ask a newbie question: how do I help test? I can set up a test server, but how do I get the code and/or patch? Thanks!

          Show
          Elizabeth Dalton added a comment - - edited I'm sorry to have to ask a newbie question: how do I help test? I can set up a test server, but how do I get the code and/or patch? Thanks!
          Hide
          Dan Poltawski added a comment -

          Hi Elizabeth,

          Thanks for volunteering! If you are familiar with checking out a git branch, you can use fred's branch: git://github.com/FMCorz/moodle.git MDL-13114-master

          Otherwise, if you can wait a week, this will hopefully get integrated and should be available for testing on qa.moodle.net.

          Show
          Dan Poltawski added a comment - Hi Elizabeth, Thanks for volunteering! If you are familiar with checking out a git branch, you can use fred's branch: git://github.com/FMCorz/moodle.git MDL-13114 -master Otherwise, if you can wait a week, this will hopefully get integrated and should be available for testing on qa.moodle.net.
          Hide
          Marina Glancy added a comment -

          Hi guys. This is really a huge work and very nice code!

          Fred, just to notice couple of things that don't break anything but may be worth looking at:

          • get_plugin_list() is deprecated, also does not make sense to cache it's result in MUC any more, it will only make it slower.
          • cache 'helper' should be persistent (I tried setting it to persistent and reduced unittest time by 30%). But you need to set some value for 'persistent_max_size' then to make it optimal. Why not have two cache definitions, one for lists of plugins/roles and another, with maximum size, - for categories and backup ids? Or maybe even three. Caches need good settings. 'simplekeys' can be used for anything except categories paths.
          Show
          Marina Glancy added a comment - Hi guys. This is really a huge work and very nice code! Fred, just to notice couple of things that don't break anything but may be worth looking at: get_plugin_list() is deprecated, also does not make sense to cache it's result in MUC any more, it will only make it slower. cache 'helper' should be persistent (I tried setting it to persistent and reduced unittest time by 30%). But you need to set some value for 'persistent_max_size' then to make it optimal. Why not have two cache definitions, one for lists of plugins/roles and another, with maximum size, - for categories and backup ids? Or maybe even three. Caches need good settings. 'simplekeys' can be used for anything except categories paths.
          Hide
          Marina Glancy added a comment -

          Fred, also the plugin name should be included in standard_plugins_list()

          Show
          Marina Glancy added a comment - Fred, also the plugin name should be included in standard_plugins_list()
          Hide
          Frédéric Massart added a comment -

          Thanks Marina,

          I have updated my patch to add the plugin to the standard ones, and I removed the use of the get_plugin_list() (and its local cache). In regard with the cache, I am not sure why there is such an improvement when executing the Unit Tests, the default cache for request is a static store, which does pretty much the same than the persistent cache.

          At first I was not going to implement MUC in that issue, but at the end I did to make it easier to invalidate the cache during Unit Tests. I don't mind improving the cache of this plugin but I don't think this is very important. The purpose of the caching here was only to prevent fetching twice the same result, it's not like the logic is used on every page of Moodle... it is only when uploading a CSV, which is already a heavy process that some caching takes place.

          Sure I could create new cache definitions, to use persistent and simpley keys. But using persistent with a request cache seems counter productive to me... the default store being a static store, and the logic to fetch the information from there being quite straightforward, what is the real deal? Well, maybe the key doesn't need to be hashed any more, but really, 30% of improvement?

          Also, that makes me think that we should actually never use persistent with a request cache. We end up caching things twice, for no apparent reason. And if I have my own specific store for request caching, then it will never be used because most of the data will be cached statically, which defeats the purpose again.

          Could we eventually delay this caching improvement to another issue, which could also handle the caching/invalidating of the exists methods from course which could be called a lot of time when incrementing shortnames?

          Cheers,
          Fred

          Show
          Frédéric Massart added a comment - Thanks Marina, I have updated my patch to add the plugin to the standard ones, and I removed the use of the get_plugin_list() (and its local cache). In regard with the cache, I am not sure why there is such an improvement when executing the Unit Tests, the default cache for request is a static store, which does pretty much the same than the persistent cache. At first I was not going to implement MUC in that issue, but at the end I did to make it easier to invalidate the cache during Unit Tests. I don't mind improving the cache of this plugin but I don't think this is very important. The purpose of the caching here was only to prevent fetching twice the same result, it's not like the logic is used on every page of Moodle... it is only when uploading a CSV, which is already a heavy process that some caching takes place. Sure I could create new cache definitions, to use persistent and simpley keys. But using persistent with a request cache seems counter productive to me... the default store being a static store, and the logic to fetch the information from there being quite straightforward, what is the real deal? Well, maybe the key doesn't need to be hashed any more, but really, 30% of improvement? Also, that makes me think that we should actually never use persistent with a request cache. We end up caching things twice, for no apparent reason. And if I have my own specific store for request caching, then it will never be used because most of the data will be cached statically, which defeats the purpose again. Could we eventually delay this caching improvement to another issue, which could also handle the caching/invalidating of the exists methods from course which could be called a lot of time when incrementing shortnames? Cheers, Fred
          Hide
          Frédéric Massart added a comment -

          Based on discussion with Sam, I've added one more commit to cache the cache loader. The persistent option also creates a singleton of the cache loader and returns it whenever the cache loader is requested again. To benefit from this improvement, but not using the persistent data which does not make much sense on request loaders (see comment above), I am now caching the cache loader instance into a static variable. It seems like Unit Tests are faster using this method.

          Show
          Frédéric Massart added a comment - Based on discussion with Sam, I've added one more commit to cache the cache loader . The persistent option also creates a singleton of the cache loader and returns it whenever the cache loader is requested again. To benefit from this improvement, but not using the persistent data which does not make much sense on request loaders (see comment above), I am now caching the cache loader instance into a static variable. It seems like Unit Tests are faster using this method.
          Hide
          Rosario Carcò added a comment -

          Fred, I feel that templatecourse and backupcourse are the same. In the logic of my own code I simply do:

          if a course-template file is given in the file or a course to copy from, the same procedure is started for "new" and "copy" courseaction. The existing course is backed up on the fly and restored to get a copy of it which becomes the new course.

          I still appreciate the big work having been done so far. Rosario

          Show
          Rosario Carcò added a comment - Fred, I feel that templatecourse and backupcourse are the same. In the logic of my own code I simply do: if a course-template file is given in the file or a course to copy from, the same procedure is started for "new" and "copy" courseaction. The existing course is backed up on the fly and restored to get a copy of it which becomes the new course. I still appreciate the big work having been done so far. Rosario
          Hide
          Frédéric Massart added a comment -

          Hi Rosario, I am not sure to follow what you mean. Here are the possible options:

          1. You upload a course backup to restore on top of the created courses.
          2. You select a course which will be backed up on the fly and restored on the created courses.
          3. In the CSV, for a specific course, you specify the path to a backup file.
          4. In the CSV, for a specific course, you specify the shortname of a course to create a backup from.

          Some options take precedence on others. So yes, you are right, those options are resulting in the same operation, but if you are offered to chose to restore from an existing course, or a backup file.

          Show
          Frédéric Massart added a comment - Hi Rosario, I am not sure to follow what you mean. Here are the possible options: You upload a course backup to restore on top of the created courses. You select a course which will be backed up on the fly and restored on the created courses. In the CSV, for a specific course, you specify the path to a backup file. In the CSV, for a specific course, you specify the shortname of a course to create a backup from. Some options take precedence on others. So yes, you are right, those options are resulting in the same operation, but if you are offered to chose to restore from an existing course, or a backup file.
          Hide
          Rosario Carcò added a comment -

          I am sorry, I was not aware that you include uploading a whole backup-file. But this is for certain another fine option.

          Show
          Rosario Carcò added a comment - I am sorry, I was not aware that you include uploading a whole backup-file. But this is for certain another fine option.
          Hide
          Elizabeth Dalton added a comment -

          Dan, I'm not as familiar as I need to be with git yet-- if I check out the master branch you indicated, does that get all of Moodle Core, or do I need to do that first? Feel free to point me toward "Moodle Git for Dummies" as appropriate. I do want to learn to do this so I can help with testing.

          Show
          Elizabeth Dalton added a comment - Dan, I'm not as familiar as I need to be with git yet-- if I check out the master branch you indicated, does that get all of Moodle Core, or do I need to do that first? Feel free to point me toward "Moodle Git for Dummies" as appropriate. I do want to learn to do this so I can help with testing.
          Hide
          Marina Glancy added a comment -

          Thanks everybody participating in the project, it has been integrated in master.

          The cache loader caching commit was not integrated, Sam has promised to raise a separate issue about request cache.

          Show
          Marina Glancy added a comment - Thanks everybody participating in the project, it has been integrated in master. The cache loader caching commit was not integrated, Sam has promised to raise a separate issue about request cache.
          Hide
          Susan Mangan added a comment -

          Is this going to be integrated into the next weekly build for 2.4.4 or is it just specific to 2.6?

          Show
          Susan Mangan added a comment - Is this going to be integrated into the next weekly build for 2.4.4 or is it just specific to 2.6?
          Hide
          Dan Marsden added a comment -

          Susan - stable releases like 2.4.x/2.5.x are only supported for bug fixes - this is a new feature so will only land in 2.6

          Show
          Dan Marsden added a comment - Susan - stable releases like 2.4.x/2.5.x are only supported for bug fixes - this is a new feature so will only land in 2.6
          Hide
          Marina Glancy added a comment -

          Susan, also it uses some 2.6-specific features like autoloading so unfortunately this plugin can't be installed on previous versions without modifications

          Show
          Marina Glancy added a comment - Susan, also it uses some 2.6-specific features like autoloading so unfortunately this plugin can't be installed on previous versions without modifications
          Hide
          Susan Mangan added a comment -

          No worries! I just wasn't sure. Thank you for letting me know. I'm happy to continue using Pier's script manually until we get to 2.6.

          Show
          Susan Mangan added a comment - No worries! I just wasn't sure. Thank you for letting me know. I'm happy to continue using Pier's script manually until we get to 2.6.
          Hide
          Dan Marsden added a comment -

          but of course... Piers original tool (which this feature is based on) should work on 2.5 if you need something in the interim:
          https://github.com/piersharding/moodle-tool_uploadcourse

          Show
          Dan Marsden added a comment - but of course... Piers original tool (which this feature is based on) should work on 2.5 if you need something in the interim: https://github.com/piersharding/moodle-tool_uploadcourse
          Hide
          Petr Škoda added a comment -

          Typos:

          • cli help says "--action=createnew", should it be "mode"?

          Not real bugs:
          1/ frontpage shortname collision always results in error - is that expected?
          2/ allow deletes,resets and allow renames are not documented, I guess help icon would help
          3/ why show shortname template if it is present in the csv file?
          4/ if I have collision in shortname but unique idnumber the template is not used - I am confused
          5/ where is the list of allowed CSV fields?
          6/ help says shortname is required, it is not - you can use template in web UI
          7/ for all update operations I would expect only idnumber or shortname + one other field to be required

          I did not find any real bugs, so feel free to create new issues (for usability improvements and improved docs) and mark this as tested.

          Show
          Petr Škoda added a comment - Typos: cli help says "--action=createnew", should it be "mode"? Not real bugs: 1/ frontpage shortname collision always results in error - is that expected? 2/ allow deletes,resets and allow renames are not documented, I guess help icon would help 3/ why show shortname template if it is present in the csv file? 4/ if I have collision in shortname but unique idnumber the template is not used - I am confused 5/ where is the list of allowed CSV fields? 6/ help says shortname is required, it is not - you can use template in web UI 7/ for all update operations I would expect only idnumber or shortname + one other field to be required I did not find any real bugs, so feel free to create new issues (for usability improvements and improved docs) and mark this as tested.
          Hide
          Frédéric Massart added a comment -

          Hi Petr,

          thanks for the thorough test!

          • I've added a commit to fix the typo

          1/ I am not sure what you mean here, I have been able to update my frontpage using the tool. That actually makes me think that should probably be prohibited, what do you think? If yes, it would be better to take care of it in another issue.
          2/ I have raised MDL-40728 to improve the UI documentation
          3/ This can be a bit confusing, the UI defines the defaults, where as the CSV defines a property for that particular course. Though, you cannot specify a shortname template (string to generate a shortname from) in the CSV itself, because it doesn't make sense as you already have the information on the same line... If you're talking about the template course (course to import content from after upload), this can be define in both the UI and the CSV.
          4/ The template is only used if it is safe to do so. The only occasion it would be used is when the user would expect it, by leaving the shortname field empty. I tried not to randomise the behaviours too much here. If you want to use a shortname template, you need to create new courses, and not provide any shortname.
          5/ For now, nowhere. MDL-40728 could add them in the UI, but Moodle Docs is probably the best place (nothing there yet).
          6/ You are right, this is wrong, where about have you seen that?
          7/ Only shortname is supposed to be required for that operation, ID number is never used as a course identifier. Could you describe how to replicate this? What other fields were mentioned as required?

          Many thanks!
          Fred

          Show
          Frédéric Massart added a comment - Hi Petr, thanks for the thorough test! I've added a commit to fix the typo 1/ I am not sure what you mean here, I have been able to update my frontpage using the tool. That actually makes me think that should probably be prohibited, what do you think? If yes, it would be better to take care of it in another issue. 2/ I have raised MDL-40728 to improve the UI documentation 3/ This can be a bit confusing, the UI defines the defaults, where as the CSV defines a property for that particular course. Though, you cannot specify a shortname template (string to generate a shortname from) in the CSV itself, because it doesn't make sense as you already have the information on the same line... If you're talking about the template course (course to import content from after upload), this can be define in both the UI and the CSV. 4/ The template is only used if it is safe to do so. The only occasion it would be used is when the user would expect it, by leaving the shortname field empty. I tried not to randomise the behaviours too much here. If you want to use a shortname template, you need to create new courses, and not provide any shortname. 5/ For now, nowhere. MDL-40728 could add them in the UI, but Moodle Docs is probably the best place (nothing there yet). 6/ You are right, this is wrong, where about have you seen that? 7/ Only shortname is supposed to be required for that operation, ID number is never used as a course identifier. Could you describe how to replicate this? What other fields were mentioned as required? Many thanks! Fred
          Hide
          Marina Glancy added a comment -

          the typo fixing in CLI is integrated.

          Show
          Marina Glancy added a comment - the typo fixing in CLI is integrated.
          Hide
          Petr Škoda added a comment - - edited

          Passing the test, the other cosmetic issues will be addressed separately.

          Show
          Petr Škoda added a comment - - edited Passing the test, the other cosmetic issues will be addressed separately.
          Hide
          Petr Škoda added a comment -

          1/ At first I accidentally added course with the same shortname as the frontpage, so I tried to select update or increment and after hitting the Preview button there was still the red X
          6/ there is only one help - it is the icon next to the Page title on the first form
          7/ For me idnumber is the external course identifier, shortname is internal identifier. So I would expect to be able to use any of them as unique identifier, their only difference is that idnumber may be empty.

          Show
          Petr Škoda added a comment - 1/ At first I accidentally added course with the same shortname as the frontpage, so I tried to select update or increment and after hitting the Preview button there was still the red X 6/ there is only one help - it is the icon next to the Page title on the first form 7/ For me idnumber is the external course identifier, shortname is internal identifier. So I would expect to be able to use any of them as unique identifier, their only difference is that idnumber may be empty.
          Hide
          Frédéric Massart added a comment -

          Thanks Petr, I have raised MDL-40737 to prevent any alteration of the front page. Thanks for spotting that! About the ID number, the original plugin didn't allow this feature, so I have to admit that it didn't occur to me. While I agree with you, I think we should leave it the way it is for now and wait for a feature request to allow the selection by ID number.

          Show
          Frédéric Massart added a comment - Thanks Petr, I have raised MDL-40737 to prevent any alteration of the front page. Thanks for spotting that! About the ID number, the original plugin didn't allow this feature, so I have to admit that it didn't occur to me. While I agree with you, I think we should leave it the way it is for now and wait for a feature request to allow the selection by ID number.
          Hide
          Damyon Wiese added a comment -

          Moodle has many old functions,
          And although they cause no malfunction,
          There comes a day,
          When they get deprecated away,
          And get and put on the list for expulsion.

          Thanks for all the reports/testing/fixes this week. This issue has been sent upstream.

          Show
          Damyon Wiese added a comment - Moodle has many old functions, And although they cause no malfunction, There comes a day, When they get deprecated away, And get and put on the list for expulsion. Thanks for all the reports/testing/fixes this week. This issue has been sent upstream.
          Hide
          Frédéric Massart added a comment -

          Many thanks to [Piers Harding, who originally created the plugin, and anyone who has been involved.

          As Dan Marsden mentioned above, for users of Moodle below 2.6, the original plugin is still available here: https://github.com/piersharding/moodle-tool_uploadcourse.

          Show
          Frédéric Massart added a comment - Many thanks to [ Piers Harding , who originally created the plugin, and anyone who has been involved. As Dan Marsden mentioned above, for users of Moodle below 2.6, the original plugin is still available here: https://github.com/piersharding/moodle-tool_uploadcourse .
          Hide
          Piers Harding added a comment -

          Hey - a big thank you Frédéric for all your hard work on getting this through and also to Dan, Petr, Marina, Damyon (especially for the poetry) and many more. I can now retire my plugin!

          Cheers,
          Piers Harding.

          Show
          Piers Harding added a comment - Hey - a big thank you Frédéric for all your hard work on getting this through and also to Dan, Petr, Marina, Damyon (especially for the poetry) and many more. I can now retire my plugin! Cheers, Piers Harding.
          Hide
          Alastair Hole added a comment -

          First off thanks to everyone for such a great new feature. A quick question - what is the recommended way to uniquely identify courses for deletion? I was expecting to be able to use the 'id' of the course but it looks not possible. If I use shortname and idnumber will it combine them? Is there an order of precedence? Cheers

          Show
          Alastair Hole added a comment - First off thanks to everyone for such a great new feature. A quick question - what is the recommended way to uniquely identify courses for deletion? I was expecting to be able to use the 'id' of the course but it looks not possible. If I use shortname and idnumber will it combine them? Is there an order of precedence? Cheers
          Hide
          Frédéric Massart added a comment -

          Hi Alastair, the plugin only uses the shortname as an identifier. Even if the ID number was used to identify the courses, you would not need to combine it with a shortname, because both will always be unique. The plugin also has a "Preview" mode, which would give you a quick overview of what actions would be undertaken when you will proceed with the CSV, so you should not end up with unexpected deletions. Cheers!

          Show
          Frédéric Massart added a comment - Hi Alastair, the plugin only uses the shortname as an identifier. Even if the ID number was used to identify the courses, you would not need to combine it with a shortname, because both will always be unique. The plugin also has a "Preview" mode, which would give you a quick overview of what actions would be undertaken when you will proceed with the CSV, so you should not end up with unexpected deletions. Cheers!
          Hide
          Alastair Hole added a comment -

          Thanks Frédéric
          Unfortunately as shortname is not constrained to be unique at a schema level (or even non-empty) we have a lot of courses that have been automatically created with no shortname, or with duplicate shortnames. Many of these do however have an ID number but I don't think this can be guaranteed to be unique either. Do you have any advice on how to tackle this? At the moment I think I will just have to modify the code to use the course ID as an identifier instead of the shortname as this seems be the only reliably unique and existent identifier.
          Cheers

          Show
          Alastair Hole added a comment - Thanks Frédéric Unfortunately as shortname is not constrained to be unique at a schema level (or even non-empty) we have a lot of courses that have been automatically created with no shortname, or with duplicate shortnames. Many of these do however have an ID number but I don't think this can be guaranteed to be unique either. Do you have any advice on how to tackle this? At the moment I think I will just have to modify the code to use the course ID as an identifier instead of the shortname as this seems be the only reliably unique and existent identifier. Cheers
          Hide
          Petr Škoda added a comment -

          Moodle short course names must be present and must be unique, if you somehow inserted them directly into the database it is your responsibility to fix them I am afraid.

          Show
          Petr Škoda added a comment - Moodle short course names must be present and must be unique, if you somehow inserted them directly into the database it is your responsibility to fix them I am afraid.
          Hide
          Frédéric Massart added a comment -

          I am surprised you didn't encounter any issues as Moodle really assumes that shortnames and ID numbers are unique... Indeed I would advise you to edit the plugin code to use IDs, though the best is probably to set proper shortnames in each of your courses.

          Show
          Frédéric Massart added a comment - I am surprised you didn't encounter any issues as Moodle really assumes that shortnames and ID numbers are unique... Indeed I would advise you to edit the plugin code to use IDs, though the best is probably to set proper shortnames in each of your courses.
          Hide
          Alastair Hole added a comment -

          Thanks for the advice - I have decided to copy the course id into the shortname where it is empty, and append it to the shortname where it is duplicated. For reference:
          UPDATE mdl_course SET shortname = id WHERE shortname = '';
          UPDATE mdl_course
          SET shortname = CONCAT(shortname,'-',id)
          WHERE shortname IN
          (
          SELECT shortname FROM
          (
          SELECT shortname
          FROM mdl_course
          GROUP BY shortname HAVING count(id) > 1
          )
          AS shortnames
          )

          Show
          Alastair Hole added a comment - Thanks for the advice - I have decided to copy the course id into the shortname where it is empty, and append it to the shortname where it is duplicated. For reference: UPDATE mdl_course SET shortname = id WHERE shortname = ''; UPDATE mdl_course SET shortname = CONCAT(shortname,'-',id) WHERE shortname IN ( SELECT shortname FROM ( SELECT shortname FROM mdl_course GROUP BY shortname HAVING count(id) > 1 ) AS shortnames )
          Hide
          Alastair Hole added a comment -

          I have a Moodle course that contains a comma in the shortname - it seems this is permitted by the Moodle interface for course creation. This obviously yields the following error from this tool:
          "There is something wrong with the format of the CSV file - please check the number of headings and columns match, and that the delimiter and file encoding are correct (don\t use comma-quoted as Moodle does not support it): Invalid CSV file format - number of columns is not constant!"

          This suggests that I may not escape the comma by simply enclosing it in quotes. Is there any way around this?
          Cheers,
          Alastair

          Show
          Alastair Hole added a comment - I have a Moodle course that contains a comma in the shortname - it seems this is permitted by the Moodle interface for course creation. This obviously yields the following error from this tool: "There is something wrong with the format of the CSV file - please check the number of headings and columns match, and that the delimiter and file encoding are correct (don\t use comma-quoted as Moodle does not support it): Invalid CSV file format - number of columns is not constant!" This suggests that I may not escape the comma by simply enclosing it in quotes. Is there any way around this? Cheers, Alastair
          Hide
          Frédéric Massart added a comment -

          I bet the problem is that your CSV is not a proper CSV. Try to open it in LibreOffice Calc or Excel and save it again. There should not be any issues with quoted fields, if there were the fields like "Summary" would be a real problem! We should actually update this message: "(don\t use comma-quoted as Moodle does not support it)" as it is incorrect.

          Show
          Frédéric Massart added a comment - I bet the problem is that your CSV is not a proper CSV. Try to open it in LibreOffice Calc or Excel and save it again. There should not be any issues with quoted fields, if there were the fields like "Summary" would be a real problem! We should actually update this message: "(don\t use comma-quoted as Moodle does not support it)" as it is incorrect.
          Hide
          Alastair Hole added a comment -

          Thanks for your reply Frédéric
          The CSV is exported from MySQL Workbench, I have pared it down to the offending line (that causes the aforementioned error):
          ---------
          shortname,deleted
          "WRLD POL, CSD",1
          --------

          According to RFC4180/2.6, "Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes." it seems as though the comma has been escaped in accordance with at least this particular definition of a proper CSV - would you agree that that is the correct method of escaping a comma?
          Thanks again
          Cheers,
          Alastair

          Show
          Alastair Hole added a comment - Thanks for your reply Frédéric The CSV is exported from MySQL Workbench, I have pared it down to the offending line (that causes the aforementioned error): --------- shortname,deleted "WRLD POL, CSD",1 -------- According to RFC4180/2.6, "Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes." it seems as though the comma has been escaped in accordance with at least this particular definition of a proper CSV - would you agree that that is the correct method of escaping a comma? Thanks again Cheers, Alastair
          Hide
          Frédéric Massart added a comment -

          Yes, I agree, and it works. I have just copied/pasted your small example into a new CSV file and I could preview the CSV without any issue. Have you selected the right encoding? Mine was UTF-8.

          Show
          Frédéric Massart added a comment - Yes, I agree, and it works. I have just copied/pasted your small example into a new CSV file and I could preview the CSV without any issue. Have you selected the right encoding? Mine was UTF-8.
          Hide
          Alastair Hole added a comment -

          Still no dice unfortunately - I am running this tool under Moodle 2.2.6 (Build: 20121112) however, do you think that might be the issue? Would you mind attaching the CSV you just created so I can try to rule out any encoding issues etc?
          Cheers

          Show
          Alastair Hole added a comment - Still no dice unfortunately - I am running this tool under Moodle 2.2.6 (Build: 20121112) however, do you think that might be the issue? Would you mind attaching the CSV you just created so I can try to rule out any encoding issues etc? Cheers
          Hide
          Alastair Hole added a comment -

          I think I'll just use tab delimited files as a workaround for now.
          Thanks for your help

          Show
          Alastair Hole added a comment - I think I'll just use tab delimited files as a workaround for now. Thanks for your help
          Hide
          Alastair Hole added a comment -

          I have had some limited success with using the tab delimited format. My issue now is that any quoted fields are unable to be located, e.g.:
          Course not deleted - missing 6 "Staff Area Old" 1

          Moodle doesn't seem to strip the quotes off so the shortname doesn't match - is this likely an issue with Moodle 2.2.6? Do quoted shortnames work for anyone else? I have a 2.5 install I can try

          Show
          Alastair Hole added a comment - I have had some limited success with using the tab delimited format. My issue now is that any quoted fields are unable to be located, e.g.: Course not deleted - missing 6 "Staff Area Old" 1 Moodle doesn't seem to strip the quotes off so the shortname doesn't match - is this likely an issue with Moodle 2.2.6? Do quoted shortnames work for anyone else? I have a 2.5 install I can try
          Hide
          Alastair Hole added a comment -

          Having tested in 2.5.0 it seems it was an issue with 2.2.6
          My solution is to use un-quoted, tab delimited data and 2.2.6 seems happy with that.
          Sorry to clutter these comments with my progress, hopefully it may be of some use to others.

          Show
          Alastair Hole added a comment - Having tested in 2.5.0 it seems it was an issue with 2.2.6 My solution is to use un-quoted, tab delimited data and 2.2.6 seems happy with that. Sorry to clutter these comments with my progress, hopefully it may be of some use to others.
          Hide
          Frédéric Massart added a comment -

          Hi Alastair. I'm glad you mentioned that you are using Moodle 2.2 because we have fixed the CSV comma/quote issues in 2.4, see MDL-34074.

          Show
          Frédéric Massart added a comment - Hi Alastair. I'm glad you mentioned that you are using Moodle 2.2 because we have fixed the CSV comma/quote issues in 2.4, see MDL-34074 .
          Hide
          Alastair Hole added a comment -

          Apologies for the confusion, we are using this in 2.2 to perform some course housekeeping prior to an upgrade to 2.5 - now working very nicely indeed with tab delimited data

          Show
          Alastair Hole added a comment - Apologies for the confusion, we are using this in 2.2 to perform some course housekeeping prior to an upgrade to 2.5 - now working very nicely indeed with tab delimited data
          Hide
          Gus Hagelberg added a comment -

          It would be great if cohorts could be added to uploaded courses. Any chance on that? I've been using the third party upload course plug in and have missed this feature.

          Show
          Gus Hagelberg added a comment - It would be great if cohorts could be added to uploaded courses. Any chance on that? I've been using the third party upload course plug in and have missed this feature.
          Hide
          Mary Cooch added a comment -

          QA test added here for inclusion in 2.6 testing MDLQA-5738 Please note: I would be very grateful if someone would check the test out, as the course create process is very new to me

          Show
          Mary Cooch added a comment - QA test added here for inclusion in 2.6 testing MDLQA-5738 Please note: I would be very grateful if someone would check the test out, as the course create process is very new to me
          Hide
          Frédéric Massart added a comment - - edited

          Removing the label docs_required. http://docs.moodle.org/26/en/Upload_courses.

          Arg... I just realised that Mary Cooch had already done some work there... I'll try to merge those!

          Show
          Frédéric Massart added a comment - - edited Removing the label docs_required . http://docs.moodle.org/26/en/Upload_courses . Arg... I just realised that Mary Cooch had already done some work there... I'll try to merge those!
          Hide
          Mary Cooch added a comment -

          Thanks for your documentation work Fred; it is very much appreciated

          Show
          Mary Cooch added a comment - Thanks for your documentation work Fred; it is very much appreciated

            Dates

            • Created:
              Updated:
              Resolved:

              Agile