Moodle
  1. Moodle
  2. MDL-11996

bulk user upload - improvements, fixes and cleanup

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.9
    • Fix Version/s: 1.9
    • Component/s: Administration
    • Labels:
      None
    • Affected Branches:
      MOODLE_19_STABLE
    • Fixed Branches:
      MOODLE_19_STABLE
    • Rank:
      34634

      Description

      todo:

      • review coding style
      • add charset conversion
      • set advanced elements in default form
      • limit auth plugins to only those that are active
      • review all older code
      • fox whitespace
      • use $USER instead of main admin for defaults
      • convert to use new file upload method from formslib
      • ignore empty rows in import file
      • add more default fields - idnumber, ajax
      • move settings above the defaults - more important first
      • make skip user default - counter is new feature, it is safer to skip anyway

        Issue Links

          Activity

          Hide
          Petr Škoda added a comment - - edited

          optionally:

          • preview cvs import
          • separator selector
          • more sanity checks and verification of file content
          • better visual feedback - table with import results, links to user accounts, etc.
          Show
          Petr Škoda added a comment - - edited optionally: preview cvs import separator selector more sanity checks and verification of file content better visual feedback - table with import results, links to user accounts, etc.
          Hide
          Dan Poltawski added a comment -

          Hi Petr,

          Do we not have a library for CSV processing now? I was doing some work on it a while back and i'm sure there was work on buffered readers?? It would be good if all the disparate bits of csv processing could be combined for something which deals with all the little gotchas

          Show
          Dan Poltawski added a comment - Hi Petr, Do we not have a library for CSV processing now? I was doing some work on it a while back and i'm sure there was work on buffered readers?? It would be good if all the disparate bits of csv processing could be combined for something which deals with all the little gotchas
          Hide
          Petr Škoda added a comment -

          Hello,
          no library yet

          I am testing a new patch just now, I have rewritten it again - working on performance/buffering too.
          I think we can extract the bits and pieces later and make proper library I am still not sure what is needed.

          Petr

          Show
          Petr Škoda added a comment - Hello, no library yet I am testing a new patch just now, I have rewritten it again - working on performance/buffering too. I think we can extract the bits and pieces later and make proper library I am still not sure what is needed. Petr
          Hide
          Dan Poltawski added a comment -

          Oh well, It would be great if we just had a single 'thing' which did the processing, so many things seem to try and achieve the same goal in moodle where CSV is concerned . (Forgetting things like BOM markers etc (understandably))

          I wrote something a while ago which would accept required/optional fields, and would just provide the data to the consumer, that feels like the way we should provide it? Rather than complex code which is embeded within the csv parsing?

          dan.

          Show
          Dan Poltawski added a comment - Oh well, It would be great if we just had a single 'thing' which did the processing, so many things seem to try and achieve the same goal in moodle where CSV is concerned . (Forgetting things like BOM markers etc (understandably)) I wrote something a while ago which would accept required/optional fields, and would just provide the data to the consumer, that feels like the way we should provide it? Rather than complex code which is embeded within the csv parsing? dan.
          Hide
          Petr Škoda added a comment -

          I did some coding for grade import several moths ago, the idea was to preprocess the data and push it into database, I think this would be a correct approach. Unfortunately we can not add new tables now

          Show
          Petr Škoda added a comment - I did some coding for grade import several moths ago, the idea was to preprocess the data and push it into database, I think this would be a correct approach. Unfortunately we can not add new tables now
          Hide
          Petr Škoda added a comment -

          Some more:

          • select processed users for bulk ops
          • different types of updates
          • special switch to allow password updates
          • role selection for typeN
          • etc., etc.

          And now the good new is I implemented all this over the weekend :-D

          Show
          Petr Škoda added a comment - Some more: select processed users for bulk ops different types of updates special switch to allow password updates role selection for typeN etc., etc. And now the good new is I implemented all this over the weekend :-D
          Hide
          Petr Škoda added a comment -

          oh, and custom profile fields...

          Show
          Petr Škoda added a comment - oh, and custom profile fields...
          Hide
          Petr Škoda added a comment -

          committed into cvs, the library is in lib/csvlib.class.php

          please test it

          Show
          Petr Škoda added a comment - committed into cvs, the library is in lib/csvlib.class.php please test it
          Hide
          Dan Poltawski added a comment -

          Cool!

          I have [quickly] adapted some existing unit tests I wrote for csv processing and attached them here. It doesn't really suite the library quite the same (because the emphasis was previously the field validation). Also, i've brutually just copied and pasted validate_user_upload_columns() for a quick test, but tests are tests I guess...

          It found that you can't specify field headers case insensitively in the header field name definitions (as apposed to the actual file) in validate_user_upload_columns().

          Enjoy

          Show
          Dan Poltawski added a comment - Cool! I have [quickly] adapted some existing unit tests I wrote for csv processing and attached them here. It doesn't really suite the library quite the same (because the emphasis was previously the field validation). Also, i've brutually just copied and pasted validate_user_upload_columns() for a quick test, but tests are tests I guess... It found that you can't specify field headers case insensitively in the header field name definitions (as apposed to the actual file) in validate_user_upload_columns(). Enjoy
          Hide
          Anthony Borrow added a comment -

          Petr - Thanks for the leaps forward in making user imports so much more powerful. Now we just need to add explanations of the new functionalities to:

          http://moodle.org/help.php?file=uploadusers.html

          and

          http://docs.moodle.org/en/Upload_users

          In order for the documentation to be as complete as possible, I was thinking we could use the sub-task list of this tracker item. But having a summary list of the improvements would be helpful if you have the time and then I would be happy to test them and document them. Peace - Anthony

          Show
          Anthony Borrow added a comment - Petr - Thanks for the leaps forward in making user imports so much more powerful. Now we just need to add explanations of the new functionalities to: http://moodle.org/help.php?file=uploadusers.html and http://docs.moodle.org/en/Upload_users In order for the documentation to be as complete as possible, I was thinking we could use the sub-task list of this tracker item. But having a summary list of the improvements would be helpful if you have the time and then I would be happy to test them and document them. Peace - Anthony
          Hide
          Petr Škoda added a comment -

          The basic docs are in cvs - click on the help icon next to the Upload users heading - http://moodle.cvs.sourceforge.net/*checkout*/moodle/moodle/lang/en_utf8/help/uploadusers2.html?revision=1.1.2.2&content-type=text%2Fplain

          Most of the new features should be fairly obvious - I tried to make the options in forms more self-explaining.

          Show
          Petr Škoda added a comment - The basic docs are in cvs - click on the help icon next to the Upload users heading - http://moodle.cvs.sourceforge.net/*checkout*/moodle/moodle/lang/en_utf8/help/uploadusers2.html?revision=1.1.2.2&content-type=text%2Fplain Most of the new features should be fairly obvious - I tried to make the options in forms more self-explaining.

            People

            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: