Moodle
  1. Moodle
  2. MDL-11996

bulk user upload - improvements, fixes and cleanup

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.9
    • Fix Version/s: 1.9
    • Component/s: Administration
    • Labels:
      None
    • Affected Branches:
      MOODLE_19_STABLE
    • Fixed Branches:
      MOODLE_19_STABLE

      Description

      todo:

      • review coding style
      • add charset conversion
      • set advanced elements in default form
      • limit auth plugins to only those that are active
      • review all older code
      • fox whitespace
      • use $USER instead of main admin for defaults
      • convert to use new file upload method from formslib
      • ignore empty rows in import file
      • add more default fields - idnumber, ajax
      • move settings above the defaults - more important first
      • make skip user default - counter is new feature, it is safer to skip anyway

        Gliffy Diagrams

          Issue Links

            Activity

            Hide
            Petr Skoda added a comment - - edited

            optionally:

            • preview cvs import
            • separator selector
            • more sanity checks and verification of file content
            • better visual feedback - table with import results, links to user accounts, etc.
            Show
            Petr Skoda added a comment - - edited optionally: preview cvs import separator selector more sanity checks and verification of file content better visual feedback - table with import results, links to user accounts, etc.
            Hide
            Dan Poltawski added a comment -

            Hi Petr,

            Do we not have a library for CSV processing now? I was doing some work on it a while back and i'm sure there was work on buffered readers?? It would be good if all the disparate bits of csv processing could be combined for something which deals with all the little gotchas

            Show
            Dan Poltawski added a comment - Hi Petr, Do we not have a library for CSV processing now? I was doing some work on it a while back and i'm sure there was work on buffered readers?? It would be good if all the disparate bits of csv processing could be combined for something which deals with all the little gotchas
            Hide
            Petr Skoda added a comment -

            Hello,
            no library yet

            I am testing a new patch just now, I have rewritten it again - working on performance/buffering too.
            I think we can extract the bits and pieces later and make proper library I am still not sure what is needed.

            Petr

            Show
            Petr Skoda added a comment - Hello, no library yet I am testing a new patch just now, I have rewritten it again - working on performance/buffering too. I think we can extract the bits and pieces later and make proper library I am still not sure what is needed. Petr
            Hide
            Dan Poltawski added a comment -

            Oh well, It would be great if we just had a single 'thing' which did the processing, so many things seem to try and achieve the same goal in moodle where CSV is concerned . (Forgetting things like BOM markers etc (understandably))

            I wrote something a while ago which would accept required/optional fields, and would just provide the data to the consumer, that feels like the way we should provide it? Rather than complex code which is embeded within the csv parsing?

            dan.

            Show
            Dan Poltawski added a comment - Oh well, It would be great if we just had a single 'thing' which did the processing, so many things seem to try and achieve the same goal in moodle where CSV is concerned . (Forgetting things like BOM markers etc (understandably)) I wrote something a while ago which would accept required/optional fields, and would just provide the data to the consumer, that feels like the way we should provide it? Rather than complex code which is embeded within the csv parsing? dan.
            Hide
            Petr Skoda added a comment -

            I did some coding for grade import several moths ago, the idea was to preprocess the data and push it into database, I think this would be a correct approach. Unfortunately we can not add new tables now

            Show
            Petr Skoda added a comment - I did some coding for grade import several moths ago, the idea was to preprocess the data and push it into database, I think this would be a correct approach. Unfortunately we can not add new tables now
            Hide
            Petr Skoda added a comment -

            Some more:

            • select processed users for bulk ops
            • different types of updates
            • special switch to allow password updates
            • role selection for typeN
            • etc., etc.

            And now the good new is I implemented all this over the weekend :-D

            Show
            Petr Skoda added a comment - Some more: select processed users for bulk ops different types of updates special switch to allow password updates role selection for typeN etc., etc. And now the good new is I implemented all this over the weekend :-D
            Hide
            Petr Skoda added a comment -

            oh, and custom profile fields...

            Show
            Petr Skoda added a comment - oh, and custom profile fields...
            Hide
            Petr Skoda added a comment -

            committed into cvs, the library is in lib/csvlib.class.php

            please test it

            Show
            Petr Skoda added a comment - committed into cvs, the library is in lib/csvlib.class.php please test it
            Hide
            Dan Poltawski added a comment -

            Cool!

            I have [quickly] adapted some existing unit tests I wrote for csv processing and attached them here. It doesn't really suite the library quite the same (because the emphasis was previously the field validation). Also, i've brutually just copied and pasted validate_user_upload_columns() for a quick test, but tests are tests I guess...

            It found that you can't specify field headers case insensitively in the header field name definitions (as apposed to the actual file) in validate_user_upload_columns().

            Enjoy

            Show
            Dan Poltawski added a comment - Cool! I have [quickly] adapted some existing unit tests I wrote for csv processing and attached them here. It doesn't really suite the library quite the same (because the emphasis was previously the field validation). Also, i've brutually just copied and pasted validate_user_upload_columns() for a quick test, but tests are tests I guess... It found that you can't specify field headers case insensitively in the header field name definitions (as apposed to the actual file) in validate_user_upload_columns(). Enjoy
            Hide
            Anthony Borrow added a comment -

            Petr - Thanks for the leaps forward in making user imports so much more powerful. Now we just need to add explanations of the new functionalities to:

            http://moodle.org/help.php?file=uploadusers.html

            and

            http://docs.moodle.org/en/Upload_users

            In order for the documentation to be as complete as possible, I was thinking we could use the sub-task list of this tracker item. But having a summary list of the improvements would be helpful if you have the time and then I would be happy to test them and document them. Peace - Anthony

            Show
            Anthony Borrow added a comment - Petr - Thanks for the leaps forward in making user imports so much more powerful. Now we just need to add explanations of the new functionalities to: http://moodle.org/help.php?file=uploadusers.html and http://docs.moodle.org/en/Upload_users In order for the documentation to be as complete as possible, I was thinking we could use the sub-task list of this tracker item. But having a summary list of the improvements would be helpful if you have the time and then I would be happy to test them and document them. Peace - Anthony
            Hide
            Petr Skoda added a comment -

            The basic docs are in cvs - click on the help icon next to the Upload users heading - http://moodle.cvs.sourceforge.net/*checkout*/moodle/moodle/lang/en_utf8/help/uploadusers2.html?revision=1.1.2.2&content-type=text%2Fplain

            Most of the new features should be fairly obvious - I tried to make the options in forms more self-explaining.

            Show
            Petr Skoda added a comment - The basic docs are in cvs - click on the help icon next to the Upload users heading - http://moodle.cvs.sourceforge.net/*checkout*/moodle/moodle/lang/en_utf8/help/uploadusers2.html?revision=1.1.2.2&content-type=text%2Fplain Most of the new features should be fairly obvious - I tried to make the options in forms more self-explaining.

              People

              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: