Moodle

bulk user upload - improvements, fixes and cleanup

Details

  • Type: Improvement Improvement
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 1.9
  • Fix Version/s: 1.9
  • Component/s: Administration
  • Labels:
    None
  • Affected Branches:
    MOODLE_19_STABLE
  • Fixed Branches:
    MOODLE_19_STABLE

Description

todo:

  • review coding style
  • add charset conversion
  • set advanced elements in default form
  • limit auth plugins to only those that are active
  • review all older code
  • fox whitespace
  • use $USER instead of main admin for defaults
  • convert to use new file upload method from formslib
  • ignore empty rows in import file
  • add more default fields - idnumber, ajax
  • move settings above the defaults - more important first
  • make skip user default - counter is new feature, it is safer to skip anyway

Issue Links

This issue will help resolve:
MDL-4499 Upload users creates multiple course enrollment Major Closed
MDL-9763 Password is overwritten when uploading a user a second time. Major Closed
MDL-4576 account with same email address Major Closed
MDL-3777 Duplicated entries when bulk uploading users Minor Closed
MDL-8941 Upload users enrolls badly formed users and reports no errors Minor Closed
MDL-10567 Upload users import parses CSV records with varying number of fields strangely. Minor Closed
MDL-4367 Incorrect character conversion during site user csv file upload Minor Closed
MDL-11819 User Import: Add ability to import custom user profile data Minor Closed
MDL-6242 Dilimiter ; and , Minor Closed
MDL-9770 Upload users tool can´t recognize custom user profile fields on uploaded csv files. Minor Closed
MDL-10458 User Upload should allow field separators other than comma Minor Closed
MDL-8228 Suggested sanity check code for uploaduser.php to prevent database corruption Minor Closed
MDL-9805 User Upload - Assign Roles Minor Closed
This issue has been marked as being related by:
MDL-11399 Users enrolled as student do not appear in participants list Critical Closed
MDL-11907 upload users doesn't support auth plugins Minor Closed
MDL-11079 CSV import allows accounts to be created with duplicated email addresses. Minor Closed
MDL-3982 Searching by username Minor Closed
MDL-10457 User Upload should check that e-mails are valid and unique Minor Closed

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Petr Škoda (skodak) added a comment - - edited

optionally:

  • preview cvs import
  • separator selector
  • more sanity checks and verification of file content
  • better visual feedback - table with import results, links to user accounts, etc.
Show
Petr Škoda (skodak) added a comment - - edited optionally:
  • preview cvs import
  • separator selector
  • more sanity checks and verification of file content
  • better visual feedback - table with import results, links to user accounts, etc.
Hide
Permalink
Dan Poltawski added a comment -

Hi Petr,

Do we not have a library for CSV processing now? I was doing some work on it a while back and i'm sure there was work on buffered readers?? It would be good if all the disparate bits of csv processing could be combined for something which deals with all the little gotchas

Show
Dan Poltawski added a comment - Hi Petr, Do we not have a library for CSV processing now? I was doing some work on it a while back and i'm sure there was work on buffered readers?? It would be good if all the disparate bits of csv processing could be combined for something which deals with all the little gotchas
Hide
Permalink
Petr Škoda (skodak) added a comment -

Hello,
no library yet

I am testing a new patch just now, I have rewritten it again - working on performance/buffering too.
I think we can extract the bits and pieces later and make proper library I am still not sure what is needed.

Petr

Show
Petr Škoda (skodak) added a comment - Hello, no library yet I am testing a new patch just now, I have rewritten it again - working on performance/buffering too. I think we can extract the bits and pieces later and make proper library I am still not sure what is needed. Petr
Hide
Permalink
Dan Poltawski added a comment -

Oh well, It would be great if we just had a single 'thing' which did the processing, so many things seem to try and achieve the same goal in moodle where CSV is concerned . (Forgetting things like BOM markers etc (understandably))

I wrote something a while ago which would accept required/optional fields, and would just provide the data to the consumer, that feels like the way we should provide it? Rather than complex code which is embeded within the csv parsing?

dan.

Show
Dan Poltawski added a comment - Oh well, It would be great if we just had a single 'thing' which did the processing, so many things seem to try and achieve the same goal in moodle where CSV is concerned . (Forgetting things like BOM markers etc (understandably)) I wrote something a while ago which would accept required/optional fields, and would just provide the data to the consumer, that feels like the way we should provide it? Rather than complex code which is embeded within the csv parsing? dan.
Hide
Permalink
Petr Škoda (skodak) added a comment -

I did some coding for grade import several moths ago, the idea was to preprocess the data and push it into database, I think this would be a correct approach. Unfortunately we can not add new tables now

Show
Petr Škoda (skodak) added a comment - I did some coding for grade import several moths ago, the idea was to preprocess the data and push it into database, I think this would be a correct approach. Unfortunately we can not add new tables now
Hide
Permalink
Petr Škoda (skodak) added a comment -

Some more:

  • select processed users for bulk ops
  • different types of updates
  • special switch to allow password updates
  • role selection for typeN
  • etc., etc.

And now the good new is I implemented all this over the weekend :-D

Show
Petr Škoda (skodak) added a comment - Some more:
  • select processed users for bulk ops
  • different types of updates
  • special switch to allow password updates
  • role selection for typeN
  • etc., etc.
And now the good new is I implemented all this over the weekend :-D
Hide
Permalink
Petr Škoda (skodak) added a comment -

oh, and custom profile fields...

Show
Petr Škoda (skodak) added a comment - oh, and custom profile fields...
Hide
Permalink
Petr Škoda (skodak) added a comment -

committed into cvs, the library is in lib/csvlib.class.php

please test it

Show
Petr Škoda (skodak) added a comment - committed into cvs, the library is in lib/csvlib.class.php please test it
Hide
Permalink
Dan Poltawski added a comment -

Cool!

I have [quickly] adapted some existing unit tests I wrote for csv processing and attached them here. It doesn't really suite the library quite the same (because the emphasis was previously the field validation). Also, i've brutually just copied and pasted validate_user_upload_columns() for a quick test, but tests are tests I guess...

It found that you can't specify field headers case insensitively in the header field name definitions (as apposed to the actual file) in validate_user_upload_columns().

Enjoy

Show
Dan Poltawski added a comment - Cool! I have [quickly] adapted some existing unit tests I wrote for csv processing and attached them here. It doesn't really suite the library quite the same (because the emphasis was previously the field validation). Also, i've brutually just copied and pasted validate_user_upload_columns() for a quick test, but tests are tests I guess... It found that you can't specify field headers case insensitively in the header field name definitions (as apposed to the actual file) in validate_user_upload_columns(). Enjoy
Hide
Permalink
Anthony Borrow added a comment -

Petr - Thanks for the leaps forward in making user imports so much more powerful. Now we just need to add explanations of the new functionalities to:

http://moodle.org/help.php?file=uploadusers.html

and

http://docs.moodle.org/en/Upload_users

In order for the documentation to be as complete as possible, I was thinking we could use the sub-task list of this tracker item. But having a summary list of the improvements would be helpful if you have the time and then I would be happy to test them and document them. Peace - Anthony

Show
Anthony Borrow added a comment - Petr - Thanks for the leaps forward in making user imports so much more powerful. Now we just need to add explanations of the new functionalities to: http://moodle.org/help.php?file=uploadusers.html and http://docs.moodle.org/en/Upload_users In order for the documentation to be as complete as possible, I was thinking we could use the sub-task list of this tracker item. But having a summary list of the improvements would be helpful if you have the time and then I would be happy to test them and document them. Peace - Anthony
Hide
Permalink
Petr Škoda (skodak) added a comment -

The basic docs are in cvs - click on the help icon next to the Upload users heading - http://moodle.cvs.sourceforge.net/*checkout*/moodle/moodle/lang/en_utf8/help/uploadusers2.html?revision=1.1.2.2&content-type=text%2Fplain

Most of the new features should be fairly obvious - I tried to make the options in forms more self-explaining.

Show
Petr Škoda (skodak) added a comment - The basic docs are in cvs - click on the help icon next to the Upload users heading - http://moodle.cvs.sourceforge.net/*checkout*/moodle/moodle/lang/en_utf8/help/uploadusers2.html?revision=1.1.2.2&content-type=text%2Fplain Most of the new features should be fairly obvious - I tried to make the options in forms more self-explaining.

People

Vote (0)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved:
Atlassian JIRA (v4.4.3#663-r165197) | Report a problem
Powered by a free Atlassian JIRA open source license for Moodle. Try JIRA - bug tracking software for your team.