Moodle

Add a function to each module to report whether a given user has any data in a given activity

Details

  • Type: New Feature New Feature
  • Status: Open Open
  • Priority: Minor Minor
  • Resolution: Unresolved
  • Affects Version/s: 1.8
  • Fix Version/s: None
  • Component/s: Administration
  • Labels:
    None
  • Affected Branches:
    MOODLE_18_STABLE

Description

This is step one to implementing true deletion of users from the database: add a new required function to activity modules.

Moodle decision not to truly delete users is correct one. We don't want any user-relating activities dangling without having an owner.

However, as time progresses, our user tables (at least those not using external authentication methods like extDB or LDAP), keep swelling. It is my observation that after a few semesters, as the courses are being recycled for new students (admittedly not all sites do that), data for many inactive/deleted students is removed from Moodle. THat means we have a number of users in the database who have no data related to them and could be cleanly and safely removed.

The problem is to identify those users. This has to be done individually by each activity module since only they know how their data is related to specific users. Hence, I suggest to required all modules to add a function which reports whether a given user has any data for a specific activity. I don't mean a specific instance of an activity in a specificcourse but any instance of a given activity in any course.

This function can be then used for user overview and for checking whether a given user has any data.

Issue Links

Activity

Hide
Eric Villard added a comment -

Hi,

I've made a little addon to clean up the database from all informations relative to some users.
It does this, preserving some data (like forums threads...) to keep the coherence of exchanges, with the goal of taking advantage of the knowledge.

A thread was created on the Using Moodle General Developper Forum to keep a track of comments, ideas, criticism.
The link is [url=http://moodle.org/mod/forum/discuss.php?d=94496]http://moodle.org/mod/forum/discuss.php?d=94496[/url]

I think the goals of what you describe and the addon I've made are similar but their ways seem to be different.

Maybe can we use the thread mentioned before to centralize the need of every body and use this issue to formalize the feature, what do you think about ?

Eric

Show
Eric Villard added a comment - Hi, I've made a little addon to clean up the database from all informations relative to some users. It does this, preserving some data (like forums threads...) to keep the coherence of exchanges, with the goal of taking advantage of the knowledge. A thread was created on the Using Moodle General Developper Forum to keep a track of comments, ideas, criticism. The link is [url=http://moodle.org/mod/forum/discuss.php?d=94496]http://moodle.org/mod/forum/discuss.php?d=94496[/url] I think the goals of what you describe and the addon I've made are similar but their ways seem to be different. Maybe can we use the thread mentioned before to centralize the need of every body and use this issue to formalize the feature, what do you think about ? Eric
Hide
Eric Villard added a comment -

Excuse me, but I was thinking to be on the linked issue.
My comment would be more coherent on the MDL-7919 than here...

Eric

Show
Eric Villard added a comment - Excuse me, but I was thinking to be on the linked issue. My comment would be more coherent on the MDL-7919 than here... Eric
Hide
Robert Brenstein added a comment -

Those two issues are cross-linked, so people should find these comments here.

You are proposing the same with one difference: I meant to support deleting only users who have nothing with their name in any activity in any course, whereas you suggest to go further and support deleting anyone, just changing the ownership of their contributions to a generic "deleted user". There may be circumstances for either approach, so as long as site admins can control the deletion mode, it may make sense to have both options.

My reasoning was that in a typical school/university Moodle site, students leave the school and thus stop using Moodle. Their data may persist for some time, but eventually the courses or their contributions are removed from active site, and the only place those old students exist is the user database. Since most schools have a decent real user rotation, the user table keeps growing for no good reason, For example, I have a case where out of 1500 active users each semester 500 are new; and I believe I could readily delete a 1000 out of the 4500 user records but I am not 100% sure there have nothing posted.

My case is not unique. If one searches Moodle forums, there are more or less regularly posted requests for user deletion.

Show
Robert Brenstein added a comment - Those two issues are cross-linked, so people should find these comments here. You are proposing the same with one difference: I meant to support deleting only users who have nothing with their name in any activity in any course, whereas you suggest to go further and support deleting anyone, just changing the ownership of their contributions to a generic "deleted user". There may be circumstances for either approach, so as long as site admins can control the deletion mode, it may make sense to have both options. My reasoning was that in a typical school/university Moodle site, students leave the school and thus stop using Moodle. Their data may persist for some time, but eventually the courses or their contributions are removed from active site, and the only place those old students exist is the user database. Since most schools have a decent real user rotation, the user table keeps growing for no good reason, For example, I have a case where out of 1500 active users each semester 500 are new; and I believe I could readily delete a 1000 out of the 4500 user records but I am not 100% sure there have nothing posted. My case is not unique. If one searches Moodle forums, there are more or less regularly posted requests for user deletion.
Hide
Eric Villard added a comment -

The actual version of the plugin we propose, can actually delete every user, the administrator wants to.
I understand what you mean. We have the same problem at the difference that the school I'm working for, don't seem to have a decent real user rotation.
Most of the teachers forget to initialize their courses after their ends, and the moodle site is growing up year after year.
Some of users informations are not interesting to keep, but some of them are, like threads in forums, contributions in wikis or glossaries.
This is the kind of informations we proposed to keep, not all. Not broking threads for coherence purpose, keeping threads, contributions in wikis and glossaries, for taking advantage of the knowledge.
Other informations are entirely and definitively deleted from the database and the moodledata directories.

I've searched for informations about user deletion on moodle.org and this tracker and found some that I can group in two categories :
1. Mass user deletion
2. Real user deletion

Maybe there are some categories we can add ?

One of my question now is to know who must have the responsibility of the deletion ?
Must be a third part dedicated to this, every modules, both, or other ?

For the moment I engage my thought in the first way, but I may be wrong.
I have a basic implementation that seems to work, but the code is not enough proper to be proposed.
I'm completely opened on the idea of making it better and integrating some comments, ideas, criticisms...

Some of the purpose of administrators can be different, but I think it's more useful and coherent to have only one feature that manages deletion, than have many.

The principle of our plugin is to browse all the tables of the database in search of users ids.
So I think it can't be too hard to retrieve some informations to tell the administrator that users he wants to delete, have something or not with their names.
And also to let the possibility for the administrator to only set the delete flag in the user table to 1 (or 0 if he wants to active an account again...)

Eric

Show
Eric Villard added a comment - The actual version of the plugin we propose, can actually delete every user, the administrator wants to. I understand what you mean. We have the same problem at the difference that the school I'm working for, don't seem to have a decent real user rotation. Most of the teachers forget to initialize their courses after their ends, and the moodle site is growing up year after year. Some of users informations are not interesting to keep, but some of them are, like threads in forums, contributions in wikis or glossaries. This is the kind of informations we proposed to keep, not all. Not broking threads for coherence purpose, keeping threads, contributions in wikis and glossaries, for taking advantage of the knowledge. Other informations are entirely and definitively deleted from the database and the moodledata directories. I've searched for informations about user deletion on moodle.org and this tracker and found some that I can group in two categories : 1. Mass user deletion 2. Real user deletion Maybe there are some categories we can add ? One of my question now is to know who must have the responsibility of the deletion ? Must be a third part dedicated to this, every modules, both, or other ? For the moment I engage my thought in the first way, but I may be wrong. I have a basic implementation that seems to work, but the code is not enough proper to be proposed. I'm completely opened on the idea of making it better and integrating some comments, ideas, criticisms... Some of the purpose of administrators can be different, but I think it's more useful and coherent to have only one feature that manages deletion, than have many. The principle of our plugin is to browse all the tables of the database in search of users ids. So I think it can't be too hard to retrieve some informations to tell the administrator that users he wants to delete, have something or not with their names. And also to let the possibility for the administrator to only set the delete flag in the user table to 1 (or 0 if he wants to active an account again...) Eric
Hide
Robert Brenstein added a comment -

if you are willing to maintain your plugin to be compatible with all supported versions of Moodle and all commonly used (not just core) modules, then more power to you and Moodle. My suggestion was that we need to extend a module API to support such functionality.

The danger for your plugin is that some modules keep their data hierarchical, that is spread through multiple tables, not only tables with actual user ids. If you replace the id with a generic "deleted user" id, then all is fine. But if you try to actively delete things, it is potentially problematic.

Show
Robert Brenstein added a comment - if you are willing to maintain your plugin to be compatible with all supported versions of Moodle and all commonly used (not just core) modules, then more power to you and Moodle. My suggestion was that we need to extend a module API to support such functionality. The danger for your plugin is that some modules keep their data hierarchical, that is spread through multiple tables, not only tables with actual user ids. If you replace the id with a generic "deleted user" id, then all is fine. But if you try to actively delete things, it is potentially problematic.
Hide
Eric Villard added a comment -

So maybe a solution is to use the Moodle Event API like suggested by Valery to notify modules of the deletion of a user.
And then, either delegate to the module the responsibility of the deletion or let the module respond to the plugin it is authorized or not to perform the deletion.

I'm very new in Moodle and so don't have enough experience with it. I'm very interested in what you talking about.
Have you one or some examples of modules that keep their data hierarchical, and can be problematic with the way I've described ?

Have you any information about the module API you suggest to extend ?

Would you be ready to work with me on the specifications of a (large) user deletion functionality that group the two categories I've mentioned before and trying to solve all of the problematic we are talking about ?
If not, do you know some people interested by working on this ?

a big thanks to you Robert

Eric

Show
Eric Villard added a comment - So maybe a solution is to use the Moodle Event API like suggested by Valery to notify modules of the deletion of a user. And then, either delegate to the module the responsibility of the deletion or let the module respond to the plugin it is authorized or not to perform the deletion. I'm very new in Moodle and so don't have enough experience with it. I'm very interested in what you talking about. Have you one or some examples of modules that keep their data hierarchical, and can be problematic with the way I've described ? Have you any information about the module API you suggest to extend ? Would you be ready to work with me on the specifications of a (large) user deletion functionality that group the two categories I've mentioned before and trying to solve all of the problematic we are talking about ? If not, do you know some people interested by working on this ? a big thanks to you Robert Eric
Hide
Robert Brenstein added a comment -

Module API is described in developer docs. Most modules keep their data in multiple tables. The bigger issue from the point of view what you suggest is that the module developers are free to change their tables as they develop newer versions, so they are a moving target for any outside actions. This is why IMHO extending the module API is the way to go. Another big problem for your approach is that different Moodle sites have different modules, particularly contrib modules, and it is next to impossible for you to support all of them.

Show
Robert Brenstein added a comment - Module API is described in developer docs. Most modules keep their data in multiple tables. The bigger issue from the point of view what you suggest is that the module developers are free to change their tables as they develop newer versions, so they are a moving target for any outside actions. This is why IMHO extending the module API is the way to go. Another big problem for your approach is that different Moodle sites have different modules, particularly contrib modules, and it is next to impossible for you to support all of them.
Hide
Robert Brenstein added a comment -

I don't care whether this is done through explicit module API or events API as Valéry suggested. This is something for core people to decide. In either case, the first step is to get this officially added to API and supported by modules. Of course, nothing stops you develop the functionality specifically for your web site where you know exactly what you have, and later generalize this once the core support in available.

Show
Robert Brenstein added a comment - I don't care whether this is done through explicit module API or events API as Valéry suggested. This is something for core people to decide. In either case, the first step is to get this officially added to API and supported by modules. Of course, nothing stops you develop the functionality specifically for your web site where you know exactly what you have, and later generalize this once the core support in available.
Hide
Eric Villard added a comment -

Thanks Robert, I'm going to learn about the module API.

A part of my code is full documented but not all, and I'm working on it. The code is not very clean, cause of the need of a quick test purpose.

Here is the function that I use to dynamically retrieve all direct references of users in the database, have you some comments ?

/**

  • Returns all the table and column names where it exists a column which name inlcudes the string "userid"
    *
  • @uses $CFG
  • @uses $db
  • @return array a 0 based indexed 2-dimensional array of the recordset
    */
    function &usercleaner_get_tables() {
    global $CFG, $db;
    $sql = "SELECT TABLE_NAME, COLUMN_NAME
    FROM INFORMATION_SCHEMA.COLUMNS
    WHERE COLUMN_NAME LIKE '%userid%'
    AND TABLE_SCHEMA = '$CFG->dbname'";
    return $db->GetArray($sql);
    }

I was going in this way because I've seen that everybody tries to uniformize the labels of the fields containing ids of users in the database.
For the moment all our tests were well performed but it's true that we don't have many contrib modules installed.
But for all the one we have installed, this way seems to work.
The major problem I've encountered was for the deletion of users files, to recreate the file pathes.

Thanks for the time you spend, and please continue your comments and criticisms, they are very trainers for me.

Eric

Show
Eric Villard added a comment - Thanks Robert, I'm going to learn about the module API. A part of my code is full documented but not all, and I'm working on it. The code is not very clean, cause of the need of a quick test purpose. Here is the function that I use to dynamically retrieve all direct references of users in the database, have you some comments ? /**
  • Returns all the table and column names where it exists a column which name inlcudes the string "userid" *
  • @uses $CFG
  • @uses $db
  • @return array a 0 based indexed 2-dimensional array of the recordset */ function &usercleaner_get_tables() { global $CFG, $db; $sql = "SELECT TABLE_NAME, COLUMN_NAME FROM INFORMATION_SCHEMA.COLUMNS WHERE COLUMN_NAME LIKE '%userid%' AND TABLE_SCHEMA = '$CFG->dbname'"; return $db->GetArray($sql); }
I was going in this way because I've seen that everybody tries to uniformize the labels of the fields containing ids of users in the database. For the moment all our tests were well performed but it's true that we don't have many contrib modules installed. But for all the one we have installed, this way seems to work. The major problem I've encountered was for the deletion of users files, to recreate the file pathes. Thanks for the time you spend, and please continue your comments and criticisms, they are very trainers for me. Eric
Hide
Eric Villard added a comment -

hi,

i've joined a standalone version in a thread of the General developer forum : http://moodle.org/mod/forum/discuss.php?d=94496#p428775

Eric

Show
Eric Villard added a comment - hi, i've joined a standalone version in a thread of the General developer forum : http://moodle.org/mod/forum/discuss.php?d=94496#p428775 Eric
Hide
Piotr Kowalski added a comment -

Recently I tried to build an ad hoc function that removes all deleted users from all tables containing columns like “userid”. It works against Postgresql database (tested on Moodle 2.0.1 databases). It isn’t a plugin and may be run independently in a query environment like pgAdmin. I believe it can be easily adapted into MySQL.
Advantages of the function: an easy way to clean up database without removing courses, groups, etc.
Disadvantages: lack of removing users’ files, loosing threads on forums and emails.
The use of it is preferred in a strictly restricted Moodle configuration like mine where I have only lessons and quizzes, and haven’t neither emails nor forums. Otherwise one can use it when awares of results.

-------- sweeping...
DROP FUNCTION IF EXISTS remove_users();
DROP TABLE IF EXISTS tabnames;
DROP TABLE IF EXISTS results CASCADE;
DROP TABLE IF EXISTS ideleted CASCADE;
---------------------------------------
CREATE temporary TABLE tabnames (tabname varchar); – temporary store of tables names
CREATE temporary TABLE ideleted (id bigint); – temporary store of users' ids
CREATE temporary TABLE results (tabname text, counts bigint); – where to keep results log
INSERT INTO ideleted SELECT id from mdl_user WHERE deleted=1;
INSERT INTO tabnames
SELECT DISTINCT table_name FROM information_schema.columns WHERE column_name ='userid' ORDER BY table_name;
CREATE FUNCTION remove_users() RETURNS void AS $$
DECLARE tabname text;
counter bigint;
BEGIN
FOR tabname IN SELECT * FROM tabnames
LOOP
EXECUTE 'select count as counts from ' || tabname || ' where userid in (select id from ideleted)' INTO counter;
IF counter IS NOT NULL AND counter > 0 THEN
– Removes users from 'tabname' table
EXECUTE 'delete from ' || tabname || ' where userid in (select id from ideleted)';
– Inserts into the 'results' log table
EXECUTE 'insert into results values (''' || tabname || ''',' || counter || ')';
END IF;
END LOOP;
– Removes users from 'mdl_user' table
DELETE FROM mdl_user WHERE deleted=1;
--additional removing from other tables (aren't shown in results log)
DELETE FROM mdl_tag_instance WHERE tiuserid IN (SELECT id FROM ideleted);
DELETE FROM mdl_message WHERE useridfrom IN (SELECT id FROM ideleted) OR useridto IN (SELECT id FROM ideleted);
DELETE FROM mdl_message_read WHERE useridfrom IN (SELECT id FROM ideleted) OR useridto IN (SELECT id FROM ideleted);
RETURN;
END;
$$ LANGUAGE plpgsql;
– Run the procedure
SELECT remove_users();
– View how many users have been removed from 'tabname' tables
SELECT * FROM results ORDER BY tabname;

Show
Piotr Kowalski added a comment - Recently I tried to build an ad hoc function that removes all deleted users from all tables containing columns like “userid”. It works against Postgresql database (tested on Moodle 2.0.1 databases). It isn’t a plugin and may be run independently in a query environment like pgAdmin. I believe it can be easily adapted into MySQL. Advantages of the function: an easy way to clean up database without removing courses, groups, etc. Disadvantages: lack of removing users’ files, loosing threads on forums and emails. The use of it is preferred in a strictly restricted Moodle configuration like mine where I have only lessons and quizzes, and haven’t neither emails nor forums. Otherwise one can use it when awares of results. -------- sweeping... DROP FUNCTION IF EXISTS remove_users(); DROP TABLE IF EXISTS tabnames; DROP TABLE IF EXISTS results CASCADE; DROP TABLE IF EXISTS ideleted CASCADE; --------------------------------------- CREATE temporary TABLE tabnames (tabname varchar); – temporary store of tables names CREATE temporary TABLE ideleted (id bigint); – temporary store of users' ids CREATE temporary TABLE results (tabname text, counts bigint); – where to keep results log INSERT INTO ideleted SELECT id from mdl_user WHERE deleted=1; INSERT INTO tabnames SELECT DISTINCT table_name FROM information_schema.columns WHERE column_name ='userid' ORDER BY table_name; CREATE FUNCTION remove_users() RETURNS void AS $$ DECLARE tabname text; counter bigint; BEGIN FOR tabname IN SELECT * FROM tabnames LOOP EXECUTE 'select count as counts from ' || tabname || ' where userid in (select id from ideleted)' INTO counter; IF counter IS NOT NULL AND counter > 0 THEN – Removes users from 'tabname' table EXECUTE 'delete from ' || tabname || ' where userid in (select id from ideleted)'; – Inserts into the 'results' log table EXECUTE 'insert into results values (''' || tabname || ''',' || counter || ')'; END IF; END LOOP; – Removes users from 'mdl_user' table DELETE FROM mdl_user WHERE deleted=1; --additional removing from other tables (aren't shown in results log) DELETE FROM mdl_tag_instance WHERE tiuserid IN (SELECT id FROM ideleted); DELETE FROM mdl_message WHERE useridfrom IN (SELECT id FROM ideleted) OR useridto IN (SELECT id FROM ideleted); DELETE FROM mdl_message_read WHERE useridfrom IN (SELECT id FROM ideleted) OR useridto IN (SELECT id FROM ideleted); RETURN; END; $$ LANGUAGE plpgsql; – Run the procedure SELECT remove_users(); – View how many users have been removed from 'tabname' tables SELECT * FROM results ORDER BY tabname;

People

Vote (1)
Watch (6)

Dates

  • Created:
    Updated: