Could we discuss the problems via Skype please? (my id is petr.skoda)

Thank you for putting work in to a specification Enrique. I have also started a spec, which I am putting on the Wiki here,
http://docs.moodle.org/en/Development:Groupings_OU

Please don't think that I am trying to compete with you - I think it's useful for us all to lay out our "stalls" (to use an English expression), and I hope we can draw together your's, Petr's and my ideas to develop a comprehensive set of requirements.

I see adding grouping support in modules as essential, but also as a later step following a process of re-working the group code, and modifying the database schema. We can also work towards site-wide groups, but note that the work Juliette White did originally showed that many users (teachers etc.) don't need this, but they do need better support for groups in courses and activities.

I've had a look at the patch Enrique - a lot of useful code. I noticed that your approach to the user-interface to set the grouping for a course module was different to ours - we have a class 'MoodleQuickForm_modgrouping', similar to the class 'MoodleQuickForm_modgroupmode', and an extra parameter to the function 'standard_coursemodule_elements(... $supportsgroupings=false)'. See,
http://docs.moodle.org/en/Development:Groupings_OU#Groupings_for_course_modules

I look forward to exchanging views on Skype and/or the forums! Yours

Nick

Your spec and mine cover different aspects of Groups/groupings, one form the point of view of the user teacher and other from de code programmer. I think we would need to put together and consolidate the three views (with Petr) in a single document, or at least coherent parts of a whole. That's what Martin D and Moodle community expects, I think.

Agreement points:
a) Grouping support at modules is essential and first task to accomplish. I think Moodle 1.9 cannot lack this feature
b) Global groups are quite problematic, mainly because backup/restore: Delay adoption
c) 6-table database structure is OK: most flexible, keeps all possibilities open for future additions. I do not see a power argument to change it.

Disagreement:
I strongly oppose making mandatory that any group must belong to a grouping. This implies to create a "default grouping" to hold groups, makes arise the problem of "not in a grouping"/orphaned groups. This is also a problem when restoring old courses: they do not have a notion of "grouping". But this Groupings-first is not built in into DB or API, is a UI-imposed restriction.

UI criticism in 1.8/1.9
I think the main point to take into account is that the proposed UI insisted in creating Grouping first and then adding groups to a grouping. In pre-1.8 Moodle GROUPS are the main actors and I think that they should remain so: allow to create and manage groups independently of groupings. ONLY if needed, the teacher may use groupings as a way to organize groups and to separate sets of groups for use in different activities.

So I defend a groups-UI separate in tabs. One tab to manage groups and adding users to groups (the "classic") and another, separate, tab to manage groupings and populate groupings with groups. In this way the old-style logic and behavior is preserved, and new features may be introduced as new tabs as required.

API Code polishing
I basically agree with the changes indicated (MV,D). I do not see the rationale to maintain a separate lib/modulelib.php. The only "extra" point at module level is that groupingid is fixed. All functions in modulelib.php perform the same task as functions on basicgroups or grouping libs. For instance:

groups_m_get_group == groups_get_group_settings() or groups_groupids_to_groups()
groups_m_get_groups_for_user() == groups_get_groups_for_user_in_grouping() provided groupingid is known and passed

My point is that, in Python words, "better explicit than implicit". All m functions perform tasks that are covered by basic functions, provided groupingid parameter is passed. Why to complicate future maintaining tasks duplicating functions?

Summary: We all agree groupings at modules is THE feature to add in 1.9. I think the main point now is in groups UI.
Should we work to re-make the classic three pane UI with formslib? I think so

my priorities seem to be different from both of you. The global groupings is the most difficult problem we are facing, that is why we should solve it first. I am not saying we should implement it first, but we should IMO find some solution first and then talk about the rest.

Originally everything was contained in courses - nothing was shared between them. It was nice and easy and everything worked fine. In 1.7 was a major breakthrough - global roles. As expected anything shared between course was difficult to implement, there are still some problems left and the resulting code is much slower. Course enrolments are one of those problems, I would really like to use global groupings to work around some problems there too.

Groups shared in groupings might seem fine when thinking about course groups only, but it will IMO cause trouble when we introduce global groups. If we do not share groups we need only minimal access control in the groupings - capabilities in system context for global groupings and in course context for course groupings. If we share groups we need to deal with it at the group level which would be much more complicated.

Shared groups in groupings - do we really need it? What are the use cases? How often do you need to reuse one group in several groupings inside one course?

• Sharing groups in groupings from one course? Not really sure we need it - more complicated UI, more complicated code.
• Sharing of groups between groupings from different courses? No. Use global groupings instead.

I agree the UI in 1.8.0 is not good, I do not think we need to discuss it much. I always wanted separate grouping and group forms because not everybody needs groupings. The orphaned groups were really bad idea, internally there should IMO always exist default course grouping which would be hidden in UI if no other groupings used in course.

I am working on my proposal too, adding some more info and explanation...

Thanks everybody for working on this issue!!!

I agree that backup/restore is a problem, but if global groups are flagged and given a unique ID in DB this problem may be overcome, Am I right?
You mention the sharing of groups and access capabilities. Well, my view on that is quiet simple: Global groups must be locked (read-only items) at course level. The only thing a user may do with a global group at course level is to select it for use. Cannot add, edit, delete users change it in any way. The only access check available and meaningful at course level is - "is user X member of group G?" Nothing more.
My understanding of roles& permissions system is minimal so, probably, I am too blunt to see the complication,. Petr, please, could you explain what do you see as showstopper in the global groups-access capabilities interaction?

You demand use cases for sharing a group in groupings (I understand have a group belonging to several groupings).

Example A (from N Hansen http://moodle.org/mod/forum/discuss.php?d=52134)
She has a course with three units and three levels of payment
Pay1 give acces to unit1
Pay2 give acces to both unit 1 and 2
Pay3 give access to al units form 1 to 3.
Let say she has three groups Pay1-3 according to the amount payed (a convenient way to classify users).
Unit 1 activities would use GroupingA=(Pay1, Pay2, Pay3) ie: any student can see and participate in these activities
Unit 2 activities would use GroupingB=(Pay2, Pay3)
Unit3 activities would use GroupingC=(Pay3) only users in Pay3 group may see the content

I agree that all cases of group sharing between groupings can me managed duplicating the group or adding those users to another group in the other grouping. This flexibility, sharing within the course, is a matter on convenience.

My criticism is the other way around. Not about having a group in several groupings, what I see essential is to have groups without groupings. Your scheme imposes that every group must belong to a grouping (and afterwards, to only one). But pre-1.8 groups did not belong to any grouping. Yes, your scheme demands a hidden "default grouping", special and different from the rest of groupings. I see this as a complication. Once groupings and groups are separated having a group in 0, 1 or n groupings are natural cases.

I wholeheartedly agree that sharing course-based groups is an aberration. Read-only global groups are for that.

hmm, that example with different levels of payments does not seem right to me, I would expect this to be handled with roles and new type of enrolment plugins (multiple instances of paypal plugin each with different enrolment role and money). This is a special case and maybe using different courses + one shared would be possible - again improved enrolment plugins doing enrolments into several courses with one payment would be useful in this case. Are there any other known uses for shared groups in groupings?

Groups without a grouping were bad from the programmers view - the problem is the speed of SQL queries, you have to combine results of two different queries. Also to get better performance we have to integrate groups more with the roles framework - you often need to know if user is in some group and has capability to do something, or get all users with capability belonging to some group. In fact the hidden grouping is a major simplification from the coding point of view and speed improvement too.

Seems we are getting to some conclusions, we do not agree on:

1/ sharing of groups in course groupings

• minuses: more complex code + needs UI
• pluses: please describe, I could not find any yet - one good use case might change my mind

2/ orphan groups

• minuses: more complex code
• pluses: please describe - me, Martin and OU think there are none

-------

The relation of roles and groups is another problematic question. Some ppl proposed to assign roles in groups - it is not reasonable IMO. There should not be any group contexts for roles.

Your proposal to use groups to restrict access to activities seems to be OK for separate groups. However we can not use it for visible groups mode because it would break backwards compatibility (Am I right?). We might add a new "restricted" mode that would allow access into that activity if member of at least one group in selected grouping or maybe a new attribute in course_module to restrict access based on group membership, specifying group mode for resources sounds strange to me. The implementation should not be hard IMO, users should understand easily too. My +1 for that.

--------------

I did not speak much about the implementation yet, I would prefer to have one library file lib/grouplib.php with public functions used from modules + private ones used from UI, the lib/accesslib.php would contain some groups related code too. The ui for global and course groupings + groups should be in /group/ directory. I do not think there should be so many functions that are calling each other, sooner or later we will have to optimize db access with SQL queries instead of PHP loops, we might do that during implementation - why not? If groups are useful ppl will be using them more - I would not like to see performance problems, roles and enrolments are causing enough trouble already. Using global groupings for course enrolment purposes might help. Special global group enrolment plugins might be another improvement. (I will have to write some use cases, that should explain it easily.)

The session code for active grouping+group will have to be rewritten too, the current code has problems when having two browser windows each with different activity+grouping. Nothing serious, but there are some other hidden legacy problems and strange coding style in older code. We should IMHO trow out all older code and some new too and do a clean+faster implementation with new API that is based on needs found during module fixing - not as before having API first and then starting module conversions . I have already removed some unused/unfinished files and functions during my last groupings cleanup.
I do not think we need to keep the legacy API anymore, there were so many changes in 1.7/1.8/1.9 that no 3rd party code works anymore without modifications anyway - though there should be detailed docs describing the conversions and how to use new API.

The upgrade code should use sql if possible only, no calls to group functions. It is very tricky, we have to watch for changed functions used during upgrade which might be using tables before they are upgraded. We already have these headaches with roles code

Example B
Grouping Project A: ProjectA01, ProjectA02, ... ProjectA25 (25 groups of 3 people)
Each 3-member team work in a different topic These groups are used in a Wiki and a forum.
Grouping Writing: Writing01, Writing02, Writing03 .. Writing12 (12 groups of 5 people)
Each 5-member team work in a different topic These groups are used in a Wiki and a forum.

The teacher detects that Project03, Project17, Writing9 and Writing12 are working on related topics and have specific needs. He decided to add some resources for them and only them to see.
Set up a Resource page as separate groups using Grouping Special =(Project03, Project17, Writing9, Writing12)

I know, this may be done with other tools:
a) the additional material could be added to the Forum, but that means the teacher needs to replicate 4 times the same message (six, eight times?) Each time he wants to update information, the replication issue arises again
b) Creating a new Special group and putting it into Special Grouping. Ok ¿But what about people changing group? Teacher need to keep track that those on Project03, Project17, Writing9 and Writing12 were added to Special and review Special each time the other groups are changed. A task on teacher shoulders that supposedly CMS tool was making easier, automatic, not more complex.

Example C
Several teachers share a course. Each teaches a "Theory" group,
Grouping Sections= Theory01, Theory02, Theory03 each with one teacher in it
Forum News uses grouping Sections as separate groups: allows announcements for all students and for each sections independently.

Grouping Labs= Lab01, Lab02, ... Lab06. Each teacher is responsible for 2 Lab groups.
Wiki Lab results uses grouping Labs for students to write lab reports

Students are distributed at Lab and Theory groups at will: there is no rule like Theory01 students go to Lab01+Lab02 groups.

Each one of three teachers wants a "Doubts" forum accessible only for the students they teach, and
In addition, they want to add content (resources) private for their students
Solution: Each teacher have a Grouping. For instance, TeacherA grouping=(Theory1+Lab01+Lab03). In this way each teacher can add activities or resources that only their set of students can access.

Alternative: creating additional groups and adding users manually, again. I anticipate that if we go this route then in a near future we will discussing ways to "clone" groups and synchronize membership from one to other: Teachers demanding that members of GroupA become members of groupB automagically. Sharing groups makes this unnecessary.

Example D: overcoming a known problem in forum
When you have a forum working in "separate groups" mode the teacher can post a message to "all participants" but those messages are "read-only": students cannot reply to that post due to different groupid.
Course with two sections
Grouping Sections= (English101, English102). Each with a teacher
Grouping Project=(Project01, Project02, Project03 ... Project12, + English101, English102)
Now students are provided with a forum for private discussions about the projects. The teachers can use English101/102 to start a discussion related to these projects that concerns to all their students.

This may be done alternatively with a separate forum for those "project but common" discussions. But most teachers (and students) want to keep topics organized. Not having a "general forum" for any type of discussions. Questions about projects in the projects forum, questions about theory in the class forum.

Example F. Using global groups
Global grouping "Foreign Students"=(Resident, Erasmus, Non-UE)
Global Grouping "Disabilities"=(Disabled) (Disabled people are entitled by law to certain benefits: specially composed texts, exams ... )
Now go to course level. A course with two sections.
Grouping Sections= (Chemistry101, Chemistry102)
The teachers have a News forum in separate groups mode for general announcements, exam warnings etc. Usually Erasmus students are in an "special regime" and are a separate target for many messages and instructions.
If group sharing across groupings were allowed it would be possible to just add global group Erasmus to course grouping Sections and have News forums to manage (Section01, Section02, Erasmus) as separate groups.
If not, teachers have to add a whole new forum "News for foreigners" using "Foreign Students" grouping. And students have to watch for news at two different forums.

You have some resources or activities specially developed for students with disabilities. Let say, special reading texts. They are configure ti use Grouping=Disabilities. You have some immigrant students with a low level of communication language that could also benefit from accessing those resources. You may duplicate those, now using a local Grouping Special. If you could create a local Grouping Special=(global Disabled + local special) they all would access to the same resources or activities.

I think this is a significant limitation, with your scheme global Groupings are shared, rather than individual global Groups. And you cannot create a grouping at course level using course-local groups and global groups together. I do not see many use cases to utilize strictly global groupings at course activities. Such rigid global groupings/groups are useful mainly for enrolement purposes, not actual use in activities.

-----------------------------------------
So, let see the disagreements

>1/ sharing of groups in course groupings
> * minuses: more complex code + needs UI
Needs a different UI, yes, actually a one much more like the classic one, Please see images at http://docs.moodle.org/en/Development:Groupings_and_Groups and working demo at http://moodle.ulpgc.es/moodle19ulpgc/course/view.php?id=3 (login as "prof01" password "moodle" or "estudiante01" password "moodle")

Code complexity? Only that you cannot deduce uniquely the groupingid from groupid. However, that's not usually the task but the other way around. Groupingid must come from $course or $cm objects

> * pluses: please describe, the above use case

I do make a point on group sharing at course and site level. I would see the potential uses of groups severely handicapped if not allowed to share groups.

2/ orphan groups
> * minuses: more complex code
I have perform the hack to get groupings twice at ULPGC site. Fisrt with 1.5 I did use a "default grouping" to hold course groups if no grouping was created explicitly. When we upgraded to 1.6 I refactored the system and I got a "simper" framework without that "default grouping". Of perhaps, is simply that the second time I was more familiar with groups functions.

> * pluses: please describe - me, Martin and OU think there are none
Mainly teacher confort. Groups and groupings are demanded and a required feature, but not all teachers need groupings. If groups are used only for separate course sections then old-style moodle is enough. Presenting a UI where first groupings are to be created and then groups added within is a complication there.

On the other hand, restoring old backup copies from 1.6, 1.5 versions into >1.8 versions would be more natural if "orphan groups" (groups with a courseid but no groupingid associated) are preserved: better backward compatibility. If not , restore functions must ensure that "default grouping" is created and associated with orphan groups coming from the backup.

I do not see this point of orphan groups as crucial. This is a minor implementation detail: choosing if modifying a piece of the code or the other bu the important features for the teacher will work either.

-----------------------------------------------

With respect to the other concerns,

SPEED
I am by no mean a DBA.
I have been inspecting the code for the most used complex functions in groups API, those that allow you to retrieve groups for a user and compare how the task woul be done with your 4-table fixed-grouping scheme or the six-table loose OU scheme. In all cases the complexity of the queries is similar. In OU scheme there are more tables, but not all are used simultaneously.

groups_get_groups($courseid): 2-table join vs 1-table query if only groupids are need, a 2-table
groups_get_groups_in_grouping($groupingid): 1-table query vs 1 if only groupids or a 2-table join for group objects
groups_get_groups_for_user($userid, $courseid): both schemes use a 3-table join
groups_get_groups_for_user_in_grouping($userid, $groupingid): 2-table join vs a 2-table join for groupids or a 3-table join for group objects
groups_is_member_of_some_group_in_grouping($userid, $groupingid): both 2-table joins
groups_get_groups_not_in_grouping($groupingid, $courseid): both 2-table joins

So, it seems that some functions may be slower for the 6-table UO scheme, but other are actually simpler. I do not think that to avoid the extra complexity of groups_get_groups_for_user_in_grouping($userid, $groupingid) using a 3-table join we should refuse to use the extra flexibility of the scheme. How much slower would be this function with one and the other scheme? Is it worth that extra speed the lack of group sharing between groupings.

-------------------
Restricting access to resources and activities
Yes, the implementation makes the restriction for "separate groups" mode only: if groups are separated and you are not in any group you are off-limits. For instance in a separate groups forum ungrouped students cannot participate in the discussions. I think that adding a new attribute "restricted" distinct from "groupmode" would break that intuitive understanding.

As for Resources, I see it as a natural extension. Think of a "group mode" Resource as a "group wiki": you have in fact several wikis, one for each group. I am not thinking in adding such multipage capability to resources in any time in a near or distant future.
For the moment: you setup the resource as "separate groups" and those "not grouped" cannot access. That's the only "use of groups" of the resource module (Resources are modules, after all). But in future other "uses of groups" may arise.

-------------------
About implementation
I agree with the general view, simplification into one library file, with a reduction of overall API functions. Indeed may are not used nor needed for modules to use groups powerfully . I would use, as in other parts of Moodle, default parameters to combine some of them.

Throwing away the legacy functions is drastic but feasible: those functions are not called that much. I can mention that I hacked modules in 1.5 and 1.6 for groupings in less than a week once I had the groups functions clear in library.

I would like to sum it a bit for Martin:

• shared groups in global groupings - NO (or maybe yes if sharing allowed on the course level, can be changed later)
• shared groups between groupings from different courses - NO (backup nightmare)
• shared groups in groupings within one course - MAYBE, MartinD decides (db structure depends on this)
• orphan groups - internally NO, automatic creation of first course grouping which is set as default and hidden in UI if only one grouping in course

Timeframe:

• local groupings fully working in 1.9
• global groupings in 2.0 (patch for 1.9.x?)
• group enrolment in 2.0

People:

• coding: Enrique?, Nick?
• code review: Petr, Sam?, Nick?
• groups code maintainer in 1.9: Enrique? (== group bugs assigned in the tracker)
• documentation: ?

Nicolas might help with the UI, if I remember it correctly he was working on the original one in 1.8.0.

• I am only able work on the groups re-factor for a limited time, so after discussion with Petr we have found a way to divide the work into tasks that can be assigned to different people.

• Task 1, implement new DB schema, upgrade and backup/restore, test, with minimal changes to the libraries to support this. So, libraries will remain unfactored here. Est time, 3.5-4 weeks, assigned to Nick, review by Petr, sam? I'm on holiday 28 July - 12 August (honeymoon!), so estimated delivery 24 August: unfortunately it won't make 1.9 Beta unless that is delayed.

• Task 2, library re-factor, assigned to Enrique or other (To Be Confirmed), review by Petr, Nick, sam?

• Task 3, grouping support in modules - initially one module, eg. Database, Wiki, Forum. Can be before or after task 2; Enrique or other (TBC), with assistance from Nick, review by Petr, Nick...

• Task 4, 5... user-interface, documentation, global groupings, group enrollment ...

• We are agreed on using Petr's DB schema, except for 1 outstanding issue,
  http://docs.moodle.org/en/Image:groupsdb.png
  http://docs.moodle.org/en/Development:Groupings_OU#Petr.27s_proposal

• As indicated by Petr in his comment above, do we want shared groups in groupings within one course? Petr and OU think not, Enrique thinks yes.

Martin, please can you consider this as soon as possible. I'd like to get started on the work

Thank you. Yours

Nick

I wholeheartly advice no: teachers do want to use a group of students at more than one activity (but not in the same grouping). Groupings are sets of groups, as groups are sets of users. At first Moodle had single-group membership, each users could be in a group and only in a group, but use demanded to extend that. It's the same thing with groups and groupings. Once you have several groups in a course you can put together those groups differently for some activities.

GroupA is (John, Peter and Mary). You, as teacher, do not want to have groupA in Grouping01 used in Activity 01, and groupA'=(John, Peter and Mary) again, in grouping02 for activity02 that engages those and groups B and C. And even groupA''=(John, Peter and Mary) again, in grouping02 for activity02 that engages those and groups G, H, R. The teacher wants one group name identifiying that people, and then use that group in different activities =different groupings.

We have been discussing the alternative approach of "cloning" groups from one grouping to other. Making a GroupA in grouping 01 and a GroupB in grouping02 that happens to have the same users at members. But this solution must imply code to keep group membership synchronized: if added a user to GroupA, GroupB automatically changes. If not doing so we are creating a mess to work with by the teachers. The teacher would need to remember, magically, that GroupB is supposed to be a copy of GroupA and manually edit B when changing A.

Groups for activities are not rigid and invariable. They may be created as course flows, and members changing as needed (even self-assigned). A different beast from groups viewed as class cohorts or enrolment-assemblage that are not usually teacher controlled and invariable during course length.

The problem of not sharing groups is even worse, from the teacher point of view, when considering global groups. It is unlikely that a teacher needs to use a globally-set criterion to distribute users into activity-based groups, at least in large universities. I do not see how grouping students by course, by year in the studies, can be used at course level in course activities (may be used for enrolment, that's another story)
In the use case above, the teacher can take advantage on including a global group in a local grouping. That

The new BD schema proposed by Nick Freear http://docs.moodle.org/en/Image:groupsdb.png WITH the grouping_groups table and without the groupingid filed in groups table would meet the requirement to allow a group to be part of several groupings. However, I do not see that scheme much more simple, or facilitating faster queries that current scheme.
--------------------

Both "simplified" DB schemas make global groupings rather than groups (assuming implicitly that groups in a global grouping are global groups) However, at site level what are really needed groupings or groups? Are global groups always organized into sets that can be put separated into different groupings? Do all courses in a site always use global groups in exactly the same groupings(if use one group in teh grouping always using the other together, too). Above I have presented use cases not in this direction.

Sharing global groups would make mandatory to label those groups as unique in the DB for proper management, mostly during backup/restore. So uniqueid, synchroleid, manualsych shoul exist also at group table, not only groupings table.

--------------------
Summarizing

• shared groups between groupings from different courses - NO (backup nightmare)
• orphan groups - internally NO, automatic creation of first course grouping which is set as default and hidden in UI if only one grouping in course

• shared groups in groupings within one course - MAYBE, MartinD decides (db structure depends on this).
• shared groups in global groupings - maybe yes if sharing allowed on the course level

------------

Dedication: I have six weeks clear, four with full-day dedication. Afterwards in part-time
I can take responsibility for maintaining code, chasing bugs etc. of the tasks assigned.

We all agree that groupings at module level are a must for 1.9. Adding group-based restriction access is trivial once module-level groupings are working.

I object that cloning (== making a copy without any link to original group/grouping) would confuse teachers, they can not use groupings now. In majority of cases they would either use the same grouping or a new one - either created by hand or cloning or generated by some group distribution tool or student group selfassignment. I do not agree that shared groupings are easier to understand than cloned groupings and that the UI is easier to make/understand for shared groups. Shared groups require more complex UI IMO.

Anyway we could implement both shared and cloning of groups at the same time. The clone UI is just one field in new group/grouping form - select group/grouping to clone.

People that really want synchronised groups from different groupings might alse create automatic group synchroniser instead of db sharing - it would not have the same backup/restore or permissions problems because there would still be two separate groups each in its own grouping. This could be done with just one field in group table - groups with the same random id in this field would be synchronised when adding/removing users into these groups. There would not be needed any core changes, just added synchroniser into group UI. You could also link groups from different courses without any internal problems if needed.

than the other way
a) Create grouping01 and GroupA within (with other groups)
b) Create Grouping02 and GroupA (the same name, empty group)
c) Make a link (new function) between GroupA (with students) and groupA (without students)

Once GroupA in groupin02 is linked to groupA in grouping01 you no longer should be able to add/remove users from it. That would break the linking logic in the first place. So you have to make checkings for that case in code and UI, synchronizing groups has side-effects too, as sharing. The best synchronizing is using the very same group, not a "clone".

You are concerned by 3rd parties, other people making their own modifications to Moodle. Well, either those modifications do work AND are valuable for an ample audience, the why not incorporate them?; Or they are too specialized and only useful locally: let that people live with their code.

I concede that I have a biased view. I do see lot of uses to shared groups within courses because I am using them. I do not see any use for sharing groups across courses because I see course level as a logical barrier. Sharing groups betwen courses has no meaning if courses has different user enrolled (groups are not names, but the collection of people). Even if two courses have the same students I doubt I would use the same groupings or groups: if they are different courses they must have different topics: need different activities and different groups because not everybody is equally able at all topics. So, I minimize risk of sharing across course boundaries. Please, Petr, can you provide some use case for sharing groups between courses? (local groups, not global).

A comparison: sharing/publishing quiz questions categories. At ULPGC teachers see that as a mess. If you have 1500 courses and some of then "publish" categories rapidly that list is a complete mess where it is very difficult to locate the category you want to use. Even more, if the question text links to course-based files (e. g. images) that are lost in the new destination course. Result, at ULPGC teachers are sharing questions by sharing exported files (XML).

Petr, apart from the sharing problem, do you see any other showstopper for the 6-table scheme?
I would propose to add groupings at module level to 1.9 HEAD as it is, this putting groupings to work at last. Then let people use and opine and make any radical changes for 2.0.

a) create a grouping1
b) define groups groupA, groupB

c) create new grouping2
d) add new groupC - select "Clone groupA" in new group form
e) modify groupC members (does not affect groupA)
f) add new groupB - select "Link to groupB" in new group form

g) create new grouping3 - select "Clone+link groups from grouping1"

h) you can change/review the linking later in any grouping

I can not see any problems you describe above - if you link two groupings, the result is sum of all members in both groups; if you add one user into grouping the internal function for adding group members could sync the rest of linked groups; if you remove user from linked group, removing function syncs all linked groups. Only several places deal with group membership - group UI, user import, restore, enrolment (any more?)- they should use library functions for any adding/removing of members anyway - ok, it might be better to move it into core
As I said before, linking has IMO fewer side effects than sharing because you still have groups attached to groupings - we could deal with the linking easily in backup/restore code IMHO.

The UI could hide the difference between sharing and linking IMO - but I think that linking would be in fact more flexible; for example you could link based on group name; you could break the link and recreate it; you could decide what to do when linking groups sum members, use one group or the other, we would require unique group names in groupings only (you could reuse group1,group15 in several groupings in one course). The nice thing is that these extensions can be all done in 3rd party code without major breaking of future upgrades. We could also implement the linking later, not in 1.9, without major db changes (one db field only in groups table). There would be no need for global groups/groupings except the enrolment purposes - but it might be solved by different means.

The more I think about it the more I like the linking - flexible, few side effects, easier db schema, easier modifications...

Global groups or linking across courses? Simple use case is big course with old style tutor groups that is being split into two courses. Or when I was at grammar school we had the same 2 groups for several classes - chemistry lab, physics lab and biology lab. They are using moodle at that school now, I bet they still use the same fixed groupings for several classes Or at the university here in Liberec they have groups of students that are the same for many laboratory practices (I do not know how to call it in English properly, sorry). This is very common it the first years, later when the students specialize the groups are not fixed anymore.

Within your scheme if a teacher wants groupA=(John, Paul & Mary) in two groupings he first create a new empty group in the second grouping, and then he links it to the former. Even both linked groups may have different name, but collect together the same students. This is a radically different concept than simply "use groupA in two groupings". Linking is an additional concept or property of groups. What you are doing is crening a NEW groups, which happens to have the same members (that's a secondary condition). Linking may have its merits and uses, but it's something completely diffrent to the groups concempt in the first place.

I think that our opposing positions arise from different views of groups. It seems that you are thinking mainly in "rigid" class-level groups. At grammar school you have third grade, and probably Third Grade A, Third grade B and third grade C. They have each a teacher assigned, even e different room. The students in Third Grade A are in A for all the course, rarely change. Those gropups are not decided by the class-teacher but come done, imposed by School management.

The main point of this issue is to get ACTIVITY-SPECIFIC GROUPS. Groupings are not and end by themselves. They are useful because they allow to use different groups at different activities. I am the teacher of Third Grade A. I want my pupils to discuss about a topic in groups of 5. I set up those groups and a forum or wike where eah group can discuss its topic without interference from the others. But net week I have other excercise, now to be done by other groups (I want 5-member groups, but not the same 5 members for all activities). That's the use of groupings. So you have many many groups. It is not unusual to have 20-30 groups in each grouping (activity).

Now the next step. GroupC of activity 01 and group H of activity 03 are working in a similar topic. I want to add extra resources for them to see (and only them). A simple solution is adding those resources for a GroupingX=(GroupA+GroupH)

There is another point of view that separate our positions: the relative weight of groups and groupings. I have GROUPS as first actors, users are distributed into Groups, existing by themselves. Only in a second step groups are organized in sets, in groupings. I can reorganize the same bunch of groups into different sets, different groupings for specialized usage (grouping may impose extra conditions to how groups behave). My impression is that you have GROUPINGS as main actors, first create grouping, then groups associated to it rigidly.

The fact is that in mixed environments, and at activity level, groups, are created at teacher will, not rigidly and top-down imposed by organization enrolment structure. I as teacher create and mix as desired.

In your example about lab practices. I do have several Laboratory practice groups in my course, and they are fixed for all lab practices (but not other activities). I have LabA and LabB, with 24 users each (all 24 are in the lab room together at the same hour). Then I have LabA01, LabA02 ... LabA12 are two-student groups (practice experiments are performed in pairs). For announcements I have a grouping (Theory, LabA, LabB). That allows me to post messages in a forum (using that grouping) taylored for each of those groups (for instance, next lab session for LabA is Wed 16.00 hours, LabB is Friday 10.00 AM). For lab reports I have a Lab Grouping=(LabA, LabB, LabA01, LabA02 ....., LabB01, LabB02

In that way each students has access to two wiki pages, one "private" for the working pair (where they write their own particular report) and a common "lab" page where all pairs can write and put in common results of experiments

I could use two wikis, one for pair-group reports and another for lab-level reports. The second would use a grouping (LabA, LabB), note I am reusing the LabA, LabB groups with the Announcement (Theory, LabA, LabB). "Theory" groups is not all students (there are several theory groups, for other teachers).

I do not know if you have Lab practices in common for students of different courses. For instance, having Lab01 for students of Chem102 and Biochem101 and Biology203. That would be best managed by a metacourse, with students going into groups within metacourse based in the child course they come from. Managing this with regular separate courses and "linking" some group of Chem102 to other group of Lab0, that will open, fully, the concept of trans-course linking or coupling, the whole issue you try to keep well apart from temptation.

Offering a UI for sharing groups WITHIN course is less dangerous in the long run. It keeps things really within course boundaries.

http://docs.moodle.org/en/Image:groupsdb.png#file

I was happy with 'group', grouping, but I'm not concerned either way. Just checking!
Thanks
Nick

Please note that groupings do not exists in Moodle 1.8.1 officially yet, we can still change them in any way now and it will not be unnatural or confusing.
Yes, I am thinking about groupings as something above groups that separates them.

I have changed the table names to make it more compatible with 1.7 or earlier.

Talking about implementation, I do have proposed one. There is a diff attached since I opened the issue. That implementation covers all discussed features but global groups (taht can be easily incorporated) and has several others. Your proposal not only reduces functionality for end user but requires writing and testing new code for "linking".

You see linking simple at implementation but i see that now add/remove functions need to check first if the
In UI, when "adding a group to a grouping" we need to check first if the group belong to another grouping. If not, go ahead, if already used, proceed to transparently make a new group and link to the former. I see those checking as more complex code, not simpler.

I see "linking" a problem for understand and use, in the long term. If I add/remove users to GroupA (that happens to be used at groupings 01, 05 and 07): But if I have GroupA in grouping 01 and GroupHH in grouping04 becomes linked to GroupA; how do the teacher realize that fact afterwards, when actually using the groups? In you previous post you comment in various ways on linking: Second group linked to former (one cannot add/remove users to linked group), or bidirectional, or linked plus add/remove individual users in the linked groups: too confusing for use.

I argue that once the "linking" mechanism is on place, it can be abused to create a nightmare of cross-linked groups just as the sharing of groups.

Reviewing you last DB scheme
http://docs.moodle.org/en/Image:groupsdb.png
I see you have further reduced it by making groupings course-associated: introducing a courseid field in groupings table.
That modification works for grouping-based activities, but that implementation cannot manage Global Groupings (groupings not associated with a courseid). So that scheme is only a transient one. When Global Groups were addressed that scheme must be dropped for other more general.

If your mandatory goal is to keep groups and groupings course-based to avoid inter-course dependencies then a tested solution is to make groups and groupings course-associated by introducing explicitly a courseid field. I have the schema
http://docs.moodle.org/en/images_en/c/ce/Groups-tables2.png
working on moodle1.6 in current ULPGC site.
This DB structure ignores global groups/groupings. It would be possible to incorporate them in future by making courseid=0 to mean Global group/grouping

With this structure in is impossible to make any 3rd party extension that introduces course-crosslinking: groups and groupings are strictly course-based. But it is possible to have a group in more than one grouping.

Petr, sincerely, I do not understand your reasons to insist in "a group can belong to just one unique grouping" when both groups and groupings are course-associated.

--------------------
Another approach to global groups: use "linking" to create local-global groups

In all discussion about global groups I have understand, always, that a global group/grouping used at a course level really meant "those users or the global group that happens to be enrolled in this course". This behavior can be emulated the other way around.
a)create a groupX
b) instead of populating it with users using the UI tool selecting users from a list, the UI provides other separate tool: Add users from Global Group. No list of users but a list of global groups and the users that actually become members of groupX are those in global groups selected that actually are enrolled at the course.
In this approach synchronization of global-linked groups is maintained at global-groups editing level: each time a global group is changed (user Added/removed) all linked local groups are tested for user add/removal.

The linking is more flexible than sharing and safer for us IMO. There are more possibilities how to implement it in UI - either by fully emulating sharing or creating new not yet imagined possibilities (which I tried to describe above). The worst thing that might happen with linking during backup/restore/import/export/upgrade is that the groups are unlinked, in case of sharing the problems could be worse. I do not understand your examples of problems with linking UI, I am sure there are ways to make it intuitive, easy to use and more flexible than sharing. Remember there are no groupings or sharing of groups yet in Moodle 1.8.1.

The implementation complexity is another topic - we could implement first only groupings + cloning of groups in 1.9. The internal code and UI would be much simpler than today. later we could implement the linking and new UI features for group distribution, student self group assignment, etc. The side effect of my last proposed scheme is much better backwards compatibility with 1.7

Global groupings are not my priority, linking+cloning would partially work around it, the enrolment trouble could be solved in a different way (not sure if the groups are the optimal way).

My +1 to work on less features for 1.9 and more polished code and better APIs. We changed API+UI+db in 1.8, then again UI in 1.8.1 because groups did not work much/groupings did not work at all yet - we can not change UI+API+db in 1.9 and in 2.0 again

There is not much time to do the coding and testing, the 1.9 beta is closer each day.

Sharing within the same course is not abomination, it's natural. We do "share" users between activities (module instances), we do not "clone" a user to have access to another module. W do "share" users between groups. If a users is in three groups the userid is added to all three groups, not "cloned" or something special. There is no exception in code to check before adding a user to a group if teh user already belongs to other groups. If I have a group in a course, why should be imposed to "clone" it to access other activities?, why to "clone" it?, an exceptional procedure to add to a second grouping?.

I think that your understandable prevention to the backup nightmare of shared groups across courses is extending beyond reasonable limits if precluding group sharing within a course.

I have module-level groupings workin on 1.6 Moodle in our production site. Both groups and groupings and ONLY local, without any trace of globally or cross-course shared groups. But with groups belonging to several groupings within a course.

I have a las proposal to try to reconcile both views. From your last scheme
http://docs.moodle.org/en/Image:groupsdb.png
You are introducing courseid into groupings table, thus tightly associating each grouping with a single course. You have groupingid in groups table thus, associating a group with a single grouping and hence with a single course.

I propose to remove groupingid from groups table and substitute with courseid. (that's the way I have both tables with 1.6). So each group would be explicitly associated with a single course. No ambiguity, no confusion for backup restore. We would need groupings_members table relating groupingid and groupid to hold the membership relation between groups and groupings. The schema is

http://docs.moodle.org/en/Image:Groups-tables2.png

In this way I think that is possible to get both goals
a) Each group and grouping is explicitly associated to a unique courseid.
b) It is possible to have a single group in a course to belong to several groupings in that course.

It is always possible to write code that behaves contrary to moodle philosophy. For instance, it is possible to add people to a group that are not enrolled in the course. Just happens that the UI only present a list of course-users. Nothing in groups_add_member() function neither in the DB structure enforce a rule not to add a userid of a non-enroled user to a group. I think that the above proposal keeps the same compromise. Courseid is clearly stated and groups can be added to several groupings.

We must provide a solution for Global Groups, since they are demanded by other people (not me neither ULPGC).
For this particular purpose I think that "linking" or cloning would be a very good idea indeed.

Global groups are managed (created, edited user add/removal) by admins or alike in a separate UI, available at site level, not course context. That means a UI (and API functions) distinct and separate from regular course-groups.

We wouldn't have "global groups" at course level. All course groups would be "local" (courseid-associated)
First we create a local Group, then instead of adding users, by use of a separate UI tool we link that local group to a "global group"
linking means that only those users that happens to be be members of the global group and, simultaneously, are enrolled in teh current curse are added to the local group.

The advantage of this approach over "generalized linking" is parsimony. Current group API would work untouched. The synchronization of membership when users were added(removed would be confined and controlled by global groups functions. E.g. each time a user is added to a global groups local groups are queried to identify and update linked local groups.

But we all agree that global groups are something for the future. We must consider them now just to ensure that we are not forcing ourselves to go back and forth about them like the grouping issue in 1.8/1.8.1, 1.9.

I think that we all agree that the only feature that is feasible to add to 1.9 are activity-level groupings (and grouping-based access that it's trivial once the former is done). And this task is waiting only for decision on DB structure. I think that
http://docs.moodle.org/en/Image:Groups-tables2.png
solves your concerns and preserves my usability demands. If we agree on that structure I can implement it for 1.9 in a couple of weeks, with UI etc. (have you seen the proposed UI?)

Even, "linking" isn' strictly needed for activity-level groupings. Linking/sharing is a usability issue. Even if using your scheme
http://docs.moodle.org/en/Image:groupsdb.png
The provisions may be done in DB tables but not used yet (I mean keep linkid and the add the linking code to add to groups functions later) but activity-level groupings must be added. That's a real teaching gain, unavailable before. The issue of if groups are shared or not is secondary.

The keypoints are:
a) Both groups and groupings tables include a "courseid" filed and thus are univocally associated with a unique course
b) Groups are related to groupings loosely, through the groups_groupings table. Then a group may belong to none, one or many groupings but remain associated to a single course.

This structure is the most backward-compatible of those discussed, it's quite similar to groups in moodle versions 1.5 and 1.6. That eases upgrading. With this structure, all kind of groups/groupings relationships can be supported at single-course level.

The advantages of this structure are:
a) There is no room for cross-course interactions at DB level
b) It's simpler and smaller than 1.8 structure
c) Most used queries for user- or course-groups are simple
d) Allows re-use and sharing of groups in multiple groupings within a course

Global groups do not fit into the scheme but can be emulated. Global groups are managed at site level separately from course-level groups. In DB global groups and groupings may be identified by courseid=0 (or added to separate tables)
All groups used in a course must be local groups, the linking mechanism guarantees that user-members are added as

Another agreement.
Grouping-based access restriction to resources and activities should not overload the "separate groups" mode due to backward compatibility issues. A new switch "grouping-restricted" (or similar name) is added to course_modules table and to common module settings along with groupmode and grouping.

Nick, Sam, what do yo think in OU about this?

Pending topics:

• library cleanup, how much pruning? how far? For 1.9 or later?
• global groups: clearly later
• UI : should be discuss the UI? There are some schreenshots at http://docs.moodle.org/en/Development:Groupings_and_Groups

Nick, if you are busy with other things I can take this (and take also bugfixing responsibility and workload)

Enrique - thanks for the latest DB schema - I think we have concensus! I have started implementing the schema changes and upgrade. Will let petr and you know when I have a patch...

A minor point, can we have the 'courseid' field immediately after 'id' in the groupings table, so it matches the 'groups' table? So, id, +courseid, name, description, +exclusivegroups, +maxgroupsize ...

Regarding library cleanup "how much pruning?" - quite a bit, some of it specified in these sections - I'll flesh this out,
http://docs.moodle.org/en/Development:Groupings_OU#Folder:_group

I think we're envisioning a single 'grouplib' ((or groupinglib)) file from a merge of: basicgrouplib.php, groupinglib.php, utillib.php. All group/db/*lib.php should go, lib/courselib, automaticlib etc. to go.

Legacylib function calls should be fazed out, but I think this file is best left separate, as is modulelib.php - we at the OU are using some of these functions and they are quite distinct. Get rid of unused functions (check against ou-moodle), and reduce number of functions generally.

Hope that's a start. Must dash - talk tomorrow. Yours
Nick

You don't want a third 'groups that are not in any grouping' option (let alone a fourth 'all groupings but not including groups that aren't in any grouping') - and this turns out to be necessary if you have groups in no grouping. Far better to ensure that all groups are in a default grouping for the course, unless specified otherwise.

Overall I preferred the schema when it required every group to be in exactly one grouping But I think I don't have any practical problems with this provided we require groups to be in at least one grouping.

I like the proposed tabbed interface (http://docs.moodle.org/en/Development:Groupings_and_Groups) with separate groupings and groups forms. It solves the sharing of groups in an easy and understandable way.

For the activities and course I think that "Legacy == no grouping == all groups in course" + all groupings should be fine.

I have no strong objection, but I do not like "default" grouping because that creates an "exception" a grouping that is different from others in behavior without clear cause (from code view)

This is a matter of business logic, usability (from users point of view) and backwards compatibility:

• Backwards compatibility: before 1.8 all groups were in flat list, without groupings. Restoring old backups will work untouched if no default grouping is allowed; otherwise restore code must create the "default" grouping

• Not all teachers need groupings. Allowing to use groups without groupings keep those teachers using Moodle just as before: upgrade do not force them to change the way they work.

• The proposed UI emphasizes Groups first. You can create a bunch of groups and use them in a grouping or not. Groupings are a secondary thing used to organize groups into sets fed to activities: A way to specify what of all groups should be used in a specific activity. But groups exists per se (talking not about implementation code but teacher mind). A teacher may have some groups created but not yet used in any grouping or any activity, waiting for their moment on course time-flow. He may slowly add users (as they make their mind about a topic, as they enroll etc) into groups (unusable at activities because not in grouping) and afterwards put those groups to use in a grouping.

• There could be a confusion about "default" grouping. We as programmers are talking abut code-default. However, the course has a groupingid meaning the grouping to use for separate/visible groups when no other is declared at module level, aka "default" grouping for users. Since this is selectable may be that "default course grouping" is not the same as "default groups grouping". This more a matter of carefully selection of names, and language-specific. But allowing "no grouping=all groups" avoids the problem.

• I agree that for activities only "all groups" or "groups of grouping X" should be allowed. That menu, at modules, should show a list of groupings plus "All groups" option. Other combinations, 'groups that are not in any grouping' option and the like simple are not considered explicitly.

In the code at the attached diff you may find examples. Unfortunately my UI demo uses pre-formslib code. I may concentrate on creating the UI with formslib while you refactor the groups libraries, if that may help.

It's based on the 18 July 'groupsdb' image, http://docs.moodle.org/en/Development:Groups - with these modifications:

• groupings table: courseid after id column, not after description.
• groupings_groups: added 'timeadded' field.

We (Tim Hunt and myself) still have big problems about orphan groups - in my/sam's absence Tim is going to discuss, I think!

Other patches to follow.

• This big patch from 23 February was used internally for ou-moodle to add groupings support centrally to 'course_modules' and to the Moodle WIKI. It would need alterations, but gives an indication of what we have done.

• In OU-moodle, we've since added groupings to: forum, quiz reports, and data activities.

Please review these 3 patches. Test the first one if possible. Many thanks

Nick

Nick

Is everyone agreed on Nick's patches above?

Schema and upgrade:

http://git.catalyst.net.nz/gitweb?p=moodle-r2.git;a=commit;h=b68d2070f280fd87945b4fe96783a68fd391f25a
http://git.catalyst.net.nz/gitweb?p=moodle-r2.git;a=commit;h=c6f949250d9fb2c67aec9ebcd771a8c9f05565de

Upgrade code to use new schema:

http://git.catalyst.net.nz/gitweb?p=moodle-r2.git;a=commit;h=9c11a1d709739859dab46e5e9605f11a4bb8dd90
http://git.catalyst.net.nz/gitweb?p=moodle-r2.git;a=commit;h=7ce3afb45182127e45df4563f9a6211b5560f535
http://git.catalyst.net.nz/gitweb?p=moodle-r2.git;a=commit;h=265eb0379f7f39e4faa58e6aabbc083e063534c5

Please review these patches and let me know if you are happy for them to go into CVS head.

-Matt

testing patch....

http://moodle.cvs.sourceforge.net/moodle/moodle/lib/grouplib.php?r1=1.21&r2=1.22