Moodle
  1. Moodle
  2. MDL-31776

Additional name fields, to allow Asian languages to flexibly display user names in Chinese characters, local phonetic system or Romanization

    Details

    • Database:
      Any
    • Testing Instructions:
      Hide

      This improvement makes a lot of changes throughout the site.

      Admin settings:

      [Site administration ► Users ► Permissions ► User policies]
      fullnamedisplay
      This setting is what most students will see.

      • Update the setting to include some of the following name fields:
        • firstnamephonetic
        • lastnamephonetic
        • middlename
        • alternatename

      Basic testing

      1. Change the admin settings above.
      2. Test that the following areas display as expected:
        The following places have been updated to show display names with the new settings:
      • Navigation block
      • "Logged in as" area in the top right hand corner.
      • Participants page
      • All of the core modules
      • Admin settings
      • Profile pages

      It is recommended that exploratory testing be done to try to eliminate as many bugs and regressions as possible.

      Exploratory testing:

      Things to test:

      • Installing Moodle
      • Upgrading Moodle
      • Participants page
      • Navigation
      • Activities
      • Admin settngs
      • Gradebook
      • Webservices
      • Backup
      Show
      This improvement makes a lot of changes throughout the site. Admin settings: [Site administration ► Users ► Permissions ► User policies] fullnamedisplay This setting is what most students will see. Update the setting to include some of the following name fields: firstnamephonetic lastnamephonetic middlename alternatename Basic testing Change the admin settings above. Test that the following areas display as expected: The following places have been updated to show display names with the new settings: Navigation block "Logged in as" area in the top right hand corner. Participants page All of the core modules Admin settings Profile pages It is recommended that exploratory testing be done to try to eliminate as many bugs and regressions as possible. Exploratory testing: Things to test: Installing Moodle Upgrading Moodle Participants page Navigation Activities Admin settngs Gradebook Webservices Backup
    • Affected Branches:
      MOODLE_23_STABLE, MOODLE_25_STABLE
    • Fixed Branches:
      MOODLE_26_STABLE
    • Pull from Repository:
    • Pull Master Branch:
      wip-MDL-31776-master
    • Story Points:
      100
    • Rank:
      166
    • Sprint:
      BACKEND Sprint 2

      Description

      Alternate name fields

      This is a proposal on behalf of The Moodle Association of Japan (http://moodlejapan.org). We are willing to work closely with Moodle HQ on this improvement and help by providing some funding for its development.

      The standard Moodle has only two fields for names "firstname" and "lastname". In countries that use a different writing system, this means that two sets of names need to be displayed, their name in the local writing system and their name in Romanization. Moodle needs a way to input both types of names, and for both the individual and the administrator (and perhaps at the course level) to determine which names appear in the various Moodle contexts.

      An ideal configuration

      I propose three additional fields: surnamephon givenphon alternatename

      For standard English and European usage, these would be ignored, but for Japanese, Chinese, Korean and perhaps other writing systems such as Thai, the various Indian languages and Arabic. Possible entries would look like this:

        lastname firstname lastphon firstphon Alternate
      Japanese 佐藤 サトウ ヒロシ Satou Hiroshi
      Korean 奎報 규보 Yi Gyubo
      Chinese 美和 ㄔㄣ ㄇㄟㄏㄨㄚ Chen MeiHua

      The site administrator would select how the $FULLNAME variable would appear. For Japanese, for example, the administrator might select lastname + firstname + alternate so that the students' names would appear in Kanji, but with the Romanization of it following the Kanji. This would allow everyone to know how the name should be pronounced. (Even Japanese need this since there are alternate pronunciations for Kanji. "神部" for example, could be read as "Kamibe" or "Kambe" depending on the preference of the user's family.)

      Any combination of the above could also be selected as the default for user names that appear in various tables.

      If the selected fields for tables is one of the Asian phonetic fields, then rather than the standard alphabet display for filtering the list by first name and last name, a similar selection display would appear for the selected alphabetic system.

      For Chinese, a selection would also need to be made for the phonetic system to use. Taiwan uses the "BoPoMoFo" (Zhuyin) system, while mainland China uses the Pinyin system.

      Student ID Display

      For many Asian languages, another problem not encountered so much in the West is people with identical names. This, plus the fact that some of the languages (namely Japanese and Chinese) have no convenient method for ordering names in Chinese characters, means that most schools rely on the student ID for ordering and identifying students rather than (or in addition to) their names. Thus, there should also be an option in the site set-up selecting the StudentID field to be displayed to the left of names on various listings such as the grade book, assignment evaluation, etc.

      Complexity & coding time:
      This problem is quite complex since it involves modifying core code in perhaps 50-100 files that do database I/O, table displays, preference selections, etc. and would need to be done in close coordination with Moodle Headquarters.

      Who would benefit from this improvement?:
      Probably every institution in Asia that has staff who do not read the local language need to be able to display names in both the local language and Roman letters. Various, rather unsatisfactory ad hoc solutions are currently used but a permanent, more elegant solution is urgently needed.

      1. 1-Files-Listing-20130204.xlsx
        47 kB
        Thomas Robb
      2. 2-Database-Tables-Listing-20130204.xlsx
        9 kB
        Thomas Robb
      3. 3-Programming-Strategy-20130204.pdf
        35 kB
        Thomas Robb
      4. fullname_places.txt
        49 kB
        Adrian Greeve
      1. bonfire-screenshot-20121207-104955-873.png
        258 kB
      2. bonfire-screenshot-20121207-105347-925.png
        121 kB
      3. bonfire-screenshot-20121207-105436-367.png
        206 kB

        Issue Links

          Activity

          Hide
          Don Hinkelman added a comment - - edited

          Hideto, Peter, and I discussed this issue with Michael de Raadt at the MoodleMoot Japan conference. He raised two other possible reasons for alternate name fields that have been requested in the past:

          • avatars/nicknames in certain contexts
          • complete anonymity in certain contexts

          These other situations would be related to the issue thus requiring a flexible way to display and order class name lists, or allow varying levels of privacy in some contexts. Michael went through several strategic approaches that I think he will suggest as soon as he is back at the office. As the Stable Branch lead developer, he will be the first to address this issue.

          At our discussion, I also pointed out that student number needs to part of the display with these name displays, as most class rosters are arranged by student number in ascending order for grading, attendance-taking, assignments, etc. Presently, at my university Moodle in Japan, we use five data types within two profile fields in order to handle class rosters.

          • profile lastname field: student number + lastname Chinese character + firstname Chinese character
          • profile firstname field: lastname phonetic (kana) alphabet + firstname phonetic alphabet

          This is not really enough because many teachers also need a roman character lastname and firstname. Our university student management system (by Fujitsu) has all three alphabets available to all teachers in printing out class rosters. All sets of numbers and names are available in reports and displays.

          Show
          Don Hinkelman added a comment - - edited Hideto, Peter, and I discussed this issue with Michael de Raadt at the MoodleMoot Japan conference. He raised two other possible reasons for alternate name fields that have been requested in the past: avatars/nicknames in certain contexts complete anonymity in certain contexts These other situations would be related to the issue thus requiring a flexible way to display and order class name lists, or allow varying levels of privacy in some contexts. Michael went through several strategic approaches that I think he will suggest as soon as he is back at the office. As the Stable Branch lead developer, he will be the first to address this issue. At our discussion, I also pointed out that student number needs to part of the display with these name displays, as most class rosters are arranged by student number in ascending order for grading, attendance-taking, assignments, etc. Presently, at my university Moodle in Japan, we use five data types within two profile fields in order to handle class rosters. profile lastname field: student number + lastname Chinese character + firstname Chinese character profile firstname field: lastname phonetic (kana) alphabet + firstname phonetic alphabet This is not really enough because many teachers also need a roman character lastname and firstname. Our university student management system (by Fujitsu) has all three alphabets available to all teachers in printing out class rosters. All sets of numbers and names are available in reports and displays.
          Hide
          Gordon Bateson added a comment -

          I would also like to add my support for dividing the proposed "alternate" field into "altfirst" and "altlast", as this would allow for sorting of the student lists on English first or last name.

          Show
          Gordon Bateson added a comment - I would also like to add my support for dividing the proposed "alternate" field into "altfirst" and "altlast", as this would allow for sorting of the student lists on English first or last name.
          Hide
          Aaron Batty added a comment -

          I second both Mr. Hinkelman's and Mr. Bateson's further suggestions. It would be great if I had 7 fields to work with: 2 for kanji, 2 for phonetic, 2 for roman, 1 for student number.

          Show
          Aaron Batty added a comment - I second both Mr. Hinkelman's and Mr. Bateson's further suggestions. It would be great if I had 7 fields to work with: 2 for kanji, 2 for phonetic, 2 for roman, 1 for student number.
          Hide
          Eric Hagley added a comment -

          As mentioned, I'm sure that every institution in Asia would benefit from this addition. I look forward to its implementation and greatly appreciate all endeavors to that end.

          Show
          Eric Hagley added a comment - As mentioned, I'm sure that every institution in Asia would benefit from this addition. I look forward to its implementation and greatly appreciate all endeavors to that end.
          Hide
          Edmund Edgar added a comment -

          +1 on the main gist. If you want Moodle used on any scale in Japan and probably much of the rest of Asia, you need a clean way to handle your students' names.

          At the risk of complicating things, a couple of things that occur to me having built a (non-Moodle) system for a school with a mixture of Japanese, Chinese and US students, and Japanese-speaking and English-speaking staff and teachers. (Maybe MAJ have already discussed all this and figured this is the optimal proposal, in which case feel free to ignore me):

          • Having one big "alternate" field looks to condemn you to never being able to display or sort by Roman last name. (Or first name if you input it in the order "last first".) Wouldn't it be better to do "lastroman" and "firstroman"?
          • Knowing what to display where is a bit fiddly if you want to do it right. (Maybe this is why the MAJ proposal punts the problem to the end user with a big old "alternate" field, that you can stuff whatever you like in?) We normally want to show the Japanese name to Japanese speakers, and the English name to English speakers. That makes your name settings language-dependent, rather than having one site-wide setting for what $FULLNAME means.
          • This goes beyond Asian languages and we probably shouldn't go there, but if you're customizing $FULLNAME you may even want to do it by role. We have some external graders who don't even get to see the students' names - all they get to see is their student IDs.
          Show
          Edmund Edgar added a comment - +1 on the main gist. If you want Moodle used on any scale in Japan and probably much of the rest of Asia, you need a clean way to handle your students' names. At the risk of complicating things, a couple of things that occur to me having built a (non-Moodle) system for a school with a mixture of Japanese, Chinese and US students, and Japanese-speaking and English-speaking staff and teachers. (Maybe MAJ have already discussed all this and figured this is the optimal proposal, in which case feel free to ignore me): Having one big "alternate" field looks to condemn you to never being able to display or sort by Roman last name. (Or first name if you input it in the order "last first".) Wouldn't it be better to do "lastroman" and "firstroman"? Knowing what to display where is a bit fiddly if you want to do it right. (Maybe this is why the MAJ proposal punts the problem to the end user with a big old "alternate" field, that you can stuff whatever you like in?) We normally want to show the Japanese name to Japanese speakers, and the English name to English speakers. That makes your name settings language-dependent, rather than having one site-wide setting for what $FULLNAME means. This goes beyond Asian languages and we probably shouldn't go there, but if you're customizing $FULLNAME you may even want to do it by role . We have some external graders who don't even get to see the students' names - all they get to see is their student IDs.
          Hide
          Adam Jenkins added a comment -

          I have been adding profile fields for each of my profiles for years now. The problem is that I cannot define what data to display in my userlist. Often the information I want to see (or sort users by) does not appear on the screen. If I could easily define the data to be displayed and use extra profile fields, this problem would be solved for me.

          Sorry if this is solved in 2.2, I'm still stuck on 1.9.

          Show
          Adam Jenkins added a comment - I have been adding profile fields for each of my profiles for years now. The problem is that I cannot define what data to display in my userlist. Often the information I want to see (or sort users by) does not appear on the screen. If I could easily define the data to be displayed and use extra profile fields, this problem would be solved for me. Sorry if this is solved in 2.2, I'm still stuck on 1.9.
          Hide
          Michael de Raadt added a comment -

          Thanks for putting this forward on the tracker. I know that this issue has been brewing up for some time, so noe it can get some attention from us at Moodle HQ as well as from the community.

          This issue has 28 votes already and I suspect that will grow. I believe this improvement would benefit more than Asian language speakers.

          I believe that displaying student ID numbers is already possible, although how this is done could be improved and made more obvious and universal. Even though it is related to the display of names, I think it should be handled separately from this issue, and if more work needs to be done, then further issues should be launched.

          Storing name data

          In relation to allowing alternate name fields to be associated with a user, there are two possibilities I have thought of.

          1. '''Adding more fields to the user profile directly.''' This would be inflexible, but simple. However, we already have a number of fields in user profiles that we are considering removing and turning into custom profile fields.
          2. '''Adding a new type of custom profile field for names.''' This would mean that administrators would need to establish the fields they want, but they would be free to create as few or as many fields as they wish. It might also be possible that a set of default custom profile fields are added with a new installation. We wouldn't need to try to create a solution to suit everyone now and into the future. This type of custom profile field, once added could be treated like other profile fields (for importing data, authentication, etc.), but could be recognised as special when defining how names would be displayed.

          Displaying names

          At the moment we have a focal point in the code where we decide how names are displayed. The full name display can be controlled by an admin. There are a few options for name order and whether first and/or last names are displayed. The name display can also be configured in the language file, so arbitrary patterns are possible. There are capabilities in place to override restrictions if partial names are displayed. This system could become more flexible so that it could be configured into more formats than what can be chosen from a list and without manipulating a language file.

          Context

          At the moment, display of full names is controlled at the site level. This is a good starting point for development, however there is the potential for names to be displayed in different ways at lower context levels, such as in a course or a module. This would allow instructors with different needs to define how names appear within a site.

          Anonymity

          An extension to this work could be to allow people to appear anonymously. If the context control is added, this could allow for names to be hidden within a course or specific module. However there is more to anonymity than simply hiding a name, and achieving anonymity could be more complex and adding alternate name fields. In my opinion I would leave anonymity out of this project, at least initially.

          Some further considerations:

          • How should alternate names appear in tables, such as the gradebook, reports and in activity result tables (assignments, quizzes, etc.)?
          • How should alternate names be sorted? Is Unicode order of characters appropriate? I think we may need a more universal means of comparing names.

          I invite all interested parties to vote, comment, propose, design and of course code. Let's make Moodle better for everyone.

          Show
          Michael de Raadt added a comment - Thanks for putting this forward on the tracker. I know that this issue has been brewing up for some time, so noe it can get some attention from us at Moodle HQ as well as from the community. This issue has 28 votes already and I suspect that will grow. I believe this improvement would benefit more than Asian language speakers. I believe that displaying student ID numbers is already possible, although how this is done could be improved and made more obvious and universal. Even though it is related to the display of names, I think it should be handled separately from this issue, and if more work needs to be done, then further issues should be launched. Storing name data In relation to allowing alternate name fields to be associated with a user, there are two possibilities I have thought of. '''Adding more fields to the user profile directly.''' This would be inflexible, but simple. However, we already have a number of fields in user profiles that we are considering removing and turning into custom profile fields. '''Adding a new type of custom profile field for names.''' This would mean that administrators would need to establish the fields they want, but they would be free to create as few or as many fields as they wish. It might also be possible that a set of default custom profile fields are added with a new installation. We wouldn't need to try to create a solution to suit everyone now and into the future. This type of custom profile field, once added could be treated like other profile fields (for importing data, authentication, etc.), but could be recognised as special when defining how names would be displayed. Displaying names At the moment we have a focal point in the code where we decide how names are displayed. The full name display can be controlled by an admin. There are a few options for name order and whether first and/or last names are displayed. The name display can also be configured in the language file, so arbitrary patterns are possible. There are capabilities in place to override restrictions if partial names are displayed. This system could become more flexible so that it could be configured into more formats than what can be chosen from a list and without manipulating a language file. Context At the moment, display of full names is controlled at the site level. This is a good starting point for development, however there is the potential for names to be displayed in different ways at lower context levels, such as in a course or a module. This would allow instructors with different needs to define how names appear within a site. Anonymity An extension to this work could be to allow people to appear anonymously. If the context control is added, this could allow for names to be hidden within a course or specific module. However there is more to anonymity than simply hiding a name, and achieving anonymity could be more complex and adding alternate name fields. In my opinion I would leave anonymity out of this project, at least initially. Some further considerations: How should alternate names appear in tables, such as the gradebook, reports and in activity result tables (assignments, quizzes, etc.)? How should alternate names be sorted? Is Unicode order of characters appropriate? I think we may need a more universal means of comparing names. I invite all interested parties to vote, comment, propose, design and of course code. Let's make Moodle better for everyone.
          Hide
          Edmund Edgar added a comment -

          "How should alternate names be sorted? Is Unicode order of characters appropriate? I think we may need a more universal means of comparing names."

          Ideally you'd want to sort lists of Japanese names by their phonetic reading ("lastphon, firstphon" above), which bears no resemblance to what you'd get if you sorted the Chinese characters in the lastname / firstname fields by any method (unicode code-point order, utf8_general_ci, utf8_japanese_ci):

          木内 [Kiuchi]
          菊地 [Kikuchi] ("ku" comes after "u" in the Japanese phonetic sort order)
          木倉 [Kikura] ("ra" comes after "chi" in the Japanese phonetic sort order)

          I think if you're sorting phonetic data you can probably rely on the database's default collation (utf8_general_ci or whatever) at least for Japanese. I'm not sure if there are any languages or cases where that assumption breaks.

          (The catch is that you may not actually have that phonetic data for everybody, in which I case I suppose falling back on whatever order utf8_general_ci thinks is best for sorting the last/first data is the bet you can do.)

          If you're displaying a list headed by Roman-alphabet names rather than Japanese ones, you'd want to sort by those and ignore the Japanese phonetic order.

          Show
          Edmund Edgar added a comment - "How should alternate names be sorted? Is Unicode order of characters appropriate? I think we may need a more universal means of comparing names." Ideally you'd want to sort lists of Japanese names by their phonetic reading ("lastphon, firstphon" above), which bears no resemblance to what you'd get if you sorted the Chinese characters in the lastname / firstname fields by any method (unicode code-point order, utf8_general_ci, utf8_japanese_ci): 木内 [Kiuchi] 菊地 [Kikuchi] ("ku" comes after "u" in the Japanese phonetic sort order) 木倉 [Kikura] ("ra" comes after "chi" in the Japanese phonetic sort order) I think if you're sorting phonetic data you can probably rely on the database's default collation (utf8_general_ci or whatever) at least for Japanese. I'm not sure if there are any languages or cases where that assumption breaks. (The catch is that you may not actually have that phonetic data for everybody, in which I case I suppose falling back on whatever order utf8_general_ci thinks is best for sorting the last/first data is the bet you can do.) If you're displaying a list headed by Roman-alphabet names rather than Japanese ones, you'd want to sort by those and ignore the Japanese phonetic order.
          Hide
          Geordie McGarty added a comment -

          This would be a MAJOR improvement for asian based Moodle users. I agree with Don Hinkelman's comments as well. I really hope this can be implemented in a near future build.

          Show
          Geordie McGarty added a comment - This would be a MAJOR improvement for asian based Moodle users. I agree with Don Hinkelman's comments as well. I really hope this can be implemented in a near future build.
          Hide
          Ronald Notestine added a comment -

          I agree this would be a big improvement. I also would like to see seven fields, including altfirst, altlast, and studentnumber. My thanks to Tom Robb for initiating this.

          Show
          Ronald Notestine added a comment - I agree this would be a big improvement. I also would like to see seven fields, including altfirst, altlast, and studentnumber. My thanks to Tom Robb for initiating this.
          Hide
          Thomas Robb added a comment -

          In response to Edmund Edgar's comments about sort order, for Kanji there is no order that is useful and rememberable, so other means are used. List ordering is normally done by the assigned student numbers, which in turn have probably be assigned in some sort of natural order. In Japan, the student numbers are normally assigned in kana order, thus, sorting the students by their ID numbers is equivalent to sorting them by their kana order, as long as the students are all in the same faculty and entered in the same year.

          In the system that I initially proposed, entries in the "lastphon" and "firstphon" categories can be used to order the various reports. There would have to be a selection in the site set-up (if not at the course-level, too) which would allow admins to specify a sort order combination (first only, last only, first-last, last-first) which would perhaps not appear in the listing itself, but would be referred to when the list is sorted by name.

          Show
          Thomas Robb added a comment - In response to Edmund Edgar's comments about sort order, for Kanji there is no order that is useful and rememberable, so other means are used. List ordering is normally done by the assigned student numbers, which in turn have probably be assigned in some sort of natural order. In Japan, the student numbers are normally assigned in kana order, thus, sorting the students by their ID numbers is equivalent to sorting them by their kana order, as long as the students are all in the same faculty and entered in the same year. In the system that I initially proposed, entries in the "lastphon" and "firstphon" categories can be used to order the various reports. There would have to be a selection in the site set-up (if not at the course-level, too) which would allow admins to specify a sort order combination (first only, last only, first-last, last-first) which would perhaps not appear in the listing itself, but would be referred to when the list is sorted by name.
          Hide
          Michael de Raadt added a comment -

          I'm noting that at this point, with 45 votes, this issue is the 29th most voted for issue in the dev backlog. I suspect, though, that the impact of this change would be very beneficial in many parts of the world.

          Shane: I've added you as a watcher on this issue as you said you were interested in working in this area at MootAu12.

          Show
          Michael de Raadt added a comment - I'm noting that at this point, with 45 votes, this issue is the 29th most voted for issue in the dev backlog. I suspect, though, that the impact of this change would be very beneficial in many parts of the world. Shane: I've added you as a watcher on this issue as you said you were interested in working in this area at MootAu12.
          Hide
          Adrian Greeve added a comment -

          I have been working on one possible solution to this improvement. It follows Michael's idea of using user profile fields. This still needs a lot of work, but you're quite welcome to load the patch and try it out for yourself. Instructions on how to use this patch are located in the comments of issue MDL-34754 (a sub issue of this one).

          Concerns that I have about this solution are that it does a lot more calls on the database, which could potentially slow down your system. If I continue to progress down this path then I will need to possible alter different pages that provide lists of students on a single page.

          I welcome your suggestions and recommendations on this issue.

          Show
          Adrian Greeve added a comment - I have been working on one possible solution to this improvement. It follows Michael's idea of using user profile fields. This still needs a lot of work, but you're quite welcome to load the patch and try it out for yourself. Instructions on how to use this patch are located in the comments of issue MDL-34754 (a sub issue of this one). Concerns that I have about this solution are that it does a lot more calls on the database, which could potentially slow down your system. If I continue to progress down this path then I will need to possible alter different pages that provide lists of students on a single page. I welcome your suggestions and recommendations on this issue.
          Hide
          Thomas Robb added a comment -

          Hello Adrian. Actually, we are not interested in a "patch" since that would only be a stop-gap solution. We need a solution that quickly allows the sys admin to determine which, of many name fields – including the idnumber – to appear in all relevant display with full capability to sort on them, as can be done with the standard two name fields. Also politically, it renders non-Western alphabets as second-class citizens. Since Moodle would not work 'out of the box' without numerous adjustments. We are looking for a total solution, well-integrated in to all aspects of the base code, but which would not impact on the performance of those who do not require these special features.

          MoodleJapan (MAJ) is about to send out a request for bids for a feasibility study. Once the study is completed, it will be submitted to Moodle HQ for comments and approval, after which MAJ and other interested parties will attempt to fund the coding.

          Show
          Thomas Robb added a comment - Hello Adrian. Actually, we are not interested in a "patch" since that would only be a stop-gap solution. We need a solution that quickly allows the sys admin to determine which, of many name fields – including the idnumber – to appear in all relevant display with full capability to sort on them, as can be done with the standard two name fields. Also politically, it renders non-Western alphabets as second-class citizens. Since Moodle would not work 'out of the box' without numerous adjustments. We are looking for a total solution, well-integrated in to all aspects of the base code, but which would not impact on the performance of those who do not require these special features. MoodleJapan (MAJ) is about to send out a request for bids for a feasibility study. Once the study is completed, it will be submitted to Moodle HQ for comments and approval, after which MAJ and other interested parties will attempt to fund the coding.
          Hide
          Adrian Greeve added a comment - - edited

          I have now created a new version with suggestions made by various people. I've created a couple of screen shots and provided the code.
          As this is a proof of concept model, the extra name fields only show up on the participants page. If we decided to go ahead with this solution then changes will have to be made in all areas of code where we want to add these fields.

          (In my haste of posting up the screen shots I accidentally added the first and last name the wrong way around.)

          Show
          Adrian Greeve added a comment - - edited I have now created a new version with suggestions made by various people. I've created a couple of screen shots and provided the code. As this is a proof of concept model, the extra name fields only show up on the participants page. If we decided to go ahead with this solution then changes will have to be made in all areas of code where we want to add these fields. (In my haste of posting up the screen shots I accidentally added the first and last name the wrong way around.)
          Hide
          Don Hinkelman added a comment -

          Hi Adrian,
          Thanks for the screen shots. It is exciting to see some movement on this issue. One important field missing from the first screenshot(#873) is the Student ID column. In Japan, and likely other countries, 90% of our reports, name lists, assignments, etc. need to be in order of student ID. All attendance is taken in this order, and grades are listed by this student ID. As Tom Robb mentioned in the issue description above, could you add a student ID column to the left of the firstname/surname column? It actually is the most important of all user identifier fields.

          Show
          Don Hinkelman added a comment - Hi Adrian, Thanks for the screen shots. It is exciting to see some movement on this issue. One important field missing from the first screenshot(#873) is the Student ID column. In Japan, and likely other countries, 90% of our reports, name lists, assignments, etc. need to be in order of student ID. All attendance is taken in this order, and grades are listed by this student ID. As Tom Robb mentioned in the issue description above, could you add a student ID column to the left of the firstname/surname column? It actually is the most important of all user identifier fields.
          Hide
          Thomas Robb added a comment -

          Listing of affected core files

          Show
          Thomas Robb added a comment - Listing of affected core files
          Hide
          Thomas Robb added a comment -

          Listing of db tables – new and altered

          Show
          Thomas Robb added a comment - Listing of db tables – new and altered
          Hide
          Thomas Robb added a comment -

          Outline of the proposed programming strategy.

          Show
          Thomas Robb added a comment - Outline of the proposed programming strategy.
          Hide
          Thomas Robb added a comment -

          MoodleJapan requested bids for a preliminary study of how "alternate names" could be added to the Moodle core code in the most efficient manner. "Version 2" in Sapporo was then charged with the study and they have come through with the following report.

          We hope that Moodle HQ will carefully consider their plan, and if it looks feasible, we are prepared to discuss the coding process – particularly "who" and "how much"! MoodleJapan does not have sufficient funds to pay for the coding on its own, but it might be feasible if Moodle HQ were willing for forgo a sufficient part of the amount that we normally donate from the proceeds of the annual Moot to this cause. We estimate the that cost will come to somewhere between $10,000 and $16,000 USD.

          I have attached their report in three separate uploads. They also provided an estimate for the total cost of the development work which comes to approximately $16,000USD.

          Show
          Thomas Robb added a comment - MoodleJapan requested bids for a preliminary study of how "alternate names" could be added to the Moodle core code in the most efficient manner. "Version 2" in Sapporo was then charged with the study and they have come through with the following report. We hope that Moodle HQ will carefully consider their plan, and if it looks feasible, we are prepared to discuss the coding process – particularly "who" and "how much"! MoodleJapan does not have sufficient funds to pay for the coding on its own, but it might be feasible if Moodle HQ were willing for forgo a sufficient part of the amount that we normally donate from the proceeds of the annual Moot to this cause. We estimate the that cost will come to somewhere between $10,000 and $16,000 USD. I have attached their report in three separate uploads. They also provided an estimate for the total cost of the development work which comes to approximately $16,000USD.
          Hide
          Adrian Greeve added a comment -

          Hello Thomas,

          I have studied the plan that you have submitted. And I have a good understanding of what you are trying to achieve. In general the concept is plausible and would work, but the addition of new functions and sql queries makes the current system more complex and difficult to extend. Ideally, the user retrieval code needs to be re-written and centralized so that there is no need to alter code in so many areas. An MDL has already been raised for this improvement (MDL-37047).

          Some of the code examples look to implement the use of the student id number. This has already been implemented with MDL-26647.

          Martin is basically the gate-keeper for what does and does not get included in core code. I have added him as a watcher so that he might make a comment on this issue. I think that Martin would be the best person to consult with about getting this project completed.

          Show
          Adrian Greeve added a comment - Hello Thomas, I have studied the plan that you have submitted. And I have a good understanding of what you are trying to achieve. In general the concept is plausible and would work, but the addition of new functions and sql queries makes the current system more complex and difficult to extend. Ideally, the user retrieval code needs to be re-written and centralized so that there is no need to alter code in so many areas. An MDL has already been raised for this improvement ( MDL-37047 ). Some of the code examples look to implement the use of the student id number. This has already been implemented with MDL-26647 . Martin is basically the gate-keeper for what does and does not get included in core code. I have added him as a watcher so that he might make a comment on this issue. I think that Martin would be the best person to consult with about getting this project completed.
          Hide
          Thomas Robb added a comment - - edited

          Hello Martin, Adrian, Michael and all

          Thank you for your response. We also believe that the plan "can work" but we are not tied to the suggested approach IF there is another, more feasible method that achieves the same result.

          We, however, are not looking at the issue from only the programming standpoint, but from two other viewpoints, 1) the matter of functionality of the program in Asian contexts and 2) a subtle ethical issue of the current version being discriminatory against the Non-Western world and culture.

          Let me elaborate on the second issue first. When Moodle was in its infancy people were happy to use it as long as it worked. Now, however, it is a mature product that should be marketable in any major country in the world. We are past the point where, in good conscience, we can say to the majority of the population of the world, "Here it is, we've made it for our own needs, but if you want to use it, you've got to either use it 'as is' or jump through your own set of hoops to make it work for you" – which could perhaps be termed "computer code imperialism". We need to face the fact that most people in the world do not have a unique "first name" and "last name" in the English sense of the terms.

          Now, on the first issue, we in MoodleJapan are willing to work with MoodleHQ to add this additional functionality. We are willing to sending someone to Perth to talk over the nitty gritty face-to-face or simply to help fund the development elsewhere. As stated in my previous message, one source of funding coud be part of the funds that we annually donate to MoodleHQ from the Moot proceeds.

          Concerning the matter of the idnumber field, I have posted a separate tracker issue concerning this (MDL-38061) since it does not actually work in the new Assignment module. Furthermore, the idnumber should be one of the fields that can be selected to appear in the left column of reports as the primary sortable field.

          Basically, it is no longer a technical issue so much as one of priorities. We hope that you can consider pushing this issue to the top of the list.

          Cheers,
          Tom

          Show
          Thomas Robb added a comment - - edited Hello Martin, Adrian, Michael and all Thank you for your response. We also believe that the plan "can work" but we are not tied to the suggested approach IF there is another, more feasible method that achieves the same result. We, however, are not looking at the issue from only the programming standpoint, but from two other viewpoints, 1) the matter of functionality of the program in Asian contexts and 2) a subtle ethical issue of the current version being discriminatory against the Non-Western world and culture. Let me elaborate on the second issue first. When Moodle was in its infancy people were happy to use it as long as it worked. Now, however, it is a mature product that should be marketable in any major country in the world. We are past the point where, in good conscience, we can say to the majority of the population of the world, "Here it is, we've made it for our own needs, but if you want to use it, you've got to either use it 'as is' or jump through your own set of hoops to make it work for you" – which could perhaps be termed "computer code imperialism". We need to face the fact that most people in the world do not have a unique "first name" and "last name" in the English sense of the terms. Now, on the first issue, we in MoodleJapan are willing to work with MoodleHQ to add this additional functionality. We are willing to sending someone to Perth to talk over the nitty gritty face-to-face or simply to help fund the development elsewhere. As stated in my previous message, one source of funding coud be part of the funds that we annually donate to MoodleHQ from the Moot proceeds. Concerning the matter of the idnumber field, I have posted a separate tracker issue concerning this ( MDL-38061 ) since it does not actually work in the new Assignment module. Furthermore, the idnumber should be one of the fields that can be selected to appear in the left column of reports as the primary sortable field. Basically, it is no longer a technical issue so much as one of priorities. We hope that you can consider pushing this issue to the top of the list. Cheers, Tom
          Hide
          Michael de Raadt added a comment -

          Hi, Thom.

          Thanks for continuing to support this issue. It's good to have vocal supporters behind such initiatives to push them along. If we've caused any frustration through our lack of progress, I apologise.

          We recognise the importance of this functionality. It is usually our goal to aim for an ideal solution, rather than a less favourable one, but perhaps an ideal solution is not immediately feasible and a compromise may be acceptable for the present time. This is not so much a matter of resources or money, but of fitting in with the constraints of the current Moodle architecture.

          We will have a meeting here next week, when Martin returns, to discuss the various options that we can pursue now and into the future.

          Show
          Michael de Raadt added a comment - Hi, Thom. Thanks for continuing to support this issue. It's good to have vocal supporters behind such initiatives to push them along. If we've caused any frustration through our lack of progress, I apologise. We recognise the importance of this functionality. It is usually our goal to aim for an ideal solution, rather than a less favourable one, but perhaps an ideal solution is not immediately feasible and a compromise may be acceptable for the present time. This is not so much a matter of resources or money, but of fitting in with the constraints of the current Moodle architecture. We will have a meeting here next week, when Martin returns, to discuss the various options that we can pursue now and into the future.
          Hide
          Martin Dougiamas added a comment -

          Hi guys,

          We met today and hashed out the issues. I guess it's been complicated over the years with a variety of approaches and also the fact that there are some other very similar requirements around that are not quite covered by the Asian languages use cases. I hope we can resolve it soon.

          I've jotted a summary of a solution at http://docs.moodle.org/dev/Alternate_name_fields and Adrian is going to flesh that out with more details.

          In particular we need to agree on exactly what new fields we will add to the user table. We need to make sure they truly covers all the use cases, because it's not going to be as flexible as using custom name fields, so I'd like to then get some feedback from those in various countries to make sure it's OK before we start.

          My starting guess is:

          • firstnamephon
          • lastnamephon
          • alternatename (useful for three-name countries too?)
          • aliasname (for handles and anonymity)

          Assuming that is all OK, I think the implementation of the main stuff would be pretty quick and safe, Adrian can easily do it. After that there will be a lot of "mopping up" to improve the display of names in various smaller pages around Moodle or to improve the efficiency with better queries - it would be nice to get help with that from the community, since they have a better idea of exactly what they want to see there.

          I have one main question ... with the phonetic fields, will sorting these by UTF actually work? We rely on the database to do sorting and I'm not sure it's enough for what you want. If so then awesome, but if not then please let's think this through, because Asian language sorting is a pretty big kettle of fish and perhaps could be considered separately to this.

          Cheers,
          Martin

          Show
          Martin Dougiamas added a comment - Hi guys, We met today and hashed out the issues. I guess it's been complicated over the years with a variety of approaches and also the fact that there are some other very similar requirements around that are not quite covered by the Asian languages use cases. I hope we can resolve it soon. I've jotted a summary of a solution at http://docs.moodle.org/dev/Alternate_name_fields and Adrian is going to flesh that out with more details. In particular we need to agree on exactly what new fields we will add to the user table. We need to make sure they truly covers all the use cases, because it's not going to be as flexible as using custom name fields, so I'd like to then get some feedback from those in various countries to make sure it's OK before we start. My starting guess is: firstnamephon lastnamephon alternatename (useful for three-name countries too?) aliasname (for handles and anonymity) Assuming that is all OK, I think the implementation of the main stuff would be pretty quick and safe, Adrian can easily do it. After that there will be a lot of "mopping up" to improve the display of names in various smaller pages around Moodle or to improve the efficiency with better queries - it would be nice to get help with that from the community, since they have a better idea of exactly what they want to see there. I have one main question ... with the phonetic fields, will sorting these by UTF actually work? We rely on the database to do sorting and I'm not sure it's enough for what you want. If so then awesome, but if not then please let's think this through, because Asian language sorting is a pretty big kettle of fish and perhaps could be considered separately to this. Cheers, Martin
          Hide
          Thomas Robb added a comment -

          Martin asks:

          "I have one main question ... with the phonetic fields, will sorting these by UTF actually work? We rely on the database to do sorting and I'm not sure it's enough for what you want. If so then awesome, but if not then please let's think this through, because Asian language sorting is a pretty big kettle of fish and perhaps could be considered separately to this."

          Response:

          There is a problem with the sortability of the various alphabets. Many alphabetical scripts do have a sorting order that is different from UTF-8. As Martin hints, this requires a separate algorithm to be implemented in order to get them into the standard order. In Thai and Indic languages, for example, a short "i" precedes that consonant that it phonetically follows so without and sorting rules, it would come out in the wrong order. For example a word pronounced "adi" is actually written as "aid". Also all of the sounds/glyphs are in the same UTF-8 order regardless of the Indic language, but the various languages have some differences in alphabetization.

          In Korean, North and South Korea have different alphabetization standards. and the common UTF-8 encoding doesn't sort properly for either.

          Nevertheless, even without an encoding to sort into the correct order, the result of a sort on the UTF-8 codes will come out in a predictable order, just not the standard one and would be usable until a sorting algorithm is implemented for that language.

          In the ideal world, there would be appropriate algorithms to create a code string in a separate field which would be referred to when sorting is requested. The sort field would be updated each time the relevant phonetic field is updated. (Naturally, if someone manually modifies the phonetic field in the database, the sort field would remain unaffected.)

          Actually, even in English we sometimes have a need for such a field. Book titles, for example, often start with "The" which many applications wish to be ignored when displaying a list of book titles. This is accomplished by having a separate sorting field. This the title field has "The Adventures of Tom Sawyer" but the alpha field has ""Adventures of Tom Sawyer, The".

          I would suggest that a new function sortlang() be built that would take 1) a character string and 2) a variable representing the desired sorting algorithm and then return the desired sort string, which can then be input into the designated sort field paired with the original character string.

          The actual rules for how to create a sort string for any given language are not so difficult to code. I've personally constructed scripts to sort Japanese kana and Khmer into their respective standard alphabetical orders. Surely such algorithms for many languages already exist on the Internet and can be compiled into a workable set of sort algorithms. The individuals in charge of the various Moodle language pack maintainers should be the first people to be queried concerning their respective sorting needs.

          In addition to applying the sortlang() function to the alternate name fields, we might also consider implementing them in the database module, where the user can designate a specific field to hold the sort field and designate the original field as a trigger: Anytime the original field is modified, the sort field is also populated with the corresponding data.

          Show
          Thomas Robb added a comment - Martin asks: "I have one main question ... with the phonetic fields, will sorting these by UTF actually work? We rely on the database to do sorting and I'm not sure it's enough for what you want. If so then awesome, but if not then please let's think this through, because Asian language sorting is a pretty big kettle of fish and perhaps could be considered separately to this." Response: There is a problem with the sortability of the various alphabets. Many alphabetical scripts do have a sorting order that is different from UTF-8. As Martin hints, this requires a separate algorithm to be implemented in order to get them into the standard order. In Thai and Indic languages, for example, a short "i" precedes that consonant that it phonetically follows so without and sorting rules, it would come out in the wrong order. For example a word pronounced "adi" is actually written as "aid". Also all of the sounds/glyphs are in the same UTF-8 order regardless of the Indic language, but the various languages have some differences in alphabetization. In Korean, North and South Korea have different alphabetization standards. and the common UTF-8 encoding doesn't sort properly for either. Nevertheless, even without an encoding to sort into the correct order, the result of a sort on the UTF-8 codes will come out in a predictable order, just not the standard one and would be usable until a sorting algorithm is implemented for that language. In the ideal world, there would be appropriate algorithms to create a code string in a separate field which would be referred to when sorting is requested. The sort field would be updated each time the relevant phonetic field is updated. (Naturally, if someone manually modifies the phonetic field in the database, the sort field would remain unaffected.) Actually, even in English we sometimes have a need for such a field. Book titles, for example, often start with "The" which many applications wish to be ignored when displaying a list of book titles. This is accomplished by having a separate sorting field. This the title field has "The Adventures of Tom Sawyer" but the alpha field has ""Adventures of Tom Sawyer, The". I would suggest that a new function sortlang() be built that would take 1) a character string and 2) a variable representing the desired sorting algorithm and then return the desired sort string, which can then be input into the designated sort field paired with the original character string. The actual rules for how to create a sort string for any given language are not so difficult to code. I've personally constructed scripts to sort Japanese kana and Khmer into their respective standard alphabetical orders. Surely such algorithms for many languages already exist on the Internet and can be compiled into a workable set of sort algorithms. The individuals in charge of the various Moodle language pack maintainers should be the first people to be queried concerning their respective sorting needs. In addition to applying the sortlang() function to the alternate name fields, we might also consider implementing them in the database module, where the user can designate a specific field to hold the sort field and designate the original field as a trigger: Anytime the original field is modified, the sort field is also populated with the corresponding data.
          Hide
          Edmund Edgar added a comment -

          Thomas, have you looked at the various Unicode database sort collations which you can tell your database to use when sorting? If so, would you say they're not up to the task?
          http://collation-charts.org/mysql60/by-charset.shtml#utf8

          Presumably even if you have a custom solution on the PHP side you're still going to need the database to be able to sort the right way for when you need to display things in pages. And since this stuff can be a bit complicated, it feels like a good problem to punt off to a database vendor...

          PS. I guess you guys have already discussed this and have good reasons for doing it this way, but it does strike me that the long-term sort problem on this scheme is going to be sorting by the names in the "alternate" field, which is only hard because we've squidged two bits of data (first and last name) into one box...

          Show
          Edmund Edgar added a comment - Thomas, have you looked at the various Unicode database sort collations which you can tell your database to use when sorting? If so, would you say they're not up to the task? http://collation-charts.org/mysql60/by-charset.shtml#utf8 Presumably even if you have a custom solution on the PHP side you're still going to need the database to be able to sort the right way for when you need to display things in pages. And since this stuff can be a bit complicated, it feels like a good problem to punt off to a database vendor... PS. I guess you guys have already discussed this and have good reasons for doing it this way, but it does strike me that the long-term sort problem on this scheme is going to be sorting by the names in the "alternate" field, which is only hard because we've squidged two bits of data (first and last name) into one box...
          Hide
          Martin Dougiamas added a comment - - edited

          Thomas, sorting in PHP isn't hard but it can be painfully inefficient when listing, say, 100,000 users in a paged format and someone wants to show page 23. This means you actually need to pull everything from the database, sort it in PHP with the custom sort and then throw most of it away.

          In SQL of course this is a trivial and quick operation using LIMIT.

          Obviously a database way of doing this would be preferred, but if it's not possible then it's just not possible and those languages that won't work with UTF need to identified as such and done using the slower PHP way.

          Edmund, yes the collations were what I was driving at ... though not sure how well they work, and how they would work on a user table, say, with mixed language data.

          Show
          Martin Dougiamas added a comment - - edited Thomas, sorting in PHP isn't hard but it can be painfully inefficient when listing, say, 100,000 users in a paged format and someone wants to show page 23. This means you actually need to pull everything from the database, sort it in PHP with the custom sort and then throw most of it away. In SQL of course this is a trivial and quick operation using LIMIT. Obviously a database way of doing this would be preferred, but if it's not possible then it's just not possible and those languages that won't work with UTF need to identified as such and done using the slower PHP way. Edmund, yes the collations were what I was driving at ... though not sure how well they work, and how they would work on a user table, say, with mixed language data.
          Hide
          Thomas Robb added a comment -

          First to respond to Edmund, the collation sequences will not do the job. All they do is to order the characters and indicate which ones should be considered as equivalent for sorting purposes. If characters are in reverse order from how they should be sorted, as in my earlier "aid/adi" example, then the string has to be pre-processed. Another example is the long vowel sign used in Japanese katakana. There is a single symbol that is used, but the value it takes for pronunciation and sorting, is the value of the preceding vowel. Thus, you can have words like "ka-" and "ki-" that need to be alphabetised as "kaa" and "kii", respectively.

          As Martin points out, sorting like this can be very cpu intensive, thus the best solution would be to have a sorting key (aka collation key) in the database which is updated anytime the corresponding display field is updated. When a sorting algorithm isn't available, then the only choice is to use the appropriate collation sequence for alphabetisation.

          Show
          Thomas Robb added a comment - First to respond to Edmund, the collation sequences will not do the job. All they do is to order the characters and indicate which ones should be considered as equivalent for sorting purposes. If characters are in reverse order from how they should be sorted, as in my earlier "aid/adi" example, then the string has to be pre-processed. Another example is the long vowel sign used in Japanese katakana. There is a single symbol that is used, but the value it takes for pronunciation and sorting, is the value of the preceding vowel. Thus, you can have words like "ka-" and "ki-" that need to be alphabetised as "kaa" and "kii", respectively. As Martin points out, sorting like this can be very cpu intensive, thus the best solution would be to have a sorting key (aka collation key) in the database which is updated anytime the corresponding display field is updated. When a sorting algorithm isn't available, then the only choice is to use the appropriate collation sequence for alphabetisation.
          Hide
          Martin Dougiamas added a comment - - edited

          Would everyone be OK if we kept sorting out of scope for this issue (and made a new issue for it), in order to help this one have a better chance for 2.5? The more I think about it the more my head hurts. eg multiple tables, multiple fields per table, multiple languages per field, database update madness when inserting new entries, clumping display into "alphabet" groups like A-Z for euro languages etc ...

          Show
          Martin Dougiamas added a comment - - edited Would everyone be OK if we kept sorting out of scope for this issue (and made a new issue for it), in order to help this one have a better chance for 2.5? The more I think about it the more my head hurts. eg multiple tables, multiple fields per table, multiple languages per field, database update madness when inserting new entries, clumping display into "alphabet" groups like A-Z for euro languages etc ...
          Hide
          Michael de Raadt added a comment -

          We had a good discussion about this over dinner tonight.

          I don't want to speak for Tom, but I think there is a stronger desire to get the alternate name fields than the sorting. Tom did suggest a strategy for allowing sorting in a second language that sounded good, but it will be quite complex and will need a bit more threshing out before it could even start to be considered for implementation. I suggest Tom starts a second issue, linking it to this issue, to cover the sorting aspect.

          We also discussed the number of fields needed. It could be argued that the alternate name should have first and last name components. It would be good to get some further discussion on that before we finalise changes. We don't want to have to do this again.

          I also raised the default for the capability to allow teachers to edit name formats. I had one vote for allowing teachers to edit name formats in courses and activities by default and two votes against (not including my vote against). I think the security considerations associated with allowing teachers to potentially override a site-wide setting, potentially revealing personal information, should cause us to set a default of not allowed with an admin able to allow a teacher to have this capability. But there may be some people with strong reasons for leaving this more open.

          Show
          Michael de Raadt added a comment - We had a good discussion about this over dinner tonight. I don't want to speak for Tom, but I think there is a stronger desire to get the alternate name fields than the sorting. Tom did suggest a strategy for allowing sorting in a second language that sounded good, but it will be quite complex and will need a bit more threshing out before it could even start to be considered for implementation. I suggest Tom starts a second issue, linking it to this issue, to cover the sorting aspect. We also discussed the number of fields needed. It could be argued that the alternate name should have first and last name components. It would be good to get some further discussion on that before we finalise changes. We don't want to have to do this again. I also raised the default for the capability to allow teachers to edit name formats. I had one vote for allowing teachers to edit name formats in courses and activities by default and two votes against (not including my vote against). I think the security considerations associated with allowing teachers to potentially override a site-wide setting, potentially revealing personal information, should cause us to set a default of not allowed with an admin able to allow a teacher to have this capability. But there may be some people with strong reasons for leaving this more open.
          Hide
          Martin Dougiamas added a comment -

          IMO, the really important consideration is to support languages with more than two names. Spanish and Arabic being obviously huge ones. I'll direct some Spanish and Arabic people here to look at the spec and comment on suitability for them.

          Show
          Martin Dougiamas added a comment - IMO, the really important consideration is to support languages with more than two names. Spanish and Arabic being obviously huge ones. I'll direct some Spanish and Arabic people here to look at the spec and comment on suitability for them.
          Hide
          Martin Dougiamas added a comment -

          David Monllao (one of our Spanish developers) is of the opinion that in Spanish they are quite used to putting both surnames in the one field. So no problem there probably, but it would be good to more confirmation of this.

          Also still need some feedback from an Arabic perspective. I'll ask Muhammad ibn Saeed ibn Abd al-Aziz al-Filasteeni about it.

          Show
          Martin Dougiamas added a comment - David Monllao (one of our Spanish developers) is of the opinion that in Spanish they are quite used to putting both surnames in the one field. So no problem there probably, but it would be good to more confirmation of this. Also still need some feedback from an Arabic perspective. I'll ask Muhammad ibn Saeed ibn Abd al-Aziz al-Filasteeni about it.
          Hide
          Eloy Lafuente (stronk7) added a comment - - edited

          Hi,

          in Spanish we certainly have 2^n surnames (accumulated from all our ancestors), although only the 1st pair is used legally.

          And they have an official order (dad 1st and then mum, by default, or the opposite, each person can decide about that legally). But once decided, the combination becomes de official one.

          So for all meanings, to have both together in an unique surname field is enough, agree with David.

          BUT, there is one little details where I see Moodle lacking other software and it's that, often, we ignore particles/articles for sorting (and displaying!) so for example:

          Michael de Raadt

          should be sorted as:

          Michael Raadt, de

          Or more complex, you can find people named:

          Eloy Lafuente (my case, my 1st surname is "Lafuente" all-together)
          Eloy la Fuente
          Eloy de Lafuente
          Eloy de la Fuente

          that, for sorting purposes (and sometimes display too), should be:

          Eloy Lafuente
          Eloy Fuente, la
          Eloy Lafuente, de
          Eloy Fuente, de la

          and that can become more complex if we put the second surname in action:

          Eloy la Fuente de la Plaza

          for sorting, could be:

          Eloy Fuente, la Plaza, de la

          But, in practice... not many software allows that elaborated processing of particles/articles and peoples use to manually:

          1) Ignore the particles thing completely.
          2) If it's required for some legal cause, then the surnames are introduced in the order they need to be sorted.

          So I'll conclude that one unique field for any pair of surnames (current way) is 95% ok. Although if available, it would be correct to have a place to those articles/particles and/or to separate surnames.. but surely as something to define as optional settings. Current behavior is perfectly good as default.

          Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - - edited Hi, in Spanish we certainly have 2^n surnames (accumulated from all our ancestors), although only the 1st pair is used legally. And they have an official order (dad 1st and then mum, by default, or the opposite, each person can decide about that legally ). But once decided, the combination becomes de official one. So for all meanings, to have both together in an unique surname field is enough, agree with David. BUT, there is one little details where I see Moodle lacking other software and it's that, often, we ignore particles/articles for sorting (and displaying!) so for example: Michael de Raadt should be sorted as: Michael Raadt, de Or more complex, you can find people named: Eloy Lafuente (my case, my 1st surname is "Lafuente" all-together) Eloy la Fuente Eloy de Lafuente Eloy de la Fuente that, for sorting purposes (and sometimes display too), should be: Eloy Lafuente Eloy Fuente, la Eloy Lafuente, de Eloy Fuente, de la and that can become more complex if we put the second surname in action: Eloy la Fuente de la Plaza for sorting, could be: Eloy Fuente, la Plaza, de la But, in practice... not many software allows that elaborated processing of particles/articles and peoples use to manually: 1) Ignore the particles thing completely. 2) If it's required for some legal cause, then the surnames are introduced in the order they need to be sorted. So I'll conclude that one unique field for any pair of surnames (current way) is 95% ok. Although if available, it would be correct to have a place to those articles/particles and/or to separate surnames.. but surely as something to define as optional settings. Current behavior is perfectly good as default. Ciao
          Hide
          Eloy Lafuente (stronk7) added a comment -

          One more point... note that, also, that the whole articles thing applies, generally, only to sorting in lists (and optionally displaying there), I mean, when one name is shown individually (post author...), then the "real" name is the preferred name to be displayed ("Michael de Raadt"). Displaying it as "Michael Raadt, de" is somehow ugly.

          Show
          Eloy Lafuente (stronk7) added a comment - One more point... note that, also, that the whole articles thing applies, generally, only to sorting in lists (and optionally displaying there), I mean, when one name is shown individually (post author...), then the "real" name is the preferred name to be displayed ("Michael de Raadt"). Displaying it as "Michael Raadt, de" is somehow ugly.
          Hide
          Thomas Robb added a comment -

          About sorting again... I found out that the Indic languages DO sort properly. Even though, as I said, the short vowel "i" precedes the consonant that it should follow, it is actually coded in the proper position for pronunciation, but displayed in front of the consonant when typed or printed. Thus there is no sorting problem to worry about. In Dutch, however, one Moodle site that I am familiar with places all of those surname prefixes after the main part of the name: firstname: Michael lastname: Raadt de; firstname; Vincent lastname: Gogh van

          For these, an expanded set of name fields would allow them to place the prefixes in a separate field so that they could be designated to appear before the main surname on screen displays but allowing the main surname to be used for sorting purposes. One kludge that can be used with two field names is to place the prefix at the end of the firstname, thus: firstname: Michael de lastname: Raadt; ; firstname; Vincent van lastname: Gogh

          Show
          Thomas Robb added a comment - About sorting again... I found out that the Indic languages DO sort properly. Even though, as I said, the short vowel "i" precedes the consonant that it should follow, it is actually coded in the proper position for pronunciation, but displayed in front of the consonant when typed or printed. Thus there is no sorting problem to worry about. In Dutch, however, one Moodle site that I am familiar with places all of those surname prefixes after the main part of the name: firstname: Michael lastname: Raadt de; firstname; Vincent lastname: Gogh van For these, an expanded set of name fields would allow them to place the prefixes in a separate field so that they could be designated to appear before the main surname on screen displays but allowing the main surname to be used for sorting purposes. One kludge that can be used with two field names is to place the prefix at the end of the firstname, thus: firstname: Michael de lastname: Raadt; ; firstname; Vincent van lastname: Gogh
          Hide
          Martin Dougiamas added a comment -

          My feeling is that perhaps sorting should still be left to a follow-up issue.

          Regarding Arabic, I got one response supporting middlenames.

          In fact the existing Alternate name field could be used to support this without too much trouble. You'd just need to define the token string to be "firstname alternatename lastname" and probably also update the language pack to make it display nice in the user profile etc. it's not totally pretty but it would work. (XA)

          Otherwise we could just add a "middlename" field explicitly and make it simple. (XB)

          XA or XB?

          Show
          Martin Dougiamas added a comment - My feeling is that perhaps sorting should still be left to a follow-up issue. Regarding Arabic, I got one response supporting middlenames. In fact the existing Alternate name field could be used to support this without too much trouble. You'd just need to define the token string to be "firstname alternatename lastname" and probably also update the language pack to make it display nice in the user profile etc. it's not totally pretty but it would work. (XA) Otherwise we could just add a "middlename" field explicitly and make it simple. (XB) XA or XB?
          Hide
          Martin Dougiamas added a comment -

          +1 for XB

          Show
          Martin Dougiamas added a comment - +1 for XB
          Hide
          Michael de Raadt added a comment -

          +1 for XB. (Better to do it now)

          Show
          Michael de Raadt added a comment - +1 for XB. (Better to do it now)
          Hide
          Thomas Robb added a comment -

          Sorry, but (XB) is not acceptable. Japanese, Chinese and Koreans do NOT have middle names. Please name the field as something that is not so ethnocentric. If there are three fields that can be "mixed and matched" in various displays, it would be an improvement on the current system. Note that there are two somewhat different issues here. One is how $fullname is formed and displayed, and the other is how each of these fields can be displayed in a selectable order (by the site admin, at least) in various reports. To take Japan as the example here, Japanese may want to sort on the IDnumber since these are typically assigned in the order of the Kana syllabary. Expats, on the other hand, would wan to sort either by Surname-Given name or Given-Surname, depending on what name they use when they call on the students in class. Will this be do-able with either the (XA) or (XB) proposal?

          Show
          Thomas Robb added a comment - Sorry, but (XB) is not acceptable. Japanese, Chinese and Koreans do NOT have middle names. Please name the field as something that is not so ethnocentric. If there are three fields that can be "mixed and matched" in various displays, it would be an improvement on the current system. Note that there are two somewhat different issues here. One is how $fullname is formed and displayed, and the other is how each of these fields can be displayed in a selectable order (by the site admin, at least) in various reports. To take Japan as the example here, Japanese may want to sort on the IDnumber since these are typically assigned in the order of the Kana syllabary. Expats, on the other hand, would wan to sort either by Surname-Given name or Given-Surname, depending on what name they use when they call on the students in class. Will this be do-able with either the (XA) or (XB) proposal?
          Hide
          Michael de Raadt added a comment -

          Hi, Tom.

          I don't think that having the field there mandates that people use it; in the same way we don't require people to use the phonetic or alternate name fields. The point is we're trying not to be ethnocentric by including all these fields. The only further step we could take would be to give generic titles to all the fields like "Alternate name 1", but I'm not sure that is wise.

          People can already sort by ID number if that field is displayed and data is filled in it. I don't think we need to mix that with the name fields.

          Show
          Michael de Raadt added a comment - Hi, Tom. I don't think that having the field there mandates that people use it; in the same way we don't require people to use the phonetic or alternate name fields. The point is we're trying not to be ethnocentric by including all these fields. The only further step we could take would be to give generic titles to all the fields like "Alternate name 1", but I'm not sure that is wise. People can already sort by ID number if that field is displayed and data is filled in it. I don't think we need to mix that with the name fields.
          Hide
          Adrian Greeve added a comment - - edited

          I agree with Michael. The names that we give the fields is simply only a suggestion as to how it could be used. If anyone feels strongly about the field name then maybe they can ask who ever does the translation for their specific language to label it something different. Naming the fields with generic terms I also think is not a good idea. If we take making the fields non 'ethnocentric' then where do we stop. Most Indonesian people do not have a last name, first name no longer seems like an appropriate term. We would have to rename all name fields to something generic.

          +1 for XB

          Show
          Adrian Greeve added a comment - - edited I agree with Michael. The names that we give the fields is simply only a suggestion as to how it could be used. If anyone feels strongly about the field name then maybe they can ask who ever does the translation for their specific language to label it something different. Naming the fields with generic terms I also think is not a good idea. If we take making the fields non 'ethnocentric' then where do we stop. Most Indonesian people do not have a last name, first name no longer seems like an appropriate term. We would have to rename all name fields to something generic. +1 for XB
          Hide
          Eloy Lafuente (stronk7) added a comment -

          I hope all this is going to be something only available if the admin opts-in.

          And it won't be shown/used/listed at all by default, so people will continue using the current & simpler way (first+second and done), correct?

          (crossing fingers...)

          My POV is that 1% of sites requiring such phonetic/romanization/sorting detail (and I think I'm exaggerating largely) cannot cause the other 99% to get complex-er/slower display/sorting criteria by default. And it's going to be complex-er/slower (I bet).

          Just my thoughts... ciao

          Show
          Eloy Lafuente (stronk7) added a comment - I hope all this is going to be something only available if the admin opts-in. And it won't be shown/used/listed at all by default, so people will continue using the current & simpler way (first+second and done), correct? (crossing fingers...) My POV is that 1% of sites requiring such phonetic/romanization/sorting detail (and I think I'm exaggerating largely) cannot cause the other 99% to get complex-er/slower display/sorting criteria by default. And it's going to be complex-er/slower (I bet). Just my thoughts... ciao
          Hide
          Don Hinkelman added a comment -

          Ahem, Eloy. Check the world population figures. Asia composes slightly more than 1% of the population. Right now Moodle is a fringe LMS in Asia mostly relegated to use by Western foreigners (me, for example). If we want to continue as a Western/Anglo/Roman LMS then the 1% figure may be correct. If Moodle is to be adopted by Asian institutions (China, India, etc.), then 60% will be a closer figure. 100% of the sites in Japan will use this feature.

          Show
          Don Hinkelman added a comment - Ahem, Eloy. Check the world population figures. Asia composes slightly more than 1% of the population. Right now Moodle is a fringe LMS in Asia mostly relegated to use by Western foreigners (me, for example). If we want to continue as a Western/Anglo/Roman LMS then the 1% figure may be correct. If Moodle is to be adopted by Asian institutions (China, India, etc.), then 60% will be a closer figure. 100% of the sites in Japan will use this feature.
          Hide
          Petr Škoda added a comment -

          Hmm, translators may decide to name the user fields in whatever way they want. In my computer it is me who decides what is the appropriate order. I would expect Moodle to allow me to configure it with some reasonable defaults based on my language or global admin settings.

          1/ add some more general name fields - display name, middle, nickname, name3, name4 or whatever else (all of these would have to be included in all user related query results - we already have API for required avatar fields, we could use that)
          2/ let translators decide how these fields are called in each language (admins may override it)
          3/ let translators decide what is the expected default name format in each language (new string)
          4/ let administrators decide what is the default and what are other options how to show names
          5/ let users select own name format preference if they do not like default
          6/ collation support in databases is an inconsistent mess, we need to add support in DML layer first, for now there is only one global collation

          Performance is the main problem here, because names are printed very often, it should imo go into the user table directly, text fields are definitely not acceptable.

          Show
          Petr Škoda added a comment - Hmm, translators may decide to name the user fields in whatever way they want. In my computer it is me who decides what is the appropriate order. I would expect Moodle to allow me to configure it with some reasonable defaults based on my language or global admin settings. 1/ add some more general name fields - display name, middle, nickname, name3, name4 or whatever else (all of these would have to be included in all user related query results - we already have API for required avatar fields, we could use that) 2/ let translators decide how these fields are called in each language (admins may override it) 3/ let translators decide what is the expected default name format in each language (new string) 4/ let administrators decide what is the default and what are other options how to show names 5/ let users select own name format preference if they do not like default 6/ collation support in databases is an inconsistent mess, we need to add support in DML layer first, for now there is only one global collation Performance is the main problem here, because names are printed very often, it should imo go into the user table directly, text fields are definitely not acceptable.
          Hide
          Michael de Raadt added a comment -

          Hi, Eloy.

          The current default is to use the format specified in the language pack. That will remain the default unless it is overridden. It will be possible for different language packs to use the new fields, keeping in mind the the fields may have no information.

          Show
          Michael de Raadt added a comment - Hi, Eloy. The current default is to use the format specified in the language pack. That will remain the default unless it is overridden. It will be possible for different language packs to use the new fields, keeping in mind the the fields may have no information.
          Hide
          Eloy Lafuente (stronk7) added a comment - - edited

          Don et all,

          I'm not saying about not to implement this.

          I'm just saying, make it opt-in, defined by lang and override-able by admin. Just that.

          Finally, pay some attention to Petr's wise words above, it will be super-cool to have all those name schemas supported.... but right now we are not able to provide properly collated sorting from database, so this is only going to work for display but not for listings (in an efficient way). And IMO the main point about this is to be used for (better) listings. Display can be achieved with just 1 field (wholename).

          Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - - edited Don et all, I'm not saying about not to implement this. I'm just saying, make it opt-in, defined by lang and override-able by admin. Just that. Finally, pay some attention to Petr's wise words above, it will be super-cool to have all those name schemas supported.... but right now we are not able to provide properly collated sorting from database, so this is only going to work for display but not for listings (in an efficient way). And IMO the main point about this is to be used for (better) listings. Display can be achieved with just 1 field (wholename). Ciao
          Hide
          Martin Dougiamas added a comment -

          Note above that sorting and collation was removed from the scope of this.

          This is just about alternate name display and I think most of what Petr mentioned has already been covered (implemented).

          Show
          Martin Dougiamas added a comment - Note above that sorting and collation was removed from the scope of this. This is just about alternate name display and I think most of what Petr mentioned has already been covered (implemented).
          Hide
          Petr Škoda added a comment -

          The linked patch would cause major performance problems, solution would require changes all over the place when fetching user data.

          At present we have perf problems in filters caused by tree based settings, this feels like the same problem. In any case course level name formats sound wrong because we base everything on contexts now.

          Display name link makes me worried, again new perf problems there - we did not solve linking of avatars properly yet.

          I personally do not like the proposed new field names much, it should be imho something more language neutral at the database level. My vote is: middlename, aliasname, altname1 and altname2.

          Show
          Petr Škoda added a comment - The linked patch would cause major performance problems, solution would require changes all over the place when fetching user data. At present we have perf problems in filters caused by tree based settings, this feels like the same problem. In any case course level name formats sound wrong because we base everything on contexts now. Display name link makes me worried, again new perf problems there - we did not solve linking of avatars properly yet. I personally do not like the proposed new field names much, it should be imho something more language neutral at the database level. My vote is: middlename, aliasname, altname1 and altname2.
          Hide
          Martin Dougiamas added a comment -

          Petr please be specific about the performance problems you see, and help this issue proceed rather than just stopping it.

          I don't think the database field names are much of an issue.

          Show
          Martin Dougiamas added a comment - Petr please be specific about the performance problems you see, and help this issue proceed rather than just stopping it. I don't think the database field names are much of an issue.
          Hide
          Petr Škoda added a comment - - edited

          Potential performance problems:
          1/ all user info must be fetched in original queries, we can not fetch it later in renderers - hardcoding it in queries is not very flexible
          2/ adding course/module level settings to user display is a problem similar to filters, luckily usernames are not displayed in navigation; in any case caching still has some costs

          These are the showstoppers for me in the current patch:
          a/ please no more course/cm level only settings, we have context tree for this since 2.0 - the first bug report would be for block settings
          b/ this has to define which fields are required, there is already a bug for languages with one name only
          c/ missing $user->displayname which would allow free style user name entry - many other systems have it, it helps with sorting too, it is completely language neutral; this alone could probably resolve 80% of problems; this could be also generated automatically from first+last+other names
          d/ I do not like the proposed new fields - see above, there are too many
          e/ PARAM_TEXT is not a general text cleaning type - multilang is not allowed in display name format, is it?
          f/ some commit messages do not include MDL, others waste space with unnecessary " usability / language : " prefix, invalid structure of git commit messages in general
          g/ we did not even finish adding of context to format_(text|string) and we have another incomplete API change, I believe that fullname() should be fully deprecated if we decide this is the way; why is it even implemented in moodlelib as functions? why not only in new renderer?
          h/ if the resulting displayname is empty it should fall back to something, we do not want empty ghost names, right?

          Show
          Petr Škoda added a comment - - edited Potential performance problems: 1/ all user info must be fetched in original queries, we can not fetch it later in renderers - hardcoding it in queries is not very flexible 2/ adding course/module level settings to user display is a problem similar to filters, luckily usernames are not displayed in navigation; in any case caching still has some costs These are the showstoppers for me in the current patch: a/ please no more course/cm level only settings, we have context tree for this since 2.0 - the first bug report would be for block settings b/ this has to define which fields are required, there is already a bug for languages with one name only c/ missing $user->displayname which would allow free style user name entry - many other systems have it, it helps with sorting too, it is completely language neutral; this alone could probably resolve 80% of problems; this could be also generated automatically from first+last+other names d/ I do not like the proposed new fields - see above, there are too many e/ PARAM_TEXT is not a general text cleaning type - multilang is not allowed in display name format, is it? f/ some commit messages do not include MDL, others waste space with unnecessary " usability / language : " prefix, invalid structure of git commit messages in general g/ we did not even finish adding of context to format_(text|string) and we have another incomplete API change, I believe that fullname() should be fully deprecated if we decide this is the way; why is it even implemented in moodlelib as functions? why not only in new renderer? h/ if the resulting displayname is empty it should fall back to something, we do not want empty ghost names, right?
          Hide
          Petr Škoda added a comment -

          oh! let's not forget user upload and auth systems, we must map external things to our internal storage. Most systems use following fields:

          • firstname - aka given name
          • lastname - aka family name
          • middlenames - there can be one or more separated by space, or just initials
          • displayname - the full name of user
          • aliasname

          I would like to summarise problems with non-latin languages:

          ==Romanisation==

          Various transcriptions of alphabets, the result is usually ASCII. examples: Russian -> English, Chinese -> English, Japanese -> English, Czech -> English, Greek -> English.

          The main purpose of this is to help people to read and write names of foreigners. Quite often this is the expected form of first+last names on sites with users that are speaking different languages. On community servers where everybody is supposed to understand English it is part of the etiquette to use ISO-8859-1 characters only - Cyrillic, Greek or Chinese alphabets would be very confusing for the majority of people there.

          In theory you might romanise to any language based on latin which is probably more similar to phonetic names in different languages. Examples: Russian -> Czech.

          ==Phonetic names==

          Some names are written in alphabets that do not have fixed phonetic forms, even native speakers need to know how to pronounce them because there might be multiple options. Examples: Japanese in Kanji -> Furigana see http://en.wikipedia.org/wiki/Japanese_name

          If I understand this correctly the real name is just one and the phonetic form is used on business cards or when meeting new people. This seems to match the intended uses of our custom profile fields, or you can type it in profile description. People that need to know how to read names may easily visit users profile.

          -------

          The way I understand this patch it tries to resolve mostly phonetic name display which can be easily done via custom profile fields, I believe there is little need to show phonetic names everywhere in Moodle, or sort by phonetic names, sorry. If really necessary it could be worked around by using phonetic forms in standard user->displayname field. So far I did not see any computer system that stores phonetic names of users separately, why do we need to be special here?

          Show
          Petr Škoda added a comment - oh! let's not forget user upload and auth systems, we must map external things to our internal storage. Most systems use following fields: firstname - aka given name lastname - aka family name middlenames - there can be one or more separated by space, or just initials displayname - the full name of user aliasname I would like to summarise problems with non-latin languages: ==Romanisation== Various transcriptions of alphabets, the result is usually ASCII. examples: Russian -> English, Chinese -> English, Japanese -> English, Czech -> English, Greek -> English. The main purpose of this is to help people to read and write names of foreigners. Quite often this is the expected form of first+last names on sites with users that are speaking different languages. On community servers where everybody is supposed to understand English it is part of the etiquette to use ISO-8859-1 characters only - Cyrillic, Greek or Chinese alphabets would be very confusing for the majority of people there. In theory you might romanise to any language based on latin which is probably more similar to phonetic names in different languages. Examples: Russian -> Czech. ==Phonetic names== Some names are written in alphabets that do not have fixed phonetic forms, even native speakers need to know how to pronounce them because there might be multiple options. Examples: Japanese in Kanji -> Furigana see http://en.wikipedia.org/wiki/Japanese_name If I understand this correctly the real name is just one and the phonetic form is used on business cards or when meeting new people. This seems to match the intended uses of our custom profile fields, or you can type it in profile description. People that need to know how to read names may easily visit users profile. ------- The way I understand this patch it tries to resolve mostly phonetic name display which can be easily done via custom profile fields, I believe there is little need to show phonetic names everywhere in Moodle, or sort by phonetic names, sorry. If really necessary it could be worked around by using phonetic forms in standard user->displayname field. So far I did not see any computer system that stores phonetic names of users separately, why do we need to be special here?
          Hide
          Michael de Raadt added a comment -

          The issue fetching additional name information was known from the outset. It will come at a cost, particularly in lists of names. We are working to make this more efficient and easier for developers to provide this information, but we will always have to allow for (plugin and core) code that does not provide the additional name fields. So for sites that choose to use additional name fields when displaying names, there will be a possible performance hit.

          The issue of mixing contexts is a hairy one. Where a user's name appears on a page, the parts of the page may be generated using different contexts (like navigation, blocks and the header). Adrian was looking at this, but I'm not sure where it was up to.

          I think the people who reported this issue and are watching it have a better understanding of the need for names in various languages. I don't think we should do half a job on this.

          Show
          Michael de Raadt added a comment - The issue fetching additional name information was known from the outset. It will come at a cost, particularly in lists of names. We are working to make this more efficient and easier for developers to provide this information, but we will always have to allow for (plugin and core) code that does not provide the additional name fields. So for sites that choose to use additional name fields when displaying names, there will be a possible performance hit. The issue of mixing contexts is a hairy one. Where a user's name appears on a page, the parts of the page may be generated using different contexts (like navigation, blocks and the header). Adrian was looking at this, but I'm not sure where it was up to. I think the people who reported this issue and are watching it have a better understanding of the need for names in various languages. I don't think we should do half a job on this.
          Hide
          Petr Škoda added a comment -

          My -5 for the patch for the reasons above.

          I was thinking about the phonetic names, I think it should be possible to allow extra names via user profiles fields, it would be slower but that would be imho acceptable cost because only sites that want to use custom names would be affected. The name format could include standard {$a->firstname, {$a->lastname, {$a->middlenames} and {$a->aliasname} and any "profile_" prefixed names such as {$a->profile_firstnamephonetic} and {$a->profile_lastnamephonetic}. This way you could even define multiple phonetic systems as requested in the description.

          Show
          Petr Škoda added a comment - My -5 for the patch for the reasons above. I was thinking about the phonetic names, I think it should be possible to allow extra names via user profiles fields, it would be slower but that would be imho acceptable cost because only sites that want to use custom names would be affected. The name format could include standard {$a->firstname, {$a->lastname, {$a->middlenames} and {$a->aliasname} and any "profile_" prefixed names such as {$a->profile_firstnamephonetic} and {$a->profile_lastnamephonetic}. This way you could even define multiple phonetic systems as requested in the description.
          Hide
          Rajesh Taneja added a comment -

          Hello Petr,

          We have changed the design to make it more efficient.
          Advantages of having this in:

          1. Extra user fields are part of user profile field set and if set, will check if user has filled them.
          2. User name display can be controlled at course/ Module level, giving more flexibility.
          3. As format is coming from course/Module, it will not go though old fullname function and will improve performance for current sites.
          4. Anonymous username problem can be sorted with this patch.
          5. Will only affect site who enable alternate user name fields (for non-optimised calls to display_name())
          Show
          Rajesh Taneja added a comment - Hello Petr, We have changed the design to make it more efficient. Advantages of having this in: Extra user fields are part of user profile field set and if set, will check if user has filled them. User name display can be controlled at course/ Module level, giving more flexibility. As format is coming from course/Module, it will not go though old fullname function and will improve performance for current sites. Anonymous username problem can be sorted with this patch. Will only affect site who enable alternate user name fields (for non-optimised calls to display_name())
          Hide
          Martin Dougiamas added a comment -

          Yes, the main reasons we rejected user profile fields way back in this issue was because:

          • doing it that way is much slower and more complicated
          • the use cases in this bug already cover 99% of the possible variations people need
          • the cost outweighs the benefit
          Show
          Martin Dougiamas added a comment - Yes, the main reasons we rejected user profile fields way back in this issue was because: doing it that way is much slower and more complicated the use cases in this bug already cover 99% of the possible variations people need the cost outweighs the benefit
          Hide
          Petr Škoda added a comment -

          ok, I give up

          Show
          Petr Škoda added a comment - ok, I give up
          Hide
          Helen Foster added a comment -

          I'm echoing Eloy's comment:

          I hope all this is going to be something only available if the admin opts-in.

          And it won't be shown/used/listed at all by default, so people will continue using the current & simpler way (first+second and done), correct?

          I don't think 'User name display can be controlled at course/ Module level, giving more flexibility' can be considered an advantage unless it is only displayed for sites that want it.

          Show
          Helen Foster added a comment - I'm echoing Eloy's comment: I hope all this is going to be something only available if the admin opts-in. And it won't be shown/used/listed at all by default, so people will continue using the current & simpler way (first+second and done), correct? I don't think 'User name display can be controlled at course/ Module level, giving more flexibility' can be considered an advantage unless it is only displayed for sites that want it.
          Hide
          Rajesh Taneja added a comment -

          Thanks Helen,

          Alternate User fields will only be displayed/checked if they are enabled at site level by admin.
          Also, only those fields will be displayed which admin wants.

          I am not sure if we took care of show/hide option at course/module level, but sure we can take care of it.

          Show
          Rajesh Taneja added a comment - Thanks Helen, Alternate User fields will only be displayed/checked if they are enabled at site level by admin. Also, only those fields will be displayed which admin wants. I am not sure if we took care of show/hide option at course/module level, but sure we can take care of it.
          Hide
          Adrian Greeve added a comment -

          This improvement will rely of a lot of changes through out moodle. If the code goes through integration, I'll be happy to create additional issues to update from the fullname function.
          Attached is a grep dump of where the fullname function is used (400+ lines).

          Show
          Adrian Greeve added a comment - This improvement will rely of a lot of changes through out moodle. If the code goes through integration, I'll be happy to create additional issues to update from the fullname function. Attached is a grep dump of where the fullname function is used (400+ lines).
          Hide
          Damyon Wiese added a comment -

          Hi Adrian,

          This is looking good - I took a look at the branch and here is a braindump of big and little things I spotted:

          • The admin settings are in the wrong section (IMO) they should be in the "User policies" section not the "Site policies" section.
          • The name of the function "displayname" is too generic - it could be the displayname of a course.
          • You are not calling displayname correctly - the second parameter is the format but you have:

          e.g.
          $OUTPUT->displayname($user, $sitecontext);
          in lots of places

          • You need to pass size as an array:

          $mform->addElement('text', 'displaynameformat', get_string('displaynameformatcourse'), 'size = 50');

          • PARAM_TEXT is not correct for displaynameformat (it's not a multilang field)
          • Need get_string for 'Allow link to the profile page'
          • displaynameformat missing from backup/restore
          • new user fields missing from backup/restore
          • new user fields missing from webservices
          • typo in $addtionalfieldempty
          • Not sure if it has been raised - but why would you not want to link the username to the profile?
          • commented code in lib/navigationlib.php
          • new displayname function is not checking 'moodle/site:viewfullnames'
          • difference between username and user_name is a bit hard to read in code
          • ID number shows up twice (need to edit both)
          • There are lots of places in the code still calling "fullname()" (grep tells me 434 uses) I think it is important that this is applied consistently in core. (Not looking back at the long list of comments right now - but I am not sure why we haven't made this backwards compatible with the fullname function).
          • You have changed the tablelib class - but not the get_extra_user_fields - IMO both are required.
          • We still have $CFG->showuseridentity - IMO it doesn't make sense to have both

          Cheers - Damyon

          Show
          Damyon Wiese added a comment - Hi Adrian, This is looking good - I took a look at the branch and here is a braindump of big and little things I spotted: The admin settings are in the wrong section (IMO) they should be in the "User policies" section not the "Site policies" section. The name of the function "displayname" is too generic - it could be the displayname of a course. You are not calling displayname correctly - the second parameter is the format but you have: e.g. $OUTPUT->displayname($user, $sitecontext); in lots of places You need to pass size as an array: $mform->addElement('text', 'displaynameformat', get_string('displaynameformatcourse'), 'size = 50'); PARAM_TEXT is not correct for displaynameformat (it's not a multilang field) Need get_string for 'Allow link to the profile page' displaynameformat missing from backup/restore new user fields missing from backup/restore new user fields missing from webservices typo in $addtionalfieldempty Not sure if it has been raised - but why would you not want to link the username to the profile? commented code in lib/navigationlib.php new displayname function is not checking 'moodle/site:viewfullnames' difference between username and user_name is a bit hard to read in code ID number shows up twice (need to edit both) There are lots of places in the code still calling "fullname()" (grep tells me 434 uses) I think it is important that this is applied consistently in core. (Not looking back at the long list of comments right now - but I am not sure why we haven't made this backwards compatible with the fullname function). You have changed the tablelib class - but not the get_extra_user_fields - IMO both are required. We still have $CFG->showuseridentity - IMO it doesn't make sense to have both Cheers - Damyon
          Hide
          Martin Dougiamas added a comment -

          This has got really complex. Adrian, Raj and Damyon have been spending a lot of time on this but it seems to be spawning new problems.

          There was talk today in HQ about dropping aliasname and any context support, because this would simplify everything a lot.

          The Anonymous feature was my particular wheelbarrow because roleplaying and anonymity were long-standing requests, will touch similar areas, and didn't look too complicated to add, however, even if we did it this way it looks like the interface would still be quite imperfect (one alias per user per site, no teacher control).

          A better interface might be some page at course level where the teacher (and optionally students) can decide their aliases and assign people to them. With this method we will still need to modify activities to support this feature but we could probably call a different function than fullname() to print the alias. So it would be quite separate and manageable as a separate project.

          So on reflection I'm OK to drop this requirement and go back to just simple site settings that modify how fullname() works for the whole site, no contexts. Basically I see that as:

          • extending $CFG->fullnamedisplay to support more user field tokens and intervening filler text (such as brackets). eg "fullname middlename lastname (firstnamephon lastnamephon)"
          • making it handle missing data intelligently (eg removing empty brackets or excess whitepace from the above example)
          • checking calls to fullname() to make sure they are efficient and preload extra fields as much as possible

          I feel sorry for all the work Adrian especially has done but better for part to land than none ...

          Show
          Martin Dougiamas added a comment - This has got really complex. Adrian, Raj and Damyon have been spending a lot of time on this but it seems to be spawning new problems. There was talk today in HQ about dropping aliasname and any context support, because this would simplify everything a lot. The Anonymous feature was my particular wheelbarrow because roleplaying and anonymity were long-standing requests, will touch similar areas, and didn't look too complicated to add, however, even if we did it this way it looks like the interface would still be quite imperfect (one alias per user per site, no teacher control). A better interface might be some page at course level where the teacher (and optionally students) can decide their aliases and assign people to them. With this method we will still need to modify activities to support this feature but we could probably call a different function than fullname() to print the alias. So it would be quite separate and manageable as a separate project. So on reflection I'm OK to drop this requirement and go back to just simple site settings that modify how fullname() works for the whole site, no contexts. Basically I see that as: extending $CFG->fullnamedisplay to support more user field tokens and intervening filler text (such as brackets). eg "fullname middlename lastname (firstnamephon lastnamephon)" making it handle missing data intelligently (eg removing empty brackets or excess whitepace from the above example) checking calls to fullname() to make sure they are efficient and preload extra fields as much as possible I feel sorry for all the work Adrian especially has done but better for part to land than none ...
          Hide
          Adrian Greeve added a comment -

          After a significant re-write I now have a patch that should satisfy the above requirements.

          Show
          Adrian Greeve added a comment - After a significant re-write I now have a patch that should satisfy the above requirements.
          Hide
          Rajesh Taneja added a comment -

          Hello Damyon,

          If it's ok with you, can I peer-review this?

          Show
          Rajesh Taneja added a comment - Hello Damyon, If it's ok with you, can I peer-review this?
          Hide
          Rajesh Taneja added a comment -

          Patch seems nice Adrian, few things which you might want to consider:

          1. get_additional_name_fields (#L15R3665), seems to be redundant, as it's just using hard-coded values. Seems, unnecessary processing. IMHO, it should either return all name related fields or should be removed.
          2. #L15R3679 you can use implode to join array. No need of substring.
          3. Rather then passing two parameters to get_additional_name_fields, can't it be achieved with one ($joinwith)?
          4. #L6R93 In this case you are using get_additional_name_fields and later (#L6R108) you have hard-coded these fields. It might be nice to have some consistency with use of alternate field names.
          5. Similar hard-coding of alternate fields has been done in other places.
          6. #L15R3632 can be cache this, as this is site based and looping though displayname for every displayed name seems redundant and process hungry.
          7. Not sure if string clean-up is required (#L15R3644), as CFG is set by admin, do we bother about cleaning this ?
          8. Typo in debugging message #L15R3623
          9. $mainuserfields (#L29R384), should include username. email is not required as that is returned by user_picture::fields
          10. $mainuserfields doesn't include alternate name fields.
          Show
          Rajesh Taneja added a comment - Patch seems nice Adrian, few things which you might want to consider: get_additional_name_fields (#L15R3665), seems to be redundant, as it's just using hard-coded values. Seems, unnecessary processing. IMHO, it should either return all name related fields or should be removed. #L15R3679 you can use implode to join array. No need of substring. Rather then passing two parameters to get_additional_name_fields, can't it be achieved with one ($joinwith)? #L6R93 In this case you are using get_additional_name_fields and later (#L6R108) you have hard-coded these fields. It might be nice to have some consistency with use of alternate field names. Similar hard-coding of alternate fields has been done in other places. #L15R3632 can be cache this, as this is site based and looping though displayname for every displayed name seems redundant and process hungry. Not sure if string clean-up is required (#L15R3644), as CFG is set by admin, do we bother about cleaning this ? Typo in debugging message #L15R3623 $mainuserfields (#L29R384), should include username. email is not required as that is returned by user_picture::fields $mainuserfields doesn't include alternate name fields.
          Hide
          Adrian Greeve added a comment -

          Hello Raj,

          Thanks for the review.
          I have considered the things that you have pointed out and:

          1. I created this function as a central location for the additional name fields. If in the future we wanted to add more, all we would have to do is add them to the array in that function. I definitely don't think that it should be removed all together. Hardcoding the additional name fields everywhere doesn't make sense at all. I can see that having all name fields in the same location would be useful. I'll add firstname and lastname to the array.
          2. I'm not sure by what you mean by joining the array on line 3679. This line is a string and I'm removing the last two characters so that there is no trailing comma.
          3. Get additional name fields has a dual purpose of returning an array and providing an sql snippet. The first parameter is set to "true" if you want the sql snippet and then you can add an additional alias if required. Removing one of the parameters would mean that it couldn't be used in both situations.
          4. You're right this can be improved and has now been fixed.
          5. Same as above.
          6. I don't think that we can cache this. We have to loop through each name and replace it with the users details for each user because each user has a unique name. Perhaps I'm missing what you mean.
          7. This isn't a screen clean up for the admin, this is so we don't display something like "Adrian () Greeve" when the middle name has not been entered. I've added a comment with a similar explanation.
          8. This has been fixed.
          9. This has now been included.
          10. This does actually include the alternate name fields as user_picture::fields() includes alternate name fields.
          Show
          Adrian Greeve added a comment - Hello Raj, Thanks for the review. I have considered the things that you have pointed out and: I created this function as a central location for the additional name fields. If in the future we wanted to add more, all we would have to do is add them to the array in that function. I definitely don't think that it should be removed all together. Hardcoding the additional name fields everywhere doesn't make sense at all. I can see that having all name fields in the same location would be useful. I'll add firstname and lastname to the array. I'm not sure by what you mean by joining the array on line 3679. This line is a string and I'm removing the last two characters so that there is no trailing comma. Get additional name fields has a dual purpose of returning an array and providing an sql snippet. The first parameter is set to "true" if you want the sql snippet and then you can add an additional alias if required. Removing one of the parameters would mean that it couldn't be used in both situations. You're right this can be improved and has now been fixed. Same as above. I don't think that we can cache this. We have to loop through each name and replace it with the users details for each user because each user has a unique name. Perhaps I'm missing what you mean. This isn't a screen clean up for the admin, this is so we don't display something like "Adrian () Greeve" when the middle name has not been entered. I've added a comment with a similar explanation. This has been fixed. This has now been included. This does actually include the alternate name fields as user_picture::fields() includes alternate name fields.
          Hide
          Adrian Greeve added a comment -

          I've made some further alterations to the code and done a comparison of a few of the pages around moodle for DB calls.

          I have a plan to include caching (MDL-38606) to try and take away more of the processing that is currently done in the fullname function.

          comparison

          Database calls

          file before after
          home page 60 60
          admin/user.php 45 45
          admin/user/user_bulk.php 44 44
          course/view.php 47 47
          enrol/users.php 109 109
          grade/report/grader/index.php 66 66
          user/index.php 57 57
          mod/assign/view.php 186 186
          mod/choice/view.php 69 69

          As you can see, there is no impact on the database calls made on each of these pages. This is because the sql statements have been updated to add the new fields where necessary. The fullname function itself does not make any database calls.

          Plugin developers will need to alter their own sql statements to include these new files. Developers that have user the user_picture::fields function to create their user statement will not have to make any improvements at all.

          Rajesh, if you could please have another look at the code and tell me what you think.

          Show
          Adrian Greeve added a comment - I've made some further alterations to the code and done a comparison of a few of the pages around moodle for DB calls. I have a plan to include caching ( MDL-38606 ) to try and take away more of the processing that is currently done in the fullname function. comparison Database calls file before after home page 60 60 admin/user.php 45 45 admin/user/user_bulk.php 44 44 course/view.php 47 47 enrol/users.php 109 109 grade/report/grader/index.php 66 66 user/index.php 57 57 mod/assign/view.php 186 186 mod/choice/view.php 69 69 As you can see, there is no impact on the database calls made on each of these pages. This is because the sql statements have been updated to add the new fields where necessary. The fullname function itself does not make any database calls. Plugin developers will need to alter their own sql statements to include these new files. Developers that have user the user_picture::fields function to create their user statement will not have to make any improvements at all. Rajesh, if you could please have another look at the code and tell me what you think.
          Hide
          Rajesh Taneja added a comment -

          Thanks for fixing this Adrian,

          Patch looks great, just a minor issue.
          As per description of fullnamedisplayprivate, it should only be visible to users with moodle/site:viewuseridentity capability, but as per #L15R3596, it is only valid if $override is true. Also, should we consider having forcealternatename, similar to forcefirstname/forcelastname CFG? (probably another issue)

          Also, please update testing instructions replacing

          • fullnameformat with fullnamedisplay
          • fullnameformatprivate with fullnamedisplayprivate
          Show
          Rajesh Taneja added a comment - Thanks for fixing this Adrian, Patch looks great, just a minor issue. As per description of fullnamedisplayprivate, it should only be visible to users with moodle/site:viewuseridentity capability, but as per #L15R3596, it is only valid if $override is true. Also, should we consider having forcealternatename, similar to forcefirstname/forcelastname CFG? (probably another issue) Also, please update testing instructions replacing fullnameformat with fullnamedisplay fullnameformatprivate with fullnamedisplayprivate
          Hide
          Adrian Greeve added a comment -

          Thank Rajesh for having another look for me.
          I've removed the second fullnamedisplayprivate setting and updated the testing instructions. I created another issue to look at putting this setting back as I think that it would be very useful.

          Sending to integration review.

          Show
          Adrian Greeve added a comment - Thank Rajesh for having another look for me. I've removed the second fullnamedisplayprivate setting and updated the testing instructions. I created another issue to look at putting this setting back as I think that it would be very useful. Sending to integration review.
          Hide
          Sam Hemelryk added a comment -

          First up this change appears to be causing a few unit tests to fail.

          moodlelib.php - fullname
          The regex added for tidying up misc characters sticks out to me.
          After mulling it over (hot cross buns + coffee) I'm just not sure that this should be there.
          To summarise the code there we are trying to clean up strings removing characters associated with the display of field is found to be empty.
          It deals with one specific situation, where a pair of characters are encompasing a field.
          There are a couple of things I don't like about this:

          • The regular expression doesn't match pairs, it matches a start character and an end character that have no relation to the other. Another words the characters '-(' may be stripped if they appear next to each others.
          • There is separation of what was a field and what was not. Even if the template only contained fields we'd still execute the regex. User content (aka field content) will still be searched and acted upon.
          • We're dealing with just one possible case of unwanted characters. Other cases such as placing hyphens or commas between fields is not dealt with. While this could be considered extending functionality I think this helps to lead onto what I think may be a better solution to this issue.

          As an example of where this would fall over consider the template "last (first, middle)".

          So what should be done. Well first - do we want to implement this magic stripping of redunant characters?
          I could imagine people wanting it to be done so perhaps yes we want it but I'd like to be sure the question has been asked.
          Truthfully I don't feel we should be doing this. I think we should be applying the template and leaving it at that. If the template envolves braces, hyphens or any character other character and they end up being redundant we don't try to fix it.
          Any other solution is going to lead to either involve unmet cases or an element of guessing.

          So really there are three options I see:

          1. Don't do it.
          2. Change it so that it only removes "template" content. This was it will only remove matching "paired" characters that appear redundant in the template.
          3. Leave it as it is, breakable magic. (I've made lots of noise for no reason )

          If options 2 or 3 are selected I think we also need many more unit tests to cover what is done.

          other things

          1. outputcompontents.php user_picture There are a couple of things here, so sub numbers it is:

          • Having to array_merge in get_all_name_fields when using self::$fields is really unfortunate. $fields is a static property that should contain all required fields. Not only do you add the calls to array_merge but in the process you have likely caused a backwards compatability issue for anyone who may have overridden the user_picture class. IMHO $fields needs to include ALL required fields.
          • You've broken user_picture::fields() if it is called without a tableprefix, extrafields, or idalias. This would be fixed by above and while it would be unlikely perhaps someone out there is calling it in this way, best no break it aye.
          • unalias will be fixed by the above as well.

          2. moodlelib.php get_enabled_additional_names
          More thorough unit tests here would reveal a bug:

          $valuearray = array('second', 'firsthalf', 'first');
          $formatstring = 'firsthalf first second';
          $expectedarray = array('0' => 'firsthalf', '10' => 'firsthalf', '16' => 'second');
          $this->assertEquals($expectedarray, order_in_string($valuearray, $formatstring));
          

          You already commented about this bug in the code

          3. moodlelib.php get_all_name_fields This could be named better I feel. We seem to have several _user_field functions already. A more consistent name may be get_all_user_name_fields.

          4. moodlelib.php get_all_name_fields Arg 1 $fields could certainly be named better. What about "$returnsql" or something that better describes what that argument does.

          5. moodlelib.php get_enabled_additional_names Naming again although looking at this function I would really question whether it is needed. Its only used in user/editlib.php and the uses there look like they could easily be factored out.

          After looking at get_all_name_fields and get_enabled_additional_names did you have in mind some plan to allow those functions to be overridden or something? There seems to be a bit of code around working with those two functions and any differences. However it appears to me that if they can't be overridden that we would always know the differences?

          Anyway, at this point I've found several reasons to send this back without getting to far into the patch and I've spent most of the morning on it.
          I'm going to reopen this now and proceed with integration reviews. Once I've got more time I'll come back to it and continue my review.

          Thanks for the work and sorry for the delay,
          Sam

          Show
          Sam Hemelryk added a comment - First up this change appears to be causing a few unit tests to fail. moodlelib.php - fullname The regex added for tidying up misc characters sticks out to me. After mulling it over (hot cross buns + coffee) I'm just not sure that this should be there. To summarise the code there we are trying to clean up strings removing characters associated with the display of field is found to be empty. It deals with one specific situation, where a pair of characters are encompasing a field. There are a couple of things I don't like about this: The regular expression doesn't match pairs, it matches a start character and an end character that have no relation to the other. Another words the characters '-(' may be stripped if they appear next to each others. There is separation of what was a field and what was not. Even if the template only contained fields we'd still execute the regex. User content (aka field content) will still be searched and acted upon. We're dealing with just one possible case of unwanted characters. Other cases such as placing hyphens or commas between fields is not dealt with. While this could be considered extending functionality I think this helps to lead onto what I think may be a better solution to this issue. As an example of where this would fall over consider the template "last (first, middle)". So what should be done. Well first - do we want to implement this magic stripping of redunant characters? I could imagine people wanting it to be done so perhaps yes we want it but I'd like to be sure the question has been asked. Truthfully I don't feel we should be doing this. I think we should be applying the template and leaving it at that. If the template envolves braces, hyphens or any character other character and they end up being redundant we don't try to fix it. Any other solution is going to lead to either involve unmet cases or an element of guessing. So really there are three options I see: Don't do it. Change it so that it only removes "template" content. This was it will only remove matching "paired" characters that appear redundant in the template. Leave it as it is, breakable magic. (I've made lots of noise for no reason ) If options 2 or 3 are selected I think we also need many more unit tests to cover what is done. other things 1. outputcompontents.php user_picture There are a couple of things here, so sub numbers it is: Having to array_merge in get_all_name_fields when using self::$fields is really unfortunate. $fields is a static property that should contain all required fields. Not only do you add the calls to array_merge but in the process you have likely caused a backwards compatability issue for anyone who may have overridden the user_picture class. IMHO $fields needs to include ALL required fields. You've broken user_picture::fields() if it is called without a tableprefix, extrafields, or idalias. This would be fixed by above and while it would be unlikely perhaps someone out there is calling it in this way, best no break it aye. unalias will be fixed by the above as well. 2. moodlelib.php get_enabled_additional_names More thorough unit tests here would reveal a bug: $valuearray = array('second', 'firsthalf', 'first'); $formatstring = 'firsthalf first second'; $expectedarray = array('0' => 'firsthalf', '10' => 'firsthalf', '16' => 'second'); $ this ->assertEquals($expectedarray, order_in_string($valuearray, $formatstring)); You already commented about this bug in the code 3. moodlelib.php get_all_name_fields This could be named better I feel. We seem to have several _user_field functions already. A more consistent name may be get_all_user_name_fields. 4. moodlelib.php get_all_name_fields Arg 1 $fields could certainly be named better. What about "$returnsql" or something that better describes what that argument does. 5. moodlelib.php get_enabled_additional_names Naming again although looking at this function I would really question whether it is needed. Its only used in user/editlib.php and the uses there look like they could easily be factored out. After looking at get_all_name_fields and get_enabled_additional_names did you have in mind some plan to allow those functions to be overridden or something? There seems to be a bit of code around working with those two functions and any differences. However it appears to me that if they can't be overridden that we would always know the differences? Anyway, at this point I've found several reasons to send this back without getting to far into the patch and I've spent most of the morning on it. I'm going to reopen this now and proceed with integration reviews. Once I've got more time I'll come back to it and continue my review. Thanks for the work and sorry for the delay, Sam
          Hide
          CiBoT added a comment -

          Moving this reopened issue out from current integration. Please, re-submit it for integration once ready.

          Show
          CiBoT added a comment - Moving this reopened issue out from current integration. Please, re-submit it for integration once ready.
          Hide
          Michael de Raadt added a comment -

          Moving these out of the old sprint which used the fixVersion field.

          Show
          Michael de Raadt added a comment - Moving these out of the old sprint which used the fixVersion field.
          Hide
          Adrian Greeve added a comment -

          Thanks Sam for the review.

          I've gone ahead and made changes as per your comments.

          1. I've fixed up the places that were causing unit test problems and run all unit tests to make sure that I haven't missed something.
          2. The addition of the regex line is per the instructions of Martin as commented here https://tracker.moodle.org/browse/MDL-31776?focusedCommentId=217808&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-217808
            We had a quick discussion about this. What I have done is greatly simplified the expression. I had some characters in there that I don't think we really needed to check. I'm only looking at different types of brackets now. I've added about nine additional tests to show how this works.
          3. I added the additional fields to the user_picture class. This has solved the other issues that you mentioned about that.
          4. I fixed up the mistake in the code for get_enabled_additional_names and added a couple more tests for this.
          5. I re-named get_all_name_fields to get_all_user_name_fields.
          6. The first argument for get_all_user_name_fields is now called $returnsql.
          7. get_enabled_additional_names has been removed and the logic for this is now only located in user/editlib.php

          If you or some other integrator would be so kind as to please have another look at this issue, I would be most grateful.

          Thanks.

          Show
          Adrian Greeve added a comment - Thanks Sam for the review. I've gone ahead and made changes as per your comments. I've fixed up the places that were causing unit test problems and run all unit tests to make sure that I haven't missed something. The addition of the regex line is per the instructions of Martin as commented here https://tracker.moodle.org/browse/MDL-31776?focusedCommentId=217808&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-217808 We had a quick discussion about this. What I have done is greatly simplified the expression. I had some characters in there that I don't think we really needed to check. I'm only looking at different types of brackets now. I've added about nine additional tests to show how this works. I added the additional fields to the user_picture class. This has solved the other issues that you mentioned about that. I fixed up the mistake in the code for get_enabled_additional_names and added a couple more tests for this. I re-named get_all_name_fields to get_all_user_name_fields. The first argument for get_all_user_name_fields is now called $returnsql. get_enabled_additional_names has been removed and the logic for this is now only located in user/editlib.php If you or some other integrator would be so kind as to please have another look at this issue, I would be most grateful. Thanks.
          Hide
          Damyon Wiese added a comment -

          Hi Adrian,

          Keep going you are almost there (but not quite).

          Here are the issues I spotted in this patch:

          Editing a user profile gives debugging warnings:

          line 3613 of /lib/moodlelib.php: call to debugging()
          line 3130 of /repository/lib.php: call to fullname()
          line 319 of /lib/form/editor.php: call to initialise_filepicker()

          Saving a user profile gives "undefined property"

          Undefined property: stdClass::$fullnamedisplayprivate in /home/damyonw/Documents/Moodle/integration/master/moodle/user/editlib.php on line 132 Call Stack: 0.0008 833248

          The issue with templates needs to be solved IMO. Sam's example is the first one I came up with too - ie:

          firstname, lastname (firstnamephonetic, lastnamephonetic)

          Which gives e.g. Sandy, Raynor (, )

          for most users without phonetic names.

          Maybe - when a field is empty - remove all punctuation surrounding the field (until you hit whitespace)

          In lib/badgeslib.php - can you also remove this line:

          $userfrom->firstname = !empty($CFG->badges_defaultissuername) ? $CFG->badges_defaultissuername : $admin->firstname;

          Changes in get_users_listing (lib/datalib.php) seems to be missing a comma in the SQL.

          In flexible table you have added support for sorting by your new name fields - but the columns in the DB do not have indexes (bad).

          These seem pretty small (except maybe the regex one) - ping me when you update the branch and I'll take another look.

          Thanks!

          Show
          Damyon Wiese added a comment - Hi Adrian, Keep going you are almost there (but not quite). Here are the issues I spotted in this patch: Editing a user profile gives debugging warnings: line 3613 of /lib/moodlelib.php: call to debugging() line 3130 of /repository/lib.php: call to fullname() line 319 of /lib/form/editor.php: call to initialise_filepicker() Saving a user profile gives "undefined property" Undefined property: stdClass::$fullnamedisplayprivate in /home/damyonw/Documents/Moodle/integration/master/moodle/user/editlib.php on line 132 Call Stack: 0.0008 833248 The issue with templates needs to be solved IMO. Sam's example is the first one I came up with too - ie: firstname, lastname (firstnamephonetic, lastnamephonetic) Which gives e.g. Sandy, Raynor (, ) for most users without phonetic names. Maybe - when a field is empty - remove all punctuation surrounding the field (until you hit whitespace) In lib/badgeslib.php - can you also remove this line: $userfrom->firstname = !empty($CFG->badges_defaultissuername) ? $CFG->badges_defaultissuername : $admin->firstname; Changes in get_users_listing (lib/datalib.php) seems to be missing a comma in the SQL. In flexible table you have added support for sorting by your new name fields - but the columns in the DB do not have indexes (bad). These seem pretty small (except maybe the regex one) - ping me when you update the branch and I'll take another look. Thanks!
          Hide
          Adrian Greeve added a comment -

          Thanks Damyon,

          I've gone back through the code and I have sorted out the various problems.

          1. The debuggging warnings that you were receiving, I am pretty sure, are due to the fact that the code wasn't rebased since the last release and the update code didn't run on your machine. This will solve:
            • user profile debugging warnings
            • Saving a user profile displaying "undefined property"
            • Undefined property: stdClass::$fullnamedisplay etc. etc.
          2. I've re-written the regular expression to handle the example that you have given and increased the unit tests to handle this problem.
          3. Is there a specific reason that this line should be removed?
            The current line is :
            $userfrom->firstname = !empty($CFG->badges_defaultissuername) ? $CFG->badges_defaultissuername : $admin->firstname; 
            

            And you want me to change it to:

            $userfrom->$addname = !empty($CFG->badges_defaultissuername) ? '' : $admin->$addname;  
            

            I was a bit worried with this if statement not being the same as the others, so I decided to leave the logic as it was. I'm happy to remove it if it won't break anything.

          4. This SQL doesn't have a comma as $extrafields already has a comma at the start of that string.
          5. I've added indexes for the additional name fields.

          Thanks.

          Show
          Adrian Greeve added a comment - Thanks Damyon, I've gone back through the code and I have sorted out the various problems. The debuggging warnings that you were receiving, I am pretty sure, are due to the fact that the code wasn't rebased since the last release and the update code didn't run on your machine. This will solve: user profile debugging warnings Saving a user profile displaying "undefined property" Undefined property: stdClass::$fullnamedisplay etc. etc. I've re-written the regular expression to handle the example that you have given and increased the unit tests to handle this problem. Is there a specific reason that this line should be removed? The current line is : $userfrom->firstname = !empty($CFG->badges_defaultissuername) ? $CFG->badges_defaultissuername : $admin->firstname; And you want me to change it to: $userfrom->$addname = !empty($CFG->badges_defaultissuername) ? '' : $admin->$addname; I was a bit worried with this if statement not being the same as the others, so I decided to leave the logic as it was. I'm happy to remove it if it won't break anything. This SQL doesn't have a comma as $extrafields already has a comma at the start of that string. I've added indexes for the additional name fields. Thanks.
          Hide
          Damyon Wiese added a comment - - edited

          Just documenting our conversation on the regex. It is better to have a simple fast rule that works for 90% of cases that a specific complex rule that is hard to explain. So preferred algorithm is:

          step 1 - replace empty fields with "EMPTY"
          
          step 2 - run a regex for [:punct:]*EMPTY[:punct:]* and replace with ''
          
          step 3 - profit :)
          
          Show
          Damyon Wiese added a comment - - edited Just documenting our conversation on the regex. It is better to have a simple fast rule that works for 90% of cases that a specific complex rule that is hard to explain. So preferred algorithm is: step 1 - replace empty fields with "EMPTY" step 2 - run a regex for [:punct:]*EMPTY[:punct:]* and replace with '' step 3 - profit :)
          Hide
          Adrian Greeve added a comment -

          The regular expression has been vastly simplified. Thanks Damyon for showing me how.
          Unit tests have been updated and the installation problem fixed.

          Show
          Adrian Greeve added a comment - The regular expression has been vastly simplified. Thanks Damyon for showing me how. Unit tests have been updated and the installation problem fixed.
          Hide
          Damyon Wiese added a comment -

          Hurray!

          We made it. Thanks Adrian for working so hard on this issue. It has been integrated for master now.

          ありがとう

          Show
          Damyon Wiese added a comment - Hurray! We made it. Thanks Adrian for working so hard on this issue. It has been integrated for master now. ありがとう
          Hide
          Michael de Raadt added a comment -

          Well done, guys.

          Show
          Michael de Raadt added a comment - Well done, guys.
          Hide
          Damyon Wiese added a comment -

          We are seeing this failure in integration -

          moodlelib_testcase::test_fullname
          Failed asserting that two strings are equal.
          — Expected
          +++ Actual
          @@ @@
          -'Fletcher å¼ Scott スコット'
          +'Fletcher å¼ Scott スコット'

          Show
          Damyon Wiese added a comment - We are seeing this failure in integration - moodlelib_testcase::test_fullname Failed asserting that two strings are equal. — Expected +++ Actual @@ @@ -'Fletcher å¼ Scott スコット' +'Fletcher å¼ Scott スコット'
          Hide
          Damyon Wiese added a comment -

          Hmm - that failure is not reproducible (locally or on the ci server). Will keep an eye on it.

          Show
          Damyon Wiese added a comment - Hmm - that failure is not reproducible (locally or on the ci server). Will keep an eye on it.
          Hide
          Adrian Greeve added a comment - - edited

          I found a notice being displayed in the forums. I quickly made a fix for it. It's based on the current integration branch.
          wip-MDL-31776-master-alternatenamesfix

          I also found a bug in the question area. I have also included a commit for this as well.

          Show
          Adrian Greeve added a comment - - edited I found a notice being displayed in the forums. I quickly made a fix for it. It's based on the current integration branch. wip- MDL-31776 -master-alternatenamesfix I also found a bug in the question area. I have also included a commit for this as well.
          Hide
          Frédéric Massart added a comment - - edited

          Failing to highlight the small issues that Adrian reported (and fixed) before I even started this testing, and also to highlight a few places where we're not using full names, but maybe should.

          I'm sure we can take care of those in another issue and pass the test.

          Please integrate Adrian's commits!

          git pull git://github.com/abgreeve/moodle.git wip-MDL-31776-master-alternatenamesfix
          

          Places where name is not displayed accordingly to fullnamedisplay

          • You are logged in as ...
          • Breadcrumb
            • Home ► My courses ► Miscellaneous ► Arkadia ► Participants ► Eric Cartman ► View profile
            • Home ► Users ► Eric Cartman ► View profile
            • Home ► Users ► Eric Cartman ► Courses ► Arkadia ► Activity reports ► Today's logs ► Activity reports ► Today's logs
            • Home ► Profile settings for Eric Cartman ► Activity reports ► Grade
          • Navigation block under:
            • Current course > Course name > Participants > Firstname Lastname
            • Users > Firstname Lastname
          • Administration block: Profile settings for Eric Cartman
          • Recent activity block: "New forum posts: Admin User ..."
          • Blog post: by Admin User - Wednesday, 10 July 2013, 2:24 PM
          • In Home ► My profile settings ► Roles ► This user's role assignments: Admin User's role assignments
          • In Home ► Site administration ► Users ► Accounts ► Browse list of users
          • In Home ► Site administration ► Users ► Accounts ► Bulk user actions

          Forum

          • Name on listing of topics (Started by column). (Home ► My courses ► Miscellaneous ► Arkadia ► General ► News forum)
          • In a topic itself: by Admin User - Wednesday, 10 July 2013, 2:42 PM
          • In a topic itself (after searching)

          Glossary

          • Browse by author

          Lesson

          • Reports > Overview

          Wiki

          • Comments on a Wiki page
          Show
          Frédéric Massart added a comment - - edited Failing to highlight the small issues that Adrian reported (and fixed) before I even started this testing, and also to highlight a few places where we're not using full names, but maybe should. I'm sure we can take care of those in another issue and pass the test. Please integrate Adrian's commits! git pull git://github.com/abgreeve/moodle.git wip-MDL-31776-master-alternatenamesfix Places where name is not displayed accordingly to fullnamedisplay You are logged in as ... Breadcrumb Home ► My courses ► Miscellaneous ► Arkadia ► Participants ► Eric Cartman ► View profile Home ► Users ► Eric Cartman ► View profile Home ► Users ► Eric Cartman ► Courses ► Arkadia ► Activity reports ► Today's logs ► Activity reports ► Today's logs Home ► Profile settings for Eric Cartman ► Activity reports ► Grade Navigation block under: Current course > Course name > Participants > Firstname Lastname Users > Firstname Lastname Administration block: Profile settings for Eric Cartman Recent activity block: "New forum posts: Admin User ..." Blog post: by Admin User - Wednesday, 10 July 2013, 2:24 PM In Home ► My profile settings ► Roles ► This user's role assignments: Admin User's role assignments In Home ► Site administration ► Users ► Accounts ► Browse list of users In Home ► Site administration ► Users ► Accounts ► Bulk user actions Forum Name on listing of topics (Started by column). (Home ► My courses ► Miscellaneous ► Arkadia ► General ► News forum) In a topic itself: by Admin User - Wednesday, 10 July 2013, 2:42 PM In a topic itself (after searching) Glossary Browse by author Lesson Reports > Overview Wiki Comments on a Wiki page
          Hide
          Marina Glancy added a comment -

          I was not integrating it and this was your luck Adrian (because I'm an evil).

          There is a lot of code duplication in this issue. This could easily be avoided by

          1) adding the 3rd parameter to the function get_all_user_name_fields() which specifies prefix for the retrieved sql fields

          2) adding a new function that does a reverse mapping, i.e. to substitute repeating pieces of code like

                      $allnames = get_all_user_name_fields();
                      foreach ($allnames as $allname) {
                          $tempname = 'creator' . $allname;
                          if (isset($question->$tempname)) {
                              $u->$allname = $question->$tempname;
                          }
                      }
          

          or the same without prefix

                      $allnames = get_all_user_name_fields();
                      foreach ($allnames as $addname) {
                          $userinfo[$reviewer->reviewerid]->$addname = $reviewer->$addname;
                      } 
          

          I can see lots of similar chunks of code.

          Well, second one is probably too much now, maybe as a suggestion for improvement. But I would very recommend to add the $prefix argument to get_all_user_name_fields() now. What do you think?

          Show
          Marina Glancy added a comment - I was not integrating it and this was your luck Adrian (because I'm an evil). There is a lot of code duplication in this issue. This could easily be avoided by 1) adding the 3rd parameter to the function get_all_user_name_fields() which specifies prefix for the retrieved sql fields 2) adding a new function that does a reverse mapping, i.e. to substitute repeating pieces of code like $allnames = get_all_user_name_fields(); foreach ($allnames as $allname) { $tempname = 'creator' . $allname; if (isset($question->$tempname)) { $u->$allname = $question->$tempname; } } or the same without prefix $allnames = get_all_user_name_fields(); foreach ($allnames as $addname) { $userinfo[$reviewer->reviewerid]->$addname = $reviewer->$addname; } I can see lots of similar chunks of code. Well, second one is probably too much now, maybe as a suggestion for improvement. But I would very recommend to add the $prefix argument to get_all_user_name_fields() now. What do you think?
          Hide
          Frédéric Massart added a comment -

          Another issue: In the gradebook when marking a student over 100. The notice about missing SQL query fields pops up.

          Show
          Frédéric Massart added a comment - Another issue: In the gradebook when marking a student over 100. The notice about missing SQL query fields pops up.
          Hide
          Adrian Greeve added a comment -

          Thanks for the comments Marina.

          I completely agree. I think these ideas are excellent.
          I've created MDL-40612 to make these improvements as they will take some time to change code through out core.

          Thanks again.

          Show
          Adrian Greeve added a comment - Thanks for the comments Marina. I completely agree. I think these ideas are excellent. I've created MDL-40612 to make these improvements as they will take some time to change code through out core. Thanks again.
          Hide
          Adrian Greeve added a comment -

          Sorry Fred, I didn't see you message earlier.

          I've created a new commit on the same branch to fix the gradebook problem.

          Thanks.

          Show
          Adrian Greeve added a comment - Sorry Fred, I didn't see you message earlier. I've created a new commit on the same branch to fix the gradebook problem. Thanks.
          Hide
          Marina Glancy added a comment -

          Patch is pulled, please re-test

          Show
          Marina Glancy added a comment - Patch is pulled, please re-test
          Hide
          Adrian Greeve added a comment -

          After having a discussion with Fred about how user names are now displayed. I think that he raises a valid point and that we should re-evaluate how names are displayed though out moodle. I've created an issue for this. MDL-40616.

          Show
          Adrian Greeve added a comment - After having a discussion with Fred about how user names are now displayed. I think that he raises a valid point and that we should re-evaluate how names are displayed though out moodle. I've created an issue for this. MDL-40616 .
          Hide
          Frédéric Massart added a comment -

          Passing as the notices were fixed, and the other places where fullname() should be used (or used differently) will be taken care of in MDL-40616.

          Show
          Frédéric Massart added a comment - Passing as the notices were fixed, and the other places where fullname() should be used (or used differently) will be taken care of in MDL-40616 .
          Hide
          Marina Glancy added a comment -

          This issue requires documentation (at least for the settings pages that are changed) and developers documentation. Also it needs to be included in release notes

          Show
          Marina Glancy added a comment - This issue requires documentation (at least for the settings pages that are changed) and developers documentation. Also it needs to be included in release notes
          Hide
          Damyon Wiese added a comment -

          a single bug fix
          a drop in a waterfall
          hear the mighty roar

          Thanks for your contribution!

          Show
          Damyon Wiese added a comment - a single bug fix a drop in a waterfall hear the mighty roar Thanks for your contribution!
          Hide
          Eloy Lafuente (stronk7) added a comment -

          Please don't forget both user and developer documentation for this.

          A mini-guide about how to move to use/support the new fields would be awesome (we are already receiving code showing users here and there and it's hard to explain the required changes/available API).

          Ciao

          Show
          Eloy Lafuente (stronk7) added a comment - Please don't forget both user and developer documentation for this. A mini-guide about how to move to use/support the new fields would be awesome (we are already receiving code showing users here and there and it's hard to explain the required changes/available API). Ciao
          Hide
          Marina Glancy added a comment -

          Adrian, please don't forget about dev docs. There are more places where this is not implemented, function fullname() displays a debug information and there is no clue in the function itself on how to change the code.

          Show
          Marina Glancy added a comment - Adrian, please don't forget about dev docs. There are more places where this is not implemented, function fullname() displays a debug information and there is no clue in the function itself on how to change the code.
          Hide
          Dan Poltawski added a comment -

          Hi Adrian,

          We've noticed a sporadic unit test failure related to this a few times on the integration server. It concerns us a little so have created MDL-40929

          love,
          Integrators
          xxx

          Show
          Dan Poltawski added a comment - Hi Adrian, We've noticed a sporadic unit test failure related to this a few times on the integration server. It concerns us a little so have created MDL-40929 love, Integrators xxx
          Hide
          Dan Poltawski added a comment -

          ps. When I was searching for a duplicate, I noticed there are a number of linked issues from here which might need tidying up/closing etc.

          Show
          Dan Poltawski added a comment - ps. When I was searching for a duplicate, I noticed there are a number of linked issues from here which might need tidying up/closing etc.
          Hide
          Adrian Greeve added a comment -

          I created the following dev doc for this issue http://docs.moodle.org/dev/Additional_name_fields

          Show
          Adrian Greeve added a comment - I created the following dev doc for this issue http://docs.moodle.org/dev/Additional_name_fields
          Hide
          Mary Cooch added a comment -

          Removing docs_required link with thanks to Adrian for this http://docs.moodle.org/26/en/Additional_name_fields and also qa_test_required as there is a test here MDLQA-6673 (passed)

          Show
          Mary Cooch added a comment - Removing docs_required link with thanks to Adrian for this http://docs.moodle.org/26/en/Additional_name_fields and also qa_test_required as there is a test here MDLQA-6673 (passed)

            People

            • Votes:
              52 Vote for this issue
              Watchers:
              33 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Agile