Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-67772

Forums don't thread properly in email clients when message id's are rewritten

XMLWordPrintable

    • MOODLE_39_STABLE, MOODLE_402_STABLE, MOODLE_403_STABLE
    • MOODLE_402_STABLE, MOODLE_403_STABLE
    • MDL-67772-email-thread-headers-MOODLE_403_STABLE
    • MDL-67772-email-thread-headers
    • Hide

      Somewhat hard to test as it requires an email setup which overwrites your ID's such as AWS SES. Also not all email clients rely on these headers, many use a combination of heuristics and still thread correctly based just on the titles (eg gmail).

      Prerequisites

      1. Set up Mailhog: 

        docker run -p 8025:8025 -p 1025:1025 mailhog/mailhog

      2. Configure Moodle to use Mailhog:
        1. Browse to "Site Administration" > "Server" > "Support contact"
        2. Set "supportemail" to something like "support@localhost.invalid"
        3. Browse to "Site Administration" > "Server" > "Outgoing mail configuration"
        4. Set "smtphosts" to "localhost:1025"
        5. Set "noreplyaddress" to something like "noreply@localhost.invalid"
      3. Thunderbird installed on your machine with the following add on installed: ImportExportTools NG
      4. Add this testing patch below which simulates an external MTA always overwriting the Message IDs on every email:

      diff --git a/lib/moodlelib.php b/lib/moodlelib.php
      		 
      index 686cf26c036..d914f0cfd7e 100644
      		 
      --- a/lib/moodlelib.php
      		 
      +++ b/lib/moodlelib.php
      		 
      @@ -6311,9 +6311,9 @@ function email_to_user($user, $from, $subject, $messagetext, $messagehtml = '',
      		 
           $messagetext = $renderer->render_from_template('core/email_text', $context);
      		 
       
      		 
           // Autogenerate a MessageID if it's missing.
      		 
      -    if (empty($mail->MessageID)) {
      		 
      +    // if (empty($mail->MessageID)) {
      		 
               $mail->MessageID = generate_email_messageid();
      		 
      -    }
      		 
      +    // }
      		 
       
      		 
           if ($messagehtml && !empty($user->mailformat) && $user->mailformat == 1) {

      Testing with MH + Thunderbird

      1. In Moodle, create a test course with a forum
      2. Go to the announcement forum
      3. Click Add discussion topic
      4. Click advanced at the bottom
      5. Add some test data and tick 'Send forum post notifications with no editing-time delay'
      6. Send several nested replies, each time clicking 'Advanced' and making sure that 'no editing time delay' option is ticked
      7. Run cron to send the emails
      8. Navigate to http://localhost:8025/
      9. For each email, download it into a local folder
      10. Open thunderbird
        1. Create a new directory under the local folders mail root
        2. Right click on the new directory
        3. Select "Importtools... > Import EML files > All EML messages from directory"
        4. Select the directory EML files were downloaded into
        5. Run through the import process
      11. Toggle View > Sort by > Threaded
      12. Confirm you see all the messages in a single thread

       

       

      Show
      Somewhat hard to test as it requires an email setup which overwrites your ID's such as AWS SES. Also not all email clients rely on these headers, many use a combination of heuristics and still thread correctly based just on the titles (eg gmail). Prerequisites Set up Mailhog: docker run -p 8025:8025 -p 1025:1025 mailhog/mailhog Configure Moodle to use Mailhog: Browse to "Site Administration" > "Server" > "Support contact" Set "supportemail" to something like "support@localhost.invalid" Browse to "Site Administration" > "Server" > "Outgoing mail configuration" Set "smtphosts" to "localhost:1025" Set "noreplyaddress" to something like "noreply@localhost.invalid" Thunderbird installed on your machine with the following add on installed: ImportExportTools NG Add this testing patch below which simulates an external MTA always overwriting the Message IDs on every email: diff --git a/lib/moodlelib.php b/lib/moodlelib.php index 686cf26c036..d914f0cfd7e 100644 --- a/lib/moodlelib.php +++ b/lib/moodlelib.php @@ - 6311 , 9 + 6311 , 9 @@ function email_to_user($user, $from, $subject, $messagetext, $messagehtml = '' ,      $messagetext = $renderer->render_from_template( 'core/email_text' , $context);         // Autogenerate a MessageID if it's missing. -     if (empty($mail->MessageID)) { +     // if (empty($mail->MessageID)) {          $mail->MessageID = generate_email_messageid(); -    } +     // }         if ($messagehtml && !empty($user->mailformat) && $user->mailformat == 1 ) { Testing with MH + Thunderbird In Moodle, create a test course with a forum Go to the announcement forum Click Add discussion topic Click advanced at the bottom Add some test data and tick 'Send forum post notifications with no editing-time delay' Send several nested replies, each time clicking 'Advanced' and making sure that 'no editing time delay' option is ticked Run cron to send the emails Navigate to http://localhost:8025/ For each email, download it into a local folder Open thunderbird Create a new directory under the local folders mail root Right click on the new directory Select "Importtools... > Import EML files > All EML messages from directory" Select the directory EML files were downloaded into Run through the import process Toggle View > Sort by > Threaded Confirm you see all the messages in a single thread    

      Sibling tracker to MDLSITE-6059. I originally though this was more of an AWS setup bug just with how moodle.org's email mta is setup, which overwrites the message id's created by moodle which breaks the threading. But then I realized that this can be gracefully worked around in a generic way in core.

      The problem is the first email in a thread would be like:

       

      Message-ID: <646082d990334df2b39@moodle.org>
      		 
      Thread-Index: 646082d990334df2b392e5aa6e91

       

      And then second email reply reference it:

      In-Reply-To: <646082d990334df2b39@moodle.org>
      		 
      References: <646082d990334df2b39@moodle.org>

      But AWS overwrites all message id's so the first email looks like:

       

      Message-ID: <0101016fa75a4c94-38bfe565-70ea-40c7-b321-c4c2368911a3-000000@us-west-2.amazonses.com>
      		 
      Thread-Index: 646082d990334df2b392e5aa6e91 

       

      But we don't actually need the message Id's to match, if we simply add the References to the initial email then most email clients will thread the anyway because they are designed to thread even with a partial subset of the emails.

      Message-ID: <0101016fa75a4c94-38bfe565-70ea-40c7-b321-c4c2368911a3-000000@us-west-2.amazonses.com>
      		 
      Thread-Index: 646082d990334df2b392e5aa6e91
      		 
      In-Reply-To: <646082d990334df2b39@moodle.org>
      		 
      References: <646082d990334df2b39@moodle.org>

       

        1. (1) 12 Passed -- (Main)MDL-67772.png
          (1) 12 Passed -- (Main)MDL-67772.png
          60 kB
        2. image-2024-01-15-10-02-19-816.png
          image-2024-01-15-10-02-19-816.png
          21 kB
        3. Patch_to_test_67772.patch
          0.8 kB
        4. thunderbird-local-testing.png
          thunderbird-local-testing.png
          52 kB

            brendanheywood Brendan Heywood
            brendanheywood Brendan Heywood
            Matthew Hilton Matthew Hilton
            Glyn (Mathew) May Glyn (Mathew) May
            Kim Jared Lucas Kim Jared Lucas
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved:

                Estimated:
                Original Estimate - 0 minutes
                0m
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 3 hours, 55 minutes
                3h 55m

                  Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.