Uploaded image for project: 'Moodle'
  1. Moodle
  2. MDL-67772

Forums don't thread properly in email clients when message id's are rewritten

    XMLWordPrintable

Details

    • MOODLE_39_STABLE, MOODLE_402_STABLE, MOODLE_403_STABLE
    • MOODLE_402_STABLE, MOODLE_403_STABLE
    • MDL-67772-email-thread-headers-MOODLE_402_STABLE
    • MDL-67772-email-thread-headers-MOODLE_403_STABLE
    • MDL-67772-email-thread-headers
    • Hide

      Somewhat hard to test as it requires an email setup which overwrites your ID's such as AWS SES. Also not all email clients rely on these headers, many use a combination of heuristics and still thread correctly based just on the titles (eg gmail).

      Prerequisites

      1. Set up Mailhog:

        docker run -p 8025:8025 -p 1025:1025 mailhog/mailhog
        

      2. Configure Moodle to use Mailhog:
        1. Browse to "Site Administration" > "Server" > "Support contact"
        2. Set "supportemail" to something like "support@localhost.invalid"
        3. Browse to "Site Administration" > "Server" > "Outgoing mail configuration"
        4. Set "smtphosts" to "localhost:1025"
        5. Set "noreplyaddress" to something like "noreply@localhost.invalid"
      3. Thunderbird installed on your machine with the following add on installed: ImportExportTools NG
      4. Add this testing patch below which simulates an external MTA always overwriting the Message IDs on every email:

      diff --git a/lib/moodlelib.php b/lib/moodlelib.php
      index 686cf26c036..d914f0cfd7e 100644
      --- a/lib/moodlelib.php
      +++ b/lib/moodlelib.php
      @@ -6311,9 +6311,9 @@ function email_to_user($user, $from, $subject, $messagetext, $messagehtml = '',
           $messagetext = $renderer->render_from_template('core/email_text', $context);
       
           // Autogenerate a MessageID if it's missing.
      -    if (empty($mail->MessageID)) {
      +    // if (empty($mail->MessageID)) {
               $mail->MessageID = generate_email_messageid();
      -    }
      +    // }
       
           if ($messagehtml && !empty($user->mailformat) && $user->mailformat == 1) {
       

      Testing with MH + Thunderbird

      1. In Moodle, create a test course with a forum
      2. Go to the announcement forum
      3. Click Add discussion topic
      4. Click advanced at the bottom
      5. Add some test data and tick 'Send forum post notifications with no editing-time delay'
      6. Send several nested replies, each time clicking 'Advanced' and making sure that 'no editing time delay' option is ticked
      7. Run cron to send the emails
      8. Navigate to http://localhost:8025/
      9. For each email, download it into a local folder
      10. Open thunderbird
        1. Create a new directory under the local folders mail root
        2. Right click on the new directory
        3. Select "Importtools... > Import EML files > All EML messages from directory"
        4. Select the directory EML files were downloaded into
        5. Run through the import process
      11. Toggle View > Sort by > Threaded
      12. Confirm you see all the messages in a single thread

       

       

      Show
      Somewhat hard to test as it requires an email setup which overwrites your ID's such as AWS SES. Also not all email clients rely on these headers, many use a combination of heuristics and still thread correctly based just on the titles (eg gmail). Prerequisites Set up Mailhog: docker run -p 8025:8025 -p 1025:1025 mailhog/mailhog Configure Moodle to use Mailhog: Browse to "Site Administration" > "Server" > "Support contact" Set "supportemail" to something like "support@localhost.invalid" Browse to "Site Administration" > "Server" > "Outgoing mail configuration" Set "smtphosts" to "localhost:1025" Set "noreplyaddress" to something like "noreply@localhost.invalid" Thunderbird installed on your machine with the following add on installed: ImportExportTools NG Add this testing patch below which simulates an external MTA always overwriting the Message IDs on every email: diff --git a/lib/moodlelib.php b/lib/moodlelib.php index 686cf26c036..d914f0cfd7e 100644 --- a/lib/moodlelib.php +++ b/lib/moodlelib.php @@ - 6311 , 9 + 6311 , 9 @@ function email_to_user($user, $from, $subject, $messagetext, $messagehtml = '' ,      $messagetext = $renderer->render_from_template( 'core/email_text' , $context);         // Autogenerate a MessageID if it's missing. -     if (empty($mail->MessageID)) { +     // if (empty($mail->MessageID)) {          $mail->MessageID = generate_email_messageid(); -    } +     // }         if ($messagehtml && !empty($user->mailformat) && $user->mailformat == 1 ) { Testing with MH + Thunderbird In Moodle, create a test course with a forum Go to the announcement forum Click Add discussion topic Click advanced at the bottom Add some test data and tick 'Send forum post notifications with no editing-time delay' Send several nested replies, each time clicking 'Advanced' and making sure that 'no editing time delay' option is ticked Run cron to send the emails Navigate to http://localhost:8025/ For each email, download it into a local folder Open thunderbird Create a new directory under the local folders mail root Right click on the new directory Select "Importtools... > Import EML files > All EML messages from directory" Select the directory EML files were downloaded into Run through the import process Toggle View > Sort by > Threaded Confirm you see all the messages in a single thread    

    Description

      Sibling tracker to MDLSITE-6059. I originally though this was more of an AWS setup bug just with how moodle.org's email mta is setup, which overwrites the message id's created by moodle which breaks the threading. But then I realized that this can be gracefully worked around in a generic way in core.

      The problem is the first email in a thread would be like:

       

      Message-ID: <646082d990334df2b39@moodle.org>
      Thread-Index: 646082d990334df2b392e5aa6e91
      

       

      And then second email reply reference it:

      In-Reply-To: <646082d990334df2b39@moodle.org>
      References: <646082d990334df2b39@moodle.org>

      But AWS overwrites all message id's so the first email looks like:

       

      Message-ID: <0101016fa75a4c94-38bfe565-70ea-40c7-b321-c4c2368911a3-000000@us-west-2.amazonses.com>
      Thread-Index: 646082d990334df2b392e5aa6e91 
      

       

      But we don't actually need the message Id's to match, if we simply add the References to the initial email then most email clients will thread the anyway because they are designed to thread even with a partial subset of the emails.

      Message-ID: <0101016fa75a4c94-38bfe565-70ea-40c7-b321-c4c2368911a3-000000@us-west-2.amazonses.com>
      Thread-Index: 646082d990334df2b392e5aa6e91
      In-Reply-To: <646082d990334df2b39@moodle.org>
      References: <646082d990334df2b39@moodle.org>

       

      Attachments

        Issue Links

          Activity

            People

              brendanheywood Brendan Heywood
              brendanheywood Brendan Heywood
              Matthew Hilton Matthew Hilton
              Mathew May Mathew May
              Kim Jared Lucas Kim Jared Lucas
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 3 hours, 55 minutes
                  3h 55m

                  Clockify

                    Error rendering 'clockify-timesheets-time-tracking-reports:timer-sidebar'. Please contact your Jira administrators.