Message-ID causes mail problem when sending to more than one person (e.g. forum subscription)

That is an interesting problem. It's not something I've heard of before. The line in question, the mail ID line, was last changed in March of 2008 so lots of people have been using it for quite some time without a problem. That would indicate something unusual is going on with your mail server.

That said, its still possible that we should modify the message ID as, even if you are the first to run into this, you're unlikely to be the last.

Update: after some googling it seems that the message ID should indeed be unique for each message sent. http://stackoverflow.com/questions/831183/is-the-message-id-email-header-unique-for-each-recipient

Could you please let us know what mail server you are using, and perhaps other details of your setup?

Hi Andrew!

After not having found any problems like mine, I also suggested that something is wrong with our mail server.

While trying to figure out the problem, I found out the following: If you send mails to multiple persons with a common mail system (googlemail etc), all mails are sent in one session (one "TO" header contains all mail addresses), while mails sent by moodle are all separated into different new sessions. I know, this can be configured under "Site administration -> Plugins -> Message outputs -> Email" (SMTP session limit), but this doesn't work - moodle always sends separated mails.

Update: Hi Michael! Long time no see! I'll gather information and come back to you soon.

Update 2: We've got a standard Linux SuSE postfix mail server, that stores mails in files (maildrop). Our mail program picks them up using pop3. But due to the same message-ID, they are recognized as "duplicate".

Kind regards,
Dominik

Adding a potential solution for this. I was a little wary of putting the user ID in the email but it should be fine as its going to the user themselves (theoretically) and they could easily get their own user ID from Moodle itself if they so desired. Plus, I wasn't sure what else to use. We could plug in a random number there but that seemed a bit crappy. Not sure why.

+1 certainly makes sense to me.

also "can be configured under "Site administration -> Plugins -> Message outputs -> Email" (SMTP session limit), but this doesn't work - moodle always sends separated mails." - is there an issue for that?

G'nite moodleverse.

Adding branches and test instructions.

Sorry, reopening.

That Message-ID header is used, in conjunction with the 'In-Reply-To' and the 'References' headers to provide nesting along a varietè of email clients.

So all them must follow the same naming schema, or it will break badly.

To keep all uses always on sync I'd suggest:

1) Create one function to calculate the messageid.
2) Use it everywhere

Also, this should be (still) be backported to 20_STABLE, if I'm not wrong. EOL is Dec 31th.

Ciao

Adding this to the next sprint as it got left behind.

I've fixed this up so that nesting of emails should all work as expected.

The main moodle.git repository has just been updated with latest weekly modifications. You may wish to rebase your PULL branches to simplify history and avoid any possible merge conflicts. This would also make integrator's life easier next week.

TIA and ciao

The most I think on this, the less logic I found to these message-ids to be globally unique.

If email A sends one email to B and C, and B is set to automatically forward to C (or B and C are the same mailbox), at the end C will receive 2 emails with the same message id.

So I really think that no software should rely on unique Message-ID headers at all, in fact, they are set by senders and no system should rely on non-trusted data in that way.

Also, I use postfix since ages ago and never have detected any problem like that, both in sent and received emails with Moodle. Strange.

In any case, and as far as by changing this it's going to cause all client using such information for threading to become crazy (lost the threads), perhaps we should consider to:

1) Do it only for master. So breakage for clients will happen only in 2.3, and not now in the middle of all stables. Or instead,

2) Do it controlled by setting, so existing sites will continue using the old way and new ones will get, by default, the new (more unique) way. And any existing site will be able to switch at any moment (causing breakage of threading). Or instead,

3) Ignore the breakage problem and do it for all branches.

BUT, in any case of the alternative 1,2,3 above:

4) I would hash the postid + userid information. I cannot imagine any benefit about having that information there in a plain way (specially the userid). Perhaps some bounce control may be using that? Can anybody find a good reason to keep them plain?

While the patch is 100% correct, I'm going to keep this open util it's decided. Perhaps I'm being too much exigent, but considering that we introduced those headers for the pure reason of providing threading for clients, and knowing that the change will break current ones... I think it's worth deciding and implementing the change is the best possible way.

Ciao

reopening for further discussion (see previous comment).

1) will break existing threads?
2) consider setting controlling it?
3) hash it?

"If email A sends one email to B and C, and B is set to automatically forward to C (or B and C are the same mailbox), at the end C will receive 2 emails with the same message id."

They wont. the message ID includes the user ID of the user being emailed. Provided B and C have different Moodle user IDs the emails will have 2 different message IDs.

Hashing the data to create the message ID seems like a good idea. Its possible someone could parse plain text message IDs to do some sort of fancy mail sorting but that seems unlikely.

And having this only go into master seems reasonable to me. Certainly breaking everyone's email threading for a point release seems bad.

Adding a setting also sounds a bit bad. It would effectively just be a toggle switch that has no obvious effect apart from breaking everyone's email threading. "toggle this every time you want to annoy your users"

I've added a second commit to the master version that uses a hash in the message ID. Note the message is meant to be formated somewhat like an email address. For some reason. http://en.wikipedia.org/wiki/Message-ID

Ive removed the stable version for the time being at least as theyre out of date and may not be integrated.

As far as it is assumed (and perhaps commented somewhere, in release notes or so), that breakage of current threads may happen for some clients, np about doing so. I think that some clients, like Thunderbird, uses to fallback to subject matching if headers don't match.

Looks perfect IMO. Both the email-like format and the master-only target. +1

Putting this up for integration. Master only.

The main moodle.git repository has just been updated with latest weekly modifications. You may wish to rebase your PULL branches to simplify history and avoid any possible merge conflicts. This would also make integrator's life easier next week.

TIA and ciao

Adding the docs_required tag as far as this (the change and potential breakage) should be commented, in release notes or Forum FAQ or wherever.

Uhm,

just saw that you're using the hash() function. While it's installed by default since php 5.1.x, it may be opted-out. So I've created MDL-31707 about that.

It does not prevent integration of this, though... so doing.

Ciao

Integrated, thanks!

Works Great,

Thanks for fixing this Andrew. Just wondering, why it's an improvement and not a bug. It might be nice to backport this to 22 and 21

Sorry, forgot to mention about docblock.
New function forum_get_email_message_id, doesn't have a docblock.

Do we fail test, if functionality works but docblock or space etc. is an issue?

Read the comments for the reason this wasn't backported to 2.2 and 2.1

No, you shouldn't fail the test for docs type issues. It should have been picked up by me or the peer reviewer. Ill open a new MDL for that.

update: MDL-31739

Hello Andrew,

I saw the comments, but the problem was reported for 2.1 and 2.2. Just wondering, if we plan to have anything done for stable branches (As, I can't see any related bug for solving this issue on stable branches.)

No. There wont be a fix for this in the stable branches as fixing this would break up everyone's existing email threads.

Some changes to Moodle should be milestones in the project by themselves.

This is not the case and your fix is not so important, but your collaboration is highly appreciated, thanks!

Closing as fixed, ciao

I've got the same problems with email notifications from forum.
Cron runs correctly, shows user processing and indicates that "n users were sent post idnumber" but only one user receives email notification.

I upgraded to Moodle 2.2.2+ (Build: 20120323) but that didn't solve the issue.
I therefore modified file ../mod/forum/lib.php as suggested, but I didn't solve the problem.

Any suggestions?

Is the email domain of all recipients the same? If so, I'd suggest to look into the mail server's log and check out whether the mails arrive or not (or if they are rejected due to a special reason) ...

No domain was not the same. I looked carefully into the mail server's log and verified that using sendmail on Moodle without a smart smtp I did not realise that that server had been closed by firewall on port 25. Now I used smart smtp in my mail program to pass through the problem. Thanks!

Please can anyone advise on what exactly should be added to the user docs about this issue, and also the release notes. I can't find any mention of it in the Using Moodle forums (yet!)

(Housekeeping)10 months on - just asking again if this needs documenting and if so, how, because otherwise perhaps the docs_required label can be removed?

i think it was mainly to say that this was fixed in the release (2.3) and that it wasn't fixed in older ones and was still an issue in older ones just to be aware of (borked email sending functionality described in description). This info is now a bit obsolete now though, nothing to do now imo.