lmtp - Not delivering message with duplicate Message-ID

[mps]

Hello,

I'm taking a look to the mailbox.log and I'm seeing this message:
"lmtp - Not delivering message with duplicate Message-ID"

The incoming message is for multiple recipients.

The users are not receiving this message? Or is there some internal control for this in Zimbra?

Version of Zimbra: 5.0.4_GA_2101.DEBIAN4.0

Thanks in advance.

[mmorse]

When LMTP receives the same message for the same user twice in a short period of time, it drops the second copy - this all takes place in memory. The server will keep that many recipient/Message-ID pairs in a FIFO map (first in, first out) dropping any duplicates it sees. Though I don't think you need to, it's possible to disable this by setting zimbraMessageIdDedupeCacheSize to 0 in the global LDAP config. Useful when you're using zmlmtpinject over and over & want it to actually make more copies. For normal operation default is 3000.

[mps]

mmorse, thanks for the explanation!

[sfilipov]

Hi,

I have an issue with messages which are not coming and after some investigation I've found what is the problem.
Fisrt at 13:55 user jatanasova have sent message with subject "info profile diageo" to user mbobeva. Message was received normally.
Then at 14:07 and 14:25 user jatanasova have sent message with subject "Fwd: info profile diageo" to user mbobeva, but messages didn't come at all. Mailbox.log says: "lmtp - Not delivering message with duplicate Message-ID".
Then at 14:40 user jatanasova have sent message with subject "Fwd: info profile diageo" to distribution list in which mbobeva is participating, but again message was stopped with the same error.

Is this normal, and why is this happening - messages are different.

And from where/how can I change zimbraMessageIdDedupeCacheSize to 0. And did this hide some risks?

Thanks in advance.
Bellow is the log:


2008-06-30 13:55:04,849 INFO [LmtpServer-4952] [ip=212.72.217.102;] lmtp - Delivering message: size=25967 bytes, nrcpts=1, sender=jatanasova@manager.bg, msgid=<!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kP tZ7CgAAAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>
2008-06-30 13:55:04,883 INFO [LmtpServer-4952] [name=mbobeva@manager.bg;mid=68;] mailop - Adding Message: id=3174, Message-ID=<!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kPtZ7 CgAAAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>, parentId=-1, folderId=2, folderName=Inbox.

2008-06-30 14:07:05,587 INFO [LmtpServer-4965] [ip=212.72.217.102;] lmtp - Delivering message: size=34200 bytes, nrcpts=1, sender=jatanasova@manager.bg, msgid=<!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kP tZ7CgAAAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>
2008-06-30 14:07:05,588 INFO [LmtpServer-4965] [name=mbobeva@manager.bg;mid=68;] lmtp - Not delivering message with duplicate Message-ID <!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kPtZ7CgA AAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>

2008-06-30 14:25:21,555 INFO [LmtpServer-4993] [ip=212.72.217.102;] lmtp - Delivering message: size=35543 bytes, nrcpts=1, sender=jatanasova@manager.bg, msgid=<!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kP tZ7CgAAAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>
2008-06-30 14:25:21,556 INFO [LmtpServer-4993] [name=mbobeva@manager.bg;mid=68;] lmtp - Not delivering message with duplicate Message-ID <!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kPtZ7CgA AAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>

2008-06-30 14:40:45,608 INFO [LmtpServer-5018] [ip=212.72.217.102;] lmtp - Delivering message: size=36629 bytes, nrcpts=9, sender=jatanasova@manager.bg, msgid=<!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kP tZ7CgAAAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>
2008-06-30 14:40:45,635 INFO [LmtpServer-5018] [name=dlukanova@manager.bg;mid=69;] mailop - Adding Message: id=2307, Message-ID=<!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kPtZ7 CgAAAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>, parentId=-1, folderId=2, folderName=Inbox.
2008-06-30 14:40:45,679 INFO [LmtpServer-5018] [name=ggoranov@manager.bg;mid=70;] mailop - Adding Message: id=2077, Message-ID=<!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kPtZ7 CgAAAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>, parentId=-1, folderId=2, folderName=Inbox.
2008-06-30 14:40:45,692 INFO [LmtpServer-5018] [name=ghristova@manager.bg;mid=71;] mailop - Adding Message: id=2108, Message-ID=<!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kPtZ7 CgAAAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>, parentId=-1, folderId=2, folderName=Inbox.
2008-06-30 14:40:45,716 INFO [LmtpServer-5018] [name=iivanova@manager.bg;mid=72;] mailop - Adding Message: id=2133, Message-ID=<!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kPtZ7 CgAAAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>, parentId=-1, folderId=2, folderName=Inbox.
2008-06-30 14:40:45,735 INFO [LmtpServer-5018] [name=jatanasova@manager.bg;mid=66;] mailop - Adding Message: id=3343, Message-ID=<!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kPtZ7 CgAAAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>, parentId=-1, folderId=2, folderName=Inbox.
2008-06-30 14:40:45,744 INFO [LmtpServer-5018] [name=mbobeva@manager.bg;mid=68;] lmtp - Not delivering message with duplicate Message-ID <!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kPtZ7CgA AAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>
2008-06-30 14:40:45,746 INFO [LmtpServer-5018] [name=mmarinova@manager.bg;mid=73;] mailop - Adding Message: id=2153, Message-ID=<!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kPtZ7 CgAAAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>, parentId=-1, folderId=2, folderName=Inbox.
2008-06-30 14:40:45,876 INFO [LmtpServer-5018] [name=sfilipov@manager.bg;mid=48;] mailop - Adding Message: id=1798, Message-ID=<!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kPtZ7 CgAAAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>, parentId=-1, folderId=1494, folderName=ADV.
2008-06-30 14:40:45,895 INFO [LmtpServer-5018] [name=sgeorgieva@manager.bg;mid=74;] mailop - Adding Message: id=2206, Message-ID=<!&!AAAAAAAAAAAYAAAAAAAAAHmF2r0UnVVHgGaOP3kPtZ7 CgAAAEAAAAPN8/SQ5MydJnMuBTWMxhf4BAAAAAA==@manager.bg>, parentId=-1, folderId=2, folderName=Inbox.

[fowler]

When LMTP receives the same message for the same user twice in a short period of time, it drops the second copy
...
it's possible to disable this by setting zimbraMessageIdDedupeCacheSize to 0 in the global LDAP config.

zmprov gcf zimbraMessageIdDedupeCacheSize
3000
zmprov mcf zimbraMessageIdDedupeCacheSize 0
zmprov gcf zimbraMessageIdDedupeCacheSize
0

NOTE: Even after flushing the config cache (zmprov fc config), the change was not reflected. It appears a restart is necessary for the change to take affect.

See also: Flushing LDAP Cache

[liuk]

Hello Mike,
we are facing similare issues here with Clients using Outlook 2010. For example it the user forward the same email again changing the attachment the message is discarded for the recipient.
Since from what I can read on RFC 2822 :
the existence of a message id is an optional (SHOULD have) requirement, but its uniqueness if it is supplied is necessary (MUST be)
I suppose that the fault is from the Outlook client.

Now my other questions are: what is the purpose of this caching ? And what can be the side effects of setting it to ZERO ?

Thanks!
Luca

[mattg1]

Hi... We just encountered this issue here too. In fact, it has been going on for a long time, but was just noticed now. Users use Outlook templates to send messages via ZCO. The messages from that template always have the same Message-ID. Apparently this is a known Outlook problem for a long time. Awesome. Anyway, I guess changing zimbraMessageIdDedupeCacheSize is a workaround, but may have unintended consequences in the case of mail loops and such. (Perhaps merely reducing the value rather than zeroing it would help the problem, while still providing some loop protection.)

Another option might be for ZCO to remove the Message-ID header (which is optional from the MUA's end) so that the Zimbra server can generate a new unique one. Or, even configure the Zimbra server to ignore client-generated Message-IDs on submitted emails. I don't know if those would violate some RFC, though.

Any other ideas besides tweaking zimbraMessageIdDedupeCacheSize ? (Or getting rid of Outlook?)

[jerryipe]

I have read the above post, i am also facing the same issue mentioned above,
Any idea about zimbraMessageIdDedupeCacheSize, what does the value 3000 means.
Any one please give an idea.

Regards,
Jerry I. George

Hi... We just encountered this issue here too. In fact, it has been going on for a long time, but was just noticed now. Users use Outlook templates to send messages via ZCO. The messages from that template always have the same Message-ID. Apparently this is a known Outlook problem for a long time. Awesome. Anyway, I guess changing zimbraMessageIdDedupeCacheSize is a workaround, but may have unintended consequences in the case of mail loops and such. (Perhaps merely reducing the value rather than zeroing it would help the problem, while still providing some loop protection.)

Another option might be for ZCO to remove the Message-ID header (which is optional from the MUA's end) so that the Zimbra server can generate a new unique one. Or, even configure the Zimbra server to ignore client-generated Message-IDs on submitted emails. I don't know if those would violate some RFC, though.

Any other ideas besides tweaking zimbraMessageIdDedupeCacheSize ? (Or getting rid of Outlook?)

[liuk]

In the release notes of 7.1.4:
Global LDAP attrib zimbraMessageIdDedupeCacheTimeout for LMTP de-dupe cache to workaround clients incorrectly using the same ID for different messages. If non-zero, zimbraMessageIdDedupeCacheSize limit is ignored. (44564)

Did you try to first reduce the zimbraMessageIdDedupeCacheSize to a small value ?

If you set it to zero users will receive duplicate e-mails under certain conditions, but there are no other side effects AFAIK.

HTH
Luca

[mattg1]

Any idea about zimbraMessageIdDedupeCacheSize, what does the value 3000 means.

The way I understand it, it means that Zimbra will remember the last 3000 Message-IDs it sees, and prevent delivery of any "duplicates" of those 3000 messages. It's up to you to figure out how much email your server sees, and adjust that value to estimate how long in seconds this might be. On my server, I had to set it to 0 to allow Outlook users to do what they wanted reliably.

Global LDAP attrib zimbraMessageIdDedupeCacheTimeout for LMTP de-dupe cache to workaround clients incorrectly using the same ID for different messages. If non-zero, zimbraMessageIdDedupeCacheSize limit is ignored. (44564)

Thanks.... I will have to look into zimbraMessageIdDedupeCacheTimeout to see if it might be an alternative for me.