serious IMAP download bug (6326)

[ericding]

I reported (but forgot to make publicly visible) bug 6326 in early March: the Zimbra IMAP server has a serious bug, IMO, in that it fails to add a newline to the end of messages which do not have a trailing newline in the Zimbra message store. While this bug doesn't cause problems with, e.g., Thunderbird, it does cause issues with fetchmail:

fetchmail: IMAP> A0008 FETCH 1 BODY.PEEK[TEXT]
fetchmail: IMAP< * 1 FETCH (BODY[TEXT] {1354}
 (1354 body octets) .fetchmail: IMAP< A0008 OK FETCH completed
fetchmail: message ericding@foo.net@mail.foo.net:1 was not the expected length (3338 actual != 3335 expected)
 not flushed

The visible result is that any such message (which includes, it seems, all messages from outside servers) winds up getting an extra right paren tagged on at the end. I'm fairly certain this is a bug in the Zimbra IMAP implementation, if I read RFC3501 properly:

All interactions transmitted by client and server are in the form of lines, that is, strings that end with a CRLF. The protocol receiver of an IMAP4rev1 client or server is either reading a line, or is reading a sequence of octets with a known count followed by a line.

So far, the bug remains unconfirmed in Bugzilla; should I just realistically expect that it remain so for the near future?

[KevinH]

Thanks for the bump.. I confirmed it, made it public, added your latest note, and assigned it to our IMAP expert.

[ericding]

Thanks, KevinH. I didn't say anything new in my forum post that wasn't already in the Bugzilla issue, but just wanted to see if I could get this bug looked at and fixed.

[dkarp]

I think you've misread RFC 3501. The RFC does not mandate that every piece of data returned by the server be CRLF-terminated, it just mandates that the server CRLF-terminate every server response, either tagged or untagged. The message body is *part* of a response, and thus the server really shouldn't be tacking on extra CRLFs.

For instance, if fetchmail had sent a command along the lines of

A0008 FETCH 1 (BODY.PEEK[TEXT] FLAGS)

the Zimbra IMAP server would have to respond with

* 1 FETCH (BODY[TEXT] {1354}<CRLF>
<the 1354 octets that comprise the message> FLAGS (\Sent))<CRLF>

The close-parenthesis you're seeing is the mate to the one after "* 1 FETCH " at the beginning of the untagged response.

It seems that fetchmail is not taking the octet count and reading that number of bytes off the wire, but is instead taking the octet count and reading *lines* until it fills its expected buffer. This is completely wrong. Thunderbird gets it right.

That being said, if fetchmail has this bug, Zimbra needs to work around it. (Which version of fetchmail are you running?) I *think* we can tack on an extra CRLF to all message bodies, but we'll need to make sure that we're not breaking anything else in the process...

[ericding]

I think you've misread RFC 3501. The RFC does not mandate that every piece of data returned by the server be CRLF-terminated, it just mandates that the server CRLF-terminate every server response, either tagged or untagged. The message body is *part* of a response, and thus the server really shouldn't be tacking on extra CRLFs.

Fair enough... I've glanced at RFC822, and it doesn't seem to insist that messages end with CRLF...

It seems that fetchmail is not taking the octet count and reading that number of bytes off the wire, but is instead taking the octet count and reading *lines* until it fills its expected buffer. This is completely wrong. Thunderbird gets it right.

Yup, from my reading of fetchmail's source code, this is exactly what it does (see the function readbody() in http://mknod.org/svn/fetchmail/trunk/transact.c). I'm using the latest version of fetchmail (6.3.2).

That being said, if fetchmail has this bug, Zimbra needs to work around it. (Which version of fetchmail are you running?) I *think* we can tack on an extra CRLF to all message bodies, but we'll need to make sure that we're not breaking anything else in the process...

Or rather, I think Zimbra needs to make sure it's not dropping the CRLF field that I think should typically be at the end of every received message body... (how is it even possible to send a message via SMTP without that CRLF?)

[ericding]

Ah... I should have read the FAQ (for fetchmail, not Zimbra)!

From http://fetchmail.berlios.de/fetchmail-FAQ.html#X8:

Blame it on that rancid pile of dung and offal called Microsoft Exchange. Due to the problem described in S2, the IMAP support in fetchmail cannot follow the IMAP protocol 100%. Most of the time it doesn't matter, but if you combine it with an SMTP server that behaves unusually, you'll get a spurious ) at message end.

One piece of software that can trigger this is the Interchange mail server, as used by, e.g., mailandnews.com. Here's what happens:

1. Someone sends mail to your account. The last line of the message contains text. So at the SMTP level, the message ends with, e.g. "blahblah\r\n.\r\n"

2. The SMTP handler sees the final "\r\n.\r\n" and recognizes the end of the message. However, instead of doing the normal thing, which is tossing out the ".\r\n" and leaving the first '\r\n' as part of the email body, Interchange throws out the whole "\r\n.\r\n", and leaves the email body without any line terminator at the end of it. RFC821 does not forbid this, though it probably should.

3. Fetchmail, or some other IMAP client, asks for the message. IMAP returns it, but it's enclosed inside parentheses, according to the protocol. The message size in bytes is also present. Because the message doesn't end with a line terminator, the IMAP client sees:

....blahblah)...

where the ')' is from IMAP.

4. Fetchmail only deals with complete lines, and can't trust the stated message size because Microsoft Exchange fscks it up.

5. As a result, fetchmail takes the final 'blahblah)' and puts it at the end of the message it forwards on. If you have verbosity on, you'll get a message about actual != expected.

There is no fix for this. The nuke mentioned in S2 looks more tempting all the time.