CentOS 4.4 imapsync problems

[timschmidt]

Hey folks... this one's got me stumped. I'm prototyping a Zimbra Network Edition install on a 30 day trial of RHEL 4 and CentOS 4.4, both under VMware. Each installation is completely independent - just multiple installs to feel out the differences. I've done a minimal install, installed compat-libstdc++-33 and compat-libstdc++-296, as well as libidn, curl, and fetchmail (as required by the Zimbra install. I've also cranked up maximum attachment size in the MTA tab of the Global Settings window to 100000k. Nothing else funny.

While attempting to import an account using imapsync, I get interesting errors (in /opt/zimbra/log/zimbra.log) on quite a few messages (as far as I can tell, only ones with attachments) and imapsync says something like:

Couldn't append msg #23 (Subject:[Fw: 11076 die set]) to folder Gill T&-D: Error trying to append: 859 NO APPEND failed

Here's a zimbra.log section:

2006-12-28 10:39:57,772 WARN  [ImapSSLServer-1] [ip=192.168.1.3;name=stevev@mail2.mtd-inc.com;] TransformationClient - Error sending document; tell convertd never mind
java.net.SocketException: Connection reset
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at com.zimbra.cs.convert.TransformationClient.copyChunks(TransformationClient.java:122)
        at com.zimbra.cs.convert.TransformationClient.extract(TransformationClient.java:156)
        at com.zimbra.cs.convert.SocketTransformationStub.doExtract(SocketTransformationStub.java:68)
        at com.zimbra.cs.convert.SocketTransformationStub.extract(SocketTransformationStub.java:106)
        at com.zimbra.cs.mime.handler.DefaultHandler.getContentImpl(DefaultHandler.java:85)
        at com.zimbra.cs.mime.MimeHandler.getContent(MimeHandler.java:257)
        at com.zimbra.cs.mime.ParsedMessage.analyzePart(ParsedMessage.java:589)
        at com.zimbra.cs.mime.ParsedMessage.analyzeMessage(ParsedMessage.java:508)
        at com.zimbra.cs.mime.ParsedMessage.analyze(ParsedMessage.java:252)
        at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.java:3054)
        at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.java:3040)
        at com.zimbra.cs.imap.ImapAppendOperation.callback(ImapAppendOperation.java:126)
        at com.zimbra.cs.operation.Operation.schedule(Operation.java:190)
        at com.zimbra.cs.imap.ImapHandler.doAPPEND(ImapHandler.java:1163)
        at com.zimbra.cs.imap.ImapHandler.executeRequest(ImapHandler.java:342)
        at com.zimbra.cs.imap.ImapHandler.processCommand(ImapHandler.java:227)
        at com.zimbra.cs.tcpserver.ProtocolHandler.processConnection(ProtocolHandler.java:231)
        at com.zimbra.cs.tcpserver.ProtocolHandler.run(ProtocolHandler.java:198)
        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:595)
2006-12-28 10:39:57,799 WARN  [ImapSSLServer-1] [ip=192.168.1.3;name=stevev@mail2.mtd-inc.com;] TransformationClient - Socket error for client at port 47350: java.net.SocketException: Broken pipe
2006-12-28 10:39:58,826 WARN  [ImapSSLServer-1] [ip=192.168.1.3;name=stevev@mail2.mtd-inc.com;] ParsedMessage - Parse error on MIME part 3 (image/jpeg, Message-ID: <000901c62284$593aa1c0$f101a8c0@mtd2ub5020gwg>)
com.zimbra.cs.mime.MimeHandlerException: cannot extract text
        at com.zimbra.cs.mime.handler.DefaultHandler.getContentImpl(DefaultHandler.java:92)
        at com.zimbra.cs.mime.MimeHandler.getContent(MimeHandler.java:257)
        at com.zimbra.cs.mime.ParsedMessage.analyzePart(ParsedMessage.java:589)
        at com.zimbra.cs.mime.ParsedMessage.analyzeMessage(ParsedMessage.java:508)
        at com.zimbra.cs.mime.ParsedMessage.analyze(ParsedMessage.java:252)
        at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.java:3054)
        at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.java:3040)
        at com.zimbra.cs.imap.ImapAppendOperation.callback(ImapAppendOperation.java:126)
        at com.zimbra.cs.operation.Operation.schedule(Operation.java:190)
        at com.zimbra.cs.imap.ImapHandler.doAPPEND(ImapHandler.java:1163)
        at com.zimbra.cs.imap.ImapHandler.executeRequest(ImapHandler.java:342)
        at com.zimbra.cs.imap.ImapHandler.processCommand(ImapHandler.java:227)
        at com.zimbra.cs.tcpserver.ProtocolHandler.processConnection(ProtocolHandler.java:231)
        at com.zimbra.cs.tcpserver.ProtocolHandler.run(ProtocolHandler.java:198)
        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:595)
Caused by: com.zimbra.cs.convert.ConversionException: Cannot connect to convertd
        at com.zimbra.cs.convert.SocketTransformationStub.doExtract(SocketTransformationStub.java:71)
        at com.zimbra.cs.convert.SocketTransformationStub.extract(SocketTransformationStub.java:106)
        at com.zimbra.cs.mime.handler.DefaultHandler.getContentImpl(DefaultHandler.java:85)
        ... 15 more
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:115)
        at java.io.DataOutputStream.writeInt(DataOutputStream.java:180)
        at com.zimbra.cs.convert.TransformationClient.copyChunks(TransformationClient.java:127)
        at com.zimbra.cs.convert.TransformationClient.extract(TransformationClient.java:156)
        at com.zimbra.cs.convert.SocketTransformationStub.doExtract(SocketTransformationStub.java:68)
        ... 17 more
2006-12-28 10:39:58,854 WARN  [ImapSSLServer-1] [ip=192.168.1.3;name=stevev@mail2.mtd-inc.com;] ParsedMessage - Message had parse errors in 1 parts (Message-Id: <000901c62284$593aa1c0$f101a8c0@mtd2ub5020gwg>, Subject: 540BWC2.jpg;540BWC1.jpg)
2006-12-28 10:39:58,855 WARN  [ImapSSLServer-1] [ip=192.168.1.3;name=stevev@mail2.mtd-inc.com;] imap - APPEND failed
com.zimbra.cs.service.ServiceException: system failure: failed to analyze part
        at com.zimbra.cs.service.ServiceException.FAILURE(ServiceException.java:174)
        at com.zimbra.cs.mime.ParsedMessage.analyzeMessage(ParsedMessage.java:517)
        at com.zimbra.cs.mime.ParsedMessage.analyze(ParsedMessage.java:252)
        at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.java:3054)
        at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.java:3040)
        at com.zimbra.cs.imap.ImapAppendOperation.callback(ImapAppendOperation.java:126)
        at com.zimbra.cs.operation.Operation.schedule(Operation.java:190)
        at com.zimbra.cs.imap.ImapHandler.doAPPEND(ImapHandler.java:1163)
        at com.zimbra.cs.imap.ImapHandler.executeRequest(ImapHandler.java:342)
        at com.zimbra.cs.imap.ImapHandler.processCommand(ImapHandler.java:227)
        at com.zimbra.cs.tcpserver.ProtocolHandler.processConnection(ProtocolHandler.java:231)
        at com.zimbra.cs.tcpserver.ProtocolHandler.run(ProtocolHandler.java:198)
        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:595)
Caused by: com.zimbra.cs.convert.ConversionException: Cannot connect to convertd
        at com.zimbra.cs.convert.SocketTransformationStub.doExtract(SocketTransformationStub.java:71)
        at com.zimbra.cs.convert.SocketTransformationStub.extract(SocketTransformationStub.java:106)
        at com.zimbra.cs.mime.handler.DefaultHandler.getContentImpl(DefaultHandler.java:85)
        at com.zimbra.cs.mime.MimeHandler.getContent(MimeHandler.java:257)
        at com.zimbra.cs.mime.ParsedMessage.analyzePart(ParsedMessage.java:589)
        at com.zimbra.cs.mime.ParsedMessage.analyzeMessage(ParsedMessage.java:508)
        ... 12 more
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:115)
        at java.io.DataOutputStream.writeInt(DataOutputStream.java:180)
        at com.zimbra.cs.convert.TransformationClient.copyChunks(TransformationClient.java:127)
        at com.zimbra.cs.convert.TransformationClient.extract(TransformationClient.java:156)
        at com.zimbra.cs.convert.SocketTransformationStub.doExtract(SocketTransformationStub.java:68)
        ... 17 more

[timschmidt]

I should mention explicitly that I see the behavior on both RHEL and CentOS. On VMs with 768Mb - 1.5Gb of ram. I've also removed all mentions of UTF-8 from /etc/sysconfig/i18n on CentOS to work around an installation problem, so it looks like this:

LANG="en_US"
SUPPORTED="en_US:en"
SYSFONT="latarcyrheb-sun16"

[dkarp]

We use Verity to extract text from attachments for building our full-text index. When Verity bombs on an attachment, we're doing the wrong thing and rejecting the IMAP APPEND (this is bug 13250 -- please vote for it!).

The easiest way around this would probably be for you to disable attachment indexing during the imapsync, then reenable it for normal operation. After the import, just use the admin console or command-line tools to reindex the imported mailboxes.

[timschmidt]

Awesome. Works well. Thanks!

[timschmidt]

Everything's working great... just wondering if receiving new mail in the account normally (not through imapsync) will trigger the bad behavior?

[dkarp]

The problem is a Thunderbird bug that occurs only when copying messages into Zimbra via Thunderbird. Regular message receipt will mark mail with the time it's received.

[timschmidt]

Just to confirm... I see the problem when importing existing mail from another (non-Zimbra) server into a freshly built Zimbra box using imapsync. Thunderbird never touches the Zimbra machine (although some of the mail on the previous server undoubtedly touched thunderbird).

--tim

[dkarp]

Please make sure that you specify the --syncinternaldates flag when you run imapsync. Check out the migration wiki page for more details.

[timschmidt]

The exact invocation of imapsync I've been using is as follows:

imapsync --nosyncacls --syncinternaldates --regexflag 's/ ?\$\$\$INDEXED###//' --authmech1 PLAIN --ssl1 --host1 $host1 --user1 "$user1" --password1 "$p1" --authmech2 PLAIN --ssl2 --host2 $host2 --user2 "$user2" --password2 "$p2"

I've also tested with --nofastio1 and 2, with no noticeable change in results.

[dkarp]

Which IMAP server are you migrating from? Can you get a dump of some of imapsync's traffic with the old IMAP server?

(The obvious workaround for you is using the Thunderbird hack and migrating to user "$user2/tb". But it'd be nice to know what IMAP server was being problematic...)