LMTP server closing connection - system non-responsive

[karol@uri.edu]

We are running zcs-NETWORK-7.1.1_GA_3196.RHEL5_64.20110527001604 with patch zcs-patch-7.1.1_GA_3222 applied. We have implemented a multi-server environment (one mail-store server, one LDAP server) and upgraded from version 6.0.12 two weeks ago. We have approx. 2800 users. We have experienced two alarming events. One manifested itself by our mail store not accepting any mail although you were able to log in using an IMAP client (happened during a "slower" time of day) and the other the mail store was accepting mail albeit very slowly and the Zimbra web interface was non-responsive ("busier" time of day). Each time, in the zimbra.log, the following message occurred over and over again responding to different pieces of mail:

Aug 16 09:50:01 norm postfix/error[27328]: 0829D1EA8B5: to=<abc@mail.uri.edu>,
relay=none, delay=0.09, delays=0.05/0/0/0.04, dsn=4.0.0, status=deferred (deliv
ery temporarily suspended: host charlie.uri.edu[131.128.1.96] refused to talk to
me: 421 charlie.uri.edu Zimbra LMTP server closing connection; service busy)

Our LMTP threads hit the upper limit each time and did not come back down until I restarted zmmailboxdctl.

[LMStone]

We have a client with a similar situation.

May I trouble you please to run as root:

netstat -an | grep CLOSING

and post back if you have more than few connections in that state and on what ports?

Thanks,
Mark

[LMStone]

Something else to try...

On our client's system somehow Amavis had been started twice. There were 20 amavis processes running and the ten "rogue" processes wouldn't talk nicely with Postfix.

This may not be your issue, but in our client's case first we did as the zimbra user:

zmamavisdctl stop

Next, we ran top, sorted by name to put all the remaining running amavis processes listed first, and as the zimbra user killed them by PID.

Finally, we then restarted amavis using "amavisdctl start" and all the errors went away.

If this isn't your specific issue, you may have "rogue" zimbra component processes running, so instead of stopping just amavis as we did, I would stop all of zimbra ("zmcontrol stop") and then run top > u > zimbra and confirm no zimbra-owned processes other than your shell are still running.

Hope that helps,
Mark

[xaqar]

We are experiencing the same LMTP issue. We have had it 3 times now that LMTP reports busy until it is restarted. We have ~9000 users.

[karol@uri.edu]

We upgraded to 7.1.2 on Sunday 8/21/11 hoping this might resolve the issue. It took a few days to manifest itself after we went from 6.0.12 to 7.1.1.
I found this posting that seemed to reference our LMTP problem and decided to take a chance and upgrade.

Zimbra Collaboration Server 7.1.2 released | Agilemail

I'll update on the status of our Zimbra deployment after a few days go by.

[xaqar]

Looks like that might be our issue as well. I'll have to do the upgrade. We rarely have fewer than 1000 IMAP sessions running.

Thanks for the info!

[stuartg.orion]

I think we're seeing the same issue, our LMTP server has locked up several times over the last few weeks.

Thanks for pointing out that fix, I'll update to 7.1.2 very sooon

[deepblue]

Same here on 7.1.1. Only zmmailboxdctl restart helps. Does someone know the bugzilla bugid for this issue? Is it really fixed in 7.1.2?

Regards
Thomas

[burnmage]

We had this happen too with 7.1.1p2 in rhel 5 64bit. Restarted the zimbra daemons and it started working again. Has anyone seen a bug related to this?

[stuartg.orion]

The release statement above implies it's related to IMAP connections, I wonder if restarting the IMAP proxy, or dropping IMAP connections in some other manner would allow things to start working again. Restarting the mailbox daemon is a big thing and takes quite a while.

I'm updating to 7.1.2 to see if that fixes it this weekend.