LmtpServer thread pool exhausted

[cayaraa]

Since upgrading to zimbra 7 we've had our lmtp thread pool exhausted 3 times on our two mailbox servers. There doesn't appear to be any large burst of email and the mailbox servers don't have a high load while its happening. Some details:

Release 7.1.1_GA_3196.RHEL5_64_20110527001604 RHEL5_64 NETWORK edition
5 zimbra servers (1 ldap, 2 proxy, 2 mailbox)

Here is the error I see in the /opt/zimbra/log/maillog.log file when the issue starts happening:

2011-08-03 22:43:11,048 WARN [LmtpServer] [] TcpServer/7025 - Thread pool is 100% utilized. 20 out of 20 threads in use.
2011-08-03 22:43:30,240 WARN [LmtpServer] [] TcpServer/7025 - Thread pool is 100% utilized. 20 out of 20 threads in use.
<snip>
2011-08-03 22:55:03,157 WARN [LmtpServer] [] TcpServer/7025 - Thread pool is 105% utilized. 21 out of 20 threads in use.
2011-08-03 22:55:03,157 ERROR [LmtpServer] [] TcpServer/7025 - cannot handle connection; thread pool exhausted
java.util.concurrent.RejectedExecutionException
at java.util.concurrent.ThreadPoolExecutor$AbortPolic y.rejectedExecution(ThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(Thr eadPoolExecutor.java:767)
at java.util.concurrent.ThreadPoolExecutor.execute(Th readPoolExecutor.java:658)
at com.zimbra.cs.tcpserver.TcpServer.run(TcpServer.ja va:202)
at java.lang.Thread.run(Thread.java:662)

This continued until I did a reload on the mailboxdctrl on that server. While this happened the proxy/mta server queued up ~1000 emails (not a huge number). When I found it this morning the load on the mailbox server was only 0.54, 0.57, 0.46 and wasn't having any IO issues that I saw. After the reload of mailboxd I requeued the emails on the mta and they all delivered without any issue within a very short amount of time. This leads me to believe it might be something with threads being locked by something or maybe some sort of thread "garbage collection" not working?

Something else to note....when it first starting happening last night I aslo saw this in the log which I don't remember seeing in version 6 when users were over quota:

2011-08-03 22:43:11,439 ERROR [LmtpServer-3607] [name=username@domain.com;mid=943;ip=xxxxx;] jsieve - Evaluation failed. Reason: null
2011-08-03 22:43:11,439 WARN [LmtpServer-3607] [name=username@domain.com;mid=943;ip=xxxxxxx;] filter - An error occurred while processing filter rules. Filing message to /Inbox.
com.zimbra.cs.filter.ZimbraSieveException
<snip>
Caused by: com.zimbra.cs.mailbox.MailServiceException: mailbox exceeded quota of 1048576000 bytes
ExceptionId:LmtpServer-3607:1312425791439:b2ae89c42b0568b1
Code:mail.QUOTA_EXCEEDED Arglimit, NUM, "1048576000")
at com.zimbra.cs.mailbox.MailServiceException.QUOTA_E XCEEDED(MailServiceException.java:318)

I've just upped the thread pool from 20 to 40 to see if that helps.

Any other thoughts on what could be causing this or how to fix it?

[cmyslinsky]

cayaraa, I am experience the same symptoms you have noted on one of my boxes and am curious if increasing zimbraLmtpNumThreads to 40 helped your concern.

Cheers
Chad

[cayaraa]

IIRC it did help but shortly after this we realized that our local drives that held the zimbra application data was being maxed out (iops wise) as apposed to the data store/index drives that we had been focusing on. We had a hunch that the thread pool errors might have been an early symptom of this but didn't have any way to know for sure.