ERROR LmtpServer Thread pool exhausted

[jgt]

2012-02-02 10:01:09,081 WARN [LmtpServer] [] TcpServer/7025 - Thread pool is 105% utilized. 21 out of 20 threads inuse.
2012-02-02 10:01:09,081 ERROR [LmtpServer] [] TcpServer/7025 - cannot handle connection; thread pool xhausted
java.util.concurrent.RejectedExecutionException
at java.util.concurrent.ThreadPoolExecutor$AbortPolic y.rejectedExecution
(ThreadPoolExecutor.java:1768)
at java.util.concurrent.ThreadPoolExecutor.reject(Thr eadPoolExecutor.jav
a:767)
at java.util.concurrent.ThreadPoolExecutor.execute(Th readPoolExecutor.ja
va:658)
at com.zimbra.cs.tcpserver.TcpServer.run(TcpServer.ja va:202)
at java.lang.Thread.run(Thread.java:662)

This results in the mail queue being deferred with the message "service temporarily suspended, server refused to talk to me 421 lmtp server closing connection; service busy "
Where can the number of threads be increased or limited?

[jgt]

We have not solved the problem, but we have found a workaround.
Originally we had about 100 accounts, 95 web mail and 5 Outlook (POP3).
Every day at the end of their shift, each of the 95 would send an email with a 10mb attachment to their supervisor(s) generally 2 of the 5 POP accounts (so at 4pm and 9pm there would be a huge peak in traffic of emails from worker@abc.com to supervisor@abc.com).
We changed the 5 POP accounts so that emails were forwarded to another domain, without saving a copy on the server, and added a second account to the supervisors' Outlook.

[yutaka]

As your log said, it seems that the number of LmtpServer Thread is not enough in your system.

You can change that with zimbraLmtpNumThreads from zmprov or admin console.

[jgt]

Has this attribute been deprecated?
I tried

zmprov ms  zimbraLmtpNumThreads 40

but all I get is the zmprov help screen.
I had already tried increasing the number of concurrent processes in the admin console, but it does not change this attribute.

[yutaka]

Please try this first to get all server names:

zmprov gas

Then pick one which has your mailbox.
And put it after "ms" as a target server like beow:

zmprov ms <server name> zimbraLmtpNumThreads 40

Then see how it goes.

[jgt]

Thanks,
That seems to have worked, at least, I found an entry in the audit.log showing the change from 20 to 40. I won't know until Monday if 40 resolves the original issue or not as there is virtually no email on the weekends.

[jgt]

I was able to change the number of threads to 40, but I think that this is a symptom of a problem rather than the cause.
By 08:10 this morning (Monday) the thread pool was exhausted 41 out of 40.

[phoenix]

Originally Posted by jgt

I was able to change the number of threads to 40, but I think that this is a symptom of a problem rather than the cause.
By 08:10 this morning (Monday) the thread pool was exhausted 41 out of 40.

What is the specific disk hardware on which the /opt/zimbra directory structure is located? Is the Zimbra version mentioned in your profile still your current version and if not, what are you using? Have you monitored the iops of your disk subsystem? Has this problem just started occurring? Have you done any updates to the operating system recently? How much RAM does this server have?

[jgt]

The system is Intel I3 with 4gb ram, and two 250gb sata drives, RAID1 (ICH10).
There have been no changes to either the operating system or Zimbra.
I believe that I have found the cause.
Three of the 5 Outlook users have about 9gb, 5gb, 3gb of of email stored within folders in Outlook. They use notebooks, and take them home. They sometimes do clean up work at home, then when they come to the office the following day, the Outlook connector tries to sync the files with the server.
Yesterday I turned off the Outlook connector, and IMAP.
This morning I have turned local delivery back on (see post #2).

[jgt]

Spoke too soon. As soon as I turned local delivery back on, the mail started queuing up and exceeded 40 threads.