[SOLVED] ZCS hang for 10 minutes when sending e-mail with a large number of recipient

[ChaSer]

ZCS hang for 10 minutes when somebody sending one e-mail with a large number (95 in my case) of recipients from internal domain.
This html e-mail message has very small size: contains 5 string of text with links to document published in Alfresco and its folder, no pictures or attachments.
Some recipients receive this message soon (in few seconds), but othes can not connect to the server through POP/IMAP/Zimbra/Web-console for 10 minutes.
Others receive this message after 10 minutes.
The result of "top" command at this time looks good: no proccessor load, no swap used, 150Mb RAM free, no disks wait etc.
"zmcontrol status" reply quickly and show all processes running.
Telnet to SMTP reply with Postfix greeting in the twinkling of an eye, but telnet to POP or IMAP not connects - hangs.
Server configuration for 190 users:
AMD Athlon 64 X2 5000+
DDR2 4Gb
2xHDD SATAII 500Gb (program RAID1)

In Zimbra web admin console I get errors like:
In "User accounts": Msg: system failure: ZimbraLdapContext Code: service.FAILURE Method: SearchDirectoryRequest Detail:soap:Receiver
In "Domains": Msg: system failure: ZimbraLdapContext Code: service.FAILURE Method: GetAllZimletsRequest Detail:soap:Receiver
In "Servers": Msg: system failure: ZimbraLdapContext Code: service.FAILURE Method: GetServerRequest Detail:soap:Receiver
In "Mail queue": Msg: system failure: exception during auth {RemoteManager: mail.local->zimbra@mail.local:22} Code: service.FAILURE Method: GetMailQueueInfoRequest Detail:soap:Receiver
etc.

If there is enough hardware resources why ZCS hangs for some time ?

[ChaSer]

I done mailing for 2500 external recipients yesterday.
The result of "top" command at this time looks good: proccessor load not exceed 80%, no swap used, 800Mb RAM free, no disks wait etc.
ZCS hangs for 1.5 hours, but some times I can send message through web console and some times can see mail queue in web admin console.
I send message to my external mailbox and get it in some seconds.
I send message from my zimbra mailbox to itself and get it in 20 minutes.
My ZCS server get about 200 "wrong email address" reply from recipients servers after send all 2500 messages.
This 200 messages appear in pending mail queue and its number dont decrease for 10 minutes.
Mail queue be in processing so I can not process it manually and be forced to reboot ZCS.
Mail queue starts to processing after rebooting.
So problem is somewhere in incoming mail processing.

Where ? Maybe some configuration hints can help ?

[phoenix]

Originally Posted by ChaSer

Where ? Maybe some configuration hints can help ?

You need to look in the log files and see what's happening on your server otherwise it's just guesswork.

[ChaSer]

Originally Posted by phoenix

You need to look in the log files and see what's happening on your server otherwise it's just guesswork.

Does Your post meen I not overload the server and it should not freeze ?
I set DEBUG mode but can not find something interesting in log files.
Is that such from mailbox.log:

2010-09-15 16:48:54,379 INFO [btpool0-7://mail.local:7071/service/admin/soap/GetInfoRequest] [name=myusername@mydomain.com;aname=zimbra;mid=76;i p=10.10.20.12;ua=zclient/6.0.8_GA_2673;] soap - GetInfoRequest
2010-09-15 16:48:54,380 INFO [btpool0-7://mail.local:7071/service/admin/soap/GetInfoRequest] [name=myusername@mydomain.com;aname=zimbra;mid=76;i p=10.10.20.12;ua=zclient/6.0.8_GA_2673;] misc - delegated access: doc=GetInfo, delegating account=zimbra, authenticated account=myusername@mydomain.com, target account=myusername@mydomain.com
2010-09-15 16:48:54,382 ERROR [btpool0-7://mail.local:7071/service/admin/soap/GetInfoRequest] [name=myusername@mydomain.com;aname=zimbra;mid=76;i p=10.10.20.12;ua=zclient/6.0.8_GA_2673;] account - can't get identities
com.zimbra.common.service.ServiceException: system failure: ZimbraLdapContext
ExceptionId:btpool0-7://mail.local:7071/service/admin/soap/GetInfoRequest:1284558534382:9a9263a23b22232b
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:248)
at com.zimbra.cs.account.ldap.ZimbraLdapContext.<init >(ZimbraLdapContext.java:416)
at com.zimbra.cs.account.ldap.ZimbraLdapContext.<init >(ZimbraLdapContext.java:366)
at com.zimbra.cs.account.ldap.LdapProvisioning.getIde ntitiesByQuery(LdapProvisioning.java:4993)
at com.zimbra.cs.account.ldap.LdapProvisioning.getAll Identities(LdapProvisioning.java:5196)
at com.zimbra.cs.service.account.GetInfo.doIdentities (GetInfo.java:246)
at com.zimbra.cs.service.account.GetInfo.handle(GetIn fo.java:164)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEng ine.java:420)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:274)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:158)
at com.zimbra.soap.SoapServlet.doWork(SoapServlet.jav a:291)
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.jav a:212)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:727)
at com.zimbra.cs.servlet.ZimbraServlet.service(Zimbra Servlet.java:181)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(Ser vletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1166)
at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(Set HeaderFilter.java:79)
at org.mortbay.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1157)
at org.mortbay.servlet.UserAgentFilter.doFilter(UserA gentFilter.java:81)
at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter .java:155)
at org.mortbay.jetty.servlet.ServletHandler$CachedCha in.doFilter(ServletHandler.java:1157)
at org.mortbay.jetty.servlet.ServletHandler.handle(Se rvletHandler.java:388)
at org.mortbay.jetty.security.SecurityHandler.handle( SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(Se ssionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(Co ntextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebA ppContext.java:418)
at org.mortbay.jetty.handler.ContextHandlerCollection .handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle (HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(Ha ndlerWrapper.java:152)
at org.mortbay.jetty.handler.rewrite.RewriteHandler.h andle(RewriteHandler.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(Ha ndlerWrapper.java:152)
at org.mortbay.jetty.handler.DebugHandler.handle(Debu gHandler.java:77)
at org.mortbay.jetty.handler.HandlerWrapper.handle(Ha ndlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(Htt pConnection.java:543)
at org.mortbay.jetty.HttpConnection$RequestHandler.co ntent(HttpConnection.java:939)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser. java:755)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpPa rser.java:218)
at org.mortbay.jetty.HttpConnection.handle(HttpConnec tion.java:405)
at org.mortbay.io.nio.SelectChannelEndPoint.run(Selec tChannelEndPoint.java:413)
at org.mortbay.thread.BoundedThreadPool$PoolThread.ru n(BoundedThreadPool.java:451)
Caused by: javax.naming.CommunicationException: Bad file descriptor [Root exception is java.net.SocketException: Bad file descriptor]
at com.sun.jndi.ldap.LdapCtx.extendedOperation(LdapCt x.java:3213)
at javax.naming.ldap.InitialLdapContext.extendedOpera tion(InitialLdapContext.java:164)
at com.zimbra.cs.account.ldap.ZimbraLdapContext.<init >(ZimbraLdapContext.java:406)
... 40 more
Caused by: java.net.SocketException: Bad file descriptor
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutp utStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStre am.java:136)
at java.io.BufferedOutputStream.flushBuffer(BufferedO utputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputS tream.java:123)
at com.sun.jndi.ldap.Connection.writeRequest(Connecti on.java:396)
at com.sun.jndi.ldap.LdapClient.extendedOp(LdapClient .java:1172)
at com.sun.jndi.ldap.LdapCtx.extendedOperation(LdapCt x.java:3160)
... 42 more

[phoenix]

Originally Posted by ChaSer

Does Your post meen I not overload the server and it should not freeze ?

I've no idea whether your server is overloaded or not as you've given no information about the hardware, is it a VM or real hardware, number of users, installed RAM, whether your user are mostly Web UI users or IMAP/POP3 clients, etc. etc... Everything is guesswork unless you give a full description of the problem, your configuration and the exact circumstances under which it occurs. Certainly your server should not be 'hanging' for 10 minutes so you can start by searching the wiki for the words 'performance' and 'zmstat' and review some of the articles on how to produce graphs to monitor your server.

Originally Posted by ChaSer

I set DEBUG mode but can not find something interesting in log files.

Where did you set Debug mode?

Originally Posted by ChaSer

Is that such from mailbox.log:

Start with this: Bug 42870 &ndash; "Bad file descriptor" SocketException causes system failure every several hours

[ChaSer]

Originally Posted by phoenix

Start with this: Bug 42870 &ndash; "Bad file descriptor" SocketException causes system failure every several hours

You are THE BEST !

My Ubuntu 10.04 max opened file limit is 382416 (cat /proc/sys/fs/file-max)
So command "ulimit -n 524288" in zimbra user .bashrc (/opt/zimbra/.bashrc) file get error ant zimbra user max opened file limit was setted to 1024.
Root limit was to 1024.

I add in end of /etc/security/limits.conf:

zimbra soft nofile 382416
zimbra hard nofile 382416
root soft nofile 382416
root hard nofile 382416

and just only restart ZCS by "zmcontrol restart".