Intermittant ZimbraLdapContext Errors...

[guyson]

I am using Zimbra 6.0.6 64bit and I have managed to install it on Ubuntu 10.04 Server. Forcing the install was actually pretty easy and only involved a small change to the install script, changing /etc/issue and pinning Perl to version 5.8 from Jaunty.

I've installed this software stack on a Dell R710 with 8G of RAM and 700G of hardware RAID5 storage.

It was running rather smooth until I began migrating my coworkers to this new setup and then major issues arose.

---

I see the following error message very often on the Admin console:

Server error encountered Message: system failure: ZimbraLdapContext Error code: service.FAILURE Method: SearchDirectoryRequest Details:soap:Receiver

This problem comes and goes. If I restart zimba is will go away for a few hours. Currently the Admin console is unusable, but I know from experience it will clear up after restarting Zimbra.

Some times the Zimbra web client will just display a message about 'zimbraLdapContext error'.

zimbra@atlanta:~$ zmcontrol -v


Release 6.0.6_GA_2324.UBUNTU8_64 UBUNTU8_64 FOSS edition.

I have kept very detailed notes of what I've tried to remedy this problem but nothing have solved it yet. I have tried increasing the number of LDAP threads but that did not seem to have much of a difference.

Any thoughts? What log file should I be looking at? What parameters should I be reviewing?

[veronica]

Increasing LDAP threads might deteriorate the problem. Default limit of 8 threads is good enough for most deployments. Can you share your zimbra.log and mailbox.log files for more analysis ?

[guyson]

Originally Posted by veronica

Increasing LDAP threads might deteriorate the problem. Default limit of 8 threads is good enough for most deployments. Can you share your zimbra.log and mailbox.log files for more analysis ?

Thanks Veronica.


I found this in /opt/zimbra/log/mailbox.log.

2010-04-26 23:45:13,901 INFO  [btpool0-77://zimbra.mydomain.com:7071/service/admin/soap/SearchDirectoryRequest] [name=guyson@mydomain.com;mid=5;ip=xxx.xxx.xxx.xxx;ua=ZimbraWebClient - FF3.0 (Linux);] soap - SearchDirectoryRe
quest
2010-04-26 23:45:13,903 INFO  [btpool0-77://zimbra.mydomain.com:7071/service/admin/soap/SearchDirectoryRequest] [name=guyson@mydomain.com;mid=5;ip=xxx.xxx.xxx.xxx;ua=ZimbraWebClient - FF3.0 (Linux);] SoapEngine - handler exc
eption
com.zimbra.common.service.ServiceException: system failure: ZimbraLdapContext
ExceptionId:btpool0-77://zimbra.mydomain.com:7071/service/admin/soap/SearchDirectoryRequest:1272336313903:089080d68022be55
Code:service.FAILURE
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:248)
        at com.zimbra.cs.account.ldap.ZimbraLdapContext.<init>(ZimbraLdapContext.java:391)
        at com.zimbra.cs.account.ldap.ZimbraLdapContext.<init>(ZimbraLdapContext.java:360)
        at com.zimbra.cs.account.ldap.LdapProvisioning.searchObjects(LdapProvisioning.java:1245)
        at com.zimbra.cs.account.ldap.LdapProvisioning.searchObjects(LdapProvisioning.java:1226)
        at com.zimbra.cs.account.ldap.LdapProvisioning.searchDirectory(LdapProvisioning.java:4475)
        at com.zimbra.cs.session.AccountSearchParams.doSearch(AccountSearchParams.java:92)
        at com.zimbra.cs.session.AdminSession.searchAccounts(AdminSession.java:74)
        at com.zimbra.cs.service.admin.SearchDirectory.handle(SearchDirectory.java:162)
        at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:419)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:273)
        at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:157)
        at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:291)
        at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:212)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:182)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
        at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:79)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
        at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:81)
        at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:132)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
        at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:230)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.handler.DebugHandler.handle(DebugHandler.java:77)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:543)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:939)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405)
        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
        at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
Caused by: javax.naming.CommunicationException: simple bind failed: zimbra.mydomain.com:389 [Root exception is java.net.SocketException: Bad file descriptor]
        at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:197)
        at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2694)
        at com.sun.jndi.ldap.LdapCtx.<init>(LdapCtx.java:293)
        at com.sun.jndi.ldap.LdapCtxFactory.getUsingURL(LdapCtxFactory.java:175)
        at com.sun.jndi.ldap.LdapCtxFactory.getUsingURLs(LdapCtxFactory.java:193)
        at com.sun.jndi.ldap.LdapCtxFactory.getLdapCtxInstance(LdapCtxFactory.java:136)
        at com.sun.jndi.ldap.LdapCtxFactory.getInitialContext(LdapCtxFactory.java:66)
        at javax.naming.spi.NamingManager.getInitialContext(NamingManager.java:667)
        at javax.naming.InitialContext.getDefaultInitCtx(InitialContext.java:288)
        at javax.naming.InitialContext.init(InitialContext.java:223)
        at javax.naming.ldap.InitialLdapContext.<init>(InitialLdapContext.java:134)
        at com.zimbra.cs.account.ldap.ZimbraLdapContext.<init>(ZimbraLdapContext.java:377)
       ... 42 more
Caused by: java.net.SocketException: Bad file descriptor
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
        at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:393)
        at com.sun.jndi.ldap.LdapClient.ldapBind(LdapClient.java:334)
        at com.sun.jndi.ldap.LdapClient.authenticate(LdapClient.java:192)
        ... 53 more

I also noticed that libldap & libc had updates. I stopped Zimbra, did those updates and rebooted. I'm aware the reboot might have been overkill, but since it was after hours I had to take full advantage of my window of opportunity.

After the system restarted, this error did not happen again. Typically the behavior is that this error will only occur after a few hours of production use.

[veronica]

Seems to be related to this bug Bug 42870 – "Bad file descriptor" SocketException causes system failure every several hours

[nexussrl]

@Veronica
ty for your response
the bug on your link as been resolved on 6.02 version, but i installed the last available version and i still have the same problem
have you another idea?
bye

[nexussrl]

anyone?

[veronica]

Last comment of the bug says "reopen the bug if this issue re-surfaces". Request you to reopen the bug with evidence of your log snippets.

[Godsend]

Each time I try to modify user account from the Admin console, I keep getting ZimbraLdapContext Error code: service.FAILURE Method: ModifyAccountRequest Details:soap:Receiver.

I am currently running Release 6.0.3_GA_1915.RHEL5_20091118110117 CentOS5 FOSS edition.

Please kindly help me out.