Zimbra failing since update to 6.0.10

[MartBrooks]

Hi

I updated our pilot Zimbra server to 6.0.10 The update process appeared to go well.

zimbra@zimbra:~$ zmcontrol -v

Release 6.0.10_GA_2692.UBUNTU10_64 UBUNTU10_64 FOSS edition.

Today I was actually planning on putting the server live for our mail. When doing one final check, I suddenly starting getting the following error when trying to access a shared folder:

A network service error has occurred.
method:	SearchRequest
msg:	system failure: ZimbraLdapContext
code:	service.FAILURE
detail:	soap:Receiver
trace:	com.zimbra.common.service.ServiceException: system failure: ZimbraLdapContext ExceptionId:btpool0-14://zimbra.clues.ltd.uk/service/soap/SearchRequest:1294077775371:aa8b9e5db59b7757 Code:service.FAILURE at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:248) at com.zimbra.cs.account.ldap.ZimbraLdapContext.(ZimbraLdapContext.java:416) at com.zimbra.cs.account.ldap.ZimbraLdapContext.(ZimbraLdapContext.java:373) at com.zimbra.cs.account.ldap.LdapProvisioning.getAccountByQuery(LdapProvisioning.java:608) at com.zimbra.cs.account.ldap.LdapProvisioning.getAccountByNameInternal(LdapProvisioning.java:776) at com.zimbra.cs.account.ldap.LdapProvisioning.getAccountByName(LdapProvisioning.java:757) at com.zimbra.cs.account.ldap.LdapProvisioning.get(LdapProvisioning.java:668) at com.zimbra.cs.account.ldap.LdapProvisioning.get(LdapProvisioning.java:653) at com.zimbra.cs.account.Provisioning.get(Provisioning.java:692) at com.zimbra.soap.ZimbraSoapContext.(ZimbraSoapContext.java:218) at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:203) at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:158) at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:291) at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:212) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:181) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166) at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:79) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:81) at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:132) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.handler.DebugHandler.handle(DebugHandler.java:77) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:543) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:939) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:413) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451) Caused by: javax.naming.CommunicationException: Bad file descriptor [Root exception is java.net.SocketException: Bad file descriptor] at com.sun.jndi.ldap.LdapCtx.extendedOperation(LdapCtx.java:3213) at javax.naming.ldap.InitialLdapContext.extendedOperation(InitialLdapContext.java:164) at com.zimbra.cs.account.ldap.ZimbraLdapContext.(ZimbraLdapContext.java:406) ... 42 more Caused by: java.net.SocketException: Bad file descriptor at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) at java.net.SocketOutputStream.write(SocketOutputStream.java:136) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) at com.sun.jndi.ldap.Connection.writeRequest(Connection.java:396) at com.sun.jndi.ldap.LdapClient.extendedOp(LdapClient.java:1172) at com.sun.jndi.ldap.LdapCtx.extendedOperation(LdapCtx.java:3160) ... 44 more
request:	

Body: {
  SearchRequest: {
    _jsns: "urn:zimbraMail",
    limit: 100,
    locale: {
      _content: "en_GB"
     },
    offset: 0,
    query: "in:"Admin's Inbox"",
    sortBy: "dateDesc",
    types: "conversation",
    tz: {
      id: "Europe/London"
     }
   }
 },
Header: {
  context: {
    _jsns: "urn:zimbra",
    account: {
      _content: "martin@clues.ltd.uk",
      by: "name"
     },
    authToken: "(removed)",
    session: {
      _content: 11,
      id: 11
     },
    userAgent: {
      name: "ZimbraWebClient - FF3.0 (Win)",
      version: "6.0.10_GA_2692"
     }
   }
 }

Sometimes I am able to see the contents of a mailbox, but then I get this similar message when clicking on a mail item:

method:	SearchConvRequest
msg:	system failure: IOException:
code:	service.FAILURE
detail:	soap:Receiver
trace:	com.zimbra.common.service.ServiceException: system failure: IOException: ExceptionId:btpool0-16://zimbra.clues.ltd.uk/service/soap/SearchConvRequest:1294077905321:aa8b9e5db59b7757 Code:service.FAILURE at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:248) at com.zimbra.cs.service.mail.SearchConv.handle(SearchConv.java:172) at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:420) at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:274) at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:158) at com.zimbra.soap.SoapServlet.doWork(SoapServlet.java:291) at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:212) at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:181) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166) at com.zimbra.cs.servlet.SetHeaderFilter.doFilter(SetHeaderFilter.java:79) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:81) at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:132) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.handler.DebugHandler.handle(DebugHandler.java:77) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:543) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:939) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:405) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:413) at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451) Caused by: java.net.SocketException: Socket closed at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293) at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:331) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1493) at com.sun.net.ssl.internal.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:103) at com.sun.net.ssl.internal.ssl.Handshaker.kickstart(Handshaker.java:626) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.kickstartHandshake(SSLSocketImpl.java:1240) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1137) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1165) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1149) at com.zimbra.common.net.CustomSSLSocket.startHandshake(CustomSSLSocket.java:90) at com.zimbra.common.net.CustomSSLSocket.getInputStream(CustomSSLSocket.java:341) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:744) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1321) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:386) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396) at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:236) at com.zimbra.common.soap.SoapHttpTransport.invoke(SoapHttpTransport.java:162) at com.zimbra.common.soap.SoapTransport.invoke(SoapTransport.java:356) at com.zimbra.common.soap.SoapTransport.invokeWithoutSession(SoapTransport.java:342) at com.zimbra.cs.service.mail.SearchConv.handle(SearchConv.java:155) ... 35 more
request:	

Body: {
  SearchConvRequest: {
    _jsns: "urn:zimbraMail",
    cid: "691ee733-1e36-4fa1-adff-293704569b54:-548",
    fetch: "691ee733-1e36-4fa1-adff-293704569b54:548",
    html: 1,
    limit: 250,
    locale: {
      _content: "en_GB"
     },
    max: 100000,
    offset: 0,
    query: "in:"Admin's Inbox"",
    read: 1,
    sortBy: "dateDesc",
    tz: {
      id: "Europe/London"
     }
   }
 },
Header: {
  context: {
    _jsns: "urn:zimbra",
    account: {
      _content: "martin@clues.ltd.uk",
      by: "name"
     },
    authToken: "(removed)",
    session: {
      _content: 11,
      id: 11
     },
    userAgent: {
      name: "ZimbraWebClient - FF3.0 (Win)",
      version: "6.0.10_GA_2692"
     }
   }
 }

Google isn't producing anything that seems to be relevant. Obviously this is a bit of a show-stopper for our pilot :/

Any ideas?

Thanks

[phoenix]

Is this server sitting behind a NAT router? Have you set-up a Split DNS and is the configuration correct? You can check the DNS & hosts config by going to that article and running all the commands in the 'Verify...' section.

If it's not a DNS etc. problem you might want to take a look at this bug (see if that's the problem): Bug 42870 – "Bad file descriptor" SocketException causes system failure every several hours

[MartBrooks]

Originally Posted by phoenix

Is this server sitting behind a NAT router? Have you set-up a Split DNS and is the configuration correct? You can check the DNS & hosts config by going to that article and running all the commands in the 'Verify...' section.

If it's not a DNS etc. problem you might want to take a look at this bug (see if that's the problem): Bug 42870 – "Bad file descriptor" SocketException causes system failure every several hours

The server runs directly connected to the Internet with a local firewall.

Thanks for the pointer to that bugzilla item, I shall try upping root's FD limit as suggested. It seems the 6.0.10 installer already added a similar entry for zimibra's FD limit when the upgrade was performed.

[phoenix]

Originally Posted by MartBrooks

The server runs directly connected to the Internet with a local firewall.

I'd still suggest you check the DNS (and hosts file) via the steps in that article, you may also need a Split DNS behind a firewall.

[MartBrooks]

I'm using a non-RFC1918 IP range, so I really don't think DNS has anything to do with it. The server can correctly resolve itself both forwards and backwards. The local filtering rules do nothing with DNS.

[LMStone]

Originally Posted by MartBrooks

I'm using a non-RFC1918 IP range, so I really don't think DNS has anything to do with it. The server can correctly resolve itself both forwards and backwards. The local filtering rules do nothing with DNS.

Different components of Zimbra also use the /etc/hosts file as well as DNS, so please confirm that everything resolves as it should by running the following and letting us know?

host `hostname`
host (logical hostname)
nslookup `hostname`
nslookup (logical hostname)

And then please also confirm that /etc/hosts is formatted correctly and contains:

127.0.0.1 localhost.localdomain localhost
your.server.public.ip servername.domain.tld servername

Hope that helps,
Mark