Network service error

[mcevoys]

Recently upgraded to FC5/ZimbraOS 4.04

Having a hard time tracking down the source of a 'network service error' I get when opening an inbox:

msg - system failure: java.lang.ClassFormatError: Invalid start_pc 2164 in LocalVariableTable in class file org/cyberneko/html/HTMLScanner$ContentScanner
com.zimbra.cs.service.ServiceException: system failure: java.lang.ClassFormatError: Invalid start_pc 2164 in LocalVariableTable in class file org/cyberneko/html/HTMLScanner$ContentScanner
at com.zimbra.cs.service.ServiceException.FAILURE(Ser viceException.java:174)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEng ine.java:272)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:162)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:84)
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.jav a:223)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:709)
at com.zimbra.cs.servlet.ZimbraServlet.service(Zimbra Servlet.java:173)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:802)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invo ke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invo ke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke( StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invok e(StandardEngineValve.java:107)
at org.apache.catalina.valves.AccessLogValve.invoke(A ccessLogValve.java:541)
at org.apache.catalina.connector.CoyoteAdapter.servic e(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(H ttp11Processor.java:869)
at org.apache.coyote.http11.Http11BaseProtocol$Http11 ConnectionHandler.processConnection(Http11BaseProt ocol.java:667)
at org.apache.tomcat.util.net.PoolTcpEndpoint.process Socket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThr ead.runIt(LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlR unnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.ClassFormatError: Invalid start_pc 2164 in LocalVariableTable in class file org/cyberneko/html/HTMLScanner$ContentScanner
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java :620)
at java.security.SecureClassLoader.defineClass(Secure ClassLoader.java:124)
at org.apache.catalina.loader.WebappClassLoader.findC lassInternal(WebappClassLoader.java:1779)
at org.apache.catalina.loader.WebappClassLoader.findC lass(WebappClassLoader.java:866)
at org.apache.catalina.loader.WebappClassLoader.loadC lass(WebappClassLoader.java:1319)
at org.apache.catalina.loader.WebappClassLoader.loadC lass(WebappClassLoader.java:1198)
at java.lang.ClassLoader.loadClassInternal(ClassLoade r.java:319)
at org.cyberneko.html.HTMLScanner.(HTMLScanner.java:434)
at org.cyberneko.html.HTMLConfiguration.(HTMLConfiguration.java:169)
at com.zimbra.cs.html.HtmlDefang.defang(HtmlDefang.ja va:93)
at com.zimbra.cs.html.HtmlDefang.defang(HtmlDefang.ja va:70)
at com.zimbra.cs.html.HtmlDefang.defang(HtmlDefang.ja va:58)
at com.zimbra.cs.service.mail.ToXML.addContent(ToXML. java:1237)
at com.zimbra.cs.service.mail.ToXML.addParts(ToXML.ja va:1188)
at com.zimbra.cs.service.mail.ToXML.addParts(ToXML.ja va:1200)
at com.zimbra.cs.service.mail.ToXML.encodeMessageAsMP (ToXML.java:581)
at com.zimbra.cs.service.mail.GetMsg.handle(GetMsg.ja va:74)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEng ine.java:261)
... 21 more

code - service.FAILURE
method - ZmCsfeCommand.prototype.invoke
detail - soap:Receiver


Any ideas?

[mcevoys]

After a short period of time, even an F5 refresh of the webpage results in a network service error:

msg - system failure: fetching search metadata
com.zimbra.cs.service.ServiceException: system failure: fetching search metadata
at com.zimbra.cs.service.ServiceException.FAILURE(Ser viceException.java:174)
at com.zimbra.cs.db.DbMailItem.search(DbMailItem.java :1888)
at com.zimbra.cs.index.DBQueryOperation.getNextChunk( DBQueryOperation.java:631)
at com.zimbra.cs.index.DBQueryOperation.peekNext(DBQu eryOperation.java:438)
at com.zimbra.cs.index.QueryOperation.hasNext(QueryOp eration.java:184)
at com.zimbra.cs.index.ItemPreloadingGrouper.bufferHi ts(ItemPreloadingGrouper.java:61)
at com.zimbra.cs.index.BufferingResultsGrouper.hasNex t(BufferingResultsGrouper.java:74)
at com.zimbra.cs.index.MsgQueryResults.internalGetNex tHit(MsgQueryResults.java:61)
at com.zimbra.cs.index.MsgQueryResults.bufferNextHit( MsgQueryResults.java:107)
at com.zimbra.cs.index.MsgQueryResults.peekNext(MsgQu eryResults.java:128)
at com.zimbra.cs.index.ZimbraQueryResultsImpl.hasNext (ZimbraQueryResultsImpl.java:69)
at com.zimbra.cs.index.HitIdGrouper.bufferHits(HitIdG rouper.java:67)
at com.zimbra.cs.index.BufferingResultsGrouper.getNex t(BufferingResultsGrouper.java:103)
at com.zimbra.cs.index.ResultsPager.forward(ResultsPa ger.java:182)
at com.zimbra.cs.index.ResultsPager.(ResultsPager.java:118)
at com.zimbra.cs.index.ResultsPager.create(ResultsPag er.java:65)
at com.zimbra.cs.service.mail.Search.handle(Search.ja va:84)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEng ine.java:261)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:162)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:84)
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.jav a:223)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:709)
at com.zimbra.cs.servlet.ZimbraServlet.service(Zimbra Servlet.java:173)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:802)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invo ke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invo ke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke( StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invok e(StandardEngineValve.java:107)
at org.apache.catalina.valves.AccessLogValve.invoke(A ccessLogValve.java:541)
at org.apache.catalina.connector.CoyoteAdapter.servic e(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(H ttp11Processor.java:869)
at org.apache.coyote.http11.Http11BaseProtocol$Http11 ConnectionHandler.processConnection(Http11BaseProt ocol.java:667)
at org.apache.tomcat.util.net.PoolTcpEndpoint.process Socket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThr ead.runIt(LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlR unnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.sql.SQLException: Communications link failure due to underlying exception:

** BEGIN NESTED EXCEPTION **

java.io.EOFException

STACKTRACE:

java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1905 )
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO. java:2351)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.ja va:2862)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:15 71)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java :1666)
at com.mysql.jdbc.Connection.execSQL(Connection.java: 2994)
at com.mysql.jdbc.PreparedStatement.executeInternal(P reparedStatement.java:936)
at com.mysql.jdbc.PreparedStatement.executeQuery(Prep aredStatement.java:1030)
at org.apache.commons.dbcp.DelegatingPreparedStatemen t.executeQuery(DelegatingPreparedStatement.java:20 5)
at com.zimbra.cs.db.DbMailItem.search(DbMailItem.java :1883)
at com.zimbra.cs.index.DBQueryOperation.getNextChunk( DBQueryOperation.java:631)
at com.zimbra.cs.index.DBQueryOperation.peekNext(DBQu eryOperation.java:438)
at com.zimbra.cs.index.QueryOperation.hasNext(QueryOp eration.java:184)
at com.zimbra.cs.index.ItemPreloadingGrouper.bufferHi ts(ItemPreloadingGrouper.java:61)
at com.zimbra.cs.index.BufferingResultsGrouper.hasNex t(BufferingResultsGrouper.java:74)
at com.zimbra.cs.index.MsgQueryResults.internalGetNex tHit(MsgQueryResults.java:61)
at com.zimbra.cs.index.MsgQueryResults.bufferNextHit( MsgQueryResults.java:107)
at com.zimbra.cs.index.MsgQueryResults.peekNext(MsgQu eryResults.java:128)
at com.zimbra.cs.index.ZimbraQueryResultsImpl.hasNext (ZimbraQueryResultsImpl.java:69)
at com.zimbra.cs.index.HitIdGrouper.bufferHits(HitIdG rouper.java:67)
at com.zimbra.cs.index.BufferingResultsGrouper.getNex t(BufferingResultsGrouper.java:103)
at com.zimbra.cs.index.ResultsPager.forward(ResultsPa ger.java:182)
at com.zimbra.cs.index.ResultsPager.(ResultsPager.java:118)
at com.zimbra.cs.index.ResultsPager.create(ResultsPag er.java:65)
at com.zimbra.cs.service.mail.Search.handle(Search.ja va:84)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEng ine.java:261)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:162)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:84)
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.jav a:223)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:709)
at com.zimbra.cs.servlet.ZimbraServlet.service(Zimbra Servlet.java:173)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:802)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invo ke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invo ke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke( StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invok e(StandardEngineValve.java:107)
at org.apache.catalina.valves.AccessLogValve.invoke(A ccessLogValve.java:541)
at org.apache.catalina.connector.CoyoteAdapter.servic e(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(H ttp11Processor.java:869)
at org.apache.coyote.http11.Http11BaseProtocol$Http11 ConnectionHandler.processConnection(Http11BaseProt ocol.java:667)
at org.apache.tomcat.util.net.PoolTcpEndpoint.process Socket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThr ead.runIt(LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlR unnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)


** END NESTED EXCEPTION **



Last packet sent to the server was 438 ms ago.

Query being executed when exception was thrown:

SELECT id, index_id, type, date, mi.id, mi.type, mi.parent_id, mi.folder_id, mi.index_id, mi.imap_id, mi.date, mi.size, mi.volume_id, mi.blob_digest, mi.unread, mi.flags, mi.tags, mi.subject, mi.metadata, mi.mod_metadata, mi.change_date, mi.mod_content FROM mailbox4.mail_item mi WHERE (type IN (5) AND folder_id IN (2) ) ORDER BY date DESC LIMIT 0, 103
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO. java:2563)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.ja va:2862)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:15 71)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java :1666)
at com.mysql.jdbc.Connection.execSQL(Connection.java: 2994)
at com.mysql.jdbc.PreparedStatement.executeInternal(P reparedStatement.java:936)
at com.mysql.jdbc.PreparedStatement.executeQuery(Prep aredStatement.java:1030)
at org.apache.commons.dbcp.DelegatingPreparedStatemen t.executeQuery(DelegatingPreparedStatement.java:20 5)
at com.zimbra.cs.db.DbMailItem.search(DbMailItem.java :1883)
... 37 more

code - service.FAILURE
method - ZmCsfeCommand.prototype.invoke
detail - soap:Receiver

[dkarp]

Looks like you're getting corruption in a bunch of places. Any chance you're having a disk fail?

[mcevoys]

Nope..disks show clean. We did have a hard power down a few days ago.

Any way I can correct the corrupted files? (I tried a reinstall, but if doesn't solve the problem).

[jholder]

Try running the zmfixperms script found in libexec. You have to run it as root.

If that doesn't work, try to re-install.

[mcevoys]

Not sure why, but it's been stable with no issues for the past 24 hours.

Perhaps just discussing the problem solved it

The reinstall (upgrade) process didn't fix it at the time, but all is well now.

Thanks