Server down: Can't get to zimbra via browser/IMAP

[mcevoys]

Nice, stable environment has been running without incident for weeks.

Today, about an hour ago, the log shows the following:

2006-12-11 12:20:31,203 INFO [http-8080-Processor100] [ip=144.186.66.218;name=mailbox@domain.org;] cache - Initializing folder and tag caches for mailbox 9
2006-12-11 12:20:31,326 FATAL [http-8080-Processor100] [ip=144.186.66.218;name=mailbox@domain.org;] mailbox - Unable to commit database transaction. Forcing server to abort.
com.zimbra.cs.service.ServiceException: system failure: committing database transaction
at com.zimbra.cs.service.ServiceException.FAILURE(Ser viceException.java:174)
at com.zimbra.cs.db.DbPool$Connection.commit(DbPool.j ava:162)
at com.zimbra.cs.mailbox.Mailbox.endTransaction(Mailb ox.java:4388)
at com.zimbra.cs.mailbox.Mailbox.getConfig(Mailbox.ja va:847)
at com.zimbra.cs.im.IMRouter.findPersona(IMRouter.jav a:109)
at com.zimbra.cs.session.Session.<init>(Session.java: 77)
at com.zimbra.cs.session.SoapSession.<init>(SoapSessi on.java:77)
at com.zimbra.cs.session.SessionCache.getNewSession(S essionCache.java:86)
at com.zimbra.soap.ZimbraSoapContext.getSession(Zimbr aSoapContext.java:441)
at com.zimbra.soap.DocumentHandler.getSession(Documen tHandler.java:180)
at com.zimbra.soap.DocumentHandler.getSession(Documen tHandler.java:163)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEng ine.java:256)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:162)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:84)
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.jav a:223)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:709)
at com.zimbra.cs.servlet.ZimbraServlet.service(Zimbra Servlet.java:173)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:802)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invo ke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invo ke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardWrapperValve.invo ke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invo ke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke( StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invok e(StandardEngineValve.java:107)
at org.apache.catalina.valves.AccessLogValve.invoke(A ccessLogValve.java:541)
at org.apache.catalina.connector.CoyoteAdapter.servic e(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(H ttp11Processor.java:869)
at org.apache.coyote.http11.Http11BaseProtocol$Http11 ConnectionHandler.processConnection(Http11BaseProt ocol.java:667)
at org.apache.tomcat.util.net.PoolTcpEndpoint.process Socket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThr ead.runIt(LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlR unnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.sql.SQLException: Communications link failure during commit(). Transaction resolution unknown.
at com.mysql.jdbc.Connection.commit(Connection.java:2 169)
at org.apache.commons.dbcp.DelegatingConnection.commi t(DelegatingConnection.java:248)
at com.zimbra.cs.db.DbPool$Connection.commit(DbPool.j ava:160)
... 31 more
2006-12-11 12:20:32,345 INFO [Thread-264] [] StatsImageServlet - Servlet StatsImageServlet shutting down
2006-12-11 12:20:32,346 INFO [Thread-264] [] soap - Servlet AdminServlet shutting down
2006-12-11 12:20:32,346 INFO [Thread-264] [] TcpServer/7025 - LmtpServer initiating shutdown
2006-12-11 12:20:32,348 INFO [LmtpServer] [] TcpServer/7025 - finished accept loop


I can't get to zimbra via any browser interface...neither as user or admin.
Loading https://domain.org stops at the Zimbra splash screen (browser claims 'done')
IMAP functionality is also down.
zmcontrol status shows all services running happily

_____________

**Edit**

It appears that when I try to access this particular mailbox, it crashes the server.

If I restart zimbra, clear all cookies from the machine trying to access the (fscked) mailbox, and login to a different mailbox, all is well...as long as I don't try to access this one mailbox.

As long as I continue to stay away from this mailbox, all seems fine. If I access it, the server falls down.

Mailbox is large (~200MB), but not active (no mail gets delivered to it...it's archive data only).

Anyone have any ideas on how to 'reindex' the mailbox to make this mailbox usable again?

Problem seems related to an earlier challenge I had here: Network service error

[dkarp]

What you're seeing is that MySQL is throwing an exception every time we try to commit a change to that mailbox. When this happens, the Zimbra server has no idea whether the change actually got applied to the database, so it halts rather than risk going forward with an incorrect view of the mailbox (which could potentially snowball into really bad inconsistencies).

This seems very related to the error you were seeing earlier, and I continue to believe that you were having some serious filesystem weirdness that didn't get completely fixed.