[SOLVED] Since update from 3.1.4 to 4.0.4, having problems.

[jml75]

Hello,

I upgraded from 3.1.4 to 4.0.4 on a debian sarge server . Upgrade went fine and server worked perfectly right after it.

A few days later, some of my users started to have problems. They could still send emails from there mail client but none would come in.

I checked in the admin console and found that all of there emails were in the deffered queue and even if I requeued them, they would return to the deffered queue.

To test, i recreated their account and emails started to work normally again for them until the next day.

On that day, some of the users that had problems the day before started to have the same problems again, but new ones started to have that problem to. Same problem, able to send emails from mail client but all incoming emails would be stuck in the deffered queue.

I tried there account in the zimbra webmail interface and found out that I could not send emails from there.

I receaved the message : A network error has occured.

And here is the complete error it gave me :

msg - system failure: indexMessage caught IOException
com.zimbra.cs.service.ServiceException: system failure: indexMessage caught IOException
at com.zimbra.cs.service.ServiceException.FAILURE(Ser viceException.java:174)
at com.zimbra.cs.index.Indexer.indexMessage(Indexer.j ava:161)
at com.zimbra.cs.mailbox.Message.reindex(Message.java :421)
at com.zimbra.cs.mailbox.Mailbox.endTransaction(Mailb ox.java:4371)
at com.zimbra.cs.mailbox.Mailbox.addMessageInternal(M ailbox.java:3318)
at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.j ava:3062)
at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.j ava:3034)
at com.zimbra.cs.mailbox.MailSender.sendMimeMessage(M ailSender.java:183)
at com.zimbra.cs.mailbox.MailSender.sendMimeMessage(M ailSender.java:119)
at com.zimbra.cs.operation.SendMsgOperation.callback( SendMsgOperation.java:73)
at com.zimbra.cs.operation.Operation.schedule(Operati on.java:190)
at com.zimbra.cs.service.mail.SendMsg.handle(SendMsg. java:109)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEng ine.java:261)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:162)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:84)
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.jav a:223)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:709)
at com.zimbra.cs.servlet.ZimbraServlet.service(Zimbra Servlet.java:173)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:802)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invo ke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invo ke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke( StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invok e(StandardEngineValve.java:107)
at org.apache.catalina.valves.AccessLogValve.invoke(A ccessLogValve.java:541)
at org.apache.catalina.connector.CoyoteAdapter.servic e(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(H ttp11Processor.java:869)
at org.apache.coyote.http11.Http11BaseProtocol$Http11 ConnectionHandler.processConnection(Http11BaseProt ocol.java:667)
at org.apache.tomcat.util.net.PoolTcpEndpoint.process Socket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThr ead.runIt(LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlR unnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: Could not create index /opt/zimbra/index/0/9/index/0 (directory already exists)
at com.zimbra.cs.index.MailboxIndex.openIndexWriter(M ailboxIndex.java:906)
at com.zimbra.cs.index.MailboxIndex.addDocument(Mailb oxIndex.java:349)
at com.zimbra.cs.index.Indexer.addDocument(Indexer.ja va:354)
at com.zimbra.cs.index.Indexer.addDocument(Indexer.ja va:324)
at com.zimbra.cs.index.Indexer.indexMessage(Indexer.j ava:156)
... 32 more
Caused by: java.io.IOException: Lock obtain timed out: Lock@/opt/zimbra/apache-tomcat-5.5.1...173-write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:58)
at org.apache.lucene.index.IndexWriter.<init>(IndexWr iter.java:223)
at org.apache.lucene.index.IndexWriter.<init>(IndexWr iter.java:173)
at com.zimbra.cs.index.MailboxIndex.openIndexWriter(M ailboxIndex.java:887)
... 36 more

code - service.FAILURE
method - ZmCsfeCommand.prototype.invoke
detail - soap:Receiver

Has any one experienced this?

Like I said, if a recreate those accounts, it works again normally but the issue eventually resurfaces.

Thanx!

[KevinH]

Looks like some indexing problems. Any disk failure or are you using some sort of NFS filesystem? Did you try to just re-index the problem accounts via the admin console?

[jml75]

Thanx Kevin for answering,

By looking at the server, there does not appear to be any disk failure and I do not use any NFS disk for my server.

I did not try the reindex options because I was not aware of it but I will certainly try it.

I will keep you posted with my results.

Thanx for your help

[dkarp]

mystery solved: unclean shutdown causes redo.log to be owned by root:root