Tomcat issue

[mutinda]

Hello Zimbrans,

Since Tuesday, my zcs (4.5.0, Open Source, FC5) has been erratic. After months of literally hassle free messaging, it's gone beserk. From what I can see in the log, tomcat has something to do with it.

Normally when I have any nagging issue, an upgrade (reinstall actually) solves it. This time, nothing doing. I'm yet to see a posting that resolves it.

I've pasted the log below:

Jul 31 10:39:12 kenyalawreports postfix/lmtp[30182]: 0B29C11FC02D: to=<ajiwaji@kenyalawreports.or.ke>, relay=none, delay=1787, status=deferred (connect to kenyalawreports.or.ke[xxx
.xxx.xxx.xxx]: Connection refused)
Jul 31 10:39:12 kenyalawreports postfix/lmtp[30178]: E78E111FC031: to=<mkitana@kenyalawreports.or.ke>, orig_to=<mkitana@kenyalaw.org>, relay=none, delay=7140, status=deferred (conn
ect to kenyalawreports.or.ke[xxx.xxx.xxx.xxx]: Connection refused)
Jul 31 10:39:12 kenyalawreports postfix/lmtp[30178]: E78E111FC031: to=<rkanyi@kenyalawreports.or.ke>, orig_to=<rkanyi@kenyalaw.org>, relay=none, delay=7140, status=deferred (connec
t to kenyalawreports.or.ke[xxx.xxx.xxx.xxx]: Connection refused)
Jul 31 10:39:12 kenyalawreports zmtomcatmgr[29453]: manager woke up from wait on tomcat/JVM with pid 29454
Jul 31 10:39:12 kenyalawreports zmtomcatmgr[29453]: manager started tomcat/JVM with pid 30459

[mutinda]

Hello Zimbrans,

Since Tuesday, my zcs (4.5.0, Open Source, FC5) has been erratic. After months of literally hassle free messaging, it's gone beserk. From what I can see in the log, tomcat has something to do with it.

Normally when I have any nagging issue, an upgrade (reinstall actually) solves it. This time, nothing doing. I'm yet to see a posting that resolves it.

I've pasted the log below:

Jul 31 10:39:12 kenyalawreports postfix/lmtp[30182]: 0B29C11FC02D: to=<ajiwaji@kenyalawreports.or.ke>, relay=none, delay=1787, status=deferred (connect to kenyalawreports.or.ke[xxx
.xxx.xxx.xxx]: Connection refused)
Jul 31 10:39:12 kenyalawreports postfix/lmtp[30178]: E78E111FC031: to=<mkitana@kenyalawreports.or.ke>, orig_to=<mkitana@kenyalaw.org>, relay=none, delay=7140, status=deferred (conn
ect to kenyalawreports.or.ke[xxx.xxx.xxx.xxx]: Connection refused)
Jul 31 10:39:12 kenyalawreports postfix/lmtp[30178]: E78E111FC031: to=<rkanyi@kenyalawreports.or.ke>, orig_to=<rkanyi@kenyalaw.org>, relay=none, delay=7140, status=deferred (connec
t to kenyalawreports.or.ke[xxx.xxx.xxx.xxx]: Connection refused)
Jul 31 10:39:12 kenyalawreports zmtomcatmgr[29453]: manager woke up from wait on tomcat/JVM with pid 29454
Jul 31 10:39:12 kenyalawreports zmtomcatmgr[29453]: manager started tomcat/JVM with pid 30459

More details: When I use a mailing list to send email, messages to addresses outside my domain get sent. The rest (local) get deferred. View log below:

Jul 31 12:41:18 kenyalawreports postfix/smtp[1089]: 9AFA511F3D30: to=<karanjathomas@yahoo.com>, orig_to=<nclrstaff@kenyalawreports.or.ke>, relay=196.xxx.xxx.xxx[196.xxx.xxx.xxx], delay
=1, status=sent (250 ok: Message 62528369 accepted)
Jul 31 12:41:18 kenyalawreports postfix/smtp[1089]: 9AFA511F3D30: to=<steveguya@yahoo.com>, orig_to=<nclrstaff@kenyalawreports.or.ke>, relay=196.xxx.xxx.xxx[196.xxx.xxx.xxx], delay=1,
status=sent (250 ok: Message 62528369 accepted)
Jul 31 12:41:18 kenyalawreports postfix/smtp[1089]: 9AFA511F3D30: to=<wefwafwa2000@yahoo.com>, orig_to=<nclrstaff@kenyalawreports.or.ke>, relay=196.xxx.xxx.xxx[196.xxx.xxx.xxx], delay=
1, status=sent (250 ok: Message 62528369 accepted)
Jul 31 12:41:33 kenyalawreports postfix/lmtp[1090]: 9AFA511F3D30: to=<aasugah@kenyalawreports.or.ke>, orig_to=<nclrstaff@kenyalawreports.or.ke>, relay=kenyalawreports.or.ke[192.xxx
.xxx.xxx], delay=16, status=deferred (lost connection with kenyalawreports.or.ke[192.xxx
.xxx.xxx] while sending end of data -- message may be sent more than once)
Jul 31 12:41:33 kenyalawreports postfix/lmtp[1090]: 9AFA511F3D30: to=<ahalonyere@kenyalawreports.or.ke>, orig_to=<nclrstaff@kenyalawreports.or.ke>, relay=kenyalawreports.or.ke[192.xxx
.xxx.xxx], delay=16, status=deferred (lost connection with kenyalawreports.or.ke[192.xxx
.xxx.xxx] while sending end of data -- message may be sent more than once)
Jul 31 12:41:33 kenyalawreports postfix/lmtp[1090]: 9AFA511F3D30: to=<ajiwaji@kenyalawreports.or.ke>, orig_to=<nclrstaff@kenyalawreports.or.ke>, relay=kenyalawreports.or.ke[192.xxx
.xxx.xxx], delay=16, status=deferred (lost connection with kenyalawreports.or.ke[192.xxx
.xxx.xxx] while sending end of data -- message may be sent more than once)
Jul 31 12:41:33 kenyalawreports postfix/lmtp[1090]: 9AFA511F3D30: to=<benchhotline@kenyalawreports.or.ke>, orig_to=<nclrstaff@kenyalawreports.or.ke>, relay=kenyalawreports.or.ke[192.xxx
.xxx.xxx], delay=16, status=deferred (lost connection with kenyalawreports.or.ke[192.xxx
.xxx.xxx] while sending end of data -- message may be sent more than once)
Jul 31 12:41:33 kenyalawreports postfix/lmtp[1090]: 9AFA511F3D30: to=<bmbatia@kenyalawreports.or.ke>, orig_to=<nclrstaff@kenyalawreports.or.ke>, relay=kenyalawreports.or.ke[192.xxx
.xxx.xxx], delay=16, status=deferred (lost connection with kenyalawreports.or.ke[192.xxx
.xxx.xxx] while sending end of data -- message may be sent more than once)
Jul 31 12:41:34 kenyalawreports zmtomcatmgr[29453]: manager woke up from wait on tomcat/JVM with pid 32288
Jul 31 12:41:34 kenyalawreports zmtomcatmgr[29453]: manager started tomcat/JVM with pid 1118

Whatever could be the matter?

[dustys]

Do you see any messages in your /opt/zimbra/log/mailbox.log? I'm assuming the logs you posted are /var/log/zimbra.log. Correct me if I'm wrong.

It looks like LMTP isn't listening or accepting connections. Can you telnet to the box on port 7025 (LMTP)?

Is this a single server setup or multiple settings? When we have had this happen, it was due to the attachment indexers giving us fits.

Thanks,
Dusty

[mutinda]

dustys,

The logs I posted are form /var/log/zimbra.log. I only saw your reply after deciding to do a backup and restore so I'll wait until that's through then give a full reply. Hopefully all will be well after. Thanks.

[mutinda]

Ok, some more info:

I did a backup and restore and after getting a few login issues (resolved by doing a fixperms) I am back to where I was. Before I go on, let me answer your questions:

1) I can't telnet using 7025, only 25.
2) I have a single server setup.

As for the logs in /opt/zimbra/log/mailbox.log, I'm not sure what should I look for there... some assistance with this please?

Now that the restore is complete, my server is back to where it was. Email is coming and going except when I send email locally to more than one address, especially using a mailing list.

Second, I believe the problem is something to do with tomcat; I had to restart it once after the restore because it gave me an error.

I have one new error message from the Admin Console which wasn't there berore: I get it when I try to view my mail queue(s):

Message: system failure: server kenyalawreports.or.ke zimbraRemoteManagementPrivateKeyPath (/opt/zimbra/.ssh/zimbra_identity) is not readable
com.zimbra.common.service.ServiceException: system failure: server kenyalawreports.or.ke zimbraRemoteManagementPrivateKeyPath (/opt/zimbra/.ssh/zimbra_identity) is not readable
at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:175)
at com.zimbra.cs.rmgmt.RemoteManager.<init>(RemoteMan ager.java:84)
at com.zimbra.cs.rmgmt.RemoteManager.getRemoteManager (RemoteManager.java:202)
at com.zimbra.cs.service.admin.GetMailQueueInfo.handl e(GetMailQueueInfo.java:55)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEng ine.java:262)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:162)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:84)
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.jav a:223)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:709)
at com.zimbra.cs.servlet.ZimbraServlet.service(Zimbra Servlet.java:162)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:802)
at org.apache.catalina.core.ApplicationFilterChain.in ternalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.do Filter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invo ke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invo ke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke( StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke (ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invok e(StandardEngineValve.java:107)
at org.apache.catalina.valves.AccessLogValve.invoke(A ccessLogValve.java:541)
at org.apache.catalina.connector.CoyoteAdapter.servic e(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(H ttp11Processor.java:869)
at org.apache.coyote.http11.Http11BaseProtocol$Http11 ConnectionHandler.processConnection(Http11BaseProt ocol.java:667)
at org.apache.tomcat.util.net.PoolTcpEndpoint.process Socket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThr ead.runIt(LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlR unnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)

Error code: service.FAILURE
Method: ZmCsfeCommand.prototype.invoke
Details:soap:Receiver

Any clues??