Zimbra stoped working

[antonio.meireles]

I had M3 (build 436) working fine (upgraded from M2) until this morning.

Suddently zimbra started misbehaving. First it doesn 't allow user to login via web mail. It Reports ...

"An unknown application error has occurred. Please correct any errors and retry. If the problem persists, please contact your System Aministrator. (INVALID_PDU)".

Second it *isn't* receiving mail, with the following in the logs...

(delivery temporarily suspended: connect to zimbra.xxx.xxx.xxx [zzz.zzz.zz.zzz]: Connection refused

although a zmcontrol status says that everithing is ok

The only abnormal situation is in the tomcat logs where i find ...

2006-01-18 16:31:00,354 INFO [main] [] RedoLogManager - Starting pre-startup crash recovery
2006-01-18 16:31:00,355 INFO [IndexWritersSweeper] [] MailboxIndex - IndexWritersSweeper thread starting
2006-01-18 16:31:02,174 FATAL [main] [] soap - Unable to start servlet
java.lang.OutOfMemoryError: Requested array size exceeds VM limit
2006-01-18 16:31:02,174 INFO [main] [] [/service] - Marking servlet AdminServlet as unavailable
2006-01-18 16:31:02,175 ERROR [main] [] [/service] - Servlet /service threw load() exception

The box is a Dell 1850 running CentOS with 4GB of RAM and *nothing* changed from yesterday to today :-S


Any HELP will be greatly appreciated

Thanxs in Advance.

António

P.S. I searched in the forums and nome of the problems mentioned seems fits this one

[marcmac]

Sound like tomcat is in a bad way, predicated by some kind of java memory problem.

Stop and restart tomcat:
su - zimbra
tomcat stop
ps auxww | grep tomcat (is it really stopped?)
tomcat start

[antonio.meireles]

Problem persists... after start/stop tomcat, server restart, whatever
Any idea ?
TIA

António

[zimbra@zimbra ~]$ tomcat stop
Using CATALINA_BASE: /opt/zimbra/tomcat
Using CATALINA_HOME: /opt/zimbra/tomcat
Using CATALINA_TMPDIR: /opt/zimbra/tomcat/temp
Using JRE_HOME: /opt/zimbra/java
Waiting ..... trying again
Using CATALINA_BASE: /opt/zimbra/tomcat
Using CATALINA_HOME: /opt/zimbra/tomcat
Using CATALINA_TMPDIR: /opt/zimbra/tomcat/temp
Using JRE_HOME: /opt/zimbra/java
Jan 18, 2006 5:03:18 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl .java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSoc ketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.j ava:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.j ava:364)
at java.net.Socket.connect(Socket.java:507)
at java.net.Socket.connect(Socket.java:457)
at java.net.Socket.<init>(Socket.java:365)
at java.net.Socket.<init>(Socket.java:178)
at org.apache.catalina.startup.Catalina.stopServer(Ca talina.java:394)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.stopServer(B ootstrap.java:320)
at org.apache.catalina.startup.Bootstrap.main(Bootstr ap.java:411)
Waiting ..... trying again
Using CATALINA_BASE: /opt/zimbra/tomcat
Using CATALINA_HOME: /opt/zimbra/tomcat
Using CATALINA_TMPDIR: /opt/zimbra/tomcat/temp
Using JRE_HOME: /opt/zimbra/java
Jan 18, 2006 5:03:23 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl .java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSoc ketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.j ava:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.j ava:364)
at java.net.Socket.connect(Socket.java:507)
at java.net.Socket.connect(Socket.java:457)
at java.net.Socket.<init>(Socket.java:365)
at java.net.Socket.<init>(Socket.java:178)
at org.apache.catalina.startup.Catalina.stopServer(Ca talina.java:394)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.stopServer(B ootstrap.java:320)
at org.apache.catalina.startup.Bootstrap.main(Bootstr ap.java:411)
Waiting ..... trying again
Using CATALINA_BASE: /opt/zimbra/tomcat
Using CATALINA_HOME: /opt/zimbra/tomcat
Using CATALINA_TMPDIR: /opt/zimbra/tomcat/temp
Using JRE_HOME: /opt/zimbra/java
Jan 18, 2006 5:03:29 PM org.apache.catalina.startup.Catalina stopServer
SEVERE: Catalina.stop:
java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl .java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSoc ketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.j ava:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.j ava:364)
at java.net.Socket.connect(Socket.java:507)
at java.net.Socket.connect(Socket.java:457)
at java.net.Socket.<init>(Socket.java:365)
at java.net.Socket.<init>(Socket.java:178)
at org.apache.catalina.startup.Catalina.stopServer(Ca talina.java:394)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(De legatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.catalina.startup.Bootstrap.stopServer(B ootstrap.java:320)
at org.apache.catalina.startup.Bootstrap.main(Bootstr ap.java:411)
Waiting ..... trying again
Giving up - killing 5151
[zimbra@zimbra ~]$ ps auxww | grep tomcat
zimbra 8387 0.0 0.0 1660 476 pts/1 R+ 17:03 0:00 grep tomcat
[zimbra@zimbra ~]$ tomcat start

[marcmac]

restarting tomcat did not help.

Have you rebooted the server?

What is this set to:
zmlocalconfig tomcat_java_options

You've got 4 gigs of ram - is this box running in a virtual server?

Is this a 64 bit machine (more importantly, are you running a 64 bit OS?)

What else is running on the box?

Which version are you running (FOSS or network?)

[antonio.meireles]

a) Yes, I have rebooted the thing
b) [zimbra@zimbra ~]$ zmlocalconfig tomcat_java_options
tomcat_java_options = -client -XX:NewRatio=2
c) this is running as a Xen VM instance. 32bits. canonic CentOS/RHEL4 install
d) the box *only* runs zimbra
e) Running FOSS version

All was working *fine* until today

See plz the private msg that i have sent you.


Thanxs in advance

António

Originally Posted by marcmac

restarting tomcat did not help.

Have you rebooted the server?

What is this set to:
zmlocalconfig tomcat_java_options

You've got 4 gigs of ram - is this box running in a virtual server?

Is this a 64 bit machine (more importantly, are you running a 64 bit OS?)

What else is running on the box?

Which version are you running (FOSS or network?)

[antonio.meireles]

I' m writing here to thank publicly to Marc Mac, and zimbra, for the awesome job he did. We get in touch on MSN, he prontified to log via ssh on my zimbra box, and promply, with another zimbra engineer, identified the problem which turned to be a corrupted (tomcat) redolog.

keep the awesome job.

Cheers, from Portugal

António

[vogon1]

Originally Posted by antonio.meireles

identified the problem which turned to be a corrupted (tomcat) redolog.

Is it known now how the corruption occurred, how it can be prevented, and if it happens how it can be resolved?
Might be useful information here.

Sietse

[marcmac]

Not known yet - but we've got the log file, and we're looking at it.

[Isi]

Zimbra Team,

Was there ever a fix for this? I have recently upgraded from a working install of M3 to the M3 Update and am experiencing this problem.

So far i have done the following:

1. My install slightly deviates from standard by changing the port from 80 to 8080 in /opt/zimbra/<tomcat>/conf/server.xml and the webapps/service/WEB-INF/web.xml files.

2. I have restarted my maching (suggested earlier in the post)

3. I have looked through the logs that I am aware of and don't see any points of concern or errors.

Any thoughts or advise i could use to get to the bottom of this?

exact error i get when trying to login:
"An unknown application error has occurred. Please correct any errors and retry. If the problem persists, please contact your System Aministrator. (INVALID_PDU)"

-- Isi

[Isi]

Zimbra Team,

I am still getting this error:

"An unknown application error has occurred. Please correct any errors and retry. If the problem persists, please contact your System Aministrator. (INVALID_PDU)"

but i think i know why.

I run my server on port 8080 and not the default of 80. To do this I modify a couple of files as suggested else where in the forum and as noted in my post above.

It seems though that after I modify these and start the zimbra service the files are modified back including the ports being changed.

Is this normal procedure? If so why? If not how can i fix it?

Here is a transcript of the process I am taking so you can see exactly what I am doing and exactly what happens (so that you see i am not crazy).

[root@abbadon ~]# service zimbra stop
Host abbadon.ibancsolutions.com
        Stopping antispam...Done
        Stopping antivirus...Done
        Stopping ldap...Done
        Stopping logger...Done
        Stopping mailbox...Done
        Stopping mta...Done
        Stopping snmp...Done
        Stopping spell...Done

[root@abbadon ~]# vi /opt/zimbra/jakarta-tomcat-5.5.7/conf/server.xml

[root@abbadon ~]# egrep '(80|443)' /opt/zimbra/jakarta-tomcat-5.5.7/conf/server.xml
    <Connector port="8080"
        enableLookups="false" redirectPort="443"
    <Connector port="8443"

[root@abbadon ~]# vi /opt/zimbra/jakarta-tomcat-5.5.7/webapps/service/WEB-INF/web.xml 

[root@abbadon ~]# egrep '(80|443)' /opt/zimbra/jakarta-tomcat-5.5.7/webapps/service/WEB-INF/web.xml 
      <param-value>8080, 8443, 7070, 7443</param-value>
      <param-value>8080, 8443, 7070, 7443, 7071</param-value>
      <param-value>8080, 8443, 7070, 7443, 7071</param-value>
      <param-value>8080, 8443, 7070, 7443, 7071</param-value>
      <param-value>8080, 8443, 7070, 7443, 7071</param-value>
      <param-value>8080, 8443, 7070, 7443, 7071</param-value>
      <param-value>8080, 8443, 7070</param-value>
      <param-value>8080, 8443, 7070, 7443, 7071</param-value>

[root@abbadon ~]# service zimbra start
Host abbadon.ibancsolutions.com
        Starting ldap...Done.
        Starting antispam...Done.
        Starting antivirus...Done.
        Starting logger...Done.
        Starting mailbox...Done.
        Starting mta...Done.
        Starting snmp...Done.
        Starting spell...Done.

[root@abbadon ~]# egrep '(80|443)' /opt/zimbra/jakarta-tomcat-5.5.7/conf/server.xml
<Server port="8005" shutdown="SHUTDOWN">
    <Connector port="80"
        enableLookups="false" redirectPort="443"
    <Connector port="443"

[root@abbadon ~]# egrep '(80|443)' /opt/zimbra/jakarta-tomcat-5.5.7/webapps/service/WEB-INF/web.xml 
      <param-value>80, 443, 7070, 7443</param-value>
      <param-value>80, 443, 7070, 7443, 7071</param-value>
      <param-value>80, 443, 7070, 7443, 7071</param-value>
      <param-value>80, 443, 7070, 7443, 7071</param-value>
      <param-value>80, 443, 7070, 7443, 7071</param-value>
      <param-value>80, 443, 7070, 7443, 7071</param-value>
      <param-value>80, 443, 7070</param-value>
      <param-value>80, 443, 7070, 7443, 7071</param-value>

Any feedback or advice is greatly appreciated.

-- Isi