The mailbox and mta dies in FC4 GA version

[drewage]

sh -x bin/zmmtactl status
+ BASE=/opt/zimbra
+ SCRIPTS='zmmtaconfigctl postfix zmsaslauthdctl'
+ case "$1" in
+ STATUS=0
+ for i in '$SCRIPTS'
+ /opt/zimbra/bin/zmmtaconfigctl status
+ R=0
+ '[' 0 -ne 0 ']'
+ for i in '$SCRIPTS'
+ /opt/zimbra/bin/postfix status
+ R=0
+ '[' 0 -ne 0 ']'
+ for i in '$SCRIPTS'
+ /opt/zimbra/bin/zmsaslauthdctl status
cat: /opt/zimbra/cyrus-sasl-2.1.21.ZIMBRA/state/saslauthd.pid: No such file or directory
+ R=1
+ '[' 1 -ne 0 ']'
+ STATUS=1
+ exit 1

sh -x bin/zmmailboxctl status
+ BASE=/opt/zimbra
+ SCRIPTS='zmmtaconfigctl tomcat mysql.server'
+ STATUS=0
+ case "$1" in
+ for i in '$SCRIPTS'
+ '[' zmmtaconfigctl = mysql.server ']'
+ /opt/zimbra/bin/zmmtaconfigctl status
+ R=0
+ '[' 0 -ne 0 ']'
+ for i in '$SCRIPTS'
+ '[' tomcat = mysql.server ']'
+ /opt/zimbra/bin/tomcat status
+ R=1
+ '[' 1 -ne 0 ']'
+ STATUS=1
+ for i in '$SCRIPTS'
+ '[' mysql.server = mysql.server ']'
+ /opt/zimbra/bin/mysqladmin status
+ R=0
+ '[' 0 -ne 0 ']'
+ exit 1

ps auxww | grep tomcat
zimbra    1916  0.0  0.0  2356  460 pts/3    R+   11:40   0:00 grep tomcat

tomcat kill -QUIT
Error: no pid in /opt/zimbra/log/tomcat.pid

The most recent stacktrace is listed as file type unknown, while all the others are shown as 'plain text document' If I try to force gedit to open it it says, it cannot automatically detect the character coding.

Hardware is on the slim side, 1ghz, but does have 1gb ram. OS is CentOS 4.2 with Zimbra GA upgraded from the previous milestone release.

[marcmac]

Ok, so tomcat is dying, which is taking down sasl.

Anything in /var/log/messages or /var/log/zimbra.log around the time of the crash? Anything in /opt/zimbra/tomcat/logs/catalina.out (if you've restarted, this will be overwritten.)

Can you send us the stacktrace that can't be opened?

How's the disk space on this box?

[drewage]

There is no /var/log/messages... /var/log/zimbra.log is full of status messages dated today, nothing back to 02-12. /opt/zimbra/tomcat/logs/catalina.out is a 0byte file.

There is plenty of diskspace, its a 40gb HD with CentOS and zimbra basically...

I copied the stacktrace over to a windows box and renamed it to a .txt file...it is attached.

Let me know what's next...

[marcmac]

the attachment is corrupted, I can't unzip it.

[drewage]

Full thread dump Java HotSpot(TM) Client VM (1.5.0_05-b05 mixed mode):

"Timer-1" daemon prio=1 tid=0x08108940 nid=0x78d9 in Object.wait() [0x9e689000..0x9e68a0b0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xa890c5c8> (a java.util.TaskQueue)
	at java.lang.Object.wait(Object.java:474)
	at java.util.TimerThread.mainLoop(Timer.java:483)
	- locked <0xa890c5c8> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:462)

"Thread-163" daemon prio=1 tid=0x08108550 nid=0x78d8 waiting on condition [0x9e70a000..0x9e70b030]
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1772)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
	at com.zimbra.cs.im.IMRouter.run(IMRouter.java:146)
	at java.lang.Thread.run(Thread.java:595)

"IncomingDirectorySweeper" prio=1 tid=0x08231318 nid=0x78be in Object.wait() [0x9e78b000..0x9e78bfb0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xa8973d98> (a com.zimbra.cs.store.FileBlobStore$IncomingDirectorySweeper)
	at com.zimbra.cs.store.FileBlobStore$IncomingDirectorySweeper.run(FileBlobStore.java:425)
	- locked <0xa8973d98> (a com.zimbra.cs.store.FileBlobStore$IncomingDirectorySweeper)

"TcpWatchdog-1" daemon prio=1 tid=0x08228c58 nid=0x78b5 in Object.wait() [0x9e88d000..0x9e88deb0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xa8976788> (a EDU.oswego.cs.dl.util.concurrent.ClockDaemon)
	at java.lang.Object.wait(Object.java:474)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon.nextTask(Unknown Source)
	- locked <0xa8976788> (a EDU.oswego.cs.dl.util.concurrent.ClockDaemon)
	at EDU.oswego.cs.dl.util.concurrent.ClockDaemon$RunLoop.run(Unknown Source)
	at java.lang.Thread.run(Thread.java:595)

"http-7071-Processor50" daemon prio=1 tid=0xa39432a0 nid=0x7758 runnable [0x9f170000..0x9f171130]
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
	- locked <0xa858de60> (a java.net.SocksSocketImpl)
	at java.net.ServerSocket.implAccept(ServerSocket.java:450)
	at com.sun.net.ssl.internal.ssl.SSLServerSocketImpl.accept(SSLServerSocketImpl.java:259)
	at org.apache.tomcat.util.net.jsse.JSSESocketFactory.acceptSocket(JSSESocketFactory.java:110)
	at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:406)
	at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
	at java.lang.Thread.run(Thread.java:595)

"http-7071-Processor49" daemon prio=1 tid=0xa3942538 nid=0x7757 in Object.wait() [0x9f1f1000..0x9f1f20b0]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xa897ae50> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:474)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
	- locked <0xa897ae50> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:595)

"http-7071-Processor48" daemon prio=1 tid=0xa39417d0 nid=0x7756 in Object.wait() [0x9f272000..0x9f273030]
	at java.lang.Object.wait(Native Method)
	- waiting on <0xa897aee8> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Object.wait(Object.java:474)
	at

Below that it appears to repeat the lines:

at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
	- locked <0xa897b738> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
	at java.lang.Thread.run(Thread.java:595)

"http-7071-Processor33" daemon prio=1 tid=0xa3934f28 nid=0x7743 in Object.wait() [0x9fa01000..0x9fa020b0]

I also reattached the zip file...

[marcmac]

we'll take a look at this.

[drewage]

Thanks!

Can I restart it now or is there anything other information you guys want ?

[marcmac]

go ahead and restart.

[drewage]

Down again....

[marcmac]

/var/log/messages, /var/log/zimbra.log, df -h, and a stacktrace?