Constant crashes, connection refused on Tomcat restart

[marcmac]

At least one problem is a zimlet that needs to be cleared out:

zmzimletctl undeploy com_zimbra_po

Also - can you verify that zimbraMtaAuthHost and zimbraMtaAuthURL are set correctly?

[Dux T]

Originally Posted by bobby

did you kill -3 <tomcat-process-id> when it hangs and check catalina.out?

Not yet, I'm autorestarting the service on failure now with a cron job that checks the status every 5 minutes ... gee it'd be cool if that was an option with zmmon any ideas on automating getting the PID?

[Dux T]

Originally Posted by marcmac

At least one problem is a zimlet that needs to be cleared out:

zmzimletctl undeploy com_zimbra_po

Also - can you verify that zimbraMtaAuthHost and zimbraMtaAuthURL are set correctly?

OK undeployed... here's the authhost & url

[zimbra@wifi ~]$ zmprov gs wifi.duxarea.com | grep zimbraMtaAuthHost
zimbraMtaAuthHost: wifi.duxarea.com
[zimbra@wifi ~]$ zmprov gs wifi.duxarea.com | grep zimbraMtaAuthURL
zimbraMtaAuthURL: https://wifi.duxarea.com:443/service/soap/

[marcmac]

zmprov gs wifi.duxarea.com | grep zimbra zimbraMailMode - should be https or mixed. If it's not, reset the auth host:
zmprov ms wifi.duxarea.com zimbraMtaAuthHost wifi.duxarea.com

[Dux T]

Originally Posted by marcmac

zmprov gs wifi.duxarea.com | grep zimbra

zimbraMailMode - should be https or mixed. If it's not, reset the auth host:
zmprov ms wifi.duxarea.com zimbraMtaAuthHost wifi.duxarea.com

zimbraMailMode: mixed

[Interkey JeffG]

Here's a stacktrace. There's a bunch of stuff about locking and waiting in here on just about every thread; this looks like some kind of deadlock issue. I'll leave it to you guys to figure out what's up.

I found a post about a previous issue regarding deadlock:

Originally Posted by dkarp

The problem (at least in the second dump) is that we're doing new-message notification to an IMAP connection running the IDLE command, and the write is blocking for some reason. Unfortunately, the lock on the Mailbox is being held by this thread, so everything else backs up waiting for the write to finish and the lock to be released. And the write never finishes.

This will almost certainly go away when we switch to our new IMAP/POP/LMTP framework, as it does network writes from a different thread pool. For now, we'll see what we can do to figure out a temporary workaround.

Thanks very much for the full thread dump!

It doesn't seem the issues are related, however. Tomcat as a whole is going down here.

[KevinH]

Originally Posted by Interkey JeffG

Here's a stacktrace. There's a bunch of stuff about locking and waiting in here on just about every thread; this looks like some kind of deadlock issue. I'll leave it to you guys to figure out what's up.

I found a post about a previous issue regarding deadlock:

It doesn't seem the issues are related, however. Tomcat as a whole is going down here.

Yep, seems you hit a deadlock in the IMAP login code. What version are you running? If your not running 3.0.1 please upgrade as in previous releases we fixed a few related deadlocks in the IMAP server code.

[Dux T]

Originally Posted by Dux T

zimbraMailMode: mixed

My crashes have reduced quite a bit by adding an extra GB of ram but they still occur ... now I also get a crash where everything shows up as "Running" but no one can log on to the webclient or IMAP through outlook.

The original crashes still happen but heres the catalina.out from the "silent" crash, I added cp to c.o.$$ in my zmrestart script

There was also output from infoTen.pl which is supposed to show CPU usage but it had a wierd error:

idle time on this box has gone backwards, and some old loaddata has been lost.

normally it looks like:

CPU data for wifi.duxarea.com User 7.22 % - System 1.38 % - Idle 91.40

[KevinH]

Dux, What version do you have? Seems you hit a deadlock in the session cache. How many users, and what interfaces? POP? IMAP? Outlook? Web UI?

[Dux T]

Originally Posted by KevinH

Dux, What version do you have? Seems you hit a deadlock in the session cache. How many users, and what interfaces? POP? IMAP? Outlook? Web UI?

Ver 3.01
76 total users
Most are IMAP via Outlook some are Web UI a couple via POP3 in outlook and outlook express ... I use everything (Outlook, Thunderbird, Web UI, Kontact) but prefer the Web UI

[zimbra@wifi ~]$ zmmtareport
Generating report


2922 messages found for 5364 recipients (264121)
351095718 total bytes
120155.96 bytes/msg
1.84 recipients/msg
90.39 delay/msg (sec)
41 bounced deliveries
137 deferred deliveries

I saw the other post regarding a Deadlock issue KevinH suggested increasing the number of IMAP threads to perhaps 4 / User ... would that apply here? Should POP be done the same way for the same ratio?