5.011 Bug - zmlogger fails to stop

[dwmtractor]

Hey everyone,

I'm putting this out there because it could drive a bunch of you nuts trying to debug something we've already discovered. . .on 5.011, when you do a zmcontrol stop, the zmlogger process doesn't stop. Each stop and restart (and for those who use open source backups, that's once a night) will leave you with a new "ghost" instance of zmlogger still running on your system, as you can see by doing a

ps ax | grep zmlogger

You can safely kill those extra processes (although I'd recommend doing it with zimbra services down unless you are absolutely sure which are the ghosts and which is the real thing).

However, if you fail to kill them, the number of running processes builds up until one day you try to log onto the admin gui and you get this familiar error:

Message: system failure: getting database logger connection Error code: service.FAILURE Method: GetServiceStatusRequest Details:soap:Receiver

NOTE, this is not the only cause of this error, but it is the one I saw today. Killing off excess zmlogger processes will eliminate the error (as will a reboot of the system if you prefer the Microsoft way )

I have submitted Bug 33730 on this issue and will let you know more if I hear/discover it.

[Paolo]

Same error with me.
5.0.11 Network edition.

I've also noticed this:

logmysql
ERROR 1040 (08004): Too many connections

Paolo

[Paolo]

A possible workaround for now could be to edit this file /opt/zimbra/conf/my.logger.cnf
and change from 10 to more this line:
max_connections = 10

Paolo

[jsnapp]

I added the follow to the bug report here Bug 33730 Updated

The 5.0.10 version of /opt/zimbra/bin/zmlogswatchctl "stop" function uses "kill -TERM" which correctly shuts down the logswatch process and it's child .swatch_script.??? process.

The 5.0.11 version of /opt/zimbra/bin/zmlogswatchctl "stop" function uses "kill -9" which shuts down the logswatch process but does NOT shut down it's child .swatch_script.??? process.

This change incorrectly allows the .swatch_script.??? processes to remain active which keeps corresponding zmlogger processes active which take up logger's mysql server connections until there are no more connections to be spared.

When no more connections are available errors can show up such as the following error seen when logging into Zimbra's admin web interface:

Server error encountered Message: system failure: getting database logger connection Error code: service.FAILURE Method: GetServiceStatusRequest Details:soap:Receiver

To fix the problem you must change "kill -9" to "kill -15" (or it's equivalent "kill -TERM") in the file /opt/zimbra/bin/zmlogswatchctl

Then as the zimbra user run the following command:
zmloggerctl restart

You will probably still have to kill old swatch_script processes and manually delete their corresponding .swatch_script.??? files from /tmp to cleanup the mess.

[dwmtractor]

I would not recommend we mark this solved until the solution has been verified by Zimbra. Not because I don't trust you, Jeremiah (and I appreciate your identifying this issue) but because I don't know if someone made the change from "kill -TERM" to "kill -9" to fix a different problem, or whether it was just an error. If the former, then merely reverting to the .10 syntax isn't moving us forward, just whacking a different mole.

FWIW

[dwmtractor]

In fact, this fix is not workable for anyone who uses backup scripts or other shell scripts that kill Zimbra using an su - zimbra type command. Calling zmcontrol stop from a script, with kill -TERM results in a failure too:

root@mail:/opt/zimbra/bin# su - zimbra "/opt/zimbra/bin/zmcontrol stop"
Host mail.mydomain.net
        Stopping stats...Done
        Stopping mta...Done
        Stopping spell...Done
        Stopping snmp...Done
        Stopping archiving...Done
        Stopping antivirus...Done
        Stopping antispam...Done
        Stopping imapproxy...Done
        Stopping mailbox...Done
        Stopping logger...FAILED
Stopping mysqld...done.
Stopping logswatch...failed.


        Stopping ldap...Done
root@mail:/opt/zimbra/bin# ps ax | grep zmlog
14040 ?        R      0:01 /usr/bin/perl /opt/zimbra/libexec/zmlogger
 8685 pts/0    S+     0:00 grep zmlog

So something needs to forcibly kill these processes, but it needs to find ALL of them and kill them. . .

[phoenix]

This command should kill all remaining processes:

kill -9 `ps -u zimbra -o "pid="`

[dwmtractor]

Originally Posted by phoenix

This command should kill all remaining processes:

kill -9 `ps -u zimbra -o "pid="`

Yeah, I'm just going to add that to my backup script. Thanks, Bill!

[lzmarine]

I see similar behavior described in thread: http://www.zimbra.com/forums/install...rade-zcs5.html. I'm investigating if the kill change fixes my issue.

[LMStone]

Curious if anyone has tried the command switch I discovered listed in the Admin Guide:

zmcontrol shutdown

We are waiting for 5.0.12 so can't test this for you.

Hope that helps,
Mark