Stats stop reason?

[bhwong]

Our Zimbra Server stats services used to keep stopping every few days in the past. We resolve it temporary by restarting the services:

zmcontrol stop
zmcontrol start

At one time, restarting the service could not get it running again and it was found that wrong permission caused it: Stats keep stopping

The weird thing is why is the wrong permission suddenly happen when we did not do any permission amendment?

Yesterday, our stats got stopped again. I search the forum and found that swap storage may cause this: STATS service stopped and cannot run

Is this normal?

btw, is it possible to restart just the stats service without having to stop all the services? This is to minimize affecting our users even if it's just a short downtime.

[Hivos]

Not much help, but I've given up on stats in our 6 server ZCS6 deployment. It will die sooner or later, no matter what you do (I've tried, really).

Otherwise, I'm a perfectly happy ZCS admin by the way

[fyd]

Originally Posted by bhwong

btw, is it possible to restart just the stats service without having to stop all the services? This is to minimize affecting our users even if it's just a short downtime.

su - zimbra ; zmstatctl stop/start

[bhwong]

Thanks fyd. However, I can't restart stats service this time...again. And it's not permission or swap storage issue. What could be the problem this time? Do I have to restart the whole server just to get it up?

Maybe I should just give up on this like Hivos did.

[fyd]

Originally Posted by bhwong

Thanks fyd. However, I can't restart stats service this time...again. And it's not permission or swap storage issue. What could be the problem this time? Do I have to restart the whole server just to get it up?

Maybe I should just give up on this like Hivos did.

I think you shouldn't restart the whole server but look into what could be the problem with stats. Did you find anything in mailbox.log, or zimbra.log when the restart is done?

[bhwong]

Here's the log from Zimbra.log just minutes before the stats is stopped:

May 31 23:59:13 zimbra postfix/smtpd[11861]: warning: 118.68.159.134: hostname adsl-dynamic-pool-xxx.hcm.fpt.vn verification failed: Name or service not known
May 31 23:59:13 zimbra postfix/smtpd[11861]: connect from unknown[118.68.159.134]
May 31 23:59:15 zimbra postfix/smtpd[11861]: NOQUEUE: reject: RCPT from unknown[118.68.159.134]: 554 5.7.1 Service unavailable; Client host [118.68.159.134] blocked using zen.spamhaus.org; http://www.spamhaus.org/query/bl?ip=118.68.159.134; from=<boomqi267@remni-avtovoz.ru> to=<biopolis3@crescendas.sg> proto=ESMTP helo=<dynamic-pool-xxx.hcm.fpt.vn>
May 31 23:59:21 zimbra postfix/smtpd[11861]: lost connection after DATA from unknown[118.68.159.134]
May 31 23:59:21 zimbra postfix/smtpd[11861]: disconnect from unknown[118.68.159.134]
May 31 23:59:49 zimbra postfix/smtpd[11861]: warning: 118.68.159.134: hostname adsl-dynamic-pool-xxx.hcm.fpt.vn verification failed: Name or service not known
May 31 23:59:49 zimbra postfix/smtpd[11861]: connect from unknown[118.68.159.134]
May 31 23:59:49 zimbra postfix/smtpd[11861]: NOQUEUE: reject: RCPT from unknown[118.68.159.134]: 554 5.7.1 Service unavailable; Client host [118.68.159.134] blocked using zen.spamhaus.org; http://www.spamhaus.org/query/bl?ip=118.68.159.134; from=<crabbiestn941@tisasoft.ru> to=<biopolis3@crescendas.sg> proto=ESMTP helo=<dynamic-pool-xxx.hcm.fpt.vn>
May 31 23:59:50 zimbra postfix/smtpd[11861]: lost connection after DATA from unknown[118.68.159.134]
May 31 23:59:50 zimbra postfix/smtpd[11861]: disconnect from unknown[118.68.159.134]
Jun 1 00:00:02 zimbra zimbramon[17074]: 17074:info: 2011-06-01 00:00:01, QUEUE: 44 7
Jun 1 00:00:07 zimbra zmmailboxdmgr[18036]: status requested
Jun 1 00:00:07 zimbra zmmailboxdmgr[18036]: status OK
Jun 1 00:00:19 zimbra zimbramon[3603]: 3603:err: Service status change: zimbra.dc1.crescendas.com stats changed from running to stopped

[bhwong]

zmstat.out is filled with:

Use of uninitialized value in subtraction (-) at /opt/zimbra/libexec/zmstat-allprocs line 118.

and:

sh: /opt/zimbra/zmstat//2011-05-31/xxx.csv.gz: Permission denied
Unable to move /opt/zimbra/zmstat/xxx.csv to /opt/zimbra/zmstat//2011-05-31/xxx.csv.gz at /opt/zimbra/zimbramon/lib/Zimbra/Mon/Zmstat.pm line 256.

[bhwong]

Ok, problem solved. It's permission issue again: /opt/zimbra/zmstat/ wrong permissions

I notice 2011-05-31 and all June folders are having root:root instead of Zimbra:Zimbra permission.

I can start stats after typing this command as root:

chown -R zimbra.zimbra /opt/zimbra/zmstat/2010-*

[fyd]

Originally Posted by bhwong

zmstat.out is filled with:

Use of uninitialized value in subtraction (-) at /opt/zimbra/libexec/zmstat-allprocs line 118.

and:

sh: /opt/zimbra/zmstat//2011-05-31/xxx.csv.gz: Permission denied
Unable to move /opt/zimbra/zmstat/xxx.csv to /opt/zimbra/zmstat//2011-05-31/xxx.csv.gz at /opt/zimbra/zimbramon/lib/Zimbra/Mon/Zmstat.pm line 256.

Good find bhwong!!

[bhwong]

Our stats stop running again! This time, I found out that zmstat-fd-real.pid is owned by root:root under /opt/zimbra/zmstat/pid and despite changing ownership to zimbra:zimbra, whenever I execute zmstatctl start, it will automatically changed back to root:root! So frustrating!

zmstatctl status reveal this:

process xxxx in /opt/zimbra/zmstat/pid/zmstat-df.pid not running.

btw, heard that this bug is resolved in version 6.0.7. Anyone can verify this?