Service stats stopped on xxx.xxx.net

[rwcanary]

I am trying to figure out why I am getting this error and what dose it mean. I have found much discussion in previous threads, but they were all related to pre 5.x and supposedly it was fixed in 5.x

If I shutdown Zimbra and restart it, then it comes back alright. For about a day then happens agian.

May 9 03:11:06 mchn39 zimbramon[3801]: 3801:err: Service status change: xxx.xxx.net stats changed from running to stopped

I haven't found anything in the log files.

[cadman]

I am having this same problem, any suggestions from anyone on where to look for possible issues?

[heinzg]

hi there!

this might sound a bit simple but how are you stopping and starting Zimbra? please post all steps from the point you login to linux, though you stop and start procedures, including std. output

cheers
Heinzg

[caio80]

same here too. Already posted this a while ago, but we couldn't come up with an answer. Logs are not helping either (or at least, they are not helping ME... :-(

[rwcanary]

I do a "service zimbra stop"
and a "service zimbra start"

[caio80]

Looks like it doesn't survive to crontab job...


May 17 01:02:05 <myhostname> zimbramon[21716]: 21716:info: 2008-05-17 01:02:01, STATUS: <myhostname>: ldap: Running
May 17 01:02:05 <myhostname> zimbramon[21716]: 21716:info: 2008-05-17 01:02:01, STATUS: <myhostname>: logger: Running
May 17 01:02:05 <myhostname> zimbramon[21716]: 21716:info: 2008-05-17 01:02:01, STATUS: <myhostname>: mailbox: Running
May 17 01:02:05 <myhostname> zimbramon[21716]: 21716:info: 2008-05-17 01:02:01, STATUS: <myhostname>: mta: Running
May 17 01:02:05 <myhostname> zimbramon[21716]: 21716:info: 2008-05-17 01:02:01, STATUS: <myhostname>: snmp: Running
May 17 01:02:05 <myhostname> zimbramon[21716]: 21716:info: 2008-05-17 01:02:01, STATUS: <myhostname>: stats: Running
May 17 01:02:08 <myhostname> zimbramon[22253]: 22253:info: Stopping services
May 17 01:02:08 <myhostname> zimbramon[22253]: 22253:info: Stopping stats
May 17 01:02:09 <myhostname> zimbramon[22253]: 22253:info: Stopping mta
May 17 01:02:10 <myhostname> saslauthd[13816]: server_exit : master exited: 13816
May 17 01:02:10 <myhostname> zimbramon[22253]: 22253:info: Stopping spell
May 17 01:02:10 <myhostname> zimbramon[22253]: 22253:info: Stopping snmp
May 17 01:02:10 <myhostname> zimbramon[22253]: 22253:info: Stopping archiving
May 17 01:02:10 <myhostname> zimbramon[22253]: 22253:info: Stopping antivirus
May 17 01:02:10 <myhostname> zimbramon[22253]: 22253:info: Stopping antispam
May 17 01:02:11 <myhostname> zimbramon[22253]: 22253:info: Stopping imapproxy
May 17 01:02:11 <myhostname> zimbramon[22253]: 22253:info: Stopping mailbox
May 17 01:02:21 <myhostname> zimbramon[22253]: 22253:info: Stopping logger
May 17 01:02:26 <myhostname> zimbramon[22253]: 22253:info: Stopping ldap
May 17 01:03:03 <myhostname> zimbramon[22593]: 22593:info: Starting services
May 17 01:03:11 <myhostname> zimbramon[22593]: 22593:info: Rewriting configs webxml mailbox amavis antispam antivirus mta sasl
May 17 01:03:21 <myhostname> zimbramon[22593]: 22593:info: Starting logger
May 17 01:03:21 <myhostname> zimbramon[22593]: 22593:info: Starting mailbox
May 17 01:03:21 <myhostname> zimbramon[22993]: 22993:info: zmmtaconfig: zmmtaconfig started on <myhostname> with loglevel=3 pid=22993
May 17 01:03:39 <myhostname> zimbramon[22593]: 22593:info: Starting snmp
May 17 01:03:39 <myhostname> zimbramon[22593]: 22593:info: Starting mta
May 17 01:03:39 <myhostname> zimbramon[23498]: 23498:info: zmmtaconfig: zmmtaconfig already running at 22993
May 17 01:03:44 <myhostname> saslauthd: auth_zimbra_init: zimbra_cert_check is off!
May 17 01:03:44 <myhostname> saslauthd: auth_zimbra_init: 1 auth urls initialized for round-robin
May 17 01:03:44 <myhostname> saslauthd[23632]: detach_tty : master pid is: 23632
May 17 01:03:44 <myhostname> saslauthd[23632]: ipc_init : listening on socket: /opt/zimbra/cyrus-sasl-2.1.22.3z/state/mux
May 17 01:03:44 <myhostname> zimbramon[22593]: 22593:info: Starting stats
May 17 01:04:06 <myhostname> zimbramon[23698]: 23698:info: 2008-05-17 01:04:02, STATUS: <myhostname>: ldap: Running
May 17 01:04:06 <myhostname> zimbramon[23698]: 23698:info: 2008-05-17 01:04:02, STATUS: <myhostname>: logger: Running
May 17 01:04:06 <myhostname> zimbramon[23698]: 23698:info: 2008-05-17 01:04:02, STATUS: <myhostname>: mailbox: Running
May 17 01:04:06 <myhostname> zimbramon[23698]: 23698:info: 2008-05-17 01:04:02, STATUS: <myhostname>: mta: Running
May 17 01:04:06 <myhostname> zimbramon[23698]: 23698:info: 2008-05-17 01:04:02, STATUS: <myhostname>: snmp: Running
May 17 01:04:06 <myhostname> zimbramon[23698]: 23698:info: 2008-05-17 01:04:02, STATUS: <myhostname>: stats: Stopped


any idea?

[mmorse]

Anything useful in /opt/zimbra/zmstat/zmstat.out?
-caio80 could you also update your profile so I know what version you're running.
-Are the rest of you running 5.0.5 yet or still 5.0.4 & 5.0.2?

[caio80]

There we go. Now I have no more secrets for you (well, maybe just a couple...)!

I won't update Zimbra to something > than 5.0.4 until I see some consistent new features or interesting bugfixes.

Anyways, I checked /opt/zimbra/zmstat/zmstat.out, and I actually found out this:

No iostat installed on this host at /opt/zimbra/libexec/zmstat-io line 27.
No iostat installed on this host at /opt/zimbra/libexec/zmstat-io line 27.
Warning: Not possible to monitor process stats
No iostat installed on this host at /opt/zimbra/libexec/zmstat-io line 27.
No iostat installed on this host at /opt/zimbra/libexec/zmstat-io line 27.
Warning: Not possible to monitor process stats
No iostat installed on this host at /opt/zimbra/libexec/zmstat-io line 27.
No iostat installed on this host at /opt/zimbra/libexec/zmstat-io line 27.
Warning: Not possible to monitor process stats
No iostat installed on this host at /opt/zimbra/libexec/zmstat-io line 27.
No iostat installed on this host at /opt/zimbra/libexec/zmstat-io line 27.
Warning: Not possible to monitor process stats
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/opt/zimbra/db/mysql.sock' (2)
Error while running '/opt/zimbra/bin/mysql --user=zimbra --password=REMOVED --database=zimbra --batch --skip-column-names'. at /opt/zimbra/zimbramon/lib/Zimbra/DB/D
B.pm line 73, <OUTPUT> line 579238.
Can't locate Zimbra/Mon/Zmstat.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi /usr/
lib64/perl5/site_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7 /usr/l
ib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.7/x86_
64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr
/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/p
erl5/5.8.8 .) at /opt/zimbra/libexec/zmstat-proc line 20.
BEGIN failed--compilation aborted at /opt/zimbra/libexec/zmstat-proc line 20.
Can't locate Zimbra/Mon/Zmstat.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi /usr/
lib64/perl5/site_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7 /usr/l
ib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.7/x86_
64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr
/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/p
erl5/5.8.8 .) at /opt/zimbra/libexec/zmstat-vm line 42.
BEGIN failed--compilation aborted at /opt/zimbra/libexec/zmstat-vm line 42.
Can't locate Zimbra/Mon/Zmstat.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi /usr/
lib64/perl5/site_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7 /usr/l
ib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.7/x86_
64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr
/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/p
erl5/5.8.8 .) at /opt/zimbra/libexec/zmstat-io line 19.
BEGIN failed--compilation aborted at /opt/zimbra/libexec/zmstat-io line 19.
Can't locate Zimbra/Mon/Zmstat.pm in @INC (@INC contains: /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.7/x86_64-linux-thread-multi /usr/
lib64/perl5/site_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/site_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl/5.8.7 /usr/l
ib/perl5/site_perl/5.8.6 /usr/lib/perl5/site_perl/5.8.5 /usr/lib/perl5/site_perl /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.7/x86_
64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.6/x86_64-linux-thread-multi /usr/lib64/perl5/vendor_perl/5.8.5/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.8 /usr
/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl/5.8.6 /usr/lib/perl5/vendor_perl/5.8.5 /usr/lib/perl5/vendor_perl /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi /usr/lib/p
erl5/5.8.8 .) at /opt/zimbra/libexec/zmstat-cpu line 38.
[...]

and so on, for a little while...

I just installed sysstat to get the missing iostat component, will see tomorrow morning whether ZCS liked it or not, and if the rest of the troubles will get corrected on cascade.

Meanwhile, thanks for your support mmorse.

[mmorse]

Bug 19345 - zimbra-core has missing dependency on sysstat

(FYI I removed your mysql password above)

[caio80]

GOTCHA!

problem was in a backup script found on the forums! ( Zimbra Backup Script ... Small Problem )

Should have checked it better before implementing it :-(... It stopped zimbra and restarted it without sourcing all the env stuff the zimbra user has. By using "su - zimbra -c" (-i flag on sudo would work as well) stats service would finally get restarted correctly by cron. Here's a snippet of the stopping section, but the situation was identical in the starting part too.

[...]
# Stop Zimbra Services

# bad method
# sudo -u zimbra $ZIMBRA_DIR/bin/zmcontrol stop

# working method
su - zimbra  -c 'zmcontrol stop'
[...]

Thanks for your support, folks!