Server status fails in Zimbra 8 Administration Cosole

[grrr]

Hello, people!

For some reason my "Server status" show that all services are down. At the same moment all services are up and running. CLI zmcontrol status reports all running.
Sound familiar? Yeah, I've made google search on this topic and know that it's frequent problem, but in my case it can't be solved for some reasons.

I'm running Zimbra Release 8.0.0_GA_5434.RHEL6_64_20120907144743 CentOS6_64 NETWORK edition on CentOS release 6.3 installed with "--platform-override" option.

• DNS working fine, hosts file contain proper records.

[root@zimbra ~]# hostname
zimbra.mydomain.com
[root@zimbra ~]# grep zimbra /etc/hosts
222.333.111.444   zimbra.mydomain.com  zimbra
127.0.0.1 zimbra.mydomain.com localhost  zimbra localhost.localdomain
[root@zimbra ~]# nslookup zimbra.mydomain.com
Server:         222.333.111.555
Address:        222.333.111.555#53

Name:   zimbra.mydomain.com
Address: 222.333.111.444

• zimbraLogHostname - points to zimbra host.

# sudo -u zimbra /opt/zimbra/bin/zmprov -l gcf zimbraLogHostname
zimbraLogHostname: zimbra.mydomain.com

• SSH keys and Certificates are valid and operational.

• No problems with permission on file system to prevent questions about /opt/zimbra/libexec/zmfixperms

• All files and directories in /opt/zimbra/zmstat/ are owned by zimbra user.

• Cron is Ok. Was trying to change time interval for zmstatuslog cronjob - didn't help.

[root@zimbra ~]# crontab -u zimbra -l | grep zmstatuslog
*/2 * * * * /opt/zimbra/libexec/zmstatuslog > /dev/null 2>&1
[root@zimbra ~]# service crond status
crond (pid  17592) is running...

[root@zimbra ~]# sudo -u zimbra /opt/zimbra/bin/zmstatctl status
Running: zmstat-df
Running: zmstat-cpu
Running: zmstat-convertd
Running: zmstat-fd
Running: zmstat-vm
Running: zmstat-mysql
Running: zmstat-allprocs
Running: zmstat-ldap
Running: zmstat-io-x
Running: zmstat-proc
Running: zmstat-io
Running: zmstat-mtaqueue

• Log files are filled with up to date info.

[root@zimbra log]# cd /var/log; ls -l | grep zimbra
-rw-------  1 zimbra root      63064 Nov  7 10:06 zimbra.log
-rw-------  1 zimbra root   12415105 Nov  7 10:06 zimbra-stats.log
-rw-------  1 zimbra root      15637 Nov  4 03:24 zimbra.log-20121104.gz
-rw-------  1 zimbra root      22114 Nov  5 03:12 zimbra.log-20121105.gz
-rw-r--r--  1 zimbra zimbra    36346 Nov  6 03:26 zimbra.log-20121106.gz
-rw-------  1 zimbra root      31567 Nov  7 03:21 zimbra.log-20121107.gz

[root@zimbra log]# grep "STATUS" /var/log/zimbra-stats.log | grep stats | tail -n 10
Nov  7 09:46:08 zimbra.mydomain.com zimbramon[9768]: 9768:info: 2012-11-07 09:46:01, STATUS: zimbra.mydomain.com: stats: Running
Nov  7 09:48:09 zimbra.mydomain.com zimbramon[11522]: 11522:info: 2012-11-07 09:48:01, STATUS: zimbra.mydomain.com: stats: Running
Nov  7 09:50:09 zimbra.mydomain.com zimbramon[13160]: 13160:info: 2012-11-07 09:50:01, STATUS: zimbra.mydomain.com: stats: Running
Nov  7 09:52:09 zimbra.mydomain.com zimbramon[15072]: 15072:info: 2012-11-07 09:52:01, STATUS: zimbra.mydomain.com: stats: Running
Nov  7 09:54:08 zimbra.mydomain.com zimbramon[16707]: 16707:info: 2012-11-07 09:54:01, STATUS: zimbra.mydomain.com: stats: Running
Nov  7 09:56:09 zimbra.mydomain.com zimbramon[18343]: 18343:info: 2012-11-07 09:56:02, STATUS: zimbra.mydomain.com: stats: Running
Nov  7 09:58:09 zimbra.mydomain.com zimbramon[20103]: 20103:info: 2012-11-07 09:58:01, STATUS: zimbra.mydomain.com: stats: Running
Nov  7 10:00:09 zimbra.mydomain.com zimbramon[21743]: 21743:info: 2012-11-07 10:00:01, STATUS: zimbra.mydomain.com: stats: Running
Nov  7 10:02:09 zimbra.mydomain.com zimbramon[23648]: 23648:info: 2012-11-07 10:02:01, STATUS: zimbra.mydomain.com: stats: Running
Nov  7 10:04:09 zimbra.mydomain.com zimbramon[25397]: 25397:info: 2012-11-07 10:04:01, STATUS: zimbra.mydomain.com: stats: Running

• SYSLOG

I've used 2 different Syslog implementations.
rsyslog-5.8.10-2.el6.x86_64
syslog-ng-3.0.5-5.x86_64

I know about /opt/zimbra/libexec/zmsyslogsetup and it's meaning. Both syslog configs are updated by zmsyslogsetup script. Have tried to run syslog-ng and rsyslog separately and both at the same time.

• And finally I got same RED X'es on Status report

[root@zimbra ~]# sudo -u zimbra /opt/zimbra/bin/zmsoap -z GetServiceStatusRequest
<GetServiceStatusResponse xmlns="urn:zimbraAdmin">
  <timezone id="Europe/Riga" displayName="Eastern European Time"/>
  <status t="1352277314" service="spell" server="zimbra.mydomain.com">0</status>
  <status t="1352277314" service="ldap" server="zimbra.mydomain.com">0</status>
  <status t="1352277314" service="snmp" server="zimbra.mydomain.com">0</status>
  <status t="1352277314" service="stats" server="zimbra.mydomain.com">0</status>
  <status t="1352277314" service="convertd" server="zimbra.mydomain.com">0</status>
  <status t="1352277314" service="mta" server="zimbra.mydomain.com">0</status>
  <status t="1352277314" service="mailbox" server="zimbra.mydomain.com">0</status>
  <status t="1352277314" service="logger" server="zimbra.mydomain.com">0</status>
  <status t="1352277314" service="antispam" server="zimbra.mydomain.com">0</status>
  <status t="1352277314" service="antivirus" server="zimbra.mydomain.com">0</status>
</GetServiceStatusResponse>
[root@zimbra ~]# service zimbra status
Host zimbra.mydomain.com
        antispam                Running
        antivirus               Running
        convertd                Running
        ldap                    Running
        logger                  Running
        mailbox                 Running
        mta                     Running
        snmp                    Running
        spell                   Running
        stats                   Running
        zmconfigd               Running

Any ideas ladies and gentlemen?

[grrr]

Have just re-initiated logger database and it solved the problem for now.

[root@zimbra ~]# cd /opt/zimbra/logger/
[root@zimbra logger]# tar -cvzg db-backup.tgz db
[root@zimbra logger]# su - zimbra
[zimbra@zimbra ~]$ /opt/zimbra/libexec/zmloggerinit
Stopping logswatch...done.
Starting logswatch...done.
[root@zimbra ~]# sudo -u zimbra /opt/zimbra/bin/zmsoap -z GetServiceStatusRequest
<GetServiceStatusResponse xmlns="urn:zimbraAdmin">
  <timezone id="Europe/Riga" displayName="Eastern European Time"/>
  <status t="1352295720" service="spell" server="zimbra.mydomain.com">1</status>
  <status t="1352295720" service="mailbox" server="zimbra.mydomain.com">1</status>
  <status t="1352295720" service="convertd" server="zimbra.mydomain.com">1</status>
  <status t="1352295720" service="logger" server="zimbra.mydomain.com">1</status>
  <status t="1352295720" service="mta" server="zimbra.mydomain.com">1</status>
  <status t="1352295720" service="stats" server="zimbra.mydomain.com">1</status>
  <status t="1352295720" service="antispam" server="zimbra.mydomain.com">1</status>
  <status t="1352295720" service="zmconfigd" server="zimbra.mydomain.com">1</status>
  <status t="1352295720" service="ldap" server="zimbra.mydomain.com">1</status>
  <status t="1352295720" service="antivirus" server="zimbra.mydomain.com">1</status>
  <status t="1352295720" service="snmp" server="zimbra.mydomain.com">1</status>
</GetServiceStatusResponse>