Server Status all red x and rrd data empty

[mingus8686]

Hello we been having this issue for months and am just getting around to addressing it. Zimbra Admin GUI shows red x for all services, though they are running and Server Statistics show "no data available"

Zimbra 6.0.7 install on CentOS 5.5 64bit

from the logger wiki page here are the answers to the following requests:

1. zmloggerctl status; echo $?; ps ax | grep zmlogger

zmloggerctl status; echo $?; ps ax | grep zmlogger
0
 8213 ?        S      0:02 /usr/bin/perl /opt/zimbra/libexec/zmlogger
 8270 ?        S      0:00 zmlogger: zmrrdfetch: server
18702 pts/1    S+     0:00 grep zmlogger

2. /opt/zimbra/libexec/zmrrdfetch -f zmmtastats

1289514960,,,,,,,,,
1289514990,,,,,,,,,
1289515020,,,,,,,,,
1289515050,,,,,,,,,
1289515080,,,,,,,,,
1289515110,,,,,,,,,
1289515140,,,,,,,,,
1289515170,,,,,,,,,
1289515200,,,,,,,,,

3. /opt/zimbra/libexec/zmrrdfetch -c -f zmmtastats

1289512800,,,,,,,,,
1289513100,,,,,,,,,
1289513400,,,,,,,,,
1289513700,,,,,,,,,
1289514000,,,,,,,,,
1289514300,,,,,,,,,
1289514600,,,,,,,,,
1289514900,,,,,,,,,
1289515200,,,,,,,,,
1289515500,,,,,,,,,

4. grep -w MTA /var/log/zimbra-stats.log

empty result

5. tail /var/log/zimbra-stats.log

Nov 11 14:41:00 mail zimbramon[3942]: :::C1E02D2C-EDE4-11DF-A27B-40F76DF06777:::46149466,3,0,0,0,0,0,0,0,0,0,0,61700,0,0,0,0,0,0,1630,0,0,0,0,0,0,0,0,0,0,0,0,0,0,2358,116,0,0,0,0,0,0,0,0,0,0,0,0,1064,0,0,0,0,0,63406,0,60893,190206,0,0,0,3,0,0,0,0,0,0,0,757,0,0,0,0,0,0,0,0,0,0,2118,0,0,0,3,0,0,0,0,0,0,0,0,0,0,0,16981,245,0,0,0,0,0,0,OFF,2122,245,9,2238,0,0,0,1,83849,0,0,0,258,1059149,9906641,55746,5025,378696,747,0,0,17273,380149,17145,6,68993,290769,374,308288,10,41,30031723,13564,0,243107,24450,0,0,0,265228288,14213,66310,2304009216,68993,3666,0,86389,5082,7946,0,0,41779200,16384,1090,16055,68993,0,0,0,0,0,1509,3157,10559489,17364,0,6694,0,0,0,0,0,0.000000,4,0,27,0,233,239,0,0,0,0,0,0,0,0,0,402994,402994,NULL,0,0,4014,0,1526,0,0,OFF,0,5,2,186,2365,227,0,0,0,,,0,0,0,0,0,0,0,0,0,NONE,0,0,0,0,0,0,0,,88766,0,0,0,0,0,4,4,1,45476 
Nov 11 14:41:01 mail zimbramon[3826]: 3826:info: zmstat io-x.csv: timestamp, sda:rrqm/s, sda:wrqm/s, sda:r/s, sda:w/s, sda:rkB/s, sda:wkB/s, sda:avgrq-sz, sda:avgqu-sz, sda:await, sda:svctm, sda:%util, sda1:rrqm/s, sda1:wrqm/s, sda1:r/s, sda1:w/s, sda1:rkB/s, sda1:wkB/s, sda1:avgrq-sz, sda1:avgqu-sz, sda1:await, sda1:svctm, sda1:%util, sda2:rrqm/s, sda2:wrqm/s, sda2:r/s, sda2:w/s, sda2:rkB/s, sda2:wkB/s, sda2:avgrq-sz, sda2:avgqu-sz, sda2:await, sda2:svctm, sda2:%util, sda3:rrqm/s, sda3:wrqm/s, sda3:r/s, sda3:w/s, sda3:rkB/s, sda3:wkB/s, sda3:avgrq-sz, sda3:avgqu-sz, sda3:await, sda3:svctm, sda3:%util, sdb:rrqm/s, sdb:wrqm/s, sdb:r/s, sdb:w/s, sdb:rkB/s, sdb:wkB/s, sdb:avgrq-sz, sdb:avgqu-sz, sdb:await, sdb:svctm, sdb:%util, sdb1:rrqm/s, sdb1:wrqm/s, sdb1:r/s, sdb1:w/s, sdb1:rkB/s, sdb1:wkB/s, sdb1:avgrq-sz, sdb1:avgqu-sz, sdb1:await, sd:::C2833B8E-EDE4-11DF-AD42-CCF66DF06777::: 
Nov 11 14:41:01 mail zimbramon[3826]: 3826:info: :::C2833B8E-EDE4-11DF-AD42-CCF66DF06777:::b1:svctm, sdb1:%util, sdb2:rrqm/s, sdb2:wrqm/s, sdb2:r/s, sdb2:w/s, sdb2:rkB/s, sdb2:wkB/s, sdb2:avgrq-sz, sdb2:avgqu-sz, sdb2:await, sdb2:svctm, sdb2:%util, sdb3:rrqm/s, sdb3:wrqm/s, sdb3:r/s, sdb3:w/s, sdb3:rkB/s, sdb3:wkB/s, sdb3:avgrq-sz, sdb3:avgqu-sz, sdb3:await, sdb3:svctm, sdb3:%util, sdc:rrqm/s, sdc:wrqm/s, sdc:r/s, sdc:w/s, sdc:rkB/s, sdc:wkB/s, sdc:avgrq-sz, sdc:avgqu-sz, sdc:await, sdc:svctm, sdc:%util, sdc1:rrqm/s, sdc1:wrqm/s, sdc1:r/s, sdc1:w/s, sdc1:rkB/s, sdc1:wkB/s, sdc1:avgrq-sz, sdc1:avgqu-sz, sdc1:await, sdc1:svctm, sdc1:%util, sdd:rrqm/s, sdd:wrqm/s, sdd:r/s, sdd:w/s, sdd:rkB/s, sdd:wkB/s, sdd:avgrq-sz, sdd:avgqu-sz, sdd:await, sdd:svctm, sdd:%util, sdd1:rrqm/s, sdd1:wrqm/s, sdd1:r/s, sdd1:w/s, sdd1:rkB/s, sdd1:wkB/s, sdd1:avgrq-sz, sdd1:avgqu-sz, sdd1:await, sdd1:svc:::C2834520-EDE4-11DF-AD42-CCF66DF06777::: 
Nov 11 14:41:01 mail zimbramon[3826]: 3826:info: :::C2834520-EDE4-11DF-AD42-CCF66DF06777:::tm, sdd1:%util, md2:rrqm/s, md2:wrqm/s, md2:r/s, md2:w/s, md2:rkB/s, md2:wkB/s, md2:avgrq-sz, md2:avgqu-sz, md2:await, md2:svctm, md2:%util, md1:rrqm/s, md1:wrqm/s, md1:r/s, md1:w/s, md1:rkB/s, md1:wkB/s, md1:avgrq-sz, md1:avgqu-sz, md1:await, md1:svctm, md1:%util, md0:rrqm/s, md0:wrqm/s, md0:r/s, md0:w/s, md0:rkB/s, md0:wkB/s, md0:avgrq-sz, md0:avgqu-sz, md0:await, md0:svctm, md0:%util:: 11/11/2010 14:41:01, 0.00, 3.93, 0.00, 7.37, 0.00, 71.60, 19.44, 0.04, 5.76, 2.71, 2.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 3.93, 0.00, 7.37, 0.00, 71.60, 19.44, 0.04, 5.76, 2.71, 2.00, 0.00, 3.93, 0.00, 7.37, 0.00, 71.60, 19.44, 0.04, 5.53, 2.50, 1.84, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.0:::C28347DC-EDE4-11DF-AD42-CCF66DF06777::: 
Nov 11 14:41:01 mail zimbramon[3826]: :::C28347DC-EDE4-11DF-AD42-CCF66DF06777:::0, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 3.93, 0.00, 7.37, 0.00, 71.60, 19.44, 0.04, 5.53, 2.50, 1.84, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 10.57, 0.00, 68.67, 13.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00 
Nov 11 14:41:01 mail zimbramon[3863]: 3863:info: zmstat io.csv: timestamp, sda:tps, sda:kB_read/s, sda:kB_wrtn/s, sda:kB_read, sda:kB_wrtn, sda1:tps, sda1:kB_read/s, sda1:kB_wrtn/s, sda1:kB_read, sda1:kB_wrtn, sda2:tps, sda2:kB_read/s, sda2:kB_wrtn/s, sda2:kB_read, sda2:kB_wrtn, sda3:tps, sda3:kB_read/s, sda3:kB_wrtn/s, sda3:kB_read, sda3:kB_wrtn, sdb:tps, sdb:kB_read/s, sdb:kB_wrtn/s, sdb:kB_read, sdb:kB_wrtn, sdb1:tps, sdb1:kB_read/s, sdb1:kB_wrtn/s, sdb1:kB_read, sdb1:kB_wrtn, sdb2:tps, sdb2:kB_read/s, sdb2:kB_wrtn/s, sdb2:kB_read, sdb2:kB_wrtn, sdb3:tps, sdb3:kB_read/s, sdb3:kB_wrtn/s, sdb3:kB_read, sdb3:kB_wrtn, sdc:tps, sdc:kB_read/s, sdc:kB_wrtn/s, sdc:kB_read, sdc:kB_wrtn, sdc1:tps, sdc1:kB_read/s, sdc1:kB_wrtn/s, sdc1:kB_read, sdc1:kB_wrtn, sdd:tps, sdd:kB_read/s, sdd:kB_wrtn/s, sdd:kB_read, sdd:kB_wrtn, sdd1:tps, sdd1:kB_read/s:::C286B476-EDE4-11DF-80B8-F1F66DF06777::: 
Nov 11 14:41:01 mail zimbramon[3863]: :::C286B476-EDE4-11DF-80B8-F1F66DF06777:::, sdd1:kB_wrtn/s, sdd1:kB_read, sdd1:kB_wrtn, md2:tps, md2:kB_read/s, md2:kB_wrtn/s, md2:kB_read, md2:kB_wrtn, md1:tps, md1:kB_read/s, md1:kB_wrtn/s, md1:kB_read, md1:kB_wrtn, md0:tps, md0:kB_read/s, md0:kB_wrtn/s, md0:kB_read, md0:kB_wrtn:: 11/11/2010 14:41:01, 7.36, 0.00, 71.58, 0, 2148, 0.00, 0.00, 0.00, 0, 0, 0.00, 0.00, 0.00, 0, 0, 7.36, 0.00, 71.58, 0, 2148, 7.36, 0.00, 71.58, 0, 2148, 0.00, 0.00, 0.00, 0, 0, 0.00, 0.00, 0.00, 0, 0, 7.36, 0.00, 71.58, 0, 2148, 0.00, 0.00, 0.00, 0, 0, 0.00, 0.00, 0.00, 0, 0, 0.00, 0.00, 0.00, 0, 0, 0.00, 0.00, 0.00, 0, 0, 10.56, 0.00, 68.64, 0, 2060, 0.00, 0.00, 0.00, 0, 0, 0.00, 0.00, 0.00, 0, 0 
Nov 11 14:41:01 mail zimbramon[3824]: 3824:info: zmstat vm.csv: timestamp, r, b, swpd, free, buff, cache, si, so, bi, bo, in, cs, us, sy, id, wa, st, MemTotal, MemFree, Buffers, Cached, SwapCached, Active, Inactive, HighTotal, HighFree, LowTotal, LowFree, SwapTotal, SwapFree, Dirty, Writeback, AnonPages, Mapped, Slab, PageTables, NFS_Unstable, Bounce, CommitLimit, Committed_AS, VmallocTotal, VmallocUsed, VmallocChunk, HugePages_Total, HugePages_Free, HugePages_Rsvd, Hugepagesize, loadavg:: 11/11/2010 14:41:01, 0, 0, 156, 1559708, 356232, 9542680, 0, 0, 0, 75, 1037, 316, 0, 0, 100, 0, 0, 16443324, 1559708, 356232, 9542680, 0, 8324108, 5633552, 0, 0, 16443324, 1559708, 11534584, 11534428, 968, 0, 4058600, 39100, 871836, 22432, 0, 0, 19756244, 14556916, 34359738367, 268640, 34359469427, 0, 0, 0, 2048, 0.13 
Nov 11 14:41:02 mail zimbramon[3822]: 3822:info: zmstat cpu.csv: timestamp, cpu:user, cpu:nice, cpu:sys, cpu:idle, cpu:iowait, cpu:irq, cpu:softirq, cpu0:user, cpu0:nice, cpu0:sys, cpu0:idle, cpu0:iowait, cpu0:irq, cpu0:softirq, cpu1:user, cpu1:nice, cpu1:sys, cpu1:idle, cpu1:iowait, cpu1:irq, cpu1:softirq, cpu2:user, cpu2:nice, cpu2:sys, cpu2:idle, cpu2:iowait, cpu2:irq, cpu2:softirq, cpu3:user, cpu3:nice, cpu3:sys, cpu3:idle, cpu3:iowait, cpu3:irq, cpu3:softirq:: 11/11/2010 14:41:02, 0.2, 0.0, 0.0, 99.7, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 99.8, 0.0, 0.0, 0.0, 0.1, 0.0, 0.1, 99.8, 0.0, 0.0, 0.0, 0.2, 0.0, 0.0, 99.7, 0.1, 0.0, 0.0, 0.3, 0.0, 0.1, 99.6, 0.0, 0.0, 0.0 
Nov 11 14:41:03 mail zimbramon[4282]: 4282:info: zmstat fd.csv: timestamp, fd_count, mailboxd_fd_count:: 11/11/2010 14:41:03, 3570, 1243

6. zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats | head -20

<GetLoggerStatsResponse xmlns="urn:zimbraAdmin">
  <hostname hn="mail.exampledomain.com">
    <stats name="zmmtastats"/>
  </hostname>
</GetLoggerStatsResponse>

7. zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats @limit=1 | head -50

<GetLoggerStatsResponse xmlns="urn:zimbraAdmin">
  <hostname hn="mail.exampledomain.com">
    <stats name="zmmtastats"/>
  </hostname>
</GetLoggerStatsResponse>

[fyd]

Once I happen to run across the same issue and I had to work a bit, and confirm all configs to fix it. I hope this helps you.

Please check if your /etc/syslog.conf has entries with zimbra logger server's hostname. And if you have mailbox servers in your setup, those servers should have it too. Just run this to see the output which will give the logger hostname of your zimbra.

$zmprov -l gcf zimbraLogHostname

My entries were broken in /etc/syslog.conf, those servers to be monitored didn't have entries in it for zimbraLogHostname. So I had to run '/opt/zimbra/bin/zmsyslogsetup' to update the syslog file on all servers. That helped me.

Since 6.0.7 zmsyslogsetup is in libexec dir. ie /opt/zimbra/libexec/zmsyslogsetup. Updating your profile with 'zmcontrol -v' helps a lot to get things fixed here

[mingus8686]

fyd, Thank you for your help.

syslog.conf looks to be in order, this is a stand alone mail server, logs, mboxes stored locally:

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*							/dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;local0.none;local1.none;auth.none;mail.none;authpriv.none;cron.none		/var/log/messages

# The authpriv file has restricted access.
authpriv.*						/var/log/secure

# Log all the mail messages in one place.
mail.*							-/var/log/maillog


# Log cron stuff
cron.*							/var/log/cron

# Everybody gets emergency messages
*.emerg							*

# Save news errors of level crit and higher in a special file.
uucp,news.crit						/var/log/spooler

# Save boot messages also to boot.log
local7.*						/var/log/boot.log
local0.*                -/var/log/zimbra.log
local1.*                -/var/log/zimbra-stats.log
auth.*                  -/var/log/zimbra.log
mail.*                -/var/log/zimbra.log

$zmprov -l gcf zimbraLogHostname

zimbraLogHostname: mail.<our hostname>

No change to /etc/syslog.conf after running /opt/zimbra/libexec/zmsyslogsetup

[fyd]

mingus8686,

I am sorry that didn't help you. Apparently I have the same problem on my test server, upgraded to 6.0.8 and all looks well except for stats. I've found lot of similar issues in different threads so its some work. Will write here if I get something. Good luck.