Server Status

[simonfishley]

Hi there

I have been testing Zimbra for a few months and we are just about ready to deploy. One thing that bothers me is that in every single install I have done, within 24 hours of installation, the server status page and server statistics pages both stop working. See below:


Can anyone shed some light on this? I have literally done nothing to this server. It is a clean install on Debian 5.0 (amd64). I installed it and left it to run to illustrate this point. Sometimes when the server is rebooted I see the status works for a while but then it stops again. That is not ideal, we don't want to have to reboot our servers just to see the status and stats.

Any advice appreciated.

Thanks
Simon

[phoenix]

Originally Posted by simonfishley

I have been testing Zimbra for a few months and we are just about ready to deploy. One thing that bothers me is that in every single install I have done, within 24 hours of installation, the server status page and server statistics pages both stop working.

Is there any particular reason you've posted this in the Zimbra Desktop forums? I'll assume you meant to put this in the ZCS forums and move it for you.

Have you tried some of the solutions in the forums? Have you also tried some of the Troubleshooting tips from the wiki?

[simonfishley]

My apologies! Brainfart.

I have tried several selected fixes, but most don't apply to my situation or they have no effect.

[phoenix]

Originally Posted by simonfishley

I have tried several selected fixes, but most don't apply to my situation or they have no effect.

Do tell us what you've tried (and what the outcome was) so we don't go ever the same ground again. If you tried the Troubleshooting tips then what was the outcome of that?

[simonfishley]

Hi Bill

Since posting my query, the indicators have all turned back to blue ticks. The next time this occurs I will re-run all the troubleshooting tips and records the results and post them here.

Thanks
Simon

[LMStone]

FWIW you are not alone in seeing that behaviour.

There are several inter-related causes, depending on your OS, but generally involve a mix of syslog setup and script issues, wrong ownership on the /opt/zimbra/zmstat subdirectories, and a few other assorted items. If you are curious, search for things like "syslog" "stats" "log rotation" etc. here on the forums and in bugzilla and you will get up to date no problem.

The good news is that all of the bugs related to this behaviour we know about (some of which we commented on or filed) are seemingly fixed and due to be released with 6.0.7, due out RSN.

Since you know Zimbra is running reliably, if it were me, I would wait a few days for 6.0.7, see if that cures things for you, and then you won't need to go through all of the various fixes to get nice blue check marks instead of those red x's.

Worst case, if 6.0.7 doesn't fix everything for you, I would expect the fix for you at that point would be just one or two minor things, instead of the four-to-six+ items you might need to investigate and/or tweak now.

Hope that helps,
Mark

P.S. FWIW, we applied all of the "fixes" by hand already to our and many of our clients' systems, but we still see the wrong ownership issue on /opt/zimbra/zmstat pop up every few days. Zimbra itself however remains as reliable as ever.

[simonfishley]

Thanks for the reply Mark. /me waits for 6.0.7.

[simonfishley]

Update.

We have upgraded to 6.0.7 and for a few days we had a functional status page. This morning however we are back to the red X's even though everything is running according to zmcontrol status.

Will not attempt the fixes - again and see if it helps.

S

[simonfishley]

Update. Well we have started moving users onto the server now. Everything is running smoothly and nice and stable except of course the Server Status page which is full of red X's.

FWIW I am posting all the info as requested in the troubleshooting section in case some one has any bright ideas. Otherwise I will just not look a the page and pretend it is not a problem.

zimbra@mailcape:/root$ zmloggerctl status; echo $?; ps ax | grep zmlogger
0
18461 ? S 0:40 /usr/bin/perl /opt/zimbra/libexec/zmlogger
18598 ? S 0:00 zmlogger: zmrrdfetch: server
24071 pts/1 R+ 0:00 grep zmlogger
zimbra@mailcape:/root$ /opt/zimbra/libexec/zmrrdfetch -c -f zmmtastats

Host: mailcape.saao.ac.za

timestamp,filter_misc,clam_events,mta_delay,mta_vo lume,filter_virus,filter_count,mta_count,filter_sp am,sendmail_events
1280154300,0,0,,44.3772533237207,0,0.0059269771156 247,0.0177809313468741,0,0
1280154600,0,0.202777555532442,0.967243999134574,1 11881.528969898,0.0197221110995546,0.9655599168665 18,2.96640186169911,0,0

SNIP

1280238900,0,0.00166550622124198,,918.913078458283 ,0,0.00999303732745191,0.0383066430885656,0,0
1280239200,0,0.00166550622124198,,918.913078458283 ,0,0.00999303732745191,0.0383066430885656,0,0
1280239500,,,,,,,,,
1280239800,,,,,,,,,
1280240100,,,,,,,,,
1280240400,,,,,,,,,
1280240700,,,,,,,,,

Host: mailsuth.saao.ac.za

timestamp,filter_misc,clam_events,mta_delay,mta_vo lume,filter_virus,filter_count,mta_count,filter_sp am,sendmail_events
1280154300,0,0,,0,0,0,0,0,0
1280154600,0,0,0,0,0,0,0,0,0
1280154900,,,,,,,,,
1280155200,,,,,,,,,
1280155500,,,,,,,,,

SNIP

1280238900,0,0,,0,0,0,0,0,0
1280239200,0,0,,0,0,0,0,0,0
1280239500,,,,,,,,,
1280239800,,,,,,,,,
1280240100,,,,,,,,,
1280240400,,,,,,,,,
1280240700,,,,,,,,,

zimbra@mailcape:/root$ grep -w MTA /var/log/zimbra-stats.log
SNIP

Jul 27 16:00:04 mailcape zimbramon[11648]: 11648:info: MTA: mailcape.saao.ac.za: filter_misc,clam_events,mta_delay,mta_volume,filte r_virus,filter_count,mta_count,filter_spam,sendmai l_events:: 0,1,1.39130434782609,551732,0,6,23,0,0
Jul 27 16:00:04 mailcape zimbramon[11648]: 11648:info: MTA: mailsuth.saao.ac.za: filter_misc,clam_events,mta_delay,mta_volume,filte r_virus,filter_count,mta_count,filter_spam,sendmai l_events:: 0,0,0,0,0,0,0,0,0
Jul 27 16:10:04 mailcape zimbramon[29626]: 29626:info: MTA: mailcape.saao.ac.za: filter_misc,clam_events,mta_delay,mta_volume,filte r_virus,filter_count,mta_count,filter_spam,sendmai l_events:: 0,1,0.576923076923077,346787,0,9,26,0,0
Jul 27 16:10:04 mailcape zimbramon[29626]: 29626:info: MTA: mailsuth.saao.ac.za: filter_misc,clam_events,mta_delay,mta_volume,filte r_virus,filter_count,mta_count,filter_spam,sendmai l_events:: 0,0,0,0,0,0,0,0,0
Jul 27 16:20:05 mailcape zimbramon[20244]: 20244:info: MTA: mailcape.saao.ac.za: filter_misc,clam_events,mta_delay,mta_volume,filte r_virus,filter_count,mta_count,filter_spam,sendmai l_events:: 0,23,0.227272727272727,48674,0,6,22,0,0
Jul 27 16:20:05 mailcape zimbramon[20244]: 20244:info: MTA: mailsuth.saao.ac.za: filter_misc,clam_events,mta_delay,mta_volume,filte r_virus,filter_count,mta_count,filter_spam,sendmai l_events:: 0,0,0,0,0,0,0,0,0

zimbra@mailcape:/root$ tail /var/log/zimbra-stats.log
Jul 27 16:22:05 mailcape zimbramon[24826]: 24826:info: 2010-07-27 16:22:01, STATUS: mailcape.saao.ac.za: snmp: Running
Jul 27 16:22:05 mailcape zimbramon[24826]: 24826:info: 2010-07-27 16:22:01, STATUS: mailcape.saao.ac.za: spell: Running
Jul 27 16:22:05 mailcape zimbramon[24826]: 24826:info: 2010-07-27 16:22:01, STATUS: mailcape.saao.ac.za: stats: Running
Jul 27 16:22:05 mailcape zimbramon[3708]: 3708:info: zmstat io.csv: timestamp, sda:tps, sda:kB_read/s, sda:kB_wrtn/s, sda:kB_read, sda:kB_wrtn, sda1:tps, sda1:kB_read/s, sda1:kB_wrtn/s, sda1:kB_read, sda1:kB_wrtn, sda2:tps, sda2:kB_read/s, sda2:kB_wrtn/s, sda2:kB_read, sda2:kB_wrtn, sda5:tps, sda5:kB_read/s, sda5:kB_wrtn/s, sda5:kB_read, sda5:kB_wrtn:: 07/27/2010 16:22:05, 7.43, 0.00, 87.20, 0, 2616, 7.43, 0.00, 87.20, 0, 2616, 0.00, 0.00, 0.00, 0, 0, 0.00, 0.00, 0.00, 0, 0
Jul 27 16:22:05 mailcape zimbramon[3704]: 3704:info: zmstat io-x.csv: timestamp, sda:rrqm/s, sda:wrqm/s, sda:r/s, sda:w/s, sda:rkB/s, sda:wkB/s, sda:avgrq-sz, sda:avgqu-sz, sda:await, sda:svctm, sda:%util, sda1:rrqm/s, sda1:wrqm/s, sda1:r/s, sda1:w/s, sda1:rkB/s, sda1:wkB/s, sda1:avgrq-sz, sda1:avgqu-sz, sda1:await, sda1:svctm, sda1:%util, sda2:rrqm/s, sda2:wrqm/s, sda2:r/s, sda2:w/s, sda2:rkB/s, sda2:wkB/s, sda2:avgrq-sz, sda2:avgqu-sz, sda2:await, sda2:svctm, sda2:%util, sda5:rrqm/s, sda5:wrqm/s, sda5:r/s, sda5:w/s, sda5:rkB/s, sda5:wkB/s, sda5:avgrq-sz, sda5:avgqu-sz, sda5:await, sda5:svctm, sda5:%util:: 07/27/2010 16:22:05, 0.00, 13.47, 0.00, 7.43, 0.00, 87.20, 23.46, 0.01, 1.11, 0.77, 0.57, 0.00, 13.47, 0.00, 7.43, 0.00, 87.20, 23.46, 0.01, 1.11, 0.77, 0.57, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0:::5522B816-998A-11DF-A08D-4B056B9A9C4A:::
Jul 27 16:22:05 mailcape zimbramon[3704]: :::5522B816-998A-11DF-A08D-4B056B9A9C4A:::.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00
Jul 27 16:22:07 mailcape zimbramon[3700]: 3700:info: zmstat cpu.csv: timestamp, cpu:user, cpu:nice, cpu:sys, cpu:idle, cpu:iowait, cpu:irq, cpu:softirq, cpu0:user, cpu0:nice, cpu0:sys, cpu0:idle, cpu0:iowait, cpu0:irq, cpu0:softirq, cpu1:user, cpu1:nice, cpu1:sys, cpu1:idle, cpu1:iowait, cpu1:irq, cpu1:softirq, cpu2:user, cpu2:nice, cpu2:sys, cpu2:idle, cpu2:iowait, cpu2:irq, cpu2:softirq, cpu3:user, cpu3:nice, cpu3:sys, cpu3:idle, cpu3:iowait, cpu3:irq, cpu3:softirq, cpu4:user, cpu4:nice, cpu4:sys, cpu4:idle, cpu4:iowait, cpu4:irq, cpu4:softirq, cpu5:user, cpu5:nice, cpu5:sys, cpu5:idle, cpu5:iowait, cpu5:irq, cpu5:softirq, cpu6:user, cpu6:nice, cpu6:sys, cpu6:idle, cpu6:iowait, cpu6:irq, cpu6:softirq, cpu7:user, cpu7:nice, cpu7:sys, cpu7:idle, cpu7:iowait, cpu7:irq, cpu7:softirq, cpu8:user, cpu8:nice, cpu8:sys, cpu8:idle, cpu8:iowait, cpu8:::5625F4EE-998A-11DF-9B9F-47056B9A9C4A:::
Jul 27 16:22:07 mailcape zimbramon[3700]: 3700:info: :::5625F4EE-998A-11DF-9B9F-47056B9A9C4A::::irq, cpu8:softirq, cpu9:user, cpu9:nice, cpu9:sys, cpu9:idle, cpu9:iowait, cpu9:irq, cpu9:softirq, cpu10:user, cpu10:nice, cpu10:sys, cpu10:idle, cpu10:iowait, cpu10:irq, cpu10:softirq, cpu11:user, cpu11:nice, cpu11:sys, cpu11:idle, cpu11:iowait, cpu11:irq, cpu11:softirq, cpu12:user, cpu12:nice, cpu12:sys, cpu12:idle, cpu12:iowait, cpu12:irq, cpu12:softirq, cpu13:user, cpu13:nice, cpu13:sys, cpu13:idle, cpu13:iowait, cpu13:irq, cpu13:softirq, cpu14:user, cpu14:nice, cpu14:sys, cpu14:idle, cpu14:iowait, cpu14:irq, cpu14:softirq, cpu15:user, cpu15:nice, cpu15:sys, cpu15:idle, cpu15:iowait, cpu15:irq, cpu15:softirq:: 07/27/2010 16:22:07, 3.7, 0.0, 0.2, 96.0, 0.0, 0.0, 0.0, 41.2, 0.0, 0.9, 57.9, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 100.0, 0.0, 0.0, 0.0, 1.4, 0.0, 0.0, 98.6, 0.0, 0.0, 0.0, 0.1, 0.0, :::5625FD2C-998A-11DF-9B9F-47056B9A9C4A:::
Jul 27 16:22:07 mailcape zimbramon[3700]: :::5625FD2C-998A-11DF-9B9F-47056B9A9C4A:::0.4, 99.5, 0.0, 0.0, 0.0, 0.3, 0.0, 0.2, 99.4, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 99.9, 0.0, 0.0, 0.0, 0.5, 0.0, 0.1, 99.4, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 100.0, 0.0, 0.0, 0.0, 0.4, 0.0, 0.2, 99.0, 0.5, 0.0, 0.0, 1.3, 0.0, 0.1, 98.6, 0.0, 0.0, 0.0, 4.2, 0.0, 0.3, 95.5, 0.0, 0.0, 0.0, 4.4, 0.0, 0.3, 95.3, 0.0, 0.0, 0.0, 3.1, 0.0, 0.2, 96.7, 0.0, 0.0, 0.0, 0.7, 0.0, 0.2, 99.0, 0.0, 0.0, 0.0, 0.8, 0.0, 0.6, 98.6, 0.0, 0.0, 0.0, 0.8, 0.0, 0.2, 99.0, 0.0, 0.0, 0.0
Jul 27 16:22:17 mailcape zimbramon[3702]: 3702:info: zmstat vm.csv: timestamp, r, b, swpd, free, buff, cache, si, so, bi, bo, in, cs, us, sy, id, wa, MemTotal, MemFree, Buffers, Cached, SwapCached, Active, Inactive, SwapTotal, SwapFree, Dirty, Writeback, AnonPages, Mapped, Slab, SReclaimable, SUnreclaim, PageTables, NFS_Unstable, Bounce, WritebackTmp, CommitLimit, Committed_AS, VmallocTotal, VmallocUsed, VmallocChunk, HugePages_Total, HugePages_Free, HugePages_Rsvd, HugePages_Surp, Hugepagesize, loadavg:: 07/27/2010 16:22:17, 0, 0, 5176344, 480500, 196364, 6980784, 0, 0, 0, 106, 1688, 3231, 4, 0, 96, 0, 16463836, 480500, 196364, 6980784, 4498256, 14176840, 1398420, 23695832, 18519488, 4940, 0, 8385092, 49240, 283616, 255232, 28384, 319052, 0, 0, 0, 31927748, 17337392, 34359738367, 312784, 34359425419, 0, 0, 0, 0, 2048, 0.74

zimbra@mailcape:/root$ zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats | head -20
<GetLoggerStatsResponse xmlns="urn:zimbraAdmin">
<hostname hn="mailcape.saao.ac.za">
<stats name="zmmtastats">
<values t="1280154630">
<stat name="filter_misc" value="0"/>
<stat name="clam_events" value="0"/>
<stat name="mta_delay" value=""/>
<stat name="mta_volume" value="0"/>
<stat name="filter_virus" value="0"/>
<stat name="filter_count" value="0"/>
<stat name="mta_count" value="0"/>
<stat name="filter_spam" value="0"/>
<stat name="sendmail_events" value="0"/>
</values>
<values t="1280154660">
<stat name="filter_misc" value="0"/>
<stat name="clam_events" value="0"/>
<stat name="mta_delay" value=""/>
<stat name="mta_volume" value="0"/>
<stat name="filter_virus" value="0"/>

zimbra@mailcape:/root$ zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats @limit=1 | head -50
<GetLoggerStatsResponse xmlns="urn:zimbraAdmin">
<hostname hn="mailcape.saao.ac.za">
<stats name="zmmtastats">
<values t="1280154900">
<stat name="filter_misc" value="0"/>
<stat name="clam_events" value="0"/>
<stat name="mta_delay" value=""/>
<stat name="mta_volume" value="0"/>
<stat name="filter_virus" value="0"/>
<stat name="filter_count" value="0"/>
<stat name="mta_count" value="0"/>
<stat name="filter_spam" value="0"/>
<stat name="sendmail_events" value="0"/>
</values>

SNIP

Cheers
Simon

[phoenix]

Can you check the folder permissions as per this bug report Bug 45010 &ndash; Stats service failing due to folder permissions getting set wrong - I know that's fixed in 6.0.7 but I'd like you to verify everything is OK.