Red X

[mhammett]

I have a 7 server setup. There are 2 LDAP, 2 MTA, 2 Mailstore, and 1 Proxy. I can send and receive mail to external domains. The only box that shows up properly in the server status is my second mailstore. All boxes show everything is up when I run zmcontrol status.

I looked at Logger (ZCS 6.0.x and later) - Zimbra :: Wiki. Should the commands in the troubleshooting section at the bottom be run on all boxes or just on the mailstore boxes which have the logging services? There are failures abound when I run those commands on non-mailstores.

[mhammett]

I realized I only needed the logger on one mailbox, so I removed it from mailbox1.

Output from the troubleshooting commands on mailbox2.

[zimbra@Mailbox2 root]$ zmloggerctl status; echo $?; ps ax | grep zmlogger
0
 3981 ?        S      0:05 /usr/bin/perl /opt/zimbra/libexec/zmlogger
 4094 ?        S      0:00 zmlogger: zmrrdfetch: server

[zimbra@Mailbox2 root]$ /opt/zimbra/libexec/zmrrdfetch -f zmmtastats

Host: mta2.ics-il.net

timestamp,filter_misc,clam_events,mta_delay,mta_volume,filter_virus,filter_count,mta_count,filter_spam,sendmail_events
1315691280,,,,,,,,,
1315691310,,,,,,,,,
1315691340,,,,,,,,,

Followed by hundreds (I'd guess) lines of data.

[zimbra@Mailbox2 root]$ /opt/zimbra/libexec/zmrrdfetch -c -f zmmtastats

Host: mta2.ics-il.net

timestamp,filter_misc,clam_events,mta_delay,mta_volume,filter_virus,filter_count,mta_count,filter_spam,sendmail_events
1315691700,0,0,,0,0,0,0,0,0
1315692000,0,0,,0,0,0,0,0,0
1315692300,0,0,,0,0,0,0,0,0
1315692600,0,0,,0,0,0,0,0,0
1315692900,,,,,,,,,
1315693200,,,,,,,,,
1315693500,,,,,,,,,
1315693800,,,,,,,,,

Same here, but maybe only dozens of lines.

[zimbra@Mailbox2 root]$ grep -w MTA /var/log/zimbra-stats.log
Sep 11 04:10:22 Mailbox2 zimbramon[17320]: 17320:info: MTA: mta2.ics-il.net: filter_misc,clam_events,mta_delay,mta_volume,filter_virus,filter_count,mta_count,filter_spam,sendmail_events:: 0,0,0,0,0,0,0,0,0
Sep 11 04:10:22 Mailbox2 zimbramon[17320]: 17320:info: MTA: mta1.ics-il.net: filter_misc,clam_events,mta_delay,mta_volume,filter_virus,filter_count,mta_count,filter_spam,sendmail_events:: 0,0,0,0,0,0,0,0,0
Sep 11 04:20:20 Mailbox2 zimbramon[19603]: 19603:info: MTA: mta2.ics-il.net: filter_misc,clam_events,mta_delay,mta_volume,filter_virus,filter_count,mta_count,filter_spam,sendmail_events:: 0,0,0,0,0,0,0,0,0
Sep 11 04:20:20 Mailbox2 zimbramon[19603]: 19603:info: MTA: mta1.ics-il.net: filter_misc,clam_events,mta_delay,mta_volume,filter_virus,filter_count,mta_count,filter_spam,sendmail_events:: 0,0,0,0,0,0,0,0,0
Sep 11 04:30:24 Mailbox2 zimbramon[19930]: 19930:info: MTA: mta2.ics-il.net: filter_misc,clam_events,mta_delay,mta_volume,filter_virus,filter_count,mta_count,filter_spam,sendmail_events:: 0,0,0,0,0,0,0,0,0
Sep 11 04:30:24 Mailbox2 zimbramon[19930]: 19930:info: MTA: mta1.ics-il.net: filter_misc,clam_events,mta_delay,mta_volume,filter_virus,filter_count,mta_count,filter_spam,sendmail_events:: 0,0,0,0,0,0,0,0,0
Sep 11 04:40:19 Mailbox2 zimbramon[22095]: 22095:info: MTA: mta2.ics-il.net: filter_misc,clam_events,mta_delay,mta_volume,filter_virus,filter_count,mta_count,filter_spam,sendmail_events:: 0,0,0,0,0,0,0,0,0
Sep 11 04:40:19 Mailbox2 zimbramon[22095]: 22095:info: MTA: mta1.ics-il.net: filter_misc,clam_events,mta_delay,mta_volume,filter_virus,filter_count,mta_count,filter_spam,sendmail_events:: 0,0,0,0,0,0,0,0,0
Sep 11 04:50:32 Mailbox2 zimbramon[23950]: 23950:info: MTA: mta2.ics-il.net: filter_misc,clam_events,mta_delay,mta_volume,filter_virus,filter_count,mta_count,filter_spam,sendmail_events:: 0,0,0,0,0,0,0,0,0
Sep 11 04:50:32 Mailbox2 zimbramon[23950]: 23950:info: MTA: mta1.ics-il.net: filter_misc,clam_events,mta_delay,mta_volume,filter_virus,filter_count,mta_count,filter_spam,sendmail_events:: 0,0,0,0,0,0,0,0,0
Sep 11 05:00:22 Mailbox2 zimbramon[26183]: 26183:info: MTA: mta2.ics-il.net: filter_misc,clam_events,mta_delay,mta_volume,filter_virus,filter_count,mta_count,filter_spam,sendmail_events:: 0,0,0,0,0,0,0,0,0
Sep 11 05:00:22 Mailbox2 zimbramon[26183]: 26183:info: MTA: mta1.ics-il.net: filter_misc,clam_events,mta_delay,mta_volume,filter_virus,filter_count,mta_count,filter_spam,sendmail_events:: 0,0,0,0,0,0,0,0,0
Sep 11 05:10:19 Mailbox2 zimbramon[28404]: 28404:info: MTA: mta2.ics-il.net: filter_misc,clam_events,mta_delay,mta_volume,filter_virus,filter_count,mta_count,filter_spam,sendmail_events:: 0,0,0,0,0,0,0,0,0

Same business, bunch more afterwards.

[zimbra@Mailbox2 root]$ tail /var/log/zimbra-stats.log
Sep 11 16:52:01 Mailbox2 zimbramon[4391]: 4391:info: zmstat io-x.csv: timestamp, sda:rrqm/s, sda:wrqm/s, sda:r/s, sda:w/s, sda:rkB/s, sda:wkB/s, sda:avgrq-sz, sda:avgqu-sz, sda:await, sda:svctm, sda:%util, sda1:rrqm/s, sda1:wrqm/s, sda1:r/s, sda1:w/s, sda1:rkB/s, sda1:wkB/s, sda1:avgrq-sz, sda1:avgqu-sz, sda1:await, sda1:svctm, sda1:%util, sda2:rrqm/s, sda2:wrqm/s, sda2:r/s, sda2:w/s, sda2:rkB/s, sda2:wkB/s, sda2:avgrq-sz, sda2:avgqu-sz, sda2:await, sda2:svctm, sda2:%util, dm-0:rrqm/s, dm-0:wrqm/s, dm-0:r/s, dm-0:w/s, dm-0:rkB/s, dm-0:wkB/s, dm-0:avgrq-sz, dm-0:avgqu-sz, dm-0:await, dm-0:svctm, dm-0:%util, dm-1:rrqm/s, dm-1:wrqm/s, dm-1:r/s, dm-1:w/s, dm-1:rkB/s, dm-1:wkB/s, dm-1:avgrq-sz, dm-1:avgqu-sz, dm-1:await, dm-1:svctm, dm-1:%util:: 09/11/2011 16:52:01, 0.00, 3.59, 0.00, 2.06, 0.00, 25.58, 24.87, 0.00, 1.53, 0.55, 0.11, 0.00, 0.00, 0:::477A74AE-DCC0-11E0-B47A-CDCED8403639:::
Sep 11 16:52:01 Mailbox2 zimbramon[4391]: :::477A74AE-DCC0-11E0-B47A-CDCED8403639:::.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 3.59, 0.00, 2.06, 0.00, 25.58, 24.87, 0.00, 1.53, 0.55, 0.11, 0.00, 0.00, 0.00, 5.65, 0.00, 25.58, 9.06, 0.01, 1.47, 0.20, 0.11, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00, 0.00
Sep 11 16:52:01 Mailbox2 zimbramon[4387]: 4387:info: zmstat vm.csv: timestamp, r, b, swpd, free, buff, cache, si, so, bi, bo, in, cs, us, sy, id, wa, st, MemTotal, MemFree, Buffers, Cached, SwapCached, Active, Inactive, HighTotal, HighFree, LowTotal, LowFree, SwapTotal, SwapFree, Dirty, Writeback, AnonPages, Mapped, Slab, PageTables, NFS_Unstable, Bounce, CommitLimit, Committed_AS, VmallocTotal, VmallocUsed, VmallocChunk, HugePages_Total, HugePages_Free, HugePages_Rsvd, Hugepagesize, loadavg:: 09/11/2011 16:52:01, 3, 0, 0, 689656, 166220, 423196, 0, 0, 0, 33, 913, 358, 2, 1, 97, 0, 0, 2058840, 689532, 166220, 423196, 0, 1109612, 183184, 0, 0, 2058840, 689532, 786424, 786424, 532, 0, 703384, 34588, 48628, 9000, 0, 0, 1815844, 1385592, 34359738367, 1016, 34359737207, 0, 0, 0, 2048, 0.21
Sep 11 16:52:02 Mailbox2 zimbramon[4385]: 4385:info: zmstat cpu.csv: timestamp, cpu:user, cpu:nice, cpu:sys, cpu:idle, cpu:iowait, cpu:irq, cpu:softirq, cpu0:user, cpu0:nice, cpu0:sys, cpu0:idle, cpu0:iowait, cpu0:irq, cpu0:softirq:: 09/11/2011 16:52:02, 4.0, 0.0, 2.1, 93.8, 0.1, 0.1, 0.0, 4.0, 0.0, 2.1, 93.8, 0.1, 0.1, 0.0
Sep 11 16:52:02 Mailbox2 zimbramon[4732]: 4732:info: zmstat fd.csv: timestamp, fd_count, mailboxd_fd_count:: 09/11/2011 16:52:02, 1530, 160
Sep 11 16:52:16 Mailbox2 zimbramon[6816]: 6816:info: 2011-09-11 16:52:01, STATUS: mailbox2.ics-il.net: mailbox: Running
Sep 11 16:52:16 Mailbox2 zimbramon[6816]: 6816:info: 2011-09-11 16:52:01, STATUS: mailbox2.ics-il.net: snmp: Running
Sep 11 16:52:16 Mailbox2 zimbramon[6816]: 6816:info: 2011-09-11 16:52:01, STATUS: mailbox2.ics-il.net: spell: Running
Sep 11 16:52:16 Mailbox2 zimbramon[6816]: 6816:info: 2011-09-11 16:52:01, STATUS: mailbox2.ics-il.net: stats: Running
Sep 11 16:52:16 Mailbox2 zimbramon[6816]: 6816:info: 2011-09-11 16:52:01, STATUS: mailbox2.ics-il.net: zmconfigd: Running

[zimbra@Mailbox2 root]$ zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats | head -20

  Logger is not enabled

[zimbra@Mailbox2 root]$ zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats @limit=1 | head -50

  Logger is not enabled

I'll have to look into those Logger is not enabled lines.

[mhammett]

I started reading through the install guide step by step.

The Logger package is installed on the one mail
server. If installed, it is automatically enabled.
Logs from all the hosts are sent to the mailbox
server where the logger package is installed.
This data is used to generate the statistics
graphs and reporting.

Well, apparently it isn't. If they claim it runs automatically, maybe that's why its so hard finding how to turn it on.

[mhammett]

I upgraded the installation to the existing version to regenerate a setup log file. I then combed through that log file looking for clues. I saw a pair of zmprov commands. One notified of the installed services while the other enabled services. Logger was the only service that was installed, but not enabled. I reissued those commands adding logging as an enabled service.

[zimbra@Mailbox2 root]$ /opt/zimbra/bin/zmprov -m -l ms mailbox2.ics-il.net zimbraServiceInstalled logger zimbraServiceInstalled mailbox zimbraServiceInstalled snmp zimbraServiceInstalled stats zimbraServiceInstalled spell
[zimbra@Mailbox2 root]$ /opt/zimbra/bin/zmprov -m -l ms mailbox2.ics-il.net zimbraServiceEnabled logger zimbraServiceEnabled mailbox zimbraServiceEnabled stats zimbraServiceEnabled snmp zimbraServiceEnabled spell

I restarted and then I got:

[zimbra@Mailbox2 root]$ zmcontrol status
Host mailbox2.ics-il.net
        logger                  Running
        mailbox                 Running
        snmp                    Running
        spell                   Running
        stats                   Running
        zmconfigd               Running

Updated copies of the logging troubleshooting commands:

[zimbra@Mailbox2 root]$ zmloggerctl status; echo $?; ps ax | grep zmlogger
0
 3358 ?        S      0:00 /usr/bin/perl /opt/zimbra/libexec/zmlogger
 3493 ?        S      0:00 zmlogger: zmrrdfetch: server
 5454 pts/0    R+     0:00 grep zmlogger

[zimbra@Mailbox2 root]$ zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats | head -20

  
    
      
        
        
        
        
        
        
        
        
        
      
      
        
        
        
        
        

[zimbra@Mailbox2 root]$ zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats @limit=1 | head -50

  
    
      
        
        
        
        
        
        
        
        
        
      
      
        
        
        
        
        
        
        
        
        
      
      
        
        
        
        
        
        
        
        
        
      
      
        
        
        
        
        
        
        
        
        
      
      
        
        

That said, I'm still full of red Xs.