zimbra install with perpetually broken logger/stats

[jptech]

my domain has been removed from all the logs, so it's not a misconfiguration.

I installed zimbra a year ago and it's been working great except for the logger and stats which has perpetually been broken. The install sat on 5.0.2 for a long time and the logger started working again. I had an issue where the server was bouncing messages from other domains addressed to one of the accounts, back to sender. It would only bounce messages to that single account. The bounce stated:

The account or domain may not exist, they may be blacklisted, or missing the proper dns entries.

I fixed that rather cleverly by renaming the account and then adding an alias to the original email address. Then changing the whole thing back. That account started working again. Since the logger was working at this point, I decided to try and fix the stats. It turns out that the raw_logs table was 'crashed'. I repaired it and ran zmlogprocess and then zmgengraphs. That broke the logger again somehow. I figured since I was in it already, I might as well upgrade to latest which is 5.0.10. So I upgraded and almost everything went off without a hitch. The whole install paused on checking for deprecated zimlets. I restarted the zimbra services and then this process completed the install. So now, I want to fix my logger.

zmcontrol status shows all services running.

Host coruscant.
        antispam                Running
        antivirus               Running
        ldap                    Running
        logger                  Running
        mailbox                 Running
        mta                     Running
        snmp                    Running
        spell                   Running
        stats                   Running

This is 5.0.10 I thought there were other services like archiving or something.

In the admin interface, server status is unavailable. it says to install the logger service.

I run 'ps aux | grep zimbra'

zimbra   12883  0.0  0.1   7376  4020 ?        SN   Aug17  48:51 /usr/bin/perl /opt/zimbra/libexec/zmmtaconfig
zimbra    2259  0.0  0.0   3052  1100 ?        SNs  00:42   0:00 /opt/zimbra/clamav/bin/freshclam --config-file=/opt/zimbra/conf/freshclam.conf -d --checks=12
zimbra    2332  0.0  5.4 123836 113072 ?       SNs  00:42   0:09 /opt/zimbra/clamav/sbin/clamd --config-file=/opt/zimbra/conf/clamd.conf
zimbra    2402  0.0  1.4 145772 29396 ?        Ssl  00:42   0:27 /opt/zimbra/openldap/libexec/slapd -l LOCAL0 -4 -u zimbra -h ldap://coruscant:389 -f /opt/zimbra/conf/slapd.conf
zimbra    3149  0.0  0.0   3036  1368 ?        S    00:43   0:00 /bin/sh /opt/zimbra/logger/mysql/bin/mysqld_safe --defaults-file=/opt/zimbra/conf/my.logger.cnf --external-locking --log-error=/opt/zimbra/log/logger_mysql_error.log --ledir=/opt/zimbra/logger/mysql/libexec
zimbra    3195  0.0  0.2   5832  4228 ?        S    00:43   0:00 /usr/bin/perl /opt/zimbra/libexec/logswatch --config-file=/opt/zimbra/conf/logswatchrc --use-cpan-file-tail --script-dir=/tmp -t /var/log/zimbra.log
zimbra    3196  0.0  0.8 106808 16904 ?        Sl   00:43   0:02 /opt/zimbra/logger/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.logger.cnf --basedir=/opt/zimbra/logger/mysql --datadir=/opt/zimbra/logger/db/data --pid-file=/opt/zimbra/logger/db/mysql.pid --skip-external-locking --port=7307 --socket=/opt/zimbra/logger/db/mysql.sock --external-locking
zimbra    3221  0.0  0.3   8520  6800 ?        S    00:43   0:00 /usr/bin/perl /tmp/.swatch_script.3195
zimbra    3257  0.0  0.2   8684  4828 ?        S    00:43   0:00 /usr/bin/perl /opt/zimbra/libexec/zmlogger
zimbra    3334  0.1  0.2   7372  5196 ?        S    00:43   0:36 /usr/bin/perl /opt/zimbra/libexec/zmmtaconfig
zimbra    3343  0.0  0.0   3040  1372 ?        S    00:43   0:00 /bin/sh /opt/zimbra/mysql/bin/mysqld_safe --defaults-file=/opt/zimbra/conf/my.cnf --external-locking --log-error=/opt/zimbra/log/mysql_error.log --ledir=/opt/zimbra/mysql/libexec
zimbra    3405  0.0  3.5 355780 72724 ?        Sl   00:43   0:11 /opt/zimbra/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --basedir=/opt/zimbra/mysql --datadir=/opt/zimbra/db/data --pid-file=/opt/zimbra/db/mysql.pid --skip-external-locking --port=7306 --socket=/opt/zimbra/db/mysql.sock --external-locking
root      4264  0.0  0.0   1576   380 ?        Ss   00:43   0:00 /opt/zimbra/libexec/zmmailboxdmgr start -Xms607m -Xmx607m -client -XX:NewRatio=2 -Djava.awt.headless=true -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -Xss256k
zimbra    4265  0.1 16.6 935564 344880 ?       Sl   00:43   0:49 /opt/zimbra/java/bin/java -Xms607m -Xmx607m -client -XX:NewRatio=2 -Djava.awt.headless=true -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -Xss256k -Djava.io.tmpdir=/opt/zimbra/mailboxd/work -Djava.library.path=/opt/zimbra/lib -Djava.endorsed.dirs=/opt/zimbra/mailboxd/common/endorsed -Dzimbra.config=/opt/zimbra/conf/localconfig.xml -Djetty.home=/opt/zimbra/mailboxd -DSTART=/opt/zimbra/mailboxd/etc/start.config -jar /opt/zimbra/mailboxd/start.jar /opt/zimbra/mailboxd/etc/jetty.properties /opt/zimbra/mailboxd/etc/jetty-setuid.xml /opt/zimbra/mailboxd/etc/jetty.xml
zimbra    5004  0.0  2.4  60356 49924 ?        Ss   00:43   0:01 amavisd (master)
zimbra    5030  0.0  2.6  65068 55684 ?        S    00:43   0:03 amavisd (ch10-avail)
zimbra    5031  0.0  2.5  63476 53940 ?        S    00:43   0:01 amavisd (ch4-avail)
zimbra    5032  0.0  2.6  63848 54456 ?        S    00:43   0:01 amavisd (ch6-avail)
zimbra    5033  0.0  2.3  60356 48552 ?        S    00:43   0:00 amavisd (virgin child)
zimbra    5034  0.0  2.5  63008 53800 ?        S    00:43   0:01 amavisd (ch7-avail)
zimbra    5036  0.0  2.5  62524 53164 ?        S    00:43   0:01 amavisd (ch5-avail)
zimbra    5037  0.0  2.6  64964 55584 ?        S    00:43   0:01 amavisd (ch6-avail)
zimbra    5038  0.0  2.5  62632 53036 ?        S    00:43   0:01 amavisd (ch6-avail)
zimbra    5039  0.0  2.5  62432 52984 ?        S    00:43   0:00 amavisd (ch2-avail)
zimbra    5040  0.0  2.5  61544 52220 ?        S    00:43   0:00 amavisd (ch1-avail)
zimbra    5115  0.0  0.2   5832  4236 ?        S    00:43   0:00 /usr/bin/perl /opt/zimbra/libexec/swatch --config-file=/opt/zimbra/conf/swatchrc --use-cpan-file-tail --script-dir=/tmp -t /var/log/zimbra.log
zimbra    5123  0.0  0.3   8508  6812 ?        S    00:43   0:00 /usr/bin/perl /tmp/.swatch_script.5115
zimbra    5124  0.0  0.3  13052  6644 ?        Ss   00:43   0:00 /opt/zimbra/httpd-2.2.8/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra    5125  0.0  0.2  13052  6084 ?        S    00:43   0:00 /opt/zimbra/httpd-2.2.8/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra    5126  0.0  0.2  13052  6084 ?        S    00:43   0:00 /opt/zimbra/httpd-2.2.8/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra    5127  0.0  0.2  13052  6084 ?        S    00:43   0:00 /opt/zimbra/httpd-2.2.8/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra    5130  0.0  0.2  13052  6084 ?        S    00:43   0:00 /opt/zimbra/httpd-2.2.8/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra    5134  0.0  0.2  13052  6084 ?        S    00:43   0:00 /opt/zimbra/httpd-2.2.8/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
root      5540  0.0  0.0   7020  1768 ?        Ss   00:44   0:00 /opt/zimbra/postfix/libexec/master
zimbra    5555  0.0  0.0   7256  1068 ?        Ss   00:44   0:00 /opt/zimbra/cyrus-sasl-2.1.22.3z/sbin/saslauthd -r -a zimbra
zimbra    5556  0.0  0.1   7996  3092 ?        S    00:44   0:00 /opt/zimbra/cyrus-sasl-2.1.22.3z/sbin/saslauthd -r -a zimbra
zimbra    5557  0.0  0.0   7256   672 ?        S    00:44   0:00 /opt/zimbra/cyrus-sasl-2.1.22.3z/sbin/saslauthd -r -a zimbra
zimbra    5559  0.0  0.0   7256   624 ?        S    00:44   0:00 /opt/zimbra/cyrus-sasl-2.1.22.3z/sbin/saslauthd -r -a zimbra
zimbra    5560  0.0  0.0   7256   624 ?        S    00:44   0:00 /opt/zimbra/cyrus-sasl-2.1.22.3z/sbin/saslauthd -r -a zimbra
zimbra    5619  0.0  0.1   5316  3708 ?        S    00:44   0:13 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-proc
zimbra    5621  0.0  0.1   5176  3528 ?        S    00:44   0:00 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-cpu
zimbra    5623  0.0  0.1   5180  3528 ?        S    00:44   0:00 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-vm
zimbra    5642  0.0  0.1   5052  3460 ?        S    00:44   0:00 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-fd
zimbra    5644  0.0  0.1   5184  3580 ?        S    00:44   0:02 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-mysql
zimbra    5653  0.0  0.1   5056  3476 ?        S    00:44   0:00 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-mtaqueue
zimbra    5984  0.0  0.0   1676   560 ?        S    00:44   0:00 /usr/bin/vmstat -n -S K 30
zimbra    7049  0.0  0.0   5280  1768 pts/0    S    08:31   0:00 su - zimbra
zimbra    7050  0.0  0.0   4528  1948 pts/0    S    08:31   0:00 -bash
zimbra    8993  0.0  0.0   2220   764 pts/0    R+   08:34   0:00 ps aux
zimbra    8994  0.0  0.0   1696   544 pts/0    S+   08:34   0:00 grep zimbra

the mysql_error.log:

080926  0:42:29  InnoDB: Starting shutdown...
080926  0:42:32  InnoDB: Shutdown completed; log sequence number 0 321428952
080926  0:42:32 [Note] /opt/zimbra/mysql/libexec/mysqld: Shutdown complete

080926 00:42:32  mysqld ended

080926 00:43:10  mysqld started
080926  0:43:10  InnoDB: Started; log sequence number 0 321428952
080926  0:43:10 [Note] /opt/zimbra/mysql/libexec/mysqld: ready for connections.
Version: '5.0.51a-log'  socket: '/opt/zimbra/db/mysql.sock'  port: 7306  Source distribution

I try to run /opt/zimbra/logger/mysql/bin/mysql directly and get this:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2)

the syslog shows repeated errors:

Sep 25 23:30:02 coruscant zimbramon[12883]: 12883:info: zmmtaconfig: Skipping Global system configuration update.
Sep 25 23:30:02 coruscant zimbramon[12883]: 12883:info: zmmtaconfig: gacf ERROR: service.FAILURE (system failure: getDirectContext) (cause: javax.naming.CommunicationException coruscant:389)
Sep 25 23:30:02 coruscant zimbramon[12883]: 12883:info: zmmtaconfig: Skipping All Reverse Proxy URLs update.
Sep 25 23:30:02 coruscant zimbramon[12883]: 12883:info: zmmtaconfig: Skipping getAllReverseProxyURLs ERROR: service.FAILURE (system failure: getDirectContext) (cause: javax.naming.CommunicationException coruscant:389)
Sep 25 23:30:03 coruscant zimbramon[12883]: 12883:info: zmmtaconfig: Skipping All Memcached Servers update.
Sep 25 23:30:03 coruscant zimbramon[12883]: 12883:info: zmmtaconfig: Skipping getAllMemcachedServers ERROR: service.FAILURE (system failure: getDirectContext) (cause: javax.naming.CommunicationException coruscant:389)
Sep 25 23:30:03 coruscant zimbramon[12883]: 12883:info: zmmtaconfig: Skipping All MTA Authentication Target URLs update.
Sep 25 23:30:03 coruscant zimbramon[12883]: 12883:info: zmmtaconfig: Skipping getAllMtaAuthURLs ERROR: service.FAILURE (system failure: getDirectContext) (cause: javax.naming.CommunicationException coruscant:389)
Sep 25 23:30:04 coruscant zimbramon[12883]: 12883:info: zmmtaconfig: Skipping Configuration for server coruscant update.
Sep 25 23:30:04 coruscant zimbramon[12883]: 12883:info: zmmtaconfig: gs:coruscant ERROR: service.FAILURE (system failure: getDirectContext) (cause: javax.naming.CommunicationException coruscant:389)
Sep 25 23:30:04 coruscant zimbramon[12883]: 12883:info: zmmtaconfig: Sleeping...Key lookup failed.

as well, the syslog no longer returns status from zimbramon.

How do I troubleshoot this?

[brian]

appears you have a more then just logger issues both are likely related to misconfigured hostnames. your ldap url should contain a fully qualified domain name. curuscant:389 should be something like coruscant.domain.tld:389

also make sure your /etc/hosts table is correct you should have an entry similar to the following
x.x.x.x coruscant.domain.tld coruscant

[jptech]

I appreciate you taking the time to read through, but my very first sentence of the post was "my domain has been removed from all the logs, so it's not a misconfiguration."

the install works in every way except logging/stats

[brian]

You should probalby read through these wiki articles on logger. They will help you troubleshoot the problem.

http://wiki.zimbra.com/index.php?title=Logger

http://wiki.zimbra.com/index.php?title=Brian-Notes

The zmmtaconfig issue maybe unrelated, but it's fairly important you resolve it as well. Those errors indicate that zmmtaconfig/zmprov couldn't talk to ldap. Do you have a firewall or iptables running? Can you execute zmprov gacf from a zimbra shell?

[jptech]

the service_status table is empty.

the logger service is running.

I've run through all the troubleshooting and so far, no red flags except the service_status table and the following:
grep STATUS /var/log/zimbra.log | tail
grep DISK /var/log/zimbra.log | tail

both return nothing.

I can execute zmprov gacf from a zimbra shell just fine.
what troubles me is the zimbraLogHostname is set to my FQDN. should it be localhost?

what is the next step?

[brian]

I've run through all the troubleshooting and so far, no red flags except the service_status table and the following:
grep STATUS /var/log/zimbra.log | tail
grep DISK /var/log/zimbra.log | tail

Run /opt/zimbra/libexec/zmstatuscheck by hand and make sure entries show up in /var/log/zimbra.log. If they don't you'll need to check your syslog configuration to make sure it's logging correctly. Without status entries in zimbra.log logger will not record any data in the database.

[jptech]

I do not have a /opt/zimbra/libexec/zmstatuscheck
It's just not there.

UPDATE:

BAH!!!! the stupid thing started working on its own!!!

[brian]

sorry meant zmstatuslog

[jptech]

sorry meant zmstatuslog

thanks, but it doesn't matter now, the gremlins fixed it.