Error in my /tmp/gengraphs.out file

[Xao]

I'm getting this error in the /tmp/gengraphs.out file:

[root@webmail /]# more /tmp/gengraphs.out
DBI connect('database=zimbra_logger;mysql_read_default _file=/opt/zimbra/conf/my.logger.cnf;mysql_socket=/opt/zimbra/logger/db/mysql.sock','z
imbra',...) failed: Can't connect to local MySQL server through socket '/opt/zimbra/logger/db/mysql.sock' (2) at /opt/zimbra/libexec/zmgengr
aphs line 120
DB: Can't connect to dbi:mysql:database=zimbra_logger;mysql_read_defaul t_file=/opt/zimbra/conf/my.logger.cnf;mysql_socket=/opt/zimbra/logger
/db/mysql.sock: Can't connect to local MySQL server through socket '/opt/zimbra/logger/db/mysql.sock' (2)


All the graphs appear to be updating correctly, except for the disk stats, but I think that one is related to another issue, anyone have any ideas?

[Xao]

I should also mention the rest of the log file entries appear to be fine, one other question though, should I have double line entries for each update like this?

Generating hour graph for mta on webmail.mydomain.com
Generating hour graph for mta on webmail.mydomain.com
Generating day graph for mta on webmail.mydomain.com
Generating day graph for mta on webmail.mydomain.com
Generating month graph for mta on webmail.mydomain.com
Generating month graph for mta on webmail.mydomain.com
Generating year graph for mta on webmail.mydomain.com
Generating year graph for mta on webmail.mydomain.com
Generating hour graph for mta on webmail
Generating hour graph for mta on webmail
Generating day graph for mta on webmail
Generating day graph for mta on webmail
Generating month graph for mta on webmail
Generating month graph for mta on webmail
Generating year graph for mta on webmail
Generating year graph for mta on webmail
Generating hour graph for mta on ALL
Generating hour graph for mta on ALL
Generating day graph for mta on ALL
Generating day graph for mta on ALL
Generating month graph for mta on ALL
Generating month graph for mta on ALL
Generating year graph for mta on ALL
Generating year graph for mta on ALL
Generating hour graph for amavis on webmail.mydomain.com
Generating day graph for amavis on webmail.mydomain.com
Generating month graph for amavis on webmail.mydomain.com
Generating year graph for amavis on webmail.mydomain.com
Generating hour graph for amavis on webmail
Generating day graph for amavis on webmail
Generating month graph for amavis on webmail
Generating year graph for amavis on webmail
Generating hour graph for amavis on ALL
Generating day graph for amavis on ALL
Generating month graph for amavis on ALL
Generating year graph for amavis on ALL
Generating hour graph for disk on webmail.mydomain.com
Generating hour graph for disk on webmail.mydomain.com
Generating day graph for disk on webmail.mydomain.com
Generating day graph for disk on webmail.mydomain.com
Generating month graph for disk on webmail.mydomain.com
Generating month graph for disk on webmail.mydomain.com
Generating year graph for disk on webmail.mydomain.com
Generating year graph for disk on webmail.mydomain.com

[brian]

Is logger running? This means it can't contact the database.

[Xao]

Is logger running? This means it can't contact the database.

Yep:

[zimbra@webmail ~]$ zmcontrol status
Host webmail.intotheoven.com
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Running
mta Running
snmp Running
spell Running
stats Running

[zimbra@webmail libexec]$ ps -ef|grep logger
zimbra 11800 1 0 Jan02 ? 00:00:00 /bin/sh /opt/zimbra/logger/mysql/bin/mysqld_safe --defaults-file=/opt/zimbra/conf/my.logger.cnf --external-locking --ledir=/opt/zimbra/logger/mysql/libexec
zimbra 11872 11800 0 Jan02 ? 00:07:25 /opt/zimbra/logger/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.logger.cnf --basedir=/opt/zimbra/logger/mysql --datadir=/opt/zimbra/logger/db/data --pid-file=/opt/zimbra/logger/db/mysql.pid --skip-external-locking --port=7307 --socket=/opt/zimbra/logger/db/mysql.sock --external-locking
zimbra 11954 11904 0 Jan02 ? 00:00:13 /usr/bin/perl /opt/zimbra/libexec/zmlogger
zimbra 21976 28343 0 20:25 pts/0 00:00:00 grep logger
[zimbra@webmail libexec]$



I'm not sure if it's a sudo issue or a variable. If I run it as user 'zimbra' it runs fine.

In the /tmp/gengraphs.out file from the cron job, the first line talks about not finding a shared library.

/opt/zimbra/java/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
DBI connect('database=zimbra_logger;mysql_read_default _file=/opt/zimbra/conf/my.logger.cnf;mysql_socket=/opt/zimbra/logger/db/mysql.sock','zimbra',...) failed: Access denied for user 'zimbra'@'localhost' (using password: NO) at /opt/zimbra/libexec/zmgengraphs line 120
DB: Can't connect to dbi:mysql:database=zimbra_logger;mysql_read_defaul t_file=/opt/zimbra/conf/my.logger.cnf;mysql_socket=/opt/zimbra/logger/db/mysql.sock: Access denied for user 'zimbra'@'localhost' (using password: NO)

[brian]

Make sure the zimbra_logger_mysql_password is defined in localconfig

zmlocalconfig -s zimbra_logger_mysql_password

And you can use it to connect to the database. If you can't connect to the database with the password chances are it's out of sync. You can reset it with the following directions.
Resetting LDAP & MySQL Passwords - Zimbra :: Wiki

[Xao]

It was set, but I reset it anyways and then verified I could access the databse with the new password.

[root@webmail libexec]# mysql --port=7307 --socket=/opt/zimbra/logger/db/mysql.sock -u zimbra -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 41 to server version: 5.0.45-log

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> \s
--------------
mysql Ver 14.7 Distrib 4.1.20, for redhat-linux-gnu (x86_64) using readline 4.3

Connection id: 41
Current database:
Current user: zimbra@localhost
SSL: Not in use
Current pager: stdout
Using outfile: ''
Using delimiter: ;
Server version: 5.0.45-log
Protocol version: 10
Connection: Localhost via UNIX socket
Server characterset: latin1
Db characterset: latin1
Client characterset: latin1
Conn. characterset: latin1
UNIX socket: /opt/zimbra/logger/db/mysql.sock
Uptime: 3 min 20 sec

Threads: 2 Questions: 208 Slow queries: 0 Opens: 14 Flush tables: 1 Open tables: 8 Queries per second avg: 1.040
--------------

mysql> exit

Same error, so I did a 'zmcontrol stop' then 'zmcontrol start' and same thing.....

It looks like 1, it can't find a shared library:

/opt/zimbra/java/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory


This library is on the system though:

[zimbra@webmail ~]$ find /opt -name libjli.so
/opt/zimbra/jdk1.6.0_02/jre/lib/amd64/jli/libjli.so

[zimbra@webmail ~]$ ldd /opt/zimbra/java/bin/java
libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000002a9566c000)
libjli.so => /opt/zimbra/jdk1.6.0_02/bin/../jre/lib/amd64/jli/libjli.so (0x0000002a95781000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000002a9588b000)
libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a9598f000)
/lib64/ld-linux-x86-64.so.2 (0x0000002a95556000)
[zimbra@webmail ~]$ ls -l /opt/zimbra/jdk1.6.0_02/bin/../jre/lib/amd64/jli/libjli.so
-rwxr-xr-x 1 root root 48335 Jun 14 2007 /opt/zimbra/jdk1.6.0_02/bin/../jre/lib/amd64/jli/libjli.so

Then the next line in the log file after the library error, I get the Can't connect to database, but it say's it's not using a password. I think this may be related to the dynamic library not working, not sure though....

DBI connect('database=zimbra_logger;mysql_read_default _file=/opt/zimbra/conf/my.logger.cnf;mysql_socket=/opt/zimbra/logger/db/mysql.sock','zimbra',...) failed: Access denied for user 'zimbra'@'localhost' (using password: NO) at /opt/zimbra/libexec/zmgengraphs line 120
DB: Can't connect to dbi:mysql:database=zimbra_logger;mysql_read_defaul t_file=/opt/zimbra/conf/my.logger.cnf;mysql_socket=/opt/zimbra/logger/db/mysql.sock: Access denied for user 'zimbra'@'localhost' (using password: NO)

[brian]

You are correct, sorry I missed that. It's using zmlocalconfig to return the zimbra_logger_mysql_password. zmlocalconfig invokes java and can't find the library so it doesn't get the password and therefore can't connect to the database.

You can try setting this at the top of zmgengraphs and see if it helps
$ENV{JAVA_HOME}="/opt/zimbra/java";

Because zmgengraphs is invoked via crontab it doesn't get the full zimbra environment, not sure why we haven't seen this before.

[Xao]

Thanks for the info. I added the line in and same thing. I put the line in right after the comment header before the code actually started.

Also every other perl script is getting the same thing, it's just most of the have stderr redirected to /dev/null (> /dev/null 2>&1) in the crontab. I removed the redirects for testing purposes.

What's odd, is most of the scripts execute at least partially, in the log files there will be different points where they bomb out with the dbase connect.

[Xao]

As an example of a similiar issue is the zmstatus that runs every two minutes....

Here is what I get in an e-mail if I don't redirect to /dev/null.....

/opt/zimbra/java/bin/java: error while loading shared libraries: libjli.so: cannot open shared object file: No such file or directory
Unable to determine enabled services from ldap.
Enabled services read from cache. Service list may be inaccurate.

But I know status works because if I'm in the admin interface, it correctly reports when services are running and when they are not. I have tested by shutting down like Postfix, etc...

Like I said, it's really odd....

[Xao]

Ok, I was able to solve this and all the other issues related to it in the cronjob by adding the following line in front of each command in the crontab:

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/zimbra/jdk1.6.0_02/jre/lib/amd64/jli

An example line looks like this:

10 * * * * LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/zimbra/jdk1.6.0_02/jre/lib/amd64/jli /opt/zimbra/libexec/zmgengraphs >> /tmp/gengraphs.out 2>&1