Guide: Multi-server logging with openSuSE 10.3 syslog-ng

[bdial]

Thought I'd share my experiences getting logging to work on a multiserver install running openSuSE 10.3 and Zimbra 5.x. Maybe some of these can be integrated into the install script for the next version. These instructions will probably be valid for other 10.x versions of openSuSE but I have not tested it. If anyone has other suggestions, fixes or what not please post them and I will modify to include them.

For the purposes of this guide, we'll use the following 3 servers

mta.domain.com - the zimbra-mta server
ldap.domain.com - the zimbra-ldap server
mailbox.domain.com - the main mailbox server, also running zimbra-logger and will be the central repository for all the servers' logs

The first server we want to setup is the mailbox/logger server. You can pretty much throw out the zmsyslogsetup script here. It attempts to use a syslog-ng.conf.in file which according to the syslog-ng.conf in openSuSE 10.3

# NOTE: The SuSEconfig script and its syslog-ng.conf.in
#       configuration template aren't used any more.

So open /etc/syslog-ng/syslog-ng.conf with your favorite text edit. The first thing you'll want to do is uncomment (remote the #) the line that says

udp(ip("0.0.0.0") port(514));

This will allow the other hosts to log to syslog-ng on the logger server. This is equivelant to adding the command line arguments -r -m 0 when you're using the standard syslog. Next, add these lines to the bottom of the file

filter f_local0       { facility(local0); }; # zimbra
destination zmail { file("/var/log/zimbra.log" owner("zimbra") ); }; # zimbra
log { source(src); filter(f_mail); destination(zmail); }; # zimbra
destination local0 { file("/var/log/zimbra.log" owner("zimbra") ); }; # zimbra
log { source(src); filter(f_local0); destination(local0); }; # zimbra
filter f_auth       { facility(auth); }; # zimbra
destination zmauth { file("/var/log/zimbra.log" owner("zimbra") ); }; # zimbra
log { source(src); filter(f_auth); destination(zmauth); }; # zimbra

This sets up the necessary logging facilities. Save that file and exit. Now we need to handle the log rotating. Zimbra will have no problem moving the zimbra.log since it has the necessary permissions, but it will not be able to restart the syslog server when it does it and therefor you'll wind up with a blank zimbra.log until root restarts syslog with it's own logrotate process. The first thing you need to do is edit /etc/sudoers down at the bottom you'll find a few entries for zimbra already. Add this one below them

%zimbra ALL=NOPASSWD:/sbin/rcsyslog restart

This allows zimbra to restart the syslog daemon. Now edit the file /opt/zimbra/conf/zmlogrotate and fine the line that says

/sbin/killall -HUP syslogd 2> /dev/null || true

change that line to say

sudo /sbin/rcsyslog restart 2> /dev/null || true

Now as a good test you should su to the zimbra user, and try the command sudo /sbin/rcsyslog restart . If all goes well, it should restart syslog and you should now have a /var/log/zimbra.log with status updates of the mailbox server currently.

Now onto the other hosts mta and ldap. Open /etc/syslog-ng/syslog-ng.conf

comment (put a # in front of) the line that says

log { source(src); filter(f_mail); destination(mail); };

This keeps the system from logging mail stuff from postfix to the local mail log cause you'll want to send it to the logger server. This is only really necessary for the mta server but I guess if it was going to be integrated into the zmsyslogsetup script might as well do it for every machine it won't hurt.

Next, add these lines at the bottom

destination zmlogger { udp("mailbox.domain.com" port(514) ); }; # zimbra
log { source(src); filter(f_mail); destination(zmlogger); }; # zimbra
filter f_local0       { facility(local0); }; # zimbra
log { source(src); filter(f_local0); destination(zmlogger); }; # zimbra
filter f_auth       { facility(auth); }; # zimbra
log { source(src); filter(f_auth); destination(zmlogger); }; # zimbra

you'll want to change the destination zmlogger statement to be the address of your logger server. Ultimately the zmsyslogsetup script should populate this with the zmLogHostname from the config like it does for the standard syslog setup. Anyway, save this file now and then restart syslog as root. You don't really need to worry about zimbra's logrotate for the otehr machines as they will not be logging locally anyway and it doesn't matter if it's broke.

You should now see status/smtp logs from the otehr hosts on your mailbox/logger server.

[LMStone]

Thanks for this post!

We can confirm this also works with SUSE Linux Enterprise Server 10 SP2, which also installs syslog-ng by default.

We recently expanded a single-server SLES10 ZCS install to a multi-server setup, and this post is what got our aggregated syslog server working for us.

One suggestion if I may... if you edit /etc/syslog-ng/syslog-ng.conf by hand, SuSEconfig scripts won't modify it at all going forward. Possibly it is safer (or not) to update /etc/syslog-ng/syslog-ng.conf.in and then let SuSEconfig regenerate /etc/syslog-ng/syslog-ng.conf each time SuSEconfig is run.

Hope that helps,
Mark

[bsteimel]

We started using syslog-ng on our Ubuntu8 box instead of syslogd. I successfully used this to reconfigure our box to work. I was getting the error "logger service not installed" in the zimbra admin interface even though it said it was the logger service was running.

The only differences were:

in all the syslog-ng config files instead of source(src) i used source(s_all) because that the source i had already configured to send to my network syslog server.

In the logrotate file i used /sbin/syslog-ng instead of rcsyslog because on ubuntu8 the rcsyslog did not exist. This also has to be changed in the sudousers file.

It has been running for a few days now and everything seems to be working properly.

[LMStone]

Is syslog-ng the default on a new 8.04 install like it is on SuSE Linux Enterprise Server 10?

Can anyone respond as to the default logging for an RHEL5 install?

If syslog-ng is becoming the new standard, perhaps that's the trigger for Zimbra to update their installation scripts to see which syslogging facility is installed and deploy the correct Zimbra syslog scripts accordingly.

Mark

[bsteimel]

no syslog-ng is not default on ubuntu8. I just like the interface in webmin and the added features. I run syslog-ng on all my servers for forwarding there logs to our central syslog server that runs splunk and syslog-ng. splunk takes care of all our logs except for snort which puts us over the 500mb/day limit for the free version of splunk. syslog-ng takes care of the snort log.

[LMStone]

Originally Posted by bsteimel

no syslog-ng is not default on ubuntu8. I just like the interface in webmin and the added features. I run syslog-ng on all my servers for forwarding there logs to our central syslog server that runs splunk and syslog-ng. splunk takes care of all our logs except for snort which puts us over the 500mb/day limit for the free version of splunk. syslog-ng takes care of the snort log.

Ah, another Splunk fan! Awesome software...

Thanks for the reply,
Mark