Zimbra 5.0.18 Upgrade to 6.0.4 - Some Issues

[vavai]

Hi Mark,

According to syslog-ng problem, the same problem also hit me with ZCS 6.0.4 with SLES 11 64 bit but what make me confused, I'm trying ZCS 6.0.3 clean install with SLES 11 64 bit and it has no problem with syslog-ng. Trying to upgrade into 6.0.4 and similar problem hit me again.

Sadly, I haven't compare the syslog-ng configuration between two version

[Klug]

Originally Posted by LMStone

First, the upgrade installer destroyed the GoDaddy commercial certs, which needed to get reinstalled.

As I said somewhere else, I had the same issue while upgrading from 5.0.18 to 5.0.21...

[LMStone]

Originally Posted by vavai

Hi Mark,

According to syslog-ng problem, the same problem also hit me with ZCS 6.0.4 with SLES 11 64 bit but what make me confused, I'm trying ZCS 6.0.3 clean install with SLES 11 64 bit and it has no problem with syslog-ng. Trying to upgrade into 6.0.4 and similar problem hit me again.

Sadly, I haven't compare the syslog-ng configuration between two version

It would be helpful if you wouldn't mind posting the syslog-ng.conf file from your working SLES/ZCS 6.0.3 system please.

Can do?

Thanks!
Mark

[LMStone]

Originally Posted by Klug

As I said somewhere else, I had the same issue while upgrading from 5.0.18 to 5.0.21...

It seemed to be a common problem, so we were ready for that one.

Happy New Year!
Mark

[quanah]

Originally Posted by LMStone

News from the front:
2) broken LDAP replication, which is in Zimbra's court at the moment.

Now that I'm off vacation, I got a chance to look at this. The problem was that 3 CA certs were provided along with the server cert. Two of the three certs went together, one did not. The one that did not hashed to the same value as one that did, and overwrote the symlink. Removing the invalid CA cert from the chain and symlinking the hash of the correct one allowed replication to work.

--Quanah

[LMStone]

Originally Posted by quanah

Now that I'm off vacation, I got a chance to look at this. The problem was that 3 CA certs were provided along with the server cert. Two of the three certs went together, one did not. The one that did not hashed to the same value as one that did, and overwrote the symlink. Removing the invalid CA cert from the chain and symlinking the hash of the correct one allowed replication to work.

--Quanah

Hi Quanah,

Hope you had a nice vacation, and thank you for jumping on this so quickly upon your return!

I appreciate you testing that LDAP replication will now work; zmlocalconfig on the servers which are LDAP replicas reports that those servers are still using the LDAP master exclusively for LDAP queries. We'll change that tonight during the maintenance window and then we can scratch this one off the list.

Thank you again, and with best regards,
Mark

[quanah]

Originally Posted by LMStone

Hi Quanah,

Hope you had a nice vacation, and thank you for jumping on this so quickly upon your return!

I appreciate you testing that LDAP replication will now work; zmlocalconfig on the servers which are LDAP replicas reports that those servers are still using the LDAP master exclusively for LDAP queries. We'll change that tonight during the maintenance window and then we can scratch this one off the list.

Thank you again, and with best regards,
Mark

Hi Mark,

It was definitely an interesting case and took a bit to track down. Hopefully the rest of your 6.0.x experience will only improve from here!

Regards,
Quanah

[LMStone]

This post is about getting SUSE's SLES10-SP3 syslog-ng to work with Zimbra 6.

One noticeable way the issue manifests itself is where the Server Status page of the Admin Console shows all big red "X"s instead of the nice blue check marks -- even though Zimbra is actually running just fine. In our case, we also found that /var/log/zimbra-stats.log never populated with data.

I want to thank Irfan in Zimbra support for his help with this issue, as well as a fellow moderator for his post Guide: Multi-server logging with openSuSE 10.3 syslog-ng which provided inspiration.

For whatever reason, the Zimbra syslog-ng setup scripts didn't work on our system. So, we took what the scripts provided (which resulted in syslog-ng being unable to start) and went from there. We also had to keep in mind that the whole stats logging has changed from ZCS 5; MySQL is gone and in its place is the new log file /var/log/zimbra-stats.log along with a number of related changes.

So here's what we did...

First, we needed to get syslog-ng working on all the servers. The code snippet below is what we have added to all of our SLES10 SP3, ZCS 6.0.4 servers. Most of it is from the Zimbra installer, but we commented out the Zimbra-supplied "zimbra_src" and changed the "log" lines at the end to use the SuSE pre-configured source of "src" instead.

Once we could restart syslog-ng reliably on all the servers, we then ran as the zimbra user the command "zmstatctl" to stop, then start the Zimbra stats services.

At that point, /var/log/zimbra-stats.log and /var/log/zimbra.log started populating with good data and all the red "X"'s in the admin console went away.

We just made these changes today, so we'll need to confirm tomorrow that the daily mail report works correctly; since the upgrade /var/log/zimbra.log was being rotated shortly after 6:00pm, but until today we were running the syslog-ng config file we had used under 5.0.18. We also will need to confirm tomorrow that all the disk/mail/etc. usage stats update in the Admin Console as well.

But so far, so good!

Here's the relevant sections of our /etc/syslog-ng/syslog-ng.conf file.

source src {
        #
        # include internal syslog-ng messages
        # note: the internal() soure is required!
        #
        internal();

        #
        # the following line will be replaced by the
        # socket list generated by SuSEconfig using
        # variables from /etc/sysconfig/syslog:
        #
        unix-dgram("/dev/log");
        unix-dgram("/var/lib/named/dev/log");

        #
        # uncomment to process log messages from network:
        #
        udp(ip("0.0.0.0") port(514));
};

#Original source line from zimbra upgrade
#source zimbra_src {  unix-stream("/dev/log"; keep-alive(yes); max-connections(20);); }; # zimbra

#Source line from Zimbra support:
#source zimbra_src { unix-stream("/dev/log" keep-alive(yes)  max-connections(20)); }; # zimbra

filter zimbra_local0 { facility(local0); }; # zimbra
filter zimbra_local1 { facility(local1); }; # zimbra
filter zimbra_auth { facility(auth); }; # zimbra
filter zimbra_mail { facility(mail); }; # zimbra

#IMPORTANT!  Ensure one and only one of the Destination sections below
#is active, and that it is the correct section for the server role!

#Destinations To Be Used on the syslog server:
#destination zimbra_mail { file("/var/log/zimbra.log" owner("zimbra")); }; # zimbra
#destination zimbra_local1 { file("/var/log/zimbra-stats.log" owner("zimbra")); }; # zimbra
#destination zimbra_local0 { file("/var/log/zimbra.log" owner("zimbra")); }; # zimbra
#destination zimbra_auth { file("/var/log/zimbra.log" owner("zimbra")); }; # zimbra

#Destinations To Be Used on remote server:
destination zimbra_mail { udp("ip address of syslog server goes here" port(514) ); }; # zimbra
destination zimbra_local1 { udp("ip address of syslog server goes here" port(514) ); }; # zimbra
destination zimbra_local0 { udp("ip address of syslog server goes here" port(514) ); }; # zimbra
destination zimbra_auth { udp("ip address of syslog server goes here" port(514) ); }; # zimbra

#Changed all sources from "source(zimbra_src);..." to "source(src);"..."
log { source(src); filter(zimbra_mail); destination(zimbra_mail); }; # zimbra
log { source(src); filter(zimbra_local0); destination(zimbra_local0); }; # zimbra
log { source(src); filter(zimbra_local1); destination(zimbra_local1); }; # zimbra
log { source(src); filter(zimbra_auth); destination(zimbra_auth); }; # zimbra

Hope that helps other SuSE Zimbra 6 multi-server installs out there!

All the best,
Mark

[quanah]

Also see https://bugzilla.zimbra.com/show_bug.cgi?id=40819

[LMStone]

Originally Posted by quanah

Also see https://bugzilla.zimbra.com/show_bug.cgi?id=40819

Thanks Quanah; I added myself to the cc list for the bug and provided a link to my post above with the changes to /etc/syslog-ng/syslog-ng.conf we found which worked.

The bug is marked as "resolved" with 6.0.4, but our syslog-ng complained of syntax errors with the ZCS-supplied "zimbra_src" source, and although the modifications Irfan provided allowed syslog-ng to start, no new data was being populated to the log files. Once we edited the last four lines to use the SuSE-supplied "src" source, then the log files started getting data.

Should this bug be reopened?

All the best,
Mark