Zimbra 5.0.18 Upgrade to 6.0.4 - Some Issues

[LMStone]

Last night/this morning we upgraded our production Zimbra system from 5.0.18 to 6.0.4, unfortunately not without some drama. So first, I have to give extraordinary kudos to Irfan at Zimbra who worked with us through the wee hours to get our system functional for end users before the sun came up this morning.

To be fair, our system has been in production since 4.0.3, and though we have been doing a lot of testing prior to this upgrade, this was probably the least smooth major version upgrade we have had with Zimbra. Not complaining mind you; having headed a multi-million dollar software development project I am of the strong opinion that Zimbra's QA is very, very good. These things happen, so I thought it would be helpful to post here some of what we are seeing. We have a support ticket open with Zimbra on these issues.

First, the upgrade installer destroyed the GoDaddy commercial certs, which needed to get reinstalled. But since the certificate directories have changed a little versus the documentation we were using, we started getting a little nervous following the documentation until we sorted that out. This seems to be a common problem, so we were prepared.

Second, we use syslog-ng on SLES10 SP3, and the upgrade installer broke that (thanks Irfan for fixing that one).

Third, even after syslogging was fixed, the Admin Console functionality remains diminished. We have no "Certificates" tab in the left-hand nav bar, and the Server Status has all red "X"s. The License tab under Servers > Global Settings has also gone missing, so we'll need to use the CLI to replace our soon-to-be-expiring ZCS license. I expect we will find a few more broken bits in the coming days.

Fourth, LDAP replication was totally broken by this upgrade (and at this point remains broken). Our additional MTA and mailbox servers, which were originally LDAP replicas for performance, now talk to the LDAP master directly. Zimbra support has escalated this issue for us. This error seems to have impacted one other poster in the forums too:
LDAP / Replication on 5.x -> 6.x Upgrade

Fifth, we are discovering as the day unfolds a few non-mission critical differences between 5.0 and 6.0 that cause confusion for end users. For example, it used to be that if you had a COS on a domain and an account which had the Documents feature enabled but where domain-level Documents had not been created, then when users logged in, the Documents tab simply would not display. Under 6.0.4, the Documents tab does display, and when users click on the Documents tab they get a jetty 500 error -- and call us. It's as if the COS settings now have new preferences over the domain settings, which now requires us to add a few more COSs. Not a big deal for sure, but some end users get very nervous over errors like these--more so the ones who have Apache experience!

Overall, nothing really show-stopping here; we keep enough spare horsepower on our servers so that our LDAP master is perfectly capable of handling the load until Zimbra figures out the issue. And again, Zimbra support has been really, really terrific.

But if you are considering upgrading a ZCS system which has been in production as long as ours, at least from our experience it appears that the upgrade to 6.0.x may not be without some bumps in the road.

Hope that helps,
Mark

[vavai]

Hi LMStone,

Thanks for sharing your upgrading experience. Just a quick question : So now you are currently running ZCS 6.0.4 on SLES 10 SP3, aren't you ?

I found some issue installing (clean install) ZCS 6.0.4 on SLES 11 64 bit regarding syslog-ng. The problem also hit me on clean install ZCS 6.0.4 on SLES 10. Would you like to share how to fix the syslog-ng problem ?

Thread regarding my experience installing ZCS on SLES 11.

[LMStone]

Originally Posted by vavai

Hi LMStone,

Thanks for sharing your upgrading experience. Just a quick question : So now you are currently running ZCS 6.0.4 on SLES 10 SP3, aren't you ?

I found some issue installing (clean install) ZCS 6.0.4 on SLES 11 64 bit regarding syslog-ng. The problem also hit me on clean install ZCS 6.0.4 on SLES 10. Would you like to share how to fix the syslog-ng problem ?

Thread regarding my experience installing ZCS on SLES 11.

Yes, we are now running ZCS-NE 6.0.4 on SLES10-SP3 (I'll update my profile shortly!)

Here is the /etc/syslog-ng/syslog-ng.conf file as tweaked by Zimbra for us earlier today:

#
# /etc/syslog-ng/syslog-ng.conf
#
# Automatically generated by SuSEconfig on Sat Apr  5 16:28:11 EDT 2008.
#
# PLEASE DO NOT EDIT THIS FILE!
#
# you can modify /etc/syslog-ng/syslog-ng.conf.in instead
#
#
#
# File format description can be found in syslog-ng.conf(5)
# and /usr/share/doc/packages/syslog-ng/syslog-ng.txt.
#

#
# Global options.
#
options { long_hostnames(off); sync(0); perm(0640); stats(3600); };

#
# 'src' is our main source definition. you can add
# more sources driver definitions to it, or define
# your own sources, i.e.:
#
#source my_src { .... };
#
source src {
	#
	# include internal syslog-ng messages
	# note: the internal() soure is required!
	#
	internal();

	#
	# the following line will be replaced by the
	# socket list generated by SuSEconfig using
	# variables from /etc/sysconfig/syslog:
	#
	unix-dgram("/dev/log");
	unix-dgram("/var/lib/named/dev/log");

	#
	# uncomment to process log messages from network:
	#
	udp(ip("0.0.0.0") port(514));
};


#
# Filter definitions
#
filter f_iptables   { facility(kern) and match("IN=") and match("OUT="); };

filter f_console    { level(warn) and facility(kern) and not filter(f_iptables)
                      or level(err) and not facility(authpriv); };

filter f_newsnotice { level(notice) and facility(news); };
filter f_newscrit   { level(crit)   and facility(news); };
filter f_newserr    { level(err)    and facility(news); };
filter f_news       { facility(news); };

filter f_mailinfo   { level(info)      and facility(mail); };
filter f_mailwarn   { level(warn)      and facility(mail); };
filter f_mailerr    { level(err, crit) and facility(mail); };
filter f_mail       { facility(mail); };

filter f_cron       { facility(cron); };

filter f_local      { facility(local0, local1, local2, local3,
                               local4, local5, local6, local7); };

filter f_acpid      { match('^\[acpid\]:'); };
filter f_netmgm     { match('^NetworkManager:'); };

filter f_messages   { not facility(news, mail) and not filter(f_iptables); };
filter f_warn       { level(warn, err, crit) and not filter(f_iptables); };
filter f_alert      { level(alert); };


#
# Most warning and errors on tty10 and on the xconsole pipe:
#
destination console  { pipe("/dev/tty10"    group(tty) perm(0620)); };
log { source(src); filter(f_console); destination(console); };

destination xconsole { pipe("/dev/xconsole" group(tty) perm(0400)); };
log { source(src); filter(f_console); destination(xconsole); };

# Enable this, if you want that root is informed immediately,
# e.g. of logins:
#
#destination root { usertty("root"); };
#log { source(src); filter(f_alert); destination(root); };


#
# News-messages in separate files:
#
destination newscrit   { file("/var/log/news/news.crit"
                              owner(news) group(news)); };
log { source(src); filter(f_newscrit); destination(newscrit); };

destination newserr    { file("/var/log/news/news.err"
                              owner(news) group(news)); };
log { source(src); filter(f_newserr); destination(newserr); };

destination newsnotice { file("/var/log/news/news.notice"
                              owner(news) group(news)); };
log { source(src); filter(f_newsnotice); destination(newsnotice); };

#
# and optionally also all in one file:
# (don't forget to provide logrotation config)
#
#destination news { file("/var/log/news.all"); };
#log { source(src); filter(f_news); destination(news); };


#
# Mail-messages in separate files:
#
destination mailinfo { file("/var/log/mail.info"); };
log { source(src); filter(f_mailinfo); destination(mailinfo); };

destination mailwarn { file("/var/log/mail.warn"); };
log { source(src); filter(f_mailwarn); destination(mailwarn); };

destination mailerr  { file("/var/log/mail.err" fsync(yes)); };
log { source(src); filter(f_mailerr);  destination(mailerr); };

#
# and also all in one file:
#
destination mail { file("/var/log/mail"); };
log { source(src); filter(f_mail); destination(mail); };

 
#
# acpid messages in one file:
#
destination acpid { file("/var/log/acpid"); };
log { source(src); filter(f_acpid); destination(acpid); flags(final); };

#
# NetworkManager messages in one file:
#
destination netmgm { file("/var/log/NetworkManager"); };
log { source(src); filter(f_netmgm); destination(netmgm); flags(final); };


#
# Cron-messages in one file:
# (don't forget to provide logrotation config)
#
#destination cron { file("/var/log/cron"); };
#log { source(src); filter(f_cron); destination(cron); };


#
# Some boot scripts use/require local[1-7]:
#
destination localmessages { file("/var/log/localmessages"); };
log { source(src); filter(f_local); destination(localmessages); };


#
# All messages except iptables and the facilities news and mail:
#
destination messages { file("/var/log/messages"); };
log { source(src); filter(f_messages); destination(messages); };


#
# Firewall (iptables) messages in one file:
#
destination firewall { file("/var/log/firewall"); };
log { source(src); filter(f_iptables); destination(firewall); };


#
# Warnings (except iptables) in one file:
#
destination warn { file("/var/log/warn" fsync(yes)); };
log { source(src); filter(f_warn); destination(warn); };

#
# Enable this, if you want to keep all messages in one file:
# (don't forget to provide logrotation config)
#
#destination allmessages { file("/var/log/allmessages"); };
#log { source(src); destination(allmessages); };


filter f_local0       { facility(local0); }; # zimbra
destination zmail { file("/var/log/zimbra.log" owner("zimbra") ); }; # zimbra 
log { source(src); filter(f_mail); destination(zmail); }; # zimbra
destination local0 { file("/var/log/zimbra.log" owner("zimbra") ); }; # zimbra
log { source(src); filter(f_local0); destination(local0); }; # zimbra
filter f_auth       { facility(auth); }; # zimbra
destination zmauth { file("/var/log/zimbra.log" owner("zimbra") ); }; # zimbra
log { source(src); filter(f_auth); destination(zmauth); }; # zimbra

The "destination local1" and related parameters for the second zimbra log file did not work; syslog refused to start.

Hope that helps!
Mark

[vavai]

Hi LMStone,

Originally Posted by LMStone

Yes, we are now running ZCS-NE 6.0.4 on SLES10-SP3 (I'll update my profile shortly!)

Here is the /etc/syslog-ng/syslog-ng.conf file as tweaked by Zimbra for us earlier today:


The "destination local1" and related parameters for the second zimbra log file did not work; syslog refused to start.

Hope that helps!
Mark

Thank you, I'll be trying the tweaked syslog-ng.conf on SLES 11+ZCS 6.0.4.

[JBerbaum]

Originally Posted by LMStone

Fourth, LDAP replication was totally broken by this upgrade (and at this point remains broken). Our additional MTA and mailbox servers, which were originally LDAP replicas for performance, now talk to the LDAP master directly. Zimbra support has escalated this issue for us.

Mark,

Thanks for the rundown. If you can, any status updates on resolving the LDAP replication issue would be much appreciated. We have production servers running since 4.x as well, and plan to move from 5.x to 6.x soon.

[LMStone]

Originally Posted by JBerbaum

Mark,

Thanks for the rundown. If you can, any status updates on resolving the LDAP replication issue would be much appreciated. We have production servers running since 4.x as well, and plan to move from 5.x to 6.x soon.

No news from Zimbra so far today, but, yes, of course, I will post updates and the solutions here when we have them, no problem.

All the best,
Mark

[LMStone]

I thought it would be helpful to post a status update on things that have changed today...

First, we determined we had a failed zimlet upgrade issue during the install, so from the cli we undeployed and (re)deployed several zimlets to match the zimlets installed in our test box, a ZCS 6.0.4 system installed fresh at 6.0.2. We are having a Jetty problem with the Social zimlet, but everything else end-user facing is OK. (Only one of our hosted clients is clamoring for the Social zimlet, and several do not want it).

Second, we are missing Server Statistics and Certificates in the left-vertical nav bar of the Admin Console, and there is no License tab in Global Settings. On initial login to the Admin Console we are greeted with a full set of red X's, even though Zimbra on all of the servers is running fine. This thread shows something similar: ZCS 6.0.2 Admin Interface Missing Features and/or Broken

Third, the whole logger/stats thing is, we think, a combination of several things (at least for us). See upgrade to 6.0.2 stats and status no longer working for example.

After reading that thread carefully, I think our case is related, but there are at least two things going on here: 1) we use syslog-ng not syslogd, and 2) I don't think the updated syslogging portion of the installer completed successfully, though there is nothing in the logs we can see to confirm that.

Syslogging is not fully functional on our system just yet in that we can't get syslog-ng (remember, we are a SuSE shop and SuSE installs syslog-ng by default) to start with the Zimbra-supplied new 6.0 settings. So for the moment we are using the 5.0 settings in /etc/syslog-ng/syslog-ng.conf:

#ZCS 5.0 Syslogging code
filter f_local0       { facility(local0); }; # zimbra
destination zmail { file("/var/log/zimbra.log" owner("zimbra") ); }; # zimbra 
log { source(src); filter(f_mail); destination(zmail); }; # zimbra
destination local0 { file("/var/log/zimbra.log" owner("zimbra") ); }; # zimbra
log { source(src); filter(f_local0); destination(local0); }; # zimbra
filter f_auth       { facility(auth); }; # zimbra
destination zmauth { file("/var/log/zimbra.log" owner("zimbra") ); }; # zimbra
log { source(src); filter(f_auth); destination(zmauth); }; # zimbra

#ZCS 6.0 Syslogging code
#Syslog won't start with this Zimbra-supplied line below; syslog-ng complains of a syntax error.:
#source zimbra_src { unix-stream("/dev/log"; keep-alive(yes); max-connections(20););}; # zimbra
#Syslog will start with this modified line below
#source zimbra_src { unix-stream("/dev/log"); }; # zimbra
#filter zimbra_local0 { facility(local0); }; # zimbra
#filter zimbra_local1 { facility(local1); }; # zimbra
#filter zimbra_auth { facility(auth); }; # zimbra
#filter zimbra_mail { facility(mail); }; # zimbra
#destination zimbra_mail { file("/var/log/zimbra.log" owner("zimbra")); }; # zimbra
#destination zimbra_local1 { file("/var/log/zimbra-stats.log" owner("zimbra")); }; # zimbra
#destination zimbra_local0 { file("/var/log/zimbra.log" owner("zimbra")); }; # zimbra
#destination zimbra_auth { file("/var/log/zimbra.log" owner("zimbra")); }; # zimbra
#log { source(zimbra_src); filter(zimbra_mail); destination(zimbra_mail); }; # zimbra
#log { source(zimbra_src); filter(zimbra_local0); destination(zimbra_local0); }; # zimbra
#log { source(zimbra_src); filter(zimbra_local1); destination(zimbra_local1); }; # zimbra
#log { source(zimbra_src); filter(zimbra_auth); destination(zimbra_auth); }; # zimbra

Even when syslog-ng was started successfully with the amended ZCS 6.0 code, /var/log/zimbra-stats.log failed to populate.

Further, we have no /opt/zimbra/logger/db/data directory; I am going to try to create the directory and initialize the sqllite db as per this thread and see what happens: upgrade to 6.0.2 stats and status no longer working

More to come!

Any tips/suggestions appreciated!

With best regards to all,
Mark

[LMStone]

News from the front:

The missing bits in the Admin Console are "fixed". Essentially, the upgrade installer (at least in our case) does not apply the correct ZCS 6.0 Global Administrator rights to such accounts existing before the upgrade.

We created a new ZCS account, clicked the "Global Administrator" tick box, and now this user account when logged in to the Admin Console sees everything that should be seen by a Global Admin.

So, now we are down to:

1) broken stats, which we know in our case is related to syslog-ng configuration issues and possibly also to the sqlite database failed initialization during the upgrade, and

2) broken LDAP replication, which is in Zimbra's court at the moment.

3) broken Social zimlet throwing multiple Jetty 403 errors on service/proxy "forbidden" when navigating to the Social tab.

Happy New Year to all,
Mark

[raj]

Mark do you think these problems were specific to ZCS + SLES10 and/or to your setup only ?


Thanks
Raj

[LMStone]

Originally Posted by raj

Mark do you think these problems were specific to ZCS + SLES10 and/or to your setup only ?


Thanks
Raj

Hi Raj!

Our stats issue is partly specific to any distro using syslog-ng instead of syslogd IMHO. Since I don't know why the sqlite data directory didn't get created during the upgrade, I can't say if that part is a Zimbra-SLES problem or not.

The admin console issues I think are due to our hosting farm having been up since 4.0.3; so much has changed in the intervening versions that these kinds of upgrade issues with user accounts' rights do not surprise me.

The LDAP replication issue has me stumped (and Zimbra too at this point).

The one area in Zimbra where we have done nothing creative is LDAP. I really expect we will find something silly as the root cause here of replication no longer working.

Our hosting business is growing but we are not going to add any new servers to this farm and will instead start a new, totally separate Zimbra farm with a fresh install of ZCS 6.0.4/5.

Hope that helps,
Mark