LDAP failure? Upgrade 4.5.6 - 4.5.9 failed

[jeschulze]

Hello,

Running Ubuntu 6.06, Zimbra has been running great for weeks, today decided to update to 4.5.9 and everything went fine until the end, when i got a error indicative of SlapD hanging around - ERROR: service.FAILURE (system failure: getDirectContext) (cause: javax.naming.CommunicationException localhost.localdomain:389)

Followed the instructions for resolving this in LDAP - Zimbra :: Wiki

restarted zimbra installation with ./install.sh, the following are the results:

root@mail:/usr/src/zcs# ./install.sh

Operations logged to /tmp/install.log.4885
Checking for existing installation...
zimbra-ldap...FOUND zimbra-ldap-4.5.9_GA_1454.UBUNTU6
zimbra-logger...FOUND zimbra-logger-4.5.9_GA_1454.UBUNTU6
zimbra-mta...FOUND zimbra-mta-4.5.9_GA_1454.UBUNTU6
zimbra-snmp...FOUND zimbra-snmp-4.5.9_GA_1454.UBUNTU6
zimbra-store...FOUND zimbra-store-4.5.9_GA_1454.UBUNTU6
zimbra-apache...FOUND zimbra-apache-4.5.9_GA_1454.UBUNTU6
zimbra-spell...FOUND zimbra-spell-4.5.9_GA_1454.UBUNTU6
zimbra-core...FOUND zimbra-core-4.5.9_GA_1454.UBUNTU6

Saving existing configuration file to /opt/zimbra/.saveconfig

Checking for prerequisites...
NPTL...FOUND
sudo...FOUND sudo-1.6.8p12-1ubuntu6
libidn11...FOUND libidn11-0.5.18-1
curl...FOUND curl-7.15.1-1ubuntu2.1
fetchmail...FOUND fetchmail-6.3.2-2ubuntu2.1
libpcre3...FOUND libpcre3-6.4-1.1ubuntu4
libgmp3c2...FOUND libgmp3c2-4.1.4-11ubuntu2
libexpat1...FOUND libexpat1-1.95.8-3
libxml2...FOUND libxml2-2.6.24.dfsg-1ubuntu1
libstdc++6...FOUND libstdc++6-4.0.3-1ubuntu5
libstdc++5...FOUND libstdc++5-1:3.3.6-10
openssl...FOUND openssl-0.9.8a-7ubuntu0.3
Checking current number of databases...

Do you want to verify message store database integrity? [Y] n

Checking for installable packages

Found zimbra-core
Found zimbra-ldap
Found zimbra-logger
Found zimbra-mta
Found zimbra-snmp
Found zimbra-store
Found zimbra-apache
Found zimbra-spell


The Zimbra Collaboration Suite appears already to be installed.
It can be upgraded with no effect on existing accounts,
or the current installation can be completely removed prior
to installation for a clean install.

Do you wish to upgrade? [Y] y

Select the packages to install
Upgrading zimbra-core
Upgrading zimbra-ldap
Upgrading zimbra-logger
Upgrading zimbra-mta
Upgrading zimbra-snmp
Upgrading zimbra-store
Upgrading zimbra-apache
Upgrading zimbra-spell
Checking required space for zimbra-core
checking space for zimbra-store

Installing:
zimbra-core
zimbra-ldap
zimbra-logger
zimbra-mta
zimbra-snmp
zimbra-store
zimbra-apache
zimbra-spell

The system will be modified. Continue? [N] y

Shutting down zimbra mail

Backing up ldap


Removing existing packages

zimbra-ldap...done
zimbra-logger...done
zimbra-mta...done
zimbra-snmp...done
zimbra-store...done
zimbra-spell...done
zimbra-apache...done
zimbra-core...done

Removing deployed webapp directories
Installing packages

zimbra-core......zimbra-core_4.5.9_GA_1454.UBUNTU6_i386.deb...done
zimbra-ldap......zimbra-ldap_4.5.9_GA_1454.UBUNTU6_i386.deb...done
zimbra-logger......zimbra-logger_4.5.9_GA_1454.UBUNTU6_i386.deb...done
zimbra-mta......zimbra-mta_4.5.9_GA_1454.UBUNTU6_i386.deb...done
zimbra-snmp......zimbra-snmp_4.5.9_GA_1454.UBUNTU6_i386.deb...done
zimbra-store......zimbra-store_4.5.9_GA_1454.UBUNTU6_i386.deb...done
zimbra-apache......zimbra-apache_4.5.9_GA_1454.UBUNTU6_i386.deb...done
zimbra-spell......zimbra-spell_4.5.9_GA_1454.UBUNTU6_i386.deb...done

Setting defaults from saved config in /opt/zimbra/.saveconfig/config.save
HOSTNAME=mail.piena.com
LDAPHOST=
LDAPPORT=
SNMPTRAPHOST=
SMTPSOURCE=
SMTPDEST=
SNMPNOTIFY=0
SMTPNOTIFY=0
LDAPROOTPW=zimbra
LDAPZIMBRAPW=zimbra

Restoring existing configuration file from /opt/zimbra/.saveconfig/config.save...done
Operations logged to /tmp/zmsetup.log.9680
ERROR: service.FAILURE (system failure: getDirectContext) (cause: javax.naming.CommunicationException localhost:389)
ERROR: service.FAILURE (system failure: getDirectContext) (cause: javax.naming.CommunicationException localhost:389)
Setting defaults...Done
Setting defaults from existing config...Warning: null valued key 'mysql_logger_root_password'
Upgrading from 4.5.6_GA_1044 to 4.5.9_GA_1454
Stopping zimbra services
Stop failed - exiting
UPGRADE FAILED - exiting

previously the upgrade had hung up on restart of mysql, (or not) I unwisely tried to copy the error out of the terminal window and this resulted in a hault of the install. So, it looks like it halfway installed to me, it just has some damaged config data? If i do a clean install will user mailboxes be preserved?

And this is why we need the network edition... >.< Help!

[jeschulze]

btw, here are the contents of the upgrade log file;

Getting installed packages
Getting local config zimbra_server_hostname
Getting local config ldap_url
zimbra_server_hostname not in ldap_url not starting slapd
Getting installed services from ldap
checking isEnabled zimbra-core
zimbra-core not in enabled cache
enabled packages
Getting local config ldap_url
zimbra_server_hostname not in ldap_url not starting slapd
Getting enabled services from ldap
Marking zimbra-core as installed. Services for zimbra-core will be enabled.
Marking zimbra-ldap as installed. Services for zimbra-ldap will be enabled.
Marking zimbra-store as installed. Services for zimbra-store will be enabled.
Marking zimbra-mta as installed. Services for zimbra-mta will be enabled.
Marking zimbra-snmp as installed. Services for zimbra-snmp will be enabled.
Marking zimbra-logger as installed. Services for zimbra-logger will be enabled.
Marking zimbra-apache as installed. Services for zimbra-apache will be enabled.
Marking zimbra-spell as installed. Services for zimbra-spell will be enabled.
Setting defaults...
Setting local config zimbra_java_home to /opt/zimbra/java
*** Running as zimbra user: /opt/zimbra/bin/zmlocalconfig -f -e zimbra_java_home='/opt/zimbra/java'
checking isEnabled zimbra-store
zimbra-store is enabled
checking isEnabled zimbra-ldap
zimbra-ldap is enabled
checking isEnabled zimbra-store
zimbra-store is enabled
Done
Setting defaults from existing config...
Getting local config zimbra_server_hostname
Getting local config ldap_master_url
Getting local config ldap_port
Getting local config ldap_host
Getting local config ldap_root_password
Getting local config mysql_root_password
Getting local config mysql_logger_root_password
Getting local config zimbra_mysql_password
Getting local config zimbra_logger_mysql_password
Getting local config tomcat_java_heap_memory_percent
Getting local config mysql_memory_percent
Getting local config tomcat_keystore_password
Getting local config tomcat_keystore_password
Getting local config tomcat_truststore_password
Getting local config tomcat_truststore_password
Upgrading from 4.5.6_GA_1044 to 4.5.9_GA_1454
Stopping zimbra services
Stop failed - exiting
UPGRADE FAILED - exiting

[jeschulze]

This is hindering our buisness.. yes, i know, we need to purchase network for support. believe me, i am trying to squeeze it out of them. But if i could get one question answered, i can not find in the wiki or on the forums:

clean install option in the install script, WILL it destroy all the accounts, or just the configuration? This needs to be back online.

[brian]

restore /opt/zimbra/conf/localconfig.xml from your backup and rerun the upgrade.

[jeschulze]

I guess I've dug my own hole; No, i did not take a backup before upgrade. And it appears the Open source backup scripts i installed were not operating properly.. and unless the install did some sort of backup first, I have none. I keep getting this when I do a zmcontrol status:


zimbra@mail:/usr/src/zcs$ zmcontrol status
Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn't exist.

That leads to "ldap_url and ldap_master_url cannot be the same on an ldap replica" when i do 'ldap'. I have looked everywhere for this variable, and tried to follow the 'revert replica to master' Wiki page, but the information it tells me to look for in slapd.conf.in does not exist; i suppose the conf file is incomplete? I am unsure how to proceed.

edit: I looked at the localconfig.xml file, and most of the ldap entries are empty. File posting difficult, attaching.

[mmorse]

Would you happen to still have OK stuff in /opt/zimbra/.saveconfig/ config.save, .keys, certs, etc?

Before you keep mucking with it, the first thing you need to do is make a backup of your /opt/zimbra
Yes it's a copy of the 'bad' but do so anyway - the simplest would be to:
zmcontrol stop
ps aux | grep zimbra (kill any remaining)
rsync -avHK /opt/zimbra/ /backup459failed/zimbra
and if you want:
tar -zcvf /tmp/mail.backup.gz -C /backup459failed/zimbra .

[jeschulze]

Yes, i do. Although i've started install a few times, i hope it is not a save of a bad config. (it is listed as 4:21 today, which the problem occured around 1 pm.)

Also I have got past the LDAP problem, although still recieving the java error. Install has been stuck at starting mysql for 5 min now.

Edit: I found a backup of the system from August. Attempting re-install from that localconfig.xml file...

Edit: Install has proceeded to the point of 'updating from GA_4.5.6' and the recieves the 'getdirectcontext' java error. Update proceeds, but then hangs at 'starting MYSQL' This is what happened last time before artificially cancelled the installation.

Edit: Cancelled install, at starting mysql for last 20 min. This was the only error: ERROR: service.FAILURE (system failure: getDirectContext) (cause: javax.naming.CommunicationException localhost:389) According to Wiki, this is a LDAP problem; but nothing checks out cause LDAP won't start because the install will not finish.

[jeschulze]

We are suffering a bit here, and I am seriously going to talk to people about acquiring a license for zimbra if this is going to be our core email system; in the meantime, is there a way to pay for a single incident of phone tech support? I have found a backup, and the system was a clean install originally. I followed procedures for upgrade and now we are offline until tomorrow, we hope. Please say there is a way to get this resolved quickly!

[mmorse]

Offered to take a look:

Originally Posted by jeschulze

Thank you so very much for your offer. I was in a bit of a panic, this is my first commercial email installation and not quite as easily repaired as a sendmail config, so I did panic a bit.

I was able to switch around the .config files during mid install and get Zimbra to start. So far it appears to be running fine, but I'll keep that in mind for the next disaster. Which I hope, doesn't come to soon!

While your still on OS be sure to backup often! -About the quickest you can do is to shutdown zimbra then rsync your /opt/zimbra.

Originally Posted by jeschulze

I have found a backup, and the system was a clean install originally. I followed procedures for upgrade

So I gather right now you've just considered the stuff lost, the question is did you make a backup of the 'bad 4.5.9' (as mentioned ^above^) to see if you can move it to a test box and pull any mail remnants out of it?

[jeschulze]

Backups; For some reason the backup script I was using, which I found in the Wiki, would bring down the system and perform the backup fine, but could not reliably bring it back up. I would come back in the morning to find half of Zimbra's services failed to start, or did start but were not operating properly. (like the MTA, mail was not getting threw.) Since this machine is in production I'd rather not 'try' other backup methods and just wait till we go to Network edition, at which point we are looking at getting a different machine for that all-together.

As for my comment on clean install, I ment that when I brought the machine up in August, it was a clean install of the OS and zimbra, so nothing should be interfearing with it. Sorry, that was misleading. Thankfully no users or there mail appears to have been lost in this 4.5.9 issue.