Upgrade to 7.1.2 Massive Failure

[blackmogu]

Hi,

I did an upgrade from 7.1.1 to 7.1.2 of the open source zimbra edition, and stupidly chose the wrong package for CentOS 6 instead of CentOS 5. The install removed all packages and then bailed, leaving the server in an unusable state.

I took a tarball of the zimbra directory before, but when I reinstall Zimbra 7.1.1 and use this tarball, nothing can connect. I get the following errors on startup :-

Starting ldap...Done.
Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn't exist.

So, I tried a few things such as changing the ldap passwords, which gives me the error:-

Updating local config and LDAP
TLS: SSL connect attempt failed with unknown errorerror:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

I am at a loss as to how to get the server to start. Any help would be appreciated !

[phoenix]

Originally Posted by blackmogu

I took a tarball of the zimbra directory before, but when I reinstall Zimbra 7.1.1 and use this tarball, nothing can connect. I get the following errors on startup

Did you run zmfixperms after you restored the backup? Id suggest you do that and then run the install again for ZCS 7.1.1.

[blackmogu]

hi,

having done that you outlined, on startup zimbra gives :-

Starting ldap...Done.
Unable to determine enabled services from ldap.
Enabled services read from cache. Service list may be inaccurate.
Starting zmconfigd...Done.
Starting logger...Failed.
Starting logswatch...ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderE xception: unable to find valid certification path to requested target)
zimbra logger service is not enabled! failed

I'm at my wits end after battling for 2 days !
Thank you for your assistance.

[phoenix]

What's the status of the Zimbra services:

zmcontrol status

Have you tried regenerating the certificates? Is your DNS records and hosts file correct? Go to the Split DNS article and run all the commands in the 'Verify...' section of that article, just for confirmation. Do you see any other errors in the log files?

[blackmogu]

My hosts file, MX record and resolving are all correct. I've read through the posts you outlined and have no issues. The server was running fine for over 2 years until now. The server is using a public IP.

/opt/zimbra/bin/zmcertmgr createca -new
** Creating /opt/zimbra/ssl/zimbra/ca/zmssl.cnf...done
** Creating CA private key /opt/zimbra/ssl/zimbra/ca/ca.key...done.
** Creating CA cert /opt/zimbra/ssl/zimbra/ca/ca.pem...done.

/opt/zimbra/bin/zmcertmgr createcrt -new -days 365
Validation days: 365
** Creating /opt/zimbra/conf/zmssl.cnf...done
** Backup /opt/zimbra/ssl/zimbra to /opt/zimbra/ssl/zimbra.20110918190900
** Generating a server csr for download self -new -keysize 1024
** Creating /opt/zimbra/conf/zmssl.cnf...done
** Backup /opt/zimbra/ssl/zimbra to /opt/zimbra/ssl/zimbra.20110918190900
** Creating server cert request /opt/zimbra/ssl/zimbra/server/server.csr...done.
** Saving server config key zimbraSSLPrivateKey...failed.
** Signing cert request /opt/zimbra/ssl/zimbra/server/server.csr...done.

/opt/zimbra/bin/zmcertmgr deploycrt self
** Saving server config key zimbraSSLCertificate...failed.
** Saving server config key zimbraSSLPrivateKey...failed.
** Installing mta certificate and key...done.
** Installing slapd certificate and key...done.
** Installing proxy certificate and key...done.
** Creating pkcs12 file /opt/zimbra/ssl/zimbra/jetty.pkcs12...done.
** Creating keystore file /opt/zimbra/mailboxd/etc/keystore...failed.

Exception in thread "main" java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(Java KeyStore.java:771)
at sun.security.provider.JavaKeyStore$JKS.engineLoad( JavaKeyStore.java:38)
at java.security.KeyStore.load(KeyStore.java:1185)
at com.zimbra.cert.MyPKCS12Import.main(MyPKCS12Import .java:98)
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(Java KeyStore.java:769)
... 3 more

** Installing CA to /opt/zimbra/conf/ca...done.

/opt/zimbra/bin/zmcertmgr deployca
** Importing CA /opt/zimbra/ssl/zimbra/ca/ca.pem into CACERTS...done.
** Saving global config key zimbraCertAuthorityCertSelfSigned...failed.
** Saving global config key zimbraCertAuthorityKeySelfSigned...failed.
** Copying CA to /opt/zimbra/conf/ca...done.

[blackmogu]

zmcontrol status gives :-

antispam Running
antivirus Running
imapproxy Running
ldap Running
logger Stopped
zmlogswatchctl is not running
mailbox Stopped
mysql.server is not running.
zmmailboxdctl is not running.
memcached Running
mta Running
snmp Running
spell Running
stats Stopped
zmconfigd Running

On starting zimbra, I also got the error :-

Starting logswatch...ERROR: service.FAILURE (system failure: unable to lookup server by name: my.host.com message: [LDAP: error code 49 - Invalid Credentials]) (cause: javax.naming.AuthenticationException [LDAP: error code 49 - Invalid Credentials])
zimbra logger service is not enabled! failed.

[LMStone]

Originally Posted by blackmogu

I took a tarball of the zimbra directory before, but when I reinstall Zimbra 7.1.1 and use this tarball, nothing can connect. I get the following errors on startup :-

Was Zimbra running when you took this tarball?

[blackmogu]

possibly - I can't be sure.

[LMStone]

Without Zimbra having been shut down before taking a tarball of /opt/zimbra, many things won't be stored in the tar file in a restorable state. MySQL, LDAP etc. keeping much of their working set in memory, and changes made in RAM are not flushed to disk immediately.

The Zimbra installer does shut down Zimbra prior to running the upgrade, so what you had in /opt/zimbra just after the upgrade bailed sounds like your only "backup".

If you restored your tar file on top of the existing /opt/zimbra tree, you may have compromised your only "backup."

There are routines to recover mailstores as I understand it, but it's a manual process and I don't have any familiarity with them.

I wish I had better news...

[blackmogu]

hi all,

Here is what I did to get around my issue - it may be of some use to someone in the future.

1. I had an old backup that was 6 months old. I restored this and upgraded it to the same zimbra version as the corrupted one.

2. I copied the data and db and store directories from the corrupted installation to this recovered one.

3. Started the recovered zimbra no problems.

4. Used imapsync to sync the accounts to a new server.

Thank the gods that this worked ! I've had a horrible weekend with no end of calls about emails not working. Moral of this story is ...

BACK UP YOUR ZIMBRA INSTALLATION PROPERLY !

Once you have a disaster like this you'll never skimp again