[SOLVED] SOLVED: Zimbra 6.0.1 stop working if SSL certificate is expired

[eaperezh]

To document this issue for the future.

I have a Centos 5 server with the following version of Zimbra: Release 6.0.1_GA_1816.RHEL5_20090911181524 CentOS5 FOSS edition.

Today (october 20, 2010) the SSL certificate installed on the server expired.
The symptoms the users had:
a- no web interface at all.
b- admin interface not available
c- zimbra desktop unable to connect

The logs showed the following:

[root@correo log]# tail zmmtaconfig.log -n 100
Wed Oct 20 14:12:34 2010 Skipping All MTA Authentication Target URLs update.
Wed Oct 20 14:12:34 2010 Skipping getAllMtaAuthURLs ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:12:36 2010 Skipping Configuration for server correo.binal.ac.pa update.
Wed Oct 20 14:12:36 2010 gs:correo.binal.ac.pa ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:12:36 2010 Sleeping...Key lookup failed.
Wed Oct 20 14:12:43 2010 Skipping Global system configuration update.
Wed Oct 20 14:12:43 2010 gacf ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:12:45 2010 Skipping All Reverse Proxy URLs update.
Wed Oct 20 14:12:45 2010 Skipping getAllReverseProxyURLs ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:12:47 2010 Skipping All Reverse Proxy Backends update.
Wed Oct 20 14:12:47 2010 Skipping getAllReverseProxyBackends ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:12:49 2010 Skipping All Memcached Servers update.
Wed Oct 20 14:12:49 2010 Skipping getAllMemcachedServers ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:12:51 2010 Skipping All MTA Authentication Target URLs update.
Wed Oct 20 14:12:51 2010 Skipping getAllMtaAuthURLs ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:12:53 2010 Skipping Configuration for server correo.binal.ac.pa update.
Wed Oct 20 14:12:53 2010 gs:correo.binal.ac.pa ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:12:53 2010 Sleeping...Key lookup failed.
Wed Oct 20 14:13:00 2010 Skipping Global system configuration update.
Wed Oct 20 14:13:00 2010 gacf ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:13:02 2010 Skipping All Reverse Proxy URLs update.
Wed Oct 20 14:13:02 2010 Skipping getAllReverseProxyURLs ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:13:04 2010 Skipping All Reverse Proxy Backends update.
Wed Oct 20 14:13:04 2010 Skipping getAllReverseProxyBackends ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:13:06 2010 Skipping All Memcached Servers update.
Wed Oct 20 14:13:06 2010 Skipping getAllMemcachedServers ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:13:08 2010 Skipping All MTA Authentication Target URLs update.
Wed Oct 20 14:13:08 2010 Skipping getAllMtaAuthURLs ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:13:10 2010 Skipping Configuration for server correo.binal.ac.pa update.
Wed Oct 20 14:13:10 2010 gs:correo.binal.ac.pa ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:13:10 2010 Sleeping...Key lookup failed.
Wed Oct 20 14:13:17 2010 Skipping Global system configuration update.
Wed Oct 20 14:13:17 2010 gacf ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:13:19 2010 Skipping All Reverse Proxy URLs update.
Wed Oct 20 14:13:19 2010 Skipping getAllReverseProxyURLs ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:13:21 2010 Skipping All Reverse Proxy Backends update.
Wed Oct 20 14:13:21 2010 Skipping getAllReverseProxyBackends ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:13:23 2010 Skipping All Memcached Servers update.
Wed Oct 20 14:13:23 2010 Skipping getAllMemcachedServers ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:13:25 2010 Skipping All MTA Authentication Target URLs update.
Wed Oct 20 14:13:25 2010 Skipping getAllMtaAuthURLs ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:13:27 2010 Skipping Configuration for server correo.binal.ac.pa update.
Wed Oct 20 14:13:27 2010 gs:correo.binal.ac.pa ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:13:27 2010 Sleeping...Key lookup failed.
Wed Oct 20 14:13:34 2010 Skipping Global system configuration update.
Wed Oct 20 14:13:34 2010 gacf ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:13:36 2010 Skipping All Reverse Proxy URLs update.
Wed Oct 20 14:13:36 2010 Skipping getAllReverseProxyURLs ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:13:38 2010 Skipping All Reverse Proxy Backends update.
Wed Oct 20 14:13:38 2010 Skipping getAllReverseProxyBackends ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.net.ssl.SSLHandshakeException sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: signature check failed)
Wed Oct 20 14:13:45 2010 Sleeping...Key lookup failed.

So after a lot of search in the forum, the error messages suggested that the problem was the SSL Certificate. So i had to regenerate the certificate. I will use a self signed one here, since my new cert has not arrived yet.

Single-Node Self-Signed Certificate

1. Begin by generating a new Certificate Authority (CA).

zmcertmgr createca -new

2. Then generate a certificate signed by the CA that expires in 365 days.

zmcertmgr createcrt -new -days 365

3. Next deploy the certificate.

zmcertmgr deploycrt self

4. Next deploy the CA.

zmcertmgr deployca

5. To finish, verify the certificate was deployed to all the services.

zmcertmgr viewdeployedcrt

Now, in order to avoid LDAP crashing about the invalid key/hash, we have to import the new CA.


Note: some other user reported in a forum that this step may be necesary:

/opt/zimbra/java/bin/keytool -delete -alias root -keystore /opt/zimbra/java/jre/lib/security/cacerts -storepass changeit

But this is the only step i used:

/opt/zimbra/java/bin/keytool -import -alias root -keystore /opt/zimbra/java/jre/lib/security/cacerts -storepass changeit -file /opt/zimbra/conf/ca/ca.pem

The command will report the owner, issuer, serial number and validity. At the question of "Trust this certificate?" please answer yes.
the command will report: "Certificate was added to keystore"

Now, please do:
su - zimbra
zmcontrol stop
zmcontrol start

After this, all will be working again.

Note to zimbra team: Where is the documentation for this? Where is the FAQ for this?
Additional note: a cron job run by zimbra, that one every month send the expiration date of the certificate, so we dont forget?

[pclyne]

thank you eaperezh your fix worked for me too. It's definitely surprising that this effectively kills Zimbra rather than just getting an expired certificate warning...

[useopenid]

I just ran into this last Friday (15 minutes before I was planning on leaving for our annual dinner! Thank you zimbra for making me miss it! :-( )

The new CA seems extraneous however. Shouldn't the orginally created one work ok, or does it have a short lifetime too?

I generated a 10 year cert to avoid the problem in the future, hopefully zimbra will have an improved process by then!

I would also note that in a multi-server environment, you want to do the ldap servers *first* and then restart zimbra there. "deploycrt" tries to put the cert in ldap (I think) and it can't remotely with the ldap server using an expired cert.

[lime]

Thanks eaperezh, it did works perfectly. Well done, and many thanks for saving time to suppot zimbra !!!

[MGSteve]

Thank you also for your post, its just happened to me on version 7.

Some things about Zimbra really surprise me, but this has to be the most unprofessional issue I've found so far.

Why no warning to the admin and secondly, why does the SSL expiring kill SMTP logins.

I can still login via POP3, but only found out about the problem when I went to send a mail.