Problem: successfully installing a commercial SSL cert breaks Zimbra NE LDAP access

[deltatango]

I just tried to deploy our commercial wild-card SSl cert from Digicert. I followed this: Installing a Network Solutions Certificate on ZCS 5.0.x - Zimbra :: Wiki and adapted it slightly:

1. I created a CSR via Zimbra Web interface

2. requested an additional Tomcat SSL cert on the DigiCert website (already using one for Apache on several websites)

3. Got files: commercial.csr and commercial.key (created by Zimbra), downloaded: DigiCertCA.crt (intermediate cert, TrustedRoot.crt (root cert), star_example.com.p7b, star_example.com.crt.

4. as root: "cat DigiCertCA.crt TrustedRoot.crt >/opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt"

5. checked the certfiles: "/opt/zimbra/bin/zmcertmgr verifycrt comm":
** Verifying /opt/zimbra/ssl/zimbra/commercial/commercial.crt against /opt/zimbra/ssl/zimbra/commercial/commercial.key
Certificate (/opt/zimbra/ssl/zimbra/commercial/commercial.crt) and private key (/opt/zimbra/ssl/zimbra/commercial/commercial.key) match.
Valid Certificate: /opt/zimbra/ssl/zimbra/commercial/commercial.crt: OK

6. deployed it: "/opt/zimbra/bin/zmcertmgr deploycrt comm commercial.crt commercial_ca.crt":
mail:/opt/zimbra/ssl/zimbra/commercial root# /opt/zimbra/bin/zmcertmgr deploycrt comm commercial.crt commercial_ca.crt
** Verifying commercial.crt against /opt/zimbra/ssl/zimbra/commercial/commercial.key
Certificate (commercial.crt) and private key (/opt/zimbra/ssl/zimbra/commercial/commercial.key) match.
Valid Certificate: commercial.crt: OK
** Copying commercial.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt
cp: /opt/zimbra/ssl/zimbra/commercial/commercial.crt and commercial.crt are identical (not copied).
** Appending ca chain commercial_ca.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt
cp: /opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt and commercial_ca.crt are identical (not copied).
** Saving server config key zimbraSSLCertificate...done.
** Saving server config key zimbraSSLPrivateKey...done.
** Installing mta certificate and key...done.
** Installing slapd certificate and key...done.
** Installing proxy certificate and key...done.
** Creating pkcs12 file /opt/zimbra/ssl/zimbra/jetty.pkcs12...done.
** Creating keystore file /opt/zimbra/mailboxd/etc/keystore...done.
** Installing CA to /opt/zimbra/conf/ca...done.

7. restarted Zimbra as user zimbra: "zmcontrol stop;zmcontrol start" but then I ge tthe dreaded error:
Starting ldap...Done.
Unable to determine enabled services from ldap.
Enabled services read from cache. Service list may be inaccurate.

After cleaning out /opt/zimbra/log/ I get this:
Starting ldap...Done.
Unable to determine enabled services from ldap.
Unable to determine enabled services. Cache is out of date or doesn't exist.

I just wanted to get fresh log files for easier diagnosing the problem but now I don't get any log messages at all ...
so it appears as if some components cannot access the LDAP store any longer.

How can that happen and what do I do now?

[brian]

your failure started at step 4. don't use /opt/zimbra/ssl/zimbra/ as the working directory, use a temporary directory.

[deltatango]

Thanks, Brian, I now moved commercial.crt and commercial_ca.crt away and tried to re-deploy the cert:
mail:/opt/zimbra/temp root# ../bin/zmcertmgr deploycrt comm commercial.crt commercial_ca.crt
** Verifying commercial.crt against /opt/zimbra/ssl/zimbra/commercial/commercial.key
Certificate (commercial.crt) and private key (/opt/zimbra/ssl/zimbra/commercial/commercial.key) match.
Valid Certificate: commercial.crt: OK
** Copying commercial.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt
** Appending ca chain commercial_ca.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt
** Saving server config key zimbraSSLCertificate...failed.
** Saving server config key zimbraSSLPrivateKey...failed.
** Installing mta certificate and key...done.
** Installing slapd certificate and key...done.
** Installing proxy certificate and key...done.
** Creating pkcs12 file /opt/zimbra/ssl/zimbra/jetty.pkcs12...done.
** Creating keystore file /opt/zimbra/mailboxd/etc/keystore...done.
** Installing CA to /opt/zimbra/conf/ca...done.

mail:/opt/zimbra/temp root# su -l zimbra
mail:~ zimbra$ zmcontrol start
Host mail.proximic.com
Starting ldap...Done.
Unable to determine enabled services from ldap.
mail:~ zimbra$

the re-deployment didn't help at all and the installation is still not working. Luckily this is a test installation, thus no production is affected. Will re-install zimbra and start anew.

[deltatango]

And the Wiki howto is then completely wrong?
-> Installing a Network Solutions Certificate on ZCS 5.0.x - Zimbra :: Wiki

[deltatango]

This is becoming ridiculous: I restored the /opt/zimbra tree from a backup I took before I touched any ssl stuff. Then I tried to install the commercial cert again:

1. created csr via web admin (I failed at several attempts to install the cert via web admin, thus I went back to command line tools for further steps) which left both the csr and key in /opt/zimbra/ssl/zimbra/commercial/

2. had my cert created by Digicert (luckily one can create additional certs for free once one has purchased a wild card cert) and downloaded

3. Instead of the Wiki document I mentioned in my previous post, I now followed this: Administration Console and CLI Certificate Tools - Zimbra :: Wiki, which also claims to be certified, thus is expected to be somewhat trustworthy.

4. as I already had a commercial.key and commercial.csr (created by the web admin) I started with "cat /tmp/ca.crt /tmp/ca_intermediary.crt > /tmp/ca_chain.crt"

5. Then wanted to verify cert and key:
/opt/zimbra/bin/zmcertmgr verifycrt comm \
/opt/zimbra/ssl/zimbra/commercial/commercial.key /tmp/commercial.crt

Which failed until I temporarily copied /tmp/ca_chain.crt to /opt/zimbra/ssl/zimbra/commercial/commercial_ca.crt, then it worked:

mail:/opt/zimbra/backup root# /opt/zimbra/bin/zmcertmgr verifycrt comm /opt/zimbra/ssl/zimbra/commercial/commercial.key /tmp/commercial.crt
** Verifying /tmp/commercial.crt against /opt/zimbra/ssl/zimbra/commercial/commercial.key
Certificate (/tmp/commercial.crt) and private key (/opt/zimbra/ssl/zimbra/commercial/commercial.key) match.
Valid Certificate: /tmp/commercial.crt: OK

Now is the time to deploy it. I removed commercial_ca.crt from /opt/zimbra/ssl/zimbra/commercial and deployed:

mail:/opt/zimbra/backup root# /opt/zimbra/bin/zmcertmgr deploycrt comm /tmp/commercial.crt /tmp/commercial_ca.crt
** Verifying /tmp/commercial.crt against /opt/zimbra/ssl/zimbra/commercial/commercial.key
Certificate (/tmp/commercial.crt) and private key (/opt/zimbra/ssl/zimbra/commercial/commercial.key) match.
Valid Certificate: /tmp/commercial.crt: OK
** Copying /tmp/commercial.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt
** Appending ca chain /tmp/commercial_ca.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt
** Saving server config key zimbraSSLCertificate...done.
** Saving server config key zimbraSSLPrivateKey...done.
** Installing mta certificate and key...done.
** Installing slapd certificate and key...done.
** Installing proxy certificate and key...done.
** Creating pkcs12 file /opt/zimbra/ssl/zimbra/jetty.pkcs12...done.
** Creating keystore file /opt/zimbra/mailboxd/etc/keystore...done.
** Installing CA to /opt/zimbra/conf/ca...done.
mail:/opt/zimbra/backup root#

Looks good, doesn't it?
Now I switched to user zimbra for restarting all the services: "zmcontrol stop", then:

mail:~ zimbra$ zmcontrol start
Host mail.example.com
Starting ldap...Done.
Unable to determine enabled services from ldap.
Enabled services read from cache. Service list may be inaccurate.

What the hell am I doing wrong here?

[quanah]

Try the following:

cd /opt/zimbra/conf/ca
rm *
cd /opt/zimbra/ssl/
mv zimbra zimbra.dead

Then re-run the cert deployment pieces.

[digicert]

We've had another customer experience this error before. They said it was resolved by re-installing the certificate and rebooting the server before running the zmcontrol commands. Specifically they said they followed these instructions:

"1. You must use the web UI admin console to generate the CSR file. You must choose "all server". Do not choose your domain.
2. After generating the CSR file, open it and generate the cert on DigiCert website. Choose “Other” as the web server type.
3. Download the three files domain.crt, DigicertCa.crt and Trustedroot.crt into */opt/zimbra/ssl/zimbra/commercial/* with root account
4. Cat the three files into the commercial_ca.crt (3 file : domain.crt, DigicertCa.crt and Trustedroot.crtnot 2 file: DigicertCa.crt and Trustedroot.crt) and rename domain.crt to commercial.crt
5. Verify this crt with root command
cd /opt/zimbra/ssl/zimbra/commercial/
/opt/zimbra/bin/zmcertmgr verifycrt comm ./commercial.key ./commercial.crt ./commercial_ca.crt
6. Add CA cert */opt/zimbra/bin/zmcertmgr addcacert ./commercial.crt*
/opt/zimbra/bin/zmcertmgr addcacert ./commercial_ca.crt
7.Deploy
/opt/zimbra/bin/zmcertmgr deploycrt comm ./commercial.crt ./commercial_ca.crt
8 After it said all is done, reboot this server. Don't stop and start with zmcontrol, just reboot. I don't know why but it successful. After reboot you can start and stop with zmcontrol you always get the correct message."

You can check if it is installed correctly at SSL Certificate Tester - Check Certificates (you can check LDAP by including :636 after the hostname)

If that still doesn't work, please give us a call (+1 800-896-7973) and we'll see if there is anything else we can try.

Dan
DigiCert Support

[steeef]

I'm having this same issue with a certificate issued by GoDaddy. I also removing the files in ca and moving the zimbra directory to zimbra.dead, but it still complains about reading service info from cache. Also, all services are unresponsive (https, imaps), and I can't seem to get more log info.

[steeef]

Also tried restarting without stopping/starting Insight, no change.

I'm forced to use the command line to create the csr, since GoDaddy requires a key length of at least 2048 bytes.

[steeef]

I think I found the solution. Run this as root:

/opt/zimbra/java/bin/keytool -import -alias root -keystore /opt/zimbra/java/jre/lib/security/cacerts -storepass changeit -file /opt/zimbra/conf/ca/commercial_ca.pem

Credit goes to cpiess: ZCS Network Install: Was working, now possibly broken due to SSL?