[SOLVED] HELP - Mailboxd not running after Cert install

[bobm]

Hi guys -
I hope somebody can help , Here's the status.

On a FC7,5.0.10GA self signed certs running server I installed a Godaddy Cert.
from the GUI - everything installed fine(or so it looked)

HTTPS to the webmail worked fine and the cert returned was the GoDaddy cert but I found I had the following:

Starting ldap...Done.
Unable to determine enabled services from ldap.
Enabled services read from cache. Service list may be inaccurate.
I have a split DNS and found the hostname was not returning the FQDN - so I fixed that - no help.
After going in circles for a while - I thought that I could do a upgrade/install.sh and go back to the original self signed certs - BAD MOVE.

I then manually installed the GD cert:

[root@mail2 commercial]# ls
commercial_ca.crt commercial.crt commercial.csr commercial.key
[root@mail2 commercial]# rm commercial_ca.crt commercial.crt ../commercial_ca.crt
[root@mail2 commercial]# !pushd
pushd /home/bobm/Cert
~/Cert /opt/zimbra/ssl/zimbra/commercial
[root@mail2 Cert]# !788
cp mail2.metromotorgroup.com.crt /opt/zimbra/ssl/zimbra/commercial/commercial.crt
[root@mail2 Cert]# !789
cat gd_cross_intermediate.crt gd_intermediate.crt gd_bundle.crt>>/opt/zimbra/ssl/zimbra/commercial_ca.crt
[root@mail2 Cert]# !793
vi /opt/zimbra/ssl/zimbra/commercial_ca.crt

------END CERTIFICATE-----------BEGIN CERTIFICATE-----

[root@mail2 Cert]# !794
/opt/zimbra/bin/zmcertmgr verifycrt comm /opt/zimbra/ssl/zimbra/commercial/commercial.key /opt/zimbra/ssl/zimbra/commercial/commercial.crt /opt/zimbra/ssl/zimbra/commercial_ca.crt
** Verifying /opt/zimbra/ssl/zimbra/commercial/commercial.crt against /opt/zimbra/ssl/zimbra/commercial/commercial.key
Certificate (/opt/zimbra/ssl/zimbra/commercial/commercial.crt) and private key (/opt/zimbra/ssl/zimbra/commercial/commercial.key) match.
Valid Certificate: /opt/zimbra/ssl/zimbra/commercial/commercial.crt: OK

[root@mail2 Cert]# !795
/opt/zimbra/bin/zmcertmgr deploycrt comm /opt/zimbra/ssl/zimbra/commercial/commercial.crt /opt/zimbra/ssl/zimbra/commercial_ca.crt
** Verifying /opt/zimbra/ssl/zimbra/commercial/commercial.crt against /opt/zimbra/ssl/zimbra/commercial/commercial.key
Certificate (/opt/zimbra/ssl/zimbra/commercial/commercial.crt) and private key (/opt/zimbra/ssl/zimbra/commercial/commercial.key) match.
Valid Certificate: /opt/zimbra/ssl/zimbra/commercial/commercial.crt: OK
** Copying /opt/zimbra/ssl/zimbra/commercial/commercial.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt
cp: `/opt/zimbra/ssl/zimbra/commercial/commercial.crt' and `/opt/zimbra/ssl/zimbra/commercial/commercial.crt' are the same file
** Appending ca chain /opt/zimbra/ssl/zimbra/commercial_ca.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt
** Installing mta certificate and key...done.
** Installing slapd certificate and key...done.
** Installing proxy certificate and key...done.
** Creating pkcs12 file /opt/zimbra/ssl/zimbra/jetty.pkcs12...done.
** Creating keystore file /opt/zimbra/mailboxd/etc/keystore...done.
** Installing CA to /opt/zimbra/conf/ca...done.

[root@mail2 commercial]# !707
/opt/zimbra/bin/zmcertmgr viewdeployedcrt
::service mta::
notBefore=Jan 9 03:23:25 2009 GMT
notAfter=Jan 9 03:23:25 2011 GMT
subject= /O=mail2.metromotorgroup.com/CN=mail2.metromotorgroup.com/OU=Domain Control Validated
issuer= /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07969287
SubjectAltName= mail2.metromotorgroup.com, www.mail2.metromotorgroup.com
::service proxy::
notBefore=Jan 9 03:23:25 2009 GMT
notAfter=Jan 9 03:23:25 2011 GMT
subject= /O=mail2.metromotorgroup.com/CN=mail2.metromotorgroup.com/OU=Domain Control Validated
issuer= /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07969287
SubjectAltName= mail2.metromotorgroup.com, www.mail2.metromotorgroup.com
::service mailboxd::
notBefore=Jan 9 03:23:25 2009 GMT
notAfter=Jan 9 03:23:25 2011 GMT
subject= /O=mail2.metromotorgroup.com/CN=mail2.metromotorgroup.com/OU=Domain Control Validated
issuer= /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07969287
SubjectAltName= mail2.metromotorgroup.com, www.mail2.metromotorgroup.com
::service ldap::
notBefore=Jan 9 03:23:25 2009 GMT
notAfter=Jan 9 03:23:25 2011 GMT
subject= /O=mail2.metromotorgroup.com/CN=mail2.metromotorgroup.com/OU=Domain Control Validated
issuer= /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure Certification Authority/serialNumber=07969287
SubjectAltName= mail2.metromotorgroup.com, www.mail2.metromotorgroup.com

Now the mailboxd is DOA: I get no error when I zmcontrol start:

[zimbra@mail2 log]$ zmcontrol start
Host mail2.metromotorgroup.com
Starting ldap...Done.
Starting logger...Done.
Starting mailbox...Done.
Starting antispam...Done.
Starting antivirus...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.
Starting stats...Done.
[zimbra@mail2 log]$ zmcontrol status
Host mail2.metromotorgroup.com
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Stopped
zmmailboxdctl is not running
mta Running
snmp Running
spell Running
stats Running
I tried to monitor the startup...
zmmailboxdctl start
.... lots of exports...

+ /opt/zimbra/bin/zmtlsctl
Setting tls mode to both
Updating /opt/zimbra/mailboxd/etc/jetty.xml.in...done.
Updating /opt/zimbra/jetty/etc/zimbra.web.xml.in...done.
Updating /opt/zimbra/jetty/etc/zimbraAdmin.web.xml.in...done.
Updating PROTOCOL MODE in /opt/zimbra/mailboxd/etc/zimbra.web.xml.in...done.
Rewriting config files for webxml and mailboxd...done.
Updating /opt/zimbra/cyrus-sasl/etc/saslauthd.conf.in...done.
Rewriting config files for cyrus-sasl...done.
Setting ldap config zimbraMailMode both for mail2.metromotorgroup.com...done.
+ sudo /opt/zimbra/libexec/zmmailboxdmgr status
+ '[' 1 = 0 ']'
+ '[' xjetty = xtomcat ']'
+ mkdir -p /opt/zimbra/mailboxd/work/service/jsp
+ mkdir -p /opt/zimbra/mailboxd/work/zimbra/jsp
+ mkdir -p /opt/zimbra/mailboxd/work/zimbraAdmin/jsp
+ mailboxd_thread_stack_size=256k
++ echo -client -XX:NewRatio=2 -Djava.awt.headless=true -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1
++ grep Xss
+ '[' -z '' ']'
+ mailboxd_java_options='-client -XX:NewRatio=2 -Djava.awt.headless=true -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -Xss256k'
+ sudo /opt/zimbra/libexec/zmmailboxdmgr start -Xms1484m -Xmx1484m -client -XX:NewRatio=2 -Djava.awt.headless=true -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -Xss256k
+ status=0
+ '[' 0 = 0 ']'
+ echo 'mailboxd started.'
mailboxd started.
+ exit 0

and this is all I see in /opt/zimbra/log/mailbox.log:

at org.mortbay.start.Main.invokeMain(Main.java:183)
at org.mortbay.start.Main.start(Main.java:497)
at org.mortbay.start.Main.main(Main.java:115)
521 INFO [Shutdown] log - Shutdown hook executing
521 INFO [Shutdown] log - Shutdown hook complete
CompilerOracle: exclude com/zimbra/cs/session/SessionMap putAndPrune
CompilerOracle: exclude com/zimbra/cs/mailbox/MailItem delete
0 INFO [main] log - Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
Zimbra server reserving server socket port=143 bindaddr=null ssl=false
Zimbra server reserving server socket port=7025 bindaddr=null ssl=false
510 WARN [main] log - Config error at <Call name="open"/>
510 WARN [main] log - Config error at <Ref id="admin"><Call name="open"/></Ref>
511 WARN [main] log - EXCEPTION
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Nativ e Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Native MethodAccessorImpl.java:39)

For my 2 cents - I still think the system does not have the right info for using the cert installed - but I don't know what to check next, and I need this up ASAP.

CAN ANYBODY - HELP....

bobm

[iway]

Is the hostname different from the cert host name? I remember this problem here in the forums. If so, you can make Zimbra ignore cert mismatch from the command line. Search the forum for this.

[bobm]

Alas - no they are the same
hostname="mail2.metromotorgroup.com"
and the cert is for "mail2.metromotorgroup.com" and "www.mail2.metromotorgroup.com"

I'm pretty sure the altname is the same as well.

thx though

bobm

[bobm]

I just noticed the the there was no message for saving the config keys as listed in : Administration Console and CLI Certificate Tools - Zimbra :: Wiki

**Appending ca chain /tmp/ca_chain.crt to
/opt/zimbra/ssl/zimbra/commercial/commercial.crt
**Saving server config key zimbraSSLCeretificate…done.
**Saving server config key zimbraSSLPrivateKey…done.
**Installing mta certificate and key…done.
my output was :
** Appending ca chain /opt/zimbra/ssl/zimbra/commercial_ca.crt to /opt/zimbra/ssl/zimbra/commercial/commercial.crt
** Installing mta certificate and key...done.
** Installing slapd certificate and key...done.
** Installing proxy certificate and key...done.

Not sure if this helps any...

thx again
bobm

[bobm]

OK - I think this may be a key...
I did not get this msg:
**Saving server config key zimbraSSLCeretificate…done.
**Saving server config key zimbraSSLPrivateKey…done.
I now do -
[zimbra@mail2 ~]$ zmprov
ERROR: zclient.IO_ERROR (invoke Remote host closed connection during handshake, server: localhost) (cause: javax.net.ssl.SSLHandshakeException Remote host closed connection during handshake)
I think that the 2 keys listed above got clobbered -

zmlocalconfig|grep SSL - has no return value

Does anybody know how I can confirm the these keys got whacked???
AND FIX IT???

[bobm]

My system is UP and running, turns out a few things were hosed so for anybody who goes through this - I'd like to share the following links and commands to debug/solve your problem.

A) the following links are very good sources of info:

5.x Commercial Certificates Guide - Zimbra :: Wiki
Installing a GoDaddy Commercial Certificate - Zimbra :: Wiki
Ajcody-Notes-SSLCerts - Zimbra :: Wiki
Administration Console and CLI Certificate Tools - Zimbra :: Wiki
Problem with Certificate can cause MTA Failure - Zimbra :: Wiki

Samhain Labs | Defending against brute force ssh attacks

B) the following commands and places to check
make sure `zmhostname` = `hostname`

Assuming your limiting outside ssh access to keep the mongal hordes from contantly trying to break in:
make sure the the following is in /etc/host.allow
localhost.localdomain
localhost
`zmhostname`


sh -x ./bin/zmsshkeygen dsa [regenerates local public/private keys]

ls -l .ssh/* [check your timestamps]

vi .ssh/auth*s [edit the auth* file and replace the OLD pub key with the new one you created above]

ssh -vi .ssh/zimbra_identity -o strictHostKeyChecking=no zimbra@mail.mydomain.com -p 22 [this will tell you a lot about how zimbra is talking to the system] use this with /var/log/secure.log