[SOLVED] 5.0.2 32-bit -> 6.0.6 64-bit - zmmailboxdctl is not running - I've tried

[ScottRose]

EDIT: I've gotten further since this was posted. You can probably just skip down to post #2 for the current situation. Thanks!

Hi All,

I've been banging my head against this one for about 4 hours now, and figured I'd ask for help.

My goal is to migrate a 5.0.2 32-bit ZCS Open Source Edition instance to 6.0.6 64-bit. I have been largely successful to this point (I believe), and the actual upgrade process went smoothly.

I've read countless threads on this issue, and can't seem to find the one that solves the problem for me. So many threads both on zimbra.com and other places that I can't cite them all here.

The problem is that the ZCS services start without error, but:

zimbra@zimbra01 [~]# zmcontrol status
Host zimbra01.mail02.codecrunchers.com
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Stopped
zmmailboxdctl is not running.
mta Running
snmp Running
spell Running
stats Running

I was getting this error in logs/zmmailboxd.out when it was trying to bind the port for IMAP SSL:

java.net.SocketException: Unbound server sockets not implemented

I read that's usually associated with a commercial SSL cert, however I am not attempting to use one. I'm good with just self-signed for this machine.

I tried following the directions here ..

Recreating a Self-Signed SSL Certificate in ZCS 4.5 & 5.0 - Zimbra :: Wiki

.. using the 5.0+ commands.

I would always get the error Keystore was tampered with, or password was incorrect, however now I'm getting additional exceptions after messing with the certs:

zimbra@zimbra01 [~/log]# zmlocalconfig -s -m nokey mailboxd_keystore_password
T1JWHAMGG6
zimbra@zimbra01 [~/log]# keytool -delete -alias jetty -keystore /opt/zimbra/mailboxd/etc/keystore -storepass T1JWHAMGG6
CompilerOracle: exclude com/zimbra/cs/session/SessionMap.putAndPrune
CompilerOracle: exclude com/zimbra/cs/mailbox/MailItem.delete
CompilerOracle: exclude org/apache/xerces/impl/XMLDocumentFragmentScannerImpl$FragmentContentDisp atcher.dispatch
keytool error: java.io.IOException: Keystore was tampered with, or password was incorrect

I believe my solution overall is to just re-deploy a new self-signed cert, but if the keystore is fubar'd then I feel a bit stuck.

I can't find much on this subject specific to 6.0.*, so I've been following 5.0.* directions for the most part.

Can someone give me a nudge in the right direction, please?

[ScottRose]

While my post was awaiting moderation, I decided to delete and re-create the keystore. That got me farther, but zmmailboxdctl would still not start. The problem seemed to be the empty node in jetty.xml for direct buffers. I set that node's contents to true in the 2 locations which it appeared, and the mailbox service was able to start. Hooray!

However, now mailbox.log reveals these exceptions:

2010-05-05 23:27:42,794 INFO  [main] [] im - Setting default XMPP domain to: zimbra01.mail02.codecrunchers.com
2010-05-05 23:27:43,669 WARN  [main] [] im - Caught service exception getting local component list
com.zimbra.common.service.ServiceException: system failure: unable to list all servers
ExceptionId:main:1273116463660:0271085d7cbedf1c
Code:service.FAILURE
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:248)
        at com.zimbra.cs.account.ldap.LdapProvisioning.getAllXMPPComponents(LdapProvisioning.java:5724)
(Full stack trace omitted)
Caused by: javax.naming.NameNotFoundException: [LDAP: error code 32 - No Such Object]; remaining name 'cn=xmppcomponents,cn=zimbra'
....
2010-05-05 23:27:43,931 INFO  [main] [] im - Started server (unencrypted) socket on port: 5269
2010-05-05 23:27:43,937 INFO  [main] [] im - Started component (unencrypted) socket on port: 10015
2010-05-05 23:27:43,938 INFO  [main] [] im - Started plain (unencrypted) socket on port: 5222
2010-05-05 23:27:43,939 INFO  [main] [] im - Started SSL (encrypted) socket on port: 5223
2010-05-05 23:27:44,171 WARN  [main] [] im - Caught ServiceExcepton trying to setup remote Disco handlers
com.zimbra.common.service.ServiceException: system failure: unable to list all servers
ExceptionId:main:1273116464170:0271085d7cbedf1c
Code:service.FAILURE
        at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:248)
        at com.zimbra.cs.account.ldap.LdapProvisioning.getAllXMPPComponents(LdapProvisioning.java:5724)
(Full stack trace omitted)
Caused by: javax.naming.NameNotFoundException: [LDAP: error code 32 - No Such Object]; remaining name 'cn=xmppcomponents,cn=zimbra'
....

(I can publish the full stack trace if need be).

The above are warnings, not erorrs, so I'm thinking the services should all start anyhow. And I can't find anything else negative in any of my logs, but there is no listener on ports 4443 or 7071 (both configured in jetty.xml).

I haven't been able to find this issue reproduced in another forum/web page. The best I could find is that the LDAP password might not be correct, so I listed it using zmlocalconfig -s ldap_root_password zimbra_ldap_password and then set it to that PW using zmldappasswd -r <CORRECT_PASSWORD> and zmldappasswd <CORRECT_PASSWORD>. (All Zimbra passwords are identical when enumerated from config, and I copied/pasted the password to zmldappasswd, double-checking it).

Of course I restarted Zimbra completely between each change.

I'm guessing that I'm missing part of an LDAP entry for xmppcomponents, though there is an entry in zimbra.ldif (complete file follows).

dn: cn=zimbra
objectclass: organizationalRole
description: Zimbra Systems Application Data
cn: zimbra

dn: cn=admins,cn=zimbra
objectclass: organizationalRole
description: admin accounts
cn: admins

dn: uid=zimbra,cn=admins,cn=zimbra
uid: zimbra
objectclass: zimbraAccount
objectclass: organizationalPerson
cn: zimbra
sn: zimbra
zimbraAccountStatus: active
zimbraIsAdminAccount: TRUE
zimbraIsSystemResource: TRUE
zimbraId: e0fafd89-1360-11d9-8661-000a95d98ef2
description: The master zimbra admin account
userPassword: {SSHA}<REDACTED>

dn: uid=zmreplica,cn=admins,cn=zimbra
uid: zmreplica
objectclass: zimbraAccount
objectclass: organizationalPerson
cn: zmreplica
sn: zmreplica
zimbraAccountStatus: active
zimbraIsSystemResource: TRUE
zimbraId: D020C0FE-4F5E-11DC-B033-C3A8E67A905E
description: The zimbra replication account
userPassword: {SSHA}<REDACTED>

dn: cn=appaccts,cn=zimbra
objectclass: organizationalRole
description: application accounts
cn: appaccts

dn: uid=zmnginx,cn=appaccts,cn=zimbra
uid: zmnginx
objectclass: zimbraAccount
objectclass: organizationalPerson
cn: zmnginx
sn: zmnginx
zimbraAccountStatus: active
zimbraIsAdminAccount: TRUE
zimbraIsSystemResource: TRUE
zimbraId: DA336C18-4F5E-11DC-8514-DCA8E67A905E
description: The zimbra nginx account
userPassword: {SSHA}<REDACTED>

dn: uid=zmpostfix,cn=appaccts,cn=zimbra
uid: zmpostfix
objectclass: zimbraAccount
objectclass: organizationalPerson
cn: zmpostfix
sn: zmpostfix
zimbraAccountStatus: active
zimbraIsSystemResource: TRUE
zimbraId: DA336C18-4F5E-11DC-8514-DCA8E67A905E
description: The zimbra postfix account
userPassword: {SSHA}<REDACTED>

dn: uid=zmamavis,cn=appaccts,cn=zimbra
uid: zmamavis
objectclass: zimbraAccount
objectclass: organizationalPerson
cn: zmamavis
sn: zmamavis
zimbraAccountStatus: active
zimbraIsSystemResource: TRUE
zimbraId: 12A0779A-88B6-11DC-AA4B-97F9DEB0B34E
description: The zimbra amavis account
userPassword: {SSHA}<REDACTED>

dn: cn=zimlets,cn=zimbra
objectclass: organizationalRole
cn: zimlets
description: for storing Zimlet configuration

dn: cn=cos,cn=zimbra
objectclass: organizationalRole
cn: cos

dn: cn=servers,cn=zimbra
objectclass: organizationalRole
cn: servers

dn: cn=xmppcomponents,cn=zimbra
objectclass: organizationalRole
cn: xmppcomponents

dn: cn=globalgrant,cn=zimbra
objectclass: zimbraAclTarget
description: acl target for global grants
cn: globalgrant

I'm not even sure if that's relevant here -- my experience with LDAP in general is just about nil.

With services started, netstat shows:

zimbra@zimbra01 [~]# netstat -nlp |more
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
tcp        0      0 10.1.1.41:389               0.0.0.0:*                   LISTEN      -
tcp        0      0 127.0.0.1:10663             0.0.0.0:*                   LISTEN      31000/zmlogger: zmr
tcp        0      0 127.0.0.1:10024             0.0.0.0:*                   LISTEN      1389/amavisd (maste
tcp        0      0 127.0.0.1:10025             0.0.0.0:*                   LISTEN      -
tcp        0      0 127.0.0.1:7306              0.0.0.0:*                   LISTEN      31096/mysqld
tcp        0      0 0.0.0.0:587                 0.0.0.0:*                   LISTEN      -
tcp        0      0 0.0.0.0:971                 0.0.0.0:*                   LISTEN      -
tcp        0      0 0.0.0.0:3310                0.0.0.0:*                   LISTEN      1527/clamd
tcp        0      0 0.0.0.0:111                 0.0.0.0:*                   LISTEN      -
tcp        0      0 0.0.0.0:465                 0.0.0.0:*                   LISTEN      -
tcp        0      0 10.1.1.41:53                0.0.0.0:*                   LISTEN      -
tcp        0      0 127.0.0.1:53                0.0.0.0:*                   LISTEN      -
tcp        0      0 0.0.0.0:25                  0.0.0.0:*                   LISTEN      -
tcp        0      0 127.0.0.1:953               0.0.0.0:*                   LISTEN      -
tcp        0      0 :::7072                     :::*                        LISTEN      -
tcp        0      0 :::7777                     :::*                        LISTEN      -
tcp        0      0 :::7780                     :::*                        LISTEN      1550/httpd
tcp        0      0 :::5222                     :::*                        LISTEN      -
tcp        0      0 :::5223                     :::*                        LISTEN      -
tcp        0      0 :::7335                     :::*                        LISTEN      -
tcp        0      0 :::110                      :::*                        LISTEN      -
tcp        0      0 :::143                      :::*                        LISTEN      -
tcp        0      0 :::7025                     :::*                        LISTEN      -
tcp        0      0 :::5269                     :::*                        LISTEN      -
tcp        0      0 :::53                       :::*                        LISTEN      -
tcp        0      0 :::22                       :::*                        LISTEN      -
tcp        0      0 ::1:953                     :::*                        LISTEN      -
tcp        0      0 :::10015                    :::*                        LISTEN      -
(UDP and UNIX socks omitted)

Any help would be most appreciated. Thanks for reading!

[soxfan]

What process did you actually use for the migration? From 5.0.2 32-bit to 6.0.6 64-bit you would need to do a couple of interim upgrades I believe. Something like...
5.0.2 32-bit --> 5.0.23 32-bit --> 6.0.6 32-bit --> 6.0.6 64-bit.

I just went through a 32-bit to 64-bit migration on 6.0.6. I essentially followed the wiki document Network Edition: Moving from 32-bit to 64-bit Server - Zimbra :: Wiki. For the most part it was a pretty smooth process, except the document fails to address self-signed certificates. I ran into the same problem as you are seeing with 'zmmailboxdctl' not running.

[ScottRose]

Sorry to keep pushing this forward by myself. For some reason a reboot did it (even though I hadn't changed anything outside /opt/zimbra since the previous boot -- or maybe I just don't remember). I'm now able to access the admin tool and the mail web client. All existing email is accessible.

However, upon trying to send a message, I get this error:

method:	SendMsgRequest
msg:	system failure: Unable to send message
code:	service.FAILURE
detail:	soap:Receiver
trace:	btpool0-8:1273162686342:30a246bb9f5e8aa1
request:	

Body: {
(Full message omitted)

Likewise, in the admin tool if I click on certain options I get SOAP errors.

E.g., selecting the server from "Configuration -> Servers" on the left nav bar, or selecting "Monitoring -> Server Statistics" results in this:

Server error encountered Message: system failure: exception during auth {RemoteManager: zimbra01.mail02.codecrunchers.com->zimbra@zimbra01.mail02.codecrunchers.com:22} Error code: service.FAILURE Details:soap:Receiver

Is this still an LDAP issue??

[ScottRose]

<offtopic>Mods: I can't help but notice that my first post in this thread was deleted.

I appreciate your desire to run a tight ship, but if I violated the ToS, duplicated a previous post, or otherwise did something wrong I'd like to know about it and not just have my posts disappear.

Also, I know I've been replying to myself here, but the notion is such that someone else could follow what I'm doing, either to resolution of their own problem or to the point of finding where in my efforts I've made a mistake.

Please feel free to delete this particular post -- I wasn't sure where and to whom I should address my concerns.