[SOLVED] zmmailboxdctl urgent & ugly problem (aren't they all? ;)

[ekkas]

I have a rather big (600 user) problem with my server after trying to migrate from Ubuntu 8.04 32bit to CentOS5.5-64 bit.
Followed Network Edition: Moving from 32-bit to 64-bit Server - Zimbra :: Wiki to a t (except for the part about commercial certificates).
Although I must add that this problem started before the migrate, partly the reason for it.
The server is running on XenServer 5.6 with 2 CPUs, 2048MB RAN (768MB assigned during install)
The only 'add-on' in CentOS 5.5-64 is XenTools installed.
IP, hostname, everything exactly the same, running on same (Xen) server.


Now I have:
zmcontrol status
Host mail.true.co.za
antispam Running
antivirus Running
imapproxy Running
ldap Running
logger Running
mailbox Stopped
zmmailboxdctl is not running.
memcached Running
mta Running
snmp Running
spell Running
stats Running

I cannot find anything obviously wrong in the log files apart from
/opt/zimbra/log/zmmailboxd.out
Zimbra server reserving server socket port=110 bindaddr=my_ip_here ssl=false
Zimbra server reserving server socket port=995 bindaddr=null ssl=true
Fatal error: exception while binding to ports
java.net.SocketException: Unbound server sockets not implemented

With Zimbra stopped :
[zimbra@mail root]$ netstat -tulpn
(No info could be read for "-p": geteuid()=500 but you should be root.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN -
tcp 0 0 196.36.9.147:53 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN -
tcp 0 0 127.0.0.1:953 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:926 0.0.0.0:* LISTEN -
tcp 0 0 :::22 :::* LISTEN -
tcp 0 0 ::1:953 :::* LISTEN -
udp 0 0 0.0.0.0:44818 0.0.0.0:* -
udp 0 0 0.0.0.0:920 0.0.0.0:* -
udp 0 0 0.0.0.0:923 0.0.0.0:* -
udp 0 0 196.36.9.147:53 0.0.0.0:* -
udp 0 0 127.0.0.1:53 0.0.0.0:* -
udp 0 0 0.0.0.0:5353 0.0.0.0:* -
udp 0 0 0.0.0.0:111 0.0.0.0:* -
udp 0 0 196.36.9.147:123 0.0.0.0:* -
udp 0 0 127.0.0.1:123 0.0.0.0:* -
udp 0 0 0.0.0.0:123 0.0.0.0:* -
udp 0 0 :::47829 :::* -
udp 0 0 :::5353 :::* -
udp 0 0 fe80::6838:48ff:fe4:123 :::* -
udp 0 0 ::1:123 :::* -
udp 0 0 :::123 :::* -

No conflicting ports?

(With Zimbra running or stopped)
Even if I run something like :
[zimbra@mail root]$ zmprov ms mail.true.co.za zimbraImapCleartextLoginEnabled TRUE zimbraPop3CleartextLoginEnabled TRUE
[] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
[] INFO: Retrying request
ERROR: zclient.IO_ERROR (invoke Connection refused, server: localhost) (cause: java.net.ConnectException Connection refused)

and /var/log/zimbra.log is full of:
Dec 4 00:06:46 mail postfix/error[9953]: 7C4795D01926: to=, relay=none, delay=337, delays=337/0.13/0/0.02, dsn=4.4.1, status=deferred (delivery temporarily suspended: connect to mail.true.co.za[196.36.9.147]:7025: Connection refused)


Also strange, don't know if it's relevant:

[zimbra@mail root]$ lsof | grep TCP
lsof: WARNING: can't stat() proc file system /var/named/chroot/proc
Output information may be incomplete.
lsof: WARNING: can't stat() ext3 file system /var/named/chroot/var/run/dbus
Output information may be incomplete.

[root@mail ~]# uname -a
Linux mail.true.co.za 2.6.18-194.26.1.el5xen #1 SMP Tue Nov 9 13:35:30 EST 2010 x86_64 x86_64 x86_64 GNU/Linux
[zimbra@mail root]$ zmcontrol -v
Release 6.0.9_GA_2686.RHEL5_64_20101115233514 CentOS5_64 FOSS edition.

$ hostname
mail.true.co.za
$ hostname -f
mail.true.co.za
$ nslookup mail.true.co.za
Name: mail.true.co.za
Address: 196.36.9.147
$ dig mx true.co.za
;; ANSWER SECTION:
true.co.za. 221 IN MX 10 mail.true.co.za.
;; ADDITIONAL SECTION:
mail.true.co.za. 2592000 IN A 196.36.9.147

$ cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
196.36.9.147 mail.true.co.za mail


[zimbra@mail root]$ zmprov -l gs mail.true.co.za | grep Bind
zimbraImapBindOnStartup: TRUE
zimbraImapBindPort: 143
zimbraImapProxyBindPort: 7143
zimbraImapSSLBindOnStartup: TRUE
zimbraImapSSLBindPort: 993
zimbraImapSSLProxyBindPort: 7993
zimbraLmtpBindOnStartup: FALSE
zimbraLmtpBindPort: 7025
zimbraMemcachedBindPort: 11211
zimbraNotifyBindPort: 7035
zimbraNotifySSLBindPort: 7036
zimbraPop3BindAddress: 196.36.9.147
zimbraPop3BindOnStartup: TRUE
zimbraPop3BindPort: 110
zimbraPop3ProxyBindPort: 7110
zimbraPop3SSLBindOnStartup: TRUE
zimbraPop3SSLBindPort: 995
zimbraPop3SSLProxyBindPort: 7995


Read every post I could find, nothing helps.
Any help would very much be appreciated!

Ekkas

[bdial]

are you sure you don't have antyhing preinstalled bound to those ports already? some pop3 daemon taht came with centos?

[ekkas]

Thanks for the reply.
Nope, not according to netstat -tulpn or lsof
I just enabled text-based internet & editors (if I remember correctly)
If I know what to look for, I might find the cause, but all I could find is in the post above...

[ekkas]

I started over, re-installed CentOS with absolute minimum (again) just using text-based-internet and editors packages on install.
Didn't add any packages that the ZCS install didn't ask for (no perl, lib..., gmp)
chkconfig sendmail off
chkconfig ip6tables off
chkconfig iptables off
chkconfig bluetooth off
chkconfig cups off
yum install sysstat
Reboot, install.
In the setup options of ZCS install, entered all passwords fom old server(except MySQL root which I couldn't find), finished install (starting servers).
Then stopped Zimbra.
Checked the xml files, started MySql and reset the MySQL passwords from CLI, using same as old server.
Did the rest, and it's working now. Phew! (wiping the stress and sweat from his forehead)

Ekkas