MTA not starting, yet I'm unable to find much debug info...

[orospakr]

Hey guys:

I've installed 4.5.6 of the Open Source edition on Ubuntu 7.04 (i386), and everything seems to have installed just fine, except that the MTA just doesn't want to start:

zimbra@mahoromatic:~$ zmcontrol status

Host orospakr.ca

        antispam                Running

        antivirus               Running

        ldap                    Running

        logger                  Running

        mailbox                 Running

        mta                     Stopped

                postfix is not running

        snmp                    Running

        spell                   Running

Starting the MTA manually produces this output, which also doesn't tell me much, unless I'm missing something:

http://orospakr.ca/stuff/zmmtactl_start.txt

And the Zimbra and /var/log/mail.err logs fill up with:

postfix/postqueue[21135]: fatal: Queue report unavailable - mail system is down

Executing mailq (as Zimbra) manually prints the same thing. Strace says it is trying to open a socket at "/opt/zimbra/postfix-2.2.9/spool/public/showq":

chdir("/opt/zimbra/postfix-2.2.9/spool") = 0

rt_sigaction(SIGPIPE, {SIG_IGN}, {SIG_IGN}, 8) = 0

getuid32()                              = 1011

socket(PF_FILE, SOCK_STREAM, 0)         = 4

fcntl64(4, F_GETFL)                     = 0x2 (flags O_RDWR)

fcntl64(4, F_SETFL, O_RDWR)             = 0

connect(4, {sa_family=AF_FILE, path="public/showq"}, 110) = -1 EACCES (Permission denied)

It appears that the problem is twofold: if I manually chmod it 777, the error becomes -ENOENT, because the socket simply isn't there.

This lead me to believe that master wasn't starting to begin with, and running it and stracing it shows:

Jul 30 12:24:06 localhost postfix/master[25252]: fatal: the master command is reserved for the superuser

So I really have no idea how to go about fixing this.

Many thanks for any help!

[orospakr]

Aha!

I've done a little more debugging, and it turns out that the Postfix 'master' program was segfaulting. I enabled coredumps and managed to get a backtrace:

mahoromatic:/opt/zimbra/postfix-2.2.9/spool# gdb ../libexec/master core 

GNU gdb 6.6-debian

Copyright (C) 2006 Free Software Foundation, Inc.

GDB is free software, covered by the GNU General Public License, and you are

welcome to change it and/or distribute copies of it under certain conditions.

Type "show copying" to see the conditions.

There is absolutely no warranty for GDB.  Type "show warranty" for details.

This GDB was configured as "i486-linux-gnu"...

Using host libthread_db library "/lib/tls/i686/cmov/libthread_db.so.1".
 

warning: Can't read pathname for load map: Input/output error.

Reading symbols from /usr/lib/libpcre.so.3...done.

Loaded symbols for /usr/lib/libpcre.so.3

Reading symbols from /opt/zimbra/lib/libldap-2.3.so.0...done.

Loaded symbols for /opt/zimbra/lib/libldap-2.3.so.0

Reading symbols from /opt/zimbra/lib/liblber-2.3.so.0...done.

Loaded symbols for /opt/zimbra/lib/liblber-2.3.so.0

Reading symbols from /opt/zimbra/lib/libmysqlclient.so.15...done.

Loaded symbols for /opt/zimbra/lib/libmysqlclient.so.15

Reading symbols from /usr/lib/libz.so.1...done.

Loaded symbols for /usr/lib/libz.so.1

Reading symbols from /lib/tls/i686/cmov/libm.so.6...done.

Loaded symbols for /lib/tls/i686/cmov/libm.so.6

Reading symbols from /opt/zimbra/cyrus-sasl-2.1.21.ZIMBRA/lib/libsasl2.so.2...done.

Loaded symbols for /opt/zimbra/cyrus-sasl/lib/libsasl2.so.2

Reading symbols from /lib/tls/i686/cmov/libpthread.so.0...done.

Loaded symbols for /lib/tls/i686/cmov/libpthread.so.0

Reading symbols from /usr/lib/i686/cmov/libssl.so.0.9.8...done.

Loaded symbols for /usr/lib/i686/cmov/libssl.so.0.9.8

Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.8...done.

Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.8

Reading symbols from /lib/tls/i686/cmov/libnsl.so.1...done.

Loaded symbols for /lib/tls/i686/cmov/libnsl.so.1

Reading symbols from /lib/tls/i686/cmov/libresolv.so.2...done.

Loaded symbols for /lib/tls/i686/cmov/libresolv.so.2

Reading symbols from /lib/tls/i686/cmov/libc.so.6...done.

Loaded symbols for /lib/tls/i686/cmov/libc.so.6

Reading symbols from /lib/tls/i686/cmov/libcrypt.so.1...done.

Loaded symbols for /lib/tls/i686/cmov/libcrypt.so.1

Reading symbols from /lib/ld-linux.so.2...done.

Loaded symbols for /lib/ld-linux.so.2

Reading symbols from /lib/tls/i686/cmov/libdl.so.2...done.

Loaded symbols for /lib/tls/i686/cmov/libdl.so.2

Reading symbols from /usr/lib/libxml2.so.2...done.

Loaded symbols for /usr/lib/libxml2.so.2

Reading symbols from /lib/tls/i686/cmov/libnss_compat.so.2...done.

Loaded symbols for /lib/tls/i686/cmov/libnss_compat.so.2

Reading symbols from /lib/tls/i686/cmov/libnss_nis.so.2...done.

Loaded symbols for /lib/tls/i686/cmov/libnss_nis.so.2

Reading symbols from /lib/tls/i686/cmov/libnss_files.so.2...done.

Loaded symbols for /lib/tls/i686/cmov/libnss_files.so.2

Reading symbols from /usr/lib/libnss_db.so.2...done.

Loaded symbols for /usr/lib/libnss_db.so.2

Reading symbols from /usr/lib/libdb-4.3.so...done.

Loaded symbols for /usr/lib/libdb-4.3.so

Core was generated by `/opt/zimbra/postfix-2.2.9/libexec/master'.

Program terminated with signal 11, Segmentation fault.

#0  0x080e8903 in __db_close_pp ()

(gdb) bt

#0  0x080e8903 in __db_close_pp ()

#1  0xb789f6ed in ?? () from /usr/lib/libnss_db.so.2

#2  0x08145780 in ?? ()

#3  0x00000000 in ?? ()

One of my theories is that it might actually be an ABI incompatibility between the NSS on Ubuntu 6.06 and that on 7.04.

[orospakr]

Well, since the breakage appeared to be occurring inside Berkeley DB, I tried disabling the "db" entries in /etc/nsswitch.conf. That was the ticket.

I'm not sure if those entries in nsswitch.conf are standard on Ubuntu Feisty 7.04, because this machine has been dist-upgraded several times.

Anyway, all is working now!

[dijichi2]

It looks like for some reason your linker is reading libdb from the system rather than zimbra resulting in API mismatch - zimbra uses 4.2 where the system has 4.3. This is a classic reason for postfix segfaulting. db entries are standard in ubuntu. good detective work tracking the problem down!!

[orospakr]

Thanks!

So, what you're saying is, Zimbra actually includes its own Berkeley db and nss, but the linker is picking up the system copies?

[phingers]

I'm having the same issue with our dev zimbra server.

Linux devmail 2.6.16.29-xen #3 SMP Sun Oct 15 13:15:34 BST 2006 x86_64 x86_64 x86_64 GNU/Linux

It says postfix isnt running, I checked the nsswitch.conf file but do not see any db entries that are not commented out.

How do i get the linker to check the right file? Or what other ideas to troubleshoot?

Thanks.

[orospakr]

See above for the solution I used. If that doesn't help, you may need to to resort to other approaches, such as LD_PRELOAD hacks, perhaps. In that case, it seems likely your x86_64 userland is to blame.