[SOLVED] Ubuntu 8.04 + Zimbra 5.0.9 + nss_ldap = segfault

[conny]

Hi,

On my test setup I'm using Zimbra 5.0.9 on Ubuntu 8.04. The setup did run several days without problems.

Today I installed Samba and configured nss/pam etc. to use the Zimbra-LDAP server together with Samba. The basic stuff is working, which means I can see Zimbra-Users and -Groups via getent passwd/group. Also I can login using such an account. ATM Samba and Zimbra are running on the same machine.

The problem is that many postfix processes (trivial-rewrite, etc) are crashing. This behavior stops as soon as I change /etc/nsswitch.conf back to using local users/groups only. I don't even have to restart Zimbra. If I change the users/groups resolution back to LDAP I get the same segfaults again.

# grep -i segfault /var/log/syslog
Aug 30 17:32:54 server1 kernel: [330652.561409] trivial-rewrite[8833]: segfault at 00ffffff eip b7977a10 esp bf8cf780 error 4
Aug 30 17:33:15 server1 kernel: [330673.788479] proxymap[8845]: segfault at 00ffffff eip b7927a10 esp bfb50a10 error 4
Aug 30 17:34:22 server1 kernel: [330740.592085] proxymap[9441]: segfault at 00ffffff eip b7929a10 esp bfeca590 error 4
Aug 30 17:34:35 server1 kernel: [330754.079062] smtpd[7977]: segfault at 00ffffff eip b7746a10 esp bff56a60 error 4
Aug 30 17:34:35 server1 kernel: [330754.101527] smtp[7869]: segfault at 00ffffff eip b78eba10 esp bf98df90 error 4
Aug 30 17:34:36 server1 kernel: [330754.318906] lmtp[7978]: segfault at 00ffffff eip b7948a10 esp bfba5a40 error 4
Aug 30 17:34:43 server1 kernel: [330761.775262] trivial-rewrite[9449]: segfault at 00ffffff eip b7975a10 esp bfa65730 error 4

I'm absolutely clueless why this happens. Maybe it's problem with shared libraries? Zimbra is using libldap-2.3 but Ubuntu uses libldap-2.4? But then again postfix is using the right (zimbra) libs, so I don't see a problem there.

Any help would be highly appreciated!

Cheers!
Conny

[conny]

Does anyone has an idea how to debug this? Is it possible to get a stacktrace of those processes?

I tested some more and it looks like it doesn't has anything to do with my pam configuration. The only factor seems to be nss. I also checked that getent passwd/group/services/etc. returns propper data when using ldap as backend. Just to make sure that all important system accounts are there, but I can't find any problems.

Does anyone tried this setup on Ubuntu 8.04?

I'm thankful for any tips and pointers!
Conny

[Cafuego]

I upgraded from zcs 5.0.8 on Ubuntu 6.06 to zcs 5.0.9 on Ubuntu 8.04 yesterday, and after the upgrade postfix subprocesses started segfaulting.

Note that this does not appear to be affecting mail delivery, they all die after processing mails.

I did a trace on one of them, proxymap, which seems to be dying most often. (Conny, see Postfix Debugging Howto) The trace is attached and contains two segfaults. Offhand I can't see anything out of the ordinary, apart from the process dying after a time() call.

[Cafuego]

Yup, I have the same problem here. This started happening after going from zcs 5.0.8 on Ubuntu 6.06 to zcs 5.0.9 on ubuntu 8.04. I used nss_ldap on both.

The postfix subprocesses seem to be segfaulting after they process mails and mail delivery does not appear to be affected.

I did an strace on proxymap and the result is attached. I can't see anyting obviously out of the ordinary (apart from the segfaults ;-)

conny: Postfix Debugging Howto - after you set debug_command, the -D flag will work.

[lithorus]

I have a working Samba intergration with Zimbra's LDAP on a Ubuntu 8.04 and Zimbra 5.0.9. Can you post your common-auth, common-password, common-session and common-account?
Here are mine :
common-account :

account sufficient      pam_unix.so
account sufficient      pam_ldap.so

common-auth :

auth    sufficient      pam_ldap.so
auth    sufficient      pam_unix.so
auth    optional        pam_smbpass.so migrate missingok

common-password :

password        sufficient      pam_unix.so
password        sufficient      pam_ldap.so
password   optional   pam_smbpass.so nullok use_authtok use_first_pass missingok

common-session :

session required        pam_mkhomedir.so skel=/etc/skel umask=0077
session sufficient      pam_unix.so
session sufficient      pam_ldap.so
session required        pam_mkhomedir.so skel=/etc/skel umask=0077
session required        pam_limits.so

Also how does your nsswitch.conf look like?

[conny]

Thanks Cafuego and Lithorus for the replies!

I'll try with your configuration again in the evening. Also I'll create a stacktrace and post my configs. Are you having Samba and Zimbra on the same machine or are you using separate servers?

[lithorus]

Thanks Cafuego and Lithorus for the replies!

I'll try with your configuration again in the evening. Also I'll create a stacktrace and post my configs. Are you having Samba and Zimbra on the same machine or are you using separate servers?

On the same machine using this guide : Zimbra Integration With Samba - Ubuntu Based
It needed a few changes since the guide is based on Ubuntu 6.06

[Cafuego]

@lithorus: I don't actually use Samba with Zimbra. I just have nss-ldap set to authenticate system users against a different ldap server altogether, not the Zimbra ldap store. Authentication of system and zimbra users works fine, it's just Zimbra's postfix that segfaults.

common-account

account	    sufficient	pam_unix.so 
account	    sufficient	pam_ldap.so use_first_pass

common-auth

auth	sufficient	pam_unix.so nullok_secure
auth	sufficient	pam_ldap.so use_first_pass

common-password

password	sufficient	pam_unix.so nullok obscure min=4 max=8 md5
password	sufficient	pam_ldap.so use_authtok
password	required	pam_deny.so

common-session

session	    required	pam_unix.so
session	    optional	pam_ldap.so use_first_pass
session	    required	pam_limits.so

nsswitch.conf

passwd:         compat ldap
group:          compat ldap
shadow:         compat

hosts:          files dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis

[lithorus]

I would suggest try chaning 'compat' to 'files' in nsswitch.conf. Also I believe you can intergrate it directly with samba and not have to do it for the whole system.

Did you btw. import the samba ldap scehemes?

Perhaps take a look at Samba & LDAP - SambaWiki

[conny]

I just spend an hour trying out different versions of my /etc/pam.d/* files. I also tried it with the ones Lithorus posted. Unfortunately I still get the segfaults.

As Cafuego suggested, I did a stacktrace (using strace) of trivial-rewrite, which is one of the processes that's segfaulting. It was the first time I used strace, so I'm absolutely no exert. Still I think the following lines are relevant.

Sep  2 21:46:54 server1 logger: open("/opt/zimbra/lib/libldap_r-2.4.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
Sep  2 21:46:54 server1 logger: open("/opt/zimbra/mysql-standard-5.0.51a-pc-linux-gnu-i686-glibc23/lib/libldap_r-2.4.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
Sep  2 21:46:54 server1 logger: open("/opt/zimbra/mysql-standard-5.0.51a-pc-linux-gnu-i686-glibc23/lib/mysql/libldap_r-2.4.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
Sep  2 21:46:54 server1 logger: open("/opt/zimbra/openldap-clibs-2.3.42.8z/lib/libldap_r-2.4.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
Sep  2 21:46:54 server1 logger: open("/opt/zimbra/openssl-0.9.8g/lib/libldap_r-2.4.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
Sep  2 21:46:54 server1 logger: open("/opt/zimbra/cyrus-sasl-2.1.22.3z/lib/libldap_r-2.4.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
Sep  2 21:46:54 server1 logger: open("/opt/zimbra/sleepycat-4.2.52.6/lib/libldap_r-2.4.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)

The code above means to me that this process is trying to load the library "libldap_r-2.4.so.2". Therefor it is looking in various Zimbra directories. However it can't be found there.
The trace continues like that:

Sep  2 21:46:54 server1 logger: access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
Sep  2 21:46:54 server1 logger: open("/usr/lib/libldap_r-2.4.so.2", O_RDONLY) = 10

So now, after not finding the library inside the Zimbra directories, the process uses the values from "/etc/ld.so.cache" and figures out that the library is available in the system directory "/usr/lib". Then this library is loaded.

The problem is that my system libraries have different version than the Zimbra libraries. For example, Zimbra contains libldap-2.3.so.0 whereas Ubuntu contains libldap_r-2.4.so.2.

If I check Zimbras postfix I see that it is linked against libldap-2.3.so.0 which is located in "/opt/zimbra/lib". Therefore this version of libldap should be used and not version 2.4.

root@server1:~# ldconfig -v | grep libldap
	libldap_r-2.4.so.2 -> libldap_r-2.4.so.2.0.5

root@server1:~# ldd /opt/zimbra/postfix/libexec/trivial-rewrite | grep libldap
	libldap-2.3.so.0 => /opt/zimbra/lib/libldap-2.3.so.0 (0xb7dd6000)

As I read the rest of the stacktrace I can see that the same behavior for many other libs. The process is looking for the following libs inside the Zimbra directories. But as it cannot find them they are loaded from the system directories.

• libldap_r-2.4.so.2
• liblber-2.4.so.2
• libkrb5.so.3
• libcom_err.so.2
• libgssapi_krb5.so.2
• libgnutls.so.13
• libk5crypto.so.3
• libkrb5support.so.0
• libkeyutils.so.1
• libtasn1.so.3
• libgcrypt.so.11
• libgpg-error.so.0

So my conclusion would be, that for some reason postfix gets confused about which version of the libraries should be loaded. It tries to load the newer libraries, which it does not provide. Instead it should stick to it's own (older) libraries, which are provided by Zimbra.

Maybe I'm on the complete wrong track here, as I said, I'm not experienced with this kind of problems. So if you think that's BS, please let me know. On the other side there must be a reason for a segfault and simply a wrong config file shouldn't be the problem.

Maybe the problem is, that I just recently moved the installation from Debian to Ubuntu (this was before my Samba experiments started) and maybe the loading of dynamically linked libraries are somehow different between those distributions.... Maybe some LD_PRELOAD magic could help?!

However, any help is still greatly appreciated.
Cheers!
Conny