LDAP / Replication on 5.x -> 6.x Upgrade

[gareth]

Help!

After upgrading a working master/slave setup, the slave is now failing to connect to the server with the following messages;

slap_client_connect: URI=ldap://zm.xxxxxxx.com:3890 Error, ldap_start_tls failed (-11) 
do_syncrepl: rid=100 rc -11 retrying

Can anyone tell me what's causing this or maybe point me at a method for debugging /finding out what the problem is.

Things I'm confident of;
a. machine names / certificates match
b. port 3890 is open and working
c. ldap traffic is negotiating the link

??

[quanah]

It sounds like the replica can't validate the cert beign provided by the master. As the zimbra user on the replica system, can you ldapsearch the master when using startTLS?

[gareth]

Erm, if you can give me a specific command to run, I can confirm yes / no .. subject to that, if I type;

ldapsearch -h -p -x

On the master, it does give me a complete listing.

On the other hand, the same on the client (using the client's local address/port) gives me an empty listing, even after I populate the client LDAP database by hand ..

Note however; on the client with LDAP populated bu hand, posix searches (libnss_ldap.conf driven) all work fine .. ls -la /home happily lists user names against each home folder ...

[quanah]

On the replica, as the zimbra user, run

zmlocalconfig -s zimbra_ldap_password

note the value

Then run:

ldapsearch -ZZ -x -H ldap://zm....master...host..:3890/ -D "uid=zimbra,cn=admins,cn=zimbra" -b "" -s base -W +

It will then prompt you for the password you got from zmlocalconfig above. If the operation succeeds, then the replica can make a startTLS connection to the master. If it doesn't, it can't, and something is wrong in the cert setup between the master and replica.

[gareth]

But I'm not sure if the results are what one might expect (?) and what it means that it works ... ???

This is the output it gives;

# extended LDIF
#
# LDAPv3
# base <> with scope baseObject
# filter: (objectclass=*)
# requesting: +
#

#
dn:
structuralObjectClass: OpenLDAProotDSE
configContext: cn=config
monitorContext: cn=Monitor
namingContexts: cn=accesslog
namingContexts:
supportedControl: 1.3.6.1.4.1.4203.1.9.1.1
supportedControl: 2.16.840.1.113730.3.4.18
supportedControl: 2.16.840.1.113730.3.4.2
supportedControl: 1.3.6.1.4.1.4203.1.10.1
supportedControl: 1.2.840.113556.1.4.319
supportedControl: 1.2.826.0.1.3344810.2.3
supportedControl: 1.3.6.1.1.13.2
supportedControl: 1.3.6.1.1.13.1
supportedControl: 1.3.6.1.1.12
supportedExtension: 1.3.6.1.4.1.1466.20037
supportedExtension: 1.3.6.1.4.1.4203.1.11.1
supportedExtension: 1.3.6.1.4.1.4203.1.11.3
supportedExtension: 1.3.6.1.1.8
supportedFeatures: 1.3.6.1.1.14
supportedFeatures: 1.3.6.1.4.1.4203.1.5.1
supportedFeatures: 1.3.6.1.4.1.4203.1.5.2
supportedFeatures: 1.3.6.1.4.1.4203.1.5.3
supportedFeatures: 1.3.6.1.4.1.4203.1.5.4
supportedFeatures: 1.3.6.1.4.1.4203.1.5.5
supportedLDAPVersion: 3
supportedSASLMechanisms: GSSAPI
supportedSASLMechanisms: PLAIN
supportedSASLMechanisms: CRAM-MD5
supportedSASLMechanisms: LOGIN
supportedSASLMechanisms: DIGEST-MD5
supportedSASLMechanisms: OTP
entryDN:
subschemaSubentry: cn=Subschema
auditContext: cn=accesslog
contextCSN: 20091110191651.932455Z#000000#000#000000

# search result
search: 3
result: 0 Success

# numResponses: 2
# numEntries: 1

[quanah]

Hm, since you were using -ZZ on the replica, it'd mean it was able to do a startTLS to the master. Let me see if I can track down what specifically error -11 indicates.

[gareth]

Many thanks .. at the moment I have to resort to copying data between servers and re-populating the slave from LDIF every time we update the master.

(had to run the update, we were suffering from a bug whereby calendar entries [potentially 'corrupt'] were screwing the entire system on an hourly basis. 6.x seems to fix this ..)

Incidentally the POSIX tools were broken on our 5.x and we had to use the SMLPDAP tools to manipulate groups etc .. again this seems fixed on 6.x.
(once we managed to get the zimlet installed )

[lweeks]

Were/are you using self-signed SSL certificates? We ran into this same issue when upgrading some test VMs using self-signed certs. The issue wound up being that the LDAP servers must share a common certificate authority for TLS to work. Copying the same CA certs/keys to the hosts and regenerating certs solved the issue.

I've filed a bug asking that this information be added to the release notes, as this surely was not the same behavior in ZCS 5:

https://bugzilla.zimbra.com/show_bug.cgi?id=45048

Larry

[gareth]

Hi, I took this up with quanah offline and between us I think we identified 3 critical bugs, typically associated with upgrading, specifically in the context of LDAP/master/slave setups. I was given some patches and eventually got it working, took many days of time.

Apparently the fixes would not be in the next release, but would be in the next but one release .. not really been tracking it since then. I've no intention of upgrading that setup again - even now all the logging is broken to the extent the online console says everything is down -even tho' it's working fine.

I reckon those 'bugs' probably cost me two man weeks of work to circumvent / fix, and the upgrade was required due to another critical bug in the calendaring system that wasn't back-patched.

Bit of a mess really ...

[lweeks]

Hi,

It didn't make sense to me that they had to share a CA -- what they really need is knowledge of the CA of the master. slapd looks in /opt/zimbra/conf/ca for that. To make replication work, each replica needs to have the CA for the certificate used by the master in that directory, with a hash linked to it. That resolves the issue.

Larry