LDAP multi-master confusion - simple test case
I'd like some more information about multi master setup. I have set aside three test machines as ldap server. Is it reasonable to have 1 master, the second as mmr and the third as a replica? What would best practice be? The documentation suggests no more than 4 multi-masters which implies that a massively scaled solution would then need some replicas also.
I was hoping to get resilience from having a multi master setup but it seems to be behaving strangely. For example I have some other test VMs and run something like this on them:
while [ true ] do ; zmprov cd testvm1.example.com ; sleep 1 ; zmprov dd testvm1.example.com ; sleep 1; done
On another test vm I continually create and delete a different domain.
On another VM I run zmprov gad continuously to watch the domains come and go.
On all the test VMs the ldap_url is set to replica, master 2, master 1 and ldap_master_url is set to master 2, master 1
By this simple test I hope to simulate read and write operations on the ldap servers.
I am not 100% confident of my configuration, the zmreplchk command says the masters are in sync but the replica is '0w 0d 0h 0m 0s ahead' (code 6)
Now the strangeness - if all servers are running, the test vms behave as expected.
If one of the masters is stopped (zmcontrol stop), one of the test VMs starts giving errors to the cd and dd commands - account.NO_SUCH_DOMAIN and account.DOMAIN_EXISTS - this continues for a long time. This looks like a bug, I might expect some odd behaviour when the set of ldap servers changes but this is a steady state where we can't create a domain because it apparently exists yet we can't delete it because it apparently doesn't exist...
With one master stopped, if the replica is stopped then this behaviour from the test VMs suddenly stops and they start creating and deleting domains happily again.
While the replica is stopped, either master can be stopped and started without problems, so long as at least one is working.
So, I am wondering if there is something wrong in my config or maybe my understanding of how multi master ldap is supposed to be used to achieve high availability. Any ideas?