Live server picked up settings from Test server

[The Fold]

I had a test server setup using some data from our live server. Our live server suddenly stopped working and taking a look at the logs looked as if it was attempting to act as if it was the test server I'd setup.

I couldn't find any evidence of any configuration files having been changed but I have a hunch that LDAP somehow replicated and caused this issue.

[uxbod]

What sort of data did you migrate to the test server, and how did you do it ?

su - zimbra
zmlocalconfig | grep -i ldap

does that contain any of the production settings ? What do your Log Files - Zimbra :: Wiki show ?

[The Fold]

I restored a full backup of the live server and performed a zmsetservername and permission fix on the restored copy.

As I'm currently trying to fix this issue I can't run a zmlocalconfig | grep -i ldap as it will be referring to yesterdays backup.

I'm currently attempting a copy of the store, db, index, and postfix folders from the broken install over the new install.

My thinking the issue was ldap related stems from when I performed an export it had the settings for the test server in the ldif file.

[uxbod]

Did you follow ZmSetServerName - Zimbra :: Wiki as it has a few checks in there aswell.

[The Fold]

I followed that page to perform the rename process on the test server, all was working ok but then the live server stopped working and checking the logs showed mail trying to relay through the test server.

I found the following thread which seems very much like the problem I was experiencing.

Problem starting up Zimbra

I managed to get everything working again by restoring everything except the db, store, index and data folders to the working directory and then copying these folders over from the broken install.

It does seem to be working ok, I am however now getting messages in the logs from slapd saying:

May 21 09:53:18 zimbra slapd[12427]: bdb(): Lock table is out of available locker entries
May 21 09:53:18 zimbra slapd[12427]: bdb_db_cache: db_open(displayName) failed: Cannot allocate memory (12)
May 21 09:53:18 zimbra slapd[12427]: <= bdb_substring_candidates: (displayName) index_param failed (12)

[uxbod]

Take a look at Performance Tuning Guidelines for Large Deployments - Zimbra :: Wiki and the section "Configuring the BDB subsystem to increase LDAP server performance" for tuning the number of lockers.

[The Fold]

I took a look at that and I don't think I should be having an issue

# /opt/zimbra/sleepycat/bin/db_stat -c -h /opt/zimbra/openldap-data | head -n 12

db_stat: Ignoring log file: /opt/zimbra/openldap-data/logs/log.0000000008: magic number 0, not 40988
db_stat: DB_ENV->log_put: 8: Invalid argument
db_stat: dbenv->close: Invalid argument
11228   Last allocated locker ID.
2147M   Current maximum unused locker ID.
9       Number of lock modes.
3000    Maximum number of locks possible.
1500    Maximum number of lockers possible.
1500    Maximum number of lock objects possible.
25      Number of current locks.
163     Maximum number of locks at any one time.
1499    Number of current lockers.
1500    Maximum number of lockers at any one time.
20      Number of current lock objects.
93      Maximum number of lock objects at any one time.

EDIT: My bad, I didn't see the

1499    Number of current lockers.
1500    Maximum number of lockers at any one time.

[The Fold]

Ok, so at about 11:30 this morning ldap decided it would die altogether saying it couldn't recover and to restore from a backup. I had altered my max locker size in DB_CONFIG but I hadn't yet restarted.

I restored the openldap-data folder from a backup made a couple of days ago and everything is now back to normal, I'm not getting any messages from ldap at all.

I'm still curious however as to how this all happened in the first place.