ldap database went from 97meg to 86gig

[ugurcacaron]

You claim to be the leader in open source messaging and collaboration. Can you describe us how it will be with such a philosophy of programming? You decide to limit the size of a file, and tell the OS to reserve space for it making everything difficult for us. If your intention is to force us to buy the non-free versions, please let us know that.

Rsync is a good and working solution for us for backup and upgrade processes. How can you insist on the claim it causes corrupted LDAP databases? Until now it did worked well for my system for restoring and upgrading from 7.2.0 to 8.0.0 and then to 8.0.1.

How can we guess the max size of our data.db? I think the best and painless solution is to limit the file size without provisioning space. Can't you do that?

Best regards.
Ugur Çaçaron
Fethiye

[davestahr]

How can we guess the max size of our data.db?

This question has been answered in this thread -- read back a few pages. I understand your frustration - I was thinking the EXACT same thing about FOSS/Enterprise and this "feature" that seemed to intentionally break all of our backup scripts, but Quanah has posted some great solutions to the problem. Just look back in this forum thread, you'll find all of your answers.

[ccelis5215]

Hi all,

Maybe a silly question., we are in ZCS 7.1.4 in production, targeted for 7.2.2. there's upgrade to OpenLDAP 2.4.33

We have to worry about 80GB size in LDAP or just as Quanah pointed in 8.0.2 it's resizing on the fly?

NOTE: As a SMB user i did a cold rsync /opt/zimbra after stop zimbra services. 10 minutes.. no harm.

ccelis

[quanah]

7.2.2 doesn't use mdb, it uses hdb, so this discussion does not apply to your installation.

[ccelis5215]

7.2.2 doesn't use mdb, it uses hdb, so this discussion does not apply to your installation.

Thanks for clarification!

ccelis

[mowglie]

dave, so thats how u do it in 3 min. makes sense too. sure, i wouldnt mind your script. cant have too many scripts now can we? !
do you want my email?

[davestahr]

sure, i wouldnt mind your script

Nothing sacred - basically the same thing you've seen in this thread and in other posts - I've modified it slightly since my version is in OpenVZ - instead of the zmcontrol stop/start, I'm actually turning off the virtual container, and doing all the rsyncs from the host rather than from the zimbra server. I'll post both scripts here.

These scripts assume you have an rsync server set up at host "backup" and you're using the rsync username "backup" with a password file stored with "400" perms at /root/rsync.passwd on the local computer.

IMPORTANT -- this script is using the --delete option. If for some reason your Zimbra installation vanishes or your files become corrupted, your backup will vanish right along with it once this script runs.

Standard "bare metal" Zimbra backup:

#!/bin/bash

rsync --exclude=data/ldap/mdb/db --delete --password-file=/root/rsync.passwd -azHpr  /opt/zimbra backup@backup::zimbra/

mydir="/opt/zimbra/ldap-backup/`date +%s`"

mkdir -p $mydir

chown zimbra $mydir

su - zimbra -c /opt/zimbra/libexec/zmslapcat $mydir

su - zimbra -c /opt/zimbra/libexec/zmslapcat -c $mydir

gzip $mydir/*

su - zimbra -c /opt/zimbra/bin/zmcontrol stop

/bin/sync

rsync --exclude=data/ldap/mdb/db --delete --password-file=/root/rsync.passwd -azHpr  /opt/zimbra backup@backup::zimbra/

su - zimbra -c /opt/zimbra/bin/zmcontrol start

For the OpenVZ version - in my case my container ID is 203 that Zimbra runs in - so that's what the 203 below refers to - my rsync server host a relating backup host "203" - you'll need to change this to suit your needs.

#!/bin/bash

rsync --exclude=opt/zimbra/data/ldap/mdb/db --delete --password-file=/root/rsync.passwd -azHpr  /vz/private/203/* backup@backup::203/

#Back up LDAP - this is just a script that contains the slapcat commands in the previous example seen above
vzctl exec 203 /opt/scripts/run-ldap-backup-as-zimbra.sh

/usr/sbin/vzctl stop 203

rsync --exclude=opt/zimbra/data/ldap/mdb/db --delete --password-file=/root/rsync.passwd -aHpr  /vz/private/203/* backup@backup::203/

/usr/sbin/vzctl start 203

[LMStone]

@Quanah: Thanks for taking the time to clarify here the issues surrounding backing up LDAP. I'd be grateful for your input please on widening the scope of this thread to encompass Zimbra Disaster Recovery in a virtualized environment.

Essentially, we are looking to deploy a supported DR solution for NAT'd Zimbra NE which results in shorter recovery times than that provided by the "zmrestoreldap + zmrestoreoffline" method in the Admin Guide. Specifically, we are looking at a Sandy/Irene/Katrina use case where the primary data center is taken offline and is expected to remain offline for some time. We want to be able to cutover to the secondary data center as quickly as practicable. E.g. Server Live sync - Zimbra :: Wiki looks interesting but is of course not supported.

Challenges and Questions:

1. The NE Admin Guides for 7.2 and 8.0 make no mention of the "install.sh -s + rsync" DR method in their Disaster Recovery sections; this method is highlighted in various posts here in the forums, Zimbra blog posts and in several non-certified wiki articles. We have used that method ourselves previously and successfully to get the new replacement Zimbra server up and running quickly with minimal downtime on the old Zimbra server. Assuming we do an LDAP export/import procedure, is this method still viable with ZCS 7.2.x and 8.0.x for DR?
2. Tools exist for various hypervisors which leverage disk+RAM snapshots. If we have a disk+RAM snapshot of a running virtualised Zimbra server (e.g. available in XenServer Enterprise), is it safe to use that disk+RAM snapshot:
   
   1. alone for DR purposes?
   2. in conjunction with restoring LDAP after restoring the snapshot at the DR site?
   3. not at all?
   4. Do the above answers change if Zimbra is stopped before taking the snapshot at the hypervisor level?
   
   
3. Related to 2. above, are "crash consistent" snapshots (in conjunction with an LDAP restore) such as those provided by PHD Virtual, Alike etc. and which leverage the disk snapshot APIs in both vSphere and XenServer safe?
4. The Admin Guide points out "zmplayredo" can be used when snapshots of subfolders within /opt/zimbra are taken at different times, but suppose we just snapshot the whole virtual machine at one time or utilize SAN replication? The Admin Guide is silent on what we need to do after running "zmplayredo" to complete the DR process. Do we then boot the Zimbra DR server, restore LDAP, start Zimbra OK and then just change public DNS at that point?

Thanks Quanah for any help/pointers you can provide here!

All the best,
Mark

[quanah]

Hi Mark,

If you start a separate thread on DR, I'll be happy to comment there. I think this thread has gone on long enough. Just PM me the link to the DR thread.

[quanah]

Just wanted to note: With ZCS 8.0.2 and later, there is a utility named "mdb_copy" that can be used to copy the database. The resulting copy will *not* be a sparse file and instead will be the actual size of the database. I.e., 25MB in size or whatever your DB size actually is.