Setting up cold backup server

[msheean]

Hi All,
I'm wanting to set up a cold backup server in case of disaster on the primary and also to have as a standby in case an upgrade does not work correctly.

I have two servers with exactly the same hardware and OS. Looking at these pages:

Ajcody-Notes-Server-Move - Zimbra :: Wiki

and

Moving ZCS to Another Server » Zimbra :: Blog

it seems possible to rsync or copy to an external harddrive to a backup server. I know that the primary would need to have zimbra shutdown before the final rsync. The problem is that with our ~250GB installation it can take a loooong time to do the rsync. Our /opt/zimbra/backup folder has millions of files and the store folder has plenty of small message files that take well over a day to rsync. We can't be without mail for that long.

I know I could rsync the /opt/zimbra/backup folder separately to save some time but still it takes hours to rsync the rest. Has anyone else run into this problem?

[bdial]

do an rsync, shut down zimbra, then do another rsync with the -delete option.

this will rsync only changes (indexes, mysql db files, new blobs), and remove data that has been deleted from the main server.

[ewilen]

I'm also finding rsync to be not-very-fast. It doesn't take days but upwards of 1.5 hours is typical, even when syncing locally to an external USB hard drive from a stopped Zimbra install.

So far I've tried rsync version 3.0.7 but that doesn't seem to have helped, in fact it may even be slower. It might be worth a try for you, though.

Another thing I'm looking into is changing the value of /proc/sys/vm/vfs_cache_pressure as mentioned at Old Nabble - Samba - rsync - sync performance falls off a cliff, but I haven't tried that yet. If that helps, then adding RAM to the system might also speed things up. (I currently have 8 Gb on the primary.)

[msheean]

Thanks for your comments. I'm trying different things. One option for rsync might be to use the "--inplace" option so that files can be copied directly instead of having to make a copy and the overwrite the destination. Might be a little risky but if your zimbra is shutdown it doesn't seem like a problem.

[ewilen]

Hm, it would help if you could break down how much time is spent on the compare vs. actual copying. If the former (which is my situation), then I doubt that --inplace would help much.

Unfortunately the only way I've found to check this is to watch the console live. None of -v, -vv, or --progress seem to break down the time spent on each phase, in a way that's readable after the fact. Correction: use --stats to get this data. Note though that it only shows the time spent before copying starts, so with rsync 3's default incremental recursion algorithm, it can be a little misleading.

Another thing to try, if the issue is caching, is to sync the data in smaller chunks overall. Maybe a script to iteratively rsync directory-by-directory, or by initial-character-in-filename, or some combination.

[msheean]

I use "time" in front of the command to see how long it takes. If I do --inplace and --exclude 'backup' to avoid copying the backup folder then it takes about 35-45 minutes. Should be ok with that if I ever need to switch to cold backup server. Thanks all.

[ewilen]

Thanks for that tip.

If you would like to automate bringing up the cold standby, a good starting point is How to: cold standby server (no cluster). I'll try to post further details when I've got things going smoothly. But in the meantime, Bug 33606 - Improve zmrestore & zmrestoreoffline performance is also worth a look. The point there is: when it comes time to restore, you can greatly improve performance by just restoring the last full backup and then playing zmplayredo once for each subsequent redolog in the incrementals, in /opt/zimbra/redolog/archive/, and finally /opt/zimbra/redolog/redo.log

Additional notes:

1. Bug 31536 – add ability to restore accounts not yet backed up (but still in redologs) means that an account added after the last backup (incremental/full) won't be restorable. So, please consider voting for that bug...and in the meantime, consider kicking off an incremental after creating new accounts.

2. As shown by the "how to" thread, you can keep the cold standby in a nearly-ready-to-go state by doing a full install of zimbra, putting it in a quiescent state, and periodically doing a restore.

3. However, the cold standby script uses zmrestore, so the mailbox process has to be running. If this is undesirable for some reason, I think one could use zmrestoreoffline instead.

4. Also, you'll want to modify the zimbra crontab on a "quiescent" cold standby, as it has a lot of stuff you only want to run on a live server.

5. Be sure to keep track of (or sync) any customizations you make to salocal.cf.in, amavisd.conf.in, etc. because they'll be lost when you install zimbra on top of the rsync'ed /opt/zimbra.

[Ash-Fox]

The problem is that with our ~250GB installation it can take a loooong time to do the rsync. Our /opt/zimbra/backup folder has millions of files and the store folder has plenty of small message files that take well over a day to rsync. We can't be without mail for that long.

I know I could rsync the /opt/zimbra/backup folder separately to save some time but still it takes hours to rsync the rest. Has anyone else run into this problem?

As I do remote backups over the Internet, here is how I get around it. I shut down the server and do a mass hardlink of all the files to a /tmp path, I then delete the Zimbra's temporary files (which aren't needed in backup obviously) in the /tmp copy and delete the pid files in the /tmp copy. Deleting the pid files is essential, because daemons like postfix will refuse to start up if their pid file is hard linked. I then immediately after start zimbra up again and do the rsync on the /tmp files.

As hardlinked files automatically delink when either side is modified, leaving the unmodified copy intact, this gets around the known issue. Since the majority of files won't get modified substantially, you shouldn't expect a major increase in space used on the drive.

Here is a simplified script of what I use:

# Stop services
/etc/init.d/zimbra stop
# A small sleep is used in case a core is not being very responsive
sleep 2s
# Make sure all Zimbra owned processes are dead.
pkill -9 -u zimbra
# A small sleep is used in case a core is not being very responsive
sleep 2s
# Make sure there is nothing in the temporary path.
rm -rf /tmp/sysbackup
# Create temporary path
mkdir /tmp/sysbackup
# Hardlink Zimra installation
cp -al /opt/zimbra /tmp/sysbackup/
# Delete pid files
rm -f /tmp/sysbackup/opt/zimbra/data/postfix/master.lock /tmp/sysbackup/opt/zimbra/openldap-*/var/run/slapd.pid /tmp/sysbackup/opt/zimbra/cyrus-sasl-*/state/saslauthd.pid /tmp/sysbackup/opt/zimbra/db/mysql.pid /tmp/sysbackup/opt/zimbra/log/zmmailboxd_java.pid /tmp/sysbackup/opt/zimbra/log/memcached.pid /tmp/sysbackup/opt/zimbra/log/*.pid /tmp/sysbackup/opt/zimbra/zmstat/pid/* /tmp/sysbackup/opt/zimbra/data/postfix/spool/pid/*
# Delete temporary files
rm -rf /tmp/sysbackup/opt/zimbra/tmp /tmp/sysbackup/opt/zimbra/data/tmp
# Start zimbra in the background
/etc/init.d/zimbra start &
# rsync to remote server
rsync --delete --partial --stats -ihHavv -e "ssh -C  -o CompressionLevel=9 -o ConnectionAttempts=15 -o ConnectTimeout=10 -o TCPKeepAlive=yes -o ServerAliveInterval=300" /tmp/sysbackup/ login@hostname:/path/to/backup/folder
# Delete temporary folder
rm -rf /tmp/sysbackup

This keeps the server down time to a minimum and shouldn't take substantially more space.

[Hivos]

Thanks for sharing this, it's a simple but very effective solution I think, the kind I like .

[ewilen]

As hardlinked files automatically delink when either side is modified, leaving the unmodified copy intact, this gets around the known issue.

Are you sure about this? It's certainly true for some modifying programs, because they don't really modify the original file: they write a modified copy and delink the original. But is it true of all the zimbra modules?