Page 1 of 3 123 LastLast
Results 1 to 10 of 22

Thread: Setting up cold backup server

  1. #1
    msheean is offline Member
    Join Date
    May 2010
    Location
    Tacoma
    Posts
    14
    Rep Power
    5

    Default Setting up cold backup server

    Hi All,
    I'm wanting to set up a cold backup server in case of disaster on the primary and also to have as a standby in case an upgrade does not work correctly.

    I have two servers with exactly the same hardware and OS. Looking at these pages:

    Ajcody-Notes-Server-Move - Zimbra :: Wiki

    and

    Moving ZCS to Another Server » Zimbra :: Blog

    it seems possible to rsync or copy to an external harddrive to a backup server. I know that the primary would need to have zimbra shutdown before the final rsync. The problem is that with our ~250GB installation it can take a loooong time to do the rsync. Our /opt/zimbra/backup folder has millions of files and the store folder has plenty of small message files that take well over a day to rsync. We can't be without mail for that long.

    I know I could rsync the /opt/zimbra/backup folder separately to save some time but still it takes hours to rsync the rest. Has anyone else run into this problem?

  2. #2
    bdial's Avatar
    bdial is offline Moderator
    Join Date
    Jul 2007
    Location
    Baltimore
    Posts
    1,649
    Rep Power
    11

    Default

    do an rsync, shut down zimbra, then do another rsync with the -delete option.

    this will rsync only changes (indexes, mysql db files, new blobs), and remove data that has been deleted from the main server.

  3. #3
    ewilen's Avatar
    ewilen is offline Moderator
    Join Date
    Jun 2008
    Location
    Berkeley, CA
    Posts
    1,474
    Rep Power
    9

    Default

    I'm also finding rsync to be not-very-fast. It doesn't take days but upwards of 1.5 hours is typical, even when syncing locally to an external USB hard drive from a stopped Zimbra install.

    So far I've tried rsync version 3.0.7 but that doesn't seem to have helped, in fact it may even be slower. It might be worth a try for you, though.

    Another thing I'm looking into is changing the value of /proc/sys/vm/vfs_cache_pressure as mentioned at Old Nabble - Samba - rsync - sync performance falls off a cliff, but I haven't tried that yet. If that helps, then adding RAM to the system might also speed things up. (I currently have 8 Gb on the primary.)

  4. #4
    msheean is offline Member
    Join Date
    May 2010
    Location
    Tacoma
    Posts
    14
    Rep Power
    5

    Default

    Thanks for your comments. I'm trying different things. One option for rsync might be to use the "--inplace" option so that files can be copied directly instead of having to make a copy and the overwrite the destination. Might be a little risky but if your zimbra is shutdown it doesn't seem like a problem.

  5. #5
    ewilen's Avatar
    ewilen is offline Moderator
    Join Date
    Jun 2008
    Location
    Berkeley, CA
    Posts
    1,474
    Rep Power
    9

    Default

    Hm, it would help if you could break down how much time is spent on the compare vs. actual copying. If the former (which is my situation), then I doubt that --inplace would help much.

    Unfortunately the only way I've found to check this is to watch the console live. None of -v, -vv, or --progress seem to break down the time spent on each phase, in a way that's readable after the fact. Correction: use --stats to get this data. Note though that it only shows the time spent before copying starts, so with rsync 3's default incremental recursion algorithm, it can be a little misleading.

    Another thing to try, if the issue is caching, is to sync the data in smaller chunks overall. Maybe a script to iteratively rsync directory-by-directory, or by initial-character-in-filename, or some combination.
    Last edited by ewilen; 05-24-2010 at 04:12 PM.

  6. #6
    msheean is offline Member
    Join Date
    May 2010
    Location
    Tacoma
    Posts
    14
    Rep Power
    5

    Default

    I use "time" in front of the command to see how long it takes. If I do --inplace and --exclude 'backup' to avoid copying the backup folder then it takes about 35-45 minutes. Should be ok with that if I ever need to switch to cold backup server. Thanks all.

  7. #7
    ewilen's Avatar
    ewilen is offline Moderator
    Join Date
    Jun 2008
    Location
    Berkeley, CA
    Posts
    1,474
    Rep Power
    9

    Default

    Thanks for that tip.

    If you would like to automate bringing up the cold standby, a good starting point is How to: cold standby server (no cluster). I'll try to post further details when I've got things going smoothly. But in the meantime, Bug 33606 - Improve zmrestore & zmrestoreoffline performance is also worth a look. The point there is: when it comes time to restore, you can greatly improve performance by just restoring the last full backup and then playing zmplayredo once for each subsequent redolog in the incrementals, in /opt/zimbra/redolog/archive/, and finally /opt/zimbra/redolog/redo.log

    Additional notes:

    1. Bug 31536 – add ability to restore accounts not yet backed up (but still in redologs) means that an account added after the last backup (incremental/full) won't be restorable. So, please consider voting for that bug...and in the meantime, consider kicking off an incremental after creating new accounts.

    2. As shown by the "how to" thread, you can keep the cold standby in a nearly-ready-to-go state by doing a full install of zimbra, putting it in a quiescent state, and periodically doing a restore.

    3. However, the cold standby script uses zmrestore, so the mailbox process has to be running. If this is undesirable for some reason, I think one could use zmrestoreoffline instead.

    4. Also, you'll want to modify the zimbra crontab on a "quiescent" cold standby, as it has a lot of stuff you only want to run on a live server.

    5. Be sure to keep track of (or sync) any customizations you make to salocal.cf.in, amavisd.conf.in, etc. because they'll be lost when you install zimbra on top of the rsync'ed /opt/zimbra.

  8. #8
    Ash-Fox is offline Intermediate Member
    Join Date
    May 2009
    Posts
    19
    Rep Power
    6

    Default

    Quote Originally Posted by msheean View Post
    The problem is that with our ~250GB installation it can take a loooong time to do the rsync. Our /opt/zimbra/backup folder has millions of files and the store folder has plenty of small message files that take well over a day to rsync. We can't be without mail for that long.

    I know I could rsync the /opt/zimbra/backup folder separately to save some time but still it takes hours to rsync the rest. Has anyone else run into this problem?
    As I do remote backups over the Internet, here is how I get around it. I shut down the server and do a mass hardlink of all the files to a /tmp path, I then delete the Zimbra's temporary files (which aren't needed in backup obviously) in the /tmp copy and delete the pid files in the /tmp copy. Deleting the pid files is essential, because daemons like postfix will refuse to start up if their pid file is hard linked. I then immediately after start zimbra up again and do the rsync on the /tmp files.

    As hardlinked files automatically delink when either side is modified, leaving the unmodified copy intact, this gets around the known issue. Since the majority of files won't get modified substantially, you shouldn't expect a major increase in space used on the drive.

    Here is a simplified script of what I use:
    Code:
    # Stop services
    /etc/init.d/zimbra stop
    # A small sleep is used in case a core is not being very responsive
    sleep 2s
    # Make sure all Zimbra owned processes are dead.
    pkill -9 -u zimbra
    # A small sleep is used in case a core is not being very responsive
    sleep 2s
    # Make sure there is nothing in the temporary path.
    rm -rf /tmp/sysbackup
    # Create temporary path
    mkdir /tmp/sysbackup
    # Hardlink Zimra installation
    cp -al /opt/zimbra /tmp/sysbackup/
    # Delete pid files
    rm -f /tmp/sysbackup/opt/zimbra/data/postfix/master.lock /tmp/sysbackup/opt/zimbra/openldap-*/var/run/slapd.pid /tmp/sysbackup/opt/zimbra/cyrus-sasl-*/state/saslauthd.pid /tmp/sysbackup/opt/zimbra/db/mysql.pid /tmp/sysbackup/opt/zimbra/log/zmmailboxd_java.pid /tmp/sysbackup/opt/zimbra/log/memcached.pid /tmp/sysbackup/opt/zimbra/log/*.pid /tmp/sysbackup/opt/zimbra/zmstat/pid/* /tmp/sysbackup/opt/zimbra/data/postfix/spool/pid/*
    # Delete temporary files
    rm -rf /tmp/sysbackup/opt/zimbra/tmp /tmp/sysbackup/opt/zimbra/data/tmp
    # Start zimbra in the background
    /etc/init.d/zimbra start &
    # rsync to remote server
    rsync --delete --partial --stats -ihHavv -e "ssh -C  -o CompressionLevel=9 -o ConnectionAttempts=15 -o ConnectTimeout=10 -o TCPKeepAlive=yes -o ServerAliveInterval=300" /tmp/sysbackup/ login@hostname:/path/to/backup/folder
    # Delete temporary folder
    rm -rf /tmp/sysbackup
    This keeps the server down time to a minimum and shouldn't take substantially more space.

  9. #9
    Hivos's Avatar
    Hivos is offline Advanced Member
    Join Date
    Aug 2009
    Location
    The Hague -- The Netherlands
    Posts
    214
    Rep Power
    5

    Default

    Thanks for sharing this, it's a simple but very effective solution I think, the kind I like .

  10. #10
    ewilen's Avatar
    ewilen is offline Moderator
    Join Date
    Jun 2008
    Location
    Berkeley, CA
    Posts
    1,474
    Rep Power
    9

    Default

    Quote Originally Posted by Ash-Fox View Post
    As hardlinked files automatically delink when either side is modified, leaving the unmodified copy intact, this gets around the known issue.
    Are you sure about this? It's certainly true for some modifying programs, because they don't really modify the original file: they write a modified copy and delink the original. But is it true of all the zimbra modules?

Page 1 of 3 123 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 658
    Last Post: 04-04-2014, 09:01 AM
  2. Need Help setting up family mail server
    By mbossmanh in forum Installation
    Replies: 1
    Last Post: 06-17-2009, 11:28 PM
  3. Replies: 9
    Last Post: 02-25-2009, 04:39 AM
  4. failed to install zimbra cos of zmmailboxd
    By aljoshab in forum Installation
    Replies: 4
    Last Post: 12-09-2008, 02:33 AM
  5. Server to Backup Server Sync.
    By Nitecon in forum Administrators
    Replies: 1
    Last Post: 10-27-2008, 01:21 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •