Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: ZCS NE Backup - Archiving To Tape Is To Slow

  1. #1
    Chewie71 is offline Trained Alumni
    Join Date
    Sep 2006
    Location
    Illinois
    Posts
    374
    Rep Power
    8

    Question ZCS NE Backup - Archiving To Tape Is To Slow

    Hi,

    I've been perusing all the backup documents in the wiki and have searched through a lot of the forum discussions regarding backup. I haven't seen any one who indicates they are having the problem we are having though.

    We would like to get tape backups of the Zimbra backups (/opt/zimbra/backup), but in it's native state it is just too slow. We think the problem is that the Zimbra backup is producing thousands (probably tens of thousands or more) of small files....and it is negatively impacting the performance of our tape backups.

    We currently have a Sun Library with 2 LTO3 drives in it, and this is sitting on 4GB fiber channel. The speed of the hardware is not the issue. We can tar a single large file off to the tape drives at 50-80MB/s. Trying to do the same with a very large subdirectory of Zimbra backups gets us results in the 10-15MB/s range initially, but gradually dropping as time goes on. At times we can see it get well below 1MB/s.

    We have 3 mailstores, each with about 250GB of backups on it. Later this Spring we will be adding 2 more mailstores, 13,000 or so users, and probably almost double our backup data(1.5-2TB). We can't realistically back this up at those slow speeds. But that's not the worst part...restores go so much slower than backups that it would probably take weeks to restore these systems if we had to come back from a major disaster.

    Our current workaround is to tar /opt/zimbra/backup to another LUN on our SAN, and then backup that single tar file to tape(one for each mailstore)... which goes much faster. This however requires us to have double the storage requirements to hold all the backups and the nightly tars.

    So...what is the answer? Has anyone else in a large enterprise environment run into this and found a workaround? I saw somewhere that we could use the '-z' parameter on zmbackup, to zip up the blobs. Will that reduce the number of really small files that get created on a zimbra backup? If so that might solve the problem for us....but I'm not sure. Anyone have any thoughts or discovered a good (very fast) way to archive your Zimbra backups off to tape?

    Thanks,
    Matt

  2. #2
    vince is offline Member
    Join Date
    Jul 2007
    Posts
    11
    Rep Power
    8

    Default

    We were running into backup issue with the Veritas Netbackup solution we have. Our backup infrastructure is pretty robust but it was having trouble backing up the /opt/zimbackups directory. What we did was create separate policies for each /opt/zimbackups/backup/sessions/full*, /opt/zimbackups/backup/sessions/incr*, and then all the rest of the /opt/zimbra directory. This seems to have worked out for us but we are still looking for cleaner ways to do this. One idea we are still looking at is possibly doing snapshots of the backup directory through the SAN instead of the netbackup agent.

    I also would be very interested in finding out if the "-z" option would make things better for backups.

    Vince

  3. #3
    Klug's Avatar
    Klug is offline Moderator
    Join Date
    Mar 2006
    Location
    Beaucaire, France
    Posts
    2,316
    Rep Power
    13

    Default

    AFAIK, the "-z" option compresses blobs while it seems your backup problems are related to the number of tiny files to backup...

  4. #4
    Rich Graves is offline Outstanding Member
    Join Date
    Jan 2007
    Location
    Minnesota
    Posts
    718
    Rep Power
    9

    Default

    I see limited advantage to snapshots in the SAN. The Zimbra backup directory is essentially append-only, so except for a modicum of protection against file system corruption and insider vandals (and that's assuming the vandals can't also kill the snapshots), SAN snapshots are little different than simply extending the retention beyond the default 30 days.

    If you need tape for longer-term archiving, look for a backup product that behaves more like "dump" than like "tar." It's usually an expensive add-on, unfortunately. Veritas seems to call theirs "Block-Level Incremental Backup (BLIB)."

    If you think you need tape for offsite DR, consider putting your backup directories offsite instead. My /opt/zimbra/backup is SAN-mounted 5km away. I have not done it yet, but what I plan to experiment with is two /opt/zimbra/backup directories on different hardware in different physical locations. All incrementals would need to be synced to both, but the destination of the full backups would rotate.

  5. #5
    liston13 is offline Intermediate Member
    Join Date
    Oct 2007
    Location
    New York City - Hell's Kitchen
    Posts
    17
    Rep Power
    7

    Default backup solution

    look into bakbone software...as i was running into similar issues.
    i move about 2TB a day.

  6. #6
    Chewie71 is offline Trained Alumni
    Join Date
    Sep 2006
    Location
    Illinois
    Posts
    374
    Rep Power
    8

    Default

    Bakbone=Netvault Backup?

    We're running an enterprise solution now, Syncsort's Backup Express, but disk to tape speeds won't go any faster than what raw tar can do. Maybe we need to look at a disk-to-disk solution (which Backup Express has but we haven't really used), and forget about the tapes?

    They also have the option to do block level backups, which might work? But they are only supporting Solaris and Windows currently with that feature...and have said Linux support is coming this spring...but I'm not holding my breath. Plus it's an expensive addon.

    What is everyone else doing?

    Matt

  7. #7
    liston13 is offline Intermediate Member
    Join Date
    Oct 2007
    Location
    New York City - Hell's Kitchen
    Posts
    17
    Rep Power
    7

    Default netvault

    correct bakbone=netvault.

    typically I run a cronjob nightly on the /opt/zimbra/backup directory syncing the folder to another disk.
    for live backups you have me there.

  8. #8
    jeffw is offline Loyal Member
    Join Date
    Oct 2007
    Posts
    88
    Rep Power
    7

    Default

    I'd like to look at this problem the other way:

    Say you had a catastrophic failure of your server/storage and needed to restore from tape.

    You'd first have to restore your tape backups to /opt/zimbra/backups, and then use the Zimbra tools to restore into the mail store.

    Creating all those small files twice would be very very very slow.

    To me it seems that the Zimbra backups should be enhanced to allow the backups to consist of larger files (say one file per mailbox, or one file per account). This way the backup to tape would be very fast, and restoring from tape back into /opt/zimbra backups would be fast as well.

    Thoughts?

    -j

  9. #9
    Chewie71 is offline Trained Alumni
    Join Date
    Sep 2006
    Location
    Illinois
    Posts
    374
    Rep Power
    8

    Default

    Making the backups consist of larger files would be a great option for us. Technically I don't know how difficult it would be for Zimbra to implement. Is there a Bugzilla RFE opened for this issue? I'll certainly vote for it if there is.

    Matt

  10. #10
    jeffw is offline Loyal Member
    Join Date
    Oct 2007
    Posts
    88
    Rep Power
    7

    Default

    Quote Originally Posted by Chewie71 View Post
    Making the backups consist of larger files would be a great option for us. Technically I don't know how difficult it would be for Zimbra to implement. Is there a Bugzilla RFE opened for this issue? I'll certainly vote for it if there is.

    Matt
    Bug Bug 16570 - TAR and split the backups option seems to be the closest to that....

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 210
    Last Post: 01-17-2012, 01:19 AM
  2. Trouble Sending mail - All Messages deferred!
    By SiteDiscovery in forum Administrators
    Replies: 7
    Last Post: 09-03-2009, 04:52 AM
  3. Replies: 41
    Last Post: 10-29-2007, 02:36 PM
  4. FYI: ZCS NE backup to fuse/sshfs mount, worked.
    By jagipson in forum Administrators
    Replies: 0
    Last Post: 09-28-2007, 06:37 AM
  5. ZCS 3.2 Beta Available
    By KevinH in forum Announcements
    Replies: 31
    Last Post: 07-07-2006, 03:46 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •