ZCS NE Backup - Archiving To Tape Is To Slow

[Chewie71]

Hi,

I've been perusing all the backup documents in the wiki and have searched through a lot of the forum discussions regarding backup. I haven't seen any one who indicates they are having the problem we are having though.

We would like to get tape backups of the Zimbra backups (/opt/zimbra/backup), but in it's native state it is just too slow. We think the problem is that the Zimbra backup is producing thousands (probably tens of thousands or more) of small files....and it is negatively impacting the performance of our tape backups.

We currently have a Sun Library with 2 LTO3 drives in it, and this is sitting on 4GB fiber channel. The speed of the hardware is not the issue. We can tar a single large file off to the tape drives at 50-80MB/s. Trying to do the same with a very large subdirectory of Zimbra backups gets us results in the 10-15MB/s range initially, but gradually dropping as time goes on. At times we can see it get well below 1MB/s.

We have 3 mailstores, each with about 250GB of backups on it. Later this Spring we will be adding 2 more mailstores, 13,000 or so users, and probably almost double our backup data(1.5-2TB). We can't realistically back this up at those slow speeds. But that's not the worst part...restores go so much slower than backups that it would probably take weeks to restore these systems if we had to come back from a major disaster.

Our current workaround is to tar /opt/zimbra/backup to another LUN on our SAN, and then backup that single tar file to tape(one for each mailstore)... which goes much faster. This however requires us to have double the storage requirements to hold all the backups and the nightly tars.

So...what is the answer? Has anyone else in a large enterprise environment run into this and found a workaround? I saw somewhere that we could use the '-z' parameter on zmbackup, to zip up the blobs. Will that reduce the number of really small files that get created on a zimbra backup? If so that might solve the problem for us....but I'm not sure. Anyone have any thoughts or discovered a good (very fast) way to archive your Zimbra backups off to tape?

Thanks,
Matt

[vince]

We were running into backup issue with the Veritas Netbackup solution we have. Our backup infrastructure is pretty robust but it was having trouble backing up the /opt/zimbackups directory. What we did was create separate policies for each /opt/zimbackups/backup/sessions/full*, /opt/zimbackups/backup/sessions/incr*, and then all the rest of the /opt/zimbra directory. This seems to have worked out for us but we are still looking for cleaner ways to do this. One idea we are still looking at is possibly doing snapshots of the backup directory through the SAN instead of the netbackup agent.

I also would be very interested in finding out if the "-z" option would make things better for backups.

Vince

[Klug]

AFAIK, the "-z" option compresses blobs while it seems your backup problems are related to the number of tiny files to backup...

[Rich Graves]

I see limited advantage to snapshots in the SAN. The Zimbra backup directory is essentially append-only, so except for a modicum of protection against file system corruption and insider vandals (and that's assuming the vandals can't also kill the snapshots), SAN snapshots are little different than simply extending the retention beyond the default 30 days.

If you need tape for longer-term archiving, look for a backup product that behaves more like "dump" than like "tar." It's usually an expensive add-on, unfortunately. Veritas seems to call theirs "Block-Level Incremental Backup (BLIB)."

If you think you need tape for offsite DR, consider putting your backup directories offsite instead. My /opt/zimbra/backup is SAN-mounted 5km away. I have not done it yet, but what I plan to experiment with is two /opt/zimbra/backup directories on different hardware in different physical locations. All incrementals would need to be synced to both, but the destination of the full backups would rotate.

[liston13]

look into bakbone software...as i was running into similar issues.
i move about 2TB a day.

[Chewie71]

Bakbone=Netvault Backup?

We're running an enterprise solution now, Syncsort's Backup Express, but disk to tape speeds won't go any faster than what raw tar can do. Maybe we need to look at a disk-to-disk solution (which Backup Express has but we haven't really used), and forget about the tapes?

They also have the option to do block level backups, which might work? But they are only supporting Solaris and Windows currently with that feature...and have said Linux support is coming this spring...but I'm not holding my breath. Plus it's an expensive addon.

What is everyone else doing?

Matt

[liston13]

correct bakbone=netvault.

typically I run a cronjob nightly on the /opt/zimbra/backup directory syncing the folder to another disk.
for live backups you have me there.

[jeffw]

I'd like to look at this problem the other way:

Say you had a catastrophic failure of your server/storage and needed to restore from tape.

You'd first have to restore your tape backups to /opt/zimbra/backups, and then use the Zimbra tools to restore into the mail store.

Creating all those small files twice would be very very very slow.

To me it seems that the Zimbra backups should be enhanced to allow the backups to consist of larger files (say one file per mailbox, or one file per account). This way the backup to tape would be very fast, and restoring from tape back into /opt/zimbra backups would be fast as well.

Thoughts?

-j

[Chewie71]

Making the backups consist of larger files would be a great option for us. Technically I don't know how difficult it would be for Zimbra to implement. Is there a Bugzilla RFE opened for this issue? I'll certainly vote for it if there is.

Matt

[jeffw]

Originally Posted by Chewie71

Making the backups consist of larger files would be a great option for us. Technically I don't know how difficult it would be for Zimbra to implement. Is there a Bugzilla RFE opened for this issue? I'll certainly vote for it if there is.

Matt

Bug Bug 16570 - TAR and split the backups option seems to be the closest to that....