Remote mirroring of /opt/zimbra/backup dir using rsync over low-bandwidth link

[Apple_Eater]

Hello all,
I've just recently changed up my Zimbra backup strategy. Previously, I was just mirroring the entire /opt/zimbra directory and had disabled the backup mechanism that is included with NE. However, I found that this can lead to inconsistencies within accounts when a backup is restored, so I've now re-enabled the native NE backup to /opt/zimbra/backup and am trying to mirror this directory using rsync.

My problem is that the sheer volume of data created by the NE backup process is overwhelming my low-bandwidth rsync setup. The rsync goes thru a VPN to an off-site location, but since the locating housing our Zimbra server only has ~2.5 MBit upstream, the transfer speed is very limited.

Each full backup is about 65G, which would take around ~25 days to transfer over the given bandwidth. Daily incrementals are closer to ~200MB, which would not be an issue to transfer daily.

My question is: What are some procedures being used to mirror the /opt/zimbra/backup directory to an offsite location? We're already using the compression built in to rsync, but this provides little relief. While it would be technically possible to have one full backup and daily incrementals indefinitely, I find this a very disturbing solution.
Is there any solution to help rsync in dealing with full backups? As noted by the size of our incrementals, the daily changes are ~200 MB. If we could somehow have rsync only transfer the *changes* between the full backups every weekend, while doing daily transfers of the incrementals, this would be an optimal solution.

Has anyone else had to deal with mirroring of /opt/zimbra/backup over low-bandwidth connections?

[bofh]

Hello,

Uhm i know your Pain. I actually dropped the NE Hotbackup feaute at all cause i still dont believe in its implementation.

So first instead of rsync maybe a clusterfilesystem would work better ... anyway

i went now to a dirtyhot / cleancold backup mix of simply the hole zimbra directory.

so what i do is start with the first d.hotbackup at for exampel 10am - maxruntime 5 hours
at 3:30pm - second run d.hotbackup - max runtime 1 hour
at 5pm - shutdown - coldbackup

(ok its in the night but the time in between is correct)
i use the same backupset for that and my own onlinebackup solution and save incremental and encrypted

to determine when todo a cold backup i use a simple shellscript as a prebackup /postbackup command
it does nothing else than look at the clock and make a shutdown at >=5pm and <6 and an startup at >=5pm and <11pm

that way ive only to backup a minimum at coldbackup so only a downtime of 3 minutes and a minimum of datatransfer
also i do not need twice the amount of data on the zimbra server

My backup is compressed encrypted less that 50% of total zimbra usage so its the absolute minumum resources usuage i could do without compromising security

since we saw thats not a bad thing to reboot the zimbra frequently and 3 minutes in the night is something everyone can life with its the best solution atm at least for me

[Apple_Eater]

Have you done any test restores using this method? I unfortunately know very little about the inner workings of Zimbra, so I am not sure how it might handle any inconsistencies, i.e. with the dirtyhot backup. Previously I had just been using rsync to mirror the /opt/zimbra directory with the system online every 12 hours. It worked well, data transfer was small, etc. But I was worried about integrity.

However I like your idea of a short downtime and doing a cold backup. So perhaps I could shutdown all zimbra services for ~10 minutes, create a tgz backup or something, and use rsync's difference transferring algorithms to do the transfer?

I would really like to stick with NE's native backup methods if possible though. While I don't mind cooking up my own solution, I like to go with the "tried and true" tested, supported solutions when available. We paid a lot of money for the Network Edition, so I hate to waste one of it's more useful features!

[bofh]

No look,

You cannot restore the hotbackups - i mean you could but trouble will be comming for shure

The Hotbackups are only made to reduce the downtime during the coldbackup
so i make 2 hotbackups to collect all the fixed data - which is not chaging till cold
then when the coldbackup starts theres only a little data to collect - usually a few hundred megs - its done within 8 min max including shutdown and startup zimbra - of course how long it takes for you its a mtter of your line


the real backup IS ONLY the cold one

- you cannot rsync one tar file
however theres a backupscript for rsync rsyncing the hole backup for you
thers also a script called ftplicity which could do that - that script actually
make an incremental copy - split up in junks - encrypt em and upload em


however i do not recommend those scripted options
i really dont - no script backup is really relyable if you do not watch it every day

mayn things can go wrong and you get no information/no logs no overview
so id recommend using a prof. backupsolution if you do what i do
otherwise stick with ne backup

but take a look at the OSS backup solutions by script
theres also one which mirror the zimbra dir on harddrive
it backups hot first - then shutdown zimbra - rsync the differences between hot and shutdown state - restart zimbra

now you got a zimbradir to comfortly backup offsite by rsync with a minimum downtime

however as i stated more than once - keep watching every day
i often run into troubles with such scripted solutions - and very often i had to purge everything and start over



btw my backup are 100% relyable - i had a few restores - about 1 in every 60 days because a user killed his mailbox

i restore em to a virtual host - which is setup in a way i can restore the last backup and instantly start zimbra to backup that mailbox and restore it to production

how ever my luck is our backup server uses also an server - server replication
so the hole backup server is mirrored and one backupmirror hangs here in my company
so the restore of that backup not long over gigabit ethernet


i know ne backup can restore a single mailbox - but i dont want to use both
(to much resources /diskspace bandwith cpu) for little benefit
because ne backup is nothing more than a scripted solution and does not have features a real backup solution have


i also tell you something from 15 years expierence
script may work - may work relyable for a time but it complicate things
after a few years your Infrastructure endup in a very complicated setup where little can break down much

keep it simple keep it clean - it will get complicated by itself over the time


i use a lot of time to replace still working solutions with newer simplier once to reduce the amount of complexity frequently

also virtualisation dont make things better it makes it worse (on the complexitiy level)

[Apple_Eater]

Bump -- still looking for an rsync solution that leverages NE's native backup functionality. I'm sure I can't be the only person that uses rsync to move the NE backups offsite?

[bofh]

Im sorry but im not shure you really should do that since your knowlege lacks a bit in that topic und such solutions arent easy to maintain.

however youre not looking for rsync - rsync can only sync files - not binary deltas - you need rdiff or dulicity or something like that

those are rsync variants who can create delta file.
deltas are differential files between file A and B
so you have to initially transfer those files and backup alter only binary differentials


and still this is a problem.
ne backupfiles ae new every day - for the bandwith you describd to much
rdiff backup on zimbrafolder itself = need coldbackup

only solution - i also ready told you - take a look on those backupscripts for OSS

they create a local copy with rsync while coldbackup to reduce downtime and then sync it up the differentials - still those files can be huge so im not shure if they finish in time

even worse - if you get a shedule problem (sync take to long) youre get easy into troubles

but really i dont get it - you spend a decnet amount on the NE version but wanna save money on backup?

why dont you simply rent a cheap dedicaded server somewhere for a few bucks and park data there or you go professional and take a decent backupsoftware you can save your trouble (or a decent cloudbackup provider)
mostly cheaper than a regular tapebbackup cost (you have to change one tape a month minimum with a 5+1 rotation so cost are min 100$ a months for tapebackup plus drive and software)

a cloudbackup gonna cost you around 80-120 bucks including space a month ...
way better than never be shre if your ass is save

[Apple_Eater]

This backup isn't about saving money, this is the only way I am aware to do it. We have to get this backed up offsite somehow, and our internet connection (the fastest available at our location) is only 2 Mbit upstream. For this same reason we couldn't have the email system hosted offsite because our employees will frequently exchange large (50-100mb) emails with eachother internally, that would take a very long time to send to a remote server just to download it again.

If I am missing something obvious please let me know, but the way I see it, the data has to get offsite "somehow". I already backup our 2TB+ datastore with unison to the same offsite location, but because of how the NE backups work, this doesn't really work for them (since they have a different name/path each time).

[bofh]

Yes youre missing something obiviouse
My posts

ive already written so one more time ill try it again....

First you cant rsync zimbra backupfiles - reason we both know
so you got 2 possible solutions

#1 the good one
Using a professional backup which can do incremental and infile delta backups

how does that work - first you make a incremental delta of the zimbra directory while zimbra is running

maybe you wanna do this twice or 3 times

then you make a copy wihting THE SAME backupset of the zimbra directory
but this time you use a prebackup command to shutdown zimbra


that way - the cold backup (when zimbra is down) only transfer the infile delta difference between the last hot backup and the state when its down

you can do this 2 way - one your backup software can do multiple shedules with individual preback post backup commands
#2 in case your software cannot do individual post / pre backup commands
you use my little shellscripts


why?
that way you make for example 3 times an incremental backup
1am - 4am - 5am
but the script will be set to 5 am shutdown
which create you automatically 2 times hotbackup (to collect data)
1 time cold backup - to make the real consistent backup



solution#2 - not so good but should work
use one of those OSS backup script in the zimbra wiki
it does the following

it will hotback zimbradir to another dir (to reduce the amount of data)
then shutdown zimbra
rsync the diff to the another dir (now you got a local backup for real)
startup zimbra (now we got a lot of time)

rsync offsite



----------
however in both cases of course the total anmoutn of data generated each day should not bigger than what you can transfer (obivisouly)
second method can have a lot of drawbacks you never reckognize

also consider a solution which CAN reduce the bandwidth backup uses
otherwise you may block your line totally


im using a solution can do both - but of course i got a big upstream so i dont have to bother about how long the hotbackup will take

but in case somethigng goes wrong i make the following to reduce the downtime

1 make 1 big hotbackup - sheduled for max runtime 5 hours
2 hours after the max runtime is ended another hotbackup - max runtime one hour (this should backup all new files some crazy users left behind)

3rd now we make the coldbackup - but just in case max runtime to 1 hour
in case that it will take longer i got here the fallback so zimbra is down max 1 hour in worst case szenario (which should NEVER happen but who knows)

worst case will be one backup not fully finished but zimbra is up and still some data to recover in case by the hotbackups


i hope i could describe understanbale what / how and why im doing and how this works


so now we calculate your data/hour trougput
lets say we make a max of 1 mbit for backups
means 8 seconds / MB gives us 7.5 MB / minute and 450MB per hour

that should be sufficent
so if you stick to the almost ident backupplan as i do you should be able to backup alsmost 3 gig during hotbackup and the rest during cold


so again in short

one backupset for all 3 backups below

Backup #1 - hotbackup (precollect data) 1am runtime 5 hours (till 6am)
Backup #2 - hotbackup (precollect data2) 6:30am runtime 1 hour (till 7:30am)
Backup #3 - coldbackup (our real backup 8am - our script above will now shutdown zimbra because its 8am - max runtime 1 hour just in case its to much we sacrifice a backup for uptime)


amount of data being backed up - 3.07 gigabyte


pps: why im doing the second hotbackup and not one with 6 hours -
it could be that the first hotback end after 30 minutes
but 2 hours later some big junks comming in
then the coldbackup would take to long - so the second hotbackup is another failsafe)

[ewilen]

Originally Posted by Apple_Eater

Bump -- still looking for an rsync solution that leverages NE's native backup functionality. I'm sure I can't be the only person that uses rsync to move the NE backups offsite?

It would be helpful to post what your actual backup command is in your zimbra crontab, and what your rsync command is.

If you use the --noZip option, then the fulls will be stored as uncompressed file/directory hierarchies, which allows hardlinking of duplicate data across backups. While this does mean that rsync has more things to compare, it also means that less data ultimately has to be sent over the wire between full backups.

Make sure you use the -H (or --hard-links) option in rsync so that hard links are preserved.

[Apple_Eater]

From crontab -l as zimbra user:

# BACKUP BEGIN
0 1 * * 6 /opt/zimbra/bin/zmbackup -f -a all --mail-report
0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i --mail-report
0 0 * * * /opt/zimbra/bin/zmbackup -del 1m --mail-report
# BACKUP END

The --noZip option sounds perfect, is there a way to implement it via the GUI? Or at least set a persistent setting so that it will survive upgrades?

Also the current behavior (with the commands listed above) is already to store the full directory structure in an unzipped format, however no hardlinks were immediately evident in the directory structure.

Also:
zmcontrol -v

Release 6.0.10_GA_2692.UBUNTU8_64 UBUNTU8_64 NETWORK edition.

We haven't made the jump to the 7.x series yet because we haven't seen any seriously compelling improvements to warrant the switch -- has the backup system had significant changes in the new series that might be useful to our situation?

Thanks for the help!