ZCS-NE : rsync /opt/zimbra/backup

Hello,

I need to keep offsite copies of my zimbra full and incremental backups with no time limitation.

For now, I use a cronjob every night calling a script :

/usr/bin/rsync -aHvz myzimbra.mycompany.com:/opt/zimbra/backup /local/dir/

It does work but it's eating my disk space, as each new full backup takes 18 GB instead of less than 2 on the server.

It's not keeping the hardlinks. I was thinking about use the --link-dest option of rsync, and filter on full backups only, but the "small full backups" (made when you create a new account) kill this solution.

So, anyone have experience with offsite copies of /opt/zimbra/backup ?

Thanks

i use rsync+tar/gz+sshfs/fuse to backup to another server.
sshfs act like nfs, but over ssh.

CAUTION: sshfs !! not shfs. shfs is very slow for huge files.
You can use tar to insert the data on stdin and redirect for a ssh connection if you dont want to compile a kernel module like sshfs.

Useful link:
SSH Filesystem

sample usage:
#create a local dir to map the remote dir
mkdir /var/tmp/tmprfs

#mount the remote dir
$BPATH/sshfs $REMOTESERVER:$REMOTEDIR /var/tmp/tmprfs -o reconnect

#use tar and gzip to compress and redirect to remote dir:
$TAR -zcvf /var/tmp/tmprfs/$BACKUPFILE -T $SELECTION --exclude-from=$EXCLUSION > /var/tmp/tmprfs/rel.ok 2>rel.err

#the errors are redirected to rel.err file, and you can check the rel.ok to view all files on backup

Quote:

---

Originally Posted by breverend

Hello,

I need to keep offsite copies of my zimbra full and incremental backups with no time limitation.

For now, I use a cronjob every night calling a script :

/usr/bin/rsync -aHvz myzimbra.mycompany.com:/opt/zimbra/backup /local/dir/

---

Is /local/dir/ always the same? If not then I think you will be copying all of /opt/zimbra/backup every time that you use a new /local/dir/

Also, your local machine is Unix/Linux, right? I'm not sure if rsync on Windows will handle hardlinks properly.

This might interest you: rsnapshot

Quote:

---

So, anyone have experience with offsite copies of /opt/zimbra/backup ?

---

Sort of...I've used rsync in the past to test various options for maintaining a cold standby (which I hope to implement before year's end). It seemed to work fine but I wasn't dealing with a great deal of data.

this would be useful to add to the wiki as storing the backups on another machine is desirable.

i'm using: rsync -avzp --delete /opt/zimbra backup-01:/usr/local/backup/mail-02

any comments as to whether these are the best rsync options to mirror the /opt/zimbra/backup on another machine?

I was wondering the same. What about tossing an -H in there for preserving hardlinks? Like:

Code:

---

rsync -avzpH --delete /opt/zimbra backup-01:/usr/local/backup/mail-02

---

I noticed they were using that here:

Moving ZCS to Another Server » Zimbra :: Blog

Rather than copying/rsyncing the backup files, why not just create a remote mounted directory where all of the backups are initially created. sshfs could be used if the remote host is across an untrusted network. To keep zimbra from deleting backups older than 1 month, edit the crontab that controls backup, and comment out the 3rd line in the backup section.

The problem with that is it would not keep any manually edited configuration files eg. spamassassin which is why people rsync against /opt/zimbra.

True, but the original poster was having issues w/ preserving hard links in /opt/zimbra/backup. Relocate the default backup location /opt/zimbra/backup outside of /opt/zimbra. /opt/zimbra can then be rsynced/backed up however one wants w/o backup tree getting in the way.

rsync also has an --exclude switch, which can be used multiple times in a commandline.

Our best practice is to have /opt/zimbra/backup be on a separate ext3 formatted JBOD. That way, if (sorry, when -- all hardware eventually fails) the Zimbra server blows up, you can simply attach the JBOD to the replacement server.

If you run a cron job to rsync things like /opt/zimbra/conf to, say, /opt/zimbra/backup/zcs-conf periodically, you are covered.

So, if you keep a spare chassis handy with an operating system already installed and your production Zimbra server blows up, you can be back up and running very, very quickly.

Hope that helps,
Mark

Thanks all for your answers.
I'm getting back on this issue. Let me be more specific :

I have 1 remote server which is my production Zimbra. Let's call it "server"
The install is standard, and I'm running ZCS 6.0.6 at the moment. I have around 30 accounts.
I also have an "office" server, which I can access physically. Let's call it "backup"

What I want to do is just replicate the content of
server:/opt/zimbra/backup
to
backup:/backups/zimbra/backup

So inside of those dir, I would have the same tree :
accounts.xml sessions tmp

And inside backup:/backups/zimbra/sessions, I want to have all full and incr backups, i.e. those on server:/opt/zimbra/backup/sessions + those which have been deleted over time.

Currently, I'm using :
on "backup" :
rsync -aHvz server:/opt/zimbra/backup /backups/zimbra

Now, maybe I'm just reading wrong info from df and du commands, but those backups are really taking a lot of diskspace :

Code:

---

root@server:/opt/zimbra/backup/sessions# du -hs *
22G        full-20100604.230010.263
612K        full-20100610.230034.511
23G        full-20100611.230008.043
23G        full-20100618.230010.257
620K        full-20100623.230039.677
23G        full-20100625.230010.855
89M        incr-20100531.230011.174
58M        incr-20100601.230010.711
109M        incr-20100602.230012.958
63M        incr-20100603.230009.654
62M        incr-20100605.230011.670
8.0M        incr-20100606.230011.065
55M        incr-20100607.230012.156
111M        incr-20100608.230011.642
78M        incr-20100609.230009.626
64M        incr-20100610.230010.017
51M        incr-20100612.230017.684
25M        incr-20100613.230010.515
106M        incr-20100614.230011.702
54M        incr-20100615.230017.320
94M        incr-20100616.230015.921
113M        incr-20100617.230010.885
218M        incr-20100619.230014.408
15M        incr-20100620.230010.343
119M        incr-20100621.230014.579
85M        incr-20100622.230016.570
81M        incr-20100623.230012.822
198M        incr-20100624.230010.893
108M        incr-20100626.230014.666
9.8M        incr-20100627.230010.251
221M        incr-20100628.230011.867
182M        incr-20100629.230009.865
298M        incr-20100630.230011.408

root@server:/opt/zimbra/backup/sessions# df -kh
Filesystem            Size  Used Avail Use% Mounted on
[...]
/dev/md1              412G  124G  268G  32% /opt

root@server:/opt/zimbra/backup/sessions# du -hs /opt/zimbra
123G        /opt/zimbra

---

There's almost only zimbra in /opt (rest is less than 1GB).
Why are those backups taking so much space ?
Over a year, it will eat 50*20GB, more than 1 TB of backups, just for less than 30 ppl ? Sounds a bit much.
I also recall seeing some 1 or 2 GB full backups a few month ago. Maybe using the wrong command ?

Anyway, as you can see, I'm loosing control over the situation (as I nearly have no more space left on my "backup" machine), so if you think I did something wrong, just tell me. For the record, I've got less than 1 year of experience on Zimbra, and my time is not fully going to it :)

Thanks !