Page 1 of 2 12 LastLast
Results 1 to 10 of 14

Thread: ZCS-NE : rsync /opt/zimbra/backup

  1. #1
    breverend is offline Intermediate Member
    Join Date
    Jun 2009
    Posts
    22
    Rep Power
    5

    Default ZCS-NE : rsync /opt/zimbra/backup

    Hello,

    I need to keep offsite copies of my zimbra full and incremental backups with no time limitation.

    For now, I use a cronjob every night calling a script :

    /usr/bin/rsync -aHvz myzimbra.mycompany.com:/opt/zimbra/backup /local/dir/

    It does work but it's eating my disk space, as each new full backup takes 18 GB instead of less than 2 on the server.

    It's not keeping the hardlinks. I was thinking about use the --link-dest option of rsync, and filter on full backups only, but the "small full backups" (made when you create a new account) kill this solution.

    So, anyone have experience with offsite copies of /opt/zimbra/backup ?

    Thanks

  2. #2
    dougz is offline Active Member
    Join Date
    Aug 2007
    Posts
    31
    Rep Power
    7

    Default

    i use rsync+tar/gz+sshfs/fuse to backup to another server.
    sshfs act like nfs, but over ssh.

    CAUTION: sshfs !! not shfs. shfs is very slow for huge files.
    You can use tar to insert the data on stdin and redirect for a ssh connection if you dont want to compile a kernel module like sshfs.

    Useful link:
    SSH Filesystem

    sample usage:
    #create a local dir to map the remote dir
    mkdir /var/tmp/tmprfs

    #mount the remote dir
    $BPATH/sshfs $REMOTESERVER:$REMOTEDIR /var/tmp/tmprfs -o reconnect

    #use tar and gzip to compress and redirect to remote dir:
    $TAR -zcvf /var/tmp/tmprfs/$BACKUPFILE -T $SELECTION --exclude-from=$EXCLUSION > /var/tmp/tmprfs/rel.ok 2>rel.err

    #the errors are redirected to rel.err file, and you can check the rel.ok to view all files on backup

  3. #3
    ewilen's Avatar
    ewilen is offline Moderator
    Join Date
    Jun 2008
    Location
    Berkeley, CA
    Posts
    1,474
    Rep Power
    8

    Default

    Quote Originally Posted by breverend View Post
    Hello,

    I need to keep offsite copies of my zimbra full and incremental backups with no time limitation.

    For now, I use a cronjob every night calling a script :

    /usr/bin/rsync -aHvz myzimbra.mycompany.com:/opt/zimbra/backup /local/dir/
    Is /local/dir/ always the same? If not then I think you will be copying all of /opt/zimbra/backup every time that you use a new /local/dir/

    Also, your local machine is Unix/Linux, right? I'm not sure if rsync on Windows will handle hardlinks properly.

    This might interest you: rsnapshot

    So, anyone have experience with offsite copies of /opt/zimbra/backup ?
    Sort of...I've used rsync in the past to test various options for maintaining a cold standby (which I hope to implement before year's end). It seemed to work fine but I wasn't dealing with a great deal of data.

  4. #4
    jsturtevant is offline Member
    Join Date
    Dec 2006
    Posts
    10
    Rep Power
    8

    Default

    this would be useful to add to the wiki as storing the backups on another machine is desirable.

    i'm using: rsync -avzp --delete /opt/zimbra backup-01:/usr/local/backup/mail-02

    any comments as to whether these are the best rsync options to mirror the /opt/zimbra/backup on another machine?

  5. #5
    mlanner is offline Special Member
    Join Date
    Nov 2007
    Location
    Berkeley, CA
    Posts
    115
    Rep Power
    7

    Default

    I was wondering the same. What about tossing an -H in there for preserving hardlinks? Like:

    Code:
    rsync -avzpH --delete /opt/zimbra backup-01:/usr/local/backup/mail-02
    I noticed they were using that here:

    Moving ZCS to Another Server » Zimbra :: Blog

  6. #6
    gdr777 is offline Junior Member
    Join Date
    Jul 2009
    Posts
    6
    Rep Power
    5

    Default

    Rather than copying/rsyncing the backup files, why not just create a remote mounted directory where all of the backups are initially created. sshfs could be used if the remote host is across an untrusted network. To keep zimbra from deleting backups older than 1 month, edit the crontab that controls backup, and comment out the 3rd line in the backup section.

  7. #7
    uxbod's Avatar
    uxbod is offline Moderator
    Join Date
    Nov 2006
    Location
    UK
    Posts
    8,016
    Rep Power
    24

    Default

    The problem with that is it would not keep any manually edited configuration files eg. spamassassin which is why people rsync against /opt/zimbra.

  8. #8
    gdr777 is offline Junior Member
    Join Date
    Jul 2009
    Posts
    6
    Rep Power
    5

    Default

    True, but the original poster was having issues w/ preserving hard links in /opt/zimbra/backup. Relocate the default backup location /opt/zimbra/backup outside of /opt/zimbra. /opt/zimbra can then be rsynced/backed up however one wants w/o backup tree getting in the way.

  9. #9
    LMStone's Avatar
    LMStone is offline Moderator
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,366
    Rep Power
    10

    Default

    rsync also has an --exclude switch, which can be used multiple times in a commandline.

    Our best practice is to have /opt/zimbra/backup be on a separate ext3 formatted JBOD. That way, if (sorry, when -- all hardware eventually fails) the Zimbra server blows up, you can simply attach the JBOD to the replacement server.

    If you run a cron job to rsync things like /opt/zimbra/conf to, say, /opt/zimbra/backup/zcs-conf periodically, you are covered.

    So, if you keep a spare chassis handy with an operating system already installed and your production Zimbra server blows up, you can be back up and running very, very quickly.

    Hope that helps,
    Mark

  10. #10
    breverend is offline Intermediate Member
    Join Date
    Jun 2009
    Posts
    22
    Rep Power
    5

    Default

    Thanks all for your answers.
    I'm getting back on this issue. Let me be more specific :

    I have 1 remote server which is my production Zimbra. Let's call it "server"
    The install is standard, and I'm running ZCS 6.0.6 at the moment. I have around 30 accounts.
    I also have an "office" server, which I can access physically. Let's call it "backup"

    What I want to do is just replicate the content of
    server:/opt/zimbra/backup
    to
    backup:/backups/zimbra/backup

    So inside of those dir, I would have the same tree :
    accounts.xml sessions tmp

    And inside backup:/backups/zimbra/sessions, I want to have all full and incr backups, i.e. those on server:/opt/zimbra/backup/sessions + those which have been deleted over time.

    Currently, I'm using :
    on "backup" :
    rsync -aHvz server:/opt/zimbra/backup /backups/zimbra

    Now, maybe I'm just reading wrong info from df and du commands, but those backups are really taking a lot of diskspace :

    Code:
    root@server:/opt/zimbra/backup/sessions# du -hs *
    22G	full-20100604.230010.263
    612K	full-20100610.230034.511
    23G	full-20100611.230008.043
    23G	full-20100618.230010.257
    620K	full-20100623.230039.677
    23G	full-20100625.230010.855
    89M	incr-20100531.230011.174
    58M	incr-20100601.230010.711
    109M	incr-20100602.230012.958
    63M	incr-20100603.230009.654
    62M	incr-20100605.230011.670
    8.0M	incr-20100606.230011.065
    55M	incr-20100607.230012.156
    111M	incr-20100608.230011.642
    78M	incr-20100609.230009.626
    64M	incr-20100610.230010.017
    51M	incr-20100612.230017.684
    25M	incr-20100613.230010.515
    106M	incr-20100614.230011.702
    54M	incr-20100615.230017.320
    94M	incr-20100616.230015.921
    113M	incr-20100617.230010.885
    218M	incr-20100619.230014.408
    15M	incr-20100620.230010.343
    119M	incr-20100621.230014.579
    85M	incr-20100622.230016.570
    81M	incr-20100623.230012.822
    198M	incr-20100624.230010.893
    108M	incr-20100626.230014.666
    9.8M	incr-20100627.230010.251
    221M	incr-20100628.230011.867
    182M	incr-20100629.230009.865
    298M	incr-20100630.230011.408
    
    root@server:/opt/zimbra/backup/sessions# df -kh
    Filesystem            Size  Used Avail Use% Mounted on
    [...]
    /dev/md1              412G  124G  268G  32% /opt
    
    root@server:/opt/zimbra/backup/sessions# du -hs /opt/zimbra
    123G	/opt/zimbra
    There's almost only zimbra in /opt (rest is less than 1GB).
    Why are those backups taking so much space ?
    Over a year, it will eat 50*20GB, more than 1 TB of backups, just for less than 30 ppl ? Sounds a bit much.
    I also recall seeing some 1 or 2 GB full backups a few month ago. Maybe using the wrong command ?

    Anyway, as you can see, I'm loosing control over the situation (as I nearly have no more space left on my "backup" machine), so if you think I did something wrong, just tell me. For the record, I've got less than 1 year of experience on Zimbra, and my time is not fully going to it

    Thanks !

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Trouble Sending mail - All Messages deferred!
    By SiteDiscovery in forum Administrators
    Replies: 7
    Last Post: 09-03-2009, 04:52 AM
  2. Mail is being queued, not delivered!
    By icepick94 in forum Administrators
    Replies: 12
    Last Post: 01-22-2009, 07:03 AM
  3. Replies: 41
    Last Post: 10-29-2007, 02:36 PM
  4. ZCS 3.2 Beta Available
    By KevinH in forum Announcements
    Replies: 31
    Last Post: 07-07-2006, 03:46 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •