Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: Of backups and restoration

  1. #1
    lcxe is offline Active Member
    Join Date
    May 2008
    Posts
    34
    Rep Power
    6

    Default Of backups and restoration

    I have a few questions about the backup and restoration process.

    Firstly is backups accessed through the web interface. If we were to be left with nothing but them (the full and incremental backups) would we be able to restore our system with just those? Or do we need to have a full backup of the /opt/zimbra directory?

    I see folders in the backups for accounts, ldap, redologs, session.xml and sys.
    If you are able to do a restore with those backups, is it done through the web interface for non-account related things, or would that be done through the command line? Also how easy is this?

    Lastly, how do the full and incremental backups work? Because they seem to be fairly inconsistent in size -- mainly talking about the full backups. Does it work more around the lines that a full backup is everything since the last full? Why is there such a disparity in sizes between the full backups (see below)? Once one of the full backups hits the deletion time does the next full backup pickup the data that that full was covering (is this why there's one full at 180G and the next closest is 16G?)?

    Code:
    zimbra@server:~/backup/sessions$ du -h --max-depth=1 .
    158M    ./incr-20080930.050005.611
    207M    ./incr-20081001.050008.233
    180G    ./full-20081004.050004.143
    258M    ./incr-20081008.050006.463
    8.1G    ./full-20081011.050007.136
    420M    ./incr-20081015.050004.579
    16G     ./full-20081018.050004.026
    365M    ./incr-20081002.050004.956
    373M    ./incr-20081003.050005.608
    278M    ./incr-20081005.050011.455
    122M    ./incr-20081006.050006.597
    284M    ./incr-20081007.050010.403
    498M    ./incr-20081009.050008.952
    400M    ./incr-20081010.050006.284
    576M    ./incr-20081012.050006.029
    138M    ./incr-20081013.050008.654
    299M    ./incr-20081014.050005.140
    5.2G    ./incr-20081016.050006.582
    446M    ./incr-20081017.050004.978
    468M    ./incr-20081019.050010.869
    92M     ./incr-20081020.050005.046
    536M    ./incr-20081021.050012.687
    337M    ./incr-20081022.050009.621
    275M    ./incr-20081023.050016.021
    356M    ./incr-20081024.050010.692
    9.8G    ./full-20081025.050007.107
    1.4G    ./incr-20081026.050006.144
    128M    ./incr-20081027.050005.500
    412M    ./incr-20081028.050005.634
    282M    ./incr-20080928.050010.209
    120M    ./incr-20080929.050006.062
    6.2G    ./full-20081009.124053.116
    233G    .
    I'm sorry for so many questions, I know it's a little unorganized, but any help is greatly appreciated!

    Thanks,
    Lucas

  2. #2
    uxbod's Avatar
    uxbod is offline Moderator
    Join Date
    Nov 2006
    Location
    UK
    Posts
    8,016
    Rep Power
    24

  3. #3
    lcxe is offline Active Member
    Join Date
    May 2008
    Posts
    34
    Rep Power
    6

    Default

    Just read through it, I think it answered a couple of my questions, but for the most part all my questions stand. The information in that doc is fairly broad where my questions are rather specific.

    It may be the case that it's not that the docs don't have the answers, more that I'm not asking the right questions. I'll definitely give them another read through. I think I'll be able to find information on the first and second sets of questions, but there didn't seem to be anything really pertaining the the questions in my third set.

  4. #4
    bdial's Avatar
    bdial is offline Moderator
    Join Date
    Jul 2007
    Location
    Baltimore
    Posts
    1,649
    Rep Power
    10

    Default

    do you have it set to do a full of all accounts? (-f -a all ?)

  5. #5
    lcxe is offline Active Member
    Join Date
    May 2008
    Posts
    34
    Rep Power
    6

    Default

    Yes, this is the current backup schedule:

    Code:
    Current Schedule:
    
            f 0 1 * * 6 -a all
            i 0 1,10,14,18 * * 0-5 -a all
            d 1m 0 0 * * *
    Note: I just changed the incremental this morning which is why it's not reflected in the files seen with the du output.

  6. #6
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    20

    Default

    Backups contain everything you need, but if you can salvage /opt/zimbra/redolog it might prove useful.
    Backup and Restore (restoring data & disaster recovery sessions)
    Network Edition Disaster Recovery - Zimbra :: Wiki

    The Open Source edition behaves the same way as does Network Edition in terms of logging the operations. The difference is in what happens with the redologs once they get to a certain size and roll over. In FOSS, the log is simply truncated. In NE, it is moved to a subdirectory /opt/zimbra/redolog/archive and later thrown into an incremental backup. The archive folder is then cleaned out during the incremental backup process.

    If you don't run incremental backups the archive folder will never get cleaned out. As if you don't run those you're also typically willing to give up point-in-time restore functionality (restoreToTime/restoreToIncrLabel/restoreToRedoSeq) in that case you don't need to keep the redologs around at all; you just need the current redo.log file for crash recovery purposes. Thus the config setting for this type of operation: zmprov mcf zimbraRedoLogDeleteOnRollover TRUE

    On FOSS you can set zimbraRedoLogDeleteOnRollover FALSE (TRUE by default) but you must remember to purge /opt/zimbra/redolog/archive/ frequently least you run out of disk space accident (script it).

    Full Backup Process-
    1. Backs up the LDAP directory.
    2. Backs up the global system data including system tables and the local config.xml file.
    3. Iterates through each account to be backed up and backs up the LDAP entries for those accounts.
    4. Places the account’s mailbox in maintenance mode to temporarily block mail delivery and user access to that mailbox.
    5. Backs up the mailbox.
    a. Creates MySQL dump for all entries related to that mailbox.
    b. Creates a backup of the index directory for that mailbox.
    c. Backs up the message directory for that mailbox.
    4. Returns that account’s mailbox to active mode and moves on to the next one.

    Incremental Backup Process-
    1. Backs up the LDAP directory.
    2. Backs up the global system data including system tables and the local config xml.
    3. Iterates through each account to be backed up and backs up the LDAP entries for those accounts.
    4. Looks in the backup target directory to find the latest full backup for the account.
    5. Moves the archive redo logs, created since the last backup, to the <backup target>/redologs directory.

    Re your different full backup sizes: Hard linking to other backup directories.

    Now we might change the default to -zip mode in GNR: Bug 31836 - backup: default to the zip option
    Fewer files make it easier to copy or rsync later, and prevents you from running out of inodes. You can also easily delete individual backups rather than running zmbackup -del, and therefore keep just a few really old backups around for whatever compliance reasons you may have.
    So easier to move off-site & separate them, but what's lost in zip backup is the ability to hard link any blob (shared among mailboxes or unshared) that was present in an earlier full backup on the same partition.

    During backup, the digest of shared blobs are added to a set. In zip mode backup we still get save space by cross linking blobs across mailboxes for shared blobs (Works better in 5.0.5+ per bug 26624). The body of a shared blob is added once to a shared-blobs zip file, then a small pointer-only entry is added to a mailbox's zip file. But once again, with zip mode backup blobs can't be shared across two distinct backups, so you lose the hard linking optimization (speed and space) for blobs that are in an earlier full backup already when working from the same disk.

    Have a read of Recent Admin Backup Tidbits - Part 1 &#187; Zimbra :: Blog
    Last edited by mmorse; 10-29-2008 at 06:38 AM.

  7. #7
    ewilen's Avatar
    ewilen is offline Moderator
    Join Date
    Jun 2008
    Location
    Berkeley, CA
    Posts
    1,474
    Rep Power
    8

    Default

    Quote Originally Posted by mmorse View Post
    Backups contain everything you need, but if you can salvage /opt/zimbra/redolog it might prove useful.
    Forgive me for piggybacking but I'd appreciate if you could elaborate on this.

    Suppose I am using rsync to copy /opt/zimbra/backup/ to a remote site for disaster preparedness. (Scenario: total loss of primary site.) What benefit if any would there be to also copying /opt/zimbra/redolog/ ?

    Is there anything else that should/could be copied for this purpose?

    I've been studying the Zimbra NE backup facilities for a bit now, and what I'm seeing suggests that if you want really up to date backups, your choices are:

    • running incrementals as frequently as feasible within Zimbra
    • employing snapshots at the storage layer as described on p. 154 of the current Administrator's Guide
    It's unclear how frequent is feasible in the first case; I've found a thread from someone who does hourly incrementals in this thread: Keeping a backup server synced with live server

    But are you suggesting that just having a copy of /opt/zimbra/redolog/ (using rsync on a live server) might be enough to recover mail up to the last few minutes...or hours...since the last the incremental?

  8. #8
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    20

    Default

    It sounds like what you really want is: Bug 31536 - add ability to restore accounts not yet backed up (but still in redologs) (GNR)

    Here's an example:
    Let's say you've taken a snapshot & stored it offsite.
    You're constantly copying the redologs somewhere offsite (I'm pretending there's no backups, or FOSS with DeleteOnRollover FALSE etc.)
    Your datacenter blows up!
    You revert to snapshot.
    As you’ve saved all the redologs since the snapshot simply use zmreplayredo --toTime.

    It's an offline only tool, more on it: Bug 22659 - ability to replay redologs from arbitrary point in time (5.0.6+)

    Bug 32415 - zmplayredo command appears to ignore mailboxId parameter (fixed in 5.0.11)

  9. #9
    lcxe is offline Active Member
    Join Date
    May 2008
    Posts
    34
    Rep Power
    6

    Default

    Thank you for your reply Mike, it was very helpful and informative.

    Also ewilen's question was quite helpful as well.

    I believe that's all my questions for now, though I may perhaps dig up this thread at a later date for more questions.

    Thanks again!

  10. #10
    ewilen's Avatar
    ewilen is offline Moderator
    Join Date
    Jun 2008
    Location
    Berkeley, CA
    Posts
    1,474
    Rep Power
    8

    Default

    Quote Originally Posted by mmorse View Post
    (BTW, what does GNR mean?)

    Thanks for the answer and sorry for the belated reply.

    After playing around with rsync'ing /opt/zimbra/backup and /opt/zimbra/redolog, followed by use of

    zmrestoreldap
    zmrestoreoffline
    zmplayredo

    (with appropriate arguments and necessary turning on/off of zimbra services before each command) I find I'm very close to what I want and I now understand the limitation you reference above. I.e., it seems that if I create a user today and my data center blows up before the next full/incremental backup, I will not be able to restore that user from backup...but I'll get practically everything else.

    I can probably live with this although it will be nice if that feature request is incorporated. I'll go over there and try to add my support for it.

    EDIT: I've just done that, but while writing up my comments it occurred to me that I could force an incremental backup whenever I add a new user, without downing the server. (a) I assume this'll get that user's account and mail over to the cold standby machine, yes? (b) Given a modest site, under 200 users, would this have much of an impact on resources, assuming the "normal" schedule for backups is 1/day? What if incrementals are scheduled 1/hour? I realize this also depends on mail trafffic, but let me put it this way, is it something you would consider feasible?
    Last edited by ewilen; 11-16-2008 at 05:53 PM.

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Backup & Restoration of SSL certs
    By JonathanDelgado in forum Administrators
    Replies: 0
    Last Post: 09-18-2008, 11:25 AM
  2. Editorial: TAKE TIME TO BACKUP
    By jholder in forum Installation
    Replies: 22
    Last Post: 07-19-2007, 07:02 PM
  3. old backups from 4.0.5_GA taking up too much space!
    By riccadus in forum Administrators
    Replies: 1
    Last Post: 04-04-2007, 12:58 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •