Of backups and restoration

[lcxe]

I have a few questions about the backup and restoration process.

Firstly is backups accessed through the web interface. If we were to be left with nothing but them (the full and incremental backups) would we be able to restore our system with just those? Or do we need to have a full backup of the /opt/zimbra directory?

I see folders in the backups for accounts, ldap, redologs, session.xml and sys.
If you are able to do a restore with those backups, is it done through the web interface for non-account related things, or would that be done through the command line? Also how easy is this?

Lastly, how do the full and incremental backups work? Because they seem to be fairly inconsistent in size -- mainly talking about the full backups. Does it work more around the lines that a full backup is everything since the last full? Why is there such a disparity in sizes between the full backups (see below)? Once one of the full backups hits the deletion time does the next full backup pickup the data that that full was covering (is this why there's one full at 180G and the next closest is 16G?)?

zimbra@server:~/backup/sessions$ du -h --max-depth=1 .
158M    ./incr-20080930.050005.611
207M    ./incr-20081001.050008.233
180G    ./full-20081004.050004.143
258M    ./incr-20081008.050006.463
8.1G    ./full-20081011.050007.136
420M    ./incr-20081015.050004.579
16G     ./full-20081018.050004.026
365M    ./incr-20081002.050004.956
373M    ./incr-20081003.050005.608
278M    ./incr-20081005.050011.455
122M    ./incr-20081006.050006.597
284M    ./incr-20081007.050010.403
498M    ./incr-20081009.050008.952
400M    ./incr-20081010.050006.284
576M    ./incr-20081012.050006.029
138M    ./incr-20081013.050008.654
299M    ./incr-20081014.050005.140
5.2G    ./incr-20081016.050006.582
446M    ./incr-20081017.050004.978
468M    ./incr-20081019.050010.869
92M     ./incr-20081020.050005.046
536M    ./incr-20081021.050012.687
337M    ./incr-20081022.050009.621
275M    ./incr-20081023.050016.021
356M    ./incr-20081024.050010.692
9.8G    ./full-20081025.050007.107
1.4G    ./incr-20081026.050006.144
128M    ./incr-20081027.050005.500
412M    ./incr-20081028.050005.634
282M    ./incr-20080928.050010.209
120M    ./incr-20080929.050006.062
6.2G    ./full-20081009.124053.116
233G    .

I'm sorry for so many questions, I know it's a little unorganized, but any help is greatly appreciated!

Thanks,
Lucas

[uxbod]

Have you read through Network Edition :: Backup & Restore Admin Guide ?

[lcxe]

Just read through it, I think it answered a couple of my questions, but for the most part all my questions stand. The information in that doc is fairly broad where my questions are rather specific.

It may be the case that it's not that the docs don't have the answers, more that I'm not asking the right questions. I'll definitely give them another read through. I think I'll be able to find information on the first and second sets of questions, but there didn't seem to be anything really pertaining the the questions in my third set.

[bdial]

do you have it set to do a full of all accounts? (-f -a all ?)

[lcxe]

Yes, this is the current backup schedule:

Current Schedule:

        f 0 1 * * 6 -a all
        i 0 1,10,14,18 * * 0-5 -a all
        d 1m 0 0 * * *

Note: I just changed the incremental this morning which is why it's not reflected in the files seen with the du output.

[mmorse]

Backups contain everything you need, but if you can salvage /opt/zimbra/redolog it might prove useful.
Backup and Restore (restoring data & disaster recovery sessions)
Network Edition Disaster Recovery - Zimbra :: Wiki

The Open Source edition behaves the same way as does Network Edition in terms of logging the operations. The difference is in what happens with the redologs once they get to a certain size and roll over. In FOSS, the log is simply truncated. In NE, it is moved to a subdirectory /opt/zimbra/redolog/archive and later thrown into an incremental backup. The archive folder is then cleaned out during the incremental backup process.

If you don't run incremental backups the archive folder will never get cleaned out. As if you don't run those you're also typically willing to give up point-in-time restore functionality (restoreToTime/restoreToIncrLabel/restoreToRedoSeq) in that case you don't need to keep the redologs around at all; you just need the current redo.log file for crash recovery purposes. Thus the config setting for this type of operation: zmprov mcf zimbraRedoLogDeleteOnRollover TRUE

On FOSS you can set zimbraRedoLogDeleteOnRollover FALSE (TRUE by default) but you must remember to purge /opt/zimbra/redolog/archive/ frequently least you run out of disk space accident (script it).

Full Backup Process-
1. Backs up the LDAP directory.
2. Backs up the global system data including system tables and the local config.xml file.
3. Iterates through each account to be backed up and backs up the LDAP entries for those accounts.
4. Places the account’s mailbox in maintenance mode to temporarily block mail delivery and user access to that mailbox.
5. Backs up the mailbox.
a. Creates MySQL dump for all entries related to that mailbox.
b. Creates a backup of the index directory for that mailbox.
c. Backs up the message directory for that mailbox.
4. Returns that account’s mailbox to active mode and moves on to the next one.

Incremental Backup Process-
1. Backs up the LDAP directory.
2. Backs up the global system data including system tables and the local config xml.
3. Iterates through each account to be backed up and backs up the LDAP entries for those accounts.
4. Looks in the backup target directory to find the latest full backup for the account.
5. Moves the archive redo logs, created since the last backup, to the <backup target>/redologs directory.

Re your different full backup sizes: Hard linking to other backup directories.

Now we might change the default to -zip mode in GNR: Bug 31836 - backup: default to the zip option
Fewer files make it easier to copy or rsync later, and prevents you from running out of inodes. You can also easily delete individual backups rather than running zmbackup -del, and therefore keep just a few really old backups around for whatever compliance reasons you may have.
So easier to move off-site & separate them, but what's lost in zip backup is the ability to hard link any blob (shared among mailboxes or unshared) that was present in an earlier full backup on the same partition.

During backup, the digest of shared blobs are added to a set. In zip mode backup we still get save space by cross linking blobs across mailboxes for shared blobs (Works better in 5.0.5+ per bug 26624). The body of a shared blob is added once to a shared-blobs zip file, then a small pointer-only entry is added to a mailbox's zip file. But once again, with zip mode backup blobs can't be shared across two distinct backups, so you lose the hard linking optimization (speed and space) for blobs that are in an earlier full backup already when working from the same disk.

Have a read of Recent Admin Backup Tidbits - Part 1 » Zimbra :: Blog

[ewilen]

Originally Posted by mmorse

Backups contain everything you need, but if you can salvage /opt/zimbra/redolog it might prove useful.

Forgive me for piggybacking but I'd appreciate if you could elaborate on this.

Suppose I am using rsync to copy /opt/zimbra/backup/ to a remote site for disaster preparedness. (Scenario: total loss of primary site.) What benefit if any would there be to also copying /opt/zimbra/redolog/ ?

Is there anything else that should/could be copied for this purpose?

I've been studying the Zimbra NE backup facilities for a bit now, and what I'm seeing suggests that if you want really up to date backups, your choices are:

• running incrementals as frequently as feasible within Zimbra
• employing snapshots at the storage layer as described on p. 154 of the current Administrator's Guide

It's unclear how frequent is feasible in the first case; I've found a thread from someone who does hourly incrementals in this thread: Keeping a backup server synced with live server

But are you suggesting that just having a copy of /opt/zimbra/redolog/ (using rsync on a live server) might be enough to recover mail up to the last few minutes...or hours...since the last the incremental?

[mmorse]

It sounds like what you really want is: Bug 31536 - add ability to restore accounts not yet backed up (but still in redologs) (GNR)

Here's an example:
Let's say you've taken a snapshot & stored it offsite.
You're constantly copying the redologs somewhere offsite (I'm pretending there's no backups, or FOSS with DeleteOnRollover FALSE etc.)
Your datacenter blows up!
You revert to snapshot.
As you’ve saved all the redologs since the snapshot simply use zmreplayredo --toTime.

It's an offline only tool, more on it: Bug 22659 - ability to replay redologs from arbitrary point in time (5.0.6+)

Bug 32415 - zmplayredo command appears to ignore mailboxId parameter (fixed in 5.0.11)

[lcxe]

Thank you for your reply Mike, it was very helpful and informative.

Also ewilen's question was quite helpful as well.

I believe that's all my questions for now, though I may perhaps dig up this thread at a later date for more questions.

Thanks again!

[ewilen]

Originally Posted by mmorse

It sounds like what you really want is: Bug 31536 - add ability to restore accounts not yet backed up (but still in redologs) (GNR)

(BTW, what does GNR mean?)

Thanks for the answer and sorry for the belated reply.

After playing around with rsync'ing /opt/zimbra/backup and /opt/zimbra/redolog, followed by use of

zmrestoreldap
zmrestoreoffline
zmplayredo

(with appropriate arguments and necessary turning on/off of zimbra services before each command) I find I'm very close to what I want and I now understand the limitation you reference above. I.e., it seems that if I create a user today and my data center blows up before the next full/incremental backup, I will not be able to restore that user from backup...but I'll get practically everything else.

I can probably live with this although it will be nice if that feature request is incorporated. I'll go over there and try to add my support for it.

EDIT: I've just done that, but while writing up my comments it occurred to me that I could force an incremental backup whenever I add a new user, without downing the server. (a) I assume this'll get that user's account and mail over to the cold standby machine, yes? (b) Given a modest site, under 200 users, would this have much of an impact on resources, assuming the "normal" schedule for backups is 1/day? What if incrementals are scheduled 1/hour? I realize this also depends on mail trafffic, but let me put it this way, is it something you would consider feasible?