Question about redo logs and disaster recovery

[dlochart]

I just finished reading the disaster recovery WIKI entry and I am confused about one thing. I see where you mount the old drive (this recovery is going to a new server) to get at the backups. And then you restore from them. Let's say that the server crashed at 3:00pm on Wednesday. Last full backup was done on Sunday and last incremental was done Tuesday night. If you restore from the last incremental you are going back to that point in time correct? In the WIKI it made no mention of copying over the redo logs that have amassed since the last incremental backup so that when you restart zimbra it will be brought up to date.

Am I totally off base with this or what. The WIKI article was informative but I still do not have that warm fuzzy that it is complete.

[dlochart]

I am sure someone has had to recover from a disaster situation in which the main server is dead and you have to make a new one. All forum posts on this subject cover a migration giving you the chance to do a full backup before moving over to a new server.

So what do you do in my scenario? I ask because I need to repeat this tomorrow on a few VM's in case something tragic happens.

mulit server config: MTA + MailStore/LDAP

Scenario
------------
It is Wednesday at 3:00 pm. Last Full backup was on Sunday and Last incremental was on Tuesday night.
MailStore crashes hard. Disk is accessible but that is it.

I see the WIKI instructions but they do not cover what to do with the redo logs that have the transactions since the Tuesday night incremental backup.

Do you copy them over to the new server after everything is installed and it will rerun them or do you need to run a CLI tool to do this in online/offline mode?

thanks

[jhahm]

Let me first explain how the redo logs are used in backup and restore in general.

As changes are made on a running ZCS server, they are logged to the /opt/zimbra/redolog/redo.log file. When it reaches a certain size, it is closed and moved to /opt/zimbra/redolog/archive directory, and a new redo.log is opened. Thus redo logs pile up in the archive directory. Then, incremental backup moves all archive/* redo logs to backup directory.

Restore always starts with data in a full backup. Then all redo logs in incremental backups since the full backup are processed to replay the changes. When all incremental backups are processed, redo replay moves to the logs in /opt/zimbra/redolog/archive. Finally, the /opt/zimbra/redolog/redo.log file is processed to bring things up to the present. Various options can be passed to the restore command to control exactly where to stop the replay, achieving point-in-time recovery.

So, in your disaster recovery scenario, you have full backup and incremental backup available copied to a server, your ZCS server crashed and a spare machine was put in. How to restore to the most recent time, assuming the crashed server's hard disk is readable?

On the new ZCS server, copy over the full and incremental backups or mount them. Let's say you mounted it at /zcsbackup. So you have /zcsbackup/sessions/full-..., incr-..., and so on. You can then stop ZCS and copy over the contents of /opt/zimbra/redolog directory from crashed server to the new server. Copy over both redo.log and archive/*.

Then in the zmrestore or zmrestoreoffline command, specify "-t /zcsbackup" to indicate where the full/incremental backups are. Also, do *not* specify "-br" options. If you specify it, redo log replay stops after the incremental backups. Also don't specify "-rf" option. That option will restore only to the full backup.

Every redo log file has a sequence number. For replay to work correctly, all redo logs that are replayed must have consecutive sequence without gap. This should always be the case on a normally running system. So the main trick in your scenario is to get the backup and redolog directory contents to look as if everything is normal.

[dlochart]

Thanks for taking the time to explain that. Now I understand and can move forward. I plan to recreate this scenario with a virtual machine as a proof of concept.

[mannes]

I made a server's upgrade from 4.5.9 > 4.5.11 > 5.0.0.2
Is it safe to delete all that I have in /redolog ??

[jhahm]

Assuming you aren't running incremental backups (which would have moved those logs away already), you can safely delete /opt/zimbra/redolog/archive/*. Don't delete /opt/zimbra/redolog/redo.log as that file is in use by mailboxd.

[mannes]

I have a ZCS Network edition, so I am currently running incremental backups. Does this influences with anything?? Or I can still safely delete /opt/zimbra/redolog/archive/* (without the redo.log)? I'm in a desperate need of disk space...

[jhahm]

Then why don't you manually run an incremental backup now? Hopefully enough space will be freed up.

If you don't get enough space back and must delete the logs, keep in mind your backups will be missing these logs. Restore will not be able to get all data back. The problem will become irrelevant after the next full backup runs, assuming you won't have to do a point in time restore to a point before the full backup.

[mannes]

Our strategy is to run a full backup every Saturday, incremental Sun-Fri and to delete all backups older than 8 days.

[mannes]

in the /opt/zimbra/backup there's still a directory with old backups from the time when we were running the 4.5.11 version. Is it safe to delete this in order to free up some disk space (along with the correspondent entry in redolog)??