ZCS Administrator's Guide Network Edition 6.0.8

Disaster Recovery for Specific Situations

General Steps for Disaster Recovery

Crash Recovery Server Startup

Restore the Zimbra Collaboration Suite Servers

Preparing the New Server

Restoring from Different Failure Scenarios

Changing Local Configuration Files after Restoring Zimbra

This section provides general guidelines for disaster recovery. You can get more information from this wiki page http://wiki.zimbra.com/index.php?title=Network_Edition_Disaster_Recovery .

General Steps for Disaster Recovery

The sequence of events to restore your mailbox store server in a general disaster scenario involving multiple machines would be as follows:

Preparation

1.	Restore your LDAP directory server to a known good state before doing anything with the mailbox store server.

2.	Put all mailboxes into maintenance mode, to prevent mail delivery and user login while restoring the mailboxes.

3.	Stop the mailbox store server, if it is running.

Recovery

4.	Reinstall the ZCS software on the mailbox server, if necessary.

5.	Restore mailboxes.

6.	Start the Zimbra server.

7.	Put all Zimbra mailboxes back in active mode.

8.	Run a full backup of the server.

Crash Recovery Server Startup

When your system is unexpectedly stopped and then restarted, on startup, the server automatically searches the redo log for any uncommitted transactions, and replays any that it finds. Replaying the redo logs brings the system to a consistent state.

Restore the Zimbra Collaboration Suite Servers

This direction would be in the case of complete machine failure.

Important: The ZCS version you install on the new server must be the same version as installed on the old server. The server can have a different operating system.

The new server hardware must meet the requirements described in the Installation Prerequisites section of the ZCS Single Server Installation Guide. Install the new operating system, making any necessary OS configuration modifications as described in the installation guide.

Disaster Recovery Changing Servers

You do the following to restore to a new server:

•	Prepare the new server

•	Block client access to the old server’s IP address with firewall rules

•	Mount any volumes that were in use on the older server

•	Delete the MySQL data that is set up in the initial installation of ZCS

•	Copy the backup files to the new server

•	Run zmrestoreldap to restore the global LDAP data

•	Run zmrestoreoffline to restore account data from the backup sessions

•	Prepare and run a new backup

Old Server Status

Two scenarios for disaster recovery are the server has died and the ZCS files cannot be accessed, or ZCS is still running, but the server hardware needs to be replaced.

If the server is not running:

1.	Block client access to the server IP address with firewall rules.

2.	Find the latest full ZCS backup session to use.

If ZCS is still running, to prepare the move to the new server:

1.	Block client access to the server’s IP address with firewall rules.

2.	Run a full backup of the old service, or if the backup is recent, run an incremental backup to get the most current incremental backup session.

3.	Run zmcontrol stop, to stop ZCS. In order to restore to the most current state, no new mail should be received after the last incremental backup has run.

4.	Change the hostname and IP address on the old server to something else. Do not turn off the server.

Preparing the New Server

Before you begin, make sure that the new server is correctly configured with the IP address and hostname and that ZCS is installed and configured with the same domain, hostname, passwords, etc. as the previous server. See the ZCS installation guide for more information about preparing the server. Before you begin to install ZCS, note the information you need from the old server including: admin account name and password, LDAP, Amavis, and Postfix passwords, spam training and non-spam training user account names, exact domain name, and the global document account name.

Installing ZCS on new server

Note: Make sure the computer time is set to the same time as the old server. Verify that the old hostname and MX DNS records resolve to the new server.

1.	Copy your ZCSLicense.xml file to a directory on the new server. You will not be able to complete the ZCS installation if the license is not on the new server.

2.	Run ./install.sh and follow the directions in the installation guide to install ZCS. Make sure that you configure the same domain, hostname, passwords as on the old server. During ZCS install, the following settings must be changed to match the original server settings:

a.	Zimbra LDAP Server. For Domain to create, identify the same default domain as on the old server.

b.	Zimbra Mailbox Server. An administrator’s account is automatically created.

•	Make sure that the account name for Admin user to create is the same name as on the original server.

•	Set the admin password to be the same as on the old server.

•	Set the LDAP password to be the same as on the old server.

•	Set the Postfix user and Amavis user passwords to be the same as on the old server

•	Change the Spam training user and the Non-spam (HAM) training user account names to be the same as the spam account names on the old server.

•	Global Document Account – This account name is automatically generated and is usually named wiki. If you changed this, change the Global Document Account name to be the same account name as on the original server.

c.	Change any other settings on the new server to match the configuration on the original server.

d.	In the main menu, set the default backup schedule and the automatic starting of servers after the configuration is complete to NO.

Restoring to the new server:

1.	Stop the new server, type zmcontrol stop.

2.	If the old server had additional storage volumes configured, mount any additional volumes now.

3.	Delete the mysql data and re initialize an empty data directory. If you do not do this, zmrestoreoffline will have errors. As zimbra, type

a.	rm -rf /opt/zimbra/db/data/*

b.	/opt/zimbra/libexec/zmmyinit

The mySQL service is now running.

4.	Copy all the files in the /backup directory from the old server or from an archive location to /opt/zimbra/backup.

5.	To restore the LDAP, type zmrestoreldap -lb <latest_label>. If you are restoring large number of accounts, you may want to run a command such as the UNIX command, nohup, so that the session does not terminate before the restore is complete.

Note: To find the LDAP session label to restore, type zmrestoreldap –lbs.

6.	Because some ZCS services are running at this point, type zmconvertctl start. This is required before running zmrestoreoffline.

7.	Sync your LDAP password from backup directory to the new production servers LDAP config. type zmlocalconfig -f -e zimbra_ldap_password=<password>.

8.	To start the offline restore, type zmrestoreoffline -sys -a all -c -br. You may want to run a command such as nohup here also. To watch the progress, tail /opt/zimbra/log/mailbox.log.

Note: Use –c on the command line so that accounts will be restored even if some accounts encounter errors during the offline restore process.

9.	Because some ZCS services are running at this point, type zmcontrol stop to stop all services.

10.	Remove any old backup sessions because these sessions are no longer valid. Type rm -rf /opt/zimbra/redolog/* /opt/zimbra/backup/*

11.	To start ZCS, type zmcontrol start.

12.	Now run a full backup, type zmbackup -f -a all.

13.	Remove the firewall rules and allow client access to the new server.

Restoring from Different Failure Scenarios

The restoration steps are similar for most server failures you may encounter. If a failure occurs, review the disaster recovery section to understand the process and then follow the steps below for the specific type of failure.

Zimbra LDAP server is corrupted

1.	Reinstall the LDAP server. See the Zimbra Installation Guide.

2.	Find the label for the LDAP session to restore. Run the zmrestoreldap -lb <label> command, with no arguments to restore all accounts, domains, servers, COS, etc. for the LDAP server.

3.	Make sure that all accounts are in active mode. From the command line, type zmprov ma zimbraAccountStatus active

Partitions become corrupted

If any partition becomes corrupted, replace the failed disk(s), then run zmrestore, to restore the latest full and incremental backup files. The zmrestore process automatically retrieves the list of all mailboxes on the specified mail host from the backup date and iterates through each mailbox to restore the mailboxes to the last known good state.

Redo log is corrupted or unreadable

If the redo log becomes unreadable for any reason, the mailboxd service stops and cannot restart. If this happens, inspect the hardware and software to find the source of the problem before proceeding.

Without the latest redo log, the Zimbra mailbox server cannot be returned to the most current state. The Zimbra mailbox data can be restored to the latest archived redo log state. A new redo log for current transactions is created after the Zimbra mailbox server is restored.

Important: The mailboxd service must not be running and all accounts must be in maintenance mode before beginning.

1.	To put all accounts into maintenance mode, from the command line, type zmprov md <domain> zimbraDomainStatus maintenance

2.	With the mailboxd service not running, type zmrestoreoffline.

The offline restore process begins by retrieving the list of all mailboxes on the specified mail host from the backup.

The offline restore than iterates through each mailbox to:

•	Delete the mailboxes on the server

•	Restore the last full backup from the backup area

•	Restore all incremental backups for that mailbox in order, since the last full backup. This involves replaying the redo logs from the backup target area

•	Replay all archived redo logs

Since the redo log for current transactions is not available, the mailbox server is returned to the state of the last archived redo log.

3.	Start mailboxd, after the offline restore is complete. From the command line, type zmcontrol startup.

4.	Once the Zimbra mailbox server is up, run a full backup of the Zimbra server. The full backup must be run immediately to have the latest data backed up, as the latest redo log is not available.

Changing Local Configuration Files after Restoring Zimbra

The localconfig.xml file, located in the /opt/zimbra/conf directory, includes the core Zimbra server configuration, such as paths and passwords, This file is backed up in full and incremental backups. When you run an incremental or full restore, the backed-up version of the localconfig.xml is renamed localconfig.xml.restore and is copied to the /opt/zimbra/conf directory.

If you have made changes since the last backup, you may need to replace the localconfig.xml file with the restored copy. Compare these files, and if the .restore file has the latest local configuration data, delete the localconfig.xml file and rename the file with the .restore extension to localconfig.xml.

ZCS Administrator's Guide Network Edition 6.0.8