Zimbra offers Open Source email server software and shared calendar for Linux and the Mac
Go Back   Zimbra :: Forums > Zimbra Collaboration Suite > Migration

Welcome to the Zimbra :: Forums!
Welcome, if you would like to post a comment please register. We also encourage you to explore all things Zimbra with our team and members of the community.

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #31 (permalink)  
Old 12-23-2005, 11:15 AM
Active Member
 
Posts: 44
Default Hot Backup using LVM Snapshots

DESCRIPTION:
If you have /opt/zimbra/ on an LVM logical volume, then you can use the LVM snapshot functionality to perform a hot backup.

DISCLAIMERS:
-This script has the potential of doing great damage--DO NOT TEST IT ON A PRODUCTION SYSTEM!
-This script does no error-checking whatsoever.
-I give no guarantees that this script will work for you and cannot be held liable for any damages that may occur.
-Use at your own risk!
-If your Zimbra server breaks, you get to keep both pieces.
-This script has NOT been tested in a production environment.
-This script has not had much testing at all.
-I have not yet verified that I can perform a successful restore from the backups that are produced by this script.
-The restore script is left as an exercise to the reader.

ASSUMPTIONS:
-Fedora Core 4
-/opt/zimbra/ is on its own logical volume.
-The volume group that contains the logical volume for /opt/zimbra has free space that can be used to create a new logical volume for the snapshot.
-The script creates a full backup called zimbra.backup.tar.gz in a directory that is named with the date and time that the script was started at.

SCRIPT:
Code:
#!/bin/bash

time=`date +%Y-%m-%d_%H-%M-%S`

#########################################
# Modify the following variables according to your installation
#########################################

# backup_dir - directory to backup to
backup_dir=/media/usbdisk/zimbra/$time

# zimbra_vol - the Logical Volume that contains /opt/zimbra
zimbra_vol=LogVol02

# vol_group - the Volume Group that contains $zimbra_vol
vol_group=VolGroup00

#########################################
# Do not change anything beyond this point
#########################################

# Output date
echo Backup started at:
date

# Create a logical volume called ZimbraBackup
lvcreate -L1000M -s -n ZimbraBackup /dev/$vol_group/$zimbra_vol

# Create a mountpoint to mount the logical volume to
mkdir -p /mnt/ZimbraBackup

# Mount the logical volume to the mountpoint
mount /dev/$vol_group/ZimbraBackup /mnt/ZimbraBackup/

# For testing only
#echo Press Enter to continue . . .
#read input

# Create the current backup
mkdir -p $backup_dir
tar zcvf $backup_dir/zimbra.backup.tar.gz /mnt/ZimbraBackup/zimbra/ > /dev/null

# Unmount /mnt/ZimbraBackup and remove the logical volume
umount /mnt/ZimbraBackup/
echo y | lvremove /dev/$vol_group/ZimbraBackup

# Output date
echo Backup ended at:
date
Reply With Quote
  #32 (permalink)  
Old 12-23-2005, 03:47 PM
Advanced Member
 
Posts: 201
Default YEs

Quote:
Originally Posted by mubley
rmvg, have you been able to get the new script working?
It seems to be working much better now down to 3.3g which is about right from my users just crazy busy with X-mas retail and all that sorry not to reply.
__________________
Computer King

http://www.computerking.ca

Sales, Service, and Hosting
Email, Data, and Web Packages
Ask about web design specials

Affiliates
http://www.computerking.ca/pages/lin...affiliates.htm
Reply With Quote
  #33 (permalink)  
Old 01-14-2006, 07:38 PM
Advanced Member
 
Posts: 201
Default restore??

Does anyone have a restore script they would like to be tested. I am installing on a fresh server with the new zimbra BETA. Would like to see if i can do a full restore of my old server backups using mubleys script to the new BETA server.
__________________
Computer King

http://www.computerking.ca

Sales, Service, and Hosting
Email, Data, and Web Packages
Ask about web design specials

Affiliates
http://www.computerking.ca/pages/lin...affiliates.htm
Reply With Quote
  #34 (permalink)  
Old 01-16-2006, 03:10 PM
Intermediate Member
 
Posts: 17
Default

I've been thinking about this since I started reading up on zimbra. I am about to begin working on the backup process but am taking a different tack:

I'm using software RAID 1 on FC4 (mdadm).

My plan is to drop a mirrored drive from my array, mount it as a backup point into the filesystem, tar it off to tape, umount the drive from the backup mount point, add it back to my array and rebuild it.

I may have to do a brief zmcontrol stop before I drop the drive from my array but I don't think it'll be necessary. I think that when I restore the tape and bring up the recovored data set, it'll just look like a dirty shutdown and replay the transaction logs. I'm speaking from only postgresql experience though. Should be no way of losing conistency though. Won't be a good solution for those running hardware RAID though for obvious reasons.

I hope I'm not missing something; seems too easy.
Reply With Quote
  #35 (permalink)  
Old 01-20-2006, 04:16 PM
Starter Member
 
Posts: 2
Default

Quote:
Originally Posted by mubley
DESCRIPTION:
If you have /opt/zimbra/ on an LVM logical volume, then you can use the LVM snapshot functionality to perform a hot backup.
Please note that this method only gives you a crash-consistent backup since the LVM snapshot freezes time. While ext3 will be in a consistent state, there's no guarantee that ZCS will have written a complete message to disk, updated metadata in the database, etc.

A better way of doing this would be to stop the zimbra services, perform the snapshot, restart the zimbra services, and then do a backup from the snapshot.

This will give you the most consistent data to restore and requires a minimum of downtime.
Reply With Quote
  #36 (permalink)  
Old 01-21-2006, 06:30 AM
Active Member
 
Posts: 44
Default Backup (with minimal Zimbra downtime) using LVM Snapshots

Quote:
Originally Posted by ChadScott
Please note that this method only gives you a crash-consistent backup since the LVM snapshot freezes time. While ext3 will be in a consistent state, there's no guarantee that ZCS will have written a complete message to disk, updated metadata in the database, etc.

A better way of doing this would be to stop the zimbra services, perform the snapshot, restart the zimbra services, and then do a backup from the snapshot.

This will give you the most consistent data to restore and requires a minimum of downtime.
Good point, ChadScott. Thanks for the feedback. The updated script is below.

DISCLAIMERS:
-This script has the potential of doing great damage--DO NOT TEST IT ON A PRODUCTION SYSTEM!
-This script does no error-checking whatsoever.
-I give no guarantees that this script will work for you and cannot be held liable for any damages that may occur.
-Use at your own risk!
-If your Zimbra server breaks, you get to keep both pieces.
-This script has NOT been tested in a production environment.
-This script has not had much testing at all.
-I have not yet verified that I can perform a successful restore from the backups that are produced by this script.
-The restore script is left as an exercise to the reader.
-This new script is no longer a hot-backup; however, as ChadScott points out, the total Zimbra downtime should be minimal.

ASSUMPTIONS:
-Fedora Core 4
-/opt/zimbra/ is on its own logical volume.
-The volume group that contains the logical volume for /opt/zimbra has free space that can be used to create a new logical volume for the snapshot.
-The script creates a full backup called zimbra.backup.tar.gz in a directory that is named with the date and time that the script was started at.

Code:
#!/bin/bash

time=`date +%Y-%m-%d_%H-%M-%S`

#########################################
# Modify the following variables according to your installation
#########################################

# backup_dir - directory to backup to
backup_dir=/media/usbdisk/zimbra/$time

# zimbra_vol - the Logical Volume that contains /opt/zimbra
zimbra_vol=LogVol02

# vol_group - the Volume Group that contains $zimbra_vol
vol_group=VolGroup00

#########################################
# Do not change anything beyond this point
#########################################

# Output date
echo Backup started at:
date

# Stop the Zimbra services
service zimbra stop

# Create a logical volume called ZimbraBackup
lvcreate -L1000M -s -n ZimbraBackup /dev/$vol_group/$zimbra_vol

# Create a mountpoint to mount the logical volume to
mkdir -p /mnt/ZimbraBackup

# Mount the logical volume to the mountpoint
mount /dev/$vol_group/ZimbraBackup /mnt/ZimbraBackup/

# Start the Zimbra services
service zimbra start

# For testing only
#echo Press Enter to continue . . .
#read input

# Create the current backup
mkdir -p $backup_dir
tar zcvf $backup_dir/zimbra.backup.tar.gz /mnt/ZimbraBackup/zimbra/ > /dev/null

# Unmount /mnt/ZimbraBackup and remove the logical volume
umount /mnt/ZimbraBackup/
echo y | lvremove /dev/$vol_group/ZimbraBackup

# Output date
echo Backup ended at:
date
Reply With Quote
  #37 (permalink)  
Old 01-31-2006, 03:18 PM
Active Member
 
Posts: 33
Default Journaling - redolog?

So doesn't the production "Network" version of Zimbra use imap event journaling to handle replication to a "hot" backup server?

I seem to remember that from one of the architecture docs.

If so I'm assuming that there is some sort of other services running in the background replicating each event to the designated hot backup server.

If this is at all on track it would seem that this could be duplicated with the OSS so long as the journal is kept in an accessible DB.

It would seem to me that it is a matter of:
- check journal for event
- copy message from filesystem spool by message-id to hot backup server
- run the same imap event on the remote server.

I could be way off base. I'm very curious about this though as I had hoped to develop a java based imap proxy with journaling for generic imap replication recently. Something that I could stick in front of Cyrus. Zimbra seems to be the solution for this though if we can figure out the replication.
Reply With Quote
  #38 (permalink)  
Old 01-31-2006, 03:26 PM
Active Member
 
Posts: 33
Default

Quote:
Originally Posted by kbaker
So doesn't the production "Network" version of Zimbra use imap event journaling to handle replication to a "hot" backup server?

I seem to remember that from one of the architecture docs.
Ok, found it in the "Zimbra Architecture Overview.pdf" in documentation section.

Quote:
Originally Posted by Zimbra Architecture Overview.pdf
Journaling: ...
Zimbra’s journaling subsystem is also the foundation for hot backup, sma
(individual mailbox, point-in-time, etc.), and replication (such as for site
disaster recovery) on commodity hardware (more on replication below.)
Quote:
Originally Posted by Page 11
Zimbra’s replication architecture is master/slave, and is based on the same Zimbra journaling subsystem used for fault tolerance, back up, and recovery. The master is responsible for journaling operation records both to the primary and spooling those same records to the secondary. An efficient binary protocol is used to steam log data from primary to secondary. Records must be committed on both primary and secondary before the primary confirms operations/receipt of email to ensure no loss of data in the event of a catastrophic failure. The secondary Zimbra server is then responsible for asynchronously “replaying” its local copy of the journal to perform all successful updates on its local databases, and as such will tend to run a
few seconds behind the primary (but, of course, with no loss of data).
So I guess the hope would be to explore this features if it is included in the OSS edition, seems likely as it is used for other services, and use it to help with the syncing.

Last edited by kbaker; 01-31-2006 at 03:30 PM..
Reply With Quote
  #39 (permalink)  
Old 02-16-2006, 09:10 AM
Active Member
 
Posts: 43
Default

I have the GA release installed on a Tiger 10.4.5 system, but two problems:

--------------------------------------------------------
admin@zimbra.mapolce.com
--------------------------------------------------------
zimbraAccountStatus: maintenance
0 /opt/zimbra/doc/hot_backup/2006-02-16_10-54-56/admin@zimbra.mapolce.com/index//index
0 /opt/zimbra/doc/hot_backup/2006-02-16_10-54-56/admin@zimbra.mapolce.com/index/
cp: /opt/zimbra/store/0/1/*: No such file or directory
0 /opt/zimbra/doc/hot_backup/2006-02-16_10-54-56/admin@zimbra.mapolce.com/store/
mysqldump: Got error: 2002: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) when trying to connect
-rw-r--r-- 1 zimbra zimbra 0 Feb 16 10:55 /opt/zimbra/doc/hot_backup/2006-02-16_10-54-56/admin@zimbra.mapolce.com/mailbox1.sql
zimbraAccountStatus: active

I also get prompted for a password when the LDAP backup portion starts:

--------------------------------------------------------
Backing up LDAP accounts database
--------------------------------------------------------
Password:
Reply With Quote
  #40 (permalink)  
Old 02-16-2006, 09:26 AM
Zimbra Employee
 
Posts: 2,103
Default

Where did that output come from?

Why is it trying to go through /tmp/mysql.sock? That seems wrong - like the zimbra environment variables are getting munged.

What happens when you run "zmslapcat" as the zimbra user?
Reply With Quote
Reply


Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes


Similar Threads

Why Join?

Registering let's you ask questions, makes it easier to search, displays any files attached to posts, and notifies you about replies.

blog.zimbra.com




 

SEO by vBSEO ©2011, Crawlability, Inc.