Full Server Backup and Restore of “open source version”

[mmorse]

Just a quick note,
This technique is possible on 5.0.9+ but there are several reasons to only use it on 5.0.10+:
We found that minor mail store corruptions like a missing blob, size mismatch, or 0 byte db entries can cause a tar operation to halt prematurely/you’ll notice missing data.

You can determine if that is the problem by running “tar tfz account.tgz” and verifying that the listing completes without a corrupt stream error message.

This is similar to bug 31937 (another error handling one was bug 31016) - a fix for truncating or padding mismatched blobs will be included in 5.0.10 NE & we currently plan to release an updated 5.0.10 FOSS as well.

ZCS-to-ZCS Migrations » Zimbra :: Blog

[SGIA]

Thanks for that mmorse.

So if I understand correctly, this functionality can be presumed stable in a future release of FOSS 5.0.10? Do you know when?

If not very soon we'll have to find a different way. I can run the tar validation test you mention here after the export is complete. Perhaps there is a way to script it and log any corruption or something.

I have had to restore mailboxes on several occasions and never ran into any trouble, some in the 600-800MB range.

Thanks again

[mmorse]

Brought it up because I noticed 5.0.9_GA_2533.SLES10_64_20080815132730 FOSS in your profile.

Timeline:
-Tar formatter was added to 5.0.9
-We released 5.0.10 FOSS
-I wrote that blog using a build between 5.0.9 > 5.0.10 experiencing no issues.
-A few people discovered that it's very iffy in 5.0.9 with some bugs (that had already been addressed in 5.0.10 FOSS).
-An additional fix was added to 5.0.10 branch.

The upcoming 5.0.10 NE release next week will contain the above corrections & we also plan to release a new 5.0.10 FOSS at that time, as several NE fixes had open source code changes.

(Personally I have yet to hit trouble with the tar formatter on the 5.0.10 FOSS binaries currently available for download - if your store and database is in good shape it works quite well.)

We also use tar for sync items in a few places now (or are in the process of replacing zip) so it'll be much refined in future releases/we will obviously stay on top of any issues with that format.

[bfillmore75]

Originally Posted by Artturi

I wondered, has any database guru ever tried to extract the relevant information from the database for some user's mailbox ?

I've played a bit with the idea consisting of extracting database info for a limited set of mailbox ids but so far, done nothing.

Artturi

I think it would be much easier to restore your Zimbra to a temporary server (I use VMWare for all mine, which would make it very easy to do) and move the specific user's deleted emails with an IMAP client from the temporary server you did the restore on to the production server.

[mmarodin]

This is a per-user full backup script (based on mubley's code, 2005), for Zimbra 5.x. Zimbra DB and LDAP are also included.
I've tested it on 5.0.12 version.

I don't want to use ...

/opt/zimbra/bin/zmmailbox -z -m user@domain.com getRestURL “//?fmt=tgz” > /tmp/account.tgz

... because with this backup job I could not restore single messages, but only full mailbox.

I hope it could be usefull for you!

#!/bin/sh
#--------

# Backup Zimbra Open Source 5.x
# by mm@rodin - 20090206

# Modify the following variables to suit your installation
export time=`date +%y%m%d`
export backup_dir=/opt/backup
export zimbra_dir=/opt/zimbra
export index_dir=$zimbra_dir/index
export store_dir=$zimbra_dir/store

# We need mysqldump.  If it doesn't exist,
# copy mysql and modify it to call mysqldump.
if [ ! -e $zimbra_dir/bin/mysqldump ]; then
   cp $zimbra_dir/bin/mysql $zimbra_dir/bin/mysqldump
   sed -i 's|/bin/mysql |/bin/mysqldump |g' $zimbra_dir/bin/mysqldump
fi

# Create the backup directory
mkdir -p $backup_dir

# Get a list of all accounts and execute
# the contents of the for-loop for each account.
for i in `$zimbra_dir/bin/zmprov gaa` ; do

# Beginning of for-loop

#  Use email address to get zimbra_id
   zimbraId=`$zimbra_dir/bin/zmprov ga $i |grep zimbraId |sed 's|zimbraId: ||g'`
   set $zimbraId

#  Use mysql to get number id
   zimbra_nr=`$zimbra_dir/bin/mysql -e "use zimbra; select id from mailbox where account_id = '$1'"`
   zimbra_nr=`echo $zimbra_nr | sed 's|id ||g'`

#  Check to make sure we got a valid number id
   if [ "$zimbra_nr" = "" ]; then
      echo $i has no mailbox.
   else

#     Set zimbraAccountStatus to "maintenance"
      $zimbra_dir/bin/zmprov ma $i zimbraAccountStatus maintenance

#     Marking user backup
      if [ ! -e $backup_dir/info-$zimbra_nr.txt ]; then
         echo "Zimbra ID:	$1" > $backup_dir/info-$zimbra_nr.txt
         echo "Mail address:	$i" >> $backup_dir/info-$zimbra_nr.txt
      fi

#     Backup index and store files
      tar -czf $backup_dir/$time-$zimbra_nr-data.tgz $zimbra_dir/index/0/$zimbra_nr $zimbra_dir/store/0/$zimbra_nr

#     Backup Mysql db
      $zimbra_dir/bin/mysqldump mboxgroup$zimbra_nr > $backup_dir/$time-$zimbra_nr.dump
      tar -czf $backup_dir/$time-$zimbra_nr.dump.tgz $backup_dir/$time-$zimbra_nr.dump
      rm -f $backup_dir/$time-$zimbra_nr.dump

#     Set zimbraAccountStatus to "active"
      $zimbra_dir/bin/zmprov ma $i zimbraAccountStatus active

#  End of if statement
   fi

# End of for-loop
done

# Backup Zimbra master database and LDAP accounts database
# First, set all accounts to maintenance.
for i in `$zimbra_dir/bin/zmprov gaa` ; do
   $zimbra_dir/bin/zmprov ma $i zimbraAccountStatus maintenance
done

$zimbra_dir/bin/mysqldump zimbra  > $backup_dir/$time-zimbra.dump
tar -czf $backup_dir/$time-zimbra.dump.tgz $backup_dir/$time-zimbra.dump
rm -f $backup_dir/$time-zimbra.dump

$zimbra_dir/bin/zmslapcat > $backup_dir/$time-zimbra.ldap
tar -czf $backup_dir/$time-zimbra.ldap.tgz $backup_dir/$time-zimbra.ldap
rm -f $backup_dir/$time-zimbra.ldap

# Setting all accounts to active mode.
for i in `$zimbra_dir/bin/zmprov gaa` ; do
   $zimbra_dir/bin/zmprov ma $i zimbraAccountStatus active
done

# Backup complete

[bonobo]

I am new to zimbra and I was wondering how Can i perform a single user backup and restoration. I used your script which ran with the following error:
mysqldump: Got error: 2002: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) when trying to connect
tar: Removing leading `/' from member names
however it created the backup folder and I was able to see the backup emails. how do you restore a single user emails from this backup? or restore selected emails back to the user inbox?

thank you

[weigenmann]

The script is giving me error as below:

./zimbrabackup.sh: line 79: /opt/zimbra/bin/zmslapcat: No such file or directory

For some unknown reason my installation does not have zmslapcat.

Regards

8 April 2009
Changing the code line
from
$zimbra_dir/bin/zmslapcat > $backup_dir/$time-zimbra.ldap

to
$zimbra_dir/libexec/zmslapcat > $backup_dir/$time-zimbra.ldap

The script works now.

[mayk.backus]

zmslapcat is located in /opt/zimbra/libexec/zmslapcat on my box.. Perhaps check your install if it you can find it there..

hth

Mayk

[sofox]

I like the LVM solution, if you can afford one or two minutes server downtime

[weigenmann]

I ran the per-user backup script and it seems to to run fine.

I checked the content of 080409-zimbra.ldap.tgz and found the following:

USAGE: zmslapcat <DIR>

That does not seem to be right does it.

Regards,
Willi Eigenmann