Per User Backup for FOSS edition

[patcsy88]

Would like to help here as well. Will be doing the test on our 5.0.2 on Dapper 6.06. Have about 70 users and more being added over the next few weeks. Currently, I am using rdiff-backup for dailies and full over the weekends but no per user backup and restore capabilites....this would do it. Will let you know...

[fdsadmin]

Hi,

Sorry there has been no updates recently, I am still working on this little project but other work commitments have taken priority plus I have been upgrading from Ubuntu Gutsy to Hardy.

I will fix the rsync issue for unused accounts and I am also aware that restoring a deleted account has some special considerations.

[m3skalina]

Hi All! Thank you for your work, it's great!

I have a zimbra 4.5.7 on a Debian Etch and i have a problem whit the script "zimbra_backup_user.sh":

- at the line 48 :

current_account_zimbraId=`zimbra_run "${zimbra_home}/bin/zmprov ga ${current_account} zimbraId | grep zimbraId | cut -d' ' -f2"`

Whit this version of Zimbra, the zmprov ga don't accept the "zimbraId" parameter. I have modified the script in this:

- at line 48 :

current_account_zimbraId=`zimbra_run "${zimbra_home}/bin/zmprov ga ${current_account} |  grep zimbraId | cut -d' ' -f2 | sed -n 1p"`

I am posting this in the case could serve to someone.

Sorry for my english, but i am italian.

[jalakas]

Hi!

First of all, i like the script a lot I't looks like it does what's needed. However; has anyone tried to restore an account backed up by this script (and how did you do it - i have an idea but if someone already tried that would be nice)?

Second, has the project moved to a wiki-page or did you just run out of time? If it has moved somewhere else, a link would be nice for newbs like me

Again, thanks for the effort.

[prasenjitbehera]

Hi Martin

I just tested your script in my zimbra mail server, congratulations everything was fine.but One thing I notice is that "rsync" gives an error if the user never logged in.as per your post u have been solved that problem so can u let me know where and what i have to chage in that script so that it will not through the error.hope
u r working on restore part also it will be help full for every one who they are using open source version.i am waiting eagerly for your restore part.

thanx a lot again for your good effort.

regard's
Prasenjit

[phpasp]

when use the script backup the user mails finished , how to restore the mail ?
thx

[phoenix]

Originally Posted by phpasp

when use the script backup the user mails finished , how to restore the mail ?
thx

Did you see the following in the original post?

Originally Posted by fdsadmin

The Script

v0.1 - 11th April 2008

• Initial release.
• Scripts to backup individual user or all users.
  
• No restore features as yet.
• No real documentation.

[keith@keithnoddle.org]

Firstly, I've only been playing with Zimbra since Friday last, so please be gentle...

I read with interest this per user backup thread and all credit to the author - good stuff!

But backups are no good without a matching restore, so I've set to and created one based upon reversing the actions the backup script takes (and un-doing a few things that don't exactly reverse!). Now, caveat emptor is the rule here because I've only undertaken limited testing against a single account on a test server. So far so good, but I'd really appreciate bigger brains that mine taking a look at the script below and telling me what I've missed.

There are a couple of gotchas which I've tried to comment but I'm sure there are others.

Hope this is of some interest.

Keith.

#!/bin/bash

#===========================================================================================
# Zimbra Restore User - based on reversing the actions of zimbra_backup_user.sh
#
# Basic Testing ONLY undertaken.
#
#    ************************
#    * USE AT YOUR OWN RISK *
#    ************************
#
#===========================================================================================

###
# Requirements : bash, whoami, su, echo, tee, cut, grep, mkdir, chown, rsync, du, zimbra 5.0.x
#
###

# Are we root?
if [ `whoami` != "root" ]; then
        echo "ERROR! ${0} must be run by the root user"
        exit 1
fi

# Output help, needs work.
function display_usage {
	echo "${0} [emailaddress] [backupDirToRestore]"
	echo ""
	echo " [emailaddress] : A valid email account on your Zimbra server."
	echo " [backupDirToRestore] : A valid directory from which to restore"
	exit 1	
}

#===========================================================================================
# New functions added to facilitate restore.
#
# Included here to avoid changes to zimbra_functions.sh - even
# though that is the logical place for them.
#===========================================================================================
# rsync from cache
# $1 [string] : file/directory path
function zimbra_restore_rsync {
	if [ ! -e ${1} ]; then
		log "WARNING! I can't rsync ${1} because it does not exist."
	else
		local DIRS=`ls -1 ${zimbra_cache}/${current_account}${1}`
		for DIR in ${DIRS}
		do
			# Skip the incoming directory
			if [ ${DIR} != "incoming" ]; then
				log "Syncing ${1}/${DIR}/${mailbox_id} from ${zimbra_cache}/${current_account}${1}/${DIR}"
				rsync -aHK ${zimbra_cache}/${current_account}${1}/${DIR} ${1}
				log "Size : `du -s -h ${zimbra_cache}/${current_account}${1}/${DIR} | cut -f1`"
			fi
		done
	fi
}

# Executes mysql restore
# $1 [string] : mysql paramters
function zimbra_sql {
	log "Restoring ${1}"
	zimbra_run "${mysql_directory}/bin/mysql -f -S ${mysql_socket} -u ${zimbra_mysql_user} --password=${zimbra_mysql_password} ${1}"
}
#===========================================================================================



# Have we got enough parameters
if [ $# -ne 2 ]; then
	display_usage
else
	current_account="${1}"
	current_backup="${2}"
fi

# Setup the zimbra variables and functions
source ./zimbra_functions.sh || exit 1

if [ ! -e ${current_backup} ]; then
	display_usage
else
	zimbra_chown ${current_backup}
fi

# Line breaking please.
IFS=$'\n'

log "Processing ${current_account}"
# Get the zimbraId
current_account_zimbraId=`zimbra_run "${zimbra_home}/bin/zmprov ga ${current_account} zimbraId | grep zimbraId | cut -d' ' -f2"`
restoreZimbraID=`grep zimbraId: ${current_backup}/ga.txt | cut -d' ' -f2`

# Don't try to restore a deleted (or otherwise damaged) account
if [ "${current_account_zimbraId}" != "${restoreZimbraID}" ]; then
	echo "***************************************************************"
	echo "* There seems to be a problem with account ${current_account} *"
	echo "* Either the account doesn't exist or the backup and          *"
	echo "* current zimbraIds don't match                               *"
	echo "*                                                             *"
	echo "*             Restore operation cancelled.                    *"
	echo "***************************************************************"
	exit 1
fi

# Ensure cache exists and cleared for current account. Savage, but it works for me.
rm -rf ${zimbra_cache}/${current_account}
mkdir -p ${zimbra_cache}/${current_account}

# Get the quotaUsed - Could be used to (optionally?) skip accounts using zero quota in the future
current_account_quotaUsed=`zimbra_run "${zimbra_home}/bin/zmprov gmi ${current_account} | grep quotaUsed | cut -d' ' -f2"`
# Get the mailboxId
mailbox_id=`zimbra_run "${zimbra_home}/bin/zmprov gmi ${current_account} | grep mailboxId | cut -d' ' -f2"`

# Get the group and volume ids for the current account.
log "Getting group_id and index_volume_id"
zimbra_create_sql "SELECT group_id, index_volume_id FROM zimbra.mailbox WHERE account_id='"${current_account_zimbraId}"'"
zimbra_execute_sql
mailbox_group_id=`echo ${zimbra_sql_result} | cut -f1`
mailbox_index_volume_id=`echo ${zimbra_sql_result} | cut -f2`
mailbox="mboxgroup${mailbox_group_id}"
log "zimbraId   : ${current_account_zimbraId}"
log "Id         : ${mailbox_id}"
log "Group      : ${mailbox_group_id}"
log "Mailbox    : ${mailbox}"
log "Quota Used : ${current_account_quotaUsed}"

# Get the volume details for the current account.
log "Getting message_volume_id and index_volume_id"
zimbra_create_sql "SELECT message_volume_id, index_volume_id FROM zimbra.current_volumes WHERE index_volume_id='"${mailbox_index_volume_id}"'"
zimbra_execute_sql
message_volume_id=`echo ${zimbra_sql_result} | cut -f1`
index_volume_id=`echo ${zimbra_sql_result} | cut -f2`

# Get the index volume name and path
log "Getting index volume path"
zimbra_create_sql "SELECT name, path FROM zimbra.volume WHERE id='"${index_volume_id}"'"
zimbra_execute_sql
index_volume_name=`echo ${zimbra_sql_result} | cut -f1`
index_volume_path=`echo ${zimbra_sql_result} | cut -f2`

# Get the message volume name and path
log "Getting message volume path"
zimbra_create_sql "SELECT name, path FROM zimbra.volume WHERE id='"${message_volume_id}"'"
zimbra_execute_sql
message_volume_name=`echo ${zimbra_sql_result} | cut -f1`
message_volume_path=`echo ${zimbra_sql_result} | cut -f2`
log "${index_volume_name} path   : ${index_volume_path}"
log "${message_volume_name} path : ${message_volume_path}"

#===========================================================================================
# No hot restore - not sure what to do with ga.txt and gsig.txt...
#===========================================================================================

# Set the account to maintenance mode (Read-only, new mail will be queued at the MTA)
zimbra_set_account_status maintenance

### Warm restore of the current account database, current account rows, index and store directories.
#
log "Restoring index_n_store tarball to ${zimbra_cache}/${current_account}"
tar zxvf ${current_backup}/index_n_store.tar.gz ${zimbra_cache}/${current_account}/ 2&> /dev/null

zimbra_restore_rsync ${index_volume_path}
zimbra_restore_rsync ${message_volume_path}

#===========================================================================================
# Note that the way the mboxgroupN is dumped by zimbra_backup_user.sh means restoring it
# will cause anything held in mboxgroupN that was created after the backup was taken to
# be lost. This could maybe do with some work to merge rather than replace the entries?
#===========================================================================================
zimbra_sql "${mailbox} < ${current_backup}/${mailbox}.sql"
zimbra_sql "zimbra < ${current_backup}/mailbox.sql"
zimbra_sql "zimbra < ${current_backup}/mailbox_metadata.sql"
zimbra_sql "zimbra < ${current_backup}/out_of_office.sql"
zimbra_sql "zimbra < ${current_backup}/scheduled_task.sql"
zimbra_sql "zimbra < ${current_backup}/table_maintenance.sql"

# Tidy up
rm -rf ${zimbra_cache}/${current_account}

#
### Warm restore ends

# Set the account to back to active mode
zimbra_set_account_status active

# Purely cosmetic
echo ""

[Ono-Sendai]

I have been working with the restore script for the past few days and I have a few questions and a couple of comments.

From what I have found the script seems to work somewhat on my test server. I have created a test user and then added various items (mail, contacts, calendar appointments, and tasks). I then performed a backup (which worked without fail) and deleted all of the items I created. I then performed a restore to see what would be restored and what would not.

I found that the mail items were restored with no problem as well as the tasks on the task list, however neither the calendar appointments nor the address book were restored.

After finding this issue I tried to determine if the missing items had in fact make it into the backup and from what I can tell the data in question is in the mboxgroupX.sql file, and upon being run the data will be inserted back into the "mail items" table. It does not appear to be doing that however. The most confusing part of the whole thing is that the other items (mail, task list) are being added back to the table and they are being added in the same insert statement.

While the restore function is running I continue to receive the same error that the /opt/zimbra_cache/{email account}/opt/zimbra/index and /store directories are not there. Initially I thought it might be a rights issue as the owner of those two directories is the zimbra account and not root as all of the other directories are, but after looking deeper into the issue I found that those directories are being completely removed at some point during the restore script running. I have checked for these files existence just prior to running the restore script and they were there, however after running the script they were gone. I'm not even sure the cache folder is required for the restore to work properly. As I understand it the cache folder functions only to allow a two part backup to take place so as to minimize the amount of time a users account will be in maintenance mode. It seems to me that the zimbra_cache folder would not be necessary at all during a restore, so perhaps this has nothing to do with the partial restore problem.

My question is this: Does this script restore contacts and calendar entries or am I expecting it do something is was not designed to do? Am I doing something incorrectly? If it is not designed to restore these items then where should they be restored from/to?

Any input would be great.

Thanks,
Ono-Sendai

[Ono-Sendai]

I have a couple of additional observations to add to my last post on this topic.

I have done some further experimenting with the restoration script and have found a couple items of note to keep in mind when using the script.

I am unsure if anyone else is even trying to use the script, and if they are whether or not they are seeing the same things I am seeing, but I wanted to get my observations out there so a more automated, idiot proof, version of the restore script might be created. I find the crippling of community (unpaid) versions of open source software rather distasteful in any form, and while I fully understand the reasoning behind the practice I do not agree with it. For this reason I would really like to see a much more robust backup/restore solution on a per user basis for the community version of Zimbra. Anyway enough of my rant.

I have found that during the course of performing the restoration procedure something strange happens to the files that are located in the /Trash folder for the given mailbox being restored. By strange I mean the files become unaccessible to either the end user via the web interface or the administrator via the CLI, however the items (mail, contacts, tasks) are still in the file. This can be seen by hovering the mouse over the trash folder in the web interface where the little informational pop-up will tell you how many items are in the folder, or by accessing the user-in-question's mailbox via the CLI where pulling up the folder information indicates that the deleted items are still in the /Trash folder. I have tried various ways of removing these "hidden" items from the /Trash folder but none of them work. Emptying the trash from the web interface does not remove the files, emptying the folder from the CLI also does not work. The folder itself can not be deleted from the CLI because it is a system folder. The only way I was able to clear the folder was to wait for the purge to happen and it was able to clear the files. I'm sure there is a way to perform the purge from the CLI manually on only one mailbox, but I am still reading over the documentation to find out how. I will also add this step to the script once I figure out how to do it.

Once the folder has been purged the contacts restore perfectly using the restore script, however it does require the additional step of purging the trash folder. I have a feeling this may not be an issue in most cases as users will only need items restored that have already been purged from their trash folder, but I think inclusion of this step in the script will be a good thing.

I have also found that the calendar reacts in a very similar way to the Trash folder. The calendar items still exist in some other calendar "trash" folder and a restore of these items causes them to become "invisible". What I mean by this is that the item itself can not be seen on the calendar, however, hovering the mouse over the calendar in question brings up the info pop-up which tells you that items exist on the calendar. You also still get reminder pop-ups that the "invisible" calendar entry appointment is coming up, which is very strange. I have been unsuccessful in correcting this problem as of yet, but most of my efforts were in the recovery of contacts and not appointments.

Also re-indexing the mailbox in question had no effect on any of the problems listed above.

I am using version 5.0.18 on Ubuntu 8.

To both authors of the backup and restore scripts. Great work on these scripts, they are quite functional as is and took quite a bit of work to get to this point. Thanks for your contributions to the community.

Ono-Sendai