Per User Backup for FOSS edition

[fdsadmin]

Hi all,

I have decided to have a go at creating a per user backup solution for the FOSS edition. Here comes a brain dump...

Currently I use the LVM backup technique to create a nightly backup of '/opt/zimbra' as a disaster recovery option. However, more of my users are using webmail and sooner or later someone is going to do something really silly and I will need a way to restore the data for an individual. Restoring the whole server for one users cock up is not really an option

Disclaimer

The scripts attached to this post are experimental and only suitable for testing purposes only on non-production servers. If you use them on a production server and it breaks, you get to keep the pieces!

Scope

Basically I want a hot/warm-ish backup solution which will give me a 24 hourly restore point for individual user accounts. If someone does something silly, then loosing 24 hours of email is fair penalty in my book If I can get reliable per user backup/restore working then I will continue to do my '/opt/zimbra' DR backup just once a week, and user backups every night.

How it works

In a nut shell...

• Dumps the 'zimbra' database.
• Get a list of all users
• Loops through each user
  
  • Figures out what volumes the user data is stored in
  • Makes a "hot" sync of the users index/store
  • Puts the account in "maintenance" mode
  • Dumps the user DB and rows from associated tables in the 'zimbra' DB
  • Re-syncs the users index/store
  • Puts the account in "active" mode
  • Creates a tarball of the synced index/store data
  • Rotates the backups

It's a bash script. I have tried to make sure there is no hard coded paths, accounts, passwords, etc. All but two configuration parameters are taken directly from Zimbra. Everything is logged to the console and log file.

The directory each account rsyncs it index/store to (the cache) is preserved between backups, so subsequent syncs are relatively quick. However, the cache must be local or mounted file system. In addition to the cache, a daily backup directory is created containing a tarball of that days index/store data plus the dumped SQL. Therefore, you need extra disk space. You can do the maths

Not sure how well this will scale, but for my 15 users it should be fine. I am planning to keep the cache on local disk (for speed) and the backup directory on NFS.

Anyone for testing?

I have created a vmware box running a snapshot of my live server. It runs ZCS 5.0.4 FOSS on Debian 4.0 32bit. I have not tested on 4.5.x (yet). Is there anyone out there who might be interested in testing on non-production servers to help iron out the wrinkles?

The Script

v0.1 - 11th April 2008

• Initial release.
• Scripts to backup individual user or all users.
• No restore features as yet.
• No real documentation.

I haven't started on a restore script yet, but getting the basics going shouldn't be to time consuming. I will be working on the restore scripts early next week.

Download the attached tarball and extract on your non-production server. 'zimbra_functions.sh' has a couple of parameters you can configure at the top to choose the cache and backup locations.

To back up a single user run the following on your Zimbra server.

./zimbra_backup_user.sh user1@example.org

To backup all users run to following on your Zimbra server.

./zimbra_backup_all_users.sh

The Logs

Below is example log output to show what it does so far.

Thu Apr 10 12:51:05 EDT 2008 : Dumping zimbra --single-transaction > /opt/zimbra_backup/zimbra.sql
Thu Apr 10 12:51:05 EDT 2008 : Getting accounts
Thu Apr 10 12:51:07 EDT 2008 : Backing up accounts
Thu Apr 10 12:51:07 EDT 2008 : Processing user1@example.org
Thu Apr 10 12:51:11 EDT 2008 : Getting group_id and index_volume_id
Thu Apr 10 12:51:11 EDT 2008 : Id      : 10
Thu Apr 10 12:51:11 EDT 2008 : Group   : 10
Thu Apr 10 12:51:11 EDT 2008 : Mailbox : mboxgroup10
Thu Apr 10 12:51:11 EDT 2008 : Getting message_volume_id and index_volume_id
Thu Apr 10 12:51:11 EDT 2008 : Getting index volume path
Thu Apr 10 12:51:11 EDT 2008 : Getting message volume path
Thu Apr 10 12:51:12 EDT 2008 : index1 path   : /opt/zimbra/index
Thu Apr 10 12:51:12 EDT 2008 : message1 path : /opt/zimbra/store
Thu Apr 10 12:51:12 EDT 2008 : Syncing /opt/zimbra/index/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/index/0
Thu Apr 10 12:51:12 EDT 2008 : Size : 3.4M
Thu Apr 10 12:51:12 EDT 2008 : Syncing /opt/zimbra/store/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/store/0
Thu Apr 10 12:51:12 EDT 2008 : Size : 144K
Thu Apr 10 12:51:16 EDT 2008 : user1@example.org is now in maintenance mode.
Thu Apr 10 12:51:16 EDT 2008 : Dumping mboxgroup10 > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/mboxgroup10.sql
Thu Apr 10 12:51:16 EDT 2008 : Dumping zimbra mailbox --no-create-info --where "id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/mailbox.sql
Thu Apr 10 12:51:16 EDT 2008 : Dumping zimbra mailbox_metadata --no-create-info --where "mailbox_id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/mailbox_metadata.sql
Thu Apr 10 12:51:16 EDT 2008 : Dumping zimbra out_of_office --no-create-info --where "mailbox_id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/out_of_office.sql
Thu Apr 10 12:51:17 EDT 2008 : Dumping zimbra scheduled_task --no-create-info --where "mailbox_id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/scheduled_task.sql
Thu Apr 10 12:51:17 EDT 2008 : Dumping zimbra table_maintenance --no-create-info --where "database_name='mboxgroup10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/table_maintenance.sql
Thu Apr 10 12:51:17 EDT 2008 : Syncing /opt/zimbra/index/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/index/0
Thu Apr 10 12:51:17 EDT 2008 : Size : 3.4M
Thu Apr 10 12:51:17 EDT 2008 : Syncing /opt/zimbra/store/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/store/0
Thu Apr 10 12:51:17 EDT 2008 : Size : 144K
Thu Apr 10 12:51:21 EDT 2008 : user1@example.org is now in active mode.
Thu Apr 10 12:51:21 EDT 2008 : Creating index_n_store tarball from /opt/zimbra_cache/user1@example.org
Thu Apr 10 12:51:26 EDT 2008 : Rotating backups
Thu Apr 10 12:51:26 EDT 2008 : Keeping 2008-04-10-1251
Thu Apr 10 12:51:26 EDT 2008 : Keeping 2008-04-10-1233
Thu Apr 10 12:51:26 EDT 2008 : Keeping 2008-04-10-1209
Thu Apr 10 12:51:26 EDT 2008 : Deleting 2008-04-10-1149

Thu Apr 10 12:51:26 EDT 2008 : Processing user2@example.org
Thu Apr 10 12:51:29 EDT 2008 : Getting group_id and index_volume_id
Thu Apr 10 12:51:29 EDT 2008 : Id      : 8
Thu Apr 10 12:51:29 EDT 2008 : Group   : 8
Thu Apr 10 12:51:29 EDT 2008 : Mailbox : mboxgroup8
Thu Apr 10 12:51:29 EDT 2008 : Getting message_volume_id and index_volume_id
Thu Apr 10 12:51:29 EDT 2008 : Getting index volume path
Thu Apr 10 12:51:29 EDT 2008 : Getting message volume path
Thu Apr 10 12:51:29 EDT 2008 : index1 path   : /opt/zimbra/index
Thu Apr 10 12:51:29 EDT 2008 : message1 path : /opt/zimbra/store
Thu Apr 10 12:51:29 EDT 2008 : Syncing /opt/zimbra/index/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/index/0
Thu Apr 10 12:51:29 EDT 2008 : Size : 104M
Thu Apr 10 12:51:29 EDT 2008 : Syncing /opt/zimbra/store/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/store/0
Thu Apr 10 12:51:30 EDT 2008 : Size : 231M
Thu Apr 10 12:51:33 EDT 2008 : user2@example.org is now in maintenance mode.
Thu Apr 10 12:51:33 EDT 2008 : Dumping mboxgroup8 > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/mboxgroup8.sql
Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra mailbox --no-create-info --where "id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/mailbox.sql
Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra mailbox_metadata --no-create-info --where "mailbox_id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/mailbox_metadata.sql
Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra out_of_office --no-create-info --where "mailbox_id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/out_of_office.sql
Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra scheduled_task --no-create-info --where "mailbox_id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/scheduled_task.sql
Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra table_maintenance --no-create-info --where "database_name='mboxgroup8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/table_maintenance.sql
Thu Apr 10 12:51:33 EDT 2008 : Syncing /opt/zimbra/index/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/index/0
Thu Apr 10 12:51:33 EDT 2008 : Size : 104M
Thu Apr 10 12:51:33 EDT 2008 : Syncing /opt/zimbra/store/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/store/0
Thu Apr 10 12:51:34 EDT 2008 : Size : 231M
Thu Apr 10 12:51:37 EDT 2008 : user2@example.org is now in active mode.
Thu Apr 10 12:51:37 EDT 2008 : Creating index_n_store tarball from /opt/zimbra_cache/user2@example.org
Thu Apr 10 12:59:08 EDT 2008 : Rotating backups
Thu Apr 10 12:59:09 EDT 2008 : Keeping 2008-04-10-1251
Thu Apr 10 12:59:09 EDT 2008 : Keeping 2008-04-10-1233
Thu Apr 10 12:59:09 EDT 2008 : Keeping 2008-04-10-1210
Thu Apr 10 12:59:09 EDT 2008 : Deleting 2008-04-10-1150
Thu Apr 10 12:59:10 EDT 2008 : Deleting 2008-04-10-1148

The Cache

Example cache directory.

user1@example.org/
`-- opt
    `-- zimbra
        |-- index
        |   `-- 0
        |       `-- 10
        |           `-- index
        |               `-- 0
        |                   |-- _6ip.cfs
        |                   |-- _6ip.del
        |                   |-- _8on.cfs
        |                   |-- _8on.del
        |                   |-- _9em.cfs
        |                   |-- _9em.del
        |                   |-- _gkx.cfs
        |                   |-- _gkx.del
        |                   |-- _gyu.cfs
        |                   |-- _gyu.del
        |                   |-- _h38.cfs
        |                   |-- _h3d.cfs
        |                   |-- deletable
        |                   `-- segments
        `-- store
            `-- 0
                `-- 10
                    `-- msg
                        |-- 0
                        |-- 1
                        `-- 2
                            |-- 10023-44804.msg
                            |-- 10062-45306.msg
                            |-- 10063-45307.msg
                            |-- 10064-45308.msg
                            |-- 10082-45404.msg
                            |-- 10083-45406.msg
                            |-- 10185-46210.msg
                            |-- 10191-46216.msg
                            |-- 10201-46401.msg
                            |-- 10203-46405.msg
                            |-- 10267-46714.msg
                            |-- 10280-46800.msg
                            |-- 10300-47000.msg
                            |-- 10301-47001.msg
                            |-- 8406-37009.msg
                            |-- 9865-44013.msg
                            |-- 9920-44304.msg
                            |-- 9951-44425.msg
                            `-- 9964-44510.msg

The Backups

Example backup directory.

user1@example.org/
|-- 2008-04-10-1209
|   |-- index_n_store.tar.bz2
|   |-- mailbox.sql
|   |-- mailbox_metadata.sql
|   |-- mboxgroup10.sql
|   |-- out_of_office.sql
|   |-- scheduled_task.sql
|   `-- table_maintenance.sql
|-- 2008-04-10-1233
|   |-- calendar.ics
|   |-- contacts.csv
|   |-- mailbox.sql
|   |-- mailbox_metadata.sql
|   |-- mboxgroup10.sql
|   |-- out_of_office.sql
|   |-- scheduled_task.sql
|   `-- table_maintenance.sql
`-- 2008-04-10-1251
    |-- index_n_store.tar.bz2
    |-- mailbox.sql
    |-- mailbox_metadata.sql
    |-- mboxgroup10.sql
    |-- out_of_office.sql
    |-- scheduled_task.sql
    `-- table_maintenance.sql

Thoughts, comments, suggestions?

Regards, Martin.

[bonadio]

Hi Martin

Looks promising, I would be happy to do some testing, I have a semi-production server with a few users running UBUNTU FOSS where
I can run some tests.

Let me know

good luck

[]s

[fdsadmin]

See the first post, I have attached v0.1 of the scripts and added some basic usage instructions.

[2l2n]

I'm really not an expert, but I launched the script and everything looks good.

Tested on Ubuntu 6.06.2 - Zimbra 5.0.4 FOSS.
I only have 12 users and a few mails. I'm going to import my 800Mo .pst on Zimbra.

To test it more seriously, can you give us a method for restoring 1 account ?

[Dirk]

Looks very interesting, once you have enough details & scripts together, consider adding the info to the wiki where it can be maintained easily.

I'll have a good look through the sourcecode shortly, but it's always good to see this kind of effort

[fdsadmin]

2l2n, I will post an updated script in the new few days which includes restore capability.

Dirk, I will be more than happy to maintain a wiki page for the script once I have got the basic backup/restore stuff working :-) For now, I will keep the first post in the topic up to date.

[bonadio]

Hi Martin

I just tested your script in my semi-production server, congratulations everything was fine:

I have 25 users with 2Gb of messages total, it took 18 min to do a full run.

One thing I notice is that "rsync" gives an error if the user never logged in and never received any message because his store directory does not exist
---------------------------------------------------------
Mon Apr 14 23:02:22 BRT 2008 : Syncing /opt/zimbra/store/0/27 to /opt/tst/zimbra_cache/user@domain.com.br/opt/zimbra/store/0
rsync: link_stat "/opt/zimbra/store/0/27" failed: No such file or directory (2)
rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9]
Mon Apr 14 23:02:22 BRT 2008 : Size : 8.0K
--------------------------------------------------------------

Maybe you can check if this directory exist before runnig rsync on it.

Even with this the script runs without any problem.

One idea for the next version, after the restore script
is to be able to run more then one user at time, so it could
improve the time needed to run the backup.

Let me know if you need any help.

[]s

[bonadio]

Oops

I just forgot, the server is running Ubuntu 6 with
Zimbra 5.0.4 FOSS

[]s

[2l2n]

I confirm the problem above : new users (never logged) cause errors during rsync.
Maybe you can add a mail function to send an error message to admin user.

Very good job

[Rich Graves]

Restoring a deleted user will also require their LDAP entry (or entries, if there are child entries for things like signatures). If you're running something like openldap/sbin/slapcat -f conf/slapd.conf -l /backup/backup.ldif as part of your nightly DR backup, then you already have this info for everyone.