Page 1 of 4 123 ... LastLast
Results 1 to 10 of 34

Thread: Per User Backup for FOSS edition

  1. #1
    fdsadmin is offline Member
    Join Date
    May 2007
    Location
    England
    Posts
    13
    Rep Power
    8

    Default Per User Backup for FOSS edition

    Hi all,

    I have decided to have a go at creating a per user backup solution for the FOSS edition. Here comes a brain dump...

    Currently I use the LVM backup technique to create a nightly backup of '/opt/zimbra' as a disaster recovery option. However, more of my users are using webmail and sooner or later someone is going to do something really silly and I will need a way to restore the data for an individual. Restoring the whole server for one users cock up is not really an option

    Disclaimer

    The scripts attached to this post are experimental and only suitable for testing purposes only on non-production servers. If you use them on a production server and it breaks, you get to keep the pieces!

    Scope

    Basically I want a hot/warm-ish backup solution which will give me a 24 hourly restore point for individual user accounts. If someone does something silly, then loosing 24 hours of email is fair penalty in my book If I can get reliable per user backup/restore working then I will continue to do my '/opt/zimbra' DR backup just once a week, and user backups every night.

    How it works

    In a nut shell...

    • Dumps the 'zimbra' database.
    • Get a list of all users
    • Loops through each user
      • Figures out what volumes the user data is stored in
      • Makes a "hot" sync of the users index/store
      • Puts the account in "maintenance" mode
      • Dumps the user DB and rows from associated tables in the 'zimbra' DB
      • Re-syncs the users index/store
      • Puts the account in "active" mode
      • Creates a tarball of the synced index/store data
      • Rotates the backups


    It's a bash script. I have tried to make sure there is no hard coded paths, accounts, passwords, etc. All but two configuration parameters are taken directly from Zimbra. Everything is logged to the console and log file.

    The directory each account rsyncs it index/store to (the cache) is preserved between backups, so subsequent syncs are relatively quick. However, the cache must be local or mounted file system. In addition to the cache, a daily backup directory is created containing a tarball of that days index/store data plus the dumped SQL. Therefore, you need extra disk space. You can do the maths

    Not sure how well this will scale, but for my 15 users it should be fine. I am planning to keep the cache on local disk (for speed) and the backup directory on NFS.

    Anyone for testing?

    I have created a vmware box running a snapshot of my live server. It runs ZCS 5.0.4 FOSS on Debian 4.0 32bit. I have not tested on 4.5.x (yet). Is there anyone out there who might be interested in testing on non-production servers to help iron out the wrinkles?

    The Script

    v0.1 - 11th April 2008
    • Initial release.
    • Scripts to backup individual user or all users.
    • No restore features as yet.
    • No real documentation.


    I haven't started on a restore script yet, but getting the basics going shouldn't be to time consuming. I will be working on the restore scripts early next week.

    Download the attached tarball and extract on your non-production server. 'zimbra_functions.sh' has a couple of parameters you can configure at the top to choose the cache and backup locations.

    To back up a single user run the following on your Zimbra server.

    Code:
    ./zimbra_backup_user.sh user1@example.org
    To backup all users run to following on your Zimbra server.

    Code:
    ./zimbra_backup_all_users.sh
    The Logs

    Below is example log output to show what it does so far.

    Code:
    Thu Apr 10 12:51:05 EDT 2008 : Dumping zimbra --single-transaction > /opt/zimbra_backup/zimbra.sql
    Thu Apr 10 12:51:05 EDT 2008 : Getting accounts
    Thu Apr 10 12:51:07 EDT 2008 : Backing up accounts
    Thu Apr 10 12:51:07 EDT 2008 : Processing user1@example.org
    Thu Apr 10 12:51:11 EDT 2008 : Getting group_id and index_volume_id
    Thu Apr 10 12:51:11 EDT 2008 : Id      : 10
    Thu Apr 10 12:51:11 EDT 2008 : Group   : 10
    Thu Apr 10 12:51:11 EDT 2008 : Mailbox : mboxgroup10
    Thu Apr 10 12:51:11 EDT 2008 : Getting message_volume_id and index_volume_id
    Thu Apr 10 12:51:11 EDT 2008 : Getting index volume path
    Thu Apr 10 12:51:11 EDT 2008 : Getting message volume path
    Thu Apr 10 12:51:12 EDT 2008 : index1 path   : /opt/zimbra/index
    Thu Apr 10 12:51:12 EDT 2008 : message1 path : /opt/zimbra/store
    Thu Apr 10 12:51:12 EDT 2008 : Syncing /opt/zimbra/index/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/index/0
    Thu Apr 10 12:51:12 EDT 2008 : Size : 3.4M
    Thu Apr 10 12:51:12 EDT 2008 : Syncing /opt/zimbra/store/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/store/0
    Thu Apr 10 12:51:12 EDT 2008 : Size : 144K
    Thu Apr 10 12:51:16 EDT 2008 : user1@example.org is now in maintenance mode.
    Thu Apr 10 12:51:16 EDT 2008 : Dumping mboxgroup10 > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/mboxgroup10.sql
    Thu Apr 10 12:51:16 EDT 2008 : Dumping zimbra mailbox --no-create-info --where "id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/mailbox.sql
    Thu Apr 10 12:51:16 EDT 2008 : Dumping zimbra mailbox_metadata --no-create-info --where "mailbox_id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/mailbox_metadata.sql
    Thu Apr 10 12:51:16 EDT 2008 : Dumping zimbra out_of_office --no-create-info --where "mailbox_id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/out_of_office.sql
    Thu Apr 10 12:51:17 EDT 2008 : Dumping zimbra scheduled_task --no-create-info --where "mailbox_id='10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/scheduled_task.sql
    Thu Apr 10 12:51:17 EDT 2008 : Dumping zimbra table_maintenance --no-create-info --where "database_name='mboxgroup10'" > /opt/zimbra_backup/user1@example.org/2008-04-10-1251/table_maintenance.sql
    Thu Apr 10 12:51:17 EDT 2008 : Syncing /opt/zimbra/index/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/index/0
    Thu Apr 10 12:51:17 EDT 2008 : Size : 3.4M
    Thu Apr 10 12:51:17 EDT 2008 : Syncing /opt/zimbra/store/0/10 to /opt/zimbra_cache/user1@example.org/opt/zimbra/store/0
    Thu Apr 10 12:51:17 EDT 2008 : Size : 144K
    Thu Apr 10 12:51:21 EDT 2008 : user1@example.org is now in active mode.
    Thu Apr 10 12:51:21 EDT 2008 : Creating index_n_store tarball from /opt/zimbra_cache/user1@example.org
    Thu Apr 10 12:51:26 EDT 2008 : Rotating backups
    Thu Apr 10 12:51:26 EDT 2008 : Keeping 2008-04-10-1251
    Thu Apr 10 12:51:26 EDT 2008 : Keeping 2008-04-10-1233
    Thu Apr 10 12:51:26 EDT 2008 : Keeping 2008-04-10-1209
    Thu Apr 10 12:51:26 EDT 2008 : Deleting 2008-04-10-1149
    
    Thu Apr 10 12:51:26 EDT 2008 : Processing user2@example.org
    Thu Apr 10 12:51:29 EDT 2008 : Getting group_id and index_volume_id
    Thu Apr 10 12:51:29 EDT 2008 : Id      : 8
    Thu Apr 10 12:51:29 EDT 2008 : Group   : 8
    Thu Apr 10 12:51:29 EDT 2008 : Mailbox : mboxgroup8
    Thu Apr 10 12:51:29 EDT 2008 : Getting message_volume_id and index_volume_id
    Thu Apr 10 12:51:29 EDT 2008 : Getting index volume path
    Thu Apr 10 12:51:29 EDT 2008 : Getting message volume path
    Thu Apr 10 12:51:29 EDT 2008 : index1 path   : /opt/zimbra/index
    Thu Apr 10 12:51:29 EDT 2008 : message1 path : /opt/zimbra/store
    Thu Apr 10 12:51:29 EDT 2008 : Syncing /opt/zimbra/index/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/index/0
    Thu Apr 10 12:51:29 EDT 2008 : Size : 104M
    Thu Apr 10 12:51:29 EDT 2008 : Syncing /opt/zimbra/store/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/store/0
    Thu Apr 10 12:51:30 EDT 2008 : Size : 231M
    Thu Apr 10 12:51:33 EDT 2008 : user2@example.org is now in maintenance mode.
    Thu Apr 10 12:51:33 EDT 2008 : Dumping mboxgroup8 > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/mboxgroup8.sql
    Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra mailbox --no-create-info --where "id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/mailbox.sql
    Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra mailbox_metadata --no-create-info --where "mailbox_id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/mailbox_metadata.sql
    Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra out_of_office --no-create-info --where "mailbox_id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/out_of_office.sql
    Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra scheduled_task --no-create-info --where "mailbox_id='8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/scheduled_task.sql
    Thu Apr 10 12:51:33 EDT 2008 : Dumping zimbra table_maintenance --no-create-info --where "database_name='mboxgroup8'" > /opt/zimbra_backup/user2@example.org/2008-04-10-1251/table_maintenance.sql
    Thu Apr 10 12:51:33 EDT 2008 : Syncing /opt/zimbra/index/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/index/0
    Thu Apr 10 12:51:33 EDT 2008 : Size : 104M
    Thu Apr 10 12:51:33 EDT 2008 : Syncing /opt/zimbra/store/0/8 to /opt/zimbra_cache/user2@example.org/opt/zimbra/store/0
    Thu Apr 10 12:51:34 EDT 2008 : Size : 231M
    Thu Apr 10 12:51:37 EDT 2008 : user2@example.org is now in active mode.
    Thu Apr 10 12:51:37 EDT 2008 : Creating index_n_store tarball from /opt/zimbra_cache/user2@example.org
    Thu Apr 10 12:59:08 EDT 2008 : Rotating backups
    Thu Apr 10 12:59:09 EDT 2008 : Keeping 2008-04-10-1251
    Thu Apr 10 12:59:09 EDT 2008 : Keeping 2008-04-10-1233
    Thu Apr 10 12:59:09 EDT 2008 : Keeping 2008-04-10-1210
    Thu Apr 10 12:59:09 EDT 2008 : Deleting 2008-04-10-1150
    Thu Apr 10 12:59:10 EDT 2008 : Deleting 2008-04-10-1148
    The Cache

    Example cache directory.

    Code:
    user1@example.org/
    `-- opt
        `-- zimbra
            |-- index
            |   `-- 0
            |       `-- 10
            |           `-- index
            |               `-- 0
            |                   |-- _6ip.cfs
            |                   |-- _6ip.del
            |                   |-- _8on.cfs
            |                   |-- _8on.del
            |                   |-- _9em.cfs
            |                   |-- _9em.del
            |                   |-- _gkx.cfs
            |                   |-- _gkx.del
            |                   |-- _gyu.cfs
            |                   |-- _gyu.del
            |                   |-- _h38.cfs
            |                   |-- _h3d.cfs
            |                   |-- deletable
            |                   `-- segments
            `-- store
                `-- 0
                    `-- 10
                        `-- msg
                            |-- 0
                            |-- 1
                            `-- 2
                                |-- 10023-44804.msg
                                |-- 10062-45306.msg
                                |-- 10063-45307.msg
                                |-- 10064-45308.msg
                                |-- 10082-45404.msg
                                |-- 10083-45406.msg
                                |-- 10185-46210.msg
                                |-- 10191-46216.msg
                                |-- 10201-46401.msg
                                |-- 10203-46405.msg
                                |-- 10267-46714.msg
                                |-- 10280-46800.msg
                                |-- 10300-47000.msg
                                |-- 10301-47001.msg
                                |-- 8406-37009.msg
                                |-- 9865-44013.msg
                                |-- 9920-44304.msg
                                |-- 9951-44425.msg
                                `-- 9964-44510.msg
    The Backups

    Example backup directory.

    Code:
    user1@example.org/
    |-- 2008-04-10-1209
    |   |-- index_n_store.tar.bz2
    |   |-- mailbox.sql
    |   |-- mailbox_metadata.sql
    |   |-- mboxgroup10.sql
    |   |-- out_of_office.sql
    |   |-- scheduled_task.sql
    |   `-- table_maintenance.sql
    |-- 2008-04-10-1233
    |   |-- calendar.ics
    |   |-- contacts.csv
    |   |-- mailbox.sql
    |   |-- mailbox_metadata.sql
    |   |-- mboxgroup10.sql
    |   |-- out_of_office.sql
    |   |-- scheduled_task.sql
    |   `-- table_maintenance.sql
    `-- 2008-04-10-1251
        |-- index_n_store.tar.bz2
        |-- mailbox.sql
        |-- mailbox_metadata.sql
        |-- mboxgroup10.sql
        |-- out_of_office.sql
        |-- scheduled_task.sql
        `-- table_maintenance.sql
    Thoughts, comments, suggestions?

    Regards, Martin.
    Attached Files Attached Files
    Last edited by fdsadmin; 04-11-2008 at 06:08 AM. Reason: v0.1 Released

  2. #2
    bonadio is offline Loyal Member
    Join Date
    Oct 2007
    Location
    Brazil
    Posts
    88
    Rep Power
    7

    Default

    Hi Martin

    Looks promising, I would be happy to do some testing, I have a semi-production server with a few users running UBUNTU FOSS where
    I can run some tests.

    Let me know

    good luck

    []s

  3. #3
    fdsadmin is offline Member
    Join Date
    May 2007
    Location
    England
    Posts
    13
    Rep Power
    8

    Default v0.1 Released

    See the first post, I have attached v0.1 of the scripts and added some basic usage instructions.

  4. #4
    2l2n is offline New Member
    Join Date
    Mar 2008
    Posts
    4
    Rep Power
    7

    Default

    I'm really not an expert, but I launched the script and everything looks good.

    Tested on Ubuntu 6.06.2 - Zimbra 5.0.4 FOSS.
    I only have 12 users and a few mails. I'm going to import my 800Mo .pst on Zimbra.

    To test it more seriously, can you give us a method for restoring 1 account ?

  5. #5
    Dirk's Avatar
    Dirk is offline Moderator
    Join Date
    May 2006
    Location
    England.
    Posts
    927
    Rep Power
    10

    Default

    Looks very interesting, once you have enough details & scripts together, consider adding the info to the wiki where it can be maintained easily.

    I'll have a good look through the sourcecode shortly, but it's always good to see this kind of effort

  6. #6
    fdsadmin is offline Member
    Join Date
    May 2007
    Location
    England
    Posts
    13
    Rep Power
    8

    Default

    2l2n, I will post an updated script in the new few days which includes restore capability.

    Dirk, I will be more than happy to maintain a wiki page for the script once I have got the basic backup/restore stuff working :-) For now, I will keep the first post in the topic up to date.

  7. #7
    bonadio is offline Loyal Member
    Join Date
    Oct 2007
    Location
    Brazil
    Posts
    88
    Rep Power
    7

    Default

    Hi Martin

    I just tested your script in my semi-production server, congratulations everything was fine:

    I have 25 users with 2Gb of messages total, it took 18 min to do a full run.

    One thing I notice is that "rsync" gives an error if the user never logged in and never received any message because his store directory does not exist
    ---------------------------------------------------------
    Mon Apr 14 23:02:22 BRT 2008 : Syncing /opt/zimbra/store/0/27 to /opt/tst/zimbra_cache/user@domain.com.br/opt/zimbra/store/0
    rsync: link_stat "/opt/zimbra/store/0/27" failed: No such file or directory (2)
    rsync error: some files could not be transferred (code 23) at main.c(977) [sender=2.6.9]
    Mon Apr 14 23:02:22 BRT 2008 : Size : 8.0K
    --------------------------------------------------------------

    Maybe you can check if this directory exist before runnig rsync on it.

    Even with this the script runs without any problem.

    One idea for the next version, after the restore script
    is to be able to run more then one user at time, so it could
    improve the time needed to run the backup.

    Let me know if you need any help.

    []s

  8. #8
    bonadio is offline Loyal Member
    Join Date
    Oct 2007
    Location
    Brazil
    Posts
    88
    Rep Power
    7

    Default

    Oops

    I just forgot, the server is running Ubuntu 6 with
    Zimbra 5.0.4 FOSS

    []s

  9. #9
    2l2n is offline New Member
    Join Date
    Mar 2008
    Posts
    4
    Rep Power
    7

    Default

    I confirm the problem above : new users (never logged) cause errors during rsync.
    Maybe you can add a mail function to send an error message to admin user.

    Very good job

  10. #10
    Rich Graves is offline Outstanding Member
    Join Date
    Jan 2007
    Location
    Minnesota
    Posts
    719
    Rep Power
    9

    Default

    Restoring a deleted user will also require their LDAP entry (or entries, if there are child entries for things like signatures). If you're running something like openldap/sbin/slapcat -f conf/slapd.conf -l /backup/backup.ldif as part of your nightly DR backup, then you already have this info for everyone.

Page 1 of 4 123 ... LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Replies: 21
    Last Post: 02-04-2010, 10:06 AM
  2. Post instsallation problems
    By Assaf in forum Installation
    Replies: 14
    Last Post: 01-29-2007, 11:38 AM
  3. Services stopped working
    By lilwong in forum Administrators
    Replies: 4
    Last Post: 08-15-2006, 09:19 AM
  4. Fedora Core 3, Clean Install - Not working!
    By pcjackson in forum Installation
    Replies: 17
    Last Post: 03-05-2006, 07:38 PM
  5. Network edition - strange behavior
    By goetzi in forum Installation
    Replies: 6
    Last Post: 11-16-2005, 03:08 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •