backup strategy

[g8se]

I'm one of the administrators for SourceForge.net: ZCS Tools, which is a script to automate doing backups of the Open Source Edition, as the Open Source Edition doesn't come with the backup tools that Network Edition has.

At the moment that script stops all of the zimbra processes, rsyncs the entire /opt/zimbra directory, and starts all of the zimbra processes.

The recent blog post on ldap made me think that we could do a lot better. But I don't know enough of how zimbra works. My rough outline below is what I think might work. Can I get some feedback on if it would, or what a better approach might be?

1. prevent changes
   • stop postfix and stop tomcat
   • write lock on mysql and slapd
2. grab account data
   • dump the slapd database to somewhere
3. grab message metadata
   • dump the mysql database to somewhere
4. grab messages
   • dump the emails somewhere
5. allow changes
   • release locks on mysql and slapd
   • start postfix

Am I missing stuff? Where is the lucene database stored, is that in mysql, or do I need an extra step to dump that? Where are the actual email messages stored?

Is there another way to prevent changes, apart from stopping postfix and tomcat? Postfix could probably be reconfigured to hold all messages. Is there a mode that you can prevent all new logins, and suspend current logins?

Cheers
Geoff

[mmorse]

Originally Posted by g8se

Where are the actual email messages stored?

Usually in /opt/zimbra/store (there's also the index) but people often make symbolic links, and you can even use the admin console to change the locations...

What's your purpose? Are you trying to push for better incremental backups, etc? ie what did the blog post make you think of?
If your killing postfix and tomcat to me it seems you might as well just do a full zmcontrol stop...you just shut down an mta and cut off web access.

Postfix could probably be reconfigured to hold all messages. Is there a mode that you can prevent all new logins, and suspend current logins?

Well there's always the easy tomcat stop or a mass locking (just no login) or maintenance (no mail delivery-sits in queue) of accounts...
-To me that sound's like your pushing for a hot backup...and before I ramble needlessly let me know what in/about the ldap blog post spurred your thoughts.

[g8se]

Well there's always the easy tomcat stop or a mass locking (just no login) or maintenance (no mail delivery-sits in queue) of accounts...
-To me that sound's like your pushing for a hot backup...and before I ramble needlessly let me know what in/about the ldap blog post spurred your thoughts.

At the moment our backup script copies /opt/zimbra. So it copies all of the binaries, which is wasteful. Would be better if we only copied the user data. It also completely stops zimbra, which isn't very userfriendly.
The blog post made me think that we could do better. We could dump less data, and we could avoid completely stopping all the servers, both of which may speed up the backup time, and will give better feedback to users if they try to login. And also not rely on incoming mail servers to spool mail for us.

I'm sure that for large installs, the ratio of program data to users data gets very small, but for new installs, it could be a significant time saving.

[mubley]

I spent some time trying to develop an open-source hot-backup script back in December 2005. Have you seen this?
Full Server Backup and Restore of “open source version” (original version 0.01)
Full Server Backup and Restore of “open source version” (version 0.02)

Unfortunately, I haven't used the script since then, so I have no idea if it works in current versions of Zimbra or not. But perhaps it may assist you in your development.

[g8se]

Originally Posted by mubley

I spent some time trying to develop an open-source hot-backup script back in December 2005. Have you seen this?
Full Server Backup and Restore of “open source version” (original version 0.01)
Full Server Backup and Restore of “open source version” (version 0.02)

Unfortunately, I haven't used the script since then, so I have no idea if it works in current versions of Zimbra or not. But perhaps it may assist you in your development.

I hadn't seen it, but it look very easy to follow. Thanks!

[Artturi]

Hello,

For point 3, I have written a script grabbing metadata for mailboxes from the mysql database (folders).

See it here :
Per User Mailbox Backup (OE Version) - ZimbraWiki

It could surely be improved a lot but I have it running every night.

Regards