Risks of Hot backup and/or power failure in OSE

[regan.wallace]

This is an odd question..

What exactly are the risks of power failure/hard shutdown/hot backup/etc in zimbra?
What services don't work right, and how complicated is recovery?


I'm wondering about using LVM snapshots, but without shutting down the server for the daily backups, and once a week/month do a "full" by doing a shutdown, lvm snapshot, startup.


I also pondered the idea of a standby server, using lsyncd but this would depend on how tolerant a slightly inconsistent state is and how easily that can be recovered.


If all you are going to say is "Bad idea, don't do it" save yourself the time to type; as I've already thought that initially to myself. Hence the "weekly fulls".

Thanks

[phoenix]

Originally Posted by regan.wallace

If all you are going to say is "Bad idea, don't do it" save yourself the time to type;...

OK.

Originally Posted by regan.wallace

Hence the "weekly fulls".

If you're prepared to lose up-to a weeks worth of email, go right ahead. If you're prepared to lose all your email then don't get a UPS.

[spectra]

Sometimes it's not cool the way the gurus of the forums answer us that are not in the KNOW.

I think your question is excellent. While I don't know the technical specs to answer in a sufficient level of detail I will share something I think was significant that relates.

I run several FOSS and I have a script that basically did the following:
- shutdown zimbra
- tar or rsync /opt/zimbra to a 2nd drive in the zimbra machine
- bring zimbra back online
- rsync the tar file that was produced to a different physical machine

Oddly enough, a buddy of mine took the zimbra plunge same time I did. His box kept getting powered off freakishly. He had power issues at his location. Next thing I know, his box ran and status showed good but could never get to GUI via web. I also had a box that I found was plugged up where the power was wacky. As a result, lots of powering off and on. In the end I lost a Zimbra install as well as my buddy.

So, I'm saying that perhaps, these power issues will cause an issue.

The other thing I notice, when you do zmcontrol stop orshutdown, the mySQL stuff associated with it is still running. I know stuff is stored there, dunno what and how much. To me, if a shutdown dont stop that, maybe something can get corrupted.

I'm not expert but I would suggest a weekly shutdown, there's a script to kill all zimbra process on the wiki, do a full backup (maybe rsync to another drive ) then bring it back up.

What I do now.
1. shutdown
2. use that script to kill all zimbra processes
3. update/upgrade my ubuntu distro packages
4. backup zimbra to the other internal drive
5. reboot system
6. verify zimbra is up

That's my $0.02

[regan.wallace]

That's what I figured and was wondering.

I used to make backups of a very large and very busy mysql table by connecting, doing a lock, making an lvm snapshot, unlocking and backing up the lvm.
It allowed perfect recovery everytime I tried, and I tested it a few times per day, for the first month, and periodically after that.

At most I'd have to do REPAIR TABLE `blah`, type work; but all in all, it was a safe way to backup since the entire database is consistent (lvm snapshot).

I just wondered if the same could be true for zimbra.

As I'm the only user of my zimbra, it makes no sense for me to go to NE; and I like learning about it and managing my own server since my old job had me accustomed to maintaining the zimbra machines.
But, BES has a retry fail count for which they use IMAP idle; after a certain unknown number of retries to connect, they give up and default to a 15min polling. If I can limit the number of times zimbra shutsdown and for how long, the retries reset and I won't have to keep messing with my mail settings.

I'll probably just do a weekly or monthly full, and just use the REST url to backup my mailbox for the "just in case".

Maybe I'll try restoring from a hot-snapshot on a test box and see what happens. I wonder if stopping the MTA would keep it safe; anything except the imap server (zmmailbox).


My current setup is:
1. Rsync into temp dir
2. Shutdown
3. Rsync again
4. Startup
5. tar and rsync to backup server

I used to have a "cold standby" that was configured with everything (including the IP) that would pull that tar down and extract it, run zmfixperms, and all that; and the eth0 was just disabled, and would do it using eth1. So a ifup eth0, and a zmcontrol start would bring it online. I used it once, worked well.


Anyway.. now I'm rambling

[phoenix]

Originally Posted by spectra

Sometimes it's not cool the way the gurus of the forums answer us that are not in the KNOW.

The o/p made it clear that he didn't want a discussion if I was going to say it was a bad idea, that's what I was going to say an that's he got.

Originally Posted by regan.wallace

If all you are going to say is "Bad idea, don't do it" save yourself the time to type; as I've already thought that initially to myself. Hence the "weekly fulls".

[lytledd]

Originally Posted by regan.wallace

This is an odd question..

I'm wondering about using LVM snapshots, but without shutting down the server for the daily backups, and once a week/month do a "full" by doing a shutdown, lvm snapshot, startup.

Thanks

I do this now, but my file system on the LVM is XFS. When you do a snapshot with that file system, LVM will do a 'file system freeze', so there will be no inconsistent state.

Doug

[regan.wallace]

Originally Posted by lytledd

I do this now, but my file system on the LVM is XFS. When you do a snapshot with that file system, LVM will do a 'file system freeze', so there will be no inconsistent state.

Doug

Have you tried restoring from one? without it being completely shutdown?

I just wonder if a perfect filesystem snapshot of a running zimbra can recover guaranteed.

[lytledd]

Originally Posted by regan.wallace

Have you tried restoring from one? without it being completely shutdown?

I'm doing a backup of the complete /opt directory, so the Zimbra service has to be stopped before the restore.

And yes, I've done a restore for testing.

Doug