Hot Backup Vs Cold Backup in CE version

[blason]

Hi Folks,

I have a question regarding OSS version, I took a backup of my zimbra server using rsync while my server is running i.e I didn't stop any of the services and even after that I managed to restore the system successfully so my question is do I really need to stop the services while taking backup? Will this cause any consequences rather than doing a cold backup?

[phoenix]

Originally Posted by blason

Hi Folks,

I have a question regarding OSS version, I took a backup of my zimbra server using rsync while my server is running i.e I didn't stop any of the services and even after that I managed to restore the system successfully so my question is do I really need to stop the services while taking backup? Will this cause any consequences rather than doing a cold backup?

Yes you do need to stop the services. You may get lucky and successfully restore a backup (you may have problems later on) but the files cannot be guaranteed to be in a consistent state when you restore them. If this is a production mail server the question I'd ask: are you prepared to lose all your mail with a failed restore? If the answer to that is "yes" then carry on without stopping the services, if the answer is "no" the...... you can see where I'm going with this, I guess.

[blason]

Well, thanks Phoenix but honestly I was not lucky for once or twice but I did each and every backup testing on my test mail server for about 10 to 15 times and I was successfully restored that every time.
And trust me that actually forced me to drop a question in forum to understand others voices. I agree that services should be stopped to clean pickup the files but that never happened with me;so wondering why cant hot backup be done with OSS version.

You wouldn't believe even I tried with rsync first while services are running then stopped the services and initiated backup again just to test the setup and I was again able to restore the entire system cleanly.

[phoenix]

Originally Posted by blason

I agree that services should be stopped to clean pickup the files but that never happened with me;so wondering why cant hot backup be done with OSS version.

I've already answered this, the files cannot be guaranteed to be in a consistent state. The fact that it has 'never' happened to you does not mean that it won't ever happen to you.

My original comment still stands, if you're happy to do that and you are also happy that one day you may need to restore the server and it won't work and you lose all your email then carry on doing what you're doing. It's totally your choice, I'm not here to convince you otherwise - you asked a question and you've had my answer. If you need further confirmation then search the forums, there are several threads from various Zimbra employees (notably the director of engineering (I think that's his title) stating the same thing.

Originally Posted by blason

You wouldn't believe even I tried with rsync first while services are running then stopped the services and initiated backup again just to test the setup and I was again able to restore the entire system cleanly.

Why wouldn't I believe it? You've said that's the case and I believe you, however, you seem not to believe my reply on why it's a bad idea.

[Tripple]

And what's the load on your testserver while you're taking a hot backup?
I'm using a script found on this site who's taking a hot backup, shut down Zimbra and take a cold backup only to keep the downtime to a strict minimum.

[blason]

Nothing much hardly 30-35 users created with arnd 1.5 Gb of individual user's pst pumped in.

I know and I am following the same stuff with rsync the entire directory then shutdown the services and rsync it again.

BTW just wondering have you guys or any one able to figure it out as what NE version does to take a HOT backup that CE can not do it? I mean just out of curiosity.

[bofh]

Hello Blason

First of all its a premier - i agree with Phoenix - first time in my life so if oyu dont believe him believe me.

So you cannot say your system is restored cleanly - you simply cant - you will see if i dont if an error occurs but theres no real proof that it did

even if you check mailbox integrity and stuff like that this isnt a garantuee or any kind of proof everything is thing

you must understand zimbra uses -
a dynamic store on filebasis - so every file in there is sorted by IDs
those IDs are stored in the mysql backend - so the relation mysql and filesystem is very important
and to make things worse you got also an ldap store for all user account information things

in addition you cannot backup mysql while its running so easy
there are already aproches todo so but its way more comlicated - you have to store transaction logs and the bin position and restore them correctly - but these are specialsed products and theres a reason why almost every major backupsoftware on linux implemented a dump by mysql dump instead of simple filecopy

SO NO hotbackup of mysql - always a bad idea
hot backup mysql on zimbra because of massive write operations - desaster waiting to happen

Even if your hotbackp on mysql - and on ldap works - imagine the follwoing -

your rsync backuped mysql - fisnish
now a mail comes ins - zimbra do its magic wirtie to filesystem and mysql
now you backup the filesystem

when you restore you restore only the file but not the mysql data (or vice versa) -
ife no idea what will happen if zimbra decides to use that "unused" id for another mail again or - maybe worse you got a mysql db entry but no data on the restore ....

(i know ife simpliefed things...)




So defently hotbackup for restore a nogo -

however you can use hotbackup to reduce the downtime

you simply make several hotbackups a day and later on a certain time one coldbackup - your downtime reduce dramatically - down to a few minutes -

another adavantage is thats usually not so bad to restart the java vm once in a while java likes reboots hehe



in order to use my hotbackup/coldbackup combination you must be able todo incremental backups and be able to restore a certain version (exactly that version where you made the coldbackup)

ther a lot of rsync script they can do that easy

i dont use rsync i use another incremental backup but i can share you my prebackup and postbackup scripts - because i use the same job with a different time squedule i decide by time on which shedule im doing the shutdown

Prebackup

#!/bin/bash


chour=$(date +%H)
#declare -i chour

if ([ "$chour" -ge 5 ] && [ "$chour" -lt 6 ])  ; then
         echo "Its" $chour "- Shutdown"
         sudo -H -u zimbra /opt/zimbra/bin/zmcontrol stop
else
echo "Its" $chour "- No shutdown"
fi

Postbackup

#!/bin/bash


chour=$(date +%H)
#declare -i chour

if ([ "$chour" -ge 5 ] && [ "$chour" -lt 11 ])  ; then
         echo "Its" $chour "- Startup"
        sudo -H -u zimbra /opt/zimbra/bin/zmcontrol start
else

echo "Its" $chour "- No Startup"
fi

They are very simple - prebackup decide when to shutdown
postbackup restarts at the same time

i dont wanted to make a start zimbra all the time if its not nessesary even if it should do no damage - i simply dont do such things

so just define your incremental rsync job always with those script pre/post
and set the script on the time of rsync backup you wanna do cold

[bofh]

PS: to your question about zm NE backup
Its quiet simple they use their transaction logs
they dump ldap and mysql and stuff like that - so they dont simply copy files

also it wasnt even easy for them to find the right way - backup had issues till this year with some major bugfixes maybe still has - its not a complain - just to show that it isnt so simple even for people how developed that storing system

pps: the biggest part of my 3minutes 25 sec downtime is that my backupsoftware read out its database and compare changes to every file
that - nad startup and shutdown are the 2 major timekiller - the amount of data with 5 hotbackups during the day (3 alone between 1am and 5:30 )
is almost nothing a few hundred megs - so downtime isnt an issue

[blason]

Thanks bofh for such an extensive explanation I must appreciate that. instead I would recommend you to write and Wiki about the background stuff happens while backup in progress or backup 101 something like that and definitely its not about disbelief on anybody's comment its about the fact that I experienced though without knowing all this background hence wanted to know what exactly happens when we initiate backup.

And regarding NE I was damn sure its just not copying the stuff, it wouldn't be, again wanted to know what exactly happens behind the curtain in NE as well.

[bofh]

i addmit iam lazy - and i hate wiki writing - feel free to take any of my stuff and post it under your name its for everyone

btw: im aware of that rsync script which uses also hot prebackup and cold backup afterwards

but i strongly recommend todo with scripts i wrote
for a simple reason - you dont have control when coldbackup runs if you got a spike of data or downtime on the backuppath/backupline

means lets say you use that script its starts at 2am - works usually 1 hour then coldbackup 10 minutes - regular downtime 3am till :10

but - if you got a lot of data or you line for backup slows down - your coldbackup might start at 10am - problem

if the line breaks (i cant say that for shure but it think it will be that way) the hotprebackup fails and cold starts
so you have to replicate all the data while its down - worst case couple hours downtime


thats why i work with those scripts - i will only go down at 5:30
and backup job runtime is set on a max of 30 minutes - so worst case 31min downtime if something bad happens bevore

worst case 1 day an incomplete backup (and i recive a warning) but not a downtime of xxx hours