[SOLVED] Backup Inquiry

[msheiny]

Hello,

I've been using Zimbra (Release 6.0.5_GA_2213.RHEL5_20100202220948 CentOS5 FOSS edition) for a few months and have been pouring through the various backup scripts that various people have posted trying to figure out the best solution for my environment.

Right now I just have a single Zimbra server and about 10 users (pretty low load overall) and am using some host-based backup agents that write the whole system to tape storage. The other day I wrote a small script to shut down zimbra during the backup procedure in order to make sure I get everything during backups.

For the longterm I really want a spare machine that is sitting around that I can enable when the primary goes down. I don't need it to be a HOT spare and can afford the hour or so of downtime that would require to bring it back up/change firewall rules/etc. I was thinking for something like this I just have the spare system sitting around, give it the same hostname (different IP thou), same O/S, same Zimbra version, then run some cronjob rsyncs daily to it from the production system. Does this sound like something viable? As long as I shutdown zimbra during the rsync should my second system still be prepared to be run as a cold spare zimbra system? Is there anything else I have to worry about taking care of?

Thanks in advance for any advice.
Mike

[raj]

yes you have all the right details..

for 2nd server you can keep everything same except the IP address..
OS, hostname, zimbra version and zimbra cold backup rsync.

to rollover..just change the ip and make sure the hostname points to same on 2nd machine when you do that and you should be ok.

you should try this atleast once when you set it up

Raj

[msheiny]

Thanks for the feedback Raj.

I've run some scripts to try to sync up the index, mailstore, and /opt/zimbra directories. The problem I am running into now is that I can not hit the web interface on the mirror system. I just keep getting a lot of loading and nothing. I've confirmed the port's open, and ran a wireshark dump but it seems to be wonky. Let me know if anyone can provide some assistance on what to look for.


Wireshark pic: Yfrog Image : yfrog.com/ndselection005p

MAIL LOG EXCERPTS:

netstat results:
tcp 0 0 :::7780 :::* LISTEN 23834/httpd off (0.00/0/0)
tcp 0 0 :::80 :::* LISTEN 23745/java off (0.00/0/0)


zimbra 23834 1 0 15:30 ? 00:00:00 /opt/zimbra/httpd-2.2.14/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra 23835 23834 0 15:30 ? 00:00:00 /opt/zimbra/httpd-2.2.14/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra 23836 23834 0 15:30 ? 00:00:00 /opt/zimbra/httpd-2.2.14/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra 23837 23834 0 15:30 ? 00:00:00 /opt/zimbra/httpd-2.2.14/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra 23838 23834 0 15:30 ? 00:00:00 /opt/zimbra/httpd-2.2.14/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
zimbra 23839 23834 0 15:30 ? 00:00:00 /opt/zimbra/httpd-2.2.14/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf


Thanks in Advance!


EDIT: Errrr. NM it seems like none of the other services are operating properly either. For instance when I telnet to the POP (port 110) I just get a blank screen and though it seems the java process is listening on that port I do not get a zimbra banner like I do on my production server. I'm going to keep tracing, if anyone can give me some good tips on where to look I'd still be very appreciative Thanks again.

[raj]

well there is whole lot of stuff which needs to be talked about regarding your setup to fix the problem you are facing.

1) are you behind Firewall and do you have split-DNS setup on your current production server ??
2) you wrote you copiued index, mailstore, and /opt/zimbra..what what i know all you need to do is copy EVERYTHING under /opt/zimbra not just selected folders...you doing that correct ??
3) on target server you need to setup the SAME HOSTNAME in /etc/hosts and seprate splitdns on the 2nd sevrer to resolve the SAME HOSTNAME to 2nd machine locally only.

* for your services to start properly the 2nd machine's hostname needs ot be same on different ip pingable locally on that 2nd machine so ldap can be connected to start services propoerly.

to test the 2nd server if correct you need to make sure following is ok

Assumptions for server 1
Server 1 Hostname: myzimbra.mydomain.com
Server 1 Local IP: 10.10.1.1

:PS: follwing command have back ticks not single quotes
[root@myzimbra ~]# host `hostname`
myzimbra.mydomain.com has address 10.10.1.1
myzimbra.mydomain.com mail is handled by 10 myzimbra.mydomain.com.

* above shows DNS / hostname resolution to your local ip for the given hostname
* this will make the LDAP connecect to 10.10.1.1 when called using hostname by Zimbra

Assumptions for server 2
Server 2 Hostname: myzimbra.mydomain.com
Server 2 Local IP: 10.10.1.2

[root@myzimbra ~]# host `hostname`
myzimbra.mydomain.com has address 10.10.1.2
myzimbra.mydomain.com mail is handled by 10 myzimbra.mydomain.com.

* above shows DNS / hostname resolution to your local ip for the given hostname
* this will make the LDAP connecect to 10.10.1.2 when called using hostname by Zimbra

once you have above working on BOTH servers ..then your full rsync of /opt/zimbra with services stopped should work on 2nd server

please make sure the servers are configured the SAME way..
you can follow this articale
Moving ZCS to Another Server » Zimbra :: Blog

let me know how it goes..
Raj

[msheiny]

Hey Raj,

Let me try to answer your questions in order.

1) Firewall/NAT - Ultimately the servers are behind a firewall/NAT from the internet but not between each other. Also I have unlimited access to both boxes in the firewall rules. I've also confirmed the firewalll is disabled on the mirror server and have tried to telnet to specific ports that zimbra should be listening on and am getting the same result. Therefore it seems highly unlikely that this is the culprit but of course I'm still tracing everything...

2) I copied everything under /opt/zimbra but my index and maillstore were symlink'd to different directories outside of /opt/zimbra. For instance mine are located under /mnt/mail/index and /mnt/mail/mailstore. So I did an rsync on all three directories.

3) both systems have the same hostname, different IPs, and after following another guide. I setup dnsmasq so that the mail.mydomain.com resolves correctly back to itself. Still doesn't seem to be working properly with that change though. Hmmm. One thing I didn't check out is the MX record. Let me play with that and see if it makes a difference.

Let me play with that last part of #3 and see what happens...

Thanks for the good lead and article Raj!

Regards,
Mike

[msheiny]

Hey Raj,

It's been a couple months since I had time to resolve this issue. But I'm roughly at the same state as before. Here are the results from the commands you requested in September The only thing I still see missing is from the ORIGINAL which lacks the MX record pointing back to itself... I was also worried there was an issue the machine so I tried installing from scratch on the mirror and was able to see the web page without issue but once again when I tried to rsync the /opt/zimbra directory over I have the issues of non responsive services. Anything else you can recommend taking a look at? Thanks for your help!!

MIRROR
192.168.0.70 - mail.X.com

host `hostname`:

mail.X.com has address 192.168.0.70
mail.X.com mail is handled by 1 mail.X.com.

cat /etc/hosts:

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
192.168.0.70 mail.X.com mail

ORIGINAL
192.168.0.72 - mail.X.com

host `hostname`:

mail.X.com has address 192.168.0.72

cat /etc/hosts:

# Do not remove the following line, or various programs
# that require network functionality will fail.
::1 localhost6.localdomain6 localhost6
192.168.0.72 mail.X.com
[PUBLIC IP] mail.X.com
127.0.0.1 localhost.localdomain localhost

[msheiny]

For anyone else searching for this, I was able to solve my issue. Turns out I had a couple of issues, one of them was I cleaned up my rsync. The second was that I needed to add an MX record that pointed back to this backup machine (using dnsmasq) - if anyone is interested in those lines, let me know.

Lastly, I started running into issues where LDAP would not start upon trying to kick-off zimbra... Well I searched through the forums and found my answer and here is my latest restore script which is working (this does not include the backup rsync pieces which are on the primary server):

#!/bin/bash
#
#
#
#
# Syncin' dinky' doo? Zimbra sync for me and you.
# This is designed to be run on the zimbra mirror system


logfile=/root/zimbrabackup.log
########################## FUNCTIONS ####################################

echo -e "\t\t*******************************\n\t\t\tZi mbra COLD Sync Script v1.0\n\t\t***************
****************\n\n"

function shutdownlocalzimbra {
### shutting down local zimbra service
echo -e "Shutting down LOCAL zimbra service....\t\c"
#/etc/init.d/zimbra stop
su - zimbra -c "/opt/zimbra/bin/zmcontrol stop"

#see if that was successful
if [ $? -eq 0 ]
then
echo "Success"
else
echo "Failed"
exit 1
fi
}


function syncer {
echo -e "\n\nNow syncing Zimbra opt folder ...."
rsync --delete -a -e "ssh -i /root/.ssh/zimbra_backup_key" root@192.168.0.72:/opt/zimbra-back
up/ /opt/zimbra/
echo "Done"
}

function bringuplocalzimbra {
### Starting up local zimbra service
echo -e "Starting LOCAL zimbra service $(date +'%T')....\t\c"
#/etc/init.d/zimbra start
su - zimbra -c "/opt/zimbra/bin/zmcontrol start"

#see if that was successful
if [ $? -eq 0 ]
then
echo "Success @ $(date +'%T')"
else
echo "Failed @ $(date +'%T')"
exit 1
fi
}

function cleanup {
echo "Cleaning up Zimbra Restore...."

#fix permissions
/opt/zimbra/libexec/zmfixperms --extended -v
# my zimbra user has different uid/guid on the mirror server, only use if your environment is similar!
sed -i 's/501/500/g' /opt/zimbra/conf/localconfig.xml

#fix database
su - zimbra -c "cd /opt/zimbra/data/ldap/hdb/db/; db_recover -v"

}

######################### MAIN ############################################
shutdownlocalzimbra
syncer
cleanup
bringuplocalzimbra