[SOLVED] Yet Another Backup Script for Community Version

[dwmtractor]

You guys who care about backup, here is a thread initiated by Zimbra that you really ought to vote on:

Vote: Hot Backups for FOSS?

Dan

[cornbread]

has anyone fitured out how to modify to go over ssh with a pre-authenticated key? something like ssh -vaz /path/to/backup domain.name:/backup/

[dwmtractor]

cornbread,

I've noticed your posts on this over the past few days and it occurs to me. . .given that what you're wanting to do is open-source backup, as you have probably realized by now, the effective OSS backup requires you to have Zimbra services stopped. Since you don't want them stopped for longer than necessary, you might really want to break up the stages of your backup anyway. By this I mean do your rsync (or other file copy process) to a local hard drive, as most of these methods do, restart Zimbra, and then (and ONLY then) back it up to the remote location. If you make the backup into a tar/compressed file, then you can use any methodology for ssh that can do ftp over ssh, which might make your scripting easier.

This will be significantly faster in terms of Zimbra downtime since the limiting factor will be HDD throughput rather than your net link throughput.

cheers,

Dan

[heinzg]

Originally Posted by dwmtractor

heinzg,

I'm not enough of a shell script guru to analyze all you have done, but from what I have read of it so far, my hat is off to you for an extremely well-thought-out script. It's great to have the development site you linked, but if I could presume to make a suggestion, it'd be nice if you would post either the script, or at least a brief discussion with link to your development page, on the wiki under the open-source backup section (here) so that when people are reading about backup methodologies, this one is added to the mix. This will make it more likely (1) that those who need it find your script, and (2) that you'll get a wider variety of people testing and providing feedback to you. Kinda what we call a win-win!

Kudos for your hard work!

Dan

Hi Dan!

Thanx for you suggestion I will put it on the wiki ASAP... promise.

heinzg

[heinzg]

Hi cornbread!


Sorry I have not gotten back to you until now, vacation so to say.

How do I do this? I already have the destination set up to do automatic ssh authentication from the machine. Which part of the script would I modify to do rsync via ssh?

has anyone fitured out how to modify to go over ssh with a pre-authenticated key? something like ssh -vaz /path/to/backup domain.name:/backup/

I would like to understand what you are wanting to do with doing the the rsync over ssh to be able to offer you the best possible solution for your backup.

I opted not to do rsync over ssh when I started this script as I would have had to split the script into to two parts. Part one on the mail host to sync to a backup to a backup host. And the second part on the backup host to then archive it. I found this to be error prone & complex, as well as resource intensive (network, cpu, overall time).

I am currently testing a version of the script which will, after the creation of the backup archive ssh's (beem) it over to a backup/archive host for save keeping against 2 disks failing on the RAID 5 volume over night (yes I have seen it happen with very expensive SCSI disks made in Hungary in 2002 by a big two letter computer company from the States, who could that be... I remember you! heinzg don't forget that fast ). would this be something you could use?

On a slower system that is running my backup script I still have less than 1 min down time for the service and that is when there is as good as no one on it... (I have not had a mail user complain yet ) I find this a good and save compromise for backing up the community edition server.

I would say in advance that if I was to sync to a remote host in my own LAN I would setup a rsync server and not ssh it, or even better use a iSCSI volume mounted on demand on 10Gigabit, then again a simple NFS share would also do, but the network usage, I would only do this kind of stuff on a dedicated backup LAN.... Ah to complex to be safe without support And when it all goes wrong in production, what is it you want, yes a good old working backup that is fast and simple to restore.

My motto: Simple is good.

please feel free to contact me if there is anything regarding this script I can help you with.

heinzg

[cornbread]

heinzg,

I think that is the best way to do it.

I thin you are experimenting with exactly what I was envisioning.

local cold backup (small downtime)
restart zimbra, tar, ssh to host:/backup


Thanks again for the great work!

[heinzg]

Hi there!

I have done a re-write of the script and updated my first post (i.e. see the first post in this thread for more info)
It now has lots of new features like: archive encryption, scp, md5 checksums, installer...

Now also to be found on the zimbra wiki

I would love to hear your feed back on which Distros you are running the script it on.

Cheers
Heinzg

[heinzg]

Hi again!

JUST A WORD OF CAUTION to those who will run this script with encryption ON

The key file (script default /etc/zmbac/noread) BACK IT UP & KEEP IT SAFE without it your archives are no more than wasted space on the disk and/or tape!!.

The key generated by the script installer is a 48*8=348bit random passphrase, so unless you have a friend at the CIA and lots of time to recover the data, your archive will be lost.

OH one more thing the passphrase should only be known by "the need to know". Your secrets are only as safe as your passphrase is public!

There is also a small flaw in my scripts security you should know about. When "dar" is creating the archive (takes a while) you can SEE THE PASSPHRASE with "top -c" or with "ps -ef" so be sure to have no interactive shell users on the system at the time of backup.

Now I can sleep better tonight having that of my chest

[mmorse]

Originally Posted by dwmtractor

Kudos for your hard work!

Nice scripting - we'll be sure to get ya on the free shirt list

[jayel]

Hello,

Nice script.
I am trying to install it but I get an Error when installing.

Instellen van dar (2.2.4-2ubuntu2) ...
Create /etc/zmbac/
install y or n: y
mkdir: created directory `/etc/zmbac'
done

Create noread
install y or n: y
'G'enerate or 'E'nter a secure passphrase
please enter G or E: G


For 'scp' to work, you have to have setup PKI authentication (passwork less login)
Should I try setup this for you?
install y or n: y
/usr/bin/ssh-copy-id: ERROR: No identities found
0
Create ssh ID? y or n: y
Please except defaults, and NO 'passphrase' !!
You must specify a key type (-t).
Usage: ssh-keygen [options]
Options:
  -a trials   Number of trials for screening DH-GEX moduli.
  -B          Show bubblebabble digest of key file.
  -b bits     Number of bits in the key to create.
  -C comment  Provide new comment.
  -c          Change comment in private and public key files.
  -e          Convert OpenSSH to IETF SECSH key file.
  -F hostname Find hostname in known hosts file.
  -f filename Filename of the key file.
  -G file     Generate candidates for DH-GEX moduli.
  -g          Use generic DNS resource record format.
  -H          Hash names in known_hosts file.
  -i          Convert IETF SECSH to OpenSSH key file.
  -l          Show fingerprint of key file.
  -M memory   Amount of memory (MB) to use for generating DH-GEX moduli.
  -N phrase   Provide new passphrase.
  -P phrase   Provide old passphrase.
  -p          Change passphrase of private key file.
  -q          Quiet.
  -R hostname Remove host from known_hosts file.
  -r hostname Print DNS resource record.
  -S start    Start point (hex) for generating DH-GEX moduli.
  -T file     Screen candidates for DH-GEX moduli.
  -t type     Specify type of key to create.
  -v          Verbose.
  -W gen      Generator to use for generating DH-GEX moduli.
  -y          Read private key file and print public key.

copying your ID to 192.168.0.244
please enter your 'root' user password on '192.168.0.244' at the prompt
0
/usr/bin/ssh-copy-id: ERROR: No identities found

Install cronjob to automate the nightly backups
install y or n: n

What does that mean ?
And will my backup file be copied to my remote backup server ?

My server is UBUNTU 6.06.


Again, Nice job

John