Full Server Backup and Restore of “open source version”

[czaveri]

I am running 4.0 GA and the following works:

zmprov ma user@domain.com zimbraAccountStatus maintenance

When I run the above command and I click on anything inside the Web UI, I am logged out with a message that session expired. When I try to login It shows that this account is under maintenance.

I sent email during this time and I didn't receive it.

I also checked that user2@domain.com can login and his emails are working.

As soon as I make the user@domain.com active he is fine - but has lost the e-mail sent to him when maintenance was on. I am hoping that the mail server will try again and user@domain.com will receive his lost email.

[czaveri]

What drew my attention to this idea were this blog entry this blog entry and this post .

What we want: Sys Admin or the User wants to backup user@domain.com account.
The idea:
Backup E-mail to a spare IMAP server.
Backup Contacts & Calendars using REST.
Restore E-mail using Imapsync.
Restore Contacts & Calendars using Imapsync.

Take full zimbra backup less often - perhaps weekly. (Zimbra stop, backup, start).

P.S. If Imapsync can sync between an IMAP server and a local Maildir structure then we might not even need the spare IMAP server.

[czaveri]

... ideas on this script (zimbraBackup.pl) or this method? It's an outcome of the attempts to learn Perl, and evaluation of Zimbra and my first Trac installation and Open Source ...

I haven't experimented with it yet and it might just blow out anything/everything/nothing... or might work!?

#!/usr/bin/perl
use strict;
use warnings;
use POSIX;

###########################################################################
# Program Name: zimbraBackup v0.01                                        #
#                                                                         #
# This script can be used to backup, individually, all accounts in the    #
# Zimbra Collaboration Suite 4.0.0 GA                                     #
#                                                                         #
# Copyright (C) 2006 Chintan Zaveri                                       #
# E-mail: smile@sis.net.in                                                #
#                                                                         #
# This program is free software; you can redistribute it and/or modify    #
# it under the terms of the GNU General Public License version 2, as      #
# published by the Free Software Foundation                               #
#                                                                         #
# This program is distributed in the hope that it will be useful,         #
# but WITHOUT ANY WARRANTY; without even the implied warranty of          #
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the           #
# GNU General Public License for more details.                            #
#                                                                         #
# You should have received a copy of the GNU General Public License along #
# with this program; if not, write to the Free Software Foundation, Inc., #
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.             #
###########################################################################

#################### CHANGE THE VARIABLES BELOW THIS!! ####################

# Change this to the correct admin password. This is the password used by you to enter the Admin section.
my $admin_password = 'youradminpassword';

# Auth method (http/https)
my $auth_method = 'http';

# Change this to the correct port number on which Zimbra is accessible
my $zimbra_port = "80";

# Change this path to your backup directory. Enter Only Absolute Path else unexpected things may happen.
my $backup_directory = '/backup';

# Change this path to the one for mysqldump. Enter Only Absolute Path else unexpected things may happen.
my $mysqldump = '/usr/bin/mysqldump';

# Change this path to the one for rsync. Enter Only Absolute Path else unexpected things may happen.
my $rsync = '/usr/bin/rsync';

# Change this path to one for wget. Enter Only Absolute Path else unexpected things may happen.
my $wget = '/usr/bin/wget';

#################### DON'T CHANGE ANYTHING BELOW THIS! ####################

# Current date and time
my $current_time = POSIX::strftime('%A-%B-%d-%Y-%H-%M', localtime(time));

# Get hostname
my $hostname = `hostname`;
chop($hostname);

# Get all accounts
my $all_accounts = `/opt/zimbra/bin/zmprov gaa`;
chop($all_accounts);
my @all_accounts = split("\n",$all_accounts);

# MySQL root user
my $mysql_root_user = 'root';

# Get password for MySQL root user
my $mysql_root_password = `/opt/zimbra/bin/zmlocalconfig -s | grep mysql_root_password`;
chop($mysql_root_password);
$mysql_root_password =~ s/mysql_root_password = //;

# Setting up backup directories
if (!(-e $backup_directory)) {
        mkdir($backup_directory, 0755) || die "Cannot create directory $backup_directory : $!";
}

if (!(-e "$backup_directory/$current_time")) {
        mkdir("$backup_directory/$current_time", 0755) || die "Cannot create directory $backup_directory/$current_time : $!";
}

if (!(-e "$backup_directory/$current_time/mysql")) {
        mkdir("$backup_directory/$current_time/mysql", 0755) || die "Cannot create directory $backup_directory/$current_time/mysql : $!";
}

if (!(-e "$backup_directory/$current_time/store")) {
        mkdir("$backup_directory/$current_time/store", 0755) || die "Cannot create directory $backup_directory/$current_time/store : $!";
}

if (!(-e "$backup_directory/$current_time/index")) {
        mkdir("$backup_directory/$current_time/index", 0755) || die "Cannot create directory $backup_directory/$current_time/index : $!";
}

if (!(-e "$backup_directory/$current_time/wget")) {
        mkdir("$backup_directory/$current_time/wget", 0755) || die "Cannot create directory $backup_directory/$current_time/wget : $!";
}

# Backup every account
foreach my $account (@all_accounts) {
        if ($account) { # Backup only if $account is not empty, that is, it should contain an e-mail address
                # Get mailbox Id
                my $id = `/opt/zimbra/bin/zmprov gmi $account | grep mailboxId`;
                chop($id); # remove trailing 0
                $id =~ s/mailboxId: //; # remove the term "mailboxId: " from the line and leave only the actual mailbox id

                # Get quota Used
                my $used = `/opt/zimbra/bin/zmprov gmi $account | grep quotaUsed`;
                chop($used); # remove trailing 0
                $used =~ s/quotaUsed: //; # remove the term "quotaUsed: " from the line and leave only the actual quota used

                if ($used) { # Only if there is anything in the account, backup, else don't
                        # Put the account in Maintenance Mode
                        exec ("/opt/zimbra/bin/zmprov ma $account zimbraAccountStatus maintenance");

                        # Use "mysqldump" to backup mailbox database corresponding to the id
                        exec ("$mysqldump --port=7306 --socket=/opt/zimbra/db/mysql.sock --user=$mysql_root_user --password=$mysql_root_password mailbox$id > $backup_directory/$current_time/mysql/$account.mailbox$id.sql");

                        # Use "rsync" to backup "/opt/zimbra/store/0/5" directory where '5' is the mailbox Id
                        exec ("$rsync -avrlHKpogDt /opt/zimbra/store/0/$id/ $backup_directory/$current_time/store/$account.mailbox$id");

                        # Use "rsync" to backup "/opt/zimbra/index/0/5" directory where '5' is the mailbox Id
                        exec ("$rsync -avrlHKpogDt /opt/zimbra/index/0/$id/ $backup_directory/$current_time/index/$account.mailbox$id");

                        # Use "wget" to backup all contacts from the REST URL "http://server/zimbra/home/user@domain.com/contacts.csv"
                        if (($zimbra_port == 80) || ($zimbra_port == 443)) {
                        exec ("cd $backup_directory/$current_time/wget/ && $wget --user admin\@$hostname --password $admin_password $auth_method://$hostname/zimbra/home/$account/contacts.csv");
                        } else {
                        exec ("cd $backup_directory/$current_time/wget/ && $wget --user admin\@$hostname --password $admin_password $auth_method://$hostname:$zimbra_port/zimbra/home/$account/contacts.csv");
                        }

                        # Use "wget" to backup all appointments from the REST URL "http://server/zimbra/home/user@domain.com/calendar.ics"
                        if (($zimbra_port == 80) || ($zimbra_port == 443)) {
                        exec ("cd $backup_directory/$current_time/wget/ && $wget --user admin\@$hostname --password $admin_password $auth_method://$hostname/zimbra/home/$account/calendar.ics");
                        } else {
                        exec ("cd $backup_directory/$current_time/wget/ && $wget --user admin\@$hostname --password $admin_password $auth_method://$hostname:$zimbra_port/zimbra/home/$account/calendar.ics");
                        }

                        # Remove the account from Maintenance Mode and make it Active
                        exec ("/opt/zimbra/bin/zmprov ma $account zimbraAccountStatus active");
                }
        }
}

=> A SAMPLE BACKUP STRATEGY FOR ZCS 4.0.0 GA <=
SAVE THE ZCS BINARY
The first step is to save the ZCS binary, you used to install or upgrade Zimbra, in a safe place.

COLD OR OFF-LINE BACKUP
Now, do a "cold" or "off-line" backup of Zimbra, following the following steps:
1) su - zimbra
2) zmcontrol stop
3) exit
4) kill -9 `ps -U zimbra|grep [0-9]|awk '{print $1}'` > /dev/null
4) rsync -avrlHKpogDt /opt/zimbra/ /backup/zimbra_cold
5) su - zimbra
6) zmcontrol start

Schedule weekly cold-backups.

HOT BACKUP
After doing the cold backup, put the "zimbraBackup.pl" as a cron job scheduled every 12 or 24 hours. (Please edit it to suit your preferred paths before assigning to cron)

Advice: If you don't frequently add new accounts (less than twice a week), you must do the cold backup every time you add new account(s).

This way, at any point of time, we have full ZCS backup, and, we have copies of individual accounts' daily backups. In short, in case of any failure, you will have close to all data in ZCS.

Limitation: Even if you follow this strategy you may lose upto 12 or 24 hours of data depending upon when the last hot backup was taken.

=> A SAMPLE RESTORE PLAN FOR ZCS 4.0.0 GA <=
To Restore the system, follow the following steps:

1. If your system was not formatted you must uninstall ZCS first. Use the stored ZCS binary for this. If your system was formatted, you don't need to uninstall ZCS, because it is not there.
2. Do a fresh install of Zimbra using this binary
3. Move the "/opt/zimbra" directory to some other place (temporarily and after all's well - delete it!)
4. Move the last Cold backup to "/opt/zimbra"
5. Restore the last Hot Backup created by "zimbraBackup.pl" using standard tools, such as, mysql, mv, cp, rm, ... etc.
6. Check how much you lost!
7. Done!

[marcmac]

Did you actually run that script? I don't see it getting past this line:

exec ("/opt/zimbra/bin/zmprov ma $account zimbraAccountStatus maintenance");

From 'man perlfunc'

exec LIST
exec PROGRAM LIST
The "exec" function executes a system command and never
returns-- use "system" instead of "exec" if you want it to
return. It fails and returns false only if the command does
not exist and it is executed directly instead of via your sys-
tem's command shell (see below).

Since it's a common mistake to use "exec" instead of "system",
Perl warns you if there is a following statement which isn't
"die", "warn", or "exit" (if "-w" is set - but you always do
that). If you really want to follow an "exec" with some other
statement, you can use one of these styles to avoid the warn-
ing:

exec ('foo') or print STDERR "couldn't exec foo: $!";
{ exec ('foo') }; print STDERR "couldn't exec foo: $!";

Aside from that problem, other things that I notice:

There's no error checking at all. None. NONE.
There's no enforcement of the user (you need to run this as zimbra).
There's no usage statement, so ./script.pl --help will kick off a backup (I get flamed for that one constantly)

Other comment:

Don't call mysql, use the perl DBI - it's even shipped with zimbra.

You should use the zmmailbox command to list the calendars, and grab them all.

I don't think you need to specify BOTH the port and the socket for mysqldump.

[czaveri]

Hi marcmac,

Thank-you very much for your comments and help.

I haven't run it yet. But, will do so today and make the corrections as pointed by you.

There's no error checking at all. None. NONE.

Will take care in v0.02

There's no enforcement of the user (you need to run this as zimbra).

I am deliberately not enforcing the "zimbra" user because once I improve on this, I want to be able to run it as any user and am planning to setuid-setgid root and have a web interface ... (that's why I am using absolute paths for zimbra's provided binaries).

There's no usage statement, so ./script.pl --help will kick off a backup (I get flamed for that one constantly)

Thanks again, marcmac. v0.02

Don't call mysql, use the perl DBI - it's even shipped with zimbra.

I am using "mysqldump" and don't know if I can use DBI for similar purpose. But if it can be, then I will make the changes.

You should use the zmmailbox command to list the calendars, and grab them all.

That looks like a perfect command. Didn't know about it. Will use it.

I don't think you need to specify BOTH the port and the socket for mysqldump.

I am specifying both because I am using the mysqldump binary that is not bundled with Zimbra. This binary would generally be compiled alongwith a mysql-server that would use the default port and default socket. If I only provide port it fails because it tries to search in the default path for the socket. It works only when I provide, both, port and socket.

Sincerely,

[czaveri]

I thought better to start with Off-line backup script. Here's one. I have tested this on my system running CentOS 4.3 and ZCS 4.0.0 GA. I also restored the entire /opt/zimbra directory from the backup and am now running on it now ...

I have around 20 users on the system and approx. 3,000 emails in all. All contacts, calendars, wiki, and everything is intact and working well.

Attached the script with README: zimbraColdBackup-Ver0.01beta-Rev15.tgz

#!/usr/bin/perl -w
use strict;
use POSIX;
use Proc::ProcessTable;
use File::Path;
use File::Rsync;
use Mail::Mailer;
my $version = "0.01beta";       # version of this program
my $revision = "15";            # revision of this program
# License
my $license = qq(
###########################################################################
# Program Name: zimbraColdBackup Ver $version Rev $revision                      #
#                                                                         #
# This script can be used to backup Zimbra Collaboration Suite 4.0.0 GA   #
#                                                                         #
# Copyright (C) 2006 Chintan Zaveri                                       #
# E-mail: smile\@sis.net.in                                                #
#                                                                         #
# This program is free software; you can redistribute it and/or modify    #
# it under the terms of the GNU General Public License version 2, as      #
# published by the Free Software Foundation                               #
#                                                                         #
# This program is distributed in the hope that it will be useful,         #
# but WITHOUT ANY WARRANTY; without even the implied warranty of          #
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the           #
# GNU General Public License for more details.                            #
#                                                                         #
# You should have received a copy of the GNU General Public License along #
# with this program; if not, write to the Free Software Foundation, Inc., #
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.             #
###########################################################################
);

###########################################################################
# Make changes BELOW this line to suit your environment / requirement     #
###########################################################################

# Change the following to the path to your backup directory. This directory must exist on your system.
my $backup_directory = '/backup';

# Change the following to the path to "rsync"
my $path_to_rsync = '/usr/bin/rsync';

# Do you want to rotate backups?
my $rotate_backups = 1; # "1" for yes, and "0" for no

# If you want to rotate backups, please specify the number of days after which it should be rotated
my $rotation_days = 7;

# You must not need to change the following, but just in case
# Specify the path to zimbra
my $zimbra_path = '/opt/zimbra';

# zimbra user
my $zimbra_user = 'zimbra';

# Do you want this script to send an e-mail after backup is done?
my $to_email = 0; # 1 for yes, 0 for no

# If you specified 1 above, please provide your email address below
my $email_address = 'you@yourdomain.com';

###########################################################################
# DON'T MAKE CHANGES BELOW THIS LINE - unless you know what you are doing #
###########################################################################
my $prog_name = $0;             # name of this program
my $option = shift;             # the first parameter passed to this program

# If any useless parameters are provided to script give error.
# We are checking only the first param
usage("Invalid parameter(s)") if (($option) && ($option ne "--help") && ($option ne "?") && ($option ne "help") && ($option ne "--confirm") && ($option ne "confirm"));

# do we need to print help?
if (! defined($option)) { $option = "nevermind"; } # This is necessary so that perl does not complain about use of uninitialized value

help("Help Screen") if ($option =~ /--help|help|\?/);

usage("Usage") if (($option ne "confirm") && ($option ne "--confirm"));

# Validate Inputs
if (($rotate_backups !~ /^\d+$/) || ($rotate_backups < 0) || ($rotate_backups > 1)) {
        die "The variable \$rotate_backups can only contain either 0 or 1";
}

if (($rotation_days !~ /^\d{1,3}$/) || ($rotation_days < 1)) {
        die "The variable \$rotation_days must contain only, either 1, 2 or 3, digits - you can't leave this blank even if you are not rotating backups";
}

if (($backup_directory eq "") || ($backup_directory !~ /^\//)) {
        die "You need to specify an absolute path to backup directory";
}

if (! (-d $backup_directory)) {
        die "There is no such directory as $backup_directory";
}

if (! (-d $zimbra_path)) {
        die "There is no such directory as $zimbra_path";
}

if (! (-e $path_to_rsync)) {
        die "The path to 'rsync' provided by you as $path_to_rsync seems wrong. Please check.";
}

if (($to_email !~ /^\d+$/) || ($to_email < 0) || ($to_email > 1)) {
        die "The variable \$to_email can only contain either 0 or 1";
}

if (! $email_address) {
        die "Please enter a valid email address";
}

# Define an array to store the status of various actions of this script. This will be sent by email.
my @results;

# Stop Zimbra
my $zimbra_stop_status = system ("su - zimbra -c '$zimbra_path/bin/zmcontrol stop'");
if ($zimbra_stop_status) {
        warn "Something was not right when trying to stop Zimbra";
        push (@results, "Stopping Zimbra: Something was not right when trying to stop Zimbra");
} else {
        push (@results, "Stopping Zimbra: Success");
}
die "Error stopping Zimbra" if ($zimbra_stop_status == -1);

# Kill all lingering Zimbra processes
my $zimbra_uid = getpwnam($zimbra_user);

my $process_table = Proc::ProcessTable->new;

# Gracefully kill lingering processes: kill -15, sleep, kill -9
foreach my $process ( @{$process_table->table} ) {
        if (($process->uid eq $zimbra_uid) || (($process->cmndline =~ /$zimbra_user/) && ($process->cmndline !~ /$prog_name/))) {
                kill -15,$process->pid; # thanks, merlyn
                sleep 10; # not sure if there'll be buffering. If you know, please improve.
                kill -9,$process->pid;
        }
}

my $current_time = POSIX::strftime('%a-%d-%b-%Y-%H-%M', localtime(time)); # current day, date, month, time, ...
my $since_epoch = time();       # seconds since epoch

# Backup Zimbra using "rsync"
my $rsync_obj = File::Rsync->new( {
        'rsync-path'    => $path_to_rsync,
        'archive'       => 1,
        'recursive'     => 1,
        'links'         => 1,
        'hard-links'    => 1,
        'keep-dirlinks' => 1,
        'perms'         => 1,
        'owner'         => 1,
        'group'         => 1,
        'devices'       => 1,
        'times'         => 1
} );
$rsync_obj->exec( { src => "$zimbra_path/", dest => "$backup_directory/$since_epoch-$current_time-zcsColdBackup" } ) or warn "rsync failed\n";

push (@results, "Backup using Rsync: Successfully created $backup_directory/$since_epoch-$current_time-zcsColdBackup");

# Now that backup is done, start Zimbra
my $zimbra_start_status = system ("su - zimbra -c '$zimbra_path/bin/zmcontrol start'");
if ($zimbra_start_status) {
        warn "Something was not right when trying to start Zimbra";
        push (@results, "Starting Zimbra: Something was not right when trying to start Zimbra");
} else {
        push (@results, "Starting Zimbra: Success");
}

# Rotate backups
if ($rotate_backups) {
        # get a list of all files from the backup directory
        opendir (DIR, $backup_directory) or die "can't opendir $backup_directory: $!";
        while (defined(my $filename = readdir(DIR))) { # actually, $filename is the name of all rsynced directories
                next if ($filename !~ /-zcsColdBackup/); # if it is not a cold backup directory, ignore it
                # if $filename is older than $rotation_days then delete it
                my @filename_parts = split("-",$filename); # cutting from hyphens
                my $allowed_age = $since_epoch - (60 * 60 * 24 * $rotation_days); # allowed age is converted to seconds
                if ($filename_parts[0] < $allowed_age) { # if the first part of $filename is not equal to or more than the allowed age
                        rmtree ("$backup_directory/$filename") || die "Cannot delete $filename"; # delete it
                        push (@results, "Backup Rotation: Removed $backup_directory/$filename");
                }
        }
        closedir (DIR);
}

if ($to_email) {
        my $email_body = join("\n", @results);
        my $mailer = Mail::Mailer->new("sendmail");
        $mailer->open( {
                'From'          =>      'root@localhost',
                'To'            =>      $email_address,
                'Subject'       =>      'Results of zimbraColdBackup'
        } ) or die "Can't open: $!\n";
        print $mailer "Date: ", $current_time, "\n\n";
        print $mailer $email_body, "\n";
        $mailer->close();
}

sub usage {
# what to print when usage is incorrect
my $options = qq(
Usage: $prog_name [--help|?|help] [--confirm]

  --help|?|help         display help
  --confirm|confirm     run script

If no parameters are provided then the script will display usage.

Please edit this file to suit your system before using it.
);

die @_, $license, $options;
}

sub help {
my $help_txt = qq(
1. OVERVIEW:

This script can help in taking cold or off-line backups of Zimbra Collaboration Suite.

2. FEATURES:

a. Take cold or off-line backups of Zimbra Collaboration Suite
b. Rotate backups
c. E-mail backup reports

3. INSTALLATION:

This script relies on the following Perl modules:

POSIX, Proc::ProcessTable, File::Path, File::Rsync, Mail::Mailer

Please ensure that these modules are available on your system before you execute the script.

These can be downloaded from http://cpan.org

Once the modules are installed edit the zimbraColdBackup.pl script to match your system / requirements.

chmod this script to 755

chown this script to root:root

Done!
);

die @_, $license, $help_txt;
}

[czaveri]

Improved this script to display/email the time it took to backup. Thus, giving us an idea how long the system stayed off-line.

Typical result looks like this:

Date: Mon-18-Sep-2006-12-57

Stopping Zimbra: Success
Backup using Rsync: Successfully created /backup/1158564436-Mon-18-Sep-2006-12-57-zcsColdBackup
Starting Zimbra: Success
The backup took: 298 seconds to complete

To restore this backup, all I need to do, as "root", is:

su - zimbra
zmcontrol stop
exit
rm -rf /opt/zimbra (or copy it somewhere temporarily)
mv /backup/1158564436-Mon-18-Sep-2006-12-57-zcsColdBackup /opt/zimbra
su - zimbra
zmcontrol start

All permissions, links, ownerships, etc. are ok and I need not do anything to restore them.

Attached: zimbraColdBackup-Ver0.01beta-Rev17.tgz

[Dirk]

Looks nice, czaveri. I'll try this on my home system tonight and report back on how well it handles a Suse10 system. Thanks.

[czaveri]

Originally Posted by Dirk

Looks nice, czaveri. I'll try this on my home system tonight and report back on how well it handles a Suse10 system. Thanks.

Hi Dirk,

Did you try this? Let me know if you have any negative feedback or doesn't work for you.

Thanks and regards,

P.S. I have put it as a cron job on two of my systems and will check daily for two weeks. Will post my results then. Today is day 3.

[Dirk]

Actually, I forgot. So, I just remoted into my home machine (I'm currently at work) and sent the script to it, chmodded it to 755 and tried it out (rather brave of me, if it goes wrong I'd had it!)

Anyway, it didnt work, this is the error:

zimbra:/opt/zimbra # ./zimbraColdBackup
Can't locate Proc/ProcessTable.pm in @INC (@INC contains: /usr/lib/perl5/5.8.7/i586-linux-thread-multi /usr/lib/perl5/5.8.7 /usr/lib/perl5/site_perl/5.8.7/i586-linux-thread-multi /usr/lib/perl5/site_perl/5.8.7 /usr/lib/perl5/site_perl /usr/lib/perl5/vendor_perl/5.8.7/i586-linux-thread-multi /usr/lib/perl5/vendor_perl/5.8.7 /usr/lib/perl5/vendor_perl .) at ./zimbraColdBackup line 4.
BEGIN failed--compilation aborted at ./zimbraColdBackup line 4.

Looks like I dont have the right perc modules installed but I've no idea how to get them. System is using Zimbra 4.0 and Suse 10.0
Any ideas?

[EDITED: adjusted comment after reading the README ]

Originally Posted by czaveri

Hi Dirk,

Did you try this? Let me know if you have any negative feedback or doesn't work for you.

Thanks and regards,

P.S. I have put it as a cron job on two of my systems and will check daily for two weeks. Will post my results then. Today is day 3.