Page 2 of 66 FirstFirst 12341252 ... LastLast
Results 11 to 20 of 659

Thread: [SOLVED] Yet Another Backup Script for Community Version

  1. #11
    heinzg is offline Loyal Member
    Join Date
    Jan 2008
    Location
    Germany
    Posts
    83
    Rep Power
    7

    Default

    Hi osiris!

    sorry to hear that it did not work out of the box for you. why is it that things that work well on Ubuntu just don't work the same on RH.

    Could you maybe send me your version info for:
    bash, su an your env setup e.g. /etc/profile, /etc/bash.bashrc, and ~/.bashrc. or maybe you want to migrate your system to Ubuntu?

    Fun aside.
    I changed the script's sudo execution to su and now pack the zmcontrol strings into a $CMD variable and then pass it onto su -

    And from your output I see that there is a problem with the with the way the $CMD construct is being passed onto su.... Nothing wild!

    On lines 148 to 152 is this $CMD construct, which I think you should have a look at maybe there is a quote sign in the $CMD variable which is not liked by su or zmcontrol

    here is a script to diagnose what is going wrong in the variable construct and to test the stop and start logic in the script. You just have to uncomment the section you want to test.

    Code:
    #!/bin/bash
    
    ZM_HOME=/opt/zimbra/
    
    ZM_USER=`ps -ef | grep "$ZM_HOME" | grep "java" | grep -v "zmmailboxdmgr" | awk '{print $1}' | head -n 1`
    if [ -z $ZM_USER ]
    then
        echo "Unable to determan the zimbra user"
        cat $LOG | mail -s "Zimbra backup error on `hostname --fqdn`" $EMAIL
        exit 1
    elif [ $ZM_USER = "root" ]
    then
        echo "Zimbra user should never be root! I'll die now..."
        # send email....
        cat $LOG | mail -s "Zimbra backup error on `hostname --fqdn`" $EMAIL
        exit 1
    fi
    
    # Command short form
    ZIM_STOP="$ZM_USER -c $ZM_HOME"bin/zmcontrol" shutdown"
    ZIM_START="$ZM_USER -c $ZM_HOME"bin/zmcontrol" start"
    ZIM_STATUS="$ZM_USER -c $ZM_HOME"bin/zmcontrol" status"
    ZIM_STAT_START="$ZM_USER -c $ZM_HOME"bin/zmstatctl" start"
    ZIM_STAT_STOP="$ZM_USER -c $ZM_HOME"bin/zmstatctl" stop"
    
    echo
    echo $ZIM_STOP
    echo
    echo $ZIM_START
    echo
    echo $ZIM_STATUS
    echo
    echo $ZIM_STAT_START
    echo
    echo $ZIM_STAT_STOP
    echo
    
    
    
    
    ########## Stopping Zimbra
    #        echo "Stopping the Zimbra server"
    #        sleep 2 && sync
    #        su - $ZIM_STOP 
    #        if [ "$?" -eq "1" ]
    #        then
    #            echo "zmcontrol shutdown had an error!"
    #                 # send email....
    #                 cat $LOG | mail -s "Zimbra backup error on `hostname --fqdn`" $EMAIL
    #        else
    #            echo
    #            echo "Zimbra server has stopped "
    #        fi
    #
    #...................................................................................
    #        echo
    ########## Starting the Zimbra server again
    #        su - $ZIM_START
    #        if [ "$?" -ne "0" ]
    #        then
    #            echo "There was an error starting Zimbra!"
    #                 # send email....
    #                 cat $LOG | mail -s "Zimbra backup error on `hostname --fqdn`" $EMAIL
    #        else
    #            echo
    #            echo "Zimbra has started"
    #        fi
    #        
    #        sleep 15
    #...................................................................................
    ######### quick hack to start the stats component
    #        # On my system I have 7 pid files whe all is running well, you might have less depending on your config
    #        SPIDS=`ls -A -1 $STATPIDBASE | wc -l`
    #        if [ "$SPIDS" != "7" ] 
    #        then 
    #            echo "I am expecting 7 pids for Stats, rebooting Stats subsystem!"
    #        su - $ZIM_STAT_STOP 
    #        if [ "$?" -ne "0" ]
    #        then
    #            echo "Stopping stats failed!"
    #            # send email....
    #            cat $LOG | mail -s "Zimbra backup error on `hostname --fqdn`" $EMAIL
    #        else
    #            echo "Stats have been stopped"
    #        fi
    #         
    #        sleep 2 
    #        su - $ZIM_STAT_START 
    #        if [ "$?" -ne "0" ]
    #        then
    #            echo "Starting stats failed!"
    #            # send email....
    #            cat $LOG | mail -s "Zimbra backup error on `hostname --fqdn`" $EMAIL
    #        else
    #            echo "Stats started..."
    #        fi
    #        else
    #            echo "Stats seems to be running fine..."
    #        fi
    #               
    #        # Status Check to see that is running 
    #        sleep 2
    #        STATUS=`su - $ZIM_STATUS  | grep Stopped`
    #        if [ "$STATUS" ] 
    #        then 
    #            echo $STATUS
    #            # send email....
    #            cat $LOG | mail -s "Zimbra backup error on `hostname --fqdn`" $EMAIL
    #        fi
    You can also send my a private message then we can try diagnose this via skype.... I would really like this scrip to be more compatible with other distros and to work without changing code in the logic section of the script.

    Cheers heinzg

  2. #12
    heinzg is offline Loyal Member
    Join Date
    Jan 2008
    Location
    Germany
    Posts
    83
    Rep Power
    7

    Default

    Hi All!

    Just to update. The script does not work unmodified on Red Hat.

    If you are having problems you'll have to change all short form constructs e.g.

    Code:
    su - $ZIM_STOP
    su - $ZIM_START
    su - $ZIM_STATUS
    su - $ZIM_STAT_START
    su - $ZIM_STAT_STOP
    to the full command e.g.
    Code:
    su - zimbra -c zmcontrol shutdown
    su - zimbra -c zmcontrol start
    Sorry for this, I will fine a better solution.

    p.s. Please post your experience with the script i.e. Distro / Works "out of the box" or Not


    1. Ubuntu 6.06.1 LTS / Works out of the box
    2. Red Hat Enterprise 5 / Does Not

    Cheers

  3. #13
    Stergil is offline Senior Member
    Join Date
    May 2007
    Posts
    71
    Rep Power
    8

    Question

    Thanks for the work you have put into the script.

    I've downloaded the latest version offered by your site and tried it out.

    I also needed to install mailutils and struggled with that for a few. I didn't want to install the exim packages that apt-get suggested when trying to install the mailutils package. To get through this I came across this forum post which worked perfectly.

    I was happy to see the link your script provided when it failed to detect dar support, since dar was new to me.

    I also thought that dar was hanging, getting hung up on dar file creation. However, that was just my machine. It took ~5min for the dar process alone to finish. So if anyone else is on older hardware have some patience

    FWIW It's running on an athlon xp 1.8ghz with 1gig ram.

    I may wind up using a variant of your script for the rsyncing and process stopping/starting. The dar archives aren't really needed for my setup. I have a tape backup on the network which just pulls data from the /backups slice. So the rsync'd dir works like a champ for me.

    I'm pretty much the only user on my server for a small home-office installation. My uncompressed install is only ~2gig.

    If you find others that are in the same situation maybe a -nodar flag or something would be useful.

    Almost forgot, running Ubuntu 6.06 .

    Edit: BTW, I needed to touch create the log file to get the script to start. After a successful backup I checked the log to find it empty.

    I also just tried a dar -t on the created archive and got the following
    Code:
    root@zimbra:/opt/share/bin/backups/dar# dar -t 09_Zimbra_Backup_03-March-2008_FULL.1.dar
    Warning, 09_Zimbra_Backup_03-March-2008_FULL.1.dar seems more to be a slice name than a base name. Do you want to replace it by 09_Zimbra_Backup_03-March-2008_FULL ? [return = OK | esc = cancel]
    Escaping...
    OK, keeping 09_Zimbra_Backup_03-March-2008_FULL.1.dar as basename
    ./09_Zimbra_Backup_03-March-2008_FULL.1.dar.1.dar is required for further operation, please provide the file. [return = OK | esc = cancel]
    Also, the TOO_MEDIA_DIR was empty after the backup was finished.
    Last edited by Stergil; 03-03-2008 at 03:14 AM.

  4. #14
    Stergil is offline Senior Member
    Join Date
    May 2007
    Posts
    71
    Rep Power
    8

    Default

    Well it looks like dar may be getting hung up afterall. When I thought the previous backup was hungup on the dar process I ran a -t on the archive in another terminal. I eventually returned to the original terminal window and saw the output summary and at that point thought it finished.

    This time around I've not done anything and it's been sitting here for more than 10minutes.
    Code:
     PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
    1152 root      39  19 15836  11m 2016 R 64.1  1.6   10:21.07 dar
    It's late and I should sleep, I'll let it run overnight to see if it's going to finish on its own. The CPU utilization keeps fluctuating which seems to hint that it's not hung. But requiring more than now 12min of processing time just to create a single archive on a local hard disk seems excessive.

  5. #15
    heinzg is offline Loyal Member
    Join Date
    Jan 2008
    Location
    Germany
    Posts
    83
    Rep Power
    7

    Default

    Hi Stergil!

    Thanks for your post, and feed back!

    To answer some of your questions in short and to ask some questions to.

    Good I see you got mail installed and working. Did the script not print out the following when not finding mailx?
    "For a "Howto" install mailx without postfix please visit the link below"
    "http://www.zimbra.com/forums/administrators/13528-sending-email-comand-line-logwatch-ubuntu-6-06-a.html#post70636"
    About dar. Dar is my choice of archive due to the the list of features it has in comparison to tar, and the list is well long, e.g. one of the features of dar is the ability to recover single files from the archive with out having to unpack the whole archive. It also has encryption builtin which I will add to the script in the next release for better security. After all we don't want our tapes and dvd being copied by others with intentions we might find offensive!

    But the main reason to use dar is for the neat way it make full and then differential backups. Meaning you save a lot of space on your storage media. I have setup dar to make compressed backups archives (gzip -9 level) you can change this be changing COMPRESS=9 to a lower amount.
    This is also one of the reasons why it seams that dar is hanging on your system. At over 2GB of data it takes quite a bit of time to create the initial catalogue, and secondly the script is running dar as a background proc in non interactive mode. So it is normal for dar to sit there for a bit seeming not to do anything.
    On your spec system and data volume I can imagine that with compression on full and all IO on the same disk, that it can take up the good part of 35 to 50 min for a full backup.
    After making a Full backup you now only have to make differential backups to capture and save only file that have changed from those in the full backup archive, plus it is a lot faster now. I would guess on your system depending on the amount of spam you get a day 5 to 10 min and about 60MB data. So to recap you save lots of space on long term storage media through compression and differential backups, Dar has also been instructed not to compress common compressed files thus a performance increase for you.
    As for a increased in performance you would have to put your SYNC_DIR on an other disk than where your ARCHIVEDIR is, the basic IO story.
    OH yes the script starts dar with “nice 19” (change 19 -> 0 for more speed) which is the lowest priority... not to hinder Zimbra server from doing important mailing work :-)

    About there not being any files in TOO_MEDIA_DIR has to do with my logic of doing the backups. The script works on a weekly rotation e.g. On Monday morning it makes a full backup and then every 24 hour there after a differential backup. Until it gets to Monday morning again where it will then transfer the backups from the last week to TOO_MEDIA_DIR where you can then write it to tape or DVD what ever. But the full backup archive should stay in ARCHIVEDIR as dar compares the files from the SYNC_DIR with those in the full backup archive to then create a differential backup file.
    So if you use my crontab suggestion you will find files in TOO_MEDIA_DIR one week after running the script for the first time.

    The bit about the LOG file. This is sort of a hack if you wish... as the script is/or should mainly be used as a root corn job you should redirect all out put from the script into the same file as stated in LOG=.
    Crontab example (run the script with -h to see the right statement for your LOG entry in the contab):
    0 3 * * 1 /bin/sh /path to script/zmbac.sh -f > /path to log dir/file.log 2>&1
    0 3 * * 2-7 /bin/sh /root/zmbac.sh -d >> /path to log dir/file.log 2>&1
    If you don't do the redirect then you will land up this mails with a subject but with no body, which should actually contain the contents of the log file.

    A sample print out of the weekly full backup should look something like this, note this system also took 38 min for the task to complete (time for a new office server...):


    Full Backup started at: 03:00

    Old backups found...old week= 08 current week= 09

    /data/Backup/zimbra_dars/08_Zimbra_Backup_01-March-2008_DIFF.1.dar moved to /data/Backup/burn/
    /data/Backup/zimbra_dars/08_Zimbra_Backup_02-March-2008_DIFF.1.dar moved to /data/Backup/burn/
    /data/Backup/zimbra_dars/08_Zimbra_Backup_25-February-2008_FULL.1.dar moved to /data/Backup/burn/
    /data/Backup/zimbra_dars/08_Zimbra_Backup_26-February-2008_DIFF.1.dar moved to /data/Backup/burn/
    /data/Backup/zimbra_dars/08_Zimbra_Backup_27-February-2008_DIFF.1.dar moved to /data/Backup/burn/
    /data/Backup/zimbra_dars/08_Zimbra_Backup_28-February-2008_DIFF.1.dar moved to /data/Backup/burn/
    /data/Backup/zimbra_dars/08_Zimbra_Backup_29-February-2008_DIFF.1.dar moved to /data/Backup/burn/

    Printing date: 03-March-2008 and software version: *into /opt/zimbra/DATE_VERSION.txt

    Doing a hotsync of /opt/zimbra/ to /mnt/fakebackup/
    sync went ok

    Stopping the Zimbra server
    Host my.mail.host
    Stopping stats...Done
    Stopping mta...Done
    Stopping spell...Done
    Stopping snmp...Done
    Stopping archiving...Done
    Stopping antivirus...Done
    Stopping antispam...Done
    Stopping imapproxy...Done
    Stopping mailbox...Done
    Stopping logger...Done
    Stopping ldap...Done

    Zimbra server has stopped

    kill any left over zimbra procs

    Doing a fast cold sync of /opt/zimbra/ & /mnt/fakebackup/ while the Zimbra server is down
    sync went ok

    Host my.mail.host
    Starting ldap...Done.
    Starting logger...Done.
    Starting mailbox...Done.
    Starting antispam...Done.
    Starting antivirus...Done.
    Starting snmp...Done.
    Starting spell...Done.
    Starting mta...Done.
    Starting stats...Done.

    Zimbra has started
    Stats seems to be running fine...

    Writing a full backup called:09_Zimbra_Backup_03-March-2008_FULL in: /data/Backup/zimbra_dars/ with file sizes of max: 4695M


    *--------------------------------------------
    *35430 inode(s) saved
    *with 1454 hard link(s) recorded
    *0 inode(s) not saved (no file change)
    *0 inode(s) failed to save (filesystem error)
    *0 files(s) ignored (excluded by filters)
    *0 files(s) recorded as deleted from reference backup
    *--------------------------------------------
    *Total number of file considered: 35430
    *--------------------------------------------
    *EA saved for 0 file(s)
    *--------------------------------------------
    Archive Created successfully


    :::::::::::::::::::::::::::::::::::::::::::::::::: :::::::::::::::::::::::::
    Full Zimbra Backup ended at: 03:38
    The the full backup took Hr:0 Min:38 Sec:39 to complete
    :::::::::::::::::::Cheers Osoffice.de for the script:::::::::::::::::::::::
    I hope this helps you some what!

    Best regards
    Heinzg

  6. #16
    heinzg is offline Loyal Member
    Join Date
    Jan 2008
    Location
    Germany
    Posts
    83
    Rep Power
    7

    Default

    Hi

    Added a Redhat compatible version on the download site, for now. Till the next release is out where all distros should be compatible (I hope).

    Cheers
    heinzg

    P.S. thank you for your feed back!

  7. #17
    madunix is offline Senior Member
    Join Date
    Feb 2008
    Location
    Germany
    Posts
    65
    Rep Power
    7

    Default

    Can you please explain in details how to use it on Redhat RHEL5 64bit with zimbra Opensource version, to backup and restore of mail boxes using the above script.

  8. #18
    Stergil is offline Senior Member
    Join Date
    May 2007
    Posts
    71
    Rep Power
    8

    Default

    Quote Originally Posted by heinzg View Post
    Good I see you got mail installed and working. Did the script not print out the following when not finding mailx?
    "For a "Howto" install mailx without postfix please visit the link below"
    "http://www.zimbra.com/forums/administrators/13528-sending-email-comand-line-logwatch-ubuntu-6-06-a.html#post70636"
    Hmm, not that I recall seeing. Grant it it was about 4am and I could have missed it

    I didn't realize the benefits of dar, thanks for the info. You were right the full backup took about an hour, but the differentials only took ~10min.

    So far things seem to be working fine. Differentials are being created, etc.

    I appreciate the extra explanation on things, it was very helpful.

  9. #19
    dwmtractor's Avatar
    dwmtractor is offline Moderator
    Join Date
    Jul 2007
    Location
    San Jose, CA
    Posts
    1,027
    Rep Power
    9

    Default

    heinzg,

    I'm not enough of a shell script guru to analyze all you have done, but from what I have read of it so far, my hat is off to you for an extremely well-thought-out script. It's great to have the development site you linked, but if I could presume to make a suggestion, it'd be nice if you would post either the script, or at least a brief discussion with link to your development page, on the wiki under the open-source backup section (here) so that when people are reading about backup methodologies, this one is added to the mix. This will make it more likely (1) that those who need it find your script, and (2) that you'll get a wider variety of people testing and providing feedback to you. Kinda what we call a win-win!

    Kudos for your hard work!

    Dan

  10. #20
    cornbread is offline Loyal Member
    Join Date
    Sep 2007
    Posts
    80
    Rep Power
    7

    Default I want to push backups to remote ssh location

    How do I do this? I already have the destination set up to do automatic ssh authentication from the machine. Which part of the script would I modify to do rsync via ssh?

Page 2 of 66 FirstFirst 12341252 ... LastLast

Thread Information

Users Browsing this Thread

There are currently 4 users browsing this thread. (0 members and 4 guests)

Similar Threads

  1. Zimbra Backup Script ... Small Problem
    By frankb in forum Administrators
    Replies: 12
    Last Post: 07-14-2008, 08:40 AM
  2. Debian - Upgrade from 4.5.6
    By magikman in forum Installation
    Replies: 1
    Last Post: 01-19-2008, 12:59 PM
  3. Backup script issues
    By SSS in forum Administrators
    Replies: 18
    Last Post: 12-03-2007, 09:56 AM
  4. 4.0.4 to 4.5.6 upgrade failed in network edition
    By chenthil in forum Administrators
    Replies: 1
    Last Post: 08-27-2007, 09:36 AM
  5. [SOLVED] Simple backup question...
    By dameron in forum Administrators
    Replies: 3
    Last Post: 08-25-2007, 09:36 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •