Page 20 of 66 FirstFirst ... 10181920212230 ... LastLast
Results 191 to 200 of 659

Thread: [SOLVED] Yet Another Backup Script for Community Version

  1. #191
    liverpoolfcfan's Avatar
    liverpoolfcfan is offline Outstanding Member
    Join Date
    Oct 2009
    Location
    Dublin, IRELAND
    Posts
    710
    Rep Power
    6

    Default

    Quote Originally Posted by chewitt View Post
    I made some minor tweaks to the script. To apply them copy the code below and paste them over the original section around line 750:

    .....

    a) I've altered the naming of the backuplist file to follow the format used by the data files, e.g. 38_Zimbra_Backup_03-October-2009_LIST.txt instead of Backuplist-03-October-2009.txt. Doing this ensures the files are matched and relocated along with the .dar and .md5 files when a full backup is run, instead of leaving them orphaned to clutter up the current backup directory.

    .....

    Christian
    I like your idea of rationalizing the names of the report files.

    I have taken your idea a little bit further to make it easier to maintain, and easier to read the script.

    Around line 500 of the script is the beginning of the do_backup function

    Code:
    function do_backup {
    	TYPE=$1
    	if [ $TYPE = "full" ]
    	then
    		PREFIX=$FULL_PREFIX
    	elif [ $TYPE = "diff" ]
    	then
    		PREFIX=$DIFF_PREFIX
    	else
    		echo "Invalid Backup Type!"
    		exit 1
    	fi	
    	
    *** Add the following line here ****
    ARCHIVENAME="$BACKUPWEEK"_"$BACKUPNAME"_"$BACKUPDATE"_"$PREFIX"
    Just after the backup type has been determined, add a line to save the entire ARCHIVENAME - i.e. the Base Name of the Backup Archive to a variable (eg. 44_Zimbra_Backup_04-November-2009_FULL)

    Then at the point where the backup archives and reporting are being created, replace all the long string concatenations with this one variable name. All that needs to be added are the extensions for the DAR archives ( .1.dar) or the REPORTS (.txt), etc.

    Saving archive to DAR fiile(s) begins around line 670 ...

    Replace all instances of "$BACKUPWEEK"_"$BACKUPNAME"_"$BACKUPDATE"_"$PREFIX " below that point with "$ARCHIVENAME"

    Here is what the subsequent code looks like.

    Code:
      echo "Writing a $TYPE backup: ""$ARCHIVENAME"
    	echo "into: $ARCHIVEDIR with file sizes of max: $ARCHIVESIZE"
    	cd $SYNC_DIR
    	if [ "$CRYPT" = "yes" ]
    	then
    	    KEY=`cat "$PASSDIR""$PASSFILE"`
    	    echo "Saving Encrypted Archive..."
    		if [ "$TYPE" = "full" ]
    		then
    			nice -19 $DAR_BIN -K bf:$KEY -s $ARCHIVESIZE -z$COMPRESS -Z "*.gz" -Z "*.zip"\
    			-Z "*.bz2" -Z "*.tgz" -Z "*.zgz" -Z "*.jar" -Z "*.tiff" \
    			-Z "*.jpg" -Z "*.png" -Z "*.gif" -Z "*.jpeg" -R `pwd` \
    			-c "$ARCHIVEDIR""$ARCHIVENAME" -Q
    		elif [ "$TYPE" = "diff" ]
    		then
    		    nice -19 $DAR_BIN -J bf:$KEY -s $ARCHIVESIZE -z$COMPRESS -Z "*.gz" -Z "*.zip"\
    			-Z "*.bz2" -Z "*.tgz" -Z "*.zgz" -Z "*.jar" -Z "*.tiff" \
    			-Z "*.jpg" -Z "*.png" -Z "*.gif" -Z "*.jpeg" -R `pwd` \
    			-c "$ARCHIVEDIR""$ARCHIVENAME" -Q\
    			-A "$CURRENTFULL" -Q
    		else
    			echo "Unkown Backup Type. Fail."
    			mail_log
    			exit 1
    		fi
    	else        
    	    echo "Saving Unencrtyped Archive..."
    		if [ "$TYPE" = "full" ]
    		then
    			nice -19 $DAR_BIN -s $ARCHIVESIZE -z$COMPRESS -Z "*.gz" -Z "*.zip"\
    		    -Z "*.bz2" -Z "*.tgz" -Z "*.zgz" -Z "*.jar" -Z "*.tiff" \
    		    -Z "*.jpg" -Z "*.png" -Z "*.gif" -Z "*.jpeg" -R `pwd` \
    		    -c "$ARCHIVEDIR""$ARCHIVENAME" -Q
    		elif [ "$TYPE" = "diff" ]
    		then
    	    	nice -19 $DAR_BIN -s $ARCHIVESIZE -z$COMPRESS -Z "*.gz" -Z "*.zip"\
    		    -Z "*.bz2" -Z "*.tgz" -Z "*.zgz" -Z "*.jar" -Z "*.tiff" \
    		    -Z "*.jpg" -Z "*.png" -Z "*.gif" -Z "*.jpeg" -R `pwd` \
    		    -c "$ARCHIVEDIR""$ARCHIVENAME" -Q\
    		    -A "$CURRENTFULL" -Q
    		else
    			echo "Unkown Backup Type. Fail."
    			mail_log
    			exit 1
    		fi
    	fi
        if [ "$?" -ne "0" ]
        then
            echo "Dar had a problem!"
            mail_log
            exit 1
        else
    	    # Create MD5 Checksums to verify archives after writing to media or network transfers
    	    cd $ARCHIVEDIR
    	    FILENAME=`ls -A "$ARCHIVENAME"*`
    	    if [ -e $FILENAME ]
    	    then 
    	    	echo "Creating MD5 Checksum for $FILENAME..."
    		    $MD5SUM_BIN -b $FILENAME > "$FILENAME".md5
    		    if [ "$?" -ne "0" ]
    		    then
    		        echo "MD5 Checksum failed!"
    		        mail_log
    		        exit 1
    		    fi
    	    else
    	    	echo "$FILENAME not found!"
    			echo "This should not happen"
    			mail_log
    			exit 1
    	    fi
    	fi
    	# DRCP Section. To scp newly created archives to a remote system
    	if [ "$DRCP" = "yes" ]
    	then
    	    CPNAME=`ls -A "$ARCHIVENAME"*`
    	    echo "copy archive to $REMOTEHOST" remote directory $REMOTEDIR
    	    scp -i /root/.ssh/id_rsa $CPNAME "$SSHUSER"@"$REMOTEHOST":"$REMOTEDIR"
    		if [ "$?" -ne "0" ]
    		then
    		    echo "Error copying archive and checksum to $REMOTEHOST"
    		    mail_log
    		    exit 1
            fi 
    	fi
        
      # over view of all the files which where backed up
      echo "Creating file listing from archive..."
      if [ "$CRYPT" = "yes" ]
    	then
    	    KEY=`cat "$PASSDIR""$PASSFILE"`
    	    nice -19 $DAR_BIN -K bf:$KEY -l "$ARCHIVEDIR""$ARCHIVENAME" -Q\
    	    > "$ARCHIVEDIR""$ARCHIVENAME".txt && gzip -9 "$ARCHIVEDIR""$ARCHIVENAME".txt
    	else        
    	    nice -19 $DAR_BIN -l "$ARCHIVEDIR""$ARCHIVENAME" -Q\
    	    > "$ARCHIVEDIR""$ARCHIVENAME".txt && gzip -9 "$ARCHIVEDIR""$ARCHIVENAME".txt
    	fi
        # Script Timer
        STOPTIME=(`date +%s`)
        RUNTIME=$(expr $STOPTIME \- $STARTTIME)
        hours=$(($RUNTIME / 3600))
        seconds=$(($RUNTIME  % 3600))
        minutes=$(($RUNTIME  / 60))
        seconds=$(($RUNTIME  % 60))
        echo
        echo "::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::" 
        echo "$TYPE Zimbra Backup ended at: `date +%H:%M`" 
        echo "Backup took Hr:$hours Min:$minutes Sec:$seconds to complete" 
        echo ":::::::::::::::Cheers Osoffice for the script:::::::::::::::::::::::"
        (cat $LOG; $UUENCODE_BIN "$ARCHIVEDIR""$ARCHIVENAME".txt.gz "$ARCHIVEDIR""$ARCHIVENAME".txt.gz) \
        | mail -c $EMAILCC -s "Zimbra $TYPE Backup Log on $HOSTNAME" $EMAIL
    }
    Last edited by liverpoolfcfan; 11-05-2009 at 04:11 AM.

  2. #192
    liverpoolfcfan's Avatar
    liverpoolfcfan is offline Outstanding Member
    Join Date
    Oct 2009
    Location
    Dublin, IRELAND
    Posts
    710
    Rep Power
    6

    Default

    For the bogus time reporting for durations over 1 hour changing "seconds" to "RUNTIME" in the 3rd last line should fix it.

    Change

    Code:
        # Script Timer
        STOPTIME=(`date +%s`)
        RUNTIME=$(expr $STOPTIME \- $STARTTIME)
        hours=$(($RUNTIME / 3600))
        seconds=$(($RUNTIME  % 3600))
        minutes=$(($RUNTIME  / 60))
        seconds=$(($RUNTIME  % 60))
    to

    Code:
        # Script Timer
        STOPTIME=(`date +%s`)
        RUNTIME=$(expr $STOPTIME \- $STARTTIME)
        hours=$(($RUNTIME / 3600))
        RUNTIME=$(($RUNTIME  % 3600))
        minutes=$(($RUNTIME  / 60))
        seconds=$(($RUNTIME  % 60))

  3. #193
    borngunners is offline Advanced Member
    Join Date
    Jun 2009
    Posts
    195
    Rep Power
    6

    Default

    liverpoolfcfan. I am sorry about your champions league trauma. Back to your script from your first post. Is it the solution to the DAR problem -A option that I am having or for something else? This is what my script looks like:
    function do_backup {
    TYPE=$1
    if [ $TYPE = "full" ]
    then
    PREFIX=$FULL_PREFIX
    elif [ $TYPE = "diff" ]
    then
    PREFIX=$DIFF_PREFIX
    else
    echo "Invalid Backup Type!"
    exit 1
    fi
    # Check for required directories and helper apps
    for REQ_BIN in $DAR_BIN $MAILX_BIN $RSYNC_BIN $SSH_BIN $MD5SUM_BIN $UUENCODE_BIN
    do
    check_req_bin $REQ_BIN
    done

    for REQ_DIR in $ZM_HOME $SYNC_DIR $ARCHIVEDIR $TO_MEDIA_DIR
    do
    check_req_dir $REQ_DIR
    done
    echo "$TYPE Backup started at: `date +%H:%M`"
    # Check to make sure we have not already done a backup today.
    CURRENTNAME2=`ls -A -1 "$ARCHIVEDIR""$BACKUPWEEK"_"$BACKUPNAME"_"$BACKUPD ATE"_"$PREFIX"*dar 2> /dev/null | head -qn1`
    CURRENTFULL=`ls -A -1 "$ARCHIVEDIR""$BACKUPWEEK"*$FULL_PREFIX*dar 2>/dev/null | cut -d . -f1 `

    if [ -f "$CURRENTNAME2" ]
    then
    echo "Full Zimbra Backup failed! FOUND A BACKUP WITH SAME NAME"
    echo "Please check why! You should only run this script once a day with the current backup date settings!"
    mail_log
    exit 1
    elif [ $CURRENTFULL ] && [ $TYPE = "full" ]
    then
    # Look for old backups and put then in directory from where you write them to some form of
    # storage media
    LAST_FULL_DAR=`ls -A -1 $ARCHIVEDIR | grep $BACKUPNAME | cut -d _ -f1 | head -n1`

    if [ -z "$LAST_FULL_DAR" ]
    then
    echo "No old backups found"
    elif [ "$LAST_FULL_DAR" -lt "$BACKUPWEEK" ]
    then
    echo "Old backups found...old week= $LAST_FULL_DAR current week= $BACKUPWEEK"
    echo
    for i in `ls -A -1 "$ARCHIVEDIR""$LAST_FULL_DAR"_"$BACKUPNAME"*`
    Please let me know if there is anything in my script causing that to happen. Also, how do I make it write to my dvd/rw without me involving. I have already setup a remote backup.

    Thanks,

  4. #194
    liverpoolfcfan's Avatar
    liverpoolfcfan is offline Outstanding Member
    Join Date
    Oct 2009
    Location
    Dublin, IRELAND
    Posts
    710
    Rep Power
    6

    Default

    Quote Originally Posted by borngunners View Post
    liverpoolfcfan. I am sorry about your champions league trauma. Back to your script from your first post. Is it the solution to the DAR problem -A option that I am having or for something else? This is what my script looks like:


    Please let me know if there is anything in my script causing that to happen. Also, how do I make it write to my dvd/rw without me involving. I have already setup a remote backup.

    Thanks,
    This was not in response to any issues you have raised. It was a response to the quoted post regarding tweaks to the existing script made by chewitt

    I will likely be looking into DVD backups myself soon too - so if I find anything out I will let you know.

  5. #195
    liverpoolfcfan's Avatar
    liverpoolfcfan is offline Outstanding Member
    Join Date
    Oct 2009
    Location
    Dublin, IRELAND
    Posts
    710
    Rep Power
    6

    Default

    Quote Originally Posted by borngunners View Post
    I am still stuck with the email issues and the DAR issue with the -A option. Can someone help me resolve these issues. I have successfully created the ssh setup to a remote location, but becauise of the DAR issue the backup is not being transfered to the remote location. Below is the error message:
    OK - I'm not an expert on this stuff. I am quite new to shell scripting too myself. But trying out the various commands in the script, I think I can see some of what the OP was trying to do.

    The -A option error on the DAR command line would indicate that there was a problem with the final section of

    Code:
    		    nice -19 $DAR_BIN -J bf:$KEY -s $ARCHIVESIZE -z$COMPRESS -Z "*.gz" -Z "*.zip"\
    			-Z "*.bz2" -Z "*.tgz" -Z "*.zgz" -Z "*.jar" -Z "*.tiff" \
    			-Z "*.jpg" -Z "*.png" -Z "*.gif" -Z "*.jpeg" -R `pwd` \
    			-c "$ARCHIVEDIR""$ARCHIVENAME" -Q\
    			-A "$CURRENTFULL" -Q
    so DAR did not like what it saw in the $CURRENTFULL variable.

    Looking back up to where $CURRENTFULL gets it's value - it is set from

    Code:
        CURRENTFULL=`ls -A -1 "$ARCHIVEDIR""$BACKUPWEEK"*$FULL_PREFIX*dar 2>/dev/null | cut -d . -f1 `
    It seems that there are two reasons for this (and correct me if I am wrong someone) is to
    (1) to verify that there is a FULL backup in the current week's folder on which to base a DIFF
    (2) get the Base Name of the FULL backup in the current backups folder in order to pass it to the DAR command

    If you run this command with just one ....FULL.1.dar file in the folder it returns a single base file name. However, if your backup goes onto a second file and you have a ....FULL.2.dar the command returns 2 lines, etc.

    I believe changing the *dar in the command to .1.dar would actually fix this problem, and allow the script to continue. By definition every backup will always have exactly one .1.dar - so it will either exist or not, and it will only exist once - so will not supply multiple base filenames to the -A option on the DAR command line

    Code:
        CURRENTFULL=`ls -A -1 "$ARCHIVEDIR""$BACKUPWEEK""$FULL_PREFIX".1.dar 2>/dev/null | cut -d . -f1 `

    Will look further into the media section to see if can see anything else that might go wrong afterwards.

    Hope this helps

  6. #196
    liverpoolfcfan's Avatar
    liverpoolfcfan is offline Outstanding Member
    Join Date
    Oct 2009
    Location
    Dublin, IRELAND
    Posts
    710
    Rep Power
    6

    Post

    Looks like another issue at the MD5 section of code ...

    Code:
    	    # Create MD5 Checksums to verify archives after writing to media or network transfers
    	    cd $ARCHIVEDIR
    	    FILENAME=`ls -A "$BACKUPWEEK"_"$BACKUPNAME"_"$BACKUPDATE"_"$PREFIX"*`
    	    if [ -e $FILENAME ]
    	    then 
    	    	echo "Creating MD5 Checksum for $FILENAME..."
    		    $MD5SUM_BIN -b $FILENAME > "$FILENAME".md5
    		    if [ "$?" -ne "0" ]
    		    then
    		        echo "MD5 Checksum failed!"
    		        mail_log
    		        exit 1
    		    fi
    	    else
    	    	echo "$FILENAME not found!"
    			echo "This should not happen"
    			mail_log
    			exit 1
    	    fi
    This needs a loop around it I believe in case the backup runs to more than one file.

    I think this should work - but cannot test it right now myself.

    Where FILENAME is currently assigned just once - make it get assigned from a loop. and add the do and done markers to the lines that need to be looped through.


    Code:
    	    # Create MD5 Checksums to verify archives after writing to media or network transfers
    	    cd $ARCHIVEDIR
    
    
    for FILENAME in `ls -A "$BACKUPWEEK"_"$BACKUPNAME"_"$BACKUPDATE"_"$PREFIX"*`
    do
    	    if [ -e $FILENAME ]
    	    then 
    	    	echo "Creating MD5 Checksum for $FILENAME..."
    		    $MD5SUM_BIN -b $FILENAME > "$FILENAME".md5
    		    if [ "$?" -ne "0" ]
    		    then
    		        echo "MD5 Checksum failed!"
    		        mail_log
    		        exit 1
    		    fi
    	    else
    	    	echo "$FILENAME not found!"
    			echo "This should not happen"
    			mail_log
    			exit 1
    	    fi
    done

  7. #197
    borngunners is offline Advanced Member
    Join Date
    Jun 2009
    Posts
    195
    Rep Power
    6

    Default

    Thanks Liverpoolfcfan. I will try it and see. I will let you know what happen...
    Quote Originally Posted by liverpoolfcfan View Post
    OK - I'm not an expert on this stuff. I am quite new to shell scripting too myself. But trying out the various commands in the script, I think I can see some of what the OP was trying to do.

    The -A option error on the DAR command line would indicate that there was a problem with the final section of

    Code:
    		    nice -19 $DAR_BIN -J bf:$KEY -s $ARCHIVESIZE -z$COMPRESS -Z "*.gz" -Z "*.zip"\
    			-Z "*.bz2" -Z "*.tgz" -Z "*.zgz" -Z "*.jar" -Z "*.tiff" \
    			-Z "*.jpg" -Z "*.png" -Z "*.gif" -Z "*.jpeg" -R `pwd` \
    			-c "$ARCHIVEDIR""$ARCHIVENAME" -Q\
    			-A "$CURRENTFULL" -Q
    so DAR did not like what it saw in the $CURRENTFULL variable.

    Looking back up to where $CURRENTFULL gets it's value - it is set from

    Code:
        CURRENTFULL=`ls -A -1 "$ARCHIVEDIR""$BACKUPWEEK"*$FULL_PREFIX*dar 2>/dev/null | cut -d . -f1 `
    It seems that there are two reasons for this (and correct me if I am wrong someone) is to
    (1) to verify that there is a FULL backup in the current week's folder on which to base a DIFF
    (2) get the Base Name of the FULL backup in the current backups folder in order to pass it to the DAR command

    If you run this command with just one ....FULL.1.dar file in the folder it returns a single base file name. However, if your backup goes onto a second file and you have a ....FULL.2.dar the command returns 2 lines, etc.

    I believe changing the *dar in the command to .1.dar would actually fix this problem, and allow the script to continue. By definition every backup will always have exactly one .1.dar - so it will either exist or not, and it will only exist once - so will not supply multiple base filenames to the -A option on the DAR command line

    Code:
        CURRENTFULL=`ls -A -1 "$ARCHIVEDIR""$BACKUPWEEK""$FULL_PREFIX".1.dar 2>/dev/null | cut -d . -f1 `

    Will look further into the media section to see if can see anything else that might go wrong afterwards.

    Hope this helps

  8. #198
    Waffles is offline Intermediate Member
    Join Date
    Nov 2009
    Posts
    16
    Rep Power
    5

    Default

    I'm also getting the:
    Parse error on command line (or included files): Invalid argument for -A option
    Dar had a problem!

    error. I think I know where the problem is but I'm not sure how to correct it.
    I have my crontab setup to run the full backup on Saturday at 12:00am and incrementals every other day at 12:00am:

    0 0 * * 6 /bin/bash /root/scripts/zmbak_v.0.5.sh -f > /var/log/zim_backup.log 2>&1
    0 0 * * 0-5 /bin/bash /root/scripts/zmbak_v.0.5.sh -d >> /var/log/zim_backup.log 2>&1
    Saturday ran fine, sunday ran fine, but Monday didn't. I'm thinking it is the BACKUPWEEK variable.

    BACKUPWEEK=`date +%W` # Week prefix used for backup weekly rotation and naming
    I'm thinking it is looking for the full backup filename with the wrong BACKUPWEEK in it.

    I suppose we could just have the full backup run on Monday at 12:00am, but because of the network file transfer we do, I would rather have that done friday night.

    any thoughts?

  9. #199
    liverpoolfcfan's Avatar
    liverpoolfcfan is offline Outstanding Member
    Join Date
    Oct 2009
    Location
    Dublin, IRELAND
    Posts
    710
    Rep Power
    6

    Default

    Quote Originally Posted by Waffles View Post
    I'm thinking it is looking for the full backup filename with the wrong BACKUPWEEK in it.

    I suppose we could just have the full backup run on Monday at 12:00am, but because of the network file transfer we do, I would rather have that done friday night.

    any thoughts?
    You are correct. It is the fact that the week changes on Monday, and the script looks for a full backup for the current week on which to base the differential backup.

    EDIT: Revised recommendation below ...

    Took a look at the script and you could add something like the following into it.

    Running with -f option would create or update the file with the "current" week for whatever date you run the script. All backups - whether that full or subsequent diff backups - would use the contents of that file as the BACKUPWEEK patameter. Note: be sure to use a file/folder that cron can write to. I tested this outside of cron - didn't test it within cron.

    Code:
    BACKUPDATE=`date +%Y%m%d`       # date format used in archive names
    # BACKUPWEEK=`date +%W`        # Week prefix used for backup weekly rotation and naming
    # VJS - Commented out above, and added below to allow Weekly baseline to be any day of week.
    #          Needs full set of tests to be run, including archiving old weeks data, etc.
    BACKUPWEEKFILE="/opt/zimbra/backup.week"
    case $1 in
    -f | --full)
    date +%W > $BACKUPWEEKFILE
    ;;
    -d | --diff)
        if [ ! -e "$BACKUPWEEKFILE" ]
        then
            echo Backup Week identifier file "$BACKUPWEEKFILE" not found !
            echo You must run a FULL backup before a DIFF backup
            exit 1
        fi
    ;;
    *)
    ;;
    esac
    BACKUPWEEK=`cat $BACKUPWEEKFILE`        # Week prefix used for backup weekly rotation and naming
    Last edited by liverpoolfcfan; 11-25-2009 at 08:35 AM.

  10. #200
    Waffles is offline Intermediate Member
    Join Date
    Nov 2009
    Posts
    16
    Rep Power
    5

    Default

    Seems like a pretty straightforward solution!

    I'll let you know how it goes! I'll set cron to do the first backup on friday and we'll go from there.

Page 20 of 66 FirstFirst ... 10181920212230 ... LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Zimbra Backup Script ... Small Problem
    By frankb in forum Administrators
    Replies: 12
    Last Post: 07-14-2008, 08:40 AM
  2. Debian - Upgrade from 4.5.6
    By magikman in forum Installation
    Replies: 1
    Last Post: 01-19-2008, 12:59 PM
  3. Backup script issues
    By SSS in forum Administrators
    Replies: 18
    Last Post: 12-03-2007, 09:56 AM
  4. 4.0.4 to 4.5.6 upgrade failed in network edition
    By chenthil in forum Administrators
    Replies: 1
    Last Post: 08-27-2007, 09:36 AM
  5. [SOLVED] Simple backup question...
    By dameron in forum Administrators
    Replies: 3
    Last Post: 08-25-2007, 09:36 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •