[SOLVED] Yet Another Backup Script for Community Version

[liverpoolfcfan]

Originally Posted by chewitt

I made some minor tweaks to the script. To apply them copy the code below and paste them over the original section around line 750:

.....

a) I've altered the naming of the backuplist file to follow the format used by the data files, e.g. 38_Zimbra_Backup_03-October-2009_LIST.txt instead of Backuplist-03-October-2009.txt. Doing this ensures the files are matched and relocated along with the .dar and .md5 files when a full backup is run, instead of leaving them orphaned to clutter up the current backup directory.

.....

Christian

I like your idea of rationalizing the names of the report files.

I have taken your idea a little bit further to make it easier to maintain, and easier to read the script.

Around line 500 of the script is the beginning of the do_backup function

function do_backup {
	TYPE=$1
	if [ $TYPE = "full" ]
	then
		PREFIX=$FULL_PREFIX
	elif [ $TYPE = "diff" ]
	then
		PREFIX=$DIFF_PREFIX
	else
		echo "Invalid Backup Type!"
		exit 1
	fi	
	
*** Add the following line here ****
ARCHIVENAME="$BACKUPWEEK"_"$BACKUPNAME"_"$BACKUPDATE"_"$PREFIX"

Just after the backup type has been determined, add a line to save the entire ARCHIVENAME - i.e. the Base Name of the Backup Archive to a variable (eg. 44_Zimbra_Backup_04-November-2009_FULL)

Then at the point where the backup archives and reporting are being created, replace all the long string concatenations with this one variable name. All that needs to be added are the extensions for the DAR archives ( .1.dar) or the REPORTS (.txt), etc.

Saving archive to DAR fiile(s) begins around line 670 ...

Replace all instances of "$BACKUPWEEK"_"$BACKUPNAME"_"$BACKUPDATE"_"$PREFIX " below that point with "$ARCHIVENAME"

Here is what the subsequent code looks like.

  echo "Writing a $TYPE backup: ""$ARCHIVENAME"
	echo "into: $ARCHIVEDIR with file sizes of max: $ARCHIVESIZE"
	cd $SYNC_DIR
	if [ "$CRYPT" = "yes" ]
	then
	    KEY=`cat "$PASSDIR""$PASSFILE"`
	    echo "Saving Encrypted Archive..."
		if [ "$TYPE" = "full" ]
		then
			nice -19 $DAR_BIN -K bf:$KEY -s $ARCHIVESIZE -z$COMPRESS -Z "*.gz" -Z "*.zip"\
			-Z "*.bz2" -Z "*.tgz" -Z "*.zgz" -Z "*.jar" -Z "*.tiff" \
			-Z "*.jpg" -Z "*.png" -Z "*.gif" -Z "*.jpeg" -R `pwd` \
			-c "$ARCHIVEDIR""$ARCHIVENAME" -Q
		elif [ "$TYPE" = "diff" ]
		then
		    nice -19 $DAR_BIN -J bf:$KEY -s $ARCHIVESIZE -z$COMPRESS -Z "*.gz" -Z "*.zip"\
			-Z "*.bz2" -Z "*.tgz" -Z "*.zgz" -Z "*.jar" -Z "*.tiff" \
			-Z "*.jpg" -Z "*.png" -Z "*.gif" -Z "*.jpeg" -R `pwd` \
			-c "$ARCHIVEDIR""$ARCHIVENAME" -Q\
			-A "$CURRENTFULL" -Q
		else
			echo "Unkown Backup Type. Fail."
			mail_log
			exit 1
		fi
	else        
	    echo "Saving Unencrtyped Archive..."
		if [ "$TYPE" = "full" ]
		then
			nice -19 $DAR_BIN -s $ARCHIVESIZE -z$COMPRESS -Z "*.gz" -Z "*.zip"\
		    -Z "*.bz2" -Z "*.tgz" -Z "*.zgz" -Z "*.jar" -Z "*.tiff" \
		    -Z "*.jpg" -Z "*.png" -Z "*.gif" -Z "*.jpeg" -R `pwd` \
		    -c "$ARCHIVEDIR""$ARCHIVENAME" -Q
		elif [ "$TYPE" = "diff" ]
		then
	    	nice -19 $DAR_BIN -s $ARCHIVESIZE -z$COMPRESS -Z "*.gz" -Z "*.zip"\
		    -Z "*.bz2" -Z "*.tgz" -Z "*.zgz" -Z "*.jar" -Z "*.tiff" \
		    -Z "*.jpg" -Z "*.png" -Z "*.gif" -Z "*.jpeg" -R `pwd` \
		    -c "$ARCHIVEDIR""$ARCHIVENAME" -Q\
		    -A "$CURRENTFULL" -Q
		else
			echo "Unkown Backup Type. Fail."
			mail_log
			exit 1
		fi
	fi
    if [ "$?" -ne "0" ]
    then
        echo "Dar had a problem!"
        mail_log
        exit 1
    else
	    # Create MD5 Checksums to verify archives after writing to media or network transfers
	    cd $ARCHIVEDIR
	    FILENAME=`ls -A "$ARCHIVENAME"*`
	    if [ -e $FILENAME ]
	    then 
	    	echo "Creating MD5 Checksum for $FILENAME..."
		    $MD5SUM_BIN -b $FILENAME > "$FILENAME".md5
		    if [ "$?" -ne "0" ]
		    then
		        echo "MD5 Checksum failed!"
		        mail_log
		        exit 1
		    fi
	    else
	    	echo "$FILENAME not found!"
			echo "This should not happen"
			mail_log
			exit 1
	    fi
	fi
	# DRCP Section. To scp newly created archives to a remote system
	if [ "$DRCP" = "yes" ]
	then
	    CPNAME=`ls -A "$ARCHIVENAME"*`
	    echo "copy archive to $REMOTEHOST" remote directory $REMOTEDIR
	    scp -i /root/.ssh/id_rsa $CPNAME "$SSHUSER"@"$REMOTEHOST":"$REMOTEDIR"
		if [ "$?" -ne "0" ]
		then
		    echo "Error copying archive and checksum to $REMOTEHOST"
		    mail_log
		    exit 1
        fi 
	fi
    
  # over view of all the files which where backed up
  echo "Creating file listing from archive..."
  if [ "$CRYPT" = "yes" ]
	then
	    KEY=`cat "$PASSDIR""$PASSFILE"`
	    nice -19 $DAR_BIN -K bf:$KEY -l "$ARCHIVEDIR""$ARCHIVENAME" -Q\
	    > "$ARCHIVEDIR""$ARCHIVENAME".txt && gzip -9 "$ARCHIVEDIR""$ARCHIVENAME".txt
	else        
	    nice -19 $DAR_BIN -l "$ARCHIVEDIR""$ARCHIVENAME" -Q\
	    > "$ARCHIVEDIR""$ARCHIVENAME".txt && gzip -9 "$ARCHIVEDIR""$ARCHIVENAME".txt
	fi
    # Script Timer
    STOPTIME=(`date +%s`)
    RUNTIME=$(expr $STOPTIME \- $STARTTIME)
    hours=$(($RUNTIME / 3600))
    seconds=$(($RUNTIME  % 3600))
    minutes=$(($RUNTIME  / 60))
    seconds=$(($RUNTIME  % 60))
    echo
    echo "::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::" 
    echo "$TYPE Zimbra Backup ended at: `date +%H:%M`" 
    echo "Backup took Hr:$hours Min:$minutes Sec:$seconds to complete" 
    echo ":::::::::::::::Cheers Osoffice for the script:::::::::::::::::::::::"
    (cat $LOG; $UUENCODE_BIN "$ARCHIVEDIR""$ARCHIVENAME".txt.gz "$ARCHIVEDIR""$ARCHIVENAME".txt.gz) \
    | mail -c $EMAILCC -s "Zimbra $TYPE Backup Log on $HOSTNAME" $EMAIL
}

[liverpoolfcfan]

For the bogus time reporting for durations over 1 hour changing "seconds" to "RUNTIME" in the 3rd last line should fix it.

Change

    # Script Timer
    STOPTIME=(`date +%s`)
    RUNTIME=$(expr $STOPTIME \- $STARTTIME)
    hours=$(($RUNTIME / 3600))
    seconds=$(($RUNTIME  % 3600))
    minutes=$(($RUNTIME  / 60))
    seconds=$(($RUNTIME  % 60))

to

    # Script Timer
    STOPTIME=(`date +%s`)
    RUNTIME=$(expr $STOPTIME \- $STARTTIME)
    hours=$(($RUNTIME / 3600))
    RUNTIME=$(($RUNTIME  % 3600))
    minutes=$(($RUNTIME  / 60))
    seconds=$(($RUNTIME  % 60))

[borngunners]

liverpoolfcfan. I am sorry about your champions league trauma. Back to your script from your first post. Is it the solution to the DAR problem -A option that I am having or for something else? This is what my script looks like:

function do_backup {
TYPE=$1
if [ $TYPE = "full" ]
then
PREFIX=$FULL_PREFIX
elif [ $TYPE = "diff" ]
then
PREFIX=$DIFF_PREFIX
else
echo "Invalid Backup Type!"
exit 1
fi
# Check for required directories and helper apps
for REQ_BIN in $DAR_BIN $MAILX_BIN $RSYNC_BIN $SSH_BIN $MD5SUM_BIN $UUENCODE_BIN
do
check_req_bin $REQ_BIN
done

for REQ_DIR in $ZM_HOME $SYNC_DIR $ARCHIVEDIR $TO_MEDIA_DIR
do
check_req_dir $REQ_DIR
done
echo "$TYPE Backup started at: `date +%H:%M`"
# Check to make sure we have not already done a backup today.
CURRENTNAME2=`ls -A -1 "$ARCHIVEDIR""$BACKUPWEEK"_"$BACKUPNAME"_"$BACKUPD ATE"_"$PREFIX"*dar 2> /dev/null | head -qn1`
CURRENTFULL=`ls -A -1 "$ARCHIVEDIR""$BACKUPWEEK"*$FULL_PREFIX*dar 2>/dev/null | cut -d . -f1 `

if [ -f "$CURRENTNAME2" ]
then
echo "Full Zimbra Backup failed! FOUND A BACKUP WITH SAME NAME"
echo "Please check why! You should only run this script once a day with the current backup date settings!"
mail_log
exit 1
elif [ $CURRENTFULL ] && [ $TYPE = "full" ]
then
# Look for old backups and put then in directory from where you write them to some form of
# storage media
LAST_FULL_DAR=`ls -A -1 $ARCHIVEDIR | grep $BACKUPNAME | cut -d _ -f1 | head -n1`

if [ -z "$LAST_FULL_DAR" ]
then
echo "No old backups found"
elif [ "$LAST_FULL_DAR" -lt "$BACKUPWEEK" ]
then
echo "Old backups found...old week= $LAST_FULL_DAR current week= $BACKUPWEEK"
echo
for i in `ls -A -1 "$ARCHIVEDIR""$LAST_FULL_DAR"_"$BACKUPNAME"*`

Please let me know if there is anything in my script causing that to happen. Also, how do I make it write to my dvd/rw without me involving. I have already setup a remote backup.

Thanks,

[liverpoolfcfan]

Originally Posted by borngunners

liverpoolfcfan. I am sorry about your champions league trauma. Back to your script from your first post. Is it the solution to the DAR problem -A option that I am having or for something else? This is what my script looks like:


Please let me know if there is anything in my script causing that to happen. Also, how do I make it write to my dvd/rw without me involving. I have already setup a remote backup.

Thanks,

This was not in response to any issues you have raised. It was a response to the quoted post regarding tweaks to the existing script made by chewitt

I will likely be looking into DVD backups myself soon too - so if I find anything out I will let you know.

[liverpoolfcfan]

Originally Posted by borngunners

I am still stuck with the email issues and the DAR issue with the -A option. Can someone help me resolve these issues. I have successfully created the ssh setup to a remote location, but becauise of the DAR issue the backup is not being transfered to the remote location. Below is the error message:

OK - I'm not an expert on this stuff. I am quite new to shell scripting too myself. But trying out the various commands in the script, I think I can see some of what the OP was trying to do.

The -A option error on the DAR command line would indicate that there was a problem with the final section of

		    nice -19 $DAR_BIN -J bf:$KEY -s $ARCHIVESIZE -z$COMPRESS -Z "*.gz" -Z "*.zip"\
			-Z "*.bz2" -Z "*.tgz" -Z "*.zgz" -Z "*.jar" -Z "*.tiff" \
			-Z "*.jpg" -Z "*.png" -Z "*.gif" -Z "*.jpeg" -R `pwd` \
			-c "$ARCHIVEDIR""$ARCHIVENAME" -Q\
			-A "$CURRENTFULL" -Q

so DAR did not like what it saw in the $CURRENTFULL variable.

Looking back up to where $CURRENTFULL gets it's value - it is set from

    CURRENTFULL=`ls -A -1 "$ARCHIVEDIR""$BACKUPWEEK"*$FULL_PREFIX*dar 2>/dev/null | cut -d . -f1 `

It seems that there are two reasons for this (and correct me if I am wrong someone) is to
(1) to verify that there is a FULL backup in the current week's folder on which to base a DIFF
(2) get the Base Name of the FULL backup in the current backups folder in order to pass it to the DAR command

If you run this command with just one ....FULL.1.dar file in the folder it returns a single base file name. However, if your backup goes onto a second file and you have a ....FULL.2.dar the command returns 2 lines, etc.

I believe changing the *dar in the command to .1.dar would actually fix this problem, and allow the script to continue. By definition every backup will always have exactly one .1.dar - so it will either exist or not, and it will only exist once - so will not supply multiple base filenames to the -A option on the DAR command line

    CURRENTFULL=`ls -A -1 "$ARCHIVEDIR""$BACKUPWEEK""$FULL_PREFIX".1.dar 2>/dev/null | cut -d . -f1 `

Will look further into the media section to see if can see anything else that might go wrong afterwards.

Hope this helps

[liverpoolfcfan]

Looks like another issue at the MD5 section of code ...

	    # Create MD5 Checksums to verify archives after writing to media or network transfers
	    cd $ARCHIVEDIR
	    FILENAME=`ls -A "$BACKUPWEEK"_"$BACKUPNAME"_"$BACKUPDATE"_"$PREFIX"*`
	    if [ -e $FILENAME ]
	    then 
	    	echo "Creating MD5 Checksum for $FILENAME..."
		    $MD5SUM_BIN -b $FILENAME > "$FILENAME".md5
		    if [ "$?" -ne "0" ]
		    then
		        echo "MD5 Checksum failed!"
		        mail_log
		        exit 1
		    fi
	    else
	    	echo "$FILENAME not found!"
			echo "This should not happen"
			mail_log
			exit 1
	    fi

This needs a loop around it I believe in case the backup runs to more than one file.

I think this should work - but cannot test it right now myself.

Where FILENAME is currently assigned just once - make it get assigned from a loop. and add the do and done markers to the lines that need to be looped through.

	    # Create MD5 Checksums to verify archives after writing to media or network transfers
	    cd $ARCHIVEDIR


for FILENAME in `ls -A "$BACKUPWEEK"_"$BACKUPNAME"_"$BACKUPDATE"_"$PREFIX"*`
do
	    if [ -e $FILENAME ]
	    then 
	    	echo "Creating MD5 Checksum for $FILENAME..."
		    $MD5SUM_BIN -b $FILENAME > "$FILENAME".md5
		    if [ "$?" -ne "0" ]
		    then
		        echo "MD5 Checksum failed!"
		        mail_log
		        exit 1
		    fi
	    else
	    	echo "$FILENAME not found!"
			echo "This should not happen"
			mail_log
			exit 1
	    fi
done

[borngunners]

Thanks Liverpoolfcfan. I will try it and see. I will let you know what happen...

Originally Posted by liverpoolfcfan

OK - I'm not an expert on this stuff. I am quite new to shell scripting too myself. But trying out the various commands in the script, I think I can see some of what the OP was trying to do.

The -A option error on the DAR command line would indicate that there was a problem with the final section of

		    nice -19 $DAR_BIN -J bf:$KEY -s $ARCHIVESIZE -z$COMPRESS -Z "*.gz" -Z "*.zip"\
			-Z "*.bz2" -Z "*.tgz" -Z "*.zgz" -Z "*.jar" -Z "*.tiff" \
			-Z "*.jpg" -Z "*.png" -Z "*.gif" -Z "*.jpeg" -R `pwd` \
			-c "$ARCHIVEDIR""$ARCHIVENAME" -Q\
			-A "$CURRENTFULL" -Q

so DAR did not like what it saw in the $CURRENTFULL variable.

Looking back up to where $CURRENTFULL gets it's value - it is set from

    CURRENTFULL=`ls -A -1 "$ARCHIVEDIR""$BACKUPWEEK"*$FULL_PREFIX*dar 2>/dev/null | cut -d . -f1 `

It seems that there are two reasons for this (and correct me if I am wrong someone) is to
(1) to verify that there is a FULL backup in the current week's folder on which to base a DIFF
(2) get the Base Name of the FULL backup in the current backups folder in order to pass it to the DAR command

If you run this command with just one ....FULL.1.dar file in the folder it returns a single base file name. However, if your backup goes onto a second file and you have a ....FULL.2.dar the command returns 2 lines, etc.

I believe changing the *dar in the command to .1.dar would actually fix this problem, and allow the script to continue. By definition every backup will always have exactly one .1.dar - so it will either exist or not, and it will only exist once - so will not supply multiple base filenames to the -A option on the DAR command line

    CURRENTFULL=`ls -A -1 "$ARCHIVEDIR""$BACKUPWEEK""$FULL_PREFIX".1.dar 2>/dev/null | cut -d . -f1 `

Will look further into the media section to see if can see anything else that might go wrong afterwards.

Hope this helps