Thank you. This did the trick!
Printable View
Thank you. This did the trick!
I have created a new version of the zmDBbac.sh script:
- I have placed an updated version of the zip file containing the scripts at the same link, zmDBbac.zip.
- The restore_user.sh script was not changed.
- Changelist:
Code:0.5 alpha:
- Automatically have Zimbra create a user's DB, if it doesn't exist (optional).
- Alternately, log users who lack a DB, if AUTOCREATE_DB is set to "no".
- Comment and text output changes for more clarity
- Standardized script indentation
I have updated the instructions in the original post to reflect the changes.
I want to share my expiercne using OSS and Backup.
Firs tof all i dont use shellscripts anymore - besides they are technically ver powerfull (specially with rsync) they have a real problem - Management.
Starting from logging / reporting / and so on i think this isnt a professional Solution so im using another Product (easy for me since we provide onlinebackup services : )
So what im doing is i run several times a day a incremental hotbackup
to reduce the downtime at cold to a minimum
but of course theres one thing - you have todo it within the same backup set - no other way to reduce coldbackup downtime to 3 minutes
so i wrote 2 little shellscript which trigger zimbrashutdown by time
its Simple - it will only shutdown and startup zimbra if a certain timelimit is reached so the decision if its a cold or hotbackup is made by system clock trough that script
prebackup
postbackupCode:#!/bin/bash
chour=$(date +%H)
if ([ "$chour" -ge 5 ] && [ "$chour" -lt 11 ]) ; then
echo "Its" $chour "- Shutdown"
sudo -H -u zimbra /opt/zimbra/bin/zmcontrol stop
else
echo "Its" $chour "- No shutdown"
fi
exit
Explanation chour is the actual hour.Code:#!/bin/bash
chour=$(date +%H)
if ([ "$chour" -ge 5 ] && [ "$chour" -lt 11 ]) ; then
echo "Its" $chour "- Startup"
sudo -H -u zimbra /opt/zimbra/bin/zmcontrol start
else
echo "Its" $chour "- No Startup"
fi
exit
it simply runs if hour is equal 5 AND less then 11 strigger startup / shutdown - so 5:00 - 10:59 - change it for your needs
i made this timewindow something really went wrong and backup run way longer than expected - so i gave him 1 hour longer than a usually fullbackup takes - just in case otherwise - lets say for some reason the coldbackup trigger a fullbackup it woudl run for hours and the postbackup script would never trigger startup again
(of course you can easyly add minutes but i dont need em i got those backups running at xx:05)
luclyly my backupsoftware collect the stdout data of that scripts so i recive everyday my reportmails with the shelloutput
one note: its not really nessesary to have a postbackup script you could rigger a zimbra start always and it shoudl do no harm - but i dont like that - maybe something will change withing the startupscript some day - so i really only want to trigger start when i have to
works for me for 8 months 100% stable
hope it helps someone
I am bit confused between zmbak_v.0.8.sh and zmDBbac scripts.
Could anybody please explain me in details about how to install and use these scripts.
Is this are two diff script?
I don't know where to start and how to restore.
Please help me.
Quote:
Originally Posted by pradip
- Change your settings to match your environment, per the first post of this thread. Things to set might include:
- Paths for the directories.
- Archive size. Unless you are actually backing up to DVDs, I would make the archive size much larger than 4395M.
- Crypt, for encrypted archives. Useful if someone else might get access to your backup.
- Email addresses.
- SSH settings if you are doing remote rsyncing to another machine.
- Run the script with the --INSTALL option, e.g.
This will make sure you have all the required tools, and install the crontab to do full and diff backups automatically. You may have to do a web search to get dar installed.Code:/usr/local/sbin/zmbak.sh --INSTALL
The zmbak.sh script does a local or remote rsync backup of your entire /opt/zimbra directory. This allows you to restore an entire Zimbra installation.Quote:
Is this are two diff script?
The zmDBbac.sh script does MySQL and LDAP dumps for each user, allowing you to restore individual users, or even individual user folders. The post on zmDBbac.sh install describes how to call it from the zmbak.sh script.
If you have to do a full restore, you simply stop Zimbra services, replace the existing /opt/zimbra directory with the backup one, restart Zimbra services. If you are restoring to a new server, you may have a couple of other things to do, e.g. set up split DNS.Quote:
I don't know where to start and how to restore.
The method for restoring individual users or folders is somewhat involved, and is described in the zmDBbac references above in this post.
Is there anyway we can schedule zmdbbac with zmbak ??
You simply call the zmdbbac script from within the zmbak script, as shown in the post. In other words, you have to modify zmbak slightly to work with zmdbbac. A cron job starts zmbak, which does a warm rsync, then shuts down Zimbra and does a cold rsync. While Zimbra is still shut down, the zmdbbac script is called, and it backs up SQL and LDAP. When it's finished, it returns to zmbak, and the main script finishes. So you still only need the one cron job to get the whole thing done.
I am still wondering why this script is taking more than 24 hours to complete backup of 300 GB of data...
The first full rsync of that amount of data will take a long time. Future rsyncs should go much faster. Dar creation, on the other hand, will continue to take a long time on every full backup.Quote:
Originally Posted by aarat
If _every_ full takes that long, some things to consider:
What is your destination, a locally mounted volume or remote? If local, does the backup volume use completely different disks than Zimbra? If remote, is it on your LAN or over the internet? Are you using SSH? What is the transfer rate shown in the backup email? Does it take 24 hours just to rsync, or to both rsync and dar? The time at the bottom of the email includes the dar time.
For me, it takes about 5 hours to full rsync and dar a 100GB /opt/zimbra to a local volume. Most of that is dar creation. After your first full rsync, future fulls will probably only need to sync 3% or so of the total. Hot sync takes about 7 minutes for 3GB, which works out to about 4hrs for 100GB. You can calculate from the sent bytes and bytes/sec in the hotsync section of the email. Cold sync takes about 1min25sec. DB and LDAP dumps takes 2-3 minutes. Downtime, for me, is about 5min30sec, which includes DB and LDAP dumps. Thus, dar takes up about 4hrs45min of the 5hr total, to dar up 100GB. Extrapolating to your 300GB, it would have taken about 12hrs for rsync and 14.25hrs for dar, total about 26.5hrs, the very first time I ran the script.
My backup volume is local and separate, but part of a RAID set, so it is not on separate disks. I backup the backup to another RAID. :)
The fake backup is happening on SATA Storage (IBM DS3500) Virtualized on Xenserver. However the dars are being created on cifs share mounted on NAS with 100mbps of LAN Connectivity to server. I am also not receving mail for backup as it ends up with /usr/lib/sendmail not found.
Please help
Here's the backup log
Quote:
============================
Sat Jan 14 00:41:52 IST 2012
Performing FULL backup
============================
full Backup started at: Sat Jan 14 00:41:52 IST 2012
Setting date & version Marker into /opt/zimbra/DATE_VERSION.txt
Doing a hotsync of /opt/zimbra/ to /opt/fakebackup/
rsync: stat "/opt/fakebackup/store/0/111/msg/1/.4154-3372.msg.83CGcp" failed: No such file or directory (2)
rsync: rename "/opt/fakebackup/store/0/111/msg/1/.4154-3372.msg.83CGcp" -> "store/0/111/msg/1/4154-3372.msg": No such file or directory (2)
rsync: link "/opt/fakebackup/store/0/118/msg/0/260-14.msg" => store/0/106/msg/0/279-52.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/124/msg/0/500-500.msg" => store/0/110/msg/0/3563-3207.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/127/msg/0/600-600.msg" => store/0/110/msg/0/3563-3207.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/132/msg/0/499-476.msg" => store/0/111/msg/2/11401-9870.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/133/msg/0/257-7.msg" => store/0/109/msg/0/264-22.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/133/msg/0/343-303.msg" => store/0/107/msg/0/3400-3086.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/135/msg/0/323-208.msg" => store/0/115/msg/1/7986-7199.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/136/msg/0/280-200.msg" => store/0/109/msg/0/3404-2904.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/137/msg/0/260-100.msg" => store/0/103/msg/0/260-100.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/137/msg/0/263-106.msg" => store/0/103/msg/0/263-108.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/137/msg/0/267-120.msg" => store/0/103/msg/0/268-118.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/137/msg/0/280-200.msg" => store/0/103/msg/0/282-202.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/144/msg/0/258-6.msg" => store/0/140/msg/0/308-106.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/144/msg/0/300-100.msg" => store/0/109/msg/0/300-100.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/144/msg/0/504-508.msg" => store/0/103/msg/0/282-202.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/144/msg/0/520-600.msg" => store/0/100/msg/0/300-300.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/144/msg/0/528-617.msg" => store/0/129/msg/0/2883-2504.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/144/msg/0/531-626.msg" => store/0/115/msg/1/8015-7332.msg fail
Session terminated, killing shell... ...killed.
/store/0/144/msg/0/535-630.msg" => store/0/100/msg/2/11804-10075.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/146/msg/0/3603-3204.msg" => store/0/138/msg/1/5866-5313.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/147/msg/0/1040-900.msg" => store/0/138/msg/1/5866-5313.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/159/msg/0/3718-3180.msg" => store/0/15/msg/0/1566-1199.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/159/msg/1/4123-4089.msg" => store/0/103/msg/0/3152-2853.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/17/msg/0/1396-1334.msg" => store/0/15/msg/2/10888-9466.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/172/msg/0/1707-1404.msg" => store/0/1/msg/0/2300-2103.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/173/msg/0/321-308.msg" => store/0/13/msg/3/12849-11118.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/177/msg/0/1073-810.msg" => store/0/1/msg/0/2301-2104.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/177/msg/0/1100-900.msg" => store/0/159/msg/0/3864-3514.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/177/msg/0/1124-1014.msg" => store/0/161/msg/0/621-505.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/184/msg/1/7005-6110.msg" => store/0/138/msg/1/5866-5313.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/185/msg/0/2740-2600.msg" => store/0/177/msg/0/1101-903.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/206/msg/0/1041-1046.msg" => store/0/183/msg/2/10063-8843.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/206/msg/0/1044-1052.msg" => store/0/183/msg/2/10067-8847.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/206/msg/0/1072-1110.msg" => store/0/183/msg/2/10086-8882.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/223/msg/0/257-2.msg" => store/0/10/msg/0/593-489.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/225/msg/0/2381-2005.msg" => store/0/115/msg/1/8015-7332.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/233/msg/0/381-204.msg" => store/0/13/msg/3/12819-11031.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/233/msg/0/389-214.msg" => store/0/13/msg/3/12822-11035.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/233/msg/0/400-300.msg" => store/0/138/msg/1/5866-5313.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/24/msg/0/3944-3327.msg" => store/0/193/msg/7/32421-28010.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/25/msg/1/5108-4168.msg" => store/0/109/msg/0/272-38.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/25/msg/1/5380-5037.msg" => store/0/140/msg/1/5010-4449.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/25/msg/1/5852-6099.msg" => store/0/170/msg/0/1079-984.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/31/msg/0/357-263.msg" => store/0/159/msg/0/3770-3283.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/31/msg/0/380-400.msg" => store/0/103/msg/0/282-202.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/32/msg/2/8436-7151.msg" => store/0/111/msg/2/11261-9495.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/32/msg/2/8596-7495.msg" => store/0/199/msg/0/347-420.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/33/msg/0/261-21.msg" => store/0/20/msg/1/8168-6798.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/41/msg/13/54513-46697.msg" => store/0/109/msg/0/264-22.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/45/msg/0/490-617.msg" => store/0/205/msg/0/1892-2025.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/45/msg/0/502-645.msg" => store/0/104/msg/2/9003-8451.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/45/msg/0/523-679.msg" => store/0/25/msg/1/6153-6909.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/45/msg/0/536-700.msg" => store/0/140/msg/1/5276-5076.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/45/msg/0/537-702.msg" => store/0/217/msg/0/2423-2233.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/45/msg/0/567-758.msg" => store/0/144/msg/0/558-676.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/45/msg/0/570-762.msg" => store/0/100/msg/3/12593-10771.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/45/msg/0/571-763.msg" => store/0/103/msg/0/3144-2837.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/46/msg/0/266-16.msg" => store/0/200/msg/0/257-2.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/46/msg/2/11500-10037.msg" => store/0/178/msg/1/6460-5553.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/46/msg/2/11812-10781.msg" => store/0/172/msg/0/1744-1486.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/5/msg/5/20492-19526.msg" => store/0/2/msg/6/24576-24774.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/50/msg/0/280-102.msg" => store/0/24/msg/0/3901-3203.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/50/msg/0/565-349.msg" => store/0/185/msg/0/2727-2484.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/50/msg/0/580-387.msg" => store/0/111/msg/2/11261-9495.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/50/msg/0/622-508.msg" => store/0/184/msg/1/7009-6117.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/50/msg/0/624-511.msg" => store/0/185/msg/0/2763-2709.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/55/msg/1/4286-3537.msg" => store/0/181/msg/2/11201-9201.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/61/msg/0/760-500.msg" => store/0/1/msg/0/2300-2103.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/64/msg/7/30203-25874.msg" => store/0/37/msg/1/8139-7331.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/64/msg/7/30244-25945.msg" => store/0/55/msg/1/4293-3545.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/64/msg/7/30696-26839.msg" => store/0/177/msg/0/1105-910.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/64/msg/7/30697-26840.msg" => store/0/159/msg/0/3910-3601.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/64/msg/7/30699-26843.msg" => store/0/32/msg/2/8495-7269.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/64/msg/7/30714-26874.msg" => store/0/202/msg/1/4876-4242.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/64/msg/7/30724-26893.msg" => store/0/118/msg/0/300-200.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/75/msg/0/640-400.msg" => store/0/10/msg/0/593-489.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/75/msg/0/641-401.msg" => store/0/166/msg/0/261-104.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/75/msg/0/660-500.msg" => store/0/10/msg/0/762-802.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/75/msg/0/661-507.msg" => store/0/10/msg/0/767-830.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/91/msg/1/7080-6102.msg" => store/0/110/msg/0/2556-2067.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/93/msg/0/257-5.msg" => store/0/203/msg/0/257-4.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/93/msg/0/280-101.msg" => store/0/203/msg/0/303-106.msg failed: File exists (17)
rsync: link "/opt/fakebackup/store/0/99/msg/0/580-800.msg" => store/0/162/msg/0/880-801.msg failed: File exists (17)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1060) [sender=3.0.7]
rsync threw a hotsync error. this is not unusual, continuing...
Disabling the Zimbra crontab...
rm: cannot remove `/opt/zimbra/crontab.blank': No such file or directory
Stopping the Zimbra server...
Host mail.ecsinfotech.com
Stopping zmconfigd...Done.
Stopping stats...Done.
Stopping mta...Done.
Stopping spell...Done.
Stopping snmp...Done.
Stopping cbpolicyd...Done.
Stopping archiving...Done.
Stopping antivirus...Done.
Stopping antispam...Done.
Stopping imapproxy...Done.
Stopping memcached...Done.
Stopping mailbox...Done.
Stopping logger...Done.
Stopping ldap...Done.
Had to kill some left over procs...
Doing a fast cold sync...
Reinstating Zimbra's crontab...
Starting Zimbra...
Host mail.ecsinfotech.com
Starting ldap...Done.
Starting zmconfigd...Done.
Starting logger...Done.
Starting mailbox...Done.
Starting memcached...Done.
Starting imapproxy...Done.
Starting antispam...Done.
Starting antivirus...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.
Starting stats...Done.
Service down time was - Hr:0 Min:7 Sec:59
Running A hack... This one to check and start Stats subsystem
Hack not needed Stats seems to be running fine...
Writing a full backup: 02_ZimbraBackup_20120114_FULL
into: /Backup/zimbra_dars/ with file sizes of max: 4395M
Saving Unencrtyped Archive...
--------------------------------------------
730400 inode(s) saved
with 8523 hard link(s) recorded
0 inode(s) changed at the moment of the backup
0 inode(s) not saved (no inode/file change)
0 inode(s) failed to save (filesystem error)
0 inode(s) ignored (excluded by filters)
0 inode(s) recorded as deleted from reference backup
--------------------------------------------
Total number of inodes considered: 730400
--------------------------------------------
EA saved for 0 inode(s)
--------------------------------------------
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.10.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.11.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.12.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.13.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.14.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.15.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.16.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.17.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.18.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.19.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.1.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.20.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.21.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.22.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.23.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.24.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.25.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.26.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.2.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.3.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.4.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.5.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.6.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.7.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.8.dar...
Creating MD5 Checksum for 02_ZimbraBackup_20120114_FULL.9.dar...
Creating file listing from archive...
:::::::::::::::::::::::::::::::::::::::::::::::::: ::::::::::::::::::
full Zimbra Backup ended at: 10:24
Backup took Hr:33 Min:42 Sec:20 to complete
:::::::::::::::Cheers Osoffice for the script:::::::::::::::::::::::
/usr/lib/sendmail: No such file or directory
. . . message not sent.