Upgrade 5->6 and the use of hardlinks in zcs backups

[Sven Meeus]

Hi,

We recently updated our Zimbra NE from 5.0.18 to 6.0.10 on RHEL5.
Next to a few minor problems with the Outlook connector and a few inconsistencies, everything went fine.

After a few days however, Our monitoring system alerted us that the filesystem assigned for backups was full, and Zimbra also spammed our operations mailbox with the same message.

We have several mount paths in our Zimbra installation, we always had:

/opt
/opt/zimbra/backup
/opt/zimbra/store
/opt/zimbra/data
/opt/zimbra/db
/opt/zimbra/index
/opt/zimbra/log
/opt/zimbra/logger
/opt/zimbra/redolog
/opt/zimbra/data/ldap

Apart from the relocated LDAP data mountpoint, nothing changed in the upgrade.

In zimbra 5, our backup partition was 500GB. This was almost sufficient to store 3 months of backups (weekly fulls and daily incrementals) thanks to the use of hardlinks.

For Zimbra 6 we raised the partition size for the backups to 750GB, thinking this would be sufficient for the growing mailbox store (170GB at this moment), which turned out not to be, as there is no use of hardlinks by default for Zimbra 6. Unfortunately, we had to find this out the hard way.

By going over the forums, the wiki and a few other places that turned up on Google when looking for 'zimbra backup hardlink' we found some interesting articles and explanations. We also updated our cron job entries to disable the zip backups:
0 1 * * 6 /opt/zimbra/bin/zmbackup --noZip -f -a all
0 1 * * 0-5 /opt/zimbra/bin/zmbackup --noZip -i
0 0 * * * /opt/zimbra/bin/zmbackup -del 3m

From what we understand is that the '--noZip' option should enable the use of hardlinks in backups again, but it doesn't.

Our last theory on this matter is that in Zimbra 5 hardlinks for backups were made towards a previous backup, where as in Zimbra 6 the hardlinks are made against the mailbox store, which obviously won't work in our case as you can not make cross mountpoint hardlinks.

At this moment, we allocated 4TB only for backups on our iSCSI storage, only for backups, where before 500GB was enough. This is far from being cost effective, but does meet our company policy of keeping at least 3 months of mailbox backups. We also will not change the mount points.

Anyone has any thoughts or suggestions on what we're missing here?
And if our last theory is correct, Zimbra Developers: Please change it back to the Zimbra 5 behaviour!

[ewilen]

Hardlinks in backups aren't made against the mailbox store. E.g. I have all of /opt/zimbra on a single mountpoint, and I do

ls -lt /opt/zimbra/backup/sessions/full-20110226.090011.258/accounts/044/39a/04439a86-f8a1-4466-8d12-b7820e09de42/blobs/1/7

(Note: you'll have to use a different path on your system, but similar structure.) I can see that the maximum links to any given message is 4. This is the same as the number of full backups currently residing on my system. (Excluding "ad-hoc" fulls caused by the addition of new accounts.) In other words, these are messages that are in all four full backups; if there were links to the mailbox store, I should see at least one message with 5 links. I checked against several paths just to be sure.

Note that hardlinks may only occur between consecutive backups, though. If you have a zipped backup in the middle, that may cause the next --noZip backup to be done "fresh" with no hardlinks. Also, I highly doubt that it matters, but my crontab has the arguments to zmbackup in a different order:

0 1 * * 6 /opt/zimbra/bin/zmbackup -f -a all --noZip  --mail-report
0 1 * * 0-5 /opt/zimbra/bin/zmbackup -i --noZip --mail-report
0 0 * * * /opt/zimbra/bin/zmbackup -del 1m --mail-report