--zipStore doesn't work.

[lcxe]

We've been trying to get our backups into a state where we would be able to easily rsync them from our live server over to our backup server. The problem is that to date we've been unable to move them over in a reasonable amount of time due to the number of files.

Somewhat recently we found out about the --zip and --zipStore switches for the zmbackup command. From what we read everything indicated that the zip options would put all the blobs into zip files, drastically reducing the number of files in the backups, because the blobs would be clumped into the zip files.

However after running the backup command with the --zip option was out of the question for us as when we tried using it, by the time we go into the office at nine in the morning the full backup had no completed after eight hours. In comparison our normal full backups typically take around half an hour to complete. We then attempted to run the backup command with the --zipStore option. This was much more successful, as it took almost the time time as the normal full backup took run. But after comparing the number of files contained within the two full backups, there was almost no difference.

Here is the output for the first unzipped backup file:

zimbra@server:~/backup/sessions$ nice ls -R full-20081202.060006.336 | wc
1454668 1370892 60737837

And the output for the zipStore backup file:

zimbra@server:~/backup/sessions$ nice ls -R full-20081203.163307.186 | wc
1461372 1376489 60965519

Further more when I explored into the zipStore backup file, there was no sign of any zips.

For the record the commands used to create the backups were:

zimbra@pat01:~$ zmbackup -f -a all
zimbra@pat01:~$ zmbackup -f --zipStore -a all

[Jesster]

Hi lcxe,

I feel your pain! I had similar issues, until I discovered it's the combination of _both_ arguments to get few files and faster backups.


use --zip --zipStore together!

HTH
-Jessie

[Dirk]

The help for zmbackup shows this:

-z,--zip                 Backs up blobs in zip files rather than as
                          individual files.
    --zipStore            When --zip option is used, store blobs
                          uncompressed.

my crontab has the command listed with just -z which the help would seem to indicate to be synonimous with both --zip and --zipStore, but the text of the help implies that --zip is an uncompressed archive and --zipStore is compressed.

I'd say that's the wrong way around anyway, --zipStore should be the uncompressed one (store the files, dont compress them)

So I'm still confused.

[lcxe]

Thank you very much for your reply Jesster, that is very helpful knowledge. I doubt I would've figured that one out no matter what I did.

I guess it would help if the command threw some sort of error message if you didn't use the --zip option along with --zipStore, because letting it proceed as if it's working it just confusing.

And to Dirk, zipStore is the switch that leaves the files uncompressed in zip format. Just plain zip will compress the files and put them into a zip format. I think the phrasing is a little poor for the description of zipStore. Reading it now I see that it tells you what Jesster had said about using the two in conjunction, and what it does. But alone it appears as something completely different.

[Jesster]

Originally Posted by lcxe

Thank you very much for your reply Jesster, that is very helpful knowledge. I doubt I would've figured that one out no matter what I did.

Glad to have helped! We struggled on the very same problem. I'm surprised the default backup options don't use --zip --zipStore because it's more efficient on the filesystem and archives.

[lcxe]

Using zipStore seems to be acting just like it did with the zip option now. I was hopeful when you mentioned it being fast in your description Jesster, but it doesn't seem to be the case for us.

We only ran it for an hour this morning to see how it compared, and in that hour it ran, it wasn't able to get by the first account it was to backup. Mind you the account is 44GB, but still. We were seeing all the accounts take a little more than 20 minutes on a normal full backup.

I feel like I'm missing something though. Because currently our full backups are only 106GB, and I'm seeing people talking about having systems with tens of thousands of users, compared to our ~40. How are they able to get their backups done in a reasonable amount of time so that their users on their system don't suffer from downtime every time a backup is done? I know they probably have multi-server systems in place to deal with this, but the underlying idea is there. How do others get their backups done with single system setups, with a decently large amount of data in a feasible amount of time?

[Jesster]

We switched to Auto-Grouped backups from Standard, this helped reduce the amount of users to backup on a daily basis and the backup window needed at night.

There's a good write up here: Backup and Restore we have several thousands of users, so Auto-Grouped is kind of a necessity.

Are you writing the backups to another disk (separate disk from /opt/zimbra/) ?

do you have Zimbra stats package installed & enabled?

if so you can create some charts to help see if there's any IO issues, I think the program is zmstat-chart, more about that here: Server Monitoring - Zimbra :: Wiki