Reading 4.5.x backups under 5.0.x

[spector@zeitgeist.com]

Rather than upgrade our ZCS system in place, which would disrupt our operations for a time, we've put up a new machine dedicated to ZCS5.0 ... the problem comes in when trying to read the ZCS4.5.x backups...

Is there a way to restore the 4.5.x backups onto the new 5.0x machine? I've tried 7 ways from sunday to do it, but 5.0 can't read them.

If this isn't supported it seems like a near-fatal oversight...

any advise or suggestions would be appreciated...

David Spector

[Klug]

Originally Posted by spector@zeitgeist.com

Rather than upgrade our ZCS system in place, which would disrupt our operations for a time, we've put up a new machine dedicated to ZCS5.0 ...

You'll get disruption anyway : if you put a brand new server next to the current one, any email sent/received with the old one while the two servers are next to each other (and both on) will not be on the new one.

Even if you think about doing a backup then restoring it, you'll have to stop the "old" server just after you've done the backup and let it down while you restore to the new server.

Originally Posted by spector@zeitgeist.com

Is there a way to restore the 4.5.x backups onto the new 5.0x machine? I've tried 7 ways from sunday to do it, but 5.0 can't read them.

Backup are not compatible from one version to the other.

[spector@zeitgeist.com]

OK.. so the only way I can up grade my Zimbra installation is to do an in-place upgrade on my production system, then move the (now 5.0) backups to the new system?

[spector@zeitgeist.com]

Originally Posted by Klug

You'll get disruption anyway : if you put a brand new server next to the current one, any email sent/received with the old one while the two servers are next to each other (and both on) will not be on the new one.


Even if you think about doing a backup then restoring it, you'll have to stop the "old" server just after you've done the backup and let it down while you restore to the new server.


Backup are not compatible from one version to the other.

Actually if I disable mail flowing into my 4.5.x server so that my MX's spool it, and then were to 1) make a full backup and 2) restore it to the new server no mail would be delivered to the 4.5.x machine so they would NOT be out of sync. What I am suggesting is not at all an unreasonable thing to do...

I hope the inability to read 4.5.x backups into or otherwise convert them so they can be read by a 5.0 system isn't an "official" Zimbra position. The ability to read old data sets (read: backups) is prime facie a part of professional software engineering...

[phoenix]

Originally Posted by spector@zeitgeist.com

I hope the inability to read 4.5.x backups into or otherwise convert them so they can be read by a 5.0 system isn't an "official" Zimbra position.

You can vote on this if you like: Bug 14002 - Add conversion tool to upgrade backup versions to allow restore on later zcs versions

[mmorse]

Alas, there's just so much different between series that you can't; it's not our fixed 'position' though, we would love to change this in the future.
To that end-
Originally intended for minor versions: Bug 14002 - Add conversion tool to upgrade backup versions to allow restore on later zcs versions
And the tougher: Bug 15750 - support for restore across major versions

While it goes without saying to take a full backup before you upgrade, it's also good practice to take a full immediately after upgrading (so you have a 'fresh' & because those old backups generally aren't useful anymore unless you install a box with a prior version somewhere).

[spector@zeitgeist.com]

So, in a nutshell, leaving the whole upgrade issue behind for a moment... basically anyone who might need to get to data stored in backups created with an older version of ZCS (say an employee left 1 major Zimbra version ago and a critical attachment is stored in their now deleted account, or maybe for a regulatory reason, like for legal discovery under a court subpoena for example ) is basically SOL.

This is a major, major design flaw. Utterly show-stopping for any serious corporate use of Zimbra.

You guys need to fix this like yesterday!! It doesn't matter how different the data are between releases -- this is not even a plausible excuse: you guys are defining the format of the data/databases Zimbra uses! If you can't (and haven't!!!) been writing tools to accomplish this, then who can? This is engineering 101 level stuff.

No one is asking for downwards conversion (i.e., 5.0.x -> 4.5.x) but conversion of old data into current-release-compatible archives is SOP. Imagine if Legato told their users "Oh, those backups you made with Legato v6...? Sorry! You can't read them with version 7... Ooopsie" ... Even Microsoft allows people to read the old data in to new versions of their programs -- Office 2008 on my Mac happy ingests and displays/converts files from the Windows and Mac office versions all the way back into the 1980s...

I am flabbergasted.


David

[mmorse]

David, I totally understand what you're saying, and encourage you to vote for / support ticket tag / comment on / cc watch those two RFE's.

Until those are completed we include the following in each set of release notes:

BACKUP BEFORE YOU BEGIN TO UPGRADE!
Before you begin the upgrade, run a full backup of the mailbox server. If there are any problems during the upgrade, you can restore to your previous version. After the upgrade, run a full backup immediately. Schema changes in this release (Network Edition 5.0.5) invalidates all old backups. See the Backup and Restore chapter in the Administration Guide

Originally Posted by spector@zeitgeist.com

or maybe for a regulatory reason, like for legal discovery under a court subpoena for example ) is basically SOL.

Without getting too off track - you can still throw up a 4.5.x box and restore, then after restoring from backup to the desired date upgrade it to the same 5.0.x version as your production server, create a backup for the affected account, move it to the produciton box, and restore to a new target account using a prefix - though you shouldn't be using the backups for compliance purposes.
Users could always delete stuff out of a folder and you'd never know it. If you're using the backups on NE coupled with the ability to restoreToTime as a 'compliance solution' - unless you restore to every last second of the day, then comparing differences, you'll miss stuff...

The Network Edition version has an excellent add-on that does envelope forking called Archiving & Discovery w/ cross mailbox search
Often referred to as ZAD for short - more on it:
Zimbra Archiving and Discovery Whitepaper.pdf
Zimbra Archiving and Discovery Release Notes.pdf
Zimbra_Archiving_Discovery_Webinar_Dec_2007.pdf

For FOSS (or NE if you'd like) many use this method of archiving - grabbing everyone's email: Enhancement Hacks - power of filters & always_bcc
In postfix it's essentially: always_bcc = catchallArchiveMonkey@domain.com
Then couple that with a filters to sort into specific folders (make one per user etc) in the 'catchallArchiveMonkey' account.

Individually you can use 'sender_bcc_maps' and 'recipient_bcc_maps' (search Postfix.org)
Here's some examples:
http://www.zimbra.com/forums/administrators/9812-solved-automatically-cc-account-possible.html#post58005
[SOLVED] Monitoring email accounts

Side Notes:
- Depending on what you're trying to accomplish zmmsgtrace may also do what you want.
- Last but not least (use appropriately & legally) there's: http://www.zimbra.com/forums/adminis...intercept.html

[spector@zeitgeist.com]

No. No. No. No. No.

Those were just "for instance" (yet, perfectly valid) examples. (We also archive both the live system and the backups to tape, so the chances of someone deleting something permanently are very low) but that's all beside the point..

The bigger point is...
You're telling me that I should keep special hardware around and spend my support people's valuable time installing/deinstalling entire ZCS versions just so I can read data from a BACKUP!?!?!?!??!

Totally Unacceptable.

If I said that in a corporate environment I would 1) be laughed at and 2) shown the door.


"ZAD" isn't acceptable either, btw, ZCS should be able to read it's own data. Period.



This isn't a matter of "voting on an RFE." This is a basic feature that is required if companies are to rely on ZCS as a professional tool!

I now have to decide what to advise my customer to do.

_David

[mmorse]

As you're NE, I strongly urge you to open a ticket and say: "Please add this as a customer request for bug 14002 & 15750" & even include a link to this thread so you don't have to repeat information.

Note: Enhancement and bug requests also funnel in through the same ticketing process.
-snip-
Support Incidents determined to be enhancement requests or bug reports do not count against your allotted Support Incidents.

Zimbra is molded into what people request - by doing so you link customer needs to future development.

Open Source Product Management: How do features get into Zimbra?