upgrade to 4.0.3 just trashed my install (without a backup)

[DGS_David]

I just can't believe what I saw on the screen, but the zimbra install just deleted /opt/zimbra without so much as a warning!

Just to add, the zimbra user ran a backup each night and these were in /opt/zimbra - which got deleted by the installer!

I have a backup, but it be old'ish so we lost plenty email on my home server but I am just astounded how this could happen soooo easily on an upgrade.

I simply ran the upgrade from
zcs-4.0.2_GA_362.FC4.tgz -> zcs-4.0.3_GA_406.FC4.tgz

upgrades always work so seemlessly, but the core package did nto install, no error etc so I just tried again, 2nd time is just wiped the /opt/zimbra

I am now without a zimbra that installs and also without any email for the last 2 months. (no backup my fault, deleteing the /opt/zimbra - well...)

Here is what hapened:

[root@gw zcs]# ./install.sh

Operations logged to /tmp/install.log.10996
Checking for existing installation...
zimbra-ldap...FOUND zimbra-ldap-4.0.2_GA_362
zimbra-logger...FOUND zimbra-logger-4.0.2_GA_362
zimbra-mta...FOUND zimbra-mta-4.0.2_GA_362
zimbra-snmp...FOUND zimbra-snmp-4.0.2_GA_362
zimbra-store...FOUND zimbra-store-4.0.2_GA_362
zimbra-apache...FOUND zimbra-apache-4.0.2_GA_362
zimbra-spell...FOUND zimbra-spell-4.0.2_GA_362
zimbra-core...FOUND zimbra-core-4.0.2_GA_362

Saving existing configuration file to /opt/zimbra/.saveconfig


PLEASE READ THIS AGREEMENT CAREFULLY BEFORE USING THE SOFTWARE.
ZIMBRA, INC. ("ZIMBRA") WILL ONLY LICENSE THIS SOFTWARE TO YOU IF YOU
FIRST ACCEPT THE TERMS OF THIS AGREEMENT. BY DOWNLOADING OR INSTALLING
THE SOFTWARE, OR USING THE PRODUCT, YOU ARE CONSENTING TO BE BOUND BY
THIS AGREEMENT. IF YOU DO NOT AGREE TO ALL OF THE TERMS OF THIS
AGREEMENT, THEN DO NOT DOWNLOAD, INSTALL OR USE THE PRODUCT.

License Terms for the Zimbra Collaboration Suite:
http://www.zimbra.com/license/collab...cense_1.0.html


Press Return to continue

Checking for prerequisites...
NPTL...FOUND
sudo...FOUND sudo-1.6.8p8-2.4
libidn...FOUND libidn-0.5.15-1
curl...FOUND curl-7.13.1-5
fetchmail...FOUND fetchmail-6.2.5.5-1
gmp...FOUND gmp-4.1.4-6
/usr/lib/libstdc++.so.5...FOUND

Checking for installable packages

Found zimbra-core
Found zimbra-ldap
Found zimbra-logger
Found zimbra-mta
Found zimbra-snmp
Found zimbra-store
Found zimbra-apache
Found zimbra-spell


The Zimbra Collaboration Suite appears already to be installed.
It can be upgraded with no effect on existing accounts,
or the current installation can be completely removed prior
to installation for a clean install.

Do you wish to upgrade? [Y]

Select the packages to install
Upgrading zimbra-core
Upgrading zimbra-ldap
Upgrading zimbra-logger
Upgrading zimbra-mta
Upgrading zimbra-snmp
Upgrading zimbra-store
Upgrading zimbra-apache
Upgrading zimbra-spell

Installing:
zimbra-core
zimbra-ldap
zimbra-logger
zimbra-mta
zimbra-snmp
zimbra-store
zimbra-apache
zimbra-spell

The system will be modified. Continue? [N] y

Shutting down zimbra mail

Backing up ldap


Removing existing packages

zimbra-ldap...done
zimbra-logger...done
zimbra-mta...done
zimbra-snmp...done
zimbra-store...done
zimbra-spell...done
zimbra-apache...done
zimbra-core...done

Removing deployed webapp directories
Installing packages

zimbra-core......zimbra-core-4.0.3_GA_406.FC4-20061016134043.i386.rpm...FAILED
###ERROR###

zimbra-core-4.0.3_GA_406.FC4-20061016134043.i386.rpm installation failed

Installation cancelled

[root@gw zcs]# ./install.sh

Operations logged to /tmp/install.log.12162
Checking for existing installation...
zimbra-ldap...NOT FOUND
zimbra-logger...NOT FOUND
zimbra-mta...NOT FOUND
zimbra-snmp...NOT FOUND
zimbra-store...NOT FOUND
zimbra-apache...NOT FOUND
zimbra-spell...NOT FOUND
zimbra-core...NOT FOUND


PLEASE READ THIS AGREEMENT CAREFULLY BEFORE USING THE SOFTWARE.
ZIMBRA, INC. ("ZIMBRA") WILL ONLY LICENSE THIS SOFTWARE TO YOU IF YOU
FIRST ACCEPT THE TERMS OF THIS AGREEMENT. BY DOWNLOADING OR INSTALLING
THE SOFTWARE, OR USING THE PRODUCT, YOU ARE CONSENTING TO BE BOUND BY
THIS AGREEMENT. IF YOU DO NOT AGREE TO ALL OF THE TERMS OF THIS
AGREEMENT, THEN DO NOT DOWNLOAD, INSTALL OR USE THE PRODUCT.

License Terms for the Zimbra Collaboration Suite:
http://www.zimbra.com/license/collab...cense_1.0.html


Press Return to continue

Checking for prerequisites...
NPTL...FOUND
sudo...FOUND sudo-1.6.8p8-2.4
libidn...FOUND libidn-0.5.15-1
curl...FOUND curl-7.13.1-5
fetchmail...FOUND fetchmail-6.2.5.5-1
gmp...FOUND gmp-4.1.4-6
/usr/lib/libstdc++.so.5...FOUND

Checking for installable packages

Found zimbra-core
Found zimbra-ldap
Found zimbra-logger
Found zimbra-mta
Found zimbra-snmp
Found zimbra-store
Found zimbra-apache
Found zimbra-spell


Select the packages to install

Install zimbra-ldap [Y]

Install zimbra-logger [Y]

Install zimbra-mta [Y]

Install zimbra-snmp [Y]

Install zimbra-store [Y]

Install zimbra-spell [Y]

Installing:
zimbra-core
zimbra-ldap
zimbra-logger
zimbra-mta
zimbra-snmp
zimbra-store
zimbra-apache
zimbra-spell

The system will be modified. Continue? [N] y

Removing /opt/zimbra
Installing packages

zimbra-core......zimbra-core-4.0.3_GA_406.FC4-20061016134043.i386.rpm...FAILED
###ERROR###

zimbra-core-4.0.3_GA_406.FC4-20061016134043.i386.rpm installation failed

Installation cancelled

Now I need to lye down, I feel sick!

[DGS_David]

As it stands the backups which were in /opt/zimbra, or the zimbra user home directory were deleted when the installed deleted /opt/zimbra. Basically apart from the original imap server which was never decomissioned we have lost all email since we migrated to zimbra 3 months ago.

We also now need to discover why the one offsite backup which was a complete /opt/zimbra/ copy also failed to load on version 3.1.4.

What went wrong this time with the upgrade though, it looks like a download issue as running "rpm -ivh" on the zimbra-core from the command line showed that the internal md5sum was wrong. I downloaded the tarball again, both the original and new tarball have the same md5sum:

[root@gw ~]# md5sum zcs-4.0.3_GA_406.FC4.tgz
e3c4d3550679a9cdb723ea05c78b03c5 zcs-4.0.3_GA_406.FC4.tgz
[root@gw ~]# md5sum zcs-4.0.3_GA_406.FC4.tgz.1
e3c4d3550679a9cdb723ea05c78b03c5 zcs-4.0.3_GA_406.FC4.tgz.1

so whatever it was that changed must have happened in the untar I guess.

A subsequent attempt at installing from the original tarball (after re-untarring) proved successful. So I got unlucky with my first attempt...with catastrophic consequences!!!

So, the upgrade ./install.sh mechanism which removes all existing packages needs to pass any RPM error messages back to the shell to at least help debug errors.

Also, if the 'upgrade' borks - the install should *never* go ahead and simply delete the /opt/zimbra directory, not unless it has explicitly told you it is about to do that - this was a failed upgrade, not a new install for goodness sake!

Anyhow, can't take this back and 'what the hell this is open source software' and should have paid if we wanted more 'mission critical backups'. But what is to say this could not happen on a full verasion of zimbra? I only have a few users, family and friends etc, but still they are very upset they just lost all there email/contacts and schedules going back at least 3 months.

[dijichi2]

i'm backing up all my data as i speak! although you should have taken backups in reality lots of us are lazy when it comes to proper dr procedures, i sympathise!

i had the same problem at the very start of the public releases, an upgrade went wrong and it deleted /opt/zimbra without warning. this is a catastrophic bug, absolutely, please file it in bugzilla. an application should never ever unconditionally delete a directory that contains the data store - perhaps this is a case for making the default data stores outside /opt/zimbra?

[DGS_David]

Well if we forget about the backup problem which is my problem and concentrate on the *real* problem...

There is no way of knowing if the rpm's are good until such time as the install/upgrade borks. Once you get to this stage, what is the correct way to continue the upgrade without the installer trashing your data?

I think the installer should check the rpm before starting the upgrade, especially if this is a one way street.

Also, it needs to be much cleverer in detecting a existing install rather than (i guess) checking if the rpm's are already installed!

[DGS_David]

Em, glad to see Zimbra feel this is important, no reply or response. This really inspires me as I start to rebuild things that a future upgrade will not return me to oblivion at some point.

Oh well, guess I'll have better backups next time although I might edit the edit the funtion removeExistingInstall() in util/utilfunc.sh before I upgrade next

My fault not good enough backups, Zimbra fault that the upgrade managed to bork and kill my install!

[phoenix]

First off, Zimbra staff don't monitor these forums on a regular basis. Next, if this is a zimbra problem then you can file it in bugzilla which is the correct place for reporting 'bugs' which is what this would be. Finally, is there anything in the logs other than the info you've posted above?

[KevinH]

Originally Posted by DGS_David

Em, glad to see Zimbra feel this is important, no reply or response. This really inspires me as I start to rebuild things that a future upgrade will not return me to oblivion at some point.

Oh well, guess I'll have better backups next time although I might edit the edit the funtion removeExistingInstall() in util/utilfunc.sh before I upgrade next

My fault not good enough backups, Zimbra fault that the upgrade managed to bork and kill my install!

We feel this is important but this is not something we've had happen ever since the GA (Feb 2006). So hard to believe it *just happened*. A couple things could have made this better although not much can be done at this time.

- move backups to somewhere else. this is basic DR practice. ie if the drive(s) would have failed you'd be in the same boat. We use rsync twice a day to push backups to a different server. We also take tape of that backup off-site in regular intervals.

- when something goes wrong try to find out what it was. look in the logs, look at the state of your install.

We will make sure to add a prompt on the /opt/zimbra removal but this won't help those that aren't moving backups off disk or off site. If they allow it to be removed.

If you can send your install logs from /tmp that would be helpful. Maybe we can figure out why the 1st install failed.

[jholder]

We get a ton of people who have downloaded and installed zimbra (155,270 from sourceforge total)

For the FC4 Binary, 160 Downloads.

As kevin has said, this hasn't happened since the early days. I agree, that there is something that triggered it.

During the install, the packages are removed, but the message store and DBs remain in the same place.

The fact that the /opt/zimbra dir was removed leads me to belive that somehow the lines from the -u switch got envoked.

You said this was the second time that the install failed. It's important that if the -u switch was kicked on without you knowing, that we find out why.

We need to get the /tmp/install.log.xxxx files from you.

Just as a clairification, we are community supported and try to help out as much as we can. However, I know the feeling. One time, I totally erased the /opt/zimbra dir, and I DIDN"T have a backup. Boy did I take a lot of heat for that.

So now, the typical rule I use is i ALWAYS backup before any type of maintenance. I even do this with all of our servers, not just zimbra.

[marcmac]

The question isn't "why did the 2nd install remove my stuff" - the question to be answered is "why did the first install, the upgrade, fail."

So - post your logs, please.

[DGS_David]

I am in the process of testing a fix and will post my results as soon as they are completed.

As I show (and mention above), the first installed failed becuase the zimbra-core rpm had a problem with the internal md5sum being incorrect, most probably caused by the untar of the downloaded tarball rather than the download itself (I downloaded twice and checksums were same for both downloads). The rpm error did not print to screen, it was only when I invoked the rpm command manually did it croak to screen. Printing these errors to terminal needs fixing IMO.

I keep saying and lets re-iterate, the biggest issue here is that I did not backup outside of /opt/zimbra and that is my fault, probably because my dependancy grew on zimbra without even realising it and was too busy in my day job. The dact my dependancy grew on Zimbra as did my family and friends is the fault of Zimbra as it shows how good the product is ;-)

But, the fact it deleted without prompt is the 2nd problem which is big in my opinion and quite simple to invoke. Simply removing the packages of your installation via rpm:

rpm -e zimbra-ldap zimbra-logger zimbra-mta zimbra-snmp zimbra-store zimbra-apache zimbra-spell zimbra-core

and then running install.sh will confuse the installer into thinking it is a new install and then simply delete /opt/zimbra/

I'll post back shortly with my fix results.
David.