[SOLVED] Re: Upgrade from 4.5.6 to 5.0.2 on RHEL4 server down

[cadman]

Greets all!

On Friday I upgraded our server from 4.5.6 to the latest version of zimbra. It was long overdue as you can tell. I was nervous so i figured i would do it Friday afternoon so if something went wrong i would have the whole weekend to fix whatever broke. However, the upgrade appeared to go fine. No error messages and upon completion everything ran fine, web interface was up, could send and receive messages, outlook clients could send and receive.. I thought all was well...

Tonight that is not the case. I just tried to check my mail from the web front end, i get a 404 not found error. I can ping the server but that is about it. when i VNC into the local lan to a windows workstation that has outlook running with the connector outlook does startup and i get no error messages, but when i try to send an email in from the outside it does not make it to that workstation (same when i try to send something out) all with no errors.

I do not have SSH open right now so i will not have physical access to the zimbra box until tomorrow morning when i go in to check things out.

My question is. Where do i start? Where does zimbra house most of its logs? are there any log files in particular that i should check out first to see if something is broken? (and if so, location of this or these logs please?)

Any advice would be greatly appreciated. I will check this posting out in the morning and see what you guys have to say.

Thanks in advance for your help!

~Steve

[mmorse]

The big one's to attach snips of:
/var/log/zimbra.log
/opt/zimbra/log/mailbox.log
/opt/zimbra/log/zmmailboxd.out
More: List of Log Files

Based on your prior threads it looks like you're NE so don't hesitate to open a ticket if you need it.
Also, be sure to update your profile so we know appropriate courses of action/commands/options to give you later on.

[cadman]

Originally Posted by mmorse

The big one's to attach snips of:
/var/log/zimbra.log
/opt/zimbra/log/mailbox.log
/opt/zimbra/log/zmmailboxd.out
More: List of Log Files

Based on your prior threads it looks like you're NE so don't hesitate to open a ticket if you need it.
Also, be sure to update your profile so we know appropriate courses of action/commands/options to give you later on.

Thanks for the prompt reply mmorse, ive updated my profile (although i'll need to run the zmcontrol -v and update that when i have access to the box) and will check the logs when i get in the office first thing in the AM tomorrow.

I have two support tickets to use as well as 1 'disaster phone support' ticket remaining. I figured i prolly shouldnt open a ticket until i can get into the box and list some logs and see what is really wrong with it. (but if i am wrong in that assumption let me know)

At least i didnt run into this problem tomorrow night, lol. I don't feel like pulling an all nighter again, heh.

[mmorse]

Originally Posted by cadman

I have two support tickets to use as well as 1 'disaster phone support' ticket remaining. I figured i prolly shouldnt open a ticket until i can get into the box and list some logs and see what is really wrong with it. (but if i am wrong in that assumption let me know)

That's a perfectly good plan. You're always free to attach some log sections to the thread tomorrow and we'll offer suggestions; just wanted to remind you it's there in-case you get too frustrated

[cadman]

Originally Posted by mmorse

That's a perfectly good plan. You're always free to attach some log sections to the thread tomorrow and we'll offer suggestions; just wanted to remind you it's there in-case you get too frustrated

Screw it, im not doing squat anyways. Im gonna drive down to the office and check it out real quick now and at least turn on SSH so i can work on it remotely.

Log snips to be comming soon to those who are bored!

)

[cadman]

crap crap crap crap crap... andddd.. crap.

GDM crashed as soon as i tried to login with the lovely error of "no disk space left"

Although i am curious as to what happened (the server had 50gigs of space on friday when i did the upgrade) this is now probably a recovery effort.

Before anyone asks:

~Yes its backed up, but i was too dumb to rsync the zimbra backup directory off the physical server. (a lesson to anyone else out there not doing this or something like it)

I logged into the president's workstation and did a local backup of his outlook to a local PST file just in case.

I am going to pull the drive and mount it from another linux workstation and hopefully pull the last full backup off the drive.

I suppose i will evaluate my options after that. (reinstall zimbra on new server and restore backup? or try to fix whatever is broken on the existing box)

yay, there went my weekend.

[cadman]

ooooo it gets better... mounting the hd from another workstation gets me all sorts of 'bad superblock' errors.. i will try fsck and other utils, but i wont put to much effort into it, disk just might be toast.

i am noticing that i can startup outlook on users workstations sans the zimbra server.. we are a pretty small installation (15 users). I can export everyones zimbra mailboxes to PST files, then just build a new zimbra server and use the PST import function to import thier inboxes back into the system. does anyone see any holes in this plan? (should i not be able to save the zimbra drive)

[jholder]

Hi Cadman,

Sorry about all the trouble.

Are you able to save /opt/zimbra/ ?
I would suggest putting your effort into getting that. You should use the pst method as a backup.

Are you a Network Edition Customer? If so, please contact Zimbra support. They're available 24/7, and can help with data recovery and save you some time.

If not, please consider upgrading. I'm not trying to make a sale, but you could use backups

[cadman]

Originally Posted by jholder

Hi Cadman,

Sorry about all the trouble.

Are you able to save /opt/zimbra/ ?
I would suggest putting your effort into getting that. You should use the pst method as a backup.

Are you a Network Edition Customer? If so, please contact Zimbra support. They're available 24/7, and can help with data recovery and save you some time.

If not, please consider upgrading. I'm not trying to make a sale, but you could use backups

Thanks for the reply, i am a NE customer. But i figure that if i cannot recover the drive, opening a support ticket will prolly not do me much good unless they can assist in some drive recovery efforts. (my own fault of course for not backing up offsite as well)

[jholder]

Get what you can on /opt/zimbra
If this is one block, it's likely that we will have a full recovery.

Are these raid? Can you rebuild?

If that doesn't work, try to rsync -avr /opt/zimbra to another location. If it hits a bad block, it will try to read anyway. It will also throw output of what failed.

You can leave that going all night, and get some rest. Then wake up, and look for errors.