Server keeps freezing/locking up

[chrishewitt]

I have a server that's recently started to freeze on an increasingly frequent basis. The box is still powered up but Zimbra services are not accessible, and more importantly, all SSH connections are being refused. This is a complicating factor as I'm in Dubai and the box is hosted 3,500 miles away in the UK and it requires "phone a friend" to power cycle the box and restore access, which can take 8-12 hours. Meanwhile I get "mails not working, can you give me a call" text messages from the small number of family and friends the server is hosting mail for.

I have no idea what is causing this. I'm at the moderately competent level of someone capable of hacking about to get stuff like Zimbra and VMware installed and working (the box also hosts a VM for a Symantec mail security appliance) but I am not an experienced Linux admin and I don't have strong knowledge of troubleshooting. With my limited knowledge I've been looking at the Zimbra logs (which show nothing) and /var/log/messages which also shows nothing - I think the logger processes are being frozen along with everything else when the box stops responding so there's no record of what's happening.

Recently the box has moved to 5.0.6 where a couple of lock-ups occurred, followed by a 'yum update', a brief move to 5.0.7 and then to 5.0.8 where several freezes have occurred in rapid succession. Again, I'm not 100% certain this is anything to do with Zimbra, or the real correlation of events, but the frequency of problems is increasing.

Your thoughts and any guidance on how to figure out what may be happening would be greatly appreciated!

Christian

[uxbod]

Are you able to get to the VMWare console ? If you can you should be able to see whether or not the server kernel is panic'ing. If it has see if somebody can grab a screenshot and post that up.

[chrishewitt]

The VMware WebConsole is inaccessible and the VMware thicker-client console is only accessible via a tunnelled SSH port. When the box freezes SSH (and any other service) refuses connections until the box is power cycled. When I get the box up again nothing appears to have been recorded in logs, although I may not be looking everywhere I need to be due to a lack of Linux experience.

Ideas? .. kernel update through yum (if there is one)?

[uxbod]

I would ask whoever manages your server to check the physical server console when this happens. I doubt a kernel update would fix it IMHO.

[chrishewitt]

It's hosted by a friend who travels a lot on business, and his wife is the person I normally have to get involved to power cycle the box. She's well trained in which buttons to press to do that, but screenshots of hung systems from the console is a little beyond her abilities

How do I go about proving this could be a kernel panic (it does sound like one) or isolating what could be causing it?

Thx.. Christian

[uxbod]

You really need to see what is happening on the console. If it is panic'ing then it happens immediately that the kernel cannot even write a message out. It may be that it is not the VM guest that is crashing but the actual server that is hosting the VM server.

[chrishewitt]

Zimbra runs directly in CentOS on the physical server, with a VMware Server instance that hosts the mail security appliance (which is RHEL packaged with Symantec apps). I think the VMware part is solid and my suspicion is focussed on the host OS - a panic matches the lack of evidence that I'm seeing left in places like /var/log/messages .. but at this point my experience runs out.

Any pointers on what I should be thinking about to solve this? .. rolling back the kernel etc. (if this is even possible?)

[uxbod]

I run CentOS5.2 aswell on the latest kernel and is it rock solid. I am afraid unless you can get a screenshot/photo of the console it will be very difficult to troubleshoot. Are you using standard disks etc ? Hardware RAID card ? Non Broadcom NIC ? (I have had issues with Broadcom before but it seems okay now)

[phoenix]

I also use CentOS5.2 without problems, this is most unlikely to be a Zimbra problem. How much RAM is on this system and how much is used by the VM? When you say this has 'recently happened', have you done any system upgrades recently?

[y@w]

What version of VMware are you running? You mentioned a web access console.. I ran the web console on GSX for a while and it kept freezing the machine like that except mine was slightly more random.