Halloween - IT Admin Horror Stories

[mmorse]

Post your favorite 'oh crap' moments here!

(Whether they be 100% true. or an exaggerated IT horror story that's taken on additional mythical proportions every time it's told.)

*OS/network/disk failure, virus/spyware/rootkit, even ID10T user error/configuration moments - not a 'my favorite bug' thread.
Props if the situation was solved by deploying Zimbra though

[uxbod]

Well ... I was working for a large multi-national running HP-UX systems and Oracle/SAP, and one day the clock struck twelve and the OS just started to disappear Down went SAP and Oracle like a sack of spuds!

Mayhem broke with the IT manager standing over my shoulder wanting to know what had happened ... I did not have a clue, and I could not even get onto the system as it was completely hosed! So the task of restoring the server began and after 30 minutes I had everything backup and running again. phewww

Until 1pm! The system disappearing again What the hell is going off, panic set it, this time I managed to keep a couple of sessions open to allow me to check the system.

And then it clicked .... I wonder .... Yep indeed, somebody had setup a cronjob AS ROOT, that attempted to 'cd' to a directory which then proceeded with a 'rm -rf *' Though the ******* other admin did not verify that the directory existed before performing the remove! Well once we had restored the system again, the cronjob was removed, and we were all running fine again.

Morale of the story is to always protect root access and ensure you have adequate backups!!!

[raj]

My 9 months old son always lies down next to me when I am working. Other day I was logged in using WinSCP (root) to one of our client’s zimbra server (more than 700 users on it)
I got distracted doing something and suddenly I hear big thump and look at my screen, It said "Are you sure you want to delete file ‘/’ ?"
My son woke up from his sleep and hit the delete key on my laptop using his leg..and thank god he did not hit twice ..lol
So now i keep him at-least 2 feet away from my laptop..but i see it him deleting one of my servers soon..just question of time.

Raj

[p24t]

One time, I was working at a large hosting place here in Texas. We had a customer who was doing online pharma sales, and the DEA wanted to investigate them. We had to make up some excuse as to why they were going to be down, don't for the life of me remember what it was.

So we've got feds wandering around the data center wanting to poke at servers. Their main system (db server probably) is running on a nice Dell 2650, with a RAID 5 in it. The DEA folks decide they want the data from this system.

So one of our techs goes out to the system. The live, running system. And he pulls out one of the drives.

Okay, but this is a RAID. Got that nice R at the beginning for Redundant. Pulling a drive out of a RAID 5 set won't really hurt anything too much.

Unless you then decide to put it back in. When you put the drive back in, the card now has to figure out what is going on. It's going to have to determine whether that is a good drive, a new drive, if it has to completely restripe, or if it can recover, etc.

Which is what they do, you pay for expensive RAID cards because they can handle that.

Unless, while the system is trying to figure out WTF you're doing to it...you pull the power.

Yeah, the DEA wasn't happy.

[uxbod]

roflao Had a user once who made a mistake within a client/server application and decided to run to the computer room and switch off the server He thought he could get there before the change actually took place