server possibly hacked - help getting Zimbra running

[r_bin!]

Our system admin has moved on to bigger n better things. For now I am watching over the system till his spot is filled, but I am no system admin.

Today our zimbra mail stopped working. When I looked in the var/log files I noticed somebody from Paris France had been trying for hours to ssh into our linux box and appears they succeeded:

Nov 3 14:05:37 smtp sshd[6536]: Failed password for invalid user raphael from 193.251.253.164 port 55685 ssh2
Nov 3 14:06:01 smtp CRON[6543]: (pam_unix) session opened for user zimbra by (uid=0)
Nov 3 14:06:07 smtp sudo: zimbra : TTY=unknown ; PWD=/opt/zimbra ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr status

Then this happened and the mail account stopped working:

Nov 4 15:53:25 smtp init: tty1 process (373) killed by signal 15
Nov 4 15:53:25 smtp init: tty2 process (3257) killed by signal 15
Nov 4 15:53:25 smtp init: tty3 process (3258) killed by signal 15
Nov 4 15:53:25 smtp init: svscan process (3259) killed by signal 15
Nov 4 15:53:27 smtp zimbramon[13151]: 13151:info: Stopping services
Nov 4 15:53:27 smtp zimbramon[13151]: 13151:info: Stopping mta
Nov 4 15:53:32 smtp postfix/postfix-script: stopping the Postfix mail system
Nov 4 15:53:32 smtp postfix/master[4139]: terminating on signal 15
Nov 4 15:53:32 smtp zimbramon[13151]: 13151:info: Stopping spell
Nov 4 15:53:40 smtp zimbramon[13151]: 13151:info: Stopping snmp
Nov 4 15:53:40 smtp zimbramon[13151]: 13151:info: Stopping antivirus
Nov 4 15:53:40 smtp zimbramon[13151]: 13151:info: Stopping antispam
Nov 4 15:53:41 smtp amavis[4170]: Net::Server: 2007/11/04-15:53:41 Server closing!
Nov 4 15:53:43 smtp zimbramon[13151]: 13151:info: Stopping imapproxy
Nov 4 15:53:45 smtp zimbramon[13151]: 13151:info: Stopping mailbox
Nov 4 15:53:47 smtp clamd[4090]: Pid file removed.
Nov 4 15:53:47 smtp clamd[4090]: Exiting (clean)
Nov 4 15:53:47 smtp clamd[4090]: --- Stopped at Sun Nov 4 15:53:47 2007

Rest of file is too long to post. But I also noticed this warning in the reboot:

Nov 4 15:56:00 smtp postfix/postfix-script: warning: not owned by root: /opt/zimbra/postfix-2.2.9/conf/main.cf

and I have no idea what this is:

desktop:/home/build/p4/main/ThirdParty/openldap/openldap-2.3.21/servers/slapd


Can somebody tell me if my system was hacked? Any idea how can I get zimbra back up? Seems logger and snmp are stopped.

Any help would be greatly appreciated. For now I just shut the server down till I can figure out what is going on.

[jholder]

Doesn't look to me like it was hacked. It was a sudo from the zimbra user.

 Nov  3 14:06:07 smtp sudo:   zimbra : TTY=unknown ; PWD=/opt/zimbra ; USER=root ; COMMAND=/opt/zimbra/libexec/zmtomcatmgr status

Zimbra was asking itself to see if everything was running, and obviously it returned that it's not running.

If you want to see why it stopped, try taking a look at /opt/zimbra/tomcat/logs/catalina.out

By the way, a good way to protect against SSH intrusions, is to block access to port 22 from the outside world.

[jholder]

To start things up, try logging into the server, and running
su - zimbra
zmcontrol start

[r_bin!]

> Doesn't look to me like it was hacked.

That's a relief! Little paranoid with all those login atempts then zimbra going down. Thanks!!

OK, I still can't get the mail program to work. Tried zmcontrol start got this:

zimbra@smtp:~$ zmcontrol start
Host smtp.(removed for privacy)
Starting logger...Done
Starting mailbox...Done.
Starting antispam...Done.
Starting antivirus...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.

But I can load the administration console, I can see all the mail accounts, just can't get any of the mail accounts to load.

I also took a look at logs/catalina.out. File is 414K, not sure what to look for. Did find some warnings though:

Nov 4, 2007 11:16:59 PM org.apache.catalina.startup.HostConfig deployDescriptor
WARNING: A docBase /opt/zimbra/apache-tomcat-5.5.15/webapps/zimbra inside the host appBase has been specified, and will be ignored
log4j:WARN No appenders could be found for logger (org.apache.catalina.session.ManagerBase).
log4j:WARN Please initialize the log4j system properly.

Any suggestions? Again thanks for the help!

[mmorse]

Welcome to the forums,
(Though unfortunate that you had to get your feet wet this way!)

Here's a list of all the zimbra related logfiles: /docs/ne/latest/administration_guide/9_Monitoring.12.1.html#1075561
tail -f /opt/zimbra/log/mailbox.log while you try to login to an account
crashes will be written to /opt/zimbra/tomcat/logs/catalina.out
(the mta and other important log is /var/log/zimbra.log)

(You can either attach them to a post in a .zip, use zimbra private pastebin - collaborative debugging tool, or send via email if you want more privacy - see our profiles for addresses)

So logger isn't critical, for troubleshooting tips see Logger - Zimbra :: Wiki
If you can get the admin console gui that's a good thing - what does a 'zmcontrol status' give you?

[r_bin!]

zmcontrol status:

antispam Running
antivirus Running
ldap Running
logger Stopped
zmlogswatchctl is not running
mailbox Running
mta Running
snmp Stopped
swatch is not running
spell Running

Interesting, there is no mailbox.log in /opt/zimbra/log

Tried to follow catalina.out, but no additional info was wrote to file when I tried to load or reload the page.

Also tried
/etc/init.d/zimbra stop
/etc/init.d/zimbra restart

[r_bin!]

This is all I got in the catalina.out tail:

zimbra@smtp:~$ tail -f /opt/zimbra/tomcat/logs/catalina.out
Zimbra server reserving server socket port=143 bindaddr=null ssl=false
Zimbra server reserving server socket port=993 bindaddr=null ssl=true
Zimbra server process is running as root, changing to user=zimbra uid=1001 gid=1001
Zimbra server process, after change, is running with uid=1001 euid=1001 gid=1001 egid=1001
Nov 5, 2007 12:38:09 AM org.apache.coyote.http11.Http11BaseProtocol start
INFO: Starting Coyote HTTP/1.1 on http-80
Nov 5, 2007 12:38:09 AM org.apache.coyote.http11.Http11BaseProtocol start
INFO: Starting Coyote HTTP/1.1 on http-7071
Nov 5, 2007 12:38:10 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 7074 ms

[mmorse]

ok before we go any further let's get you a backup:
su zimbra
zmcontrol stop
switch back to root
ps aux | grep zimbra (kill any remaining kill -9 pid)
mkdir /backup
rsync -avHK /opt/zimbra/ /backup/zimbra

and if you want you can tar it up as well:
mkdir /backuptar
tar -zcvf /backuptar/zimbra.date.backup.gz -C /backup/zimbra .

then
su zimbra
zmcontrol start

side note - let's find out if your on NE: zmcontrol -v will give us your zimbra version - if there's 'network' in the string you can open a support ticket as well)
Plus the network edition has hot, automatic backups

[r_bin!]

Release 4.0.4_GA_457.UBUNTU6 UBUNTU6

Still working on backups...

[mmorse]

He let me know that he's completed his rsync, (had trouble with tar & just needed a syntax correction). Reports that he's currently taring and also copying the tar to a safeplace.

hm...sounds like this prior sys admin is a little behind...4.5's been out for a while and we're rounding on v5
-In case he needs it, anyone got 4.0.4 Ubuntu FOSS available? (don't see it in sourceforge anymore)