Succesfull hacking attempts on Zimbra mailboxes (webmail)

[TomG]

Since a couple of months, we suffer from accounts being hacked by spambots who sent lots of spam (which in some cases causes our outgoing smtp server being blacklisted).

To my understanding, this is what happens:
• Spambots try to logon to the Zimbra webmail interface. It seems sometimes they succeed, even when some users have complex random passwords with special characters (&,#!,…).
• Once logged on, they start to send batch mails to a lot of e-mail addresses at ounce. Most of the times, e-mailadresses on yahoo.com, Verizon.net and Comcast.net are targeted.
• What we’ve just find out, is that the spambot also configures a forwarding of all further incoming e-mail to another address (in this case: ldlexus43@gmail.com) without leaving a copy in the actual mailbox. They do so, to make sure the user won’t be suspicious about the returning non-deliverable messages. Of course, this also causes the user not to receive valid e-mail in his/her mailbox anymore. Also, there are no traces of spam in sent items in the user’s mailbox.
The first hacked accounts we’ve encountered still had the sent spam mails in their place and the non-deliverables in the inbox. They getting smarter and smarter.
• When logging on to the admin webinterface, the last logon date doesn’t correspond to the date the incident happened. For some reason, the last logon date is not updated when the spambots do their job.
• We could trace this because our Zimbra servers are using a smart host for all outgoing e-mail (who were used to sent the spam) and and a couple of separate mx anti-virus/spam servers to handle all incoming e-mail (who received the non-deliverables) routed to the Zimbra servers.
• Unfortunately, I can’t find out the source of the spambots. What ip address did they use to logon to the webinterface? I couldn’t find those details in the Zimbra log files. Is there a specific location where webmail logons (and failed attempts) are logged to? I am aware there’s a setting that causes accounts being locked after x failed logon attempts within x minutes. But this will really cause some frustrations at our users and helpdesk.
• We’re currently using Zimbra ZCS 6.0.7 but are planning to upgrade to 7.1.0 in august. Is there a better protection in ZCS 7 against accounts being hacked?
• Of course, we would like to know if there is some advice to prevent such successful hacking attempts in the future.

Thanks in advance

[LaFong]

Originally Posted by TomG

• Is there a specific location where webmail logons (and failed attempts) are logged to?

/opt/zimbra/log/audit.log and /opt/zimbra/log/mailbox.log

• Of course, we would like to know if there is some advice to prevent such successful hacking attempts in the future.

You can use fail2ban to ban IP addresses or hosts. I use it in conjunction with iptables. For mail, here's my jail.conf entries:

[zimbra-account]

enabled  = true
filter   = zimbra
action   = iptables-allports[name=Zimbra-account]
           sendmail[name=Zimbra-account, dest=it@somewhere.com]
logpath  = /opt/zimbra/log/mailbox.log
bantime  = -1
maxretry = 5


[zimbra-audit]

enabled  = true
filter   = zimbra
action   = iptables-allports[name=Zimbra-audit]
           sendmail[name=Zimbra-audit, dest=it@somewhere.com]
logpath  = /opt/zimbra/log/audit.log
bantime  = -1
maxretry = 5

[zimbra-recipient]
              
enabled  = true
filter   = zimbra
action   = iptables-allports[name=Zimbra-recipient]
sendmail[name=Zimbra-recipient, dest=it@somewhere.com]
logpath  = /var/log/maillog
findtime = 604800
bantime  = 172800
maxretry = 5

[postfix]

enabled  = true
filter   = postfix
action   = iptables-multiport[name=Postfix, port=smtp, protocol=tcp]
           sendmail-buffered[name=Postfix, dest=it@somewhere.com]
logpath  = /var/log/maillog
bantime  = 172800
maxretry = 5

[postfix-connections]

enabled  = true
filter   = postfix-connections
action   = iptables[name=Postfix-Connections, port=smtp, protocol=tcp]
           sendmail[name=Postfix-Connections, dest=it@somewhere.com]
logpath  = /var/log/maillog
bantime  = %(permanent)s
ignoreip = 127.0.0.1

[sasl-iptables]

enabled  = true
filter   = sasl
action   = iptables-allports[name=sasl]
           sendmail[name=sasl, dest=it@somewhere.com]
logpath  = /var/log/maillog
bantime  = -1
maxretry = 5

# Stop phpMyAdmin probes

[apache-phpMyAdmin]

enabled = true
filter = apache-phpMyAdmin
action = iptables-allports[name=phpMyAdmin]
         sendmail-buffered[name=phpMyAdmin, lines=5, dest=it@somewhere.com]
logpath = /opt/zimbra/log/httpd_access.log
bantime = -1
maxretry = 1

Make sure to exempt your own networks from getting banned. In jail.conf:

ignoreip = 127.0.0.1 172.16.99.0/24 172.16.98.0/24 10.1.1.0/24

I added some time aliases to the top of jail.conf:

# Time aliases
## bantime of 3600 = 60*60 = one hour
## bantime of 86400 = 60*60*24 = one day
## bantime of 604800 = 60*60*24*7 = one week
## bantime of 2592000 = 60*60*24*30 = (approx) one month
## bantime of 31536000 = 60*60*24*365 = (approx) one year
one_hour  = 3600
one_day   = 86400
one_hour  = 3600
one_day   = 86400
one_week  = 604800
one_month = 2592000
one_year  = 31536000
permanent = -1
## Use the aliases like this:
## findtime = %(one_week)s
## bantime = %(one_week)s
## the "s" is for string, and must be included

and the filter in /etc/fail2ban/filter.d/zimbra.conf:

# Fail2Ban configuration file
#
# Author: 
#
# $Revision: 1 $
#

[Definition]

# Option:  failregex
# Notes.:  regex to match the password failures messages in the logfile. The
#          host must be matched by a group named "host". The tag "<HOST>" can
#          be used for standard IP/hostname matching and is only an alias for
#          (?:::f{4,6}:)?(?P<host>[\w\-.^_]+)
# Values:  TEXT
#
failregex = \[ip=<HOST>;\] account - authentication failed for .* \(no such account\)$
                        \[ip=<HOST>;\] security - cmd=Auth; .* error=authentication failed for .*, invalid password;$
                        ;oip=<HOST>;.* security - cmd=Auth; .* protocol=soap; error=authentication failed for .* invalid password;$
                        \[oip=<HOST>;.* SoapEngine - handler exception: authentication failed for .*, account not found$
                        WARN .*;ip=<HOST>;ua=ZimbraWebClient .* security - cmd=AdminAuth; .* error=authentication failed for .*;$
                        NOQUEUE: reject: RCPT from .*\[<HOST>\]: 550 5.1.1 .*: Recipient address rejected:

# .*\[ip=<HOST>;\] .* - authentication failed for .* \(invalid password\)
# 
# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex =

You can use a bantime of -1 to not timeout the ban, or some high number in seconds to ban for a specific time span. The banned IPs get dropped when you restart the server or fail2ban, unless you add code to reinstate the ban list across restarts. If you do that, your ban list might eventually grow huge.

I added some stuff to the /etc/init.d/fail2ban init script, to maintain IP bans across restarts:

[root@mail init.d]# diff -u fail2ban.orig fail2ban
--- fail2ban.orig       2010-02-14 09:01:56.000000000 -0700
+++ fail2ban    2011-12-22 16:17:35.000000000 -0700
@@ -37,10 +37,20 @@
         echo_failure
     fi
     echo
+    if [ -x /tmp/banned_ips ] ; then /tmp/banned_ips >/dev/null ; fi
     return $RETVAL
 }
 
 stop() {
+    echo "Saving current banned IPs..."
+    if [ -e /tmp/banned_ips ] ; then rm -f /tmp/banned_ips ; fi
+    jails=$(fail2ban-client status | grep Jail\ list: | sed 's/.*Jail list:\t\+//;s/,//g')
+    for jail in ${jails}; do
+       for ip in $(fail2ban-client status ${jail}|grep IP\ list|sed 's/.*IP list:\t//'); do
+           echo "fail2ban-client set ${jail} banip ${ip}" >>/tmp/banned_ips
+       done
+    done
+    if [ -e /tmp/banned_ips ] ; then chmod 770 /tmp/banned_ips ; fi
     echo -n $"Stopping fail2ban: "
     getpid
     RETVAL=$?

My full /etc/init.d/fail2ban init script:

#!/bin/bash
#
# chkconfig: - 92 08
# description: Fail2ban daemon
#              http://fail2ban.sourceforge.net/wiki/index.php/Main_Page
# process name: fail2ban-server
#
#
# Author: Tyler Owen
#

# Source function library.
. /etc/init.d/functions

# Check that the config file exists
[ -f /etc/fail2ban/fail2ban.conf ] || exit 0

FAIL2BAN="/usr/bin/fail2ban-client"

RETVAL=0

getpid() {
    pid=`ps -eo pid,comm | grep fail2ban- | awk '{ print $1 }'`
}

start() {
    echo -n $"Starting fail2ban: "
    getpid
    if [ -z "$pid" ]; then
        $FAIL2BAN -x start > /dev/null
        RETVAL=$?
    fi
    if [ $RETVAL -eq 0 ]; then
        touch /var/lock/subsys/fail2ban
        echo_success
    else
        echo_failure
    fi
    echo
    if [ -x /tmp/banned_ips ] ; then /tmp/banned_ips >/dev/null ; fi
    return $RETVAL
}

stop() {
    echo "Saving current banned IPs..."
    if [ -e /tmp/banned_ips ] ; then rm -f /tmp/banned_ips ; fi
    jails=$(fail2ban-client status | grep Jail\ list: | sed 's/.*Jail list:\t\+//;s/,//g')
    for jail in ${jails}; do
       for ip in $(fail2ban-client status ${jail}|grep IP\ list|sed 's/.*IP list:\t//'); do
           echo "fail2ban-client set ${jail} banip ${ip}" >>/tmp/banned_ips
       done
    done
    if [ -e /tmp/banned_ips ] ; then chmod 770 /tmp/banned_ips ; fi
    echo -n $"Stopping fail2ban: "
    getpid
    RETVAL=$?
    if [ -n "$pid" ]; then
        $FAIL2BAN stop > /dev/null
    sleep 1
    getpid
    if [ -z "$pid" ]; then
        rm -f /var/lock/subsys/fail2ban
        echo_success
    else
        echo_failure
    fi
    else
        echo_failure
    fi
    echo
    return $RETVAL
}

# See how we were called.
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  status)
        getpid
        if [ -n "$pid" ]; then
                echo "Fail2ban (pid $pid) is running..."
                $FAIL2BAN status
        else
                RETVAL=1
                echo "Fail2ban is stopped"
        fi
        ;;
  restart)
        stop
        start
        ;;
  *)
        echo $"Usage: $0 {start|stop|status|restart}"
        exit 1
        ;;
esac

exit $RETVAL

If you want to maintain banned IPs across restarts, you will probably need to patch /usr/share/fail2ban/server/filter.py in order for the banip directive to work:

[root@mail server]# diff -u fail2ban-0.8.4/fail2ban/server/filter.py filter.py
--- fail2ban-0.8.4/fail2ban/server/filter.py 2011-12-21 17:11:38.000000000 -0700
+++ filter.py   2011-12-22 16:30:13.000000000 -0700
@@ -31,7 +31,7 @@
 from mytime import MyTime
 from failregex import FailRegex, Regex, RegexException
 
-import logging, re, os
+import logging, re, os, time
 
 # Gets the instance of the logger.
 logSys = logging.getLogger("fail2ban.filter")
@@ -191,7 +191,7 @@
 
        def addBannedIP(self, ip):
                unixTime = time.time()
-               self.failManager.addFailure(FailTicket(ip, unixTime))
+               self.jail.putFailTicket(FailTicket(ip, unixTime))
                return ip
 
        ##

This patch is for v0.8.4. I'm using fail2ban 0.8.6, downloaded from github, so YMMV.

Note also regarding maintaining bans, that you may get a flood of emails from fail2ban after restarts, as it reinstates bans. I set up a filter in my email client to trash those:

Fail2Ban IP renewals:

• Match All
  • Subject contains "[Fail2Ban]"
  • Body contains "(0 failures)"
• Perform these actions:
  • Move Message to Trash

I changed the sasl.conf filter, as the default wasn't working for me:

failregex = \[<HOST>\]: SASL (?:LOGIN|PLAIN|(?:CRAM|DIGEST)-MD5) authentication failed

I added a postfix-connections.conf fail2ban filter, for IP addresses that flood postfix over the default 50 connection limit:

failregex = warning: Connection concurrency limit exceeded: 51 from (.*)\[<HOST>\]

Note also that if a legitimate user repeatedly fails to login correctly, they will get banned too. You would need to remove them from the ban list using iptables:

[root@mail fail2ban]# iptables -L --line-numbers
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
fail2ban-Zimbra-audit  tcp  --  anywhere             anywhere  

Chain fail2ban-Zimbra-audit (1 references)
num target     prot opt      source                 destination         
1   DROP       all  --       bad.spammer.com        anywhere            
2   DROP       all  --       legitimate.user.com    anywhere

or, you can display just IP addresses, to avoid the slow reverse DNS lookup:

[root@mail fail2ban]# iptables -nL --line-numbers
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
fail2ban-Zimbra-audit  tcp  --  0.0.0.0/0             0.0.0.0/0  

Chain fail2ban-Zimbra-audit (1 references)
num  target     prot opt   source               destination         
1    DROP       all  --    66.77.88.99          0.0.0.0/0            
2    DROP       all  --    11.33.55.77          0.0.0.0/0

You would remove the ban on the legitimate user like so:

iptables -D fail2ban-Zimbra-audit 2

[TomG]

Many thanks, LaFong! It seems your much more helpfull than Zimbra support itself.

Anyway, we've found out why a couple of mailboxes were abused: some users received an phising e-mail. Some users filled in the form - of course with their e-mailaddress and password. In that case, a ipban or account lockout feature won't help.

[phoenix]

Originally Posted by TomG

Many thanks, LaFong! It seems your much more helpfull than Zimbra support itself.

The people that answer on these forums are not Zimbra Support, these are Community Forums and people that post here do so in their spare time.

Originally Posted by TomG

Anyway, we've found out why a couple of mailboxes were abused: some users received an phising e-mail. Some users filled in the form - of course with their e-mailaddress and password. In that case, a ipban or account lockout feature won't help.

I'm glad you've got an answer but you could have found the same information by searching the forums, there's several threads on the topic of 'hacked mailboxes' and including resolutions.

[LaFong]

Wow, put their password on a web form, huh? So far, no user has ever done that here. If they're that foolish, I hope you're using password complexity rules and/or something like fail2ban to protect users from themselves.

[TomG]

Unfortunately, some of them are so foolish.

About 5 of our 3000 Zimbra uers (could be more, but 5 were abused this way) filled in the form. Website is already deleted, I've sent an abuse report to the hoster company.

[hateball]

We also got targeted by some spambots using dictionary attacks recently, and I hadnt thought about setting up fail2ban to monitor Zimbra, only ssh... Big thanks to LaFong for the nice filter