Spam storm on my ZCS... ideas welcome!

[Labsy]

Hi,

today I noticed enormous "Deferred" queue on my ZCS 7.1.4 - there were over 20.000 mails in Deferred queue, which is way above normal. Normally, I find 10-100 mails there, most are recipient errors and expired mail addresses.

After investigating a bit, I found all these deferred mail to be NDR and refusals from other mail servers, mostly from Yahoo and Hotmail:

Gigabytes! of such logs:

Jan 25 09:33:19 zimbra postfix/smtp[31036]: 8E6EFC1205B: host mta2.am0.yahoodns.net[98.139.54.60] refused to talk to me: 421 4.7.0 [TS01] Messages from my-zimbra-ip temporarily deferred due to user complaints - 4.16.55.1; see http://postmaster.yahoo.com/421-ts01.html
Jan 25 09:33:19 zimbra postfix/qmgr[2491]: A9536C124CA: from=<okuvow@yahoo.com>, size=1529, nrcpt=21 (queue active)
Jan 25 09:33:19 zimbra postfix/smtp[31324]: 33BA9C12062: host mx1.mail.eu.yahoo.com[77.238.177.9] refused to talk to me: 421 4.7.0 [TS01] Messages from my-zimbra-ip temporarily deferred due to user complaints - 4.16.55.1; see http://postmaster.yahoo.com/421-ts01.html

I tried to pinpoint the SOURCE of these mails, meaning which user is compromised... but here I stuck. How to find DOMAIN (I have 300+ domains on this server) and USER (2000+ users)?
What to look for in logs?
Which logs?

Ideas welcome!

[phoenix]

Search the forums for details on some techniques for blocking NDR spam, you should also reject unlisted recipients (details in the wiki article on improving the Anti-Spam system). If all your deferred queue is spam then block incoming mail at your firewall and purge the deferred queue after that's done open the firewall again.

[Labsy]

Hi Phoenix,

thanx for quick reply.

I successfully applied few hacks from Wiki and forum:

vim /opt/zimbra/conf/zmmta.cf
POSTCONF smtpd_reject_unlisted_recipient            no
CHANGED TO:
POSTCONF smtpd_reject_unlisted_recipient            yes

Works, OK.

vim /opt/zimbra/conf/postfix_recipient_restrictions.cf
ADDED:
reject_unknown_recipient_domain
reject_unverified_recipient

Works, OK

vim /opt/zimbra/postfix/conf/master.cf
bounce    unix  -       -       n       -       0       bounce     
CHANGED TO:
bounce    unix  -       -       n       -       0       discard

Hmmm, this one did not survive zmcontrol restart
But I think this setting is cruical.

Any idea how to set it up to survive reboot/restart?

Beside that, after purging 20.000+ deferred queue, applying above (2 of 3) hacks, rebooting...
here it is how it looks now - messages per hour:

[Labsy]

Just to confirm:
is it possible that above mentioned 1st and 2nd change, which both survive reboot, changed the behavior of my ZCS in the manner, that:
- BEFORE changes ZCS kept NDRs in its own deferred queue and senders did NOT receive it,
- while AFTER changes sender receives NDR and Greylist 421 responses?

Why asking? Because many clients, which were used to send mailings around, now complain about not being able to send mail to some recipients to which they were able before. But I think it is just the fact, that before they did not know that some of recipients were either non-existent or had mailbox full or something like.

Now, my concern:
If ZCS is now, after I've done changes, sending out NDRs..well, isnt' that just what I wanted to avoid?

[Labsy]

I am receiving many complaints from users after those changes were applied:
- some users say, that they receive GREYLISTING response from some servers (and their mail is then delivered)
- others say, that when sending to multiple recipients, their mail client (Outlook Express) sends out only part of e-mails, until it finds one non-existent address. Then it stops and nobody knows which mails were sent out sucesfully

I myself do not have neither 500 recipients to test, neither Win XP with Outlook Express to test...huh, bad luck.

On the other hand, if I turn those settings as they were, I'll receive thousands of bounces again. If I leave settings, users have problems.
I am stuck.
any idea?

[Labsy]

Seems like this change adds too strict behavoiur rule to mail server, so end-user receives all errors and warnings if on recipient's side is something wrong.
User's get confused, so I do NOT recommend doing THIS:

vim /opt/zimbra/conf/postfix_recipient_restrictions.cf
ADD:
reject_unknown_recipient_domain    <-- Seems OK
reject_unverified_recipient        <-- Do NOT add this!

[Labsy]

Sorry for almost duplicate threads, but I really need advice on how to stop ZCS from spamming around.
After last step described above, I get thousands of messages in deferred queue, and who knows how much spam has already passed thru successfully...

Thousands of such can be found in /var/log/mail.info (except of "zimbra" name, all other IP are real, and none of them is known to me):

Jan 31 09:51:34 zimbra postfix/qmgr[9685]: 75E9920C0F9: from=<hjkjdy@live.com>, size=1604, nrcpt=16 (queue active)
Jan 31 09:51:37 zimbra postfix/smtp[20034]: 75E9920C0F9: to=<betfred@cellarman.dabsol.co.uk>, relay=cellarman.dabsol.co.uk[109.203.99.180]:25, delay=13089, delays=13086/0.11/2.6/0.11, dsn=4.0.0, status=deferred (host cellarman.dabsol.co.uk[109.203.99.180] said: 451 Temporary local problem - please try later (in reply to RCPT TO command))
Jan 31 09:52:05 zimbra postfix/smtp[20052]: 75E9920C0F9: to=<dave@mostlikelytofail.com>, relay=none, delay=13117, delays=13086/0.12/31/0, dsn=4.4.1, status=deferred (connect to mostlikelytofail.com[74.220.199.6]:25: Connection timed out)

Is it just me having problems?
Or even worse - might it be I have problems because of upgrading to ZCS 7.1.4?

Until lately I only had few dozens of malformed mail, sent from my legitimate users, in deferred queue.
But lately, there are thousands of them, with FROM and TO addresses of unknown and foolish addresses.

Server is NOT open relay. Nothing of config has changed in past years.
Please, help.

[raj]

looks like you have compromised accounts...which are using SMTP AUTH to relay emails using your servers
Read the following thread and my replies to find out the compromised accounts

People spamming via my zimbra server

Raj

[Labsy]

looks like you have compromised accounts...which are using SMTP AUTH to relay emails using your servers
Read the following thread and my replies to find out the compromised accounts...

Hi Raj,
thank you for hint.
I have examined my logs with your search string, and increased it for 1 zero:

tail -n 1000000 /var/log/mail.info | grep "sasl_username=" > /tmp/smtplogins.log

But I found nothing special:
- there were about 500 rows found in last 1.000.000 bytes of log, which is quite normal operation (I think)
- repetitions were not more than 3-5 in a row from the same user, which is also quite normal
- most active users are those half dozen, which send out mailings to few hundreds of recipients...also quite normal

QUESTION: does each "sasl_username=" log row equals to 1 mail sent out? Ok, to 1 or more recipients.

QUESTION 2: I still think I would discover a lot about my problem if I could somehow view full body of deferred mail. Any idea how to display them?

[Labsy]

Here is another example, how it looks in /var/log/mail.info

INTERESTING: 123.456.789.1 in this log is IP of my WEB HOSTING cisco firewall. So most probably one of the users has leak in WEB form (too weak captcha).
But despite that...well, my WEB HOSTING users are all forced to authenticate...so there should be some relation with one of ZIMBRA accounts.

BUT HEY...how can I find out, which user is this?
I checked LOG for "sasl_username", but none of web users were authenticated +/- 1 hour to this log!?
Who is sending out SPAM?
How to find the account name?

Jan 31 07:16:53 zimbra postfix/smtpd[10539]: 7B35B20C066: client=cisco-out.myFirewall.com[123.456.789.1]
Jan 31 07:16:53 zimbra postfix/cleanup[10531]: 7B35B20C066: message-id=<20120131061653.7B35B20C066@zimbra.hostname.com>
Jan 31 07:16:54 zimbra postfix/qmgr[9685]: 7B35B20C066: from=<isax@mail.com>, size=1117, nrcpt=16 (queue active)
Jan 31 07:16:54 zimbra postfix/cleanup[10531]: 7B6E920C07C: message-id=<20120131061653.7B35B20C066@zimbra.hostname.com>
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<adehomont@aol.com>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<adeczyk@clerk.com>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<adehn@freizeit-kids.de>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<adefish@fsmail.net>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<adekweiss@gmx.de>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<ade.casino@googlemail.com>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<ade@hotmail.com>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<adeelnadeem786@hotmail.com>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<ade5@hotmail.de>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<adeenko@rambler.ru>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<adehn@t-online.de>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<adeilza@tele.ch>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<addys@web.de>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<adedeoglu@web.de>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<adehn@web.de>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/smtp[10532]: 7B35B20C066: to=<adedic00@yahoo.de>, relay=127.0.0.1[127.0.0.1]:10024, delay=1.1, delays=0.55/0/0.01/0.53, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 7B6E920C07C)
Jan 31 07:16:54 zimbra postfix/qmgr[9685]: 7B35B20C066: removed