How does HTTP IP based throttling exactly works?
Release 8.0.4.GA.5737.UBUNTU12.64 UBUNTU12_64 FOSS edition.
We have one z-push + zimbra backend (foss activesync) servers that need to do a lot of http requests to our mailbox node. It gets 503 pages frequently.
The aparent initial solution (as recommended in z-push backend zimbra release notes) is to add the z-push server to zimbraHttpThrottleSafeIPs attribute and restart mailboxd service thanks to zmmailboxdctl. It's explained in DoSFilter Zimbra Wiki page. By the way if you want to understand if zimbra server ips and/or localhost ips are added or not when establishing your own ips (not default) the answer is yes. If I'm not mistaken the associated source code can be found at: Github - zimbra-sources / main / ZimbraServer / src / java / com / zimbra / cs / servlet / DoSFilter.java (Fetched at 19 November 2013).
I still get 503 pages frequently even if the z-push server ip is found in zimbraHttpThrottleSafeIPs attribute.
I can assure you that the ip is there because when checking mailbox.log file you can see something like:
where the ip is found.
Configured whitelist IPs = 22.214.171.124,126.96.36.199,188.8.131.52,184.108.40.206,127.0.0.1,::1
I was afraid that 503 page was because of Zimbra mailbox server being busy but that's not true because I find at /opt/zimbra/log/zmmailboxd.out file:
Further investigation about zimbraHttpThrottleSafeIPs
2014-01-04 04:35:49.817:WARN:oejs.DoSFilter:DOS ALERT: ip=220.127.116.11,session=null,user=null
In the DoSFilter Zimbra Wiki page they point to: Zimbra - Bug 66921 - throttling to prevent DoS attacks.
Here you can find that maybe not only witelist IPs are taken into account but before them taken into account there are two throttles based on account and session:
For HTTP connections, the DosFilter throttles based on the following.
if (valid authToken is present)
Throttle based on account
else if (JSESSIONID is present with the request)
Throttle based on session
else if remotePort set to false
Throttle based on IP address
else if remotePort set to true
Throttle based on IP+port (essentially, a connection)
I have even found what I think it's the associated source code for this algorithm: Github - zimbra-sources / main / ZimbraServer / src / java / com / zimbra / cs / servlet / ZimbraQoSFilter.java (Fetched at 19 November 2013).
Not related with my problem but I'm going to add here that you cannot add an entire subnet in this attribute, just single IPs. Bug 85183 - Allow network addresses in zimbraHttpThrottleSafeIPs
Further investigation about Throttling in Jetty
It seems that upstream jetty has a fixed setting for whitelisting ips in its DoS named ipWhiteList. Check Jetty Denial of Service Filter if you want more information.
But according to Bug 79530 - Use LDAP attribute for DOS filter whitelist they managed to convert it into a LDAP attribute and thus the zimbraHttpThrottleSafeIPs attribute appears.
In the mean time there is the zimbraThrottleSafeHosts attribute which I'm not very sure if it's replaced by zimbraHttpThrottleSafeIPs or not. This quote from Bug 85020 - RFE: Add whitelist for imap_throttle_*_limits is not very meaninful:
Hosts to ignore during IP based throttling. Typically should list nginx hostname and any other mailbox servers which can proxy to this server.
0) How do avoid DoS filter to show 503 pages for a given ip (independent of user / session)?
1) Is what I have just explained in Further investigation about zimbraHttpThrottleSafeIPs the expected behaviour?
2) Can't we force a ip to excluded in DoS filter bypassing the account and session checks?
3) If I wanted to make concurrent http requests per account bigger what's the zimbra-way of setting LC.servlet_max_concurrent_http_requests_per_accoun t value? (This is maybe a workaround to avoid my 503 problems).
4) If I added ipWhiteList parametre to jetty.xml (or jetty.xml.in) manually (Check my former Further investigation about Throttling in Jetty) would it bypass account and session checks?
5) zimbraHttpThrottleSafeIPs / zimbraThrottleSafeHosts differences.
6) Any place where the Imap / Pop3 / Lmtp throttle ldap attributes (I have found them in other several ldap bugs) are explained?
7) Do the zimbra-proxy servers attempt to do any DoS filtering? And if they do the IP exceptions are also handled by zimbraHttpThrottleSafeIPs ?
So these are my questions. I thought it was more appropiated to ask help here than going around filling new issues. Hope that my gathered information helps someone else too. If you find that anything is wrong please complain, as I say, it's not obvious to gather the expected overall behaviour.
Workaround for 502 errors
What worked for me, as we have a ZCS Multiserver with zimbra-proxy was to run the suggested commands at:
Zimbra Bug 80135 - Improved proxy timeout defaults.
So the errors might not be about DDoS protection but about another issues after all.
Hope it helps you and you give feedback about your results.
I'm not marking as solved this thread because I don't have a nice explanation on what I asked for yet.
Comma separated for Whitelisted ips is the right way to go
That's wrong! I spent around two hours in the Zimbra source code (Not sure if it was 8.0.6 version or the one from Github) to confirm that the original suggested way of adding these ips (comma separated) was the right one. I have not updated the Zimbra ticket with my findings but that is what I found.
Originally Posted by liverpoolfcfan