Results 1 to 7 of 7

Thread: How does HTTP IP based throttling exactly works?

  1. #1
    adrian.gibanel.btactic is offline Senior Member
    Join Date
    Nov 2010
    Posts
    53
    Rep Power
    4

    Question How does HTTP IP based throttling exactly works?

    Zimbra version
    Release 8.0.4.GA.5737.UBUNTU12.64 UBUNTU12_64 FOSS edition.

    Initial problem
    We have one z-push + zimbra backend (foss activesync) servers that need to do a lot of http requests to our mailbox node. It gets 503 pages frequently.

    Initial solution
    The aparent initial solution (as recommended in z-push backend zimbra release notes) is to add the z-push server to zimbraHttpThrottleSafeIPs attribute and restart mailboxd service thanks to zmmailboxdctl. It's explained in DoSFilter Zimbra Wiki page. By the way if you want to understand if zimbra server ips and/or localhost ips are added or not when establishing your own ips (not default) the answer is yes. If I'm not mistaken the associated source code can be found at: Github - zimbra-sources / main / ZimbraServer / src / java / com / zimbra / cs / servlet / DoSFilter.java (Fetched at 19 November 2013).

    What happens
    I still get 503 pages frequently even if the z-push server ip is found in zimbraHttpThrottleSafeIPs attribute.
    I can assure you that the ip is there because when checking mailbox.log file you can see something like:
    Code:
    Configured whitelist IPs = 1.2.3.4,5.6.7.8,3.6.9.12,5.10.15.20,127.0.0.1,::1
    where the ip is found.
    I was afraid that 503 page was because of Zimbra mailbox server being busy but that's not true because I find at /opt/zimbra/log/zmmailboxd.out file:
    Code:
    2014-01-04 04:35:49.817:WARN:oejs.DoSFilter:DOS ALERT: ip=1.2.3.4,session=null,user=null
    Further investigation about zimbraHttpThrottleSafeIPs
    In the DoSFilter Zimbra Wiki page they point to: Zimbra - Bug 66921 - throttling to prevent DoS attacks.

    Here you can find that maybe not only witelist IPs are taken into account but before them taken into account there are two throttles based on account and session:
    For HTTP connections, the DosFilter throttles based on the following.

    if (valid authToken is present)
    Throttle based on account
    else if (JSESSIONID is present with the request)
    Throttle based on session
    else if remotePort set to false
    Throttle based on IP address
    else if remotePort set to true
    Throttle based on IP+port (essentially, a connection)
    .

    I have even found what I think it's the associated source code for this algorithm: Github - zimbra-sources / main / ZimbraServer / src / java / com / zimbra / cs / servlet / ZimbraQoSFilter.java (Fetched at 19 November 2013).

    Not related with my problem but I'm going to add here that you cannot add an entire subnet in this attribute, just single IPs. Bug 85183 - Allow network addresses in zimbraHttpThrottleSafeIPs

    Further investigation about Throttling in Jetty
    It seems that upstream jetty has a fixed setting for whitelisting ips in its DoS named ipWhiteList. Check Jetty Denial of Service Filter if you want more information.

    But according to Bug 79530 - Use LDAP attribute for DOS filter whitelist they managed to convert it into a LDAP attribute and thus the zimbraHttpThrottleSafeIPs attribute appears.

    In the mean time there is the zimbraThrottleSafeHosts attribute which I'm not very sure if it's replaced by zimbraHttpThrottleSafeIPs or not. This quote from Bug 85020 - RFE: Add whitelist for imap_throttle_*_limits is not very meaninful:
    Hosts to ignore during IP based throttling. Typically should list nginx hostname and any other mailbox servers which can proxy to this server.

    QUESTIONS

    0) How do avoid DoS filter to show 503 pages for a given ip (independent of user / session)?
    1) Is what I have just explained in Further investigation about zimbraHttpThrottleSafeIPs the expected behaviour?
    2) Can't we force a ip to excluded in DoS filter bypassing the account and session checks?
    3) If I wanted to make concurrent http requests per account bigger what's the zimbra-way of setting LC.servlet_max_concurrent_http_requests_per_accoun t value? (This is maybe a workaround to avoid my 503 problems).
    4) If I added ipWhiteList parametre to jetty.xml (or jetty.xml.in) manually (Check my former Further investigation about Throttling in Jetty) would it bypass account and session checks?
    5) zimbraHttpThrottleSafeIPs / zimbraThrottleSafeHosts differences.
    6) Any place where the Imap / Pop3 / Lmtp throttle ldap attributes (I have found them in other several ldap bugs) are explained?
    7) Do the zimbra-proxy servers attempt to do any DoS filtering? And if they do the IP exceptions are also handled by zimbraHttpThrottleSafeIPs ?

    So these are my questions. I thought it was more appropiated to ask help here than going around filling new issues. Hope that my gathered information helps someone else too. If you find that anything is wrong please complain, as I say, it's not obvious to gather the expected overall behaviour.
    Adrian Gibanel, IT Manager

    "be free, be innovative, bTactic"

    Av. Balmes, 34 4rt 1a B | Lleida, Spain 25006

    www.btactic.com

  2. #2
    Join Date
    Feb 2014
    Posts
    2
    Rep Power
    1

    Default

    Hello Folks,

    I have exactly the same problem. I am using Zimbra 8.0.6GA and am currently developing a synchronisation Java program, which should synchronise external events with zimbra calendars. For this I am using the SOAP API via the Server jars
    - zimbra-native.jar
    - zimbracommon.jar
    - zimbrasoap.jar
    - zimbrastore.jar
    - zimbraclient.jar
    to access the server the same way, as any Zimbra desktop client does it.

    While developing testcases I discovered, that when firing a lot of SOAP requests from my machine, I recently get a "HTTP/1.1 503 Service Unavailable" response from the server.

    At the same time, the mailbox.log on the server also documents this with the following logging line:
    Code:
    2014-02-18 11:52:04.948:WARN:oejs.DoSFilter:DOS ALERT: ip=192.168.1.203,session=null,user=null
    However my ips are parts of the whitelist and I also tried to configure
    - zimbraHttpDosFilterMaxRequestsPerSec to 200
    - zimbraHttpDosFilterDelayMillis to 0

    Still I am getting the above described error.

    Has anybody experience in howto correctly disable the DoS Filter for clients accessing Zimbra via the SOAP API?

    Thanx in advance,

    Gabriel

  3. #3
    adrian.gibanel.btactic is offline Senior Member
    Join Date
    Nov 2010
    Posts
    53
    Rep Power
    4

    Smile Workaround for 502 errors

    What worked for me, as we have a ZCS Multiserver with zimbra-proxy was to run the suggested commands at:

    Zimbra Bug 80135 - Improved proxy timeout defaults.

    So the errors might not be about DDoS protection but about another issues after all.

    Hope it helps you and you give feedback about your results.

    I'm not marking as solved this thread because I don't have a nice explanation on what I asked for yet.
    Adrian Gibanel, IT Manager

    "be free, be innovative, bTactic"

    Av. Balmes, 34 4rt 1a B | Lleida, Spain 25006

    www.btactic.com

  4. #4
    Join Date
    Feb 2014
    Posts
    2
    Rep Power
    1

    Default

    Quote Originally Posted by adrian.gibanel.btactic View Post
    What worked for me, as we have a ZCS Multiserver with zimbra-proxy was to run the suggested commands at:

    Zimbra Bug 80135 - Improved proxy timeout defaults.

    So the errors might not be about DDoS protection but about another issues after all.

    Hope it helps you and you give feedback about your results.

    I'm not marking as solved this thread because I don't have a nice explanation on what I asked for yet.
    I investigated in the sourcecode of zimbra which is responsible for writing the log entries mentioned above. This piece of code ( https://github.com/Zimbra-Community/...DoSFilter.java) is actually an override of the jetty DoS Filter and provides the possibility to configure the IPs to whitelist via a Zimbra LDAP Configuration attribute.

    The interesting thing is however that the other configuration attributes like zimbraHttpDosFilterMaxRequestsPerSec and zimbraHttpDosFilterDelayMillis are not fed into the filter. (At least by the version of the filter, whose sourcecode ist hosted on GitHub, but I also decompiled the jar of the Zimbra system which I am using and found no difference).

    What worked for me was to remove the filter completly from the "service" webapp in the jetty. (Via editing the web.xml in jetty/webapps/service/WEB-INF)

    Now all my soap requests are working also in batchmode... I still have to figure out, if changing the DoS filter attributes directly in the web.xml, will make my configuration work (with enabled DoS Filter).

    Cheers,
    Gabriel

  5. #5
    adrian.gibanel.btactic is offline Senior Member
    Join Date
    Nov 2010
    Posts
    53
    Rep Power
    4

    Thumbs up

    Quote Originally Posted by gabriel.gruber@workflow.at View Post
    The interesting thing is however that the other configuration attributes like zimbraHttpDosFilterMaxRequestsPerSec and zimbraHttpDosFilterDelayMillis are not fed into the filter. (At least by the version of the filter, whose sourcecode ist hosted on GitHub, but I also decompiled the jar of the Zimbra system which I am using and found no difference).
    I think your description does not apply to my problem.
    I recommend you to start a bug in bugzilla because you seem to have found something that does not work as expected by the current documentation.

    Please keep us updated with the bug number just in case it actually has to do something with my problem. I'll probably add my experience there anyways.

    Thank you for your debug efforts!
    Adrian Gibanel, IT Manager

    "be free, be innovative, bTactic"

    Av. Balmes, 34 4rt 1a B | Lleida, Spain 25006

    www.btactic.com

  6. #6
    liverpoolfcfan's Avatar
    liverpoolfcfan is offline Outstanding Member
    Join Date
    Oct 2009
    Location
    Dublin, IRELAND
    Posts
    698
    Rep Power
    6

    Default

    It appears that the Wiki documentation related to this https://wiki.zimbra.com/wiki/DoSFilter is incorrect.

    The wiki page states that

    IP addresses should be comma separated. To modify:
    zmprov mcf zimbraHttpThrottleSafeIPs 10.1.2.3,192.168.4.5
    Based on a the comments on a couple of bugzilla tickets ( https://bugzilla.zimbra.com/show_bug.cgi?id=85016 and https://bugzilla.zimbra.com/show_bug.cgi?id=79530 ) it appears these IP addresses should be added one at a time with the "+" operator (for adding multi-values to LDAP)

    zmprov mcf +zimbraHttpThrottleSafeIPs 10.1.2.3
    zmprov mcf +zimbraHttpThrottleSafeIPs 192.168.4.5
    I don't have an 8.0.4/5/6/7 to test this with - but a user of the z-push zimbra backend confirmed this had fixed the issue for him.

    Please report back if this works for you, and hopefully one of the moderators/zimbra employees can correct the Wiki page.

    Note: if you already have a comma separated value set, note down the list of IP addresses, then remove the existing value by setting
    zmprov mcf zimbraHttpThrottleSafeIPs "" (without the "+")

    then add each one back separately using the "+" in each case.

  7. #7
    adrian.gibanel.btactic is offline Senior Member
    Join Date
    Nov 2010
    Posts
    53
    Rep Power
    4

    Post Comma separated for Whitelisted ips is the right way to go

    Quote Originally Posted by liverpoolfcfan View Post
    It appears that the Wiki documentation related to this https://wiki.zimbra.com/wiki/DoSFilter is incorrect.

    The wiki page states that


    Based on a the comments on a couple of bugzilla tickets ( https://bugzilla.zimbra.com/show_bug.cgi?id=85016 and https://bugzilla.zimbra.com/show_bug.cgi?id=79530 ) it appears these IP addresses should be added one at a time with the "+" operator (for adding multi-values to LDAP)

    That's wrong! I spent around two hours in the Zimbra source code (Not sure if it was 8.0.6 version or the one from Github) to confirm that the original suggested way of adding these ips (comma separated) was the right one. I have not updated the Zimbra ticket with my findings but that is what I found.
    Adrian Gibanel, IT Manager

    "be free, be innovative, bTactic"

    Av. Balmes, 34 4rt 1a B | Lleida, Spain 25006

    www.btactic.com

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. How to disable POP3 throttling
    By djpark in forum Administrators
    Replies: 7
    Last Post: 03-24-2014, 06:53 AM
  2. 7.1.1 throttling?
    By ScottChapman in forum Administrators
    Replies: 0
    Last Post: 06-20-2011, 07:12 AM
  3. Replies: 0
    Last Post: 02-08-2010, 01:02 AM
  4. Proxy Upgrade - http no longer works only https
    By bonoboslr in forum Administrators
    Replies: 1
    Last Post: 11-24-2008, 12:38 AM
  5. Replies: 6
    Last Post: 11-18-2006, 12:42 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •