Results 1 to 6 of 6

Thread: Scaling ZCS regionally - performance solutions

  1. #1
    j2b's Avatar
    j2b
    j2b is offline Special Member
    Join Date
    Sep 2008
    Location
    Latvia
    Posts
    164
    Rep Power
    6

    Cool Scaling ZCS regionally - performance solutions

    Some time ago I was watching Zimbra webinar on issue "Why ZCS is better than Gmail". While I was expecting more focus on interface benefits and ZCS offered services, more or less, Zimbra CS turned out to be very good solution (alternative) as a local deployment, e.g. leaving out of game aka ZCS service providers. At least that was my understanding of this positioning, as the benefits were leaning more towards code, software and deployment ease and control. Although, from the other side, ZCS, practically, is the only packaged solution capable of multidomains and service providers oriented solutions, including stable and very usable web interface, including different mobile/offline solutions. Honestly, our customers say, that they like ZCS web client better, than Gmail. And for this I think, we have to thankyou Microsoft Why? Our customers say, that Gmail interface is something you have to learn, but in ZCS - everything looks very familiar to MS Outlook .

    Nevertheless, this was not my main point of discussion. My question or discussion offer is connected to scaling ZCS as a provider regionally. I understand, that there are very many other connected issues for that, like international or customer internet access speeds, as well as their computer power and ISP offered speeds, but experimenting and testing in different manner, I came to issue, that I have to solve ZCS Web client operational speed. Sometimes our customers complain, that ZCS web client is very slow, and in this case, they compare it to Gmail interface speed, which is much better, taking in mind related networking issues, etc. Due to this, I was testing zimbra on RedHat Enterprise Cloud, increasing machines to beefy values, without any significant impact at the customer end. And I was starting to think about reginal distribution. I would be glad to have discussion with people of some experience in this field.

    My initial thoughts of solution findings come down to the following possibilities (not in specific order for importance/effectivity):

    1. Building whole ZCS stack closer to actual customers (e.g. opening accounts in local datacenter companies, and servicing such customers from there).
    2. Moving ZCS Web client static files to CDNs (by implementation of hand made rewrite rules for such files).
    3. Creating distributed network of Zimbra Proxy servers (leaving main mailboxes in current datacenter, due to lower service and management costs).
    4. Creating distributed newtork of 3-rd party Proxy servers with cashing, tuned to authenticated sessions, etc.


    SOME ASSUMPTIONS ON VARIATIONS ABOVE

    Overall, speed issue is connected with time, from entering login info, till loading full web application. Afterwards, operations are understandable, if, for example, user is operating with large attachments, or so. But feeling of first screen being slow, lays down this speed increasion issue.


    ZCS CLOSER TO CUSTOMERS

    This solution would be double, tripple, etc. in administration tasks, as well it'l increase operational costs. But main benefit will be only for customers working locally only (regarding specific region of deployment). Mainly our issues with ZCS speed raised with customers working in one company in different regions, and travelling arround, where in such case this increase of opeeration speed in a region, will give only small benefit, comparing to possible additional operational costs.


    ZCS STATIC FILES TO CDN

    Although, ZCS web client interface is only arround 550-600KB in weight, Yslow testing raises 2 issues - CDN usage for static files and Decreasing of number of DOM elements. While the last task would be impossible, I think, that rewriting URLs with some kind of proxy implementation, could be done, to move these static graphic and script files to CDN, for distribution. Although, can not comment on real speed increasing benefits. May be somebody can comment on this?


    ZCS PROXY DISTRIBUTION NETWORK

    While ZCS has its own solution for proxy, it is probably wise to use it for this aim, thus potentially leaving data transfers between regional proxy and main mailbox in central datacenter. And again, what could be expected increases or benefits in terms of web client speed? Probably exploring ZCS Proxy Memcache settings, there could be some gains of performance too, if tuning can help.


    3-RD PARTY PROXY NETWORK

    Referencing this way of solution, it is known, that memcache is usually used for non-authenticated sessions - e.g. the same static files or publicly available data. But there are solutions for implementations of authenticated user caching, which probably could be a solution in ZCS web client case, to gain more speed, depending on the region, a user is accessing his/her account via web client.

    Or may be your practice shows, that web interface speeds are not so related to internet networking, but more relay on ZCS inside network and performance tuning between main servers - proxy, MTAs, Mailbox and LDAP (all run separately). Some times I have a feeling, that LDAP server is the narrowest link. While LDAP requests are relatively small, it somehow gives a feeling, that this timeout is connected with authentication process. And afterwards it is successfull, it starts to load web application. But increasing of technical parameters for LDAP, didn't give any increase in this process. I was testing this on small multiserver installation - arround 60 accounts, including IMAPs, POPs and web access. Currently LDAP runs on 2GB RAM, but I was increasing up to 8GB, with no feelable difference. Although, locally, where datacenter is located, Webclient is rather quick, even on middlesized DSL connections.

    Can we compile some recomendations on these issues?

  2. #2
    Krishopper is offline Dedicated Member
    Join Date
    Dec 2006
    Location
    Minneapolis MN
    Posts
    777
    Rep Power
    9

    Default

    Quote Originally Posted by j2b View Post
    ZCS CLOSER TO CUSTOMERS

    This solution would be double, tripple, etc. in administration tasks, as well it'l increase operational costs. But main benefit will be only for customers working locally only (regarding specific region of deployment). Mainly our issues with ZCS speed raised with customers working in one company in different regions, and travelling arround, where in such case this increase of opeeration speed in a region, will give only small benefit, comparing to possible additional operational costs.
    We are doing this for our customers. We have several locations, and we allow customers to manually choose where they want their mailboxes created.

    We also added an option for customers so the mailboxes can follow them whereever they go, if they travel. For customers that chose that option, we have systems set up to monitor the Zimbra log files to see where they log in from. If we notice their Mailbox is in Dallas, TX, and they start logging in from an IP in France, we will magically move their mailbox to our Amsterdam server.

    Leading into your CDN point, we provide our users with a single login address, regardless of where their mailbox is. Our DNS servers also respond with the IP address from our datacenter that is closest to them, so they will hit that first. So the login pages come up significantly faster, and hopefully their mailbox is already local to them so when they get redirect to the right mailbox server, things will still be fast.

    Now, Zimbra doesn't have a lot of this "geography balancing" stuff built into it, so we had to customize quite a bit of it, but it can be done and it can be done pretty easily.
    01 Networks, LLC / Cybernetik.net
    Zimbra NE and OSS Cloud Hosting
    Shared Web Hosting
    Consulting Services

  3. #3
    j2b's Avatar
    j2b
    j2b is offline Special Member
    Join Date
    Sep 2008
    Location
    Latvia
    Posts
    164
    Rep Power
    6

    Default

    Krishopper, thank you for your reply. Analysis your response raised a couple of questions, if you don't mind.

    We have several locations, and we allow customers to manually choose where they want their mailboxes created.
    Was your decision based on the fact, that it would be more easy to manage ZCS installations, by not tearing them into peaces, or in fact there were no real benefit in speed gain serving static files in regions locally, leaving only data to mailboxes transfered over international internet channels to main datacenter?

    ...option for customers so the mailboxes can follow them whereever they go...
    This is really interesting solution. I suspect, that this is done not only for web client, but for IMAPs/POPs too. I'd state our customer profile and positioning below, but my concern is about movement of large mailboxes over international lines. Even between datacenters in one city we had problems with data movements. Is this included in NE (as your profile says), or this is in-house solution or any other third party solution? I assume it could be some storage distribution way.

    OUR POTENTIALS AND ISSUES

    When starting design of our services four years ago, we noticed, that it would be hard to offer ZCS NE solutions to customers in our region. There are different reasons. Main of them are:

    - extensively Microsoft oriented existing contracts
    - vast amount of free or semi/free (read cheap) local e-mail service providers, including Gmail.com, which services are used very often
    - very big part of local region market is aggregated by small and in many cases un-professional e-mailing solutions

    All that turns out, that big companies (with stakeholders not from local regions) are dedicated to locally deployed collaboration systems by Microsoft or IBM Lotus Notes Domino, where would be very hard to dive in, especially if they have international e-mail and security policies (which, in fact, turns out, that nobody is complying with them), or the other significant part, who are paying very, very small money for bulk e-mail service, and even do not know how to work with other services outside POP and Inbox folder. They are more happy to see troubling adverts from local mass providers, receive and generate massive amount of spam, and not even bother about that, not talking about necessity to get e-mail with company domain.

    Only during last 2 years Gmail became popular, which turned a little bit of light on this business, teaching people that they can do more with less.

    Due to these facts, we started our project orienting on middle sized and small companies, from which most interesting are such with small distributed offices (including regionally arround Europe) or mobile workers, which travel a lot. And because of the fact, that we have to keep our operating costs low, we decided that it would be ok to popularize ZCS OS version, as our costs for service are more infrastructure and personalized (localized) support oriented. And in some cases, if any request is received, we can deploy NE solutions too.

    Our biggest competitor for now is Gmail (not comparing financial figures, to say). Just because availability and fast service speed. And we have to comply with their stated market figures.

    Our best cooperation customers are these small distributed companies, and we are looking for availability to increase our service speed in countries arround the Europe, they are operating. In most cases, there are only 3-10 users in specific region for now, and this does not let us create full blown ZCS installations locally. That is why I noted, that supporting and operation costs will be too high. The worst thing is that international lines between EU countries (I mean regular ones, which mainly are used by internet providers) are very small, thus leading to slow download speeds from our central location. And that is why I started to look for two most effective things - static file serving from CDNs, or providing a network of ZCS or 3-rd party Proxy solutions arround. E.g. if there is a customer, it really does not cost too much to place 1x or better 2x proxies in the region and make DNS changes. But I lack of possibility to test it.

  4. #4
    Krishopper is offline Dedicated Member
    Join Date
    Dec 2006
    Location
    Minneapolis MN
    Posts
    777
    Rep Power
    9

    Default

    Was your decision based on the fact, that it would be more easy to manage ZCS installations, by not tearing them into peaces, or in fact there were no real benefit in speed gain serving static files in regions locally, leaving only data to mailboxes transfered over international internet channels to main datacenter?
    It's really not that difficult for us to manage several ZCS installations. They are all part of the same Zimbra cluster, so we always have to make sure the directory server is available, but we do have multiple masters replicated to help with that. Having at least 2 ZCS servers at each location allows us to migrate mailbox back and forth between nodes when we do upgrades so we don't have to subject our customers outages. Regarding the international transport, we don't have many mailbox moves that happen to cross International lines. Our customers have departments spread across the world so they generally stay pretty static in location. We do treat all of the datacenters equally so there isn't one we consider a "main" one.

    This is really interesting solution. I suspect, that this is done not only for web client, but for IMAPs/POPs too. I'd state our customer profile and positioning below, but my concern is about movement of large mailboxes over international lines. Even between datacenters in one city we had problems with data movements. Is this included in NE (as your profile says), or this is in-house solution or any other third party solution? I assume it could be some storage distribution way.
    For simplicity, we are using POP/IMAP proxy, and DNS always returns the closest to the customer. So sometimes there is some haul across the backbone to get them their actual mailbox contents. We are running NE and OSS; our larger installation is actually OSS, since we can sell mailboxes at a cheaper rate. I was going for a "Like Google Apps, but better" motto with these services. Because we are using OSS, we had to build custom solutions to migrate mailboxes between nodes (since its not supported in OSS, only in NE). Spending quite a bit of time learning the API, how the database and stores work, we have it down pretty well. It does require a lot of testing between upgrades to make sure Zimbra didn't break any of our processes.

    [quote]Our biggest competitor for now is Gmail (not comparing financial figures, to say). Just because availability and fast service speed. And we have to comply with their stated market figures.[quote]

    They're a pretty big competition to everyone. They have an amazing spam filter, and their mail service is great. However, if you look at the fine print of their SLA, you could have a big winning advertising factor right there.

    Our best cooperation customers are these small distributed companies, and we are looking for availability to increase our service speed in countries arround the Europe, they are operating. In most cases, there are only 3-10 users in specific region for now, and this does not let us create full blown ZCS installations locally. That is why I noted, that supporting and operation costs will be too high. The worst thing is that international lines between EU countries (I mean regular ones, which mainly are used by internet providers) are very small, thus leading to slow download speeds from our central location. And that is why I started to look for two most effective things - static file serving from CDNs, or providing a network of ZCS or 3-rd party Proxy solutions arround. E.g. if there is a customer, it really does not cost too much to place 1x or better 2x proxies in the region and make DNS changes. But I lack of possibility to test it.
    Are you working with your own datacenters, dedicated servers, VPS's, colocation? We have private networks set up between our servers, so we do have a few servers out there that are simple VPS's to keep the costs down. They use multiple IPSEC VPN's with dynamic routing protocols to our other servers to create a fully routed internal network. That way we can keep an eye on traffic and route internal traffic differently or over cheaper links.
    01 Networks, LLC / Cybernetik.net
    Zimbra NE and OSS Cloud Hosting
    Shared Web Hosting
    Consulting Services

  5. #5
    j2b's Avatar
    j2b
    j2b is offline Special Member
    Join Date
    Sep 2008
    Location
    Latvia
    Posts
    164
    Rep Power
    6

    Default

    Regarding first quote of previous comment:

    But can you confirm difference for better performance, if in region locally there would be a couple of ZCS Proxy servers, which redirect traffic over int internet lines to other country, where main mailboxes are located? Or the gain in login screen and first application load would be insignifficant, comparing to work with data, which any way should be transferred internationally? Let's say, if technically speaking such improvement in speed would be only 10%, than it would be considered as a base of project to distribute mailbox sesrvers and incur such costs.

    ... It does require a lot of testing between upgrades to make sure Zimbra didn't break any of our processes...
    Yes, this was really my concern, at least in my practice, is that it is better to keep ZCS stack together without special modifications, to comply with later upgrades, although last movement from v6 to v7 was painfull. My intent was to create any additions or modifications to traffic routing outside ZCS stack, as much as it is possible, to leave ZCS intact.

    They're a pretty big competition to everyone. They have an amazing spam filter, and their mail service is great. However, if you look at the fine print of their SLA, you could have a big winning advertising factor right there.
    Yes, I could only agree. As for last year reports, assessed value of Google - 33 billion $, with own sattelite above us. But in my case, why I stated such loud statement, is that in fact about 3 years ago, Gmail was rearely used in our location (the Baltics), and increased its popularity only during last years. Not taking in mind, that it is still very hard to convince somebody to use it for bussiness (paid accounts). At least for now.

    Concerning SLA factor - yes and now. If we still compare, what could be done by $33 billion company, and small, like ours, in fact, increasing SLA statements for us would mean do more job for less price (i.e. overwork), and it probably would not cost Google too much in some time to increase SLA statements. But at the moment - yes. We only can hope, that Google will not want to eat out all possible providers, thus leaving at least some space for alternative developments and services. That is why we are orienting on personalized support, which Google does not have.

    Are you working with your own datacenters, dedicated servers, VPS's, colocation?
    A kind of combination of above, except own datacenters. We ar really small company (startup) with some kind of specialization, mentioned above, and struggling for growth and prosperity. Own datacenters would cost us too much, and according to my 4years old studies, it would be really expensive to invest in own technology and support it, while you develop new business. It could be viable after achieving some cashflow milestones, but for now, we use 2 datacenters, where collocation is used for our servers with deployment of VMware virtual datacenter (e.g. datacenter in datacenter). And for expansion, we are renting VPSes on cloud architecture, to provide backups, storage etc. At the moment it costs less to rent virtual infrastructure (including double paths, FC connections, huge storages and part of physical datacenter) rather build your own. But I hope it'll change in future

  6. #6
    Krishopper is offline Dedicated Member
    Join Date
    Dec 2006
    Location
    Minneapolis MN
    Posts
    777
    Rep Power
    9

    Default

    But can you confirm difference for better performance, if in region locally there would be a couple of ZCS Proxy servers, which redirect traffic over int internet lines to other country, where main mailboxes are located? Or the gain in login screen and first application load would be insignifficant, comparing to work with data, which any way should be transferred internationally? Let's say, if technically speaking such improvement in speed would be only 10%, than it would be considered as a base of project to distribute mailbox sesrvers and incur such costs.
    The initial load is faster if the mailbox is local to the user, but the rest of the experience isn't too bad since everything is AJAX based. I spent a month in Europe accessing my account from the US, and I honestly couldn't notice much of a difference in speed; it was still very prompt and fast.

    Yes, this was really my concern, at least in my practice, is that it is better to keep ZCS stack together without special modifications, to comply with later upgrades, although last movement from v6 to v7 was painfull. My intent was to create any additions or modifications to traffic routing outside ZCS stack, as much as it is possible, to leave ZCS intact.
    Well, we did leave the entire ZCS stack in tact the way it was. Our external processes use the API, more specifically the sync protocol, to get mailbox migrations done. There are a few cases where we touch the database manually from external scripts, but we are working on ways around that.

    A kind of combination of above, except own datacenters. We ar really small company (startup) with some kind of specialization, mentioned above, and struggling for growth and prosperity. Own datacenters would cost us too much, and according to my 4years old studies, it would be really expensive to invest in own technology and support it, while you develop new business. It could be viable after achieving some cashflow milestones, but for now, we use 2 datacenters, where collocation is used for our servers with deployment of VMware virtual datacenter (e.g. datacenter in datacenter). And for expansion, we are renting VPSes on cloud architecture, to provide backups, storage etc. At the moment it costs less to rent virtual infrastructure (including double paths, FC connections, huge storages and part of physical datacenter) rather build your own. But I hope it'll change in future
    Then you should easily be able to add a few new VPS's in countries where you have a few mailboxes. Obviously you will need to make sure that its worth the expense to add a node in a particular location and make sure enough people will use it to make it worth your time. But for administration purposes, there are plenty of tools to automate administration/patching/etc. so you can have a pretty hands-off experience. Be sure to use good monitoring software to keep an eye on network and disk utilization issues, Also pay very close attention to the vendor's SLA's to ensure that they meet or exceed the SLA's you are providing to your customers. Your SLA is only as good as your VPS provider, or datacenter provider.
    01 Networks, LLC / Cybernetik.net
    Zimbra NE and OSS Cloud Hosting
    Shared Web Hosting
    Consulting Services

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Trouble Sending mail - All Messages deferred!
    By SiteDiscovery in forum Administrators
    Replies: 7
    Last Post: 09-03-2009, 04:52 AM
  2. Replies: 41
    Last Post: 10-29-2007, 02:36 PM
  3. poor performance ZCS NE vs. ZCS OSS
    By fisch09 in forum Administrators
    Replies: 5
    Last Post: 08-19-2007, 06:17 AM
  4. ZCS 3.2 Beta Available
    By KevinH in forum Announcements
    Replies: 31
    Last Post: 07-07-2006, 03:46 PM
  5. ZCS Connector Performance
    By art_hebert in forum Zimbra Connector for Outlook
    Replies: 1
    Last Post: 03-06-2006, 06:33 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •