100-500 user machine specs

[rotorboy]

Greetings.
I'm currently evaluating Zimbra for use in our hosting environment, to offer as a service to our clients. I have a concern about memory usage and some bugs and would love some feedback before we get too far down this road.

I'm testing the system on a CentOS 5.1 x86_64 VPS with 1.5GB RAM allocated. with only 5 or 6 users, currently none of which are logged in, the RAM usage is the entire 1.5GB + 360MB Swap. I know Java based applications tend to suck RAM like nothing else, but this seems ridiculous to me... is this really normal?

What will I see when I have 100 or 500 active users? What kind of machine specs will I need to be able to provide quality service to these users?

What kind of mileage would I get out of a Core2Q 6600 w/8GB RAM?

And regarding bugs... I see people posting on the forums and most issues seem to get a response - very encouraging. Can anyone comment on how they've been doing with this?

Much appreciated!

[Bill Brock]

I run 125 users on machine with 4GB(3.2 usable). 2.2GB usage is the most I have seen. It has never had to use the swap file. I run SuSE 10.2.

I have 35 users on a machine with 2GB and 1.3GB is all I have ever seen it use. Again, it's never had to start swapping. SuSE 10.2 on this one also. I ran this machine with 1.5GB for a while and the most me usage I saw was 1.14GB and it used a very small amount of swap. I don't remember the exact amount.

Are you running any other services on your machine? I'm surprised to see it go into swapping with no active mail going on.

[rotorboy]

Thanks a ton for your response!

I have disabled processes as outlined in the performance tuning. I think that RedHat (and thus CentOS) report RAM usage a little differently than some distributions do, but at the end of the day swap is swap and swap is bad!

I'm going to try bumping the VPS's RAM to 2GB and see what effect that has. Fortunately the machine that the VPS is on isn't being used for anything else yet ;0)

On the final machine I was planning to use 3ware 9650 series SATA II RAID 1 or RAID 10. I just read that SATA is discouraged... but is SATA RAID 1 or 10 via 3ware discouraged too? The performance document doesn't really touch on this.

[mmorse]

You will probably fare just fine. What's discouraged is RAID 5 solutions, as with lots of small files it's just too slow on writes for larger setups.
This guide is generally for systems much bigger than your's: Performance Tuning Guidelines for Large Deployments - Zimbra :: Wiki
See also: Memory usage on Zimbra 5.0.2 NE on RHEL 5.1?

[rotorboy]

Ok. I've increased the alloted RAM to 2GB and it's used up all that plus 670MB swap... and there's only 3-5 people checking it out so far. Java based applications have always been RAM/CPU heavy but this really does seem incredible... what is this RAM actually caching?

I'm finding it difficult to believe that if we buy into this system and build a machine with 4-8GB RAM it will actually support more than a couple dozen active users. I'm imagining most out of the 100-500 users will be on this at least during business hours every day...

[Bill Brock]

Neither one of my servers, 35 users with 2GB and 125 users with 3.2 GB, ever has to use the swap file.

Can you look at memory usage by PID and see what is eating up so much of your memory? I just can't believe Zimbra alone is using that much. But I could be wrong.

[rotorboy]

Here's what I'm seeing:
top - 09:50:15 up 1 day, 10 min, 1 user, load average: 0.29, 0.30, 0.22
Tasks: 116 total, 2 running, 114 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2045952k total, 1940788k used, 105164k free, 89260k buffers
Swap: 1081336k total, 607780k used, 473556k free, 363360k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2980 zimbra 18 0 1174m 883m 8552 S 0 44.2 3:34.55 java
2632 zimbra 25 0 614m 110m 11m S 0 5.5 0:21.89 java
3175 zimbra 18 0 151m 58m 716 S 0 2.9 0:04.89 clamd
1616 zimbra 19 0 594m 48m 4064 S 0 2.4 0:09.49 slapd
3146 zimbra 15 0 240m 47m 3480 S 0 2.4 0:00.43 amavisd
3154 zimbra 16 0 240m 47m 3520 S 0 2.4 0:00.44 amavisd
2659 zimbra 18 0 341m 37m 3544 S 0 1.9 0:20.30 mysqld
1500 root 15 0 256m 36m 3428 R 0 1.8 0:24.31 yum-updatesd
8754 zimbra 15 0 98188 9932 1792 S 0 0.5 0:00.15 perl
8708 zimbra 15 0 98060 9876 1792 S 0 0.5 0:00.12 perl
1170 root 15 0 23588 8048 300 S 0 0.4 0:00.00 restorecond
8886 zimbra 18 0 40900 7192 1136 S 0 0.4 0:16.76 zmmtaconfig
2219 zimbra 15 0 212m 5956 2696 S 0 0.3 0:11.34 mysqld
8750 zimbra 22 0 87636 5672 1708 S 0 0.3 0:00.06 swatch
8662 zimbra 21 0 87640 5648 1708 S 0 0.3 0:00.05 logswatch
8784 zimbra 16 0 94904 5648 2180 S 0 0.3 0:00.15 zmlogger
1156 root 12 -3 120m 3840 1548 S 0 0.2 0:01.17 python

Java is gobbling up RAM and Swap... like it has nothing better to do... Yum is using a bit, but mostly it's all zambra related processes...

[Bill Brock]

Here's my mem usage. I pulled it from Webmin instead of top. I have 25 users (not 35 like I originally said).

Display : PID User Memory CPU Search Run..

Real memory: 1.98 GB total / 968.19 MB free Swap space: 2.01 GB total / 2.01 GB free

Process ID Owner Size Command
6077 zimbra 860544 kB /opt/zimbra/jdk1.6.0_04/bin/java -Xms608m -Xmx608m -client -XX:NewRatio=2 -Djava ...
3576 zimbra 390584 kB /opt/zimbra/openldap/libexec/slapd -l LOCAL0 -4 -u zimbra -h ldap://webmail.broc ...
4089 zimbra 342712 kB /opt/zimbra/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --based ...
3981 zimbra 105296 kB /opt/zimbra/logger/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.logg ...
6197 zimbra 74676 kB /opt/zimbra/clamav/sbin/clamd --config-file /opt/zimbra/conf/clamd.conf
32563 zimbra 65424 kB amavisd (ch10-avail)
4720 zimbra 64772 kB amavisd (ch18-avail)
30179 zimbra 64592 kB amavisd (ch9-avail)
8947 zimbra 63944 kB amavisd (ch19-avail)
12780 zimbra 63932 kB amavisd (ch14-avail)
32559 zimbra 63540 kB amavisd (ch10-avail)
28103 zimbra 63408 kB amavisd (ch8-avail)
14784 zimbra 62968 kB amavisd (ch1-avail)
7912 zimbra 62860 kB amavisd (ch5-avail)
13168 zimbra 62048 kB amavisd (ch2-avail)
6139 zimbra 58440 kB amavisd (master)
6241 zimbra 12960 kB /opt/zimbra/httpd-2.2.6/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
6242 zimbra 12960 kB /opt/zimbra/httpd-2.2.6/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
6243 zimbra 12960 kB /opt/zimbra/httpd-2.2.6/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
6244 zimbra 12960 kB /opt/zimbra/httpd-2.2.6/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
6260 zimbra 12960 kB /opt/zimbra/httpd-2.2.6/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
6261 zimbra 12960 kB /opt/zimbra/httpd-2.2.6/bin/httpd -k start -f /opt/zimbra/conf/httpd.conf
18862 root 11472 kB /usr/libexec/webmin/proc/index_size.cgi
3155 root 11264 kB /usr/sbin/snmpd -r -A -LF d /var/log/net-snmpd.log -p /var/run/snmpd.pid
3013 root 9948 kB /sbin/auditd -n
11498 postfix 9084 kB smtpd -n smtp -t inet -u
18110 postfix 8996 kB smtpd -n smtp -t inet -u
16016 postfix 8992 kB smtpd -n smtp -t inet -u
18108 postfix 8992 kB smtpd -n smtp -t inet -u
4870 zimbra 8496 kB /usr/bin/perl /tmp/.swatch_script.4861
4808 zimbra 8356 kB /usr/bin/perl /tmp/.swatch_script.4803
3498 root 7248 kB /usr/bin/perl /usr/libexec/webmin/miniserv.pl /etc/webmin/miniserv.conf
4926 zimbra 7244 kB /usr/bin/perl /opt/zimbra/libexec/zmmtaconfig
6356 postfix 7080 kB qmgr -l -t fifo -u
6740 postfix 7032 kB tlsmgr -l -t unix -u
6741 postfix 7024 kB anvil -l -t unix -u
6343 root 7008 kB /opt/zimbra/postfix-2.4.3.4z/libexec/master
6358 zimbra 6972 kB /opt/zimbra/cyrus-sasl-2.1.22.3z/sbin/saslauthd -r -a zimbra
6361 zimbra 6972 kB /opt/zimbra/cyrus-sasl-2.1.22.3z/sbin/saslauthd -r -a zimbra
6362 zimbra 6972 kB /opt/zimbra/cyrus-sasl-2.1.22.3z/sbin/saslauthd -r -a zimbra
6363 zimbra 6972 kB /opt/zimbra/cyrus-sasl-2.1.22.3z/sbin/saslauthd -r -a zimbra
6364 zimbra 6972 kB /opt/zimbra/cyrus-sasl-2.1.22.3z/sbin/saslauthd -r -a zimbra
17298 postfix 6864 kB trivial-rewrite -n rewrite -t unix -u
10385 postfix 6724 kB showq -t unix -u
7474 postfix 6720 kB pickup -l -t fifo -u
3487 root 5884 kB /usr/sbin/sshd -o PidFile=/var/run/sshd.init.pid
4861 zimbra 5816 kB /usr/bin/perl /opt/zimbra/libexec/swatch --config-file=/opt/zimbra/conf/swatchrc ...
4803 zimbra 5812 kB /usr/bin/perl /opt/zimbra/libexec/logswatch --config-file=/opt/zimbra/conf/logsw ...
2338 haldaemon 5680 kB /usr/sbin/hald --daemon=yes
6398 zimbra 5172 kB /usr/bin/perl -w /opt/zimbra/libexec/zmstat-proc
6400 zimbra 5048 kB /usr/bin/perl -w /opt/zimbra/libexec/zmstat-cpu
6402 zimbra 5044 kB /usr/bin/perl -w /opt/zimbra/libexec/zmstat-vm
6408 zimbra 5044 kB /usr/bin/perl -w /opt/zimbra/libexec/zmstat-mysql
6411 zimbra 4912 kB /usr/bin/perl -w /opt/zimbra/libexec/zmstat-mtaqueue
3062 root 4716 kB /usr/sbin/cupsd
3219 root 4428 kB /usr/sbin/powersaved -d -f /var/run/acpid.socket -v 3
2302 messagebus 3548 kB /usr/bin/dbus-daemon --system
2327 root 3140 kB /usr/sbin/polkitd
2339 root 2948 kB hald-runner
3878 zimbra 2900 kB /bin/sh /opt/zimbra/logger/mysql/bin/mysqld_safe --defaults-file=/opt/zimbra/con ...
3995 zimbra 2900 kB /bin/sh /opt/zimbra/mysql/bin/mysqld_safe --defaults-file=/opt/zimbra/conf/my.cn ...
2966 root 2104 kB /sbin/syslog-ng
6480 root 2060 kB /sbin/mingetty --noclear tty1
6484 root 2060 kB /sbin/mingetty tty2
6488 root 2060 kB /sbin/mingetty tty3
6492 root 2060 kB /sbin/mingetty tty5
6490 root 2056 kB /sbin/mingetty tty4
6503 root 2056 kB /sbin/mingetty tty6
2943 mdnsd 2048 kB /usr/sbin/mdnsd -f /etc/nss_mdns.conf -b
2407 haldaemon 2024 kB hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
2418 haldaemon 2024 kB hald-addon-keyboard: listening on /dev/input/event1
938 root 1964 kB /sbin/udevd --daemon
3011 root 1960 kB /usr/sbin/cron
2689 root 1828 kB hald-addon-storage: polling /dev/hda
2317 root 1824 kB /sbin/resmgrd
2969 root 1720 kB /sbin/klogd -c 1 -x -x
6605 zimbra 1672 kB /usr/bin/vmstat -n -S K 30
2980 nobody 1632 kB /sbin/portmap
2691 root 1612 kB /sbin/dhcpcd -C -D -K -R -N -t 999999 -h webmail -c /etc/sysconfig/network/scrip ...
2274 root 1584 kB /sbin/acpid
6076 root 1584 kB /opt/zimbra/libexec/zmmailboxdmgr start -Xms608m -Xmx608m -client -XX:NewRatio=2 ...
3156 root 1440 kB zmd /usr/lib/zmd/zmd.exe --sleep 82199
1 root 740 kB init [3]
2 root 0 kB [migration/0]
3 root 0 kB [ksoftirqd/0]
4 root 0 kB [events/0]
5 root 0 kB [khelper]
6 root 0 kB [kthread]
9 root 0 kB [kblockd/0]
10 root 0 kB [kacpid]
87 root 0 kB [cqueue/0]
88 root 0 kB [kseriod]
130 root 0 kB [pdflush]
131 root 0 kB [pdflush]
132 root 0 kB [kswapd0]
133 root 0 kB [aio/0]
380 root 0 kB [kpsmoused]
766 root 0 kB [scsi_eh_0]
890 root 0 kB [reiserfs/0]
1266 root 0 kB [khubd]
2372 root 0 kB [kauditd]

[reza225]

i think we're around the 150 user mark with Dual Core Opteron 2.4ghz, 4gb of RAM and a SATA RAID 10 1TB setup. Runs pretty good so far, my only issue is that since it's in a VM Host there is a bit of an I/O bottleneck. I'm hoping this issue will be resolved when we move to ESX server.
I would really suggest though that if you're going to be building a large scale solution to get the NE edition and go with some SCSI driives. The NE edition has the HSM which will allow you to store recent email on fast disks (SCSI) and then move older email (30+ days) to a seperate SATA store.

top - 22:28:12 up 1 day, 15:42, 1 user, load average: 0.47, 0.59, 0.79
Tasks: 109 total, 3 running, 106 sleeping, 0 stopped, 0 zombie
Cpu(s): 49.2% us, 28.6% sy, 0.0% ni, 21.6% id, 0.0% wa, 0.3% hi, 0.3% si
Mem: 3606552k total, 2838424k used, 768128k free, 60316k buffers
Swap: 2031608k total, 2920k used, 2028688k free, 837096k cached

after a week or two our swap will creep around 300-500mb. it doesn't seem to affect performance a whole lot, once you're getting around 1gb of swap you should probably add more ram.

[dwmtractor]

Rotorboy,

I notice you never posted your Zimbra version, just your OS. Adding the version information may help.

However the issue of RAM as you show it does not seem to mirror the usage most folks experience. My own system supports 35 users on an old Pentium-III 1.4GHz with 2GB and I never see the swap get touched. However, even when I was testing with 2 users, the system ground terribly until I upped the RAM to 2 GB; that really seems to be a magic number for even eval usage, and the RAM doesn't seem to get hit that much harder for 20-30 users than it does on idle, in my system at least.

I am wondering if your VM/VPS may be part of the problem. While I know people do run Zimbra in VMWare and probably other virtual systems as well, I also have observed odd problems for some VM users that don't show up in a vanilla OS installed on a hardware machine. Just a guess and maybe wildly off base, but worth thinking about. . .

Cheers,

Dan