Connectivity problems: Connection timeout

[andre.byng]

Hi All,

Firstly thanks for any help provided, this is my first post here so I'll be happy to receive any feedback!

Now, we have had a Zimbra 5.0.6 installation for about 20 months now, and it has around 800 accounts in it now. In the last week or so we have been getting complaints from clients about the connection timing out when sending out messages. When the mail client tried to copy the message to the Sent folder the connection would time out. But since the problem appears to have spread to most of the accounts, and I have been able to replicate the error myself. The connection seems to timeout quite randomly, but it's happening a lot (around 50% of the time).

Looking at mailboxd.log, there's occasionally the following exception coming up (not sure if it's related):

2010-01-18 10:53:55,012 INFO [ImapServer-16212] [] ProtocolHandler - Exception occurred while handling connection
java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream. java:129)
at com.sun.net.ssl.internal.ssl.InputRecord.readFully (InputRecord.java:293)
at com.sun.net.ssl.internal.ssl.InputRecord.read(Inpu tRecord.java:331)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRec ord(SSLSocketImpl.java:782)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDat aRecord(SSLSocketImpl.java:739)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(A ppInputStream.java:75)
at java.io.BufferedInputStream.fill(BufferedInputStre am.java:218)
at java.io.BufferedInputStream.read(BufferedInputStre am.java:235)
at com.zimbra.cs.tcpserver.TcpServerInputStream.readL ine(TcpServerInputStream.java:73)
at com.zimbra.cs.imap.TcpImapRequest.continuation(Tcp ImapRequest.java:54)
at com.zimbra.cs.imap.TcpImapHandler.processCommand(T cpImapHandler.java:107)
at com.zimbra.cs.tcpserver.ProtocolHandler.processCon nection(ProtocolHandler.java:160)
at com.zimbra.cs.tcpserver.ProtocolHandler.run(Protoc olHandler.java:128)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Wo rker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:595)



The Datacenter hosting the machine as well as the client's ISP have reported they are not having any sort of network problems. Could this simply be due to too high usage on the server? The CPU usage has been around 50% on average during working hours. Are there any memory or CPU settings I can change to optimize Zimbra? It's the only service running on that server, so we can crank it up to full!

Please let me know if I can provide any more info!

Thanks a lot!
Andre, the desperate sysadmin!

[phoenix]

Originally Posted by andre.byng

The Datacenter hosting the machine as well as the client's ISP have reported they are not having any sort of network problems. Could this simply be due to too high usage on the server? The CPU usage has been around 50% on average during working hours. Are there any memory or CPU settings I can change to optimize Zimbra? It's the only service running on that server, so we can crank it up to full!

What's the specification of this server specifically how much RAM?

[andre.byng]

Hi Bill,

The machine has 4GB of RAM, and it's a dual Xeon X3210 2.13GHz

Andre

[phoenix]

I'd suggest that RAM is a bit on the low side for a busy server, have you done any diagnostics on the server load? Are most of your connections via fat clients (T'bird etc and IMAP) or via the web client? Have you tried running zmstat for diagnosing any problems? If not, search the wiki for 'zmstat'.

[andre.byng]

Bill, my mistake. It's actually a single Quad core Xeon.

Regards,
Andre

[andre.byng]

Yes most connections are made through IMAP on Outlook and Thunderbird. We are running an SNMP monitoring which shows the load being typically between 4 and 6 during a business day.

I have gotten a few stat's csv files from /opt/zimbra/zmstat but haven't run zmstat. I'll look into it right now.

Regards,
Andre

[andre.byng]

Bill,

Are there any tune-up that can be made via system variables and such so that we have short term improvement while we look into getting more memory onto the system?

Also, how much memory would you recommend?

Regards,
Andre

[phoenix]

Originally Posted by andre.byng

Are there any tune-up that can be made via system variables and such so that we have short term improvement while we look into getting more memory onto the system?

I forgot to ask what type of HD subsystem is on this server, any RAID?

Have a look at this article: Performance Tuning Guidelines for Large Deployments - Zimbra :: Wiki - there are tips in there for improving performance even on smaller systems, look particularly at the Memory Allocation and JVM options.

Originally Posted by andre.byng

Also, how much memory would you recommend?

That would depend on how busy the server is, without exact figures it's difficult to estimate but 2GB-4GB should make the system more responsive. As I mentioned earlier zmstat will give you lots of detailed statistics on the server behaviour.

[andre.byng]

Yeah Bill there's a RAID-1 running on the server, mirroring 2 1TB disks.

I have tried the JVM recommended options listed on the Zimbra Wiki link but when running zmcontrol stop/start, the mailbox wouldn't come back up. So I have changed the settings back.

The command used was the following:

zmlocalconfig -e mailboxd_java_options="-server -XX:NewRatio=2 -Djava.awt.headless=true -XX:MaxPermSize=128m -XX:SoftRefLRUPolicyMSPerMB=1 -XX:+UseConcMarkSweepGC -XX:PermSize=128m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/zimbra/log -XX:ErrorFile=/opt/zimbra/log/jvm_errors -Xmn250m"

Did I mess something up?

[phoenix]

Originally Posted by andre.byng

Yeah Bill there's a RAID-1 running on the server, mirroring 2 1TB disks.

The recommended RAID level is RAID 10 for any number of users more than 100(ish), you can get away with other RAID levels but increased activity of 800 users would lead me to believe you'd have an I/O problem. Check the zmstats command that I pointed you to earlier, it gives great detail on the activity of your server and you need to determine if there's any bottleneck in your system.

Originally Posted by andre.byng

I have tried the JVM recommended options listed on the Zimbra Wiki link but when running zmcontrol stop/start, the mailbox wouldn't come back up. So I have changed the settings back.

Start with the memory allocations first, have a look at the following two settings:

zmlocalconfig | grep mailboxd_java_heap_memory_percent
mailboxd_java_heap_memory_percent = 40
zmlocalconfig | grep mailboxd_java_heap_new_size_percent
mailboxd_java_heap_new_size_percent = 25

You can tweak them slightly to reduce their memory usage, do one at a time and see if there's any improvements and start with java_heap_memory_percent. Don't reduce the percentage too low or you'll also impact performance and make a note of the default settings on your server (my output above shows what the default should be).