LMTP Problems

[secbreak]

Hello,

I recently installed Zimbra and i'm having problems.
A few minutes after i started zimbra all the messages gives the following error.

May 11 17:12:10 stanley postfix/lmtp[15502]: C06625145C7: to=, relay=stanley.mail.com[192.168.220.8]:7025, delay=1438, delays=836/0.07/0.01/602, dsn=4.4.2, status=deferred (conversation with stanley.mail.com[192.168.220.8] timed out while sending end of data -- message may be sent more than once)

When i stop and start zimbra the messages start to flow, but after a few minutes the error came back again.

More info:
+100 users
Release 5.0.14_GA_2850.UBUNTU8_64 UBUNTU8_64 FOSS edition
Ubuntu 8.04.2 64bits
Xeon Dual Core 2.4Ghz
5Gb RAM



Thanks.

[phoenix]

Have you disabled Amavis on this server? What does the following show:

zmcontrol status

[secbreak]

Hi, result:

antivirus Running
ldap Running
logger Running
mailbox Running
mta Running
snmp Running
spell Running
stats Running

[phoenix]

When you get this problem if you then go to the Admin UI, can you see the email deferred? If it is deferred, doe re-queueing the email get it to be delivered? What other messages are in the log files when this happens? Do you have a LAN DNS server? Is it on the Zimbra server or another machine? Is this a new install or an update? Have you always had this problem or has it just started?

[secbreak]

Yes, i can see them on the AdminUI.
Re-queuing the messages doesn't work.
I didn't find any other errors messages on my logs.
I use a Split-DNS configuration.
All my email system is on the zimbra machine.
It's a fresh new install, i never used zimbra before.
I can say for sure that the problem only happens when the server gets a lot of messages.

PS: When the error started, i did a telnet on localhost port 7025 and i didn't get any banner, but when the messages are being delivered i got the banner.

Thank you for help.

[phoenix]

Any firewall or SElinux (AppArmor) active on this system? Is this Zimbra server a VM or real hardware? Please post the output of the following commands (run on the Zimbra server):

cat /etc/hosts
cat /etc/resolv.conf
dig yourdomain.com mx 
dig yourdomain.com any
host `hostname`  <-- type that exactly and use backticks not single quotes

[secbreak]

No firewall or apparmor.
It's a real hardware.

cat /etc/hosts

127.0.0.1       localhost.localdomain localhost
192.168.220.8   stanley.mydomain.com   stanley

cat /etc/resolv.conf

search mydomain.com
nameserver 192.168.220.8

dig mydomain.com mx

; <<>> DiG 9.4.2-P2 <<>> mydomain.com mx
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42830
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2

;; QUESTION SECTION:
;mydomain.com.                 IN      MX

;; ANSWER SECTION:
mydomain.com.          604800  IN      MX      10 stanley.mydomain.com.

;; AUTHORITY SECTION:
mydomain.com.          604800  IN      NS      mydomain.com.

;; ADDITIONAL SECTION:
stanley.mydomain.com.  604800  IN      A       192.168.220.8
mydomain.com.          604800  IN      A       192.168.220.8

;; Query time: 0 msec
;; SERVER: 192.168.220.8#53(192.168.220.8)
;; WHEN: Tue May 12 11:31:30 2009
;; MSG SIZE  rcvd: 101

dig mydomain.com any

; <<>> DiG 9.4.2-P2 <<>> mydomain.com any
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53347
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;mydomain.com.                 IN      ANY

;; ANSWER SECTION:
mydomain.com.          604800  IN      SOA     mydomain.com. root.mydomain.com. 2009021801 604800 86400 2419200 604800
mydomain.com.          604800  IN      NS      mydomain.com.
mydomain.com.          604800  IN      A       192.168.220.8
mydomain.com.          604800  IN      MX      10 stanley.mydomain.com.

;; ADDITIONAL SECTION:
mydomain.com.          604800  IN      A       192.168.220.8
stanley.mydomain.com.  604800  IN      A       192.168.220.8

;; Query time: 0 msec
;; SERVER: 192.168.220.8#53(192.168.220.8)
;; WHEN: Tue May 12 11:32:23 2009
;; MSG SIZE  rcvd: 158

host `hostname`

stanley.mydomain.com has address 192.168.220.8

[secbreak]

Now, the log shows:

May 12 12:20:02 stanley postfix/qmgr[18830]: EF86D5141A8: to=, relay=none, delay=2262, delays=1359/903/0/0, dsn=4.3.0, status=deferred (mail transport unavailable)

zmcontrol status

Host stanley.finnet.com.br
        antispam                Running
        antivirus               Running
        ldap                    Running
        logger                  Running
        mailbox                 Running
        mta                     Running
        snmp                    Running
        spell                   Running
        stats                   Running

[secbreak]

New info
Im running zmlmtpinject (at the zimbra machine) with -z 10 -t 30 delivering a dir with 200 messages to 2 mailbox.

LmtpInject Finished
submitted=1980 failed=0
maximum concurrent active connections: 30
27.82s, 14.05ms/msg, 71.18mps            
average message zie = 66.02KB

When i run with -t 50

LmtpInject Finished
submitted=1980 failed=0
maximum concurrent active connections: 50
328.18s, 165.75ms/msg, 6.03mps
average message zie = 66.02KB

[secbreak]

I made some config changes:

zmprov ms stanley.finnet.com.br zimbraLmtpNumThreads 40
zmlocalconfig -e postfix_lmtp_destination_concurrency_limit=20
zmlocalconfig -e mailboxd_java_options="-client -XX:NewRatio=2 -XX:MaxPermSize=128m -Djava.awt.headless=true -XX:SoftRefLRUPolicyMSPerMB=1 -XX:+UseParallelGC"
fs.file-max=16384
kernel.threads-max=2048
net.ipv4.tcp_fin_timeout=15
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1 
mysql  table_cache = 1200
zmlocalconfig -e zimbra_index_max_uncommitted_operations=0
zmprov ms zimbraMessageCacheSize 104857600
ldap threads 16
ldap cachesize 50000
ldap idlcachesize 50000
ldap set_cachesize 0 524288000 1

The situation is much better than before, but the server is taking so long to deliver the messages to the mailbox, about 20-25min at peak moments.

I only have about 110 users but the volume of messages is huge, i have several systems that send notifies to distributions lists (with around 30 mail each). And it's in these moments that the errors appears.

May 11 17:12:10 stanley postfix/lmtp[15502]: C06625145C7: to=, relay=stanley.mail.com[192.168.220.8]:7025, delay=1438, delays=836/0.07/0.01/602, dsn=4.4.2, status=deferred (conversation with stanley.mail.com[192.168.220.8] timed out while sending end of data -- message may be sent more than once)

Can someone with a large implementation tells me some hints to tuning my system please?

PS: I'm using 2 SATA Disks in RAID (mirroring), and i'm thinking that's could be the problem.