Slow Queue Processing and SA TIMED OUT Errors

[sberkman]

All,

Recently started seeing the following errors while trying to troubleshoot slow queue processing:

Apr 16 16:02:02 hostname amavis[13605]: (13605-07-2) (!)SA TIMED OUT, backtrace: at /opt/zimbra/zimbramon/lib/Mail/SpamAssassin/Plugin/WLBLEval.pm line 292\n\teval {...} called at /opt/zimbra/zimbramon/lib/Mail/SpamAssassin/Plugin/WLBLEval.pm line 292\n\tMail::SpamAssassin::Plugin::WLBLEval::_check_whitelist('Mail::SpamAssassin::Plugin::WLBLEval=HASH(0xa469a9c)', 'HASH(0x8de9220)', 'admin@reignmaker.net') called at /opt/zimbra/zimbramon/lib/Mail/SpamAssassin/Plugin/WLBLEval.pm line 60\n\tMail::SpamAssassin::Plugin::WLBLEval::check_from_in_blacklist('Mail::SpamAssassin::Plugin::WLBLEval=HASH(0xa469a9c)', 'Mail::SpamAssassin::PerMsgStatus=HASH(0xaa21a64)') called at (eval 695) line 7\n\tMail::SpamAssassin::PerMsgStatus::check_from_in_blacklist('Mail::SpamAssassin::PerMsgStatus=HASH(0xaa21a64)') called at (eval 693) line 47\n\teval {...} called at (eval 693) line 46\n\tMail::SpamAssassin::Plugin::Check::_eval_tests_type9_prineg900_set3('Mail::SpamAssassi[...]

I have searched around and found some people with similar issues but no answers.

We are running Zimbra 5.02, but I don't know where to find the exact version number.

Thanks,

-Scott

[uxbod]

To get the exact version run

su - zimbra
zmcontrol -v

as SA is timing out it could be a DNS issue. From the ZCS server what happen is you run

dig reignmaker.net mx

[sberkman]

Thanks for the reply.

Here are the version details:

zimbra@hostname:~$ zmcontrol -v

Release 5.0.1_GA_1902.UBUNTU6 UBUNTU6 NETWORK edition

As for the DNS possibly timing out, I looked into this before I even knew this was an SA issue, and didn't find much of anything. There is a libcrypto error but I am not sure if that is involved at all, it doesn't seem to effect lookup time. Here is a lookup so you can see the response time:

zimbra@hostname:~$ dig mx reignmaker.net
dig: /opt/zimbra/openssl/lib/libcrypto.so.0.9.8: no version information available (required by /usr/lib/libdns.so.21)

; <<>> DiG 9.3.2 <<>> mx reignmaker.net
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 24165
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;reignmaker.net.                        IN      MX

;; ANSWER SECTION:
reignmaker.net.         1449    IN      MX      10 mail.reignmaker.net.

;; ADDITIONAL SECTION:
mail.reignmaker.net.    364     IN      A       216.246.192.39

;; Query time: 2 msec
;; SERVER: 10.1.1.15#53(10.1.1.15)
;; WHEN: Thu Apr 17 13:07:43 2008
;; MSG SIZE  rcvd: 69

Is there any way to get more detailed debug output from SA or the WLBLEval plugin to see why or on what step it is timing out?

Thanks,

-Scott

[phoenix]

As for the DNS possibly timing out, I looked into this before I even knew this was an SA issue, and didn't find much of anything. There is a libcrypto error but I am not sure if that is involved at all, it doesn't seem to effect lookup time. Here is a lookup so you can see the response time:

Do you have a firewall or SElinux enabled on this system? You should certainly disable SElinux if it's on. I also assume it's on a public IP rather than a LAN? I'm making that assumption but I see that your DNS server is on a LAN IP, is the Zimbra server really on a public IP?

[sberkman]

We are not using SELinux, but we do have an IPTables firewall in place as well as some ACLs on routers in front of the system. I have tried clearing the iptables rules however, and I get the same SA timeouts in the amavis log messages, so I don't think this is related.

The server itself has 2 interfaces, 1 public and one private. The public interface is where all mail and web traffic go, but we manage and monitor the system from the LAN side.

Also yes the DNS server we are using is on the LAN side as well.

-Scott

[phoenix]

What do the following return:

host `hostname`  <-- backticks not single quotes in both commands
host `zmhostname`

you can run them both as the zimbra user, please post the results.

[sberkman]

Here you are, I have anonymized the results a little for security, please let me know if this is a problem:

zimbra@hostname:/etc$ host `hostname`
host: /opt/zimbra/openssl/lib/libcrypto.so.0.9.8: no version information available (required by /usr/lib/libdns.so.21)
hostname.domain.local has address 10.1.1.120
zimbra@krusty:/etc$ host `zmhostname`
host: /opt/zimbra/openssl/lib/libcrypto.so.0.9.8: no version information available (required by /usr/lib/libdns.so.21)
hostname.domain.local has address 10.1.1.120

-Scott

[Bill Brock]

If your DNS server is on the LAN, do you have port 53 ported to it on your firewall/router/gateway? To do recursive lookups, this needs to be.

[phoenix]

Here you are, I have anonymized the results a little for security, please let me know if this is a problem:

zimbra@hostname:/etc$ host `hostname`
host: /opt/zimbra/openssl/lib/libcrypto.so.0.9.8: no version information available (required by /usr/lib/libdns.so.21)
hostname.domain.local has address 10.1.1.120
zimbra@krusty:/etc$ host `zmhostname`
host: /opt/zimbra/openssl/lib/libcrypto.so.0.9.8: no version information available (required by /usr/lib/libdns.so.21)
hostname.domain.local has address 10.1.1.120

-Scott

Yes, that's effectively your problem. From those results I assume that you are forwarding port 25 to the LAN IP? What you've done is created the Zimbra server on a LAN IP which requires a Split DNS so that your email can get delivered correctly and the host file settings match your correct IP for the server. Is there any possibility that you could move the Zimbra installation to another server inside the LAN? It's not really a good idea to have it on the firewall.

[sberkman]

There are no firewall issues here. I have done plenly of testing and there are definately not any DNS issues involved here in terms of the server's ability to perform lookups:

admin@hostname:~# dig google.com

; <<>> DiG 9.3.2 <<>> google.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 44222
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 4, ADDITIONAL: 0

;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             300     IN      A       72.14.207.99
google.com.             300     IN      A       64.233.167.99
google.com.             300     IN      A       64.233.187.99

;; AUTHORITY SECTION:
google.com.             345600  IN      NS      ns3.google.com.
google.com.             345600  IN      NS      ns4.google.com.
google.com.             345600  IN      NS      ns1.google.com.
google.com.             345600  IN      NS      ns2.google.com.

;; Query time: 27 msec
;; SERVER: 10.1.1.20#53(10.1.1.20)
;; WHEN: Thu Apr 17 15:29:32 2008
;; MSG SIZE  rcvd: 148

admin@hostname:~# dig cnn.com

; <<>> DiG 9.3.2 <<>> cnn.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58512
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 4, ADDITIONAL: 0

;; QUESTION SECTION:
;cnn.com.                       IN      A

;; ANSWER SECTION:
cnn.com.                300     IN      A       64.236.16.20
cnn.com.                300     IN      A       64.236.16.52
cnn.com.                300     IN      A       64.236.24.12
cnn.com.                300     IN      A       64.236.29.120

;; AUTHORITY SECTION:
cnn.com.                600     IN      NS      twdns-02.ns.aol.com.
cnn.com.                600     IN      NS      twdns-03.ns.aol.com.
cnn.com.                600     IN      NS      twdns-04.ns.aol.com.
cnn.com.                600     IN      NS      twdns-01.ns.aol.com.

;; Query time: 88 msec
;; SERVER: 10.1.1.20#53(10.1.1.20)
;; WHEN: Thu Apr 17 15:29:39 2008
;; MSG SIZE  rcvd: 188

admin@hostname:~# dig mx reignmaker.net

; <<>> DiG 9.3.2 <<>> mx reignmaker.net
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29733
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; QUESTION SECTION:
;reignmaker.net.                        IN      MX

;; ANSWER SECTION:
reignmaker.net.         197     IN      MX      10 mail.reignmaker.net.

;; ADDITIONAL SECTION:
mail.reignmaker.net.    653     IN      A       216.246.192.39

;; Query time: 1 msec
;; SERVER: 10.1.1.20#53(10.1.1.20)
;; WHEN: Thu Apr 17 15:29:46 2008
;; MSG SIZE  rcvd: 69

-Scott