Upgrade to 4.0.3 and mail delivery slows to a crawl

[stbain]

We finally upgraded to the latest version over the course of this weekend. I received a report of slow mail delivery, and when I logged in to take a look I found that the mail delivery has all but come to a standstill. Have a look:

[zimbra@hostname bin]$ ./qshape | head
                               T   5  10  20  40   80  160  320  640  1280 1280+
                     TOTAL 30778 232 212 408 401 1006 2166 3826 7900 12677  1950
               domain1.com  2270  14  11  18  31   72  121  262  681   948   112
               domain2.com  2009  14  15  21  33   67  148  270  477   856   108
               domain3.com  1711  12  10  16  19   45  106  208  382   801   112
               domain4.com  1607   8  14  17  17   76  135  189  374   688    89
               domain5.com  1371  13  21  30  26   41   85  135  385   543    92
               domain6.com  1366  13  11  12  13   54   90  184  346   565    78
               domain7.com   966  11   3  16  12   29   91   92  231   428    53
               domain8.com   913  10   5  10  13   31   73   97  241   390    43
[zimbra@hostname bin]$ ./qshape active | head
                                    T  5 10 20  40  80  160  320  640 1280 1280+
                          TOTAL 22641  1  1  9 232 641 1788 3109 5889 9439  1532
                    domain1.com  1699  0  0  0  15  47  108  212  524  710    83
                    domain2.com  1455  0  0  0   9  52  118  209  358  628    81
                    domain3.com  1163  0  0  0   8  35  125  151  280  490    74
                    domain4.com  1161  0  0  1   7  30   79  151  262  556    75
                    domain5.com   992  0  0  1  23  26   70  112  284  394    82
                    domain6.com   992  0  0  0   6  34   75  153  246  416    62
                    domain7.com   730  0  0  3   8  20   70   84  180  322    43
                    domain8.com   660  0  0  0   6  15   66   71  175  283    44
[zimbra@hostname bin]$ ./qshape incoming | head
                                   T   5  10  20  40  80 160 320  640 1280 1280+
                          TOTAL 8113 259 231 418 234 324 352 744 1932 3108   511
                    domain1.com  556  17  16  28  20  19  26  63  115  219    33
                    domain2.com  554  11  12  15  17  14  27  55  115  248    40
                    domain3.com  552  16  14  22  14  24  15  43  138  230    36
                    domain4.com  443  18   9  22  12  21  26  40   92  179    24
                    domain5.com  379  13  14  16  10  16  14  33   98  145    20
                    domain6.com  376   5  13  34  11  16  13  24  102  136    22
                    domain7.com  247   5   8  12   8  16   6  23   65   95     9
                    domain8.com  236   6  11   9   7   6   9  21   51  103    13
[zimbra@hostname bin]$

I've tried turning off antispam/antivirus thinking the scanning was causing the problem, but that doesn't seem to speed things up at all. My queues are continuing to grow at a very rapid rate. We're running ZCS Open Source 4.0.3 on CentOS 4.3. Anyone seen similar problems?

[phoenix]

You don't say what else is running on that server but could this be the problem? If that's not it see what's consuming your cpu cycles.

[stbain]

Other than Zimbra, I'm only running a local caching nameserver (solely for Zimbra's use) and an unused VMware server. According to `top`, the server's procs (dual P4) are approximately 50% idle with load averages around 5.

[stbain]

Just to make sure, I checked for any unicode related processes:

[root@hostname log]# ps ax | grep unicode
10309 pts/0    S+     0:00 grep unicode
[root@hostname log]#

also... here is my /etc/sysconfig/i18n file:

[root@hostname log]# cat /etc/sysconfig/i18n
LANG="en_US.UTF-8"
SUPPORTED="en_US.UTF-8:en_US:en"
SYSFONT="latarcyrheb-sun16"
[root@hostname log]#

Should I remove the UTF-8 and just stick with en_US? Reboot afterwards?

[stbain]

Followup:

I upgraded CentOS to latest and greatest including the newest kernel update. Deliveries were delayed by over 20 hours, but they're slowly dwindling down. Most of the 4000 remaining messages are in the active queue. At the current rate, it looks like queue will be back to normal sometime this afternoon.

Any advice on how to speed up delivery for messages waiting in the active queue? I tried flushing, but that doesn't seem to help at all.