[SOLVED] total zimbra cs startup time 1.5 hours

[morozovvs]

After few month of using Zimbra Collaboration Suite i've found that startup time became near 90 minutes (93 last time). After installation and test usage this time was less than 2 minutes.
"zmcontrol status" command takes 3 minutes.
During startup time processor usage by zimbra components is near 0 (by "top" utility), disk activity low, 20-100 block read/write operations in 10 seconds (by "iostat")
I don't know, what was the reason of this. Every night cron launches backup script based on "Open Source Edition Backup Procedure", "Simple Shell Script Method" - shutdown zimbra, rsync files to another storage, startup server. Because night is more than 90 minutes, nobody have seen this problem.
Server work 24/7 without restart.
Total mail size on server near 16 GB, 20 GB with database and other software. 19 account, 11 with mailbox more than 100Mb. Statistic is not accumulated because it didn't work after installation, it had big size, it wasn't needed.

CPU is Xeon E5420, RAM 2 GB + swap 2GB, Mail filesystem 60GB, 66% free.

Startup part from /var/log/zimbra.log is in attachment - server started from init after reboot.

[dalmate]

Follow your log, I don't see any errors.
Can you try to disable AV/AS service to check if because of them your server starts too slowly?

[tonster]

Originally Posted by morozovvs

After few month of using Zimbra Collaboration Suite i've found that startup time became near 90 minutes (93 last time). After installation and test usage this time was less than 2 minutes.
"zmcontrol status" command takes 3 minutes.
During startup time processor usage by zimbra components is near 0 (by "top" utility), disk activity low, 20-100 block read/write operations in 10 seconds (by "iostat")
I don't know, what was the reason of this. Every night cron launches backup script based on "Open Source Edition Backup Procedure", "Simple Shell Script Method" - shutdown zimbra, rsync files to another storage, startup server. Because night is more than 90 minutes, nobody have seen this problem.
Server work 24/7 without restart.
Total mail size on server near 16 GB, 20 GB with database and other software. 19 account, 11 with mailbox more than 100Mb. Statistic is not accumulated because it didn't work after installation, it had big size, it wasn't needed.

CPU is Xeon E5420, RAM 2 GB + swap 2GB, Mail filesystem 60GB, 66% free.

Startup part from /var/log/zimbra.log is in attachment - server started from init after reboot.

The better file to look at is likely the mailbox.log. You're also running the bare minimum amount of RAM in the system, and if the system is swapping alot then you're likely found your problem. What does the free output look like?

[morozovvs]

There are no errors. The problem is that startup takes a long time. I'll try to disable Debian's clamav. There are no other working AV daemons.

Output of memory usage tool:

# free
             total       used       free     shared    buffers     cached
Mem:       2061752    1894308     167444          0      12988     274656
-/+ buffers/cache:    1606664     455088
Swap:      1951856     439048    1512808

I'll try reduce memory usage by shutdown of some non-zimbra daemons and removing them from startup. After shutdown of them i have following:

# free
             total       used       free     shared    buffers     cached
Mem:       2061752    1709736     352016          0      22788     295396
-/+ buffers/cache:    1391552     670200
Swap:      1951856     401484    1550372

About mailbox.log compared to zimbra.log:
I've stared Zimbra at 13:37, but first record in mailbox.log appeared at 14:32

2010-04-20 12:32:51,638 INFO  [Shutdown] [] log - Shutdown hook complete
2010-04-20 14:32:20,788 INFO  [main] [] soap - Servlet SoapServlet starting up
2010-04-20 14:32:20,835 INFO  [main] [] soap - Adding service AccountService to SoapServlet
2010-04-20 14:32:21,051 INFO  [main] [] soap - Adding service MailService to SoapServlet
2010-04-20 14:32:21,173 INFO  [main] [] soap - Adding service IMService to SoapServlet
...

I don't know if there some ldap log.
stacktrace.*.* contains some exceptions (most of them are TIMED_WAITING), but without timestamps.

I'll look tomorrow at logs, if the problem is in memory.

[tonster]

I wouldn't say the problem is necessarily in memory so much as it is in lack of it. As you can see in the free output, you're always swapping. Committing stuff to disk and having to access it from disk is going to be a lot slower than doing everything from memory. You'd do well to increase your RAM when you can and it would likely improve things quite a bit.

[morozovvs]

I've disabled swap. So now there are only RAM is used. Also i've decreased quantity of amavis processes from 10 to 5, disabled one AV scaner.
It didn't change anything. Startup time is still almost same.
Night backup finish main work at 00:45 and launches zimbra. The first postfix response is near 02:18.
I'ts also notable that 'zmcontrol status' command takes 2-3 minutes.
All services, including postfix, amavis, mailbox work fast.

# free
             total       used       free     shared    buffers     cached
Mem:       2061752    1717704     344048          0      32484     358628
-/+ buffers/cache:    1326592     735160
Swap:            0          0          0

[tonster]

Disabling swap is a pretty bad idea. That doesn't change the fact that the system wants to use more memory than it has, it just guarantees that at some point you're going to run out of memory and instead of swapping to disk, it's going to just start killing processes to get more memory.

[morozovvs]

Possibly i've found the reason. It's not a memory problem. I think it was Linux ip6tables firewall (it prevented almost all incoming connections by ipv6 protocol). As i saw before, if application require ip connection, it tries IPv6 first, and if it fails, it tries IPv4.
I've disabled IPv6 firewall (and also disabled IPv6 Internet connection). Now 'zmcontrol status' takes few seconds. Later i'll check Zimbra startup time.

[morozovvs]

Yes, the reason was in IPv6 firewall. After disabling of IPv6 firewall (after resetting rules to accept everything) zimbra starts in less than 2 minutes.

[emman]

i know this is an old thread but i found it to be of help in identifying our problem. zmprov, as well as other related zm* commands, takes a very long time to run. and i found out that our IPv6 firewall is the culprit through this thread.

with the advent of the World IPv6 Day, we enabled IPv6 on our Zimbra installation, following several sources online (see below if interested). we successfully enabled it (dual-stacked with v4), and we modified our firewall definitions for v6. and then this problem came out. since our v4 firewall is working w/o any issues before, i assumed that the same definitions would fit v6. question would be, are there ports other than the ones indicated here that should be enabled for IPv6? thanks in advance. will investigate this further tomorrow.


resources for enabling IPv6 on Zimbra:

• Zimbra Box using IPv6
• SMTP and IPv6: How to with Zimbra (and postfix)
• Virbl (by BIT)