Zimbra server crashed

[goetzi]

Hi,

I didn't find my error messages somewhere else in the forums, if there was a similar problem, I am sorry for this thread.

This morning I recognised that my Zimbra server wasn't running, so I tried a "zmcontrol status". It didn't work:

Cannot determine services - exiting

After a "zmcontrol start" the server was running.

I looked into zimbra.log and I am quite worried.

I attached a part of zimbra.log. How can I look into the zimbra.log of yesterday? The server has been crashed in the time after 8 PM, that was the time I received the last mail.

[marcmac]

Looks like ldap crashed.

The logs are rotated daily, so there should be a zimbra.log.1, zimbra.log.2, etc.

[goetzi]

Originally Posted by marcmac

Looks like ldap crashed.

The logs are rotated daily, so there should be a zimbra.log.1, zimbra.log.2, etc.

I have attached the part of the log file from where the server was crashing until ~4 hours after it.

Can you see why LDAP crashed?

There seems to be a problem with "4-mail.net[62.2.100.201]" (in the log at Feb 27 20:21). This is my backup-MX server (I don't know if this information is relevant, but maybe you can need it).

Thanks for your help,
goetzi

[marcmac]

Originally Posted by goetzi

I have attached the part of the log file from where the server was crashing until ~4 hours after it.

Can you see why LDAP crashed?

There seems to be a problem with "4-mail.net[62.2.100.201]" (in the log at Feb 27 20:21). This is my backup-MX server (I don't know if this information is relevant, but maybe you can need it).

Thanks for your help,
goetzi

I can't tell why it crashed from this log - apparently it died between
Feb 27 20:20:52 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: ldap: Running


and
Feb 27 20:31:00 zimbra postfix/smtpd[20569]: warning: dict_ldap_lookup: Search error -5: Timed out


It's logging disk space, which looks ok. What can you tell me about your hardware? (Virtual machine?)

Also - can you send me the contents of /var/log/messages between 20:20 and 20:35 on that date?

(It looks like you reduced the frequency of status checking in the crontab - is that correct?)

Also - did you have any DNS problems around this time?

This is merely statistical:

Feb 27 20:32:36 zimbra postfix/anvil[20769]: statistics: max connection rate 1/60s for (smtp:62.2.100.201) at Feb 27 20:22:05
Feb 27 20:33:36 zimbra postfix/anvil[20769]: statistics: max connection count 1 for (smtp:62.2.100.201) at Feb 27 20:22:05


Reported by the postfix anvil service (man 8 anvil on your zimbra box for more info).

[goetzi]

What can you tell me about your hardware? (Virtual machine?)

No, it's no virtual machine.
It's a PIII 600 Mhz with 386 MB RAM and 18 GB RAID 5 (3 Disks with 9 GB).

Also - can you send me the contents of /var/log/messages between 20:20 and 20:35 on that date?

Feb 27 20:20:08 zimbra crond(pam_unix)[20526]: session opened for user zimbra by (uid=0)
Feb 27 20:20:08 zimbra crond(pam_unix)[20529]: session opened for user zimbra by (uid=0)
Feb 27 20:20:10 zimbra crond(pam_unix)[20527]: session opened for user zimbra by (uid=0)
Feb 27 20:20:10 zimbra crond(pam_unix)[20528]: session opened for user zimbra by (uid=0)
Feb 27 20:20:29 zimbra zimbramon[20558]: 20558:info: 2006-02-27 20:20:26, DISK: : dev: /dev/ida/c0d0p3, mp: /, tot: 16139, ava
il: 5771
Feb 27 20:20:34 zimbra zimbramon[20558]: 20558:info: 2006-02-27 20:20:26, DISK: : dev: /dev/ida/c0d0p1, mp: /boot, tot: 97, av
ail: 81
Feb 27 20:20:37 zimbra crond(pam_unix)[20528]: session closed for user zimbra
Feb 27 20:20:45 zimbra zimbramon[20556]: 20556:info: 2006-02-27 20:20:26, QUEUE: 0 0
Feb 27 20:20:48 zimbra crond(pam_unix)[20526]: session closed for user zimbra
Feb 27 20:20:51 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: antispam: Running
Feb 27 20:20:52 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: antivirus: Running
Feb 27 20:20:52 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: ldap: Running
Feb 27 20:20:53 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: logger: Running
Feb 27 20:20:55 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: mailbox: Running
Feb 27 20:20:55 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: mta: Running
Feb 27 20:20:55 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: snmp: Running
Feb 27 20:20:56 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: spell: Running
Feb 27 20:20:54 zimbra crond(pam_unix)[15857]: session closed for user zimbra
Feb 27 20:21:15 zimbra crond(pam_unix)[20755]: session opened for user zimbra by (uid=0)
Feb 27 20:22:25 zimbra crond(pam_unix)[20794]: session opened for user zimbra by (uid=0)
Feb 27 20:23:53 zimbra crond(pam_unix)[20804]: session opened for user zimbra by (uid=0)
Feb 27 20:24:32 zimbra crond(pam_unix)[20871]: session opened for user zimbra by (uid=0)
Feb 27 20:26:19 zimbra crond(pam_unix)[20926]: session opened for user zimbra by (uid=0)
Feb 27 20:28:59 zimbra crond(pam_unix)[20940]: session opened for user zimbra by (uid=0)
Feb 27 20:29:34 zimbra crond(pam_unix)[20945]: session opened for user zimbra by (uid=0)
Feb 27 20:30:47 zimbra kernel: oom-killer: gfp_mask=0x1d2
Feb 27 20:30:51 zimbra kernel: Mem-info:
Feb 27 20:30:51 zimbra kernel: DMA per-cpu:
Feb 27 20:30:51 zimbra kernel: cpu 0 hot: low 2, high 6, batch 1
Feb 27 20:30:51 zimbra kernel: cpu 0 cold: low 0, high 2, batch 1
Feb 27 20:30:51 zimbra kernel: Normal per-cpu:
Feb 27 20:30:51 zimbra kernel: cpu 0 hot: low 32, high 96, batch 16
Feb 27 20:30:51 zimbra kernel: cpu 0 cold: low 0, high 32, batch 16
Feb 27 20:30:51 zimbra kernel: HighMem per-cpu: empty
Feb 27 20:30:51 zimbra kernel:
Feb 27 20:30:51 zimbra kernel: Free pages:         592kB (0kB HighMem)
Feb 27 20:30:51 zimbra kernel: Active:9098 inactive:37654 dirty:0 writeback:0 unstable:0 free:148 slab:22514 mapped:46423 page
tables:25807
Feb 27 20:30:51 zimbra kernel: DMA free:24kB min:24kB low:48kB high:72kB active:212kB inactive:280kB present:16384kB pages_sca
nned:2994 all_unreclaimable? yes
Feb 27 20:30:54 zimbra kernel: protections[]: 0 0 0
Feb 27 20:30:57 zimbra kernel: Normal free:568kB min:596kB low:1192kB high:1788kB active:36180kB inactive:150336kB present:376
816kB pages_scanned:328267 all_unreclaimable? yes
Feb 27 20:30:59 zimbra kernel: protections[]: 0 0 0
Feb 27 20:31:02 zimbra kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scann
ed:0 all_unreclaimable? no
Feb 27 20:31:06 zimbra kernel: protections[]: 0 0 0
Feb 27 20:31:08 zimbra kernel: DMA: 6*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 24kB
Feb 27 20:31:09 zimbra kernel: Normal: 0*4kB 1*8kB 3*16kB 0*32kB 0*64kB 4*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5
68kB
Feb 27 20:31:12 zimbra kernel: HighMem: empty
Feb 27 20:31:13 zimbra kernel: Swap cache: add 304430, delete 304430, find 16703/26938, race 0+4
Feb 27 20:31:13 zimbra kernel: 0 bounce buffer pages
Feb 27 20:31:15 zimbra kernel: Free swap:            0kB
Feb 27 20:31:17 zimbra kernel: 98300 pages of RAM
Feb 27 20:31:18 zimbra kernel: 0 pages of HIGHMEM
Feb 27 20:31:19 zimbra kernel: 1784 reserved pages
Feb 27 20:31:19 zimbra kernel: 11076 pages shared
Feb 27 20:31:20 zimbra kernel: 0 pages swap cached
Feb 27 20:31:22 zimbra kernel: Out of Memory: Killed process 20633 (java).
Feb 27 20:31:29 zimbra crond(pam_unix)[20957]: session opened for user zimbra by (uid=0)
Feb 27 20:31:29 zimbra crond(pam_unix)[20951]: session opened for user zimbra by (uid=0)
Feb 27 20:31:29 zimbra crond(pam_unix)[20948]: session opened for user zimbra by (uid=0)
Feb 27 20:33:32 zimbra crond(pam_unix)[20958]: session opened for user zimbra by (uid=0)
Feb 27 20:35:44 zimbra kernel: oom-killer: gfp_mask=0x1d2
Feb 27 20:35:44 zimbra kernel: Mem-info:
Feb 27 20:35:44 zimbra kernel: DMA per-cpu:
Feb 27 20:35:44 zimbra kernel: cpu 0 hot: low 2, high 6, batch 1
Feb 27 20:35:44 zimbra kernel: cpu 0 cold: low 0, high 2, batch 1
Feb 27 20:35:44 zimbra kernel: Normal per-cpu:
Feb 27 20:35:44 zimbra kernel: cpu 0 hot: low 32, high 96, batch 16
Feb 27 20:35:44 zimbra kernel: cpu 0 cold: low 0, high 32, batch 16
Feb 27 20:35:44 zimbra kernel: HighMem per-cpu: empty
Feb 27 20:35:44 zimbra kernel:
Feb 27 20:35:44 zimbra kernel: Free pages:         568kB (0kB HighMem)
Feb 27 20:35:44 zimbra kernel: Active:156 inactive:46434 dirty:0 writeback:0 unstable:0 free:142 slab:22517 mapped:46309 paget
ables:25835
Feb 27 20:35:44 zimbra kernel: DMA free:24kB min:24kB low:48kB high:72kB active:368kB inactive:124kB present:16384kB pages_sca
nned:1246 all_unreclaimable? yes
Feb 27 20:35:46 zimbra kernel: protections[]: 0 0 0
Feb 27 20:35:51 zimbra kernel: Normal free:544kB min:596kB low:1192kB high:1788kB active:256kB inactive:185612kB present:37681
6kB pages_scanned:859980 all_unreclaimable? yes
Feb 27 20:36:08 zimbra kernel: protections[]: 0 0 0
Feb 27 20:36:08 zimbra crond(pam_unix)[20964]: session opened for user zimbra by (uid=0)
Feb 27 20:36:08 zimbra crond(pam_unix)[20960]: session opened for user zimbra by (uid=0)
Feb 27 20:36:12 zimbra kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scann
ed:0 all_unreclaimable? no
Feb 27 20:36:12 zimbra crond(pam_unix)[20974]: session opened for user zimbra by (uid=0)
Feb 27 20:36:12 zimbra crond(pam_unix)[20979]: session opened for user zimbra by (uid=0)
Feb 27 20:36:12 zimbra crond(pam_unix)[20970]: session opened for user zimbra by (uid=0)
Feb 27 20:36:12 zimbra crond(pam_unix)[20982]: session opened for user zimbra by (uid=0)
Feb 27 20:36:12 zimbra crond(pam_unix)[20977]: session opened for user zimbra by (uid=0)
Feb 27 20:36:13 zimbra crond(pam_unix)[20956]: session opened for user zimbra by (uid=0)
Feb 27 20:36:14 zimbra kernel: protections[]: 0 0 0
Feb 27 20:36:15 zimbra kernel: DMA: 6*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 24kB
Feb 27 20:36:17 zimbra kernel: Normal: 0*4kB 0*8kB 2*16kB 0*32kB 0*64kB 4*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5
44kB
Feb 27 20:36:17 zimbra kernel: HighMem: empty
Feb 27 20:36:18 zimbra kernel: Swap cache: add 307028, delete 307028, find 17081/27658, race 0+19
Feb 27 20:36:24 zimbra kernel: 0 bounce buffer pages
Feb 27 20:36:28 zimbra kernel: Free swap:            0kB
Feb 27 20:36:43 zimbra zimbramon[20986]: 20986:info: 2006-02-27 20:36:19, DISK: : dev: /dev/ida/c0d0p3, mp: /, tot: 16139, ava
il: 5771
Feb 27 20:36:31 zimbra kernel: 98300 pages of RAM
Feb 27 20:36:33 zimbra kernel: 0 pages of HIGHMEM
Feb 27 20:36:34 zimbra kernel: 1784 reserved pages
Feb 27 20:36:34 zimbra kernel: 11098 pages shared
Feb 27 20:36:34 zimbra kernel: 0 pages swap cached

May it be a problem with my memory ("zimbra kernel: Out of Memory: Killed process 20633 (java).")?
I know that 386 MB isn't very much, but it always worked right (I only have 3 users on my system at the moment).

(It looks like you reduced the frequency of status checking in the crontab - is that correct?)

No, I didn't do anything there.

Also - did you have any DNS problems around this time?

Not that I know. All other things worked well at this time.

[marcmac]

Yeah, it's the memory killing you.

I'd suggest the following:

Disable the logger package:
zmprov ms SERVERNAME -zimbraServiceEnabled logger

- Edit the zimbra crontab, remove zmlogprocess and zmgengraphs

- Cut down the frequency of zmstatuslog and zmdisklog

Edit amavisd.conf.in and reduce max_servers

Reduce zimbraLmtpNumThreads (via zmprov) on the server

Disable the spell server, or reduce the number http processes (in conf/httpd.conf, spareservers, minspareservers, maxspareservers - reduce to 2 or 3)

[goetzi]

I did the following things:

Disable the logger package:
zmprov ms SERVERNAME -zimbraServiceEnabled logger

Edit amavisd.conf.in and reduce max_servers

Reduce zimbraLmtpNumThreads (via zmprov) on the server

It is running well now, no more crashes. I waited with this post so long because I wanted to watch the server some time before posting a wrong "success story".

You guys are just too cool. Instead of telling me to upgrade my server hardware you give me detailled and very good instructions how to cut down Zimbra so it fits to my hardware. I don't know many companies which such a good and competent support
(This had to be said)