Results 1 to 7 of 7

Thread: Zimbra server crashed

  1. #1
    goetzi is offline Project Contributor
    Join Date
    Nov 2005
    Location
    Austria
    Posts
    223
    Rep Power
    9

    Default Zimbra server crashed

    Hi,

    I didn't find my error messages somewhere else in the forums, if there was a similar problem, I am sorry for this thread.

    This morning I recognised that my Zimbra server wasn't running, so I tried a "zmcontrol status". It didn't work:
    Code:
    Cannot determine services - exiting
    After a "zmcontrol start" the server was running.

    I looked into zimbra.log and I am quite worried.

    I attached a part of zimbra.log. How can I look into the zimbra.log of yesterday? The server has been crashed in the time after 8 PM, that was the time I received the last mail.
    Attached Files Attached Files

  2. #2
    marcmac is offline Expert Member
    Join Date
    Sep 2005
    Posts
    2,103
    Rep Power
    13

    Default

    Looks like ldap crashed.

    The logs are rotated daily, so there should be a zimbra.log.1, zimbra.log.2, etc.

  3. #3
    goetzi is offline Project Contributor
    Join Date
    Nov 2005
    Location
    Austria
    Posts
    223
    Rep Power
    9

    Default

    Quote Originally Posted by marcmac
    Looks like ldap crashed.

    The logs are rotated daily, so there should be a zimbra.log.1, zimbra.log.2, etc.
    I have attached the part of the log file from where the server was crashing until ~4 hours after it.

    Can you see why LDAP crashed?

    There seems to be a problem with "4-mail.net[62.2.100.201]" (in the log at Feb 27 20:21). This is my backup-MX server (I don't know if this information is relevant, but maybe you can need it).

    Thanks for your help,
    goetzi
    Attached Files Attached Files

  4. #4
    marcmac is offline Expert Member
    Join Date
    Sep 2005
    Posts
    2,103
    Rep Power
    13

    Default

    Quote Originally Posted by goetzi
    I have attached the part of the log file from where the server was crashing until ~4 hours after it.

    Can you see why LDAP crashed?

    There seems to be a problem with "4-mail.net[62.2.100.201]" (in the log at Feb 27 20:21). This is my backup-MX server (I don't know if this information is relevant, but maybe you can need it).

    Thanks for your help,
    goetzi
    I can't tell why it crashed from this log - apparently it died between
    Feb 27 20:20:52 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: ldap: Running


    and
    Feb 27 20:31:00 zimbra postfix/smtpd[20569]: warning: dict_ldap_lookup: Search error -5: Timed out


    It's logging disk space, which looks ok. What can you tell me about your hardware? (Virtual machine?)

    Also - can you send me the contents of /var/log/messages between 20:20 and 20:35 on that date?

    (It looks like you reduced the frequency of status checking in the crontab - is that correct?)

    Also - did you have any DNS problems around this time?

    This is merely statistical:

    Feb 27 20:32:36 zimbra postfix/anvil[20769]: statistics: max connection rate 1/60s for (smtp:62.2.100.201) at Feb 27 20:22:05
    Feb 27 20:33:36 zimbra postfix/anvil[20769]: statistics: max connection count 1 for (smtp:62.2.100.201) at Feb 27 20:22:05


    Reported by the postfix anvil service (man 8 anvil on your zimbra box for more info).

  5. #5
    goetzi is offline Project Contributor
    Join Date
    Nov 2005
    Location
    Austria
    Posts
    223
    Rep Power
    9

    Default

    What can you tell me about your hardware? (Virtual machine?)
    No, it's no virtual machine.
    It's a PIII 600 Mhz with 386 MB RAM and 18 GB RAID 5 (3 Disks with 9 GB).


    Also - can you send me the contents of /var/log/messages between 20:20 and 20:35 on that date?
    Code:
    Feb 27 20:20:08 zimbra crond(pam_unix)[20526]: session opened for user zimbra by (uid=0)
    Feb 27 20:20:08 zimbra crond(pam_unix)[20529]: session opened for user zimbra by (uid=0)
    Feb 27 20:20:10 zimbra crond(pam_unix)[20527]: session opened for user zimbra by (uid=0)
    Feb 27 20:20:10 zimbra crond(pam_unix)[20528]: session opened for user zimbra by (uid=0)
    Feb 27 20:20:29 zimbra zimbramon[20558]: 20558:info: 2006-02-27 20:20:26, DISK: : dev: /dev/ida/c0d0p3, mp: /, tot: 16139, ava
    il: 5771
    Feb 27 20:20:34 zimbra zimbramon[20558]: 20558:info: 2006-02-27 20:20:26, DISK: : dev: /dev/ida/c0d0p1, mp: /boot, tot: 97, av
    ail: 81
    Feb 27 20:20:37 zimbra crond(pam_unix)[20528]: session closed for user zimbra
    Feb 27 20:20:45 zimbra zimbramon[20556]: 20556:info: 2006-02-27 20:20:26, QUEUE: 0 0
    Feb 27 20:20:48 zimbra crond(pam_unix)[20526]: session closed for user zimbra
    Feb 27 20:20:51 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: antispam: Running
    Feb 27 20:20:52 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: antivirus: Running
    Feb 27 20:20:52 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: ldap: Running
    Feb 27 20:20:53 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: logger: Running
    Feb 27 20:20:55 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: mailbox: Running
    Feb 27 20:20:55 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: mta: Running
    Feb 27 20:20:55 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: snmp: Running
    Feb 27 20:20:56 zimbra zimbramon[17100]: 17100:info: 2006-02-27 20:19:13, STATUS: zimbra.glatter-goetz.de: spell: Running
    Feb 27 20:20:54 zimbra crond(pam_unix)[15857]: session closed for user zimbra
    Feb 27 20:21:15 zimbra crond(pam_unix)[20755]: session opened for user zimbra by (uid=0)
    Feb 27 20:22:25 zimbra crond(pam_unix)[20794]: session opened for user zimbra by (uid=0)
    Feb 27 20:23:53 zimbra crond(pam_unix)[20804]: session opened for user zimbra by (uid=0)
    Feb 27 20:24:32 zimbra crond(pam_unix)[20871]: session opened for user zimbra by (uid=0)
    Feb 27 20:26:19 zimbra crond(pam_unix)[20926]: session opened for user zimbra by (uid=0)
    Feb 27 20:28:59 zimbra crond(pam_unix)[20940]: session opened for user zimbra by (uid=0)
    Feb 27 20:29:34 zimbra crond(pam_unix)[20945]: session opened for user zimbra by (uid=0)
    Feb 27 20:30:47 zimbra kernel: oom-killer: gfp_mask=0x1d2
    Feb 27 20:30:51 zimbra kernel: Mem-info:
    Feb 27 20:30:51 zimbra kernel: DMA per-cpu:
    Feb 27 20:30:51 zimbra kernel: cpu 0 hot: low 2, high 6, batch 1
    Feb 27 20:30:51 zimbra kernel: cpu 0 cold: low 0, high 2, batch 1
    Feb 27 20:30:51 zimbra kernel: Normal per-cpu:
    Feb 27 20:30:51 zimbra kernel: cpu 0 hot: low 32, high 96, batch 16
    Feb 27 20:30:51 zimbra kernel: cpu 0 cold: low 0, high 32, batch 16
    Feb 27 20:30:51 zimbra kernel: HighMem per-cpu: empty
    Feb 27 20:30:51 zimbra kernel:
    Feb 27 20:30:51 zimbra kernel: Free pages:         592kB (0kB HighMem)
    Feb 27 20:30:51 zimbra kernel: Active:9098 inactive:37654 dirty:0 writeback:0 unstable:0 free:148 slab:22514 mapped:46423 page
    tables:25807
    Feb 27 20:30:51 zimbra kernel: DMA free:24kB min:24kB low:48kB high:72kB active:212kB inactive:280kB present:16384kB pages_sca
    nned:2994 all_unreclaimable? yes
    Feb 27 20:30:54 zimbra kernel: protections[]: 0 0 0
    Feb 27 20:30:57 zimbra kernel: Normal free:568kB min:596kB low:1192kB high:1788kB active:36180kB inactive:150336kB present:376
    816kB pages_scanned:328267 all_unreclaimable? yes
    Feb 27 20:30:59 zimbra kernel: protections[]: 0 0 0
    Feb 27 20:31:02 zimbra kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scann
    ed:0 all_unreclaimable? no
    Feb 27 20:31:06 zimbra kernel: protections[]: 0 0 0
    Feb 27 20:31:08 zimbra kernel: DMA: 6*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 24kB
    Feb 27 20:31:09 zimbra kernel: Normal: 0*4kB 1*8kB 3*16kB 0*32kB 0*64kB 4*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5
    68kB
    Feb 27 20:31:12 zimbra kernel: HighMem: empty
    Feb 27 20:31:13 zimbra kernel: Swap cache: add 304430, delete 304430, find 16703/26938, race 0+4
    Feb 27 20:31:13 zimbra kernel: 0 bounce buffer pages
    Feb 27 20:31:15 zimbra kernel: Free swap:            0kB
    Feb 27 20:31:17 zimbra kernel: 98300 pages of RAM
    Feb 27 20:31:18 zimbra kernel: 0 pages of HIGHMEM
    Feb 27 20:31:19 zimbra kernel: 1784 reserved pages
    Feb 27 20:31:19 zimbra kernel: 11076 pages shared
    Feb 27 20:31:20 zimbra kernel: 0 pages swap cached
    Feb 27 20:31:22 zimbra kernel: Out of Memory: Killed process 20633 (java).
    Feb 27 20:31:29 zimbra crond(pam_unix)[20957]: session opened for user zimbra by (uid=0)
    Feb 27 20:31:29 zimbra crond(pam_unix)[20951]: session opened for user zimbra by (uid=0)
    Feb 27 20:31:29 zimbra crond(pam_unix)[20948]: session opened for user zimbra by (uid=0)
    Feb 27 20:33:32 zimbra crond(pam_unix)[20958]: session opened for user zimbra by (uid=0)
    Feb 27 20:35:44 zimbra kernel: oom-killer: gfp_mask=0x1d2
    Feb 27 20:35:44 zimbra kernel: Mem-info:
    Feb 27 20:35:44 zimbra kernel: DMA per-cpu:
    Feb 27 20:35:44 zimbra kernel: cpu 0 hot: low 2, high 6, batch 1
    Feb 27 20:35:44 zimbra kernel: cpu 0 cold: low 0, high 2, batch 1
    Feb 27 20:35:44 zimbra kernel: Normal per-cpu:
    Feb 27 20:35:44 zimbra kernel: cpu 0 hot: low 32, high 96, batch 16
    Feb 27 20:35:44 zimbra kernel: cpu 0 cold: low 0, high 32, batch 16
    Feb 27 20:35:44 zimbra kernel: HighMem per-cpu: empty
    Feb 27 20:35:44 zimbra kernel:
    Feb 27 20:35:44 zimbra kernel: Free pages:         568kB (0kB HighMem)
    Feb 27 20:35:44 zimbra kernel: Active:156 inactive:46434 dirty:0 writeback:0 unstable:0 free:142 slab:22517 mapped:46309 paget
    ables:25835
    Feb 27 20:35:44 zimbra kernel: DMA free:24kB min:24kB low:48kB high:72kB active:368kB inactive:124kB present:16384kB pages_sca
    nned:1246 all_unreclaimable? yes
    Feb 27 20:35:46 zimbra kernel: protections[]: 0 0 0
    Feb 27 20:35:51 zimbra kernel: Normal free:544kB min:596kB low:1192kB high:1788kB active:256kB inactive:185612kB present:37681
    6kB pages_scanned:859980 all_unreclaimable? yes
    Feb 27 20:36:08 zimbra kernel: protections[]: 0 0 0
    Feb 27 20:36:08 zimbra crond(pam_unix)[20964]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:08 zimbra crond(pam_unix)[20960]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:12 zimbra kernel: HighMem free:0kB min:128kB low:256kB high:384kB active:0kB inactive:0kB present:0kB pages_scann
    ed:0 all_unreclaimable? no
    Feb 27 20:36:12 zimbra crond(pam_unix)[20974]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:12 zimbra crond(pam_unix)[20979]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:12 zimbra crond(pam_unix)[20970]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:12 zimbra crond(pam_unix)[20982]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:12 zimbra crond(pam_unix)[20977]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:13 zimbra crond(pam_unix)[20956]: session opened for user zimbra by (uid=0)
    Feb 27 20:36:14 zimbra kernel: protections[]: 0 0 0
    Feb 27 20:36:15 zimbra kernel: DMA: 6*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 24kB
    Feb 27 20:36:17 zimbra kernel: Normal: 0*4kB 0*8kB 2*16kB 0*32kB 0*64kB 4*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5
    44kB
    Feb 27 20:36:17 zimbra kernel: HighMem: empty
    Feb 27 20:36:18 zimbra kernel: Swap cache: add 307028, delete 307028, find 17081/27658, race 0+19
    Feb 27 20:36:24 zimbra kernel: 0 bounce buffer pages
    Feb 27 20:36:28 zimbra kernel: Free swap:            0kB
    Feb 27 20:36:43 zimbra zimbramon[20986]: 20986:info: 2006-02-27 20:36:19, DISK: : dev: /dev/ida/c0d0p3, mp: /, tot: 16139, ava
    il: 5771
    Feb 27 20:36:31 zimbra kernel: 98300 pages of RAM
    Feb 27 20:36:33 zimbra kernel: 0 pages of HIGHMEM
    Feb 27 20:36:34 zimbra kernel: 1784 reserved pages
    Feb 27 20:36:34 zimbra kernel: 11098 pages shared
    Feb 27 20:36:34 zimbra kernel: 0 pages swap cached
    May it be a problem with my memory ("zimbra kernel: Out of Memory: Killed process 20633 (java).")?
    I know that 386 MB isn't very much, but it always worked right (I only have 3 users on my system at the moment).

    (It looks like you reduced the frequency of status checking in the crontab - is that correct?)
    No, I didn't do anything there.

    Also - did you have any DNS problems around this time?
    Not that I know. All other things worked well at this time.

  6. #6
    marcmac is offline Expert Member
    Join Date
    Sep 2005
    Posts
    2,103
    Rep Power
    13

    Default

    Yeah, it's the memory killing you.

    I'd suggest the following:

    Disable the logger package:
    zmprov ms SERVERNAME -zimbraServiceEnabled logger

    - Edit the zimbra crontab, remove zmlogprocess and zmgengraphs

    - Cut down the frequency of zmstatuslog and zmdisklog

    Edit amavisd.conf.in and reduce max_servers

    Reduce zimbraLmtpNumThreads (via zmprov) on the server

    Disable the spell server, or reduce the number http processes (in conf/httpd.conf, spareservers, minspareservers, maxspareservers - reduce to 2 or 3)

  7. #7
    goetzi is offline Project Contributor
    Join Date
    Nov 2005
    Location
    Austria
    Posts
    223
    Rep Power
    9

    Default

    I did the following things:

    Disable the logger package:
    zmprov ms SERVERNAME -zimbraServiceEnabled logger
    Edit amavisd.conf.in and reduce max_servers
    Reduce zimbraLmtpNumThreads (via zmprov) on the server
    It is running well now, no more crashes. I waited with this post so long because I wanted to watch the server some time before posting a wrong "success story".

    You guys are just too cool. Instead of telling me to upgrade my server hardware you give me detailled and very good instructions how to cut down Zimbra so it fits to my hardware. I don't know many companies which such a good and competent support
    (This had to be said)

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. QUE Failure
    By tbullock in forum Administrators
    Replies: 31
    Last Post: 07-30-2008, 12:17 PM
  2. Replies: 9
    Last Post: 03-01-2008, 08:21 PM
  3. svn version still won't start
    By kinaole in forum Developers
    Replies: 0
    Last Post: 10-04-2006, 06:47 AM
  4. Logger
    By jholder in forum Installation
    Replies: 24
    Last Post: 03-31-2006, 11:50 AM
  5. Monitoring : Data not yet avalaible
    By s3nz3x in forum Installation
    Replies: 7
    Last Post: 11-30-2005, 07:18 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •