ZCS 7.0 GA build 3077 random crashes and reboots

[SiliconJunkie]

Running ZCS 7.0 GA build 3077 on Ubuntu 10.04 LTS 64-bit, with Xeon 2.8 GHz CPU, 4GB RAM, 1TB disk with 600+GB free space.

Experiencing random crashes and system reboots for no apparent reason. Information in zimbra.log is of no use. Any idea where else I can look for some relevant data as to why this might be happening?

[John Siu]

Check syslog?

[SiliconJunkie]

Syslog only shows that the reboot happened, so not much help there.


Here's an excerpt from syslog:

Mar 18 10:25:01 mail CRON[18837]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Mar 18 10:25:01 mail CRON[18838]: (zimbra) CMD (/opt/zimbra/libexec/zmcheckduplicatemysqld -e > /dev/null 2>&1)
Mar 18 10:26:01 mail CRON[18876]: (zimbra) CMD (/opt/zimbra/libexec/zmstatuslog)
Mar 18 10:28:01 mail CRON[19350]: (zimbra) CMD (/opt/zimbra/libexec/zmstatuslog)
Mar 18 10:58:58 mail kernel: imklog 4.2.0, log source = /proc/kmsg started.
Mar 18 10:58:58 mail rsyslogd: [origin software="rsyslogd" swVersion="4.2.0" x-pid="954" x-info="http://www.rsyslog.com"] (re)start
Mar 18 10:58:58 mail rsyslogd: rsyslogd's groupid changed to 103
Mar 18 10:58:58 mail rsyslogd: rsyslogd's userid changed to 101
Mar 18 10:58:58 mail rsyslogd-2039: Could no open output file '/dev/xconsole' [try http://www.rsyslog.com/e/2039 ]
Mar 18 10:58:58 mail kernel: [    0.000000] Initializing cgroup subsys cpuset
Mar 18 10:58:58 mail kernel: [    0.000000] Initializing cgroup subsys cpu
Mar 18 10:58:58 mail kernel: [    0.000000] Linux version 2.6.32-28-server (buildd@allspice) (gcc version 4.4.3 (Ubuntu 4.4.3-4ubuntu5) ) #55-Ubuntu SMP Mon Jan 10 23:57:16 UTC 2011 (Ubuntu 2.6.32-28.55-server 2.6.32.27+drm33.12)
Mar 18 10:58:58 mail kernel: [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-2.6.32-28-server root=UUID=a08d71f3-97ca-4ec5-aa99-090d3f1dc68b ro quiet
Mar 18 10:58:58 mail kernel: [    0.000000] KERNEL supported cpus:
Mar 18 10:58:58 mail kernel: [    0.000000]   Intel GenuineIntel
Mar 18 10:58:58 mail kernel: [    0.000000]   AMD AuthenticAMD
Mar 18 10:58:58 mail kernel: [    0.000000]   Centaur CentaurHauls
Mar 18 10:58:58 mail kernel: [    0.000000] BIOS-provided physical RAM map:
Mar 18 10:58:58 mail kernel: [    0.000000]  BIOS-e820: 0000000000000000 - 000000000009a400 (usable)
Mar 18 10:58:58 mail kernel: [    0.000000]  BIOS-e820: 000000000009a400 - 00000000000a0000 (reserved)
Mar 18 10:58:58 mail kernel: [    0.000000]  BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Mar 18 10:58:58 mail kernel: [    0.000000]  BIOS-e820: 0000000000100000 - 000000009da60000 (usable)
Mar 18 10:58:58 mail kernel: [    0.000000]  BIOS-e820: 000000009da60000 - 000000009db34000 (ACPI NVS)
Mar 18 10:58:58 mail kernel: [    0.000000]  BIOS-e820: 000000009db34000 - 000000009f6a4000 (ACPI data)
Mar 18 10:58:58 mail kernel: [    0.000000]  BIOS-e820: 000000009f6a4000 - 000000009f6df000 (reserved)
Mar 18 10:58:58 mail kernel: [    0.000000]  BIOS-e820: 000000009f6df000 - 000000009f78a000 (ACPI data)
Mar 18 10:58:58 mail kernel: [    0.000000]  BIOS-e820: 000000009f78a000 - 000000009f7df000 (ACPI NVS)
Mar 18 10:58:58 mail kernel: [    0.000000]  BIOS-e820: 000000009f7df000 - 000000009f800000 (ACPI data)
Mar 18 10:58:58 mail kernel: [    0.000000]  BIOS-e820: 000000009f800000 - 00000000b0000000 (reserved)
Mar 18 10:58:58 mail kernel: [    0.000000]  BIOS-e820: 00000000fed1c000 - 00000000fed20000 (reserved)
Mar 18 10:58:58 mail kernel: [    0.000000]  BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved)
Mar 18 10:58:58 mail kernel: [    0.000000]  BIOS-e820: 0000000100000000 - 0000000260000000 (usable)
Mar 18 10:58:58 mail kernel: [    0.000000] DMI 2.5 present.

...and so continues boot-time information until it's finished logging that data...

So, you can see that at 10:28:01, the machine was processing zmstatuslog..then all of a sudden quit, until 10:58 when I had to power-cycle the box to get mail back up.

What is the database integrity check routine for ZCS 7.0?

I would paste in a dmesg, but it is too large to fit.

[SiliconJunkie]

...never mind, I found it.

...as zimbra

./libexec/zmdbintegrityreport

[SiliconJunkie]

OK, what does this mean?

Database errors found.
/opt/zimbra/mysql/bin/mysqlcheck --defaults-file=/opt/zimbra/conf/my.cnf -S /opt/zimbra/db/mysql.sock -A -C -s -u root --password=OBFUSCATED
mysql.general_log
 Error    : You can't use locks with log tables.
 mysql.slow_log
 Error    : You can't use locks with log tables.

This e-mail was in the "administrator" mailbox on Feb 28th, which is the date this server was placed into production. There have been no such reports since then.

[phoenix]

OK, what does this mean?

site:zimbra.com +"You can't use locks with log tables" - Yahoo! Search Results

[SiliconJunkie]

Thanks for the link Phoenix - glad to learn that it's a harmless message.

Now, if I can just figure out why the server reboots itself about every 36 hours, I'll be in great shape.

[SiliconJunkie]

OK, what does this mean?

Database errors found.
/opt/zimbra/mysql/bin/mysqlcheck --defaults-file=/opt/zimbra/conf/my.cnf -S /opt/zimbra/db/mysql.sock -A -C -s -u root --password=OBFUSCATED
mysql.general_log
 Error    : You can't use locks with log tables.
 mysql.slow_log
 Error    : You can't use locks with log tables.

This e-mail was in the "administrator" mailbox on Feb 28th, which is the date this server was placed into production. There have been no such reports since then.

Oops... actually saw two more of these - looks like they are running weekly, as they should be, via crontab.

[phoenix]

Thanks for the link Phoenix - glad to learn that it's a harmless message.

Now, if I can just figure out why the server reboots itself about every 36 hours, I'll be in great shape.

Have you considered it might be a hardware (specifically RAM) problem? What's running on the server at the time it reboots? Do you have any monitoring in place that might give you some clue as to the services running, ram usage etc.?

[SiliconJunkie]

Hi, Phoenix.

Yeah, I've considered that it might be a RAM problem.

I've increased the amount of RAM on the machine to 8GB. I just checked, and 3.1 GB is used, the rest is free.

The box is only running ZCS, and DNS for name resolution for ZCS.

I'm beginning to think there might be a bad module in the mix. I'm going to change out the RAM later this week, and see what happens.