Mailbox and logger not starting - SLES9 M3 Beta436

[DanielP]

First of all, thanks for a great product!

After fresh (smooth btw) install of M3 Beta 436 on SLES9 SP3, the mailbox and logger does not start(even tough the /etc/init.d/zimbra start says it does).

Starting ldap...Done.
Starting antispam...Done.
Starting antivirus...Done.
Starting logger...Done.
Starting mailbox...Done.
Starting mta...Done.
Starting spell...Done.

However running:
zmcontrol status

gives

antispam Running
antivirus Running
ldap Running
logger Stopped
mailbox Stopped
mta Running
spell Running

and
zmcontrol stop
Stopping antispam...Done
Stopping antivirus...Done
Stopping ldap...Done
Stopping logger...FAILED
Error: no pid in '/opt/zimbra/logger/db/mysql.pid'
/opt/zimbra/bin/zmlogswatchctl: line 59: kill: (22748) - No such process
Stopping mailbox...Done
Stopping mta...FAILED
/opt/zimbra/cyrus-sasl-2.1.21.ZIMBRA/sbin/saslauthd: no process killed
Stopping snmp...Done
Stopping spell...Done

So, the mysqld server is not starting at all. Also, the webserver does not listen to anything but 7780.

What could be the problem here? The machine ran a previous (M3) beta just fine.

/Daniel

[DanielP]

Hi again,

Sorry for replying on my own post. I have also tried this with daily build 381. Same problem. Could it be some external problem? I did change ipadress on the machine.... however, DNS are updated, and everything gets resolved correctly.(using short name, FQDN and by ping, host and dig a hostname).
SLES own Postfix is stopped. Machine is rebooted and problem persists. Could it be any lock files preventing mysql to start?

The aspell works btw (port 7780) but apache does not server on any other ports.

zmcontrol stop says:
Stopping antispam...Done
Stopping antivirus...Done
Stopping ldap...Done
Stopping logger...FAILED
Error: no pid in '/opt/zimbra/logger/db/mysql.pid'
/opt/zimbra/bin/zmlogswatchctl: line 59: kill: (339) - No such process
Stopping mailbox...Done
Stopping mta...FAILED
/opt/zimbra/cyrus-sasl-2.1.21.ZIMBRA/sbin/saslauthd: no process killed
Stopping snmp...Done
Stopping spell...Done

Altough ps -aux shows that saslauthd should be started:

zimbra 653 0.0 0.1 4336 1168 ? Ss 14:54 0:00 /opt/zimbra/cyrus-sasl-2.1.21.ZIMBRA/sbin/saslauthd -r -a zimbra
zimbra 654 0.0 0.1 4336 1168 ? S 14:54 0:00 /opt/zimbra/cyrus-sasl-2.1.21.ZIMBRA/sbin/saslauthd -r -a zimbra
zimbra 655 0.0 0.1 4336 1168 ? S 14:54 0:00 /opt/zimbra/cyrus-sasl-2.1.21.ZIMBRA/sbin/saslauthd -r -a zimbra
zimbra 656 0.0 0.1 4336 1168 ? S 14:54 0:00 /opt/zimbra/cyrus-sasl-2.1.21.ZIMBRA/sbin/saslauthd -r -a zimbra
zimbra 657 0.0 0.1 4336 1168 ? S 14:54 0:00 /opt/zimbra/cyrus-sasl-2.1.21.ZIMBRA/sbin/saslauthd -r -a zimbra

Any pointers on where start investigating?

/D

[marcmac]

Since the mailbox and logger services refuse to start, I suspect mysql problems. (Mailbox = mysql+tomcat, logger=mysql+swatch).

Run zmcontrol stop, and ps auxww | grep sql - you should see nothing.

run zmcontrol start, and again ps auxww | grep sql - you should (if it's failing) see nothing.

If sql is running, it's a status check failure.

If it's not - as zimbra -
sh -x bin/mysql.server start

Any errors output? Perhaps missing library?

[DanielP]

As far as i can see, sh -x does not show any errors. However, this is the last lines:

++ export zimbra_uid=1001
++ zimbra_uid=1001
++ export zimbra_user=zimbra
++ zimbra_user=zimbra
++ export zimlet_directory=/opt/zimbra/tomcat/webapps/service/zimlet
++ zimlet_directory=/opt/zimbra/tomcat/webapps/service/zimlet
+ export zmsetvars=true
+ zmsetvars=true
+ zmassert -d /opt/zimbra/mysql
+ '[' -d /opt/zimbra/mysql ']'
+ return
+ zmassert -x /opt/zimbra/mysql/bin/mysqld_safe
+ '[' -x /opt/zimbra/mysql/bin/mysqld_safe ']'
+ return
+ zmassert -r /opt/zimbra/conf/my.cnf
+ '[' -r /opt/zimbra/conf/my.cnf ']'
+ return
++ dirname /opt/zimbra/db/mysql.pid
+ zmassert -d /opt/zimbra/db
+ '[' -d /opt/zimbra/db ']'
+ return
+ cd /opt/zimbra/mysql
zimbra@mail:~> + /opt/zimbra/mysql/bin/mysqld_safe --defaults-file=/opt/zimbra/conf/my.cnf --ledir=/opt/zimbra/mysql/bin
Starting mysqld daemon with databases from /opt/zimbra/db/data
STOPPING server from pid file /opt/zimbra/db/mysql.pid
060112 15:37:01 mysqld ended

So, it seems like it is trying to start, and then stops it immediatly.

Also:
ls db/data/
ib_logfile0 ib_logfile1 ibdata1 mail-bin.000001 mail-bin.000002 mail-bin.index mysql test

All libraries specified in the quickstart should be present.

Update:

mysqld.log shows:
060112 15:40:17 mysqld started
060112 15:40:17 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
060112 15:40:17 InnoDB: Starting log scan based on checkpoint at
InnoDB: log sequence number 0 36808.
InnoDB: Doing recovery: scanned up to log sequence number 0 43676
060112 15:40:17 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Last MySQL binlog file position 0 79, file name ./mail-bin.000001
060112 15:40:17 InnoDB: Flushing modified pages from the buffer pool...
060112 15:40:17 InnoDB: Started; log sequence number 0 43676
060112 15:40:17 [ERROR] Fatal error: Can't open privilege tables: Table 'mysql.host' doesn't exist
060112 15:40:17 mysqld ended

Maybe the install did not work as expected?

/D

[marcmac]

your installation failed - mysql init didn't work.


Do you have another mysql instance on this server?

What's in /tmp/install.log. from the installation?

[DanielP]

Mysql is not installed on this machine. Postfix is off.
I removed everything i could find of the zimbra installation(after running install.sh -u) i.e grep -r zimbra /etc/* and rm -rf /opt/zimbra and /var/log/zimbra. Restart of crond and syslogd

I then did a fresh install, without SNMP.
Problem persists.

The install log is as followed.

cat install.log.15284

Preparing packages for installation...
zimbra-core-3.0.0_M3_436.SuSEES9-20060106155022
ERROR: service.FAILURE (system failure: getDirectContext) (cause: javax.naming.CommunicationException localhost:389)
zimbra 0:off 1:off 2:off 3:on 4:off 5:on 6:off
Loading classes to share ... done.
Rewriting and unlinking classes ... done.
Calculating hash values for String objects .. done.
Calculating fingerprints ... done.
Removing unshareable information ... done.
Moving most read-only objects to shared space at 0x6d8f0000 ... done.
Moving common symbols to shared space at 0x6dc6a520 ... done.
Moving remaining symbols to shared space at 0x6dd95640 ... done.
Moving string char arrays to shared space at 0x6dd960d0 ... done.
Moving additional symbols to shared space at 0x6de17728 ... done.
Read-only space ends at 0x6de6b5e8, 5748200 bytes.
Moving read-write objects to shared space at 0x6e0f0000 ... done.
Moving String objects to shared space at 0x6e669ec0 ... done.
Read-write space ends at 0x6e6a9dc0, 6004160 bytes.
Updating references to shared objects ... done.

Preparing packages for installation...
zimbra-ldap-3.0.0_M3_436.SuSEES9-20060106155022
Preparing packages for installation...
zimbra-logger-3.0.0_M3_436.SuSEES9-20060106155022
Preparing packages for installation...
zimbra-mta-3.0.0_M3_436.SuSEES9-20060106155022
umount2: No such file or directory
umount: /opt/zimbra/amavisd/tmp: not mounted
mv: cannot stat `/opt/zimbra/amavisd': No such file or directory
chown: cannot access `/opt/zimbra/perdition-1.17': No such file or directory
Preparing packages for installation...
zimbra-store-3.0.0_M3_436.SuSEES9-20060106155022
Preparing packages for installation...
zimbra-apache-3.0.0_M3_436.SuSEES9-20060106155022
Preparing packages for installation...
zimbra-spell-3.0.0_M3_436.SuSEES9-20060106155022

At the error above, it tries to bind ldap to localhost...could it be a resolve issue after all?

/D

[marcmac]

what does
host localhost
return?

With the name you're configuring the box as in zimbra
host

[DanielP]

zimbra@mail:~> host localhost
localhost has address 127.0.0.1
zimbra@mail:~> host mail.inside.domain.se
mail.inside.domain.se has address 192.168.1.1
zimbra@mail:~> host inside.domain.se
zimbra@mail:~>

Could the problem be that inside.domain.se dont have A record?

inside.domain.se is the domain, and all inside hosts are in that domain. The zimbra install is configured to setup inside.domain.se as the default domain.(altough, i tried with mail.inside.domain.se aswell without any luck)

/D

[marcmac]

You running a firewall on this box?

Try initializing mysql by hand:
as zimbra, /opt/zimbra/libexec/zmmyinit

Should re-init the sql db and start sql.

[DanielP]

No, firewall is turned off(until successful installation)

mail:/tmp # iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
mail:/tmp # su - zimbra
zimbra@mail:~> /opt/zimbra/libexec/zmmyinit
* Creating required directories
* Generating mysql config /opt/zimbra/conf/my.cnf
* Creating database in /opt/zimbra/db/data
* Starting mysql server
* Loading schema /opt/zimbra/db/db.sql
ERROR 2013 (HY000): Lost connection to MySQL server during query
* Loading version from /opt/zimbra/db/versions-init.sql
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/opt/zimbra/db/mysql.sock' (2)
* Setting random passwd for mysql root user in zimbra localconfig
* Setting random passwd for mysql zimbra user in zimbra localconfig
* Changing mysql root user password
/opt/zimbra/mysql/bin/mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/opt/zimbra/db/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/opt/zimbra/db/mysql.sock' exists!
* Changing mysql zimbra user password
ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/opt/zimbra/db/mysql.sock' (2)
**** PASSWORD CHANGE FAILED

It still cant connect. Other stuff as telnet localhost 25 works as expected.

/Daniel