ZCS: Duplicate mysqld processes detected!

[henrik_b]

Hi,

after fixing the last problem. Now Zimbra sends an email to admin@ every five minutes with the following content:

More then 1 mysqld processes are running
Parent processes include: 15410 15542 15530

This should be investigated immediately as it may lead
to database corruption.

Using ps I can see various mysqld-processes, but all are child-processes of an initial call of one mysqld_safe process. (there is a second set of processes for logger, that looks just identical)

What may cause that? Looking through the logs I couldn't find anything special.

Henrik

[bobby]

do the processes 15410 15542 15530 still exist?

[henrik_b]

do the processes 15410 15542 15530 still exist?

yes.

15410 would be /opt/zimbra/mysql/bin/mysqld_safe
15530 and 15542 are /opt/zimbra/mysql/libexec/mysqld processes and children of 15410.

Henrik

[mmorse]

Version string? (zmcontrol -v)
Paste the output of ps -ef
Run a db integrity report on it (/opt/zimbra/libexec/zmdbintegrityreport)

[henrik_b]

Version string? (zmcontrol -v)

zmcontrol -v
Release 4.5.10_GA_1575.DEBIAN3.1 DEBIAN3.1

Paste the output of ps -ef

Doesn't look pretty here, paste it somewhere without linebreaks. Obviously I've got other process numbers now. Zimbra now complains about 7169 7158 7089.

ps ef
PID TTY STAT TIME COMMAND
5302 pts/1 S 0:00 su - zimbra
5303 pts/1 S 0:00 \_ -su TERM=xterm PATH=/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games LANG=de_DE.UTF-8 MAIL=/var/mail/zimbra HOME=/opt/zimbra SHELL=/bin/bash USER=zimbra LOGNAME=zimbra
8141 pts/1 R+ 0:00 \_ ps ef JAVA_JVM_VERSION=1.5 SHELL=/bin/bash TERM=xterm PERL5LIB=/opt/zimbra/zimbramon/lib/x86_64-linux-thread-multi:/opt/zimbra/zimbramon/lib:/opt/zimbra/zimbramon/lib/i386-linux-thread-multi:/opt/zimbra/zimbramon/lib
7802 pts/1 S 0:00 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-mtaqueue ldap_connect_timeout=30000 JAVA_JVM_VERSION=1.5 debug_mailboxindex_use_new_locking=true zimbra_user=zimbra ldap_is_master=true nio_write_buffer_compaction_percent=50 postf
7800 pts/1 S 0:00 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-mysql ldap_connect_timeout=30000 JAVA_JVM_VERSION=1.5 debug_mailboxindex_use_new_locking=true zimbra_user=zimbra ldap_is_master=true nio_write_buffer_compaction_percent=50 postfix_
7794 pts/1 S 0:00 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-vm ldap_connect_timeout=30000 JAVA_JVM_VERSION=1.5 debug_mailboxindex_use_new_locking=true zimbra_user=zimbra ldap_is_master=true nio_write_buffer_compaction_percent=50 postfix_hea
7940 pts/1 S 0:00 \_ /usr/bin/vmstat -n -S K 30 ldap_connect_timeout=30000 JAVA_JVM_VERSION=1.5 debug_mailboxindex_use_new_locking=true zimbra_user=zimbra ldap_is_master=true nio_write_buffer_compaction_percent=50 postfix_header_checks=pcre:
7792 pts/1 S 0:00 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-cpu ldap_connect_timeout=30000 JAVA_JVM_VERSION=1.5 debug_mailboxindex_use_new_locking=true zimbra_user=zimbra ldap_is_master=true nio_write_buffer_compaction_percent=50 postfix_he
7790 pts/1 S 0:00 /usr/bin/perl -w /opt/zimbra/libexec/zmstat-proc ldap_connect_timeout=30000 JAVA_JVM_VERSION=1.5 debug_mailboxindex_use_new_locking=true zimbra_user=zimbra ldap_is_master=true nio_write_buffer_compaction_percent=50 postfix_h
7626 pts/1 S 0:00 /usr/bin/perl /opt/zimbra/libexec/swatch --config-file=/opt/zimbra/conf/swatchrc --use-cpan-file-tail --script-dir=/tmp -t /var/log/zimbra.log ldap_connect_timeout=30000 JAVA_JVM_VERSION=1.5 debug_mailboxindex_use_new_lockin
7638 pts/1 S 0:00 \_ /usr/bin/perl /tmp/.swatch_script.7626 ldap_connect_timeout=30000 JAVA_JVM_VERSION=1.5 debug_mailboxindex_use_new_locking=true zimbra_user=zimbra ldap_is_master=true nio_write_buffer_compaction_percent=50 postfix_header_
7089 pts/1 S 0:00 /bin/sh /opt/zimbra/mysql/bin/mysqld_safe --defaults-file=/opt/zimbra/conf/my.cnf --external-locking --ledir=/opt/zimbra/mysql/libexec ldap_connect_timeout=30000 zimbra_user=zimbra debug_mailboxindex_use_new_locking=true JAV
7158 pts/1 S 0:00 \_ /opt/zimbra/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --basedir=/opt/zimbra/mysql --datadir=/opt/zimbra/db/data --pid-file=/opt/zimbra/db/mysql.pid --skip-external-locking --port=7306 --socket=/opt/zim
7169 pts/1 S 0:00 \_ /opt/zimbra/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --basedir=/opt/zimbra/mysql --datadir=/opt/zimbra/db/data --pid-file=/opt/zimbra/db/mysql.pid --skip-external-locking --port=7306 --socket=/opt
7171 pts/1 S 0:00 \_ /opt/zimbra/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --basedir=/opt/zimbra/mysql --datadir=/opt/zimbra/db/data --pid-file=/opt/zimbra/db/mysql.pid --skip-external-locking --port=7306 --socket=
7173 pts/1 S 0:00 \_ /opt/zimbra/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --basedir=/opt/zimbra/mysql --datadir=/opt/zimbra/db/data --pid-file=/opt/zimbra/db/mysql.pid --skip-external-locking --port=7306 --socket=
7176 pts/1 S 0:00 \_ /opt/zimbra/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --basedir=/opt/zimbra/mysql --datadir=/opt/zimbra/db/data --pid-file=/opt/zimbra/db/mysql.pid --skip-external-locking --port=7306 --socket=
7180 pts/1 S 0:00 \_ /opt/zimbra/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --basedir=/opt/zimbra/mysql --datadir=/opt/zimbra/db/data --pid-file=/opt/zimbra/db/mysql.pid --skip-external-locking --port=7306 --socket=
7227 pts/1 S 0:00 \_ /opt/zimbra/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --basedir=/opt/zimbra/mysql --datadir=/opt/zimbra/db/data --pid-file=/opt/zimbra/db/mysql.pid --skip-external-locking --port=7306 --socket=
7228 pts/1 S 0:00 \_ /opt/zimbra/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --basedir=/opt/zimbra/mysql --datadir=/opt/zimbra/db/data --pid-file=/opt/zimbra/db/mysql.pid --skip-external-locking --port=7306 --socket=
7230 pts/1 S 0:00 \_ /opt/zimbra/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --basedir=/opt/zimbra/mysql --datadir=/opt/zimbra/db/data --pid-file=/opt/zimbra/db/mysql.pid --skip-external-locking --port=7306 --socket=
7231 pts/1 S 0:00 \_ /opt/zimbra/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --basedir=/opt/zimbra/mysql --datadir=/opt/zimbra/db/data --pid-file=/opt/zimbra/db/mysql.pid --skip-external-locking --port=7306 --socket=
7456 pts/1 S 0:00 \_ /opt/zimbra/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --basedir=/opt/zimbra/mysql --datadir=/opt/zimbra/db/data --pid-file=/opt/zimbra/db/mysql.pid --skip-external-locking --port=7306 --socket=
7967 pts/1 S 0:00 \_ /opt/zimbra/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.cnf --basedir=/opt/zimbra/mysql --datadir=/opt/zimbra/db/data --pid-file=/opt/zimbra/db/mysql.pid --skip-external-locking --port=7306 --socket=
7083 pts/1 S 0:00 /usr/bin/perl /opt/zimbra/libexec/zmmtaconfig ldap_connect_timeout=30000 zimbra_user=zimbra debug_mailboxindex_use_new_locking=true JAVA_JVM_VERSION=1.5 postfix_header_checks=pcre:/opt/zimbra/conf/postfix_header_checks nio_w
7065 pts/1 S 0:00 /usr/bin/perl /opt/zimbra/libexec/logswatch --config-file=/opt/zimbra/conf/logswatchrc --use-cpan-file-tail --script-dir=/tmp -t /var/log/zimbra.log ldap_connect_timeout=30000 zimbra_user=zimbra debug_mailboxindex_use_new_lo
7082 pts/1 S 0:00 \_ /usr/bin/perl /tmp/.swatch_script.7065 ldap_connect_timeout=30000 zimbra_user=zimbra debug_mailboxindex_use_new_locking=true JAVA_JVM_VERSION=1.5 postfix_header_checks=pcre:/opt/zimbra/conf/postfix_header_checks nio_writ
7151 pts/1 S 0:00 \_ /usr/bin/perl /opt/zimbra/libexec/zmlogger ldap_connect_timeout=30000 zimbra_user=zimbra debug_mailboxindex_use_new_locking=true JAVA_JVM_VERSION=1.5 postfix_header_checks=pcre:/opt/zimbra/conf/postfix_header_checks
7058 pts/1 S 0:00 /bin/sh /opt/zimbra/logger/mysql/bin/mysqld_safe --defaults-file=/opt/zimbra/conf/my.logger.cnf --external-locking --ledir=/opt/zimbra/logger/mysql/libexec ldap_connect_timeout=30000 zimbra_user=zimbra debug_mailboxindex_use
7202 pts/1 S 0:00 \_ /opt/zimbra/logger/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.logger.cnf --basedir=/opt/zimbra/logger/mysql --datadir=/opt/zimbra/logger/db/data --pid-file=/opt/zimbra/logger/db/mysql.pid --skip-external-lo
7255 pts/1 S 0:00 \_ /opt/zimbra/logger/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.logger.cnf --basedir=/opt/zimbra/logger/mysql --datadir=/opt/zimbra/logger/db/data --pid-file=/opt/zimbra/logger/db/mysql.pid --skip-externa
7256 pts/1 S 0:00 \_ /opt/zimbra/logger/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.logger.cnf --basedir=/opt/zimbra/logger/mysql --datadir=/opt/zimbra/logger/db/data --pid-file=/opt/zimbra/logger/db/mysql.pid --skip-ext
7257 pts/1 S 0:00 \_ /opt/zimbra/logger/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.logger.cnf --basedir=/opt/zimbra/logger/mysql --datadir=/opt/zimbra/logger/db/data --pid-file=/opt/zimbra/logger/db/mysql.pid --skip-ext
7259 pts/1 S 0:00 \_ /opt/zimbra/logger/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.logger.cnf --basedir=/opt/zimbra/logger/mysql --datadir=/opt/zimbra/logger/db/data --pid-file=/opt/zimbra/logger/db/mysql.pid --skip-ext
7262 pts/1 S 0:00 \_ /opt/zimbra/logger/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.logger.cnf --basedir=/opt/zimbra/logger/mysql --datadir=/opt/zimbra/logger/db/data --pid-file=/opt/zimbra/logger/db/mysql.pid --skip-ext
7271 pts/1 S 0:00 \_ /opt/zimbra/logger/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.logger.cnf --basedir=/opt/zimbra/logger/mysql --datadir=/opt/zimbra/logger/db/data --pid-file=/opt/zimbra/logger/db/mysql.pid --skip-ext
7272 pts/1 S 0:00 \_ /opt/zimbra/logger/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.logger.cnf --basedir=/opt/zimbra/logger/mysql --datadir=/opt/zimbra/logger/db/data --pid-file=/opt/zimbra/logger/db/mysql.pid --skip-ext
7273 pts/1 S 0:00 \_ /opt/zimbra/logger/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.logger.cnf --basedir=/opt/zimbra/logger/mysql --datadir=/opt/zimbra/logger/db/data --pid-file=/opt/zimbra/logger/db/mysql.pid --skip-ext
7274 pts/1 S 0:00 \_ /opt/zimbra/logger/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.logger.cnf --basedir=/opt/zimbra/logger/mysql --datadir=/opt/zimbra/logger/db/data --pid-file=/opt/zimbra/logger/db/mysql.pid --skip-ext
7353 pts/1 S 0:00 \_ /opt/zimbra/logger/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.logger.cnf --basedir=/opt/zimbra/logger/mysql --datadir=/opt/zimbra/logger/db/data --pid-file=/opt/zimbra/logger/db/mysql.pid --skip-ext
8061 pts/1 S 0:00 \_ /opt/zimbra/logger/mysql/libexec/mysqld --defaults-file=/opt/zimbra/conf/my.logger.cnf --basedir=/opt/zimbra/logger/mysql --datadir=/opt/zimbra/logger/db/data --pid-file=/opt/zimbra/logger/db/mysql.pid --skip-ext

Shutdown mysql and run a db integrity report on it (/opt/zimbra/libexec/zmdbintegrityreport)

doesn't work when its down.

/opt/zimbra/libexec/zmdbintegrityreport -v
Generating report
No errors found

Greetings Henrik

[henrik_b]

Any ideas about this? For me the behaviour of mysqld looks normal, why does zimbra complains about this?

Henrik