Upgrade 4.5.1 -> 4.5.2 Network Edition on RH Failed

[CJ.deb]

Hi,

tested two times to upgrade to 4.5.2 (from 4.5.1)

But failed with following error:

Fri Feb 23 20:43:03 2007: Starting mysql
ERROR 1017 (HY000) at line 1: Can't find file: 'config' (errno: 2)
Error while running '/opt/zimbra/bin/mysql --user=zimbra --password=[password here] --database=zimbra --batch --skip-column-names'. at /opt/zimbra/libexec/scripts/Migrate.pm line 220.

Fortunately downgrade afterwards was OK

Any idea ?

Cheers
Christian

[jholder]

Lets take a look at a couple things. Brian will have more info than I but what's your
df -h look like?

Also, try running the zmfixperms script located in the /opt/zimbra/libexec dir, and let me know how it works out.

-john

[brian]

Are you sure your 4.5.1 install is functioning correctly? That error indicates the zimbra db config table is missing or corrupt. Either way your install wouldn't really function without it.

Make surethe config table is there.
mysql -u zimbra -Dzimbra -e "show tables;"

mysql HY000 1017 is file not found so make sure /opt/zimbra/db/data/zimbra/config.{frm,ibd} exists

[CJ.deb]

* Are you sure your 4.5.1 install is functioning correctly?

Yes, using it daily for testing incl. mobile and Outlook

zmcontrol status
Host abcd
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Running
mta Running
snmp Running
spell Running

* what's your df -h look like?

Easy ;-)

Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
536G 49G 460G 10% /
/dev/sda1 99M 52M 43M 55% /boot
none 4.0G 0 4.0G 0% /dev/shm

* try running the zmfixperms script located in the /opt/zimbra/libexec dir, and let me know how it works out.

/opt/zimbra/libexec/

As zimbra
zmfixperms (Operation not permitted)

As root
zmfixperms (No output at all)

* Make sure the config table is there.
mysql -u zimbra -Dzimbra -e "show tables;"

su zimbra
$ mysql -u zimbra -Dzimbra -e "show tables;"
+-------------------+
| Tables_in_zimbra |
+-------------------+
| config |
| current_volumes |
| mailbox |
| mailbox_metadata |
| out_of_office |
| service_status |
| table_maintenance |
| volume |
+-------------------+

* mysql HY000 1017 is file not found so make sure /opt/zimbra/db/data/zimbra/config.{frm,ibd} exists

cd /opt/zimbra/db/data/zimbra
$ ll
total 1124
-rw-rw---- 1 zimbra zimbra 8674 Jul 12 2006 config.frm
-rw-rw---- 1 zimbra zimbra 98304 Jan 17 07:38 config.ibd

* OK - tried again :

Did a fresh Download ....

Install :

:-( :-( :-( :-( No Change :-( :-( :-( :-(

The Zimbra Collaboration Suite appears already to be installed.
It can be upgraded with no effect on existing accounts,
or the current installation can be completely removed prior
to installation for a clean install.

Do you wish to upgrade? [Y]

Select the packages to install
Upgrading zimbra-core
Upgrading zimbra-ldap
Upgrading zimbra-logger
Upgrading zimbra-mta
Upgrading zimbra-snmp
Upgrading zimbra-store
Upgrading zimbra-apache
Upgrading zimbra-spell

Installing:
zimbra-core
zimbra-ldap
zimbra-logger
zimbra-mta
zimbra-snmp
zimbra-store
zimbra-apache
zimbra-spell

The system will be modified. Continue? [N] y

Shutting down zimbra mail

Backing up ldap


Removing existing packages

zimbra-ldap...done
zimbra-logger...done
zimbra-mta...done
zimbra-snmp...done
zimbra-store...done
zimbra-spell...done
zimbra-apache...done
zimbra-core...done

Removing deployed webapp directories
Installing packages

zimbra-core......zimbra-core-4.5.2_GA_699.RHEL4-20070216151950.i386.rpm...done
zimbra-ldap......zimbra-ldap-4.5.2_GA_699.RHEL4-20070216151950.i386.rpm...done
zimbra-logger......zimbra-logger-4.5.2_GA_699.RHEL4-20070216151950.i386.rpm...done
zimbra-mta......zimbra-mta-4.5.2_GA_699.RHEL4-20070216151950.i386.rpm...done
zimbra-snmp......zimbra-snmp-4.5.2_GA_699.RHEL4-20070216151950.i386.rpm...done
zimbra-store......zimbra-store-4.5.2_GA_699.RHEL4-20070216151950.i386.rpm...done
zimbra-apache......zimbra-apache-4.5.2_GA_699.RHEL4-20070216151950.i386.rpm...done
zimbra-spell......zimbra-spell-4.5.2_GA_699.RHEL4-20070216151950.i386.rpm...done

Setting defaults from saved config in /opt/zimbra/.saveconfig/config.save
HOSTNAME=abcd
LDAPHOST=abcd
LDAPPORT=389
SNMPTRAPHOST=abcd
SMTPSOURCE=abcd
SMTPDEST=abcd
SNMPNOTIFY=yes
SMTPNOTIFY=yes
LDAPROOTPW=abcd
LDAPZIMBRAPW=abcd

=====>>>>>>>>>>

Restoring existing configuration file from /opt/zimbra/.saveconfig/config.save...Restoring backup schedule...done
Operations logged to /tmp/zmsetup.log.14032
Setting defaults...Setting defaults from existing config...Done
Upgrading from 4.5.1_GA_660 to 4.5.2_GA_699
Sat Feb 24 16:48:57 2007: Stopping zimbra services
Sat Feb 24 16:49:02 2007: Starting mysql
ERROR 1017 (HY000) at line 1: Can't find file: 'config' (errno: 2)
Error while running '/opt/zimbra/bin/mysql --user=zimbra --password=[Password Here] --database=zimbra --batch --skip-column-names'. at /opt/zimbra/libexec/scripts/Migrate.pm line 220.

[brian]

How much memory does your server have? Your buffer settings in /opt/zimbra/conf/my.cnf might be set too high, during the initial install. Check the output in /opt/zimbra/db/data/.err for malloc errors, on startup. We had another report of this same error and that was the cause.

[CJ.deb]

Originally Posted by brian

How much memory does your server have? Your buffer settings in /opt/zimbra/conf/my.cnf might be set too high, during the initial install. Check the output in /opt/zimbra/db/data/.err for malloc errors, on startup. We had another report of this same error and that was the cause.

Hi Brain

8GB memory -> can be set to high ;-)
Hope for your advise.

cheers
Christian


* cat /proc/meminfo
MemTotal: 8308584 kB
MemFree: 5386764 kB
Buffers: 100784 kB
Cached: 1158656 kB
SwapCached: 0 kB
Active: 2032176 kB
Inactive: 756248 kB
HighTotal: 7470840 kB
HighFree: 4771520 kB
LowTotal: 837744 kB
LowFree: 615244 kB
SwapTotal: 2031608 kB
SwapFree: 2031608 kB
Dirty: 608 kB
Writeback: 0 kB
Mapped: 1518228 kB
Slab: 109328 kB
CommitLimit: 6185900 kB
Committed_AS: 6350096 kB
PageTables: 7888 kB
VmallocTotal: 106488 kB
VmallocUsed: 3000 kB
VmallocChunk: 102984 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB

* /opt/zimbra/conf/my.cnf

[mysqld]

basedir = /opt/zimbra/mysql
datadir = /opt/zimbra/db/data
socket = /opt/zimbra/db/mysql.sock
pid-file = /opt/zimbra/db/mysql.pid
bind-address = localhost
port = 7306
user = zmbra

skip-external-locking

log-slow-queries = /opt/zimbra/log/myslow.log
long-query-time = 1
log-long-format
log-queries-not-using-indexes
log-bin

thread_cache = 71
max_connections = 71

# We do a lot of writes, query cache turns out to be not useful.
query_cache_type = 0

sort_buffer_size = 1048576
read_buffer_size = 1048576

# Increase the size of the table cache, since each mailbox has its
# own set of tables
table_cache = 500

innodb_buffer_pool_size = 2552400690
innodb_log_file_size = 104857600
innodb_log_buffer_size = 8388608
innodb_file_per_table

[mysqld_safe]

err-log = /opt/zimbra/log/mysqld.log

__________________________________________________ ________

* cat /opt/zimbra/db/data/.err

Sorry couldn't give the upgrade a new try at the moment.

Just attached the "gibberish" out of the error file. Will give it a clean try asap.

------------------------------------------------------
InnoDB: Error in opening ./ibdata1
070224 16:52:45 InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.0/...ror-codes.html
InnoDB: Could not open or create data files.
InnoDB: If you tried to add new data files, and it failed here,
InnoDB: you should now edit innodb_data_file_path in my.cnf back
InnoDB: to what it was, and remove the new ibdata files InnoDB created
InnoDB: in this failed attempt. InnoDB only wrote those files full of
InnoDB: zeros, but did not yet use them in any way. But be careful: do not
InnoDB: remove old data files which contain your precious data!
070224 16:52:45 [Note] /opt/zimbra/mysql/libexec/mysqld: ready for connections.
Version: '5.0.33-log' socket: '/opt/zimbra/db/mysql.sock' port: 7306 Source distribution
InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
070224 16:52:46 InnoDB: Unable to open the first data file
InnoDB: Error in opening ./ibdata1
070224 16:52:46 InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.0/...ror-codes.html
InnoDB: Could not open or create data files.
InnoDB: If you tried to add new data files, and it failed here,
InnoDB: you should now edit innodb_data_file_path in my.cnf back
InnoDB: to what it was, and remove the new ibdata files InnoDB created
InnoDB: in this failed attempt. InnoDB only wrote those files full of
InnoDB: zeros, but did not yet use them in any way. But be careful: do not
InnoDB: remove old data files which contain your precious data!
070224 16:52:46 [ERROR] Can't start server: Bind on TCP/IP port: Address already in use
070224 16:52:46 [ERROR] Do you already have another mysqld server running on port: 7306 ?
070224 16:52:46 [ERROR] Aborting

070224 16:52:46 [Note] /opt/zimbra/mysql/libexec/mysqld: Shutdown complete

InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
070224 16:52:46 InnoDB: Unable to open the first data file
InnoDB: Error in opening ./ibdata1
070224 16:52:46 InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.0/...ror-codes.html
InnoDB: Could not open or create data files.
InnoDB: If you tried to add new data files, and it failed here,
InnoDB: you should now edit innodb_data_file_path in my.cnf back
InnoDB: to what it was, and remove the new ibdata files InnoDB created
InnoDB: in this failed attempt. InnoDB only wrote those files full of
InnoDB: zeros, but did not yet use them in any way. But be careful: do not
InnoDB: remove old data files which contain your precious data!
070224 16:52:46 [ERROR] Can't start server: Bind on TCP/IP port: Address already in use
070224 16:52:46 [ERROR] Do you already have another mysqld server running on port: 7306 ?
070224 16:52:46 [ERROR] Aborting

070224 16:52:46 [Note] /opt/zimbra/mysql/libexec/mysqld: Shutdown complete

070224 16:52:46 mysqld ended

070224 16:52:46 mysqld ended

070224 16:53:05 InnoDB: We now intentionally generate a seg fault so that
InnoDB: on Linux we get a stack trace.
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=0
read_buffer_size=1044480
max_used_connections=0
max_connections=71
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 145123 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbfe5988c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x816df93
0x8366ac9
0x8366b9a
0x8363e05
0x8320e05
0x832551a
0x8292755
0x822ab43
0x8220a32
0x816e30e
0x8171f21
0x1d8de3
0x80e63f1
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/us...ack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
070224 16:53:05 mysqld ended

070224 16:54:53 mysqld started
070224 16:54:53 [Warning] One can only use the --user switch if running as root

070224 16:54:53 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=zmail-bin' to avoid this problem.
070224 16:54:54 InnoDB: Error: cannot allocate 161631436 bytes of
InnoDB: memory with malloc! Total allocated memory
InnoDB: by InnoDB 2740576092 bytes. Operating system errno: 12
InnoDB: Check if you should increase the swap file or
InnoDB: ulimits of your operating system.
InnoDB: On FreeBSD check you have compiled the OS with
InnoDB: a big enough maximum process size.
InnoDB: Note that in most 32-bit computers the process
InnoDB: memory space is limited to 2 GB or 4 GB.
InnoDB: We keep retrying the allocation for 60 seconds...
070224 16:55:54 InnoDB: We now intentionally generate a seg fault so that
InnoDB: on Linux we get a stack trace.
mysqld got signal 11;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=0
read_buffer_size=1044480
max_used_connections=0
max_connections=71
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_connections = 145123 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd=(nil)
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
Cannot determine thread, fp=0xbfecb90c, backtrace may not be correct.
Stack range sanity check OK, backtrace follows:
0x816df93
0x8366ac9
0x8366b9a
0x8363e05
0x8320e05
0x832551a
0x8292755
0x822ab43
0x8220a32
0x816e30e
0x8171f21
0x1d8de3
0x80e63f1
New value of fp=(nil) failed sanity check, terminating stack trace!
Please read http://dev.mysql.com/doc/mysql/en/us...ack-trace.html and follow instructions on how to resolve the stack trace. Resolved
stack trace is much more helpful in diagnosing the problem, so please do
resolve it
The manual page at http://www.mysql.com/doc/en/Crashing.html contains
information that should help you find out what is causing the crash.
070224 16:55:54 mysqld ended

070224 17:00:26 mysqld started
070224 17:00:26 [Warning] One can only use the --user switch if running as root

070224 17:00:26 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=zmail-bin' to avoid this problem.
070224 17:00:28 InnoDB: Started; log sequence number 0 242054503
070224 17:00:28 [ERROR] Can't start server: Bind on TCP/IP port: Address already in use
070224 17:00:28 [ERROR] Do you already have another mysqld server running on port: 7306 ?
070224 17:00:28 [ERROR] Aborting

070224 17:00:28 InnoDB: Starting shutdown...
070224 17:00:30 InnoDB: Shutdown completed; log sequence number 0 242054503
070224 17:00:30 [Note] /opt/zimbra/mysql/libexec/mysqld: Shutdown complete

070224 17:00:30 mysqld ended

070224 17:44:20 mysqld started
070224 17:44:20 [Warning] One can only use the --user switch if running as root

070224 17:44:20 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=zmail-bin' to avoid this problem.
070224 17:44:21 InnoDB: Started; log sequence number 0 242054503
070224 17:44:21 [ERROR] Can't start server: Bind on TCP/IP port: Address already in use
070224 17:44:21 [ERROR] Do you already have another mysqld server running on port: 7306 ?
070224 17:44:21 [ERROR] Aborting

070224 17:44:21 InnoDB: Starting shutdown...
070224 17:44:24 InnoDB: Shutdown completed; log sequence number 0 242054503
070224 17:44:24 [Note] /opt/zimbra/mysql/libexec/mysqld: Shutdown complete

070224 17:44:24 mysqld ended

070224 17:47:55 [Note] /opt/zimbra/mysql/libexec/mysqld: Normal shutdown

070224 17:47:55 [Note] /opt/zimbra/mysql/libexec/mysqld: Shutdown complete

070224 17:47:55 mysqld ended

070224 17:52:53 mysqld started
070224 17:52:53 [Warning] One can only use the --user switch if running as root

070224 17:52:53 [Warning] No argument was provided to --log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '--log-bin=zmail-bin' to avoid this problem.
070224 17:52:55 InnoDB: Started; log sequence number 0 242054503
070224 17:52:55 [Note] /opt/zimbra/mysql/libexec/mysqld: ready for connections.
Version: '5.0.26-log' socket: '/opt/zimbra/db/mysql.sock' port: 7306 Source distribution
A mysqld process already exists at Wed Feb 28 07:46:15 CET 2007

[phoenix]

Try reducing the "innodb_buffer_pool_size " to something less than 2GB, that should fix the problem.

[CJ.deb]

Originally Posted by phoenix

Try reducing the "innodb_buffer_pool_size " to something less than 2GB, that should fix the problem.

- RESOLVED -

TA !

=> innodb_buffer_pool_size = 1992294400
Did the trick

:-)

Cheers
Christian