Error Starting Zimbra after IP Change

[wdimmit]

I recently had to re-ip one of my Zimbra boxes. After changing the DNS records and the hosts file entry, Zimbra will not restart properly. When I try to start (from a clean reboot), the startup process dies after "Starting MTA:" with the following message:

postmap: fatal: bad numerical configuration: message_size_limit =
postalias: fatal: bad numerical configuration: message_size_limit =
postfix: fatal: bad numerical configuration: message_size_limit =

I've verified that the LDAP and mySQL servers are indeed running and can be seen by Zimbra.

Tomcat also fails to start, with the following showing up in /var/log/zimbra.log after trying to manually start it:

Oct 31 14:59:48 postal zmtomcatmgr[20996]: stale pid 28885 found in /opt/zimbra/log/zmtomcatmgr.pid: No such process
Oct 31 14:59:48 postal zmtomcatmgr[20996]: assuming no other instance is running
Oct 31 14:59:48 postal zmtomcatmgr[20997]: wrote manager pid 20997 to /opt/zimbra/log/zmtomcatmgr.pid
Oct 31 14:59:48 postal zmtomcatmgr[20997]: manager started tomcat/JVM with pid 20998
Oct 31 14:59:54 postal zmtomcatmgr[20997]: manager woke up from wait on tomcat/JVM with pid 20998
Oct 31 14:59:54 postal zmtomcatmgr[20997]: manager started tomcat/JVM with pid 21018
Oct 31 15:00:00 postal zmtomcatmgr[20997]: manager woke up from wait on tomcat/JVM with pid 21018
Oct 31 15:00:00 postal zmtomcatmgr[20997]: tomcat/JVM exited twice in 6 seconds (tolerance=60)

And corresponding to the above in catalina.out is a repeating:

Oct 31, 2006 2:59:56 PM org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-443
Oct 31, 2006 2:59:56 PM org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-7071
Oct 31, 2006 2:59:56 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1258 ms
Oct 31, 2006 2:59:56 PM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Oct 31, 2006 2:59:56 PM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/5.5.15
Oct 31, 2006 2:59:56 PM org.apache.catalina.core.StandardHost start
INFO: XML validation disabled
Oct 31, 2006 2:59:57 PM org.apache.catalina.startup.HostConfig deployDescriptor
WARNING: A docBase /opt/zimbra/apache-tomcat-5.5.15/webapps/zimbra inside the host appBase has been specified, and will be ignored
log4j:WARN No appenders could be found for logger (org.apache.catalina.session.ManagerBase).
log4j:WARN Please initialize the log4j system properly.
Zimbra server process is running as root, changing to user=zimbra uid=500 gid=500
Zimbra server process, after change, is running with uid=500 euid=500 gid=500 egid=500

I've dug through the forums and wiki and have not been able to find a similar case, and have tried solutions to similar problems (firewall is not an issue, SElinux is disabled, DNS and hosts files both report correct IP, there are no port collisions, zmfixperms doesn't seem to find any problems). I've also reinstalled over-the-top, as I was able to correct a strange problem some time ago by reinstalling.

This is ZCS 4.0.3 running on Centos 4 (2.6.9-42 Kernel). Any help would be hugely appreciated.

Thank you,
Will

[czaveri]

Why stale PIDs? Did you change the IP while Zimbra was running?

Perhaps, deleting stale PID files should help?!

[wdimmit]

I just gave that a go (cleared out zmtomcatmgr.pid), but there was no change. As a side note, after shutting down Zimbra, the only pid file left is that (zmtomcatmgr.pid) one, and it has different permissions than the rest of the zimbra pid's:

-rw-r--r--  1 root root 5 Oct 31 15:46 zmtomcatmgr.pid

I *think* the "stale pid" messages are from tomcat restarting itself. Thank you for the suggestion though!

Thanks,
Will

[czaveri]

Perhaps, you have already looked at these. Pasting them here anyways:

Tomcat crashes after upgrade to 4.0.2
Tomcat not starting
fatal: Queue report unavailable - mail system is down
Tomcat !! Newbee

I am not saying that these contain a solution. But they might.

[wdimmit]

Thanks again for the tips. Most of those dealt with port conflicts, and I've got this server stripped down to the bone (service wise), at the moment to remove that as a possibility.

Two additional notes:
When I reinstall over-the-top, I am asked to reset the spam and ham email addresses every time, they do *not* seem to be carrying over.
Does anyone have any idea whether this (below) is a symptom or a cause?

postmap: fatal: bad numerical configuration: message_size_limit =
postalias: fatal: bad numerical configuration: message_size_limit =
postfix: fatal: bad numerical configuration: message_size_limit =

Again, thanks,
Will

[jholder]

Take a look at this post:
Zimbra will not start after upgrade to 3.1.2

[wdimmit]

I've reset the LDAP passwords (twice, just to be sure). I've not seen anything that seems to indicate an auth problem with either mySQL or LDAP. When I run 'zmprov gaa' (or zmprov + anything else) I get:

[] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
[] INFO: Retrying request
[] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
[] INFO: Retrying request
[] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
[] INFO: Retrying request
ERROR: zclient.IO_ERROR (invoke Connection refused, server: localhost) (cause: java.net.ConnectException Connection refused)

Which I've attributed to tomcat being down. As a side note, something I've done during the course of all this has caused " Sleeping...Couldn't contact LDAP return=-1" to pop up periodically in the "zimbra" user's session. I've not yet found what process or log that message belongs to.

Thanks for all the suggestions!
Will

[jholder]

Can you telnet to localhost:389 or host.domain.com:389 ?

Also, could you post your zmlocalconfig ?

[jholder]

Also, try
zmprov mcf zimbraMtaMaxMessageSize 10240000

[wdimmit]

Telnet to LDAP (389) at "localhost" fails. Telnet to 389 @ "FQDN" connects right up. zmlocalconfig is as follows:

av_notify_user = admin@host.domain.com
ldap_cache_account_maxage = 15
ldap_cache_account_maxsize = 5000
ldap_cache_cos_maxage = 15
ldap_cache_cos_maxsize = 100
ldap_cache_domain_maxage = 15
ldap_cache_domain_maxsize = 100
ldap_cache_server_maxage = 15
ldap_cache_server_maxsize = 100
ldap_cache_timezone_maxsize = 100
ldap_cache_zimlet_maxage = 15
ldap_cache_zimlet_maxsize = 100
ldap_connect_pool_debug = false
ldap_connect_pool_initsize = 1
ldap_connect_pool_maxsize = 25
ldap_connect_pool_prefsize = ${ldap_connect_pool_maxsize}
ldap_connect_pool_timeout = 120000
ldap_connect_timeout = 10000
ldap_host = host.domain.com
ldap_is_master = true
ldap_log_level = 0
ldap_master_url = ldap://host.domain.com:389
ldap_port = 389
ldap_root_password = *
ldap_url = ldap://host.domain.com:389
localized_msgs_directory = ${zimbra_home}/conf/msgs
logger_mysql_bind_address = localhost
logger_mysql_data_directory = ${zimbra_home}/logger/db/data
logger_mysql_directory = ${zimbra_home}/logger/mysql
logger_mysql_mycnf = ${zimbra_home}/conf/my.logger.cnf
logger_mysql_pidfile = ${zimbra_home}/logger/db/mysql.pid
logger_mysql_port = 7307
logger_mysql_socket = ${zimbra_home}/logger/db/mysql.sock
mysql_bind_address = localhost
mysql_data_directory = ${zimbra_db_directory}/data
mysql_directory = ${zimbra_home}/mysql
mysql_innodb_log_buffer_size = 8388608
mysql_innodb_log_file_size = 104857600
mysql_logger_root_password = *
mysql_memory_percent = 40
mysql_mycnf = ${zimbra_home}/conf/my.cnf
mysql_pidfile = ${zimbra_db_directory}/mysql.pid
mysql_port = 7306
mysql_read_buffer_size = 1048576
mysql_root_password = *
mysql_socket = ${zimbra_db_directory}/mysql.sock
mysql_sort_buffer_size = 1048576
mysql_table_cache = 500
nio_imap_enable = false
nio_imap_log_buffers = false
nio_imap_write_queue_max_size = 10240000
nio_imap_write_queue_max_size_unauth = 20480
nio_write_buffer_compaction_percent = 50
postfix_alias_maps = hash:/etc/aliases
postfix_broken_sasl_auth_clients = yes
postfix_command_directory = /opt/zimbra/postfix-${postfix_version}/sbin
postfix_daemon_directory = /opt/zimbra/postfix-${postfix_version}/libexec
postfix_header_checks = pcre:/opt/zimbra/conf/postfix_header_checks
postfix_mailq_path = /opt/zimbra/postfix-${postfix_version}/sbin/mailq
postfix_manpage_directory = /opt/zimbra/postfix-${postfix_version}/man
postfix_newaliases_path = /opt/zimbra/postfix-${postfix_version}/sbin/newaliases
postfix_queue_directory = /opt/zimbra/postfix-${postfix_version}/spool
postfix_sender_canonical_maps = ldap:/opt/zimbra/conf/ldap-scm.cf
postfix_sendmail_path = /opt/zimbra/postfix-${postfix_version}/sbin/sendmail
postfix_smtpd_client_restrictions = reject_unauth_pipelining
postfix_smtpd_data_restrictions = reject_unauth_pipelining
postfix_smtpd_helo_required = yes
postfix_smtpd_tls_cert_file = ${zimbra_home}/conf/smtpd.crt
postfix_smtpd_tls_key_file = ${zimbra_home}/conf/smtpd.key
postfix_smtpd_tls_loglevel = 3
postfix_transport_maps = ldap:/opt/zimbra/conf/ldap-transport.cf
postfix_version = 2.2.9
postfix_virtual_alias_domains = ldap://opt/zimbra/conf/ldap-vad.cf
postfix_virtual_alias_maps = ldap:/opt/zimbra/conf/ldap-vam.cf
postfix_virtual_mailbox_domains = ldap:/opt/zimbra/conf/ldap-vmd.cf
postfix_virtual_mailbox_maps = ldap:/opt/zimbra/conf/ldap-vmm.cf
postfix_virtual_transport = error
smtp_destination = admin@host.domain.com
smtp_notify = yes
smtp_source = admin@host.domain.com
snmp_notify = yes
snmp_trap_host = host.domain.com
ssl_allow_untrusted_certs = TRUE
stats_img_folder = ${zimbra_home}/zimbramon/rrdtool/work
tomcat_directory = ${zimbra_home}/tomcat
tomcat_java_heap_memory_percent = 30
tomcat_java_home = ${zimbra_java_home}
tomcat_java_options = -client -XX:NewRatio=2
tomcat_keystore = ${tomcat_directory}/conf/keystore
tomcat_pidfile = ${zimbra_log_directory}/tomcat.pid
wiki_enabled = false
wiki_user = wiki
zimbra_admin_service_port = 7071
zimbra_attrs_directory = ${zimbra_home}/conf/attrs
zimbra_auth_always_send_refer = false
zimbra_db_directory = ${zimbra_home}/db
zimbra_extension_common_directory = ${zimbra_home}/lib/ext-common
zimbra_extension_directory = ${zimbra_home}/lib/ext
zimbra_gid = 500
zimbra_home = /opt/zimbra
zimbra_index_directory = ${zimbra_home}/index
zimbra_index_idle_flush_time = 600
zimbra_index_lru_size = 100
zimbra_index_max_uncommitted_operations = 200
zimbra_java_home = ${zimbra_home}/java
zimbra_ldap_password = *
zimbra_ldap_user = zimbra
zimbra_ldap_userdn = uid=zimbra,cn=admins,cn=zimbra
zimbra_log4j_properties = ${zimbra_home}/conf/log4j.properties
zimbra_log_directory = ${zimbra_home}/log
zimbra_logger_mysql_password = *
zimbra_mtareport_max_recipients = 50
zimbra_mtareport_max_senders = 50
zimbra_mysql_connector_maxActive = 10
zimbra_mysql_password = *
zimbra_mysql_user = zimbra
zimbra_server_hostname = host.domain.com
zimbra_spam_report_queue_size = 100
zimbra_store_directory = ${zimbra_home}/store
zimbra_store_sweeper_max_age = 480
zimbra_throttle_op_concurrency = 1000,1000,1000,1000,1000
zimbra_tmp_directory = /tmp/zimbra
zimbra_uid = 500
zimbra_user = zimbra
zimbra_zmprov_default_soap_server = localhost
zimbra_zmprov_default_to_ldap = false
zimlet_directory = ${tomcat_directory}/webapps/service/zimlet

Thanks,
Will