Was Rock Solid, Now Tomcat will not start.

[venaro]

Greetings!

I've had Release 4.0.2_GA_362.FC5_20060922122246 FC5 FOSS edition running for a few months with no issues.

The server was recently moved and had an ip address change. It came back up fine, but we had a power outtage today and now tomcat will not start. I've noticed a line in the catalina.out referencing the old local ip address, but do not know how to change it, or if that it the source of my pain.

I've searched all the threads that seemed to relate. I've backed up and tried going through an upgrade reinstall with no improvement.

Help me obiwan(s), your my only hope.

Logfiles below. Happy to dig deeper if you need more info.

Venaro

#status
<snip>
[zimbra@zeplin libexec]$ zmcontrol status
Host venaro.net
antispam Running
antivirus Running
ldap Running
logger Running
mailbox Stopped
tomcat is not running
mta Running
snmp Running
spell Running
[zimbra@zeplin libexec]$
<snip>

# zmlocalconfig
<snip>
[zimbra@zeplin libexec]$ zmlocalconfig
av_notify_user = admin@venaro.net
ldap_cache_account_maxage = 15
ldap_cache_account_maxsize = 5000
ldap_cache_cos_maxage = 15
ldap_cache_cos_maxsize = 100
ldap_cache_domain_maxage = 15
ldap_cache_domain_maxsize = 100
ldap_cache_server_maxage = 15
ldap_cache_server_maxsize = 100
ldap_cache_timezone_maxsize = 100
ldap_cache_zimlet_maxage = 15
ldap_cache_zimlet_maxsize = 100
ldap_connect_pool_debug = false
ldap_connect_pool_initsize = 1
ldap_connect_pool_maxsize = 50
ldap_connect_pool_prefsize = 0
ldap_connect_pool_timeout = 120000
ldap_connect_timeout = 30000
ldap_host = venaro.net
ldap_is_master = true
ldap_log_level = 0
ldap_master_url = ldap://venaro.net:389
ldap_port = 389
ldap_root_password = *
ldap_url = ldap://venaro.net:389
localized_msgs_directory = ${zimbra_home}/conf/msgs
logger_mysql_bind_address = localhost
logger_mysql_data_directory = ${zimbra_home}/logger/db/data
logger_mysql_directory = ${zimbra_home}/logger/mysql
logger_mysql_mycnf = ${zimbra_home}/conf/my.logger.cnf
logger_mysql_pidfile = ${zimbra_home}/logger/db/mysql.pid
logger_mysql_port = 7307
logger_mysql_socket = ${zimbra_home}/logger/db/mysql.sock
mysql_bind_address = localhost
mysql_data_directory = ${zimbra_db_directory}/data
mysql_directory = ${zimbra_home}/mysql
mysql_innodb_log_buffer_size = 8388608
mysql_innodb_log_file_size = 104857600
mysql_logger_root_password = *
mysql_memory_percent = 40
mysql_mycnf = ${zimbra_home}/conf/my.cnf
mysql_pidfile = ${zimbra_db_directory}/mysql.pid
mysql_port = 7306
mysql_read_buffer_size = 1048576
mysql_root_password = *
mysql_socket = ${zimbra_db_directory}/mysql.sock
mysql_sort_buffer_size = 1048576
mysql_table_cache = 500
nio_imap_enable = false
nio_imap_log_buffers = false
nio_imap_write_queue_max_size = 10240000
nio_imap_write_queue_max_size_unauth = 20480
nio_write_buffer_compaction_percent = 50
postfix_alias_maps = hash:/etc/aliases
postfix_broken_sasl_auth_clients = yes
postfix_command_directory = /opt/zimbra/postfix-${postfix_version}/sbin
postfix_daemon_directory = /opt/zimbra/postfix-${postfix_version}/libexec
postfix_header_checks = pcre:/opt/zimbra/conf/postfix_header_checks
postfix_mailq_path = /opt/zimbra/postfix-${postfix_version}/sbin/mailq
postfix_manpage_directory = /opt/zimbra/postfix-${postfix_version}/man
postfix_newaliases_path = /opt/zimbra/postfix-${postfix_version}/sbin/newaliases
postfix_queue_directory = /opt/zimbra/postfix-${postfix_version}/spool
postfix_sender_canonical_maps = ldap:/opt/zimbra/conf/ldap-scm.cf
postfix_sendmail_path = /opt/zimbra/postfix-${postfix_version}/sbin/sendmail
postfix_smtpd_client_restrictions = reject_unauth_pipelining
postfix_smtpd_data_restrictions = reject_unauth_pipelining
postfix_smtpd_helo_required = yes
postfix_smtpd_tls_cert_file = ${zimbra_home}/conf/smtpd.crt
postfix_smtpd_tls_key_file = ${zimbra_home}/conf/smtpd.key
postfix_smtpd_tls_loglevel = 1
postfix_transport_maps = ldap:/opt/zimbra/conf/ldap-transport.cf
postfix_version = 2.2.9
postfix_virtual_alias_domains = ldap://opt/zimbra/conf/ldap-vad.cf
postfix_virtual_alias_maps = ldap:/opt/zimbra/conf/ldap-vam.cf
postfix_virtual_mailbox_domains = ldap:/opt/zimbra/conf/ldap-vmd.cf
postfix_virtual_mailbox_maps = ldap:/opt/zimbra/conf/ldap-vmm.cf
postfix_virtual_transport = error
smtp_destination = admin@venaro.net
smtp_notify = yes
smtp_source = admin@venaro.net
snmp_notify = yes
snmp_trap_host = venaro.net
ssl_allow_untrusted_certs = TRUE
stats_img_folder = ${zimbra_home}/logger/db/work
tomcat_directory = ${zimbra_home}/tomcat
tomcat_java_heap_memory_percent = 30
tomcat_java_home = ${zimbra_java_home}
tomcat_java_options = -client -XX:NewRatio=2
tomcat_keystore = ${tomcat_directory}/conf/keystore
tomcat_pidfile = ${zimbra_log_directory}/tomcat.pid
wiki_enabled = false
wiki_user = wiki
zimbra_admin_service_port = 7071
zimbra_attrs_directory = ${zimbra_home}/conf/attrs
zimbra_auth_always_send_refer = false
zimbra_db_directory = ${zimbra_home}/db
zimbra_extension_common_directory = ${zimbra_home}/lib/ext-common
zimbra_extension_directory = ${zimbra_home}/lib/ext
zimbra_gid = 511
zimbra_home = /opt/zimbra
zimbra_index_directory = ${zimbra_home}/index
zimbra_index_idle_flush_time = 600
zimbra_index_lru_size = 100
zimbra_index_max_uncommitted_operations = 200
zimbra_java_home = ${zimbra_home}/java
zimbra_ldap_password = *
zimbra_ldap_user = zimbra
zimbra_ldap_userdn = uid=zimbra,cn=admins,cn=zimbra
zimbra_log4j_properties = ${zimbra_home}/conf/log4j.properties
zimbra_log_directory = ${zimbra_home}/log
zimbra_logger_mysql_password = *
zimbra_mtareport_max_recipients = 50
zimbra_mtareport_max_senders = 50
zimbra_mysql_connector_maxActive = 16
zimbra_mysql_password = *
zimbra_mysql_user = zimbra
zimbra_server_hostname = venaro.net
zimbra_spam_report_queue_size = 100
zimbra_store_directory = ${zimbra_home}/store
zimbra_store_sweeper_max_age = 480
zimbra_throttle_op_concurrency = 1000,1000,1000,1000,1000
zimbra_tmp_directory = /tmp/zimbra
zimbra_uid = 510
zimbra_user = zimbra
zimbra_zmprov_default_soap_server = localhost
zimbra_zmprov_default_to_ldap = false
zimlet_directory = ${tomcat_directory}/webapps/service/zimlet
[zimbra@zeplin libexec]$
<snip>

#zimbra.log

<snip>
Nov 16 00:35:40 zeplin postfix/qmgr[3520]: 5108CAE0121: to=<chris@venaro.net>, relay=none, delay=0, status=deferred (delivery temporarily suspended: connect to venaro.net[192.168.2.101]: Connection refused)
Nov 16 00:35:40 zeplin postfix/qmgr[3520]: 5431FAE0122: removed
Nov 16 00:36:03 zeplin zmtomcatmgr[5220]: status requested
Nov 16 00:36:03 zeplin zmtomcatmgr[5220]: stale pid 3197 found in /opt/zimbra/log/zmtomcatmgr.pid: No such process
Nov 16 00:36:03 zeplin zmtomcatmgr[5220]: assuming no other instance is running
Nov 16 00:36:03 zeplin zmtomcatmgr[5220]: no manager process is running
Nov 16 00:36:03 zeplin zimbramon[5074]: 5074:info: 2006-11-16 00:36:01, STATUS: venaro.net: antispam: Running
Nov 16 00:36:03 zeplin zimbramon[5074]: 5074:info: 2006-11-16 00:36:01, STATUS: venaro.net: antivirus: Running
Nov 16 00:36:03 zeplin zimbramon[5074]: 5074:info: 2006-11-16 00:36:01, STATUS: venaro.net: ldap: Running
Nov 16 00:36:03 zeplin zimbramon[5074]: 5074:info: 2006-11-16 00:36:01, STATUS: venaro.net: logger: Running
Nov 16 00:36:03 zeplin zimbramon[5074]: 5074:info: 2006-11-16 00:36:01, STATUS: venaro.net: mailbox: Stopped
Nov 16 00:36:03 zeplin zimbramon[5074]: 5074:info: 2006-11-16 00:36:01, STATUS: venaro.net: mta: Running
Nov 16 00:36:03 zeplin zimbramon[5074]: 5074:info: 2006-11-16 00:36:01, STATUS: venaro.net: snmp: Running
Nov 16 00:36:03 zeplin zimbramon[5074]: 5074:info: 2006-11-16 00:36:01, STATUS: venaro.net: spell: Running
Nov 16 00:38:03 zeplin zmtomcatmgr[5738]: status requested
Nov 16 00:38:03 zeplin zmtomcatmgr[5738]: stale pid 3197 found in /opt/zimbra/log/zmtomcatmgr.pid: No such process
Nov 16 00:38:03 zeplin zmtomcatmgr[5738]: assuming no other instance is running
Nov 16 00:38:03 zeplin zmtomcatmgr[5738]: no manager process is running
Nov 16 00:38:04 zeplin zimbramon[5592]: 5592:info: 2006-11-16 00:38:01, STATUS: venaro.net: antispam: Running
Nov 16 00:38:04 zeplin zimbramon[5592]: 5592:info: 2006-11-16 00:38:01, STATUS: venaro.net: antivirus: Running
Nov 16 00:38:04 zeplin zimbramon[5592]: 5592:info: 2006-11-16 00:38:01, STATUS: venaro.net: ldap: Running
Nov 16 00:38:04 zeplin zimbramon[5592]: 5592:info: 2006-11-16 00:38:01, STATUS: venaro.net: logger: Running
Nov 16 00:38:04 zeplin zimbramon[5592]: 5592:info: 2006-11-16 00:38:01, STATUS: venaro.net: mailbox: Stopped
Nov 16 00:38:04 zeplin zimbramon[5592]: 5592:info: 2006-11-16 00:38:01, STATUS: venaro.net: mta: Running
Nov 16 00:38:04 zeplin zimbramon[5592]: 5592:info: 2006-11-16 00:38:01, STATUS: venaro.net: snmp: Running
Nov 16 00:38:04 zeplin zimbramon[5592]: 5592:info: 2006-11-16 00:38:01, STATUS: venaro.net: spell: Running
Nov 16 00:38:55 zeplin postfix/anvil[4901]: statistics: max connection rate 1/60s for (smtp:207.7.149.45) at Nov 16 00:35:34
Nov 16 00:38:55 zeplin postfix/anvil[4901]: statistics: max connection count 1 for (smtp:207.7.149.45) at Nov 16 00:35:34
Nov 16 00:38:55 zeplin postfix/anvil[4901]: statistics: max cache size 1 at Nov 16 00:35:34
Nov 16 00:40:01 zeplin zimbramon[5956]: 5956:info: 2006-11-16 00:40:01, QUEUE: 2179 23
Nov 16 00:40:03 zeplin zimbramon[5961]: 5961:info: 2006-11-16 00:40:01, DISK: venaro.net: dev: /dev/mapper/VolGroup01-LogVol01, mp: /, tot: 6031, avail: 10%
Nov 16 00:40:03 zeplin zimbramon[5961]: 5961:info: 2006-11-16 00:40:01, DISK: venaro.net: dev: /dev/sdc1, mp: /boot, tot: 99, avail: 80
Nov 16 00:40:03 zeplin zimbramon[5961]: 5961:info: 2006-11-16 00:40:01, DISK: venaro.net: dev: /dev/mapper/VolGroup00-LogVol00, mp: /flats, tot: 180475, avail: 69%
<snip>



# catalina.out
>snip<
Nov 16, 2006 12:34:49 AM org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-443
Nov 16, 2006 12:34:49 AM org.apache.coyote.http11.Http11BaseProtocol init
INFO: Initializing Coyote HTTP/1.1 on http-7071
Nov 16, 2006 12:34:49 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 1664 ms
Nov 16, 2006 12:34:49 AM org.apache.catalina.core.StandardService start
INFO: Starting service Catalina
Nov 16, 2006 12:34:49 AM org.apache.catalina.core.StandardEngine start
INFO: Starting Servlet Engine: Apache Tomcat/5.5.15
Nov 16, 2006 12:34:49 AM org.apache.catalina.core.StandardHost start
INFO: XML validation disabled
Nov 16, 2006 12:34:50 AM org.apache.catalina.startup.HostConfig deployDescriptor
WARNING: A docBase /opt/zimbra/apache-tomcat-5.5.15/webapps/zimbra inside the host appBase has been specified, and will be ignored
log4j:WARN No appenders could be found for logger (org.apache.catalina.session.ManagerBase).
log4j:WARN Please initialize the log4j system properly.
Zimbra server reserving server socket port=110 bindaddr=192.168.11.3 ssl=false
Fatal error: terminating: PrivilegedServlet init failed
java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSock etChannelImpl.java:119)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAd aptor.java:59)
at com.zimbra.cs.util.NetUtil.newBoundServerSocket(Ne tUtil.java:85)
at com.zimbra.cs.util.NetUtil.reserveServerSocket(Net Util.java:108)
at com.zimbra.cs.servlet.PrivilegedServlet.init(Privi legedServlet.java:91)
at javax.servlet.GenericServlet.init(GenericServlet.j ava:211)
at org.apache.catalina.core.StandardWrapper.loadServl et(StandardWrapper.java:1105)
at org.apache.catalina.core.StandardWrapper.load(Stan dardWrapper.java:932)
at org.apache.catalina.core.StandardContext.loadOnSta rtup(StandardContext.java:3915)
at org.apache.catalina.core.StandardContext.start(Sta ndardContext.java:4176)
at org.apache.catalina.core.ContainerBase.addChildInt ernal(ContainerBase.java:759)
at org.apache.catalina.core.ContainerBase.addChild(Co ntainerBase.java:739)
at org.apache.catalina.core.StandardHost.addChild(Sta ndardHost.java:524)
at org.apache.catalina.startup.HostConfig.deployDirec tory(HostConfig.java:910)
at org.apache.catalina.startup.HostConfig.deployDirec tories(HostConfig.java:873)
at org.apache.catalina.startup.HostConfig.deployApps( HostConfig.java:474)
<snip>

[kirme3]

When you changed it's IP address, did you update your hosts file? DNS?

[venaro]

absolutley. Everything was working great until a recent power failure.

At this point I have copied off /opt/zimbra to /backup/ and have done a complete reinstall. Everything is up and running as it should be, but now I am challenged with how to get all that user data and messages from the old copy to the new install.

Ideas?

[mcevoys]

Sounds like the same process as the recent migration I did:

Install a clean copy of zimbra
Copy your entire backed-up /opt/zimbra folder to the clean copy
Rerun install and do an upgrade procedure

All should be well.

[venaro]

I have tried the proceedure you mention below, and the problem with a misconfiugred tomcat turns backup. the catalinia.out once again showed that zimbra server was trying to reserve the old IP and port 110, which is not a valid ip on in the new environment.

question. should I be copying the entire original /opt/zimbra back as root (which is what I did) and secondly, what are the recommeded cp args? I used cp -rfv



Thanks

Originally Posted by mcevoys

Sounds like the same process as the recent migration I did:

Install a clean copy of zimbra
Copy your entire backed-up /opt/zimbra folder to the clean copy
Rerun install and do an upgrade procedure

All should be well.

[venaro]

Is there just no way to do this or are there really no modorators or zimbra staff available to help out with this?

[kirme3]

Instead of copying the ENTIRE structure back...try everything but /opt/zimbra/tomcat or /opt/zimbra/conf...since this error is in catalina.out. I believe as long as the User Data is there, you should be able to pull the different configs from each piece.

The user data is generally in mysql, db, store, and openldap.

I'm not too sure if anywhere else.

Definitely have a backup of things first, though.

[venaro]

Thanks for the reply.

I have tried copying back, only the /store /index and /db directories but that causes the tomcat error again.

[dkarp]

Originally Posted by venaro

Zimbra server reserving server socket port=110 bindaddr=192.168.11.3 ssl=false
Fatal error: terminating: PrivilegedServlet init failed
java.net.BindException: Cannot assign requested address

My bet is that you've still got zimbraPop3BindAddress set to the old IP. You should also check zimbraPop3SSLBindAddress, zimbraImapBindAddress, and zimbraImapSSLBindAddress.

[venaro]

Originally Posted by dkarp

My bet is that you've still got zimbraPop3BindAddress set to the old IP. You should also check zimbraPop3SSLBindAddress, zimbraImapBindAddress, and zimbraImapSSLBindAddress.

Sweet. What file are those configurations in?

Thanks