Hey All,

I've googling the forums for much of the last 2 days and I've decided to give up and just ask.

I run the zimbra server (among others) at a small video production company, and while I'm savvy enough to get it running, I'm by no means an expert. Two days ago we had a nasty power outage, and when I finally got the servers back on our stand alone zimbra server was giving this message on startup:

Code:
Unable to determine enabled services. Cache is out of date or doesn't exist.
A few searches suggested that it was a DNS issue. Zimbra was working before the crash, but our firewall (with dnsmasq) was acting flaky, so I decided to implement the split DNS setup.

Outputs to standard questions:
Code:
zimbra@mail:/$ dig productivemedia.com mx

; <<>> DiG 9.6-ESV-R4 <<>> productivemedia.com mx
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16727
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 13, ADDITIONAL: 1

;; QUESTION SECTION:
;productivemedia.com.		IN	MX

;; ANSWER SECTION:
productivemedia.com.	900	IN	MX	10 mail.productivemedia.com.
productivemedia.com.	900	IN	MX	20 mail2.productivemedia.com.

;; AUTHORITY SECTION:
.			25807	IN	NS	k.root-servers.net.
.			25807	IN	NS	m.root-servers.net.
.			25807	IN	NS	i.root-servers.net.
.			25807	IN	NS	f.root-servers.net.
.			25807	IN	NS	g.root-servers.net.
.			25807	IN	NS	c.root-servers.net.
.			25807	IN	NS	l.root-servers.net.
.			25807	IN	NS	j.root-servers.net.
.			25807	IN	NS	e.root-servers.net.
.			25807	IN	NS	h.root-servers.net.
.			25807	IN	NS	b.root-servers.net.
.			25807	IN	NS	d.root-servers.net.
.			25807	IN	NS	a.root-servers.net.

;; ADDITIONAL SECTION:
mail.productivemedia.com. 172800 IN	A	192.168.2.13

;; Query time: 47 msec
;; SERVER: 192.168.2.13#53(192.168.2.13)
;; WHEN: Mon Feb 27 15:43:55 2012
;; MSG SIZE  rcvd: 307
The entry for mail2.productivemedia.com is a server I started on EC2 to hold us over.

Code:
zimbra@mail:/$ dig productivemedia.com any

; <<>> DiG 9.6-ESV-R4 <<>> productivemedia.com any
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8748
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 13, ADDITIONAL: 1

;; QUESTION SECTION:
;productivemedia.com.		IN	ANY

;; ANSWER SECTION:
productivemedia.com.	776	IN	MX	20 mail2.productivemedia.com.
productivemedia.com.	776	IN	MX	10 mail.productivemedia.com.

;; AUTHORITY SECTION:
.			25683	IN	NS	b.root-servers.net.
.			25683	IN	NS	c.root-servers.net.
.			25683	IN	NS	e.root-servers.net.
.			25683	IN	NS	g.root-servers.net.
.			25683	IN	NS	a.root-servers.net.
.			25683	IN	NS	m.root-servers.net.
.			25683	IN	NS	h.root-servers.net.
.			25683	IN	NS	j.root-servers.net.
.			25683	IN	NS	l.root-servers.net.
.			25683	IN	NS	d.root-servers.net.
.			25683	IN	NS	k.root-servers.net.
.			25683	IN	NS	i.root-servers.net.
.			25683	IN	NS	f.root-servers.net.

;; ADDITIONAL SECTION:
mail.productivemedia.com. 172800 IN	A	192.168.2.13

;; Query time: 0 msec
;; SERVER: 192.168.2.13#53(192.168.2.13)
;; WHEN: Mon Feb 27 15:45:59 2012
;; MSG SIZE  rcvd: 307
Code:
zimbra@mail:~$ host $(hostname)
mail.productivemedia.com has address 192.168.2.13
mail.productivemedia.com mail is handled by 10 mail.productivemedia.com.
Did I miss something there? The internal address for the server is 192.168.2.13. The external address is actually internet routable.

I had assumed that I had gotten this right so I decided to check the ldap server. After a ton of searching I think I found the right log which read:

Code:
2012-02-25 16:48:42,590 INFO  [main] [] soap - Servlet SoapServlet starting up
2012-02-25 16:48:42,633 INFO  [main] [] soap - Adding service AccountService to SoapServlet
2012-02-25 16:48:42,689 INFO  [main] [] soap - Adding service MailService to SoapServlet
2012-02-25 16:48:42,807 INFO  [main] [] soap - Adding service IMService to SoapServlet
2012-02-25 16:48:42,818 INFO  [main] [] misc - version=6.0.13_GA_2918 release=20110513140800 builddate=20110513-1410 buildhost=zre-debian5.eng.vmware.com
2012-02-25 16:48:42,819 INFO  [main] [] misc - LANG environment is set to: en_US.UTF-8
2012-02-25 16:48:42,819 INFO  [main] [] misc - System property java.home=/opt/zimbra/jdk1.6.0_25/jre
2012-02-25 16:48:42,819 INFO  [main] [] misc - System property java.runtime.version=1.6.0_25-b06
2012-02-25 16:48:42,819 INFO  [main] [] misc - System property java.version=1.6.0_25
2012-02-25 16:48:42,819 INFO  [main] [] misc - System property java.vm.info=mixed mode
2012-02-25 16:48:42,819 INFO  [main] [] misc - System property java.vm.name=Java HotSpot(TM) Server VM
2012-02-25 16:48:42,819 INFO  [main] [] misc - System property java.vm.version=20.0-b11
2012-02-25 16:48:42,819 INFO  [main] [] misc - System property os.arch=i386
2012-02-25 16:48:42,819 INFO  [main] [] misc - System property os.name=Linux
2012-02-25 16:48:42,819 INFO  [main] [] misc - System property os.version=2.6.26-2-686
2012-02-25 16:48:42,819 INFO  [main] [] misc - System property sun.arch.data.model=32
2012-02-25 16:48:42,819 INFO  [main] [] misc - System property sun.cpu.endian=little
2012-02-25 16:48:42,819 INFO  [main] [] misc - System property sun.cpu.isalist=
2012-02-25 16:48:42,819 INFO  [main] [] misc - System property sun.os.patch.level=unknown
2012-02-25 16:48:42,835 INFO  [main] [] system - Setting mysql connector property: maxActive=100
2012-02-25 16:48:42,844 INFO  [main] [] system - Setting mysql connector property: maxActive=100
2012-02-25 16:48:42,851 INFO  [main] [] dbconn - instantiating DB connection factory class com.zimbra.cs.db.ZimbraConnectionFactory
2012-02-25 16:48:43,227 FATAL [main] [] system - Config initialization failed
com.zimbra.common.service.ServiceException: system failure: unable to lookup server by name: mail.productivemedia.com message: [LDAP: error code 49 - Invalid Credentials]
ExceptionId:main:1330206523225:2ddcc87a8945c723
Code:service.FAILURE
	at com.zimbra.common.service.ServiceException.FAILURE(ServiceException.java:248)
	at com.zimbra.cs.account.ldap.LdapProvisioning.getServerByName(LdapProvisioning.java:2749)
	at com.zimbra.cs.account.ldap.LdapProvisioning.getServerByNameInternal(LdapProvisioning.java:2727)
	at com.zimbra.cs.account.ldap.LdapProvisioning.getLocalServer(LdapProvisioning.java:3338)
	at com.zimbra.cs.util.Config.init(Config.java:81)
	at com.zimbra.cs.util.Config.initConfig(Config.java:92)
	at com.zimbra.cs.util.Config.getString(Config.java:104)
	at com.zimbra.cs.db.Versions.checkDBVersion(Versions.java:66)
	at com.zimbra.cs.db.Versions.checkVersions(Versions.java:62)
	at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:162)
	at com.zimbra.cs.util.Zimbra.startup(Zimbra.java:123)
	at com.zimbra.soap.SoapServlet.init(SoapServlet.java:125)
	at javax.servlet.GenericServlet.init(GenericServlet.java:241)
	at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:440)
	at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685)
	at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
	at org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1254)
	at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
	at org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:471)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
	at org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
	at org.mortbay.jetty.handler.DebugHandler.doStart(DebugHandler.java:127)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
	at org.mortbay.jetty.Server.doStart(Server.java:224)
	at org.mortbay.setuid.SetUIDServer.doStart(SetUIDServer.java:158)
	at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
	at org.mortbay.xml.XmlConfiguration.main(XmlConfiguration.java:985)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.mortbay.start.Main.invokeMain(Main.java:194)
	at org.mortbay.start.Main.start(Main.java:534)
	at org.mortbay.start.Main.start(Main.java:441)
	at org.mortbay.start.Main.main(Main.java:119)
Caused by: javax.naming.AuthenticationException: [LDAP: error code 49 - Invalid Credentials]; remaining name 'cn=mail.productivemedia.com,cn=servers,cn=zimbra'
	at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3041)
	at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2987)
	at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2789)
	at com.sun.jndi.ldap.LdapCtx.connect(LdapCtx.java:2703)
	at com.sun.jndi.ldap.LdapCtx.ensureOpen(LdapCtx.java:2602)
	at com.sun.jndi.ldap.LdapCtx.ensureOpen(LdapCtx.java:2576)
	at com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1919)
	at com.sun.jndi.ldap.LdapCtx.doSearchOnce(LdapCtx.java:1911)
	at com.sun.jndi.ldap.LdapCtx.c_getAttributes(LdapCtx.java:1304)
	at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:213)
	at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:121)
	at javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:133)
	at javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:128)
	at com.zimbra.cs.account.ldap.ZimbraLdapContext.getAttributes(ZimbraLdapContext.java:655)
	at com.zimbra.cs.account.ldap.LdapProvisioning.getServerByName(LdapProvisioning.java:2740)
	... 43 more
I thought that might still be a DNS issue, but now I'm thinking that it's actually the system being unable to look up the server name in the ldap database . . . Maybe? Right?

so I checked netstat to see if if Slapd was available to connect to:
Code:
mail:/opt/zimbra/log# netstat -na | grep LISTEN
tcp        0      0 192.168.2.13:389        0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:7306          0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN     
tcp        0      0 192.168.2.13:53         0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:46357           0.0.0.0:*               LISTEN     
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:953           0.0.0.0:*               LISTEN     
tcp6       0      0 :::53                   :::*                    LISTEN     
tcp6       0      0 :::22                   :::*                    LISTEN     
tcp6       0      0 ::1:953                 :::*                    LISTEN     
unix  2      [ ACC ]     STREAM     LISTENING     17326    /opt/zimbra/db/mysql.sock
unix  2      [ ACC ]     STREAM     LISTENING     5851     /var/run/acpid.socket
unix  2      [ ACC ]     STREAM     LISTENING     15428    /opt/zimbra/openldap-2.4.25.3z/var/run/ldapi
I saw the that 389 was bound, so I tried to telnet to it, and there is something there, but I have no idea how to speak ldap, so I bailed out:
Code:
mail:/# telnet 192.168.2.13 389
Trying 192.168.2.13...
Connected to 192.168.2.13.
Escape character is '^]'.
I did download apache directory studio to test the connection outside of zimbra. I got my ldap password:

Code:
zimbra@mail:/root$ zmlocalconfig -s ldap_root_password
ldap_root_password = 0lPEfrT4Q
The rest of my ldap local config settings look like this:
Code:
zimbra@mail:/root$ zmlocalconfig -s | grep ldap
ldap_accesslog_cachefree = 1
ldap_accesslog_cachesize = 10000
ldap_accesslog_checkpoint = 64 5
ldap_accesslog_dncachesize = 0
ldap_accesslog_idlcachesize = 10000
ldap_accesslog_shmkey = 0
ldap_amavis_password = 0lPEfrT4Q
ldap_bind_url = 
ldap_cache_account_maxage = 15
ldap_cache_account_maxsize = 20000
ldap_cache_cos_maxage = 15
ldap_cache_cos_maxsize = 100
ldap_cache_domain_maxage = 15
ldap_cache_domain_maxsize = 100
ldap_cache_external_domain_maxage = 15
ldap_cache_external_domain_maxsize = 2000
ldap_cache_group_maxage = 15
ldap_cache_group_maxsize = 2000
ldap_cache_mime_maxage = 15
ldap_cache_reverseproxylookup_domain_maxage = 15
ldap_cache_reverseproxylookup_domain_maxsize = 100
ldap_cache_reverseproxylookup_server_maxage = 15
ldap_cache_reverseproxylookup_server_maxsize = 100
ldap_cache_right_maxage = 15
ldap_cache_right_maxsize = 100
ldap_cache_server_maxage = 15
ldap_cache_server_maxsize = 100
ldap_cache_timezone_maxsize = 100
ldap_cache_xmppcomponent_maxage = 15
ldap_cache_xmppcomponent_maxsize = 100
ldap_cache_zimlet_maxage = 15
ldap_cache_zimlet_maxsize = 100
ldap_common_loglevel = 49152
ldap_common_require_tls = 0
ldap_common_threads = 8
ldap_common_toolthreads = 1
ldap_common_writetimeout = 0
ldap_connect_pool_debug = false
ldap_connect_pool_initsize = 1
ldap_connect_pool_master = false
ldap_connect_pool_maxsize = 50
ldap_connect_pool_prefsize = 0
ldap_connect_pool_timeout = 120000
ldap_connect_timeout = 30000
ldap_db_cachefree = 1
ldap_db_cachesize = 10000
ldap_db_checkpoint = 64 5
ldap_db_dncachesize = 0
ldap_db_idlcachesize = 10000
ldap_db_shmkey = 0
ldap_deref_aliases = always
ldap_host = mail.productivemedia.com
ldap_is_master = true
ldap_master_url = ldap://mail.productivemedia.com:389
ldap_nginx_password = 0lPEfrT4Q
ldap_overlay_accesslog_logpurge = 01+00:00  00+04:00
ldap_overlay_syncprov_checkpoint = 20 10
ldap_overlay_syncprov_sessionlog = 500
ldap_port = 389
ldap_postfix_password = 0lPEfrT4Q
ldap_read_timeout = 30000
ldap_replication_password = 0lPEfrT4Q
ldap_root_password = 0lPEfrT4Q
ldap_starttls_supported = 1
ldap_url = ldap://mail.productivemedia.com:389
postfix_sender_canonical_maps = proxy:ldap:${zimbra_home}/conf/ldap-scm.cf
postfix_transport_maps = proxy:ldap:${zimbra_home}/conf/ldap-transport.cf
postfix_virtual_alias_domains = proxy:ldap:${zimbra_home}/conf/ldap-vad.cf
postfix_virtual_alias_maps = proxy:ldap:${zimbra_home}/conf/ldap-vam.cf
postfix_virtual_mailbox_domains = proxy:ldap:${zimbra_home}/conf/ldap-vmd.cf
postfix_virtual_mailbox_maps = proxy:ldap:${zimbra_home}/conf/ldap-vmm.cf
zimbra_class_provisioning = com.zimbra.cs.account.ldap.LdapProvisioning
zimbra_ldap_password = 0lPEfrT4Q
zimbra_ldap_user = zimbra
zimbra_ldap_userdn = uid=zimbra,cn=admins,cn=zimbra
zimbra_zmprov_default_to_ldap = false
But when I connect using Apache Directory Studio using: uid=zimbra,cn=admins,cn=zimbra
as the login, and
0lPEfrT4Q
as the password I still get:
Code:
Error while opening connection
 - [LDAP: error code 49 - Invalid Credentials]
Which says its slapd that doesn't agree with my password . . . And I'm stuck.

Any Help? Please?

Thanks In advance!