Admin panel wrongly reports 1 up and 1 down

[vmagecic]

I set up two mail servers, the first "mail1" is the primary mx, the second "mail11" is the secondary, backup mx. I enabled ldap replication to mail11 and got both servers up and running. Everything seems to be going smoothly without a hitch except for one thing. When I log into the admin panel, mail1 is reported as being down, even though it's not, maill11 is reported as being up and functional.

This is where it gets weird.

I followed these two threads,
[SOLVED] Stats again marked down in Admin console
[SOLVED] Web Admin UI Server Status...offline?

And everything mentioned in both posts I have tried and mail1 is still reported as being down.

I can send mail to and from it with no problems. GetServiceStatusRequest reports mail1 as being down, though.

zimbra@mail11:~/conf/nginx/includes$ zmsoap -z GetServiceStatusRequest
<GetServiceStatusResponse xmlns="urn:zimbraAdmin">
  <timezone id="America/New_York" displayName="Eastern Standard Time"/>
  <status t="1294243680" service="spell" server="mail11.pacpub.com">1</status>
  <status t="1294243680" service="mailbox" server="mail11.pacpub.com">1</status>
  <status t="1294243680" service="logger" server="mail11.pacpub.com">1</status>
  <status t="1294243680" service="mta" server="mail11.pacpub.com">1</status>
  <status t="1294243680" service="imapproxy" server="mail11.pacpub.com">1</status>
  <status t="1294243680" service="antispam" server="mail11.pacpub.com">1</status>
  <status t="1294243680" service="stats" server="mail11.pacpub.com">1</status>
  <status t="1294243680" service="ldap" server="mail11.pacpub.com">1</status>
  <status t="1294243680" service="memcached" server="mail11.pacpub.com">1</status>
  <status t="1294243680" service="antivirus" server="mail11.pacpub.com">1</status>
  <status t="1294243680" service="snmp" server="mail11.pacpub.com">1</status>
  <status t="1294243759" service="antivirus" server="mail1.pacpub.com">0</status>
  <status t="1294243759" service="antispam" server="mail1.pacpub.com">0</status>
  <status t="1294243759" service="logger" server="mail1.pacpub.com">0</status>
  <status t="1294243759" service="mailbox" server="mail1.pacpub.com">0</status>
  <status t="1294243759" service="memcached" server="mail1.pacpub.com">0</status>
  <status t="1294243759" service="mta" server="mail1.pacpub.com">0</status>
  <status t="1294243759" service="stats" server="mail1.pacpub.com">0</status>
  <status t="1294243759" service="imapproxy" server="mail1.pacpub.com">0</status>
  <status t="1294243759" service="snmp" server="mail1.pacpub.com">0</status>
  <status t="1294243759" service="ldap" server="mail1.pacpub.com">0</status>
  <status t="1294243759" service="spell" server="mail1.pacpub.com">0</status>
</GetServiceStatusResponse>

zmcontrol and zmstatctl both report everything is running on both servers.

I thought it might be a hostname issue, so I added each servers hostname to /etc/hosts, this fixed one of the other problems I had, but mail1 is still reported as being down.

zmswatch.out just as the started log and nothing more. `cat zmlogswatch.out | grep STATUS` reports everything running.

And this is how it is on both servers, virtually identical, except for mail11 having ldap replica from mail1.

What am I missing?

EDIT: I found that mail1's zimbraLogHostname was set to mail11 instead of mail1. I changed that and now mail1 is reported as being up, but with "data is stale" next to each service and mail11 is reported as being down.

I'm pretty sure, now, that it's a misconfiguration somewhere. Does anyone have any tips on which settings I should be looking at?

[vmagecic]

Intrestingly, I just took mail11 down and the status' for mail1 started to display correctly. Once mail11 came back up, everything was wrong again.

I'm really thinking is some hostname misconfiguration somewhere, I'm just not sure where.

[vmagecic]

I ended up reinstalling zimbra on both servers and once I finished, mail1 is now reported correctly, however, mail11 is down reported as being down. I did happen to find the following error message when I use zmsoap.

zimbra@mail11:~$ zmsoap -z GetLoggerStatsRequest
[] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused
[] INFO: Retrying request
java.net.ConnectException: Connection refused

But, when I run it from mail1 I get this..

zimbra@mail1:~$ zmsoap -z GetLoggerStatsRequest
<GetLoggerStatsResponse xmlns="urn:zimbraAdmin">
  <hostname hn="mail1.pacpub.com"/>
  <hostname hn="mail11.pacpub.com"/>
</GetLoggerStatsResponse>

Which is right. Any ideas on what might be causing this problem with mail11?