Hello, all. We're having a bit of a crisis here just as we are moving from test to production. I am not an experienced mail administrator and am brand new to Zimbra. It seems we have suddenly lost two days worth of data - mail, calendar appointments, even a complete new domain we created. The emails I see in my mail box are emails I deleted. I get NO_SUCH_BLOB errors when I click on them.
We thought all was running smoothly until we looked at our zimbra.log file and realized it was 17GB in size! It was filled with errors such as:
Code:
May 7 22:25:56 zimbra01 zimbramon[2695]: 2695:info: zmmtaconfig: Skipping Global system configuration No data returned.
May 7 22:25:56 zimbra01 zimbramon[2695]: 2695:info: zmmtaconfig: Skipping Configuration for server zimbra01.ssiservices.biz No data returned.
May 7 22:25:56 zimbra01 zimbramon[2695]: 2695:info: zmmtaconfig: Sleeping...Key lookup failed.
May 7 22:25:59 smtp01 zimbramon[2393]: 2393:info: zmmtaconfig: Skipping Global system configuration update.
May 7 22:25:59 smtp01 zimbramon[2393]: 2393:info: zmmtaconfig: gacf ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cause: javax.naming.Commu
May 7 22:26:00 smtp01 zimbramon[2393]: 2393:info: zmmtaconfig: Skipping All Reverse Proxy URLs update.
May 7 22:26:00 smtp01 zimbramon[2393]: 2393:info: zmmtaconfig: Skipping getAllReverseProxyURLs ERROR: service.FAILURE (system failure: ZimbraLdapContext)
May 7 22:26:00 smtp01 zimbramon[2393]: 2393:info: zmmtaconfig: Skipping All Reverse Proxy Backends update.
May 7 22:26:00 smtp01 zimbramon[2393]: 2393:info: zmmtaconfig: Skipping getAllReverseProxyBackends ERROR: service.FAILURE (system failure: ZimbraLdapCont
May 7 22:26:01 smtp01 zimbramon[2393]: 2393:info: zmmtaconfig: Skipping All Memcached Servers update.
May 7 22:26:01 smtp01 zimbramon[2393]: 2393:info: zmmtaconfig: Skipping getAllMemcachedServers ERROR: service.FAILURE (system failure: ZimbraLdapContext)
May 7 22:26:02 zimbra01 zimbramon[2695]: 2695:info: zmmtaconfig: Skipping Local server configuration No data returned.
May 7 22:26:02 zimbra01 zimbramon[2695]: 2695:info: zmmtaconfig: Skipping Global system configuration No data returned.
May 7 22:26:02 zimbra01 zimbramon[2695]: 2695:info: zmmtaconfig: Skipping Configuration for server zimbra01.ssiservices.biz No data returned.
May 7 22:26:02 zimbra01 zimbramon[2695]: 2695:info: zmmtaconfig: Sleeping...Key lookup failed.
May 7 22:26:02 smtp01 zimbramon[2393]: 2393:info: zmmtaconfig: Skipping All MTA Authentication Target URLs update.
May 7 22:26:02 smtp01 zimbramon[2393]: 2393:info: zmmtaconfig: Skipping getAllMtaAuthURLs ERROR: service.FAILURE (system failure: ZimbraLdapContext) (cau
May 7 22:26:03 smtp01 zimbramon[2393]: 2393:info: zmmtaconfig: Skipping Configuration for server smtp01.ssiservices.biz update.
May 7 22:26:03 smtp01 zimbramon[2393]: 2393:info: zmmtaconfig: gs:smtp01.ssiservices.biz ERROR: service.FAILURE (system failure: ZimbraLdapContext) (caus
May 7 22:26:03 smtp01 zimbramon[2393]: 2393:info: zmmtaconfig: Sleeping...Key lookup failed.
May 7 22:26:07 zimbra01 zimbramon[2695]: 2695:info: zmmtaconfig: Skipping Local server configuration No data returned.
May 7 22:26:07 zimbra01 zimbramon[2695]: 2695:info: zmmtaconfig: Skipping Global system configuration No data returned.
May 7 22:26:07 zimbra01 zimbramon[2695]: 2695:info: zmmtaconfig: Skipping Configuration for server zimbra01.ssiservices.biz No data returned.
May 7 22:26:07 zimbra01 zimbramon[2695]: 2695:info: zmmtaconfig: Sleeping...Key lookup failed.
Some searching suggested running zmfixperms. We did that but the errors seemed to persist.
We then noticed the cron log was also very large. It was filled with errors such as:
Code:
May 5 14:12:01 zimbra01 crond[9123]: Cannot make/remove an entry for the specified session
May 5 14:12:01 zimbra01 crond[9123]: CRON (zimbra) ERROR: failed to open PAM security session: Success
May 5 14:12:01 zimbra01 crond[9123]: CRON (zimbra) ERROR: cannot set security context
May 5 14:14:01 zimbra01 crond[12627]: Cannot make/remove an entry for the specified session
May 5 14:14:01 zimbra01 crond[12627]: CRON (zimbra) ERROR: failed to open PAM security session: Success
May 5 14:14:01 zimbra01 crond[12627]: CRON (zimbra) ERROR: cannot set security context
May 5 14:15:01 zimbra01 crond[14336]: Cannot make/remove an entry for the specified session
May 5 14:15:01 zimbra01 crond[14336]: CRON (zimbra) ERROR: failed to open PAM security session: Success
May 5 14:15:01 zimbra01 crond[14336]: CRON (zimbra) ERROR: cannot set security context
May 5 14:16:01 zimbra01 crond[16038]: Cannot make/remove an entry for the specified session
May 5 14:16:01 zimbra01 crond[16038]: CRON (zimbra) ERROR: failed to open PAM security session: Success
May 5 14:16:01 zimbra01 crond[16038]: CRON (zimbra) ERROR: cannot set security context
May 5 14:18:01 zimbra01 crond[18985]: Cannot make/remove an entry for the specified session
May 5 14:18:01 zimbra01 crond[18985]: CRON (zimbra) ERROR: failed to open PAM security session: Success
May 5 14:18:01 zimbra01 crond[18985]: CRON (zimbra) ERROR: cannot set security context
We then listed the zimbra user crontab and ran the commands manually as the zimbra user to see which one was failing. None failed.
We then restarted the server and it did not seem to shut down cleanly. The zimbra service said it failed on shutdown. The server did appear to come up clean. However, when I went into my email, I was shocked to see two days missing! I see such phenomena in the logs but related to migration.
I really do not know where to begin but we need to get these two days worth of data back. We have not yet implemented the archiving feature. We are running GA16 on CentOS 5.3 within a VServer. What do we do? Thanks - John