Clock reset - emails missing...

[madcat]

Our Zimbra server had some clock issues (most likely vmware skew issues) causing it to move forward over a month. Now that I reset the clock back the emails are missing! They don't show up in any users GUI. If I "reindex" a mailbox the numbers appear correct but the emails still don't show up. The calendar is missing recent items as well.

Any thoughts on what I need to do?

Thanks!

[madcat]

Another data point...

The /opt/zimbra/data directory only contains no messages between Dec 28 and Feb 20 (today). Does that mean that Zimbra deleted those files when the clock was set back? Sounds like I might need to recover from backup. Has this happened to anyone before?

[MidSpeck]

Originally Posted by madcat

Another data point...

The /opt/zimbra/data directory only contains no messages between Dec 28 and Feb 20 (today). Does that mean that Zimbra deleted those files when the clock was set back? Sounds like I might need to recover from backup. Has this happened to anyone before?

Where you are utilizing VMware, I almost wonder if somehow the machine was reverted to an old snapshot. But I guess that doesn't make a whole lot of sense if you clock was a month ahead rather than a month behind.

I have not experienced this issue though, sorry.

Edit: I just checked on one of my VMware guest operating systems and its clock had skewed in the forward direction, rather than behind. So I guess it's possible for that to happen.

[dkarp]

Originally Posted by madcat

The /opt/zimbra/data directory only contains no messages between Dec 28 and Feb 20 (today). Does that mean that Zimbra deleted those files when the clock was set back? Sounds like I might need to recover from backup. Has this happened to anyone before?

Do you mean the /opt/zimbra/store directory? And wouldn't it make sense that there were no files in that date range -- no messages were deposited while the clock was set to those times, right?

[madcat]

Yes, /opt/zimbra/store (not data), sorry about that.

I pulled a backup image of the machine from the day before and it was in the same state (only files from Dec 28 and earlier in store/). I'm going to grab another image from the last known running state.

But I believe I have found out what happened. We had some issues a few months ago when the there was a power outage and the dbm files were corrupt (LDAP). I removed the lock/tmp files and things sprang back to life. It looks like a system admin did the same thing this weekend to try and recover from this weekends power outage.

Looks like I'll need to pull the new emails over from one image into the other to keep things running. Can I just copy over the store files and re-index?

[dkarp]

Originally Posted by madcat

But I believe I have found out what happened. We had some issues a few months ago when the there was a power outage and the dbm files were corrupt (LDAP). I removed the lock/tmp files and things sprang back to life. It looks like a system admin did the same thing this weekend to try and recover from this weekends power outage.

I'm a bit confused. How would removing the lockfiles cause months of messages to disappear?

Originally Posted by madcat

Looks like I'll need to pull the new emails over from one image into the other to keep things running. Can I just copy over the store files and re-index?

No, as there's also the database to consider. You can use zmlmtpinject or zmmailbox addMessage to do this, though.