Rebooted my server, now everything is gone since easter!

[telefunken]

Hi, I desperatly need your help!

My virtual server has been running a bit slow lately, and I was seeing some high load on the mailserver guest running Zimbra. Therefore I did a proper shutdown of zimbra using zmcontrol stop, and then rebooted. Now that everything is back up, I tried to log in with no avail. I tried again with my old password that I had to change a month, maybe two months ago. The old password worked, and when I login the last mail in my mailbox is from several months ago! I logged in to the admin interface, and I see right away that three domains I added in about the same timespan is gone, and I assume the users on them aswell.

I have no full backup either because the remote server I copy them to (run by my friend) have not been accepting them lately apparently...

So here I am now, with no new emails since february and no working backup...what do I do?!

[bdial]

do you use snapshotting on the vm because it sounds like it was rolled back somehow

[telefunken]

Yes that occured to me to, but I'm using VMware server, and there is no "snapshot manager" or any log. I also checked the /var/log folder (inside the VM), and there is rotated logs there from the last week, so I don't think that's what happened.
Another thing that doesn't add up is the fact that I have mail from one of my friends asking about the password for one of the accounts setup on one of the domains now missing from the server...That means the domain was created that date, but it's not there now...

[phoenix]

Which version of VMware server is this? VMware Server 2 does have a snapshot facility but you only get one snapshot. It's also nota good idea to run VMware Server with a production system, you'd be far better using ESXi if you want a reliable VM.

If you have no full backup of your system or Zimbra then the likelihood is that you mail is gone. What about the ZImbra log files, is there anything in there to indicate what happened? I assume the clock on your system is set correctly (the VM and the host)?

[telefunken]

Yes, I'm aware about the VMware snapshots, what I meant is that you can't see what snapshots there are, since you only can have one at a time. Therefore I checked the logs in /var on the mailserver and if it was a snapshot, there would be logs from february, and then all of a sudden from 26 of may. However this was not the case, there where several logs from last week.

The usage of VMware server was supposed to be a short term solution, but it's still on there since august last year. However I have a new Xenserver that I will be migrating to shortly.
This server is run to support my own, and some friends and a cuple of associations email/domains, so there are no paying customers.
However it's still a real pita for everone, and a bit embarassing for me...

I've browsed the logs, and I can't se anything strange really...The clock was off with maybe 5 minutes, since I never got vmware tools to sync with the host properly.

[phoenix]

So, in the zimbra logs is there anything that would indicate whether they are old logs or current logs? I honestly can't imagine how this could have happened unless there was a problem with your clock. How about the size of the mailboxes, do they still appear to be the same size? Have a look in the /opt/zimbra/store directory and see if the email is still in there, you can probably recover it if it's still there. If you want to find mail for a specific account then read this article: Account mailbox database structure - Zimbra :: Wiki

[telefunken]

-rw-r--r-- 1 syslog adm 190286 2009-05-27 10:44 zimbra.log
-rw-r----- 1 syslog adm 1525419 2009-05-24 06:47 zimbra.log.0
-rw-r--r-- 1 syslog adm 123798 2009-05-27 06:54 zimbra.log.1.gz
-rw-r--r-- 1 syslog adm 138737 2009-05-26 06:38 zimbra.log.2.gz
-rw-r--r-- 1 syslog adm 121122 2009-05-25 06:46 zimbra.log.3.gz
-rw-r----- 1 syslog adm 245 2009-05-24 06:48 zimbra.log.4.gz

This tells me that it probably isn't a "snapshot problem", since there are logs from before the incident.
I can't find mailboxes for the accounts that had been setup with the domains now missing...

[phoenix]

Did you look in the store directory to see if there's any recent 'missing' mail in there?

[telefunken]

I've checked, and no there is no recent mail...

[uxbod]

What about the graphs in the Admin GUI do they show data all the way through the last few months ? Also check /opt/zimbra/log audit.log for any spurious activity.