Missing Blob Error

[segleaur]

So I've searched around to see if other people have had an issue with the infamous "missing blob" error, and it seems that I'm up a creek without a paddle.

I've had some reliablity issues lately with my server and I'm guessing that at somepoint (about a week or so ago), all the older emails (I migrated to Zimbra about 2 weeks ago) that were working are now inaccessible. I can see them in the message list, but it seems that either the database or the index cannot find them. I am using the OSS version 4.0.4 on Mac OS X (PPC variant). And I'm now getting this error:

msg - system failure: missing blob for id: 1203, change: 1124
com.zimbra.cs.service.ServiceException: system failure: missing blob for id: 1203, change: 1124
	at com.zimbra.cs.service.ServiceException.FAILURE(ServiceException.java:174)
	at com.zimbra.cs.mailbox.MailItem.getBlob(MailItem.java:632)
	at com.zimbra.cs.mailbox.MessageCache.fetchFromStore(MessageCache.java:213)
	at com.zimbra.cs.mailbox.MessageCache.getMimeMessage(MessageCache.java:179)
	at com.zimbra.cs.mailbox.Message.getMimeMessage(Message.java:270)
	at com.zimbra.cs.service.mail.ToXML.encodeMessageAsMP(ToXML.java:532)
	at com.zimbra.cs.service.mail.Search.addMessageHit(Search.java:280)
	at com.zimbra.cs.service.mail.SearchConv.putHits(SearchConv.java:175)
	at com.zimbra.cs.service.mail.SearchConv.handle(SearchConv.java:101)
	at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEngine.java:261)
	at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:162)
	at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.java:84)
	at com.zimbra.soap.SoapServlet.doPost(SoapServlet.java:223)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
	at com.zimbra.cs.servlet.ZimbraServlet.service(ZimbraServlet.java:173)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
	at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
	at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:667)
	at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
	at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
	at java.lang.Thread.run(Thread.java:613)

code - service.FAILURE
method - ZmCsfeCommand.prototype.invoke
detail - soap:Receiver
line - 55149
sourceURL - http://webmail.mechanus.org/zimbra/js/ZimbraMail_all.js.zgz?v=061115183750

Now, as far as I understand, everytime I restart the Zimbra service, it should reindex those emails, but it seems that that's not happening. Any way to force that to happen? Have I just lost all my emails?

I'm now migrating to a new server, so when I run imapsync, it moves all the pointers (for a lack of a better word) to the new server, but no content - so all those inaccessible emails are now from unknown senders, with no subjects and 0KB.

So, my worry (beyond the fact that the original Zimbra installation cannot "see" those emails) is that IF I do a two step migration (migrating again the old mail from Cyrus to the new Zimbra instance and then whatever the Old Zimbra installation really can grok to the new Zimbra instance) is what kind of junk will i get? Will i have to weed out all the Unknown Sender/No Subject/0KB emails?

It's not a permissions problem (at least not one I can find) - store/ and index/ both belong to zimbra, as well as all the underlying directories.

Any suggestions?

[dkarp]

msg - system failure: missing blob for id: 1203, change: 1124

Zimbra stores data in 3 places: the MySQL DB, the Lucene index, and the actual message files on disk. If you can click on a folder in the web UI and get results, it means MySQL is up and happy. If you open a message and get the error you've listed, it means that the message files on disk aren't where they're supposed to be. And it sounds like it's happened for all your old mail, which makes no sense unless somebody was moving data directories around at some point.

Now, as far as I understand, everytime I restart the Zimbra service, it should reindex those emails, but it seems that that's not happening.

A) Zimbra doesn't reindex all your mail at startup, as that'd make startup take the better part of a day.
B) This is a filesystem issue, not an index issue, so that wouldn't even help.

Is there any chance you can put the old store folder back where it used to be?

So, my worry (beyond the fact that the original Zimbra installation cannot "see" those emails) is that IF I do a two step migration (migrating again the old mail from Cyrus to the new Zimbra instance and then whatever the Old Zimbra installation really can grok to the new Zimbra instance) is what kind of junk will i get? Will i have to weed out all the Unknown Sender/No Subject/0KB emails?

The two-stage migrate should work, but you'll have to weed out the 0-length messages yourself. SEARCH SMALLER 1 in IMAP or searching on "smaller:1" in the Zimbra UI should pick these malformed messages up.

[segleaur]

dkarp - thanks for the heads up. apparently, mac os x (i'm guessing not 100% sure here) deals with the software mirror with external drives in a strange way. i didn't check to see if the store/ and index/ actually contained data - the directory tree was there, so naturally, i assumed everything was OK. Disk Admin said the array was fine. It seems though, that at some point - one of the drives must of have taken longer to come up, and mac os x zero'ed out the actually files. strange stuff. but thanks for the tip. everything is OK now.