[SOLVED] Indexing and subsequent searches causing crashing in 7

[ddebiase]

Zimbra 7 CentOS 5 x64 - 8GB RAM / 2 Core Opteron CPU

Upgraded to Version 7 FOSS this AM from 6.0.10 without issue.

I've since tried to reindex two mailboxes in the admin console - both did not finish and threw errors.. After this I tried searching the mailboxes in the webmail and had these errors:

___________________________
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(Abs tractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStr ingBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java: 119)
at java.lang.StringBuilder.append(StringBuilder.java: 115)
at com.zimbra.common.mime.HeaderUtils$ByteBuilder.app endTo(HeaderUtils.java:212)
at com.zimbra.common.mime.InternetAddress.parse(Inter netAddress.java:450)
at com.zimbra.common.mime.InternetAddress.parse(Inter netAddress.java:493)
at com.zimbra.common.mime.InternetAddress.parse(Inter netAddress.java:323)
at com.zimbra.common.mime.InternetAddress.<init>(Inte rnetAddress.java:68)
at com.zimbra.common.mime.InternetAddress.parseHeader (InternetAddress.java:237)
at com.zimbra.common.mime.InternetAddress.parseHeader (InternetAddress.java:162)
at com.zimbra.cs.index.analysis.RFC822AddressTokenStr eam.<init>(RFC822AddressTokenStream.java:95)
at com.zimbra.cs.mime.ParsedMessage.getMainBodyLucene Document(ParsedMessage.java:834)
at com.zimbra.cs.mime.ParsedMessage.analyzeNonBodyPar ts(ParsedMessage.java:413)
at com.zimbra.cs.mime.ParsedMessage.analyzeFully(Pars edMessage.java:439)
at com.zimbra.cs.mailbox.Message.generateIndexData(Me ssage.java:1078)
at com.zimbra.cs.mailbox.IndexHelper.indexItemList(In dexHelper.java:844)
at com.zimbra.cs.mailbox.IndexHelper.indexDeferredIte msInternal(IndexHelper.java:361)
at com.zimbra.cs.mailbox.IndexHelper.indexDeferredIte ms(IndexHelper.java:278)
at com.zimbra.cs.mailbox.IndexHelper.search(IndexHelp er.java:132)
at com.zimbra.cs.mailbox.Mailbox.search(Mailbox.java: 3743)
at com.zimbra.cs.service.mail.Search.doSearch(Search. java:134)
at com.zimbra.cs.service.mail.Search.handle(Search.ja va:114)
at com.zimbra.soap.SoapEngine.dispatchRequest(SoapEng ine.java:401)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:276)
at com.zimbra.soap.SoapEngine.dispatch(SoapEngine.jav a:157)
at com.zimbra.soap.SoapServlet.doWork(SoapServlet.jav a:294)
at com.zimbra.soap.SoapServlet.doPost(SoapServlet.jav a:215)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:727)
at com.zimbra.cs.servlet.ZimbraServlet.service(Zimbra Servlet.java:208)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:820)

______________________________

At this point I had to zmcontrol restart. Both mailboxes had 30,000+ items. I had a third mailbox index properly and that can be searched without error. That mailbox also had a large number of items.

How can I isolate this problem and remove the offending mail item or configure my system to compensate?

[phoenix]

Sorry, wrong question. Is this a VM or real hardware?

[ddebiase]

It's bare metal

[ddebiase]

Just to update...

Installed 4GB additional of RAM, now at 12GB

changed:


mailboxd_java_heap_new_size_percent = 35
mailboxd_java_heap_size = 5000

Java memory usage is steady for the reindex, then hits the same item and jumps to the max size and shortly crashes.

[FritzBrause]

Could you try and set the logging level to "DEBUG" for lucene:

Log Files - Zimbra :: Wiki

This might help to determine the exact email the indexer is going nuts about. My best guess is that this will be a message with an huge attachment or maybe a password protected word document or the like.

P.S. Don't forget to switch the Log level back to normal.

[ddebiase]

Thanks for the information Andreas! I actually rolled back to 6.0.10 yesterday and the reindex worked well. Then I just upgraded again an hour ago, tried again with the same results. So there is something different in the way 7.0.0 handles this vs. 6.0.10, or so it seems.

I just rolled back again when I saw your message, and now I'm just probably going to wait for 7.0.1. I will keep your instructions handy as your suggestion makes complete sense, and I'm happy to scrap the offending message if this persists. If any case I'll update this and mark as solved (if appropriate) with my results in a few weeks.

Best Regards,

Dan

[phoenix]

Originally Posted by ddebiase

Thanks for the information Andreas! I actually rolled back to 6.0.10 yesterday and the reindex worked well. Then I just upgraded again an hour ago, tried again with the same results.

It would help if you filed a bug report on the problem (do a search first to see if it's an already known bug).

[ddebiase]

Bill,

Done. I'll be happy to help try and track this down for anyone at Zimbra if interested. I just have some limited windows of maintenance for the next few weeks.

Best,

Dan

[phoenix]

Thanks for filing that, I'll just post the link to that bug in case anyone else sees the same problem: Bug 56685 &ndash; Reindex mailbox in 7.0.0 causes crash, corrupts index

[ddebiase]

Turns out I had a malformed mail header that caused the problem. There will be a fix in 7.0.1