Recovery of tgz mail file fails

[the_griz]

I am trying to recover a very large archived 28GB mail file created with version ZCS 5.0.23. I am now at 7.0.0.

I have run this command before and had success, but not now.

$ zmmailbox -z -v -m test@mail.myserver.com postRestURL "//?fmt=tgz&resolve=reset" 2009archivemail.tgz

ERROR: zclient.IO_ERROR (Broken pipe) (cause: java.net.SocketException Broken pipe)
com.zimbra.common.zclient.ZClientException: Broken pipe
ExceptionId:main:1299865829541:0969031b97372ea6
Code:zclient.IO_ERROR
at com.zimbra.common.zclient.ZClientException.IO_ERRO R(ZClientException.java:45)
at com.zimbra.cs.zclient.ZMailbox.postRESTResource(ZM ailbox.java:2499)
at com.zimbra.cs.zclient.ZMailboxUtil.doPostRestURL(Z MailboxUtil.java:2794)
at com.zimbra.cs.zclient.ZMailboxUtil.execute(ZMailbo xUtil.java:1137)
at com.zimbra.cs.zclient.ZMailboxUtil.main(ZMailboxUt il.java:2675)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutp utStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStre am.java:136)
at com.sun.net.ssl.internal.ssl.OutputRecord.writeBuf fer(OutputRecord.java:297)
at com.sun.net.ssl.internal.ssl.OutputRecord.write(Ou tputRecord.java:286)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRe cordInternal(SSLSocketImpl.java:743)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRe cord(SSLSocketImpl.java:731)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write (AppOutputStream.java:59)
at java.io.BufferedOutputStream.write(BufferedOutputS tream.java:105)
at org.apache.commons.httpclient.methods.InputStreamR equestEntity.writeRequest(InputStreamRequestEntity .java:171)
at org.apache.commons.httpclient.methods.EntityEnclos ingMethod.writeRequestBody(EntityEnclosingMethod.j ava:499)
at org.apache.commons.httpclient.HttpMethodBase.write Request(HttpMethodBase.java:2114)
at org.apache.commons.httpclient.HttpMethodBase.execu te(HttpMethodBase.java:1096)
at org.apache.commons.httpclient.HttpMethodDirector.e xecuteWithRetry(HttpMethodDirector.java:398)
at org.apache.commons.httpclient.HttpMethodDirector.e xecuteMethod(HttpMethodDirector.java:171)
at org.apache.commons.httpclient.HttpClient.executeMe thod(HttpClient.java:397)
at com.zimbra.common.httpclient.HttpClientUtil.execut eMethod(HttpClientUtil.java:42)
at com.zimbra.common.httpclient.HttpClientUtil.execut eMethod(HttpClientUtil.java:38)
at com.zimbra.cs.zclient.ZMailbox.postRESTResource(ZM ailbox.java:2491)
... 3 more

Any suggested ideas on how to recover this mail file or send it to zimbra a different way? I am going to try some stuff in AJCody's wiki page on migration (Ajcody-Migration-Notes - Zimbra :: Wiki) but thought I would ask in case anyone has resolved this in the past.

Thanks!
Tom

[tdslot]

Hello,

Try disable Attachent indexing in Administration console on Class of Service in Advanced tab. The same problem I have. And upgrade to 7.0.1

Tadas

[the_griz]

Thanks for the advice.

Upgraded to 7.0.1 and unchecked Attachment Indexing.

While the import ran longer, the same "Broken Pipe" error comes out.

For immediacy I came up with a work-around. I untarred the tgz file and ran a script that sequentially ran through each resulting folder and .eml file and sent them via zmlmtpinject into the desired account. The basic command I used was:

e.g. If the command recovers archive msgs in Inbox!540.....

/opt/zimbra/bin/zmlmtpinject /path/to/archive/Inbox\!540/*.eml -r recovery-account\@myserver.com

This worked well, but took hours to run.

If anyone knows how this can be done with the original command:
$ zmmailbox -z -v -m test@mail.myserver.com postRestURL "//?fmt=tgz&resolve=reset" 2009archivemail.tgz

.... please let me know.

Thanks!
Tom

[tdslot]

Did you restart server?

su - zimbra -c 'zmcontrol restart'

[the_griz]

Server was restarted with the upgrade. As well as twice thereafter.

Problem still occurs when passing a large .tgz file.

Thanks!
Tom

[carlswart]

I ran into the same problems trying to import my 5.1GB tgz file.

It seems that if you change the resolve= parameter of the command to 'skip' or you just simply remove the &resolve=reset portion of the command that the import will continue from where the error occurred.

The resolve= parameter has several options:

* “skip” ignores duplicates of old items, it’s also the default conflict-resolution.
* “modify” changes old items.
* “reset” will delete the old sub folder (or entire mailbox if /).
* “replace” will delete and re-enter them.

» Zimbra :: Blog

[the_griz]

Here is the solution I came up with recovering large archived TGZ files:

1. Create an account to recover the TGZ file to in ZCS.

2. Extract the TGZ file to a directory
- tar xvfz myarchive.tgz

3. Chown the entire directory to zimbra. e.g. chown -R zimbra.zimbra <directory>

Note: The structure of the extracted data is <Folder Name>!<Incremented Number>. In my case, I had all mail in the Inbox of this archived account. The TGZ was 44GB. The tar had Inbox!1 through Inbox!1052. You will have to recover each folder and its incremented subfolder into its specific folder in Zimbra.

Note 2: The contents of each subfolder was each message in eml format and a metadata file. The file names were constructed <sequential number>-<subject text>.eml and <sequential number>-<subject text>.meta. This causes an issue.

3. Some subject lines have non-ascii characters, spaces, apostrophes, etc. that confuse zmmailbox when it imports them. We need to get rid of them. The best tool I found that worked this issue well was ren-regexp.pl which I downloaded from https://github.com/msabramo/ren-rege.../ren-regexp.pl. It worked great.

Note: In reality, the filenames of the eml files are can be anything. I just found it easier to keep the sequential leading numbers and clean up the trailing subject-based filenames.

4. Here's the script I ran over the Inbox files. It's not elegant but it works well to import the email message by message. Run this as zimbra user.

Specific to my case: I ran this over my Inbox!1 -> Inbox!1052. Substitute folder names as appropriate.

==================================================

#!/bin/bash

for i in {1..1052};

do
echo "******************************";
echo "* Inbox!$i *";
echo "******************************";

cd <path to folders>/Inbox\!$i

#GET RID OF
ren-regexp.pl -q "s/\'//g" "s/ /_/g" "s/\!//g" "s/[^[:ascii:]]/_/g"

#REMOVE UNNECESSARY METADATA FILES
rm -f *.meta

#IMPORT THEM INTO YOUR NEW ACCOUNT AND INTO FOLDER /Inbox
zmmailbox -z -m <account>@<yourmailserver.com> addMessage /Inbox *.eml;

done
==================================================

5. I hope this helps. It took about 22 hrs to re-import a 40GB TGZ (~580K messages).