HSM Failure

[pereljon]

We just tried to run HSM for the first time, however after getting about 1/2 way done the session failed with the following error.

com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=11, message=43559, path=/Volumes/Storage_Drive/store/0/11/msg/10/43559-50835.msg} to volume 3 ExceptionId:BlobMover:1220947696108:00204c89752c5a cb Code:service.FAILURE

Has anyone seen this error? Any suggestions?

[stuartg.orion]

We've just had pretty much the same error on our very first HSM session.


com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=128, message=8560, path=/opt/zimbra/store/0/128/msg/2/8560-33056.msg} to volume 3 ExceptionId:BlobMover:1223928818440:c04111e2fbe74a 37 Code:service.FAILURE

Anyone know what this means? Do we have to run a repair tool on a mmailbox?

[mmorse]

Anything interesting about that blob's contents? (I presume it's there/pretty sure we handle missing blob errors much more gracefully.)

His was halfway through, so we can probably rule out write permissions at the HSM volume (I think we throw another error for that anyways/not going off to check at the moment).

[mmorse]

Could you also post the full stack from the mailbox.log? Thanks.

[stuartg.orion]

I can't see any thing strange about the BLOB. Is there a tool to get info on a BLOB file?

The stack trace:

2008-10-14 09:13:38,510 INFO [BlobMover] [name=larrys@orionhealth.com;] hsm - Finished moving blobs for all mailboxes.
2008-10-14 09:13:38,510 ERROR [BlobMover] [name=larrys@orionhealth.com;] hsm - An error occurred while running BlobMover
com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=128, message=8560, path=/opt/zimbra/store/0/128/msg/2/8560-33056.msg} to volume 3
ExceptionId:BlobMover:1223928818440:c04111e2fbe74a 37
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:253)
at com.zimbra.cs.hsm.BlobMover.moveBlobs(BlobMover.ja va:220)
at com.zimbra.cs.hsm.BlobMover.moveBlobs(BlobMover.ja va:125)
at com.zimbra.cs.hsm.BlobMoverThread.run(BlobMoverThr ead.java:29)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.io.IOException: link(/opt/zimbra/store_hsm/0/451/msg/0/1497-2891.msg, /opt/zimbra/store_hsm/0/128/msg/2/8560-33056.msg): No such file or directory
at com.zimbra.znative.IO.link0(Native Method)
at com.zimbra.znative.IO.link(IO.java:88)
at com.zimbra.cs.store.FileBlobStore.link(FileBlobSto re.java:253)
at com.zimbra.cs.hsm.BlobMover.moveBlobs(BlobMover.ja va:213)
... 3 more

[stuartg.orion]

Ok so reading the stacktrace now I see that the blob this blob is a copy of isn't where it should be.

[bjared]

...and the fix for that is... ??? I don't understand what it's doing with the files in the link() statement. In my error, neither parameter of the link() function exist:

ls: /opt/zimbra/hsm01/0/173/msg/0/2396-13956.msg: No such file or directory
ls: /opt/zimbra/hsm01/0/174/msg/0/2523-11289.msg: No such file or directory

And the second one's /opt/zimbra/store/0 equivilant exists, but not the first one. Why would it complain about files not existing on the HSM volume unless it's moving blobs then verifying they've moved. If the first one doesn't exist on the zimbra store, then...where did it go? I don't understand.

[Rich Graves]

Where I'm running into this, the blobs are shared -- copied to more than one user, and single-instance-stored.

Note the link count of 8, second column of ls -l.

An error occurred: com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=1001, message=59060, path=/opt/zimbra/store5/0/1001/msg/14/59060-241000.msg} to volume 7 ExceptionId:BlobMover:1249663883484:fabfd081fa2bfc84 Code:service.FAILURE
Moved 400808 blobs dated earlier than Sun May 24 07:45:55 CDT 2009
  to volume 7.

[zimbra@mail2 ~]$ ls -l /opt/zimbra/store5/0/1001/msg/14/59060-241000.msg
-rw-r-----  8 zimbra zimbra 13145 May 23 22:18 /opt/zimbra/store5/0/1001/msg/14/59060-241000.msg

[Rich Graves]

...of course, among the 400,808 blobs that were moved successfully, surely some had multiple links.

[Klug]

It just happened on a customer's server.

The fix (at least the fix that worked for us) is quite simple:

• look into /opt/zimbra/log/mailbox.log to get the full error message (not just the one zmhsm prints)
• in the full error message, you can see the account that has the problem (while in the zmhsm error message you only see the mailbox number): in stuartg.orion's post above the account with the problem is "larrys@orionhealth.com")
• reindex this account (CLI or admin web interface)
• restart HSM session (CLi or admin web interface)

CLI command to reindex is:

su - zimbra
zmprov rim user@domain.tld start

CLI command to restart HSM session is:

su - zimbra
zmhsm -t

CLI command to display HSM status is:

su - zimbra
zmhsm -u