Page 1 of 2 12 LastLast
Results 1 to 10 of 14

Thread: HSM Failure

  1. #1
    pereljon's Avatar
    pereljon is offline Junior Member
    Join Date
    Jul 2007
    Posts
    6
    Rep Power
    8

    Default HSM Failure

    We just tried to run HSM for the first time, however after getting about 1/2 way done the session failed with the following error.

    com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=11, message=43559, path=/Volumes/Storage_Drive/store/0/11/msg/10/43559-50835.msg} to volume 3 ExceptionId:BlobMover:1220947696108:00204c89752c5a cb Code:service.FAILURE

    Has anyone seen this error? Any suggestions?
    Attached Images Attached Images

  2. #2
    stuartg.orion is offline Senior Member
    Join Date
    May 2007
    Location
    Auckland
    Posts
    68
    Rep Power
    8

    Default

    We've just had pretty much the same error on our very first HSM session.


    com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=128, message=8560, path=/opt/zimbra/store/0/128/msg/2/8560-33056.msg} to volume 3 ExceptionId:BlobMover:1223928818440:c04111e2fbe74a 37 Code:service.FAILURE

    Anyone know what this means? Do we have to run a repair tool on a mmailbox?

  3. #3
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    Anything interesting about that blob's contents? (I presume it's there/pretty sure we handle missing blob errors much more gracefully.)

    His was halfway through, so we can probably rule out write permissions at the HSM volume (I think we throw another error for that anyways/not going off to check at the moment).
    Last edited by mmorse; 10-13-2008 at 09:37 PM.

  4. #4
    mmorse's Avatar
    mmorse is offline Moderator
    Join Date
    May 2006
    Location
    USA
    Posts
    6,242
    Rep Power
    21

    Default

    Could you also post the full stack from the mailbox.log? Thanks.

  5. #5
    stuartg.orion is offline Senior Member
    Join Date
    May 2007
    Location
    Auckland
    Posts
    68
    Rep Power
    8

    Default

    I can't see any thing strange about the BLOB. Is there a tool to get info on a BLOB file?

    The stack trace:

    2008-10-14 09:13:38,510 INFO [BlobMover] [name=larrys@orionhealth.com;] hsm - Finished moving blobs for all mailboxes.
    2008-10-14 09:13:38,510 ERROR [BlobMover] [name=larrys@orionhealth.com;] hsm - An error occurred while running BlobMover
    com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=128, message=8560, path=/opt/zimbra/store/0/128/msg/2/8560-33056.msg} to volume 3
    ExceptionId:BlobMover:1223928818440:c04111e2fbe74a 37
    Code:service.FAILURE
    at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:253)
    at com.zimbra.cs.hsm.BlobMover.moveBlobs(BlobMover.ja va:220)
    at com.zimbra.cs.hsm.BlobMover.moveBlobs(BlobMover.ja va:125)
    at com.zimbra.cs.hsm.BlobMoverThread.run(BlobMoverThr ead.java:29)
    at java.lang.Thread.run(Thread.java:595)
    Caused by: java.io.IOException: link(/opt/zimbra/store_hsm/0/451/msg/0/1497-2891.msg, /opt/zimbra/store_hsm/0/128/msg/2/8560-33056.msg): No such file or directory
    at com.zimbra.znative.IO.link0(Native Method)
    at com.zimbra.znative.IO.link(IO.java:88)
    at com.zimbra.cs.store.FileBlobStore.link(FileBlobSto re.java:253)
    at com.zimbra.cs.hsm.BlobMover.moveBlobs(BlobMover.ja va:213)
    ... 3 more

  6. #6
    stuartg.orion is offline Senior Member
    Join Date
    May 2007
    Location
    Auckland
    Posts
    68
    Rep Power
    8

    Default

    Ok so reading the stacktrace now I see that the blob this blob is a copy of isn't where it should be.

  7. #7
    bjared's Avatar
    bjared is offline Senior Member
    Join Date
    Jun 2006
    Location
    Indianapolis, IN
    Posts
    51
    Rep Power
    9

    Default

    ...and the fix for that is... ??? I don't understand what it's doing with the files in the link() statement. In my error, neither parameter of the link() function exist:

    ls: /opt/zimbra/hsm01/0/173/msg/0/2396-13956.msg: No such file or directory
    ls: /opt/zimbra/hsm01/0/174/msg/0/2523-11289.msg: No such file or directory

    And the second one's /opt/zimbra/store/0 equivilant exists, but not the first one. Why would it complain about files not existing on the HSM volume unless it's moving blobs then verifying they've moved. If the first one doesn't exist on the zimbra store, then...where did it go? I don't understand.

  8. #8
    Rich Graves is offline Outstanding Member
    Join Date
    Jan 2007
    Location
    Minnesota
    Posts
    719
    Rep Power
    9

    Default

    Where I'm running into this, the blobs are shared -- copied to more than one user, and single-instance-stored.

    Note the link count of 8, second column of ls -l.

    Code:
    An error occurred: com.zimbra.common.service.ServiceException: system failure: Unable to copy MailboxBlob: {mailbox=1001, message=59060, path=/opt/zimbra/store5/0/1001/msg/14/59060-241000.msg} to volume 7 ExceptionId:BlobMover:1249663883484:fabfd081fa2bfc84 Code:service.FAILURE
    Moved 400808 blobs dated earlier than Sun May 24 07:45:55 CDT 2009
      to volume 7.
    
    [zimbra@mail2 ~]$ ls -l /opt/zimbra/store5/0/1001/msg/14/59060-241000.msg
    -rw-r-----  8 zimbra zimbra 13145 May 23 22:18 /opt/zimbra/store5/0/1001/msg/14/59060-241000.msg

  9. #9
    Rich Graves is offline Outstanding Member
    Join Date
    Jan 2007
    Location
    Minnesota
    Posts
    719
    Rep Power
    9

    Default

    ...of course, among the 400,808 blobs that were moved successfully, surely some had multiple links.

  10. #10
    Klug's Avatar
    Klug is offline Moderator
    Join Date
    Mar 2006
    Location
    Beaucaire, France
    Posts
    2,322
    Rep Power
    13

    Default

    It just happened on a customer's server.

    The fix (at least the fix that worked for us) is quite simple:
    • look into /opt/zimbra/log/mailbox.log to get the full error message (not just the one zmhsm prints)
    • in the full error message, you can see the account that has the problem (while in the zmhsm error message you only see the mailbox number): in stuartg.orion's post above the account with the problem is "larrys@orionhealth.com")
    • reindex this account (CLI or admin web interface)
    • restart HSM session (CLi or admin web interface)


    CLI command to reindex is:
    Code:
    su - zimbra
    zmprov rim user@domain.tld start
    CLI command to restart HSM session is:
    Code:
    su - zimbra
    zmhsm -t
    CLI command to display HSM status is:
    Code:
    su - zimbra
    zmhsm -u
    Last edited by Klug; 09-21-2009 at 06:39 AM.

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Installation zimbra Initializing ldap...failed. (28416)
    By farrukh.nadeem in forum Installation
    Replies: 10
    Last Post: 08-14-2009, 06:52 AM
  2. Replies: 8
    Last Post: 10-19-2008, 09:52 AM
  3. Replies: 4
    Last Post: 08-04-2008, 06:17 PM
  4. Error loading on Mac OS X 10.4.10 server PPC
    By qprcanada in forum Installation
    Replies: 7
    Last Post: 10-26-2007, 06:25 AM
  5. system failure: getDirectContext
    By avisser in forum Installation
    Replies: 3
    Last Post: 10-12-2005, 05:32 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •