Page 1 of 3 123 LastLast
Results 1 to 10 of 22

Thread: Backups to NAS via CIFS Failing Intermittently

  1. #1
    LMStone's Avatar
    LMStone is offline Moderator
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,366
    Rep Power
    10

    Default Backups to NAS via CIFS Failing Intermittently

    Good Morning,

    We are seeing intermittent errors with backups (we are testing using a Buffalo NAS device).

    The two types of errors affecting perhaps 10% of the accounts in a system wide backup are "bad file descriptor" and "unable to remove /opt/zimbra/backup/tmp/..." (though there is nothing in ~/backup/tmp).

    Accounts in the Admin UI displaying this errors are not restorable. Other accounts restore fine.

    Not sure where to start debugging, so I thought I'd post here!

    Our setup is that we have Buffalo NAS device on the same subnet exposing a Samba share. We mount the share on the Zimbra box and then create a symlink to the mount point from /opt/zimbra/backup, after mv-ing the existing /opt/zimbra/backup somewhere else.

    I looked through previous posts here with "bad file descriptor" content, and didn't see anything that applied directly.

    I grepped zimbra.log for "btpool" and nothing looked out of sorts there either.

    We have fsck'd the Buffalo device and still get the same problem.

    We did find this article:
    Java Bad File Descriptor Close Bug

    but I wouldn't know where to begin looking at Zimbra's code, nor even if the backups are being done via Java somehow.

    The network connection between the two devices is about three feet, with known good Cat6 cables and an HP switch whose log is showing no errors.

    Other servers using the NAS have no such problems. No other servers are accessing the NAS device during the Zimbra backup period.

    Any ideas would be appreciated; NAS storage is much cheaper than DASD and makes off-site replication easier as well.

    Thanks!
    Mark

  2. #2
    uxbod's Avatar
    uxbod is offline Moderator
    Join Date
    Nov 2006
    Location
    UK
    Posts
    8,016
    Rep Power
    24

    Default

    Hi Mark,

    are you using the same NIC for both servicing ZCS requests and the NAS backup share ?

    Do you get the same problem if you use NFS ?

  3. #3
    LMStone's Avatar
    LMStone is offline Moderator
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,366
    Rep Power
    10

    Default

    Thanks for the fast reply!

    Quote Originally Posted by uxbod View Post
    Hi Mark,

    are you using the same NIC for both servicing ZCS requests and the NAS backup share ?
    Yes. On-board GB NIC on an HP DL-360G4p. We've run these servers as web servers with more than twenty IPs on a NIC, no problem. What are thinking?


    Do you get the same problem if you use NFS ?
    This NAS does not support NFS; Samba, AppleTalk, HTTP and FTP only.

    Zimbra docs discourage NFS use, and we've found it to be pretty slow in other use cases, so we tend to avoid NFS.

    Ideas?

    All the best,
    Mark

  4. #4
    uxbod's Avatar
    uxbod is offline Moderator
    Join Date
    Nov 2006
    Location
    UK
    Posts
    8,016
    Rep Power
    24

    Default

    Quote Originally Posted by LMStone View Post
    What are thinking?
    Utilisation of the NIC with respect to load. Also, how is the server for memory ? Does anything show up in dmesg ? What about in /var/log/messages ?

  5. #5
    langs's Avatar
    langs is offline Special Member
    Join Date
    Sep 2006
    Location
    Brisbane
    Posts
    132
    Rep Power
    8

    Default

    Quote Originally Posted by LMStone View Post
    Zimbra docs discourage NFS use, and we've found it to be pretty slow in other use cases, so we tend to avoid NFS.
    Out of curiosity where do they say that and why? Doesn't make a lot of sense to me, NFS is well proven and a hell of a lot better then Samba.

    I have all my backups going to a NFS mounted target on my SAN, does 300gb nightly without issue and fast.. don't know where the slow comes into it.

    my bet is your smb setup is having issues, are you using the NE version or the open source? If you are using the open source version with Rsync you will see "bad file descriptor" if you use -a, as it will want full *Nix permissions that a smb mount doesn't handle.
    Last edited by langs; 05-11-2008 at 04:42 PM.

  6. #6
    Rich Graves is offline Outstanding Member
    Join Date
    Jan 2007
    Location
    Minnesota
    Posts
    718
    Rep Power
    9

    Default

    There are two major branches of code for smb/cifs mounts on Linux. Are you using smbfs or cifs? cifs is almost certainly better, but whichever it is, try the other.

    There are several cifs tuning options... try turning off opslocks and directio, since they have historically been buggy and likely won't gain you anything with a single client doing sequential I/O.

    You probably need to live with the hardware you've got, but for your next purchase, consider iSCSI. High-end iSCSI is expensive, but low-end ought to be fine for backup applications.

  7. #7
    LMStone's Avatar
    LMStone is offline Moderator
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,366
    Rep Power
    10

    Default

    Quote Originally Posted by langs View Post
    Out of curiosity where do they say that and why? Doesn't make a lot of sense to me, NFS is well proven and a hell of a lot better then Samba.
    The install docs list services to turn off/avoid; NFS is one of them.

    Quote Originally Posted by langs View Post
    I have all my backups going to a NFS mounted target on my SAN, does 300gb nightly without issue and fast.. don't know where the slow comes into it.
    We have used and continue to use NFS in a variety of deployments just fine. We just find cifs to be able to transfer larger files (>50MB) faster than NFS.

    Quote Originally Posted by langs View Post
    my bet is your smb setup is having issues, are you using the NE version or the open source? If you are using the open source version with Rsync you will see "bad file descriptor" if you use -a, as it will want full *Nix permissions that a smb mount doesn't handle.
    I suspect you are right. If so, either we'll need to tweak the mount options or we went a little too low end on this particular NAS server.

    But, the java article in my original post led me to believe we ourselves may not be entirely to blame, hence my post to see if anyone has had this issue with Zimbra. Not casting a stone, just asking.

    Thanks for your post!
    Mark

    P.S. We are on NE. Sorry, I don't post the version we are running anymore after we got into the habit of keeping our profile up to date. :-)

  8. #8
    uxbod's Avatar
    uxbod is offline Moderator
    Join Date
    Nov 2006
    Location
    UK
    Posts
    8,016
    Rep Power
    24

    Default

    Quote Originally Posted by LMStone View Post
    But, the java article in my original post led me to believe we ourselves may not be entirely to blame, hence my post to see if anyone has had this issue with Zimbra. Not casting a stone, just asking
    From a quick Google it would appear a lot of people have this issue with CIFS and Samba. I did see the Java one, but also when the machine is under high load it can happen. Certainly a interesting problem

  9. #9
    uxbod's Avatar
    uxbod is offline Moderator
    Join Date
    Nov 2006
    Location
    UK
    Posts
    8,016
    Rep Power
    24

    Default

    Out of interest which model are you trying ? I have been looking at the iSCSI one.

  10. #10
    LMStone's Avatar
    LMStone is offline Moderator
    Join Date
    Sep 2006
    Location
    477 Congress Street | Portland, ME 04101
    Posts
    1,366
    Rep Power
    10

    Default

    Quote Originally Posted by uxbod View Post
    Out of interest which model are you trying ? I have been looking at the iSCSI one.
    Low-end: Linkstation Pro Duo. Ethernet, not iSCSI. For about $310 you get a RAID1 500GB device that does Samba, http, ftp and AppleTalk (heretofore) quite reliably. We've put a number of these out at smaller clients' sites.

    We also like these devices for a simple disaster recovery plan: one of these devices can be set as the backup target of another one. So, for companies with branch offices, you can locate the primary one in the server rack at the main office, and put a second one in the telco closet at a branch office.

    They keep themselves in sync, so if the main office burns down, all the data is in the branch office OK. The sync works well over WAN connections too.

    As the units are not much bigger than the two disk drives inside them, we train clients to try to grab them (if it is safe to do so) when the office fire alarm goes off, even if only for a fire drill.

    Plus, if you need more than 500GB storage, you can plug in external USB hard drives to these devices.

    Where else can you do disk-to-disk-to-(WAN)-disk backups for about $1.25 per GB all in?

    Much cheaper than a pair of Clarions doing SAN replication!

    All the best,
    Mark

Page 1 of 3 123 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. [SOLVED] Backups failing, "unable to read metadata for account"
    By smcgrath1111 in forum Administrators
    Replies: 10
    Last Post: 04-10-2008, 03:15 PM
  2. NAS and CIFS
    By ronnyek in forum Administrators
    Replies: 11
    Last Post: 12-14-2006, 09:17 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •