Backups to NAS via CIFS Failing Intermittently

[LMStone]

Good Morning,

We are seeing intermittent errors with backups (we are testing using a Buffalo NAS device).

The two types of errors affecting perhaps 10% of the accounts in a system wide backup are "bad file descriptor" and "unable to remove /opt/zimbra/backup/tmp/..." (though there is nothing in ~/backup/tmp).

Accounts in the Admin UI displaying this errors are not restorable. Other accounts restore fine.

Not sure where to start debugging, so I thought I'd post here!

Our setup is that we have Buffalo NAS device on the same subnet exposing a Samba share. We mount the share on the Zimbra box and then create a symlink to the mount point from /opt/zimbra/backup, after mv-ing the existing /opt/zimbra/backup somewhere else.

I looked through previous posts here with "bad file descriptor" content, and didn't see anything that applied directly.

I grepped zimbra.log for "btpool" and nothing looked out of sorts there either.

We have fsck'd the Buffalo device and still get the same problem.

We did find this article:
Java Bad File Descriptor Close Bug

but I wouldn't know where to begin looking at Zimbra's code, nor even if the backups are being done via Java somehow.

The network connection between the two devices is about three feet, with known good Cat6 cables and an HP switch whose log is showing no errors.

Other servers using the NAS have no such problems. No other servers are accessing the NAS device during the Zimbra backup period.

Any ideas would be appreciated; NAS storage is much cheaper than DASD and makes off-site replication easier as well.

Thanks!
Mark

[uxbod]

Hi Mark,

are you using the same NIC for both servicing ZCS requests and the NAS backup share ?

Do you get the same problem if you use NFS ?

[LMStone]

Thanks for the fast reply!

Originally Posted by uxbod

Hi Mark,

are you using the same NIC for both servicing ZCS requests and the NAS backup share ?

Yes. On-board GB NIC on an HP DL-360G4p. We've run these servers as web servers with more than twenty IPs on a NIC, no problem. What are thinking?

Do you get the same problem if you use NFS ?

This NAS does not support NFS; Samba, AppleTalk, HTTP and FTP only.

Zimbra docs discourage NFS use, and we've found it to be pretty slow in other use cases, so we tend to avoid NFS.

Ideas?

All the best,
Mark

[uxbod]

Originally Posted by LMStone

What are thinking?

Utilisation of the NIC with respect to load. Also, how is the server for memory ? Does anything show up in dmesg ? What about in /var/log/messages ?

[langs]

Originally Posted by LMStone

Zimbra docs discourage NFS use, and we've found it to be pretty slow in other use cases, so we tend to avoid NFS.

Out of curiosity where do they say that and why? Doesn't make a lot of sense to me, NFS is well proven and a hell of a lot better then Samba.

I have all my backups going to a NFS mounted target on my SAN, does 300gb nightly without issue and fast.. don't know where the slow comes into it.

my bet is your smb setup is having issues, are you using the NE version or the open source? If you are using the open source version with Rsync you will see "bad file descriptor" if you use -a, as it will want full *Nix permissions that a smb mount doesn't handle.

[Rich Graves]

There are two major branches of code for smb/cifs mounts on Linux. Are you using smbfs or cifs? cifs is almost certainly better, but whichever it is, try the other.

There are several cifs tuning options... try turning off opslocks and directio, since they have historically been buggy and likely won't gain you anything with a single client doing sequential I/O.

You probably need to live with the hardware you've got, but for your next purchase, consider iSCSI. High-end iSCSI is expensive, but low-end ought to be fine for backup applications.

[LMStone]

Originally Posted by langs

Out of curiosity where do they say that and why? Doesn't make a lot of sense to me, NFS is well proven and a hell of a lot better then Samba.

The install docs list services to turn off/avoid; NFS is one of them.

Originally Posted by langs

I have all my backups going to a NFS mounted target on my SAN, does 300gb nightly without issue and fast.. don't know where the slow comes into it.

We have used and continue to use NFS in a variety of deployments just fine. We just find cifs to be able to transfer larger files (>50MB) faster than NFS.

Originally Posted by langs

my bet is your smb setup is having issues, are you using the NE version or the open source? If you are using the open source version with Rsync you will see "bad file descriptor" if you use -a, as it will want full *Nix permissions that a smb mount doesn't handle.

I suspect you are right. If so, either we'll need to tweak the mount options or we went a little too low end on this particular NAS server.

But, the java article in my original post led me to believe we ourselves may not be entirely to blame, hence my post to see if anyone has had this issue with Zimbra. Not casting a stone, just asking.

Thanks for your post!
Mark

P.S. We are on NE. Sorry, I don't post the version we are running anymore after we got into the habit of keeping our profile up to date. :-)

[uxbod]

Originally Posted by LMStone

But, the java article in my original post led me to believe we ourselves may not be entirely to blame, hence my post to see if anyone has had this issue with Zimbra. Not casting a stone, just asking

From a quick Google it would appear a lot of people have this issue with CIFS and Samba. I did see the Java one, but also when the machine is under high load it can happen. Certainly a interesting problem

[uxbod]

Out of interest which model are you trying ? I have been looking at the iSCSI one.

[LMStone]

Originally Posted by uxbod

Out of interest which model are you trying ? I have been looking at the iSCSI one.

Low-end: Linkstation Pro Duo. Ethernet, not iSCSI. For about $310 you get a RAID1 500GB device that does Samba, http, ftp and AppleTalk (heretofore) quite reliably. We've put a number of these out at smaller clients' sites.

We also like these devices for a simple disaster recovery plan: one of these devices can be set as the backup target of another one. So, for companies with branch offices, you can locate the primary one in the server rack at the main office, and put a second one in the telco closet at a branch office.

They keep themselves in sync, so if the main office burns down, all the data is in the branch office OK. The sync works well over WAN connections too.

As the units are not much bigger than the two disk drives inside them, we train clients to try to grab them (if it is safe to do so) when the office fire alarm goes off, even if only for a fire drill.

Plus, if you need more than 500GB storage, you can plug in external USB hard drives to these devices.

Where else can you do disk-to-disk-to-(WAN)-disk backups for about $1.25 per GB all in?

Much cheaper than a pair of Clarions doing SAN replication!

All the best,
Mark