java.io.FileNotFoundException: ... (Read-only file system)

[dfriestedt]

I'm getting the following error message over and over in mailbox.log. Every day around 2AM zimbra is spitting out this message (and then again every 15 minutes or so...) and then spools up emails and does not deliver. I need to reboot the server to get Zimbra working again. zmcontrol stop then start does not work....

Any thoughts on what is wrong?

2009-01-22 09:17:07,415 WARN  [LmtpServer-485] [name=XXX@XXX.com;mid=13;] lmtp - try again for XXX@XXX.com: exception occurred
java.io.FileNotFoundException: /opt/zimbra/store1/incoming/1232464905537-567.msg (Read-only file system)
        at java.io.FileOutputStream.open(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:131)
        at com.zimbra.cs.store.FileBlobStore.storeIncoming(FileBlobStore.java:137)
        at com.zimbra.cs.store.StoreManager.storeIncoming(StoreManager.java:74)
        at com.zimbra.cs.mailbox.Mailbox.addMessageInternal(Mailbox.java:4891)
        at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.java:4683)
        at com.zimbra.cs.mailbox.Mailbox.addMessage(Mailbox.java:4639)
        at com.zimbra.cs.filter.ZimbraMailAdapter.addMessage(ZimbraMailAdapter.java:414)
        at com.zimbra.cs.filter.ZimbraMailAdapter.doDefaultFiling(ZimbraMailAdapter.java:408)
        at com.zimbra.cs.filter.ZimbraMailAdapter.executeActions(ZimbraMailAdapter.java:266)
        at org.apache.jsieve.SieveFactory.evaluate(SieveFactory.java:159)
        at com.zimbra.cs.filter.RuleManager.applyRules(RuleManager.java:196)
        at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliverMessageToLocalMailboxes(ZimbraLmtpBackend.java:380)
        at com.zimbra.cs.lmtpserver.ZimbraLmtpBackend.deliver(ZimbraLmtpBackend.java:138)
        at com.zimbra.cs.lmtpserver.LmtpHandler.processMessageData(LmtpHandler.java:375)
        at com.zimbra.cs.lmtpserver.TcpLmtpHandler.continueDATA(TcpLmtpHandler.java:67)
        at com.zimbra.cs.lmtpserver.LmtpHandler.doDATA(LmtpHandler.java:364)
        at com.zimbra.cs.lmtpserver.LmtpHandler.processCommand(LmtpHandler.java:174)
        at com.zimbra.cs.lmtpserver.TcpLmtpHandler.processCommand(TcpLmtpHandler.java:61)
        at com.zimbra.cs.tcpserver.ProtocolHandler.processConnection(ProtocolHandler.java:160)
        at com.zimbra.cs.tcpserver.ProtocolHandler.run(ProtocolHandler.java:128)
        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:595)

[uxbod]

Are you performing a backup to a remote system via NFS ? Anything in /var/log/messages around that time ? What version of ZCS are you running

su - zimbra
zmcontrol -v

[bdial]

have you fsck'd the filesystem lately? if there are major problems with the filesystem, it will often cause it to remount read-only.

[dfriestedt]

I'm running:

Release 5.0.11_GA_2695.UBUNTU6_64 UBUNTU6_64 NETWORK edition

Regarding NFS - I'm not using it but I think you are on to something....

I've got a separate RAID 5 array mounted to a /backup directory. Zimbra drops the backups into that directory every night. Then, I have a remote server rsync the data from that /backup to the remote server. I've got that running on crontab and it starts every day at 4AM. I got the java.io.FileNotFoundException: ... (Read-only file system) error at 4:15AM, so shortly after rsync runs. The remote server is on a comcast cable internet connection, but it drops out all the time. So, I wonder if rsync started up, comcast dropped out and then it somehow screwed up my Zimbra server...

I'll run fsck and see what comes back too.

[uxbod]

Read about Wiki :: NFS Issues under disk. You could try using CIFS instead.