Problems - post 5.0.4 upgrade

[bradb21]

I've been running 5.0.2 for several months on this current server with no major issues. I upgraded yesterday morning to 5.0.4 and the upgrade was pretty painless (like they've always been). So less than 24 hous after the upgrade I get a call from my wife saying she and her employees can not access their email. I was already at my job, and I don't have remote access to her servers from work, so I said they would have to tough it out until I got out of work. I was getting "internal error" when I tried to access the webmail remotely. I finally talked my wife thought doing a zmcontrol status (all services said running), and then I had her do a zmcontrol stop and start.... this seems to have fixed the problem and everyone can access their mail now.

So now I'm trying to comb the logs looking for what the heck happened, and I get to my /opt/zimbra/log/mailbox.log (which seems to rotate daily)..... well it's 7.1gb in size on my 10 users server. So I'm willing to bet this will point to my problem, but I don't even know what I can do with a log this size. Any ideas???

I'm not really wanting this to happen again, as my server was totally stable prior to my 5.0.4 upgrade and before that when I was using OSS (now a paying customer).

[mmorse]

Well search the logs for error/fatal/failure/severe, but you might check zmmailboxd.out for crash related material - we need some kinda error to work from.

ZCS Version: Zimbra 5.0.4 (Network Edition) on Ubuntu 6.06 LTS Server

I would also apply the fix(s) as in Bug 24897 - update /etc/pam.d/su on ubuntu so ulimits get set properly then note this post: [SOLVED] Too many open files error after increasing zimbra_session_limit_imap

[bradb21]

Originally Posted by mmorse

Well search the logs for error/fatal/failure/severe, but you might check zmmailboxd.out for crash related material - we need some kinda error to work from.
I would also apply the fix(s) as in Bug 24897 - update /etc/pam.d/su on ubuntu so ulimits get set properly then note this post: [SOLVED] Too many open files error after increasing zimbra_session_limit_imap

Yes, this log is just so huge it's tough to work with. I just shutdown Zimbra and I'm trying to gzip up this log so I can move it off the server, and I can play with it more without effecting my server performance. I'll post some as soon as I get this log off the server.

Thanks!

[bradb21]

This log was really much to large for me to deal with. I had to get my server back up so I did some quick greps for the keywords you gave me and piped to a new log file. I canceled the greps so I could get my server back up. I did get some information, but don't know if it will help. I've gone ahead and made the changes from those two links you provided.

This is grep for error (you can see by timestamps this is just a stream of errors)

2008-03-31 06:46:57,720 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,723 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,726 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,728 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,731 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,734 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,737 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,740 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,743 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,747 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,752 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,755 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,758 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,760 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,763 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,766 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,769 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,772 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,779 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,782 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH: 
2008-03-31 06:46:57,784 WARN  [ImapSSLServer-35] [name=tracey@domainname.com.com;mid=10;ip=192.168.11.174;] imap - ignoring error during UID FETCH:

This is a sample of grep for failure. (judging from the size of my exports this made up a larger % of log)

com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1471
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1472
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1473
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1474
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1476
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1477
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1478
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1479
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1480
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1481
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1483
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1484
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1485
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1486
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1487
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1489
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1491
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1492
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1494
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1495
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1496
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 1497

[bradb21]

I went and made the changes, but I don't know how to tell if it did anything (basically just uncommented out that line).

To fix this problem, edit the file: /etc/pam.d/su

You fill find a line that looks similar to the following:

# session required pam_limits.so

You will want to change it so that it reads:

session required pam_limits.so

[mmorse]

su - zimbra
ulimit -n

It does seem pretty similar to the first post in that other thread:

Originally Posted by jeffw

And here's a sample from the mailbox.log:
2008-03-25 17:12:03,935 WARN [ImapSSLServer-10] [name=user@domain.org;mid=3;ip=x.x.x.x;] imap - ignoring error during UID FETCH:
com.zimbra.common.service.ServiceException: system failure: IOException while retrieving content for item 264795
ExceptionId:ImapSSLServer-10:1206479523935:13358c51319891c4
Code:service.FAILURE
at com.zimbra.common.service.ServiceException.FAILURE (ServiceException.java:253)
at com.zimbra.cs.mailbox.MessageCache.getMimeMessage( MessageCache.java:252)
at com.zimbra.cs.mailbox.Message.getMimeMessage(Messa ge.java:284)
at com.zimbra.cs.imap.ImapMessage.getMimeMessage(Imap Message.java:176)
at com.zimbra.cs.imap.ImapHandler.fetch(ImapHandler.j ava:2937)
at com.zimbra.cs.imap.ImapHandler.doFETCH(ImapHandler .java:2788)
at com.zimbra.cs.imap.ImapHandler.executeRequest(Imap Handler.java:402)
at com.zimbra.cs.imap.TcpImapHandler.processCommand(T cpImapHandler.java:151)
at com.zimbra.cs.tcpserver.ProtocolHandler.processCon nection(ProtocolHandler.java:160)
at com.zimbra.cs.tcpserver.ProtocolHandler.run(Protoc olHandler.java:128)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Wo rker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.FileNotFoundException: /opt/zimbra/store/0/3/msg/64/264795-237760.msg (Too many open files)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.jav a:106)
at com.zimbra.cs.store.FileBlobStore.getContent(FileB lobStore.java:371)
at com.zimbra.cs.store.FileBlobStore.getContent(FileB lobStore.java:365)
at com.zimbra.cs.mailbox.MessageCache.fetchFromStore( MessageCache.java:270)
at com.zimbra.cs.mailbox.MessageCache.getMimeMessage( MessageCache.java:214)
... 10 more

[mmorse]

Any "Too many open files" references?

[bradb21]

[quote=mmorse;85028]su - zimbra
ulimit -n

This is my result

root@mailhost:~# su - zimbra
zimbra@mailhost:~$ ulimit -n
524288
zimbra@mailhost:~$

I was not able to get the log off the server, and had to whack it. I made the change to that file, but it looks like when I do the ulimit... that is probably not the output you were expecting???

[mmorse]

That's the correct ulimit (remember you corrected /etc/pam.d/su and/or /etc/pam.d/common-session)

[bradb21]

Just verifying.....

I sent and changed "/etc/pam.d/su".... by basically uncommented out the line saying: session required pam_limits.so

used to read:
#session required pam_limits.so

now reads:
session required pam_limits.so