mobile calender Internal Server Error

[LMStone]

BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD

You're addressing the symptom and not the problem. It's only a matter of time before the cache is cleared, and the compiling fails again
BAD BAD BAD BAD

OK, I hear you John but I replied before I read your post!

I'll turn on debugging and see why compiling is failing. In the meantime, I'll keep my backup "just in case".

I want to return to be in your good graces! :-)

BTW, won't this be fixed in 5.0.5?

All the best,
Mark

[LMStone]

BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD

You're addressing the symptom and not the problem. It's only a matter of time before the cache is cleared, and the compiling fails again
BAD BAD BAD BAD

So John, when does the cache get cleared? Does it get cleared if one does a zmcontrol stop/start?

I ask because we just did that, and none of the .class files in the directory I mentioned were recreated. Earlier today, when we tested a zmmailboxdctl stop/start. it seems some of the class files were recreated. But again, not the displayMessage_tag.class file which as you can see from the directory listing below is still the file we restored from backup earlier today.

We opened a support case earlier today on this BTW, so I'm not trying to "double-dip" as it were. But if you do have some info that could help Jason Bryan, I'm sure the both of us would be appreciative.

Thanks!
Mark

ls -al
total 376
drwxr-x--- 2 zimbra zimbra 4096 Apr 14 15:29 .
drwxr-x--- 11 zimbra zimbra 4096 Apr 14 15:29 ..
-rw-r----- 1 zimbra zimbra 20870 Apr 14 14:12 attachments_tag.class
-rw-r----- 1 zimbra zimbra 11418 Apr 14 14:12 body_tag.class
-rw-r----- 1 zimbra zimbra 63704 Apr 6 21:38 displayMessage_tag.class
-rw-r----- 1 zimbra zimbra 22391 Apr 14 14:12 messageAction_tag$messageAction_tagHelper.class
-rw-r----- 1 zimbra zimbra 49064 Apr 14 14:12 messageAction_tag.class
-rw-r----- 1 zimbra zimbra 7700 Apr 14 14:12 messageIframe_tag.class
-rw-r----- 1 zimbra zimbra 2485 Apr 14 15:29 messageListViewToolbar_tag$messageListViewToolbar_ tagHelper.class
-rw-r----- 1 zimbra zimbra 34692 Apr 14 15:29 messageListViewToolbar_tag.class
-rw-r----- 1 zimbra zimbra 27122 Apr 14 15:29 messageListView_tag$messageListView_tagHelper.clas s
-rw-r----- 1 zimbra zimbra 42421 Apr 14 15:29 messageListView_tag.class
-rw-r----- 1 zimbra zimbra 2433 Apr 14 15:29 messageViewToolbar_tag$messageViewToolbar_tagHelpe r.class
-rw-r----- 1 zimbra zimbra 27670 Apr 14 15:29 messageViewToolbar_tag.class
-rw-r----- 1 zimbra zimbra 14155 Apr 14 15:29 messageView_tag$messageView_tagHelper.class
-rw-r----- 1 zimbra zimbra 23435 Apr 14 15:29 messageView_tag.class
zimbra@viognier:~/jetty/work/zimbra/jsp/org/apache/jsp/tag/web/message>

[Rich Graves]

I just did the same thing to myself, wiping the jetty/work/ cache... fortunately on a 5.0.5 test system, not our production 5.0.4. So this is a learning opportunity.

What should I do?

I notice that /zimbra/h/calendar, /zimbra/m/mocalendar, /zimbra/m/main, and /zimbra/m/main?action=compose work. /zimbra/m/mosearch, /zimbra/h/search, and /zimbra/h/search?action=compose are broken.

2008-04-23 12:11:41,547 WARN  [btpool0-11] [] log - /zimbra/h/search: java.io.FileNotFoundException: no such file: /opt/zimbra/jetty-6.1.5/work/zimbra/jsp/org/apache/jsp/tag/web/message/displayMessage_tag.class
java.io.FileNotFoundException: no such file: /opt/zimbra/jetty-6.1.5/work/zimbra/jsp/org/apache/jsp/tag/web/message/displayMessage_tag.class
2008-04-23 12:13:16,144 WARN  [btpool0-13] [] log - /zimbra/h/search: java.io.FileNotFoundException: no such file: /opt/zimbra/jetty-6.1.5/work/zimbra/jsp/org/apache/jsp/tag/web/message/displayMessage_tag.class
java.io.FileNotFoundException: no such file: /opt/zimbra/jetty-6.1.5/work/zimbra/jsp/org/apache/jsp/tag/web/mobile/moDisplayMessage_tag.class
2008-04-23 12:15:11,851 WARN  [btpool0-9] [] log - /zimbra/m/mosearch: java.io.FileNotFoundException: no such file: /opt/zimbra/jetty-6.1.5/work/zimbra/jsp/org/apache/jsp/tag/web/mobile/moDisplayMessage_tag.class

[Rich Graves]

Damnit. Now it works.

I had restarted mailboxd twice and verified it still broken. Then I turned on zmprov aal for everything but POP and IMAP (which one is relevant? soap? misc?), shut mailboxd down again, emptied mailbox.log to reduce the noise and disk space, and restarted, at which point it worked with nothing of interest logged. If there was anything, it's in the mailbox.log that I deleted. :-(

So unless there's something interesting in the jetty logs, I got nothing. Oh well. Mark, did the support case you opened conclude happily?

[LMStone]

Damnit. Now it works.

I had restarted mailboxd twice and verified it still broken. Then I turned on zmprov aal for everything but POP and IMAP (which one is relevant? soap? misc?), shut mailboxd down again, emptied mailbox.log to reduce the noise and disk space, and restarted, at which point it worked with nothing of interest logged. If there was anything, it's in the mailbox.log that I deleted. :-(

So unless there's something interesting in the jetty logs, I got nothing. Oh well. Mark, did the support case you opened conclude happily?

Rich,

We upgraded to 5.0.5 as a "fix".

We ran out of time during the maintenance window to test deleting the jetty/work directory contents and see if they get recreated correctly when restarting the mailbox. We'll try that this weekend and then update here and our support ticket accordingly.

Hope your "fix" sticks as well!

All the best,
Mark

[Rich Graves]

I can reproduce the problem with 5.0.5. Stop mailboxd, rm -rf /opt/zimbra/jetty/work/*, start mailboxd, and get random errors in /h/ and /m/.

Wait sometimes as long as 15 minutes, and the errors go away.

I guess the server has better things to do at startup than recompile JSPs.

[LMStone]

I can reproduce the problem with 5.0.5. Stop mailboxd, rm -rf /opt/zimbra/jetty/work/*, start mailboxd, and get random errors in /h/ and /m/.

Wait sometimes as long as 15 minutes, and the errors go away.

I guess the server has better things to do at startup than recompile JSPs.

Rich,

Can you do an ls > rightafterrestart.txt on the jetty/work directory, and then when the errors go away, do another ls > allisOKnow.txt and do a diff on the two text files?

Would be nice to know if the "lazy loading" feature in ZCS 5 applies to recreating the jar files as well...

All the best,
Mark

P.S. If you are running 5.0.5 now shouldn't you update your profile?

[Rich Graves]

Can you do an ls > rightafterrestart.txt on the jetty/work directory, and then when the errors go away

It's hard to know exactly "when the errors go away," but I'll see if I can find anything, time permitting. General lesson seems to be "it's not safe to wipe out the work directory on a production server."

P.S. If you are running 5.0.5 now shouldn't you update your profile?

Production 5.0.4, testing 5.0.5. I wouldn't be offering to restart production so willy-nilly.

[jholder]

Can you guys open tickets? We really need to figure out why it's happening on like 2% of systems. Very odd.

Usually, wiping the work dir can be a good thing to ensure that you're getting the latest version of the files.

When the work dir is wiped (and as long as perms are correct), the jsp's simply recompile. In the cases on this thread, they're not.

IF you get a chance, please do a diff on your /opt/zimbra/jetty/webapps and a off the shelf webapps dir. I'm getting the feeling that one of our apps/tags aren't getting updated.

[LMStone]

Can you guys open tickets? We really need to figure out why it's happening on like 2% of systems. Very odd.

Usually, wiping the work dir can be a good thing to ensure that you're getting the latest version of the files.

When the work dir is wiped (and as long as perms are correct), the jsp's simply recompile. In the cases on this thread, they're not.

IF you get a chance, please do a diff on your /opt/zimbra/jetty/webapps and a off the shelf webapps dir. I'm getting the feeling that one of our apps/tags aren't getting updated.

John,

We already have a support ticket open and were asked to upgrade to 5.0.5 before proceeding further.

We'll test this weekend during our maintenance window:

mkdir /opt/zimbra/jetty/work-original
rsync -avz /opt/zimbra/jetty/work/ /opt/zimbra/jetty/work-original/
zmcontrol stop
rm -rf /opt/zimbra/jetty/work/*
zmcontrol start

and report back what happens.

If we get errors, I'll do a diff on the contents of ~/jetty/work and ~/jetty/work-original.

Anything else I should be doing here?

Thanks,
Mark