mystery solved: unclean shutdown causes redo.log to be owned by root:root

[anand]

Today we finally tracked down why on unclean shutdown of the tomcat JVM (mailbox server process) would cause redo.log and even some other files to be owned by root:root.

Turns out there is a bug in linux kernel circa 2.6.5 (can't find a specific bug report on it) which would cause threads created before a setuid() call to have wrong privileges. (See zimbra bug 8158 for details.)

This was showing up on our SuSE Linux Enterprise Server v9 build box. We just upgraded the SLES box and problem is fixed in the latest kernel on SLES9.

I know this has come up and forums a few times, specially with Debian. So be sure to check your kernel version - because to be security hygienic you really don't want those threads running root privileged.

[Crexis]

Ha-HA! and you guys thought I was just some kind of looney (with reference to my bugzilla report). Nice one, good to know you guys have got a bit of terrier in you (won't let go until it's dead)

[jholder]

Loony Toon j/k
Good job

[PNE]

I do not know if this is the right place for comments, but please read this. I have noticed several times that files in opt/zimbra/amavisd/.spamassassin/ directory have incorrect permissions, in fact the same you mentioned (owned by root/root, read only). This happens quite randomly, let's say once a month. I do not think it is related with Zimbra shutdown or restart, it just happens while Zimbra is running. As an effect, bayes stops working, while other spamassassin tests work, just as DSPAM does. After I manually set permissions back to zimbra/zimbra, bayes will start working, no need to restart Zimbra. I am using Centos 4.3 with 2.6.9-34.0.1 kernel, which is much newer version than you suspect to cause the bug. Today I got notification from Centos and upgrading to 2.6.9-34.0.2. Later I will see, it this helped, but as I wrote above, problem appears to be very random.

Does anyone here have the same problem? And is it related to the bug mentionded in this thread?

[claros]

Originally Posted by PNE

I do not know if this is the right place for comments, but please read this. I have noticed several times that files in opt/zimbra/amavisd/.spamassassin/ directory have incorrect permissions, in fact the same you mentioned (owned by root/root, read only). This happens quite randomly, let's say once a month. I do not think it is related with Zimbra shutdown or restart, it just happens while Zimbra is running. As an effect, bayes stops working, while other spamassassin tests work, just as DSPAM does. After I manually set permissions back to zimbra/zimbra, bayes will start working, no need to restart Zimbra. I am using Centos 4.3 with 2.6.9-34.0.1 kernel, which is much newer version than you suspect to cause the bug. Today I got notification from Centos and upgrading to 2.6.9-34.0.2. Later I will see, it this helped, but as I wrote above, problem appears to be very random.

Does anyone here have the same problem? And is it related to the bug mentionded in this thread?

Ciao,
I'm testing zimbra since a week, but will try to contribute...
With the startup script included in /opt/zimbra/libexec/zimbra service will never shutdown in a clean way under RHEL and Centos.
In this way you could loose your file permission and have a strange modification in zimbra config files.
Seem to have solved creating a semaphore file under /var/lock/subsys. Testing with kernel 2.6.9-34.0.2smp
My startup/shutdown script is attached, just the modified part without comments and license

Greetings
Claudio

[PNE]

Claros thanks for your help. And for Zimbra staff - can you please check Claros' solution and confirm the problem itself and this one solution?

It seems to me that you consider this "mysterious problem" solved (no one posted anything after my post since almost one month ago), but for sure it is not! Just today my Zimbra instal got stuck just because permissions on amavisd/.spamassassin/ got corrupted again (see my last post for more). The result was long timeouts for content checking and subsequently overall server performance drop/overload/jam. Messages not passing through, users complaining etc... Really bad monday morning.

[KevinH]

PNE, claros - What Distro and version are you running and what is the exact Kernel version?

To date we've only seen this on SuSE and Debian. SuSE kernel upgrade fixes it. Debain doesn't have a patch available yet.

[PNE]

Running 3.1.4 GA on fully patched Centos 4.3, kernel 2.6.9-34.0.2, as I wrote in one of posts above.

[KevinH]

Originally Posted by PNE

Running 3.1.4 GA on fully patched Centos 4.3, kernel 2.6.9-34.0.2, as I wrote in one of posts above.

Can you reproduce this everytime? I assume you installed the RedHat EL4 build on CentOS right?

[PNE]

I don't know if the naming is right, but I've downloaded and installed CentOS 4.2 Server CD, just necessary services + GUI (X-Window + Gnome), then updated using yum to 4.3. CentOS Server distribution is described at centos.org. I think many users here use the same OS.

No, I cannot reproduce this behaviour. I don't know next time the problem will occur. It happens occasionally let's say once or twice a month, it has happenned overnight or in daytime, without restarting Zimbra or OS.