Statistics collection problems (vserver)

[gracedman]

Hello, all. We have installed Zimbra 5.0.16_GA_2921.RHEL5_64_20090429021719 CentOS5_64 NETWORK edition on a VServer with reasonable success. However, we have never received any statistics. In pulling apart zmstatctl, we have found some interesting issues and are hoping someone can tell us what they mean and how to fix them.

We see this regularly in zmstat.out:
Use of uninitialized value in string ne at /opt/zimbra/libexec/zmstat-io line 87

I do not know perl well at all but it looks to me like the variable
$line is defined. I do not think this is peculiar to our environment.
What would cause this?

Some issues may be related to the fact we have installed on a very skeleton CentOS installation and may have some undocumented missing dependencies.
Whenever we run zmlogprocess, we get this:

[zimbra@zimbra01 zmstat]$ /opt/zimbra/libexec/zmlogprocess
Pruning service/server entries older then 2009-06-04 17:20:56
zimbra maps to zimbra.ssiservices.biz
smtp maps to smtp.ssiservices.biz
Processing mta entries from 3907047
Found 18 messages
Inserted 18 rows
Saving ID 3907215
Deleting processed postfix logs from raw_logs...Done
Processing clamd entries from 0
Found 0 entries
Deleting processed clamd logs from raw_logs...Done
Processing amavis entries from 0
DBD::mysql::st execute failed: Error writing file '/tmp/MY6zdFMa' (Errcode: 28) at /opt/zimbra/libexec/zmlogprocess line 1134.
Error executing select id from raw_logs where app='amavis' and id > '0' order by id asc with
Error writing file '/tmp/MY6zdFMa' (Errcode: 28)
Error writing file '/tmp/MY6zdFMa' (Errcode: 28) at /opt/zimbra/libexec/zmlogprocess line 1135.

Can't call method "fetchall_arrayref" on an undefined value at /opt/zimbra/libexec/zmlogprocess line 885.

We then installed perl-DBD-MySQL but this has not eliminated the
problem. What would cause this error? If it is of any consequence, we
are not running antispam on zimbra but we are on smtp.

We thought this might be related to the fact our /tmp is mounted
nosuid,noexec,nodev so we removed those restriction but see the same
results. We also verified that the zimbra user can write to /tmp.

When zmstat-proc runs, we get:
Warning: Not possible to monitor process stats

This could be related to our environment. For security reasons,
vservers do not have full access to /proc. An even more severe instance
was zmstat-vm which filled zmstat.out with errors about needing the
mount /proc. This was because we did not expose /proc/vmstat by
default. Once we did, the errors went away in zmstat-vm but not
zmstat-proc.

Does anyone have any idea why we are seeing these issues? We are wondering if they are related to the problem we have where all logging seems to stop at 4:02 every morning (the same time as logrotate runs its daily routine) and where we lose all data from 4:02 in the morning if we restart the zimbra service

Any help would be greatly appreciated as we are still very new to Zimbra. Thanks - John

[gracedman]

We have solved some but not all of the issues. The database issue was not vserver specific per se. It was more our fairly strict security profile. The bottom line was not enough /tmp space. Here are the details.

As a security practice, we build skeleton systems and only install necessary services. We did not pick up several dependencies for Zimbra including the perl-DBD-MySQL. Either they are not properly documented assuming a fat installation or we were sloppy. This built up a huge backlog of unprocessed data.

As a security and performance practice, we create /tmp in RAM and, because it is a separate partition, flag it noexec,nosuid,nodev to avoid using /tmp as a major attack vector for executing malicious code. Because it is a RAM partition, it was limited to 512 MB in this case. The large backlog needed more space than that to process its data.

We moved /tmp temporarily to disk and processed the backlog using zmlogprocess and zmgengraphs. Now we have stats. As far as I can tell, we are now running smoothly with the RAM based /tmp having cleared the backlog.

We did have one vserver specific issue. zmstat-vm runs vmstat which must have access to /proc/vmstat. This is not enabled by default in vserver. We thus had to create an /etc/vservers/.defaults/apps/vprocunhide/file file with a line "/proc/vmstat" in it. Unfortunately, this is an all or nothing approach so all the vservers on the same host now have access to /proc/vmstat (there are ways around this). Moreover the statistics are not virtualized, i.e., they reflect the data for the entire host system including all guest processes. But, at least it works.

We still do not know why we are getting the zmstat-proc and zmstat-io errors. Any thoughts? Thanks - John