[SOLVED] Problem with Network Edition: Moving from 32-bit to 64-bit Server

[drwho]

Hi

I am trying to migrate 6.0.6 Network Edition from Ubuntu 8.0.4 32bit to Centos 5.4 64bit.

I have followed the document "Network Edition: Moving from 32-bit to 64-bit Server" (Network Edition: Moving from 32-bit to 64-bit Server - Zimbra :: Wiki).

I have tried this twice, each time with the same result

The installation of ZCS on the new machine ran OK and I checked that the installation was stable (ie I could get to the admin page and zmcontrol status showed all apps running)

In passing, contrary to the instructions I noticed that the following values were absent from both the 32bit and 64bit versions of localconfig.xml:

zimbra_logger_mysql_password
mailboxd_keystore_base_password

I followed the instructions without any problems.

When I run zmcontrol start I see this:

Starting ldap...Done.
Starting logger...Done.
Starting convertd...Done.
Starting mailbox...Done.
Starting antispam...Done.
Starting antivirus...Done.
Starting snmp...Done.
Starting spell...Done.
Starting mta...Done.
Starting stats...Done.

OK so far, but I can't get to the admin screen on ~:7071/zimbraAdmin

So I run zmcontrol status and see this - some processes have stopped:

antispam Running
antivirus Running
convertd Running
ldap Running
logger Running
mailbox Stopped
zmmtaconfig is running.
mailboxd is not running.
zmmailboxdctl is not running
mta Running
snmp Running
spell Running
stats Running

I've had a non-expert look at the logs but can't find anything obvious.

I don't have apache installed.

Any suggestions would be welcome...

richard

[Hivos]

Just some random shots: Did you change hostname perhaps? Fix your hosts file? Is there another process running on port 25? ( netstat -platune | grep 25 )

[soxfan]

I attempted a similar migration on Saturday and had similar issues. See my post here. I think the problem has something to do with certificates (self-signed in my case). Unfortunately I don't have a solution yet. If you come up with something please post back; I will do the same.

BTW, I noticed a similar thing with the missing variables in the localconfig.xml file. I wasn't overly concerned with the logger variable, but the mailboxd_keystore_base_password caught my attention when my problems arose. In my case the variable existed on my 32-bit system, but wasn't in my 64-bit file at all (as noted in my post).

[maxxer]

Once I had issues on a similar migration. On the new server I needed to regenerate the self-signed certs.

[drwho]

Hi

Section 4 of "Network Edition: Moving from 32-bit to 64-bit Server" deals with commercial certificates; I ignored that because I'm not using any.

The problem appears to be that the new installation uses a newly generated certificate - not one copied from the old system. So in that situation the value of mailboxd_keystore_password in localconfig.xml must NOT be changed.

When I changed this back to the value for the new installation, I found that the mailbox processes remained stable and I am now using Zimbra.

However: an hour later (midnight) I received this email:

Apr 26 00:00:03 mailserver zimbramon[24303]: 24303:err: Service status change: mailserver.xxxxxxx.co.uk stats changed from running to stopped

And sure enough, zmcontrol status shows stats is stopped - Anyone any thoughts on this?

In addition, I am seeing weird CPU behaviour:

It looks like a square-wave. predominantly ~2 minutes at 2% CPU followed by ~2 minutes at 100% CPU - and this is in a very lightly loaded environment (only 3 users in the office at present, none doing much with email)


If anyone from Zimbra is reading this, may I comment that the document in question is described as "certified documentation" and I am dissappointed that it is inaccurate and that after 2 days of hard work I still have problems. This type of migration is likely to be a fairly common occurence, and the documentation really ought to to tested.

I have identified two issues in the section dealing with localconfig.xml (2 values not present - c and f in the document - and one (d) which shouldn't be changed if the user is using Zimbra-generated certificates. Please update the document for the benefit of others!


richard

[Dirk]

To lend my voice to the choir here, I've also attempted this migration following those instructions and ended up with a non functional server, so yes I think it would be an exceedingly good idea if it was reviewed by Zimbra support engineers.

[soxfan]

Well, in a test environment anyway it seems like if I leave the 'mailboxd_keystore_password' alone (ie do not copy value from 32-bit system) and get rid of the 'mailboxd_keystore_base_password' value things seem to be working. As I mentioned previously, the 'mailboxd_keystore_base_password' existed on the 32-bit side, but did not get created by the installation on the 64-bit side.

It is going to be tough for me to know if everything is really OK, because I have my new 64-bit Zimbra server segregated from my live network. Until I actually see live email flowing in and out I won't feel 100% comfortable with the setup, but it seems like things are moving in the right direction anyway.

However: an hour later (midnight) I received this email:

Apr 26 00:00:03 mailserver zimbramon[24303]: 24303:err: Service status change: mailserver.xxxxxxx.co.uk stats changed from running to stopped

And sure enough, zmcontrol status shows stats is stopped - Anyone any thoughts on this?

Could this have been a 'cron' job or something that temporarily stopped the stats service in order to do something? Seems kind of strange that it would happen right at midnight.

It looks like a square-wave. predominantly ~2 minutes at 2% CPU followed by ~2 minutes at 100% CPU - and this is in a very lightly loaded environment (only 3 users in the office at present, none doing much with email)

Have you tried monitoring with 'top' (or similar) to see what process is taking all the CPU?

[drwho]

Originally Posted by drwho

However: an hour later (midnight) I received this email:

Apr 26 00:00:03 mailserver zimbramon[24303]: 24303:err: Service status change: mailserver.xxxxxxx.co.uk stats changed from running to stopped

And sure enough, zmcontrol status shows stats is stopped - Anyone any thoughts on this?

UPDATE - "stats stopped" problem

After some Googling, I found what appears to be a solution (works for me so far - stats still running after several hours):

as root:

chown zimbra:zimbra /opt/zimbra/zmstat -R

That tree was owned by root. I can't imagine how that could be - I didn't touch the directory; it was created by the installer. I thought running zmfixperms was supposed to fix permissions - or maybe it is the culprit??

richard

[drwho]

Originally Posted by soxfan

Have you tried monitoring with 'top' (or similar) to see what process is taking all the CPU?

Thinking back, I had this problem with the old installation. It was something to do with over-zealous monitoring processes. There was a method to turn them off or change the scheduling.

What I find a bit odd is that a lightly loaded system is capable of being so demanding on the CPU. My instinct is that something somewhere is being very inefficient. Seem to be a variety of java processes with different PID's which keep coming and going. Call me old-fashioned, but maybe a case for re-writing in c++ !!

richard

[soxfan]

Thinking back, I had this problem with the old installation. It was something to do with over-zealous monitoring processes. There was a method to turn them off or change the scheduling.

Right could be the logger process...

Here's a link with instructions for disabling totally:
Zimbra slow in very small deployment

and another with instructions on changing some parameter settings, so that it doesn't run as often:
High CPU spikes every 1 minute