[SOLVED] Zimbra + Apache 2 very slow!

[jakob.m]

Hi,

a couple of weeks ago I set up a new Zimbra Server with a Apache2 Webserver working on the same machine.
Since I transferred all old data from my old webserver and all old mailuser and mails to the new Server, and all about 30 users are using the new server now frequently, the server is becoming slower and slower.
Since I don`t think it`s because of those 30 users, i know them and i don`t think they are like every 5 minutes refreshing mails or website, I think it`s a general problem of Zimbra and Apache2 working on the same machine.

The problem is, if I restart the server, everyhing works fine and very fast for about 2 to 3 hours. Then it slows down for some reason. After about 6 hours the website-speed is that slow that one can`t even load a simple html-site within 2 minutes.

Since I´m no pro, and I know here are lot´s of people knowing about this much more than I do, it would be nice if one could answer me and help me solving my problem.

Since I don`t know where to start, please tell me wich log`s you need to see.

General Serverstats:

Prozessor : Intel Core™2 Duo2 x 2,93 GHz
RAM: 4 GB DDR2-RAM
HDD: 2x 320 GB SATA 2
Network: 1 GBit/s
OS: Ubuntu Hardy Heron v8.04
----
MySQL:
Array ( [0] => Uptime: 2458 [1] => Threads: 1 [2] => Questions: 470 [3] => Slow queries: 0 [4] => Opens: 374 [5] => Flush tables: 1 [6] => Open tables: 64 [7] => Queries per second avg: 0.191 )

I attached some log-files, maybe they will help. Please tell me wich I need to attach additionally.

[host and hostname are correct configured since i did this couple of weeks ago with help from this forum.]

[edit]

By the way, webmin shows this Server-Status abfter 50 min. uptime:

System uptime 	0 hours, 50 minutes
Running processes 	145
CPU load averages 	1.18 (1 min) 0.66 (5 mins) 0.55 (15 mins)
CPU usage 	0% user, 0% kernel, 0% IO, 99% idle
Real memory 	3.83 GB total, 1.31 GB used	
Virtual memory 	7.31 GB total, 0 bytes used	
Local disk space 	451.25 GB total, 86.91 GB used

After 3:43 hours uptime (Still fast)

Processor information 	Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz, 2 cores
System uptime 	3 hours, 43 minutes
Running processes 	141
CPU load averages 	0.41 (1 min) 0.49 (5 mins) 0.45 (15 mins)
CPU usage 	1% user, 0% kernel, 0% IO, 99% idle
Real memory 	3.83 GB total, 1.40 GB used	
Virtual memory 	7.31 GB total, 0 bytes used	
Local disk space 	451.25 GB total, 86.92 GB used

[dalmate]

Can you show me your output when type command while the server is slow:
top
or maybe you can disable the antivirus and antispam services of zimbra to check if they made that problems.

[FritzBrause]

Zimbra is intended to be run as the only application on a dedicated Server. Other memory hungry softare stacks like a full blown Apache with PHP, Ruby etc. compete for ressources. If you're running your Webserver on the same hardware like Zimbra is there another MySQL Server running on the same machine too?

There may be an IO bottleneck - which it is in most cases. If your server is becoming slow again, you could check with

iostat -dx 2

what your harddisks are doing.

[jakob.m]

Ok,
the server is now again very very slow:
Here the output of webmin status:

System uptime 	22 hours, 32 minutes
Running processes 	146
CPU load averages 	0.68 (1 min) 0.59 (5 mins) 0.54 (15 mins)
CPU usage 	1% user, 0% kernel, 0% IO, 98% idle
Real memory 	3.83 GB total, 1.77 GB used
	
Virtual memory 	7.31 GB total, 1.56 MB used
	
Local disk space 	451.25 GB total, 98.62 GB used

even typing in ssh is very very slow and is displayd with a huge delay.

top:

Tasks: 148 total,   1 running, 146 sleeping,   0 stopped,   1 zombie
Cpu(s):  1.0%us,  0.2%sy,  0.0%ni, 98.6%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   4020172k total,  3885360k used,   134812k free,   185224k buffers
Swap:  7667688k total,     1596k used,  7666092k free,  1853368k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 8170 zimbra    20   0 2120m 686m  12m S    2 17.5   8:15.25 java
 6096 zimbra    20   0 1546m 228m 4888 S    0  5.8   0:44.61 mysqld
    1 root      20   0  4016  688  604 S    0  0.0   0:00.24 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd
    3 root      RT   0     0    0    0 S    0  0.0   0:00.39 migration/0
    4 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/0
    5 root      RT   0     0    0    0 S    0  0.0   0:00.36 migration/1
    6 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/1
    7 root      20   0     0    0    0 S    0  0.0   0:02.49 events/0
    8 root      20   0     0    0    0 S    0  0.0   0:00.75 events/1
    9 root      20   0     0    0    0 S    0  0.0   0:00.00 khelper
   15 root      20   0     0    0    0 S    0  0.0   0:00.00 async/mgr
  188 root      20   0     0    0    0 S    0  0.0   0:00.00 sync_supers
  190 root      20   0     0    0    0 S    0  0.0   0:00.00 bdi-default
  191 root      20   0     0    0    0 S    0  0.0   0:00.12 kblockd/0
  192 root      20   0     0    0    0 S    0  0.0   0:00.11 kblockd/1
  193 root      20   0     0    0    0 S    0  0.0   0:00.00 kacpid
  194 root      20   0     0    0    0 S    0  0.0   0:00.00 kacpi_notify
  195 root      20   0     0    0    0 S    0  0.0   0:00.00 kacpi_hotplug
  296 root      20   0     0    0    0 S    0  0.0   0:00.00 ata/0
  297 root      20   0     0    0    0 S    0  0.0   0:00.00 ata/1
  298 root      20   0     0    0    0 S    0  0.0   0:00.00 ata_aux
  302 root      20   0     0    0    0 S    0  0.0   0:00.00 ksuspend_usbd
  307 root      20   0     0    0    0 S    0  0.0   0:00.00 khubd
  310 root      20   0     0    0    0 S    0  0.0   0:00.00 kseriod
  348 root      20   0     0    0    0 S    0  0.0   0:00.00 rpciod/0
  349 root      20   0     0    0    0 S    0  0.0   0:00.00 rpciod/1
  381 root      20   0     0    0    0 S    0  0.0   0:03.47 kswapd0
  382 root      20   0     0    0    0 S    0  0.0   0:00.00 aio/0
  383 root      20   0     0    0    0 S    0  0.0   0:00.00 aio/1
  384 root      20   0     0    0    0 S    0  0.0   0:00.00 nfsiod
  386 root      20   0     0    0    0 S    0  0.0   0:00.00 crypto/0
  387 root      20   0     0    0    0 S    0  0.0   0:00.00 crypto/1
  756 www-data  20   0  279m 9364 1604 S    0  0.2   0:00.00 apache2
 1078 root      20   0     0    0    0 S    0  0.0   0:00.00 iscsi_eh
 1085 root      20   0     0    0    0 S    0  0.0   0:00.00 fc_rport_eq
 1086 root      20   0     0    0    0 S    0  0.0   0:00.00 fnic_event_wq
 1102 root      20   0     0    0    0 S    0  0.0   0:00.00 scsi_eh_0
 1118 root      20   0     0    0    0 S    0  0.0   0:00.00 scsi_eh_1
 1120 root      20   0     0    0    0 S    0  0.0   0:00.00 scsi_eh_2
 1230 root      20   0     0    0    0 S    0  0.0   0:00.00 kpsmoused
 1261 root      20   0     0    0    0 S    0  0.0   0:00.00 usbhid_resumer
 1268 root      20   0     0    0    0 S    0  0.0   0:02.91 kjournald
 1272 postfix   20   0 47500 2248 1704 S    0  0.1   0:00.00 pickup
 1383 root      16  -4 16848  460  384 S    0  0.0   0:00.32 udevd
 2776 ntp       18  -2 23360 1160 1008 S    0  0.0   0:00.02 ntpd
 3186 root      20   0  3860  508  500 S    0  0.0   0:00.00 getty
 3187 root      20   0  3860  508  500 S    0  0.0   0:00.00 getty
 3189 root      20   0  3860  508  500 S    0  0.0   0:00.00 getty
 3190 root      20   0  3860  508  500 S    0  0.0   0:00.00 getty
 3192 root      20   0  3860  508  500 S    0  0.0   0:00.00 getty
 3196 root      20   0     0    0    0 S    0  0.0   0:04.22 flush-8:0
 3230 syslog    20   0 12292  716  564 S    0  0.0   0:03.40 syslogd
 3252 root      20   0  8128  500  476 S    0  0.0   0:00.55 dd
 3255 klog      20   0  4932 1664  432 S    0  0.0   0:00.17 klogd
 3276 messageb  20   0 21184  940  716 S    0  0.0   0:00.00 dbus-daemon
 3299 root      20   0 50912 1156  680 S    0  0.0   0:00.66 sshd

Could you tell me how to shutdown the antivirus and antispam?
/etc/init.d/spamassasin stop ??

By the way, it seems as if the server slows down every time, after the automatic backup to a backup-ftp took place. For this I use reoback. I saves the whole /opt/zimbra and /var/www folder by zipping and transfeering it via ftp. How can I check this?

[edit]

yes there is a mysql-server running, feeding a joomla-installation and a phpbb3 forum so not that much data.
i had to restart the server now since the webpage wasn`t accassable anymore, if it slows down the next time, i will test again.

[bofh]

if youve only one host i would strongly recommend to work with kvm.
with kvm you loosing almost nothing in cpu speed - but of course maybe some io slowdowns on the virtual harddisk but here you can do some things to improve...

its also easier to monitor which services doing trouble and easy to upgrade to a new server if nessesary...
and here ife to say that zimbras storage management is very very nice.

you need only basic harddiskspace and can archive the rest to an external source without anyone notice that their old data is on a different server..

so its alsomost a dreamland of virtualisation (and im not such a biig fan of virtualisation..) ..

however its strongly recommend to do that and do not let run other services than zimbra..


also for security reasons its better. if onyl one php application is compromised maybe your whole zimbra server is in danger.

[jakob.m]

at this stage there is no possibility to change again the server setup since every data and every user of a porject is using this server now. I have to live with it and therefore find the reason why the server os slowing down that much.

[bofh]

why you have to live with it?
you can easy setup a kvm and a new guest install without downtime, copy the config and the backups for an initial setup and make the switch and last datatransfer in the night.
shouldnt take longer than 4 to 5 hours total.

you could also do that on a different machine und just transfer the virtual host at times. i mean you cant run this server forever :-) so sooner or later you got a upgrade anyway but this gonna make it easier

you can use internal ip for the guest and set everything up and test it while the prod. system is still up. use a port forwarding from the external ip to the internal that way you can simply turn the switch whenever you want.

cant see the problem here. maintainance on productive systems like that happens all the time without or - with min downtime for a couple minutes.

if anything is wrong and didnt work properly you can easily switch back to the already running system.

alternative you put the apache on the virtual host (i would do both) and seperate it from the rest of the system.

i mean solving that problem (if it is easly solveable without downtime) is just using time to fix problems caused by an suboptimal groundsetup. you will run in other problems anyway sooner or later... which leeds to additional fixing.

the the solution i told you will fix all of those - so it is a fix of the current problem too, just cleaner
and dont be scared, kvm is very easy since it is in the kernel and it is a full not paravirt. so much easier to setup and handle (not kernel patches or so - its straightforward and even easier then setup an proper apache)

[jakob.m]

okay,
i will do this, but within the next 2 weeks i have to much trouble at work so this has to wait. Until then, it would be nice if I could find the slow-down-problem.
Is anybody using reoback as well and has the same problem?

[bofh]

what happen when you restart the apache process?
is it faster then?

if not
try same thing with zimbra (restart all services) and tell us.

btw: you backup is horrible wrong.
zimbra uses a mysql backend to store data. you cant backup it as a regular file (i mean you can but you dont like the result)

search for the backupscript in this forum
make a backup (if possible directly to another lokation like a san store / fileserver or whatever) if not possible at least in a different directory like "archiv"
backup this with your script to ftp


BTW: More im reading your post more im scared for your 30 users :-)
what exactly do you do? is it your emplayers server, do you make your own service in your freetime? whats the background...

i can tell you this, get rid of webmin. it might be a good choice for things like samba but defently not a good diea to manage your whole system with it (thats why no webmin in debian - to unstable, sometimes its scrables conf files). ok you cn add user and stuff but everything else do please on the commandshell

another thing is install htop (aptitude install htop) gives a a nice graphical shell virew of processes and cpu/ram load.


also please check your DNS thing. your result (if they are accurate) dont show anything which shuold cause that problem.
you cant have an IO bottleneck if absoluty nothing happens on the machine

so what exactly is slow? is it the website loaded from apache or is it zimbra or both?

[jakob.m]

Hi,

this server is for a project and since i`m no pro, please don`t treat me as one. I don`t want to sell anythig or stuff like that. I´m doing this in my free time to help organizing a project for about 30 students! Btw. the server is running with student-licenses.

The backup i set up was because zimbra stores its own backups in the /opt/zimbra path and since nothing or nothing realy big except the backups changes within this directory, I thougth backing up the whole /opt/zimbra directory would be sufficient (btw. backup is done incremental)

I don`t actually use webmin since I think it`s horrible organized. I set it up for samba but than dind`t actually use it. I will uninstall webmin.

Restarting Zimbra and/or Apache doesn`t speed up the system. Every webthing is slow. Doesn`t matter if i use the Zimbra Webserver by opening the webmail-tool or if I have a look at the website wich is provided by apache2.

I will install htop and if it slows down again, I will post the output.
I wanted to install apachetop for mintoring apache seperately but for some reason, even though it has the right logs, it doesn`t show anything.