Administrators GUI server status - red crosses no services running

[Hennie]

Hi All,
I am having a similar problem as mentioned many times by other admins whereas in the Admin GUI, the server status pages reflects red crosses on all services indicating that they arenot running. This is not the case as doing zmcontrol status shows all services as running.

I have restarted the server and also regenerated SSH keys, even checked for locked zimbra accounts. See below the info as requested on the logger wiki page (http://wiki.zimbra.com/index.php?tit....x_and_later):

1. zmloggerctl status; echo $?; ps ax | grep zmlogger
2747 pts/0 R+ 0:00 grep zmlogger
7535 ? S 14:02 /usr/bin/perl /opt/zimbra/libexec/zmlogger

2. /opt/zimbra/libexec/zmrrdfetch -f zmmtastats
----------cut-----------
1269937950,,,,,,,,,,
1269937980,,,,,,,,,,
1269938010,,,,,,,,,,
1269938040,,,,,,,,,,
1269938070,,,,,,,,,,
1269938100,,,,,,,,,,
1269938130,,,,,,,,,,
1269938160,,,,,,,,,,
1269938190,,,,,,,,,,
1269938220,,,,,,,,,,
1269938250,,,,,,,,,,
1269938280,,,,,,,,,,
1269938310,,,,,,,,,,
1269938340,,,,,,,,,,
1269938370,,,,,,,,,,
1269938400,,,,,,,,,,
1269938430,,,,,,,,,,
1269938460,,,,,,,,,,
1269938490,,,,,,,,,,
1269938520,,,,,,,,,,
1269938550,,,,,,,,,,
1269938580,,,,,,,,,,
1269938610,,,,,,,,,,
--------cut---------

3. /opt/zimbra/libexec/zmrrdfetch -c -f zmmtastats
---------cut---------
1269928500,,,,,,,,,,
1269928800,,,,,,,,,,
1269929100,,,,,,,,,,
1269929400,,,,,,,,,,
1269929700,,,,,,,,,,
1269930000,,,,,,,,,,
1269930300,,,,,,,,,,
1269930600,,,,,,,,,,
1269930900,,,,,,,,,,
1269931200,,,,,,,,,,
1269931500,,,,,,,,,,
1269931800,,,,,,,,,,
1269932100,,,,,,,,,,
1269932400,,,,,,,,,,
1269932700,,,,,,,,,,
1269933000,,,,,,,,,,
1269933300,,,,,,,,,,
1269933600,,,,,,,,,,
1269933900,,,,,,,,,,
1269934200,,,,,,,,,,
1269934500,,,,,,,,,,
1269934800,,,,,,,,,,
1269935100,,,,,,,,,,
1269935400,,,,,,,,,,
1269935700,,,,,,,,,,
1269936000,,,,,,,,,,
1269936300,,,,,,,,,,
1269936600,,,,,,,,,,
1269936900,,,,,,,,,,
1269937200,,,,,,,,,,
1269937500,,,,,,,,,,
1269937800,,,,,,,,,,
1269938100,,,,,,,,,,
1269938400,,,,,,,,,,
1269938700,,,,,,,,,,
1269939000,,,,,,,,,,
-------cut-----------

4. grep -w MTA /var/log/zimbra-stats.log
----"NOTHING"------ (is this the problem?)

5. tail /var/log/zimbra-stats.log
Mar 30 10:46:30 zmail01 zimbramon[840]: 840:info: zmstat mysql.csv: timestamp, Aborted_clients, Aborted_connects, Binlog_cache_disk_use, Binlog_cache_use, Bytes_received, Bytes_sent, Com_admin_commands, Com_alter_db, Com_alter_table, Com_analyze, Com_backup_table, Com_begin, Com_call_procedure, Com_change_db, Com_change_master, Com_check, Com_checksum, Com_commit, Com_create_db, Com_create_function, Com_create_index, Com_create_table, Com_create_user, Com_dealloc_sql, Com_delete, Com_delete_multi, Com_do, Com_drop_db, Com_drop_function, Com_drop_index, Com_drop_table, Com_drop_user, Com_execute_sql, Com_flush, Com_grant, Com_ha_close, Com_ha_open, Com_ha_read, Com_help, Com_insert, Com_insert_select, Com_kill, Com_load, Com_load_master_data, Com_load_master_table, Com_lock_tables, Com_optimize, Com_preload_keys, Com_prepare_sql, Com_purge:::BC56A9BC-3BD8-11DF-8A22-376811BD5AC3:::
Mar 30 10:46:30 zmail01 zimbramon[840]: 840:info: :::BC56A9BC-3BD8-11DF-8A22-376811BD5AC3:::, Com_purge_before_date, Com_rename_table, Com_repair, Com_replace, Com_replace_select, Com_reset, Com_restore_table, Com_revoke, Com_revoke_all, Com_rollback, Com_savepoint, Com_select, Com_set_option, Com_show_binlog_events, Com_show_binlogs, Com_show_charsets, Com_show_collations, Com_show_column_types, Com_show_create_db, Com_show_create_table, Com_show_databases, Com_show_errors, Com_show_fields, Com_show_grants, Com_show_innodb_status, Com_show_keys, Com_show_logs, Com_show_master_status, Com_show_ndb_status, Com_show_new_master, Com_show_open_tables, Com_show_privileges, Com_show_processlist, Com_show_slave_hosts, Com_show_slave_status, Com_show_status, Com_show_storage_engines, Com_show_tables, Com_show_triggers, Com_show_variables, Com_show_warnings, Com_slave_start, Com_slave_sto:::BC56B452-3BD8-11DF-8A22-376811BD5AC3:::
Mar 30 10:46:30 zmail01 zimbramon[840]: 840:info: :::BC56B452-3BD8-11DF-8A22-376811BD5AC3::, Com_stmt_close, Com_stmt_execute, Com_stmt_fetch, Com_stmt_prepare, Com_stmt_reset, Com_stmt_send_long_data, Com_truncate, Com_unlock_tables, Com_update, Com_update_multi, Com_xa_commit, Com_xa_end, Com_xa_prepare, Com_xa_recover, Com_xa_rollback, Com_xa_start, Compression, Connections, Created_tmp_disk_tables, Created_tmp_files, Created_tmp_tables, Delayed_errors, Delayed_insert_threads, Delayed_writes, Flush_commands, Handler_commit, Handler_delete, Handler_discover, Handler_prepare, Handler_read_first, Handler_read_key, Handler_read_next, Handler_read_prev, Handler_read_rnd, Handler_read_rnd_next, Handler_rollback, Handler_savepoint, Handler_savepoint_rollback, Handler_update, Handler_write, Innodb_buffer_pool_pages_data, Innodb_buffer_pool_pages_dirty, Innodb_buffer_pool_pages_flush:::BC56B89E-3BD8-11DF-8A22-376811BD5AC3:::
Mar 30 10:46:30 zmail01 zimbramon[840]: 840:info: :::BC56B89E-3BD8-11DF-8A22-376811BD5AC3:::ed, Innodb_buffer_pool_pages_free, Innodb_buffer_pool_pages_misc, Innodb_buffer_pool_pages_total, Innodb_buffer_pool_read_ahead_rnd, Innodb_buffer_pool_read_ahead_seq, Innodb_buffer_pool_read_requests, Innodb_buffer_pool_reads, Innodb_buffer_pool_wait_free, Innodb_buffer_pool_write_requests, Innodb_data_fsyncs, Innodb_data_pending_fsyncs, Innodb_data_pending_reads, Innodb_data_pending_writes, Innodb_data_read, Innodb_data_reads, Innodb_data_writes, Innodb_data_written, Innodb_dblwr_pages_written, Innodb_dblwr_writes, Innodb_log_waits, Innodb_log_write_requests, Innodb_log_writes, Innodb_os_log_fsyncs, Innodb_os_log_pending_fsyncs, Innodb_os_log_pending_writes, Innodb_os_log_written, Innodb_page_size, Innodb_pages_created, Innodb_pages_read, Innodb_pages_written, Innodb_row_lock_current_wai:::BC56BD30-3BD8-11DF-8A22-376811BD5AC3:::
Mar 30 10:46:30 zmail01 zimbramon[840]: 840:info: :::BC56BD30-3BD8-11DF-8A22-376811BD5AC3:::ts, Innodb_row_lock_time, Innodb_row_lock_time_avg, Innodb_row_lock_time_max, Innodb_row_lock_waits, Innodb_rows_deleted, Innodb_rows_inserted, Innodb_rows_read, Innodb_rows_updated, Key_blocks_not_flushed, Key_blocks_unused, Key_blocks_used, Key_read_requests, Key_reads, Key_write_requests, Key_writes, Last_query_cost, Max_used_connections, Not_flushed_delayed_rows, Open_files, Open_streams, Open_tables, Opened_tables, Prepared_stmt_count, Qcache_free_blocks, Qcache_free_memory, Qcache_hits, Qcache_inserts, Qcache_lowmem_prunes, Qcache_not_cached, Qcache_queries_in_cache, Qcache_total_blocks, Queries, Questions, Rpl_status, Select_full_join, Select_full_range_join, Select_range, Select_range_check, Select_scan, Slave_open_temp_tables, Slave_retried_transactions, Slave_running, Slow_launch:::BC56C1C2-3BD8-11DF-8A22-376811BD5AC3:::
Mar 30 10:46:30 zmail01 zimbramon[840]: 840:info: :::BC56C1C2-3BD8-11DF-8A22-376811BD5AC3:::_threads, Slow_queries, Sort_merge_passes, Sort_range, Sort_rows, Sort_scan, Ssl_accept_renegotiates, Ssl_accepts, Ssl_callback_cache_hits, Ssl_cipher, Ssl_cipher_list, Ssl_client_connects, Ssl_connect_renegotiates, Ssl_ctx_verify_depth, Ssl_ctx_verify_mode, Ssl_default_timeout, Ssl_finished_accepts, Ssl_finished_connects, Ssl_session_cache_hits, Ssl_session_cache_misses, Ssl_session_cache_mode, Ssl_session_cache_overflows, Ssl_session_cache_size, Ssl_session_cache_timeouts, Ssl_sessions_reused, Ssl_used_session_cache_entries, Ssl_verify_depth, Ssl_verify_mode, Ssl_version, Table_locks_immediate, Table_locks_waited, Tc_log_max_pages_used, Tc_log_page_size, Tc_log_page_waits, Threads_cached, Threads_connected, Threads_created, Threads_running, Uptime:: 03/30/2010 10:46:30,0,0,0,0,61650613,2:::BC56C640-3BD8-11DF-8A22-376811BD5AC3:::
Mar 30 10:46:30 zmail01 zimbramon[840]: :::BC56C640-3BD8-11DF-8A22-376811BD5AC3:::36496390,16,0,0,0,0,0,0,0,0,0,0,988 00,0,0,0,0,0,0,9784,0,0,0,0,0,0,0,0,0,0,0,0,0,0,11 501,515,0,0,0,0,0,0,0,0,0,0,0,0,10891,0,0,0,0,0,10 6521,0,93507,319366,0,0,0,16,0,0,0,0,0,0,0,501,0,0 ,0,0,0,0,0,0,0,0,1394,0,0,0,16,0,0,0,0,0,0,0,0,0,0 ,0,52501,20,0,0,0,0,0,0,OFF,1411,20,5,1085,0,0,0,1 ,183995,0,0,0,28,586110,2815459,338343,23380,25571 7,6027,0,0,37,277519,20853,1488,144315,287130,625, 308608,8,12,15666905,19086,0,1049260,31897,0,0,0,3 34663680,19969,112838,4784804352,144315,3349,0,120 564,8727,10682,0,0,54889472,16384,560,20293,144315 ,0,30,30,30,1,17169,18334,3362760,57493,0,6682,13, 171,18,46,0,0.000000,17,0,35,0,251,257,0,0,0,0,0,0 ,0,0,0,706336,706336,NULL,0,0,4061,0,1046,0,0,OFF, 0,12,0,597,17770,9,0,0,0,,,0,0,0,0,0,0,0,0,0,NONE, 0,0,0,0,0,0,0,,180349,0,0,0,0,0,17,17,1,30128
Mar 30 10:46:30 zmail01 zimbramon[695]: 695:info: zmstat vm.csv: timestamp, r, b, swpd, free, buff, cache, si, so, bi, bo, in, cs, us, sy, id, wa, MemTotal, MemFree, Buffers, Cached, SwapCached, Active, Inactive, SwapTotal, SwapFree, Dirty, Writeback, AnonPages, Mapped, Slab, SReclaimable, SUnreclaim, PageTables, NFS_Unstable, Bounce, CommitLimit, Committed_AS, VmallocTotal, VmallocUsed, VmallocChunk, HugePages_Total, HugePages_Free, HugePages_Rsvd, HugePages_Surp, Hugepagesize, loadavg:: 03/30/2010 10:46:30, 0, 0, 92, 215792, 492048, 10777864, 0, 0, 3, 1749, 1192, 3579, 25, 2, 72, 1, 16461628, 215792, 492048, 10777864, 32, 5499092, 10313488, 658624, 658532, 3776, 0, 4542500, 42068, 354512, 271940, 82572, 23784, 0, 0, 8889436, 14453992, 34359738367, 39340, 34359698683, 0, 0, 0, 0, 2048, 1.39
Mar 30 10:46:31 zmail01 zimbramon[813]: 813:info: zmstat fd.csv: timestamp, fd_count:: 03/30/2010 10:46:31, 4736
Mar 30 10:46:31 zmail01 zimbramon[693]: 693:info: zmstat cpu.csv: timestamp, cpu:user, cpu:nice, cpu:sys, cpu:idle, cpu:iowait, cpu:irq, cpu:softirq, cpu0:user, cpu0:nice, cpu0:sys, cpu0:idle, cpu0:iowait, cpu0:irq, cpu0:softirq, cpu1:user, cpu1:nice, cpu1:sys, cpu1:idle, cpu1:iowait, cpu1:irq, cpu1:softirq, cpu2:user, cpu2:nice, cpu2:sys, cpu2:idle, cpu2:iowait, cpu2:irq, cpu2:softirq, cpu3:user, cpu3:nice, cpu3:sys, cpu3:idle, cpu3:iowait, cpu3:irq, cpu3:softirq:: 03/30/2010 10:46:31, 24.5, 0.0, 2.0, 72.4, 0.8, 0.1, 0.2, 31.8, 0.0, 2.3, 64.6, 0.3, 0.2, 0.9, 20.1, 0.0, 1.5, 77.7, 0.7, 0.0, 0.0, 23.1, 0.0, 2.1, 73.3, 1.6, 0.0, 0.0, 23.2, 0.0, 2.2, 74.0, 0.6, 0.0, 0.1

6. zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats | head -20
<GetLoggerStatsResponse xmlns="urn:zimbraAdmin">
<hostname hn="host.mydomain.com">
<stats name="zmmtastats"/>
</hostname>
</GetLoggerStatsResponse>

7. zmsoap -z GetLoggerStatsRequest stats/@name=zmmtastats @limit=1 | head -50
<GetLoggerStatsResponse xmlns="urn:zimbraAdmin">
<hostname hn="host.mydomain.com">
<stats name="zmmtastats"/>
</hostname>
</GetLoggerStatsResponse>


I will appreciate any help as I have battled with this now for a week!

[uxbod]

First of all it would be helpful if you were to update your profile with the following so we know which version of ZCS you are running

su - zimbra
zmcontrol -v

[Hennie]

Hi uxbod,

Sorry 'bout that. See below:

Release 6.0.1_GA_1816.UBUNTU8_64 UBUNTU8_64 FOSS edition.

Thanks.

[uxbod]

Thanks As the zimbra user what happens if you do

ssh -i .ssh/zimbra_identity -o strictHostKeyChecking=no zimbra@MAIL.DOMAIN.COM

When you regenerated the ssh keys did you push them out with

su - zimbra
zmupdateauthkeys

[Hennie]

Hi,
ssh -i .ssh/zimbra_identity -o strictHostKeyChecking=no zimbra@MAIL.DOMAIN.COM does not come up with any error messages, I just get empty line (not dropping bck to command line).

Yes, I have pushed the keys with zmupdateauthkeys.

Need to know if this could be serious or the start of something going wrong?

[uxbod]

Are you using syslog or rsyslog ?

[Hennie]

I'm sorry, how do I know which I'm using?

[Hennie]

Nobody have any answers on this?

Hennie

[phoenix]

Nobody have any answers on this?

You haven't answered the question you were asked earlier about which logger you're using, check your server and see which one is installed and running. You should also upgrade to the ost recent version of ZImbra and see if the problem still occurs.

[breejesh]

just check whether you have sysstat package installed or not. it should be there else you wont get the server status.