Zimbra-Logger includes tools for syslog aggregation, reporting, and message tracing. Installing the Logger package is optional, but if you do not install Logger, Server Statistics and Server Status information is not captured and message tracing is not available.
In environments with more than one Zimbra server, Logger is enabled on only one mailbox server. This server is designated as the monitor host. The Zimbra monitor host is responsible for checking the status of all the other Zimbra servers and presenting this information on the Zimbra administration console. The information updates every 10 minutes. You can modify this time from the crontab.
|
5.
|
Stop and start the daemon. Type sudo launchctl unload /System/Library/LaunchDaemons/com.apple.syslogd.plist
sudo launchctl load /System/Library/LaunchDaemons/com.apple.syslogd.plist
|
The Server Status page lists all servers and services, their status, and when the server status was last checked. The servers include the MTA, LDAP, and mailbox server. The services include MTA, LDAP, Mailbox, SNMP, Anti-Spam, Anti-Virus, Spell checker, and Logger.
To start a server if it is not running, use the zmcontrol CLI command. You can stop and start services from the administration console,
Servers>
Services tab.
If the Zimbra-logger package is installed on a Zimbra mailbox server. Server Statistics shows bar graphs of the message count, message volume, anti-spam, and anti-virus activity. The information is displayed for the last 48 hours, and 30, 60, and 365 days.
When Server Statistics is selected in the Navigation pane, consolidated statistics for all mailbox servers is displayed. Selecting a specific server in the expanded view shows statistics for that server only, including disk usage for that server.
|
•
|
Message Count counts message transactions. A transaction is defined as either the SMTP receipt of a message per person (by Postfix) or a LMTP delivery of it (by mailboxd) per person. For example, if a message is sent to three people, six transactions are displayed. Three for SMTP to Postfix and three four LMTP to mailboxd. The message count is increased by six. The last 48 hours shows the count per hour. The last x days shows the count per day.
|
|
•
|
Message Volume displays the aggregate size in bytes of transactions sent and received per hour and per day. Graphs show the total inbound data by volume in KB, MB, or GB. The scale changes depending on the volumes in question.
|
|
•
|
Anti-Spam/Anti-Virus Activity displays the number of messages that were checked for spam or viruses and the number of messages that were tagged as spam or deemed to contain a virus. The AS/AV count is increased by one per message scanned. One message sent to three people counts as only one message processed by AS/AV.
|
|
•
|
Disk for a selected server displays the disk used and the disk space available. The information is displayed for the last hour, day, month, and year.
|
|
•
|
Session displays information about the active Web client, administrator and IMAP sessions. You can see how many active sessions are opened, who is logged on, when the session was created and the last time the session was accessed.
|
Each email message includes a header that shows the path of an email from its origin to destination. This information is used to trace a message’s route when there is a problem with the message. The Zimbra email message header can be viewed from the Zimbra Web Client Message view. Right-click on a message and select
Show Original.
|
•
|
Date - The date and time the message was sent. When you specify time, you can specify range by adding start and stop time to search for messages.
|
|
•
|
From - The name of the sender and the email address
|
|
•
|
To - The name of the recipient and the email address
|
|
•
|
Message-ID - Unique number used for tracing mail routing
|
|
•
|
Received: from - The name and IP address the message was sent from. The header displays Received: from information from the MTA to the LMTP and from the local host.
|
The CLI utility, zmmsgtrace is run to find email messages by the follow:
The following message trace example was looking for messages sent from sender, jdoe, to recipient address, aol.com, any time within the last 30 days. The details show that two messages were sent, and it shows to whom the messages were sent.
$ zmmsgtrace -s jdoe -r aol.com Tracing messages from jdoe to aol.com Message ID 17357409.1128717619728.JavaMail.companya@example.com jdoe@example.com --> kumsh@aol.com Recipient kumsh@aol.com 2005-01-07 13:40:19 - example.com (10.10.000.20) --> 2005-01-07 13:40:20 - example --> 000.0.0.1 (100.0.0.0) status sent 2005-01-07 13:40:20 Passed by amavisd on example (CLEAN) HITS: -5.773 in 539 ms 2005-01-07 13:40:20 - localhost.localdomain (100.0.0.1) --> example 2005-01-07 13:40:20 - example --> mta02.example.com (0.00.000.00) status sent Message ID 3836172.14011130514432170.JavaMail.root@example.com jdoe@example.com --> harma@aol.com lt@hotmail.com Recipient harma@aol.com 2005-01-28 08:47:13 - localhost.localdomain (000.0.0.1) --> example 2005-01-28 08:47:13 - example --> mta02.example.com ( 0.70.000.09) status sent 2 messages found
|
When the Logger package is installed, a daily mail report is automatically scheduled in the crontab. The Zimbra daily mail report includes the following information:
You should regularly review your disks capacity and when disks are getting full you should take preventative measures to maintain service. To alert administrators when the mailbox server disk is almost full, an email message is sent to the admin account. The default is to send out warning alerts when the threshold is 85% and a critical alert when the threshold is 95%.
|
•
|
vm.csv: Linux VM statistics (from the vmstat command)
|
These files are in a standard CSV format that can be loaded into Excel for viewing and charting. They are archived to subdirectories of
/opt/zimbra/zmstat every day at midnight. You can change the time in the crontab.
The zmstat-chart CLI is used to collect statistical information for the CPU, IO, mailboxd, MTAqueue, MySQL, and other components and to run a script on the .csv files to display the usage details in various charts.
This command reads data from the .csv files in /opt/zimbra/zmstat/<date> and writes files to the directory specified with the
-d option. The default chart parameters are specified in
/opt/zimbra/conf/zmstat-chart.xml.
If you do not want to generate all the charts or want to add other charts that are not generated by default, you can specify an alternate chart conf file with the -c option. For more information about using the
zmstat-chart command, see
Appendix A Command-Line Utilities.
CPU utilization is tracked both at the server level and the process level. the following is a sample process CPU graph:
ZCS stats track the amount of memory used by each process i the system. This information can be used to determine how system memory is being allocated between the various processes
ZCS tracks the percentage of time that the Java Virtual Machine spends on garbage collection. If the JVM is spending more than a few percent of its time on garbage collection, consider increasing the amount of memory allocated to the server Java process.
Higher numbers indicate that MySQL is able to get data from memory instead of going to disk. If the high rate is below 990, MySQL is hitting the disk harder than it should be. Investigate the following issues:
|
•
|
Run EXPLAIN on some of the SQL statements in /opt/zimbra/log/myslow.log, to see if they are causing InnoDB to read a large amount of data.
|
If you are having problems with mail delivery, you can view the mail queues from the administration console Monitoring Mail Queues page to see if you can fix the mail delivery problem. When you open mail queues, the content of the Deferred, Incoming, Active, Hold, and Corrupt queues at that point in time can be viewed. You can view the number of messages and where they are coming from and going to. For description of these queues, see
“Zimbra MTA Message Queues” .
For each queue, the Summary pane shows a summary of messages by receiver domain, origin IP, sender domain, receiver address, sender address, and for the Deferred queue, by error type. You can select any of the summaries to see detailed envelope information by message in the Messages pane.
|
•
|
Hold, to move all messages in the queue being viewed to the Hold queue. Messages stay in this queue until the administrator moves them.
|
|
•
|
Release, to remove all message from the Hold queue. Messages are moved to the Deferred queue.
|
|
•
|
Requeue all messages in the queue being viewed. Requeuing messages can be used to send messages that were deferred because of a configuration problem that has been fixed. Messages are re-evaluated and earlier penalties are forgotten.
|
|
•
|
Delete all messages in the queue being viewed.
|
The Zimbra MTA, Postfix queue file IDs are reused. If you requeue or delete a message, note the message envelope information, not the queue ID. It is possible that when you refresh the mail queues, the queue ID could be used on a different message.
In addition to moving individual messages in a specific queue, you can flush the server. When you click the Flush button, delivery is immediately attempted for all messages in the Deferred, Incoming and Active queues.
Mailbox quotas apply to email messages, attachments, calendar appointments, tasks, briefcase files, and document notebooks in a user’s account. When an account quota is reached all mail messages are rejected. Users must delete mail from their account to get below their quota limit, or you can increase their quota. This includes emptying their Trash.
You can check mailbox quotas for individual accounts from Server Statistics on the administration console. The Mailbox Quota tab gives you an instant view of the following information for each account:
From a COS or Account, you can configure a quota threshold that, when reached, triggers sending a warning message alerting users that they are about to reach their mailbox quota.
To guard against simple password harvest attacks, a ZCS account authentication password policy can be configured to insure strong passwords and a failed login policy can be set to lockout accounts that fail to log in after the maximum number of attempts.These policies protect against targeted account attacks, but do not provide visibility into dictionary and distributed based attacks.
The zmauditwatch script attempts to detect these more advanced attacks by looking at where the authentication failures are coming from and how frequently they are happening for all accounts on a Zimbra mailbox server and sends an email alert to the administrator’s mailbox.
|
•
|
IP/Account hash check. The default is to send an email alert if 10 authenticating failures from an IP/account combination occur within a 60 second window.
|
|
•
|
Account check. The default is to send an email alert if 15 authentication failures from any IP address occur within a 60 second window. This check attempts to detect a distributed hijack based attack on a single account.
|
|
•
|
IP check. The default is to send an email alert if 20 authentication failures to any account occur within a 60 second window. This check attempts to detect a single host based attack across multiple accounts.
|
|
•
|
Total authentication failure check. The default is to send an email alert if 1000 auth failures from any IP address to any account occurs within 60 seconds. The default should be modified to be 1% of the active accounts on the mailbox server.
|
Configure zimbra_swatch_notice_user with the email address that should receive the alerts
.
The Zimbra Collaboration Suite logs its activities and errors to a combination of system logs through the syslog daemon as well as Zimbra specific logs on the local file system. The logs described below are the primary logs that are used for analysis and troubleshooting.
|
•
|
audit.log. This log contains authentication activity of users and administrators and login failures. In addition, it logs admin activity to be able to track configuration changes.
|
|
•
|
clamd.log. This log contains activity from the antivirus application clamd.
|
|
•
|
freshclam.log. This log contains log information related to the updating of the clamd virus definitions.
|
|
•
|
logger_myslow.log. This slow query log consists of all SQL statements that took more than long_query_time seconds to execute. Note: long_query_time is defined in /opt/zimbra/my.logger.cnf.
|
|
•
|
mailbox.log. This log is a mailboxd log4j server log containing the logs from the mailbox server. This includes the mailbox store, LMTP server, IMAP and POP servers, and Index server. (Note: prior to ZCS 4.5, this log was called /opt/zimbra/log/zimbra.log.)
|
|
•
|
myslow.log. This slow query log consists of all SQL statements from the mailbox server that took more then long_query_time seconds to execute. Note: long_query_time is defined in /opt/zimbra/my.cnf.
|
|
•
|
spamtrain.log. This log contains output from zmtrainsa during regularly scheduled executions from the cron.
|
|
•
|
sync.log. This log contains information about ZCS mobile sync operations.
|
|
•
|
/var/log/zimbra.log. The Zimbra syslog details the activities of the Zimbra MTA (Postfix, amavisd, antispam, antivirus), Logger, Authentication (cyrus-sasl), and Directory (OpenLDAP). By default LDAP activity is logged to Zimbra.log.
|
Zimbra modifies the systems syslog daemon to capture data from the mail and local syslog facility to
/var/log/zimbra.log. This allows syslogd to capture data from several ZCS components including Postfix, Amavis, ClamAV, mailboxd, zmmtaconfig, and logger. The SNMP module uses the data from the log file to generate traps for critical errors. The zmlogger daemon also collects a subset of the data in this file to provide statistics on the utilization of ZCS via the administration console.
By default, mailboxd is configured to log its output to /opt/ZCS/log/mailboxd.log. You can enable mailboxd to take advantage of a centralized syslogd infrastructure by enabling the following either globally or by server
The Zimbra server uses log4j, a Java logging package as the log manager. By default, the Zimbra server has
log4j configured to log to the local file system. You can configure
log4j to direct output to another location. Go to the Log4j website for information about using log4j.
The logging level is set by default to include logs that are generated for INFO, WARNING, ERROR and FATAL. When problems start to occur, you can turn on the DEBUG log level.
When enabling DEBUG, you can specify a specific category to debug. For example, to see debug details for POP activity, you would type
logger.com.zimbra.pop=DEBUG.
The mailbox.log file logs every action taken on the mailbox server, including authentication sessions, LMTP, POP3, and IMAP servers, and Index server. Review the mailbox.log to find information about the health of your server and to help identify problems.
Mailbox.log records valid and invalid login attempts, account activity such as opening email, deleting items, creating items, indexing of new mail, server activities including start and stop. The progress of an activity on the mail server is logged as INFO and if the expected results of the activity fails and errors occurs, an exception is written to the log.
The example below is a record showing that on June 25, 2007, the zimbra server with an IP address of 127.0.0.1 was in the process of deleting backups that were created on Monday, June 18, 2007 at 8 seconds after midnight Pacific Daylight Time (PDT) or older than that date.
If an error occurs during the progress of an activity, a handler exception is added to the end of the basic log record to notify you that an event occurred during the execution of the process that disrupted the normal flow. This signals that some type of error was detected.
Sometimes a stack trace is displayed after the exceptions notification. A stack logs the process in detail. A stack trace is a report of the threads and monitors in the zimbra’s
mailboxd service. This information aids in debugging, as the trace shows where the error occurred. The last few entries in the stack often indicate the origin of the problem. When the
caused by descriptor is included in the log line, this is the root of the error. In the example below, the error was caused by 501, bad address syntax.
at com.example.cs.mailbox. MailServiceException.internal_SEND_FAILURE (MailServiceException.java:412)
at com.example.cs.mailbox. MailServiceException.SEND_ABORTED_ADDRESS_ FAILURE MailServiceException.java:416)
Caused by: com.example.cs.mailbox.MailSender$SafeSendFailedException :501 Bad address syntax
com.sun.mail.smtp.SMTPAddressFailedException: 501 Bad address syntax
at com.sun.mail.smtp.SMTPTransport.rcptTo(SMTPTransport.java:1196)
at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:584)
at javax.mail.Transport.send0(Transport.java:169)
at javax.mail.Transport.send(Transport.java:98)
at com.example.cs.mailbox.MailSender.sendMessage(MailSender.java:409)
at com.example.cs.mailbox.MailSender.sendMimeMessage(MailSender.java:262)
|
The mailbox.log files rotate daily. The mailbox log files are saved in /opt/zimbra/log. Previous mailbox.log file names include the date the file was made. The log without a date is the current log file. You can backup and remove these files.
To review the mailbox.log for errors, search for the email address or the service that is experiencing the problem. Also, search for WARN or ERROR log levels, read the text of the message. When you find the error review the records, tracing the events that happened before the problem was recorded.
When your system crashes, look for the startup message and after finding that message, look for errors before the startup message date. This example shows an out-of-memory error on June 17, 2007.
When you are looking for an error in mail delivery, start by looking for the “LmtpServer” service. This example includes a stack trace report with a
caused by explanation that the recipient address was rejected as the address must be a fully-qualified address.
2007-06-25 10:47:43,008 INFO [LmtpServer-250] [name=bigen@example.com;mid=30;msgid=<1291804360.35481182793659172.JavaMail.root@dogfood.example.com>;] lmtp - rejecting message bigen@example.com: exception occurred
at com.zimbra.cs.mailbox.MailServiceException.internal_SEND_FAILURE (MailServiceException.java:412)
at com.zimbra.cs.mailbox.MailServiceException.SEND_FAILURE(MailServi ceException.java:424)
at com.zimbra.cs.filter.zimbraMailAdapter.executeActions(zimbraMailA dapter.java:286)
at org.apache.jsieve.SieveFactory.evaluate(SieveFactory.java:151)
at com.zimbra.cs.filter.RuleManager.applyRules(RuleManager.java:177)
at com.zimbra.cs.lmtpserver.zimbraLmtpBackend.deliverMessageToLocal Mailboxes(zimbraLmtpBackend.java:325)
at com.zimbra.cs.lmtpserver.zimbraLmtpBackend.deliver(zimbraLmtpBack end.java:140)
at com.zimbra.cs.lmtpserver.LmtpHandler.doDATA(LmtpHandler.java:441)
at com.zimbra.cs.lmtpserver.LmtpHandler.processCommand(LmtpHandler. java:205)
at com.zimbra.cs.tcpserver.ProtocolHandler.processConnection(Protoc olHandler.java:231)
at com.zimbra.cs.tcpserver.ProtocolHandler.run(ProtocolHandler.java :198)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unkn own Source)
at java.lang.Thread.run(Thread.java:619)
|
Caused by: com.zimbra.cs.mailbox.MailSender$SafeSendFailedException: 504 <too>: Recipient address rejected: need fully-qualified address
com.sun.mail.smtp.SMTPAddressFailedException: 504 <too>: Recipient address rejected: need fully-qualified address
at com.sun.mail.smtp.SMTPTransport.rcptTo(SMTPTransport.java:1196)
at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:584)
at javax.mail.Transport.send0(Transport.java:169)
at javax.mail.Transport.send(Transport.java:120)
at com.zimbra.cs.filter.zimbraMailAdapter.executeActions(zimbraMailAdapter.java:281)
|
Mailbox.log logs any successful or unsuccessful login attempts from IMAP, POP3 or ZWC. When you are looking for a login error, start by looking for “Auth.” This example shows that someone from IP address 10.10.131.10 was trying to log in as admin on the Zimbra Web Client, using Firefox 2.0 in a Windows OS. Permission was denied because it was not an admin account.
at com.zimbra.common.service.ServiceException.PERM_DENIED(ServiceExc eption.java:205)
at com.zimbra.cs.service.admin.Auth.handle(Auth.java:103)
|
When you are looking for a log because of an IMAP or POP issue, look for “ImapServer/Pop3Server.” This example shows a fatal IMAP server error occurred while trying to connect siress@example.com.
Zimbra includes an installer package with SNMP monitoring. This package should be run on every server (Zimbra, OpenLDAP, and Postfix) that is part of the Zimbra configuration.
The ZCS error message generates SNMP traps when a service is stopped or is started. You can capture these messages using third-party SNMP monitoring software and direct selected messages to a pager or other alert system.
The MySQL database is automatically checked weekly to verify the health of the database. This check takes about an hour. If any errors are found, a report is sent to the administrator’s account. The report name that runs the MySQL check is
zmbintegrityreport, and the crontab is automatically configured to run this report once a week.
Copyright © 2008 Zimbra Inc.
