ZCS Administrator's Guide, Open Source Edition, 6.0.8
Table of Contents Previous Next Index


Monitoring Zimbra Servers : Log Files

Log Files
The Zimbra Collaboration Suite logs its activities and errors to a combination of system logs through the syslog daemon as well as Zimbra specific logs on the local file system. The logs described below are the primary logs that are used for analysis and troubleshooting.
Local logs containing Zimbra activity are in the /opt/zimbra/log directory.
audit.log. This log contains authentication activity of users and administrators and login failures. In addition, it logs admin activity to be able to track configuration changes.
clamd.log. This log contains activity from the antivirus application clamd.
freshclam.log. This log contains log information related to the updating of the clamd virus definitions.
logger_myslow.log. This slow query log consists of all SQL statements that took more than long_query_time seconds to execute. Note: long_query_time is defined in /opt/zimbra/my.logger.cnf.
mailbox.log. This log is a mailboxd log4j server log containing the logs from the mailbox server. This includes the mailbox store, LMTP server, IMAP and POP servers, and Index server. (Note: prior to ZCS 4.5, this log was called /opt/zimbra/log/zimbra.log.)
myslow.log. This slow query log consists of all SQL statements from the mailbox server that took more then long_query_time seconds to execute. Note: long_query_time is defined in /opt/zimbra/my.cnf.
spamtrain.log. This log contains output from zmtrainsa during regularly scheduled executions from the cron.
sync.log. This log contains information about ZCS mobile sync operations.
Other logs include:
/opt/zimbra/jetty/logs/. This is where Jetty-specific activity is logged.
/opt/zimbra/db/data. <hostname>.err. This is the message store database error log.
/opt/zimbra/logger/db/data. <hostname>.err. This is the Logger database error log.
ZCS activity logged to System syslog
/var/log/zimbra.log. The Zimbra syslog details the activities of the Zimbra MTA (Postfix, amavisd, antispam, antivirus), Logger, Authentication (cyrus-sasl), and Directory (OpenLDAP). By default LDAP activity is logged to Zimbra.log.
Syslog
Zimbra modifies the systems syslog daemon to capture data from the mail and local syslog facility to /var/log/zimbra.log. This allows syslogd to capture data from several ZCS components including Postfix, Amavis, ClamAV, mailboxd, zmmtaconfig, and logger. The SNMP module uses the data from the log file to generate traps for critical errors. The zmlogger daemon also collects a subset of the data in this file to provide statistics on the utilization of ZCS via the administration console.
By default, mailboxd is configured to log its output to /opt/ZCS/log/mailboxd.log. You can enable mailboxd to take advantage of a centralized syslogd infrastructure by enabling the following either globally or by server
zmprov mcf zimbraLogToSysLog True
Using log4j to Configure Logging
The Zimbra server uses log4j, a Java logging package as the log manager. By default, the Zimbra server has log4j configured to log to the local file system. You can configure log4j to direct output to another location. Go to the Log4j website for information about using log4j.
Logging Levels
The logging level is set by default to include logs that are generated for INFO, WARNING, ERROR and FATAL. When problems start to occur, you can turn on the DEBUG log level.
To change the logging levels, edit the log4j properties, log4j.logger.zimbra.
When enabling DEBUG, you can specify a specific category to debug. For example, to see debug details for POP activity, you would type logger.zimbra.pop=DEBUG.
The following categories are pre-defined in log4j:
 
Changes to the log level take affect immediately.
Table 1 zimbra Logging Levels
SNMP Trap?
The FATAL level designates very severe error events that will lead the application to abort or impact a large number of users. For example, being unable to contact the MySQL database.
The ERROR level designates error events that might still allow the application to continue running or impact a single user. For example, a single mailbox having a corrupt index or being unable to delete a message from a mailbox.
The WARN level designates potentially harmful situations but are usually recoverable or can be ignored. For example, user log in failed.
The INFO level designates information messages that highlights the progress of the application, basic transaction-level logging. For example, server start-ups, mailbox creation/deletion, account creation.
Events that would generally be useful to help a customer debug problems.
Reviewing mailbox.log Records
The mailbox.log file logs every action taken on the mailbox server, including authentication sessions, LMTP, POP3, and IMAP servers, and Index server. Review the mailbox.log to find information about the health of your server and to help identify problems.
Mailbox.log records valid and invalid login attempts, account activity such as opening email, deleting items, creating items, indexing of new mail, server activities including start and stop. The progress of an activity on the mail server is logged as INFO and if the expected results of the activity fails and errors occurs, an exception is written to the log.
Note: You can set up logging options for a single account in order to trace account activity for one user without filing up mailbox.log with log messages for unrelated accounts. See : Appendix A Command-Line Utilities, zmprov miscellaneous.
Reading records in the log
The example below is a record showing that on June 25, 2007, the zimbra server with an IP address of 127.0.0.1 was in the process of deleting backups that were created on Monday, June 18, 2007 at 8 seconds after midnight Pacific Daylight Time (PDT) or older than that date.
Note: Component thread number identifies which thread managed by mailboxd is performing the action logged.
Handler Exceptions and Stack Traces
If an error occurs during the progress of an activity, a handler exception is added to the end of the basic log record to notify you that an event occurred during the execution of the process that disrupted the normal flow. This signals that some type of error was detected.
 
007-06-25 00:00:10,379 INFO [btpool0-1064] [name=nriers@example.com; mid=228;ip=72.255.38.207;ua=zimbra Desktop/0.38;] SoapEngine - handler exception
Sometimes a stack trace is displayed after the exceptions notification. A stack logs the process in detail. A stack trace is a report of the threads and monitors in the zimbra’s mailboxd service. This information aids in debugging, as the trace shows where the error occurred. The last few entries in the stack often indicate the origin of the problem. When the caused by descriptor is included in the log line, this is the root of the error. In the example below, the error was caused by 501, bad address syntax.
 
at com.example.cs.mailbox.MailServiceException.internal_SEND_FAILURE (MailServiceException.java:412)
at com.example.cs.mailbox.MailServiceException.SEND_ABORTED_ADDRESS_ FAILURE MailServiceException.java:416)
Caused by: com.example.cs.mailbox.MailSender$SafeSendFailedException :501 Bad address syntax
com.sun.mail.smtp.SMTPAddressFailedException: 501 Bad address syntax
at com.sun.mail.smtp.SMTPTransport.rcptTo(SMTPTransport.java:1196)
at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:584)
at javax.mail.Transport.send0(Transport.java:169)
at javax.mail.Transport.send(Transport.java:98)
at com.example.cs.mailbox.MailSender.sendMessage(MailSender.java:409)
at com.example.cs.mailbox.MailSender.sendMimeMessage(MailSender.java:262)
... 30 more
Mailbox log files
The mailbox.log files rotate daily. The mailbox log files are saved in /opt/zimbra/log. Previous mailbox.log file names include the date the file was made. The log without a date is the current log file. You can backup and remove these files.
mailbox.log examples
To review the mailbox.log for errors, search for the email address or the service that is experiencing the problem. Also, search for WARN or ERROR log levels, read the text of the message. When you find the error review the records, tracing the events that happened before the problem was recorded.
The following are examples of the three areas that can register exceptions, service, account and email.
Service Error - System Crashing
When your system crashes, look for the startup message and after finding that message, look for errors before the startup message date. This example shows an out-of-memory error on June 17, 2007.
 
Look for errors before the startup message.
 
2007-06-17 20:11:34,194 FATAL [btpool0-3335] [name=samd@example.com;aname=abcadmin@example.com;mid=142;ip=66.92.25.194;ua=zimbraConnectorForBES/5.0.207;] system - handler exception
Mail Error - Mail Delivery problem
When you are looking for an error in mail delivery, start by looking for the “LmtpServer” service. This example includes a stack trace report with a caused by explanation that the recipient address was rejected as the address must be a fully-qualified address.
 
2007-06-25 10:47:43,008 INFO [LmtpServer-250] [name=bigen@example.com;mid=30;msgid=<1291804360.35481182793659172.JavaMail.root@dogfood.example.com>;] lmtp - rejecting message bigen@example.com: exception occurred
at com.zimbra.cs.mailbox.MailServiceException.internal_SEND_FAILURE (MailServiceException.java:412)
at com.zimbra.cs.mailbox.MailServiceException.SEND_FAILURE(MailServi ceException.java:424)
at com.zimbra.cs.filter.zimbraMailAdapter.executeActions(zimbraMailA dapter.java:286)
at org.apache.jsieve.SieveFactory.evaluate(SieveFactory.java:151)
at com.zimbra.cs.filter.RuleManager.applyRules(RuleManager.java:177)
at com.zimbra.cs.lmtpserver.zimbraLmtpBackend.deliverMessageToLocal Mailboxes(zimbraLmtpBackend.java:325)
at com.zimbra.cs.lmtpserver.zimbraLmtpBackend.deliver(zimbraLmtpBack end.java:140)
at com.zimbra.cs.lmtpserver.LmtpHandler.doDATA(LmtpHandler.java:441)
at com.zimbra.cs.lmtpserver.LmtpHandler.processCommand(LmtpHandler. java:205)
at com.zimbra.cs.tcpserver.ProtocolHandler.processConnection(Protoc olHandler.java:231)
at com.zimbra.cs.tcpserver.ProtocolHandler.run(ProtocolHandler.java :198)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unkn own Source)
at java.lang.Thread.run(Thread.java:619)
 
Caused by: com.zimbra.cs.mailbox.MailSender$SafeSendFailedException: 504 <too>: Recipient address rejected: need fully-qualified address
com.sun.mail.smtp.SMTPAddressFailedException: 504 <too>: Recipient address rejected: need fully-qualified address
at com.sun.mail.smtp.SMTPTransport.rcptTo(SMTPTransport.java:1196)
at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:584)
at javax.mail.Transport.send0(Transport.java:169)
at javax.mail.Transport.send(Transport.java:120)
at com.zimbra.cs.filter.zimbraMailAdapter.executeActions(zimbraMailAdapter.java:281)
... 10 more
Account Error- Log in error
Mailbox.log logs any successful or unsuccessful login attempts from IMAP, POP3 or ZWC. When you are looking for a login error, start by looking for “Auth.” This example shows that someone from IP address 10.10.131.10 was trying to log in as admin on the Zimbra Web Client, using Firefox 2.0 in a Windows OS. Permission was denied because it was not an admin account.
 
2007-06-25 09:16:11,483 INFO [btpool0-251] [ip=10.10.131.10;ua=zimbraWebClient - FF2.0 (Win);] SoapEngine - handler exception
at com.zimbra.common.service.ServiceException.PERM_DENIED(ServiceExc eption.java:205)
at com.zimbra.cs.service.admin.Auth.handle(Auth.java:103)
Account Errors - IMAP or POP related
When you are looking for a log because of an IMAP or POP issue, look for “ImapServer/Pop3Server.” This example shows a fatal IMAP server error occurred while trying to connect siress@example.com.
 
mailbox.log.2007-06-19:2007-06-19 15:33:56,832 FATAL [ImapServer-2444] [name=sires@example.com;ip=127.0.0.1;] system - Fatal error occurred while handling connection

Monitoring Zimbra Servers : Log Files

Table of Contents Previous Next Index
ZCS Administrator's Guide, Open Source Edition, 6.0.8
Copyright © 2010 Zimbra Inc.