Results 1 to 7 of 7

Thread: InnoDB plugin error

  1. #1
    Paladinemishakal is offline Intermediate Member
    Join Date
    Jun 2012
    Posts
    21
    Rep Power
    3

    Default InnoDB plugin error

    Hi All,

    This morning I encounter a problem with my Zimbra email servers which one of the mailstore got an error with the InnoDB plugin and my users informed that they are unable to send emails. My Zimbra is Release 7.2.0_GA_2669.UBUNTU10_64 UBUNTU10_64 FOSS edition.

    I checked on the mailstore server and found that the Mailbox service is stopped and when I tried to restart just the mailbox service, it is not able to.
    Code:
    Host mail05
            logger                  Running
            mailbox                Stopped
                    zmmailboxdctl is not running.
            memcached          Running
            snmp                   Running
            spell                    Running
            stats                   Running
            zmconfigd            Running
    Checking the system memory did not show the system is low on memory:
    Code:
    zimbra@mail05:~$ top
    top - 11:04:34 up 25 days,  4:33,  3 users,  load average: 0.16, 1.09, 1.56
    Tasks: 194 total,   1 running, 193 sleeping,   0 stopped,   0 zombie
    Cpu(s):  0.6%us,  0.1%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    Mem:   8196500k total,  3356120k used,  4840380k free,    11768k buffers
    Swap:  3906552k total,  3902688k used,     3864k free,    93084k cached
    I checked in /opt/zimbra/log/mysql_error.log, I got this error:
    Code:
    130710 10:40:32 mysqld_safe Number of processes running now: 0
    130710 10:40:32 mysqld_safe mysqld restarted
    InnoDB: The InnoDB memory heap is disabled
    InnoDB: Mutexes and rw_locks use GCC atomic builtins
    InnoDB: Compressed tables use zlib 1.2.3.3
    130710 10:40:33  InnoDB: Initializing buffer pool, size = 2.3G
    InnoDB: mmap(2574188544 bytes) failed; errno 12
    130710 10:40:33InnoDB: Fatal error: cannot allocate the memory for the buffer pool
    130710 10:40:33 [ERROR] Plugin 'InnoDB' init function returned error.
    130710 10:40:33 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
    130710 10:40:33 [Note] Event Scheduler: Loaded 0 events
    130710 10:40:33 [Note] /opt/zimbra/mysql/libexec/mysqld: ready for connections.
    Version: '5.1.58-log'  socket: '/opt/zimbra/db/mysql.sock'  port: 7306  Source distribution
    130710 10:52:16 [Note] /opt/zimbra/mysql/libexec/mysqld: Normal shutdown
    
    130710 10:52:16 [Note] Event Scheduler: Purging the queue. 0 events
    130710 10:52:16 [Note] /opt/zimbra/mysql/libexec/mysqld: Shutdown complete
    Then I check in /var/log/syslog and I notice the kernel is terminating the mysql process:
    Code:
    Jul 10 10:57:47 mail05 kernel: [2175982.795204] active_anon:1652360 inactive_anon:323485 isolated_anon:0
    Jul 10 10:57:47 mail05 kernel: [2175982.795205]  active_file:212 inactive_file:110 isolated_file:64
    Jul 10 10:57:47 mail05 kernel: [2175982.795207]  unevictable:0 dirty:6 writeback:7 unstable:0
    Jul 10 10:57:47 mail05 kernel: [2175982.795208]  free:11795 slab_reclaimable:5366 slab_unreclaimable:5429
    Jul 10 10:57:47 mail05 kernel: [2175982.795210]  mapped:213 shmem:25 pagetables:7378 bounce:0
    Jul 10 10:57:47 mail05 kernel: [2175982.795213] Node 0 DMA free:15832kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15268kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:8kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
    Jul 10 10:57:47 mail05 kernel: [2175982.795228] lowmem_reserve[]: 0 3000 8050 8050
    Jul 10 10:57:47 mail05 kernel: [2175982.795234] Node 0 DMA32 free:24340kB min:4272kB low:5340kB high:6408kB active_anon:2318556kB inactive_anon:578716kB active_file:112kB inactive_file:136kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3072096kB mlocked:0kB dirty:0kB writeback:4kB mapped:596kB shmem:0kB slab_reclaimable:7316kB slab_unreclaimable:1644kB kernel_stack:464kB pagetables:5348kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
    Jul 10 10:57:47 mail05 kernel: [2175982.795249] lowmem_reserve[]: 0 0 5050 5050
    Jul 10 10:57:47 mail05 kernel: [2175982.795254] Node 0 Normal free:7008kB min:7192kB low:8988kB high:10788kB active_anon:4290884kB inactive_anon:715224kB active_file:736kB inactive_file:304kB unevictable:0kB isolated(anon):0kB isolated(file):256kB present:5171200kB mlocked:0kB dirty:24kB writeback:24kB mapped:256kB shmem:100kB slab_reclaimable:14148kB slab_unreclaimable:20064kB kernel_stack:4144kB pagetables:24164kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:1862 all_unreclaimable? yes
    Jul 10 10:57:47 mail05 kernel: [2175982.795270] lowmem_reserve[]: 0 0 0 0
    Jul 10 10:57:47 mail05 kernel: [2175982.795275] Node 0 DMA: 2*4kB 2*8kB 2*16kB 3*32kB 3*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15832kB
    Jul 10 10:57:47 mail05 kernel: [2175982.795289] Node 0 DMA32: 288*4kB 169*8kB 186*16kB 215*32kB 97*64kB 17*128kB 7*256kB 0*512kB 2*1024kB 0*2048kB 0*4096kB = 24584kB
    Jul 10 10:57:47 mail05 kernel: [2175982.795302] Node 0 Normal: 888*4kB 5*8kB 0*16kB 7*32kB 6*64kB 7*128kB 2*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 7144kB
    Jul 10 10:57:47 mail05 kernel: [2175982.795315] 12339 total pagecache pages
    Jul 10 10:57:47 mail05 kernel: [2175982.795318] 11896 pages in swap cache
    Jul 10 10:57:47 mail05 kernel: [2175982.795321] Swap cache stats: add 3655879, delete 3643983, find 4050699/4347053
    Jul 10 10:57:47 mail05 kernel: [2175982.795323] Free swap  = 0kB
    Jul 10 10:57:47 mail05 kernel: [2175982.795325] Total swap = 3906552kB
    Jul 10 10:57:47 mail05 kernel: [2175982.814370] 2097136 pages RAM
    Jul 10 10:57:47 mail05 kernel: [2175982.814374] 48075 pages reserved
    Jul 10 10:57:47 mail05 kernel: [2175982.814376] 6659 pages shared
    Jul 10 10:57:47 mail05 kernel: [2175982.814378] 2034160 pages non-shared
    Jul 10 10:57:47 mail05 kernel: [2175982.814382] Out of memory: kill process 2278 (mysqld_safe) score 30210120 or a child
    Jul 10 10:57:47 mail05 kernel: [2175982.846519] Killed process 2471 (mysqld)
    Googling on the internet, I found the following results:
    MySQL InnoDB crash post-mortem - Database Administrators Stack Exchange
    InnoDB: The InnoDB memory heap is disabled | Nix World
    MySQL :: InnoDB Plugin 1.0 for MySQL 5.1 User

    I proceed to add the mentioned option into /opt/zimbra/conf/my.cnf
    # Added to allow InnoDB to use system to allocate memory - 10 Jul 2013
    innodb_use_sys_malloc = 0

    Restart zimbra and monitored the mysql_error.log and it shows the mysql is working fine again.
    Code:
    130710 11:46:27 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
    InnoDB: Mutexes and rw_locks use GCC atomic builtins
    InnoDB: Compressed tables use zlib 1.2.3.3
    130710 11:46:27  InnoDB: Initializing buffer pool, size = 2.3G
    130710 11:46:28  InnoDB: Completed initialization of buffer pool
    130710 11:46:28  InnoDB: highest supported file format is Barracuda.
    InnoDB: Log scan progressed past the checkpoint lsn 33536688489
    130710 11:46:28  InnoDB: Database was not shut down normally!
    InnoDB: Starting crash recovery.
    InnoDB: Reading tablespace information from the .ibd files...
    InnoDB: Restoring possible half-written data pages from the doublewrite
    InnoDB: buffer...
    InnoDB: Doing recovery: scanned up to log sequence number 33537319114
    130710 11:46:42  InnoDB: Starting an apply batch of log records to the database...
    InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
    InnoDB: Apply batch completed
    130710 11:46:49 InnoDB Plugin 1.0.17 started; log sequence number 33537319114
    130710 11:46:49 [Note] Event Scheduler: Loaded 0 events
    130710 11:46:49 [Note] /opt/zimbra/mysql/libexec/mysqld: ready for connections.
    Version: '5.1.58-log'  socket: '/opt/zimbra/db/mysql.sock'  port: 7306  Source distribution
    I want to know if I have applied the correct way to fix this InnoDB plugin issue?
    Is there something else I have missed?

    Regards.

  2. #2
    Paladinemishakal is offline Intermediate Member
    Join Date
    Jun 2012
    Posts
    21
    Rep Power
    3

    Default

    Hi All,

    At around 5pm today, the mysql database was restarted by the kernel again but it is not loading the InnoDB plugin. I need to know how do I resolve this issue.

    I would like to know how do I calculate the memory requirement for mysql with reference to the following on the /opt/zimbra/conf/my.cnf
    Code:
    # (Num mailbox groups * Num tables in each group) + padding
    table_cache = 1200
    
    innodb_buffer_pool_size        = 2517990604
    innodb_log_file_size           = 524288000
    innodb_log_buffer_size         = 8388608
    innodb_file_per_table
    
    # Added to allow InnoDB to use system to allocate memory - 10 Jul 2013 LG
    innodb_use_sys_malloc = 0
    
    # Value is: 200 + max_connections + 2 * table_cache
    innodb_open_files              = 2710
    
    innodb_max_dirty_pages_pct     = 30
    innodb_flush_method            = O_DIRECT
    innodb_flush_log_at_trx_commit = 0
    max_allowed_packet             = 16777216
    Regards.
    Last edited by phoenix; 07-10-2013 at 03:23 AM. Reason: fixed code tag

  3. #3
    phoenix is online now Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,579
    Rep Power
    57

    Default

    The one thing you haven't done is give details about how many users there are on your server nor what type of clients they're using nor how many users are active at any one time. You're also missing (I think) some further information from the my.cnf file.You would seem to have some performance problem as the swap file seems to be fully use and I wouldn't have expected that. Has this problem just started happening? Have you read the wiki article on performance tuning for large deployments? Have you recently done any software updates on this server? Is this server dedicated to ZCS or does it run anything else?
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  4. #4
    Paladinemishakal is offline Intermediate Member
    Join Date
    Jun 2012
    Posts
    21
    Rep Power
    3

    Default

    Hi All,

    I found this document about Performance Tuning Guidelines for Large Deployments for Zimbra and under the MySQL section, it mentioned about setting InnoDB cache setting to about 40% of RAM in the system. This mailstore have about 532 user accounts

    I proceed to made the following changes in /opt/zimbra/conf/my.cnf:
    1. Increase the innodb_buffer_pool_size to 3.2 GB as I have 8 GB of RAM,
    2. Comment the previously set option "innodb_use_sys_malloc".
    Code:
    # (Num mailbox groups * Num tables in each group) + padding
    table_cache = 1200
    
    #innodb_buffer_pool_size        = 2517990604
    innodb_buffer_pool_size        = 3435973840
    innodb_log_file_size           = 524288000
    innodb_log_buffer_size         = 8388608
    innodb_file_per_table
    
    # Added to allow InnoDB to use system to allocate memory - 10 Jul 2013 LG
    #innodb_use_sys_malloc = 0
    
    # Value is: 200 + max_connections + 2 * table_cache
    innodb_open_files              = 2710
    
    innodb_max_dirty_pages_pct     = 30
    innodb_flush_method            = O_DIRECT
    innodb_flush_log_at_trx_commit = 0
    max_allowed_packet             = 16777216
    Restarting Zimbra, I saw in the /opt/zimbra/log/mysql_error.log, the service is able to start:
    Code:
    130710 18:53:04 mysqld_safe mysqld from pid file /opt/zimbra/db/mysql.pid ended
    130710 18:53:56 mysqld_safe Starting mysqld daemon with databases from /opt/zimbra/db/data
    InnoDB: The InnoDB memory heap is disabled
    InnoDB: Mutexes and rw_locks use GCC atomic builtins
    InnoDB: Compressed tables use zlib 1.2.3.3
    130710 18:53:56  InnoDB: Initializing buffer pool, size = 3.2G
    130710 18:53:56  InnoDB: Completed initialization of buffer pool
    130710 18:53:56  InnoDB: highest supported file format is Barracuda.
    130710 18:53:57 InnoDB Plugin 1.0.17 started; log sequence number 33602488948
    130710 18:53:57 [Note] Event Scheduler: Loaded 0 events
    130710 18:53:57 [Note] /opt/zimbra/mysql/libexec/mysqld: ready for connections.
    Version: '5.1.58-log'  socket: '/opt/zimbra/db/mysql.sock'  port: 7306  Source distribution
    The item in bold red is not found when I use the "innodb_use_sys_malloc" and mysql is able to start up.

    Can anyone advise if I am doing the right thing?

  5. #5
    phoenix is online now Zimbra Consultant & Moderator
    Join Date
    Sep 2005
    Location
    Vannes, France
    Posts
    23,579
    Rep Power
    57

    Default

    Quote Originally Posted by Paladinemishakal View Post
    Can anyone advise if I am doing the right thing?
    ...... and yet you still haven't given any information about your configuration nor any answers to the questions I asked, I guess I'm wasting my time.
    Regards


    Bill


    Acompli: A new adventure for Co-Founder KevinH.

  6. #6
    Paladinemishakal is offline Intermediate Member
    Join Date
    Jun 2012
    Posts
    21
    Rep Power
    3

    Default

    Quote Originally Posted by phoenix View Post
    The one thing you haven't done is give details about how many users there are on your server nor what type of clients they're using nor how many users are active at any one time. You're also missing (I think) some further information from the my.cnf file.You would seem to have some performance problem as the swap file seems to be fully use and I wouldn't have expected that. Has this problem just started happening? Have you read the wiki article on performance tuning for large deployments? Have you recently done any software updates on this server? Is this server dedicated to ZCS or does it run anything else?
    1. There are about 532 users on this server and we are using a mix of Zimbra desktops, Thunderbird, iOS mail client and Android mail client.
    2. How do I check how many users are active on this mailstore?
    3. Full Details of my.cnf:
    zimbra@mail05:~/conf$ more my.cnf

    [mysqld]

    basedir = /opt/zimbra/mysql
    datadir = /opt/zimbra/db/data
    socket = /opt/zimbra/db/mysql.sock
    pid-file = /opt/zimbra/db/mysql.pid
    bind-address = localhost
    port = 7306
    user = zimbra
    tmpdir = /opt/zimbra/data/tmp

    external-locking
    slow_query_log = 1
    slow_query_log_file = /opt/zimbra/log/myslow.log
    ignore-builtin-innodb

    plugin-load=innodb=ha_innodb_plugin.so;innodb_trx=ha_inno db_plugin.so;innodb_locks=ha_innodb_plugin.so;inno db_lock_waits=ha_innodb
    _plugin.so;innodb_cmp=ha_innodb_plugin.so;innodb_c mp_reset=ha_innodb_plugin.so;innodb_cmpmem=ha_inno db_plugin.so;innodb_cmpmem_res
    et=ha_innodb_plugin.so

    long-query-time = 1
    log-queries-not-using-indexes

    thread_cache_size = 110
    max_connections = 110

    # We do a lot of writes, query cache turns out to be not useful.
    query_cache_type = 0

    sort_buffer_size = 1048576
    read_buffer_size = 1048576

    # (Num mailbox groups * Num tables in each group) + padding
    table_cache = 1200

    #innodb_buffer_pool_size = 2517990604
    innodb_buffer_pool_size = 3435973840
    innodb_log_file_size = 524288000
    innodb_log_buffer_size = 8388608
    innodb_file_per_table

    # Added to allow InnoDB to use system to allocate memory - 10 Jul 2013 LG
    #innodb_use_sys_malloc = 0

    # Value is: 200 + max_connections + 2 * table_cache
    innodb_open_files = 2710

    innodb_max_dirty_pages_pct = 30
    innodb_flush_method = O_DIRECT
    innodb_flush_log_at_trx_commit = 0
    max_allowed_packet = 16777216

    [mysqld_safe]

    err-log = /opt/zimbra/log/mysqld.log
    pid-file = /opt/zimbra/db/mysql.pid


    4. Just google and read the performance tunning guide and done the changes after you replied
    5. This issue just popped up today.
    6. Start of Jun, just done a "apt-get update and upgrade".
    7. This server is dedicated to ZCS as a mailstore.

  7. #7
    Paladinemishakal is offline Intermediate Member
    Join Date
    Jun 2012
    Posts
    21
    Rep Power
    3

    Default

    This morning, my 2 other mailstores also faced the same issue which I have to increase the innodb_buffer_pool_size to the same as mail05.

    So I need to know what is causing the kernel to kill mysql? Could this be a database issue?

    Quote Originally Posted by Paladinemishakal View Post
    1. There are about 532 users on this server and we are using a mix of Zimbra desktops, Thunderbird, iOS mail client and Android mail client.
    2. How do I check how many users are active on this mailstore?
    3. Full Details of my.cnf:
    zimbra@mail05:~/conf$ more my.cnf

    [mysqld]

    basedir = /opt/zimbra/mysql
    datadir = /opt/zimbra/db/data
    socket = /opt/zimbra/db/mysql.sock
    pid-file = /opt/zimbra/db/mysql.pid
    bind-address = localhost
    port = 7306
    user = zimbra
    tmpdir = /opt/zimbra/data/tmp

    external-locking
    slow_query_log = 1
    slow_query_log_file = /opt/zimbra/log/myslow.log
    ignore-builtin-innodb

    plugin-load=innodb=ha_innodb_plugin.so;innodb_trx=ha_inno db_plugin.so;innodb_locks=ha_innodb_plugin.so;inno db_lock_waits=ha_innodb
    _plugin.so;innodb_cmp=ha_innodb_plugin.so;innodb_c mp_reset=ha_innodb_plugin.so;innodb_cmpmem=ha_inno db_plugin.so;innodb_cmpmem_res
    et=ha_innodb_plugin.so

    long-query-time = 1
    log-queries-not-using-indexes

    thread_cache_size = 110
    max_connections = 110

    # We do a lot of writes, query cache turns out to be not useful.
    query_cache_type = 0

    sort_buffer_size = 1048576
    read_buffer_size = 1048576

    # (Num mailbox groups * Num tables in each group) + padding
    table_cache = 1200

    #innodb_buffer_pool_size = 2517990604
    innodb_buffer_pool_size = 3435973840
    innodb_log_file_size = 524288000
    innodb_log_buffer_size = 8388608
    innodb_file_per_table

    # Added to allow InnoDB to use system to allocate memory - 10 Jul 2013 LG
    #innodb_use_sys_malloc = 0

    # Value is: 200 + max_connections + 2 * table_cache
    innodb_open_files = 2710

    innodb_max_dirty_pages_pct = 30
    innodb_flush_method = O_DIRECT
    innodb_flush_log_at_trx_commit = 0
    max_allowed_packet = 16777216

    [mysqld_safe]

    err-log = /opt/zimbra/log/mysqld.log
    pid-file = /opt/zimbra/db/mysql.pid


    4. Just google and read the performance tunning guide and done the changes after you replied
    5. This issue just popped up today.
    6. Start of Jun, just done a "apt-get update and upgrade".
    7. This server is dedicated to ZCS as a mailstore.

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. zcs 7.2.0 on ubuntu 10 64-bit & Innodb
    By SVD in forum Installation
    Replies: 4
    Last Post: 06-11-2012, 05:17 AM
  2. Total Innodb/MySQL failure
    By rotorboy in forum Administrators
    Replies: 8
    Last Post: 03-19-2012, 08:07 AM
  3. SQL: 'SHOW INNODB STATUS' Error
    By carnold in forum Administrators
    Replies: 3
    Last Post: 01-17-2012, 04:28 AM
  4. Zcs 7 'show Innodb Status'
    By adeelarifbhatti in forum Installation
    Replies: 1
    Last Post: 06-19-2011, 07:54 AM
  5. [SOLVED] More innodb corruption
    By Insanity5902 in forum Migration
    Replies: 5
    Last Post: 01-10-2009, 09:44 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •