Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-5594

If TOKUDB crashes, no other engine starts, MariaDB does not start !

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 10.0.7
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Environment:
      Ubuntu 12.4 LTS
      MariaDB 10.0.7

      Description

      If TokuDB has at start crash you can not get MariaDB
      up nor reapaisr TokuDB to get MariaDB up.

      MariaDB could go up whit all other engines even tokuDB
      or what so ever dose not start.

      What I tried was to copy /var/lib/mysql and /media/SDD1/tokudb (tokudb database at sdd) to replication server /var/lib/mysql and /media/SDD2/tokudb (tokudb database at sdd, name is difrent at every disk sdd1, sdd2,... ext)

      root@mpi1:/media/SDD1/tokudb# /opt/mariadb-10.0.7/bin/mysqld
      140130 18:00:04 [Warning] option 'table_definition_cache': unsigned value 100 adjusted to 400
      2014-01-30 18:00:04 7f33b133b780 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.
      140130 18:00:04 [Note] InnoDB: The InnoDB memory heap is disabled
      140130 18:00:04 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      140130 18:00:04 [Note] InnoDB: Compressed tables use zlib 1.2.3.4
      140130 18:00:04 [Note] InnoDB: CPU does not support crc32 instructions
      140130 18:00:04 [Note] InnoDB: Initializing buffer pool, size = 8.0M
      140130 18:00:04 [Note] InnoDB: Completed initialization of buffer pool
      140130 18:00:04 [Note] InnoDB: Highest supported file format is Barracuda.
      140130 18:00:04 [Note] InnoDB: 128 rollback segment(s) are active.
      140130 18:00:04 [Note] InnoDB: Waiting for purge to start
      140130 18:00:04 [Note] InnoDB: 5.6.10 started; log sequence number 7062765079
      140130 18:00:04 [Note] CONNECT: connect.ini=/var/lib/mysql/connect.ini
      profile file /var/lib/mysql/connect.ini not found
      Thu Jan 30 18:00:04 2014 Tokudb recovery starting in env /var/lib/mysql/
      Thu Jan 30 18:00:04 2014 Tokudb recovery scanning backward from 17775
      Thu Jan 30 18:00:04 2014 Tokudb recovery bw_end_checkpoint at 17775 timestamp 1391095211182665 xid 17771 (bw_newer)
      Thu Jan 30 18:00:04 2014 Tokudb recovery bw_begin_checkpoint at 17771 timestamp 1391095211182654 (bw_between)
      Thu Jan 30 18:00:04 2014 Tokudb recovery turning around at begin checkpoint 17771 time 11
      Thu Jan 30 18:00:04 2014 Tokudb recovery starts scanning forward to 17775 from 17771 left 4 (fw_between)
      /mpi3/S3/mariadb-10.0.7/storage/tokudb/ft-index/ft/ft-serialize.cc:712 toku_deserialize_ft_from: Assertion `!((r0==0 && checkpoint_lsn_0.lsn > max_acceptable_lsn.lsn) && (r1==0 && checkpoint_lsn_1.lsn > max_acceptable_lsn.lsn))' failed (errno=2)
      : No such file or directory
      Backtrace: (Note: toku_do_assert=0x0x7f33946d7120)
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0x9c718)[0x7f33946d4718]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0x9c8a0)[0x7f33946d48a0]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z24toku_deserialize_ft_fromi10__toku_lsnPP2ft+0x37b)[0x7f33947090db]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z35toku_read_ft_and_store_in_cachefileP9ft_handleP9cachefile10__toku_lsnPP2ft+0x90)[0x7f3394709210]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0xb8c4e)[0x7f33946f0c4e]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z28toku_ft_handle_open_recoveryP9ft_handlePKciiP10cachetableP7tokutxn7FILENUM10__toku_lsn+0x28)[0x7f33946f1178]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z14tokudb_recoverP13__toku_db_envPFvS0_P7tokutxnEPFvS0_P10cachetableEP10tokuloggerPKcSC_PFiP9__toku_dbPK10__toku_dbtSH_EPFiSE_SH_SH_SH_PFvSH_PvESK_EPFiSE_SE_P9DBT_ARRAYSQ_SH_SH_EPFiSE_SE_SQ_SH_SH_Em+0x2193)[0x7f33946fc2a3]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0xc5d09)[0x7f33946fdd09]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0x47162)[0x7f339467f162]
      /opt/mariadb-10.0.7/bin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x5b)[0x6dff9b]
      /opt/mariadb-10.0.7/bin/mysqld[0x5b9170]
      /opt/mariadb-10.0.7/bin/mysqld(_Z11plugin_initPiPPci+0x64b)[0x5bd79b]
      /opt/mariadb-10.0.7/bin/mysqld[0x53b775]
      /opt/mariadb-10.0.7/bin/mysqld(_Z11mysqld_mainiPPc+0x3da)[0x53df7a]
      /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f33af6e176d]
      /opt/mariadb-10.0.7/bin/mysqld[0x534279]
      Engine status function not available
      Memory usage:
      Arena 0:
      system bytes     =          0
      in use bytes     =          0
      Total (incl. mmap):
      system bytes     =          0
      in use bytes     =          0
      max mmap regions =          0
      max mmap bytes   =          0
      140130 18:00:04 [ERROR] mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
      
      To report this bug, see http://kb.askmonty.org/en/reporting-bugs
      
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed, 
      something is definitely wrong and this may fail.
      
      Server version: 10.0.7-MariaDB-log
      key_buffer_size=16777216
      read_buffer_size=1048576
      max_used_connections=0
      max_threads=102
      thread_count=0
      It is possible that mysqld could use up to 
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 227356 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
      
      Thread pointer: 0x0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x0 thread_stack 0x40000
      mysys/stacktrace.c:247(my_print_stacktrace)[0xa7b526]
      sql/signal_handler.cc:153(handle_fatal_signal)[0x6ddd88]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f33b08dfcb0]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x35)[0x7f33af6f6425]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x17b)[0x7f33af6f9b8b]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0x9c81b)[0x7f33946d481b]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0x9c8a0)[0x7f33946d48a0]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z24toku_deserialize_ft_fromi10__toku_lsnPP2ft+0x37b)[0x7f33947090db]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z35toku_read_ft_and_store_in_cachefileP9ft_handleP9cachefile10__toku_lsnPP2ft+0x90)[0x7f3394709210]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0xb8c4e)[0x7f33946f0c4e]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z28toku_ft_handle_open_recoveryP9ft_handlePKciiP10cachetableP7tokutxn7FILENUM10__toku_lsn+0x28)[0x7f33946f1178]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(_Z14tokudb_recoverP13__toku_db_envPFvS0_P7tokutxnEPFvS0_P10cachetableEP10tokuloggerPKcSC_PFiP9__toku_dbPK10__toku_dbtSH_EPFiSE_SH_SH_SH_PFvSH_PvESK_EPFiSE_SE_P9DBT_ARRAYSQ_SH_SH_EPFiSE_SE_SQ_SH_SH_Em+0x2193)[0x7f33946fc2a3]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0xc5d09)[0x7f33946fdd09]
      /opt/mariadb-10.0.7/bin/lib/plugin/ha_tokudb.so(+0x47162)[0x7f339467f162]
      sql/handler.cc:497(ha_initialize_handlerton(st_plugin_int*))[0x6dff9b]
      sql/sql_plugin.cc:1358(plugin_initialize)[0x5b9170]
      sql/sql_plugin.cc:1650(plugin_init(int*, char**, int))[0x5bd79b]
      sql/mysqld.cc:4747(init_server_components)[0x53b775]
      sql/mysqld.cc:5344(mysqld_main(int, char**))[0x53df7a]
      /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f33af6e176d]
      /opt/mariadb-10.0.7/bin/mysqld[0x534279]
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      root@mpi1:/media/SDD1/tokudb# 
      

        Gliffy Diagrams

          Attachments

            Activity

            Hide
            serg Sergei Golubchik added a comment -

            There are two issues here.

            1. TokuDB crash. TokuDB developers will comment on that.

            2. MariaDB cannot continue. Currently plugins run in the server address space. To isolate plugin crashes, MariaDB would need to run every plugin in a sandbox, so that a plugin failure wouldn't affect the whole server. We currently have no plans of doing it, it would be a big a complex feature, and most probably it would negatively impact the performance of the server.

            Show
            serg Sergei Golubchik added a comment - There are two issues here. 1. TokuDB crash. TokuDB developers will comment on that. 2. MariaDB cannot continue. Currently plugins run in the server address space. To isolate plugin crashes, MariaDB would need to run every plugin in a sandbox, so that a plugin failure wouldn't affect the whole server. We currently have no plans of doing it, it would be a big a complex feature, and most probably it would negatively impact the performance of the server.
            Hide
            serg Sergei Golubchik added a comment -

            Rich Prohaska, could you please comment on the TokuDB plugin crash above?

            Show
            serg Sergei Golubchik added a comment - Rich Prohaska , could you please comment on the TokuDB plugin crash above?
            Hide
            boy007 Joni-Pekka Kurronen added a comment -

            Reason for crash, how I got start up whitout crash:

            before locking tables and get master status
            I stopped slave and reseted it.

            After that I made rsync copy. Directory names exaltly
            like master for tokudb.

            Then I could start whitout errors.

            Show
            boy007 Joni-Pekka Kurronen added a comment - Reason for crash, how I got start up whitout crash: before locking tables and get master status I stopped slave and reseted it. After that I made rsync copy. Directory names exaltly like master for tokudb. Then I could start whitout errors.
            Hide
            serg Sergei Golubchik added a comment -

            The bug as reported "if tokudb crashes, it takes the whole mysqld process down" will not be fixed anytime soon.

            Show
            serg Sergei Golubchik added a comment - The bug as reported "if tokudb crashes, it takes the whole mysqld process down" will not be fixed anytime soon.

              People

              • Assignee:
                Unassigned
                Reporter:
                boy007 Joni-Pekka Kurronen
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved: