Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-6970

MariaDB 10.0.13 crash while using partitioning

    Details

    • Type: Bug
    • Status: Stalled
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 10.0.13, 10.0
    • Fix Version/s: 10.0
    • Component/s: Partitioning, Query Cache
    • Labels:
      None
    • Environment:
      Oracle Linux 6.5 2.6.32-431.29.2.el6.x86_64
      Server version: 10.0.13-MariaDB-wsrep-log MariaDB Server, wsrep_25.10.r4123
    • Sprint:
      10.0.20

      Description

      141029  4:00:04 [ERROR] mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
      
      To report this bug, see http://kb.askmonty.org/en/reporting-bugs
      
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,.
      something is definitely wrong and this may fail.
      
      Server version: 10.0.13-MariaDB-wsrep-log
      key_buffer_size=2097152
      read_buffer_size=2097152
      max_used_connections=513
      max_threads=514
      thread_count=105
      It is possible that mysqld could use up to.
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2118249 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
      
      Thread pointer: 0x0x7f8c75a70008
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0x7f8b15707ce0 thread_stack 0x48000
      /usr/sbin/mysqld(my_print_stacktrace+0x2b)[0xb9541b]
      /usr/sbin/mysqld(handle_fatal_signal+0x398)[0x744b48]
      /lib64/libpthread.so.0[0x379e80f710]
      /usr/sbin/mysqld[0x94bc7e]
      /usr/sbin/mysqld[0x8a25b7]
      /usr/sbin/mysqld(_ZN7handler13ha_index_nextEPh+0x10c)[0x74987c]
      /usr/sbin/mysqld(_ZN7handler15read_range_nextEv+0x29)[0x74a259]
      /usr/sbin/mysqld[0xb6ffb5]
      /usr/sbin/mysqld(_ZN7handler21multi_range_read_nextEPPv+0xb2)[0x6ca3a2]
      /usr/sbin/mysqld(_ZN18QUICK_RANGE_SELECT8get_nextEv+0x52)[0x815632]
      /usr/sbin/mysqld[0x8331d8]
      /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x1b2)[0x602232]
      /usr/sbin/mysqld[0x61931d]
      /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0x6da)[0x62be2a]
      /usr/sbin/mysqld(_ZN4JOIN4execEv+0x11)[0x62e001]
      /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x1dd)[0x62abcd]
      /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x28d)[0x62e35d]
      /usr/sbin/mysqld[0x5d3329]
      /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x543e)[0x5debee]
      /usr/sbin/mysqld[0x5e1752]
      /usr/sbin/mysqld[0x5e210b]
      /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x16d0)[0x5e3c30]
      /usr/sbin/mysqld(_Z10do_commandP3THD+0x132)[0x5e4402]
      /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x54b)[0x6a31cb]
      /usr/sbin/mysqld(handle_one_connection+0x42)[0x6a32c2]
      /lib64/libpthread.so.0[0x379e8079d1]
      /lib64/libc.so.6(clone+0x6d)[0x379e4e886d]
      Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_
      
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      141029 04:00:04 mysqld_safe Number of processes running now: 0
      141029 04:00:04 mysqld_safe WSREP: not restarting wsrep node automatically
      141029 04:00:04 mysqld_safe mysqld from pid file /nmc/mysql/mysql.pid ended
      

        Gliffy Diagrams

          Attachments

          1. 02122014.sql
            659 kB
          2. 02122014.txt
            4 kB
          3. dump.sql
            981 kB
          4. log_event.txt
            9 kB
          5. mdev6464-my.cnf.single
            3 kB
          6. mdev6464-schema.sql
            104 kB
          7. mysql.err
            6 kB
          8. MySQLCurrentSettings.txt
            15 kB
          9. neominder_sqlcrashlog.txt
            4 kB
          10. procedures.txt
            7 kB
          11. show_create.txt
            3 kB
          12. show_index.txt
            1.0 kB
          13. show_status.txt
            1 kB

            Issue Links

              Activity

              Hide
              neominder Thomas Daugherty added a comment - - edited

              Just wanted to comment that I'm having this same issue. Stored procedures to drop/create partitions runs once per day. A few times a week a node will crash within a minute of the procedure running. This is a database for a Zabbix install, I'm using the stored procedures referenced above. Please let me know if there are any data you'd like me to submit.

              Edit: Using 10.0.16

              Show
              neominder Thomas Daugherty added a comment - - edited Just wanted to comment that I'm having this same issue. Stored procedures to drop/create partitions runs once per day. A few times a week a node will crash within a minute of the procedure running. This is a database for a Zabbix install, I'm using the stored procedures referenced above. Please let me know if there are any data you'd like me to submit. Edit: Using 10.0.16
              Hide
              jgilberAZ Jeff Gilbertson added a comment -

              Just a comment to say that I'm having the issue, too, exactly as described above.

              Show
              jgilberAZ Jeff Gilbertson added a comment - Just a comment to say that I'm having the issue, too, exactly as described above.
              Hide
              ddb0faz DD Brown added a comment -

              This seems to be coming more of an issue in our environment. What started out as once every couple of months seems to be increasing in frequency. It is now happening several times a week.

              Show
              ddb0faz DD Brown added a comment - This seems to be coming more of an issue in our environment. What started out as once every couple of months seems to be increasing in frequency. It is now happening several times a week.
              Hide
              neominder Thomas Daugherty added a comment -

              Elena Stepanova and Vicentiu Ciorbaru,

              Would it be possible to get this bug looked at again? It's really causing havoc for us. Its happening twice a week or more right after the create/drop partition stored procedure runs.

              neominder_sqlcrashlog.txt

              Show
              neominder Thomas Daugherty added a comment - Elena Stepanova and Vicentiu Ciorbaru , Would it be possible to get this bug looked at again? It's really causing havoc for us. Its happening twice a week or more right after the create/drop partition stored procedure runs. neominder_sqlcrashlog.txt
              Hide
              eugeno Eugen O added a comment -

              Hi,
              I've hit the same situation, stack trace looks like this :

              150827  5:28:54 [ERROR] mysqld got signal 11 ;
              This could be because you hit a bug. It is also possible that this binary
              or one of the libraries it was linked against is corrupt, improperly built,
              or misconfigured. This error can also be caused by malfunctioning hardware.
              
              To report this bug, see http://kb.askmonty.org/en/reporting-bugs
              
              We will try our best to scrape up some info that will hopefully help
              diagnose the problem, but since we have already crashed,
              something is definitely wrong and this may fail.
              
              Server version: 10.0.20-MariaDB-log
              key_buffer_size=536870912
              read_buffer_size=131072
              max_used_connections=62
              max_threads=1502
              thread_count=34
              It is possible that mysqld could use up to
              key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3822319 K  bytes of memory
              Hope that's ok; if not, decrease some variables in the equation.
              
              Thread pointer: 0x0x19a89bf8
              Attempting backtrace. You can use the following information to find out
              where mysqld died. If you see no messages after this, something went
              terribly wrong...
              stack_bottom = 0x7f8e7a7dcd78 thread_stack 0x48000
              mysys/stacktrace.c:247(my_print_stacktrace)[0xb81c6e]
              sql/signal_handler.cc:153(handle_fatal_signal)[0x726cab]
              /lib64/libpthread.so.0(+0xf710)[0x7f8eafa8d710]
              sql/sql_plist.h:128(I_P_List<MDL_ticket, I_P_List_adapter<MDL_ticket, &(MDL_ticket::next_in_lock), &(MDL_ticket::prev_in_lock)>, I_P_List_null_counter, I_P_List_fast_push_back<MDL_ticket> >::remove(MDL_ticket*))[0x68c800]
              sql/sql_plist.h:126(I_P_List<MDL_ticket, I_P_List_adapter<MDL_ticket, &(MDL_ticket::next_in_context), &(MDL_ticket::prev_in_context)>, I_P_List_null_counter, I_P_List_no_push_back<MDL_ticket> >::remove(MDL_ticket*))[0x68c968]
              sql/sql_parse.cc:5196(mysql_execute_command(THD*))[0x5acb05]
              sql/sql_parse.cc:6529(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x5b325d]
              sql/sql_parse.cc:1310(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x5b4f49]
              sql/sql_connect.cc:1378(do_handle_one_connection(THD*))[0x68218e]
              sql/sql_connect.cc:1295(handle_one_connection)[0x682262]
              /lib64/libpthread.so.0(+0x79d1)[0x7f8eafa859d1]
              /lib64/libc.so.6(clone+0x6d)[0x7f8eae19f8fd]
              
              Trying to get some variables.
              Some pointers may be invalid and cause the dump to abort.
              Query (0x7f8b14004c60): is an invalid pointer
              Connection ID (thread ID): 1357300
              
              Status: NOT_KILLED
              
              Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
              
              The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
              information that should help you find out what is causing the crash.
              150827 05:28:55 mysqld_safe Number of processes running now: 0
              150827 05:28:55 mysqld_safe mysqld restarted
              150827  5:28:56 [Note] /usr/sbin/mysqld (mysqld 10.0.20-MariaDB-log) starting as process 25031 ...
              150827  5:28:56 [Note] InnoDB: Using mutexes to ref count buffer pool pages
              150827  5:28:56 [Note] InnoDB: The InnoDB memory heap is disabled
              150827  5:28:56 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
              150827  5:28:56 [Note] InnoDB: Memory barrier is not used
              150827  5:28:56 [Note] InnoDB: Compressed tables use zlib 1.2.3
              150827  5:28:56 [Note] InnoDB: Using Linux native AIO
              150827  5:28:56 [Note] InnoDB: Using CPU crc32 instructions
              150827  5:28:56 [Note] InnoDB: Initializing buffer pool, size = 12.0G
              150827  5:28:56 [Note] InnoDB: Completed initialization of buffer pool
              150827  5:28:57 [Note] InnoDB: Highest supported file format is Barracuda.
              150827  5:28:57 [Note] InnoDB: Log scan progressed past the checkpoint lsn 114661609834
              150827  5:28:57 [Note] InnoDB: Database was not shutdown normally!
              150827  5:28:57 [Note] InnoDB: Starting crash recovery.
              150827  5:28:57 [Note] InnoDB: Reading tablespace information from the .ibd files...
              150827  5:28:58 [Note] InnoDB: Restoring possible half-written data pages
              150827  5:28:58 [Note] InnoDB: from the doublewrite buffer...
              InnoDB: Doing recovery: scanned up to log sequence number 114661791274
              150827  5:28:59 [Note] InnoDB: Starting an apply batch of log records to the database...
              InnoDB: Progress in percent: 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
              InnoDB: Apply batch completed
              150827  5:29:00 [Note] InnoDB: 128 rollback segment(s) are active.
              150827  5:29:00 [Note] InnoDB: Waiting for purge to start
              150827  5:29:00 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.24-72.2 started; log sequence number 114661791274
              150827  5:29:00 [Note] Plugin 'FEEDBACK' is disabled.
              150827  5:29:00 [Note] Server socket created on IP: '::'.
              150827  5:29:00 [ERROR] mysqld: Table './mysql/event' is marked as crashed and should be repaired
              150827  5:29:00 [Warning] Checking table:   './mysql/event'
              150827  5:29:00 [ERROR] mysql.event: 1 client is using or hasn't closed the table properly
              150827  5:29:00 [Note] Event Scheduler: Loaded 1 event
              150827  5:29:00 [Note] Event Scheduler: scheduler thread started with id 1
              150827  5:29:00 [Note] /usr/sbin/mysqld: ready for connections.
              
              

              At that hour partitioning was occuring. Let me know if anything else is needed.

              Show
              eugeno Eugen O added a comment - Hi, I've hit the same situation, stack trace looks like this : 150827 5:28:54 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http: //kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.20-MariaDB-log key_buffer_size=536870912 read_buffer_size=131072 max_used_connections=62 max_threads=1502 thread_count=34 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3822319 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x19a89bf8 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this , something went terribly wrong... stack_bottom = 0x7f8e7a7dcd78 thread_stack 0x48000 mysys/stacktrace.c:247(my_print_stacktrace)[0xb81c6e] sql/signal_handler.cc:153(handle_fatal_signal)[0x726cab] /lib64/libpthread.so.0(+0xf710)[0x7f8eafa8d710] sql/sql_plist.h:128(I_P_List<MDL_ticket, I_P_List_adapter<MDL_ticket, &(MDL_ticket::next_in_lock), &(MDL_ticket::prev_in_lock)>, I_P_List_null_counter, I_P_List_fast_push_back<MDL_ticket> >::remove(MDL_ticket*))[0x68c800] sql/sql_plist.h:126(I_P_List<MDL_ticket, I_P_List_adapter<MDL_ticket, &(MDL_ticket::next_in_context), &(MDL_ticket::prev_in_context)>, I_P_List_null_counter, I_P_List_no_push_back<MDL_ticket> >::remove(MDL_ticket*))[0x68c968] sql/sql_parse.cc:5196(mysql_execute_command(THD*))[0x5acb05] sql/sql_parse.cc:6529(mysql_parse(THD*, char *, unsigned int , Parser_state*))[0x5b325d] sql/sql_parse.cc:1310(dispatch_command(enum_server_command, THD*, char *, unsigned int ))[0x5b4f49] sql/sql_connect.cc:1378(do_handle_one_connection(THD*))[0x68218e] sql/sql_connect.cc:1295(handle_one_connection)[0x682262] /lib64/libpthread.so.0(+0x79d1)[0x7f8eafa859d1] /lib64/libc.so.6(clone+0x6d)[0x7f8eae19f8fd] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7f8b14004c60): is an invalid pointer Connection ID (thread ID): 1357300 Status: NOT_KILLED Optimizer switch : index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http: //dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 150827 05:28:55 mysqld_safe Number of processes running now: 0 150827 05:28:55 mysqld_safe mysqld restarted 150827 5:28:56 [Note] /usr/sbin/mysqld (mysqld 10.0.20-MariaDB-log) starting as process 25031 ... 150827 5:28:56 [Note] InnoDB: Using mutexes to ref count buffer pool pages 150827 5:28:56 [Note] InnoDB: The InnoDB memory heap is disabled 150827 5:28:56 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 150827 5:28:56 [Note] InnoDB: Memory barrier is not used 150827 5:28:56 [Note] InnoDB: Compressed tables use zlib 1.2.3 150827 5:28:56 [Note] InnoDB: Using Linux native AIO 150827 5:28:56 [Note] InnoDB: Using CPU crc32 instructions 150827 5:28:56 [Note] InnoDB: Initializing buffer pool, size = 12.0G 150827 5:28:56 [Note] InnoDB: Completed initialization of buffer pool 150827 5:28:57 [Note] InnoDB: Highest supported file format is Barracuda. 150827 5:28:57 [Note] InnoDB: Log scan progressed past the checkpoint lsn 114661609834 150827 5:28:57 [Note] InnoDB: Database was not shutdown normally! 150827 5:28:57 [Note] InnoDB: Starting crash recovery. 150827 5:28:57 [Note] InnoDB: Reading tablespace information from the .ibd files... 150827 5:28:58 [Note] InnoDB: Restoring possible half-written data pages 150827 5:28:58 [Note] InnoDB: from the doublewrite buffer... InnoDB: Doing recovery: scanned up to log sequence number 114661791274 150827 5:28:59 [Note] InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percent: 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed 150827 5:29:00 [Note] InnoDB: 128 rollback segment(s) are active. 150827 5:29:00 [Note] InnoDB: Waiting for purge to start 150827 5:29:00 [Note] InnoDB: Percona XtraDB (http: //www.percona.com) 5.6.24-72.2 started; log sequence number 114661791274 150827 5:29:00 [Note] Plugin 'FEEDBACK' is disabled. 150827 5:29:00 [Note] Server socket created on IP: '::'. 150827 5:29:00 [ERROR] mysqld: Table './mysql/event' is marked as crashed and should be repaired 150827 5:29:00 [Warning] Checking table: './mysql/event' 150827 5:29:00 [ERROR] mysql.event: 1 client is using or hasn't closed the table properly 150827 5:29:00 [Note] Event Scheduler: Loaded 1 event 150827 5:29:00 [Note] Event Scheduler: scheduler thread started with id 1 150827 5:29:00 [Note] /usr/sbin/mysqld: ready for connections. At that hour partitioning was occuring. Let me know if anything else is needed.

                People

                • Assignee:
                  cvicentiu Vicentiu Ciorbaru
                  Reporter:
                  blacksmith Sergii
                • Votes:
                  5 Vote for this issue
                  Watchers:
                  6 Start watching this issue

                  Dates

                  • Created:
                    Updated:

                    Agile