Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-8333

mysqld got signal 11

        Details

        • Type: Bug
        • Status: Closed
        • Priority: Critical
        • Resolution: Fixed
        • Affects Version/s: 10.1.5
        • Fix Version/s: 10.1.6
        • Component/s: OTHER
        • Labels:
          None
        • Environment:

          Description

          MariaDB is running as as slave, currently with very little activity.
          At the time of the crash there was a script running doing a "check table" on all tables in all databases. There was no other client connected at this time.
          The server failure cannot be repeted. Running th script a second and third time was fine.
          Server crashed with message:

          150618  1:50:07 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.5-MariaDB-log
key_buffer_size=67108864
read_buffer_size=131072
max_used_connections=5
max_threads=502
thread_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1168091 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x7fef8fefb008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fef90491e58 thread_stack 0x48000
mysys/stacktrace.c:247(my_print_stacktrace)[0x7fef931dcbf9]
sql/signal_handler.cc:153(handle_fatal_signal)[0x7fef92def98d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7fef924498d0]
sql/sql_plist.h:181(I_P_List_iterator)[0x7fef92da6360]
mysys/lf_hash.c:125(lfind)[0x7fef931e3a5c]
include/my_atomic.h:281(my_atomic_storeptr)[0x7fef931e4459]
sql/table_cache.cc:1084(tdc_iterate(THD*, char (*)(void*, void*), void*, bool))[0x7fef92da7fb3]
sql/table_cache.cc:267(tc_add_table(THD*, TABLE*))[0x7fef92da833a]
sql/sql_base.cc:2572(open_table(THD*, TABLE_LIST*, Open_table_context*))[0x7fef92c42f0d]
sql/sql_base.cc:4019(open_and_process_table)[0x7fef92c46557]
sql/sql_base.cc:5184(open_and_lock_tables(THD*, DDL_options_st const&, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*))[0x7fef92c46c1d]
sql/sql_admin.cc:446(mysql_admin_table)[0x7fef92d576d6]
sql/sql_admin.cc:1210(Sql_cmd_check_table::execute(THD*))[0x7fef92d59046]
sql/sql_parse.cc:5573(mysql_execute_command(THD*))[0x7fef92c839fc]
sql/sql_parse.cc:7173(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x7fef92c8a334]
sql/sql_parse.cc:1470(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x7fef92c8cb80]
sql/sql_parse.cc:1095(do_command(THD*))[0x7fef92c8d29b]
sql/sql_connect.cc:1347(do_handle_one_connection(THD*))[0x7fef92d4c80c]
sql/sql_connect.cc:1260(handle_one_connection)[0x7fef92d4c9e7]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7fef924420a4]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fef9085c04d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7fef5eb49020): is an invalid pointer
Connection ID (thread ID): 17307
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_conditio

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
150618 01:50:07 mysqld_safe Number of processes running now: 0
150618 01:50:07 mysqld_safe mysqld restarted
2015-06-18  1:50:07 139723066570624 [Note] /usr/local/mysql/bin/mysqld (mysqld 10.1.5-MariaDB-log) starting as process 16564 ...

            Gliffy Diagrams

              Attachments

                Issue Links

                relates to

                Bug - A problem which impairs or prevents the functions of the product. MDEV-8339 Server crash during table cache eviction

                • Major - Major loss of function.
                • Closed

                Bug - A problem which impairs or prevents the functions of the product. MDEV-8370 mysqld got signal 11 with aggressive parallel replication

                • Major - Major loss of function.
                • Closed

                  Activity

                  Ascending order - Click to sort in descending order
                  Hide
                  Permalink
                  elenst Elena Stepanova added a comment -

                  Thanks for the report.

                  Here is a variation of the concurrent test from MDEV-8339, but adjust for CHECK TABLE:

                  RQG grammar
                  query_init:
	CREATE TABLE IF NOT EXISTS t1 (`pk` INT AUTO_INCREMENT PRIMARY KEY, `c` VARCHAR(32), KEY(`c`)) ENGINE=MyISAM; CREATE TABLE IF NOT EXISTS t2 LIKE t1; CREATE TABLE IF NOT EXISTS t3 LIKE t1; CREATE TABLE IF NOT EXISTS t4 LIKE t1;

query:
	INSERT INTO { 't'.(int(rand(500))+5) } (`c`) VALUES ( _english ), ( _english ), ( _english ), ( _english ), ( _english ) |
	DELETE FROM { 't'.(int(rand(500))+5) } WHERE `c` = _english ORDER BY `pk` LIMIT _digit | 
	CHECK TABLE tbl;

tbl:
	t1 | t2 | t3 | t4;

                  The command line is the same.

                  Stack trace from debug build of 10.1 commit 93fc04ff1dc613a9ad75ccc57988a6b57b94b6cb
                  # 2015-06-19T22:43:05 [18731] #3  <signal handler called>
# 2015-06-19T22:43:05 [18731] #4  0x00007fc5b55e4af9 in TDC_element::free_tables_back (this=0x7fc58fcf1810) at 10.1/sql/table_cache.h:110
# 2015-06-19T22:43:05 [18731] #5  0x00007fc5b55e259f in tc_add_table_callback (element=0x7fc58fcf1810, arg=0x7fc5b4dfc550) at 10.1/sql/table_cache.cc:237
# 2015-06-19T22:43:05 [18731] #6  0x00007fc5b5c74d2d in lfind (head=0x7fc58f8203b0, cs=0x0, hashnr=0, key=0x7fc5b4dfc550 "test", keylen=0, cursor=0x7fc5b4dfc3d0, pins=0x7fc5b215f180, callback=0x7fc5b55e2565 <tc_add_table_callback(TDC_element*, tc_add_table_arg*)>) at 10.1/mysys/lf_hash.c:125
# 2015-06-19T22:43:05 [18731] #7  0x00007fc5b5c759a4 in lf_hash_iterate (hash=0x7fc5b671b040, pins=0x7fc5b215f180, action=0x7fc5b55e2565 <tc_add_table_callback(TDC_element*, tc_add_table_arg*)>, argument=0x7fc5b4dfc550) at 10.1/mysys/lf_hash.c:517
# 2015-06-19T22:43:05 [18731] #8  0x00007fc5b55e4649 in tdc_iterate (thd=0x7fc58f70c070, action=0x7fc5b55e2565 <tc_add_table_callback(TDC_element*, tc_add_table_arg*)>, argument=0x7fc5b4dfc550, no_dups=false) at 10.1/sql/table_cache.cc:1082
# 2015-06-19T22:43:05 [18731] #9  0x00007fc5b55e278c in tc_add_table (thd=0x7fc58f70c070, table=0x7fc58f8e8e70) at 10.1/sql/table_cache.cc:265
# 2015-06-19T22:43:05 [18731] #10 0x00007fc5b53af91d in open_table (thd=0x7fc58f70c070, table_list=0x7fc58f82c150, ot_ctx=0x7fc5b4dfcb80) at 10.1/sql/sql_base.cc:2572
# 2015-06-19T22:43:05 [18731] #11 0x00007fc5b53b2070 in open_and_process_table (thd=0x7fc58f70c070, lex=0x7fc58f70f968, tables=0x7fc58f82c150, counter=0x7fc5b4dfcc24, flags=0, prelocking_strategy=0x7fc5b4dfcc90, has_prelocking_list=false, ot_ctx=0x7fc5b4dfcb80) at 10.1/sql/sql_base.cc:4016
# 2015-06-19T22:43:05 [18731] #12 0x00007fc5b53b3165 in open_tables (thd=0x7fc58f70c070, options=..., start=0x7fc5b4dfcc08, counter=0x7fc5b4dfcc24, flags=0, prelocking_strategy=0x7fc5b4dfcc90) at 10.1/sql/sql_base.cc:4526
# 2015-06-19T22:43:05 [18731] #13 0x00007fc5b53b433e in open_and_lock_tables (thd=0x7fc58f70c070, options=..., tables=0x7fc58f82c150, derived=true, flags=0, prelocking_strategy=0x7fc5b4dfcc90) at 10.1/sql/sql_base.cc:5184
# 2015-06-19T22:43:05 [18731] #14 0x00007fc5b53a7e3b in open_and_lock_tables (thd=0x7fc58f70c070, tables=0x7fc58f82c150, derived=true, flags=0) at 10.1/sql/sql_base.h:530
# 2015-06-19T22:43:05 [18731] #15 0x00007fc5b55665ae in mysql_admin_table(THD *, TABLE_LIST *, HA_CHECK_OPT *, const char *, thr_lock_type, bool, bool, uint, int (*)(THD *, TABLE_LIST *, HA_CHECK_OPT *), struct {...}, int (*)(THD *, TABLE_LIST *, HA_CHECK_OPT *)) (thd=0x7fc58f70c070, tables=0x7fc58f82c150, check_opt=0x7fc58f710848, operator_name=0x7fc5b5d8abf4 "check", lock_type=TL_READ_NO_INSERT, open_for_modify=false, repair_table_use_frm=false, extra_open_options=32, prepare_func=0, operator_func=(int (handler::*)(handler * const, THD *, HA_CHECK_OPT *)) 0x7fc5b566153c <handler::ha_check(THD*, st_ha_check_opt*)>, view_operator_func=0x7fc5b550aa6d <view_check(THD*, TABLE_LIST*, st_ha_check_opt*)>) at 10.1/sql/sql_admin.cc:447
# 2015-06-19T22:43:05 [18731] #16 0x00007fc5b55691bf in Sql_cmd_check_table::execute (this=0x7fc58f82c738, thd=0x7fc58f70c070) at 10.1/sql/sql_admin.cc:1208
# 2015-06-19T22:43:05 [18731] #17 0x00007fc5b5422ac3 in mysql_execute_command (thd=0x7fc58f70c070) at 10.1/sql/sql_parse.cc:5573
# 2015-06-19T22:43:05 [18731] #18 0x00007fc5b5426dff in mysql_parse (thd=0x7fc58f70c070, rawbuf=0x7fc58f82c088 "CHECK TABLE t2", length=14, parser_state=0x7fc5b4dfe1f0) at 10.1/sql/sql_parse.cc:7173
# 2015-06-19T22:43:05 [18731] #19 0x00007fc5b5415d78 in dispatch_command (command=COM_QUERY, thd=0x7fc58f70c070, packet=0x7fc58f7ed071 "", packet_length=14) at 10.1/sql/sql_parse.cc:1470
# 2015-06-19T22:43:05 [18731] #20 0x00007fc5b5414a98 in do_command (thd=0x7fc58f70c070) at 10.1/sql/sql_parse.cc:1093
# 2015-06-19T22:43:05 [18731] #21 0x00007fc5b555640c in do_handle_one_connection (thd_arg=0x7fc58f70c070) at 10.1/sql/sql_connect.cc:1347
# 2015-06-19T22:43:05 [18731] #22 0x00007fc5b5556151 in handle_one_connection (arg=0x7fc58f70c070) at 10.1/sql/sql_connect.cc:1258
# 2015-06-19T22:43:05 [18731] #23 0x00007fc5b4acab50 in start_thread (arg=<optimized out>) at pthread_create.c:304
# 2015-06-19T22:43:05 [18731] #24 0x00007fc5b2b5f95d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
                  Show
                  elenst Elena Stepanova added a comment - Thanks for the report. Here is a variation of the concurrent test from MDEV-8339 , but adjust for CHECK TABLE: RQG grammar query_init: CREATE TABLE IF NOT EXISTS t1 (`pk` INT AUTO_INCREMENT PRIMARY KEY, `c` VARCHAR(32), KEY(`c`)) ENGINE=MyISAM; CREATE TABLE IF NOT EXISTS t2 LIKE t1; CREATE TABLE IF NOT EXISTS t3 LIKE t1; CREATE TABLE IF NOT EXISTS t4 LIKE t1; query: INSERT INTO { 't'.(int(rand(500))+5) } (`c`) VALUES ( _english ), ( _english ), ( _english ), ( _english ), ( _english ) | DELETE FROM { 't'.(int(rand(500))+5) } WHERE `c` = _english ORDER BY `pk` LIMIT _digit | CHECK TABLE tbl; tbl: t1 | t2 | t3 | t4; The command line is the same. Stack trace from debug build of 10.1 commit 93fc04ff1dc613a9ad75ccc57988a6b57b94b6cb # 2015-06-19T22:43:05 [18731] #3 <signal handler called> # 2015-06-19T22:43:05 [18731] #4 0x00007fc5b55e4af9 in TDC_element::free_tables_back (this=0x7fc58fcf1810) at 10.1/sql/table_cache.h:110 # 2015-06-19T22:43:05 [18731] #5 0x00007fc5b55e259f in tc_add_table_callback (element=0x7fc58fcf1810, arg=0x7fc5b4dfc550) at 10.1/sql/table_cache.cc:237 # 2015-06-19T22:43:05 [18731] #6 0x00007fc5b5c74d2d in lfind (head=0x7fc58f8203b0, cs=0x0, hashnr=0, key=0x7fc5b4dfc550 "test", keylen=0, cursor=0x7fc5b4dfc3d0, pins=0x7fc5b215f180, callback=0x7fc5b55e2565 <tc_add_table_callback(TDC_element*, tc_add_table_arg*)>) at 10.1/mysys/lf_hash.c:125 # 2015-06-19T22:43:05 [18731] #7 0x00007fc5b5c759a4 in lf_hash_iterate (hash=0x7fc5b671b040, pins=0x7fc5b215f180, action=0x7fc5b55e2565 <tc_add_table_callback(TDC_element*, tc_add_table_arg*)>, argument=0x7fc5b4dfc550) at 10.1/mysys/lf_hash.c:517 # 2015-06-19T22:43:05 [18731] #8 0x00007fc5b55e4649 in tdc_iterate (thd=0x7fc58f70c070, action=0x7fc5b55e2565 <tc_add_table_callback(TDC_element*, tc_add_table_arg*)>, argument=0x7fc5b4dfc550, no_dups=false) at 10.1/sql/table_cache.cc:1082 # 2015-06-19T22:43:05 [18731] #9 0x00007fc5b55e278c in tc_add_table (thd=0x7fc58f70c070, table=0x7fc58f8e8e70) at 10.1/sql/table_cache.cc:265 # 2015-06-19T22:43:05 [18731] #10 0x00007fc5b53af91d in open_table (thd=0x7fc58f70c070, table_list=0x7fc58f82c150, ot_ctx=0x7fc5b4dfcb80) at 10.1/sql/sql_base.cc:2572 # 2015-06-19T22:43:05 [18731] #11 0x00007fc5b53b2070 in open_and_process_table (thd=0x7fc58f70c070, lex=0x7fc58f70f968, tables=0x7fc58f82c150, counter=0x7fc5b4dfcc24, flags=0, prelocking_strategy=0x7fc5b4dfcc90, has_prelocking_list=false, ot_ctx=0x7fc5b4dfcb80) at 10.1/sql/sql_base.cc:4016 # 2015-06-19T22:43:05 [18731] #12 0x00007fc5b53b3165 in open_tables (thd=0x7fc58f70c070, options=..., start=0x7fc5b4dfcc08, counter=0x7fc5b4dfcc24, flags=0, prelocking_strategy=0x7fc5b4dfcc90) at 10.1/sql/sql_base.cc:4526 # 2015-06-19T22:43:05 [18731] #13 0x00007fc5b53b433e in open_and_lock_tables (thd=0x7fc58f70c070, options=..., tables=0x7fc58f82c150, derived=true, flags=0, prelocking_strategy=0x7fc5b4dfcc90) at 10.1/sql/sql_base.cc:5184 # 2015-06-19T22:43:05 [18731] #14 0x00007fc5b53a7e3b in open_and_lock_tables (thd=0x7fc58f70c070, tables=0x7fc58f82c150, derived=true, flags=0) at 10.1/sql/sql_base.h:530 # 2015-06-19T22:43:05 [18731] #15 0x00007fc5b55665ae in mysql_admin_table(THD *, TABLE_LIST *, HA_CHECK_OPT *, const char *, thr_lock_type, bool, bool, uint, int (*)(THD *, TABLE_LIST *, HA_CHECK_OPT *), struct {...}, int (*)(THD *, TABLE_LIST *, HA_CHECK_OPT *)) (thd=0x7fc58f70c070, tables=0x7fc58f82c150, check_opt=0x7fc58f710848, operator_name=0x7fc5b5d8abf4 "check", lock_type=TL_READ_NO_INSERT, open_for_modify=false, repair_table_use_frm=false, extra_open_options=32, prepare_func=0, operator_func=(int (handler::*)(handler * const, THD *, HA_CHECK_OPT *)) 0x7fc5b566153c <handler::ha_check(THD*, st_ha_check_opt*)>, view_operator_func=0x7fc5b550aa6d <view_check(THD*, TABLE_LIST*, st_ha_check_opt*)>) at 10.1/sql/sql_admin.cc:447 # 2015-06-19T22:43:05 [18731] #16 0x00007fc5b55691bf in Sql_cmd_check_table::execute (this=0x7fc58f82c738, thd=0x7fc58f70c070) at 10.1/sql/sql_admin.cc:1208 # 2015-06-19T22:43:05 [18731] #17 0x00007fc5b5422ac3 in mysql_execute_command (thd=0x7fc58f70c070) at 10.1/sql/sql_parse.cc:5573 # 2015-06-19T22:43:05 [18731] #18 0x00007fc5b5426dff in mysql_parse (thd=0x7fc58f70c070, rawbuf=0x7fc58f82c088 "CHECK TABLE t2", length=14, parser_state=0x7fc5b4dfe1f0) at 10.1/sql/sql_parse.cc:7173 # 2015-06-19T22:43:05 [18731] #19 0x00007fc5b5415d78 in dispatch_command (command=COM_QUERY, thd=0x7fc58f70c070, packet=0x7fc58f7ed071 "", packet_length=14) at 10.1/sql/sql_parse.cc:1470 # 2015-06-19T22:43:05 [18731] #20 0x00007fc5b5414a98 in do_command (thd=0x7fc58f70c070) at 10.1/sql/sql_parse.cc:1093 # 2015-06-19T22:43:05 [18731] #21 0x00007fc5b555640c in do_handle_one_connection (thd_arg=0x7fc58f70c070) at 10.1/sql/sql_connect.cc:1347 # 2015-06-19T22:43:05 [18731] #22 0x00007fc5b5556151 in handle_one_connection (arg=0x7fc58f70c070) at 10.1/sql/sql_connect.cc:1258 # 2015-06-19T22:43:05 [18731] #23 0x00007fc5b4acab50 in start_thread (arg=<optimized out>) at pthread_create.c:304 # 2015-06-19T22:43:05 [18731] #24 0x00007fc5b2b5f95d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
                  Hide
                  Permalink
                  svoj Sergey Vojtovich added a comment -

                  Fixed in scope of MDEV-8339.

                  Show
                  svoj Sergey Vojtovich added a comment - Fixed in scope of MDEV-8339 .

                    People

                    • Assignee:
                      svoj Sergey Vojtovich
                      Reporter:
                      Cyberagent technik Cyberagent technik
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      5 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: