Details
-
Type:
Bug
-
Status: Open
-
Priority:
Major
-
Resolution: Unresolved
-
Affects Version/s: 10.0.20
-
Fix Version/s: None
-
Component/s: Platform RedHat
-
Labels:None
-
Environment:CentOS 6.6 x86_64
Description
Hello!
We experienced multiple random mysqld crashes with signal 11 on a single node of our master-master setup. We run on mariadb 10.0.20 on CentOS 6.6. The node is a VM running on CloudStack. The underlying hypervisor does not seem to have any problem/issue.
We cannot reproduce the issue but we enabled core dump in order to capture more information on the crash.
There were previous crashes back in July, but we started compiling on August 22th when we had 5 crashes on the same day.
For crash 1 to 6, we do not have core dumps but for crash 7 and 8 we have uploaded them on ftp.askmonty.org as: mariadb-bug-crash_on_select.tgz
We have conducted a first analysis of the core dumps but could not find an obvious common cause.
Crash 7 investigation
The following query and database were where the server cored:
(gdb) print do_command::thd->query_string
$23 = {string = {str = 0x7f5b22cfb020 "SELECT `session`.* FROM `session` WHERE (`lifetime` > 1440751751 AND `uid` = 'odrfv7h7mrmnq76eicedeumn16') LIMIT 1", length = 114},
cs = 0x1410a60}
(gdb) print do_command::thd->db
$24 = 0x7f5b22c3a288 "117localeyes"
The backtrace is:
(gdb) bt
#0 0x00007f5d029278ac in pthread_kill () from /lib64/libpthread.so.0
#1 0x000000000071fa29 in handle_fatal_signal (sig=11) at /home/buildbot/buildbot/build/sql/signal_handler.cc:262
#2 <signal handler called>
#3 row_update_prebuilt_trx (prebuilt=0x7f5b1e360068, trx=0x7f5b22d02468) at /home/buildbot/buildbot/build/storage/xtradb/row/row0mysql.cc:996
#4 0x00000000008b94aa in update_thd (this=0x7f5af9415020, thd=0x7f5b24ef8008, lock_type=0) at /home/buildbot/buildbot/build/storage/xtradb/handler/ha_innodb.cc:2702
#5 ha_innobase::external_lock (this=0x7f5af9415020, thd=0x7f5b24ef8008, lock_type=0) at /home/buildbot/buildbot/build/storage/xtradb/handler/ha_innodb.cc:13237
#6 0x0000000000720f04 in handler::ha_external_lock (this=0x7f5b1e360068, thd=0x7f5b24ef8008, lock_type=0) at /home/buildbot/buildbot/build/sql/handler.cc:5913
#7 0x00000000007f6a89 in lock_external (thd=0x7f5b24ef8008, sql_lock=0x7f5b0d846308, flags=<value optimized out>) at /home/buildbot/buildbot/build/sql/lock.cc:360
#8 mysql_lock_tables (thd=0x7f5b24ef8008, sql_lock=0x7f5b0d846308, flags=<value optimized out>) at /home/buildbot/buildbot/build/sql/lock.cc:309
#9 0x00000000007f6ed5 in mysql_lock_tables (thd=0x7f5b24ef8008, tables=0x7f5b22cfc108, count=<value optimized out>, flags=0)
at /home/buildbot/buildbot/build/sql/lock.cc:276
#10 0x0000000000553d7c in lock_tables (thd=0x7f5b24ef8008, tables=<value optimized out>, count=<value optimized out>, flags=0)
at /home/buildbot/buildbot/build/sql/sql_base.cc:5404
#11 0x00000000005591d6 in open_and_lock_tables (thd=0x7f5b24ef8008, tables=0x7f5b22cfb2d8, derived=true, flags=0, prelocking_strategy=<value optimized out>)
at /home/buildbot/buildbot/build/sql/sql_base.cc:5174
#12 0x0000000000595ba7 in open_and_lock_tables (thd=0x7f5b24ef8008, all_tables=0x7f5b22cfb2d8) at /home/buildbot/buildbot/build/sql/sql_base.h:490
#13 execute_sqlcom_select (thd=0x7f5b24ef8008, all_tables=0x7f5b22cfb2d8) at /home/buildbot/buildbot/build/sql/sql_parse.cc:5219
#14 0x0000000000598847 in mysql_execute_command (thd=0x7f5b24ef8008) at /home/buildbot/buildbot/build/sql/sql_parse.cc:2562
#15 0x000000000059fc86 in mysql_parse (thd=0x7f5b24ef8008,
rawbuf=0x7f5b22cfb020 "SELECT `session`.* FROM `session` WHERE (`lifetime` > 1440751751 AND `uid` = 'odrfv7h7mrmnq76eicedeumn16') LIMIT 1",
length=<value optimized out>, parser_state=0x7f5d01147bd0) at /home/buildbot/buildbot/build/sql/sql_parse.cc:6529
#16 0x00000000005a1bb7 in dispatch_command (command=COM_QUERY, thd=0x7f5b24ef8008, packet=<value optimized out>, packet_length=114)
at /home/buildbot/buildbot/build/sql/sql_parse.cc:1308
#17 0x00000000005a22f9 in do_command (thd=0x7f5b24ef8008) at /home/buildbot/buildbot/build/sql/sql_parse.cc:999
#18 0x000000000066c7d4 in do_handle_one_connection (thd_arg=0x7f5b24ef8008) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1378
#19 0x000000000066c912 in handle_one_connection (arg=0x7f5b24fa5008) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1293
#20 0x00007f5d029229d1 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f5d018a18fd in clone () from /lib64/libc.so.6
The crash was a segmentation fault occurring at line 996ish of storage/xtradb/row/row0mysql.cc. It appears that it is in these lines:
https://github.com/MariaDB/server/blob/10.0/storage/xtradb/row/row0mysql.cc#L996
if (prebuilt->sel_graph) {
prebuilt->sel_graph->trx = trx;
}
The value of prebuilt->sel_graph at this point is certainly outside of allocated memory:
(gdb) print row_update_prebuilt_trx::prebuilt->sel_graph $3 = (que_fork_t *) 0x404c000000000000
The full dump of the prebuilt is:
(gdb) print *row_update_prebuilt_trx::prebuilt
$6 = {magic_n = 78540783, table = 0x7f5b24b3f268, index = 0x7f5b24b3f768, trx = 0x7f5b22d02468, sql_stat_start = 1, mysql_has_locked = 0, clust_index_was_generated = 1,
index_usable = 1, read_just_key = 0, used_in_HANDLER = 0, template_type = 0, n_template = 4, null_bitmap_len = 0, need_to_access_clustered = 1, templ_contains_blob = 1,
mysql_template = 0x404c000000000000, heap = 0x7f5b1e360000, ins_node = 0x7f5b1e360808, ins_upd_rec_buff = 0x7f5b1e3608c0 "", default_rec = 0x7f5b23fb8040 "",
hint_need_to_fetch_extra_cols = 0, upd_node = 0x0, trx_id = 0, ins_graph = 0x7f5b1e360cb0, upd_graph = 0x0, pcur = {btr_cur = {index = 0x7f5b24b3f768, page_cur = {
rec = 0x7f5c96290070 "supremumc\032", block = 0x7f5c594d5200}, purge_node = 0x0, left_block = 0x0, thr = 0x0, flag = 0, tree_height = 0, up_match = 0, up_bytes = 0,
low_match = 0, low_bytes = 0, n_fields = 0, n_bytes = 0, fold = 0, path_arr = 0x0}, latch_mode = 1, old_stored = 908467085, old_rec = 0x7f5b1e2b466d "",
old_n_fields = 1, rel_pos = 3, block_when_stored = 0x7f5c594d5200, modify_clock = 52, pos_state = BTR_PCUR_IS_POSITIONED, search_mode = 1, trx_if_known = 0x0,
old_rec_buf = 0x7f5b1e2b4668 "", buf_size = 16}, clust_pcur = {btr_cur = {index = 0x0, page_cur = {rec = 0x0, block = 0x0}, purge_node = 0x0, left_block = 0x0,
thr = 0x0, flag = 0, tree_height = 0, up_match = 0, up_bytes = 0, low_match = 0, low_bytes = 0, n_fields = 0, n_bytes = 0, fold = 0, path_arr = 0x0}, latch_mode = 3,
old_stored = 122766467, old_rec = 0x0, old_n_fields = 0, rel_pos = 0, block_when_stored = 0x0, modify_clock = 0, pos_state = BTR_PCUR_NOT_POSITIONED, search_mode = 0,
trx_if_known = 0x0, old_rec_buf = 0x0, buf_size = 0}, sel_graph = 0x404c000000000000, search_tuple = 0x7f5b1e3603d8, row_id = "\000\000\000\021\022\376",
fts_doc_id = 0, clust_ref = 0x7f5b1e360558, select_lock_type = 5, stored_select_lock_type = 5, row_read_type = 0, new_rec_locks = 0, mysql_prefix_len = 785,
mysql_row_len = 785, n_rows_fetched = 2574, fetch_direction = 1, fetch_cache = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, keep_other_fields_on_keyread = 0,
fetch_cache_first = 0, n_fetch_cached = 0, blob_heap = 0x0, old_vers_heap = 0x7f5b21c5b340, in_fts_query = false, autoinc_last_value = 0, autoinc_increment = 1,
autoinc_offset = 0, autoinc_error = DB_SUCCESS, idx_cond = 0x0, idx_cond_n_cols = 0, magic_n2 = 78540783, innodb_api = 0, innodb_api_rec = 0x0, srch_key_val1 = 0x0,
srch_key_val2 = 0x0, srch_key_val_len = 0}
Notice that the mysql_template has the same value as sel_graph.
=========================
Crash 8 invstigation:
The core happened whilst performing the following query from the 069shop database:
SELECT DISTINCT 1 AS `status`, `e`.`entity_id`, `e`.`type_id`, `e`.`attribute_set_id`, `cat_index`.`position` AS `cat_index_position`, `mile_price_index`.`fb_miles`, `mile_price_index`.`fb_home_miles`, `mile_price_index`.`min_price` AS `min_miles`, `mile_price_index`.`max_price` AS `max_miles`, `e`.`name`, `e`.`short_description`, `e`.`price`, `e`.`special_price`, `e`.`special_from_date`, `e`.`special_to_date`, `e`.`small_image`, `e`.`thumbnail`, `e`.`news_from_date`, `e`.`news_to_date`, `e`.`url_key`, `e`.`required_options`, `e`.`image_label`, `e`.`small_image_label`, `e`.`thumbnail_label`, `e`.`msrp_enabled`, `e`.`msrp_display_actual_price_type`, `e`.`msrp`, `e`.`tax_class_id`, `e`.`price_type`, `e`.`weight_type`, `e`.`price_view`, `e`.`shipment_type`, `e`.`links_purchased_separately`, `e`.`links_exist`, `e`.`giftcard_amounts`, `e`.`allow_open_amount`, `e`.`open_amount_min`, `e`.`open_amount_max`, `e`.`brand_id`, `e`.`brand_id_value`, `e`.`fb_miles`, `e`.`fb_home_miles`, `e`.`label_id`, `e`.`label_id_value`, `e`.`scale`, `e`.`scale_value`, `e`.`delivery_rule_ids`, `e`.`target_group_ids`, `e`.`warranty`, `e`.`is_home_page`, `e`.`show_in_campaign`, `e`.`campaign_slider_image`, `e`.`campaign_main_image`, `e`.`campaign_text`, `e`.`fb_home_earn`, `e`.`fb_home_earn_promo`, `e`.`fb_home_earn_promo_code`, `e`.`fb_onboard_earn`, `e`.`fb_onboard_earn_promo`, `e`.`fb_onboard_earn_promo_code`, `e`.`color`, `e`.`your_surprise_id`, `e`.`prologistica_number`, `e`.`waterproof`, `e`.`prologistica_send`, `e`.`local_price`, `price_index`.`price`, `price_index`.`tax_class_id`, `price_index`.`final_price`, IF(price_index.tier_price IS NOT NULL, LEAST(price_index.min_price, price_index.tier_price), price_index.min_price) AS `minimal_price`, `price_index`.`min_price`, `price_index`.`max_price`, `price_index`.`tier_price` FROM `catalog_product_flat_7` AS `e`\n INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id AND cat_index.store_id='7' AND cat_index.visibility IN(2, 4) AND cat_index.category_id='168'\n LEFT JOIN `klm_flyingblue_index` AS `mile_price_index` ON e.entity_id = mile_price_index.entity_id\n INNER JOIN `catalog_product_index_price` AS `price_index` ON price_index.entity_id = e.entity_id AND price_index.website_id = '2' AND price_index.customer_group_id = 0 WHERE (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('160', '168'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('160', '168'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('160', '168'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('160', '168'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('160', '168'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('160', '168'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('160', '168'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('160', '168')))))
The backtrace for the crash from gdb:
(gdb) bt
#0 0x00007f5978c1f8ac in pthread_kill () from /lib64/libpthread.so.0
#1 0x000000000071fa29 in handle_fatal_signal (sig=11) at /home/buildbot/buildbot/build/sql/signal_handler.cc:262
#2 <signal handler called>
#3 build_template_field (prebuilt=0x7f57947cf068, clust_index=0x7f5793f95168, index=0x7f5793f95668, table=0x7f5794470608, field=0x7f57944e8888, i=0)
at /home/buildbot/buildbot/build/storage/xtradb/handler/ha_innodb.cc:6979
#4 0x00000000008aaf68 in ha_innobase::build_template (this=0x7f57944bb020, whole_row=false) at /home/buildbot/buildbot/build/storage/xtradb/handler/ha_innodb.cc:7302
#5 0x00000000008ae856 in ha_innobase::change_active_index (this=0x7f57944bb020, keynr=2) at /home/buildbot/buildbot/build/storage/xtradb/handler/ha_innodb.cc:8771
#6 0x0000000000824b30 in ha_index_init (this=0x7f576fe55d80) at /home/buildbot/buildbot/build/sql/handler.h:2618
#7 QUICK_RANGE_SELECT::reset (this=0x7f576fe55d80) at /home/buildbot/buildbot/build/sql/opt_range.cc:11693
#8 0x00000000005be465 in join_init_read_record (tab=0x7f579478cce0) at /home/buildbot/buildbot/build/sql/sql_select.cc:18698
#9 0x00000000005dc8b4 in sub_select (join=0x7f5794545130, join_tab=0x7f579478cce0, end_of_records=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:17812
#10 0x00000000006a9316 in join_tab_execution_startup (tab=<value optimized out>) at /home/buildbot/buildbot/build/sql/opt_subselect.cc:5072
#11 0x00000000005dc888 in sub_select (join=0x7f5794545130, join_tab=0x7f579478a8d0, end_of_records=false) at /home/buildbot/buildbot/build/sql/sql_select.cc:17803
#12 0x00000000005dc65c in evaluate_join_record (join=0x7f5794545130, join_tab=0x7f579478a5a8, error=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:18037
#13 0x00000000005dc8cb in sub_select (join=0x7f5794545130, join_tab=0x7f579478a5a8, end_of_records=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:17815
#14 0x00000000005dc65c in evaluate_join_record (join=0x7f5794545130, join_tab=0x7f579478a280, error=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:18037
#15 0x00000000005dc8cb in sub_select (join=0x7f5794545130, join_tab=0x7f579478a280, end_of_records=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:17815
#16 0x00000000005dc65c in evaluate_join_record (join=0x7f5794545130, join_tab=0x7f5794789f58, error=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:18037
#17 0x00000000005dc8cb in sub_select (join=0x7f5794545130, join_tab=0x7f5794789f58, end_of_records=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:17815
#18 0x00000000005dc65c in evaluate_join_record (join=0x7f5794545130, join_tab=0x7f5794789c30, error=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:18037
#19 0x00000000005dc8cb in sub_select (join=0x7f5794545130, join_tab=0x7f5794789c30, end_of_records=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:17815
#20 0x00000000005dc65c in evaluate_join_record (join=0x7f5794545130, join_tab=0x7f5794789908, error=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:18037
#21 0x00000000005dc8cb in sub_select (join=0x7f5794545130, join_tab=0x7f5794789908, end_of_records=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:17815
#22 0x00000000005dc65c in evaluate_join_record (join=0x7f5794545130, join_tab=0x7f57947895e0, error=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:18037
#23 0x00000000005dc8cb in sub_select (join=0x7f5794545130, join_tab=0x7f57947895e0, end_of_records=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:17815
#24 0x00000000005dc65c in evaluate_join_record (join=0x7f5794545130, join_tab=0x7f57947892b8, error=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:18037
#25 0x00000000005dc8cb in sub_select (join=0x7f5794545130, join_tab=0x7f57947892b8, end_of_records=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:17815
#26 0x00000000005dc65c in evaluate_join_record (join=0x7f5794545130, join_tab=0x7f5794788f90, error=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:18037
#27 0x00000000005dc8cb in sub_select (join=0x7f5794545130, join_tab=0x7f5794788f90, end_of_records=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:17815
#28 0x00000000005dcfdd in do_select (join=0x7f5794545130, fields=0x0, table=0x7f57945fa020, procedure=0x0) at /home/buildbot/buildbot/build/sql/sql_select.cc:17477
#29 0x00000000005ef5c0 in JOIN::exec_inner (this=0x7f5794545130) at /home/buildbot/buildbot/build/sql/sql_select.cc:2622
#30 0x00000000005ee19a in exec (thd=0x7f576ffd0008, rref_pointer_array=0x7f576ffd4190, tables=0x7f5794664020, wild_num=0, fields=<value optimized out>, conds=0x7f5794566548, og_num=0, order=0x0,
group=0x0, having=0x0, proc_param=0x0, select_options=2147748609, result=0x7f5794545110, unit=0x7f576ffd3828, select_lex=0x7f576ffd3f18) at /home/buildbot/buildbot/build/sql/sql_select.cc:2368
#31 mysql_select (thd=0x7f576ffd0008, rref_pointer_array=0x7f576ffd4190, tables=0x7f5794664020, wild_num=0, fields=<value optimized out>, conds=0x7f5794566548, og_num=0, order=0x0, group=0x0,
having=0x0, proc_param=0x0, select_options=2147748609, result=0x7f5794545110, unit=0x7f576ffd3828, select_lex=0x7f576ffd3f18) at /home/buildbot/buildbot/build/sql/sql_select.cc:3308
#32 0x00000000005f1dcd in handle_select (thd=0x7f576ffd0008, lex=0x7f576ffd3760, result=0x7f5794545110, setup_tables_done_option=0) at /home/buildbot/buildbot/build/sql/sql_select.cc:373
#33 0x0000000000595bf0 in execute_sqlcom_select (thd=0x7f576ffd0008, all_tables=0x7f5794664020) at /home/buildbot/buildbot/build/sql/sql_parse.cc:5274
#34 0x0000000000598847 in mysql_execute_command (thd=0x7f576ffd0008) at /home/buildbot/buildbot/build/sql/sql_parse.cc:2562
#35 0x000000000059fc86 in mysql_parse (thd=0x7f576ffd0008,
rawbuf=0x7f5794540020 "SELECT DISTINCT 1 AS `status`, `e`.`entity_id`, `e`.`type_id`, `e`.`attribute_set_id`, `cat_index`.`position` AS `cat_index_position`, `mile_price_index`.`fb_miles`, `mile_price_index`.`fb_home_miles`"..., length=<value optimized out>, parser_state=0x7f576f291bd0) at /home/buildbot/buildbot/build/sql/sql_parse.cc:6529
#36 0x00000000005a1bb7 in dispatch_command (command=COM_QUERY, thd=0x7f576ffd0008, packet=<value optimized out>, packet_length=3808) at /home/buildbot/buildbot/build/sql/sql_parse.cc:1308
#37 0x00000000005a22f9 in do_command (thd=0x7f576ffd0008) at /home/buildbot/buildbot/build/sql/sql_parse.cc:999
#38 0x000000000066c7d4 in do_handle_one_connection (thd_arg=0x7f576ffd0008) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1378
#39 0x000000000066c912 in handle_one_connection (arg=0x7f57702d1008) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1293
#40 0x00007f5978c1a9d1 in start_thread () from /lib64/libpthread.so.0
#41 0x00007f5977b998fd in clone () from /lib64/libc.so.6
The crash appears to have occurred in a different place than the other core dump.
prebuilt at the point of crash contains:
(gdb) print *build_template_field::prebuilt
$3 = {magic_n = 78540783, table = 0x7f5793f94c68, index = 0x7f5793f95668, trx = 0x7f57944da668, sql_stat_start = 1, mysql_has_locked = 1, clust_index_was_generated = 0, index_usable = 1,
read_just_key = 1, used_in_HANDLER = 0, template_type = 1, n_template = 1, null_bitmap_len = 1, need_to_access_clustered = 0, templ_contains_blob = 0, mysql_template = 0x403f000000000000,
heap = 0x7f57947cf000, ins_node = 0x0, ins_upd_rec_buff = 0x0, default_rec = 0x7f5798de8cc8 "\377", hint_need_to_fetch_extra_cols = 0, upd_node = 0x0, trx_id = 0, ins_graph = 0x0, upd_graph = 0x0,
pcur = {btr_cur = {index = 0x7f5793f95668, page_cur = {rec = 0x7f588174ad7e "", block = 0x7f583ef23b00}, purge_node = 0x0, left_block = 0x0, thr = 0x0, flag = BTR_CUR_BINARY, tree_height = 2,
up_match = 2, up_bytes = 0, low_match = 1, low_bytes = 3, n_fields = 2, n_bytes = 0, fold = 1763120805103625962, path_arr = 0x0}, latch_mode = 1, old_stored = 908467085,
old_rec = 0x7f57943b966e "", old_n_fields = 6, rel_pos = 2, block_when_stored = 0x7f583ef23b00, modify_clock = 0, pos_state = BTR_PCUR_IS_POSITIONED, search_mode = 2,
trx_if_known = 0x7f57944da668, old_rec_buf = 0x7f57943b9668 "", buf_size = 24}, clust_pcur = {btr_cur = {index = 0x0, page_cur = {rec = 0x0, block = 0x0}, purge_node = 0x0, left_block = 0x0,
thr = 0x0, flag = 0, tree_height = 0, up_match = 0, up_bytes = 0, low_match = 0, low_bytes = 0, n_fields = 0, n_bytes = 0, fold = 0, path_arr = 0x0}, latch_mode = 3, old_stored = 122766467,
old_rec = 0x0, old_n_fields = 0, rel_pos = 0, block_when_stored = 0x0, modify_clock = 0, pos_state = BTR_PCUR_NOT_POSITIONED, search_mode = 0, trx_if_known = 0x0, old_rec_buf = 0x0,
buf_size = 0}, sel_graph = 0x403f000000000000, search_tuple = 0x7f57947cf400, row_id = "\000\000\000\000\000", fts_doc_id = 0, clust_ref = 0x7f57947cf5e0, select_lock_type = 5,
stored_select_lock_type = 5, row_read_type = 0, new_rec_locks = 0, mysql_prefix_len = 0, mysql_row_len = 19, n_rows_fetched = 7, fetch_direction = 1, fetch_cache = {0x7f57981a93ec "",
0x7f57981a9407 "", 0x7f57981a9422 "", 0x7f57981a943d "", 0x7f57981a9458 "\220\250", 0x7f57981a9473 "", 0x7f57981a948e "", 0x7f57981a94a9 ""}, keep_other_fields_on_keyread = 0,
fetch_cache_first = 0, n_fetch_cached = 0, blob_heap = 0x0, old_vers_heap = 0x0, in_fts_query = false, autoinc_last_value = 0, autoinc_increment = 1, autoinc_offset = 0,
autoinc_error = DB_SUCCESS, idx_cond = 0x0, idx_cond_n_cols = 0, magic_n2 = 78540783, innodb_api = 0, innodb_api_rec = 0x0, srch_key_val1 = 0x7f57947cf3d8 "", srch_key_val2 = 0x7f57947cf3ea "",
srch_key_val_len = 18}
Notice that both sel_graph and mysql_template are invalid pointers (and the same value).
Please let us know if we can help you further to identify the root cause.
Cheers,
--thomas
Gliffy Diagrams
Attachments
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Please find attached the my.cnf files of that server.
We have had more crashes.
On production with an update this time (sel_graph set to an impropoer value):
(gdb) bt
#0 0x00007fe5f69e48ac in pthread_kill () from /lib64/libpthread.so.0
#1 0x000000000071fa29 in handle_fatal_signal (sig=11) at /home/buildbot/buildbot/build/sql/signal_handler.cc:262
#2 <signal handler called>
#3 row_update_prebuilt_trx (prebuilt=0x7fe3ef356068, trx=0x7fe411d50468) at /home/buildbot/buildbot/build/storage/xtradb/row/row0mysql.cc:996
#4 0x00000000008ba11b in update_thd (this=0x7fe3ef2e8020, flag=212, is_analyze=false) at /home/buildbot/buildbot/build/storage/xtradb/handler/ha_innodb.cc:2702
#5 ha_innobase::info_low (this=0x7fe3ef2e8020, flag=212, is_analyze=false) at /home/buildbot/buildbot/build/storage/xtradb/handler/ha_innodb.cc:11890
#6 0x0000000000600f1f in get_schema_tables_record (thd=0x7fe402949008, tables=0x7fe418e5a500, table=0x7fe3eeeac020, res=<value optimized out>, db_name=<value optimized out>, table_name=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_show.cc:5193
#7 0x00000000005f8e5c in fill_schema_table_by_open (thd=0x7fe402949008, is_show_fields_or_keys=false, table=0x7fe3eeeac020, schema_table=0x1389be0, orig_db_name=0x7fe3eeecf9b8, orig_table_name=0x7fe418d0aff0, open_tables_state_backup=0x7fe5f49ee6d0, can_deadlock=false)
at /home/buildbot/buildbot/build/sql/sql_show.cc:4298
#8 0x00000000005fd5c7 in get_all_tables (thd=0x7fe402949008, tables=0x7fe411c4e8b0, cond=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_show.cc:4921
#9 0x00000000005f545a in get_schema_tables_result (join=0x7fe3ee4651f8, executed_place=PROCESSED_BY_JOIN_EXEC) at /home/buildbot/buildbot/build/sql/sql_show.cc:8166
#10 0x00000000005ef4b5 in JOIN::exec_inner (this=0x7fe3ee4651f8) at /home/buildbot/buildbot/build/sql/sql_select.cc:2532
#11 0x00000000005ee19a in exec (thd=0x7fe402949008, rref_pointer_array=0x7fe40294d190, tables=0x7fe411c4e8b0, wild_num=0, fields=<value optimized out>, conds=0x7fe411c4f448, og_num=2, order=0x7fe411c4f7d8, group=0x7fe411c4f680, having=0x0, proc_param=0x0, select_options=2684619520, result=0x7fe3ee4651d8, unit=
0x7fe40294c828, select_lex=0x7fe40294cf18) at /home/buildbot/buildbot/build/sql/sql_select.cc:2368
#12 mysql_select (thd=0x7fe402949008, rref_pointer_array=0x7fe40294d190, tables=0x7fe411c4e8b0, wild_num=0, fields=<value optimized out>, conds=0x7fe411c4f448, og_num=2, order=0x7fe411c4f7d8, group=0x7fe411c4f680, having=0x0, proc_param=0x0, select_options=2684619520, result=0x7fe3ee4651d8, unit=0x7fe40294c828,
select_lex=0x7fe40294cf18) at /home/buildbot/buildbot/build/sql/sql_select.cc:3308
#13 0x00000000005f1dcd in handle_select (thd=0x7fe402949008, lex=0x7fe40294c760, result=0x7fe3ee4651d8, setup_tables_done_option=0) at /home/buildbot/buildbot/build/sql/sql_select.cc:373
#14 0x0000000000595bf0 in execute_sqlcom_select (thd=0x7fe402949008, all_tables=0x7fe411c4e8b0) at /home/buildbot/buildbot/build/sql/sql_parse.cc:5274
#15 0x0000000000598847 in mysql_execute_command (thd=0x7fe402949008) at /home/buildbot/buildbot/build/sql/sql_parse.cc:2562
#16 0x000000000059fc86 in mysql_parse (thd=0x7fe402949008, rawbuf=0x7fe411c4e020 "SELECT ENGINE,SUM(DATA_LENGTH),COUNT(ENGINE) FROM information_schema.TABLES WHERE TABLE_SCHEMA NOT IN ('information_schema','mysql') AND ENGINE IS NOT NULL GROUP BY ENGINE ORDER BY ENGINE ASC", length=<value optimized out>,
parser_state=0x7fe5f49f0bd0) at /home/buildbot/buildbot/build/sql/sql_parse.cc:6529
#17 0x00000000005a1bb7 in dispatch_command (command=COM_QUERY, thd=0x7fe402949008, packet=<value optimized out>, packet_length=191) at /home/buildbot/buildbot/build/sql/sql_parse.cc:1308
#18 0x00000000005a22f9 in do_command (thd=0x7fe402949008) at /home/buildbot/buildbot/build/sql/sql_parse.cc:999
#19 0x000000000066c7d4 in do_handle_one_connection (thd_arg=0x7fe402949008) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1378
#20 0x000000000066c912 in handle_one_connection (arg=0x7fe4170b8008) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1293
#21 0x00007fe5f69df9d1 in start_thread () from /lib64/libpthread.so.0
#22 0x00007fe5f595e8fd in clone () from /lib64/libc.so.6
(gdb) print *row_update_prebuilt_trx::prebuilt
$1 = {magic_n = 78540783, table = 0x7fe41a59ba68, index = 0x7fe41a59cbe8, trx = 0x7fe411d50468, sql_stat_start = 1, mysql_has_locked = 0, clust_index_was_generated = 0, index_usable = 1, read_just_key = 0, used_in_HANDLER = 0, template_type = 3, n_template = 0, null_bitmap_len = 0, need_to_access_clustered = 0,
templ_contains_blob = 1, mysql_template = 0x7fe4174f5168, heap = 0x7fe3ef356000, ins_node = 0x0, ins_upd_rec_buff = 0x0, default_rec = 0x7fe3eb4d2120 "", hint_need_to_fetch_extra_cols = 0, upd_node = 0x0, trx_id = 0, ins_graph = 0x0, upd_graph = 0x0, pcur = {btr_cur = {index = 0x7fe41a59cbe8, page_cur = {
rec = 0x7fe4f2620070 "supremum\016", block = 0x7fe4bc654c00}, purge_node = 0x0, left_block = 0x0, thr = 0x0, flag = 0, tree_height = 0, up_match = 0, up_bytes = 0, low_match = 0, low_bytes = 0, n_fields = 0, n_bytes = 0, fold = 0, path_arr = 0x0}, latch_mode = 1, old_stored = 908467085,
old_rec = 0x7fe3ef3f616d "", old_n_fields = 2, rel_pos = 3, block_when_stored = 0x7fe4bc654c00, modify_clock = 1, pos_state = BTR_PCUR_IS_POSITIONED, search_mode = 1, trx_if_known = 0x0, old_rec_buf = 0x7fe3ef3f6168 "", buf_size = 16}, clust_pcur = {btr_cur = {index = 0x0, page_cur = {rec = 0x0, block = 0x0},
purge_node = 0x0, left_block = 0x0, thr = 0x0, flag = 0, tree_height = 0, up_match = 0, up_bytes = 0, low_match = 0, low_bytes = 0, n_fields = 0, n_bytes = 0, fold = 0, path_arr = 0x0}, latch_mode = 3, old_stored = 122766467, old_rec = 0x0, old_n_fields = 0, rel_pos = 0, block_when_stored = 0x0,
modify_clock = 0, pos_state = BTR_PCUR_NOT_POSITIONED, search_mode = 0, trx_if_known = 0x0, old_rec_buf = 0x0, buf_size = 0}, sel_graph = 0x402c000000000000, search_tuple = 0x7fe3ef3563e8, row_id = "\000\000\000\000\000", fts_doc_id = 0, clust_ref = 0x7fe3ef356598, select_lock_type = 5,
stored_select_lock_type = 2, row_read_type = 0, new_rec_locks = 0, mysql_prefix_len = 1552, mysql_row_len = 1552, n_rows_fetched = 8, fetch_direction = 1, fetch_cache = {0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, keep_other_fields_on_keyread = 0, fetch_cache_first = 0, n_fetch_cached = 0, blob_heap = 0x0,
old_vers_heap = 0x0, in_fts_query = false, autoinc_last_value = 0, autoinc_increment = 1, autoinc_offset = 0, autoinc_error = DB_SUCCESS, idx_cond = 0x0, idx_cond_n_cols = 0, magic_n2 = 78540783, innodb_api = 0, innodb_api_rec = 0x0, srch_key_val1 = 0x7fe3ef3563d8 "", srch_key_val2 = 0x7fe3ef3563e0 "",
srch_key_val_len = 8}
And as well as 1 crash on a preproduction environment (so a different node so this is definitvely a bug in the mariadb code) during a security scan:
(gdb) bt
#0 0x00007f037af958ac in pthread_kill () from /lib64/libpthread.so.0
#1 0x000000000071fa29 in handle_fatal_signal (sig=11) at /home/buildbot/buildbot/build/sql/signal_handler.cc:262
#2 <signal handler called>
#3 row_update_prebuilt_trx (prebuilt=0x7f0182ecc068, trx=0x7f019c853268) at /home/buildbot/buildbot/build/storage/xtradb/row/row0mysql.cc:996
#4 0x00000000008b94aa in update_thd (this=0x7f0182ee6020, thd=0x7f019eb7a008, lock_type=0) at /home/buildbot/buildbot/build/storage/xtradb/handler/ha_innodb.cc:2702
#5 ha_innobase::external_lock (this=0x7f0182ee6020, thd=0x7f019eb7a008, lock_type=0) at /home/buildbot/buildbot/build/storage/xtradb/handler/ha_innodb.cc:13237
#6 0x0000000000720f04 in handler::ha_external_lock (this=0x7f0182ecc068, thd=0x7f019eb7a008, lock_type=0) at /home/buildbot/buildbot/build/sql/handler.cc:5913
#7 0x00000000007f6a89 in lock_external (thd=0x7f019eb7a008, sql_lock=0x7f0181f56888, flags=<value optimized out>) at /home/buildbot/buildbot/build/sql/lock.cc:360
#8 mysql_lock_tables (thd=0x7f019eb7a008, sql_lock=0x7f0181f56888, flags=<value optimized out>) at /home/buildbot/buildbot/build/sql/lock.cc:309
#9 0x00000000007f6ed5 in mysql_lock_tables (thd=0x7f019eb7a008, tables=0x7f0182fc1888, count=<value optimized out>, flags=0) at /home/buildbot/buildbot/build/sql/lock.cc:276
#10 0x0000000000553d7c in lock_tables (thd=0x7f019eb7a008, tables=<value optimized out>, count=<value optimized out>, flags=0) at /home/buildbot/buildbot/build/sql/sql_base.cc:5404
#11 0x00000000005591d6 in open_and_lock_tables (thd=0x7f019eb7a008, tables=0x7f019c4306f0, derived=true, flags=0, prelocking_strategy=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_base.cc:5174
#12 0x0000000000595ba7 in open_and_lock_tables (thd=0x7f019eb7a008, all_tables=0x7f019c4306f0) at /home/buildbot/buildbot/build/sql/sql_base.h:490
#13 execute_sqlcom_select (thd=0x7f019eb7a008, all_tables=0x7f019c4306f0) at /home/buildbot/buildbot/build/sql/sql_parse.cc:5219
#14 0x0000000000598847 in mysql_execute_command (thd=0x7f019eb7a008) at /home/buildbot/buildbot/build/sql/sql_parse.cc:2562
#15 0x000000000059fc86 in mysql_parse (thd=0x7f019eb7a008, rawbuf=0x7f019c42f020 "SELECT DISTINCT COUNT(DISTINCT e.entity_id) FROM `catalog_product_flat_7` AS `e`\n INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id AND cat_index.store_id="...,
length=<value optimized out>, parser_state=0x7f037b36cbd0) at /home/buildbot/buildbot/build/sql/sql_parse.cc:6529
#16 0x00000000005a1bb7 in dispatch_command (command=COM_QUERY, thd=0x7f019eb7a008, packet=<value optimized out>, packet_length=2475) at /home/buildbot/buildbot/build/sql/sql_parse.cc:1308
#17 0x00000000005a22f9 in do_command (thd=0x7f019eb7a008) at /home/buildbot/buildbot/build/sql/sql_parse.cc:999
#18 0x000000000066c7d4 in do_handle_one_connection (thd_arg=0x7f019eb7a008) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1378
#19 0x000000000066c912 in handle_one_connection (arg=0x7f019a83b008) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1293
#20 0x00007f037af909d1 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f0379f0f8fd in clone () from /lib64/libc.so.6
(gdb) print *row_update_prebuilt_trx::prebuilt
$1 = {magic_n = 78540783, table = 0x7f019e01fd68, index = 0x7f019e020768, trx = 0x7f019c853268, sql_stat_start = 1, mysql_has_locked = 0, clust_index_was_generated = 0, index_usable = 1, read_just_key = 0, used_in_HANDLER = 0, template_type = 1, n_template = 3, null_bitmap_len = 1, need_to_access_clustered = 0,
templ_contains_blob = 0, mysql_template = 0x7f019c4c85e8, heap = 0x7f0182ecc000, ins_node = 0x0, ins_upd_rec_buff = 0x0, default_rec = 0x7f01799828c8 "\377", hint_need_to_fetch_extra_cols = 0, upd_node = 0x0, trx_id = 0, ins_graph = 0x0, upd_graph = 0x0, pcur = {btr_cur = {index = 0x7f019e020768, page_cur = {
rec = 0x7f030ae0431e "", block = 0x7f02d1b69d80}, purge_node = 0x0, left_block = 0x0, thr = 0x0, flag = BTR_CUR_BINARY, tree_height = 2, up_match = 2, up_bytes = 0, low_match = 1, low_bytes = 3, n_fields = 2, n_bytes = 0, fold = 1763120737502627629, path_arr = 0x0}, latch_mode = 1, old_stored = 908467085,
old_rec = 0x7f019b58c16e "", old_n_fields = 6, rel_pos = 2, block_when_stored = 0x7f02d1b69d80, modify_clock = 3613, pos_state = BTR_PCUR_IS_POSITIONED, search_mode = 2, trx_if_known = 0x7f0199c9c268, old_rec_buf = 0x7f019b58c168 "", buf_size = 24}, clust_pcur = {btr_cur = {index = 0x0, page_cur = {rec = 0x0,
block = 0x0}, purge_node = 0x0, left_block = 0x0, thr = 0x0, flag = 0, tree_height = 0, up_match = 0, up_bytes = 0, low_match = 0, low_bytes = 0, n_fields = 0, n_bytes = 0, fold = 0, path_arr = 0x0}, latch_mode = 3, old_stored = 122766467, old_rec = 0x0, old_n_fields = 0, rel_pos = 0,
block_when_stored = 0x0, modify_clock = 0, pos_state = BTR_PCUR_NOT_POSITIONED, search_mode = 0, trx_if_known = 0x0, old_rec_buf = 0x0, buf_size = 0}, sel_graph = 0x402c000000000000, search_tuple = 0x7f0182ecc400, row_id = "\000\000\000\000\000", fts_doc_id = 0, clust_ref = 0x7f0182ecc5e0,
select_lock_type = 5, stored_select_lock_type = 5, row_read_type = 0, new_rec_locks = 0, mysql_prefix_len = 17, mysql_row_len = 19, n_rows_fetched = 24, fetch_direction = 1, fetch_cache = {0x7f019ea7d3ac "", 0x7f019ea7d3c7 "", 0x7f019ea7d3e2 "", 0x7f019ea7d3fd "", 0x7f019ea7d418 "X\245", 0x7f019ea7d433 "",
0x7f019ea7d44e "", 0x7f019ea7d469 ""}, keep_other_fields_on_keyread = 0, fetch_cache_first = 0, n_fetch_cached = 0, blob_heap = 0x0, old_vers_heap = 0x0, in_fts_query = false, autoinc_last_value = 0, autoinc_increment = 1, autoinc_offset = 0, autoinc_error = DB_SUCCESS, idx_cond = 0x0, idx_cond_n_cols = 0,
magic_n2 = 78540783, innodb_api = 0, innodb_api_rec = 0x0, srch_key_val1 = 0x7f0182ecc3d8 "", srch_key_val2 = 0x7f0182ecc3ea "", srch_key_val_len = 18}
Same behavior as before with sel_graph value out of bounds...
Would you like me to send the core files generated by those 2 crashes or do you already have enough information with the core files already uploaded?
Cheers,
--thomas
Hi,
The last crash looks very different, looks more like a problem on SQL-optimizer than InnoDB. Can you report that on different MDEV and upload core. This leads me to think is there a bug on InnoDB code base or in how SQL-layer is executing the actual query. Can you provide output from explain plan for select ... and show create table from used tables ?
In surface this looks like memory corruption/overwrite but if it repeats on two different machines, HW problem is not likely, but can you check dmesg output from both machines.
Does crash happen on every case you run above query or just randomly?
Can you also provide full, unedited error log from (1) InnoDB crash and (2) from SQL-layer crash.
R: Jan
Hello Jan,
I agree that the last crash is not linked with this case.
I have updated my last comment to withdraw that last crash.
Regarding the case I created we have now 8 crashes with segfault 11 each time on a different query as reported to you earlier.
Most of the time it happens on select and sometimes on update.
We cannot reproduce the crash but the backtrace is showing the same signature: sel_graph and when present mysql_template get a wrong memory address.
Please find below the unedited error log for another crash on our preproduction system on friday 11/09:
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: 150911 11:16:51 [ERROR] mysqld got signal 11 ;
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: This could be because you hit a bug. It is also possible that this binary
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: or one of the libraries it was linked against is corrupt, improperly built,
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: or misconfigured. This error can also be caused by malfunctioning hardware.
Sep 11 11:16:51 eggxdb001 mysqld-mariadb:
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: To report this bug, see http://kb.askmonty.org/en/reporting-bugs
Sep 11 11:16:51 eggxdb001 mysqld-mariadb:
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: We will try our best to scrape up some info that will hopefully help
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: diagnose the problem, but since we have already crashed,
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: something is definitely wrong and this may fail.
Sep 11 11:16:51 eggxdb001 mysqld-mariadb:
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: Server version: 10.0.20-MariaDB-log
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: key_buffer_size=805306368
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: read_buffer_size=2097152
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: max_used_connections=20
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: max_threads=802
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: thread_count=13
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: It is possible that mysqld could use up to
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5729758 K bytes of memory
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: Hope that's ok; if not, decrease some variables in the equation.
Sep 11 11:16:51 eggxdb001 mysqld-mariadb:
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: Thread pointer: 0x0x7f0ed0703008
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: Attempting backtrace. You can use the following information to find out
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: where mysqld died. If you see no messages after this, something went
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: terribly wrong...
Sep 11 11:16:51 eggxdb001 mysqld-mariadb: stack_bottom = 0x7f1001cefe10 thread_stack 0x48000
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: mysys/stacktrace.c:247(my_print_stacktrace)[0xbcb2ee]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: sql/signal_handler.cc:153(handle_fatal_signal)[0x71fa8c]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: /lib64/libpthread.so.0(+0xf710)[0x7f10ac9fd710]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: row/row0mysql.cc:996(row_update_prebuilt_trx(row_prebuilt_t*, trx_t*))[0x93f28f]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: handler/ha_innodb.cc:13246(ha_innobase::external_lock(THD*, int))[0x8b94aa]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: sql/handler.cc:5913(handler::ha_external_lock(THD*, int))[0x720f04]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: sql/lock.cc:360(lock_external)[0x7f6a89]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: sql/lock.cc:276(mysql_lock_tables(THD*, TABLE**, unsigned int, unsigned int))[0x7f6ed5]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: sql/sql_base.cc:5404(lock_tables(THD*, TABLE_LIST*, unsigned int, unsigned int))[0x553d7c]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: sql/sql_base.cc:5174(open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*))[0x5591d6]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: sql/sql_parse.cc:5219(execute_sqlcom_select)[0x595ba7]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: sql/sql_parse.cc:2562(mysql_execute_command(THD*))[0x598847]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: sql/sql_parse.cc:6529(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59fc86]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: sql/sql_parse.cc:1310(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x5a1bb7]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: sql/sql_parse.cc:999(do_command(THD*))[0x5a22f9]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: sql/sql_connect.cc:1378(do_handle_one_connection(THD*))[0x66c7d4]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: sql/sql_connect.cc:1295(handle_one_connection)[0x66c912]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: /lib64/libpthread.so.0(+0x79d1)[0x7f10ac9f59d1]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: /lib64/libc.so.6(clone+0x6d)[0x7f10ab9748fd]
Sep 11 11:16:52 eggxdb001 mysqld-mariadb:
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: Trying to get some variables.
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: Some pointers may be invalid and cause the dump to abort.
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: Query (0x7f0ecc4a1020): SELECT DISTINCT COUNT(DISTINCT e.entity_id) FROM `catalog_product_flat_8` AS `e` INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id AND cat_index.store_id='8' AND cat_index.visibility IN(2, 4) AND cat_index.category_id='161' INNER JOIN `catalog_product_index_price` AS `price_index` ON price_index.entity_id = e.entity_id AND price_index.website_id = '2' AND price_index.customer_group_id = 0 WHERE (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('160', '163'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('160', '163'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('160', '163'))))) AND
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('160', '163'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('160', '163'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('160', '163'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('160', '163')))))
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: Connection ID (thread ID): 139282
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: Status: NOT_KILLED
Sep 11 11:16:52 eggxdb001 mysqld-mariadb:
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on
Sep 11 11:16:52 eggxdb001 mysqld-mariadb:
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: information that should help you find out what is causing the crash.
Sep 11 11:16:52 eggxdb001 mysqld-mariadb: Writing a core file
Sep 11 11:16:54 eggxdb001 mysqld_safe-mariadb: Number of processes running now: 0
Sep 11 11:16:54 eggxdb001 mysqld_safe-mariadb: mysqld restarted
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] /db/mariadb/bin/mysqld (mysqld 10.0.20-MariaDB-log) starting as process 4029 ...
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: Using mutexes to ref count buffer pool pages
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: The InnoDB memory heap is disabled
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: Memory barrier is not used
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: Compressed tables use zlib 1.2.3
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: Using Linux native AIO
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: Using CPU crc32 instructions
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: Initializing buffer pool, size = 5.4G
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: Completed initialization of buffer pool
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: Highest supported file format is Barracuda.
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: Log scan progressed past the checkpoint lsn 49540562459
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: Database was not shutdown normally!
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: Starting crash recovery.
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: Reading tablespace information from the .ibd files...
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: Restoring possible half-written data pages
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: 150911 11:16:54 [Note] InnoDB: from the doublewrite buffer...
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 49545805312
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 49551048192
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 49556291072
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 49561533952
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 49566776832
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 49572019712
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 49577262592
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 49582505472
Sep 11 11:16:54 eggxdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 49587748352
Sep 11 11:16:55 eggxdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 49588752829
Sep 11 11:16:55 eggxdb001 mysqld-mariadb: 150911 11:16:55 [Note] InnoDB: Starting an apply batch of log records to the database...
Sep 11 11:16:55 eggxdb001 mysqld-mariadb: InnoDB: Progress in percent: 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
Sep 11 11:16:55 eggxdb001 mysqld-mariadb: InnoDB: Apply batch completed
Sep 11 11:16:55 eggxdb001 mysqld-mariadb: InnoDB: Last MySQL binlog file position 0 82173679, file name /db/admin/binlog/mariadb-bin.000033
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] InnoDB: 128 rollback segment(s) are active.
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] InnoDB: Waiting for purge to start
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.24-72.2 started; log sequence number 49588752829
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] Plugin 'FEEDBACK' is disabled.
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] Recovering after a crash using /db/admin/binlog/mariadb-bin
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] Starting crash recovery...
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] Crash recovery finished.
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] Server socket created on IP: '10.132.19.201'.
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] Event Scheduler: Loaded 0 events
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] Reading Master_info: 'master-eggxdb002.info' Relay_info:'/db/admin/binlog/relay-bin-eggxdb002.info'
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] Initialized Master_info from 'master-eggxdb002.info'
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] Added new Master_info 'eggxdb002' to hash table
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] Started replication for 'eggxdb002'
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] Reading of all Master_info entries succeded
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] Added new Master_info '' to hash table
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] /db/mariadb/bin/mysqld: ready for connections.
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: Version: '10.0.20-MariaDB-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] Master 'eggxdb002': Slave SQL thread initialized, starting replication in log 'mariadb-bin.000032' at position 83218702, relay log '/db/admin/binlog/relay-bin-eggxdb002.000082' position: 41895424
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Note] Master 'eggxdb002': Slave I/O thread: connected to master 'replication@eggxdb002:3306',replication started in log 'mariadb-bin.000032' at position 83218702
Sep 11 11:16:56 eggxdb001 mysqld-mariadb: 150911 11:16:56 [Warning] IP address '10.132.15.17' could not be resolved: Name or service not known
Sep 11 11:17:04 eggxdb001 mysqld-mariadb: 150911 11:17:04 [Note] Start binlog_dump to slave_server(3), pos(mariadb-bin.000033, 82173679)
Sep 11 11:17:04 eggxdb001 mysqld-mariadb: 150911 11:17:04 [Note] Start binlog_dump to slave_server(2), pos(mariadb-bin.000033, 82173679)
Is that sufficient regarding the innodb and sql layer unedited log crash ?
Cheers,
--thomas
Hello Jan,
We had another crash this morning.
mysqld core post-mortem:
(gdb) bt #0 0x00007f9caf3818ac in pthread_kill () from /lib64/libpthread.so.0 #1 0x000000000071fa29 in handle_fatal_signal (sig=11) at /home/buildbot/buildbot/build/sql/signal_handler.cc:262 #2 <signal handler called> #3 que_fork_get_first_thr (buf=0x7f9ac9f7ac20 "\377", mode=2, prebuilt=0x7f9ab8be2068, match_mode=1, direction=0) at /home/buildbot/buildbot/build/storage/xtradb/include/que0que.ic:63 #4 row_search_for_mysql (buf=0x7f9ac9f7ac20 "\377", mode=2, prebuilt=0x7f9ab8be2068, match_mode=1, direction=0) at /home/buildbot/buildbot/build/storage/xtradb/row/row0sel.cc:4058 #5 0x00000000008b06ff in ha_innobase::index_read (this=0x7f9ac9eae820, buf=0x7f9ac9f7ac20 "\377", key_ptr=<value optimized out>, key_len=<value optimized out>, find_flag=HA_READ_KEY_EXACT) at /home/buildbot/buildbot/build/storage/xtradb/handler/ha_innodb.cc:8525 #6 0x0000000000720b5d in handler::ha_index_read_map (this=0x7f9ac9dc1c68, buf=0x7f9ac9f7ac20 "\377", key=0x7f9ac9f58048 "\b", keypart_map=3, find_flag=HA_READ_KEY_EXACT) at /home/buildbot/buildbot/build/sql/handler.cc:2621 #7 0x0000000000724b0b in handler::read_range_first (this=0x7f9ac9eae820, start_key=<value optimized out>, end_key=<value optimized out>, eq_range_arg=<value optimized out>, sorted=<value optimized out>) at /home/buildbot/buildbot/build/sql/handler.cc:5399 #8 0x00000000006a4e45 in handler::multi_range_read_next (this=0x7f9ac9eae820, range_info=0x7f9ae32344d0) at /home/buildbot/buildbot/build/sql/multi_range_read.cc:296 #9 0x00000000006a4ef7 in Mrr_simple_index_reader::get_next (this=0x7f9ac9eaee18, range_info=0x7f9ae32344d0) at /home/buildbot/buildbot/build/sql/multi_range_read.cc:328 #10 0x00000000006a510f in DsMrr_impl::dsmrr_next (this=0x7f9ac9eaecd8, range_info=0x7f9ae32344d0) at /home/buildbot/buildbot/build/sql/multi_range_read.cc:1415 #11 0x0000000000823982 in QUICK_RANGE_SELECT::get_next (this=0x7f9abb0efb80) at /home/buildbot/buildbot/build/sql/opt_range.cc:11775 #12 0x0000000000841c35 in rr_quick (info=0x7f9ac9f181c8) at /home/buildbot/buildbot/build/sql/records.cc:345 #13 0x00000000005dc8b4 in sub_select (join=0x7f9ab8864130, join_tab=0x7f9ac9f18120, end_of_records=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:17812 #14 0x00000000006a9316 in join_tab_execution_startup (tab=<value optimized out>) at /home/buildbot/buildbot/build/sql/opt_subselect.cc:5072 #15 0x00000000005dc888 in sub_select (join=0x7f9ab8864130, join_tab=0x7f9ac9f15c30, end_of_records=false) at /home/buildbot/buildbot/build/sql/sql_select.cc:17803 #16 0x00000000005dc65c in evaluate_join_record (join=0x7f9ab8864130, join_tab=0x7f9ac9f15908, error=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:18037 #17 0x00000000005dc8cb in sub_select (join=0x7f9ab8864130, join_tab=0x7f9ac9f15908, end_of_records=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:17815 #18 0x00000000005dc65c in evaluate_join_record (join=0x7f9ab8864130, join_tab=0x7f9ac9f155e0, error=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:18037 #19 0x00000000005dc8cb in sub_select (join=0x7f9ab8864130, join_tab=0x7f9ac9f155e0, end_of_records=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:17815 #20 0x00000000005dc65c in evaluate_join_record (join=0x7f9ab8864130, join_tab=0x7f9ac9f152b8, error=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:18037 #21 0x00000000005dc8cb in sub_select (join=0x7f9ab8864130, join_tab=0x7f9ac9f152b8, end_of_records=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:17815 #22 0x00000000005dc65c in evaluate_join_record (join=0x7f9ab8864130, join_tab=0x7f9ac9f14f90, error=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:18037 #23 0x00000000005dc8cb in sub_select (join=0x7f9ab8864130, join_tab=0x7f9ac9f14f90, end_of_records=<value optimized out>) at /home/buildbot/buildbot/build/sql/sql_select.cc:17815 #24 0x00000000005dcfdd in do_select (join=0x7f9ab8864130, fields=0x0, table=0x7f9ac9f99020, procedure=0x0) at /home/buildbot/buildbot/build/sql/sql_select.cc:17477 #25 0x00000000005ef5c0 in JOIN::exec_inner (this=0x7f9ab8864130) at /home/buildbot/buildbot/build/sql/sql_select.cc:2622 #26 0x00000000005ee19a in exec (thd=0x7f9abb1c1008, rref_pointer_array=0x7f9abb1c5190, tables=0x7f9ab6c16020, wild_num=0, fields=<value optimized out>, conds=0x7f9ac9fd0548, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748609, result=0x7f9ab8864110, unit=0x7f9abb1c4828, select_lex=0x7f9abb1c4f18) at /home/buildbot/buildbot/build/sql/sql_select.cc:2368 #27 mysql_select (thd=0x7f9abb1c1008, rref_pointer_array=0x7f9abb1c5190, tables=0x7f9ab6c16020, wild_num=0, fields=<value optimized out>, conds=0x7f9ac9fd0548, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748609, result=0x7f9ab8864110, unit=0x7f9abb1c4828, select_lex=0x7f9abb1c4f18) at /home/buildbot/buildbot/build/sql/sql_select.cc:3308 #28 0x00000000005f1dcd in handle_select (thd=0x7f9abb1c1008, lex=0x7f9abb1c4760, result=0x7f9ab8864110, setup_tables_done_option=0) at /home/buildbot/buildbot/build/sql/sql_select.cc:373 #29 0x0000000000595bf0 in execute_sqlcom_select (thd=0x7f9abb1c1008, all_tables=0x7f9ab6c16020) at /home/buildbot/buildbot/build/sql/sql_parse.cc:5274 #30 0x0000000000598847 in mysql_execute_command (thd=0x7f9abb1c1008) at /home/buildbot/buildbot/build/sql/sql_parse.cc:2562 #31 0x000000000059fc86 in mysql_parse (thd=0x7f9abb1c1008, rawbuf=0x7f9ac9fae020 "SELECT DISTINCT 1 AS `status`, `e`.`entity_id`, `e`.`type_id`, `e`.`attribute_set_id`, `cat_index`.`position` AS `cat_index_position`, `mile_price_index`.`fb_miles`, `mile_price_index`.`fb_home_miles`"..., length=<value optimized out>, parser_state=0x7f9ae3236bd0) at /home/buildbot/buildbot/build/sql/sql_parse.cc:6529 #32 0x00000000005a1bb7 in dispatch_command (command=COM_QUERY, thd=0x7f9abb1c1008, packet=<value optimized out>, packet_length=3808) at /home/buildbot/buildbot/build/sql/sql_parse.cc:1308 #33 0x00000000005a22f9 in do_command (thd=0x7f9abb1c1008) at /home/buildbot/buildbot/build/sql/sql_parse.cc:999 #34 0x000000000066c7d4 in do_handle_one_connection (thd_arg=0x7f9abb1c1008) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1378 #35 0x000000000066c912 in handle_one_connection (arg=0x7f9abb386008) at /home/buildbot/buildbot/build/sql/sql_connect.cc:1293 #36 0x00007f9caf37c9d1 in start_thread () from /lib64/libpthread.so.0 #37 0x00007f9cae2fb8fd in clone () from /lib64/libc.so.6 (gdb) print *row_search_for_mysql::prebuilt $1 = {magic_n = 78540783, table = 0x7f9ad10b59e8, index = 0x7f9ad10b63e8, trx = 0x7f9ac9dc1c68, sql_stat_start = 1, mysql_has_locked = 1, clust_index_was_generated = 0, index_usable = 1, read_just_key = 1, used_in_HANDLER = 0, template_type = 1, n_template = 3, null_bitmap_len = 1, need_to_access_clustered = 0, templ_contains_blob = 0, mysql_template = 0x7f9ac9d576e8, heap = 0x7f9ab8be2000, ins_node = 0x0, ins_upd_rec_buff = 0x0, default_rec = 0x7f9ab98170c8 "\377", hint_need_to_fetch_extra_cols = 0, upd_node = 0x0, trx_id = 0, ins_graph = 0x0, upd_graph = 0x0, pcur = {btr_cur = {index = 0x7f9ad10b63e8, page_cur = {rec = 0x7f9b8dd89a5e "", block = 0x7f9b74371300}, purge_node = 0x0, left_block = 0x0, thr = 0x0, flag = BTR_CUR_BINARY, tree_height = 2, up_match = 2, up_bytes = 0, low_match = 1, low_bytes = 3, n_fields = 0, n_bytes = 0, fold = 0, path_arr = 0x0}, latch_mode = 1, old_stored = 908467085, old_rec = 0x7f9ac9cabaee "", old_n_fields = 6, rel_pos = 2, block_when_stored = 0x7f9b74371300, modify_clock = 1114, pos_state = BTR_PCUR_IS_POSITIONED, search_mode = 2, trx_if_known = 0x7f9ac9dc1c68, old_rec_buf = 0x7f9ac9cabae8 "", buf_size = 24}, clust_pcur = {btr_cur = {index = 0x0, page_cur = {rec = 0x0, block = 0x0}, purge_node = 0x0, left_block = 0x0, thr = 0x0, flag = 0, tree_height = 0, up_match = 0, up_bytes = 0, low_match = 0, low_bytes = 0, n_fields = 0, n_bytes = 0, fold = 0, path_arr = 0x0}, latch_mode = 3, old_stored = 122766467, old_rec = 0x0, old_n_fields = 0, rel_pos = 0, block_when_stored = 0x0, modify_clock = 0, pos_state = BTR_PCUR_NOT_POSITIONED, search_mode = 0, trx_if_known = 0x0, old_rec_buf = 0x0, buf_size = 0}, sel_graph = 0x4028000000000000, search_tuple = 0x7f9ab8be2400, row_id = "\000\000\000\000\000", fts_doc_id = 0, clust_ref = 0x7f9ab8be25e0, select_lock_type = 5, stored_select_lock_type = 5, row_read_type = 0, new_rec_locks = 0, mysql_prefix_len = 17, mysql_row_len = 19, n_rows_fetched = 0, fetch_direction = 1, fetch_cache = {0x7f9ab84985ec "", 0x7f9ab8498607 "", 0x7f9ab8498622 "", 0x7f9ab849863d "", 0x7f9ab8498658 "\220\200", 0x7f9ab8498673 "", 0x7f9ab849868e "", 0x7f9ab84986a9 ""}, keep_other_fields_on_keyread = 0, fetch_cache_first = 0, n_fetch_cached = 0, blob_heap = 0x0, old_vers_heap = 0x0, in_fts_query = false, autoinc_last_value = 0, autoinc_increment = 1, autoinc_offset = 0, autoinc_error = DB_SUCCESS, idx_cond = 0x0, idx_cond_n_cols = 0, magic_n2 = 78540783, innodb_api = 0, innodb_api_rec = 0x0, srch_key_val1 = 0x7f9ab8be23d8 "", srch_key_val2 = 0x7f9ab8be23ea "", srch_key_val_len = 18}
As you can see the sel_graph value is set out of bounds.
Here is the full/unedited mysqld error log:
Sep 23 08:21:22 eggpdb001 mysqld-mariadb: 150923 8:21:22 [ERROR] mysqld got signal 11 ; Sep 23 08:21:22 eggpdb001 mysqld-mariadb: This could be because you hit a bug. It is also possible that this binary Sep 23 08:21:22 eggpdb001 mysqld-mariadb: or one of the libraries it was linked against is corrupt, improperly built, Sep 23 08:21:22 eggpdb001 mysqld-mariadb: or misconfigured. This error can also be caused by malfunctioning hardware. Sep 23 08:21:22 eggpdb001 mysqld-mariadb: Sep 23 08:21:22 eggpdb001 mysqld-mariadb: To report this bug, see http://kb.askmonty.org/en/reporting-bugs Sep 23 08:21:22 eggpdb001 mysqld-mariadb: Sep 23 08:21:22 eggpdb001 mysqld-mariadb: We will try our best to scrape up some info that will hopefully help Sep 23 08:21:22 eggpdb001 mysqld-mariadb: diagnose the problem, but since we have already crashed, Sep 23 08:21:22 eggpdb001 mysqld-mariadb: something is definitely wrong and this may fail. Sep 23 08:21:22 eggpdb001 mysqld-mariadb: Sep 23 08:21:22 eggpdb001 mysqld-mariadb: Server version: 10.0.20-MariaDB-log Sep 23 08:21:22 eggpdb001 mysqld-mariadb: key_buffer_size=805306368 Sep 23 08:21:22 eggpdb001 mysqld-mariadb: read_buffer_size=2097152 Sep 23 08:21:22 eggpdb001 mysqld-mariadb: max_used_connections=42 Sep 23 08:21:22 eggpdb001 mysqld-mariadb: max_threads=802 Sep 23 08:21:22 eggpdb001 mysqld-mariadb: thread_count=14 Sep 23 08:21:22 eggpdb001 mysqld-mariadb: It is possible that mysqld could use up to Sep 23 08:21:22 eggpdb001 mysqld-mariadb: key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5729758 K bytes of memory Sep 23 08:21:22 eggpdb001 mysqld-mariadb: Hope that's ok; if not, decrease some variables in the equation. Sep 23 08:21:22 eggpdb001 mysqld-mariadb: Sep 23 08:21:22 eggpdb001 mysqld-mariadb: Thread pointer: 0x0x7f9abb1c1008 Sep 23 08:21:22 eggpdb001 mysqld-mariadb: Attempting backtrace. You can use the following information to find out Sep 23 08:21:22 eggpdb001 mysqld-mariadb: where mysqld died. If you see no messages after this, something went Sep 23 08:21:22 eggpdb001 mysqld-mariadb: terribly wrong... Sep 23 08:21:22 eggpdb001 mysqld-mariadb: stack_bottom = 0x7f9ae3236e10 thread_stack 0x48000 Sep 23 08:21:23 eggpdb001 mysqld-mariadb: mysys/stacktrace.c:247(my_print_stacktrace)[0xbcb2ee] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/signal_handler.cc:153(handle_fatal_signal)[0x71fa8c] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: /lib64/libpthread.so.0(+0xf710)[0x7f9caf384710] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: include/que0que.ic:63(que_fork_get_first_thr)[0x955c27] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: handler/ha_innodb.cc:8525(ha_innobase::index_read(unsigned char*, unsigned char const*, unsigned int, ha_rkey_function))[0x8b06ff] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/handler.cc:2621(handler::ha_index_read_map(unsigned char*, unsigned char const*, unsigned long, ha_rkey_function))[0x720b5d] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/handler.cc:5399(handler::read_range_first(st_key_range const*, st_key_range const*, bool, bool))[0x724b0b] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/multi_range_read.cc:297(handler::multi_range_read_next(void**))[0x6a4e45] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/multi_range_read.cc:328(Mrr_simple_index_reader::get_next(void**))[0x6a4ef7] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/multi_range_read.cc:1415(DsMrr_impl::dsmrr_next(void**))[0x6a510f] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/opt_range.cc:11777(QUICK_RANGE_SELECT::get_next())[0x823982] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/records.cc:345(rr_quick)[0x841c35] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_select.cc:17813(sub_select(JOIN*, st_join_table*, bool))[0x5dc8b4] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/opt_subselect.cc:5072(join_tab_execution_startup(st_join_table*))[0x6a9316] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_select.cc:17803(sub_select(JOIN*, st_join_table*, bool))[0x5dc888] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_select.cc:18037(evaluate_join_record)[0x5dc65c] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_select.cc:17815(sub_select(JOIN*, st_join_table*, bool))[0x5dc8cb] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_select.cc:18037(evaluate_join_record)[0x5dc65c] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_select.cc:17815(sub_select(JOIN*, st_join_table*, bool))[0x5dc8cb] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_select.cc:18037(evaluate_join_record)[0x5dc65c] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_select.cc:17815(sub_select(JOIN*, st_join_table*, bool))[0x5dc8cb] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_select.cc:18037(evaluate_join_record)[0x5dc65c] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_select.cc:17815(sub_select(JOIN*, st_join_table*, bool))[0x5dc8cb] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_select.cc:17478(do_select)[0x5dcfdd] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_select.cc:2624(JOIN::exec_inner())[0x5ef5c0] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_select.cc:2370(JOIN::exec())[0x5ee19a] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_select.cc:373(handle_select(THD*, LEX*, select_result*, unsigned long))[0x5f1dcd] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_parse.cc:5275(execute_sqlcom_select)[0x595bf0] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_parse.cc:2562(mysql_execute_command(THD*))[0x598847] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_parse.cc:6529(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x59fc86] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_parse.cc:1310(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x5a1bb7] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_parse.cc:999(do_command(THD*))[0x5a22f9] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_connect.cc:1378(do_handle_one_connection(THD*))[0x66c7d4] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: sql/sql_connect.cc:1295(handle_one_connection)[0x66c912] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: /lib64/libpthread.so.0(+0x79d1)[0x7f9caf37c9d1] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: /lib64/libc.so.6(clone+0x6d)[0x7f9cae2fb8fd] Sep 23 08:21:23 eggpdb001 mysqld-mariadb: Sep 23 08:21:23 eggpdb001 mysqld-mariadb: Trying to get some variables. Sep 23 08:21:23 eggpdb001 mysqld-mariadb: Some pointers may be invalid and cause the dump to abort. Sep 23 08:21:23 eggpdb001 mysqld-mariadb: Query (0x7f9ac9fae020): SELECT DISTINCT 1 AS `status`, `e`.`entity_id`, `e`.`type_id`, `e`.`attribute_set_id`, `cat_index`.`position` AS `cat_index_position`, `mile_price_index`.`fb_miles`, `mile_price_index`.`fb_home_miles`, `mile_price_index`.`min_price` AS `min_miles`, `mile_price_index`.`max_price` AS `max_miles`, `e`.`name`, `e`.`short_description`, `e`.`price`, `e`.`special_price`, `e`.`special_from_date`, `e`.`special_to_date`, `e`.`small_image`, `e`.`thumbnail`, `e`.`news_from_date`, `e`.`news_to_date`, `e`.`url_key`, `e`.`required_options`, `e`.`image_label`, `e`.`small_image_label`, `e`.`thumbnail_label`, `e`.`msrp_enabled`, `e`.`msrp_display_actual_price_type`, `e`.`msrp`, `e`.`tax_class_id`, `e`.`price_type`, `e`.`weight_type`, `e`.`price_view`, `e`.`shipment_type`, `e`.`links_purchased_separately`, `e`.`links_exist`, `e`.`giftcard_amounts`, `e`.`allow_open_amount`, `e`.`open_amount_min`, `e`.`open_amount_max`, `e`.`brand_id`, `e`.`brand_id_value`, `e`.`fb_miles`, `e`.`fb_home_miles`, `e`.`label_ Sep 23 08:21:23 eggpdb001 mysqld-mariadb: id`, `e`.`label_id_value`, `e`.`scale`, `e`.`scale_value`, `e`.`delivery_rule_ids`, `e`.`target_group_ids`, `e`.`warranty`, `e`.`is_home_page`, `e`.`show_in_campaign`, `e`.`campaign_slider_image`, `e`.`campaign_main_image`, `e`.`campaign_text`, `e`.`fb_home_earn`, `e`.`fb_home_earn_promo`, `e`.`fb_home_earn_promo_code`, `e`.`fb_onboard_earn`, `e`.`fb_onboard_earn_promo`, `e`.`fb_onboard_earn_promo_code`, `e`.`color`, `e`.`your_surprise_id`, `e`.`prologistica_number`, `e`.`waterproof`, `e`.`prologistica_send`, `e`.`local_price`, `price_index`.`price`, `price_index`.`tax_class_id`, `price_index`.`final_price`, IF(price_index.tier_price IS NOT NULL, LEAST(price_index.min_price, price_index.tier_price), price_index.min_price) AS `minimal_price`, `price_index`.`min_price`, `price_index`.`max_price`, `price_index`.`tier_price` FROM `catalog_product_flat_8` AS `e` INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id AND cat_index.store_id='8' AND cat_index.visibility IN(2, Sep 23 08:21:23 eggpdb001 mysqld-mariadb: 4) AND cat_index.category_id='128' LEFT JOIN `egg_flyingblue_index` AS `mile_price_index` ON e.entity_id = mile_price_index.entity_id INNER JOIN `catalog_product_index_price` AS `price_index` ON price_index.entity_id = e.entity_id AND price_index.website_id = '2' AND price_index.customer_group_id = 0 WHERE (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('128', '131'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('128', '131'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('128', '131'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_i Sep 23 08:21:23 eggpdb001 mysqld-mariadb: d in ('128', '131'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('128', '131'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('128', '131'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('128', '131'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('128', '131'))))) Sep 23 08:21:23 eggpdb001 mysqld-mariadb: Connection ID (thread ID): 4264662 Sep 23 08:21:23 eggpdb001 mysqld-mariadb: Status: NOT_KILLED Sep 23 08:21:23 eggpdb001 mysqld-mariadb: Sep 23 08:21:23 eggpdb001 mysqld-mariadb: Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on Sep 23 08:21:23 eggpdb001 mysqld-mariadb: Sep 23 08:21:23 eggpdb001 mysqld-mariadb: The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains Sep 23 08:21:23 eggpdb001 mysqld-mariadb: information that should help you find out what is causing the crash. Sep 23 08:21:23 eggpdb001 mysqld-mariadb: Writing a core file Sep 23 08:22:20 eggpdb001 mysqld_safe-mariadb: Number of processes running now: 0 Sep 23 08:22:20 eggpdb001 mysqld_safe-mariadb: mysqld restarted Sep 23 08:22:20 eggpdb001 mysqld-mariadb: 150923 8:22:20 [Note] /db/mariadb/bin/mysqld (mysqld 10.0.20-MariaDB-log) starting as process 28790 ... Sep 23 08:22:20 eggpdb001 mysqld-mariadb: 150923 8:22:20 [Note] InnoDB: Using mutexes to ref count buffer pool pages Sep 23 08:22:20 eggpdb001 mysqld-mariadb: 150923 8:22:20 [Note] InnoDB: The InnoDB memory heap is disabled Sep 23 08:22:20 eggpdb001 mysqld-mariadb: 150923 8:22:20 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins Sep 23 08:22:20 eggpdb001 mysqld-mariadb: 150923 8:22:20 [Note] InnoDB: Memory barrier is not used Sep 23 08:22:20 eggpdb001 mysqld-mariadb: 150923 8:22:20 [Note] InnoDB: Compressed tables use zlib 1.2.3 Sep 23 08:22:20 eggpdb001 mysqld-mariadb: 150923 8:22:20 [Note] InnoDB: Using Linux native AIO Sep 23 08:22:20 eggpdb001 mysqld-mariadb: 150923 8:22:20 [Note] InnoDB: Using CPU crc32 instructions Sep 23 08:22:20 eggpdb001 mysqld-mariadb: 150923 8:22:20 [Note] InnoDB: Initializing buffer pool, size = 5.4G Sep 23 08:22:20 eggpdb001 mysqld-mariadb: 150923 8:22:20 [Note] InnoDB: Completed initialization of buffer pool Sep 23 08:22:20 eggpdb001 mysqld-mariadb: 150923 8:22:20 [Note] InnoDB: Highest supported file format is Barracuda. Sep 23 08:22:20 eggpdb001 mysqld-mariadb: 150923 8:22:20 [Note] InnoDB: Log scan progressed past the checkpoint lsn 127456692121 Sep 23 08:22:20 eggpdb001 mysqld-mariadb: 150923 8:22:20 [Note] InnoDB: Database was not shutdown normally! Sep 23 08:22:20 eggpdb001 mysqld-mariadb: 150923 8:22:20 [Note] InnoDB: Starting crash recovery. Sep 23 08:22:20 eggpdb001 mysqld-mariadb: 150923 8:22:20 [Note] InnoDB: Reading tablespace information from the .ibd files... Sep 23 08:22:29 eggpdb001 mysqld-mariadb: 150923 8:22:29 [Note] InnoDB: Restoring possible half-written data pages Sep 23 08:22:29 eggpdb001 mysqld-mariadb: 150923 8:22:29 [Note] InnoDB: from the doublewrite buffer... Sep 23 08:22:29 eggpdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 127461934592 Sep 23 08:22:30 eggpdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 127467177472 Sep 23 08:22:30 eggpdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 127472420352 Sep 23 08:22:30 eggpdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 127477663232 Sep 23 08:22:30 eggpdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 127482906112 Sep 23 08:22:31 eggpdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 127488148992 Sep 23 08:22:31 eggpdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 127493391872 Sep 23 08:22:31 eggpdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 127498634752 Sep 23 08:22:32 eggpdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 127503877632 Sep 23 08:22:32 eggpdb001 mysqld-mariadb: InnoDB: Doing recovery: scanned up to log sequence number 127504718797 Sep 23 08:22:32 eggpdb001 mysqld-mariadb: InnoDB: Transaction 293292002 was in the XA prepared state. Sep 23 08:22:32 eggpdb001 mysqld-mariadb: InnoDB: Transaction 293292002 was in the XA prepared state. Sep 23 08:22:32 eggpdb001 mysqld-mariadb: InnoDB: 1 transaction(s) which must be rolled back or cleaned up Sep 23 08:22:32 eggpdb001 mysqld-mariadb: InnoDB: in total 0 row operations to undo Sep 23 08:22:32 eggpdb001 mysqld-mariadb: InnoDB: Trx id counter is 293292288 Sep 23 08:22:32 eggpdb001 mysqld-mariadb: 150923 8:22:32 [Note] InnoDB: Starting an apply batch of log records to the database... Sep 23 08:22:41 eggpdb001 mysqld-mariadb: InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 Sep 23 08:22:41 eggpdb001 mysqld-mariadb: InnoDB: Apply batch completed Sep 23 08:22:41 eggpdb001 mysqld-mariadb: InnoDB: Last MySQL binlog file position 0 14904564, file name /db/admin/binlog/mariadb-bin.000436 Sep 23 08:22:41 eggpdb001 mysqld-mariadb: InnoDB: Starting in background the rollback of uncommitted transactions Sep 23 08:22:41 eggpdb001 mysqld-mariadb: 2015-09-23 08:22:41 7f1b87bff700 InnoDB: Rollback of non-prepared transactions completed Sep 23 08:22:41 eggpdb001 mysqld-mariadb: 150923 8:22:41 [Note] InnoDB: 128 rollback segment(s) are active. Sep 23 08:22:41 eggpdb001 mysqld-mariadb: 150923 8:22:41 [Note] InnoDB: Waiting for purge to start Sep 23 08:22:41 eggpdb001 mysqld-mariadb: 150923 8:22:41 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.24-72.2 started; log sequence number 127504718797 Sep 23 08:22:41 eggpdb001 mysqld-mariadb: 150923 8:22:41 [Note] Plugin 'FEEDBACK' is disabled. Sep 23 08:22:41 eggpdb001 mysqld-mariadb: 150923 8:22:41 [Note] Recovering after a crash using /db/admin/binlog/mariadb-bin Sep 23 08:22:41 eggpdb001 mysqld-mariadb: 150923 8:22:41 [Note] Starting crash recovery... Sep 23 08:22:41 eggpdb001 mysqld-mariadb: 2015-09-23 08:22:41 7f1d69ede760 InnoDB: Starting recovery for XA transactions... Sep 23 08:22:41 eggpdb001 mysqld-mariadb: 2015-09-23 08:22:41 7f1d69ede760 InnoDB: Transaction 293292002 in prepared state after recovery Sep 23 08:22:41 eggpdb001 mysqld-mariadb: 2015-09-23 08:22:41 7f1d69ede760 InnoDB: Transaction contains changes to 3 rows Sep 23 08:22:41 eggpdb001 mysqld-mariadb: 2015-09-23 08:22:41 7f1d69ede760 InnoDB: 1 transactions in prepared state after recovery Sep 23 08:22:41 eggpdb001 mysqld-mariadb: 150923 8:22:41 [Note] Found 1 prepared transaction(s) in InnoDB Sep 23 08:22:41 eggpdb001 mysqld-mariadb: 150923 8:22:41 [Note] Crash recovery finished. Sep 23 08:22:41 eggpdb001 mysqld-mariadb: 150923 8:22:41 [Note] Server socket created on IP: '10.132.16.201'. Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [ERROR] mysqld: Table './mysql/user' is marked as crashed and should be repaired Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [ERROR] mysqld: Table 'user' is marked as crashed and should be repaired Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [Warning] Checking table: './mysql/user' Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [ERROR] mysql.user: 1 client is using or hasn't closed the table properly Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [ERROR] mysqld: Table './mysql/db' is marked as crashed and should be repaired Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [ERROR] mysqld: Table 'db' is marked as crashed and should be repaired Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [Warning] Checking table: './mysql/db' Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [ERROR] mysql.db: 1 client is using or hasn't closed the table properly Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [Note] Event Scheduler: Loaded 0 events Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [Note] Reading Master_info: 'master-eggpdb002.info' Relay_info:'/db/admin/binlog/relay-bin-eggpdb002.info' Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [Note] Initialized Master_info from 'master-eggpdb002.info' Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [Note] Added new Master_info 'eggpdb002' to hash table Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [Note] Started replication for 'eggpdb002' Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [Note] Reading of all Master_info entries succeded Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [Note] Added new Master_info '' to hash table Sep 23 08:22:42 eggpdb001 mysqld-mariadb: 150923 8:22:42 [Note] /db/mariadb/bin/mysqld: ready for connections.
Is there any more trace or information we could provide to get a better grasp on this issue?
More core files can be uploaded to you if that could help.
Please let us know.
Kind Regards,
--thomas
Hello Jan,
We had 6 crashes in less than 1-hour this morning.
I have created compilation of core + binlog + mysql log for each crash and uploaded them to ftp.askmonty.org as:
MDEV-8736_crash_20151005-0821.tgz MDEV-8736_crash_20151005-0853.tgz MDEV-8736_crash_20151005-0905.tgz-2 MDEV-8736_crash_20151005-0935.tgz MDEV-8736_crash_20151005-0936.tgz MDEV-8736_crash_20151005-0941.tgz
You can find here the queries which provoked a crash in the last month:
Sep 10 14:09:40 Query (0x7fe411c4e020): SELECT ENGINE,SUM(DATA_LENGTH),COUNT(ENGINE) FROM information_schema.TABLES WHERE TABLE_SCHEMA NOT IN ('information_schema','mysql') AND ENGINE IS NOT NULL GROUP BY ENGINE ORDER BY ENGINE ASC
Sep 23 08:21:23 Query (0x7f9ac9fae020): SELECT DISTINCT 1 AS `status`, `e`.`entity_id`, `e`.`type_id`, `e`.`attribute_set_id`, `cat_index`.`position` AS `cat_index_position`, `mile_price_index`.`ca_miles`, `mile_price_index`.`ca_home_miles`, `mile_price_index`.`min_price` AS `min_miles`, `mile_price_index`.`max_price` AS `max_miles`, `e`.`name`, `e`.`short_description`, `e`.`price`, `e`.`special_price`, `e`.`special_from_date`, `e`.`special_to_date`, `e`.`small_image`, `e`.`thumbnail`, `e`.`news_from_date`, `e`.`news_to_date`, `e`.`url_key`, `e`.`required_options`, `e`.`image_label`, `e`.`small_image_label`, `e`.`thumbnail_label`, `e`.`msrp_enabled`, `e`.`msrp_display_actual_price_type`, `e`.`msrp`, `e`.`tax_class_id`, `e`.`price_type`, `e`.`weight_type`, `e`.`price_view`, `e`.`shipment_type`, `e`.`links_purchased_separately`, `e`.`links_exist`, `e`.`giftcard_amounts`, `e`.`allow_open_amount`, `e`.`open_amount_min`, `e`.`open_amount_max`, `e`.`brand_id`, `e`.`brand_id_value`, `e`.`ca_miles`, `e`.`ca_home_miles`, `e`.`label_id`, `e`.`label_id_value`, `e`.`scale`, `e`.`scale_value`, `e`.`delivery_rule_ids`, `e`.`target_group_ids`, `e`.`warranty`, `e`.`is_home_page`, `e`.`show_in_campaign`, `e`.`campaign_slider_image`, `e`.`campaign_main_image`, `e`.`campaign_text`, `e`.`ca_home_earn`, `e`.`ca_home_earn_promo`, `e`.`ca_home_earn_promo_code`, `e`.`ca_onboard_earn`, `e`.`ca_onboard_earn_promo`, `e`.`ca_onboard_earn_promo_code`, `e`.`color`, `e`.`your_surprise_id`, `e`.`prologistica_number`, `e`.`waterproof`, `e`.`prologistica_send`, `e`.`local_price`, `price_index`.`price`, `price_index`.`tax_class_id`, `price_index`.`final_price`, IF(price_index.tier_price IS NOT NULL, LEAST(price_index.min_price, price_index.tier_price), price_index.min_price) AS `minimal_price`, `price_index`.`min_price`, `price_index`.`max_price`, `price_index`.`tier_price` FROM `catalog_product_flat_8` AS `e` INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id AND cat_index.store_id='8' AND cat_index.visibility IN(2,4) AND cat_index.category_id='128' LEFT JOIN `egg_cat_index` AS `mile_price_index` ON e.entity_id = mile_price_index.entity_id INNER JOIN `catalog_product_index_price` AS `price_index` ON price_index.entity_id = e.entity_id AND price_index.website_id = '2' AND price_index.customer_group_id = 0 WHERE (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('128', '131'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('128', '131'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('128', '131'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('128', '131'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('128', '131'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('128', '131'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('128', '131'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('128', '131')))))
Sep 27 22:50:13 Query (0x7f1b854bf020): SELECT 1 AS `status`, `e`.`entity_id`, `e`.`type_id`, `e`.`attribute_set_id`, `cat_index`.`position` AS `cat_index_position`, `mile_price_index`.`ca_miles`, `mile_price_index`.`ca_home_miles`, `mile_price_index`.`min_price` AS `min_miles`, `mile_price_index`.`max_price` AS `max_miles`, `e`.`name`, `e`.`short_description`, `e`.`price`, `e`.`special_price`, `e`.`special_from_date`, `e`.`special_to_date`, `e`.`small_image`, `e`.`thumbnail`, `e`.`news_from_date`, `e`.`news_to_date`, `e`.`url_key`, `e`.`required_options`, `e`.`image_label`, `e`.`small_image_label`, `e`.`thumbnail_label`, `e`.`msrp_enabled`, `e`.`msrp_display_actual_price_type`, `e`.`msrp`, `e`.`tax_class_id`, `e`.`price_type`, `e`.`weight_type`, `e`.`price_view`, `e`.`shipment_type`, `e`.`links_purchased_separately`, `e`.`links_exist`, `e`.`giftcard_amounts`, `e`.`allow_open_amount`, `e`.`open_amount_min`, `e`.`open_amount_max`, `e`.`brand_id`, `e`.`brand_id_value`, `e`.`ca_miles`, `e`.`ca_home_miles`, `e`.`label_id`, `e`.`label_id_value`, `e`.`scale`, `e`.`scale_value`, `e`.`delivery_rule_ids`, `e`.`target_group_ids`, `e`.`warranty`, `e`.`is_home_page`, `e`.`show_in_campaign`, `e`.`campaign_slider_image`, `e`.`campaign_main_image`, `e`.`campaign_text`, `e`.`ca_home_earn`, `e`.`ca_home_earn_promo`, `e`.`ca_home_earn_promo_code`, `e`.`ca_onboard_earn`, `e`.`ca_onboard_earn_promo`, `e`.`ca_onboard_earn_promo_code`, `e`.`color`, `e`.`your_surprise_id`, `e`.`prologistica_number`, `e`.`waterproof`, `e`.`prologistica_send`, `e`.`local_price`, `price_index`.`price`, `price_index`.`tax_class_id`, `price_index`.`final_price`, IF(price_index.tier_price IS NOT NULL, LEAST(price_index.min_price, price_index.tier_price), price_index.min_price) AS `minimal_price`, `price_index`.`min_price`, `price_index`.`max_price`, `price_index`.`tier_price` FROM `catalog_product_flat_1` AS `e` INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id AND cat_index.store_id=1 AND cat_index.visibility IN(2, 4) AND cat_index.category_id='33' LEFT JOIN `egg_cat_index` AS `mile_price_index` ON e.entity_id = mile_price_index.entity_id INNER JOIN `catalog_product_index_price` AS `price_index` ON price_index.entity_id = e.entity_id AND price_index.website_id = '1' AND price_index.customer_group_id = 0 ORDER BY `e`.`name` asc LIMIT 12 OFFSET 132
Sep 27 22:57:00 Query (0x7f2430033020): SELECT `egg_bitly_urls`.* FROM `egg_bitly_urls` WHERE (url='https://shop.egg.com/nl_en/comfort-bag-for-him.html')
Sep 27 23:23:30 Query (0x7f63fb4eb020): SELECT DISTINCT 1 AS `status`, `e`.`entity_id`, `e`.`type_id`, `e`.`attribute_set_id`, `cat_index`.`position` AS `cat_index_position`, `mile_price_index`.`ca_miles`, `mile_price_index`.`ca_home_miles`, `mile_price_index`.`min_price` AS `min_miles`, `mile_price_index`.`max_price` AS `max_miles`, `e`.`name`, `e`.`short_description`, `e`.`price`, `e`.`special_price`, `e`.`special_from_date`, `e`.`special_to_date`, `e`.`small_image`, `e`.`thumbnail`, `e`.`news_from_date`, `e`.`news_to_date`, `e`.`url_key`, `e`.`required_options`, `e`.`image_label`, `e`.`small_image_label`, `e`.`thumbnail_label`, `e`.`msrp_enabled`, `e`.`msrp_display_actual_price_type`, `e`.`msrp`, `e`.`tax_class_id`, `e`.`price_type`, `e`.`weight_type`, `e`.`price_view`, `e`.`shipment_type`, `e`.`links_purchased_separately`, `e`.`links_exist`, `e`.`giftcard_amounts`, `e`.`allow_open_amount`, `e`.`open_amount_min`, `e`.`open_amount_max`, `e`.`brand_id`, `e`.`brand_id_value`, `e`.`ca_miles`, `e`.`ca_home_miles`, `e`.`label_id`, `e`.`label_id_value`, `e`.`scale`, `e`.`scale_value`, `e`.`delivery_rule_ids`, `e`.`target_group_ids`, `e`.`warranty`, `e`.`is_home_page`, `e`.`show_in_campaign`, `e`.`campaign_slider_image`, `e`.`campaign_main_image`, `e`.`campaign_text`, `e`.`ca_home_earn`, `e`.`ca_home_earn_promo`, `e`.`ca_home_earn_promo_code`, `e`.`ca_onboard_earn`, `e`.`ca_onboard_earn_promo`, `e`.`ca_onboard_earn_promo_code`, `e`.`color`, `e`.`your_surprise_id`, `e`.`prologistica_number`, `e`.`waterproof`, `e`.`prologistica_send`, `e`.`local_price`, `price_index`.`price`, `price_index`.`tax_class_id`, `price_index`.`final_price`, IF(price_index.tier_price IS NOT NULL, LEAST(price_index.min_price, price_index.tier_price), price_index.min_price) AS `minimal_price`, `price_index`.`min_price`, `price_index`.`max_price`, `price_index`.`tier_price` FROM `catalog_product_flat_8` AS `e` INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id AND cat_index.store_id='8' AND cat_index.visibility IN(2,4) AND cat_index.category_id='158' LEFT JOIN `egg_cat_index` AS `mile_price_index` ON e.entity_id = mile_price_index.entity_id INNER JOIN `catalog_product_index_price` AS `price_index` ON price_index.entity_id = e.entity_id AND price_index.website_id = '2' AND price_index.customer_group_id = 0 WHERE (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '159'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '159'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '159'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '159'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '159'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '159'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '159'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '159')))))
Sep 27 23:23:48 Query (0x7f4959845020): SELECT DISTINCT 1 AS `status`, `e`.`entity_id`, `e`.`type_id`, `e`.`attribute_set_id`, `cat_index`.`position` AS `cat_index_position`, `mile_price_index`.`ca_miles`, `mile_price_index`.`ca_home_miles`, `mile_price_index`.`min_price` AS `min_miles`, `mile_price_index`.`max_price` AS `max_miles`, `e`.`name`, `e`.`short_description`, `e`.`price`, `e`.`special_price`, `e`.`special_from_date`, `e`.`special_to_date`, `e`.`small_image`, `e`.`thumbnail`, `e`.`news_from_date`, `e`.`news_to_date`, `e`.`url_key`, `e`.`required_options`, `e`.`image_label`, `e`.`small_image_label`, `e`.`thumbnail_label`, `e`.`msrp_enabled`, `e`.`msrp_display_actual_price_type`, `e`.`msrp`, `e`.`tax_class_id`, `e`.`price_type`, `e`.`weight_type`, `e`.`price_view`, `e`.`shipment_type`, `e`.`links_purchased_separately`, `e`.`links_exist`, `e`.`giftcard_amounts`, `e`.`allow_open_amount`, `e`.`open_amount_min`, `e`.`open_amount_max`, `e`.`brand_id`, `e`.`brand_id_value`, `e`.`ca_miles`, `e`.`ca_home_miles`, `e`.`label_id`, `e`.`label_id_value`, `e`.`scale`, `e`.`scale_value`, `e`.`delivery_rule_ids`, `e`.`target_group_ids`, `e`.`warranty`, `e`.`is_home_page`, `e`.`show_in_campaign`, `e`.`campaign_slider_image`, `e`.`campaign_main_image`, `e`.`campaign_text`, `e`.`ca_home_earn`, `e`.`ca_home_earn_promo`, `e`.`ca_home_earn_promo_code`, `e`.`ca_onboard_earn`, `e`.`ca_onboard_earn_promo`, `e`.`ca_onboard_earn_promo_code`, `e`.`color`, `e`.`your_surprise_id`, `e`.`prologistica_number`, `e`.`waterproof`, `e`.`prologistica_send`, `e`.`local_price`, `price_index`.`price`, `price_index`.`tax_class_id`, `price_index`.`final_price`, IF(price_index.tier_price IS NOT NULL, LEAST(price_index.min_price, price_index.tier_price), price_index.min_price) AS `minimal_price`, `price_index`.`min_price`, `price_index`.`max_price`, `price_index`.`tier_price` FROM `catalog_product_flat_8` AS `e` INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id AND cat_index.store_id='8' AND cat_index.visibility IN(2,4) AND cat_index.category_id='158' LEFT JOIN `egg_cat_index` AS `mile_price_index` ON e.entity_id = mile_price_index.entity_id INNER JOIN `catalog_product_index_price` AS `price_index` ON price_index.entity_id = e.entity_id AND price_index.website_id = '2' AND price_index.customer_group_id = 0 WHERE (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '154'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '154'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '154'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '154'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '154'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '154'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '154'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('158', '154')))))
Oct 2 22:00:08 Query (0x7f2096a64020): INSERT INTO `sales_bestsellers_aggregated_daily` (`period`, `store_id`, `product_id`, `product_name`, `product_price`, `qty_ordered`) SELECT STRAIGHT_JOIN DATE(CASE WHEN (`source_table`.`created_at` between '2015-03-29T01:00:00+0000' and '2015-10-02T22:00:07+02:00') THEN DATE_ADD(`source_table`.`created_at`, INTERVAL 7200 SECOND) ELSE DATE_ADD(`source_table`.`created_at`, INTERVAL 3600 SECOND)END ) AS `period`, `source_table`.`store_id`, `order_item`.`product_id`, MIN(IFNULL(product_name.value, product_default_name.value)) AS `product_name`, MIN(IFNULL((IFNULL(product_price.value, product_default_price.value)), 0)) * MIN(IFNULL(source_table.base_to_global_rate, 0)) AS `product_price`, SUM(order_item.qty_ordered) AS `qty_ordered` FROM `sales_flat_order` AS `source_table` INNER JOIN `sales_flat_order_item` AS `order_item` ON order_item.order_id = source_table.entity_id INNER JOIN `catalog_product_entity` AS `product` ON product.entity_id = order_item.product_id AND product.entity_type_id = 4 AND product.type_id NOT IN('grouped', 'configurable', 'bundle') LEFT JOIN `catalog_product_entity_varchar` AS `product_name` ON product_name.entity_id = product.entity_id AND product_name.store_id = source_table.store_id AND product_name.entity_type_id = 4 AND product_name.attribute_id = '65' LEFT JOIN `catalog_product_entity_varchar` AS `product_default_name` ON product_default_name.entity_id = product.entity_id AND product_default_name.store_id = 0 AND product_default_name.entity_type_id = 4 AND product_default_name.attribute_id = '65' LEFT JOIN `catalog_product_entity_decimal` AS `product_price` ON product_price.entity_id = product.entity_id AND product_price.store_id = source_table.store_id AND product_price.entity_type_id = 4 AND product_price.attribute_id = '69' LEFT JOIN `catalog_product_entity_decimal` AS `product_default_price` ON product_default_price.entity_id = product.entity_id AND product_default_price.store_id = 0 AND product_default_price.entity_type_id = 4 AND product_default_price.attribute_id = '69' WHERE (source_table.state != 'canceled') GROUP BY DATE(CASE WHEN (`source_table`.`created_at` between '2015-03-29T01:00:00+0000' and '2015-10-02T22:00:07+02:00') THEN DATE_ADD(`source_table`.`created_at`, INTERVAL 7200 SECOND) ELSE DATE_ADD(`source_table`.`created_at`, INTERVAL 3600 SECOND)END ), `source_table`.`store_id`, `order_item`.`product_id` HAVING (period LIKE '2015-10-01' OR period LIKE '2015-10-02' OR period LIKE '2015-09-30' OR period LIKE '2015-09-27' OR period LIKE '2015-09-24' OR period LIKE '2015-09-23' OR period LIKE '2015-09-29' OR period LIKE '2015-03-12' OR period LIKE '2015-09-22' OR period LIKE '2015-09-19' OR period LIKE '2015-09-12' OR period LIKE '2015-08-25' OR period LIKE '2015-01-10' OR period LIKE '2015-09-05' OR period LIKE '2015-09-15' OR period LIKE '2015-09-25' OR period LIKE '2015-09-21' OR period LIKE '2015-09-08' OR period LIKE '2015-08-12' OR period LIKE '2015-09-04' OR period LIKE '2015-09-16') ON DUPLICATE KEY UPDATE `period` = VALUES(`period`), `store_id` = VALUES(`store_id`), `product_id` = VALUES(`product_id`), `product_name` = VALUES(`product_name`), `product_price` = VALUES(`product_price`), `qty_ordered` = VALUES(`qty_ordered`)
Oct 3 09:17:05 Query (0x7f54603e1020): SELECT DISTINCT 1 AS `status`, `e`.`entity_id`, `e`.`type_id`, `e`.`attribute_set_id`, `cat_index`.`position` AS `cat_index_position`, `mile_price_index`.`ca_miles`, `mile_price_index`.`ca_home_miles`, `mile_price_index`.`min_price` AS `min_miles`, `mile_price_index`.`max_price` AS `max_miles`, `e`.`name`, `e`.`short_description`, `e`.`price`, `e`.`special_price`, `e`.`special_from_date`, `e`.`special_to_date`, `e`.`small_image`, `e`.`thumbnail`, `e`.`news_from_date`, `e`.`news_to_date`, `e`.`url_key`, `e`.`required_options`, `e`.`image_label`, `e`.`small_image_label`, `e`.`thumbnail_label`, `e`.`msrp_enabled`, `e`.`msrp_display_actual_price_type`, `e`.`msrp`, `e`.`tax_class_id`, `e`.`price_type`, `e`.`weight_type`, `e`.`price_view`, `e`.`shipment_type`, `e`.`links_purchased_separately`, `e`.`links_exist`, `e`.`giftcard_amounts`, `e`.`allow_open_amount`, `e`.`open_amount_min`, `e`.`open_amount_max`, `e`.`brand_id`, `e`.`brand_id_value`, `e`.`ca_miles`, `e`.`ca_home_miles`, `e`.`label_id`, `e`.`label_id_value`, `e`.`scale`, `e`.`scale_value`, `e`.`delivery_rule_ids`, `e`.`target_group_ids`, `e`.`warranty`, `e`.`is_home_page`, `e`.`show_in_campaign`, `e`.`campaign_slider_image`, `e`.`campaign_main_image`, `e`.`campaign_text`, `e`.`ca_home_earn`, `e`.`ca_home_earn_promo`, `e`.`ca_home_earn_promo_code`, `e`.`ca_onboard_earn`, `e`.`ca_onboard_earn_promo`, `e`.`ca_onboard_earn_promo_code`, `e`.`color`, `e`.`your_surprise_id`, `e`.`prologistica_number`, `e`.`waterproof`, `e`.`prologistica_send`, `e`.`local_price`, `price_index`.`price`, `price_index`.`tax_class_id`, `price_index`.`final_price`, IF(price_index.tier_price IS NOT NULL, LEAST(price_index.min_price, price_index.tier_price), price_index.min_price) AS `minimal_price`, `price_index`.`min_price`, `price_index`.`max_price`, `price_index`.`tier_price` FROM `catalog_product_flat_8` AS `e` INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id AND cat_index.store_id='8' AND cat_index.visibility IN(2,4) AND cat_index.category_id='128' LEFT JOIN `egg_cat_index` AS `mile_price_index` ON e.entity_id = mile_price_index.entity_id INNER JOIN `catalog_product_index_price` AS `price_index` ON price_index.entity_id = e.entity_id AND price_index.website_id = '2' AND price_index.customer_group_id = 0 WHERE (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('131', '128'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('131', '128'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('131', '128'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('131', '128'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('131', '128'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('131', '128'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('131', '128'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '8') AND (cat_prod.category_id in ('131', '128')))))
Oct 5 08:21:38 Query (0x7fb1c642c020): SELECT DISTINCT 1 AS `status`, `e`.`entity_id`, `e`.`type_id`, `e`.`attribute_set_id`, `cat_index`.`position` AS `cat_index_position`, `mile_price_index`.`ca_miles`, `mile_price_index`.`ca_home_miles`, `mile_price_index`.`min_price` AS `min_miles`, `mile_price_index`.`max_price` AS `max_miles`, `e`.`name`, `e`.`short_description`, `e`.`price`, `e`.`special_price`, `e`.`special_from_date`, `e`.`special_to_date`, `e`.`small_image`, `e`.`thumbnail`, `e`.`news_from_date`, `e`.`news_to_date`, `e`.`url_key`, `e`.`required_options`, `e`.`image_label`, `e`.`small_image_label`, `e`.`thumbnail_label`, `e`.`msrp_enabled`, `e`.`msrp_display_actual_price_type`, `e`.`msrp`, `e`.`tax_class_id`, `e`.`price_type`, `e`.`weight_type`, `e`.`price_view`, `e`.`shipment_type`, `e`.`links_purchased_separately`, `e`.`links_exist`, `e`.`giftcard_amounts`, `e`.`allow_open_amount`, `e`.`open_amount_min`, `e`.`open_amount_max`, `e`.`brand_id`, `e`.`brand_id_value`, `e`.`ca_miles`, `e`.`ca_home_miles`, `e`.`label_id`, `e`.`label_id_value`, `e`.`scale`, `e`.`scale_value`, `e`.`delivery_rule_ids`, `e`.`target_group_ids`, `e`.`warranty`, `e`.`is_home_page`, `e`.`show_in_campaign`, `e`.`campaign_slider_image`, `e`.`campaign_main_image`, `e`.`campaign_text`, `e`.`ca_home_earn`, `e`.`ca_home_earn_promo`, `e`.`ca_home_earn_promo_code`, `e`.`ca_onboard_earn`, `e`.`ca_onboard_earn_promo`, `e`.`ca_onboard_earn_promo_code`, `e`.`color`, `e`.`your_surprise_id`, `e`.`prologistica_number`, `e`.`waterproof`, `e`.`prologistica_send`, `e`.`local_price`, `price_index`.`price`, `price_index`.`tax_class_id`, `price_index`.`final_price`, IF(price_index.tier_price IS NOT NULL, LEAST(price_index.min_price, price_index.tier_price), price_index.min_price) AS `minimal_price`, `price_index`.`min_price`, `price_index`.`max_price`, `price_index`.`tier_price` FROM `catalog_product_flat_7` AS `e` INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id AND cat_index.store_id='7' AND cat_index.visibility IN(2,4) AND cat_index.category_id='168' LEFT JOIN `egg_cat_index` AS `mile_price_index` ON e.entity_id = mile_price_index.entity_id INNER JOIN `catalog_product_index_price` AS `price_index` ON price_index.entity_id = e.entity_id AND price_index.website_id = '2' AND price_index.customer_group_id = 0 WHERE (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('163', '168'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('163', '168'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('163', '168'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('163', '168'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('163', '168'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('163', '168'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('163', '168'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('163', '168')))))
Oct 5 08:53:21 Query (0x7fd70e8eb020): SELECT cid, data, created, expire, serialized FROM cache WHERE cid IN ('schema:runtime:1')
Oct 5 09:05:52 Query (0x7f47ca851020): SELECT DISTINCT 1 AS `status`, `e`.`entity_id`, `e`.`type_id`, `e`.`attribute_set_id`, `cat_index`.`position` AS `cat_index_position`, `mile_price_index`.`ca_miles`, `mile_price_index`.`ca_home_miles`, `mile_price_index`.`min_price` AS `min_miles`, `mile_price_index`.`max_price` AS `max_miles`, `e`.`name`, `e`.`short_description`, `e`.`price`, `e`.`special_price`, `e`.`special_from_date`, `e`.`special_to_date`, `e`.`small_image`, `e`.`thumbnail`, `e`.`news_from_date`, `e`.`news_to_date`, `e`.`url_key`, `e`.`required_options`, `e`.`image_label`, `e`.`small_image_label`, `e`.`thumbnail_label`, `e`.`msrp_enabled`, `e`.`msrp_display_actual_price_type`, `e`.`msrp`, `e`.`tax_class_id`, `e`.`price_type`, `e`.`weight_type`, `e`.`price_view`, `e`.`shipment_type`, `e`.`links_purchased_separately`, `e`.`links_exist`, `e`.`giftcard_amounts`, `e`.`allow_open_amount`, `e`.`open_amount_min`, `e`.`open_amount_max`, `e`.`brand_id`, `e`.`brand_id_value`, `e`.`ca_miles`, `e`.`ca_home_miles`, `e`.`label_id`, `e`.`label_id_value`, `e`.`scale`, `e`.`scale_value`, `e`.`delivery_rule_ids`, `e`.`target_group_ids`, `e`.`warranty`, `e`.`is_home_page`, `e`.`show_in_campaign`, `e`.`campaign_slider_image`, `e`.`campaign_main_image`, `e`.`campaign_text`, `e`.`ca_home_earn`, `e`.`ca_home_earn_promo`, `e`.`ca_home_earn_promo_code`, `e`.`ca_onboard_earn`, `e`.`ca_onboard_earn_promo`, `e`.`ca_onboard_earn_promo_code`, `e`.`color`, `e`.`your_surprise_id`, `e`.`prologistica_number`, `e`.`waterproof`, `e`.`prologistica_send`, `e`.`local_price`, `price_index`.`price`, `price_index`.`tax_class_id`, `price_index`.`final_price`, IF(price_index.tier_price IS NOT NULL, LEAST(price_index.min_price, price_index.tier_price), price_index.min_price) AS `minimal_price`, `price_index`.`min_price`, `price_index`.`max_price`, `price_index`.`tier_price` FROM `catalog_product_flat_7` AS `e` INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id AND cat_index.store_id='7' AND cat_index.visibility IN(2,4) AND cat_index.category_id='172' LEFT JOIN `egg_cat_index` AS `mile_price_index` ON e.entity_id = mile_price_index.entity_id INNER JOIN `catalog_product_index_price` AS `price_index` ON price_index.entity_id = e.entity_id AND price_index.website_id = '2' AND price_index.customer_group_id = 0 WHERE (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('167', '172'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('167', '172'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('167', '172'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('167', '172'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('167', '172'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('167', '172'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('167', '172'))))) AND (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('167', '172')))))
Oct 5 09:35:54 Query (0x7f4cd78ff020): SELECT 1 AS `status`, `e`.`entity_id`, `e`.`type_id`, `e`.`attribute_set_id`, `cat_index`.`position` AS `cat_index_position`, `mile_price_index`.`ca_miles`, `mile_price_index`.`ca_home_miles`, `mile_price_index`.`min_price` AS `min_miles`, `mile_price_index`.`max_price` AS `max_miles`, `e`.`name`, `e`.`short_description`, `e`.`price`, `e`.`special_price`, `e`.`special_from_date`, `e`.`special_to_date`, `e`.`small_image`, `e`.`thumbnail`, `e`.`news_from_date`, `e`.`news_to_date`, `e`.`url_key`, `e`.`required_options`, `e`.`image_label`, `e`.`small_image_label`, `e`.`thumbnail_label`, `e`.`msrp_enabled`, `e`.`msrp_display_actual_price_type`, `e`.`msrp`, `e`.`tax_class_id`, `e`.`price_type`, `e`.`weight_type`, `e`.`price_view`, `e`.`shipment_type`, `e`.`links_purchased_separately`, `e`.`links_exist`, `e`.`giftcard_amounts`, `e`.`allow_open_amount`, `e`.`open_amount_min`, `e`.`open_amount_max`, `e`.`brand_id`, `e`.`brand_id_value`, `e`.`ca_miles`, `e`.`ca_home_miles`, `e`.`label_id`, `e`.`label_id_value`, `e`.`scale`, `e`.`scale_value`, `e`.`delivery_rule_ids`, `e`.`target_group_ids`, `e`.`warranty`, `e`.`is_home_page`, `e`.`show_in_campaign`, `e`.`campaign_slider_image`, `e`.`campaign_main_image`, `e`.`campaign_text`, `e`.`ca_home_earn`, `e`.`ca_home_earn_promo`, `e`.`ca_home_earn_promo_code`, `e`.`ca_onboard_earn`, `e`.`ca_onboard_earn_promo`, `e`.`ca_onboard_earn_promo_code`, `e`.`color`, `e`.`your_surprise_id`, `e`.`prologistica_number`, `e`.`waterproof`, `e`.`prologistica_send`, `e`.`local_price`, `price_index`.`price`, `price_index`.`tax_class_id`, `price_index`.`final_price`, IF(price_index.tier_price IS NOT NULL, LEAST(price_index.min_price, price_index.tier_price), price_index.min_price) AS `minimal_price`, `price_index`.`min_price`, `price_index`.`max_price`, `price_index`.`tier_price` FROM `catalog_product_flat_1` AS `e` INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id AND cat_index.store_id=1 AND cat_index.visibility IN(2, 4) AND cat_index.category_id='3' LEFT JOIN `egg_cat_index` AS `mile_price_index` ON e.entity_id = mile_price_index.entity_id INNER JOIN `catalog_product_index_price` AS `price_index` ON price_index.entity_id = e.entity_id AND price_index.website_id = '1' AND price_index.customer_group_id = 0 ORDER BY `e`.`name` asc LIMIT 12 OFFSET 36
Oct 5 09:36:47 Query (0x7f9e4402d020): SELECT DISTINCT `e`.`attribute_set_id` FROM `catalog_product_flat_7` AS `e` INNER JOIN `catalog_category_product_index` AS `cat_index` ON cat_index.product_id=e.entity_id AND cat_index.store_id='7' AND cat_index.visibility IN(2, 4) AND cat_index.category_id='172' LEFT JOIN `egg_cat_index` AS `mile_price_index` ON e.entity_id = mile_price_index.entity_id INNER JOIN `catalog_product_index_price` AS `price_index` ON price_index.entity_id = e.entity_id AND price_index.website_id = '2' AND price_index.customer_group_id = 0 WHERE (e.entity_id in ((SELECT `cat_prod`.`product_id` FROM `catalog_category_product_index` AS `cat_prod` WHERE (cat_prod.store_id = '7') AND (cat_prod.category_id in ('172', '162')))))
Oct 5 09:41:25 Query (0x7fecac8da020): REPAIR TABLE `catalog_category_entity_varchar`
There are some patterns in the query that seems to trigger the crash however they do not seem to be the root cause: I tried to reproduce the issue running all those queries more than thousand of times on a test sytem but did not succeed to reproduce that SEGFAULT 11 crash.
Is there anything else I could provide to help getting closer to the root cause?
Any tips on how to proceed?
Thanks in advance for your help.
Cheers,
--thomas
Hi,
There is some patters, true, is there anything special on used tables, e.g. they are partitioned or use compression ? To really help you I would need a quite repeatable case i.e. database and set of queries that at least mostly repeat the crash. Output from *row_search_for_mysql::prebuilt->trx could also help. Clearly something happens to sel_graph either it is freed or overwritten.
Thomas Moreau,
Thanks for the report. Please also provide your cnf file(s).
Jan Lindström,
Could you please take a look at this to see if, by any chance, you can figure out and fix the problem by stack traces alone. If not, maybe you can give some hints on the test flow which could help reproduce it.