[MDEV-6455] MariaDB/TokuDB crash during "join update/set" operation - JIRA

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 10.0.12
Fix Version/s: 10.0.14
Component/s: Storage Engine - TokuDB
Labels:
- tokudb
Environment:
CentOS 6.5 running in Citrix Xen VM, 16 GB RAM, 4 CPU cores, 200GB HDD

Description

During a somewhat big "join update/set" operation, MariaDB/TokuDB crashes for unknown reason.
While watching the processlist, the initial join-query succeeds with some million rows; but the update operation causes the crash after some 100.000s updates.
Here is information from the mysqld.log file:

/home/buildbot/buildbot/build/mariadb-10.0.12/storage/tokudb/ft-index/ft/ft-flusher.cc:1949 toku_ft_flush_node_on_background_thread: Assertion `toku_bnc_n_entries(BNC(parent, childnum))>0' failed (errno=0)
: Success
Backtrace: (Note: toku_do_assert=0x0x7fa91e8791a0)
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z19db_env_do_backtraceP8_IO_FILE+0x1b)[0x7fa91e876b2b]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x71c53)[0x7fa91e876c53]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x71ccd)[0x7fa91e876ccd]
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z39toku_ft_flush_node_on_background_threadP2ftP6ftnode+0x270)[0x7fa91e902490]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0xfd610)[0x7fa91e902610]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x109430)[0x7fa91e90e430]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0xdfcda)[0x7fa91e8e4cda]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0xe0268)[0x7fa91e8e5268]
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z20toku_ft_root_put_msgP2ftP6ft_msgP11txn_gc_info+0x531)[0x7fa91e8e6081]
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z19toku_ft_send_insertP9ft_handleP10__toku_dbtS2_P6xids_t11ft_msg_typeP11txn_gc_info+0x2f)[0x7fa91e8e718f]
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z20toku_ft_maybe_insertP9ft_handleP10__toku_dbtS2_P7tokutxnb10__toku_lsnb11ft_msg_type+0x1ce)[0x7fa91e9381ae]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x45e51)[0x7fa91e84ae51]
/usr/lib64/mysql/plugin/ha_tokudb.so(_Z19env_update_multipleP13__toku_db_envP9__toku_dbP13__toku_db_txnP10__toku_dbtS6_S6_S6_jPS2_PjjP9DBT_ARRAYjSA_+0xeb8)
[0x7fa91e9315a8]
/usr/lib64/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudb10update_rowEPKhPh+0x510)[0x7fa91e869c10]
/usr/sbin/mysqld(_ZN7handler13ha_update_rowEPKhPh+0x162)[0x72eb32]
/usr/sbin/mysqld(_ZN12multi_update10do_updatesEv+0x63f)[0x66562f]
/usr/sbin/mysqld(_ZN12multi_update8send_eofEv+0x288)[0x6659d8]
/usr/sbin/mysqld[0x60c74e]
/usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa3d)[0x61f5cd]
/usr/sbin/mysqld(_ZN4JOIN4execEv+0x11)[0x621441]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x1dd)
[0x61e00d]
/usr/sbin/mysqld(_Z18mysql_multi_updateP3THDP10TABLE_LISTP4ListI4ItemES6_PS4_y15enum_duplicatesbP18st_select_lex_unitP13st_select_lexPP12multi_update+0x1ba
)[0x664e5a]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x4667)[0x5d3a27]
/usr/sbin/mysqld[0x5d5b12]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1b20)[0x5d7cd0]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x453)[0x6950e3]
/usr/sbin/mysqld(handle_one_connection+0x42)[0x6951b2]
/lib64/libpthread.so.0[0x3b530079d1]
/lib64/libc.so.6(clone+0x6d)[0x3b52ce8b5d]

Gliffy Diagrams

Attachments

Activity

Ascending order - Click to sort in descending order

Hide

Permalink

Thomas Trede added a comment - 2014-07-18 00:25 - edited

more information from logfile:

Server version: 10.0.12-MariaDB
key_buffer_size=33554432
read_buffer_size=131072
max_used_connections=35
max_threads=1002
thread_count=2
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2233828 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x7fa92f90b008
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fa8e6ffed10 thread_stack 0x30000
(my_addr_resolve failure: fork)
/usr/sbin/mysqld(my_print_stacktrace+0x2b) [0xb6c14b]
/usr/sbin/mysqld(handle_fatal_signal+0x398) [0x7255f8]
/lib64/libpthread.so.0() [0x3b5300f710]
/lib64/libc.so.6(gsignal+0x35) [0x3b52c32925]
/lib64/libc.so.6(abort+0x175) [0x3b52c34105]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x71c58) [0x7fa91e876c58]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x71ccd) [0x7fa91e876ccd]
/usr/lib64/mysql/plugin/ha_tokudb.so(toku_ft_flush_node_on_background_thread(ft*, ftnode*)+0x270) [0x7fa91e902490]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0xfd610) [0x7fa91e902610]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x109430) [0x7fa91e90e430]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0xdfcda) [0x7fa91e8e4cda]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0xe0268) [0x7fa91e8e5268]
/usr/lib64/mysql/plugin/ha_tokudb.so(toku_ft_root_put_msg(ft*, ft_msg*, txn_gc_info*)+0x531) [0x7fa91e8e6081]
/usr/lib64/mysql/plugin/ha_tokudb.so(toku_ft_send_insert(ft_handle*, __toku_dbt*, __toku_dbt*, xids_t*, ft_msg_type, txn_gc_info*)+0x2f) [0x7fa91e8e718f]
/usr/lib64/mysql/plugin/ha_tokudb.so(toku_ft_maybe_insert(ft_handle*, __toku_dbt*, __toku_dbt*, tokutxn*, bool, __toku_lsn, bool, ft_msg_type)+0x1ce) [0x7fa91e9381ae]
/usr/lib64/mysql/plugin/ha_tokudb.so(+0x45e51) [0x7fa91e84ae51]
/usr/lib64/mysql/plugin/ha_tokudb.so(env_update_multiple(__toku_db_env*, __toku_db*, __toku_db_txn*, __toku_dbt*, __toku_dbt*, __toku_dbt*, __toku_dbt*, unsigned int, __toku_db**, unsigned int*, unsigned int, DBT_ARRAY*, unsigned int, DBT_ARRAY*)+0xeb8) [0x7fa91e9315a8]
/usr/lib64/mysql/plugin/ha_tokudb.so(ha_tokudb::update_row(unsigned char const*, unsigned char*)+0x510) [0x7fa91e869c10]
/usr/sbin/mysqld(handler::ha_update_row(unsigned char const*, unsigned char*)+0x162) [0x72eb32]
/usr/sbin/mysqld(multi_update::do_updates()+0x63f) [0x66562f]
/usr/sbin/mysqld(multi_update::send_eof()+0x288) [0x6659d8]
/usr/sbin/mysqld() [0x60c74e]
/usr/sbin/mysqld(JOIN::exec_inner()+0xa3d) [0x61f5cd]
/usr/sbin/mysqld(JOIN::exec()+0x11) [0x621441]
/usr/sbin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x1dd) [0x61e00d]
/usr/sbin/mysqld(mysql_multi_update(THD*, TABLE_LIST*, List<Item>*, List<Item>*, Item*, unsigned long long, enum_duplicates, bool, st_select_lex_unit*, st_select_lex*, multi_update**)+0x1ba) [0x664e5a]
/usr/sbin/mysqld(mysql_execute_command(THD*)+0x4667) [0x5d3a27]
/usr/sbin/mysqld() [0x5d5b12]
/usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1b20) [0x5d7cd0]
/usr/sbin/mysqld(do_handle_one_connection(THD*)+0x453) [0x6950e3]
/usr/sbin/mysqld(handle_one_connection+0x42) [0x6951b2]
/lib64/libpthread.so.0() [0x3b530079d1]
/lib64/libc.so.6(clone+0x6d) [0x3b52ce8b5d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0x7fa91fdf5020): is an invalid pointer
Connection ID (thread ID): 39
Status: NOT_KILLED

Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on

Show

Thomas Trede added a comment - 2014-07-18 00:25 - edited more information from logfile: Server version: 10.0.12-MariaDB key_buffer_size=33554432 read_buffer_size=131072 max_used_connections=35 max_threads=1002 thread_count=2 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 2233828 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7fa92f90b008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fa8e6ffed10 thread_stack 0x30000 (my_addr_resolve failure: fork) /usr/sbin/mysqld(my_print_stacktrace+0x2b) [0xb6c14b] /usr/sbin/mysqld(handle_fatal_signal+0x398) [0x7255f8] /lib64/libpthread.so.0() [0x3b5300f710] /lib64/libc.so.6(gsignal+0x35) [0x3b52c32925] /lib64/libc.so.6(abort+0x175) [0x3b52c34105] /usr/lib64/mysql/plugin/ha_tokudb.so(+0x71c58) [0x7fa91e876c58] /usr/lib64/mysql/plugin/ha_tokudb.so(+0x71ccd) [0x7fa91e876ccd] /usr/lib64/mysql/plugin/ha_tokudb.so(toku_ft_flush_node_on_background_thread(ft*, ftnode*)+0x270) [0x7fa91e902490] /usr/lib64/mysql/plugin/ha_tokudb.so(+0xfd610) [0x7fa91e902610] /usr/lib64/mysql/plugin/ha_tokudb.so(+0x109430) [0x7fa91e90e430] /usr/lib64/mysql/plugin/ha_tokudb.so(+0xdfcda) [0x7fa91e8e4cda] /usr/lib64/mysql/plugin/ha_tokudb.so(+0xe0268) [0x7fa91e8e5268] /usr/lib64/mysql/plugin/ha_tokudb.so(toku_ft_root_put_msg(ft*, ft_msg*, txn_gc_info*)+0x531) [0x7fa91e8e6081] /usr/lib64/mysql/plugin/ha_tokudb.so(toku_ft_send_insert(ft_handle*, __toku_dbt*, __toku_dbt*, xids_t*, ft_msg_type, txn_gc_info*)+0x2f) [0x7fa91e8e718f] /usr/lib64/mysql/plugin/ha_tokudb.so(toku_ft_maybe_insert(ft_handle*, __toku_dbt*, __toku_dbt*, tokutxn*, bool, __toku_lsn, bool, ft_msg_type)+0x1ce) [0x7fa91e9381ae] /usr/lib64/mysql/plugin/ha_tokudb.so(+0x45e51) [0x7fa91e84ae51] /usr/lib64/mysql/plugin/ha_tokudb.so(env_update_multiple(__toku_db_env*, __toku_db*, __toku_db_txn*, __toku_dbt*, __toku_dbt*, __toku_dbt*, __toku_dbt*, unsigned int, __toku_db**, unsigned int*, unsigned int, DBT_ARRAY*, unsigned int, DBT_ARRAY*)+0xeb8) [0x7fa91e9315a8] /usr/lib64/mysql/plugin/ha_tokudb.so(ha_tokudb::update_row(unsigned char const*, unsigned char*)+0x510) [0x7fa91e869c10] /usr/sbin/mysqld(handler::ha_update_row(unsigned char const*, unsigned char*)+0x162) [0x72eb32] /usr/sbin/mysqld(multi_update::do_updates()+0x63f) [0x66562f] /usr/sbin/mysqld(multi_update::send_eof()+0x288) [0x6659d8] /usr/sbin/mysqld() [0x60c74e] /usr/sbin/mysqld(JOIN::exec_inner()+0xa3d) [0x61f5cd] /usr/sbin/mysqld(JOIN::exec()+0x11) [0x621441] /usr/sbin/mysqld(mysql_select(THD*, Item***, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x1dd) [0x61e00d] /usr/sbin/mysqld(mysql_multi_update(THD*, TABLE_LIST*, List<Item>*, List<Item>*, Item*, unsigned long long, enum_duplicates, bool, st_select_lex_unit*, st_select_lex*, multi_update**)+0x1ba) [0x664e5a] /usr/sbin/mysqld(mysql_execute_command(THD*)+0x4667) [0x5d3a27] /usr/sbin/mysqld() [0x5d5b12] /usr/sbin/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x1b20) [0x5d7cd0] /usr/sbin/mysqld(do_handle_one_connection(THD*)+0x453) [0x6950e3] /usr/sbin/mysqld(handle_one_connection+0x42) [0x6951b2] /lib64/libpthread.so.0() [0x3b530079d1] /lib64/libc.so.6(clone+0x6d) [0x3b52ce8b5d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fa91fdf5020): is an invalid pointer Connection ID (thread ID): 39 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on

22 older comments

Hide

Permalink

Thomas Trede added a comment - 2014-08-26 10:38

Hello Elena,
for the time being, I prefer to keep the coredumps private.
Thank you & kind regards,
Thomas

Show

Thomas Trede added a comment - 2014-08-26 10:38 Hello Elena, for the time being, I prefer to keep the coredumps private. Thank you & kind regards, Thomas

Hide

Permalink

Rich Prohaska added a comment - 2014-08-26 20:46

Thanks for getting the core files. We examined them and found the bug in the TokuDB fractal tree indexing code. We are tracking the fix in Tokutek's bug tracking system as https://tokutek.atlassian.net/browse/FT-590.

Show

Rich Prohaska added a comment - 2014-08-26 20:46 Thanks for getting the core files. We examined them and found the bug in the TokuDB fractal tree indexing code. We are tracking the fix in Tokutek's bug tracking system as https://tokutek.atlassian.net/browse/FT-590 .

Hide

Permalink

Thomas Trede added a comment - 2014-09-03 11:57

Hello Rich,
fine, that you have found the root-cause/bug for this issue.
Now in the Tokutek bug tracking, they tell, that this will be fixed in tokudb 7.5.0.
In MariaDB 10 we are somewhere at tokudb-version 7.1.5; when will we see this fix in MariaDB?
It looks like, that this is very, very far away...
Thank you & kind regards,
Thomas

Show

Thomas Trede added a comment - 2014-09-03 11:57 Hello Rich, fine, that you have found the root-cause/bug for this issue. Now in the Tokutek bug tracking, they tell, that this will be fixed in tokudb 7.5.0. In MariaDB 10 we are somewhere at tokudb-version 7.1.5; when will we see this fix in MariaDB? It looks like, that this is very, very far away... Thank you & kind regards, Thomas

Hide

Permalink

Rich Prohaska added a comment - 2014-09-04 02:57

TokuDB 7.5.0 will be available in the middle of sept from Tokutek and I expect that it will be pulled into Mariadb 10 sometime after that.

If you want to experiment on your own, the patch to the tokudb source is very simple. See the commit here: https://tokutek.atlassian.net/browse/FT-590

Show

Rich Prohaska added a comment - 2014-09-04 02:57 TokuDB 7.5.0 will be available in the middle of sept from Tokutek and I expect that it will be pulled into Mariadb 10 sometime after that. If you want to experiment on your own, the patch to the tokudb source is very simple. See the commit here: https://tokutek.atlassian.net/browse/FT-590

Hide

Permalink

Rich Prohaska added a comment - 2014-10-03 03:08

Fixed 10.0.14

Show

Rich Prohaska added a comment - 2014-10-03 03:08 Fixed 10.0.14

People

Assignee:

Unassigned

Reporter:

Thomas Trede

Votes:

0 Vote for this issue

Watchers:

4 Start watching this issue

Dates

Created:

2014-07-18 00:22

Updated:

2014-11-14 19:02

Resolved:

2014-11-14 19:02

Agile

View on Board