Details
Critical
Description
Hi,
our server MariaDB 10.0.14 crashed today, 1 months after upgrade from MySQL 5.1.61, which had 2 years uptime.
This server contains only 2 databases with low load and total size about 1GB. One database with Magento e-shop database and one with our database (100x InnoDB tables, 1x Memory, 1x CONNECT and 1x TokuDB).
I think, that it is not related to CONNECT, or TokuDB engine.
MariaDB crashed during standard mysqldump-based backup proces.
Below is also backtrace and MySQL process-list 1-3 seconds before crash.
There was no load, just backup simple InnoDB tables of Magento database.
Thank you for your help.
141024 2:21:13 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see http://kb.askmonty.org/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.0.14-MariaDB key_buffer_size=134217728 read_buffer_size=4194304 max_used_connections=69 max_threads=202 thread_count=4 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4272191 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x7fa9570f3008 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x7fa954579e28 thread_stack 0x80000 /usr/sbin/mysqld(my_print_stacktrace+0x26)[0xb16713] /usr/sbin/mysqld(handle_fatal_signal+0x398)[0x6f3658] /lib64/libpthread.so.0(+0x10150)[0x7fa9fa18b150] /usr/sbin/mysqld[0x88facb] /usr/sbin/mysqld[0x89079f] /usr/sbin/mysqld[0x890a1b] /usr/sbin/mysqld(_ZN7handler22ha_rnd_init_with_errorEb+0x15)[0x6f84a5] /usr/sbin/mysqld(_Z16init_read_recordP11READ_RECORDP3THDP5TABLEP10SQL_SELECTibb+0x4c2)[0x7c9f52] /usr/sbin/mysqld(_Z21join_init_read_recordP13st_join_table+0x70)[0x5f1d10] /usr/sbin/mysqld(_Z10sub_selectP4JOINP13st_join_tableb+0x161)[0x5f1ed1] /usr/sbin/mysqld[0x5fd82d] /usr/sbin/mysqld(_ZN4JOIN10exec_innerEv+0xa08)[0x60e508] /usr/sbin/mysqld(_ZN4JOIN4execEv+0x9)[0x610019] /usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x148)[0x60d068] /usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x294)[0x610314] /usr/sbin/mysqld[0x5bff2b] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x50fe)[0x5cac5e] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x1fc)[0x5cd46c] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x1530)[0x5ce9f0] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x19d)[0x674abd] /usr/sbin/mysqld(handle_one_connection+0x40)[0x674b80] /lib64/libpthread.so.0(+0x8bb5)[0x7fa9fa183bb5] /lib64/libc.so.6(clone+0x6d)[0x7fa9f8e76d7d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7fa9634ba020): is an invalid pointer Connection ID (thread ID): 328170 Status: NOT_KILLED Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_me rge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_so rt_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash.
MySQL process list 1-3 seconds before crash.
+--------+-----------------+-----------+-----------------+---------+------+-----------------------------+--------------------------------------------------------------------+----------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
+--------+-----------------+-----------+-----------------+---------+------+-----------------------------+--------------------------------------------------------------------+----------+
| 1 | event_scheduler | localhost | | Daemon | 14 | Waiting for next activation | | 0.000 |
| 328170 | root | localhost | client_cz_new | Query | 1 | Sending data | SELECT /*!40001 SQL_NO_CACHE */ * FROM `ig_cashondelivery_foreign` | 0.000 |
| 328175 | root | localhost | | Query | 0 | init | show processlist | 0.000 |
| 328176 | root | localhost | | Sleep | 1 | | | 0.000 |
+--------+-----------------+-----------+-----------------+---------+------+-----------------------------+--------------------------------------------------------------------+----------+
Gliffy Diagrams
Attachments
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Hi Elena,
backup procedure is executed every 6 hours. It happen just once.
After re-run everything works fine.
I uploaded "my.cnf" and our backup script "make-daily-mysql-backup" into ftp.askmonty.org/private (by jan.reges@siteone.cz).
Btw, table "ig_cashondelivery_foreign" is empty, so it's not used. We store whole process-list every 2-3 seconds and it was last before crash.
Also, maybe it can depends. We have some UDF:
SELECT * FROM mysql.func; lib_mysqludf_preg_info 0 lib_mysqludf_preg.so function preg_capture 0 lib_mysqludf_preg.so function preg_check 2 lib_mysqludf_preg.so function preg_replace 0 lib_mysqludf_preg.so function preg_rlike 2 lib_mysqludf_preg.so function preg_position 2 lib_mysqludf_preg.so function microtime 1 mysql_udf_microtime.so function microtime 1 mysql_udf_microtime.so
Hi Ján,
Thanks for the info.
There are several things here I have difficulties explaining.
First, if the table ig_cashondelivery_foreign is empty, it should not take 1 second or more to select from it, and it should be quite hard to catch it at 'Sending data' phase.
Since you have lots of processlist logs (normal ones, not crash-related) – can you grep them to see if select from this table appears often?
Secondly, I don't see anything in the backup process that should cause the stack trace like that. You had 4 connections, is it possible that one of the two others woke up and ran something? E.g. event scheduler started working? (The time is odd, which makes it rather unlikely but not impossible). Or could it be the other connection, the one that's sleeping in the processlist?
Finally, about your select from mysql.func – it looks strange too. The first column in the table, 'name', is a primary key, so you should not have two functions called 'microtime'. Do you really, or is it a copy-paste error? If you seem to have two, could you please run check table mysql.func extended?
I presume you could not provide the data dump, otherwise you would have, right?
Then, could you please see in one of the dumps which table is queried next after ig_cashondelivery_foreign, and send both tables' structures (SHOW CREATE TABLE .. output and SHOW INDEX IN .. output)? Or, if it's not a table but some other object, it's even more interesting.
Thanks.
Closing for now as incomplete, please comment to re-open if you have more information.
Hi,
How often do you run the same backup procedure? Did it happen before, or was it the first attempt after the upgrade to 10.0.14?
Did you re-run the backup after the crash? If you did, did it succeed?
How confidential is the data? Would you be able to upload it to our ftp.askmonty.org/private? (only MariaDB developers will have access to it)?
If you can't provide all data, maybe at least client_cz_new.ig_cashondelivery_foreign which appears in the processlist – it is no guarantee that it was the last, but there is a chance.
In any case, please also attach your cnf file(s) and the exact mysqldump command your process uses (or the backup script which performs it).
Thank you.