Details
Description
Found while analyzing test failures for MDEV-5089.
To reproduce these problems compile MariaDB as described in MDEV-5089 with -DUSE_MUTEX_INSTEAD_OF_RW_LOCKS. Run tokudb and tokudb_bugs test suites.
First, there are attempts to lock read-locked num_DBs_lock. It is locked in ha_tokudb::start_bulk_insert() and then in ha_tokudb::update_row() or ha_tokudb::acquire_table_lock(). It may be not a problem, but as this situation is handled in ha_tokudb::write_row() it should be checked at least.
Second, ha_tokudb::write_row() unlocks not locked num_DBs_lock (in case of error in the very beginning of the function). This should be a problem according to the pthread_rwlock_unlock manual: results are undefined if the read-write lock rwlock is not held by the calling thread.
A rough patching solving the above issues attached.
Gliffy Diagrams
Attachments
Issue Links
- is part of
-
MDEV-5089 possible deadlocks between rwlocks and mutexes
-
- Closed
-
- links to
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Hi Rich,
there seem to be no corresponding cmake variable. Either hack sources (include/my_pthread.h) or compile with CFLAGS=-DUSE_MUTEX_INSTEAD_OF_RW_LOCKS.
I want to reproduce case (1). If you know of a particular tokudb test that exposes these locking problems, please let me know. Otherwise, I will run all of the test and try to discover one.
The error cleanup in described in case (2) is unfortunate. Will fix that one.
IIRC any of the following tests should expose the problem:
tokudb_bugs.1795
tokudb_bugs.1833
tokudb_bugs.2548
tokudb_bugs.5003
tokudb_bugs.tokudb_mrr
tokudb_bugs.tokudb_mrr2
tokudb.auto_increment
tokudb.auto_increment_boundary
tokudb.auto_increment_boundary_traditional
tokudb.i_s_tokudb_lock_waits_released
tokudb.i_s_tokudb_locks_released
tokudb.nested_txn_autocommit
tokudb.nested_txn_begin
tokudb.type_timestamp
I hit this problem first. Is there is patch for it?
Version: '5.5.37-MariaDB-debug' socket: '/tmp/rfp.sock' port: 30000 Source distribution
[New Thread 0x7ffff7b16700 (LWP 11193)]
safe_mutex: Found wrong usage of mutex '(&that->m_rwlock)' and 'LOCK_global_system_variables'
#2 0x0000000000cb87d4 in safe_mutex_lock (mp=0x15605e0 <LOCK_system_variables_hash>, my_flags=0, file=0xd61ec0 "/home/prohaska/mdev6162/mariadb-5.5.37/include/mysql/psi/mysql_thread.h", line=761) at /ho\
me/prohaska/mdev6162/mariadb-5.5.37/mysys/thr_mutex.c:266
#3 0x000000000060ebc2 in inline_mysql_rwlock_rdlock (that=0x15605e0 <LOCK_system_variables_hash>, src_file=0xd621f0 "/home/prohaska/mdev6162/mariadb-5.5.37/sql/sql_plugin.cc", src_line=2916) at /home/pr\
ohaska/mdev6162/mariadb-5.5.37/include/mysql/psi/mysql_thread.h:761
#4 0x0000000000615d97 in intern_sys_var_ptr (thd=0x7ffdb6502060, offset=24, global_lock=false) at /home/prohaska/mdev6162/mariadb-5.5.37/sql/sql_plugin.cc:2916
#5 0x0000000000616956 in sys_var_pluginvar::real_value_ptr (this=0x7ffff7120ad8, thd=0x7ffdb6502060, type=OPT_SESSION) at /home/prohaska/mdev6162/mariadb-5.5.37/sql/sql_plugin.cc:3226
#6 0x0000000000616a1e in sys_var_pluginvar::do_value_ptr (this=0x7ffff7120ad8, thd=0x7ffdb6502060, type=OPT_SESSION, base=0x7ffff7b14330) at /home/prohaska/mdev6162/mariadb-5.5.37/sql/sql_plugin.cc:3255
#7 0x0000000000618fa8 in sys_var_pluginvar::session_value_ptr (this=0x7ffff7120ad8, thd=0x7ffdb6502060, base=0x7ffff7b14330) at /home/prohaska/mdev6162/mariadb-5.5.37/sql/sql_plugin.cc:289
#8 0x000000000056d6ae in sys_var::value_ptr (this=0x7ffff7120ad8, thd=0x7ffdb6502060, type=OPT_SESSION, base=0x7ffff7b14330) at /home/prohaska/mdev6162/mariadb-5.5.37/sql/set_var.cc:254
#9 0x00000000006782e4 in show_status_array (thd=0x7ffdb6502060, wild=0x0, variables=0x7ffd8587e1c8, value_type=OPT_SESSION, status_var=0x0, prefix=0xd6d6c3 "", table=0x7ffd8584c078, ucase_names=false, c\
ond=0x0) at /home/prohaska/mdev6162/mariadb-5.5.37/sql/sql_show.cc:2659
#10 0x00000000006878db in fill_variables (thd=0x7ffdb6502060, tables=0x7ffd858453a0, cond=0x0) at /home/prohaska/mdev6162/mariadb-5.5.37/sql/sql_show.cc:6943
Yes, see first comment in MDEV-5089.
A workaround:
=== modified file 'sql/sql_plugin.cc'
--- sql/sql_plugin.cc 2014-04-09 12:28:07 +0000
+++ sql/sql_plugin.cc 2014-04-24 10:14:36 +0000
@@ -2913,7 +2913,10 @@ static uchar *intern_sys_var_ptr(THD* th
{
uint idx;
- mysql_rwlock_rdlock(&LOCK_system_variables_hash);
+#undef pthread_mutex_lock
+#undef pthread_mutex_unlock
+// mysql_rwlock_rdlock(&LOCK_system_variables_hash);
+// pthread_mutex_lock(&LOCK_system_variables_hash.m_rwlock.mutex);
thd->variables.dynamic_variables_ptr= (char*)
my_realloc(thd->variables.dynamic_variables_ptr,
@@ -2973,7 +2976,8 @@ static uchar *intern_sys_var_ptr(THD* th
thd->variables.dynamic_variables_size=
global_system_variables.dynamic_variables_size;
- mysql_rwlock_unlock(&LOCK_system_variables_hash);
+// pthread_mutex_unlock(&LOCK_system_variables_hash.m_rwlock.mutex);
+// mysql_rwlock_unlock(&LOCK_system_variables_hash);
}
DBUG_RETURN((uchar*)thd->variables.dynamic_variables_ptr + offset);
}
=== modified file 'sql/sql_show.cc'
--- sql/sql_show.cc 2014-04-15 15:41:08 +0000
+++ sql/sql_show.cc 2014-04-24 10:14:36 +0000
@@ -6889,11 +6889,15 @@ int fill_variables(THD *thd, TABLE_LIST
COND *partial_cond= make_cond_for_info_schema(cond, tables);
- mysql_rwlock_rdlock(&LOCK_system_variables_hash);
+#undef pthread_mutex_lock
+#undef pthread_mutex_unlock
+// mysql_rwlock_rdlock(&LOCK_system_variables_hash);
+ pthread_mutex_lock(&LOCK_system_variables_hash.m_rwlock.mutex);
res= show_status_array(thd, wild, enumerate_sys_vars(thd, sorted_vars, option_type),
option_type, NULL, "", tables->table,
upper_case_names, partial_cond);
- mysql_rwlock_unlock(&LOCK_system_variables_hash);
+ pthread_mutex_unlock(&LOCK_system_variables_hash.m_rwlock.mutex);
+// mysql_rwlock_unlock(&LOCK_system_variables_hash);
DBUG_RETURN(res);
}
tokudb 7.1.6 will include these bug fixes. Thanks.
Can I define a cmake variable or do I need to hack the source to define USE_MUTEX_INSTEAD_OF_RW_LOCKS?