Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-6083

Assertion `! (&(&LOCK_open)->m_mutex)->count || ! pthread_equal(pth read_self(), (&(&LOCK_open)->m_mutex)->thread)' fails in intern_sys_var_ptr on server shutdown after uninstalling TokuDB plugin at runtime

        Details

        • Type: Bug
        • Status: Closed
        • Priority: Minor
        • Resolution: Fixed
        • Affects Version/s: 5.5.36
        • Fix Version/s: 5.5.38
        • Component/s: None
        • Labels:
          None
        • Sprint:
          serg-test-sprint-1

          Description

          On 10.0, the assertion failure doesn't happen, but there is some uncleanness on the shutdown still (e.g. seen with valgrind), so I suppose it might be affected also, thus filing both for 5.5 and 10.0.

          # It's not necessary to install TokuDB at runtime, 
# instead the server can be run with --plugin-load=ha_tokudb

INSTALL SONAME 'ha_tokudb';
DROP TABLE IF EXISTS t1;
CREATE TABLE t1 (a INT) ENGINE=TokuDB;
INSERT INTO t1 VALUES (1),(2);
UNINSTALL SONAME 'ha_tokudb';

# Let server to shutdown (with MTR, it happens on its own)

          5.5/sql/sql_plugin.cc:2903: uchar* intern_sys_var_ptr(THD*, int, bool): Assertion `! (&(&LOCK_open)->m_mutex)->count || ! pthread_equal(pthread_self(), (&(&LOCK_open)->m_mutex)->thread)' failed.
140412 20:09:03 [ERROR] mysqld got signal 6 ;

          #6  0x00007f6db3ad1621 in *__GI___assert_fail (assertion=0xd8d6c0 "! (&(&LOCK_open)->m_mutex)->count || ! pthread_equal(pthread_self(), (&(&LOCK_open)->m_mutex)->thread)", file=<optimized out>, line=2903, function=0xd8e500 "uchar* intern_sys_var_ptr(THD*, int, bool)") at assert.c:81
#7  0x00000000006476df in intern_sys_var_ptr (thd=0x0, offset=136, global_lock=false) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:2903
#8  0x00000000006482ce in sys_var_pluginvar::real_value_ptr (this=0x7f6db23fe248, thd=0x0, type=OPT_GLOBAL) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:3220
#9  0x000000000064809a in plugin_vars_free_values (vars=0x7f6db33fb090) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:3156
#10 0x0000000000642f11 in plugin_del (plugin=0x7f6db33f53a8) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:1202
#11 0x0000000000643161 in reap_plugins () at /home/elenst/bzr/5.5/sql/sql_plugin.cc:1247
#12 0x000000000064342a in plugin_unlock (thd=0x0, plugin=0x7f6d9e01d460) at /home/elenst/bzr/5.5/sql/sql_plugin.cc:1314
#13 0x00000000006f08a4 in TABLE_SHARE::destroy (this=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/table.cc:426
#14 0x00000000006f0aa2 in free_table_share (share=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/table.cc:480
#15 0x00000000005cedb8 in table_def_free_entry (share=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/sql_base.cc:368
#16 0x0000000000cb57d2 in my_hash_delete (hash=0x15877a0, record=0x7f6daf67ad78 "\002") at /home/elenst/bzr/5.5/mysys/hash.c:604
#17 0x00000000005cfb57 in release_table_share (share=0x7f6daf67ad78) at /home/elenst/bzr/5.5/sql/sql_base.cc:794
#18 0x00000000006f7452 in closefrm (table=0x7f6d9e1e7c60, free_share=true) at /home/elenst/bzr/5.5/sql/table.cc:2761
#19 0x00000000005d013a in intern_close_table (table=0x7f6d9e1e7c60) at /home/elenst/bzr/5.5/sql/sql_base.cc:928
#20 0x00000000005d01a7 in free_cache_entry (table=0x7f6d9e1e7c60) at /home/elenst/bzr/5.5/sql/sql_base.cc:951
#21 0x00000000005d0525 in close_cached_tables (thd=0x0, tables=0x0, wait_for_refresh=false, timeout=31536000) at /home/elenst/bzr/5.5/sql/sql_base.cc:1058
#22 0x00000000005ceef1 in table_def_start_shutdown () at /home/elenst/bzr/5.5/sql/sql_base.cc:410
#23 0x00000000005845e4 in clean_up (print_message=true) at /home/elenst/bzr/5.5/sql/mysqld.cc:1821
#24 0x00000000005843d0 in unireg_end () at /home/elenst/bzr/5.5/sql/mysqld.cc:1734
#25 0x00000000005842f2 in kill_server (sig_ptr=0x0) at /home/elenst/bzr/5.5/sql/mysqld.cc:1662
#26 0x000000000058430f in kill_server_thread (arg=0x7f6d9dbfee48) at /home/elenst/bzr/5.5/sql/mysqld.cc:1685
#27 0x0000000000b61755 in pfs_spawn_thread (arg=0x7f6d9dc271c0) at /home/elenst/bzr/5.5/storage/perfschema/pfs.cc:1015
#28 0x00007f6db5031b50 in start_thread (arg=<optimized out>) at pthread_create.c:304
#29 0x00007f6db3b80a7d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112

          Stack trace from:

          revision-id: knielsen@knielsen-hq.org-20140411084611-oygkym1990hgbuhe
date: 2014-04-11 10:46:11 +0200
build-date: 2014-04-12 21:42:44 +0400
revno: 4133
branch-nick: 5.5

          Valgrind warnings with 10.0:

          ==2806== 212 bytes in 1 blocks are still reachable in loss record 9 of 10
==2806==    at 0x4C28CCE: malloc (vg_replace_malloc.c:292)
==2806==    by 0xE3853F: sf_malloc (safemalloc.c:115)
==2806==    by 0xE276A6: my_malloc (my_malloc.c:100)
==2806==    by 0xD63B40C: ???
==2806==    by 0xD63EEA2: ???
==2806==    by 0xD64F7DE: ???
==2806==    by 0x869CC7: handler::ha_external_lock(THD*, int) (handler.cc:5843)
==2806==    by 0x93F723: lock_external(THD*, TABLE**, unsigned int) (lock.cc:357)
==2806==    by 0x93F4A3: mysql_lock_tables(THD*, st_mysql_lock*, unsigned int) (lock.cc:309)
==2806==    by 0x93F3B4: mysql_lock_tables(THD*, TABLE**, unsigned int, unsigned int) (lock.cc:276)
==2806==    by 0x61937F: lock_tables(THD*, TABLE_LIST*, unsigned int, unsigned int) (sql_base.cc:5310)
==2806==    by 0x618B3A: open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*) (sql_base.cc:5081)
==2806==    by 0x60D1E2: open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int) (sql_base.h:485)
==2806==    by 0x655BE1: mysql_insert(THD*, TABLE_LIST*, List<Item>&, List<List<Item> >&, List<Item>&, List<Item>&, enum_duplicates, bool) (sql_insert.cc:722)
==2806==    by 0x676262: mysql_execute_command(THD*) (sql_parse.cc:3471)
==2806==    by 0x67E48E: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:6452)

          Stack trace from:

          revision-id: elenst@montyprogram.com-20140410221003-dea2djgc31gbu3lu
date: 2014-04-11 02:10:03 +0400
build-date: 2014-04-12 21:43:05 +0400
revno: 4148
branch-nick: 10.0

            Gliffy Diagrams

              Attachments

                Issue Links

                relates to

                Bug - A problem which impairs or prevents the functions of the product. MDEV-6212 TokuDB: memory leak in external_lock

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Open

                  Activity

                  Ascending order - Click to sort in descending order
                  Hide
                  Permalink
                  serg Sergei Golubchik added a comment -

                  This is not specific to TokuDB, any engine would do.

                  Show
                  serg Sergei Golubchik added a comment - This is not specific to TokuDB, any engine would do.
                  Hide
                  Permalink
                  svoj Sergey Vojtovich added a comment -

                  Assertion failure in 5.5 and memory leak in 10.0 are different problems.

                  Assertion failure must not affect 10.0 because:
                  1. there is no such assertion in 10.0
                  2. there is no LOCK_open in 10.0
                  3. even per-share mutex is not held while calling intern_sys_var_ptr.

                  As for the valgrind warning: it looks like a memory leak in TokuDB.

                  Show
                  svoj Sergey Vojtovich added a comment - Assertion failure in 5.5 and memory leak in 10.0 are different problems. Assertion failure must not affect 10.0 because: 1. there is no such assertion in 10.0 2. there is no LOCK_open in 10.0 3. even per-share mutex is not held while calling intern_sys_var_ptr. As for the valgrind warning: it looks like a memory leak in TokuDB.
                  Hide
                  Permalink
                  svoj Sergey Vojtovich added a comment -

                  Sergei, please review fix for this bug. I didn't add a test case because we need a storage engine with PLUGIN_VAR_MEMALLOC variable. I could extend example engine if you like.

                  Show
                  svoj Sergey Vojtovich added a comment - Sergei, please review fix for this bug. I didn't add a test case because we need a storage engine with PLUGIN_VAR_MEMALLOC variable. I could extend example engine if you like.
                  Hide
                  Permalink
                  svoj Sergey Vojtovich added a comment -

                  Reported TokuDB memory leak as MDEV-6212.

                  Show
                  svoj Sergey Vojtovich added a comment - Reported TokuDB memory leak as MDEV-6212 .
                  Hide
                  Permalink
                  serg Sergei Golubchik added a comment -

                  Thanks! Ok to push!

                  Show
                  serg Sergei Golubchik added a comment - Thanks! Ok to push!
                  Hide
                  Permalink
                  svoj Sergey Vojtovich added a comment -

                  Fixed in 5.5.38:

                  revno: 4194
revision-id: svoj@mariadb.org-20140506081303-81bm6s8jvazblaqo
parent: sanja@askmonty.org-20140527064501-rkk8nxwgchdzy1gx
committer: Sergey Vojtovich <svoj@mariadb.org>
branch nick: 5.5-mdev6083
timestamp: Tue 2014-05-06 12:13:03 +0400
message:
  MDEV-6083 - Assertion `! (&(&LOCK_open)->m_mutex)->count ||
              ! pthread_equal(pth read_self(), (&(&LOCK_open)->m_mutex)->thread)'
              fails in intern_sys_var_ptr on server shutdown after uninstalling
              TokuDB plugin at runtime

  This assertion was introduced by patch for MDEV-5089 to ensure proper lock order
  among LOCK_open and LOCK_global_system_variables: LOCK_open must not be held
  while acquiring LOCK_global_system_variables.

  intern_sys_var_ptr() may be called while freeing storage engine variables with
  PLUGIN_VAR_MEMALLOC flag (when destroying table share after storage engine was
  uninstalled). In this case LOCK_open is held, which is harmless because we need
  global value pointer and thus won't acquire LOCK_global_system_variables.

  Relaxed assertion so it is valid only for session variables.
                  Show
                  svoj Sergey Vojtovich added a comment - Fixed in 5.5.38: revno: 4194 revision-id: svoj@mariadb.org-20140506081303-81bm6s8jvazblaqo parent: sanja@askmonty.org-20140527064501-rkk8nxwgchdzy1gx committer: Sergey Vojtovich <svoj@mariadb.org> branch nick: 5.5-mdev6083 timestamp: Tue 2014-05-06 12:13:03 +0400 message: MDEV-6083 - Assertion `! (&(&LOCK_open)->m_mutex)->count || ! pthread_equal(pth read_self(), (&(&LOCK_open)->m_mutex)->thread)' fails in intern_sys_var_ptr on server shutdown after uninstalling TokuDB plugin at runtime This assertion was introduced by patch for MDEV-5089 to ensure proper lock order among LOCK_open and LOCK_global_system_variables: LOCK_open must not be held while acquiring LOCK_global_system_variables. intern_sys_var_ptr() may be called while freeing storage engine variables with PLUGIN_VAR_MEMALLOC flag (when destroying table share after storage engine was uninstalled). In this case LOCK_open is held, which is harmless because we need global value pointer and thus won't acquire LOCK_global_system_variables. Relaxed assertion so it is valid only for session variables.

                    People

                    • Assignee:
                      svoj Sergey Vojtovich
                      Reporter:
                      elenst Elena Stepanova
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      4 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved:

                        Agile