Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-8206

OQGraph crash caused by SELECT queries

    Details

      Description

      I deployed MariaDB 10.0.19 on our servers and encountered a crash that seems to happen irregularly after SELECT queries on an OQGraph table. It was reproducably happening on two servers, both running CentOS 7 (in a master-slave setup). I have so far been unable to reproduce the crash on any other system or even CentOS 7 inside a VM. I since was forced to revert those servers back to 5.5, but I would appreciate any pointers on how I could reproduce the crash I got.

        Gliffy Diagrams

          Attachments

          1. db_history_minimal.sql
            19 kB
          2. general.log.gz
            92 kB
          3. general2.log
            0.3 kB
          4. mysql-error.log
            9 kB
          5. mysql-error2.log
            8 kB
          6. oqgraph-crash.log
            3 kB
          7. server.cnf
            1 kB
          8. version_history.sql
            0.5 kB

            Activity

            Hide
            elenst Elena Stepanova added a comment -

            So, you downgraded both servers where the problem was observed, and you are not getting the crash anymore?
            If so, it's going to be hard to reproduce it.

            If you still have a server where it happens, please enable the general log for a while (until the next crash). After the server crashes, store both the general log and error log, this way we'll be able to find the guilty query.

            Show
            elenst Elena Stepanova added a comment - So, you downgraded both servers where the problem was observed, and you are not getting the crash anymore? If so, it's going to be hard to reproduce it. If you still have a server where it happens, please enable the general log for a while (until the next crash). After the server crashes, store both the general log and error log, this way we'll be able to find the guilty query.
            Hide
            pprkut Heinz Wiesinger added a comment -

            I tried reproducing it on a similar setup running CentOS 6, but no dice. I can't reproduce it on any other setup so far. I'm afraid the affected servers are blocked for the next 1-2 months and I can't touch them during that period. That's why I was hoping to be able to reproduce it elsewhere :/

            I can say however that the query itself didn't really matter. Any kind of SELECT query on the oqgraph table could cause the crash, from a simple "SELECT * FROM table" to a dijkstra shortest path query, just not every time.

            Show
            pprkut Heinz Wiesinger added a comment - I tried reproducing it on a similar setup running CentOS 6, but no dice. I can't reproduce it on any other setup so far. I'm afraid the affected servers are blocked for the next 1-2 months and I can't touch them during that period. That's why I was hoping to be able to reproduce it elsewhere :/ I can say however that the query itself didn't really matter. Any kind of SELECT query on the oqgraph table could cause the crash, from a simple "SELECT * FROM table" to a dijkstra shortest path query, just not every time.
            Hide
            elenst Elena Stepanova added a comment -

            Closing for now as not reproducible. Please comment to re-open if you encounter the problem again.

            Show
            elenst Elena Stepanova added a comment - Closing for now as not reproducible. Please comment to re-open if you encounter the problem again.
            Hide
            pprkut Heinz Wiesinger added a comment -

            Managed to finally reproduce this again, error and general log attached.
            This now happened on a single server setup running 10.0.21 on CentOS 6.

            Show
            pprkut Heinz Wiesinger added a comment - Managed to finally reproduce this again, error and general log attached. This now happened on a single server setup running 10.0.21 on CentOS 6.
            Hide
            pprkut Heinz Wiesinger added a comment -

            Got a debug version of mariadb installed on the affected server and managed to reproduce the crash. General log and error log attached as well, should contain more useful information now.

            Show
            pprkut Heinz Wiesinger added a comment - Got a debug version of mariadb installed on the affected server and managed to reproduce the crash. General log and error log attached as well, should contain more useful information now.
            Hide
            monty Michael Widenius added a comment -

            pprkut (on Freenode IRC) has still this issue and think he can reproduce it)

            Show
            monty Michael Widenius added a comment - pprkut (on Freenode IRC) has still this issue and think he can reproduce it)
            Hide
            elenst Elena Stepanova added a comment -

            Heinz Wiesinger, thanks.
            Can you paste the structure of 'version_history' (and underlying tables if there are any)? Even better if you can provide the data dump.
            Please also attach your cnf file.
            Thanks.

            Stack trace from the error log
            mysqld: /opt/mariadb-10.0.21/mariadb-10.0.21/sql/handler.cc:2473: THD* handler::ha_thd() const: Assertion `!table || !table->in_use || table->in_use == _current_thd()' failed.
            150925 15:07:07 [ERROR] mysqld got signal 6 ;
            
            sql/handler.cc:2474(handler::ha_thd() const)[0x861bcc]
            handler/ha_innodb.cc:13024(ha_innobase::extra(ha_extra_function))[0xa1b01b]
            /opt/mariadb-10.0.21/mariadb-10.0.21/sql/lib/plugin/ha_oqgraph.so(_ZN10ha_oqgraph5extraE17ha_extra_function+0x3b)[0x7ff5b193675d]
            sql/table_cache.cc:342(tc_acquire_table)[0x7ead7c]
            sql/table_cache.cc:765(tdc_acquire_share(THD*, char const*, char const*, char const*, unsigned int, unsigned int, unsigned int, TABLE**))[0x7ebc6b]
            sql/sql_base.cc:2371(open_table(THD*, TABLE_LIST*, st_mem_root*, Open_table_context*))[0x603792]
            sql/sql_base.cc:4032(open_and_process_table)[0x606441]
            sql/sql_base.cc:4566(open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*))[0x607552]
            sql/sql_base.cc:5209(open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*))[0x608640]
            sql/sql_base.h:490(open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int))[0x5fc65e]
            sql/sql_parse.cc:5219(execute_sqlcom_select)[0x66d748]
            sql/sql_parse.cc:2562(mysql_execute_command(THD*))[0x6659b1]
            sql/sql_parse.cc:6531(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x6705d6]
            sql/sql_parse.cc:1310(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x662b43]
            sql/sql_parse.cc:999(do_command(THD*))[0x661dc9]
            sql/sql_connect.cc:1378(do_handle_one_connection(THD*))[0x784376]
            sql/sql_connect.cc:1294(handle_one_connection)[0x7840d5]
            /lib64/libpthread.so.0(+0x7a51)[0x7ff5b3475a51]
            /lib64/libc.so.6(clone+0x6d)[0x7ff5b1e5893d]
            
            Trying to get some variables.
            Some pointers may be invalid and cause the dump to abort.
            Query (0x7ff584823088): SELECT COUNT(*) FROM version_history
            Connection ID (thread ID): 5
            Status: NOT_KILLED
            
            Show
            elenst Elena Stepanova added a comment - Heinz Wiesinger , thanks. Can you paste the structure of 'version_history' (and underlying tables if there are any)? Even better if you can provide the data dump. Please also attach your cnf file. Thanks. Stack trace from the error log mysqld: /opt/mariadb-10.0.21/mariadb-10.0.21/sql/handler.cc:2473: THD* handler::ha_thd() const: Assertion `!table || !table->in_use || table->in_use == _current_thd()' failed. 150925 15:07:07 [ERROR] mysqld got signal 6 ; sql/handler.cc:2474(handler::ha_thd() const)[0x861bcc] handler/ha_innodb.cc:13024(ha_innobase::extra(ha_extra_function))[0xa1b01b] /opt/mariadb-10.0.21/mariadb-10.0.21/sql/lib/plugin/ha_oqgraph.so(_ZN10ha_oqgraph5extraE17ha_extra_function+0x3b)[0x7ff5b193675d] sql/table_cache.cc:342(tc_acquire_table)[0x7ead7c] sql/table_cache.cc:765(tdc_acquire_share(THD*, char const*, char const*, char const*, unsigned int, unsigned int, unsigned int, TABLE**))[0x7ebc6b] sql/sql_base.cc:2371(open_table(THD*, TABLE_LIST*, st_mem_root*, Open_table_context*))[0x603792] sql/sql_base.cc:4032(open_and_process_table)[0x606441] sql/sql_base.cc:4566(open_tables(THD*, TABLE_LIST**, unsigned int*, unsigned int, Prelocking_strategy*))[0x607552] sql/sql_base.cc:5209(open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int, Prelocking_strategy*))[0x608640] sql/sql_base.h:490(open_and_lock_tables(THD*, TABLE_LIST*, bool, unsigned int))[0x5fc65e] sql/sql_parse.cc:5219(execute_sqlcom_select)[0x66d748] sql/sql_parse.cc:2562(mysql_execute_command(THD*))[0x6659b1] sql/sql_parse.cc:6531(mysql_parse(THD*, char*, unsigned int, Parser_state*))[0x6705d6] sql/sql_parse.cc:1310(dispatch_command(enum_server_command, THD*, char*, unsigned int))[0x662b43] sql/sql_parse.cc:999(do_command(THD*))[0x661dc9] sql/sql_connect.cc:1378(do_handle_one_connection(THD*))[0x784376] sql/sql_connect.cc:1294(handle_one_connection)[0x7840d5] /lib64/libpthread.so.0(+0x7a51)[0x7ff5b3475a51] /lib64/libc.so.6(clone+0x6d)[0x7ff5b1e5893d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (0x7ff584823088): SELECT COUNT(*) FROM version_history Connection ID (thread ID): 5 Status: NOT_KILLED
            Hide
            pprkut Heinz Wiesinger added a comment -

            Uploaded the files as requested. Please let me know if you need anything else.

            Show
            pprkut Heinz Wiesinger added a comment - Uploaded the files as requested. Please let me know if you need anything else.

              People

              • Assignee:
                Unassigned
                Reporter:
                pprkut Heinz Wiesinger
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: