Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-7685

MariaDB - server crashes when inserting more rows than available space on disk

    Details

      Gliffy Diagrams

        Attachments

          Issue Links

            Activity

            Hide
            jplindst Jan Lindström added a comment -

            btr_page_split_and_insert() assumes that the operation of this
            function must always succeed, we cannot reverse it: therefore enough
            free disk space (2 pages) must be guaranteed to be available before
            this function is called.

            Show
            jplindst Jan Lindström added a comment - btr_page_split_and_insert() assumes that the operation of this function must always succeed, we cannot reverse it: therefore enough free disk space (2 pages) must be guaranteed to be available before this function is called.
            Hide
            jplindst Jan Lindström added a comment -

            commit a7fd11b31d52b62ef7b61783bb83a5e62271307b
            Author: Jan Lindström <jan.lindstrom@mariadb.com>
            Date: Mon Mar 9 18:21:48 2015 +0200

            MDEV-7685: MariaDB - server crashes when inserting more rows than
            available space on disk

            Add error handling when disk full situation happens and
            intentionally bring server down with stacktrace because
            on all cases InnoDB can't continue anyway.

            Show
            jplindst Jan Lindström added a comment - commit a7fd11b31d52b62ef7b61783bb83a5e62271307b Author: Jan Lindström <jan.lindstrom@mariadb.com> Date: Mon Mar 9 18:21:48 2015 +0200 MDEV-7685 : MariaDB - server crashes when inserting more rows than available space on disk Add error handling when disk full situation happens and intentionally bring server down with stacktrace because on all cases InnoDB can't continue anyway.
            Show
            nirbhay_c Nirbhay Choubey added a comment - Patch : https://github.com/MariaDB/server/commit/a7fd11b31d52b62ef7b61783bb83a5e62271307b
            Hide
            nirbhay_c Nirbhay Choubey added a comment -

            10.0.20 still seem to be affected.

            See : https://bugzilla.suse.com/show_bug.cgi?id=919053

            MariaDB [test]> create table t1(a int,b int) engine=innodb data directory='/data1';
            Query OK, 0 rows affected (0.41 sec)
            
            MariaDB [test]> \! ls -lah /data1/test
            total 99K
            drwxrwx--- 2 mysql mysql 1.0K Jul  8 17:26 .
            drwxrwxrwx 4 root  root  1.0K Jul  8 17:26 ..
            -rw-rw---- 1 mysql mysql  96K Jul  8 17:26 t1.ibd
            MariaDB [test]> call innodb_insert_proc(14700);
            ERROR 2013 (HY000): Lost connection to MySQL server during query
            

            Coredump:

            [Thread debugging using libthread_db enabled]
            Using host libthread_db library "/lib64/libthread_db.so.1".
            Core was generated by `/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib6'.
            Program terminated with signal SIGSEGV, Segmentation fault.
            #0  0x00007fbab00020d1 in pthread_kill () from /lib64/libpthread.so.0
            Missing separate debuginfos, use: zypper install glibc-debuginfo-2.19-20.3.x86_64 libaio1-debuginfo-0.3.109-17.15.x86_64 libgcc_s1-debuginfo-4.8.3+r212056-11.2.x86_64 libopenssl1_0_0-debuginfo-1.0.1i-25.1.x86_64 libstdc++6-debuginfo-4.8.3+r212056-11.2.x86_64 libz1-debuginfo-1.2.8-5.1.x86_64
            (gdb) bt
            #0  0x00007fbab00020d1 in pthread_kill () from /lib64/libpthread.so.0
            #1  0x0000000000720184 in handle_fatal_signal (sig=11)
                at /usr/src/debug/mariadb-10.0.20/sql/signal_handler.cc:262
            #2  <signal handler called>
            #3  0x0000000000a80686 in btr_page_split_and_insert (flags=flags@entry=0, 
                cursor=cursor@entry=0x7fba9c39a970, offsets=offsets@entry=0x7fba9c39a910, 
                heap=heap@entry=0x7fba9c39a930, tuple=tuple@entry=0x7fba7403c928, n_ext=<optimized out>, 
                mtr=mtr@entry=0x7fba9c39ad10) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/btr/btr0btr.cc:3101
            #4  0x0000000000a8da44 in btr_cur_pessimistic_insert (flags=flags@entry=0, 
                cursor=cursor@entry=0x7fba9c39a970, offsets=offsets@entry=0x7fba9c39a910, 
                heap=heap@entry=0x7fba9c39a930, entry=entry@entry=0x7fba7403c928, rec=rec@entry=0x7fba9c39a9f0, 
                big_rec=big_rec@entry=0x7fba9c39a920, n_ext=n_ext@entry=0, thr=thr@entry=0x7fba74043ec8, 
                mtr=mtr@entry=0x7fba9c39ad10) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/btr/btr0cur.cc:1731
            #5  0x0000000000a0dfdf in row_ins_clust_index_entry_low (flags=flags@entry=0, mode=<optimized out>, 
                mode@entry=33, index=index@entry=0x7fba7403ef98, n_uniq=n_uniq@entry=0, 
                entry=entry@entry=0x7fba7403c928, n_ext=n_ext@entry=0, thr=thr@entry=0x7fba74043ec8)
                at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:2512
            #6  0x0000000000a0ea39 in row_ins_clust_index_entry (index=index@entry=0x7fba7403ef98, 
                entry=entry@entry=0x7fba7403c928, thr=thr@entry=0x7fba74043ec8, n_ext=n_ext@entry=0)
                at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:2920
            #7  0x0000000000a0efe2 in row_ins_index_entry (thr=0x7fba74043ec8, entry=0x7fba7403c928, 
                index=0x7fba7403ef98) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:2997
            #8  row_ins_index_entry_step (thr=0x7fba74043ec8, node=<optimized out>)
                at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:3074
            #9  row_ins (thr=<optimized out>, node=<optimized out>)
                at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:3214
            #10 row_ins_step (thr=thr@entry=0x7fba74043ec8)
                at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:3339
            #11 0x0000000000a1a608 in row_insert_for_mysql (
                mysql_rec=mysql_rec@entry=0x7fba74031bf0 "\371\335\067", prebuilt=0x7fba740437b8)
                at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0mysql.cc:1369
            #12 0x0000000000983a41 in ha_innobase::write_row (this=0x7fba74031480, 
                record=0x7fba74031bf0 "\371\335\067")
                at /usr/src/debug/mariadb-10.0.20/storage/xtradb/handler/ha_innodb.cc:7593
            ---Type <return> to continue, or q <return> to quit---
            #13 0x0000000000729459 in handler::ha_write_row (this=0x7fba74031480, 
                buf=0x7fba74031bf0 "\371\335\067") at /usr/src/debug/mariadb-10.0.20/sql/handler.cc:5984
            #14 0x00000000005c0c5f in write_record (thd=thd@entry=0x4519c08, table=table@entry=0x7fba74030ac8, 
                info=info@entry=0x7fba9c39b640) at /usr/src/debug/mariadb-10.0.20/sql/sql_insert.cc:1848
            #15 0x00000000005c429a in mysql_insert (thd=thd@entry=0x4519c08, table_list=0x7fba7402a270, 
                fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false)
                at /usr/src/debug/mariadb-10.0.20/sql/sql_insert.cc:965
            #16 0x00000000005dc2d9 in mysql_execute_command (thd=0x4519c08)
                at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:3449
            #17 0x0000000000813d54 in sp_instr_stmt::exec_core (this=0x7fba7402c0a0, thd=<optimized out>, 
                nextp=0x7fba9c39ce40) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:3204
            #18 0x0000000000819add in sp_lex_keeper::reset_lex_and_exec_core (this=this@entry=0x7fba7402c0e0, 
                thd=thd@entry=0x4519c08, nextp=nextp@entry=0x7fba9c39ce40, open_tables=open_tables@entry=false, 
                instr=instr@entry=0x7fba7402c0a0) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:2971
            #19 0x0000000000819fb4 in sp_instr_stmt::execute (this=0x7fba7402c0a0, thd=0x4519c08, 
                nextp=0x7fba9c39ce40) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:3120
            #20 0x00000000008169af in sp_head::execute (this=this@entry=0x7fba74028ae0, thd=thd@entry=0x4519c08, 
                merge_da_on_success=merge_da_on_success@entry=true)
                at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:1371
            #21 0x0000000000817e17 in sp_head::execute_procedure (this=this@entry=0x7fba74028ae0, 
                thd=thd@entry=0x4519c08, args=args@entry=0x451e0d0)
                at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:2159
            #22 0x00000000005d91e1 in mysql_execute_command (thd=thd@entry=0x4519c08)
                at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:4701
            #23 0x00000000005df668 in mysql_parse (thd=0x4519c08, rawbuf=<optimized out>, length=30, 
                parser_state=0x7fba9c39e730) at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:6529
            #24 0x00000000005e16c7 in dispatch_command (command=COM_QUERY, thd=0x4519c08, packet=<optimized out>, 
                packet_length=<optimized out>) at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:1308
            #25 0x00000000005e21c0 in do_command (thd=<optimized out>)
                at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:999
            #26 0x0000000000697742 in do_handle_one_connection (thd_arg=thd_arg@entry=0x4519c08)
                at /usr/src/debug/mariadb-10.0.20/sql/sql_connect.cc:1378
            #27 0x00000000006977c7 in handle_one_connection (arg=0x4519c08)
                at /usr/src/debug/mariadb-10.0.20/sql/sql_connect.cc:1293
            ---Type <return> to continue, or q <return> to quit---
            #28 0x00007fbaafffd0a4 in start_thread () from /lib64/libpthread.so.0
            #29 0x00007fbaaeeca08d in clone () from /lib64/libc.so.6
            (gdb) quit
            
            Show
            nirbhay_c Nirbhay Choubey added a comment - 10.0.20 still seem to be affected. See : https://bugzilla.suse.com/show_bug.cgi?id=919053 MariaDB [test]> create table t1(a int ,b int ) engine=innodb data directory='/data1'; Query OK, 0 rows affected (0.41 sec) MariaDB [test]> \! ls -lah /data1/test total 99K drwxrwx--- 2 mysql mysql 1.0K Jul 8 17:26 . drwxrwxrwx 4 root root 1.0K Jul 8 17:26 .. -rw-rw---- 1 mysql mysql 96K Jul 8 17:26 t1.ibd MariaDB [test]> call innodb_insert_proc(14700); ERROR 2013 (HY000): Lost connection to MySQL server during query Coredump: [ Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1" . Core was generated by `/usr/sbin/mysqld --basedir=/usr --datadir=/ var /lib/mysql --plugin-dir=/usr/lib6'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007fbab00020d1 in pthread_kill () from /lib64/libpthread.so.0 Missing separate debuginfos, use: zypper install glibc-debuginfo-2.19-20.3.x86_64 libaio1-debuginfo-0.3.109-17.15.x86_64 libgcc_s1-debuginfo-4.8.3+r212056-11.2.x86_64 libopenssl1_0_0-debuginfo-1.0.1i-25.1.x86_64 libstdc++6-debuginfo-4.8.3+r212056-11.2.x86_64 libz1-debuginfo-1.2.8-5.1.x86_64 (gdb) bt #0 0x00007fbab00020d1 in pthread_kill () from /lib64/libpthread.so.0 #1 0x0000000000720184 in handle_fatal_signal (sig=11) at /usr/src/debug/mariadb-10.0.20/sql/signal_handler.cc:262 #2 <signal handler called> #3 0x0000000000a80686 in btr_page_split_and_insert (flags=flags@entry=0, cursor=cursor@entry=0x7fba9c39a970, offsets=offsets@entry=0x7fba9c39a910, heap=heap@entry=0x7fba9c39a930, tuple=tuple@entry=0x7fba7403c928, n_ext=<optimized out>, mtr=mtr@entry=0x7fba9c39ad10) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/btr/btr0btr.cc:3101 #4 0x0000000000a8da44 in btr_cur_pessimistic_insert (flags=flags@entry=0, cursor=cursor@entry=0x7fba9c39a970, offsets=offsets@entry=0x7fba9c39a910, heap=heap@entry=0x7fba9c39a930, entry=entry@entry=0x7fba7403c928, rec=rec@entry=0x7fba9c39a9f0, big_rec=big_rec@entry=0x7fba9c39a920, n_ext=n_ext@entry=0, thr=thr@entry=0x7fba74043ec8, mtr=mtr@entry=0x7fba9c39ad10) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/btr/btr0cur.cc:1731 #5 0x0000000000a0dfdf in row_ins_clust_index_entry_low (flags=flags@entry=0, mode=<optimized out>, mode@entry=33, index=index@entry=0x7fba7403ef98, n_uniq=n_uniq@entry=0, entry=entry@entry=0x7fba7403c928, n_ext=n_ext@entry=0, thr=thr@entry=0x7fba74043ec8) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:2512 #6 0x0000000000a0ea39 in row_ins_clust_index_entry (index=index@entry=0x7fba7403ef98, entry=entry@entry=0x7fba7403c928, thr=thr@entry=0x7fba74043ec8, n_ext=n_ext@entry=0) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:2920 #7 0x0000000000a0efe2 in row_ins_index_entry (thr=0x7fba74043ec8, entry=0x7fba7403c928, index=0x7fba7403ef98) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:2997 #8 row_ins_index_entry_step (thr=0x7fba74043ec8, node=<optimized out>) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:3074 #9 row_ins (thr=<optimized out>, node=<optimized out>) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:3214 #10 row_ins_step (thr=thr@entry=0x7fba74043ec8) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0ins.cc:3339 #11 0x0000000000a1a608 in row_insert_for_mysql ( mysql_rec=mysql_rec@entry=0x7fba74031bf0 "\371\335\067" , prebuilt=0x7fba740437b8) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/row/row0mysql.cc:1369 #12 0x0000000000983a41 in ha_innobase::write_row ( this =0x7fba74031480, record=0x7fba74031bf0 "\371\335\067" ) at /usr/src/debug/mariadb-10.0.20/storage/xtradb/handler/ha_innodb.cc:7593 ---Type < return > to continue , or q < return > to quit--- #13 0x0000000000729459 in handler::ha_write_row ( this =0x7fba74031480, buf=0x7fba74031bf0 "\371\335\067" ) at /usr/src/debug/mariadb-10.0.20/sql/handler.cc:5984 #14 0x00000000005c0c5f in write_record (thd=thd@entry=0x4519c08, table=table@entry=0x7fba74030ac8, info=info@entry=0x7fba9c39b640) at /usr/src/debug/mariadb-10.0.20/sql/sql_insert.cc:1848 #15 0x00000000005c429a in mysql_insert (thd=thd@entry=0x4519c08, table_list=0x7fba7402a270, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore= false ) at /usr/src/debug/mariadb-10.0.20/sql/sql_insert.cc:965 #16 0x00000000005dc2d9 in mysql_execute_command (thd=0x4519c08) at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:3449 #17 0x0000000000813d54 in sp_instr_stmt::exec_core ( this =0x7fba7402c0a0, thd=<optimized out>, nextp=0x7fba9c39ce40) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:3204 #18 0x0000000000819add in sp_lex_keeper::reset_lex_and_exec_core ( this = this @entry=0x7fba7402c0e0, thd=thd@entry=0x4519c08, nextp=nextp@entry=0x7fba9c39ce40, open_tables=open_tables@entry= false , instr=instr@entry=0x7fba7402c0a0) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:2971 #19 0x0000000000819fb4 in sp_instr_stmt::execute ( this =0x7fba7402c0a0, thd=0x4519c08, nextp=0x7fba9c39ce40) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:3120 #20 0x00000000008169af in sp_head::execute ( this = this @entry=0x7fba74028ae0, thd=thd@entry=0x4519c08, merge_da_on_success=merge_da_on_success@entry= true ) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:1371 #21 0x0000000000817e17 in sp_head::execute_procedure ( this = this @entry=0x7fba74028ae0, thd=thd@entry=0x4519c08, args=args@entry=0x451e0d0) at /usr/src/debug/mariadb-10.0.20/sql/sp_head.cc:2159 #22 0x00000000005d91e1 in mysql_execute_command (thd=thd@entry=0x4519c08) at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:4701 #23 0x00000000005df668 in mysql_parse (thd=0x4519c08, rawbuf=<optimized out>, length=30, parser_state=0x7fba9c39e730) at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:6529 #24 0x00000000005e16c7 in dispatch_command (command=COM_QUERY, thd=0x4519c08, packet=<optimized out>, packet_length=<optimized out>) at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:1308 #25 0x00000000005e21c0 in do_command (thd=<optimized out>) at /usr/src/debug/mariadb-10.0.20/sql/sql_parse.cc:999 #26 0x0000000000697742 in do_handle_one_connection (thd_arg=thd_arg@entry=0x4519c08) at /usr/src/debug/mariadb-10.0.20/sql/sql_connect.cc:1378 #27 0x00000000006977c7 in handle_one_connection (arg=0x4519c08) at /usr/src/debug/mariadb-10.0.20/sql/sql_connect.cc:1293 ---Type < return > to continue , or q < return > to quit--- #28 0x00007fbaafffd0a4 in start_thread () from /lib64/libpthread.so.0 #29 0x00007fbaaeeca08d in clone () from /lib64/libc.so.6 (gdb) quit
            Hide
            nirbhay_c Nirbhay Choubey added a comment -

            Reopening for further investigation.

            Show
            nirbhay_c Nirbhay Choubey added a comment - Reopening for further investigation.
            Hide
            jplindst Jan Lindström added a comment -

            Tested with 10.0.20 and I got (/mnt/jan is about 7M):

            MariaDB Version 10.0.20-MariaDB-debug
             - SSL connections supported
             - binaries are debug compiled
            Collecting tests...
            Installing system database...
            
            ==============================================================================
            
            TEST                                      RESULT   TIME (ms) or COMMENT
            --------------------------------------------------------------------------
            
            worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
            create table t1(a int,b int) engine=innodb data directory='/mnt/jan';
            create procedure innodb_insert_proc (repeat_count int)
            begin
            declare current_num int;
            set current_num = 0;
            while current_num < repeat_count do
            insert into t1 values(current_num, current_num);
            set current_num = current_num + 1;
            end while;
            end//
            commit;
            set autocommit=0;
            call innodb_insert_proc(14700);
            call innodb_insert_proc(14700);
            innodb.innodb-out-of-disk 'innodb_plugin' [ fail ]
                    Test ended at 2015-07-21 13:45:40
            
            CURRENT_TEST: innodb.innodb-out-of-disk
            mysqltest: At line 18: query 'call innodb_insert_proc(14700)' failed: 1114: The table 't1' is full
            
            Show
            jplindst Jan Lindström added a comment - Tested with 10.0.20 and I got (/mnt/jan is about 7M): MariaDB Version 10.0.20-MariaDB-debug - SSL connections supported - binaries are debug compiled Collecting tests... Installing system database... ============================================================================== TEST RESULT TIME (ms) or COMMENT -------------------------------------------------------------------------- worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 create table t1(a int,b int) engine=innodb data directory='/mnt/jan'; create procedure innodb_insert_proc (repeat_count int) begin declare current_num int; set current_num = 0; while current_num < repeat_count do insert into t1 values(current_num, current_num); set current_num = current_num + 1; end while; end// commit; set autocommit=0; call innodb_insert_proc(14700); call innodb_insert_proc(14700); innodb.innodb-out-of-disk 'innodb_plugin' [ fail ] Test ended at 2015-07-21 13:45:40 CURRENT_TEST: innodb.innodb-out-of-disk mysqltest: At line 18: query 'call innodb_insert_proc(14700)' failed: 1114: The table 't1' is full
            Hide
            jplindst Jan Lindström added a comment -

            Note there are some historical cases where InnoDB assumes some operations to succeed. In these cases there is no rollback, thus it will crash.

            Show
            jplindst Jan Lindström added a comment - Note there are some historical cases where InnoDB assumes some operations to succeed. In these cases there is no rollback, thus it will crash.

              People

              • Assignee:
                jplindst Jan Lindström
                Reporter:
                jplindst Jan Lindström
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 hours
                  2h