Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-444

Crash in ALTER TABLE ADD PRIMARY KEY ..., ADD INDEX

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 5.5.25
    • Fix Version/s: 5.5.27
    • Component/s: None
    • Labels:
      None
    • Environment:
      OpenSuse 11.2 64-bit

      Description

      When attempting to add indexes to a smaller version of table lineitem from DBT3, the ALTER TABLE crashes the server. How to reproduce:

      • Use DBT3 scale 10
      • Run the following statements:

      create table t_ps_partkey as
      (select distinct ps_partkey, ps_suppkey
      from partsupp
      where ps_partkey in (select p_partkey from part where p_name like 'forest%'));

      alter table t_ps_partkey add index i_ps_partkey(ps_partkey, ps_suppkey);

      create table lineitem_small as
      (select distinct lineitem.* from lineitem, t_ps_partkey
      where l_partkey = ps_partkey
      and l_suppkey = ps_suppkey
      and l_shipdate >= date('1994-01-01')
      and l_shipdate < date('1994-01-01') + interval '1' year );

      ALTER TABLE lineitem_small
      ADD PRIMARY KEY (l_orderkey, l_linenumber),
      ADD INDEX i_l_shipdate(l_shipdate),
      ADD INDEX i_l_suppkey_partkey (l_partkey, l_suppkey),
      ADD INDEX i_l_partkey (l_partkey),
      ADD INDEX i_l_suppkey (l_suppkey),
      ADD INDEX i_l_receiptdate (l_receiptdate),
      ADD INDEX i_l_orderkey (l_orderkey),
      ADD INDEX i_l_orderkey_quantity (l_orderkey, l_quantity),
      ADD INDEX i_l_commitdate (l_commitdate);

      The last statement results in a crash:

      InnoDB: DEBUG: update_statistics for dbt3/lineitem_small.
      120809 23:51:18 InnoDB: Assertion failure in thread 140347972777728 in file buf0buf.ic line 515
      InnoDB: Failing assertion: mutex_own(&buf_pool->LRU_list_mutex)
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.

      The stack trace is:

      #0 0x00007fa5b074ad95 in raise () from /lib64/libc.so.6
      #1 0x00007fa5b074c2ab in abort () from /lib64/libc.so.6
      #2 0x0000000000b365f3 in buf_page_set_sticky (bpage=0x7fa55b773cc0) at /home/tsk/mprog/src/5.5/storage/xtradb/include/buf0buf.ic:515
      #3 0x0000000000b494aa in buf_flush_yield (buf_pool=0x4bedf08, bpage=0x7fa55b773cc0) at /home/tsk/mprog/src/5.5/storage/xtradb/buf/buf0lru.c:399
      #4 0x0000000000b495a9 in buf_flush_try_yield (buf_pool=0x4bedf08, bpage=0x7fa55b773cc0, processed=1024) at /home/tsk/mprog/src/5.5/storage/xtradb/buf/buf0lru.c:445
      #5 0x0000000000b498b9 in buf_flush_or_remove_pages (buf_pool=0x4bedf08, id=30) at /home/tsk/mprog/src/5.5/storage/xtradb/buf/buf0lru.c:569
      #6 0x0000000000b4992c in buf_flush_dirty_pages (buf_pool=0x4bedf08, id=30) at /home/tsk/mprog/src/5.5/storage/xtradb/buf/buf0lru.c:600
      #7 0x0000000000b49e2f in buf_LRU_flush_or_remove_pages (id=30, buf_remove=BUF_REMOVE_FLUSH_NO_WRITE) at /home/tsk/mprog/src/5.5/storage/xtradb/buf/buf0lru.c:789
      #8 0x0000000000b74ada in fil_delete_tablespace (id=30, evict_all=0) at /home/tsk/mprog/src/5.5/storage/xtradb/fil/fil0fil.c:2383
      #9 0x0000000000add415 in row_drop_table_for_mysql (name=0x7fa5200f0098 "dbt3/lineitem_small@00232", trx=0x7fa520063ac8, drop_db=0) at /home/tsk/mprog/src/5.5/storage/xtradb/row/row0mysql.c:3565
      #10 0x0000000000ad73e9 in row_merge_drop_table (trx=0x7fa520063ac8, table=0x7fa5200683d8) at /home/tsk/mprog/src/5.5/storage/xtradb/row/row0merge.c:2634
      #11 0x0000000000ac2c9a in ha_innobase::final_add_index (this=0x7fa5200deee8, add_arg=0x7fa5200da970, commit=true) at /home/tsk/mprog/src/5.5/storage/xtradb/handler/handler0alter.cc:1023
      #12 0x00000000006c51c6 in mysql_alter_table (thd=0x7e38870, new_db=0x7fa520006cb8 "dbt3", new_name=0x7fa5200066a8 "lineitem_small", create_info=0x7fa54f0fdd00, table_list=0x7fa5200066f0, alter_info=0x7fa54f0fde00, order_num=0, order=0x0, ignore=false, require_online=false) at /home/tsk/mprog/src/5.5/sql/sql_table.cc:6946
      #13 0x000000000093f508 in Alter_table_statement::execute (this=0x7fa5200074b8, thd=0x7e38870) at /home/tsk/mprog/src/5.5/sql/sql_alter.cc:106
      #14 0x000000000062f6ee in mysql_execute_command (thd=0x7e38870) at /home/tsk/mprog/src/5.5/sql/sql_parse.cc:4459
      #15 0x000000000063272a in mysql_parse (thd=0x7e38870, rawbuf=0x7fa5200062c8 "ALTER TABLE lineitem_small\n ADD PRIMARY KEY (l_orderkey, l_linenumber),\n ADD INDEX i_l_shipdate(l_shipdate),\n ADD INDEX i_l_suppkey_partkey (l_partkey, l_suppkey),\n ADD INDEX i_l_partkey (l_pa"..., length=435, parser_state=0x7fa54f0fe660) at /home/tsk/mprog/src/5.5/sql/sql_parse.cc:5736

      The crash is in the function: buf_page_set_sticky(), on the line:
      ut_ad(mutex_own(&buf_pool->LRU_list_mutex));

      p buf_pool->LRU_list_mutex
      $5 = {
      event = 0x4bee840,
      lock_word = 0 '\000',
      waiters = 0,
      list =

      { prev = 0x4bee150, next = 0x4bedf08 }

      ,
      cfile_name = 0xe0b8f0 "/home/src/5.5/storage/xtradb/buf/buf0buf.c",
      cline = 1267,
      thread_id = 18446744073709551615,
      magic_n = 979585,
      count_os_wait = 0,
      count_using = 3578904,
      count_spin_loop = 0,
      count_spin_rounds = 0,
      count_os_yield = 0,
      lspent_time = 0,
      lmax_spent_time = 0,
      mutex_type = 0,
      cmutex_name = 0xe0c1ce "&buf_pool->LRU_list_mutex",
      pfs_psi = 0x0
      }

        Gliffy Diagrams

          Attachments

            Activity

            Hide
            timour Timour Katchaounov added a comment -

            p mutex_own(&buf_pool->LRU_list_mutex)
            $9 = 0

            Show
            timour Timour Katchaounov added a comment - p mutex_own(&buf_pool->LRU_list_mutex) $9 = 0
            Hide
            timour Timour Katchaounov added a comment -

            In order to reproduce the crash, it is sufficient to run only this reduced statement:

            ALTER TABLE lineitem_small ADD PRIMARY KEY (l_orderkey, l_linenumber);

            Show
            timour Timour Katchaounov added a comment - In order to reproduce the crash, it is sufficient to run only this reduced statement: ALTER TABLE lineitem_small ADD PRIMARY KEY (l_orderkey, l_linenumber);
            Hide
            elenst Elena Stepanova added a comment - - edited

            I haven't got the crash so far. Here is what I did in my latest (cleanest) attempt:

            • start server with innodb-file-per-table=1 and innodb-file-format=Barracuda, tmpdir pointing at a disk with more space, the rest is default;
            • feed mariadb-tools/dbt3_benchmark/dbt3_mysql/make-dbt3-db_pre_create_PK.sql to the server. No modifications were done to the script, apart from pointing it to SF10 data;
            • stop server (to backup the database);
            • start server with the same exact parameters as before;
            • run create statements table t_ps_partkey, alter table t_ps_partkey, create table lineitem_small as given in the description;
            • run ALTER TABLE lineitem_small ADD PRIMARY KEY as given in the comment;

            The last ALTER TABLE finished fairly fast (in ~30 seconds), no problems or errors in the log file.
            After that I tried to drop the PK and run the big ALTER TABLE lineitem_small from the description, adding many indexes, which also went all right.

            I was running it on perro, which is openSUSE 11.3 x86_64 8 Gb.

            A couple observations:

            First, despite what was discussed on IRC, it seems to depend on tmpdir after all. In previous dirty attempts I had been having a problem with disk space on alter table until I switched tmpdir from /tmp (which had about 2 Gb of free space) to the main storage which had about 17 Gb.
            Second, when I was having the disk problems, I was getting an error "wrong key", which was preceded by the server complaining into the error log about not enough disk space (error 28). I'm wondering if in certain cases it asserts instead of aborting the operation.

            Timour, you were going to check the tmpdir when you were reproducing the issue. Did you have enough space in there?
            Also, could you please provide the entire command line for the server startup (or your cnf config if you are using one). I am wondering if it behaves differently for me because I'm using a different set of parameters.

            Show
            elenst Elena Stepanova added a comment - - edited I haven't got the crash so far. Here is what I did in my latest (cleanest) attempt: start server with innodb-file-per-table=1 and innodb-file-format=Barracuda, tmpdir pointing at a disk with more space, the rest is default; feed mariadb-tools/dbt3_benchmark/dbt3_mysql/make-dbt3-db_pre_create_PK.sql to the server. No modifications were done to the script, apart from pointing it to SF10 data; stop server (to backup the database); start server with the same exact parameters as before; run create statements table t_ps_partkey, alter table t_ps_partkey, create table lineitem_small as given in the description; run ALTER TABLE lineitem_small ADD PRIMARY KEY as given in the comment; The last ALTER TABLE finished fairly fast (in ~30 seconds), no problems or errors in the log file. After that I tried to drop the PK and run the big ALTER TABLE lineitem_small from the description, adding many indexes, which also went all right. I was running it on perro, which is openSUSE 11.3 x86_64 8 Gb. A couple observations: First, despite what was discussed on IRC, it seems to depend on tmpdir after all. In previous dirty attempts I had been having a problem with disk space on alter table until I switched tmpdir from /tmp (which had about 2 Gb of free space) to the main storage which had about 17 Gb. Second, when I was having the disk problems, I was getting an error "wrong key", which was preceded by the server complaining into the error log about not enough disk space (error 28). I'm wondering if in certain cases it asserts instead of aborting the operation. Timour, you were going to check the tmpdir when you were reproducing the issue. Did you have enough space in there? Also, could you please provide the entire command line for the server startup (or your cnf config if you are using one). I am wondering if it behaves differently for me because I'm using a different set of parameters.
            Hide
            timour Timour Katchaounov added a comment -

            I changed the temporary directory to my home dir, where there was ~25 GB free space.
            Still, just executing the statement that adds a PK results in a crash.

            The server is a debug build, built with ./BUILD/compile-amd64-debug-max-no-ndb

            The server was started from the source tree with the following command line:

            ./sql/mysqld --no-defaults --skip-networking --datadir=/home/<user>/dbdata/dbt3s10-mdb-5.5-a --tmpdir=/home/<user>/tmp/db --socket=/home/<user>/dbdata/dbt3s10-mdb-5.5-a/mysql.sock --lc-messages-dir=/home/<user>/mprog/src/5.5/sql/share --character-sets-dir=/home/<user>/mprog/src/5.5/sql/share/charsets --lower-case-table-names=1 --key_buffer_size=64M --table_cache=256 --sort_buffer_size=64M --join_buffer_size=512M --read_buffer_size=16M --innodb_buffer_pool_size=1024M --innodb_additional_mem_pool_size=128M --innodb_log_buffer_size=64M --innodb_log_file_size=256M --innodb_file_per_table --innodb_data_file_path=ibdata1:128M:autoextend

            Show
            timour Timour Katchaounov added a comment - I changed the temporary directory to my home dir, where there was ~25 GB free space. Still, just executing the statement that adds a PK results in a crash. The server is a debug build, built with ./BUILD/compile-amd64-debug-max-no-ndb The server was started from the source tree with the following command line: ./sql/mysqld --no-defaults --skip-networking --datadir=/home/<user>/dbdata/dbt3s10-mdb-5.5-a --tmpdir=/home/<user>/tmp/db --socket=/home/<user>/dbdata/dbt3s10-mdb-5.5-a/mysql.sock --lc-messages-dir=/home/<user>/mprog/src/5.5/sql/share --character-sets-dir=/home/<user>/mprog/src/5.5/sql/share/charsets --lower-case-table-names=1 --key_buffer_size=64M --table_cache=256 --sort_buffer_size=64M --join_buffer_size=512M --read_buffer_size=16M --innodb_buffer_pool_size=1024M --innodb_additional_mem_pool_size=128M --innodb_log_buffer_size=64M --innodb_log_file_size=256M --innodb_file_per_table --innodb_data_file_path=ibdata1:128M:autoextend
            Hide
            elenst Elena Stepanova added a comment - - edited

            Attached mdev-444.data – the load file for the provided test case. ~12 Mb uncompressed, contains 100K rows.

            Show
            elenst Elena Stepanova added a comment - - edited Attached mdev-444.data – the load file for the provided test case. ~12 Mb uncompressed, contains 100K rows.
            Hide
            elenst Elena Stepanova added a comment -

            Reproducible on the current MariaDB 5.5 revno 3492 with XtraDB, percona-server revno 288.
            Not reproducible on MariaDB 5.5 with InnoDB plugin, current MySQL 5.5, 5.6, MariaDB 5.3.
            No crash on a release build.

            1. Test case
            2. (please note that you need to download and extract
            3. the attached file mdev-444.data, put it in <basedir>/mysql-test).
            4. run as
            5. perl ./mtr -mysqld=innodb_log_buffer_size=64M --mysqld=innodb_log_file_size=64M --mysqld=-innodb_buffer_pool_size=128M main.mdev-444
            6. where t/mdev-444.test is the test file

            --source include/have_innodb.inc

            SET GLOBAL innodb_file_per_table=ON;

            USE test;
            DROP TABLE IF EXISTS t1;

            CREATE TABLE t1 (
            `l_orderkey` int(11) NOT NULL DEFAULT '0',
            `l_partkey` int(11) DEFAULT NULL,
            `l_suppkey` int(11) DEFAULT NULL,
            `l_linenumber` int(11) NOT NULL DEFAULT '0',
            `l_quantity` double DEFAULT NULL,
            `l_extendedprice` double DEFAULT NULL,
            `l_discount` double DEFAULT NULL,
            `l_tax` double DEFAULT NULL,
            `l_returnflag` char(1) DEFAULT NULL,
            `l_linestatus` char(1) DEFAULT NULL,
            `l_shipDATE` date DEFAULT NULL,
            `l_commitDATE` date DEFAULT NULL,
            `l_receiptDATE` date DEFAULT NULL,
            `l_shipinstruct` char(25) DEFAULT NULL,
            `l_shipmode` char(10) DEFAULT NULL,
            `l_comment` varchar(44) DEFAULT NULL
            ) ENGINE=InnoDB DEFAULT CHARSET=latin1;

            LOAD DATA LOCAL INFILE 'mdev-444.data' INTO TABLE t1;

            --enable_reconnect
            --append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect
            restart
            EOF
            --shutdown_server 60
            --source include/wait_until_connected_again.inc

            ALTER TABLE t1 ADD PRIMARY KEY (l_orderkey, l_linenumber);

            1. Cleanup
              DROP TABLE t1;
            1. End of test case
            Show
            elenst Elena Stepanova added a comment - Reproducible on the current MariaDB 5.5 revno 3492 with XtraDB, percona-server revno 288. Not reproducible on MariaDB 5.5 with InnoDB plugin, current MySQL 5.5, 5.6, MariaDB 5.3. No crash on a release build. Test case (please note that you need to download and extract the attached file mdev-444.data, put it in <basedir>/mysql-test). run as perl ./mtr - mysqld= innodb_log_buffer_size=64M --mysqld= innodb_log_file_size=64M --mysqld= -innodb_buffer_pool_size=128M main.mdev-444 where t/mdev-444.test is the test file --source include/have_innodb.inc SET GLOBAL innodb_file_per_table=ON; USE test; DROP TABLE IF EXISTS t1; CREATE TABLE t1 ( `l_orderkey` int(11) NOT NULL DEFAULT '0', `l_partkey` int(11) DEFAULT NULL, `l_suppkey` int(11) DEFAULT NULL, `l_linenumber` int(11) NOT NULL DEFAULT '0', `l_quantity` double DEFAULT NULL, `l_extendedprice` double DEFAULT NULL, `l_discount` double DEFAULT NULL, `l_tax` double DEFAULT NULL, `l_returnflag` char(1) DEFAULT NULL, `l_linestatus` char(1) DEFAULT NULL, `l_shipDATE` date DEFAULT NULL, `l_commitDATE` date DEFAULT NULL, `l_receiptDATE` date DEFAULT NULL, `l_shipinstruct` char(25) DEFAULT NULL, `l_shipmode` char(10) DEFAULT NULL, `l_comment` varchar(44) DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=latin1; LOAD DATA LOCAL INFILE 'mdev-444.data' INTO TABLE t1; --enable_reconnect --append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect restart EOF --shutdown_server 60 --source include/wait_until_connected_again.inc ALTER TABLE t1 ADD PRIMARY KEY (l_orderkey, l_linenumber); Cleanup DROP TABLE t1; End of test case
            Hide
            elenst Elena Stepanova added a comment -
            Show
            elenst Elena Stepanova added a comment - Also filed for Percona Server as https://bugs.launchpad.net/percona-server/+bug/1038225
            Hide
            wlad Vladislav Vaintroub added a comment -

            After conversation with Laurinas on IRC, found out that fix for the bug will be included into the next Percona Server, which should be released ASAP. It could make sense to wait and merge the new XtraDB before MariaDB release, since it has at least one more critical bugfix

            Show
            wlad Vladislav Vaintroub added a comment - After conversation with Laurinas on IRC, found out that fix for the bug will be included into the next Percona Server, which should be released ASAP. It could make sense to wait and merge the new XtraDB before MariaDB release, since it has at least one more critical bugfix
            Hide
            serg Sergei Golubchik added a comment -

            merged with XtraDB from Percona-Server-5.5.27-rel28.1

            Show
            serg Sergei Golubchik added a comment - merged with XtraDB from Percona-Server-5.5.27-rel28.1

              People

              • Assignee:
                wlad Vladislav Vaintroub
                Reporter:
                timour Timour Katchaounov
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 0 minutes
                  0m
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 3 hours
                  3h