Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 5.5.39, 10.0.13
-
Component/s: Storage Engine - TokuDB
-
Labels:
-
Environment:Ubuntu 64bit 14.4 LTS: linux-kernel-3.13.0-24-generic
Description
I have a table partitioned by range created with the tokudb engine on mariadb 10.0.12. When I have one client inserting into this table everything works as expected. However, as soon as I have a second client inserting more data into this table I get the following error: Failed to read auto-increment value from storage engine. When I see the table status the Auto_increment value is 2049. My auto increment description is "id bigint not null auto_increment".
I can reproduce this problem with the below table and Java client side code. Another tokudb user on the tokudb-user group performed the test on Percona Server 5.6, which did not fail: https://groups.google.com/forum/#!topic/tokudb-user/2pnjQxuuvUo
CREATE TABLE `auto_inc_test` (
`id` bigint(20) NOT NULL AUTO_INCREMENT,
`time` bigint(20) NOT NULL,
PRIMARY KEY (`time`,`id`)
) ENGINE=TokuDB AUTO_INCREMENT=1001 DEFAULT CHARSET=latin1 `compression`='tokudb_zlib'
ALTER TABLE order_books partition by range (time) (
partition p0 values less than (1507495611880)
);
public class MultipleClientsInsertingDataIntoPartitionedTokudbTable { public static final String INSERT_STATEMENT = "INSERT INTO test.auto_inc_test (time) values (?)"; private final CountDownLatch countDownLatch; public MultipleClientsInsertingDataIntoPartitionedTokudbTable() { countDownLatch = new CountDownLatch(2); } public static void main(String[] args) throws InterruptedException { MultipleClientsInsertingDataIntoPartitionedTokudbTable multipleClientsInsertingDataIntoPartitionedTokudbTable = new MultipleClientsInsertingDataIntoPartitionedTokudbTable(); multipleClientsInsertingDataIntoPartitionedTokudbTable.start(); } private void start() throws InterruptedException { DataSource dataSource = createDataSource(); new Thread(new InsertRunnable(dataSource)).start(); new Thread(new InsertRunnable(dataSource)).start(); countDownLatch.await(); } private DataSource createDataSource() { String jdbcUrl = "jdbc:mysql://xxx.xxx.xxx.xxx/test?tcpNoDelay=true&tcpKeepAlive=true&rewriteBatchedStatements=true"; String username = "user"; String password = "password"; MysqlDataSource mysqlDataSource = new MysqlDataSource(); mysqlDataSource.setURL(jdbcUrl); mysqlDataSource.setUser(username); mysqlDataSource.setPassword(password); return mysqlDataSource; } private void insertData(DataSource dataSource) { try (Connection connection = dataSource.getConnection(); PreparedStatement preparedStatement = connection .prepareStatement(INSERT_STATEMENT)) { connection.setAutoCommit(false); for (int i = 0; i < 1000; i++) { preparedStatement.setLong(1, System.currentTimeMillis()); preparedStatement.addBatch(); } preparedStatement.executeBatch(); } catch (SQLException e) { e.printStackTrace(); } } private class InsertRunnable implements Runnable { private final DataSource dataSource; private InsertRunnable(DataSource dataSource) { this.dataSource = dataSource; } @Override public void run() { for (int i = 0; i < 1000; i++) { insertData(dataSource); } countDownLatch.countDown(); } } }
Gliffy Diagrams
Attachments
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
MariaDB will not create an InnoDB table with an auto increment column that is not the first column in the primary key. This tells me that there are likely to be bugs in MariaDB's handling of auto increment since no other engines (except TokuDB) supports it.
See the following example.
MariaDB [test]> CREATE TABLE `auto_inc_test` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `time` bigint(20) NOT NULL, PRIMARY KEY (`time`,`id`) ) ENGINE=tokudb;
Query OK, 0 rows affected (0.05 sec)
MariaDB [test]> drop table auto_inc_test;
Query OK, 0 rows affected (0.02 sec)
MariaDB [test]> CREATE TABLE `auto_inc_test` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `time` bigint(20) NOT NULL, PRIMARY KEY (`time`,`id`) ) ENGINE=innodb;
ERROR 1075 (42000): Incorrect table definition; there can be only one auto column and it must be defined as a key
"no other engine" is not exactly correct. The table flag HA_AUTO_PART_KEY is responsible for this feature, and it is announced by Blackhole, Federated, FederatedX, Aria, MyISAM, Merge, and TokuDB engines. InnoDB and XtraDB, indeed, do not have it.
This test program deadlocks. One thread owns the partition auto inc mutex and is blocked trying to get a tokudb lock on a PK. The other threads owns a tokudb lock on a PK that conflicts with the first thread, and is blocked trying to get the partition auto inc mutex. Eventually, tokudb's lock timeout expires, and the first thread gets a lock timeout error. This hits the debug assert (on a debug build), or returns an error to the app (on a release build).
Rich Prohaska, thanks. Could you elaborate a bit, please? Where the tokudb lock on PK is taken and on what code execution path is it taken before the partition auto inc mutex?
given table mdev6605 (t bigint not null, id bigint not null auto_increment, primary key(t,id)) engine=tokudb partition by range(t) (partition p0 values less than (1));
t1: begin; insert into mdev6605 (t) values (0);
t1 holds a lock on the pk. the pk lock taken by ha_tokudb::index_read, handler::index_read_map, handler::get_auto_increment, ha_partition::get_auto_increment call stack.
t2: begin; insert into mdev6605 (t) values (0);
t2 holds the partition auto inc mutex, blocked on the tokudb pk
t1: insert into mdev6605 (t) values (0);
t1 blocked on the partition auto inc mutex
eventually t2 times out in tokudb's lock manager, t2 rolls back, and the deadlock is removed.
Thanks. Simple test case without partitioning:
create table t1 (a int auto_increment, b bigint(20), primary key (b,a)) engine=tokudb; start transaction; insert t1 (b) values (1); connect(con2,localhost,root); start transaction; insert t1 (b) values (1); -- crash!
That means, partitioning autoinc mutex is not the issue here. handler::get_auto_increment() crashes, because it doesn't expect an error, but, I believe, lock timeout is a perfectly valid status and it should be expected. Compare with
create table t1 (a int auto_increment, b bigint(20), primary key (b,a)) engine=tokudb; start transaction; insert t1 select max(a)+1, 1 from t1 where b=1; connect(con2,localhost,root); start transaction; insert t1 select max(a)+1, 1 from t1 where b=1;
This has the same semantics, and the second insert also times out on a lock wait.
The crash is on a debug assert. The test case just gets an error on a release build.
True. And the deadlock is caused by the partitioning auto-inc mutex. But the underlying error will happen in debug or release builds, with or without partitioning. See the second test case, the one with insert ... select. The second thread needs to read the value from the index, and the first thread keeps it locked. There can be no workaround, if the first thread doesn't commit — the second will time out eventually.
I'll fix the deadlock, though, so that the first thread would be able to commit.
Reproducible on current 5.5 and 10.0 trees.
On debug versions, the test case causes the assertion failure:
Slightly tweaked Java class (imports added, connection parameters changed to MTR-like). Apparently, Java 7 is required for it to work.
Also slightly tweaked version of SQL (table name fixed, COMPRESSION clause removed):
CREATE TABLE `auto_inc_test` ( `id` bigint(20) NOT NULL AUTO_INCREMENT, `time` bigint(20) NOT NULL, PRIMARY KEY (`time`,`id`) ) ENGINE=TokuDB AUTO_INCREMENT=1001 DEFAULT CHARSET=latin1; ALTER TABLE auto_inc_test partition by range (time) ( partition p0 values less than (1507495611880) );Couldn't reproduce on the Tokutek MariaDB 5.5.38 build.