Kristian Nielsen added a comment - 2015-06-11 12:50 So this turns out to actually be an old bug. It just happened to trigger more easily after patch for MDEV-8294 . The race is that there is a small window where SHOW GLOBAL STATUS may count twice the values of a thread that is exiting. This happens also for normal threads, not just the slave thread. Here is a test case for latest 10.0, triggers with the following sleep patched in: diff --git a/sql/mysqld.cc b/sql/mysqld.cc index e05c0b6..75af8f7 100644 --- a/sql/mysqld.cc +++ b/sql/mysqld.cc @@ -2759,6 +2759,7 @@ void unlink_thd(THD *thd) thd->add_status_to_global(); +my_sleep(2000000); mysql_mutex_lock(&LOCK_thread_count); thd->unlink(); /* CREATE TABLE t1 (a INT PRIMARY KEY); connect (c1,localhost,root,,); INSERT INTO t1 VALUES (1); INSERT INTO t1 VALUES (2); INSERT INTO t1 VALUES (3); --let $cid= `SELECT CONNECTION_ID()` --connection default eval KILL CONNECTION $cid; --sleep 1 SHOW GLOBAL STATUS LIKE 'Com_insert'; --sleep 2 SHOW GLOBAL STATUS LIKE 'Com_insert'; DROP TABLE t1; The first SHOW GLOBAL STATUS counts the thread twice: CREATE TABLE t1 (a INT PRIMARY KEY); INSERT INTO t1 VALUES (1); INSERT INTO t1 VALUES (2); INSERT INTO t1 VALUES (3); KILL CONNECTION 4; SHOW GLOBAL STATUS LIKE 'Com_insert'; Variable_name Value Com_insert 6 SHOW GLOBAL STATUS LIKE 'Com_insert'; Variable_name Value Com_insert 3 DROP TABLE t1;

Kristian Nielsen added a comment - 2015-06-11 13:05 The race occurs because there is a window of time between a thread does THD::add_status_to_global() and THD::unlink(). During this window, calc_sum_of_all_status() can run and count the contribution of the thread twice.

Kristian Nielsen added a comment - 2015-06-11 13:31 The original buildbot test failure can be reproduced easily by injecting the following sleep in the code: diff --git a/sql/slave.cc b/sql/slave.cc index 4129c4c..32e6bd4 100644 --- a/sql/slave.cc +++ b/sql/slave.cc @@ -4837,6 +4837,7 @@ log '%s' at position %s, relay log '%s' position: %s%s", RPL_LOG_NAME, mysql_cond_broadcast(&rli->stop_cond); DBUG_EXECUTE_IF("simulate_slave_delay_at_terminate_bug38694", sleep(5);); mysql_mutex_unlock(&rli->run_lock); // tell the world we are done +my_sleep(500000); /* Deactivate the parallel replication thread pool, if there are now no more

Kristian Nielsen added a comment - 2015-06-11 13:39 Monty said he would fix this during this week

Kristian Nielsen added a comment - 2015-06-15 09:27 Monty, I've pushed a temporary work-around for the test failure in rpl.rpl_000011: http://lists.askmonty.org/pipermail/commits/2015-June/008039.html Please remove that work-around from the testcase when you push the fix for this underlying issue.

Michael Widenius added a comment - 2015-06-26 00:38 Fixed by adding a marker if we have added the thread counters to the global counters. Fix will be pushed into 5.5 tree

Michael Widenius added a comment - 2015-06-26 15:08 Fix pushed into 5.5 When merged to 10.0, the above temporary fix should be removed and this bug closed.