Race in InnoDB/XtraDB mutex implementation can stall or hang the server

Description

This failure is seen in Buildbot. It is a race that occurs somewhat rarely,
but it can be consistently reproduced manually by running tests in a loop.

The failure can occur in different tests, for example like this:

http://buildbot.askmonty.org/buildbot/builders/bintar-trusty-p8-debug/builds/173

More failures are seen in a cross-reference search:

http://buildbot.askmonty.org/buildbot/reports/cross_reference#branch=&revision=&platform=-p8&dt=&bbnum=&typ=&info=&fail_name=&fail_variant=&fail_info_short=&fail_info_full=Failed+to+start+mysqld&limit=100

It looks like it only fails in debug builds on Power8, but this is not 100%
confirmed.

Looking into the error log, we consistently see that it hangs during creation
of the Doublewrite buffer:

Normally there should be a message after this that the doublewrite buffer was
created successfully, but when the error occurs, instead the server hangs at
this point.

Here is a GDB stacktrace from when it hangs:

Here is how I reproduced it, running the test in a loop and waiting for it to
hang:

Environment

None

Status

Assignee

Sergey Vojtovich

Reporter

Kristian Nielsen

Labels

Fix versions

Affects versions

Priority

Blocker
Configure