Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Cannot Reproduce
-
Affects Version/s: 5.5.34, 5.5.35
-
Fix Version/s: 5.5.39
-
Component/s: None
-
Labels:None
-
Environment:SW: Centos 6.5 x86_64 + Official MariaDB repo HW: 2xIntel(R) Xeon(R) CPU E5-2430, 64GB RAM, 4x2GB 7200RPM SATA Disk in RAID10
Description
When the crash happens, there are two sets of queries running on the server.
One set is populating Business Intelligence aggregate tables, the other is refreshing a development copy of a database, using parallel loading.
The two sets of queries are running on different, independent databases.
The crash does not occur when only one set of queries (e.g. only the db load queries, or only the BI aggregator queries ) are running.
The tables that I try to drop are in compressed format.
The system is IO bound when the crash occurs (iostat consistently shows 100% disk utilization), but not CPU bound.
The crash occurs when I try to drop several big (multi gigabyte) tables in parallel, as you can see in the attached error log.
Gliffy Diagrams
Attachments
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Long semaphore waits for Jan's expert reading (logs are attached).
FYI, I've just had a very similar crash on Oracle Mysql 5.6.17. I can attach the logs for it if it is interesting to you.
Hi,
Why they are running these huge inserts like:
---TRANSACTION 705FC, ACTIVE 1492 sec
mysql tables in use 9, locked 9
69176 lock struct(s), heap size 6175160, 3804151 row lock(s)
If these inserts can't be splitted, you need to increase the long semaphore wait to higher number.
R: Jan
These are the queries that transform the raw data to the star schema required by the BI software. It may be possible to re-write them, but since mysql 5.5, and MariaDB 10.x does not exhibit the crash under the same circumstances, they are probably triggering a bug specific to mysql 5.6 (sometimes crashes) and MariaDB 5.5 (crashes every time).
However, accoring to our tests the inserts in themselves are not the problem, the crash only happens when they are running in parallel with the drop table statements.
revno: 4230
committer: Jan Lindström <jplindst@mariadb.org>
branch nick: 5.5
timestamp: Wed 2014-07-23 09:04:59 +0300
message:
MDEV-5673: Crash while parallel dropping multiple tables under heavy load
Improve long semaphore wait output to include all semaphore waits
and try to find out if there is a sequence of waiters.
We have upgraded to 10.0.8, with the same configiuration, and haven't experienced the problem since then.