Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 10.0.11, 10.1.0
-
Fix Version/s: 10.0.18
-
Component/s: Storage Engine - OQGRAPH
-
Labels:
-
Environment:CentOS 6, GTID based replication, 1 master 1 slave
Description
I've been updating one of my applications from OQGraphv2 to v3 and while initial tests showed everything working, I'm now getting troublesome crashes when I put some load on the OQGraph table.
The application is basically a webservice (in PHP) that queries the OQGraph table (schema attached) and returns the result. The query in itself should be fairly simple:
SELECT db.* FROM db_history AS db INNER JOIN version_history AS v ON db.nodeID = v.linkid WHERE origid = 1 AND destid = 3 AND latch = 'dijkstras';
When doing single sequential requests in the browser, everything works fine. But once I started to put some load on the webservice (using siege as load testing tool), MariaDB crashes quickly after 1-2 requests (crash dump attached), always with the same crash dump.
I should note that the database setup is a master-slave replication setup. I'm not sure that has anything to do with the crashes I'm seeing though.
I can reproduce this fairly easily and reliably in my test environment, on both the slave and the master node, but have not been successful so far in producing a test case that does not involve running siege.
Gliffy Diagrams
Attachments
Issue Links
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Yes. If you plan to fix the bug (or do some change, whatever) in both 10.0 anf 10.1, then you only need to do it and provide a pull request for 10.0 — I'll take care of propagating it to 10.1. Of course, if you want something to be done in 10.1 only, then you just create a pull request for 10.1.
I did a forward merge test to 10.1, there are a couple of instances where there is a merge conflict that needs to be resolved C,B or similar
I pushed a branch that has the correct result if you need something to compare against
https://github.com/pastcompute/server/tree/10.1-oqgraph-6282-6345-6784-cherrypick-test
I checked pull/16 and pull/17. Both fix the problem and result in a crash-free load test here 
Thank you very much!
No worries!
BTW pull #16 has been cancelled although it is a good test -
My understanding is that #17 will get merged into 10.0 branch first, then Sergei will at some point merge 10.0 -> 10.1
Hello Heinz
Is it possible you can rebuild MariaDB with debugging turned on and try again? Having had a very quick look at the code, a bunch of inline functions seem to be optimised out of the stack trace between oqgraph::cursor::seek_to() and the function where the crash seems to happen which makes it hard to see what could be happening
FWIW I am guessing something somewhere has not been done with proper thread-safety. In that vein, is there any form of logging with timing information produced by your web service that might be available?
The other thing is turning on mariadb logging, if you are able to get mysqld to run with -#d,oq-debug,info on the command line and attach those logs as well...
in the meantime I'll see if I can repeat this on my system, although I suspect it may take a little while