Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Critical
-
Resolution: Fixed
-
Affects Version/s: 5.5.28a-galera
-
Fix Version/s: 5.5.29-galera
-
Component/s: None
-
Labels:
-
Environment:Cent OS 5.8 x64
Description
/etc/init.d/mysql stop hangs forever.
Version: '5.5.28a-MariaDB' socket: '/var/lib/mysql/mysql.sock' port: 53306 MariaDB Server, wsrep_23.7rc1.rXXXX
130205 6:27:44 [Note] WSREP: Member 1 (devdb03) synced with group.
130205 6:27:44 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 277387)
130205 6:27:44 [Note] WSREP: Synchronized with group, ready for connections
130205 6:27:44 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 6:29:22 [Note] /usr/sbin/mysqld: Normal shutdown
130205 6:29:22 [Note] WSREP: Stop replication
130205 6:29:22 [Note] WSREP: Closing send monitor...
130205 6:29:22 [Note] WSREP: Closed send monitor.
130205 6:29:22 [Note] WSREP: gcomm: terminating thread
130205 6:29:22 [Note] WSREP: gcomm: joining thread
130205 6:29:22 [Note] WSREP: gcomm: closing backend
130205 6:29:23 [Note] WSREP: view(view_id(NON_PRIM,8e7f4cb4-6f5c-11e2-0800-4990a8ab3479,4) memb
joined {
} left {
} partitioned
)
130205 6:29:23 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
130205 6:29:23 [Note] WSREP: view((empty))
130205 6:29:23 [Note] WSREP: gcomm: closed
130205 6:29:23 [Note] WSREP: Flow-control interval: [16, 16]
130205 6:29:23 [Note] WSREP: Received NON-PRIMARY.
130205 6:29:23 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 277388)
130205 6:29:23 [Note] WSREP: Received self-leave message.
130205 6:29:23 [Note] WSREP: Flow-control interval: [0, 0]
130205 6:29:23 [Note] WSREP: Received SELF-LEAVE. Closing connection.
130205 6:29:23 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 277388)
130205 6:29:23 [Note] WSREP: RECV thread exiting 0: Success
130205 6:29:23 [Note] WSREP: New cluster view: global state: 34b486ae-6f0c-11e2-0800-5cdefb8ea919:277388, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
130205 6:29:23 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 6:29:23 [Note] WSREP: New cluster view: global state: 34b486ae-6f0c-11e2-0800-5cdefb8ea919:277388, view# -1: non-Primary, number of nodes: 0, my index: -1, protocol version 2
130205 6:29:23 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
130205 6:29:23 [Note] WSREP: applier thread exiting (code:0)
130205 6:29:23 [Note] WSREP: recv_thread() joined.
130205 6:29:23 [Note] WSREP: Closing slave action queue.
130205 6:29:23 [Note] WSREP: applier thread exiting (code:5)
Gliffy Diagrams
Attachments
Issue Links
- links to
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
I think I figured it out. The problem is in the "thread_handling=pool-of-threads" setting. If I remove it then the shutdown goes just fine. With the "thread_handling=pool-of-threads", the unlink_thd() is not called (probably because one_thread_per_connection_scheduler() is not called.
So yes, it was MariaDB specific at the end 
Thank you, Aleksey.
It is totally reproducible with thread pool indeed.
Reproducible on current maria-5.5-galera tree as well (revno 3378).
To reproduce, it's enough to start server with
--wsrep_cluster_address=gcomm:// --wsrep_provider=<path_to_the_library>/libgalera_smm.so --thread-handling=pool-of-threads
wait till it starts, then try to shut it down.
Seppo,
Wlad can consult from the thread pool side, please contact him if needed.
Committed a simple fix, which enables graceful shutdown of wsrep replicator when thread pool scheduler is being used.
We have conflicting work in upstream, which deals with dynamic slave thread management. This fix is therefore quite temporary and will be re-factored latest by when wsrep API #24 will be supported.
Fix was committed in revision: http://bazaar.launchpad.net/~maria-captains/maria/maria-5.5-galera/revision/3379
Hi Aleksey,
Does it happen every time, or does it depend on the workload / number of connections during the shutdown?
At the first glance it looks like the upstream bug https://bugs.launchpad.net/codership-mysql/+bug/1099742 , but i'm not 100% sure since their error log seems much shorter (or they didn't quote the whole thing), and we don't have anything else to compare. Would you be able to run strace and/or gdb bt so we could see if it stops at the same place as in the other bug?
Thanks