Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-8044

sigsegv during XtraDB shutdown

        Details

        • Type: Bug
        • Status: Closed
        • Priority: Minor
        • Resolution: Cannot Reproduce
        • Affects Version/s: 10.1.4
        • Fix Version/s: N/A
        • Component/s: Storage Engine - XtraDB
        • Labels:
          None
        • Environment:
          CentOS 6
        • Sprint:
          10.1.6-1

          Description

          There's clearly not much info to go on here, but this task can act as a stub in case we find more to work with. The user is seeing sigsegv each time the server shuts down, after "InnoDB: Starting shutdown" and before "InnoDB: Shutdown completed". There is no useful backtrace.

          150423 17:12:28 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
150423 17:12:28 mysqld_safe WSREP: Running position recovery with --log_error='/var/lib/mysql/wsrep_recovery.kYi8Y1' --pid-file='...'
150423 17:12:41 mysqld_safe WSREP: Position recovery skipped
150423 17:12:42 [Note] InnoDB: Using mutexes to ref count buffer pool pages
150423 17:12:42 [Note] InnoDB: The InnoDB memory heap is disabled
150423 17:12:42 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
150423 17:12:42 [Note] InnoDB: Memory barrier is not used
150423 17:12:42 [Note] InnoDB: Compressed tables use zlib 1.2.3
150423 17:12:42 [Note] InnoDB: Using Linux native AIO
150423 17:12:42 [Note] InnoDB: Using CPU crc32 instructions
150423 17:12:42 [Note] InnoDB: Initializing buffer pool, size = 200.0G
150423 17:12:50 [Note] InnoDB: Completed initialization of buffer pool
150423 17:12:51 [Note] InnoDB: Highest supported file format is Barracuda.
150423 17:12:51 [Note] InnoDB: Redo log crypto: unencrypted key ver.
150423 17:12:51 [Note] InnoDB: Log scan progressed past the checkpoint lsn 121695848220106
150423 17:12:51 [Note] InnoDB: Database was not shutdown normally!
150423 17:12:51 [Note] InnoDB: Starting crash recovery.
150423 17:12:51 [Note] InnoDB: Reading tablespace information from the .ibd files...
150423 17:12:51 [Note] InnoDB: Restoring possible half-written data pages
150423 17:12:51 [Note] InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 121695848220116
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 401474766, file name mysql-bin.084373
InnoDB: Last MySQL binlog file position 0 150702, file name ./mysql-bin.025303
150423 17:12:54 [Note] InnoDB: 128 rollback segment(s) are active.
150423 17:12:54 [Note] InnoDB: Waiting for purge to start
150423 17:12:54 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.22-72.0 started; log sequence number 121695848220116
150423 17:12:54 [Note] Server socket created on IP: '0.0.0.0'.
150423 17:12:54 [Note] WSREP: Read nil XID from storage engines, skipping position init
150423 17:12:54 [Note] WSREP: wsrep_load(): loading provider library 'none'
150423 17:12:54 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.1.4-MariaDB-wsrep-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MariaDB Server, wsrep_25.10.r4144
150423 17:12:54 [Note] Slave I/O thread: connected to master 'replic@venom:3306',replication starts at GTID position '205-37100-103818655,0-37100-1297299746,504-37100-775544,503-37100-697202,81-37100-1979,805-37100-169,206-37100-106185681,309-37100-941659,807-37100-164,308-37100-597508,207-37100-103810546,311-37100-846659,314-37100-587201,806-37100-184,302-37100-71829638,214-37100-1094,500-37100-772002,208-37100-28120,513-37100-550392,904-37100-784,209-37100-962,36-37100-109836652,907-37100-696,30-37100-140906688,502-37100-749228,310-37100-442922,31-37100-142509720,210-37100-1081,912-37100-249,810-37100-27,211-37100-962,313-37100-625695,913-37100-277,32-37100-135991140,501-37100-778681,312-37100-486452,33-37100-135071016,801-37100-7498,517-37100-563773,304-37100-72239237,213-37100-891,2001-37100-1739,306-37100-70223908,34-37100-133513877,201-37100-105166098,902-37100-906,35-37100-132901879,514-37100-950440,519-37100-550327,908-37100-727,510-37100-1087405,202-37100-107537995,512-37100-492831,905-37100-557,203-37100-105579213,515-37
150423 17:13:47 [Note] Slave: received end packet from server, apparent master shutdown:
150423 17:13:47 [Note] Slave I/O thread: Failed reading log event, reconnecting to retry, log 'mysql-bin.025219' at position 40124; GTID position '205-37100-103818655,0-37100-1297301770,504-37100-775544,503-37100-697202,81-37100-1979,805-37100-169,206-37100-106185681,309-37100-
150423 17:13:47 [ERROR] Slave I/O: error reconnecting to master 'replic@venom:3306' - retry-time: 60 retries: 86400 message: Can't connect to MySQL server on 'venom' (111 "Connection refused"), Internal MariaDB error code: 2003
150423 17:13:51 [Note] /usr/sbin/mysqld: Normal shutdown

150423 17:13:51 [Note] Event Scheduler: Purging the queue. 0 events
150423 17:13:51 [Note] Slave I/O thread exiting, read up to log 'mysql-bin.025219', position 40124; GTID position 205-37100-103818655,0-37100-1297301770,504-37100-775544,503-37100-697202,81-37100-1979,805-37100-169,206-37100-106185681,309-37100-941659,807-37100-164,308-37100-597508,207-37100-103810546,311-37100-846659,314-37100-587201,806-37100-184,302-37100-71829638,214-37100-1094,500-37100-772002,208-37100-28120,513-37100-550392,904-37100-784,209-37100-962,36-37100-109836652,907-37100-696,30-37100-140906688,502-37100-749228,310-37100-442922,31-37100-142509720,210-37100-1081,912-37100-249,810-37100-27,211-37100-962,313-37100-625695,913-37100-277,32-37100-135991140,501-37100-778681,312-37100-486452,33-37100-135071016,801-37100-7498,517-37100-563773,304-37100-72239237,213-37100-891,2001-37100-1739,306-37100-70223908,34-37100-133513877,201-37100-105166098,902-37100-906,35-37100-132901879,514-37100-950440,519-37100-550327,908-37100-727,510-37100-1087405,202-37100-107537995,512-37100-492831,905-37100-557,203-37100-105579213,515-37100-
150423 17:13:51 [Note] InnoDB: FTS optimize thread exiting.
150423 17:13:51 [Note] InnoDB: Starting shutdown...
150423 17:13:51 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see http://kb.askmonty.org/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.1.4-MariaDB-wsrep-log
key_buffer_size=2147483648
read_buffer_size=8388608
max_used_connections=27
max_threads=402
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 11985247 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48000
150423 17:13:52 mysqld_safe Number of processes running now: 0
150423 17:13:52 mysqld_safe mysqld restarted

            Gliffy Diagrams

              Attachments

                Activity

                Ascending order - Click to sort in descending order
                Hide
                Permalink
                kolbe Kolbe Kegel added a comment -

                This occurred when 10.1.4 was running on tables created under a 10.1.3 server.

                Show
                kolbe Kolbe Kegel added a comment - This occurred when 10.1.4 was running on tables created under a 10.1.3 server.
                Hide
                Permalink
                kolbe Kolbe Kegel added a comment -

                This sigsegv-on-shutdown issue eventually was replaced by an assertion on crash recovery:

                InnoDB: Apply batch completed
InnoDB: In a MySQL replication slave the last master binlog file
InnoDB: position 0 401474766, file name mysql-bin.084373
InnoDB: Last MySQL binlog file position 0 71036301, file name ./mysql-bin.025331
InnoDB: Cleaning up trx with id 158495032187
2015-04-23 17:43:17 7f3479664820  InnoDB: Assertion failure in thread 139863351773216 in file trx0trx.cc line 467
InnoDB: Failing assertion: trx->update_undo == NULL
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
150423 17:43:17 [ERROR] mysqld got signal 6 ;
                Show
                kolbe Kolbe Kegel added a comment - This sigsegv-on-shutdown issue eventually was replaced by an assertion on crash recovery: InnoDB: Apply batch completed InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 0 401474766, file name mysql-bin.084373 InnoDB: Last MySQL binlog file position 0 71036301, file name ./mysql-bin.025331 InnoDB: Cleaning up trx with id 158495032187 2015-04-23 17:43:17 7f3479664820 InnoDB: Assertion failure in thread 139863351773216 in file trx0trx.cc line 467 InnoDB: Failing assertion: trx->update_undo == NULL InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 150423 17:43:17 [ERROR] mysqld got signal 6 ;
                Hide
                Permalink
                jplindst Jan Lindström added a comment -

                Can't repeat with this information, I would either need a database that crashes on crash recovery or more detailed steps how to repeat.

                Show
                jplindst Jan Lindström added a comment - Can't repeat with this information, I would either need a database that crashes on crash recovery or more detailed steps how to repeat.

                  People

                  • Assignee:
                    jplindst Jan Lindström
                    Reporter:
                    kolbe Kolbe Kegel
                  • Votes:
                    0 Vote for this issue
                    Watchers:
                    2 Start watching this issue

                    Dates

                    • Created:
                      Updated:
                      Resolved:

                      Agile