Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 5.5.29-galera
    • Fix Version/s: 5.5.32-galera
    • Component/s: None
    • Environment:
      RHEL

      Description

      We are in the process of migrating to a MariaDB Galera cluster from a MySQL 5.1 Replication environment. The first step is to use one Galera node as a replication slave to the existing environment. This process was running smoothly until it started crashing with the attached stack trace. Restarting the node did not solve the issue. Stopping the slave process (skip-slave-start) will allow the node to start, but crashes again when we resume replication.

        Gliffy Diagrams

          Attachments

            Activity

            Hide
            seppo Seppo Jaakola added a comment -

            This looks quite similar as case reported here: https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1134892

            Show
            seppo Seppo Jaakola added a comment - This looks quite similar as case reported here: https://bugs.launchpad.net/percona-xtradb-cluster/+bug/1134892
            Hide
            seppo Seppo Jaakola added a comment -

            I can confirm that this issue is a duplicate of the one linked above.

            The problem happens when replicating a transaction which modifies tables with cascading foreign key constraints. Can you pinpoint which parent-child table pair were part of this crashing transaction? If so, please show the table definitions for inspection.

            Also, attach your my.cnf

            Show
            seppo Seppo Jaakola added a comment - I can confirm that this issue is a duplicate of the one linked above. The problem happens when replicating a transaction which modifies tables with cascading foreign key constraints. Can you pinpoint which parent-child table pair were part of this crashing transaction? If so, please show the table definitions for inspection. Also, attach your my.cnf
            Hide
            seppo Seppo Jaakola added a comment -

            The revision used in this case is #3390 in mariadb-galera branch and is tagged with: mariadb-galera-5.5.29.
            Corresponding upstream revision in wsrep-5.5-23 is: #3853

            Show
            seppo Seppo Jaakola added a comment - The revision used in this case is #3390 in mariadb-galera branch and is tagged with: mariadb-galera-5.5.29. Corresponding upstream revision in wsrep-5.5-23 is: #3853
            Hide
            wfong Will Fong added a comment -

            Hello. Attached is the schema for the transaction in question:

            MariaDB [ped]> show create table 1\G

                                                                • 1. row ***************************
                                                                  Table: 1
                                                                  Create Table: CREATE TABLE `1` (
                                                                  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
                                                                  `user_id` int(10) unsigned NOT NULL,
                                                                  PRIMARY KEY (`id`),
                                                                  KEY `user_id` (`user_id`),
                                                                  CONSTRAINT `2_FK` FOREIGN KEY (`user_id`) REFERENCES `2` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
                                                                  ) ENGINE=InnoDB AUTO_INCREMENT=2000986 DEFAULT CHARSET=utf8
                                                                  1 row in set (0.00 sec)

            MariaDB [ped]> show create table 2\G

                                                                • 1. row ***************************
                                                                  Table: 2
                                                                  Create Table: CREATE TABLE `2` (
                                                                  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
                                                                  `first_name` varchar(45) NOT NULL,
                                                                  `last_name` varchar(45) NOT NULL,
                                                                  PRIMARY KEY (`id`),
                                                                  KEY `part_of_firstname` (`first_name`(3)),
                                                                  KEY `part_of_lastname` (`last_name`(3))
                                                                  ) ENGINE=InnoDB AUTO_INCREMENT=3134511 DEFAULT CHARSET=utf8
                                                                  1 row in set (0.00 sec)

            MariaDB [ped]> show create table 3\G

                                                                • 1. row ***************************
                                                                  Table: 3
                                                                  Create Table: CREATE TABLE `3` (
                                                                  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
                                                                  `user_id` int(10) unsigned NOT NULL,
                                                                  `time` datetime NOT NULL DEFAULT '1970-01-01 00:00:00',
                                                                  `4` int(10) unsigned DEFAULT NULL,
                                                                  `5` int(10) unsigned DEFAULT NULL,
                                                                  PRIMARY KEY (`id`),
                                                                  KEY `user_id` (`user_id`,`time`),
                                                                  KEY `4_FK` (`4`),
                                                                  KEY `5_FK` (`5`),
                                                                  CONSTRAINT `1_FK` FOREIGN KEY (`5`) REFERENCES `1` (`id`) ON DELETE SET NULL ON UPDATE CASCADE,
                                                                  CONSTRAINT `5_FK` FOREIGN KEY (`4`) REFERENCES `6` (`id`) ON DELETE SET NULL ON UPDATE CASCADE,
                                                                  CONSTRAINT `2_FK` FOREIGN KEY (`user_id`) REFERENCES `2` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
                                                                  ) ENGINE=InnoDB AUTO_INCREMENT=182241820 DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED
                                                                  1 row in set (0.00 sec)

            MariaDB [ped]> show create table 6\G

                                                                • 1. row ***************************
                                                                  Table: 6
                                                                  Create Table: CREATE TABLE `6` (
                                                                  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
                                                                  `user_id` int(10) unsigned NOT NULL,
                                                                  `short_name` varchar(15) DEFAULT NULL,
                                                                  PRIMARY KEY (`id`),
                                                                  UNIQUE KEY `user_short_name_UNIQUE` (`user_id`,`short_name`),
                                                                  KEY `2_FK` (`user_id`),
                                                                  CONSTRAINT `2_FK` FOREIGN KEY (`user_id`) REFERENCES `2` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
                                                                  ) ENGINE=InnoDB AUTO_INCREMENT=7550174 DEFAULT CHARSET=utf8
                                                                  1 row in set (0.00 sec)

            The transaction contains row-based DELETE statements on tables 1 and 3.

            I'll grab the my.cnf file as soon as possible.

            Thanks,
            -will

            Show
            wfong Will Fong added a comment - Hello. Attached is the schema for the transaction in question: MariaDB [ped] > show create table 1\G 1. row *************************** Table: 1 Create Table: CREATE TABLE `1` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `user_id` int(10) unsigned NOT NULL, PRIMARY KEY (`id`), KEY `user_id` (`user_id`), CONSTRAINT `2_FK` FOREIGN KEY (`user_id`) REFERENCES `2` (`id`) ON DELETE CASCADE ON UPDATE CASCADE ) ENGINE=InnoDB AUTO_INCREMENT=2000986 DEFAULT CHARSET=utf8 1 row in set (0.00 sec) MariaDB [ped] > show create table 2\G 1. row *************************** Table: 2 Create Table: CREATE TABLE `2` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `first_name` varchar(45) NOT NULL, `last_name` varchar(45) NOT NULL, PRIMARY KEY (`id`), KEY `part_of_firstname` (`first_name`(3)), KEY `part_of_lastname` (`last_name`(3)) ) ENGINE=InnoDB AUTO_INCREMENT=3134511 DEFAULT CHARSET=utf8 1 row in set (0.00 sec) MariaDB [ped] > show create table 3\G 1. row *************************** Table: 3 Create Table: CREATE TABLE `3` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `user_id` int(10) unsigned NOT NULL, `time` datetime NOT NULL DEFAULT '1970-01-01 00:00:00', `4` int(10) unsigned DEFAULT NULL, `5` int(10) unsigned DEFAULT NULL, PRIMARY KEY (`id`), KEY `user_id` (`user_id`,`time`), KEY `4_FK` (`4`), KEY `5_FK` (`5`), CONSTRAINT `1_FK` FOREIGN KEY (`5`) REFERENCES `1` (`id`) ON DELETE SET NULL ON UPDATE CASCADE, CONSTRAINT `5_FK` FOREIGN KEY (`4`) REFERENCES `6` (`id`) ON DELETE SET NULL ON UPDATE CASCADE, CONSTRAINT `2_FK` FOREIGN KEY (`user_id`) REFERENCES `2` (`id`) ON DELETE CASCADE ON UPDATE CASCADE ) ENGINE=InnoDB AUTO_INCREMENT=182241820 DEFAULT CHARSET=utf8 ROW_FORMAT=COMPRESSED 1 row in set (0.00 sec) MariaDB [ped] > show create table 6\G 1. row *************************** Table: 6 Create Table: CREATE TABLE `6` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `user_id` int(10) unsigned NOT NULL, `short_name` varchar(15) DEFAULT NULL, PRIMARY KEY (`id`), UNIQUE KEY `user_short_name_UNIQUE` (`user_id`,`short_name`), KEY `2_FK` (`user_id`), CONSTRAINT `2_FK` FOREIGN KEY (`user_id`) REFERENCES `2` (`id`) ON DELETE CASCADE ON UPDATE CASCADE ) ENGINE=InnoDB AUTO_INCREMENT=7550174 DEFAULT CHARSET=utf8 1 row in set (0.00 sec) The transaction contains row-based DELETE statements on tables 1 and 3. I'll grab the my.cnf file as soon as possible. Thanks, -will
            Hide
            seppo Seppo Jaakola added a comment -

            I was able to reproduce according to suggested scenario:

            1. Start standard replication from MySQL master to first Galera node.
            2. Command STOP SLAVE on the Galera node (slave).
            3. Execute ALTER TABLE '3' ROW_FORMAT=compressed on the slave.
            4. Command START SLAVE on the Galera node (slave)
            => Galera node (slave) crashes soon after

            This happened with 5.5.29 release, but not anymore with 5.5.31 RC. I have a targeted fix for 5.5.29, which I probably merge for 5.5.31 during today.

            Show
            seppo Seppo Jaakola added a comment - I was able to reproduce according to suggested scenario: 1. Start standard replication from MySQL master to first Galera node. 2. Command STOP SLAVE on the Galera node (slave). 3. Execute ALTER TABLE '3' ROW_FORMAT=compressed on the slave. 4. Command START SLAVE on the Galera node (slave) => Galera node (slave) crashes soon after This happened with 5.5.29 release, but not anymore with 5.5.31 RC. I have a targeted fix for 5.5.29, which I probably merge for 5.5.31 during today.
            Hide
            seppo Seppo Jaakola added a comment -

            Merged last fixes related to this issue in revision: http://bazaar.launchpad.net/~maria-captains/maria/maria-5.5-galera/revision/3400

            Show
            seppo Seppo Jaakola added a comment - Merged last fixes related to this issue in revision: http://bazaar.launchpad.net/~maria-captains/maria/maria-5.5-galera/revision/3400
            Hide
            ratzpo Rasmus Johansson added a comment -

            Reopened to be able to move to correct fix version

            Show
            ratzpo Rasmus Johansson added a comment - Reopened to be able to move to correct fix version

              People

              • Assignee:
                seppo Seppo Jaakola
                Reporter:
                wfong Will Fong
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 days
                  2d