Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 5.5.40
    • Fix Version/s: N/A
    • Labels:
      None
    • Environment:
      Debian Wheezy 64bit, 14 GB RAM

      Description

      I've got two crashes in short period of time, attaching logs. Don't know what else to add.

        Gliffy Diagrams

          Attachments

          1. my.cnf
            6 kB
          2. mysql1.log
            5 kB
          3. mysql2.log
            9 kB

            Activity

            Hide
            elenst Elena Stepanova added a comment -

            Hi,

            First, could you please include a bigger part of the error logs, not starting from 'signal 11', but preferably starting from the server startup and up to the restart? (In your case the first message after startup is "[Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release.").

            Second, how high is the load on the server? Would it be possible to turn on general logging for one session, to find which query it actually crashes on?

            Third, while the first stack trace is non-informative at all, the second one shows that the crash happened upon table creation. Does table creation happen often on that server? Do you, by any chance, know what and how was created during this time? There were 68 threads running at the moment, so I understand it's unlikely that you know what was executed at the moment, but just in case...

            Thanks.

            Show
            elenst Elena Stepanova added a comment - Hi, First, could you please include a bigger part of the error logs, not starting from 'signal 11', but preferably starting from the server startup and up to the restart? (In your case the first message after startup is " [Warning] 'THREAD_CONCURRENCY' is deprecated and will be removed in a future release."). Second, how high is the load on the server? Would it be possible to turn on general logging for one session, to find which query it actually crashes on? Third, while the first stack trace is non-informative at all, the second one shows that the crash happened upon table creation. Does table creation happen often on that server? Do you, by any chance, know what and how was created during this time? There were 68 threads running at the moment, so I understand it's unlikely that you know what was executed at the moment, but just in case... Thanks.
            Hide
            azurit azurit added a comment -

            1.) There's nothing else in the logs except notes that IPs cannot be resolved.

            2.) About 70 qps.

            3.) After the first crash i found this in logs:

            Nov 13 11:43:09 server01 mysqld: 141113 11:43:09 InnoDB: Error: table `<censored>`.`serial_menu` does not exist in the InnoDB internal
            Nov 13 11:43:09 server01 mysqld: InnoDB: data dictionary though MySQL is trying to drop it.
            Nov 13 11:43:09 server01 mysqld: InnoDB: Have you copied the .frm file of the table to the
            Nov 13 11:43:09 server01 mysqld: InnoDB: MySQL database directory from another database?
            Nov 13 11:43:09 server01 mysqld: InnoDB: You can look for further help from
            Nov 13 11:43:09 server01 mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting.html
            Nov 13 11:43:09 server01 mysqld: 141113 11:43:09 InnoDB: Error creating file './<censored>/serial_menu.ibd'.
            Nov 13 11:43:09 server01 mysqld: 141113 11:43:09 InnoDB: Operating system error number 17 in a file operation.
            Nov 13 11:43:09 server01 mysqld: InnoDB: Error number 17 means 'File exists'.
            Nov 13 11:43:09 server01 mysqld: InnoDB: Some operating system error numbers are described at
            Nov 13 11:43:09 server01 mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html
            Nov 13 11:43:09 server01 mysqld: InnoDB: The file already exists though the corresponding table did not
            Nov 13 11:43:09 server01 mysqld: InnoDB: exist in the InnoDB data dictionary. Have you moved InnoDB
            Nov 13 11:43:09 server01 mysqld: InnoDB: .ibd files around without using the SQL commands
            Nov 13 11:43:09 server01 mysqld: InnoDB: DISCARD TABLESPACE and IMPORT TABLESPACE, or did
            Nov 13 11:43:09 server01 mysqld: InnoDB: mysqld crash in the middle of CREATE TABLE? You can
            Nov 13 11:43:09 server01 mysqld: InnoDB: resolve the problem by removing the file './<censored>/serial_menu.ibd'
            Nov 13 11:43:09 server01 mysqld: InnoDB: under the 'datadir' of MySQL.

            This was probably the table which cased the crash. This is the current table structure (but i cannot tell if it's the same as it was while server crashed):
            CREATE TABLE `serial_menu` (
            `id` int(11) NOT NULL AUTO_INCREMENT,
            `serial_id` int(11) NOT NULL,
            `name` varchar(30) NOT NULL,
            `slug` varchar(50) NOT NULL,
            `page_id` int(10) NOT NULL,
            `visible` int(1) NOT NULL DEFAULT '1',
            `default` int(1) NOT NULL DEFAULT '0',
            `order` int(5) NOT NULL,
            PRIMARY KEY (`id`),
            KEY `serial_id` (`serial_id`),
            KEY `page_id` (`page_id`)
            ) ENGINE=MyISAM DEFAULT CHARSET=utf8

            Here is also other structure of a table from different database but with, probably, the same tables:

            CREATE TABLE `serial_menu` (
            `id` int(11) NOT NULL AUTO_INCREMENT,
            `serial_id` int(11) NOT NULL,
            `name` varchar(30) NOT NULL,
            `slug` varchar(50) NOT NULL,
            `page_id` int(10) NOT NULL,
            `visible` int(1) NOT NULL DEFAULT '1',
            `default` int(1) NOT NULL DEFAULT '0',
            `order` int(5) NOT NULL,
            PRIMARY KEY (`id`),
            KEY `serial_id` (`serial_id`),
            KEY `page_id` (`page_id`),
            CONSTRAINT `serial_menu_ibfk_1` FOREIGN KEY (`serial_id`) REFERENCES `serials` (`id`) ON DELETE CASCADE,
            CONSTRAINT `serial_menu_ibfk_2` FOREIGN KEY (`page_id`) REFERENCES `serial_pages` (`id`) ON DELETE CASCADE
            ) ENGINE=InnoDB DEFAULT CHARSET=utf8

            Show
            azurit azurit added a comment - 1.) There's nothing else in the logs except notes that IPs cannot be resolved. 2.) About 70 qps. 3.) After the first crash i found this in logs: Nov 13 11:43:09 server01 mysqld: 141113 11:43:09 InnoDB: Error: table `<censored>`.`serial_menu` does not exist in the InnoDB internal Nov 13 11:43:09 server01 mysqld: InnoDB: data dictionary though MySQL is trying to drop it. Nov 13 11:43:09 server01 mysqld: InnoDB: Have you copied the .frm file of the table to the Nov 13 11:43:09 server01 mysqld: InnoDB: MySQL database directory from another database? Nov 13 11:43:09 server01 mysqld: InnoDB: You can look for further help from Nov 13 11:43:09 server01 mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting.html Nov 13 11:43:09 server01 mysqld: 141113 11:43:09 InnoDB: Error creating file './<censored>/serial_menu.ibd'. Nov 13 11:43:09 server01 mysqld: 141113 11:43:09 InnoDB: Operating system error number 17 in a file operation. Nov 13 11:43:09 server01 mysqld: InnoDB: Error number 17 means 'File exists'. Nov 13 11:43:09 server01 mysqld: InnoDB: Some operating system error numbers are described at Nov 13 11:43:09 server01 mysqld: InnoDB: http://dev.mysql.com/doc/refman/5.5/en/operating-system-error-codes.html Nov 13 11:43:09 server01 mysqld: InnoDB: The file already exists though the corresponding table did not Nov 13 11:43:09 server01 mysqld: InnoDB: exist in the InnoDB data dictionary. Have you moved InnoDB Nov 13 11:43:09 server01 mysqld: InnoDB: .ibd files around without using the SQL commands Nov 13 11:43:09 server01 mysqld: InnoDB: DISCARD TABLESPACE and IMPORT TABLESPACE, or did Nov 13 11:43:09 server01 mysqld: InnoDB: mysqld crash in the middle of CREATE TABLE? You can Nov 13 11:43:09 server01 mysqld: InnoDB: resolve the problem by removing the file './<censored>/serial_menu.ibd' Nov 13 11:43:09 server01 mysqld: InnoDB: under the 'datadir' of MySQL. This was probably the table which cased the crash. This is the current table structure (but i cannot tell if it's the same as it was while server crashed): CREATE TABLE `serial_menu` ( `id` int(11) NOT NULL AUTO_INCREMENT, `serial_id` int(11) NOT NULL, `name` varchar(30) NOT NULL, `slug` varchar(50) NOT NULL, `page_id` int(10) NOT NULL, `visible` int(1) NOT NULL DEFAULT '1', `default` int(1) NOT NULL DEFAULT '0', `order` int(5) NOT NULL, PRIMARY KEY (`id`), KEY `serial_id` (`serial_id`), KEY `page_id` (`page_id`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8 Here is also other structure of a table from different database but with, probably, the same tables: CREATE TABLE `serial_menu` ( `id` int(11) NOT NULL AUTO_INCREMENT, `serial_id` int(11) NOT NULL, `name` varchar(30) NOT NULL, `slug` varchar(50) NOT NULL, `page_id` int(10) NOT NULL, `visible` int(1) NOT NULL DEFAULT '1', `default` int(1) NOT NULL DEFAULT '0', `order` int(5) NOT NULL, PRIMARY KEY (`id`), KEY `serial_id` (`serial_id`), KEY `page_id` (`page_id`), CONSTRAINT `serial_menu_ibfk_1` FOREIGN KEY (`serial_id`) REFERENCES `serials` (`id`) ON DELETE CASCADE, CONSTRAINT `serial_menu_ibfk_2` FOREIGN KEY (`page_id`) REFERENCES `serial_pages` (`id`) ON DELETE CASCADE ) ENGINE=InnoDB DEFAULT CHARSET=utf8
            Hide
            elenst Elena Stepanova added a comment -

            Please also attach your cnf file(s).

            I assume your answer about the load implies you cannot turn on the general log.
            Do you happen to have any kind of monitoring, e.g. periodic processlist output?

            Show
            elenst Elena Stepanova added a comment - Please also attach your cnf file(s). I assume your answer about the load implies you cannot turn on the general log. Do you happen to have any kind of monitoring, e.g. periodic processlist output?
            Hide
            azurit azurit added a comment - - edited

            Attaching.

            I activated logging of that user via audit plugin but it didn't crash yet.

            Show
            azurit azurit added a comment - - edited Attaching. I activated logging of that user via audit plugin but it didn't crash yet.
            Hide
            elenst Elena Stepanova added a comment -

            Still can't reproduce it even with the server options from the cnf file, lets hope the audit plugin will show us something.

            Show
            elenst Elena Stepanova added a comment - Still can't reproduce it even with the server options from the cnf file, lets hope the audit plugin will show us something.
            Hide
            elenst Elena Stepanova added a comment -

            Please comment to re-open if you get new information. Without it, the closest that we have among known bugs is MDEV-7049. The visible part of the 2nd stack trace here is pretty much the same as in MDEV-7049 (except for mysql_create_table_no_lock, but that's because the test case there creates a temporary table; if it's changed to a normal table, there will be mysql_create_table). However, MDEV-7049 is only reproducible on a debug build, I couldn't get either a crash or valgrind warnings on a release build, so probably it's not exactly the same problem.

            Show
            elenst Elena Stepanova added a comment - Please comment to re-open if you get new information. Without it, the closest that we have among known bugs is MDEV-7049 . The visible part of the 2nd stack trace here is pretty much the same as in MDEV-7049 (except for mysql_create_table_no_lock, but that's because the test case there creates a temporary table; if it's changed to a normal table, there will be mysql_create_table). However, MDEV-7049 is only reproducible on a debug build, I couldn't get either a crash or valgrind warnings on a release build, so probably it's not exactly the same problem.

              People

              • Assignee:
                Unassigned
                Reporter:
                azurit azurit
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved: