Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-6952

query in relation with replication freeze

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: 10.0.14
    • Fix Version/s: N/A
    • Labels:
    • Environment:
      unbuntu 12.04

      Description

      Hello,

      I made a cleaner to supress data from a database, to test the script I put it at max and I got a freeze on a drop of temporary table (on slave).

       DROP TEMPORARY TABLE IF EXISTS `PRODUCTION`.`DELETE_PROD_PRODUITS_BABEL` /* generated by server */
      

      It's the 3rd time I meet with this problem.

      show full processlist;
      +------+------------------+------------------+------------+---------+--------+----------------------------------+----------------------------------------------------------------------------------------------------+----------+
      | Id   | User             | Host             | db         | Command | Time   | State                            | Info                                                                                               | Progress |
      +------+------------------+------------------+------------+---------+--------+----------------------------------+----------------------------------------------------------------------------------------------------+----------+
      |    2 | event_scheduler  | localhost        | NULL       | Daemon  | 280218 | Waiting on empty queue           | NULL                                                                                               |    0.000 |
      |  228 | system user      |                  | NULL       | Connect | 274596 | Waiting for master to send event | NULL                                                                                               |    0.000 |
      |  229 | system user      |                  | PRODUCTION | Connect |  83613 | init                             | DROP TEMPORARY TABLE IF EXISTS `PRODUCTION`.`DELETE_PROD_PRODUITS_BABEL` /* generated by server */ |    0.000 |
      | 7908 | dba              | 10.10.16.7:32875 | mysql      | Query   |  83610 | init                             | SHOW MASTER STATUS                                                                                 |    0.000 |
      | 7910 | debian-sys-maint | localhost        | NULL       | Query   |  30808 | init                             | flush logs                                                                                         |    0.000 |
      | 7913 | dba              | 10.10.16.7:35808 | mysql      | Query   |  13804 | init                             | SHOW MASTER STATUS                                                                                 |    0.000 |
      | 7914 | dba              | 10.10.16.7:35826 | mysql      | Query   |  13751 | init                             | SHOW MASTER STATUS                                                                                 |    0.000 |
      | 7915 | dba              | 10.10.16.7:35847 | mysql      | Query   |  13287 | init                             | SHOW MASTER LOGS                                                                                   |    0.000 |
      | 7917 | dba              | 10.10.16.9:40147 | NULL       | Killed  |  10819 | Killing slave                    | stop slave                                                                                         |    0.000 |
      | 7922 | dba              | 10.10.16.9:40176 | NULL       | Killed  |     30 | init                             | SHOW MASTER LOGS                                                                                   |    0.000 |
      | 7925 | dba              | 10.10.16.9:40179 | NULL       | Query   |      0 | init                             | show full processlist                                                                              |    0.000 |
      +------+------------------+------------------+------------+---------+--------+----------------------------------+----------------------------------------------------------------------------------------------------+----------+
      11 rows in set (0.00 sec)
      

      and strange things : (empty)

       show engine innodb status;
      +--------+------+--------+
      | Type   | Name | Status |
      +--------+------+--------+
      | InnoDB |      |        |
      +--------+------+--------+
      1 row in set (0.00 sec)
      

      After it all tools / function in relation with replication are totally frozen.

        Gliffy Diagrams

          Attachments

            Activity

            Hide
            elenst Elena Stepanova added a comment -

            Hi,

            By saying that it's the 3rd time you've seen the problem, do you mean that you have a reliable way to reproduce it, or that it happens sporadically and has happened 3 times so far?

            If you have a way to reproduce it, can you share it? The initial data dump, configuration, and the set of statements to execute on the master.
            The datadump can be uploaded to our FTP at ftp.askmonty.org/private so that only MariaDB developers have access to it.

            If it's sporadic, please attach configuration files from master and slave, and provide some more details:

            • what is your replication configuration/topology?
            • what is SHOW CREATE for this temporary table in question?
            • does it exist on the slave when the DROP is attempted?
            • how much data does it contain?
            • did you check your system logs for disk and other errors?
            • does the slave error log contain any errors/warnings? (please attach it too).

            Thanks.

            Show
            elenst Elena Stepanova added a comment - Hi, By saying that it's the 3rd time you've seen the problem, do you mean that you have a reliable way to reproduce it, or that it happens sporadically and has happened 3 times so far? If you have a way to reproduce it, can you share it? The initial data dump, configuration, and the set of statements to execute on the master. The datadump can be uploaded to our FTP at ftp.askmonty.org/private so that only MariaDB developers have access to it. If it's sporadic, please attach configuration files from master and slave, and provide some more details: what is your replication configuration/topology? what is SHOW CREATE for this temporary table in question? does it exist on the slave when the DROP is attempted? how much data does it contain? did you check your system logs for disk and other errors? does the slave error log contain any errors/warnings? (please attach it too). Thanks.
            Hide
            Aurelien_LEQUOY Aurélien LEQUOY added a comment - - edited

            well it always happen after i can't predict at witch time exactly.

            Got the same problem on 2 differents servers.

            The 2 first times it seem to happear with flush logs (debian-sys-maint).

            I relaunch the process with more log & trace.

            Show
            Aurelien_LEQUOY Aurélien LEQUOY added a comment - - edited well it always happen after i can't predict at witch time exactly. Got the same problem on 2 differents servers. The 2 first times it seem to happear with flush logs (debian-sys-maint). I relaunch the process with more log & trace.
            Hide
            Aurelien_LEQUOY Aurélien LEQUOY added a comment -

            schema on my monitoring tool : https://pbs.twimg.com/media/B09--cBIAAAcoBt.png

            M = Binlog
            S = Replicate

            all servers have binlog + log_slave_update

            this script is executed with SET @@skip_replication = ON on dba-test-sa-03
            and dba-test-sa-02 have replicate events marked for skip = filter_on_slave

            Show
            Aurelien_LEQUOY Aurélien LEQUOY added a comment - schema on my monitoring tool : https://pbs.twimg.com/media/B09--cBIAAAcoBt.png M = Binlog S = Replicate all servers have binlog + log_slave_update this script is executed with SET @@skip_replication = ON on dba-test-sa-03 and dba-test-sa-02 have replicate events marked for skip = filter_on_slave
            Hide
            elenst Elena Stepanova added a comment - - edited

            So, it does not always happen with the same DROP TABLE statement?
            If this DROP TABLE is always involved, could you please reply to other questions from my previous comment?
            If it happens on totally different queries, then the questions are different, first of all – did you check the disk space where the slave's @@datadir and @@tmpdir are located?
            also, what does the error log of this slave say?

            Regarding the empty InnoDB status – what kind of GUI do you use? I read about a similar problem with SQLYog, although not a recent one. Did you try to run it via the regular MySQL client? (It will probably hang, but it would be useful to know for sure).

            Show
            elenst Elena Stepanova added a comment - - edited So, it does not always happen with the same DROP TABLE statement? If this DROP TABLE is always involved, could you please reply to other questions from my previous comment? If it happens on totally different queries, then the questions are different, first of all – did you check the disk space where the slave's @@datadir and @@tmpdir are located? also, what does the error log of this slave say? Regarding the empty InnoDB status – what kind of GUI do you use? I read about a similar problem with SQLYog, although not a recent one. Did you try to run it via the regular MySQL client? (It will probably hang, but it would be useful to know for sure).
            Hide
            Aurelien_LEQUOY Aurélien LEQUOY added a comment -

            Can close this bug, I identified the problem and was an hardware issue

            Show
            Aurelien_LEQUOY Aurélien LEQUOY added a comment - Can close this bug, I identified the problem and was an hardware issue
            Hide
            elenst Elena Stepanova added a comment -

            We don't have a proper resolution value for an environmental problem, so I'm closing it as "Cannot Reproduce".
            If it turns out that the problem still exist after fixing the hardware, please comment to re-open.

            Show
            elenst Elena Stepanova added a comment - We don't have a proper resolution value for an environmental problem, so I'm closing it as "Cannot Reproduce". If it turns out that the problem still exist after fixing the hardware, please comment to re-open.

              People

              • Assignee:
                Unassigned
                Reporter:
                Aurelien_LEQUOY Aurélien LEQUOY
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 1 day, 4 hours
                  1d 4h
                  Remaining:
                  Remaining Estimate - 1 day, 4 hours
                  1d 4h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified