Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-8494

mysqldump ignore-table doesn't ignore mysql.slow_log or mysql.general_log

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 10.0.20
    • Fix Version/s: N/A
    • Component/s: Scripts & Clients
    • Labels:
      None
    • Environment:
      Debian Jessie X64

      Description

      When doing a full DB dump with mysqldump some tables are dumped regardles of their appearance in --ignore-table.

      /usr/bin/mysqldump --defaults-file=/etc/mysql/mydump.cnf -u<user> -p<password> --add-locks --extended-insert --add-locks --add-drop-database --add-drop-table --single-transaction --routines --triggers --all-databases --all-tablespaces --allow-keywords --complete-insert --create-options --events --dump-date>/root/db.sql

      cat /etc/mysql/mydump.cnf

      [mysqldump]
      ignore-table=mysql.general_log
      ignore-table=mysql.slow_log
      

      Still contains the two tables, other mysql.* tables can be ignored, worse in some cases it also contains the drop statement which causes an import to fail! (when dumping the whole server the export includes "/!40000 DROP DATABASE IF EXISTS `mysql`/;" which breaks if we attempt to import it).

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

              Hide
              GieltjE Michiel Hazelhof added a comment -

              mysqldump version: mysqldump Ver 10.15 Distrib 10.0.20-MariaDB, for debian-linux-gnu (x86_64)
              Command: /usr/bin/mysqldump --all-databases --add-drop-database --add-drop-table --ignore-table=mysql.general_log --ignore-table=mysql.slow_log --ignore-table=mysql.column_stats > test.sql

              Expected result: A dumb of all databases without general_log, slow_low and column_stats
              Result: Only column_stats is properly excluded.

              Show
              GieltjE Michiel Hazelhof added a comment - mysqldump version: mysqldump Ver 10.15 Distrib 10.0.20-MariaDB, for debian-linux-gnu (x86_64) Command: /usr/bin/mysqldump --all-databases --add-drop-database --add-drop-table --ignore-table=mysql.general_log --ignore-table=mysql.slow_log --ignore-table=mysql.column_stats > test.sql Expected result: A dumb of all databases without general_log, slow_low and column_stats Result: Only column_stats is properly excluded.
              Hide
              elenst Elena Stepanova added a comment - - edited

              As said above, I don't see this command either drop general_log and slow_log or dump their contents. It only adds CREATE TABLE IF NOT EXISTS statements which should not do any harm. This specifics of log tables was introduced in MySQL 5.5 and documented e.g. here: http://dev.mysql.com/doc/refman/5.5/en/mysqldump.html

              Before MySQL 5.5.25, mysqldump does not dump the general_log or slow_query_log tables for dumps of the mysql database. As of 5.5.25, the dump includes statements to recreate those tables so that they are not missing after reloading the dump file. Log table contents are not dumped.

              If you observe anything different, please attach the resulting dump file. If the essence of the request is that CREATE TABLE IF NOT EXISTS statements must not be there, again, I can convert it into a feature request, if you can provide an example of them being harmful.

              Show
              elenst Elena Stepanova added a comment - - edited As said above, I don't see this command either drop general_log and slow_log or dump their contents. It only adds CREATE TABLE IF NOT EXISTS statements which should not do any harm. This specifics of log tables was introduced in MySQL 5.5 and documented e.g. here: http://dev.mysql.com/doc/refman/5.5/en/mysqldump.html Before MySQL 5.5.25, mysqldump does not dump the general_log or slow_query_log tables for dumps of the mysql database. As of 5.5.25, the dump includes statements to recreate those tables so that they are not missing after reloading the dump file. Log table contents are not dumped. If you observe anything different, please attach the resulting dump file. If the essence of the request is that CREATE TABLE IF NOT EXISTS statements must not be there, again, I can convert it into a feature request, if you can provide an example of them being harmful.
              Hide
              GieltjE Michiel Hazelhof added a comment -

              The resulting dump file:

              -- MySQL dump 10.15  Distrib 10.0.20-MariaDB, for debian-linux-gnu (x86_64)
              --
              -- Host: localhost    Database: 
              -- ------------------------------------------------------
              -- Server version	10.0.20-MariaDB-1~jessie-log
              
              /*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
              /*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */;
              /*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */;
              /*!40101 SET NAMES utf8 */;
              /*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */;
              /*!40103 SET TIME_ZONE='+00:00' */;
              /*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */;
              /*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
              /*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
              /*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;
              
              --
              -- Current Database: `mysql`
              --
              
              /*!40000 DROP DATABASE IF EXISTS `mysql`*/;
              
              CREATE DATABASE /*!32312 IF NOT EXISTS*/ `mysql` /*!40100 DEFAULT CHARACTER SET utf8 */;
              
              USE `mysql`;
              
              <snip>
              
              --
              -- Table structure for table `general_log`
              --
              
              /*!40101 SET @saved_cs_client     = @@character_set_client */;
              /*!40101 SET character_set_client = utf8 */;
              CREATE TABLE IF NOT EXISTS `general_log` (
                `event_time` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
                `user_host` mediumtext NOT NULL,
                `thread_id` bigint(21) unsigned NOT NULL,
                `server_id` int(10) unsigned NOT NULL,
                `command_type` varchar(64) NOT NULL,
                `argument` mediumtext NOT NULL
              ) ENGINE=CSV DEFAULT CHARSET=utf8 COMMENT='General log';
              /*!40101 SET character_set_client = @saved_cs_client */;
              
              --
              -- Table structure for table `slow_log`
              --
              
              /*!40101 SET @saved_cs_client     = @@character_set_client */;
              /*!40101 SET character_set_client = utf8 */;
              CREATE TABLE IF NOT EXISTS `slow_log` (
                `start_time` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
                `user_host` mediumtext NOT NULL,
                `query_time` time(6) NOT NULL,
                `lock_time` time(6) NOT NULL,
                `rows_sent` int(11) NOT NULL,
                `rows_examined` int(11) NOT NULL,
                `db` varchar(512) NOT NULL,
                `last_insert_id` int(11) NOT NULL,
                `insert_id` int(11) NOT NULL,
                `server_id` int(10) unsigned NOT NULL,
                `sql_text` mediumtext NOT NULL,
                `thread_id` bigint(21) unsigned NOT NULL
              ) ENGINE=CSV DEFAULT CHARSET=utf8 COMMENT='Slow log';
              /*!40101 SET character_set_client = @saved_cs_client */;
              /*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */;
              
              /*!40101 SET SQL_MODE=@OLD_SQL_MODE */;
              /*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */;
              /*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */;
              /*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
              /*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */;
              /*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */;
              /*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */;
              
              -- Dump completed on 2015-07-19 17:34:14
              
              
              Show
              GieltjE Michiel Hazelhof added a comment - The resulting dump file: -- MySQL dump 10.15 Distrib 10.0.20-MariaDB, for debian-linux-gnu (x86_64) -- -- Host: localhost Database: -- ------------------------------------------------------ -- Server version 10.0.20-MariaDB-1~jessie-log /*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */; /*!40101 SET @OLD_CHARACTER_SET_RESULTS=@@CHARACTER_SET_RESULTS */; /*!40101 SET @OLD_COLLATION_CONNECTION=@@COLLATION_CONNECTION */; /*!40101 SET NAMES utf8 */; /*!40103 SET @OLD_TIME_ZONE=@@TIME_ZONE */; /*!40103 SET TIME_ZONE='+00:00' */; /*!40014 SET @OLD_UNIQUE_CHECKS=@@UNIQUE_CHECKS, UNIQUE_CHECKS=0 */; /*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */; /*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */; /*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */; -- -- Current Database: `mysql` -- /*!40000 DROP DATABASE IF EXISTS `mysql`*/; CREATE DATABASE /*!32312 IF NOT EXISTS*/ `mysql` /*!40100 DEFAULT CHARACTER SET utf8 */; USE `mysql`; <snip> -- -- Table structure for table `general_log` -- /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE IF NOT EXISTS `general_log` ( `event_time` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), `user_host` mediumtext NOT NULL, `thread_id` bigint(21) unsigned NOT NULL, `server_id` int(10) unsigned NOT NULL, `command_type` varchar(64) NOT NULL, `argument` mediumtext NOT NULL ) ENGINE=CSV DEFAULT CHARSET=utf8 COMMENT='General log'; /*!40101 SET character_set_client = @saved_cs_client */; -- -- Table structure for table `slow_log` -- /*!40101 SET @saved_cs_client = @@character_set_client */; /*!40101 SET character_set_client = utf8 */; CREATE TABLE IF NOT EXISTS `slow_log` ( `start_time` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6), `user_host` mediumtext NOT NULL, `query_time` time(6) NOT NULL, `lock_time` time(6) NOT NULL, `rows_sent` int(11) NOT NULL, `rows_examined` int(11) NOT NULL, `db` varchar(512) NOT NULL, `last_insert_id` int(11) NOT NULL, `insert_id` int(11) NOT NULL, `server_id` int(10) unsigned NOT NULL, `sql_text` mediumtext NOT NULL, `thread_id` bigint(21) unsigned NOT NULL ) ENGINE=CSV DEFAULT CHARSET=utf8 COMMENT='Slow log'; /*!40101 SET character_set_client = @saved_cs_client */; /*!40103 SET TIME_ZONE=@OLD_TIME_ZONE */; /*!40101 SET SQL_MODE=@OLD_SQL_MODE */; /*!40014 SET FOREIGN_KEY_CHECKS=@OLD_FOREIGN_KEY_CHECKS */; /*!40014 SET UNIQUE_CHECKS=@OLD_UNIQUE_CHECKS */; /*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */; /*!40101 SET CHARACTER_SET_RESULTS=@OLD_CHARACTER_SET_RESULTS */; /*!40101 SET COLLATION_CONNECTION=@OLD_COLLATION_CONNECTION */; /*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */; -- Dump completed on 2015-07-19 17:34:14
              Hide
              elenst Elena Stepanova added a comment -

              So, there are indeed only CREATE statements for general_log and slow_log. Why are they a problem?

              I do see a real problem which your scenario reveals – that mysql schema should not be dropped while log_output=TABLE and either general log or slow log is enabled. I think mysqldump should still drop the schema if it's told to do so, but it should turn off this kind of logging first, and restore the value afterwards. It's a little cumbersome to check, but it's doable.

              However, so far I do not see a problem with (not) ignoring mysql.slow_log or mysql.general_log. I'm not a big fan of the implicit logic for general/slow log tables introduced in MySQL while fixing http://bugs.mysql.com/bug.php?id=26121 and http://bugs.mysql.com/bug.php?id=45740, but it's already there, and making it even more complicated will only make things worse. Removing these CREATE TABLE IF NOT EXISTS statements would not make the ERROR 1580 you are getting to go away, but it would cause a problem later.

              Not writing the DROP DATABASE `mysql` statement, as you suggest, would solve the problem in your particular case, but it would actually reduce mysqldump functionality for no good reason, which is not desirable.

              Currently I see three reasonable ways to use ignore-table options, all of which mysqldump serves well enough.

              1) I have db1 schema, in which t1, t2, ... t10 are valuable tables, while tGarbage is just some trash which should never exist after loading the clean dump, no matter whether it existed before loading it or not. Then, I can create the dump with --add-drop-database --ignore-table=db1.tGarbage, and it will do exactly that.

              2) I have db2 schema, in which t1, t2, ... t10 are valuable tables, and tLocal is a very valuable table which should not be touched by the dump no matter what.Then, I can create the dump with --add-drop-table --ignore-table=db2.tLocal, and it will do exactly that.

              3) I have both db1 and db2, for db1 I want to do (1), and for db2 I want to do (2). Since the settings are essentially different, I can run mysqldump twice, separately for each schema, and get the desired result.

              If we do as you suggest and forbid dropping the schema if ignore-table is provided, in general case we will achieve nothing, but will lose the scenario (1).

              Show
              elenst Elena Stepanova added a comment - So, there are indeed only CREATE statements for general_log and slow_log. Why are they a problem? I do see a real problem which your scenario reveals – that mysql schema should not be dropped while log_output=TABLE and either general log or slow log is enabled. I think mysqldump should still drop the schema if it's told to do so, but it should turn off this kind of logging first, and restore the value afterwards. It's a little cumbersome to check, but it's doable. However, so far I do not see a problem with (not) ignoring mysql.slow_log or mysql.general_log. I'm not a big fan of the implicit logic for general/slow log tables introduced in MySQL while fixing http://bugs.mysql.com/bug.php?id=26121 and http://bugs.mysql.com/bug.php?id=45740 , but it's already there, and making it even more complicated will only make things worse. Removing these CREATE TABLE IF NOT EXISTS statements would not make the ERROR 1580 you are getting to go away, but it would cause a problem later. Not writing the DROP DATABASE `mysql` statement, as you suggest, would solve the problem in your particular case, but it would actually reduce mysqldump functionality for no good reason, which is not desirable. Currently I see three reasonable ways to use ignore-table options, all of which mysqldump serves well enough. 1) I have db1 schema, in which t1 , t2 , ... t10 are valuable tables, while tGarbage is just some trash which should never exist after loading the clean dump, no matter whether it existed before loading it or not. Then, I can create the dump with --add-drop-database --ignore-table=db1.tGarbage , and it will do exactly that. 2) I have db2 schema, in which t1 , t2 , ... t10 are valuable tables, and tLocal is a very valuable table which should not be touched by the dump no matter what.Then, I can create the dump with --add-drop-table --ignore-table=db2.tLocal , and it will do exactly that. 3) I have both db1 and db2 , for db1 I want to do (1), and for db2 I want to do (2). Since the settings are essentially different, I can run mysqldump twice, separately for each schema, and get the desired result. If we do as you suggest and forbid dropping the schema if ignore-table is provided, in general case we will achieve nothing, but will lose the scenario (1).
              Hide
              elenst Elena Stepanova added a comment -

              So, as described above, there is indeed an issue with mysqldump --add-drop-database when mysql schema is involved and TABLE logging is turned on; but it has already been filed as MDEV-4875 (upstream issue https://bugs.mysql.com/bug.php?id=69970). Otherwise, I don't see a bug in here, so I'm closing it as a duplicate of MDEV-4875. If you disagree, please comment, and if you make your case, the report will be re-opened.

              Show
              elenst Elena Stepanova added a comment - So, as described above, there is indeed an issue with mysqldump --add-drop-database when mysql schema is involved and TABLE logging is turned on; but it has already been filed as MDEV-4875 (upstream issue https://bugs.mysql.com/bug.php?id=69970 ). Otherwise, I don't see a bug in here, so I'm closing it as a duplicate of MDEV-4875 . If you disagree, please comment, and if you make your case, the report will be re-opened.

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  GieltjE Michiel Hazelhof
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: