Uploaded image for project: 'MariaDB Server'
  1. MariaDB Server
  2. MDEV-2964

LP:944422 - mysql_upgrade destroys Maria tables?

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Upgrading tables from 5.2 to 5.3.5 with mysql_upgrade seems to cause MySQL to zerofill tables...

      [Note] Zerofilling moved table: '.\database\technikervermerke'

      which then seems to destroy some tables. At least when you run a check, it states:

      Found row with transaction id 808 when max transaction id according to maria_control_file is 84

      This issue might be linked to:
      http://askmonty.org/worklog/Maria-BackLog/?tid=126
      http://bugs.mysql.com/bug.php?id=42379 (status: "not supported")

      Attached is an archive with the 5.2-state as well as 5.3-state table `technikervermerke`.

      If the upgrade causes corrupt tables, this bug should be fixed. If it is an erroneus message, it should be disabled.

        Gliffy Diagrams

          Attachments

            Activity

            Hide
            nbrnhardt nbrnhardt added a comment -

            Re: mysql_upgrade destroys Maria tables?
            There is really something going wrong when Maria table logs aren't identical to what they are expected to be. So this is not limited to an uprade, I think.

            Following scenario to set up / sync new slave servers (5.2.10):

            • stop master
            • delete the log files (that is, we have a lot of aria_xxx files)
            • copy the data dir to the slaves
            • restart master, and start the slaves servers.

            On the MASTER following happens:

            As soon as the first user uses a a table, MariaDB seems to notice some differences (between table and (non-existing) log?), and issues in the error log:

            120327 7:04:52 [Note] Zerofilling moved table: '.\dbv\belegliste'

            120327 7:04:54 [Warning] Checking table: '.\dbv\belegliste'

            Not only limited to this table, but here it does so here hundreds of times during the next hour.

            Then the server got restarted and an REPAIR TABLE belegliste was issued:

            120327 8:10:23 [Warning] Checking table: '.\dbv\belegliste'

            120327 8:10:23 [Warning] Recovering table: '.\dbv\belegliste'

            120327 8:10:23 [Note] Retrying repair of: '.\dbv\belegliste' with keycache

            120327 8:10:23 [ERROR] Couldn't repair table: dbv.belegliste

            After another restart, the error log tells:

            120327 8:11:10 [ERROR] MySQL: Table '.\dbv\belegliste' is marked as crashed and last (automatic?) repair failed

            On another REPAIR TABLE, the result is:

            120327 8:11:37 [Note] Found 6004 of 0 rows when repairing '.\dbv\belegliste'

            After issuing a REPAIR TABLE to all Maria tables, the database is working fine again.

            On the SLAVE, `belegliste` gives a "Got error: 176 when reading datafile". Couldn't find anything on Google.

            It seems that on a read or write of a MariaDB table, the database server compares the table to the log file that isn't there anymore. Is there any sort of counter that tells MariaDB that a table is 'corrupt'? More specificly, is there a counter stored in the table to sync the position of the log files? If the log files change or get deleted, this counter should automatically re-synced with the new logs.

            Any ideas?

            If you need the table `belegliste`, I would upload it on FTP.

            Show
            nbrnhardt nbrnhardt added a comment - Re: mysql_upgrade destroys Maria tables? There is really something going wrong when Maria table logs aren't identical to what they are expected to be. So this is not limited to an uprade, I think. Following scenario to set up / sync new slave servers (5.2.10): stop master delete the log files (that is, we have a lot of aria_xxx files) copy the data dir to the slaves restart master, and start the slaves servers. On the MASTER following happens: As soon as the first user uses a a table, MariaDB seems to notice some differences (between table and (non-existing) log?), and issues in the error log: 120327 7:04:52 [Note] Zerofilling moved table: '.\dbv\belegliste' 120327 7:04:54 [Warning] Checking table: '.\dbv\belegliste' Not only limited to this table, but here it does so here hundreds of times during the next hour. Then the server got restarted and an REPAIR TABLE belegliste was issued: 120327 8:10:23 [Warning] Checking table: '.\dbv\belegliste' 120327 8:10:23 [Warning] Recovering table: '.\dbv\belegliste' 120327 8:10:23 [Note] Retrying repair of: '.\dbv\belegliste' with keycache 120327 8:10:23 [ERROR] Couldn't repair table: dbv.belegliste After another restart, the error log tells: 120327 8:11:10 [ERROR] MySQL: Table '.\dbv\belegliste' is marked as crashed and last (automatic?) repair failed On another REPAIR TABLE, the result is: 120327 8:11:37 [Note] Found 6004 of 0 rows when repairing '.\dbv\belegliste' After issuing a REPAIR TABLE to all Maria tables, the database is working fine again. On the SLAVE, `belegliste` gives a "Got error: 176 when reading datafile". Couldn't find anything on Google. It seems that on a read or write of a MariaDB table, the database server compares the table to the log file that isn't there anymore. Is there any sort of counter that tells MariaDB that a table is 'corrupt'? More specificly, is there a counter stored in the table to sync the position of the log files? If the log files change or get deleted, this counter should automatically re-synced with the new logs. Any ideas? If you need the table `belegliste`, I would upload it on FTP.
            Hide
            monty Michael Widenius added a comment -

            Re: mysql_upgrade destroys Maria tables?
            The reason you get the 'zerofilling table' is that you removed the aria_log_control file. This is something one should not normally do as it contains some critical information for Aria.

            If the aria_log_control file is removed, the Aria engine is re-initialized and transaction id's are starting from 1 again.
            The effect of this is that all Aria data files are regarded as 'moved' from another server and have to be zerofilled to be usable.

            Zerofilling means that all transaction id's are removed from the table and that all unused data is set to 0.
            The first part is needed for the table to work, the second is mainly useful if one wants to compress and archive the table.
            No real data is removed from the table. I will document this in the knowledgebase shortly.

            In no case should one get the error: Found row with transaction id 808; This is a bug and we will fix it as soon as I get a repeatable test case for it (Elena said she can repeat it and will produce one for me).

            Show
            monty Michael Widenius added a comment - Re: mysql_upgrade destroys Maria tables? The reason you get the 'zerofilling table' is that you removed the aria_log_control file. This is something one should not normally do as it contains some critical information for Aria. If the aria_log_control file is removed, the Aria engine is re-initialized and transaction id's are starting from 1 again. The effect of this is that all Aria data files are regarded as 'moved' from another server and have to be zerofilled to be usable. Zerofilling means that all transaction id's are removed from the table and that all unused data is set to 0. The first part is needed for the table to work, the second is mainly useful if one wants to compress and archive the table. No real data is removed from the table. I will document this in the knowledgebase shortly. In no case should one get the error: Found row with transaction id 808; This is a bug and we will fix it as soon as I get a repeatable test case for it (Elena said she can repeat it and will produce one for me).
            Hide
            elenst Elena Stepanova added a comment -

            Re: mysql_upgrade destroys Maria tables?
            To reproduce the error message "Found row with transaction id..", do the following:

            • start server (no specific server parameters required);
            • run
              mysql -uroot -e "DROP DATABASE IF EXISTS db; CREATE DATABASE db; USE db; \
              CREATE TABLE t ( i INT ) ENGINE=Aria; BEGIN; INSERT INTO t VALUES (1),(2); COMMIT; \
              BEGIN; INSERT INTO t VALUES (1),(2); COMMIT;"
              (adjust connection parameter such as port or socket, if needed)
            • stop server, e.g. using mysqladmin -uroot shutdown
            • remove aria_log* files from the datadir;
            • start server on the same datadir;
            • run
              mysql -uroot -e "CHECK TABLE db.t"
              see the output:

            ---------------------------------------------------------------------------------------------------------------+

            Table Op Msg_type Msg_text

            ---------------------------------------------------------------------------------------------------------------+

            db.t check error Found row with transaction id 5 when max transaction id according to aria_control_file is 1
            db.t check error Corrupt

            ---------------------------------------------------------------------------------------------------------------+

            (the actual transaction id in the message might be different depending on the pre-conditions of your test).

            Please note that it's important either to use MySQL client in the batch mode (with -e, as described), or start it in interactive mode with -A.

            Show
            elenst Elena Stepanova added a comment - Re: mysql_upgrade destroys Maria tables? To reproduce the error message "Found row with transaction id..", do the following: start server (no specific server parameters required); run mysql -uroot -e "DROP DATABASE IF EXISTS db; CREATE DATABASE db; USE db; \ CREATE TABLE t ( i INT ) ENGINE=Aria; BEGIN; INSERT INTO t VALUES (1),(2); COMMIT; \ BEGIN; INSERT INTO t VALUES (1),(2); COMMIT;" (adjust connection parameter such as port or socket, if needed) stop server, e.g. using mysqladmin -uroot shutdown remove aria_log* files from the datadir; start server on the same datadir; run mysql -uroot -e "CHECK TABLE db.t" see the output: ------ ----- -------- --------------------------------------------------------------------------------------------+ Table Op Msg_type Msg_text ------ ----- -------- --------------------------------------------------------------------------------------------+ db.t check error Found row with transaction id 5 when max transaction id according to aria_control_file is 1 db.t check error Corrupt ------ ----- -------- --------------------------------------------------------------------------------------------+ (the actual transaction id in the message might be different depending on the pre-conditions of your test). Please note that it's important either to use MySQL client in the batch mode (with -e, as described), or start it in interactive mode with -A.
            Hide
            monty Michael Widenius added a comment -

            Re: mysql_upgrade destroys Maria tables?
            The bug was that check table (and also optimize and related admin functions) didn't check if table was moved and needed zerofill before starting to work on them.
            I have now fixed that check table will detect that a table is moved and give a proper error in this case.
            I also added more warning messages when zerofill happens.

            This is pushed into 5.1 and I will merge it to 5.2 and 5.3.

            Show
            monty Michael Widenius added a comment - Re: mysql_upgrade destroys Maria tables? The bug was that check table (and also optimize and related admin functions) didn't check if table was moved and needed zerofill before starting to work on them. I have now fixed that check table will detect that a table is moved and give a proper error in this case. I also added more warning messages when zerofill happens. This is pushed into 5.1 and I will merge it to 5.2 and 5.3.
            Hide
            ratzpo Rasmus Johansson added a comment -

            Launchpad bug id: 944422

            Show
            ratzpo Rasmus Johansson added a comment - Launchpad bug id: 944422

              People

              • Assignee:
                monty Michael Widenius
                Reporter:
                nbrnhardt nbrnhardt
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: