Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 10.0.0
    • Labels:
    • Global Rank:
      391

      Description

      This task is about implementing a way for a MariaDB replication slave to
      replicate from more than one master simultaneously.

      Each master is handled by a specific replicator instance in the slave
      server. Each replicator instance consists of separate I/O thread, SQL thread,
      and associated state and configuration. In efffect, several replication slaves
      are running at the same time, each replicating from a separate master, but all
      replicating into a common data store (typically, but not necessarily to
      separate databases/tables).

      A replicator instance is identified with a user-chosen name used in
      replication SQL statements such as CHANGE MASTER TO ...
      This name is also included in file names to distinguish the files that keep
      the replication state (relay logs, master.info, relay-log.info). This way,
      each separate instance can be configured separately, but otherwise the same
      way as existing single-source replication.

      In order to remain backwards-compatible with existing third-party scripts
      etc., the replicator instance name is made optional in all existing
      replication statements. If it is omitted, the name "default" is used, and for
      this particular name, master.info and the other files retain their old names
      to allow seamless upgrades of slaves.

      In this worklog, there is no extra conflict resolution proposed. The effect of
      updates from one master on replication from another master will be the same as
      the effect of direct user SQL queries on the slave server, ie. it is the
      responsibility of the user/DBA to ensure no conflicts occur. If a conflict
      causes replication of an event to fail (with duplicate key violation eg.), the
      corresponding slave SQL thread will stop, requiring manual intervention to fix
      and start.

      An easy and typical way to avoid conflicts would be to eg. to use separate
      databases for each master->slave replication channel. RBR idempotent slave
      application can also be used to help resolve conflicts, for example.

      See also MySQL WL#1697: http://forge.mysql.com/worklog/task.php?id=1697

      High-Level Specification

      Here is a preliminary list of things that need to be changed/extended to
      handle multi-source:

      File names
      ----------

      These files used to store replication state must be extended to include the
      replicator instance name. For "default", as a special case, the old name
      should be used for backwards compatibility:

      HOST-relay-bin.XXXXXX
      HOST-relay-bin.index
      relay-log.info
      master.info

      SQL statements
      --------------

      These statements need to be extended to take an optional replicator instance
      name. If omitted, "default" is used:

      CHANGE MASTER TO
      LOAD DATA FROM MASTER
      LOAD TABLE xxx FROM MASTER
      MASTER_POS_WAIT(file, position, timeout)
      RESET SLAVE
      SHOW SLAVE STATUS
      START SLAVE
      START SLAVE UNTIL ...
      STOP SLAVE

      System variables
      ----------------

      These system variables need to handle multiple replicator instances. Some can
      remain global, and apply to all instances. Some will need to be per-instance,
      and can probably use structured system variables, same was as for multiple key
      caches:

      SQL_SLAVE_SKIP_COUNTER
      log_slave_updates
      master_connect_retry
      master_info_file
      master_retry_count
      max_relay_log_size
      read_only
      relay_log
      relay_log_index
      relay_log_info_file
      relay_log_purge
      relay_log_space_limit
      replicate_do_db
      replicate_do_table
      replicate_ignore_db
      replicate_ignore_table
      replicate_rewrite_db
      replicate_same_server_id
      replicate_wild_do_table
      replicate_wild_ignore_table
      report_host
      report_port
      skip_slave_start
      slave_compressed_protocol
      slave_load_tmpdir
      slave_net_timeout
      slave_skip_errors

      The deprecated options master_host, master_port, master_user, master_password,
      and master_ssl_* should not be supported for multi source replication.

      Low-Level Design

      A main part of this worklog will be to modify the server code so that all
      slave objects and data can have multiple instances, not use global variables,
      etc. And so that all user-visible interfaces (SQL statements, system
      variables, status variables, files) are extended to support multiple
      replicator instances.

      Another major part is to carefully develop the feature and not the least test
      it for full backwards compatibility in the case where only a single, default
      master is used. If this is not done correctly, a lot of users will get
      problems when slaves are upgraded and their monitoring scripts, management
      scripts, or replication state breaks.

      1.
      Multi-source: sql_slave_skip_counter doesn't work for a connection with a non-default name Technical task Closed Michael Widenius  
       
      2.
      Multi-source: Behavior of "RESET SLAVE 'name'" is inconsistent with the normal RESET SLAVE Technical task Closed Michael Widenius  
       
      3.
      Multi-source: sync_with_master doesn't accept a space after the comma Technical task Closed Michael Widenius

      100%

      Original Estimate - Not Specified Original Estimate - Not Specified
      Time Spent - 5 minutes
       
      4.
      Multi-source: [patch] Get rid of sleep in multi_source.simple test Technical task Closed Michael Widenius  
       
      5.
      Multi-source: Non-descriptive error message in the error log on creating a duplicate replication configuration Technical task Closed Michael Widenius  
       
      6.
      Multi-source: Slave allows multiple masters to have the same server id (might be bad for complicated replication setups) Technical task Closed Michael Widenius  
       
      7.
      Multi-source: SHOW RELAYLOG EVENTS doesn't work for named connections Technical task Closed Michael Widenius  
       
      8.
      Multi-source: More status variables need to be either local for a replication connection, or truly global across all slaves Technical task Closed Michael Widenius  
       
      9.
      Multi-source: "Freeing overrun buffer" error and valgrind warnings on change master with a named connection Technical task Closed Michael Widenius  
       
      10.
      Multi-source: [tests] Basic tests to review and add to the multi_source suite Technical task Closed Michael Widenius  
       
      11.
      Multi-source: RESET SLAVE 'name' ALL creates master.info.index file instead of updating multi-master.info Technical task Closed Michael Widenius  
       
      12.
      Multi-source: Memory loss warnings on an attempt to create a duplicate connection with a different name Technical task Closed Michael Widenius  
       
      13. Multi-source: sync_with_master without the 2nd parameter always attempts to synchronize with connection '', regardless the current default_master_connection Technical task Open Michael Widenius  
       
      14. Multi-source: Semisync replication is not fully supported for multiple masters and can cause replication failure and relay log corruption Technical task Open Michael Widenius

      3%

      Original Estimate - 2 days, 4 hours Original Estimate - 2 days, 4 hours
      Time Spent - 45 minutes Remaining Estimate - 2 days, 4 hours
       

        Activity

        Hide
        Michael Widenius added a comment -

        Have not been able to repeat the problem with memory allocation until very late last night. Will fix tomorrow.
        The problem with sql_slave_skip_counter was that this variable is not yet multi-source aware.

        Have fixed the following issues:

        • "Slave_IO_state and Connection_name still seem to be switched in SHOW FULL SLAVE STATUS"
        • --sync_with_master 0, 'master1' doesn't work
        • Made sql_slave_skip_counter

        It's intentionally that RESET SLAVE removes the master configuration (this comes from the original patch).
        The reason this method was used, is probably that there was no other logical way to remove a connection.
        This does however create the problem of how to be able to remove the relay logs for a named connection.
        The suggestions I have regarding this are:

        • Let RESET SLAVE remove relay logs and the connection but FLUSH SLAVE would only remove relay logs.
        • Add a DROP SLAVE 'connection_name' command.
        Show
        Michael Widenius added a comment - Have not been able to repeat the problem with memory allocation until very late last night. Will fix tomorrow. The problem with sql_slave_skip_counter was that this variable is not yet multi-source aware. Have fixed the following issues: "Slave_IO_state and Connection_name still seem to be switched in SHOW FULL SLAVE STATUS" --sync_with_master 0, 'master1' doesn't work Made sql_slave_skip_counter It's intentionally that RESET SLAVE removes the master configuration (this comes from the original patch). The reason this method was used, is probably that there was no other logical way to remove a connection. This does however create the problem of how to be able to remove the relay logs for a named connection. The suggestions I have regarding this are: Let RESET SLAVE remove relay logs and the connection but FLUSH SLAVE would only remove relay logs. Add a DROP SLAVE 'connection_name' command.
        Hide
        Elena Stepanova added a comment -

        Cannot we still use "RESET SLAVE 'name' ALL" to remove everything and "RESET SLAVE 'name'" to only remove the logs and position? It would be consistent with what we have for single-source: now 'RESET SLAVE ALL' removes the entire configuration, while 'RESET SLAVE' only removes logs and the position.

        For the memory problem, I'll try to create a shorter test case, to reduce the amount of noise in the trace. The problem itself is reproducible on my machines and on perro, both with the plain cmake build and with BUILD/compile-pentium-debug-max. So if it doesn't happen on your boxes, maybe you could try there. I can set up the environment so you would only have to run the test.

        Show
        Elena Stepanova added a comment - Cannot we still use "RESET SLAVE 'name' ALL" to remove everything and "RESET SLAVE 'name'" to only remove the logs and position? It would be consistent with what we have for single-source: now 'RESET SLAVE ALL' removes the entire configuration, while 'RESET SLAVE' only removes logs and the position. For the memory problem, I'll try to create a shorter test case, to reduce the amount of noise in the trace. The problem itself is reproducible on my machines and on perro, both with the plain cmake build and with BUILD/compile-pentium-debug-max. So if it doesn't happen on your boxes, maybe you could try there. I can set up the environment so you would only have to run the test.
        Hide
        Elena Stepanova added a comment -

        With the new usage of max_relay_log_size we are getting extra warnings in the standard test suite (as below). Should we add the warnings to the result files, or are you planning to change the algorithm somehow?

        CURRENT_TEST: rpl.rpl_deadlock_innodb
        — mysql-test/suite/rpl/r/rpl_deadlock_innodb.result 2012-09-28 03:23:07.482695000 +0400
        +++ mysql-test/suite/rpl/r/rpl_deadlock_innodb.reject 2012-10-01 22:52:05.107989763 +0400
        @@ -76,6 +76,8 @@

            • Test lock wait timeout and purged relay logs ***
              SET @my_max_relay_log_size= @@global.max_relay_log_size;
              SET global max_relay_log_size=0;
              +Warnings:
              +Warning 1292 Truncated incorrect max_relay_log_size value: '0'
              include/stop_slave.inc
              DELETE FROM t2;
              CHANGE MASTER TO MASTER_LOG_POS=<master_pos_begin>;

        mysqltest: Result length mismatch

        Show
        Elena Stepanova added a comment - With the new usage of max_relay_log_size we are getting extra warnings in the standard test suite (as below). Should we add the warnings to the result files, or are you planning to change the algorithm somehow? CURRENT_TEST: rpl.rpl_deadlock_innodb — mysql-test/suite/rpl/r/rpl_deadlock_innodb.result 2012-09-28 03:23:07.482695000 +0400 +++ mysql-test/suite/rpl/r/rpl_deadlock_innodb.reject 2012-10-01 22:52:05.107989763 +0400 @@ -76,6 +76,8 @@ Test lock wait timeout and purged relay logs *** SET @my_max_relay_log_size= @@global.max_relay_log_size; SET global max_relay_log_size=0; +Warnings: +Warning 1292 Truncated incorrect max_relay_log_size value: '0' include/stop_slave.inc DELETE FROM t2; CHANGE MASTER TO MASTER_LOG_POS=<master_pos_begin>; mysqltest: Result length mismatch
        Hide
        Michael Widenius added a comment -

        The warning for max_relay_log_size is ok in my opinion. Now it works like max_binlog_size and other variables.

        Show
        Michael Widenius added a comment - The warning for max_relay_log_size is ok in my opinion. Now it works like max_binlog_size and other variables.
        Hide
        Michael Widenius added a comment -

        Pushed into 10.0-base

        Show
        Michael Widenius added a comment - Pushed into 10.0-base

          People

          • Assignee:
            Michael Widenius
            Reporter:
            Rasmus Johansson
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 9 weeks, 4 hours Original Estimate - 9 weeks, 4 hours
              9w 4h
              Remaining:
              Time Spent - 4 weeks, 4 days, 4 hours, 50 minutes Remaining Estimate - 4 weeks, 1 day
              4w 1d
              Logged:
              Time Spent - 4 weeks, 4 days, 4 hours, 50 minutes Remaining Estimate - 4 weeks, 1 day
              4w 4d 4h 50m