We're updating the issue view to help you get more done.Learn more

Multi-source replication

This task is about implementing a way for a MariaDB replication slave to
replicate from more than one master simultaneously.

Each master is handled by a specific replicator instance in the slave
server. Each replicator instance consists of separate I/O thread, SQL thread,
and associated state and configuration. In efffect, several replication slaves
are running at the same time, each replicating from a separate master, but all
replicating into a common data store (typically, but not necessarily to
separate databases/tables).

A replicator instance is identified with a user-chosen name used in
replication SQL statements such as CHANGE MASTER TO ...
This name is also included in file names to distinguish the files that keep
the replication state (relay logs, master.info, relay-log.info). This way,
each separate instance can be configured separately, but otherwise the same
way as existing single-source replication.

In order to remain backwards-compatible with existing third-party scripts
etc., the replicator instance name is made optional in all existing
replication statements. If it is omitted, the name "default" is used, and for
this particular name, master.info and the other files retain their old names
to allow seamless upgrades of slaves.

In this worklog, there is no extra conflict resolution proposed. The effect of
updates from one master on replication from another master will be the same as
the effect of direct user SQL queries on the slave server, ie. it is the
responsibility of the user/DBA to ensure no conflicts occur. If a conflict
causes replication of an event to fail (with duplicate key violation eg.), the
corresponding slave SQL thread will stop, requiring manual intervention to fix
and start.

An easy and typical way to avoid conflicts would be to eg. to use separate
databases for each master->slave replication channel. RBR idempotent slave
application can also be used to help resolve conflicts, for example.

See also MySQL WL#1697: http://forge.mysql.com/worklog/task.php?id=1697

High-Level Specification

Here is a preliminary list of things that need to be changed/extended to
handle multi-source:

File names
----------

These files used to store replication state must be extended to include the
replicator instance name. For "default", as a special case, the old name
should be used for backwards compatibility:

HOST-relay-bin.XXXXXX
HOST-relay-bin.index
relay-log.info
master.info

SQL statements
--------------

These statements need to be extended to take an optional replicator instance
name. If omitted, "default" is used:

CHANGE MASTER TO
LOAD DATA FROM MASTER
LOAD TABLE xxx FROM MASTER
MASTER_POS_WAIT(file, position, timeout)
RESET SLAVE
SHOW SLAVE STATUS
START SLAVE
START SLAVE UNTIL ...
STOP SLAVE

System variables
----------------

These system variables need to handle multiple replicator instances. Some can
remain global, and apply to all instances. Some will need to be per-instance,
and can probably use structured system variables, same was as for multiple key
caches:

SQL_SLAVE_SKIP_COUNTER
log_slave_updates
master_connect_retry
master_info_file
master_retry_count
max_relay_log_size
read_only
relay_log
relay_log_index
relay_log_info_file
relay_log_purge
relay_log_space_limit
replicate_do_db
replicate_do_table
replicate_ignore_db
replicate_ignore_table
replicate_rewrite_db
replicate_same_server_id
replicate_wild_do_table
replicate_wild_ignore_table
report_host
report_port
skip_slave_start
slave_compressed_protocol
slave_load_tmpdir
slave_net_timeout
slave_skip_errors

The deprecated options master_host, master_port, master_user, master_password,
and master_ssl_* should not be supported for multi source replication.

Low-Level Design

A main part of this worklog will be to modify the server code so that all
slave objects and data can have multiple instances, not use global variables,
etc. And so that all user-visible interfaces (SQL statements, system
variables, status variables, files) are extended to support multiple
replicator instances.

Another major part is to carefully develop the feature and not the least test
it for full backwards compatibility in the case where only a single, default
master is used. If this is not done correctly, a lot of users will get
problems when slaves are upgraded and their monitoring scripts, management
scripts, or replication state breaks.

Status

Assignee

Michael Widenius

Reporter

Rasmus Johansson