We're updating the issue view to help you get more done. 

Replace with dummy events an event that is not understood by a slave to which it should be sent

Description

When new features are added we may want to add new binary log events. But
adding new events presents problems for replicating against older slaves that
do not understand the new event.

This is needed for eg. MDEV-181. The event that annotates row events with
their original query (introduced in 5.2 or 5.3 I forget which) is another
example.

In MySQL 5.6+, there is a flag to mark an event as ignorable - a 5.6+ slave
will ignore such an event, even if it does not know of it. But this does not
work for 5.5 or earlier versions (of either MariaDB or MySQL).

MariaDB 5.5 slave can tolerate that master just omits sending the event,
leaving a "hole" in the binlog stream. However, MariaDB <5.5 or any MySQL will
get confused on such holes, and will fail shortly after (because the
accumulated size of events received no longer match the master's binlog). In
fact, this happens when enabling row annotation events in MariaDB master and
using a slave that does not understand row annotation events. This bug is
fixed for mariadb 5.5 slave but remains for old mariadb slave or for mysql
slave.

The proposed solution here is that the master will for such older slaves not
just omit the event. Instead it will replace the event with a dummy event of
the same size - an empty query event (just a comment). The old slave will
execute the event (with no effect) and avoid becoming confused or failing on
unknown event.

If the master detects that the slave is new enough to understand the original
event (ie. understands the ignorable event flag), it will not substitute the
dummy event.


As dummy event we will use an empty query holding just a comment, like this:

1 "# Dummy event replacing event type %u that slave cannot handle."

The comment will be padded with spaces as needed, or truncated if the original
event is shorter (the whole idea is to supply the slave with a dummy event of
the same length, as old slaves do not tolerate holes in the binlog stream).

The query event will have no status variables, so it is just the fixed headers
+ one null byte (required for empty db) + the comment. This gives a minimum
event length of 34 bytes with a query consisting of a single character '#'.

For shorter events, we can instead replace with a user_var event. The variable
will be called @`!dummyvar` (or a shorter prefix of this that fits the length
of the original event, possibly just @`!`). This allows to go down to an event
length of just 25 bytes, 19 bytes of fixed header plus 6 bytes of data. I do
not think there exists events shorter than this, but if there does, they can
not be replaced with dummy events using this mechanism.


To help the master know what events the slave does and does not understand, we
will extend the slave to announce this.

The slave will, when connecting, set a user variable
@mariadb_slave_capability. The master will read the value of this variable
and use it to determine the capabilities of the slave. This is more robust and
flexible than relying on version numbers, since ideally any version master
should be able to replicate against any version slave. When we release the
code in a master, it is hard to predict what slave capabilities every possible
version of future releases will have - especially with multiple forks/branches
of MySQL being developed individually. And announcing the slave capabilities
explicitly allows the slave code full flexibility in working correctly against
older masters.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 /* MySQL or old MariaDB slave with no announced capability. */ #define MARIA_SLAVE_CAPABILITY_UNKNOWN 0 /* MariaDB >= 5.3, which understands ANNOTATE_ROWS_EVENT. */ #define MARIA_SLAVE_CAPABILITY_ANNOTATE 1 /* MariaDB >= 5.5. This version has the capability to tolerate events omitted from the binlog stream without breaking replication (MySQL slaves fail because they mis-compute the offsets into the master's binlog). */ #define MARIA_SLAVE_CAPABILITY_TOLERATE_HOLES 2 /* MariaDB > 5.5, which knows about binlog_checkpoint_log_event. */ #define MARIA_SLAVE_CAPABILITY_BINLOG_CHECKPOINT 3 /* MariaDB server which understands MySQL 5.6 ignorable events. This server can tolerate receiving any event with the LOG_EVENT_IGNORABLE_F flag set. */ #define MARIA_SLAVE_CAPABILITY_IGNORABLE 4

We can additionally push into 5.3 and 5.5 the functionality of the slave of
setting @mariadb_slave_capability - this is a safe change, and will help
mariadb masters >5.5 to work optimally against older MariaDB slaves.

The capability is a simple integer, as capabilities are added in new releases
but not removed.

The master can use the capability number to know when it must not send a
certain event to a certain slave. If it must not send, it can check if the
capability is at least MARIA_SLAVE_CAPABILITY_TOLERATE_HOLES. If so, master
can just skip the event and not send it to the slave - the slave is able to
deal with the resulting hole in the binlog stream. But if not - master will
replace the event with a dummy event, as implemented in this task, allowing
the slave to ignore the event and replicate correctly.

Environment

None

Status

Assignee

Kristian Nielsen

Reporter

Kristian Nielsen

Labels

None

External issue ID

None

External issue ID

None

Time tracking

70h

Priority

Major