Incomparable GTID Sets crash Replication after MySQL failover

[sandaru1]

Summary

When Readyset reconnects to a new MySQL server during the catch-up phase, replication dies permanently with "Incomparable GTID sets: neither is a subset of the other".

Description

This occurs when:

1. Readyset snapshots multiple tables at different GTID positions, creating a catch-up window {uuid-a: 1-N} → {uuid-a: 1-N+K}.
2. During catch-up, the upstream MySQL fails and a replica is promoted.
3. The replica was behind at failover time, so it does not have all uuid-a events up to N+K.
4. After promotion, writes to the new primary generate uuid-b events.
5. Readyset accumulates current = {uuid-a: 1-N+delta, uuid-b: 1-J} - incomparable to target = {uuid-a: 1-N+K} because current has a foreign uuid-b that target doesn't know about, and is also missing uuid-a events.

Steps to reproduce

See the attached files.

docker-compose.yml

haproxy.cfg

reproduce.sh

Logs

2026-03-19T22:02:06.079408Z  INFO replicators::mysql_connector::connector: Starting binlog replication offset=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6905
2026-03-19T22:02:06.106248Z  INFO replicators::noria_adapter: Catching up start=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6905 end=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6906
2026-03-19T22:02:06.111422Z  INFO replicators::noria_adapter: Streaming replication started position=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6906 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6907@0]
2026-03-19T22:02:15.021541Z  WARN replicators::noria_adapter: Replicator stopped during streaming replication position=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6973 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6974@1]
2026-03-19T22:02:15.021575Z  WARN replicators::noria_adapter: Restarting adapter after error encountered error=Error during replication: Binlog error before position GTID:49cbdc7a-23dc-11f1-b319-0242ac150002:1-6974 at replicators/src/mysql_connector/connector.rs:1419:32: Error during replication: Binlog error before position GTID:49cbdc7a-23dc-11f1-b319-0242ac150002:1-6974 at replicators/src/mysql_connector/connector.rs:525:28: Received EOF (254) from MySQL server
2026-03-19T22:02:15.021854Z ERROR replicators: Error in replication, will retry after timeout error=Error during replication: Binlog error before position GTID:49cbdc7a-23dc-11f1-b319-0242ac150002:1-6974 at replicators/src/mysql_connector/connector.rs:1419:32: Error during replication: Binlog error before position GTID:49cbdc7a-23dc-11f1-b319-0242ac150002:1-6974 at replicators/src/mysql_connector/connector.rs:525:28: Received EOF (254) from MySQL server timeout_sec=1
2026-03-19T22:02:16.045931Z  INFO replicators::noria_adapter: GTID mode enabled via --enable-gtid, server confirms gtid_mode=ON
2026-03-19T22:02:16.049356Z  INFO replicators::mysql_connector::connector: Crash recovery: will skip already-applied events pending_gtid=49cbdc7a-23dc-11f1-b319-0242ac150002:6907@0 events_to_skip=0
2026-03-19T22:02:16.050985Z  INFO replicators::mysql_connector::connector: Starting binlog replication offset=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6906 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6907@0]
2026-03-19T22:02:16.052526Z  INFO replicators::noria_adapter: Catching up start=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6906 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6907@0] end=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6973 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6974@1]
2026-03-19T22:02:16.052557Z  WARN replicators::mysql_connector::connector: Auto-finalizing implicitly-committed GTID before starting new transaction incoming_gtid=49cc52b8-23dc-11f1-86c6-0242ac150003:1@0
2026-03-19T22:02:16.053246Z  WARN replicators::mysql_connector::connector: Auto-finalizing implicitly-committed GTID before starting new transaction incoming_gtid=49cc52b8-23dc-11f1-86c6-0242ac150003:2@0
2026-03-19T22:02:16.053258Z  WARN replicators::noria_adapter: Replicator stopped during catch-up phase position=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6907 [pending: 49cc52b8-23dc-11f1-86c6-0242ac150003:1@0]
2026-03-19T22:02:16.053269Z  WARN replicators::noria_adapter: Restarting adapter after error encountered error=Error during replication: Incomparable GTID sets: neither is a subset of the other (self=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6907,49cc52b8-23dc-11f1-86c6-0242ac150003:1 [pending: 49cc52b8-23dc-11f1-86c6-0242ac150003:2@0], other=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6973 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6974@1])
2026-03-19T22:02:16.053283Z ERROR replicators: Error in replication, will retry after timeout error=Error during replication: Incomparable GTID sets: neither is a subset of the other (self=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6907,49cc52b8-23dc-11f1-86c6-0242ac150003:1 [pending: 49cc52b8-23dc-11f1-86c6-0242ac150003:2@0], other=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6973 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6974@1]) timeout_sec=1
2026-03-19T22:02:17.069882Z  INFO replicators::noria_adapter: GTID mode enabled via --enable-gtid, server confirms gtid_mode=ON
2026-03-19T22:02:17.074474Z  INFO replicators::mysql_connector::connector: Crash recovery: will skip already-applied events pending_gtid=49cbdc7a-23dc-11f1-b319-0242ac150002:6907@0 events_to_skip=0
2026-03-19T22:02:17.079066Z  INFO replicators::mysql_connector::connector: Starting binlog replication offset=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6906 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6907@0]
2026-03-19T22:02:17.081383Z  INFO replicators::noria_adapter: Catching up start=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6906 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6907@0] end=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6973 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6974@1]
2026-03-19T22:02:17.081456Z  WARN replicators::mysql_connector::connector: Auto-finalizing implicitly-committed GTID before starting new transaction incoming_gtid=49cc52b8-23dc-11f1-86c6-0242ac150003:1@0
2026-03-19T22:02:17.081539Z  WARN replicators::mysql_connector::connector: Auto-finalizing implicitly-committed GTID before starting new transaction incoming_gtid=49cc52b8-23dc-11f1-86c6-0242ac150003:2@0
2026-03-19T22:02:17.081549Z  WARN replicators::noria_adapter: Replicator stopped during catch-up phase position=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6907 [pending: 49cc52b8-23dc-11f1-86c6-0242ac150003:1@0]
2026-03-19T22:02:17.081562Z  WARN replicators::noria_adapter: Restarting adapter after error encountered error=Error during replication: Incomparable GTID sets: neither is a subset of the other (self=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6907,49cc52b8-23dc-11f1-86c6-0242ac150003:1 [pending: 49cc52b8-23dc-11f1-86c6-0242ac150003:2@0], other=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6973 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6974@1])
2026-03-19T22:02:17.081574Z ERROR replicators: Error in replication, will retry after timeout error=Error during replication: Incomparable GTID sets: neither is a subset of the other (self=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6907,49cc52b8-23dc-11f1-86c6-0242ac150003:1 [pending: 49cc52b8-23dc-11f1-86c6-0242ac150003:2@0], other=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6973 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6974@1]) timeout_sec=1
2026-03-19T22:02:18.091832Z  INFO replicators::noria_adapter: GTID mode enabled via --enable-gtid, server confirms gtid_mode=ON
2026-03-19T22:02:18.095112Z  INFO replicators::mysql_connector::connector: Crash recovery: will skip already-applied events pending_gtid=49cbdc7a-23dc-11f1-b319-0242ac150002:6907@0 events_to_skip=0
2026-03-19T22:02:18.097536Z  INFO replicators::mysql_connector::connector: Starting binlog replication offset=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6906 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6907@0]
2026-03-19T22:02:18.098989Z  INFO replicators::noria_adapter: Catching up start=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6906 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6907@0] end=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6973 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6974@1]
2026-03-19T22:02:18.099009Z  WARN replicators::mysql_connector::connector: Auto-finalizing implicitly-committed GTID before starting new transaction incoming_gtid=49cc52b8-23dc-11f1-86c6-0242ac150003:1@0
2026-03-19T22:02:18.099111Z  WARN replicators::mysql_connector::connector: Auto-finalizing implicitly-committed GTID before starting new transaction incoming_gtid=49cc52b8-23dc-11f1-86c6-0242ac150003:2@0
2026-03-19T22:02:18.099117Z  WARN replicators::noria_adapter: Replicator stopped during catch-up phase position=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6907 [pending: 49cc52b8-23dc-11f1-86c6-0242ac150003:1@0]
2026-03-19T22:02:18.099125Z  WARN replicators::noria_adapter: Restarting adapter after error encountered error=Error during replication: Incomparable GTID sets: neither is a subset of the other (self=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6907,49cc52b8-23dc-11f1-86c6-0242ac150003:1 [pending: 49cc52b8-23dc-11f1-86c6-0242ac150003:2@0], other=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6973 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6974@1])
2026-03-19T22:02:18.099137Z ERROR replicators: Error in replication, will retry after timeout error=Error during replication: Incomparable GTID sets: neither is a subset of the other (self=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6907,49cc52b8-23dc-11f1-86c6-0242ac150003:1 [pending: 49cc52b8-23dc-11f1-86c6-0242ac150003:2@0], other=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6973 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6974@1]) timeout_sec=1
2026-03-19T22:02:19.112369Z  INFO replicators::noria_adapter: GTID mode enabled via --enable-gtid, server confirms gtid_mode=ON
2026-03-19T22:02:19.115386Z  INFO replicators::mysql_connector::connector: Crash recovery: will skip already-applied events pending_gtid=49cbdc7a-23dc-11f1-b319-0242ac150002:6907@0 events_to_skip=0
2026-03-19T22:02:19.118592Z  INFO replicators::mysql_connector::connector: Starting binlog replication offset=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6906 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6907@0]
2026-03-19T22:02:19.119956Z  INFO replicators::noria_adapter: Catching up start=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6906 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6907@0] end=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6973 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6974@1]
2026-03-19T22:02:19.119971Z  WARN replicators::mysql_connector::connector: Auto-finalizing implicitly-committed GTID before starting new transaction incoming_gtid=49cc52b8-23dc-11f1-86c6-0242ac150003:1@0
2026-03-19T22:02:19.120045Z  WARN replicators::mysql_connector::connector: Auto-finalizing implicitly-committed GTID before starting new transaction incoming_gtid=49cc52b8-23dc-11f1-86c6-0242ac150003:2@0
2026-03-19T22:02:19.120053Z  WARN replicators::noria_adapter: Replicator stopped during catch-up phase position=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6907 [pending: 49cc52b8-23dc-11f1-86c6-0242ac150003:1@0]
2026-03-19T22:02:19.120062Z  WARN replicators::noria_adapter: Restarting adapter after error encountered error=Error during replication: Incomparable GTID sets: neither is a subset of the other (self=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6907,49cc52b8-23dc-11f1-86c6-0242ac150003:1 [pending: 49cc52b8-23dc-11f1-86c6-0242ac150003:2@0], other=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6973 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6974@1])
2026-03-19T22:02:19.120073Z ERROR replicators: Error in replication, will retry after timeout error=Error during replication: Incomparable GTID sets: neither is a subset of the other (self=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6907,49cc52b8-23dc-11f1-86c6-0242ac150003:1 [pending: 49cc52b8-23dc-11f1-86c6-0242ac150003:2@0], other=49cbdc7a-23dc-11f1-b319-0242ac150002:1-6973 [pending: 49cbdc7a-23dc-11f1-b319-0242ac150002:6974@1]) timeout_sec=1
2026-03-19T22:02:20.128548Z  INFO replicators::noria_adapter: GTID mode enabled via --enable-gtid, server confirms gtid_mode=ON
2026-03-19T22:02:20.130780Z  INFO replicators::mysql_connector::connector: Crash recovery: will skip already-applied events pending_gtid=49cbdc7a-23dc-11f1-b319-0242ac150002:6907@0 events_to_skip=0