The TiDB Binlog component collects TiDB binlogs and synchronizes them to the downstream, such as TiDB and MySQL. This component is similar to the master-slave replication of MySQL in function. It collects binlogs generated by each TiDB instance and sorts them by transaction commit time to globally synchronize data to downstream. TiDB Binlog can be used to synchronize data to other databases, and to backup and restore TiDB data in quasi-real time. As one of the core components of TiDB, TiDB Binlog has been running stably in hundreds of production environments for a long time.

In order to facilitate users and developers to better understand and use TiDB Binlog components, as well as secondary development based on TiDB Binlog components for more business scenarios, we decided to officially open source TiDB Binlog components today.

TiDB Binlog function scenarios

  • Quasi-real-time data synchronization: Synchronizes TiDB data to other databases or message queues (such as TiDB/MySQL/MariaDB/Kafka).
  • Quasi-real-time backup and recovery: Incremental backup of TiDB cluster data to external systems. The backup data can be used to restore data to any point in time in case of a system failure or other scenarios.

TiDB Binlog architecture

TiDB Binlog core features

  • Supports replication mode similar to MySQL ROW.
  • Synchronizes data downstream in quasi-real-time and in the chronological order of transaction commits.
  • The distributed architecture supports flexible horizontal expansion and high service availability.
  • Data is highly reliable, and the system persists data to local disks in real time.
  • Supports multiple output modes, as follows:
    • File: The system writes binlog to the file system in quasi-real time as an incremental backup file. The incremental backup file can be used to restore data to any point in time.
    • Message queue: Output to Kafka according to binlog slave Protocol.
    • Downstream target database: TiDB/MySQL/MariaDB.

TiDB Binlog code and documentation resources

  • TiDB Binlog source code
  • TiDB Binlog User manual
  • In-depth understanding of the implementation principle of TiDB Binlog component
  • Customize the output mode or output to other downstream storage systems

Welcome to participate in TiDB Binlog design, research and development, testing and jointly promote TiDB Binlog to more mature, more stable. We will publish the TiDB Binlog source guide in the near future.