简介

It is a commonly observed pattern for applications to utilize multiple heterogeneous databases where each is used to serve a specific need such as storing the canonical form of data or providing advanced search capabilities. For applications it is hence desired to keep multi-ple databases in sync. We have observed a series of distinct patterns that have tried to solve this problem such as dual-writes and dis-tributed transactions. However, these approaches have limitations with regard to feasibility, robustness, and maintenance. An alterna-tive approach that has recently emerged is to utilize Change-Data-Capture (CDC) in order to capture changed rows from a database’s transaction log and eventually deliver them downstream with low latency. In order to solve the data synchronization problem one also needs to replicate the full state of a database and transaction logs typically do not contain the full history of changes. At the same time, there are use cases that require high availability of the transaction log events so that databases stay as closely in-sync as possible.

一个应用程序使用多个异构数据库是一种常见的模式，每个数据库都用于满足特定的需求，例如存储规范形式的数据或者提供高级搜索的数据源。因此，对于应用程序来说，需要保持多个数据库同步。我们观察到存在多种解决这个问题的方式，例如双写和分布式事务。然而，这些方法在可行性、稳健性和维护方面存在局限性，最近出现的一种替代方法是捕获变更数据(CDC) ，从数据库的事务日志中捕获变更的数据，并低延迟的传输给下游。为了解决数据同步问题，还需要复制数据库的完整状态，而事务日志通常不包含完整的更改历史记录。 同时，为了保障数据的同步，有些场景需要事务日志事件的高可用性。

To address the above challenges, we developed a novel CDC framework for databases, namely DBLog. DBLog utilizes a water-mark based approach that allows us to interleave transaction log events with rows that we directly select from tables to capture the full state. Our solution allows log events to continue progress without stalling while processing selects. Selects can be triggered at any time on all tables, a specific table, or for specific primary keys of a table. DBLog executes selects in chunks and tracks progress, allowing them to pause and resume. The watermark approach does not use locks and has minimum impact on the source. DBLog is currently used in production by tens of microservices at Netflix.
To address the above challenges, we developed a novel CDC
为了应对上述挑战，我们开发了一种新型的 CDC 数据库框架，即 DBLog。 DBLog 利用基于Watermark的方法，允许我们将事务日志事件与我们直接从表中选择的行交错以捕获完整状态。 我们的解决方案允许日志事件继续进行，而不会在处理选择时停止。 可以随时在所有表、特定表或表的特定主键上触发选择。 DBLog 以块的形式执行选择并跟踪进度，允许它们暂停和恢复。 水印方法不使用锁，对源的影响最小。 DBLog 目前被 Netflix 的数十个微服务用于生产。

介绍

Netfix uses hundreds of microservices performing trillions of oper-ations per day in the data layer. Since there is no single database design that fits all the needs, each of the microservices can utilize multiple heterogeneous databases. For example, a service can use MySQL, PostgreSQL, Aurora or Cassandra for the operational data and Elasticsearch for its indexing capabilities. To be able to keep multiple databases in sync we developed a data enrichment and synchronization platform namely Delta [7]. One of the key require-ments is to have low propagation delays from the source to the derived stores and that the flow of events is highly available. A key requirement to achieve that is having Change-Data-Capture (CDC) that allows capturing changed rows from a database in near real-time and eventually propagating those rows to downstream consumers [11]. CDC is becoming increasingly popular for use cases that require keeping multiple heterogeneous databases in sync [8, 12, 16] and addresses challenges that exist with traditional techniques like dual-writes and distributed transactions [13].

Netfix 使用数百个微服务，每天在数据层执行数万亿次操作。由于没有适合所有需求的单一数据库设计，因此每个微服务都可以利用多个异构数据库。例如，一项服务可以使用 MySQL、PostgreSQL、Aurora 或 Cassandra 来处理操作数据，并使用 Elasticsearch 来提供索引功能。为了能够使多个数据库保持同步，我们开发了一个数据丰富和同步平台，即 Delta [7]。关键要求之一是从源到派生存储的传播延迟低，并且事件流高度可用。实现这一目标的一个关键要求是拥有更改数据捕获 (CDC)，它允许从数据库中近乎实时地捕获更改的行，并最终将这些行传播给下游消费者 [11]。对于需要保持多个异构数据库同步的用例 [8、12、16]，CDC 正变得越来越流行，并解决了双写和分布式事务等传统技术存在的挑战 [13]。

In database systems, the transaction log typically has limited retention and it is not guaranteed to contain the full history of changes. Therefore, the full state of a database needs to be captured as well. While operating data synchronization in production at Netflix, we identified some requirements in regards to the full state capture. We wanted to (a) trigger the full state capture at any point in time. That is because the full state may not only be needed initially and may be needed at any time afterwards. For instance if the database is restored from a backup or for repairs if there is data loss or corruption downstream. There are also cases where only a subset of data needs to be repaired, for example if a specific set of rows has been identified to be corrupt downstream. (b) pause or resume at any time so that full state capture does not need to start from the beginning for large tables after restarting the process. (c) capture transaction log events and the full state side by side without stalling one or the other. There are use cases that require high availability of transaction log events so that the replication lag to the source is kept to a minimum. (d) prevent time-travel, by preserving the order of history when transmitting events to a derived datastore. This way an earlier version of a row (like the residential address of a member account) is not delivered after a later version. Hence, a solution had to combine transaction log events and the full state in a way that preserves the history of changes. (e) offer this as a platform. Hence it was crucial to minimize the impact on the source database. Otherwise this can hinter adoption of the platform, especially for use cases that have high traffic. In that regard we want to avoid primitives such as table locks which can block application write traffic. (f) function across a variety of Relational Database Management Systems (RDMBS), such as MySQL, PostgreSQL, Aurora [19] etc, that we use in production. In order to achieve that we wanted to avoid using vendor specific features.