The Distributed SQL Blog

Thoughts on distributed databases, open source, and cloud native

YugabyteDB CDC: Using JDBC Sink Connector with Kafka

It’s been a while since YugabyteDB came out with Change Data Capture (CDC) using Debezium, an open source distributed platform. You can just fire it up as a plugin for Kafka connect and then it will start publishing all the changes in your database to a Kafka topic. 

The Debezium Connector for YugabyteDB CDC is our addition to the list of Debezium connectors. It also provides support for reading changes from a YugabyteDB database.

In this blog, we’ll discover how to publish data to a Kafka topic and then read those messages to another sink, in this case a PostgreSQL database. But note that this blog is a continuation of Debezium and CDC in YugabyteDB. However, in this instance, we’ll follow the same approach to publish our data to Kafka.

First steps with YugabyteDB CDC

Assuming that you have now configured the source connector which would publish the changes to a Kafka topic, we can proceed forward.

1. Start a PostgreSQL instance

docker run -d --rm --name postgresql -p 5432:5432 \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres debezium/example-postgres:1.6

2. Deploy the configuration for the JDBC sink connector

First steps with YugabyteDB CDC

Note that we are using our custom message transformer `YBExtractNewRecordState`. This transformer reads the message from the Kafka topic and then applies it on the sink as needed.

3. Insert two rows in YugabyteDB

First steps with YugabyteDB CDC

4. Connect to PostgreSQL

First steps with YugabyteDB CDC

Enter password: postgres

Finally, check if the records you inserted in YugabyteDB have been replicated here as well:

First steps with YugabyteDB CDC

And voila! You have successfully replicated the data from YugabyteDB to a Postgres database using YugabyteDB CDC. Additionally, you can now try out more DML operations.

Conclusion

In this blog, we have successfully set up CDC on a YugabyteDB database and published the data to a sink database using the Debezium Connector for YugabyteDB. To read more about CDC in YugabyteDB, head over to our Docs site.

Want to grow your YugabyteDB skills? Enroll in free online training courses and certification from Yugabyte University, the best resource for advancing your career in distributed SQL.

Related Posts