YugabyteDB CDC: Using JDBC Sink Connector with Kafka

It’s been a while since YugabyteDB came out with Change Data Capture (CDC) using Debezium, an open source distributed platform. You can just fire it up as a plugin for Kafka connect and then it will start publishing all the changes in your database to a Kafka topic. 

The Debezium Connector for YugabyteDB CDC is our addition to the list of Debezium connectors. It also provides support for reading changes from a YugabyteDB database.

In this blog, we’ll discover how to publish data to a Kafka topic and then read those messages to another sink, in this case a PostgreSQL database. But note that this blog is a continuation of Debezium and CDC in YugabyteDB. However, in this instance, we’ll follow the same approach to publish our data to Kafka.

First steps with YugabyteDB CDC

Assuming that you have now configured the source connector which would publish the changes to a Kafka topic, we can proceed forward.

1. Start a PostgreSQL instance

docker run -d --rm --name postgresql -p 5432:5432 \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres debezium/example-postgres:1.6

2. Deploy the configuration for the JDBC sink connector

Note that we are using our custom message transformer `YBExtractNewRecordState`. This transformer reads the message from the Kafka topic and then applies it on the sink as needed.

3. Insert two rows in YugabyteDB

4. Connect to PostgreSQL

Enter password: postgres

Finally, check if the records you inserted in YugabyteDB have been replicated here as well:

And voila! You have successfully replicated the data from YugabyteDB to a Postgres database using YugabyteDB CDC. Additionally, you can now try out more DML operations.

Conclusion

In this blog, we have successfully set up CDC on a YugabyteDB database and published the data to a sink database using the Debezium Connector for YugabyteDB. To read more about CDC in YugabyteDB, head over to our Docs site.

Want to grow your YugabyteDB skills? Enroll in free online training courses and certification from Yugabyte University, the best resource for advancing your career in distributed SQL.