The Distributed SQL Blog

Thoughts on distributed databases, open source and cloud native

Develop IoT Apps with Confluent Kafka, KSQL, Spring Boot & Distributed SQL

In our previous post “5 Reasons Why Apache Kafka Needs a Distributed SQL Database”, we highlighted why Kafka-based data services need a distributed SQL database like YugaByte DB as their highly scalable, long-term persistent data store. In this post, we show how Confluent Kafka, KSQL, Spring Boot and YugaByte DB can be integrated to develop an application for managing Internet-of-Things (IoT) sensor data.

The Scenario – IoT-Enabled Fleet Management

A trucking company wants to track its fleet of IoT-enabled vehicles that are delivering shipments across the country. The vehicles are of different types (such as 18 Wheelers, buses, large trucks) and follow 3 delivery routes (Route-37, Route-82, Route-43). In particular, the company wants to track:

  • Overall distribution of the vehicle types per delivery route.
  • Most recent (say, in the past 30 seconds) subset of these vehicle types per shipment delivery route.
  • List of vehicles near road closures, so that they can predict delays in deliveries.

Below is a snapshot of the real-time, auto-refreshing dashboard that this app will render to the end user to visualize the above three tracking aspects.

Application Architecture

In addition to Confluent Kafka as the streaming platform, the application has the following components:

  • Data Store: YugaByte DB for storing raw events from a Kafka stream as well as the aggregates from the KSQL Data Processor.
  • Data Producer: Program to simulate vehicle events being written into the Kafka stream.
  • Data Processor: KSQL reading from Data Producer, computing the aggregates and storing results in the Data Store.
  • Data Dashboard: Spring Boot app using web sockets, jQuery and Bootstrap to display output of Data Processor.

Below is the architectural diagram showing how these components fit together. We refer to this as the Confluent Kafka, KSQL and YugaByte DB stack or CKY stack.

We will now look at each of these components in detail.

Data Store

This layer stores all the user data. YugaByte DB is used as the database and YugaByte Cloud Query Language (YCQL) is used as the database API. All the data is stored in the keyspace TrafficKeySpace. There is a Origin_Table table for storing the raw events.

Note the default_time_to_live value set to 3600 seconds to ensure that raw events get auto-deleted after 1 hour. This is to ensure that the raw events do not consume up all the storage in the database and are efficiently deleted from the database a short while after their aggregates have been computed.

There are three tables that hold the data to be used for the user-facing display:

  • Total_Traffic for the lifetime traffic information
  • Window_Traffic for the last 30 seconds of traffic and
  • poi_traffic for the traffic near a point of interest (road closures).

The data processor constantly updates these tables, and the dashboard reads from them.

Below are the schemas for these tables.

Data Producer

This contains the program that generates simulated test data and publishes it to the Kafka topic iot-data-event. This emulates the data received from the connected vehicles using a message broker in the real world.

A single data point is a JSON payload and looks as follows:

The Kafka Connect YugaByte DB Sink Connector reads the above iot-data-event topic, transforms each such event into a YCQL INSERT statement and then calls YugaByte DB to persist the event in the TrafficKeySpace.Origin_Table table.

Data Processor

KSQL is the streaming SQL engine for Apache Kafka. It provides an easy-to-use yet powerful interactive SQL interface for stream processing on Kafka, without the need to write code in programming languages such as Java or Python. It supports a wide range of streaming operations, including data filtering, transformations, aggregations, joins, windowing, and sessionization.

The first step in using KSQL is to create a STREAM from the raw events of iot-data-event as shown below.

Various aggregations/queries can now be run on the above stream with results of each type of query stored in a new Kafka topic of its own. This application uses 3 such queries/topics. Thereafter, the Kafka Connect YugaByte DB Sink Connector reads these 3 topics and persists the results into the 3 corresponding tables in YugaByte DB.

The Kafka Connect YugaByte DB Sink Connector is used for storing both the raw events as well as the aggregate data (that’s generated using KSQL). It computes the following:

  • A breakdown by vehicle type and the shipment route across all the vehicles and shipments done so far.
  • Compute the above breakdown only for active shipments. This is done by computing the breakdown by vehicle type and shipment route for the last 30 seconds.
  • Detect the vehicles which are within a 20 mile radius of a given Point of Interest (POI), which represents a road-closure.

Data Dashboard

This is a Spring Boot application which queries the data from YugaByte DB and pushes the data to the webpage using Web Sockets and jQuery. The data is pushed to the web page in fixed intervals so data will be refreshed automatically. The main UI page uses bootstrap.js to display the dashboard containing charts and tables.

We create entity classes for the three tables Total_Traffic, Window_Traffic and poi_traffic, and Data Access Object (DAO) interfaces for all the entities extending CassandraRepository. For example, we create the DAO class for TotalTrafficData entity as follows.

In order to connect to YugaByte DB cluster and get connection for database operations, we also write a DatabaseConfig class.

Note that currently the Dashboard does not access the raw events table and relies only on the data stored in the aggregates tables.

Summary

This application is a blueprint for building IoT applications using Confluent Kafka, KSQL, Spring Boot and YugaByte DB. While this post focused on a local cluster deployment, the Kafka brokers and YugaByte DB nodes can be horizontally scaled in a real cluster deployment to get more application throughput and fault tolerance. The instructions to build and run the application, as well as the source code can be found in the yb-iot-fleet-management GitHub repo.

What’s Next?

  • Compare YugaByte DB in depth to databases like CockroachDB, Google Cloud Spanner and MongoDB.
  • Get started with YugaByte DB on macOS, Linux, Docker, and Kubernetes.
  • Contact us to learn more about licensing, pricing or to schedule a technical overview.

Related Posts