Distributed SQL Change Management with Liquibase and YugabyteDB on GKE
Liquibase is an open source and extensible change management project that supports a variety of databases including Snowflake, MySQL, and PostgreSQL via JDBC. Liquibase allows users to easily define changes in SQL, XML, JSON, and YAML. These changes are then managed in a version control system so the changes can be documented, ordered, and standardized. For more information on the features and benefits of Liquibase, check out their documentation site.
In this blog post we’ll show you how to:
- Install a 3 node YugabyteDB cluster on Google Kubernetes Engine
- Build the sample Northwind database
- Install and configure Liquibase
- Create a simple changeset and verify the results
- Explore how changes are documented and managed in Liquibase
New to distributed SQL or YugabyteDB? Read on.
What is Distributed SQL?
Distributed SQL databases are becoming popular with organizations interested in moving data infrastructure to the cloud or cloud native environments. This is often motivated by the desire to reduce TCO or move away from the horizontal scaling limitations of monolithic RDBMS like Oracle, PostgreSQL, MySQL, and SQL Server. The basic characteristics of Distributed SQL are:
- They must have a SQL API for querying and modeling data, with support for traditional RDBMS features like foreign keys, partial indexes, stored procedures, and triggers.
- Automatic distributed query execution so that no single node becomes a bottleneck.
- Should support automatic and transparent distributed data storage. This includes indexes, which should be sharded across multiple nodes of the cluster so that no single node becomes a bottleneck. Data distribution ensures high performance and high availability.
- Distributed SQL systems should also provide for strongly consistent replication and distributed ACID transactions.
For a deeper discussion about what Distributed SQL is, check out, “What is Distributed SQL?”
What is YugabyteDB?
YugabyteDB is an open source, high-performance distributed SQL database built on a scalable and fault-tolerant design inspired by Google Spanner. YugabyteDB is PostgreSQL wire compatible, cloud native, offers deep integration with GraphQL projects, plus supports advanced RDBMS features like stored procedures, triggers, and UDFs.
Got questions? Make sure to ask them in our YugabyteDB Slack channel. Ok, let’s dive in…
Step 1: Install YugabyteDB on a GKE Cluster using Helm 3
In this section we are going to install YugabyteDB on the cluster. The complete steps are documented here. We’ll assume you already have a GKE cluster up and running as a starting point.
The first thing to do is to add the charts repository.
$ helm repo add yugabytedb https://charts.yugabyte.com
Now, fetch the updates.
$ helm repo update
Create a namespace. In this case we’ll call it yb-demo.
$ kubectl create namespace yb-demo
We are now ready to install YugabyteDB. In the command below we’ll be specifying values for a resource constrained environment.
$ helm install yb-demo yugabytedb/yugabyte \ --set resource.master.requests.cpu=1,resource.master.requests.memory=1Gi,\ resource.tserver.requests.cpu=1,resource.tserver.requests.memory=1Gi,\ enableLoadBalancer=True --namespace yb-demo --wait
To check the status of the cluster, execute the below command:
$ kubectl get services --namespace yb-demo
Note the external-IP for yb-tserver-service which we are going to use to establish a connection between YugabyteDB and Liquibase. From the screenshot above we can see that the IP is
34.72.XX.XX and the YSQL port is
Step 2: Creating the Northwind sample database
The next step is to download a sample schema and data. You can find a variety of sample databases that are compatible with YugabyteDB in our Docs. For the purposes of this tutorial we are going to use the Northwind sample database. The Northwind database contains the sales data for a fictitious company called “Northwind Traders,” which imports and exports specialty foods from around the world. The Northwind database is an excellent tutorial schema for a small-business ERP, with customers, orders, inventory, purchasing, suppliers, shipping, employees, and single-entry accounting.
Connect to the yb-tserver-pod by running the following command:
$ kubectl exec -n yb-demo -it yb-tserver-0 /bin/bash
To download the schema and data files, run the following commands:
$ wget https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/sample/northwind_ddl.sql $ wget https://raw.githubusercontent.com/yugabyte/yugabyte-db/master/sample/northwind_data.sql
Note: If the Google Cloud Shell tells you that the wget command does not exist, you can execute:
$ yum install wget -y
To connect to the YSQL service exit out of the pod shell and run the following command:
$ exit $ kubectl exec -n yb-demo -it yb-tserver-0 -- ysqlsh -h yb-tserver-0.yb-tservers.yb-demo
Create a database and connect to it using the following commands:
yugabyte=# CREATE DATABASE northwind; northwind=# \c northwind;
We can now create the database objects and load them with data using the files we downloaded to yb-tserver-pod using the following commands:
northwind=# \i 'northwind_ddl.sql'; northwind=# \i 'northwind_data.sql';
Verify that the tables are created by running the following command:
northwind-# \d List of relations Schema | Name | Type | Owner --------+------------------------+-------+---------- public | categories | table | yugabyte public | customer_customer_demo | table | yugabyte public | customer_demographics | table | yugabyte public | customers | table | yugabyte public | employee_territories | table | yugabyte public | employees | table | yugabyte public | order_details | table | yugabyte public | orders | table | yugabyte public | products | table | yugabyte public | region | table | yugabyte public | shippers | table | yugabyte public | suppliers | table | yugabyte public | territories | table | yugabyte public | us_states | table | yugabyte (14 rows)
Verify we have data by issuing a simple
northwind=# SELECT count(*) FROM products; count ------- 77 (1 row)
By default, a YugabyteDB installation doesn’t have a password setup for the default
yugabyte user. Specifying one is done the same way you’d do it in PostgreSQL.
northwind=# ALTER ROLE yugabyte WITH PASSWORD 'password';
Step 3: Install and configure Liquibase
Locate the appropriate version of Liquibase for your platform from their downloads page.
For the purposes of this demo, we’ll be installing Liquibase locally on a Mac.
Create a Liquibase project
On my Mac I created a new folder called LiquibaseYugabyteDB.
Download the appropriate PostgreSQL driver
Visit https://jdbc.postgresql.org/download.html and download the appropriate JDBC driver for your environment. I downloaded JDBC 4.2 (42.2.14) and placed it in the LiquidbaseYugabyteDB folder.
Create a changelog file
Next, in this same folder I created a file called dbchangelog.xml file. This changelog contains the sequence of changesets, each one of which makes small changes to the structure of the database. Add the following boilerplate to this file to get started.
<?xml version="1.0" encoding="UTF-8"?> <databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd"> </databaseChangeLog>
Create a properties file
Now in the same directory, create a file called liquibase.properties and add the following content to it that is reflective of your environment.
changeLogFile: ../LiquibaseYugabyteDB/dbchangelog.xml url: jdbc:postgresql://34.72.XX.XX:5433/northwind username: yugabyte password: password driver: org.postgresql.Driver classpath: ../LiquibaseYugabyteDB/postgresql-42.2.14.jar
Step 4: Create a changeset and verify the results
Create a changeset
To create a changeset, return to the dbchangelog.xml file and add the following copy highlighted in red. In the changeset we are going to create a promotions table in the Northwind database with columns for the name of the promotion and the discounted amount.
<?xml version="1.0" encoding="UTF-8"?> <databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd"> <changeSet id="1" author="jguerrero"> <createTable tableName="promotions"> <column name="id" type="serial"> <constraints primaryKey="true" nullable="false"/> </column> <column name="name" type="varchar(50)"> <constraints nullable="false"/> </column> <column name="discount" type="numeric(3,2)"/> </createTable> </changeSet> </databaseChangeLog>
The SQL equivalent of the changeset above would be:
CREATE TABLE promotions (id serial PRIMARY KEY, name VARCHAR(50) UNIQUE NULL, discount NUMERIC (3,2) );
Execute the changeset
update command locally in the directory you created to send the changeset to the database.
$ liquibase update Liquibase Community 4.0.0 by Datical Starting Liquibase at 11:00:28 (version 4.0.0 #19 built at 2020-07-13 19:45+0000) Liquibase: Update has been successful.
Verify the results
To verify that the promotions table was created we can open up a YSQL shell and insert two records into the table and select the data out.
INSERT INTO promotions (id, name, discount) VALUES (DEFAULT, 'Memorial Day Promotion', 3.50), (DEFAULT, 'International Coffee Day Promotion', 5.35); SELECT * FROM promotions;
Step 5: Managing changes
Along with the promotions table, you should now see two additional tables that Liquibase has created in the Northwind database.
The databasechangelog table
This table keeps a record of all the changesets that were deployed. This means that the next time you deploy again, the changesets in the changelog will be compared with the databasechangelog tracking table and only the new changesets that were not found in the databasechangelog will be deployed. You will notice that a new row was created in that table with the changeset information we have just deployed. For this example:
The databasechangelock table
This table is used internally by Liquibase to manage access to the changelog table during deployment. So, nothing to see here!
Step 6: Issue a second changeset
Create a second changeset
Let’s create an additional change set. In this case let’s add two more columns to the promotions table. A start_date column and a stop_date column so we know when a promotion starts and ends. Instead of XML, let’s use SQL to initiate the changes. To do this, create a file in your Liquibase project directory called changelog.sql.
Add the following commands to this file:
--liquibase formatted sql --changeset jimmy:1 ALTER TABLE promotions ADD COLUMN start_date TIMESTAMP, ADD COLUMN stop_date TIMESTAMP;
Issue the following command to send the change to the database.
$ liquibase --changeLogFile=changelog.sql update
Verify the change
We should now be able to see a second record in the databasechangelog tracking table.
We should also be able to see the new start_time and stop_date columns in the promotions table.
That’s it! You now have a PostgreSQL-compatible, 3 node YugabyteDB cluster running on GKE whose changes are now being managed and tracked by Liquibase. To learn more about supported third-party PostgreSQL tools that work with YugabyteDB, check out the YugabyteDB Documentation or join our community Slack and hit us up there.