The Distributed SQL Blog

Thoughts on distributed databases, open source and cloud native

YugabyteDB Community Update, Tricks and Tips – Nov 22, 2019

Welcome to this week’s community update where we recap a few interesting questions that have popped up in the last week or so on the YugabyteDB Slack channel, the Forum, GitHub or Stackoverflow. We’ll also review upcoming events, new blogs and documentation that has been published since the last update. Ok, let’s dive right in:

How do data compactions work in YugabyteDB?

YugabyteDB uses a tiered compaction strategy that has the advantage of lower disk write (IO) amplification. Normally, compactions require 50% free space, but with YugabyteDB this is fixed by splitting the table into multiple smaller tablets. Compactions are throttled and prioritized so they don’t exhaust system resources. For more information check out the Docs on “Throttled Compactions” plus “Small and Large Compaction Queues.”

How does YugabyteDB test its code base?

Our tests are split into several categories:

  • Module/class-level unit tests
  • End-to-end functional tests
  • Mini-cluster tests which simulate a multi-node setup on a single node
  • Integration tests that run on real multi-node setup.

The range of things we test includes testing for correctness of functionality, correctness under failure modes like the crash of node/process, network partition etc. (e.g., Jepsen tests) and performance tests to catch unexpected regressions.

Every diff triggers an automated build, which uses our in-house, distributed build toolchain spreading work across an autoscaling GCP instance group and an in-office fleet of servers, for parallelizing our Darwin efforts. For each build, once complete, a Spark job acts as our distributed task coordination framework and runs our entire in-repo test suite, somewhere around 4,000 tests, across C++ and Java. The tests are run against binaries from multiple compilers and build targets/systems including release and debug builds. Additionally, the tests are run on two special targets: CentOS Clang AddressSanitizer (ASAN) and CentOS Clang ThreadSanitizer (TSAN). The former runs the tests under an address sanitizer which checks for memory leaks/errors, and the latter instruments memory accesses to detect potential data races.

Finally, we use Jenkins pipelines to stitch everything together and another in-house webapp – aptly named Detective – to link diffs to their respective Jenkins jobs and logs and perform some automated test failure analysis and regression detection and let engineers “play Sherlock” on code failures. All of this happens automatically. On every version. For every diff in review. And once again, after landing… All the time!

We are continuously improving our testing and delivery pipeline and view it as a key strength that allows us to move as fast as we do with development, while ensuring that every release of YugabyteDB is as reliable, robust and performant as our users deserve.

Do I need an external load balancer with YugabyteDB?

Currently, we recommend a load balancer for users who make use of the YSQL API, but it is not required when using the YCQL API. Cluster aware clients for YSQL are getting ready to enter “release candidate” stage. You can learn more about our work in this area by checking the following:

New Documentation, Blogs, Tutorials, and Videos

New Blogs

New Videos

New Docs

Upcoming Meetups and Conferences

PostgreSQL Meetups

Distributed SQL Webinars

AWS re:Invent

We are Hiring!

YugaByte is growing fast and we’d like you to help us keep the momentum going! Check out our currently open positions:

Our team consists of domain experts from leading software companies such as Facebook, Oracle, Nutanix, Google and LinkedIn. We have come a long way in a short time but we cannot rest on our past accomplishments. We need your ideas and skills to make us better at every function that is necessary to create the next great software company. All while having tons of fun and blazing new trails!

Get Started

Ready to start exploring YugabyteDB features? Getting up and running locally on your laptop is fast. Visit our quickstart page to get started.

What’s Next?

  • Compare YugabyteDB in depth to databases like CockroachDB, Google Cloud Spanner and MongoDB.
  • Get started with YugabyteDB on macOS, Linux, Docker, and Kubernetes.
  • Contact us to learn more about licensing, pricing or to schedule a technical overview.

Related Posts