“Litmus is a chaos engineering tool for Kubernetes. Litmus provides easy-to-use, ready made pipeline jobs that can be used to build a pipeline to help harden the applications that use Cassandra. Litmus helps in validating the functionality of Cassandra, OpenEBS, and Kubernetes itself.”
In this post, I will detail some of the Litmus jobs that are related to Cassandra. But first, let’s start with a bit of recap of Cassandra, OpenEBS, and Litmus itself.
Apache Cassandra is an open-source, NoSQL, highly scalable, high-performance distributed database management system designed to handle an enormous amount of structured data across commodity servers.
OpenEBS is an open-source CNCF Sandbox project for container-attached and container-native storage on Kubernetes. OpenEBS adopts the Container Attached Storage (CAS) approach, where each workload is provided with a dedicated storage controller.
Why use OpenEBS for Cassandra HA?
There are several notable benefits to using OpenEBS as the underlying storage for Cassandra statefulset:
Chaos Engineering practices are quickly becoming a standard component of any CI/CD pipeline to ensure faster and reliable software development. This approach identifies possible failures before they become outages. Practicing Chaos Engineering in DevOps pipelines can increase the confidence in the system and enable application developers to make their code concrete.
Litmus is an Open-source Chaos engineering framework designed for validating behavior by inserting chaos into the applications running in the Kubernetes environment. It acts as preventive medicine for “diseases” that can occur in the production environment.
Though Litmus has been designed to be easily extendable and work with different persistent storage solutions, OpenEBS is the primary consumer of these tests. As a result, the majority of chaos functions are directed against the OpenEBS storage resources (target pod, replica pod, storage pools, etc.) and their corresponding state validations. For more information about Litmus, visit https://litmuschaos.io/
Litmus experiments are broadly categorized into four types:
Here, we will specifically focus on the Litmus deployers and chaos experiments currently available to build a CI/CD pipeline that can harden Cassandra's application running on OpenEBS and Kubernetes.
Ideally, developers should concentrate on implementing test cases for the business logic involved in the application pods. Pipelines for hardening the rest of the components of the stack, such as Cassandra deployment, OpenEBS implementation, and Kubernetes/OpenShift implementation, can be built through Litmus books. Later in this post, we provide a sample implementation and example litmus books for reference.
Stages of Cassandra HA Pipeline
The Pipeline has been constructed through various GitLab stages such as
The jobs in the above pipeline continue to evolve across releases.
Litmus Experiment for Deploying Cassandra HA
Litmus is based on ansible playbooks that typically create Kubernetes jobs to execute the desired test. Users can get started with Litmus by following this procedure.
Now, let me illustrate the usage of a few Litmus experiments.
Cassandra HA Deployer
Here is the litmus experiment for deploying Cassandra Statefulset in the Kubernetes environment.
In the above litmus experiment, set the parameters according to requirements before executing it. The job can be created by running the following simple kubectl command:
kubectl create -f cassandra_litmus.ymlLitmus Experiment for Pumping Traffic
This litmus experiment writes data into Cassandra instances using the tool `cassandra-stress`.
Chaos Job-Cassandra Pod Failure
The intention of this litmus experiment is to force a failure in one of the Cassandra pods and analyze the system’s behavior.
Make sure to update the environmental variables with the required test parameters before creating the job.
Contributing to Cassandra Chaos Chart
Litmus has introduced a new chaos operator and the infrastructure to run chaos in production against stateful applications such as Cassandra. If anyone in the Cassandra community is interested in contributing to the helm chart of Cassandra chaos, take a look at the issue
Summary
Implementing CI/CD pipelines for applications based on Cassandra, OpenEBS, and Kubernetes using Litmus is very simple, as illustrated in this discussion.
If you would like to contribute to Litmus, feel free to join our community slack channel slack.openebs.io and visit the #litmus channel.
Important Links
Litmus website: https://litmuschaos.io/
Litmus docs: https://docs.litmuschaos.io/
Litmus repo: https://github.com/litmuschaos/litmus
Issue for creating Cassandra chaos chart: https://github.com/litmuschaos/chaos-charts/issues/7
This article was first published on May 22, 2019 on MayaData's Medium Account