GitHub Chaos Actions in Your CI/CD workflow [Part-1]

In this blog, I will be talking about setting GitHub Chaos Actions in your CI workflow. Before jumping in, let's do a quick recap on LitmusChaos. Litmus is a framework for practicing chaos engineering in cloud-native environments. Litmus provides a chaos operator, a large set of chaos experiments on its hub, detailed documentation, and a friendly community. Litmus is very easy to use. You can also set up a very quick demo environment to install and run Litmus experiments.

What is Github Chaos Action?

Github actions automate the chaos execution on an application in the same place where the code is stored. You can write individual tasks along with chaos actions and combine them to create a custom GitHub workflow. GitHub Workflows are custom automated processes that you can set up in your repository to build, test, package, or deploy any code project on GitHub. Including the Github chaos actions in your workflow YAML, you can test your application's performance/resiliency in a much simpler and better way. To know more, visit the Github chaos actions repository.

What do we need Github Chaos Action?

GitHub Action makes the execution of chaos on an application in a much simpler and easier way. This action could be triggered based on the parameter listed in the GitHub workflow YAML. This action can be performed over an application that can be deployed in the cluster or already present in the cluster. Github chaos action helps to fix the weaknesses, leading to increased resilience of the system, which helps to gain confidence in developers and SRE. It takes a cloud-native approach to create, manage, and monitor the chaos.

Pre-requisites

• Kubernetes 1.11 or later.
• A workflow in your Github Repository
• Application pod in a healthy state
• Pre-requisites of the Chaos Experiment you are executing. You can find this under the experiment section in litmus docs.

A Sample Github Chaos Action Workflow

The workflow file is stored under .github/workflows directory of your repository, and it uses YAML syntax and must have either a .yml or .yaml file extension. We just need to follow the below-mentioned steps to create a sample GitHub workflow for performing chaos engineering on a Kubernetes application using Github chaos action.

1. Setup name and event to trigger the GitHub workflow

name: Push
on:
 push:
 branches: [ master ]
 pull_request:
 branches: [ master ]

This will trigger the GitHub workflow on a push or pull request basis on the master branch. Similarly, it can also be on a scheduled basis.

on:
 schedule:
 - cron: ‘*/15 * * * *’

2. Setup jobs in the workflow

jobs:
 build:
 runs-on: ubuntu-latest

A GitHub workflow run is made up of one or more jobs. Each job runs in an environment specified by runs-on. So in the above case, each job will run on the ubuntu-latest environment. By default jobs run in parallel to run them sequentially, we need to add

needs: <Job-Name>.
jobs:
 build:
 runs-on: ubuntu-latest
needs: job1
 steps: 
 — uses: actions/checkout@v2

3. Setup different tasks in a job

A job contains a sequence of tasks called Steps. Steps can be used to run commands, tasks, or actions itself. Not all steps run actions, but all actions run as a step.

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@master

Using job steps, we can run the chaos action over an application, and for doing so, we need to use our Github-chaos-actions in a below-mentioned way:

A sample pod deletes experiment workflow:

.github/workflows/main.yml
name: main
on:
 push:
 branches: [ master ]
jobs:
 build:
 runs-on: ubuntu-latest
steps:
 — uses: actions/checkout@master

 — name: Running Litmus pod delete chaos experiment
 uses: mayadata-io/github-chaos-actions@master
 env:
 ##Pass kubeconfig data from secret in base 64 encoded form 
 KUBE_CONFIG_DATA: $
 ##If litmus is not installed
 INSTALL_LITMUS: true
 ##Give application info under chaos
 APP_NS: default
 APP_LABEL: run=nginx
 APP_KIND: deployment
 EXPERIMENT_NAME: pod-delete
 ##Custom image can also been used
 EXPERIMENT_IMAGE: litmuschaos/ansible-runner
 EXPERIMENT_IMAGE_TAG: latest
 IMAGE_PULL_POLICY: Always     
 TOTAL_CHAOS_DURATION: 30
 CHAOS_INTERVAL: 10
 FORCE: false
 ##Select true if you want to uninstall litmus after chaos
 LITMUS_CLEANUP: true

This is a complete single job running a chaos action over an application for every push on branch master, and the kubeconfig of the cluster and application details are provided in the form of secrets and environment variables, respectively. Other Variables are chaos action variables that change based on the chaos experiment we are running.

How to connect the cluster with chaos actions?

1. To connect a healthy cluster with the chaos action, we need to pass the kubeconfig of the cluster in the base 64 encoded form as done in the above example. For doing this we need to follow these steps: Convert the kubeconfig of your cluster in base 64 encoded form using this command:

cat $HOME/.kube/config | base64

• Add a secret KUBE_CONFIG_DATA with the value containing the output of the above command.
• Pass this secret as ENV in Github chaos actions to connect your cluster with the action.

- name: Running Litmus pod delete chaos experiment
 uses: mayadata-io/github-chaos-actions@master
 env:
 KUBE_CONFIG_DATA: $

1. Creating a Cluster in the workflow itself is also one of the ways of performing chaos actions. We can even create a cluster in one of the job steps and then use Github-chaos-actions in further steps. In this case, we don't need to pass the kubeconfig externally through action's ENV. Instead, you can setup kubeconfig as an environment variable in base 64 encoded form using:

- name: "Setup kubeconfig ENV for Github Chaos Action"
  run: echo ::set-env name=KUBE_CONFIG_DATA::$(base64 -w 0 ~/.kube/config)

The KUBE_CONFIG_DATA will be accessible to the Chaos Action, which is running on a different container. Please visit [Part-2] of the GitHub action series. We have discussed it in a more detailed way.

Monitor the GitHub workflow jobs:

Once the workflow triggers, we need to monitor the workflow. For monitoring the different jobs in the workflow:

• Click on the Actions option present in the repository containing workflow.
  
  
• It will show all the workflow runs. Select the one which triggered recently
  
  
• Check the job logs of the workflow.
  

Details of chaos action run

• To get the details of what happened when the chaos action was running, check the logs on the Github-chaos-actions from the job logs.
• The Github-chaos-action contains the logs of the process involved in running the chaos actions.
• It also includes the logs of the experiment pod, which will give a verbose idea of what was happening at the time of chaos injection.
• At the end of the logs, there will be a check of the verdict whether the chaos result verdict is Pass or Fail (It will be a pass for the successful execution of the experiment).
• To know more about the terms like Chaos Experiment, Chaos Engine, and Chaos Result, visit litmus docs.

To know more about GitHub Chaos Actions and different ways to setup Chaos Actions in your GitHub workflow, read Part-2 now.

Conclusion

We can automate the chaos execution on a review application to check its resiliency in the same place where the codes are stored using Github Action. It provides an easier way to perform different chaos experiments, which helps in gaining confidence.

Are you an SRE or a Kubernetes enthusiast? Does Chaos Engineering excite you?

Join Our Community On Slack For Detailed Discussion, Feedback & Regular Updates On Chaos Engineering For Kubernetes: https://kubernetes.slack.com/messages/CNXNB0ZTN

(#litmus channel on the Kubernetes workspace)

Check out the Litmus Chaos GitHub repo and do share your feedback: https://github.com/litmuschaos/litmus

Submit a pull request if you identify any necessary changes.