Hardening your GitLab Deployment when Running OpenEBS Beneath

GitLab is an excellent tool used to manage git repositories on a centralized server. It is also great for building On-Premise cloud native CI/CD platforms. GitLab is composed of different stateful components like Gitaly, Redis, PostgreSQL, Minio and Prometheus.

GitLab is an excellent tool used to manage git repositories

Use Case:

GitLab runs with various stateful components such as Redis, PostgreSQl, Minio and Prometheus, all of which consume kubernetes volumes for storing pipeline related data. Securing data for the stateful components is a major concern when using Gitlab in a production environment. To improve availability of data and fault tolerance support for kubernetes volumes, a proper storage solution is required, and this is where OpenEBS comes into play.

Pros of using GitLab with OpenEBS:

• OpenEBS acts a single storage platform for all stateful applications, including Gitaly, Redis, PostgreSQL, Minio and Prometheus.
• OpenEBS provides HA to the volumes. Node loss, reboots and kubernetes upgrades will not affect the availability of persistent storage to the stateful applications.
• Storage is scalable on demand. You can start with small storage for all the databases required by GitLab and easily scale it on demand.
• Supports features like volume snapshots, which allows users to take a snapshot of data (mostly sensitive) and make multiple clones.

Steps to Install Gitlab on OpenEBS

• Prerequisites: A properly-functioning kubernetes cluster with helm installed.
• If OpenEBS is not installed in your K8s cluster, this can be done from here. If OpenEBS is already installed, move on to the next step.
• Add gitlab repo using helm:

helm repo add gitlab-openshift https://charts.gitlab.io/

• Create a yaml file named helm_options.yml to define the use of an OpenEBS storage class for the gitlab components deployments. The helm_options.yml will look like:
  gitlab:

gitaly:
persistence:
storageClass: openebs-cstor-sparse
size: 10Gi
postgresql:
persistence:
storageClass: openebs-cstor-sparse
size: 8Gi
minio:
persistence:
storageClass: openebs-cstor-sparse
size: 10Gi
redis:
persistence:
storageClass: openebs-cstor-sparse
size: 5Gi
prometheus:
persistence:
storageClass: openebs-cstor-sparse
size: 10Gi
storageClass: openebs-cstor-sparse
size: 10Gi

• Execute the following for gitlab installation:

helm upgrade — install gitlab-openshift gitlab/gitlab -f 
helm_options.yml \ 
— timeout 600 \ 
— set global.hosts.domain=<dns_name> \ 
— set global.hosts.externalIP=<master_ip> \ 
— set nginx-ingress.enabled=false \ 
— set registry.enabled=false \ 
— set certmanager-issuer.email=<email address>

Note: dns_name will be like example.io or example.com where gitlab dashboard will be hosted

Executing the steps above will deploy gitlab on k8s cluster in the default namespace using OpenEBS storage class (one of them is openebs-cstor-sparse).

Resilience and Fault Tolerance

After deploying GitLab using OpenEBS storage class, it’s always a good idea to perform some chaos on the GitLab components that use OpenEBS-cstor volume. This well help to verify how resilient and fault-tolerant the GitLab infrastructure is. For this reason, there is a tool called LITMUS. Users can run multiple litmus-books to induce chaos against different GitLab components.

LITMUS:
Litmus is a chaos engineering framework for stateful workloads on Kubernetes. The primary objective of Litmus is to ensure consistent and reliable behavior of workloads running in Kubernetes. To learn more about litmus, visit the official repository here.

There are different sets of litmus-books designed to perform tests (experiments or chaos) on either the application side or on OpenEBS components. Again, this will help to test and verify the resilience and fault tolerance of any application deployed on top of OpenEBS.

OpenEBS itself performs chaos engineering using litmus on a broader scale against different application (obviously GitLab is one of them) for quality assurance. OpenEBS uses GitLab pipelines in which litmus tests are systematically grouped in stages. In each individual stage, there are a certain number of jobs that execute a litmus-book to perform functional and chaos tests against applications using OpenEBS. You can view the running pipelines here.

As mentioned, OpenEBS experiments with different applications using litmus. It maintains a dedicated pipeline for gitlab as well, where different GitLab component failures are performed (like Unicorn pod failure, Prometheus target kill etc..).

• CLUSTER-SETUP: The very first stage of the pipeline where various cluster’s health checks are performed to ensure the cluster is healthy and ready to proceed further for performing tests.
• FUNCTIONAL: In this stage, application-specific functional tests are performed, such as scaling up the gitlab-unicorn pod.
• CHAOS: This stage is composed of tests that perform GitLab component failures, such as target-kill of minio, Prometheus, Redis etc.
• CLEANUP: This is the final stage of the pipeline where deprovisioning of test-related resources is performed and the k8s cluster is reverted back to its initial healthy state.

Summary:

Deploying GitLab using OpenEBS as storage is a trusted and reliable solution for any production environment. Volume management and data availability is not an overhead issue anymore when using OpenEBS as storage solution.

This article was first published on Jun 29, 2019 on MayaData's Medium Account