Achieving cross zone HA in GKE

Why a multi-zone cluster?

GKE has multiple zones within each region, when we set up a cluster by default all the nodes get deployed in a single zone. Though rare, GCP data centres of a zone may face an outage, which might cause a business disruption lasting for a few minutes or for hours so in order to increase resiliency we can deploy the nodes of a cluster in different zones of a region. To know about setting up a multi-zone cluster you can follow Google Cloud Docs.

But there are certain limitations with GKE multi-zone cluster as well,

  • When a node goes down a new node will come up automatically (if auto scaling is enabled), but the problem lies with data availability in case the attached disk is ephemeral. When a node goes down the data associated with the node is lost.
  • Secondly, even if your application is capable of performing replication of data on its own, it will be more time consuming thereby resulting in a negative impact on the performance.

Hence, in any of the cases, it is advisable to provision OpenEBS volumes.

How is replication done with OpenEBS?

OpenEBS needs a minimum of 3 replicas to run the OpenEBS cluster with high availability. If a node fails, OpenEBS will manage the data to be replicated to a new disk. In the meantime, your workload can access data from one of the replicas thereby ensuring better performance and no downtime.

Getting started with OpenEBS Enterprise Platform is simple all you need is to follow the below-mentioned steps.



Getting Started with OpenEBS Enterprise Platform

Install OpenEBS on your cluster using the following command:

kubectl apply -f

In order to verify whether the namespace is created and all pods are working as expected, run the following command:

kubectl get ns

The OpenEBS namespace should be in Active State. (By default pods are created in OpenEBS namespace)

NAME         STATUS        AGE
default      Active        8m
kube-public  Active        8m
kube-system  Active        8m
openebs      Active        11s

Run the following command to check all OpenEBS pods are in running state using the following command: By default OpenEBS pods are created in openebs namespace.

kubectl get pod -n openebs

An easier way to monitor pods and other components that will be used in further steps is with Director Online or Director onPrem


Creating and Attaching Disks 

Next, you need to create a disk and attach them to the desired VMs, this can be done either from the console or using command line arguments.

In order to get the detailed steps to create and attach disks in GCP click the corresponding link.

  1. Creation of disk
  2. Attaching the disk


Provisioning OpenEBS volumes

First, Create a cStorPool:

For creation of a cStorPool you will need to specify the block devices, to view the disks attached in step 2 execute the following command

kubectl get blockdevice -n openebs_namespace

Now, copy the below snippet with your block devices specified in a file, say cstor-pool-config.yaml

kind: StoragePoolClaim
  name: cstor-disk-pool
  annotations: |
      - name: PoolResourceRequests
        value: |-
            memory: 2Gi
      - name: PoolResourceLimits
        value: |-
            memory: 4Gi
  name: cstor-disk-pool
  type: disk
    poolType: striped
  ## Replace the following with actual blockDevice CRs from your cluster using "kubectl get bd -n openebs" .
   - blockdevice-66a74896b61c60dcdaf7c7a76fde0ebb
    - blockdevice-b34b3f97840872da9aa0bac1edc9578a
    - blockdevice-ce41f8f5fa22acb79ec56292441dc207

Now, apply the above yaml file using the following command:

kubectl apply -f cstor-pool-config.yaml

To verify execute :

kubectl get spc

The output should have a cStor-pool created

NAME              AGE
cstor-disk-pool   1m

Also, Verify csp status:

kubectl get csp

The output status of the csp must be Healthy.
For further confirmation that things are working as expected execute the following command:

kubectl get pods -n openebs | grep poolname

Here, the name of pool is cstor-disk-pool so the command to be executed is:

kubectl get pods -n openebs | grep cstor-disk-pool

The output should display pods in running state.

cstor-disk-pool-c4qj-664bd98889-pclpb          3/3     Running   0          42m
cstor-disk-pool-c4ql-5fcb7646d6-jlk6w          3/3     Running   0          42m
cstor-disk-pool-gke5-76d97dc449-8px9c          3/3     Running   0          42m

Next, You have to create a StorageClass and mention about the StoragePoolClaim and ReplicaCount as the number of cStor Volume Replica is getting created.

Copy the YAML file given below, say cstor-sc.yaml

kind: StorageClass
  name: openebs-sc-statefulset
  annotations: cstor |
      - name: StoragePoolClaim
        value: "cstor-disk-pool"
      - name: ReplicaCount
        value: "3"    

To apply the above yaml execute:

kubectl apply -f cstor-sc.yaml

To verify execute:

kubectl get sc

The output must contain the StorageClass name specified in the yaml file.

NAME                            PROVISIONER                         AGE
openebs-sc-statefulset        1m

Next, you need a PVC spec or volumeClaimTemplate to use the StorageClass that is pointing to a pool with real disks.

apiVersion: v1
kind: PersistentVolumeClaim
  name: cstor-pvc-mysql-large
  storageClassName: openebs-sc-statefulset
    - ReadWriteOnce
      storage: 500Gi

Now, things are all set to deploy an application and get started.

Jeffry Molanus
Jeffry prior to being CTO at MayaData has worked at several other startups in the storage industry. He worked on several scale-out object storage products as well as traditional NAS and SAN storage solutions where he held technical leadership roles. At MayaData, his primary focus is to make sure the product is flexible and scalable. At the same time, robust enough to be integrated seamlessly into modern-day infrastructure where he believes, containers will have a dominant role. Jeffry holds a master's degree in electrical engineering with a focus on distributed control engineering from the University of Twente in the Netherlands. When he is not working with code, he practices martial arts.
Oum Kale
Oum Kale is an SDE intern at Mayadata. He loves to write programs and play chess. He is fascinated by the extensive impact computers can have on solving real-world problems.
Abhishek Raj
Abhishek is a Customer Success Engineer at Mayadata. He is currently working with Kubernetes and Docker.