OpenEBS Dynamic Volume Provisioning on ZFS

OpenEBS’ ZFS driver binds a ZFS file system into a Kubernetes environment and allows users to provision and de-provision volumes dynamically. This blog will demonstrate how to deploy a Percona application on the ZFS storage system with OpenEBS.

OpenEBS Dynamic Volume Provisioning on FS

ZFS combines the features of a file system and volume manager. Uniquely, this means ZFS can host a file system spanning multiple drives (a pool). Capacity can be added to a pool by adding another drive. ZFS handles partitioning and formatting. ZFS features include:

  • Pooled storage
  • Scalability to 16 Exabyte file & 256 Quadrillion Zettabytes storage
  • Copy-on-write
  • Replication
  • De-duplication
  • Snapshots
  • Data integrity verification and automatic repair
  • RAID-Z

(Note: ZFS’ history technical evolution and legal intricacies is both complex and interesting, and above all, it is a proof of the singular staying power of a healthy open source community.)

Setup

We will be using GKE with Kubernetes 1.14+ version with Ubuntu 18.4 installed on each node. We have to set up the node with ZFS utility once the cluster is up and running. We need to install zfsutils-linux on each node and use ZFS commands to set up a storage pool.

$ apt-get install -y zfsutils-linux

Attach the disk (if not already attached) to the nodes:

$ gcloud compute instances attach-disk <node-name> --disk <disk-name> --zone us-central1-a

Create the zpool using the disks:

$ zpool create <pool name> mirror /dev/sdb /dev/sdc

Check the zpool status:

$  zfs list
NAME     USED  AVAIL REFER  MOUNTPOINT
zfspv-pool   644K 96.4G 176K  /zfspv-pool

Install OpenEBS ZFS driver:

kubectl apply -f
https://raw.githubusercontent.com/openebs/zfs-localpv/master/deploy/zfs-operator.yaml

Check that the driver is installed:

$ kubectl get pods -n kube-system -l role=openebs-zfs
NAME                   READY   STATUS RESTARTS   AGE
openebs-zfs-controller-0   4/4 Running   0 5h28m
openebs-zfs-node-4d94n 2/2 Running   0 5h28m
openebs-zfs-node-gssh8 2/2 Running   0 5h28m
openebs-zfs-node-twmx8 2/2 Running   0 5h28m

Now we can deploy the Percona application.

Percona YAML:

$ cat percona.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: openebs-zfspv
allowVolumeExpansion: true
parameters:
 blocksize: "4k"
 compression: "on"
 dedup: "on"
 thinprovision: "yes"
 poolname: POOL_NAME
provisioner: openebs.io/zfs
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
 name: csi-zfspv
spec:
 storageClassName: openebs-zfspv
 accessModes:
- ReadWriteOnce
 resources:
requests:
   storage: 4Gi
---
apiVersion: v1
kind: ConfigMap
metadata:
 annotations:
 name: sqltest
 namespace: default
data:
 sql-test.sh: |
#!/bin/bash
DB_PREFIX="Inventory"
DB_SUFFIX=`echo $(mktemp) | cut -d '.' -f 2`
DB_NAME="${DB_PREFIX}_${DB_SUFFIX}"
echo -e "\nWaiting for mysql server to start accepting connections.."
retries=10;wait_retry=30
for i in `seq 1 $retries`; do
   mysql -uroot -pk8sDem0 -e 'status' > /dev/null 2>&1
   rc=$?
   [ $rc -eq 0 ] && break
   sleep $wait_retry
done
if [ $rc -ne 0 ];
then
   echo -e "\nFailed to connect to db server after trying for $(($retries * $wait_retry))s, exiting\n"
   exit 1
fi
mysql -uroot -pk8sDem0 -e "CREATE DATABASE $DB_NAME;"
mysql -uroot -pk8sDem0 -e "CREATE TABLE Hardware (id INTEGER, name VARCHAR(20), owner VARCHAR(20),description VARCHAR(20));" $DB_NAME
mysql -uroot -pk8sDem0 -e "INSERT INTO Hardware (id, name, owner, description) values (1, "dellserver", "basavaraj", "controller");" $DB_NAME
mysql -uroot -pk8sDem0 -e "DROP DATABASE $DB_NAME;"
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
 name: percona
 labels:
name: percona
spec:
 replicas: 1
 selector:
matchLabels:
   name: percona
 template:
metadata:
   labels:
     name: percona
spec:
   affinity:
     nodeAffinity:
       requiredDuringSchedulingIgnoredDuringExecution:
         nodeSelectorTerms:
         - matchExpressions:
           - key: kubernetes.io/hostname
             operator: In
             values:
             - NODE_NAME
   containers:
     - resources:
       name: percona
       image: openebs/tests-custom-percona:latest
       imagePullPolicy: IfNotPresent
       args:
         - "--ignore-db-dir"
         - "lost+found"
       env:
         - name: MYSQL_ROOT_PASSWORD
           value: k8sDem0
       ports:
         - containerPort: 3306
           name: percona
       volumeMounts:
         - mountPath: /var/lib/mysql
           name: demo-vol1
         - mountPath: /sql-test.sh
           subPath: sql-test.sh
           name: sqltest-configmap
       livenessProbe:
         exec:
           command: ["bash", "sql-test.sh"]
         initialDelaySeconds: 30
         periodSeconds: 1
         timeoutSeconds: 10
   volumes:
     - name: demo-vol1
       persistentVolumeClaim:
         claimName: csi-zfspv
     - name: sqltest-configmap
       configMap:
         name: sqltest
---
apiVersion: v1
kind: Service
metadata:
 name: percona-mysql
 labels:
name: percona-mysql
spec:
 ports:
- port: 3306
   targetPort: 3306
 selector:
   name: percona

Just patch the above YAML with the NODE_NAME where you want to run your application and POOL_NAME which you want to use for the storage. Then apply the configuration:

$ kubectl apply -f percona.yaml
storageclass.storage.k8s.io/openebs-zfspv created
persistentvolumeclaim/csi-zfspv created
configmap/sqltest created
deployment.apps/percona created
service/percona-mysql created

Check the status of the Pod.

$ kubectl get po
NAME                   READY   STATUS RESTARTS   AGE
percona-7456dc6f7b-nnfcz   1/1 Running   0 67s

Check the PVC

kubectl get pvc
NAME    STATUS   VOLUME                               CAPACITY   ACCESS MODES   STORAGECLASS AGE
csi-zfspv   Bound pvc-ecdb16e2-e03a-11e9-b418-42010a80006b   4Gi RWO        openebs-zfspv   5m39s

Using the ZFS list command, we can go the <NODE_NAME> and see that a volume has been created in the pool <POOL_NAME> :-

$ zfs list
NAME                                              USED  AVAIL REFER  MOUNTPOINT
zfspv-pool                                        115M  96.3G 176K  /zfspv-pool
zfspv-pool/pvc-ecdb16e2-e03a-11e9-b418-42010a80006b   102M 96.3G 102M  -

Summary

As demonstrated in this blog, OpenEBS makes it easy for Kubernetes applications to take advantage of the many technical advantages of ZFS storage.

Important links

https://github.com/openebs/zfs-localpv

Please drop your valuable comments or feed backs in the comment section below. Interested in our Use Cases and Solutions? Click here

Jeffry Molanus
Jeffry prior to being CTO at MayaData has worked at several other startups in the storage industry. He worked on several scale out object storage products as well as traditional NAS and SAN storage solutions where he held technical leadership roles. At MayaData, his primary focus will be around making sure the product is flexible and scalable yet robust enough to be integrated seamlessly into modern day infrastructure where he believes, containers will have a dominant role. Jeffry holds a master degree in electrical engineering with a focus on distributed control engineering from the University Twente in the Netherlands. When he is not working with code, he practices martial arts.
Saumya Sharma
I am a Software Engineer turned Digital Marketer. Being from a Computer Science background, Software Engineering never thrilled me. I believe learning is a continuous process, and it's never too late to start a new phase of life. I am currently working on SEO and blog setups. In my free time, I love to read and write. I have a blog named The Inked Perception, where I post my poetry. I also love to travel and binge-watch web series.
Pawan Sharma
It's been an amazing experience in Software Engineering because of my love for coding. I have a very good system understanding and I enjoy solving puzzles. In my free time, I read books, play table tennis and watch tv series