Updated September 7th 2021: This blog is updated with the the latest guide on OpenEBS Local PV, please refer to https://github.com/openebs/dynamic-localpv-provisioner.
Before reading this post, please read my previous post for instructions on setting up the ZFS-LocalPV for dynamically provisioning the volumes on the ZFS storage. Here, we will focus on how we can resize the volumes provisioned by ZFS-LocalPV without restarting the application.
Please make sure you have installed the ZFS-LocalPV Driver version v0.5 or later:
$ kubectl apply -f
https://raw.githubusercontent.com/openebs/zfs-localpv/v0.5.x/deploy/zfs-operator.yamlMake sure you are using k8s version 1.16+ as this feature is in beta. In Kubernetes 1.14 and 1.15, this feature was in alpha status and required enabling the following feature gate:
--feature-gates=ExpandCSIVolumes=true
Also for Kubernetes 1.14 and 1.15, online expansion feature gate has to be enabled explicitly:
--feature-gates=ExpandInUsePersistentVolumes=true
The ZFS-LocalPV CSI driver supports ONLINE volume expansion, which means, if the application is using the volume, you can perform the volume expansion. This also means that the volume expansion will be performed when an application is using that volume. So, if an application is not running and we have performed the resize operation, the Driver will wait for the application to start for the resize operation to complete.
Create the StorageClass with allowVolumeExpansion as true. We can resize only those volumes which are using the StorageClass with this flag as true.
$ cat sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: openebs-zfspv
allowVolumeExpansion: true
parameters:
poolname: "zfspv-pool"
provisioner: zfs.csi.openebs.io
$ kubectl apply -f sc.yaml
storageclass.storage.k8s.io/openebs-zfspv createdCreate the PVC using the above StorageClass:
$ cat pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: csi-zfspv
spec:
storageClassName: openebs-zfspv
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
$ kubectl apply -f pvc.yaml
persistentvolumeclaim/csi-zfspv createdNow deploy the application using the above PVC. Here, we will be using below Percona application:
$ cat percona.yaml
apiVersion: v1
kind: ConfigMap
metadata:
annotations:
name: sqltest
namespace: default
data:
sql-test.sh: |
#!/bin/bash
DB_PREFIX="Inventory"
DB_SUFFIX=`echo $(mktemp) | cut -d '.' -f 2`
DB_NAME="${DB_PREFIX}_${DB_SUFFIX}"
echo -e "\nWaiting for mysql server to start accepting connections.."
retries=10;wait_retry=30
for i in `seq 1 $retries`; do
mysql -uroot -pk8sDem0 -e 'status' > /dev/null 2>&1
rc=$?
[ $rc -eq 0 ] && break
sleep $wait_retry
done
if [ $rc -ne 0 ];
then
echo -e "\nFailed to connect to db server after trying for $(($retries * $wait_retry))s, exiting\n"
exit 1
fi
mysql -uroot -pk8sDem0 -e "CREATE DATABASE $DB_NAME;"
mysql -uroot -pk8sDem0 -e "CREATE TABLE Hardware (id INTEGER, name VARCHAR(20), owner VARCHAR(20),description VARCHAR(20));" $DB_NAME
mysql -uroot -pk8sDem0 -e "INSERT INTO Hardware (id, name, owner, description) values (1, "dellserver", "basavaraj", "controller");" $DB_NAME
mysql -uroot -pk8sDem0 -e "DROP DATABASE $DB_NAME;"
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: percona
labels:
name: percona
spec:
replicas: 1
selector:
matchLabels:
name: percona
template:
metadata:
labels:
name: percona
spec:
containers:
- resources:
name: percona
image: openebs/tests-custom-percona:latest
imagePullPolicy: IfNotPresent
args:
- "--ignore-db-dir"
- "lost+found"
env:
- name: MYSQL_ROOT_PASSWORD
value: k8sDem0
ports:
- containerPort: 3306
name: percona
volumeMounts:
- mountPath: /var/lib/mysql
name: demo-vol1
- mountPath: /sql-test.sh
subPath: sql-test.sh
name: sqltest-configmap
livenessProbe:
exec:
command: ["bash", "sql-test.sh"]
initialDelaySeconds: 30
periodSeconds: 1
timeoutSeconds: 10
volumes:
- name: demo-vol1
persistentVolumeClaim:
claimName: csi-zfspv
- name: sqltest-configmap
configMap:
name: sqltest
---
apiVersion: v1
kind: Service
metadata:
name: percona-mysql
labels:
name: percona-mysql
spec:
ports:
- port: 3306
targetPort: 3306
selector:
name: perconaApply the above YAML to deploy the Percona application:
$ kubectl apply -f percona.yaml
configmap/sqltest created
deployment.apps/percona created
service/percona-mysql createdNow the setup is ready and the application is running:
$ kubectl get po
NAME READY STATUS RESTARTS AGE
percona-9449b4b9c-48qpw 1/1 Running 0 38sCheck the volume size at the application size:
$ kubectl exec -it percona-9449b4b9c-48qpw bash
root@percona-9449b4b9c-48qpw:/# df -h
Filesystem Size Used Avail Use% Mounted on
none 91G 18G 69G 21% /
tmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda1 91G 18G 69G 21% /etc/hosts
shm 64M 0 64M 0% /dev/shm
/dev/zd0 4.9G 234M 4.7G 5% /var/lib/mysql
tmpfs 3.9G 12K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 3.9G 0 3.9G 0% /sys/firmwareFrom above o/p we can see that the volume has been created of size 5Gi and it is attached to the application at the given mount point (/var/lib/mysql).
Here, we just have to update the PVC with the new size and apply it. Please note that volume shrinking is not supported, so you have to change the size to a higher value. Here, in our case, we will update the size to 8Gi
$ cat pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: csi-zfspv
spec:
storageClassName: openebs-zfspv
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 8GiApply the above YAML to perform the resize:
$ kubectl apply -f pvc.yaml
persistentvolumeclaim/csi-zfspv configuredNow, we can keep checking the PVC for the new size to be updated, it may take a while. Once resize operation is done we can see the PVC output with the updated size:
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
csi-zfspv Bound pvc-9b5c22a5-29be-428e-aa96-5e183c1c4c62 8Gi RWO openebs-zfspv 33mWe can also exec into the application pod and verify that the new size is visible to the application:
$ kubectl exec -it percona-9449b4b9c-48qpw bash
root@percona-9449b4b9c-48qpw:/# df -h
Filesystem Size Used Avail Use% Mounted on
none 91G 18G 69G 21% /
tmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda1 91G 18G 69G 21% /etc/hosts
shm 64M 0 64M 0% /dev/shm
/dev/zd0 7.9G 237M 7.6G 3% /var/lib/mysql
tmpfs 3.9G 12K 3.9G 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 3.9G 0 3.9G 0% /sys/firmwareI hope you find this post useful. Feel free to contact me with any feedback or questions by using the comment section below.