This blog will explain how to migrate an application Jiva to CSPC cStor.
CSPC is a CSI driver implementation for the OpenEBS CStor storage engine.
The current implementation supports the following for CStor Volumes:
This blog will be using 1 master and 3 worker node clusters. In this blog, we will be discussing how to install OpenEBS, and a MySQL application will be deployed on JIva, and then the MySQL application will be migrated to CSPC cStor.
Note: We will need some downtime to perform this migration. Once the application’s backup is taken, there should not be any new write to the application. So, it is advised to stop any new writes to the application before taking backup. It can vary depending on the size of the volume, n/w bandwidth, resource limits.
Use the below command to verify iSCSI is running on all the nodes.
root@demo-2:~# systemctl status iscsid
● iscsid.service - iSCSI initiator daemon (iscsid)
Loaded: loaded (/lib/systemd/system/iscsid.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2020-10-14 20:10:17 IST; 19s ago
Docs: man:iscsid(8)
Process: 9288 ExecStart=/sbin/iscsid (code=exited, status=0/SUCCESS)
Process: 9285 ExecStartPre=/lib/open-iscsi/startup-checks.sh (code=exited, status=0/SUCCESS)
Main PID: 9290 (iscsid)
Tasks: 2 (limit: 4915)
CGroup: /system.slice/iscsid.service
├─9289 /sbin/iscsid
└─9290 /sbin/iscsidThis blog will be using operator yaml to install OpenEBS. Click here if a user wants to install OpenEBS using Helm.
First, get the operator yaml using the below command.
wget https://openebs.github.io/charts/openebs-operator.yamlAs mentioned in the prerequisites, we have 2 disks on each node (/dev/sdb, /dev/sdc). We will be using an exclude filter to exclude /dev/sdb as we will use it for JIva, and NDM will not detect it as blockdevices. Users can avoid this if they want to manage blockdevices by themselves.
Edit the operator yaml and add an exclude path filter in the openebs-ndm-config, as shown below.
apiVersion: v1
kind: ConfigMap
metadata:
name: openebs-ndm-config
namespace: openebs
labels:
openebs.io/component-name: ndm-config
data:
# udev-probe is default or primary probe which should be enabled to run ndm
# filterconfigs contails configs of filters - in their form fo include
# and exclude comma separated strings
node-disk-manager.config: |
probeconfigs:
- key: udev-probe
name: udev probe
state: true
- key: seachest-probe
name: seachest probe
state: false
- key: smart-probe
name: smart probe
state: true
filterconfigs:
- key: os-disk-exclude-filter
name: os disk exclude filter
state: true
exclude: "/,/etc/hosts,/boot"
- key: vendor-filter
name: vendor filter
state: true
include: ""
exclude: "CLOUDBYT,OpenEBS"
- key: path-filter
name: path filter
state: true
include: ""
exclude: "loop,/dev/fd0,/dev/sr0,/dev/ram,/dev/dm-,/dev/md,/dev/rbd,/dev/sdb"Now apply the operator yaml.
root@demo-1:~# kubectl apply -f openebs-operator.yaml
namespace/openebs created
serviceaccount/openebs-maya-operator created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/openebs-maya-operator created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/openebs-maya-operator created
deployment.apps/maya-apiserver created
service/maya-apiserver-service created
deployment.apps/openebs-provisioner created
deployment.apps/openebs-snapshot-operator created
configmap/openebs-ndm-config created
daemonset.apps/openebs-ndm created
deployment.apps/openebs-ndm-operator created
deployment.apps/openebs-admission-server created
deployment.apps/openebs-localpv-provisioner createdVerify all the OpenEBS pods are running.
root@demo-1:~# kubectl get pods -n openebs
NAME READY STATUS RESTARTS AGE
maya-apiserver-559484b978-f6tsv 1/1 Running 2 107s
openebs-admission-server-68b67858cb-q6rsz 1/1 Running 0 107s
openebs-localpv-provisioner-84d94fc75-42r2c 1/1 Running 0 107s
openebs-ndm-c4clw 1/1 Running 0 107s
openebs-ndm-operator-75b957bf74-9zdt6 1/1 Running 0 107s
openebs-ndm-qhjwn 1/1 Running 0 107s
openebs-ndm-zj2pc 1/1 Running 0 107s
openebs-provisioner-664d994494-w4h6b 1/1 Running 0 107s
openebs-snapshot-operator-59c97c6cfc-dsw2q 2/2 Running 0 107sList the blockdevices as below. If the user has not used the exclude filter then 6 blockdevices will be listed.
root@demo-1:~# kubectl get bd -n openebs
NAME NODENAME SIZE CLAIMSTATE STATUS AGE
blockdevice-c5214e4ed934156c492e8d2a52922cbc demo-2 42948607488 Unclaimed Active 2m45s
blockdevice-ca60348232f4fdb6392fbf6b86f0dba1 demo-4 42948607488 Unclaimed Active 2m42s
blockdevice-d50d6f154651d96a5cb4389ec34b10b6 demo-3 42948607488 Unclaimed Active 2m45sNow let’s create a Jiva storage pool.
These steps (Step-1 to Step-2) are to be performed on all the worker/storage nodes.
Step-1: Identify the disk using -- lsblk
root@demo-2:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
loop0 7:0 0 44.9M 1 loop /snap/gtk-common-themes/1440
loop1 7:1 0 89.1M 1 loop /snap/core/8268
loop2 7:2 0 62.1M 1 loop /snap/gtk-common-themes/1506
loop3 7:3 0 55.3M 1 loop /snap/core18/1885
loop4 7:4 0 3.7M 1 loop /snap/gnome-system-monitor/127
loop5 7:5 0 14.8M 1 loop /snap/gnome-characters/399
loop6 7:6 0 162.9M 1 loop /snap/gnome-3-28-1804/145
loop7 7:7 0 956K 1 loop /snap/gnome-logs/100
loop8 7:8 0 2.2M 1 loop /snap/gnome-system-monitor/148
loop9 7:9 0 956K 1 loop /snap/gnome-logs/81
loop10 7:10 0 54.7M 1 loop /snap/core18/1668
loop11 7:11 0 2.5M 1 loop /snap/gnome-calculator/826
loop12 7:12 0 4.2M 1 loop /snap/gnome-calculator/544
loop13 7:13 0 276K 1 loop /snap/gnome-characters/570
loop14 7:14 0 160.2M 1 loop /snap/gnome-3-28-1804/116
loop15 7:15 0 217.9M 1 loop /snap/gnome-3-34-1804/60
loop16 7:16 0 97.7M 1 loop /snap/core/10126
sda 8:0 0 100G 0 disk
└─sda1 8:1 0 100G 0 part /
sdb 8:16 0 20G 0 disk
sdc 8:32 0 40G 0 disk
sr0 11:0 1 1024M 0 romStep-2: Format the disk and mount them as shown below.
root@demo-2:~# mkfs.ext4 /dev/sdb
mke2fs 1.44.1 (24-Mar-2018)
Creating filesystem with 5242880 4k blocks and 1310720 inodes
Filesystem UUID: 90a938bc-c056-4bfb-8a5e-29fe2c23266f
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
root@demo-2:~# mkdir /home/openebs-jiva
root@demo-2:~# mount /dev/sdb /home/openebs-jiva
root@demo-2:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
fd0 2:0 1 4K 0 disk
loop0 7:0 0 3.7M 1 loop /snap/gnome-system-monitor/127
loop1 7:1 0 4.2M 1 loop /snap/gnome-calculator/544
loop2 7:2 0 14.8M 1 loop /snap/gnome-characters/399
loop3 7:3 0 2.5M 1 loop /snap/gnome-calculator/826
loop4 7:4 0 956K 1 loop /snap/gnome-logs/81
loop5 7:5 0 2.2M 1 loop /snap/gnome-system-monitor/148
loop6 7:6 0 160.2M 1 loop /snap/gnome-3-28-1804/116
loop7 7:7 0 62.1M 1 loop /snap/gtk-common-themes/1506
loop8 7:8 0 956K 1 loop /snap/gnome-logs/100
loop9 7:9 0 44.9M 1 loop /snap/gtk-common-themes/1440
loop10 7:10 0 89.1M 1 loop /snap/core/8268
loop11 7:11 0 217.9M 1 loop /snap/gnome-3-34-1804/60
loop12 7:12 0 276K 1 loop /snap/gnome-characters/570
loop13 7:13 0 55.3M 1 loop /snap/core18/1885
loop14 7:14 0 54.7M 1 loop /snap/core18/1668
loop15 7:15 0 162.9M 1 loop /snap/gnome-3-28-1804/145
loop16 7:16 0 97.7M 1 loop /snap/core/10126
sda 8:0 0 100G 0 disk
└─sda1 8:1 0 100G 0 part /
sdb 8:16 0 20G 0 disk /home/openebs-jiva
sdc 8:32 0 40G 0 disk
sr0 11:0 1 1024M 0 romNow create a jiva_storagepool.yaml by adding the below content. In the path, we have mentioned the path we had mounted the disk in the previous step.
apiVersion: openebs.io/v1alpha1
kind: StoragePool
metadata:
name: jiva-pool
type: hostdir
spec:
path: "/home/openebs-jiva"Apply the jiva_storagepool.yaml and verify the pool has been created.
root@demo-1:~# kubectl apply -f jiva_storagepool.yaml
storagepool.openebs.io/jiva-pool created
root@demo-1:~# kubectl get storagepool
NAME AGE
default 4m22s
jiva-pool 13sCreate a jiva_storageclass.yaml with the below content. Mentioned the Jiva storagepool we created earlier in the StoragePool value.
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: jiva-sc
annotations:
openebs.io/cas-type: jiva
cas.openebs.io/config: |
- name: ReplicaCount
value: "3"
- name: StoragePool
value: jiva-pool
provisioner: openebs.io/provisioner-iscsi
Apply the jiva_storageclass.yaml and verify the storage class is created.
root@demo-1:~# kubectl apply -f jiva_storageclass.yaml
storageclass.storage.k8s.io/jiva-sc created
root@demo-1:~# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
jiva-sc openebs.io/provisioner-iscsi Delete Immediate false 6s
openebs-device openebs.io/local Delete WaitForFirstConsumer false 10m
openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 10m
openebs-jiva-default openebs.io/provisioner-iscsi Delete Immediate false 10m
openebs-snapshot-promoter volumesnapshot.external-storage.k8s.io/snapshot-promoter Delete Immediate false 10mCreate a secret by using the following command. The secret will be used in the MySQL application. The user will need to replace YOUR_PASSWORD with the password the user wants to use.
kubectl create secret generic mysql-pass --from-literal=username=root --from-literal=password=YOUR_PASSWORDCreate a mysql.yaml with the below details. This yaml will create a service and PVC for MySQL and a MySQL deployment. The user has to mention the Jiva storage class created earlier in the spec:storageClassName.
apiVersion: v1
kind: Service
metadata:
name: mysql
labels:
app: mysql
spec:
ports:
- port: 3306
selector:
app: mysql
tier: mysql
clusterIP: None
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
labels:
app: mysql
spec:
storageClassName: jiva-sc
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
name: mysql
labels:
app: mysql
spec:
selector:
matchLabels:
app: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: mysql
spec:
containers:
- image: mysql:5.6
name: mysql
env:
- name: MYSQL_ROOT_PASSWORD
valueFrom:
secretKeyRef:
name: mysql-pass
key: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claimNow apply mysql.yaml and verify the MySQL pod is running.
root@demo-1:~# kubectl apply -f mysql.yaml
service/mysql created
persistentvolumeclaim/mysql-pv-claim created
deployment.apps/mysql created
root@demo-1:~# kubectl get pods
NAME READY STATUS RESTARTS AGE
mysql-6c85844c87-rpqb5 1/1 Running 0 38sNow execute into the MySQL pod and write some data into it. We can verify the data are intact after the migration.
root@demo-1:~# kubectl exec -it mysql-6c85844c87-rpqb5 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@mysql-6c85844c87-rpqb5:/# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.49 MySQL Community Server (GPL)
Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> select * from KUBERA;
+--------+------------+-----------+------+---------+--------------+
| name | city | infra | vpc | storage | region |
+--------+------------+-----------+------+---------+--------------+
| ABN | SAN JOSE | AWS | YES | 50G | US-EAST-1 |
| IGL | SANTACLARA | AZURE | YES | 40G | DOWN-SOUTH-2 |
| LGI | KENTUCKY | AWS | NO | 20G | US-WEST-2 |
| BOEING | BRUSSELS | OPENSTACK | NO | 80G | ASIA-PACIFIC |
| NIFTY | SANTACRUZ | AWS | YES | 67G | EU-WEST-1 |
| ABN | SAN JOSE | AWS | YES | 50G | US-EAST-1 |
| IGL | SANTACLARA | AZURE | YES | 40G | DOWN-SOUTH-2 |
| LGI | KENTUCKY | AWS | NO | 20G | US-WEST-2 |
| BOEING | BRUSSELS | OPENSTACK | NO | 80G | ASIA-PACIFIC |
| NIFTY | SANTACRUZ | AWS | YES | 67G | EU-WEST-1 |
+--------+------------+-----------+------+---------+--------------+
10 rows in set (0.00 sec)Apply the latest cstor-operator.yaml by using the below command.
kubectl apply -f https://raw.githubusercontent.com/openebs/charts/gh-pages/cstor-operator.yamlThe user will see the below output once cstor-operator.yaml is applied. Verify the csi pods are running, as shown in the below output.
root@demo-1:~# kubectl apply -f https://raw.githubusercontent.com/openebs/charts/gh-pages/cstor-operator.yaml
namespace/openebs unchanged
serviceaccount/openebs-maya-operator unchanged
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/openebs-cstor-operator created
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/openebs-cstor-operator created
clusterrole.rbac.authorization.k8s.io/openebs-cstor-migration created
clusterrolebinding.rbac.authorization.k8s.io/openebs-cstor-migration created
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/csinodeinfos.csi.storage.k8s.io created
customresourcedefinition.apiextensions.k8s.io/cstorvolumeattachments.cstor.openebs.io created
customresourcedefinition.apiextensions.k8s.io/volumesnapshotclasses.snapshot.storage.k8s.io created
customresourcedefinition.apiextensions.k8s.io/volumesnapshotcontents.snapshot.storage.k8s.io created
customresourcedefinition.apiextensions.k8s.io/volumesnapshots.snapshot.storage.k8s.io created
Warning: storage.k8s.io/v1beta1 CSIDriver is deprecated in v1.19+, unavailable in v1.22+; use storage.k8s.io/v1 CSIDriver
csidriver.storage.k8s.io/cstor.csi.openebs.io created
clusterrolebinding.rbac.authorization.k8s.io/openebs-cstor-csi-snapshotter-binding created
clusterrole.rbac.authorization.k8s.io/openebs-cstor-csi-snapshotter-role created
serviceaccount/openebs-cstor-csi-controller-sa created
clusterrole.rbac.authorization.k8s.io/openebs-cstor-csi-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/openebs-cstor-csi-provisioner-binding created
statefulset.apps/openebs-cstor-csi-controller created
clusterrole.rbac.authorization.k8s.io/openebs-cstor-csi-attacher-role created
clusterrolebinding.rbac.authorization.k8s.io/openebs-cstor-csi-attacher-binding created
clusterrole.rbac.authorization.k8s.io/openebs-cstor-csi-cluster-registrar-role created
clusterrolebinding.rbac.authorization.k8s.io/openebs-cstor-csi-cluster-registrar-binding created
serviceaccount/openebs-cstor-csi-node-sa created
clusterrole.rbac.authorization.k8s.io/openebs-cstor-csi-registrar-role created
clusterrolebinding.rbac.authorization.k8s.io/openebs-cstor-csi-registrar-binding created
configmap/openebs-cstor-csi-iscsiadm created
daemonset.apps/openebs-cstor-csi-node created
customresourcedefinition.apiextensions.k8s.io/cstorpoolclusters.cstor.openebs.io created
customresourcedefinition.apiextensions.k8s.io/cstorpoolinstances.cstor.openebs.io created
customresourcedefinition.apiextensions.k8s.io/cstorvolumeconfigs.cstor.openebs.io created
customresourcedefinition.apiextensions.k8s.io/cstorvolumepolicies.cstor.openebs.io created
customresourcedefinition.apiextensions.k8s.io/cstorvolumereplicas.cstor.openebs.io created
customresourcedefinition.apiextensions.k8s.io/cstorvolumes.cstor.openebs.io created
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
customresourcedefinition.apiextensions.k8s.io/cstorbackups.openebs.io configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
customresourcedefinition.apiextensions.k8s.io/cstorcompletedbackups.openebs.io configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
customresourcedefinition.apiextensions.k8s.io/cstorrestores.openebs.io configured
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
customresourcedefinition.apiextensions.k8s.io/upgradetasks.openebs.io configured
deployment.apps/cspc-operator created
deployment.apps/cvc-operator created
service/cvc-operator-service created
deployment.apps/openebs-cstor-admission-server created
root@demo-1:~# kubectl get pods -n openebs -l role=openebs-cstor-csi
NAME READY STATUS RESTARTS AGE
openebs-cstor-csi-controller-0 7/7 Running 0 51s
openebs-cstor-csi-node-5qwh5 2/2 Running 0 51s
openebs-cstor-csi-node-mhgrd 2/2 Running 0 51s
openebs-cstor-csi-node-mljvw 2/2 Running 0 51sUsers need to specify cStor pool intent in a CSPC YAML to provision cStor pools on nodes. In this article, the user is going to provision 3 stripe cStor pools. Let us prepare a CSPC YAML now.
Following command list all block devices which represent the user’s attached disks as mentioned earlier.
kubectl get bd -n kuberaSample Output
root@demo-1:~# kubectl get bd -n openebs
NAME NODENAME SIZE CLAIMSTATE STATUS AGE
blockdevice-c5214e4ed934156c492e8d2a52922cbc demo-2 42948607488 Unclaimed Active 125m
blockdevice-ca60348232f4fdb6392fbf6b86f0dba1 demo-4 42948607488 Unclaimed Active 125m
blockdevice-d50d6f154651d96a5cb4389ec34b10b6 demo-3 42948607488 Unclaimed Active 125mNow the user has to pick 1 block device from each node to form a CSPC YAML. Users can pick multiple block devices from one node. The user has to update the hostname with the respective blockDeviceName carefully.
Sample CSPC yaml:
apiVersion: cstor.openebs.io/v1
kind: CStorPoolCluster
metadata:
name: cspc-stripe
namespace: openebs
spec:
pools:
- nodeSelector:
kubernetes.io/hostname: "demo-2"
dataRaidGroups:
- blockDevices:
- blockDeviceName: "blockdevice-c5214e4ed934156c492e8d2a52922cbc"
poolConfig:
dataRaidGroupType: "stripe"
- nodeSelector:
kubernetes.io/hostname: "demo-3"
dataRaidGroups:
- blockDevices:
- blockDeviceName: "blockdevice-d50d6f154651d96a5cb4389ec34b10b6"
poolConfig:
dataRaidGroupType: "stripe"
- nodeSelector:
kubernetes.io/hostname: "demo-4"
dataRaidGroups:
- blockDevices:
- blockDeviceName: "blockdevice-ca60348232f4fdb6392fbf6b86f0dba1"
poolConfig:
dataRaidGroupType: "stripe"
Now apply and verify the CSPC components are created and running, as shown below.
root@demo-1:~# kubectl apply -f cspc.yaml
cstorpoolcluster.cstor.openebs.io/cspc-stripe created
root@demo-1:~# kubectl get cspc -n openebs
NAME HEALTHYINSTANCES PROVISIONEDINSTANCES DESIREDINSTANCES AGE
cspc-stripe 3 3 11s
root@demo-1:~# kubectl get cspi -n openebs
NAME HOSTNAME FREE CAPACITY READONLY PROVISIONEDREPLICAS HEALTHYREPLICAS STATUS AGE
cspc-stripe-6564 demo-2 38500M 38500056k false 0 0 ONLINE 31s
cspc-stripe-dkb6 demo-4 38500M 38500053k false 0 0 ONLINE 30s
cspc-stripe-rwd4 demo-3 38500M 38500053k false 0 0 ONLINE 30s
Create storageclass for cStor
Now create cstor_sc.yaml by using the below contents. Mention the CSPC name that we created earlier in the parameters:cstorPoolCluster.
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: csi-cstor-sc
provisioner: cstor.csi.openebs.io
allowVolumeExpansion: true
parameters:
cas-type: cstor
cstorPoolCluster: cspc-stripe
replicaCount: "1"Now apply the cstor_sc.yaml and verify the storage class is created.
root@demo-1:~# kubectl create -f cstor_sc.yaml
storageclass.storage.k8s.io/csi-cstor-sc created
root@demo-1:~# kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
csi-cstor-sc cstor.csi.openebs.io Delete Immediate true 6s
jiva-sc openebs.io/provisioner-iscsi Delete Immediate false 5h55m
openebs-device openebs.io/local Delete WaitForFirstConsumer false 6h5m
openebs-hostpath openebs.io/local Delete WaitForFirstConsumer false 6h5m
openebs-jiva-default openebs.io/provisioner-iscsi Delete Immediate false 6h5m
openebs-snapshot-promoter volumesnapshot.external-storage.k8s.io/snapshot-promoter Delete Immediate false 6h5m
Migrating MySQL application from Jiva to CSPC cStor
In this blog, we will be using Velero with Restic for taking a complete backup of the MySQL application, .including complete data, and all the dependencies. We will need a Minio application locally to store the backup we will take.
Let’s deploy a Minio application using the below yaml in the velero namespace.
Create velero namespace using below yaml.
kubectl create ns veleroapiVersion: apps/v1
kind: Deployment
metadata:
# This name uniquely identifies the Deployment
name: minio
labels:
app: minio
spec:
selector:
matchLabels:
app: minio
strategy:
type: Recreate
template:
metadata:
labels:
# Label is used as selector in the service.
app: minio
spec:
# Refer to the PVC created earlier
volumes:
- name: storage1
persistentVolumeClaim:
# Name of the PVC created earlier
claimName: minio-pv-claim1
containers:
- name: minio
# Pulls the default Minio image from Docker Hub
image: minio/minio
args:
- server
- /storage1
env:
# Minio access key and secret key
- name: MINIO_ACCESS_KEY
value: "minio"
- name: MINIO_SECRET_KEY
value: "minio123"
ports:
- containerPort: 9000
# Mount the volume into the pod
volumeMounts:
- name: storage1 # must match the volume name, above
mountPath: "/storage1"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: minio-pv-claim1
labels:
app: minio
spec:
storageClassName: openebs-hostpath
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Service
metadata:
name: minio
labels:
app: minio
spec:
ports:
- port: 9000
nodePort: 32701
protocol: TCP
selector:
app: minio
sessionAffinity: None
type: NodePortCreate a minio.yaml using the above contents and apply it. In this yaml we have chosen to run Minio on openebs-hostpath storageclass. Users can also choose Jiva or cStor for that. Once the yaml is applied, verify the minio pod is running, as shown below.
root@demo-1:~# kubectl apply -f minio.yaml -n velero
deployment.apps/minio created
persistentvolumeclaim/minio-pv-claim1 created
service/minio created
root@demo-1:~# kubectl get pods -n velero
NAME READY STATUS RESTARTS AGE
minio-c45648fb4-tbmrn 1/1 Running 0 58s
root@demo-1:~# kubectl get pvc -n velero
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
minio-pv-claim1 Bound pvc-8ff5c6f4-66af-459a-bb84-0bf7cd04a0e7 10Gi RWO jiva-sc 62s
root@demo-1:~# kubectl get svc -n velero
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 443/TCP 24h
minio NodePort 10.103.229.48 9000:32701/TCP 67sUsers can access Minio UI from the browser using the below pattern.
http://<worker node IP>:<NodePort>
Example:
http://10.41.107.21:32701
Login to minio and create a bucket named the velero
Create a Velero-specific credentials file (credentials-velero) in your local directory:
[default]
aws_access_key_id = minio
aws_secret_access_key = minio123Install velero binary using the below commands.
wget https://github.com/vmware-tanzu/velero/releases/download/v1.5.1/velero-v1.5.1-linux-amd64.tar.gztar -zxvf velero-v1.5.1-linux-amd64.tar.gzcd velero-v1.5.1-linux-amd64/
root@demo-1:~/velero-v1.5.1-linux-amd64# cp velero /usr/local/bin/
root@demo-1:~/velero-v1.5.1-linux-amd64# velero version
Client:
Version: v1.5.1
Git commit: 87d86a45a6ca66c6c942c7c7f08352e26809426c
Install velero components using the below command.
velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.0.0 --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=true --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc:9000 --snapshot-location-config region="default" --use-resticOnce the above command is applied, the user will observe the below output. Verify velero and restic pods are running in the velero namespace, as shown below.
root@demo-1:~# velero install --provider aws --plugins velero/velero-plugin-for-aws:v1.0.0 --bucket velero --secret-file ./credentials-velero --use-volume-snapshots=true --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://minio.velero.svc:9000 --snapshot-location-config region="default" --use-restic
CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: created
VolumeSnapshotLocation/default: attempting to create resource
VolumeSnapshotLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: created
DaemonSet/restic: attempting to create resource
DaemonSet/restic: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.
root@demo-1:~# kubectl get pods -n velero
NAME READY STATUS RESTARTS AGE
restic-n2h8v 1/1 Running 0 40s
restic-n587c 1/1 Running 0 40s
restic-qcscn 1/1 Running 0 40s
velero-b54b7f5b8-zn9b4 1/1 Running 0 40sNow we will create a configmap using the below contents. We need to create the config map to restore the MySQL application in a different storage class. In this blog, the MySQL application runs on Jiva storageclass jiva-sc and we will migrate it to csi-cstor-sc.
User has mention this in data section of configmap yaml in <source storageclass>: <destination storageclass> pattern as shown below.
apiVersion: v1
kind: ConfigMap
metadata:
# any name can be used; Velero uses the labels (below)
# to identify it rather than the name
name: change-storage-class-config
# must be in the velero namespace
namespace: velero
# the below labels should be used verbatim in your
# ConfigMap.
labels:
# this value-less label identifies the ConfigMap as
# config for a plugin (i.e., the built-in restore item action plugin)
velero.io/plugin-config: ""
# this label identifies the name and kind of plugin
# that this ConfigMap is for.
velero.io/change-storage-class: RestoreItemAction
data:
# add 1+ key-value pairs here, where the key is the old
# storage class name and the value is the new storage
# class name.
jiva-sc: csi-cstor-scUsing the above content, create velero_cm.yaml and apply it. Verify the configMap created in the velero namespace.
root@demo-1:~# kubectl apply -f velero_cm.yaml
configmap/change-storage-class-config created
root@demo-1:~# kubectl get cm -n velero
NAME DATA AGE
change-storage-class-config 1 13sNow the user has to make sure that all the components of MySQL have a common label. We will use that common label to take backup. Here, the MySQL app has components like pod, deployment, PVC, SVC, and secret have common label app=mysql
root@demo-1:~# kubectl get pods --show-labels
NAME READY STATUS RESTARTS AGE LABELS
mysql-6c85844c87-rpqb5 1/1 Running 0 31m app=mysql,pod-template-hash=6c85844c87
root@demo-1:~# kubectl get deploy --show-labels
NAME READY UP-TO-DATE AVAILABLE AGE LABELS
mysql 1/1 1 1 31m app=mysql
root@demo-1:~# kubectl get pvc --show-labels
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE LABELS
mysql-pv-claim Bound pvc-b2fcadc1-b588-44cd-9914-926de41dde2c 10Gi RWO jiva-sc 31m app=mysql
root@demo-1:~# kubectl get svc --show-labels
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE LABELS
kubernetes ClusterIP 10.96.0.1 443/TCP 32h component=apiserver,provider=kubernetes
mysql ClusterIP None 3306/TCP 31m app=mysql
root@demo-1:~# kubectl get secret --show-labels
NAME TYPE DATA AGE LABELS
default-token-vg85s kubernetes.io/service-account-token 3 32h
mysql-pass Opaque 2 35m app=mysqlNow we have to add an annotation to the MySQL pod. This annotation will make sure that velero also takes the backup of the volume attached to the pod. The value of the annotation is the volume mount name of the pod. Here it is mysql-persistent-storage.
kubectl annotate pod mysql-6c85844c87-rpqb5 backup.velero.io/backup-volumes=mysql-persistent-storageNow, everything in place, we need to take a complete of the MySQL application. Use the below command to take a complete backup.
Note: Once a backup of the application is taken, there should not be any new write to the application. So, it is advised to stop any new writes to the application before taking backup.
velero backup create --selector Example:
root@demo-1:~# velero backup create mysql-backup --selector app=mysql
Backup request "mysql-backup" submitted successfully.
Run `velero backup describe mysql-backup` or `velero backup logs mysql-backup` for more details.Now check the backup status, as shown below. It should complete without any errors.
root@demo-1:~# velero backup get
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
mysql-backup Completed 0 0 2020-10-15 05:00:14 +0530 IST 29d default app=mysqlNow we will restore the application from the backup we took in the previous step. We will use namespace mapping to restore the application to another namespace as the MySQL application is already running in the default namespace. Users can also delete the application in its current namespace and restore it to the same namespace it was running. However, we will restore the application from the default namespace to the openebs namespace. Use the below command to restore the application.
velero restore create --from-backup --namespace-mappings :Example:
root@demo-1:~# velero restore create mysql-restore --from-backup mysql-backup --namespace-mappings default:openebs
Restore request "mysql-restore" submitted successfully.
Run `velero restore describe mysql-restore` or `velero restore logs mysql-restore` for more details.Now verify the restore should complete without any errors.
root@demo-1:~# velero restore get
NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR
mysql-restore mysql-backup Completed 2020-10-15 05:01:42 +0530 IST 2020-10-15 05:02:52 +0530 IST 0 0 2020-10-15 05:01:42 +0530 IST Now, we can see MySQL pod is running in openebs namespace.
root@demo-1:~# kubectl get pvc -n openebs
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
mysql-pv-claim Bound pvc-ffc10bec-c033-4387-b16e-f7dd36e1f5eb 10Gi RWO csi-cstor-sc 86s
root@demo-1:~# kubectl get pods -n openebs
NAME READY STATUS RESTARTS AGE
cspc-operator-5b755bcb85-h6zjb 1/1 Running 0 6h1m
cspc-stripe-6564-6fb46bb8d9-h65r5 3/3 Running 0 119m
cspc-stripe-dkb6-68c7ff54fc-vr9q6 3/3 Running 0 119m
cspc-stripe-rwd4-5499687b74-wfk4m 3/3 Running 0 119m
cvc-operator-5c6ff4f749-nhr24 1/1 Running 0 6h1m
maya-apiserver-559484b978-f6tsv 1/1 Running 2 7h56m
mysql-6c85844c87-rpqb5 1/1 Running 0 94s
openebs-admission-server-68b67858cb-q6rsz 1/1 Running 0 7h56m
openebs-cstor-admission-server-65549c6c8c-v2s4g 1/1 Running 0 6h1m
openebs-cstor-csi-controller-0 7/7 Running 0 6h1m
openebs-cstor-csi-node-5qwh5 2/2 Running 0 6h1m
openebs-cstor-csi-node-mhgrd 2/2 Running 0 6h1m
openebs-cstor-csi-node-mljvw 2/2 Running 0 6h1m
openebs-localpv-provisioner-84d94fc75-42r2c 1/1 Running 0 7h56m
openebs-ndm-c4clw 1/1 Running 0 7h56m
openebs-ndm-operator-75b957bf74-9zdt6 1/1 Running 0 7h56m
openebs-ndm-qhjwn 1/1 Running 0 7h56m
openebs-ndm-zj2pc 1/1 Running 0 7h56m
openebs-provisioner-664d994494-w4h6b 1/1 Running 0 7h56m
openebs-snapshot-operator-59c97c6cfc-dsw2q 2/2 Running 0 7h56m
pvc-b2fcadc1-b588-44cd-9914-926de41dde2c-ctrl-6f86c8b84c-bjf7f 2/2 Running 0 51m
pvc-b2fcadc1-b588-44cd-9914-926de41dde2c-rep-1-5d7bc57d44-dw65r 1/1 Running 0 51m
pvc-b2fcadc1-b588-44cd-9914-926de41dde2c-rep-2-7567494c45-8brwf 1/1 Running 0 51m
pvc-b2fcadc1-b588-44cd-9914-926de41dde2c-rep-3-c7bfc4b-g4bgg 1/1 Running 0 51m
pvc-ffc10bec-c033-4387-b16e-f7dd36e1f5eb-target-76cbdcdfb8wp2f2 3/3 Running 0 94s
sjr-pvc-1713f365-464a-4a6e-8f46-50586f9c4a1e-ky9a-jplr5 0/1 Completed 0 51m
sjr-pvc-1713f365-464a-4a6e-8f46-50586f9c4a1e-o3tw-tgtc5 0/1 Completed 0 51m
sjr-pvc-1713f365-464a-4a6e-8f46-50586f9c4a1e-trj6-vf6t6 0/1 Completed 0 51m
sjr-pvc-8ff5c6f4-66af-459a-bb84-0bf7cd04a0e7-6yio-crjqn 0/1 Completed 0 83m
sjr-pvc-8ff5c6f4-66af-459a-bb84-0bf7cd04a0e7-o8y4-5dx2s 0/1 Completed 0 83m
sjr-pvc-8ff5c6f4-66af-459a-bb84-0bf7cd04a0e7-x6dr-tj5qq 0/1 Completed 0 83mNow, we can execute the pod and verify the data are intact after the restoration.
Example:
root@demo-1:~# kubectl exec -it mysql-6c85844c87-rpqb5 -n openebs bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@mysql-6c85844c87-rpqb5:/# mysql -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.49 MySQL Community Server (GPL)
Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases;
+---------------------+
| Database |
+---------------------+
| information_schema |
| #mysql50#lost+found |
| mayadata |
| mysql |
| performance_schema |
+---------------------+
5 rows in set (0.00 sec)
mysql> use mayadata;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+--------------------+
| Tables_in_mayadata |
+--------------------+
| KUBERA |
+--------------------+
1 row in set (0.00 sec)
mysql> select * from KUBERA;
+--------+------------+-----------+------+---------+--------------+
| name | city | infra | vpc | storage | region |
+--------+------------+-----------+------+---------+--------------+
| ABN | SAN JOSE | AWS | YES | 50G | US-EAST-1 |
| IGL | SANTACLARA | AZURE | YES | 40G | DOWN-SOUTH-2 |
| LGI | KENTUCKY | AWS | NO | 20G | US-WEST-2 |
| BOEING | BRUSSELS | OPENSTACK | NO | 80G | ASIA-PACIFIC |
| NIFTY | SANTACRUZ | AWS | YES | 67G | EU-WEST-1 |
| ABN | SAN JOSE | AWS | YES | 50G | US-EAST-1 |
| IGL | SANTACLARA | AZURE | YES | 40G | DOWN-SOUTH-2 |
| LGI | KENTUCKY | AWS | NO | 20G | US-WEST-2 |
| BOEING | BRUSSELS | OPENSTACK | NO | 80G | ASIA-PACIFIC |
| NIFTY | SANTACRUZ | AWS | YES | 67G | EU-WEST-1 |
+--------+------------+-----------+------+---------+--------------+
10 rows in set (0.00 sec)I hope this blog will help you to migrate applications from Jiva to CSPC cStor. Thank you for reading, and please provide any feedback below or on Twitter. For more details on OpenEBS installation and troubleshooting, visit https://docs.openebs.io/. You can even reach out to us on OpenEBS slack channel here.