Easy Kubernetes storage for ElasticSearch

How to run the ElasticSearch operator (ECK) using OpenEBS LocalPV auto-provisioned disks. 
In the previous blog, we discussed the advantages of using OpenEBS LocalPV with the new operator for ElasticSearch, ECK. In this blog article, we will give step-by-step instructions to get started with deploying ElasticSearch, connecting it to MayaData DirectorOnline for free monitoring and scaling up the ElasticSearch cluster with auto-provisioning of the disks.

ElasticSearch and Kubernetes

Prepare your cluster for monitoring

OpenEBS Director gives a free forever tier for Kubernetes visibility (monitoring, logging and topology views). Start your Kubernetes visibility journey by connecting your Kubernetes to OpenEBS Director. Example steps are shown below.

1. Sign in using GitHub or Google or username/passwordSign in using GitHub or Google or username/password


2. Choose the location of your cluster and try to connect
Choose the location of your cluster and try to connect

3. You will be offered a command that you need to run on your Kubernetes shell for auto-connecting
command that you need to run on your Kubernetes shell for auto-connecting

root@openebs-ci-master:~# kubectl apply -f https://director.mayadata.io/v3/scripts/EB77CE273CFEAACE0EA8:1546214400000:jwUHp9bpttm9I5qgr1E7GrXMU.yaml
namespace/maya-system created
limitrange/maya-system-limit-range created
serviceaccount/maya-io created
clusterrolebinding.rbac.authorization.k8s.io/maya-io created
secret/maya-credentials-aaca9c75 created
job.batch/cluster-register created
daemonset.extensions/maya-io-agent created
deployment.apps/status-agent created
deployment.apps/kube-state-metrics created
service/kube-state-metrics created
configmap/fluentd-forwarder created
configmap/fluentd-aggregator created
deployment.apps/fluentd-aggregator created
service/fluentd-aggregator created
daemonset.apps/fluentd-forwarder created
configmap/cortex-agent-config created
deployment.extensions/cortex-agent created
service/cortex-agent-service created
deployment.apps/weave-scope-app created
service/weave-scope-app created
daemonset.extensions/weave-scope-agent created
deployment.apps/upgrade-controller created
root@openebs-ci-master:~#

4. Once the command is executed, your cluster is connected and existing applications are displayed if any are discovered.

Once the command is executed, your cluster is connected and existing applications are displayed if any are discovered.

Install OpenEBS

Choose the default installation mode of OpenEBS. Run the following command as Kubernetes admin.

helm install --namespace openebs --name openebs stable/openebs --version 1.1.0

Disk Pre-requisites

OpenEBS LocalPV provisioning requires the disks to be available on the nodes on which the ElasticSearch data pods are going to be scheduled. If more than one disk is available, OpenEBS LocalPV provisioner chooses the appropriate disk based on the requested size. So, the pre-requisite is to have at least one free disk (not mounted) on the node. 

As seen on Kubernetes shell:

root@openebs-ci-master:~# kubectl get bd -n openebs -o wide
NAME SIZE CLAIMSTATE STATUS AGE
blockdevice-3735b49b888bbf54406240a44aa7ed55   107374182400 Unclaimed Active 22h
blockdevice-b63e2e62040513698454d33818a5f630   107374182400 Unclaimed Active 22h
blockdevice-c2eb81ef31f4ca8cb1885aee675c23d0   107374182400 Unclaimed Active 22h
blockdevice-c3e24a19d11dd5ee1478c7bd8bf1ebdc   107374182400 Unclaimed Active 22h
blockdevice-e4faa44472434b2f8c3e0ef9533d3032   107374182400 Unclaimed Active 22h
blockdevice-ff5e6f96801909b8a572bc92a5e5c11c   107374182400 Unclaimed Active 22h

As seen on OpenEBS Director (Cluster -> Topology -> Hosts)
As seen on OpenEBS Director (Cluster -> Topology -> Hosts)

There are six worker nodes with an unclaimed disk on each node.
Each of them is unclaimed as seen below:

There are six worker nodes with an unclaimed disk on each node.


Prepare  OpenEBS LocalPV StorageClass to be used in your ECK operator

Create a LocalPV storageClass on your Kubernetes

cat <<EOF | kubectl apply -f -
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
 name: elastic-openebs-localpv-sc
 annotations:
   openebs.io/cas-type: local
   cas.openebs.io/config: |
     - name: StorageType
       value: "device"
     - name: FSType
       value: ext4
provisioner: openebs.io/local
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
EOF
Note: You can also use OpenEBS LocalPV in hostpath mode if you are sharing the underlying disk with other applications.

You need to append or use this StorageClass elastic-openebs-localpv-sc in the ECK operator as shown later in this blog.

You need to append or use this StorageClass elastic-openebs-localpv-sc in the ECK operator


Get Started with the ECK Operator

First, install the CRDs and the operator itself.
(https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html#k8s-deploy-elasticsearch)

kubectl apply -f https://download.elastic.co/downloads/eck/0.9.0/all-in-one.yaml

Verify if the elastic-operator is running
Verify if the elastic-operator is running
Start ElasticSearch using ECK with OpenEBS LocalPV storage class

Start ElasticSearch using ECK with OpenEBS LocalPV storage class

cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
kind: Elasticsearch
metadata:
 name: openebs-ci-elastic
spec:
 version: 7.2.0
 nodes:
- nodeCount: 3
 config:
   node.master: true
   node.data: true
   node.ingest: true
 volumeClaimTemplates:
 - metadata:
   name: elasticsearch-data # note: elasticsearch-data must be the name of the Elasticsearch volume
   spec:
     accessModes:
   - ReadWriteOnce
     resources:
     requests:
       storage: 90Gi
       storageClassName: elastic-openebs-localpv-sc
EOF

Note: Storage specific configurations include only storage size and StorageClass name

storage: 90Gi
storageClassName: elastic-openebs-localpv-sc


Verifying ElasticSearch

kubectl get elastic
NAME HEALTH NODES VERSION PHASE AGE
elasticsearch.elasticsearch.k8s.elastic.co/openebs-ci-elastic   green 3 7.2.0 Operational 106s
Verifying ElasticSearch

We have used NodeCount as 3. On the three Kubernetes worker nodes, OpenEBS LocalPV provisioner would have mounted the matching disks as required for each ElasticSearch Data Node. 

Application+PVC view
Application+PVC view

Disk / Block Device (BD) / Block Device Claim (BDC) view
Disk / Block Device (BD) / Block Device Claim (BDC) view

The three disks are automatically claimed and attached to PVC by the OpenEBS LocalPV provisioner.

Scaling up ElasticSearch

It is done easily by changing the nodeCount in the ElasticSearch CR spec.

root@openebs-ci-master:~# kubectl edit elasticsearch openebs-ci-elastic
 - -
           nodes:
         - config:
         node.data: true
         node.ingest: true
         node.master: true
         nodeCount: 3
         podTemplate:
           metadata:
             creationTimestamp: null
           spec:
             containers: null
         volumeClaimTemplates:
         - metadata:
           creationTimestamp: null
           name: elasticsearch-data       

Change the nodeCount to 4.
Save it.
And Elastic operator now scales the cluster to 4 nodes.

Verifying scaled up setup

Verify if the ElasticSearch cluster has 4th node and LocalPV is auto-provisioned

root@openebs-ci-master:~# kubectl get elastic

NAME HEALTH NODES VERSION PHASE AGE
elasticsearch.elasticsearch.k8s.elastic.co/openebs-ci-elastic   green 4 7.2.0 Operational 13m

Application view
Application view

PVC/BD/BDC view
PVC/BD/BDC viewSummary: 

We have demonstrated the following

  • The new Elastic operator is the most simple way to manage ElasticSearch clusters
  • Using OpenEBS LocalPV StorageClass, it is most simple to auto-provision and auto-scale the nodes with OpenEBS doing the provisioning logic behind the scene. 

OpenEBS also provides the following additional services for LocalPV and ElasticSearch:

  • Monitoring of disks health and usage parameters from the Director console
  • Monitoring of ElasticSearch performance and status from the Director console

Getting Help: 

Getting help for managing OpenEBS and ElasticSearch monitoring is easy. Sign up for free at OpenEBS Director and/or join our amazing slack community.

Authors:

1. Shashank Ranjan
SRE, Data Operations 
MayaData Inc
Twitter handle: @shashankranjan9


2. Uma Mukkara
COO
MayaData Inc
Twitter handle: @uma_mukkara

Utkarsh Mani Tripathi
Utkarsh is a maintainer of jiva project and has contributed in building both control and data plane of OpenEBS. He loves to learn about file-system, distributed systems and networking. Currently, he is mainly focusing on enhancing jiva and maya-exporter In his free time, he loves to write poems and make lip smacking dishes
Chuck Piercey
Chuck Piercey is a Silicon Valley product manager with experience shipping more than 15 products in several different market segments representing a total of $2.5Bn revenue under both commercial and open source business models. Most recently he has been working for MayaData, Inc. focused on software-defined storage, network, and compute for Kubernetes environments. Chuck occasionally writes articles about the technology industry.
Sagar Kumar
Sagar is a software engineer at Mayadata who loves coding and solving real-world problems. He has been playing with Kubernetes for the last couple of years. Currently, he is focused on building OpenEBS Director as the go-to solution for OpenEBS users. In his free time, he loves playing cricket and traveling.