How to monitor OpenEBS cStor pools?

In the previous blog, we discussed how to monitor OpenEBS volumes. In that post, I primarily talked about monitoring volume controllers of the storage engines jiva and cstor. In this post, I want to cover how to monitor cstor-pools using Prometheus. From this, we can pull metrics from maya-exporter sidecar of cstor-pool pods and Grafana to visualize the metrics.

Monitoring cstor pools

Monitoring cstor pools

It’s been almost a year since we released cstor, and we feel there is a need for a continuous monitoring tool to monitor cstor-pool containers to obtain details about various metrics such as health of cstor-pool container, number of reads/writes, number of io’s, rebuilding status, throughput, latency etc. So, we introduced maya-exporter in the 0.9.0 release as a sidecar of cstor-pool pod as well, which collects metrics from cstor-pool container.

Prerequisites

  • Openebs version > 0.9.0
  • cstor pool must be up and running
  • PVC (*optional)

PVC is optional because the pool should be available before creating pvc’s.

Steps:

  1. Run kubectl create -f openebs-monitoring-pg.yaml
  2. Run kubectl get pods -o wide to verify and determine where Prometheus and Grafana instances are running.
  3. Run kubectl get nodes -o wide to obtain the node’s IP.
  4. Run kubectl get svc and note the port allocated to Prometheus and Grafana services.
  5. Open your browser, then open <nodeip:nodeport>(32514 for Prometheus and 32515 for Grafana). Here, you will be able to see the Prometheus’s expression browser and Grafana’s UI on your browser.
  6. Enter your login information, then add the data source name as prometheus. Select datasource type Prometheus and pass the <nodeip:nodeport> of Prometheus in the url field. Then click on run and test.
  7. Go to the next tab (Dashboard) and import the desired dashboard. You should be able to get the preloaded dashboard of prometheus.
  8. To create a pool dashboard, go to the import button and paste the json.
  9. You should now be able to obtain the graph of various metrics that provide info from OpenEBS pools and volumes in the UI by clicking the desired items from the dropdown menu.

Note: I am using NodePort here only for demonstration purposes. Please use service names or ingress to access if you wish to expose the Grafana and Prometheus.

If you have already read my previous blog, many of the steps are very similar. The only difference is that the dashboard json for pool metrics is different and we have added configurations to pull metrics from OpenEBS pools.

I hope you have enjoyed this post and your pool dashboard is up and running. If you encounter any issues, don’t hesitate to reach out to us on slack or comment. There are many other important metrics that are planned for future releases, which will help in visualizing OpenEBS from various perspectives.

That’s all folks! 😃

 

 

Utkarsh Mani Tripathi
Utkarsh is a maintainer of jiva project and has contributed in building both control and data plane of OpenEBS. He loves to learn about file-system, distributed systems and networking. Currently, he is mainly focusing on enhancing jiva and maya-exporter In his free time, he loves to write poems and make lip smacking dishes
Chuck Piercey
Chuck Piercey is a Silicon Valley product manager with experience shipping more than 15 products in several different market segments representing a total of $2.5Bn revenue under both commercial and open source business models. Most recently he has been working for MayaData, Inc. focused on software-defined storage, network, and compute for Kubernetes environments. Chuck occasionally writes articles about the technology industry.
Sagar Kumar
Sagar is a software engineer at Mayadata who loves coding and solving real-world problems. He has been playing with Kubernetes for the last couple of years. Currently, he is focused on building OpenEBS Director as the go-to solution for OpenEBS users. In his free time, he loves playing cricket and traveling.