MayaOnline—Tool to manage storage for Stateful workloads on Kubernetes

In October 2017, the term MayaOnline was coined for the first time at MayaData Inc. The increasing usage of our open-sourced OpenEBS Container Attached Storage (CAS) engine for stateful workloads in Kubernetes, created needs to visualize, monitor, and manage the storage components of the various Kubernetes clusters, where our customers were deploying OpenEBS engine. We began our initial experiments on building a prototype for an enterprise SaaS platform and launched an alpha version in December 2017 at the Kubecon event.

The initial version of the product had the following features:

• Import an existing OpenEBS-enabled K8s cluster.
• A dedicated Prometheus engine which collected the storage capacity and I/O metrics from the OpenEBS volumes in a time-series format.
• A dedicated alert-manager which configured the rules for generating alerts whenever the metrics reached the threshold limits.
• A dedicated Grafana service with industry-standard dashboards for visualizing the metrics.
• A dedicated chat-ops server which was integrated with Slack so that the cluster configuration and real-time alerts could be sent to the administrator’s slack channels.

We opened the portal for reviews, and through it, our users provided encouraging feedback which led us to launch the beta version by mid of May 2018 with additional features:

• Integration with Weave Scope for better visualization and easy sifting through the various K8s resources including storage support in Weave Scope.
• Support for OpenEBS-enabled OpenShift clusters.
• A multi-tenant Prometheus and Grafana engine.
• Additional dashboards for aggregate metrics.
• A dedicated chat bot called MuleBot.
• Enhanced user interface.

In the last couple of months, OpenEBS has evolved tremendously. A new open-source initiative, i.e., Node Disk Manager (NDM) has been started for common management of K8s disks (physical and virtual) for on-premise and cloud deployments. OpenEBS 0.7 (read blog) now supports cStor — a faster and more resilient storage engine for enterprise workloads, in addition to its existing engine Jiva. Then there is Litmus which introduces chaos to test the resiliency of stateful workloads in K8s (read Chaos Engineering).

Our users are really excited about these developments, and we are providing them with a new version of MayaOnline which integrates all of the above features for providing data agility (read MDAP). Some of the cool new features include:

• End-to-end storage visualization.
• Ability to take snapshots and restore data from OpenEBS volumes.
• Launch Litmus chaos tests on K8s applications.
• Enhanced user interface.

Let me talk more about each of them.

End-to-End Storage Visualization

MayaOnline beta already had Scope integration where the user could get a holistic view of all the K8s storage constructs and resources like Persistent Volumes, Persistent Volume Claims and Storage Classes which she has configured to satisfy the storage needs of her workloads. The new MayaOnline goes one step further and integrates NDM into scope. Now a user can view or manage the connectivity between the PVs and the core storage components like StoragePools and Disks!! This feature provides the right perspective to the user to understand the complex relationships between the various resources which in turn facilitates easy management and debugging.

No other product in the market can provide this level of control to the users.

Snapshots and Restore

Ability to take application-consistent snapshots has always been the need of the hour for the developers, and they expect all storage solutions to make provisions for the same. Along with the E2E visualization, a MayaOnline user can now browse/navigate to the volume of his choice and take snapshots with a single click of a button. What’s more? The snapshots themselves can be viewed as K8s resources and support control operations like creating clones and restoring them!!

Trigger snapshot control

Litmus Integration

Chaos engineering is the buzzword these days, and we at MayaData joined the bandwagon by developing Litmus — a first of its kind easy-to-develop Behavioral-Driven-Development Test framework where chaos tests can be written in plain English language!! Also, application developers and testers can use Litmus to induce real-time chaos like node failures, network failures, and pod failures to name a few. MayaOnline integrates with Litmus and provides an easy interface for the developers and users to develop, modify, launch and view the results of Litmus tests to see how their applications fare in real-time chaos constraints.

Enhanced User Interface

As a part of our continuous efforts to improve the user experience, we ran through rapid design sprints with extended teams and sought feedback from external users. This led to a re-branding effort, and the resultant new screens are more intuitive and user-friendly.

In summary, MayaOnline is a single place for all the tools that are required for a developer/administrator to use K8s clusters and stateful applications. Shortly, we will be adding more DevOps-centric features like GitOps integration, CI-CD pipelines, blue-green deployments along with a fully functional end-to-end monitoring and management portal.

To conclude, we’d like to offer our sincere and humble thanks to you, our users and well-wishers without whom this journey wouldn’t have been possible, and we will strive to excel at meeting your expectations.

If you have not already tried out MayaOnline, please do so at https://mayaonline.io/. We would love to hear about your experiences.

Cheers!!

This post was originally published on Aug 30, 2018, on MayaData's Medium account.