CLI Changes for OpenEBS 'cstor' storage engine

OpenEBS is the leading block storage solution on Kubernetes because of its Container Attached Storage (CAS) architecture. The latest release of the OpenEBS ‘cStor’ storage engine CLI natively supports ZFS, combining the best technical features of ZFS storage on individual servers with the usability features of the OpenEBS Kubernetes storage mesh that can span on-premise and multiple cloud servers.

CLI enhanced for ZFS

In this blog, we'll discuss how the latest release of the ‘cStor’ command-line interface (CLI) has been enhanced to support ZFS.

ZFS combines the features of a file system and volume manager. Uniquely, this means ZFS can host a file system spanning multiple drives (a pool). Capacity can be added to a pool by adding another drive. ZFS handles partitioning and formatting. ZFS features include:

  • Pooled storage
  • Scalability to 16 Exabyte file & 256 Quadrillion Zettabytes storage
  • Copy-on-write
  • Replication
  • Deduplication
  • Snapshots
  • Data integrity verification and automatic repair
  • RAID-Z

(Note: ZFS’ history and legal intricacies are complex and interesting, and it proves the singular power of a healthy open source community.) From an OpenEBS point of view, a key ZFS feature is that it ensures data is always consistent on the disk.

In cStor, we run ZFS in userspace which gives us more flexibility as there is no kernel dependency. There are several cStor changes we made in ZFS CLI to support running in userspace.

The existing model has userspace and kernel space architecture where the ZFS file system runs in kernel space, and zfs/zpool binaries run in userspace, and they interact via ioctl (system call). Since we run ZFS in userspace we can not use existing CLI (zfs/zpool) as those libraries directly interact with the kernel.

The existing zfs/zpool has been there for years and is feature-rich. To make these usable for cStor we introduced a Unix domain socket server and client architecture. The Unix domain server runs on the cStor side (zrepl binary) running the zfs filesystem in userspace, and the cli (zfs/zpool) runs as a client to this server. These two components make a socket call instead of an ioctl call which will route the request to the cStor instead of going to the kernel.

ZFS/ZPool and ZREPL

Once ZREPL gets the request, it will process this by calling an appropriate ioctl handler.

We have introduced a new -- enable-uzfs config option, through which you can switch to the Unix domain model from the ioctl model.

$ ./configure --enable-uzfs=yes

That is still not sufficient, so whenever a kernel space switch happens, the kernel does the copy-in of the data to copy the data from user space. This data should be sent from the zfs/zpool CLI to the Unix domain server (zrepl), and that copy-in call should be changed to socket read, similarly while returning to userspace, the kernel does the copy-out of the data, which now should be sent on the wire. So the zrepl sends this data to the cli (zfs/zpool), and once this transition completes, the call has been processed successfully.

All these changes can be seen in the cStor repository below (forked from zfs on Linux https://github.com/zfsonlinux/zfs).

https://github.com/openebs/cStor

Summary

The latest release of the OpenEBS ‘cStor’ storage engine CLI natively supports ZFS, combining the best technical features of ZFS storage on individual servers with the usability features of an OpenEBS’s Kubernetes storage mesh that can span on-premise and multiple cloud servers.

Please drop your valuable comments or feed backs in the comment section below. Interested in our White papers and Case studies? Click here

Prateek Pandey
Software Engineer at MayaData
Murat Karslioglu
VP @OpenEBS & @MayaData_Inc. Murat Karslioglu is a serial entrepreneur, technologist, and startup advisor with over 15 years of experience in storage, distributed systems, and enterprise hardware development. Prior to joining MayaData, Murat worked at Hewlett Packard Enterprise / 3PAR Storage in various advanced development projects including storage file stack performance optimization and the storage management stack for HPE’s Hyper-converged solution. Before joining HPE, Murat led virtualization and OpenStack integration projects within the Nexenta CTO Office. Murat holds a Bachelor’s Degree in Industrial Engineering from the Sakarya University, Turkey, as well as a number of IT certifications. When he is not in his lab, he loves to travel, advise startups, and spend time with his family. Lives to innovate! Opinions my own!
Evan Powell
Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO & Chairman of OpenEBS/MayaData. ML and DevOps and Python, oh my!