CLI Changes for OpenEBS 'cstor' storage engine

OpenEBS is the leading block storage solution on Kubernetes because of its Container Attached Storage (CAS) architecture. The latest release of the OpenEBS ‘cStor’ storage engine CLI natively supports ZFS, combining the best technical features of ZFS storage on individual servers with the usability features of the OpenEBS Kubernetes storage mesh that can span on-premise and multiple cloud servers.

CLI enhanced for ZFS

In this blog, we'll discuss how the latest release of the ‘cStor’ command-line interface (CLI) has been enhanced to support ZFS.

ZFS combines the features of a file system and volume manager. Uniquely, this means ZFS can host a file system spanning multiple drives (a pool). Capacity can be added to a pool by adding another drive. ZFS handles partitioning and formatting. ZFS features include:

  • Pooled storage
  • Scalability to 16 Exabyte file & 256 Quadrillion Zettabytes storage
  • Copy-on-write
  • Replication
  • Deduplication
  • Snapshots
  • Data integrity verification and automatic repair
  • RAID-Z

(Note: ZFS’ history and legal intricacies are complex and interesting, and it proves the singular power of a healthy open source community.) From an OpenEBS point of view, a key ZFS feature is that it ensures data is always consistent on the disk.

In cStor, we run ZFS in userspace which gives us more flexibility as there is no kernel dependency. There are several cStor changes we made in ZFS CLI to support running in userspace.

The existing model has userspace and kernel space architecture where the ZFS file system runs in kernel space, and zfs/zpool binaries run in userspace, and they interact via ioctl (system call). Since we run ZFS in userspace we can not use existing CLI (zfs/zpool) as those libraries directly interact with the kernel.

The existing zfs/zpool has been there for years and is feature-rich. To make these usable for cStor we introduced a Unix domain socket server and client architecture. The Unix domain server runs on the cStor side (zrepl binary) running the zfs filesystem in userspace, and the cli (zfs/zpool) runs as a client to this server. These two components make a socket call instead of an ioctl call which will route the request to the cStor instead of going to the kernel.

ZFS/ZPool and ZREPL

Once ZREPL gets the request, it will process this by calling an appropriate ioctl handler.

We have introduced a new -- enable-uzfs config option, through which you can switch to the Unix domain model from the ioctl model.

$ ./configure --enable-uzfs=yes

That is still not sufficient, so whenever a kernel space switch happens, the kernel does the copy-in of the data to copy the data from user space. This data should be sent from the zfs/zpool CLI to the Unix domain server (zrepl), and that copy-in call should be changed to socket read, similarly while returning to userspace, the kernel does the copy-out of the data, which now should be sent on the wire. So the zrepl sends this data to the cli (zfs/zpool), and once this transition completes, the call has been processed successfully.

All these changes can be seen in the cStor repository below (forked from zfs on Linux https://github.com/zfsonlinux/zfs).

https://github.com/openebs/cStor

Summary

The latest release of the OpenEBS ‘cStor’ storage engine CLI natively supports ZFS, combining the best technical features of ZFS storage on individual servers with the usability features of an OpenEBS’s Kubernetes storage mesh that can span on-premise and multiple cloud servers.

Please drop your valuable comments or feed backs in the comment section below. Interested in our White papers and Case studies? Click here

Utkarsh Mani Tripathi
Utkarsh is a maintainer of jiva project and has contributed in building both control and data plane of OpenEBS. He loves to learn about file-system, distributed systems and networking. Currently, he is mainly focusing on enhancing jiva and maya-exporter In his free time, he loves to write poems and make lip smacking dishes
Chuck Piercey
Chuck Piercey is a Silicon Valley product manager with experience shipping more than 15 products in several different market segments representing a total of $2.5Bn revenue under both commercial and open source business models. Most recently he has been working for MayaData, Inc. focused on software-defined storage, network, and compute for Kubernetes environments. Chuck occasionally writes articles about the technology industry.
Sagar Kumar
Sagar is a software engineer at Mayadata who loves coding and solving real-world problems. He has been playing with Kubernetes for the last couple of years. Currently, he is focused on building OpenEBS Director as the go-to solution for OpenEBS users. In his free time, he loves playing cricket and traveling.