CLI Changes for OpenEBS 'cstor' storage engine

OpenEBS is the leading block storage solution on Kubernetes because of its Container Attached Storage (CAS) architecture. The latest release of the OpenEBS ‘cStor’ storage engine CLI natively supports ZFS, combining the best technical features of ZFS storage on individual servers with the usability features of the OpenEBS Kubernetes storage mesh that can span on-premise and multiple cloud servers.

In this blog, we'll discuss how the latest release of the ‘cStor’ command-line interface (CLI) has been enhanced to support ZFS.

ZFS combines the features of a file system and volume manager. Uniquely, this means ZFS can host a file system spanning multiple drives (a pool). Capacity can be added to a pool by adding another drive. ZFS handles partitioning and formatting. ZFS features include:

• Pooled storage
• Scalability to 16 Exabyte file & 256 Quadrillion Zettabytes storage
• Copy-on-write
• Replication
• Deduplication
• Snapshots
• Data integrity verification and automatic repair
• RAID-Z

(Note: ZFS’ history and legal intricacies are complex and interesting, and it proves the singular power of a healthy open source community.) From an OpenEBS point of view, a key ZFS feature is that it ensures data is always consistent on the disk.

In cStor, we run ZFS in userspace which gives us more flexibility as there is no kernel dependency. There are several cStor changes we made in ZFS CLI to support running in userspace.

The existing model has userspace and kernel space architecture where the ZFS file system runs in kernel space, and zfs/zpool binaries run in userspace, and they interact via ioctl (system call). Since we run ZFS in userspace we can not use existing CLI (zfs/zpool) as those libraries directly interact with the kernel.

The existing zfs/zpool has been there for years and is feature-rich. To make these usable for cStor we introduced a Unix domain socket server and client architecture. The Unix domain server runs on the cStor side (zrepl binary) running the zfs filesystem in userspace, and the cli (zfs/zpool) runs as a client to this server. These two components make a socket call instead of an ioctl call which will route the request to the cStor instead of going to the kernel.

Once ZREPL gets the request, it will process this by calling an appropriate ioctl handler.

We have introduced a new -- enable-uzfs config option, through which you can switch to the Unix domain model from the ioctl model.

$ ./configure --enable-uzfs=yes

That is still not sufficient, so whenever a kernel space switch happens, the kernel does the copy-in of the data to copy the data from user space. This data should be sent from the zfs/zpool CLI to the Unix domain server (zrepl), and that copy-in call should be changed to socket read, similarly while returning to userspace, the kernel does the copy-out of the data, which now should be sent on the wire. So the zrepl sends this data to the cli (zfs/zpool), and once this transition completes, the call has been processed successfully.

All these changes can be seen in the cStor repository below (forked from zfs on Linux https://github.com/zfsonlinux/zfs).

https://github.com/openebs/cStor

Summary

The latest release of the OpenEBS ‘cStor’ storage engine CLI natively supports ZFS, combining the best technical features of ZFS storage on individual servers with the usability features of an OpenEBS’s Kubernetes storage mesh that can span on-premise and multiple cloud servers.

Please drop your valuable comments or feed backs in the comment section below. Interested in our White papers and Case studies? Click here