CLI Changes for OpenEBS 'cstor' storage engine

OpenEBS is the leading block storage solution on Kubernetes because of its Container Attached Storage (CAS) architecture. The latest release of the OpenEBS ‘cStor’ storage engine CLI natively supports ZFS, combining the best technical features of ZFS storage on individual servers with the usability features of the OpenEBS Kubernetes storage mesh that can span on-premise and multiple cloud servers.

CLI enhanced for ZFS

In this blog, we'll discuss how the latest release of the ‘cStor’ command-line interface (CLI) has been enhanced to support ZFS.

ZFS combines the features of a file system and volume manager. Uniquely, this means ZFS can host a file system spanning multiple drives (a pool). Capacity can be added to a pool by adding another drive. ZFS handles partitioning and formatting. ZFS features include:

  • Pooled storage
  • Scalability to 16 Exabyte file & 256 Quadrillion Zettabytes storage
  • Copy-on-write
  • Replication
  • Deduplication
  • Snapshots
  • Data integrity verification and automatic repair
  • RAID-Z

(Note: ZFS’ history and legal intricacies are complex and interesting, and it proves the singular power of a healthy open source community.) From an OpenEBS point of view, a key ZFS feature is that it ensures data is always consistent on the disk.

In cStor, we run ZFS in userspace which gives us more flexibility as there is no kernel dependency. There are several cStor changes we made in ZFS CLI to support running in userspace.

The existing model has userspace and kernel space architecture where the ZFS file system runs in kernel space, and zfs/zpool binaries run in userspace, and they interact via ioctl (system call). Since we run ZFS in userspace we can not use existing CLI (zfs/zpool) as those libraries directly interact with the kernel.

The existing zfs/zpool has been there for years and is feature-rich. To make these usable for cStor we introduced a Unix domain socket server and client architecture. The Unix domain server runs on the cStor side (zrepl binary) running the zfs filesystem in userspace, and the cli (zfs/zpool) runs as a client to this server. These two components make a socket call instead of an ioctl call which will route the request to the cStor instead of going to the kernel.

ZFS/ZPool and ZREPL

Once ZREPL gets the request, it will process this by calling an appropriate ioctl handler.

We have introduced a new -- enable-uzfs config option, through which you can switch to the Unix domain model from the ioctl model.

$ ./configure --enable-uzfs=yes

That is still not sufficient, so whenever a kernel space switch happens, the kernel does the copy-in of the data to copy the data from user space. This data should be sent from the zfs/zpool CLI to the Unix domain server (zrepl), and that copy-in call should be changed to socket read, similarly while returning to userspace, the kernel does the copy-out of the data, which now should be sent on the wire. So the zrepl sends this data to the cli (zfs/zpool), and once this transition completes, the call has been processed successfully.

All these changes can be seen in the cStor repository below (forked from zfs on Linux https://github.com/zfsonlinux/zfs).

https://github.com/openebs/cStor

Summary

The latest release of the OpenEBS ‘cStor’ storage engine CLI natively supports ZFS, combining the best technical features of ZFS storage on individual servers with the usability features of an OpenEBS’s Kubernetes storage mesh that can span on-premise and multiple cloud servers.

Please drop your valuable comments or feed backs in the comment section below. Interested in our White papers and Case studies? Click here

Jeffry Molanus
Jeffry prior to being CTO at MayaData has worked at several other startups in the storage industry. He worked on several scale out object storage products as well as traditional NAS and SAN storage solutions where he held technical leadership roles. At MayaData, his primary focus will be around making sure the product is flexible and scalable yet robust enough to be integrated seamlessly into modern day infrastructure where he believes, containers will have a dominant role. Jeffry holds a master degree in electrical engineering with a focus on distributed control engineering from the University Twente in the Netherlands. When he is not working with code, he practices martial arts.
Saumya Sharma
I am a Software Engineer turned Digital Marketer. Being from a Computer Science background, Software Engineering never thrilled me. I believe learning is a continuous process, and it's never too late to start a new phase of life. I am currently working on SEO and blog setups. In my free time, I love to read and write. I have a blog named The Inked Perception, where I post my poetry. I also love to travel and binge-watch web series.
Pawan Sharma
It's been an amazing experience in Software Engineering because of my love for coding. I have a very good system understanding and I enjoy solving puzzles. In my free time, I read books, play table tennis and watch tv series