Managing Ephemeral Storage on Kubernetes with OpenEBS

Kubernetes uses volume abstractions to attach containers and pods as storage for data management. The types of volumes in Kubernetes is usually categorized as - 

• Persistent volumes that have their lifetimes independent of pods, and enable the provisioning of storage for applications that need data to persist between applications. 
• Ephemeral volumes are designed for applications that do not require persistent data. 

In this article, we delve into ephemeral storage options, types and managing options while exploring how OpenEBS helps provision ephemeral storage for Kubernetes workloads. 

How Kubernetes handles Ephemeral Storage

Some pods host transient applications - those that require data storage but do not require the data to persist across container restarts. These include applications that extract read-only input data in files such as secret keys and configuration data. Cache services typically move non-crucial data into limited memory storage without impacting performance, meaning the volume attached to their pods only need to last the pod’s lifetime. 

Kubernetes uses ephemeral volumes to handle the storage needs of these transient pods. Ephemeral volumes allow pods to be started and stopped without being limited to the location of persistent volume. Kubernetes nodes have local ephemeral storage which is attached to the RAM or local writable devices. Pods typically use ephemeral volumes for caching, scratch space and logs. 

Let us understand this deeper by learning how Kubernetes manages ephemeral storage. 

Ephemeral Storage Options

Ephemeral storage is hosted in an unstructured volume that is shared between the container runtime, the operating system, and all pods running in the node. The volumes prevent these entities from using the node’s local storage excessively. Ephemeral storage is always hosted on the local storage’s primary partition. There are two ways of creating this partition:

Root

The root storage directory is shared between the OS, user pods and Kubernetes system daemons. The root partition holds the /var/log/ directory and the kubelet root directory, which is /var/lib/kubelet/ by default. Pods can consume this partition using container-writeable layers, container logs, image layers and EmptyDir volumes. The kubelet service administers and manages isolation and shared access of the root partition. As the root partition is ephemeral, it does not guarantee durability, disk IOPS and other performance metrics. 

Runtime

The runtime partition is an optional ephemeral storage that container runtimes can use to implement overlay file systems. When implementing isolation for the partition, runtime then provides shared access. This partition stores container-writable layers and image layers. Once the runtime partition is created, these layers are written onto it by default and not the root partition. 

Types of Ephemeral Volumes in Kubernetes

Kubernetes supports multiple types of ephemeral volumes depending on the intended purpose. These include:

Generic ephemeral volumes

These volumes can be provisioned using any storage driver that supports the dynamic provisioning of persistent volumes. These volumes provide a pod-level directory for scratch data that is discarded after provisioning. Some features of generic ephemeral volumes include: 

• Supports both local and network-attached storage
• Supports size restrictions for pods
• Depending on parameters and the storage driver used, these volumes may contain some initial data
• Depending on the storage driver used, generic ephemeral volumes may implement snapshotting, cloning, storage capacity tracking and resizing

CSI ephemeral volumes

These volumes are provisioned by third-party Container Storage Interface (CSI) drivers. If the driver is written for ephemeral volumes that support dynamic provisioning, the CSI driver can also provision generic volumes. These third-party volumes offer numerous benefits as they can offer functionality that are not typically supported by Kubernetes. CSI ephemeral volumes are created together with other resources when a pod is scheduled, and is managed locally on the node.

Kubernetes also supports case-specific ephemeral volumes such as those that inject Kubernetes data into a pod. These include configMap, secret and downwardAPI volumes. The emptyDir volume is an ephemeral storage that comes locally within the Kubelet directory, and is always empty when the pod starts up. 

Managing Ephemeral Storage

Once pods have been scheduled, the scheduler ensures that resource requests do not exceed the maximum allocated ephemeral storage for each node. The kubelet service manages local ephemeral storage as a Kubernetes resource, and monitors resource usage. While doing so, kubelet scans for the use of storage in directories holding node logs, writable container layers and emptyDir volumes. If the pod consumes more storage above the resource limits, the kubelet triggers its eviction.

The kubelet also marks a container’s writable layer and log usage if it exceeds the resource limit, then evicts the pod hosting the container. For pod-level isolation, the kubelet compares the total storage resource usage plus emptyDir volumes against the total resource limits. If total consumption exceeds total resource limits, the pod is marked for eviction. 

This allows cluster administrators to manage ephemeral storage for applications by setting resource quotas while limiting the number of requests and storage ranges for active pods.

Provisioning Generic Ephemeral Volumes 

The YAML specification for a pod attached to a generic ephemeral volume with 2Gi storage, with multiple access modes would look similar to:

Provisioning CSI Ephemeral Volumes

When compared to generic provisioning, the pod manifest that uses CSI ephemeral storage would look similar to:

kind:Pod
apiVersion: v1
metadata:
  name:darwin-csi-app
spec:
  containers:
- name:my-frontend
      image: busybox
      volumeMounts:
- mountPath:"/data"
        name:darwin-csi-inline-vol
      command:["sleep","1000000"]
  volumes:(1)
- name:darwin-csi-inline-vol
      csi:
        driver:inline.storage.kubernetes.io
        volumeAttributes:
          foo: bar

The configuration file above includes mountPath and driver specifications that point to the CSI driver used to implement ephemeral storage.

Monitoring ephemeral volumes

It is possible to set up a monitoring tool to monitor storage usage on the volumes where containers store their data. This volume is typically located in /var/lib/docker or /var/lib/origin. The /bin/df tool is one such tool that can be used to check resource usage on these disks. Cluster administrators can use the df -h command to show human-readable values of available and used storage in these locations. For instance, to check resource consumption on /var/lib:

How OpenEBS helps manage ephemeral storage

OpenEBS sets the ephemeral storage requests from its components against each pod container and sidecar. This feature is enabled by default in Kubernetes 1.13 +, and can be set by enabling the feature-gate flag in earlier versions. This value is added to all components in the OpenEBS control plane in the YAML spec of the OpenEBS operator before it is applied to the cluster. Once administrators specify ephemeral storage with this approach, transient pods will be evicted gracefully by Kubernetes.

An example is the AuxResourceRequests setting that allows administrators to specify minimum ephemeral storage requests for container sidecars. The YAML spec below sets sidecars with an ephemeral storage request limit of 50Mi. 

- name:  AuxResourceRequests
  value: |-
      ephemeral-storage: "50Mi"

Conclusion:

Ephemeral storage is a crucial component for Kubernetes applications that process transient data. Kubernetes enables ephemeral volumes to enable transient pods to stop and restart gracefully regardless of the Persistent Volume’s location. Every Kubernetes node has ephemeral storage that is locally attached to writable devices or RAM. Pods can use this storage for caching, logs and scratch space. 

OpenEBS helps automate the management of ephemeral storage by enabling administrators to set storage requests against sidecars and containers. This setting helps avoid the erroneous eviction of transient pods from the cluster. 

To know more about how OpenEBS can help your organization simplify the provisioning of storage for Kubernetes applications, drop us a message here.