Backward compatibility in Kubernetes operators can be an afterthought

I'm an engineer at MayaData, a company dedicated to enabling data agility through the use of Kubernetes as a data layer and the sponsor of the CNCF project OpenEBS and other open source projects. Take it for a test drive via free and easy to use management software by registering here.

Overview - Backward compatibility in Kubernetes operators

There are scenarios when a Kubernetes operators can be used to manage multiple Kubernetes clusters. OpenEBS Director is one such multitenant visualization and management software for OpenEBS storage. OpenEBS Director manages multiple OpenEBS clusters from a single instance. It comes with two flavors SaaS and On-Premise. In the SaaS version, Kubernetes clusters can be connected to Director online. However, if SaaS is not an option, these clusters can be connected to an on-premise Director installation.

OpenEBS Director being a multitenant software the connected clusters can be expected to have different versions of OpenEBS. Managing multiple versions of OpenEBS is challenging due to the differences in the Kubernetes resources that are supported.  Once  the Director is upgraded, its agents get upgraded automatically in all its connected clusters. These upgraded agents are now supposed to act based on the version of OpenEBS that the Kubernetes cluster is using.

We shall dig deep on this problem statement from the perspective of CStor. CStor is one of the OpenEBS data engines. CStor is popular amongst workloads that need efficient capacity management, replicas, point in time snapshots, incremental backups, etc. Its pool management is operated via an operator that works based on a custom resource named CStorPoolCluster (CSPC). The CSPC API version was v1alpha1 in OpenEBS version 1.9. However, with OpenEBS 1.10, the CSPC API version is now v1.

CStorClusterConfig is another custom resource that operates on CSPC and manages the advanced life cycles (i.e., day-0, day-1, day-2 operations) of CStor pools. Director in turn relies on CStorClusterConfig to manage CStor pools. Note that the Kubernetes controllers that watch for CStorClusterConfig are available in CstorPoolAuto repo.

Since CStorClusterConfig is used by Director it needs to maintain backward compatibility with multiple versions of OpenEBS to manage CStor pools. CstorPoolAuto uses the Metacontroller SDK under the hood to implement Kubernetes reconciliation. One of the advantages of using Metacontroller is its ability to handle backward compatibility. Metacontroller allows you to write the Kubernetes controller(s) in a declarative way. We need to write controller specification(s) declaratively, and corresponding instances will be created & run by Metacontroller using these specifications. Hence, the term “meta,” 

We used labelSelector which is one of the declarative controller features from Metacontroller to achieve backward compatibility. This feature is a simple way to let a controller manage specific instances of a custom resource. In our case, we have CStorClusterConfig custom resource which manages the life cycle of CStor pools. We had to create two declarative controllers, one to manage v1 CStorPoolCluster and another to manage v1alpha1 CStorPoolCluster via CStorClusterConfig.

For older custom resources we do not have any label with respect to CStorPoolCluster version so we use below configuration to manage v1alpha1 CStorPoolCluster.

labelSelector:
  matchExpressions:
    - key: cspc.openebs.io/version
      operator: DoesNotExist

To manage v1 CStorPoolCluster we introduced cspc.openebs.io/version  as a new label  in CStorClusterConfig that was set with value v1. In other words to manage v1 CStorPoolCluster, CStorClusterConfig should contain cspc.openebs.io/version=v1 label. 

Following is the controller configuration to manage v1 CStorPoolCluster:

labelSelector:
  matchLabels:
    cspc.openebs.io/version: v1

With these controller configurations if CStorClusterConfig contains cspc.openebs.io/version=v1 label then it is handled by the controller which is assigned to manage v1 CStorPoolClutser. If CStorClusterConfig does not contain cspc.openebs.io/version label then it goes to the controller which should manage v1alpha1 CStorPoolClutser.

This was the sample Metacontroller config used earlier

apiVersion: metac.openebs.io/v1alpha1
kind: GenericController
spec:
  watch:
    apiVersion: dao.mayadata.io/v1alpha1
    resource: cstorclusterconfigs

Following is the updated Metacontroller specification that supports both v1alpha1 as well as v1 versions of CStorPoolCluster.

Controller to support v1alpha1

apiVersion: metac.openebs.io/v1alpha1
kind: GenericController
spec:
  watch:
    apiVersion: dao.mayadata.io/v1alpha1
    resource: cstorclusterconfigs
    labelSelector:
      matchExpressions:
        - key: cspc.openebs.io/version
          operator: DoesNotExist

Controller to support v1

apiVersion: metac.openebs.io/v1alpha1
kind: GenericController
spec:
  watch:
    apiVersion: dao.mayadata.io/v1alpha1
    resource: cstorclusterconfigs
    labelSelector:
      matchLabels:
        cspc.openebs.io/version: v1