Building Kubernetes Operators Can be Simple

What is a Kubernetes operator?

A Kubernetes operator is nothing but an object that extends Kubernetes functionality using the atomic and modular nature of Kubernetes to create a controller that responds to a specific set of events and executes some custom actions as a result of that.

A Kubernetes operator usually works with Custom Resource Definitions(CRDs), resources that the User creates to solve complex business problems.

An operator can be thought of as an alternative to the human operator, someone who is aware of the technical details of the application and also knows the business requirements and acts accordingly.

Why Metacontroller?

Metacontroller is an add-on for Kubernetes that makes it easy to write and deploy custom controllers without the need to deal with Kubernetes itself.

Metacontroller has following advantages when implementing custom Kubernetes controllers over other SDKs:

• Developers do not need to learn & code against Kubernetes APIs. Metacontroller frees developers from learning Kubernetes API machinery and instead allows them to focus on solving problems in the application domain.
• No dependency on code generation. Metacontroller implements dynamic clientset & makes use of unstructured instances as its webhook payload. This, in turn, avoids the need to deal with code generation completely.
• Metacontroller aligns the development by focusing on writing idempotent code. This code is responsible for returning the desired state of resources as its response to the reconciliation request. In other words, logic does not need to handle CRUD operations against the resource and instead delegate the same to Metacontroller by just returning the desired states. Metacontroller, on its part, checks if these resources are new or need to be updated or perhaps need to be deleted.
• Metacontroller makes use of its own server-side apply logic. This logic follows convention over configuration. In other words, developers are not expected to provide merge hints in the CRD and yet expect their custom resources to exhibit merge results similar to that of Kubernetes native resources.
• The business logic for the operator can be written in any programming language since Metacontroller understands webhooks. This is one of its important features since the controller logic is now agnostic to any programming language. Take a look at the project examples to understand more about this.
  
  

How to write a Kubernetes operator using Metacontroller?

I have been using Metacontroller to write an operator for OpenEBS, which is responsible for deploying and managing various OpenEBS components, whether we want to change some configuration for some of the components or all of the components, their upgrade, etc.

We have made use of “Golang” as a programming language to write business logic. I will try to provide a glimpse of how easy it has been for me to develop this operator using Metacontroller, where I had to focus only on the business logic and a configuration file i.e., metac.yaml.

The steps involved in writing an operator using Metacontroller are as follows:

• Define the Custom Resource Definition and the custom resource schema, which will be watched by the operator.
• Import Metacontroller into the code where you are writing the business logic. Metacontroller is used as a library here. So no need to expose the business logic as webhook(s).
• Define metac.yaml config file, which contains the configuration for the GenericController (one of the supported meta controllers).
• Write the business logic that gets invoked in every reconcile.
• Deploy the operator as a Kubernetes StatefulSet.

1. Custom Resource Definition and Schema:
   

   Defining or creating an instance of the custom resource is a way to define the intent to the operator, which in turn reads and executes the required action in order to get to the desired state as defined in the custom resource.

   OpenEBS operator watches this resource to deploy or update the OpenEBS configuration.

   An example of OpenEBS Custom resource definition:

   apiVersion: apiextensions.k8s.io/v1beta1
   kind: CustomResourceDefinition
   metadata:
     name: openebses.dao.mayadata.io
   spec:
     group: dao.mayadata.io
     version: v1alpha1
     scope: Namespaced
     names:
       plural: openebses
       singular: openebs
       kind: OpenEBS
       shortNames:
       - openebs

   An example schema for OpenEBS CR can be found here, which defines all the possible configurations that can be provided to deploy/update OpenEBS components.

2. Metacontroller config (/config/metac.yaml)
   

   Metacontroller config defines various custom controllers that will run inside the operator. We define one config instance for each controller, all in the same metac.yaml file.

   A sample metac.yaml config for a custom controller could look like:

   apiVersion: metac.openebs.io/v1alpha1
   kind: GenericController
   metadata:
     name: sync-openebs
     namespace: openebs-operator
   spec:
     updateAny: true
     watch:
       apiVersion: dao.mayadata.io/v1alpha1
       resource: openebses
     attachments:
       - apiVersion: dao.mayadata.io/v1alpha1
         resource: openebses
       - apiVersion: apps/v1
         resource: daemonsets
         updateStrategy:
           method: InPlace
         advancedSelector:
           selectorTerms:
             - matchReferenceExpressions:
                 - key: metadata.annotations.dao\.mayadata\.io/openebs-uid
                   refKey: metadata.uid # match this ann value against watch UID
             - matchReferenceExpressions:
                 - key: metadata.annotations.openebs-upgrade\.dao\.mayadata\.io/openebs-uid
                   refKey: metadata.uid # match this ann value against watch UID
       - apiVersion: apps/v1
         resource: deployments
         updateStrategy:
           method: InPlace
         advancedSelector:
           selectorTerms:
             - matchReferenceExpressions:
                 - key: metadata.annotations.dao\.mayadata\.io/openebs-uid
                   refKey: metadata.uid # match this ann value against watch UID
             - matchReferenceExpressions:
                 - key: metadata.annotations.openebs-upgrade\.dao\.mayadata\.io/openebs-uid
                   refKey: metadata.uid # match this ann value against watch UID
     hooks:
       sync:
         inline:
           funcName: sync/openebs

This declarative specification means that your code never has to talk to the Kubernetes API, so you don’t need to import any Kubernetes client library, nor depend on any code provided by Kubernetes. You merely receive JSON describing the observed state of the resources and return JSON describing your desired state of the resources. Metacontroller handles all interactions with the Kubernetes API. It runs a level-triggered reconciliation loop.

Metacontroller config talks in terms of “watch” and “attachments”. Watch is the kind/resource that is under watch, while attachments are the resources that form the desired state based on this watch.

Following are the definition of the keywords found in the above config:

• updateAny tells the custom controller to update the attachments even if these were not created (i.e., owned) by this controller.
• updateStrategy defines the update strategy to be followed by the custom controller to update the attachments. InPlace, Recreate, RollingUpdate are some of the supported update strategies.
• advancedSelector is used to filter the attachments based on the needs of the controller. For example, the advancedSelector defined in the above config tells Metacontroller to filter out those deployments created by the particular watch instance, i.e., OpenEBS custom resource.

Inline function maps the function to be invoked to handle this controller’s reconciliation. Inline function is the one that defines the overall business logic & returns the desired state in response.

hooks:
  sync:
    inline:
      funcName: sync/openebs

A sample code snippet which registers this inline function from the controller main code is:

package main

import (
  "flag"

  "openebs.io/metac/controller/generic"
  "openebs.io/metac/start"

  "mayadata.io/openebs-upgrade/controller/openebs"
)

// main function is the entry point of this binary.
//
// This registers various controller (i.e. kubernetes reconciler)
// handler functions. Each handler function gets triggered due
// to any change (add, update or delete) to the configured watch.
func main() {
  flag.Set("logtostderr", "true")
  flag.Parse()

  generic.AddToInlineRegistry("sync/openebs", openebs.Sync)
  start.Start()
}

The line that interests us is the one that registers and maps the Go function openebs.Sync against the named inline hook defined in the config.

generic.AddToInlineRegistry("sync/openebs", openebs.Sync)

The function openebs.Sync does not care if the components have to be created or updated or deleted since this is taken care of by Metacontroller.

A sample code snippet for openebs.Sync function:

// Sync implements the idempotent logic to reconcile OpenEBS
//
// NOTE:
//     SyncHookRequest is the payload received as part of reconcile
// request. Similarly, SyncHookResponse is the payload sent as a
// response as part of reconcile request.
//
// NOTE:
// SyncHookRequest uses OpenEBS as the watched resource.
// SyncHookResponse has the resources that form the desired state
// w.r.t the watched resource.
//
// NOTE:
// Returning error will panic this process. We would rather want this
// controller to run continuously. Hence, the errors are logged and at
// the same time, and these errors are posted against OpenEBS's
// status.
func Sync(request *generic.SyncHookRequest, response *generic.SyncHookResponse) error {
  // Nothing needs to be done if there are no attachments in the request
  //
  // NOTE:
  //     It is expected to have OpenEBS as an attachment
  // resource as well as the resource under watch.
  if request.Attachments == nil || request.Attachments.IsEmpty() {
     response.SkipReconcile = true
     return nil
  }

  glog.V(3).Infof(
     "Will reconcile OpenEBS %s %s:",
     request.Watch.GetNamespace(), request.Watch.GetName(),
  )

  // construct the error handler
  errHandler := &reconcileErrHandler{
     openebs:      request.Watch,
     hookResponse: response,
  }

  var observedOpenEBS *unstructured.Unstructured
  var observedOpenEBSComponents []*unstructured.Unstructured
  for _, attachment := range request.Attachments.List() {
     // this watch resource must be present in the list of attachments
     if request.Watch.GetUID() == attachment.GetUID() &&
        attachment.GetKind() == string(types.KindOpenEBS) {
        // this is the required OpenEBS
        observedOpenEBS = attachment
        // add this to the response later after completion of its
        // reconcile logic
        continue
     }
     // If the attachments are not of Kind: OpenEBS then it will be
     // considered as an OpenEBS component.
     if attachment.GetKind() != string(types.KindOpenEBS) {
        observedOpenEBSComponents = append(observedOpenEBSComponents, attachment)
     }
  }

  if observedOpenEBS == nil {
     errHandler.handle(
        errors.Errorf("Can't reconcile: OpenEBS not found in attachments"),
     )
     return nil
  }

  // reconciler is the one that will perform reconciliation of
  // OpenEBS resource
  reconciler, err :=
     NewReconciler(
        ReconcilerConfig{
           ObservedOpenEBS:           observedOpenEBS,
           observedOpenEBSComponents: observedOpenEBSComponents,
        })
  if err != nil {
     errHandler.handle(err)
     return nil
  }
  resp, err := reconciler.Reconcile()
  if err != nil {
     errHandler.handle(err)
     return nil
  }
  // add all the desired OpenEBS components as attachments in the response
  if resp.DesiredOpenEBSComponets != nil {
     for _, desiredOpenEBSComponent := range resp.DesiredOpenEBSComponets {
        response.Attachments = append(response.Attachments, desiredOpenEBSComponent)
     }
  }

  glog.V(2).Infof(
     "OpenEBS %s %s reconciled successfully: %s",
     request.Watch.GetNamespace(), request.Watch.GetName(),
     metac.GetDetailsFromResponse(response),
  )

  // construct the success handler
  successHandler := reconcileSuccessHandler{
     openebs:      request.Watch,
     hookResponse: response,
  }
  successHandler.handle()

  return nil
}

Complete implementation of this operator can be found at Github.

Deploy the operator:

A sample operator YAML to deploy the above operator:

apiVersion: v1
kind: Namespace
metadata:
  name: openebs-operator

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: openebs-upgrade
  namespace: openebs-operator
spec:
  replicas: 1
  selector:
    matchLabels:
      name: openebs-upgrade
  template:
    metadata:
      labels:
        name: openebs-upgrade
    spec:
      containers:
      - name: openebs-upgrade
        image: sagarkrsd/openebs-upgrade:latest
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        command: ["/usr/bin/openebs-upgrade"]
        args:
        - --logtostderr
        - --run-as-local
        - -v=5
        - --discovery-interval=40s
        - --cache-flush-interval=240s
        resources:
      serviceAccountName: openebsupgrade

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: openebsupgrade
  namespace: openebs-operator

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: openebsupgrade
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: openebsupgrade
subjects:
- kind: ServiceAccount
  name: openebsupgrade
  namespace: openebs-operator
roleRef:
  kind: ClusterRole
  name: openebsupgrade
  apiGroup: rbac.authorization.k8s.io

Conclusion

It's relatively straightforward to create and deploy a custom Operator and custom Kubernetes types using Metacontroller.

Some Useful Links:

• https://github.com/AmitKumarDas/metac
• https://github.com/GoogleCloudPlatform/metacontroller
• https://metacontroller.app/