A Kubernetes operator is nothing but an object that extends Kubernetes functionality using the atomic and modular nature of Kubernetes to create a controller that responds to a specific set of events and executes some custom actions as a result of that.
A Kubernetes operator usually works with Custom Resource Definitions(CRDs), resources that the User creates to solve complex business problems.
An operator can be thought of as an alternative to the human operator, someone who is aware of the technical details of the application and also knows the business requirements and acts accordingly.
Metacontroller is an add-on for Kubernetes that makes it easy to write and deploy custom controllers without the need to deal with Kubernetes itself.
Metacontroller has following advantages when implementing custom Kubernetes controllers over other SDKs:
I have been using Metacontroller to write an operator for OpenEBS, which is responsible for deploying and managing various OpenEBS components, whether we want to change some configuration for some of the components or all of the components, their upgrade, etc.
We have made use of “Golang” as a programming language to write business logic. I will try to provide a glimpse of how easy it has been for me to develop this operator using Metacontroller, where I had to focus only on the business logic and a configuration file i.e., metac.yaml.
The steps involved in writing an operator using Metacontroller are as follows:
Defining or creating an instance of the custom resource is a way to define the intent to the operator, which in turn reads and executes the required action in order to get to the desired state as defined in the custom resource.
OpenEBS operator watches this resource to deploy or update the OpenEBS configuration.
An example of OpenEBS Custom resource definition:
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: openebses.dao.mayadata.io
spec:
group: dao.mayadata.io
version: v1alpha1
scope: Namespaced
names:
plural: openebses
singular: openebs
kind: OpenEBS
shortNames:
- openebsAn example schema for OpenEBS CR can be found here, which defines all the possible configurations that can be provided to deploy/update OpenEBS components.
Metacontroller config defines various custom controllers that will run inside the operator. We define one config instance for each controller, all in the same metac.yaml file.
A sample metac.yaml config for a custom controller could look like:
apiVersion: metac.openebs.io/v1alpha1
kind: GenericController
metadata:
name: sync-openebs
namespace: openebs-operator
spec:
updateAny: true
watch:
apiVersion: dao.mayadata.io/v1alpha1
resource: openebses
attachments:
- apiVersion: dao.mayadata.io/v1alpha1
resource: openebses
- apiVersion: apps/v1
resource: daemonsets
updateStrategy:
method: InPlace
advancedSelector:
selectorTerms:
- matchReferenceExpressions:
- key: metadata.annotations.dao\.mayadata\.io/openebs-uid
refKey: metadata.uid # match this ann value against watch UID
- matchReferenceExpressions:
- key: metadata.annotations.openebs-upgrade\.dao\.mayadata\.io/openebs-uid
refKey: metadata.uid # match this ann value against watch UID
- apiVersion: apps/v1
resource: deployments
updateStrategy:
method: InPlace
advancedSelector:
selectorTerms:
- matchReferenceExpressions:
- key: metadata.annotations.dao\.mayadata\.io/openebs-uid
refKey: metadata.uid # match this ann value against watch UID
- matchReferenceExpressions:
- key: metadata.annotations.openebs-upgrade\.dao\.mayadata\.io/openebs-uid
refKey: metadata.uid # match this ann value against watch UID
hooks:
sync:
inline:
funcName: sync/openebsThis declarative specification means that your code never has to talk to the Kubernetes API, so you don’t need to import any Kubernetes client library, nor depend on any code provided by Kubernetes. You merely receive JSON describing the observed state of the resources and return JSON describing your desired state of the resources. Metacontroller handles all interactions with the Kubernetes API. It runs a level-triggered reconciliation loop.
Metacontroller config talks in terms of “watch” and “attachments”. Watch is the kind/resource that is under watch, while attachments are the resources that form the desired state based on this watch.
Following are the definition of the keywords found in the above config:
Inline function maps the function to be invoked to handle this controller’s reconciliation. Inline function is the one that defines the overall business logic & returns the desired state in response.
hooks:
sync:
inline:
funcName: sync/openebsA sample code snippet which registers this inline function from the controller main code is:
package main
import (
"flag"
"openebs.io/metac/controller/generic"
"openebs.io/metac/start"
"mayadata.io/openebs-upgrade/controller/openebs"
)
// main function is the entry point of this binary.
//
// This registers various controller (i.e. kubernetes reconciler)
// handler functions. Each handler function gets triggered due
// to any change (add, update or delete) to the configured watch.
func main() {
flag.Set("logtostderr", "true")
flag.Parse()
generic.AddToInlineRegistry("sync/openebs", openebs.Sync)
start.Start()
}The line that interests us is the one that registers and maps the Go function openebs.Sync against the named inline hook defined in the config.
generic.AddToInlineRegistry("sync/openebs", openebs.Sync)
The function openebs.Sync does not care if the components have to be created or updated or deleted since this is taken care of by Metacontroller.
A sample code snippet for openebs.Sync function:
// Sync implements the idempotent logic to reconcile OpenEBS
//
// NOTE:
// SyncHookRequest is the payload received as part of reconcile
// request. Similarly, SyncHookResponse is the payload sent as a
// response as part of reconcile request.
//
// NOTE:
// SyncHookRequest uses OpenEBS as the watched resource.
// SyncHookResponse has the resources that form the desired state
// w.r.t the watched resource.
//
// NOTE:
// Returning error will panic this process. We would rather want this
// controller to run continuously. Hence, the errors are logged and at
// the same time, and these errors are posted against OpenEBS's
// status.
func Sync(request *generic.SyncHookRequest, response *generic.SyncHookResponse) error {
// Nothing needs to be done if there are no attachments in the request
//
// NOTE:
// It is expected to have OpenEBS as an attachment
// resource as well as the resource under watch.
if request.Attachments == nil || request.Attachments.IsEmpty() {
response.SkipReconcile = true
return nil
}
glog.V(3).Infof(
"Will reconcile OpenEBS %s %s:",
request.Watch.GetNamespace(), request.Watch.GetName(),
)
// construct the error handler
errHandler := &reconcileErrHandler{
openebs: request.Watch,
hookResponse: response,
}
var observedOpenEBS *unstructured.Unstructured
var observedOpenEBSComponents []*unstructured.Unstructured
for _, attachment := range request.Attachments.List() {
// this watch resource must be present in the list of attachments
if request.Watch.GetUID() == attachment.GetUID() &&
attachment.GetKind() == string(types.KindOpenEBS) {
// this is the required OpenEBS
observedOpenEBS = attachment
// add this to the response later after completion of its
// reconcile logic
continue
}
// If the attachments are not of Kind: OpenEBS then it will be
// considered as an OpenEBS component.
if attachment.GetKind() != string(types.KindOpenEBS) {
observedOpenEBSComponents = append(observedOpenEBSComponents, attachment)
}
}
if observedOpenEBS == nil {
errHandler.handle(
errors.Errorf("Can't reconcile: OpenEBS not found in attachments"),
)
return nil
}
// reconciler is the one that will perform reconciliation of
// OpenEBS resource
reconciler, err :=
NewReconciler(
ReconcilerConfig{
ObservedOpenEBS: observedOpenEBS,
observedOpenEBSComponents: observedOpenEBSComponents,
})
if err != nil {
errHandler.handle(err)
return nil
}
resp, err := reconciler.Reconcile()
if err != nil {
errHandler.handle(err)
return nil
}
// add all the desired OpenEBS components as attachments in the response
if resp.DesiredOpenEBSComponets != nil {
for _, desiredOpenEBSComponent := range resp.DesiredOpenEBSComponets {
response.Attachments = append(response.Attachments, desiredOpenEBSComponent)
}
}
glog.V(2).Infof(
"OpenEBS %s %s reconciled successfully: %s",
request.Watch.GetNamespace(), request.Watch.GetName(),
metac.GetDetailsFromResponse(response),
)
// construct the success handler
successHandler := reconcileSuccessHandler{
openebs: request.Watch,
hookResponse: response,
}
successHandler.handle()
return nil
}Complete implementation of this operator can be found at Github.
A sample operator YAML to deploy the above operator:
apiVersion: v1
kind: Namespace
metadata:
name: openebs-operator
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: openebs-upgrade
namespace: openebs-operator
spec:
replicas: 1
selector:
matchLabels:
name: openebs-upgrade
template:
metadata:
labels:
name: openebs-upgrade
spec:
containers:
- name: openebs-upgrade
image: sagarkrsd/openebs-upgrade:latest
imagePullPolicy: Always
ports:
- containerPort: 8080
command: ["/usr/bin/openebs-upgrade"]
args:
- --logtostderr
- --run-as-local
- -v=5
- --discovery-interval=40s
- --cache-flush-interval=240s
resources:
serviceAccountName: openebsupgrade
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: openebsupgrade
namespace: openebs-operator
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: openebsupgrade
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: openebsupgrade
subjects:
- kind: ServiceAccount
name: openebsupgrade
namespace: openebs-operator
roleRef:
kind: ClusterRole
name: openebsupgrade
apiGroup: rbac.authorization.k8s.ioIt's relatively straightforward to create and deploy a custom Operator and custom Kubernetes types using Metacontroller.