Static Pods in Kubernetes

If you are involved in the Kubernetes world, you know that the very first thing we encounter is the concept of the pod. If you are not familiar with what a Kubernetes Pod is, I would recommend to first check out the Kubernetes docs:
https://kubernetes.io/docs/concepts/workloads/pods/pod/

Static Pod in KubernetesStatic Pod in Kubernetes

This tutorial assumes that you have a basic understanding of Kubernetes and how pods work. Additionally, you should have access to a Kubernetes cluster to do the lab described here. I will perform this lab on a 3 node GKE cluster and try to create a Static Pod. The nodes are on Ubuntu 16.04, with systemd as an init system used to bootstrap and manage the Kubernetes components.
Now, let us jump right in and learn exactly what a static pod is and how we can create it!

Static pods are a type of pod that is not observed or managed via kube-apiserver and is directly bound to the Kubelet daemon on the node. We will do some more analysis on the characteristics of this kind of pod at the end of this tutorial. Before that, let us first create the static pod using the following steps.

  1. First, log onto one of your worker nodes.
  2. On that same node, create a pod manifest yaml by executing the following command:
    ashutosh_kumar@gke-cstor-it-default-pool-e84f5225-083w:~$ kubectl run static-pod --image=sonasingh46/node-web-app:latest --generator=run-pod/v1 --dry-run -o yaml > static.yaml
  3. Execute the following command on the same node to locate the kubelet systemd service file.
    ashutosh_kumar@gke-cstor-it-default-pool-e84f5225-083w:~$ sudo systemctl status kubelet


    ● kubelet.service - Kubernetes kubelet
    Loaded: loaded (/etc/systemd/system/kubelet.service; disabled; vendor preset: enabled)
    Active: active (running) since Thu 2018-12-06 19:14:32 UTC; 2min 25s ago
    Main PID: 2551 (kubelet)
    Tasks: 16
    Memory: 56.0M
    CPU: 8.278s
    CGroup: /system.slice/kubelet.service
    └─2551 /home/kubernetes/bin/kubelet --v=2 --allow-privileged=true --cloud-provider=gce --experimental-mounter-path=/home/kubernetes/containerized_mounter/m
    ...
  4. Edit the service file by typing vim /etc/systemd/system/kubelet.service and provide the path of the pod manifest that we created to the flag (add the flag if not present) --pod-manifest-path and save it. Note that the value of the flag is the address of the directory where we have kept our pod manifest yaml. After editing, my kubelet service file will look like this:
    [Unit]
    Description=Kubernetes kubelet
    Requires=network-online.target
    After=network-online.target

    [Service]
    Restart=always
    RestartSec=10
    EnvironmentFile=/etc/default/kubelet
    ExecStart=/home/kubernetes/bin/kubelet $KUBELET_OPTS \
    --pod-manifest-path=/home/ashutosh_kumar

    [Install]
    WantedBy=multi-user.target
  5. Run the following commands:

    { sudo systemctl daemon-reload
       sudo systemctl restart kubelet
    }

That’s it! We have now created a static pod! In just a few seconds you will see a pod spawn on the node. To view it, run kubectl get pod from the machine where your kubeconfig is configured.

ashutosh@miracle:~$ kubectl get pod
NAME READY STATUS RESTARTS AGE
static-pod-gke-cstor-it-default-pool-e84f5225-083w 1/1 Running 0 54s

Try deleting the pod, and you’ll find that you cannot!

You can add as many pod manifest yaml’s to the path as you want that you have provided to the pod-manifest-path flag. As many manifests as you have, that is how many pods Kubelet daemon will create. Actually, Kubelet daemon keeps watching the provided path constantly, and addition and removal of a pod manifest yaml will create and delete static pods, respectively.

Now, if you delete the pod manifest yaml that was created, soon you will see that the pod disappears. Before removing this manifest yaml, let us do one more experiment. At the end of the lab we can remove the manifest yaml to observe the static pod deletion.

If you cordon the node on which the static pod is running, it will not be evicted, as static pods are not scheduled by kube-scheduler and hence are ignored. Run the following command to cordon the node on which your static pod is running:

kubectl drain gke-cstor-it-default-pool-e84f5225-083w --ignore-daemonsets

You will see that static pod is still in the running state.

At this point, if you are familiar with DaemonSet in Kubernetes, you probably see some similarity between a static pod and a DaemonSet pod. Normally, the node which runs a pod is selected by kube-scheduler, but the pods of the DaemonSet controller have their node already selected by setting the node name on the pod spec.nodeName field. Hence DaemonSet pods are also ignored by the kube-scheduler.

If you want to learn more about DaemonSet in Kubernetes, feel free to go through the following link:

https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

In the case of static pods, the narrative is similar, but the node name is given to the pod by the kubelet daemon.

To learn more about static pods, visit the following link:
https://kubernetes.io/docs/tasks/administer-cluster/static-pod/

Summary:

  1. Pods scheduled by kube-scheduler will be evicted in cases where the node is drained or goes down.
  2. The un-schedulable field of a node is not respected by the DaemonSet controller or static pods. Static pods and pods managed by DaemonSet controllers are therefore not evicted in case of a node drain.
  3. Static pods and pods managed by DaemonSet controllers are not scheduled by the kube-scheduler. One important point to note here is that whenever the spec.nodeName field of the pod is set already (which is the case with DaemonSet pods and static pods), the pod is ignored by the kube-scheduler.
  4. In fact, you can create a pod by specifying the node name of your choice in the spec.nodeName field of your pod manifest. It will be ignored by the kube-scheduler, but it is somewhat different from a static pod as you will be able to delete this pod using kubectl delete.
  5. You cannot delete the static pod by requesting it to kube-apiserver, i.e. kubectl delete will not delete the static pod.
  6. Kubelet creates the mirror pod on the kube-apiserver in case of static pod creation, but this is only for visibility and cannot be controlled from kube-apiserver.

I hope you enjoyed this tutorial! Please feel free to comment or offer suggestions or corrections if any or if there are any crucial point that I missed.

Thank you folks, and see you in the next post!


This article was first published on Dec 7, 2018 on MayaData's Medium Account

 

Utkarsh Mani Tripathi
Utkarsh is a maintainer of jiva project and has contributed in building both control and data plane of OpenEBS. He loves to learn about file-system, distributed systems and networking. Currently, he is mainly focusing on enhancing jiva and maya-exporter In his free time, he loves to write poems and make lip smacking dishes
Chuck Piercey
Chuck Piercey is a Silicon Valley product manager with experience shipping more than 15 products in several different market segments representing a total of $2.5Bn revenue under both commercial and open source business models. Most recently he has been working for MayaData, Inc. focused on software-defined storage, network, and compute for Kubernetes environments. Chuck occasionally writes articles about the technology industry.
Evan Powell
Founding CEO of a few companies including StackStorm (BRCD) and Nexenta — and CEO &Chairman of OpenEBS / MayaData. ML and DevOps and Python, oh my!