Static Pods in Kubernetes

I'm an engineer at MayaData, a company dedicated to enabling data agility through the use of Kubernetes as a data layer and the sponsor of the CNCF project OpenEBS and other open source projects. Take it for a test drive via free and easy to use management software by registering here.

If you are involved in the Kubernetes world, you know that the very first thing we encounter is the concept of the pod. If you are not familiar with what a Kubernetes Pod is, I would recommend to first check out the Kubernetes docs:
https://kubernetes.io/docs/concepts/workloads/pods/pod/

Static Pod in Kubernetes

This tutorial assumes that you have a basic understanding of Kubernetes and how pods work. Additionally, you should have access to a Kubernetes cluster to do the lab described here. I will perform this lab on a 3 node GKE cluster and try to create a Static Pod. The nodes are on Ubuntu 16.04, with systemd as an init system used to bootstrap and manage the Kubernetes components.
Now, let us jump right in and learn exactly what a static pod is and how we can create it!

Static pods are a type of pod that is not observed or managed via kube-apiserver and is directly bound to the Kubelet daemon on the node. We will do some more analysis on the characteristics of this kind of pod at the end of this tutorial. Before that, let us first create the static pod using the following steps.

1. First, log onto one of your worker nodes.
2. On that same node, create a pod manifest yaml by executing the following command:
   

   ashutosh_kumar@gke-cstor-it-default-pool-e84f5225-083w:~$ kubectl run static-pod --image=sonasingh46/node-web-app:latest --generator=run-pod/v1 --dry-run -o yaml > static.yaml

3. Execute the following command on the same node to locate the kubelet systemd service file.
   

   ashutosh_kumar@gke-cstor-it-default-pool-e84f5225-083w:~$ sudo systemctl status kubelet

   
   
   

   ● kubelet.service - Kubernetes kubelet Loaded: loaded (/etc/systemd/system/kubelet.service; disabled; vendor preset: enabled) Active: active (running) since Thu 2018-12-06 19:14:32 UTC; 2min 25s ago Main PID: 2551 (kubelet) Tasks: 16 Memory: 56.0M CPU: 8.278s CGroup: /system.slice/kubelet.service └─2551 /home/kubernetes/bin/kubelet --v=2 --allow-privileged=true --cloud-provider=gce --experimental-mounter-path=/home/kubernetes/containerized_mounter/m ...

4. Edit the service file by typing vim /etc/systemd/system/kubelet.service and provide the path of the pod manifest that we created to the flag (add the flag if not present) --pod-manifest-path and save it. Note that the value of the flag is the address of the directory where we have kept our pod manifest yaml. After editing, my kubelet service file will look like this:
   

   [Unit] Description=Kubernetes kubelet Requires=network-online.target After=network-online.target [Service] Restart=always RestartSec=10 EnvironmentFile=/etc/default/kubelet ExecStart=/home/kubernetes/bin/kubelet $KUBELET_OPTS \ --pod-manifest-path=/home/ashutosh_kumar [Install] WantedBy=multi-user.target

5. Run the following commands:
   
   

   { sudo systemctl daemon-reload    sudo systemctl restart kubelet }

That’s it! We have now created a static pod! In just a few seconds you will see a pod spawn on the node. To view it, run kubectl get pod from the machine where your kubeconfig is configured.

Try deleting the pod, and you’ll find that you cannot!

You can add as many pod manifest yaml’s to the path as you want that you have provided to the pod-manifest-path flag. As many manifests as you have, that is how many pods Kubelet daemon will create. Actually, Kubelet daemon keeps watching the provided path constantly, and addition and removal of a pod manifest yaml will create and delete static pods, respectively.

Now, if you delete the pod manifest yaml that was created, soon you will see that the pod disappears. Before removing this manifest yaml, let us do one more experiment. At the end of the lab we can remove the manifest yaml to observe the static pod deletion.

If you cordon the node on which the static pod is running, it will not be evicted, as static pods are not scheduled by kube-scheduler and hence are ignored. Run the following command to cordon the node on which your static pod is running:

You will see that static pod is still in the running state.

At this point, if you are familiar with DaemonSet in Kubernetes, you probably see some similarity between a static pod and a DaemonSet pod. Normally, the node which runs a pod is selected by kube-scheduler, but the pods of the DaemonSet controller have their node already selected by setting the node name on the pod spec.nodeName field. Hence DaemonSet pods are also ignored by the kube-scheduler.

If you want to learn more about DaemonSet in Kubernetes, feel free to go through the following link:

https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/

In the case of static pods, the narrative is similar, but the node name is given to the pod by the kubelet daemon.

To learn more about static pods, visit the following link:
https://kubernetes.io/docs/tasks/administer-cluster/static-pod/

Summary:

1. Pods scheduled by kube-scheduler will be evicted in cases where the node is drained or goes down.
2. The un-schedulable field of a node is not respected by the DaemonSet controller or static pods. Static pods and pods managed by DaemonSet controllers are therefore not evicted in case of a node drain.
3. Static pods and pods managed by DaemonSet controllers are not scheduled by the kube-scheduler. One important point to note here is that whenever the spec.nodeName field of the pod is set already (which is the case with DaemonSet pods and static pods), the pod is ignored by the kube-scheduler.
4. In fact, you can create a pod by specifying the node name of your choice in the spec.nodeName field of your pod manifest. It will be ignored by the kube-scheduler, but it is somewhat different from a static pod as you will be able to delete this pod using kubectl delete.
5. You cannot delete the static pod by requesting it to kube-apiserver, i.e. kubectl delete will not delete the static pod.
6. Kubelet creates the mirror pod on the kube-apiserver in case of static pod creation, but this is only for visibility and cannot be controlled from kube-apiserver.

I hope you enjoyed this tutorial! Please feel free to comment or offer suggestions or corrections if any or if there are any crucial point that I missed.

Thank you folks, and see you in the next post!

This article was first published on Dec 7, 2018 on MayaData's Medium Account