Install Kubernetes

When to use it

To build a Kubernetes cluster on servers you already run, the way the Kubernetes documentation sets one up with kubeadm. Run it once with Control plane on the server that will manage the cluster, and with Worker node on every server that will run workloads. The Kubernetes Node Setup template updates the server's packages first and then runs this playbook.

Supported on Ubuntu 22.04 and later, Debian 12 and later, and RHEL-compatible 9 and later, on x86_64 and aarch64. Kubernetes 1.35 and later run only on cgroup v2, which rules out RHEL 8 and Ubuntu 20.04.

Choosing the options

Kubernetes version — the three minor versions the Kubernetes project supports: 1.37 (the newest), 1.36 and 1.35. Every server in a cluster should run the same one. Choose 1.37 for a new cluster.

Role

• Worker node, ready to join a cluster — installs everything a node needs and stops before joining. Any server size works.
• Control plane, starting a new cluster — also starts the cluster with kubeadm init and installs the pod network. Needs at least 2 CPUs and 2 GB of memory.

Pod network — for a control plane:

• Flannel — the simplest pod network. Choose it unless you need network policies.
• Calico — also enforces Kubernetes network policies, which restrict which pods may talk to each other.

Pod network range — for a control plane. The addresses pods get, from /16 to /24. 10.244.0.0/16 suits most setups; choose another range only if it overlaps the network your servers are on.

Run workloads on the control plane — for a cluster of one server. Off keeps the control plane for Kubernetes itself, which is right once you have workers.

Open the Kubernetes ports in the firewall — when firewalld or ufw is active, opens what Kubernetes needs: 6443, 2379–2380, 10250, 10257 and 10259 on a control plane; 10250, 10256 and 30000–32767 on a worker; and the pod network's traffic between servers (8472/udp for Flannel, 179/tcp, 4789/udp and IP-in-IP for Calico). Off leaves the firewall as it is.

What it does on the server

1. Turns swap off, now and at boot, keeping a copy of /etc/fstab.
2. Loads the overlay and br_netfilter kernel modules and turns on IP forwarding (/etc/modules-load.d/k8s.conf, /etc/sysctl.d/k8s.conf).
3. On the RHEL family, sets SELinux to permissive, as the Kubernetes installation guide requires.
4. Installs containerd: the distribution's containerd on Ubuntu and Debian, containerd.io from Docker's repository on the RHEL family. A containerd already installed, such as Docker's, is used.
5. Adds the chosen version's repository from pkgs.k8s.io, after checking its signing key, and installs kubelet, kubeadm and kubectl, held at that version so routine package updates never move the cluster.
6. Configures containerd for Kubernetes: the CRI plugin on, the systemd cgroup driver, and the pause image kubeadm expects. The previous configuration is kept beside it.
7. For a control plane: runs kubeadm init, sets up kubectl for root, installs Flannel v0.28.9 or Calico v3.32.2 from their official manifests (checked against pinned checksums), and waits until the node is Ready.

The join command kubeadm init prints is never shown in the run, because it carries a token that lets any machine join the cluster. kubeadm's full output is kept on the server in /root/kubeadm-init.log, readable only by root.

After the run

Add a worker: on the control plane, over SSH, run kubeadm token create --print-join-command --ttl 1h. It prints a kubeadm join command valid for an hour. Run that command on the worker over SSH, then check on the control plane with kubectl get nodes: the worker shows as Ready within a few minutes. Use SSH rather than the Command Runner for these two commands: the Command Runner keeps every command and its output in the run history, and the join command carries a token.

Use the cluster: kubectl works for root on the control plane. To use it from your computer, copy /etc/kubernetes/admin.conf from the control plane to ~/.kube/config on your computer, and treat that file as a password: it gives full control of the cluster.

Upgrade later: upgrades are done with kubeadm upgrade, one minor version at a time. This playbook never changes an installed version.

Common problems

"A control plane needs at least 2 CPUs and 1700 MB of memory" — kubeadm refuses to start a control plane on less. Use a larger server, or make this one a worker.

"This server uses cgroup v1." — Kubernetes 1.35 and later need cgroup v2. Remove systemd.unified_cgroup_hierarchy=0 from the kernel command line, or use a newer distribution.

"Kubernetes … is installed here. Moving to … is an upgrade" — The server already runs another Kubernetes version. Run the playbook with that version, or upgrade with kubeadm upgrade.

"This server has already joined a cluster as a worker" — To turn it into a control plane, remove it from its cluster (kubectl drain and kubectl delete node on the control plane) and run kubeadm reset -f on it first.

"kubeadm could not initialise the control plane" — kubeadm's own reason is in the output (the lines with [ERROR name the check that failed). What it had set up was undone, so fix the cause and run the playbook again.

"The control plane is running, but the node was not Ready after 7 minutes." — Usually the pod network's images are still downloading on a slow connection. The pods listed show what is still starting. Check again with kubectl get pods -A a few minutes later; run the playbook again if a pod is failing.

A worker stays NotReady after joining — Its pod network pod is still starting or cannot reach the control plane. Check kubectl get pods -n kube-system -o wide on the control plane, and that the workers can reach the control plane on port 6443 and each other on the pod network's ports.

"…'s own containerd package is installed, and Docker's packages would replace it." (Install Docker, after Kubernetes) — Kubernetes uses the distribution's containerd on Ubuntu and Debian. Keep Docker and Kubernetes on separate servers there.