Domain 4Kubernetes EngineeringCORE

Multi-Node Cluster Bootstrapping: Production Kubeadm Setup across RHEL & Ubuntu Nodes

Comprehensive production setup guide for bootstrapping a multi-node Kubernetes cluster across heterogeneous RHEL and Ubuntu nodes using Kubeadm, Containerd with systemd cgroups, and network pre-flight tuning.

Domain: Domain 4: Kubernetes Cluster Engineering•Updated: 2026-09-28
#Kubernetes#Kubeadm#Containerd#CKA#RHEL#Ubuntu#CRI#systemd

Multi-Node Cluster Bootstrapping: Production Kubeadm Setup across RHEL & Ubuntu Nodes


Executive Summary

While lightweight distributions like Minikube or K3s provide convenient single-node sandboxes, mastering Certified Kubernetes Administrator (CKA) skills and production operations requires deploying upstream Kubernetes via kubeadm directly across multi-node Linux environments.

Operating across a heterogeneous node fleet (mixing Ubuntu 24.04 control-plane nodes with RHEL 9 worker nodes) introduces key challenges: distinct kernel module initialization (overlay, br_netfilter), differing SELinux/AppArmor profiles, and systemd cgroup driver configuration under Containerd.

This guide details the complete cluster bootstrap lifecycle: OS pre-flight preparation, CRI runtime initialization, control-plane initialization with kubeadm init, worker joins, and initial cluster verification.


1. Architectural Overview & Node Topology

+----------------------------------------------------------------------------------------------------+
|                                KUBERNETES MULTI-NODE CLUSTER TOPOLOGY                               |
+----------------------------------------------------------------------------------------------------+
|                                        VLAN 40: `10.0.40.0/24`                                     |
|                                                     |                                              |
|      +----------------------------------------------+---------------------------------------+      |
|      |                                              |                                       |      |
|  [ k8s-cp-01 (Control Plane) ]            [ k8s-worker-01 (Worker) ]            [ k8s-worker-02 ]   |
|  10.0.40.10                               10.0.40.11                            10.0.40.12          |
|  - Ubuntu 24.04 LTS (2 vCPU, 4GB RAM)     - Ubuntu 24.04 (4 vCPU, 8GB RAM)      - RHEL 9.4 (4 vCPU) |
|  - kube-apiserver, etcd, controller-mgr   - kubelet, kube-proxy, containerd     - kubelet, kube-proxy|
|  - Pod Network CIDR: `10.244.0.0/16`      - Workload Pods                       - Workload Pods     |
+----------------------------------------------------------------------------------------------------+

Cluster Resource Allocation Matrix

Node Hostname Role OS Distribution IP Address Specs (vCPU / RAM / Disk)
k8s-cp-01 Control Plane (Master) Ubuntu 24.04 LTS 10.0.40.10 2 vCPU / 4 GiB RAM / 40 GiB NVMe
k8s-worker-01 Worker Node Ubuntu 24.04 LTS 10.0.40.11 4 vCPU / 8 GiB RAM / 80 GiB NVMe
k8s-worker-02 Worker Node RHEL 9.4 10.0.40.12 4 vCPU / 8 GiB RAM / 80 GiB NVMe

2. Step-by-Step Implementation

Step 1: Pre-Flight Node Configuration (All Nodes)

Disable swap (mandatory for kubelet), load prerequisite kernel modules, and configure sysctl packet forwarding across all nodes:

# 1. Disable swap memory
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

# 2. Forwarding IPv4 and letting iptables see bridged traffic
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

sudo modprobe overlay
sudo modprobe br_netfilter

# 3. Apply sysctl parameters
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

sudo sysctl --system

Step 2: Install Containerd & Configure Systemd Cgroups

Install containerd and generate the default configuration with SystemdCgroup = true:

# Generate default containerd config
sudo mkdir -p /etc/containerd
sudo containerd config default | sudo tee /etc/containerd/config.toml

# Set SystemdCgroup to true in config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

# Restart and enable containerd
sudo systemctl restart containerd
sudo systemctl enable --now containerd

Step 3: Install Kubeadm, Kubelet & Kubectl (v1.31)

On Ubuntu / Debian Nodes:

sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl gpg
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.31/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.31/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

On RHEL / Fedora Nodes:

cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.31/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.31/rpm/repodata/repomd.xml.key
EOF
sudo setenforce 0
sudo sed -i --follow-symlinks 's/SELINUX=enforcing/SELINUX=permissive/g' /etc/sysconfig/selinux
sudo dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes
sudo systemctl enable --now kubelet

Step 4: Control Plane Initialization (kubeadm init)

Execute the initialization command on k8s-cp-01:

sudo kubeadm init \
  --apiserver-advertise-address=10.0.40.10 \
  --pod-network-cidr=10.244.0.0/16 \
  --node-name=k8s-cp-01

Configure kubectl access for the non-root administrative user:

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Step 5: Joining Worker Nodes (kubeadm join)

On k8s-worker-01 and k8s-worker-02, execute the token join command generated by the control plane:

sudo kubeadm join 10.0.40.10:6443 \
  --token <join-token> \
  --discovery-token-ca-cert-hash sha256:<ca-cert-hash>

3. Deep Dive & Troubleshooting Analysis

+-----------------------------------------------------------------------------------+
|                           TROUBLESHOOTING MATRIX                                  |
+-------------------------------------+---------------------------------------------+
| Failure / Symptom                   | Root Cause & Technical Fix                  |
+-------------------------------------+---------------------------------------------+
| 1. `kubelet` fails with cgroup error| Driver mismatch: Containerd used `cgroupfs` |
|                                     | while kubelet used `systemd`. Set `Systemd- |
|                                     | Cgroup = true` in `/etc/containerd/config`. |
+-------------------------------------+---------------------------------------------+
| 2. `br_netfilter` missing error     | `sysctl` parameters fail because kernel     |
|                                     | module not loaded. Add to `/etc/modules-    |
|                                     | load.d/k8s.conf` and run `modprobe`.        |
+-------------------------------------+---------------------------------------------+
| 3. RHEL Worker `SELinux` port block | SELinux in Enforcing mode blocked kubelet   |
|                                     | container volume access. Set to Permissive  |
|                                     | or install `container-selinux`.             |
+-------------------------------------+---------------------------------------------+

4. Verification & Health Checks

Verify node health and cluster status from the control plane:

kubectl get nodes -o wide

Expected Status Output:

NAME            STATUS     ROLES           AGE     VERSION   INTERNAL-IP   OS-IMAGE             KERNEL-VERSION
k8s-cp-01       NotReady   control-plane   5m12s   v1.31.0   10.0.40.10    Ubuntu 24.04 LTS     6.8.0-40-generic
k8s-worker-01   NotReady   <none>          2m45s   v1.31.0   10.0.40.11    Ubuntu 24.04 LTS     6.8.0-40-generic
k8s-worker-02   NotReady   <none>          1m18s   v1.31.0   10.0.40.12    Red Hat Ent. 9.4     5.14.0-427.el9

Note on NotReady Status: Nodes remain in NotReady status until a Container Network Interface (CNI) like Cilium or Calico is installed to manage Pod IP routing.


5. Key Technical Lessons & Next Steps

  1. Unified Cgroup Drivers: Always enforce SystemdCgroup = true under Containerd to match kubelet defaults on modern Linux kernels.
  2. Heterogeneous Fleet Management: Abstracting pre-flight kernel configurations through Ansible playbooks ensures identical node state regardless of OS family.
  3. Next Chapter: 02-cilium-ebpf-cni-networking.md — Next-Gen CNI: Deploying Cilium with eBPF, Kube-Proxy Replacement, and L7 NetworkPolicies.