Domain 4Kubernetes EngineeringADVANCED

Kubernetes The Hard Way: Bootstrapping a Bare-Metal Cluster from Scratch

A step-by-step walkthrough of bootstrapping a Kubernetes v1.32 cluster from scratch without automated installers, covering PKI, etcd, control plane components, containerd worker runtimes, static CNI routing, and cluster validation.

Domain: Domain 4: Kubernetes Cluster Engineering•Updated: 2026-10-31
#Kubernetes#k8s#etcd#containerd#PKI#RBAC#CNI#Debian#Homelab#Linux

Kubernetes The Hard Way: Bootstrapping a Bare-Metal Cluster from Scratch

Abstract

Modern managed Kubernetes distributions—such as kubeadm, k3s, EKS, and GKE—abstract away the intricate underlying mechanics that keep container orchestration systems running. While these tools streamline production deployments, they hide the essential relationships between control plane components, public key infrastructure (PKI), container runtime interfaces, and bare-metal routing.

This post documents the step-by-step journey of manually bootstrapping a fully functional Kubernetes v1.32 cluster on four bare-metal or virtualized Debian 12 nodes. By manually issuing TLS certificates, configuring etcd, assembling control plane binaries (kube-apiserver, kube-controller-manager, kube-scheduler), wiring worker runtimes (containerd, runc, CNI plugins), and managing Linux static routing tables, you will gain a deep, production-grade operational understanding of Kubernetes mechanics from the ground up.


1. Prerequisites (docs/01-prerequisites.md)

  • System Requirements: Overview of the four Debian 12 (Bookworm) virtual or physical instances required for the lab topology:
Name Description CPU RAM Storage
jumpbox Administration host 1 vCPU 512 MB 10 GB
server Kubernetes control plane 1 vCPU 2 GB 20 GB
node-0 Kubernetes worker node 1 vCPU 2 GB 20 GB
node-1 Kubernetes worker node 1 vCPU 2 GB 20 GB
  • OS Verification: Validating release metadata via /etc/os-release across all target instances.

This section covers the provisioning of a 4-node infrastructure layout using Proxmox LXC containers connected to a dedicated virtual bridge (vmbr2).

[ jumpbox (10.0.0.1) ] ──┐
[ server  (10.0.0.2) ] ──┼──► [ Proxmox Bridge: vmbr2 ] (10.0.0.254/24)
[ node-0  (10.0.0.3) ] ──┤                │ (IPTables SNAT / NAT)
[ node-1  (10.0.0.4) ] ──┘                ▼
                               [ Proxmox Bridge: vmbr0 ] (192.168.1.x)
                                          │
                                          ▼
                               [ Gateway Router / Internet ]

1.1 High-Level Node Architecture

Hostname CT ID Role vCPU Memory Disk IP Address Subnet
jumpbox 901 Administration & CA Authority 1 512 MB 4 GB 10.0.0.1 10.0.0.0/24
server 902 Kubernetes Control Plane (etcd, apiserver, etc.) 1 1024 MB 8 GB 10.0.0.2 10.0.0.0/24
node-0 903 Worker Node 0 (containerd, kubelet, kube-proxy) 1 512 MB 4 GB 10.0.0.3 10.0.0.0/24
node-1 904 Worker Node 1 (containerd, kubelet, kube-proxy) 1 512 MB 4 GB 10.0.0.4 10.0.0.0/24

1.2 Technical Understanding & Deep Dive

Virtual Bridges vs. Physical Egress Routing

Linux bridge interfaces (vmbr0, vmbr2) act as virtual Layer 2 switches inside the kernel. When a container sends a packet to an external destination (e.g., 1.1.1.1), the kernel consults its host routing table to find the appropriate egress interface.

In a nested virtual network where physical NIC nic0 is enslaved under vmbr0, outbound packets exiting the host use vmbr0 as their master network device. Defining an IP masquerading rule targeting nic0 directly fails because outbound packets are attached to the vmbr0 logical interface before leaving the system.

+--------------------------------------------------------------------------+
| Proxmox Host (hyproxmox)                                                 |
|                                                                          |
|  [ LXC: jumpbox ] -- (veth) --> [ Bridge: vmbr2 ] (10.0.0.254)          |
|                                         |                                |
|                                   (IP Forwarding)                        |
|                                         v                                |
|  [ LAN / WAN ] <-- (nic0) -- [ Bridge: vmbr0 ] (192.168.1.x) <--- NAT    |
+--------------------------------------------------------------------------+

SNAT / IP Masquerading Mechanism

Because the 10.0.0.0/24 subnet consists of non-routable RFC 1918 private IP addresses, external routers on your local network or the wider internet do not know how to send return traffic to 10.0.0.1.

Source Network Address Translation (SNAT)—specifically MASQUERADE in iptables—rewrites the IP header of every outbound packet originating from 10.0.0.0/24. It changes the source address from the internal container IP (e.g., 10.0.0.1) to the host’s LAN IP on vmbr0 (e.g., 192.168.1.x). The host tracks these active connection states in its conntrack table so return traffic is routed seamlessly back to the originating container.

Reverse Path Filtering (rp_filter)

Linux includes a security feature called Reverse Path Filtering (rp_filter). The kernel checks whether an incoming packet on an interface arrives via the same path that would be used to reach the source IP. If asymmetric paths exist or multiple virtual bridges overlap on similar subnets, rp_filter drops packets as potential spoofed traffic. Setting net.ipv4.conf.all.rp_filter = 0 disables these strict checks on custom bridged setups.


1.3 Step-by-Step Implementation

Step 1: Provision LXC Containers via Proxmox CLI

Run the automated provisioning commands on your Proxmox host (root@hyproxmox):

# Extract the exact Ubuntu 24.04 LXC template path
TEMPLATE=$(pveam list local | grep ubuntu-24.04 | awk '{print $1}' | head -n 1)
STORAGE="local-lvm"
BRIDGE="vmbr2"
GATEWAY="10.0.0.254"
ROOT_PASS="ChangeMe123!"

# 1. jumpbox (ID: 901, IP: 10.0.0.1)
pct create 901 $TEMPLATE \
  --hostname jumpbox \
  --ostype ubuntu \
  --cores 1 \
  --memory 512 \
  --swap 0 \
  --rootfs ${STORAGE}:4 \
  --net0 name=eth0,bridge=${BRIDGE},ip=10.0.0.1/24,gw=${GATEWAY} \
  --unprivileged 1 \
  --features nesting=1,keyctl=1 \
  --password "$ROOT_PASS" \
  --onboot 1

# 2. server (ID: 902, IP: 10.0.0.2)
pct create 902 $TEMPLATE \
  --hostname server \
  --ostype ubuntu \
  --cores 1 \
  --memory 1024 \
  --swap 0 \
  --rootfs ${STORAGE}:8 \
  --net0 name=eth0,bridge=${BRIDGE},ip=10.0.0.2/24,gw=${GATEWAY} \
  --unprivileged 1 \
  --features nesting=1,keyctl=1 \
  --password "$ROOT_PASS" \
  --onboot 1

# 3. node-0 (ID: 903, IP: 10.0.0.3)
pct create 903 $TEMPLATE \
  --hostname node-0 \
  --ostype ubuntu \
  --cores 1 \
  --memory 512 \
  --swap 0 \
  --rootfs ${STORAGE}:4 \
  --net0 name=eth0,bridge=${BRIDGE},ip=10.0.0.3/24,gw=${GATEWAY} \
  --unprivileged 1 \
  --features nesting=1,keyctl=1 \
  --password "$ROOT_PASS" \
  --onboot 1

# 4. node-1 (ID: 904, IP: 10.0.0.4)
pct create 904 $TEMPLATE \
  --hostname node-1 \
  --ostype ubuntu \
  --cores 1 \
  --memory 512 \
  --swap 0 \
  --rootfs ${STORAGE}:4 \
  --net0 name=eth0,bridge=${BRIDGE},ip=10.0.0.4/24,gw=${GATEWAY} \
  --unprivileged 1 \
  --features nesting=1,keyctl=1 \
  --password "$ROOT_PASS" \
  --onboot 1
  • --unprivileged 1: Runs the container in an isolated user namespace for security.
  • --features nesting=1,keyctl=1: Enables container nesting and keyring functionality required for running systemd services and container runtimes (such as containerd) inside LXCs.

Step 2: Start Containers and Configure Host NAT Rules

Start all instances concurrently:

pct start 901 && pct start 902 && pct start 903 && pct start 904

Apply runtime network rules on root@hyproxmox to allow IP forwarding and source translation over master bridge vmbr0:

# Clean up any misplaced rules pointing to the physical enslaved slave port
iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o nic0 -j MASQUERADE 2>/dev/null || true

# Append MASQUERADE rule targeting the active egress master bridge
iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE

# Permit local host traffic and forward traffic across the bridge
iptables -I INPUT -i vmbr2 -j ACCEPT
iptables -I FORWARD -i vmbr2 -j ACCEPT
iptables -I FORWARD -o vmbr2 -j ACCEPT

Step 3: Persist Network and Kernel Configurations

To ensure bridge forwarding and NAT rules persist across host reboots, append the following configurations on root@hyproxmox:

Update /etc/network/interfaces:

auto vmbr2
iface vmbr2 inet static
        address 10.0.0.254/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0
        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE
        post-up iptables -I INPUT -i vmbr2 -j ACCEPT
        post-up iptables -I FORWARD -i vmbr2 -j ACCEPT
        post-up iptables -I FORWARD -o vmbr2 -j ACCEPT
        post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE
        post-down iptables -D INPUT -i vmbr2 -j ACCEPT
        post-down iptables -D FORWARD -i vmbr2 -j ACCEPT
        post-down iptables -D FORWARD -o vmbr2 -j ACCEPT

Create /etc/sysctl.d/99-k8s-lxc.conf:

net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.vmbr2.rp_filter = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

Apply sysctl changes immediately:

sysctl --system


1.4 Verification & Validation

1. Verify LXC Status on Proxmox Host

Inspect the status of running containers:

pct list
VMID       Status     Lookups            Name                
901        running    10.0.0.1           jumpbox             
902        running    10.0.0.2           server              
903        running    10.0.0.3           node-0              
904        running    10.0.0.4           node-1              

2. Verify Packet Matching on NAT Rules

Confirm that outbound packets from 10.0.0.0/24 are hitting the correct MASQUERADE target:

iptables -t nat -L POSTROUTING -n -v | grep MASQUERADE
  12   840 MASQUERADE  all  --  *      vmbr0   10.0.0.0/24          0.0.0.0/0

3. Verify Local Bridge and Public WAN Reachability

Test ping connectivity from jumpbox (CT 901) to the host gateway (10.0.0.254) and a public address (1.1.1.1):

pct exec 901 -- ping -c 3 10.0.0.254
PING 10.0.0.254 (10.0.0.254) 56(84) bytes of data.
64 bytes from 10.0.0.254: icmp_seq=1 ttl=64 time=0.048 ms
64 bytes from 10.0.0.254: icmp_seq=2 ttl=64 time=0.039 ms
64 bytes from 10.0.0.254: icmp_seq=3 ttl=64 time=0.038 ms

--- 10.0.0.254 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2045ms
rtt min/avg/max/mdev = 0.038/0.041/0.048/0.005 ms
pct exec 901 -- ping -c 3 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=56 time=11.2 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=56 time=10.8 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=56 time=10.9 ms

--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 10.841/10.980/11.214/0.165 ms

4. Verify HTTP and DNS Resolution

Perform a curl request to confirm external HTTPS connectivity and working DNS resolution:

pct exec 901 -- curl -I https://github.com
HTTP/2 200 
server: GitHub.com
date: Sat, 31 Oct 2026 10:36:00 GMT
content-type: text/html; charset=utf-8
vary: X-PJAX, X-PJAX-Container, Accept-Encoding, Accept, X-Requested-With

2. Jumpbox Provisioning (docs/02-jumpbox.md)

  • Admin Tooling: Installing standard utilities (wget, curl, vim, openssl, git) on the jumpbox.
  • Repository Sync: Cloning the core repository and establishing the base working environment.
  • Binary Staging: Staging architecture-specific binaries (etcd, kubectl, kube-apiserver, containerd, runc, cni-plugins) in downloads/.
  • Local Client Tooling: Installing and verifying kubectl on the administrative workstation.

Setting Up the Jumpbox


2.1 Section Overview & Context

Before bootstrapping a Kubernetes cluster from the ground up, you need a dedicated administrative environment—the Jumpbox.

In the broader architecture of Kubernetes The Hard Way, the jumpbox acts as your central command center and artifact repository. Rather than installing build tools, fetching binaries, or generating Public Key Infrastructure (PKI) certificates on individual cluster nodes, you perform all preparation work here.

                  +-----------------------------------+
                  |          ADMIN JUMPBOX            |
                  |            10.0.0.1               |
                  |                                   |
                  |  [Local Artifact Cache]           |
                  |   ├── downloads/client/           |
                  |   ├── downloads/controller/       |
                  |   └── downloads/worker/           |
                  +-----------------+-----------------+
                                    |
          +-------------------------+-------------------------+
          |                         |                         |
          v                         v                         v
+------------------+      +------------------+      +------------------+
|  Control Plane   |      |     Worker 0     |      |     Worker 1     |
|   10.0.0.10-12   |      |    10.0.0.20     |      |    10.0.0.21     |
+------------------+      +------------------+      +------------------+

High-Level Architecture Summary

Setting up the jumpbox accomplishes three primary goals:

  • Centralized Administration: Provides a isolated, uniform environment (10.0.0.1) to execute provisioning scripts, manage SSH keys, generate TLS certificates, and issue cluster-wide commands.
  • Local Artifact Caching: Downloads over 500 MB of core binaries (etcd, containerd, kube-apiserver, kubelet, cni-plugins, etc.) once. These binaries are organized by node role (client, controller, worker), eliminating redundant external downloads when staging cluster nodes.
  • Management Tooling: Installs client tools such as kubectl locally, enabling direct control-plane communication once the API server is online.

2.2 Technical Understanding & Deep Dive: The “Why”

Why Use a Dedicated Jumpbox?

In production environments, control plane instances and worker nodes should be locked down minimal operating system images with restricted external internet access. Staging all binaries, configuration templates, and cryptographic assets on a dedicated jumpbox allows you to maintain strict security boundaries while providing a single source of truth for cluster assets.

Architecture-Aware Downloading (dpkg --print-architecture)

Kubernetes components are distributed as compiled Go binaries tied to specific CPU architectures (e.g., amd64, arm64). By dynamically querying your system’s architecture using dpkg --print-architecture, the bootstrap process automatically selects the correct download manifest file (downloads-amd64.txt vs. downloads-arm64.txt). This avoids architecture mismatch errors during binary execution (such as Exec format error).

Binary Role Segregation

Instead of dumping all binaries into a single directory, we extract and group them into strict role-based directories:

Target Directory Binaries Included Functional Role
downloads/client/ kubectl, etcdctl CLI client tools used by administrators to interact with Kubernetes and etcd.
downloads/controller/ etcd, kube-apiserver, kube-controller-manager, kube-scheduler Core control plane services responsible for cluster consensus, API exposure, state control loops, and Pod scheduling.
downloads/worker/ kubelet, kube-proxy, containerd, crictl, runc Node-level runtime execution engines, container orchestration agents, and networking proxies.
downloads/cni-plugins/ bridge, loopback, host-local, etc. Container Network Interface binaries that manage Pod networking, IP allocation, and virtual interfaces.

Archive Extraction Strategy: Strip Components

When extracting multi-file distribution archives like etcd or containerd, the archive root often contains nested directory hierarchies (e.g., etcd-v3.6.0-rc.3-linux-amd64/etcd). Using tar --strip-components 1 removes leading directory paths during extraction, allowing binaries to land cleanly into targeted target directories without unwanted subfolder nesting.


2.3 Step-by-Step Implementation & Command Breakdown

All commands are executed on the jumpbox (10.0.0.1) as the root user to ensure consistent access permissions throughout the setup.

Step 1: Log In to the Jumpbox & Check Connectivity

Access your jumpbox container via Proxmox LXC enter or SSH:

# If accessing directly from Proxmox VE host:
pct enter 901

# Or via SSH from your workstation:
ssh root@10.0.0.1

Verify outbound network access:

ping -c 2 github.com

Step 2: Install Required Utilities

Update local repository metadata and install essential administrative tools:

apt-get update && apt-get -y install wget curl vim openssl git
  • wget / curl: Used to retrieve remote binaries, configuration files, and API endpoints.
  • openssl: Essential for inspecting and generating PKI TLS certificates.
  • git: Required to clone the source configuration repository.

Step 3: Clone the Kubernetes The Hard Way Repository

Clone the upstream repository to obtain setup scripts, systemd unit templates, and binary download manifests:

git clone --depth 1 https://github.com/kelseyhightower/kubernetes-the-hard-way.git
cd kubernetes-the-hard-way
  • --depth 1: Performs a shallow clone, downloading only the latest revision to save time and bandwidth.

Verify your working directory:

pwd

Expected Output:

/root/kubernetes-the-hard-way

Step 4: Download & Organize Kubernetes Binaries

Inspect your system architecture manifest list:

cat downloads-$(dpkg --print-architecture).txt

Download all cluster components into the downloads/ cache directory:

wget -q --show-progress \
  --https-only \
  --timestamping \
  -P downloads \
  -i downloads-$(dpkg --print-architecture).txt
Command Flag Breakdown:
  • -q --show-progress: Hides verbose output while rendering a clean progress bar.
  • --https-only: Enforces secure HTTPS transport for binary verification.
  • --timestamping (-N): Re-downloads files only if the remote file is newer than the local copy.
  • -P downloads: Saves all retrieved files into the downloads directory.
  • -i <file>: Reads target download URLs directly from the specified manifest file.

Verify the downloaded archives (~500 MB total):

ls -lh downloads

Extract binaries and route them into their target role directories:

{
  ARCH=$(dpkg --print-architecture)
  mkdir -p downloads/{client,cni-plugins,controller,worker}

  # Worker runtime & CNI binaries
  tar -xvf downloads/crictl-v1.32.0-linux-${ARCH}.tar.gz -C downloads/worker/
  tar -xvf downloads/containerd-2.1.0-beta.0-linux-${ARCH}.tar.gz --strip-components 1 -C downloads/worker/
  tar -xvf downloads/cni-plugins-linux-${ARCH}-v1.6.2.tgz -C downloads/cni-plugins/

  # etcd binaries
  tar -xvf downloads/etcd-v3.6.0-rc.3-linux-${ARCH}.tar.gz -C downloads/ \
    --strip-components 1 \
    etcd-v3.6.0-rc.3-linux-${ARCH}/etcdctl \
    etcd-v3.6.0-rc.3-linux-${ARCH}/etcd

  # Categorize binaries into role directories
  mv downloads/{etcdctl,kubectl} downloads/client/
  mv downloads/{etcd,kube-apiserver,kube-controller-manager,kube-scheduler} downloads/controller/
  mv downloads/{kubelet,kube-proxy} downloads/worker/
  mv downloads/runc.${ARCH} downloads/worker/runc
}

Clean up tarballs and grant execution permissions across all categorized binaries:

rm -rf downloads/*.gz downloads/*.tgz
chmod +x downloads/{client,cni-plugins,controller,worker}/*

Step 5: Install kubectl on the Jumpbox

Install kubectl into your system PATH to allow immediate administrative access to remote clusters:

cp downloads/client/kubectl /usr/local/bin/
chmod +x /usr/local/bin/kubectl

2.4 Verification & Validation

Verify that your local workspace is organized correctly and that the client tools are ready.

1. Validate Organized Directory Structure

Inspect the contents of the downloads/ directory:

ls -l downloads/*

Expected Output:

downloads/client:
total 162M
-rwxr-xr-x 1 root root  98M Mar 27 00:00 etcdctl
-rwxr-xr-x 1 root root  65M Mar 27 00:00 kubectl

downloads/cni-plugins:
total 72M
-rwxr-xr-x 1 root root 4.1M Mar 27 00:00 bandwidth
-rwxr-xr-x 1 root root 4.6M Mar 27 00:00 bridge
-rwxr-xr-x 1 root root 11M  Mar 27 00:00 dhcp
-rwxr-xr-x 1 root root 4.3M Mar 27 00:00 firewall
-rwxr-xr-x 1 root root 3.7M Mar 27 00:00 host-device
-rwxr-xr-x 1 root root 3.6M Mar 27 00:00 host-local
-rwxr-xr-x 1 root root 4.3M Mar 27 00:00 ipvlan
-rwxr-xr-x 1 root root 3.8M Mar 27 00:00 loopback
-rwxr-xr-x 1 root root 4.4M Mar 27 00:00 macvlan
-rwxr-xr-x 1 root root 4.1M Mar 27 00:00 ptp
-rwxr-xr-x 1 root root 3.9M Mar 27 00:00 sbr
-rwxr-xr-x 1 root root 3.1M Mar 27 00:00 static
-rwxr-xr-x 1 root root 4.2M Mar 27 00:00 tap
-rwxr-xr-x 1 root root 3.8M Mar 27 00:00 tuning
-rwxr-xr-x 1 root root 4.3M Mar 27 00:00 vlan
-rwxr-xr-x 1 root root 4.7M Mar 27 00:00 vrf

downloads/controller:
total 252M
-rwxr-xr-x 1 root root  24M Mar 27 00:00 etcd
-rwxr-xr-x 1 root root 124M Mar 27 00:00 kube-apiserver
-rwxr-xr-x 1 root root  68M Mar 27 00:00 kube-controller-manager
-rwxr-xr-x 1 root root  36M Mar 27 00:00 kube-scheduler

downloads/worker:
total 165M
-rwxr-xr-x 1 root root  52M Mar 27 00:00 containerd
-rwxr-xr-x 1 root root 7.1M Mar 27 00:00 containerd-shim-runc-v2
-rwxr-xr-x 1 root root  14M Mar 27 00:00 crictl
-rwxr-xr-x 1 root root  24M Mar 27 00:00 kube-proxy
-rwxr-xr-x 1 root root  54M Mar 27 00:00 kubelet
-rwxr-xr-x 1 root root  14M Mar 27 00:00 runc

2. Verify kubectl Client Binary

Confirm that kubectl is installed globally and functioning:

kubectl version --client

Expected Output:

Client Version: v1.32.3
Kustomize Version: v5.5.0

The administrative jumpbox (10.0.0.1) is fully provisioned with essential utilities, repository templates, binary caching directories, and the kubectl client tool (v1.32.3).


3. Provisioning Compute Resources (docs/03-compute-resources.md)

  • Machine Database: Creating machines.txt to track IP address allocations, FQDNs, hostnames, and assigned Pod CIDRs (10.200.x.0/24).
  • SSH Connectivity: Enabling root SSH login across instances, generating SSH keypairs on the jumpbox, and distributing public keys.
  • Hostname Configuration: Setting FQDN hostnames using hostnamectl and generating host lookup tables across all node /etc/hosts files.

Before installing container runtimes, configuring control plane services, or issuing TLS certificates, you must establish a reliable compute and networking baseline. In this section, you will lay the groundwork for a 4-node cluster comprising one control plane node (server), two worker nodes (node-0, node-1), and a dedicated administrative node (jumpbox).

Installing Kubernetes from scratch requires total administrative control and predictable network routes across all participating systems. To achieve this without relying on external cloud providers or automated provisioning scripts, you will:

  • Define an authoritative machine database tracking static IPs, Hostnames, FQDNs, and Pod CIDR allocations.
  • Enable root administrative access and set up an SSH key-based trust model for non-interactive orchestration from the jumpbox.
  • Configure system hostnames and establish a static local host lookup table across every node to ensure deterministic cluster-wide name resolution without an external DNS server.
                  +-----------------------------------+
                  |        jumpbox (10.0.0.1)         |
                  |  (Orchestration & Administration) |
                  +-----------------+-----------------+
                                    |
            +-----------------------+-----------------------+
            |                       |                       |
            v                       v                       v
+-----------------------+ +-------------------+ +-------------------+
|  server (10.0.0.2)    | | node-0 (10.0.0.3)| | node-1 (10.0.0.4)|
| Control Plane / etcd  | | Worker Node       | | Worker Node       |
| (No Pod Subnet)       | | Subnet:           | | Subnet:           |
|                       | |   10.200.0.0/24   | |   10.200.1.0/24   |
+-----------------------+ +-------------------+ +-------------------+

3.1 TECHNICAL UNDERSTANDING & DEEP DIVE (THE “WHY”)

1. Machine Database & Pod Subnet Allocation

Kubernetes imposes a strict networking requirement: every Pod must receive a unique, routable IP address across the entire cluster. To enforce this without complex dynamic allocation during initial boot, each worker node is assigned a non-overlapping IP subnet—a Pod CIDR (e.g., 10.200.0.0/24).

  • Node node-0 receives 10.200.0.0/24, reserving IPs 10.200.0.1 through 10.200.0.254 exclusively for Pods running on node-0.
  • Node node-1 receives 10.200.1.0/24, reserving IPs 10.200.1.1 through 10.200.1.254 for Pods running on node-1.

Storing these parameters in a structured file (machines.txt) creates a single source of truth. Automated shell loops (while read ...) parse this data dynamically throughout subsequent setup steps.

2. Root SSH Access & Passwordless Key Authentication

Bootstrapping a control plane requires creating system directories, installing systemd unit files, managing etcd datastores, and configuring kernel modules—actions that demand superuser (root) privileges. While disabling root SSH access is standard security practice, optimizing for clarity and operational speed during low-level cluster creation justifies using direct SSH key access from a trusted administrative jumpbox.

Using an unpassphrased RSA key pair allows shell scripts on the jumpbox to push files via scp and execute commands via ssh without stopping for password prompts.

3. Hostname Resolution without External DNS

Kubernetes components rely heavily on hostnames:

  • kubelet registers itself to the cluster control plane using its local hostname.
  • kube-apiserver validates client identity by cross-referencing Hostnames and FQDNs against Subject Alternative Names (SANs) embedded inside TLS certificates.
  • etcd nodes coordinate consensus using member hostnames.

Rather than introducing an external DNS server (which adds failure domain complexity to a bootstrap environment), local host resolution is enforced statically via /etc/hosts across all machines.


3.2 STEP-BY-STEP IMPLEMENTATION & COMMAND BREAKDOWN

Step 1: Create the Machine Database

On the jumpbox (10.0.0.1), navigate to your working directory and create the machines.txt source of truth file:

cat <<EOF > machines.txt
10.0.0.2 server.kubernetes.local server
10.0.0.3 node-0.kubernetes.local node-0 10.200.0.0/24
10.0.0.4 node-1.kubernetes.local node-1 10.200.1.0/24
EOF

Inspect the generated database:

cat machines.txt
10.0.0.2 server.kubernetes.local server
10.0.0.3 node-0.kubernetes.local node-0 10.200.0.0/24
10.0.0.4 node-1.kubernetes.local node-1 10.200.1.0/24

Schema Breakdown:

  • Column 1 (IPV4_ADDRESS): The primary host IP address accessible over the local network.
  • Column 2 (FQDN): Fully Qualified Domain Name used for certificate SAN alignment.
  • Column 3 (HOSTNAME): Short hostname assigned to the node operating system.
  • Column 4 (POD_SUBNET): The dedicated /24 CIDR range reserved for container allocation on worker nodes.

Step 2: Configure Root SSH Access on Cluster Nodes

Log into each node (server, node-0, node-1) via standard user SSH, elevate to root, and permit root logins in the OpenSSH daemon configuration.

  1. Elevate to root:
su - root

Note on su -: The - (login shell) flag clears the caller’s environment and loads a clean environment with root’s native $PATH, $HOME, and system variables.

  1. Enable root login in /etc/ssh/sshd_config:
sed -i \
  's/^#*PermitRootLogin.*/PermitRootLogin yes/' \
  /etc/ssh/sshd_config
  • sed -i: Modifies the file in-place without writing a secondary output file.
  • 's/^#*PermitRootLogin.*/PermitRootLogin yes/': Uses regular expressions to locate any variation of the PermitRootLogin key (whether active or commented out with #) and replaces the line with PermitRootLogin yes.
  1. Restart the SSH daemon to apply changes:
systemctl restart sshd

Step 3: Generate and Distribute SSH Keys

From the jumpbox (10.0.0.1), generate a secure 4096-bit RSA key pair without a passphrase and distribute the public key to all nodes.

  1. Generate the SSH key pair on jumpbox:
ssh-keygen -t rsa -b 4096 -f /root/.ssh/id_rsa -N ""
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
  • -t rsa -b 4096: Requests RSA key generation using a 4096-bit length.
  • -f /root/.ssh/id_rsa: Sets the target path for the private key file.
  • -N "": Configures an empty passphrase for unprompted script execution.
  1. Distribute the public key to each cluster machine:
while read IP FQDN HOST SUBNET; do
  ssh-copy-id root@${IP}
done < machines.txt

Enter the root password for each node when prompted. ssh-copy-id appends /root/.ssh/id_rsa.pub into the target host’s /root/.ssh/authorized_keys file and sets correct permissions (600).

  1. Verify non-interactive execution across all machines:
while read IP FQDN HOST SUBNET; do
  ssh -n root@${IP} hostname
done < machines.txt
server
node-0
node-1
  • The -n flag: Redirects stdin from /dev/null. Without -n, the ssh binary consumes the remaining lines of machines.txt directly from standard input, causing the loop to exit after processing only the first entry.

Step 4: Configure Hostnames across the Cluster

Set hostnames on each target host dynamically from the jumpbox:

while read IP FQDN HOST SUBNET; do
    CMD="sed -i 's/^127.0.1.1.*/127.0.1.1\t${FQDN} ${HOST}/' /etc/hosts"
    ssh -n root@${IP} "$CMD"
    ssh -n root@${IP} hostnamectl set-hostname ${HOST}
    ssh -n root@${IP} systemctl restart systemd-hostnamed
done < machines.txt

Command Breakdown:

  • sed -i 's/^127.0.1.1...*/...': Updates local loopback definitions in /etc/hosts to match the target FQDN and short hostname, preventing administrative utilities like sudo from raising unable to resolve host errors.
  • hostnamectl set-hostname ${HOST}: Interacts with systemd-hostnamed to set the runtime kernel hostname parameter (sysctl kernel.hostname) and writes the configuration persistently to /etc/hostname.
  • systemctl restart systemd-hostnamed: Forces background administrative daemons to adopt the updated hostname string immediately.

Step 5: Build and Distribute the Host Lookup Table

Create a consolidated static lookup table containing host mapping entries for all cluster nodes, then apply it locally and remotely.

  1. Generate the static host entries on jumpbox:
echo "" > hosts
echo "# Kubernetes The Hard Way" >> hosts
while read IP FQDN HOST SUBNET; do
    ENTRY="${IP} ${FQDN} ${HOST}"
    echo $ENTRY >> hosts
done < machines.txt
  1. Inspect the generated file:
cat hosts

# Kubernetes The Hard Way
10.0.0.2 server.kubernetes.local server
10.0.0.3 node-0.kubernetes.local node-0
10.0.0.4 node-1.kubernetes.local node-1
  1. Append host entries to the local /etc/hosts file on jumpbox:
cat hosts >> /etc/hosts
  1. Distribute and append host entries to all cluster nodes:
while read IP FQDN HOST SUBNET; do
  scp hosts root@${HOST}:~/
  ssh -n root@${HOST} "cat hosts >> /etc/hosts"
done < machines.txt

3.3 REAL-WORLD TROUBLESHOOTING & PITFALLS


VERIFICATION & VALIDATION

Verify that hostnames are correctly configured and reachable using their short names from the jumpbox.

  1. Verify Fully Qualified Domain Name resolution on each node:
while read IP FQDN HOST SUBNET; do
  ssh -n root@${IP} hostname --fqdn
done < machines.txt
server.kubernetes.local
node-0.kubernetes.local
node-1.kubernetes.local
  1. Verify hostname-based SSH connectivity from the jumpbox without raw IP addresses:
for host in server node-0 node-1; do
  ssh root@${host} hostname
done
server
node-0
node-1
  1. Verify cross-node host lookup entry in /etc/hosts:
ssh root@node-0 "ping -c 1 server"
PING server.kubernetes.local (10.0.0.2) 56(84) bytes of data.
64 bytes from server.kubernetes.local (10.0.0.2): icmp_seq=1 ttl=64 time=0.185 ms

--- server.kubernetes.local ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.185/0.185/0.185/0.185 ms

4. PKI & Certificate Authority (docs/04-certificate-authority.md)

  • Certificate Authority Setup: Bootstrapping a self-signed root CA (ca.key, ca.crt) using OpenSSL and ca.conf.

  • Component Certificates: Generating TLS client and server certificate pairs for:

  • Administrative identity: admin

  • Control plane services: kube-api-server, kube-controller-manager, kube-scheduler, service-accounts

  • Worker components: kube-proxy, and node-specific kubelet certs (node-0, node-1)

  • Key Distribution: Copying target certificates and private keys to /var/lib/kubelet/ on workers and ~/ on the server.

Kubernetes operates on a Zero-Trust network architecture:

  • no component trusts any other component by default.
  • Every internal connection—from kubectl issuing commands to the API server, to worker node daemons reporting pod statuses
  • is encrypted in transit and authenticated using Mutual TLS (mTLS).

In this section, you will establish the foundational Public Key Infrastructure (PKI) for your Kubernetes cluster from the jumpbox provisioning node (10.0.0.1). You will create a self-signed Root Certificate Authority (CA) and generate all client/server X.509 certificate pairs required by the Kubernetes control plane and worker nodes. Finally, you will distribute these sensitive security credentials to their respective host nodes (10.0.0.2, 10.0.0.3, and 10.0.0.4).

                     +---------------------------------------+
                     |                JUMPBOX                |
                     |              (10.0.0.1)               |
                     |  Generates Root CA, component certs,  |
                     |  and stores client admin credentials  |
                     +-------------------+-------------------+
                                         |
                     +-------------------+-------------------+
                     | SCP                                   | SCP
                     v                                       v
+------------------------------------+   +------------------------------------+
|               SERVER               |   |           NODE-0 / NODE-1          |
|             (10.0.0.2)             |   |       (10.0.0.3 / 10.0.0.4)        |
|        Control Plane Host          |   |         Worker Node Hosts          |
| - ca.crt & ca.key                  |   | - ca.crt                           |
| - kube-api-server.crt / .key       |   | - kubelet.crt (node-specific)      |
| - service-accounts.crt / .key      |   | - kubelet.key (node-specific)      |
+------------------------------------+   +------------------------------------+

4.1 Technical Understanding & Deep Dive (The “Why”)

Before executing commands, it is crucial to understand how Kubernetes uses digital certificates for identity, authorization, and encryption.

1. Mutual TLS (mTLS) vs. Standard TLS

In traditional web browsing, TLS is one-way: your browser verifies the server’s identity via a public certificate, but the server relies on passwords or cookies to identify you. In Kubernetes mTLS:

  • Server Verification: The server presents a certificate to prove its identity to the client.
  • Client Verification: The client presents its own certificate to prove its identity to the server.
  • Common Trust Anchor: Both parties validate each other against the same Root Certificate Authority (ca.crt).

2. X.509 Certificate Fields mapped to Kubernetes RBAC

Kubernetes does not maintain an internal user database or user table. Instead, the kube-apiserver extracts user identity directly from the subject fields of presented X.509 client certificates:

  • Common Name (CN): Interpreted by Kubernetes as the User Identity (e.g., CN=admin or CN=system:node:node-0).
  • Organization (O): Interpreted by Kubernetes as Group Membership (e.g., O=system:masters grants cluster-wide administrative privileges; O=system:nodes identifies worker node groups).

3. Subject Alternative Names (SANs)

When a client connects to an HTTPS service like kube-apiserver, TLS verification checks that the IP address or domain name requested matches the certificate. The Subject Alternative Names (SAN) field explicitly lists every valid IP (10.0.0.2, 127.0.0.1) and domain name (kubernetes, kubernetes.default, kubernetes.default.svc.cluster.local) through which kube-apiserver can be reached.

4. The Special Case: service-accounts Key Pair

Unlike the other certificate pairs generated in this section, service-accounts.key and service-accounts.crt are not used for TLS network encryption. Instead:

  • kube-controller-manager uses service-accounts.key (RSA private key) to sign asymmetric JWT tokens issued to Service Accounts inside Pods.
  • kube-apiserver uses service-accounts.crt (public key) to verify those token signatures when Pods authenticate against the API.

5. Inventory of Generated File Extensions

Extension File Type Purpose Security Level
.key Private Key Secret key used to sign tokens, decrypt traffic, and prove identity ownership. Strictly Secret
.csr Certificate Signing Request Temporary request containing identity attributes (CN, O, SANs) sent to the CA for signing. Temporary (Disposable)
.crt Public Certificate Signed public certificate containing the public key and identity attributes. Distributed freely for identity verification. Public / Distributed

4.2 Step-by-Step Implementation & Command Breakdown

All commands in this section are executed from the jumpbox (10.0.0.1) inside the /root/kubernetes-the-hard-way directory.

Step 1: Create the Certificate Authority (CA)

First, provision the Root CA private key and self-signed root certificate.

cd /root/kubernetes-the-hard-way

{
  openssl genrsa -out ca.key 4096
  openssl req -x509 -new -sha512 -noenc \
    -key ca.key -days 3653 \
    -config ca.conf \
    -out ca.crt
}
Command & Flag Breakdown:
  • genrsa -out ca.key 4096: Generates an unencrypted 4096-bit RSA private key (ca.key).
  • req -x509: Outputs a self-signed root certificate (ca.crt) rather than a signing request.
  • -sha512: Specifies SHA-512 hashing for signature security.
  • -noenc: Prevents encrypting the private key with a passphrase, allowing unattended component startup.
  • -days 3653: Sets certificate validity to 10 years.
  • -config ca.conf: References OpenSSL presets for CA subject fields.

Step 2: Generate Component Certificates and Keys

Define the list of required cluster identities and loop through them to generate private keys, signing requests, and signed certificates:

certs=(
  "admin" "node-0" "node-1"
  "kube-proxy" "kube-scheduler"
  "kube-controller-manager"
  "kube-api-server"
  "service-accounts"
)

for i in ${certs[*]}; do
  openssl genrsa -out "${i}.key" 4096

  openssl req -new -key "${i}.key" -sha256 \
    -config "ca.conf" -section ${i} \
    -out "${i}.csr"

  openssl x509 -req -days 3653 -in "${i}.csr" \
    -copy_extensions copyall \
    -sha256 -CA "ca.crt" \
    -CAkey "ca.key" \
    -CAcreateserial \
    -out "${i}.crt"
done
Critical Loop Execution Details:
  • -section ${i}: Directs OpenSSL to read specific sections from ca.conf (e.g., [node-0], [kube-api-server]) to populate exact CN, O, and SAN settings.
  • -copy_extensions copyall: Mandatory flag. Copies SAN extension fields defined in the CSR into the final signed .crt file. Omitting this drops SANs and causes TLS connection failures.
  • -CAcreateserial: Creates ca.srl to track issued serial numbers and prevent certificate collision.

Step 3: Verify Local Certificate Output

List all generated PKI files on the jumpbox:

ls -1 *.crt *.key *.csr
Expected Terminal Output:
admin.crt
admin.csr
admin.key
ca.crt
ca.key
kube-api-server.crt
kube-api-server.csr
kube-api-server.key
kube-controller-manager.crt
kube-controller-manager.csr
kube-controller-manager.key
kube-proxy.crt
kube-proxy.csr
kube-proxy.key
kube-scheduler.crt
kube-scheduler.csr
kube-scheduler.key
node-0.crt
node-0.csr
node-0.key
node-1.crt
node-1.csr
node-1.key
service-accounts.crt
service-accounts.csr
service-accounts.key

Step 4: Distribute Certificates to Worker Nodes

Transfer ca.crt and node-specific keypairs to worker nodes node-0 (10.0.0.3) and node-1 (10.0.0.4), placing them directly in the default kubelet directory /var/lib/kubelet/:

for host in node-0 node-1; do
  ssh root@${host} mkdir -p /var/lib/kubelet/

  scp ca.crt root@${host}:/var/lib/kubelet/

  scp ${host}.crt \
    root@${host}:/var/lib/kubelet/kubelet.crt

  scp ${host}.key \
    root@${host}:/var/lib/kubelet/kubelet.key
done

Step 5: Distribute Certificates to Control Plane Server

Transfer control plane certificates to the server host (10.0.0.2):

scp \
  ca.key ca.crt \
  kube-api-server.key kube-api-server.crt \
  service-accounts.key service-accounts.crt \
  root@server:~/

4.3 Summary of Host Placement & Roles

Node Host IP Address Target Directory Files Copied Functionality
server 10.0.0.2 /root/ ca.crt, ca.key, kube-api-server.crt, .key, service-accounts.crt, .key Serves API endpoint on port 6443, authenticates mTLS requests, and signs/verifies Service Account JWTs.
node-0 10.0.0.3 /var/lib/kubelet/ ca.crt, kubelet.crt (renamed from node-0.crt), kubelet.key (renamed from node-0.key) Authenticates node-0 (CN=system:node:node-0, O=system:nodes) to kube-apiserver and secures local endpoint (10250).
node-1 10.0.0.4 /var/lib/kubelet/ ca.crt, kubelet.crt (renamed from node-1.crt), kubelet.key (renamed from node-1.key) Authenticates node-1 (CN=system:node:node-1, O=system:nodes) to kube-apiserver and secures local endpoint (10250).
jumpbox 10.0.0.1 /root/kubernetes-the-hard-way/ admin.*, kube-proxy.*, kube-scheduler.*, kube-controller-manager.* Retains remaining client keypairs to generate .kubeconfig authentication files in the next lab.

4.4 Real-World Troubleshooting & Pitfalls

Gotcha 1: Missing SAN Extensions in Signed Certificates

  • Symptom: Clients or nodes fail to connect to kube-apiserver, logging errors like: x509: certificate is valid for 127.0.0.1, not 10.0.0.2 or x509: certificate relies on legacy Common Name field, not SANs.
  • Root Cause: Executing openssl x509 without -copy_extensions copyall causes OpenSSL to strip the SAN fields defined in the CSR when emitting the signed .crt.
  • Fix: Always verify certificate SANs using OpenSSL before deployment, ensuring -copy_extensions copyall was passed during generation.

Gotcha 2: Incorrect CN or O Mappings Leading to RBAC Failures

  • Symptom: Worker nodes join the cluster but show 403 Forbidden errors in kubelet.log when trying to post node status.
  • Root Cause: Typos in ca.conf under section blocks (e.g., setting CN=node-0 instead of CN=system:node:node-0, or omitting O=system:nodes).
  • Fix: Ensure exact prefixing for system components as specified by Kubernetes Node Authorizer specs.

4.5 Verification & Validation

Inspect the generated certificates on the jumpbox using openssl x509 to confirm identity fields (CN, O) and Subject Alternative Names (SANs).

1. Verify Administrative Client Certificate (admin.crt)

Check that admin.crt carries superuser administrative privileges (O=system:masters):

openssl x509 -in admin.crt -text -noout | grep -E "Subject:"
Expected Terminal Output:
Subject: C=US, ST=Washington, L=Seattle, O=system:masters, OU=Kubernetes The Hard Way, CN=admin

2. Verify API Server Certificate SANs (kube-api-server.crt)

Check that kube-api-server.crt contains all necessary IP addresses and hostnames:

openssl x509 -in kube-api-server.crt -text -noout | grep -A 1 "Subject Alternative Name"
Expected Terminal Output:
            X509v3 Subject Alternative Name: 
                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.0.0.2, IP Address:127.0.0.1

3. Verify Remote Distribution on Control Plane (server)

Execute a remote check on server (10.0.0.2) to ensure files arrived safely:

ssh root@server "ls -l /root/*.crt /root/*.key"
Expected Terminal Output:
-rw-r--r-- 1 root root 2098 Oct 31 10:50 /root/ca.crt
-rw------- 1 root root 3243 Oct 31 10:50 /root/ca.key
-rw-r--r-- 1 root root 2212 Oct 31 10:50 /root/kube-api-server.crt
-rw------- 1 root root 3243 Oct 31 10:50 /root/kube-api-server.key
-rw-r--r-- 1 root root 2086 Oct 31 10:50 /root/service-accounts.crt
-rw------- 1 root root 3243 Oct 31 10:50 /root/service-accounts.key
Actual Terminal Output run:
# 04 certificate authority

root@jumpbox:~/kubernetes-the-hard-way# certs=(
  "admin" "node-0" "node-1"
  "kube-proxy" "kube-scheduler"
  "kube-controller-manager"
  "kube-api-server"
  "service-accounts"
)
root@jumpbox:~/kubernetes-the-hard-way# for i in ${certs[*]}; do
  openssl genrsa -out "${i}.key" 4096

  openssl req -new -key "${i}.key" -sha256 \
    -config "ca.conf" -section ${i} \
    -out "${i}.csr"

  openssl x509 -req -days 3653 -in "${i}.csr" \
    -copy_extensions copyall \
    -sha256 -CA "ca.crt" \
    -CAkey "ca.key" \
    -CAcreateserial \
    -out "${i}.crt"
done
Certificate request self-signature ok
subject=CN = admin, O = system:masters
Certificate request self-signature ok
subject=CN = system:node:node-0, O = system:nodes, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:node:node-1, O = system:nodes, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:kube-proxy, O = system:node-proxier, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:kube-scheduler, O = system:system:kube-scheduler, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:kube-controller-manager, O = system:kube-controller-manager, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = kubernetes, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = service-accounts
root@jumpbox:~/kubernetes-the-hard-way# 

root@jumpbox:~/kubernetes-the-hard-way# ls -1 *.crt *.key *.csr
admin.crt
admin.csr
admin.key
ca.crt
ca.key
kube-api-server.crt
kube-api-server.csr
kube-api-server.key
kube-controller-manager.crt
kube-controller-manager.csr
kube-controller-manager.key
kube-proxy.crt
kube-proxy.csr
kube-proxy.key
kube-scheduler.crt
kube-scheduler.csr
kube-scheduler.key
node-0.crt
node-0.csr
node-0.key
node-1.crt
node-1.csr
node-1.key
service-accounts.crt
service-accounts.csr
service-accounts.key
root@jumpbox:~/kubernetes-the-hard-way# for host in node-0 node-1; do
  ssh root@${host} mkdir /var/lib/kubelet/

  scp ca.crt root@${host}:/var/lib/kubelet/

  scp ${host}.crt \
    root@${host}:/var/lib/kubelet/kubelet.crt

  scp ${host}.key \
    root@${host}:/var/lib/kubelet/kubelet.key
done
ca.crt                                                                 100% 1899    13.6MB/s   00:00    
node-0.crt                                                             100% 2147    13.0MB/s   00:00    
node-0.key                                                             100% 3272    14.7MB/s   00:00    
ca.crt                                                                 100% 1899    12.9MB/s   00:00    
node-1.crt                                                             100% 2147    14.1MB/s   00:00    
node-1.key                                                             100% 3272    18.4MB/s   00:00    
root@jumpbox:~/kubernetes-the-hard-way# scp \
  ca.key ca.crt \
  kube-api-server.key kube-api-server.crt \
  service-accounts.key service-accounts.crt \
  root@server:~/
ca.key                                                                 100% 3272    14.9MB/s   00:00    
ca.crt                                                                 100% 1899    19.4MB/s   00:00    
kube-api-server.key                                                    100% 3272    26.7MB/s   00:00    
kube-api-server.crt                                                    100% 2354    23.4MB/s   00:00    
service-accounts.key                                                   100% 3268    37.2MB/s   00:00    
service-accounts.crt                                                   100% 2004    23.8MB/s   00:00    
root@jumpbox:~/kubernetes-the-hard-way# 

4.6 Section Summary

You have successfully provisioned a Public Key Infrastructure (PKI) for the cluster, issued X.509 client/server certificates mapped to Kubernetes RBAC identities, and securely distributed credentials to control plane and worker nodes. In the next section, you will use the remaining client certificates on the jumpbox to generate authentication .kubeconfig files for kubectl, kube-proxy, kube-scheduler, and kube-controller-manager.


5. Kubernetes Configuration Files (docs/05-kubernetes-configuration-files.md)

  • Kubeconfig Generation: Packaging API server endpoints and embedded TLS credentials using kubectl config:

  • node-0.kubeconfig and node-1.kubeconfig (using Node Authorizer credentials)

  • kube-proxy.kubeconfig

  • kube-controller-manager.kubeconfig

  • kube-scheduler.kubeconfig

  • admin.kubeconfig

  • File Distribution: Pushing generated .kubeconfig files to their respective destinations on worker nodes and control plane controllers.

5.1 Section Overview & Context

In a Kubernetes cluster built from scratch, components do not communicate over open, unauthenticated channels. Every core daemon—from the kubelet agents running on worker nodes to the kube-scheduler on the control plane—must present a verified identity when making API requests to kube-apiserver.

While TLS certificates (generated in the previous lab) provide cryptographic proof of identity, Kubernetes components require a standardized configuration wrapper to locate the API server, verify its server certificate, and attach client credentials. This manifest is a kubeconfig file.

In this section, you will generate and distribute self-contained kubeconfig files for all core system components and administrative users across your 4-node Proxmox LXC infrastructure:

                                  +-----------------------------+
                                  |           JUMPBOX           |
                                  |          (10.0.0.1)         |
                                  | Generates ALL .kubeconfigs  |
                                  +--------------+--------------+
                                                 |
                             +-------------------+-------------------+
                             | SCP                                   | SCP
                             v                                       v
+--------------------------+                   +--------------------------+
|          SERVER          |                   |      NODE-0 / NODE-1      |
|        (10.0.0.2)        |                   |   (10.0.0.3 / 10.0.0.4)  |
| /root/                   |                   | /var/lib/kubelet/        |
| - admin.kubeconfig       |                   | - kubeconfig (node-X)    |
| - kube-controller-       |                   | /var/lib/kube-proxy/     |
|   manager.kubeconfig     |                   | - kubeconfig             |
| - kube-scheduler.        |                   +--------------------------+
|   kubeconfig             |
+--------------------------+

5.2 Technical Understanding & Deep Dive (The “Why”)

Anatomy of a kubeconfig File

A kubeconfig file is a structured YAML document defined by the client.authentication.k8s.io/v1 API schema. It organises cluster access around three distinct structures:

  1. clusters: Defines where to connect (API Server URL endpoint, e.g., [https://server.kubernetes.local:6443](https://server.kubernetes.local:6443)) and how to verify the server (the Root CA certificate).
  2. users: Defines who is connecting (client public certificate data and private key data).
  3. contexts: Binds a specific User identity to a specific Cluster profile under a designated context name (e.g., default). Setting current-context instructs kubectl or system daemons which identity to present by default.
apiVersion: v1
kind: Config
preferences: {}
clusters:
- cluster:
    certificate-authority-data: <BASE64_ENCODED_CA_CERT>
    server: https://server.kubernetes.local:6443
  name: kubernetes-the-hard-way
users:
- name: system:node:node-0
  user:
    client-certificate-data: <BASE64_ENCODED_CLIENT_CERT>
    client-key-data: <BASE64_ENCODED_CLIENT_KEY>
contexts:
- context:
    cluster: kubernetes-the-hard-way
    user: system:node:node-0
  name: default
current-context: default

The Four Key kubectl config Subcommands

Instead of manually editing YAML files—which is prone to indentation and syntax errors—you construct kubeconfig files using the kubectl config command utility:

  • kubectl config set-cluster: Appends or updates an entry in the clusters array.
  • kubectl config set-credentials: Appends or updates authentication credentials in the users array.
  • kubectl config set-context: Defines a logical mapping rule in the contexts array.
  • kubectl config use-context: Sets the current-context field in the root of the configuration document.

Why Use --embed-certs=true?

By default, passing --certificate-authority=ca.crt or --client-certificate=node-0.crt to kubectl config stores local filesystem path references inside the generated file. If you move that .kubeconfig file to a target node without also copying the associated .crt and .key files to the exact same directory path, authentication fails.

Setting --embed-certs=true instructs kubectl to read the target certificate and key files, base64-encode their raw contents, and write them directly into the YAML parameters:

  • certificate-authority-data
  • client-certificate-data
  • client-key-data

This renders every generated .kubeconfig file 100% standalone, self-contained, and portable.

Identity, RBAC, and the Node Authorizer

Kubernetes lacks a database of static user accounts. When a client presents a TLS certificate to kube-apiserver, the API server decrypts the client certificate and inspects its X.509 fields:

  • Subject Common Name (CN): Interpreted as the User.
  • Organization (O): Interpreted as the Group(s).

The API server then passes these values to its configured authorization modules:

  • Worker Nodes (node-0.kubeconfig, node-1.kubeconfig): Signed with CN=system:node:node-0 and O=system:nodes. Requests containing this credential are processed by the Node Authorizer. This specialized authorizer strictly enforces least privilege: node-0 can only read/write Pod specs, secrets, and status updates assigned to node-0. It is blocked from accessing resources belonging to node-1.
  • System Components (kube-proxy, kube-controller-manager, kube-scheduler): Configured with dedicated system users (CN=system:kube-proxy, CN=system:kube-scheduler). These match pre-defined cluster Role-Based Access Control (RBAC) system bindings (cluster-admin, system:kube-scheduler, etc.).
  • Admin User (admin.kubeconfig): Signed with CN=admin and O=system:masters. The system:masters organization group bypasses RBAC checks, granting full root-level control over the cluster.

Target Endpoint Differences: Public Domain vs. Local Loopback

Notice the difference in API server endpoints across the generated files:

  • Component Configs (server.kubernetes.local:6443): System daemons running across different nodes (node-0, node-1) communicate with kube-apiserver over the internal network using the cluster’s domain name or control plane IP address.
  • Admin Config (127.0.0.1:6443): Administrative operations using admin.kubeconfig are typically performed directly on the control plane server host (server at 10.0.0.2) or via SSH tunnels. Pointing to 127.0.0.1 ensures administrative access remains local to the control plane, bypassing external ingress routes and network dependency failures.

5.3 Step-by-Step Implementation & Command Breakdown

All commands in this section must be executed from your jumpbox (10.0.0.1) inside the working directory where your CA and TLS certificates were generated (/root/kubernetes-the-hard-way).

Step 1: Generate Kubelet Configuration Files for Worker Nodes

Run the following loop to construct node-0.kubeconfig and node-1.kubeconfig:

for host in node-0 node-1; do
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=https://server.kubernetes.local:6443 \
    --kubeconfig=${host}.kubeconfig

  kubectl config set-credentials system:node:${host} \
    --client-certificate=${host}.crt \
    --client-key=${host}.key \
    --embed-certs=true \
    --kubeconfig=${host}.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=system:node:${host} \
    --kubeconfig=${host}.kubeconfig

  kubectl config use-context default \
    --kubeconfig=${host}.kubeconfig
done
Command Breakdown:
  • --server=[https://server.kubernetes.local:6443](https://server.kubernetes.local:6443): Directs worker node traffic to the control plane API server.
  • --client-certificate=${host}.crt: Uses the node-specific TLS certificate (CN=system:node:node-0).
  • --kubeconfig=${host}.kubeconfig: Writes the generated configuration into an isolated, node-specific file.

Step 2: Generate the kube-proxy Configuration File

Generate the configuration file for the network proxy daemon running on worker nodes:

{
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=https://server.kubernetes.local:6443 \
    --kubeconfig=kube-proxy.kubeconfig

  kubectl config set-credentials system:kube-proxy \
    --client-certificate=kube-proxy.crt \
    --client-key=kube-proxy.key \
    --embed-certs=true \
    --kubeconfig=kube-proxy.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=system:kube-proxy \
    --kubeconfig=kube-proxy.kubeconfig

  kubectl config use-context default \
    --kubeconfig=kube-proxy.kubeconfig
}

Step 3: Generate the kube-controller-manager Configuration File

Generate the configuration file for the background controller loops running on the control plane:

{
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=https://server.kubernetes.local:6443 \
    --kubeconfig=kube-controller-manager.kubeconfig

  kubectl config set-credentials system:kube-controller-manager \
    --client-certificate=kube-controller-manager.crt \
    --client-key=kube-controller-manager.key \
    --embed-certs=true \
    --kubeconfig=kube-controller-manager.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=system:kube-controller-manager \
    --kubeconfig=kube-controller-manager.kubeconfig

  kubectl config use-context default \
    --kubeconfig=kube-controller-manager.kubeconfig
}

Step 4: Generate the kube-scheduler Configuration File

Generate the configuration file for the cluster workload scheduler:

{
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=https://server.kubernetes.local:6443 \
    --kubeconfig=kube-scheduler.kubeconfig

  kubectl config set-credentials system:kube-scheduler \
    --client-certificate=kube-scheduler.crt \
    --client-key=kube-scheduler.key \
    --embed-certs=true \
    --kubeconfig=kube-scheduler.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=system:kube-scheduler \
    --kubeconfig=kube-scheduler.kubeconfig

  kubectl config use-context default \
    --kubeconfig=kube-scheduler.kubeconfig
}

Step 5: Generate the admin Configuration File

Generate the superuser administrative configuration file for cluster management:

{
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=https://127.0.0.1:6443 \
    --kubeconfig=admin.kubeconfig

  kubectl config set-credentials admin \
    --client-certificate=admin.crt \
    --client-key=admin.key \
    --embed-certs=true \
    --kubeconfig=admin.kubeconfig

  kubectl config set-context default \
    --cluster=kubernetes-the-hard-way \
    --user=admin \
    --kubeconfig=admin.kubeconfig

  kubectl config use-context default \
    --kubeconfig=admin.kubeconfig
}

Step 6: Distribute Configuration Files

With all six kubeconfig files generated on the jumpbox, distribute them to their respective destinations using scp.

  1. Distribute to Worker Nodes (node-0 at 10.0.0.3 and node-1 at 10.0.0.4):
for host in node-0 node-1; do
  ssh root@${host} "mkdir -p /var/lib/{kube-proxy,kubelet}"

  scp kube-proxy.kubeconfig \
    root@${host}:/var/lib/kube-proxy/kubeconfig

  scp ${host}.kubeconfig \
    root@${host}:/var/lib/kubelet/kubeconfig
done
  1. Distribute to Control Plane Server (server at 10.0.0.2):
scp admin.kubeconfig \
  kube-controller-manager.kubeconfig \
  kube-scheduler.kubeconfig \
  root@server:~/
File Placement Matrix
Target Host Host IP Target Path Transferred File Renamed To Executing Service
node-0 10.0.0.3 /var/lib/kubelet/ node-0.kubeconfig kubeconfig kubelet
node-0 10.0.0.3 /var/lib/kube-proxy/ kube-proxy.kubeconfig kubeconfig kube-proxy
node-1 10.0.0.4 /var/lib/kubelet/ node-1.kubeconfig kubeconfig kubelet
node-1 10.0.0.4 /var/lib/kube-proxy/ kube-proxy.kubeconfig kubeconfig kube-proxy
server 10.0.0.2 /root/ admin.kubeconfig admin.kubeconfig Administrative CLI (kubectl)
server 10.0.0.2 /root/ kube-controller-manager.kubeconfig kube-controller-manager.kubeconfig kube-controller-manager
server 10.0.0.2 /root/ kube-scheduler.kubeconfig kube-scheduler.kubeconfig kube-scheduler

5.4 Real-World Troubleshooting & Pitfalls

Gotcha 1: Omission of --embed-certs=true

  • Symptom: Services start up on worker nodes (node-0, node-1), but instantly fail during API handshake with file errors: error: failed to load kubeconfig: open ca.crt: no such file or directory
  • Root Cause: The kubeconfig was created without --embed-certs=true. It contains path references like certificate-authority: ca.crt instead of raw base64 data under certificate-authority-data.
  • Fix: Regenerate all kubeconfig files using the --embed-certs=true flag before distribution.

Gotcha 2: Node Authorizer Rejection (403 Forbidden)

  • Symptom: kubelet fails to register with kube-apiserver, logging: Node "node-0" is unauthorized to modify node "node-0"
  • Root Cause: The user identity configured in set-credentials did not strictly follow the format system:node:<node-name> during certificate generation, or the context bound the wrong username.
  • Fix: Verify that the certificate Common Name (CN) matches system:node:node-0 and that the credential set in set-context references this exact user string.

Gotcha 3: Secure Permission Denied on Target Directories

  • Symptom: scp transfers fail with No such file or directory or Permission denied.
  • Root Cause: Destination system paths (/var/lib/kubelet, /var/lib/kube-proxy) do not exist on clean base OS installations.
  • Fix: Always ensure target directories are created beforehand using ssh root@${host} "mkdir -p /var/lib/{kube-proxy,kubelet}".

Verification & Validation

1. Confirm Local Generation on Jumpbox

Verify that all six .kubeconfig files were generated successfully in your working directory:

ls -1 *.kubeconfig
Expected Output:
admin.kubeconfig
kube-controller-manager.kubeconfig
kube-proxy.kubeconfig
kube-scheduler.kubeconfig
node-0.kubeconfig
node-1.kubeconfig

2. Validate Embedded Data Integrity

Inspect the structure of a generated file (e.g., admin.kubeconfig) to confirm that certificates are embedded as raw base64 data rather than path references:

kubectl config view --kubeconfig=admin.kubeconfig
Expected Output:
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://127.0.0.1:6443
  name: kubernetes-the-hard-way
contexts:
- context:
    cluster: kubernetes-the-hard-way
    user: admin
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: admin
  user:
    client-certificate-data: DATA+OMITTED
    client-key-data: DATA+OMITTED

3. Verify Remote File Distribution

Verify that files are in place on worker nodes (node-0 and node-1) and the control plane server (server):

ssh root@node-0 "ls -la /var/lib/kubelet/kubeconfig /var/lib/kube-proxy/kubeconfig"
ssh root@server "ls -la ~/admin.kubeconfig ~/kube-controller-manager.kubeconfig ~/kube-scheduler.kubeconfig"
Expected Output:
-rw------- 1 root root 6240 Oct 31 10:57 /var/lib/kube-proxy/kubeconfig
-rw------- 1 root root 6284 Oct 31 10:57 /var/lib/kubelet/kubeconfig

-rw------- 1 root root 6210 Oct 31 10:57 /root/admin.kubeconfig
-rw------- 1 root root 6320 Oct 31 10:57 /root/kube-controller-manager.kubeconfig
-rw------- 1 root root 6280 Oct 31 10:57 /root/kube-scheduler.kubeconfig

5.5 Section Summary

You have successfully generated self-contained Kubernetes configuration files for every system component and distributed them across your control plane and worker nodes. Each daemon now possesses a dedicated, portable identity manifest configured for mutual TLS authentication and strict role-based access control.

Next, you will generate the cluster-wide data encryption configuration and key to secure secret data at rest in etcd.


6. Data Encryption at Rest (docs/06-data-encryption-keys.md)

  • Encryption Key Generation: Extracting a 32-byte Base64 key from /dev/urandom.
  • Config Creation: Generating encryption-config.yaml with the aescbc provider specification to secure Kubernetes Secrets inside etcd.

5.1 Section Overview & Context

In a production-grade Kubernetes cluster, securing persistent state is just as critical as securing network traffic. Kubernetes relies on etcd as its primary key-value datastore for all cluster state, configurations, and sensitive credentials like Secret objects.

By default, etcd stores data as unencrypted plaintext. If an attacker gains unauthorized access to etcd backups, storage snapshots, or the underlying filesystem on the control plane host (server at 10.0.0.2), they can read raw authentication tokens, private keys, and API credentials.

+-----------------------------------------------------------------------------------+
|                                  JUMPBOX (10.0.0.1)                               |
|  1. Generate 256-bit Key  -->  2. Render Config  -->  3. SCP to Control Plane    |
+-----------------------------------------------------------------------------------+
                                                                 |
                                                                 v
+-----------------------------------------------------------------------------------+
|                               CONTROL PLANE (10.0.0.2)                            |
|  /var/lib/kubernetes/encryption-config.yaml                                      |
|                                                                                   |
|    +-------------------+       Encrypts        +-----------------------------+    |
|    |  kube-apiserver   | --------------------> | etcd (/var/lib/etcd)        |    |
|    |  (v1.32)          | <-------------------- | (Ciphertext stored on disk) |    |
|    +-------------------+       Decrypts        +-----------------------------+    |
+-----------------------------------------------------------------------------------+

In this step, you will generate a cryptographically secure 256-bit symmetric key on the jumpbox (10.0.0.1) and render an EncryptionConfiguration manifest. This manifest instructs kube-apiserver (v1.32) to transparently encrypt Secret resources in-flight using AES-CBC mode before persisting them into etcd.


5.2 Technical Understanding & Deep Dive (The “Why”)

Base64 Encoding vs. Cryptographic Encryption

A common misconception among Kubernetes operators is that Kubernetes Secret objects are encrypted by default because their values appear as encoded strings (e.g., dGVzdA==) when queried with kubectl get secret -o yaml.

Base64 is an encoding scheme designed for safe data transport over text protocols—it provides zero confidentiality. Anyone with read access to the cluster API or etcd can instantly decode Base64 strings using standard tools (echo "dGVzdA==" | base64 -d). True Encryption at Rest requires transforming plaintext data into unreadable ciphertext using a secret key and a cryptographic algorithm.

In-Flight Envelope Encryption via kube-apiserver

Kubernetes handles encryption at the API layer through kube-apiserver:

  1. Write Path: When a client posts a Secret payload to kube-apiserver, the API server intercepts the request, reads its active EncryptionConfiguration, encrypts the payload in memory using the defined key, and writes the resulting ciphertext to etcd.
  2. Read Path: When an authorized client requests a Secret, kube-apiserver reads the ciphertext from etcd, decrypts it using the matching key, and returns the payload to the authenticated client over TLS.

Deconstructing encryption-config.yaml

The EncryptionConfiguration structure defines which resources are protected and which encryption algorithms (providers) are applied:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <BASE64_ENCRYPTION_KEY>
      - identity: {}
  • resources: Specifies the array of Kubernetes API resources to encrypt. Here, only secrets are targeted.
  • providers: An ordered list of encryption providers evaluated sequentially:
  • aescbc (AES-CBC with PKCS#7 padding): Placed first. All new write operations to etcd are encrypted using AES in Cipher Block Chaining mode with key1.
  • identity: {}: Placed second. Represents plaintext (no encryption). This allows kube-apiserver to read existing, unencrypted secrets stored in etcd prior to enabling encryption. Upon updating an unencrypted secret, kube-apiserver automatically rewrites it using the primary provider (aescbc).

5.3 Step-by-Step Implementation & Command Breakdown

Perform all commands from the repository root directory on your jumpbox (10.0.0.1).

Step 1: Generate the Symmetric Encryption Key

Generate a 32-byte (256-bit) cryptographically secure random key and store it in an environment variable:

cd /root/kubernetes-the-hard-way

export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
Command Breakdown:
  • /dev/urandom: The Linux kernel entropy device providing non-blocking pseudo-random bytes.
  • head -c 32: Extracts exactly 32 bytes (256 bits) from the entropy stream.
  • | base64: Pipes the raw binary data into base64 to encode it as a 44-character ASCII string compatible with YAML configuration files.
  • export ENCRYPTION_KEY=...: Assigns the value to the shell environment for tool substitution.

Step 2: Render encryption-config.yaml

Substitute the environment variable into the configuration template using envsubst:

envsubst < configs/encryption-config.yaml > encryption-config.yaml
Command Breakdown:
  • envsubst: Scans the input stream for references like $ENCRYPTION_KEY and replaces them with the actual environment string.
  • < configs/encryption-config.yaml: Standard input redirection feeding the template file.
  • > encryption-config.yaml: Standard output redirection writing the rendered manifest to disk.

Step 3: Inspect the Rendered Configuration

Verify that the template populated correctly with your Base64 string:

cat encryption-config.yaml
Realistic Terminal Output:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: 4Xz8vK9mP2qR5wT8yU1vC3bN6mQ9xZ2aL5pK8jH3fG4=
      - identity: {}

Step 4: Transfer the Manifest to the Control Plane

Copy the configuration file over SSH to the control plane host (server at 10.0.0.2):

scp encryption-config.yaml root@server:~/
Realistic Terminal Output:
encryption-config.yaml                        100%  243   187.4KB/s   00:00

Placement & Component Matrix

Host Machine File Transferred Destination Path Target Component Operational Purpose
server (10.0.0.2) encryption-config.yaml /root/ (staged for /var/lib/kubernetes/) kube-apiserver Instructs API server to encrypt/decrypt Secret payloads stored in etcd.

5.4 Real-World Troubleshooting & Pitfalls

Pitfall 1: envsubst: command not found

  • Symptom: Executing envsubst fails with bash: envsubst: command not found.
  • Root Cause: The GNU gettext-base utility package is not installed on the jumpbox LXC container.
  • Fix: Install the utility package via apt:
apt-get update && apt-get install -y gettext-base

Pitfall 2: Blank secret: Field or Invalid Key Length

  • Symptom: Inspecting encryption-config.yaml shows secret: (empty) or a key shorter than 44 characters.
  • Root Cause: Using head -n 32 instead of head -c 32. The -n flag attempts to read 32 lines instead of 32 bytes, which fails or produces wrong byte counts when handling raw binary data from /dev/urandom.
  • Fix: Re-export the key using -c 32 explicitly and re-render:
export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
envsubst < configs/encryption-config.yaml > encryption-config.yaml

Pitfall 3: SCP Permission Denied on Host Transfer

  • Symptom: scp encryption-config.yaml root@server:~/' returns Permission denied (publickey)`.
  • Root Cause: Missing SSH key setup between jumpbox and server or improper permissions on /root/.ssh/authorized_keys on server.
  • Fix: Verify connectivity and key propagation:
ssh root@server "mkdir -p ~/.ssh && chmod 700 ~/.ssh"
scp encryption-config.yaml root@server:~/

5.5 Verification & Validation

Verify that the key generation and configuration deployment were executed accurately before proceeding to etcd bootstrapping.

1. Validate Secret String Character Count

A valid 256-bit Base64-encoded key must be exactly 44 characters long (including trailing = padding characters):

grep "secret:" encryption-config.yaml | awk '{print $2}' | wc -c
Expected Output:
45

(Note: wc -c counts 44 ASCII characters plus 1 trailing newline character).

2. Confirm File Presence on Control Plane

Query server (10.0.0.2) remotely via SSH to ensure the manifest was delivered cleanly:

ssh root@server "ls -la ~/encryption-config.yaml"
Expected Output:
-rw-r--r-- 1 root root 243 Oct 31 11:18 /root/encryption-config.yaml

5.6 Section Summary

You have generated a symmetric 256-bit encryption key and staged the EncryptionConfiguration manifest on the control plane host (server). This ensures kube-apiserver is equipped to secure Kubernetes secrets at rest once the control plane services are started.


7. Bootstrapping etcd (docs/07-bootstrapping-etcd.md)

  • Etcd Installation: Installing etcd and etcdctl binaries on the server node.
  • Configuration & Systemd: Configuring /etc/etcd/ certificates, deploying etcd.service, and starting the systemd unit.
  • Cluster Verification: Confirming cluster member health and status using etcdctl member list.

Kubernetes components are entirely stateless. They do not store cluster state in local memory or on container disk images. Instead, all persistent state—every Pod definition, Service specification, Secret, ConfigMap, and Role-Based Access Control (RBAC) policy—is stored in etcd.

In this section, you will bootstrap a single-node etcd database cluster on your control plane server (server at 10.0.0.2).

+-----------------------------------------------------------------------+
|                         CONTROL PLANE (server)                        |
|                                                                       |
|  +---------------------+                   +-----------------------+  |
|  |   kube-apiserver    |  <-- mTLS (2379)  |         etcd          |  |
|  | (Stateless Gateway) |                   |  (State Storage Engine|  |
|  +---------------------+                   +-----------------------+  |
|                                                        |              |
|                                               WAL / Snapshot DB       |
|                                            (/var/lib/etcd: 0700)      |
+-----------------------------------------------------------------------+

Bootstrapping etcd involves provisioning the executable binaries, configuring Mutual TLS (mTLS) authentication certificates, defining a dedicated systemd unit file, and hardening the persistent storage directory on disk.


7.1 Technical Understanding & Deep Dive: The “Why”

1. Raft Consensus and Persistent Data Storage

etcd is a strongly consistent, distributed key-value store implementing the Raft consensus algorithm. Raft ensures that all writes are committed to a Write-Ahead Log (WAL) before updating the main database file (db) located in /var/lib/etcd.

Even in a single-node setup, etcd maintains strict transaction logs to guarantee data durability across system reboots. Because etcd holds sensitive cluster secrets in plaintext or encrypted formats, directory permissions must be restricted to 0700 (rwx------), ensuring only the root user or dedicated etcd service accounts can inspect raw database files.

2. The Isolation of kube-apiserver

In a healthy Kubernetes architecture, no component communicates with etcd directly except the kube-apiserver. Controller managers, schedulers, and worker node kubelet daemons all direct their read and write requests to kube-apiserver, which acts as a secure, validating gateway in front of etcd.

3. Network Ports and Dual-Listener Architecture

etcd utilizes two distinct networking ports:

  • Port 2379 (Client Traffic): Used for client-to-etcd communication (e.g., kube-apiserver querying state or etcdctl performing administrative tasks).
  • Port 2380 (Peer Traffic): Used exclusively for node-to-node replication, leader election, and Raft consensus synchronization across cluster peers.
4. PKI and Mutual TLS (mTLS) Authentication

To prevent unauthorized access, etcd enforces mTLS on both client and peer interfaces. Under mTLS:

  1. The server presents its certificate (kube-api-server.crt) to prove its identity to the client.
  2. The client (kube-apiserver or etcdctl) presents its own client certificate signed by the Certificate Authority (ca.crt).
  3. Connection is granted only if both parties successfully validate each other’s certificates against the shared CA.

7.2 Step-by-Step Implementation & Command Breakdown

Step 1: Copy Binaries and Configuration to the Control Plane

From your management host (jumpbox at 10.0.0.1), transfer the pre-downloaded binaries and systemd service template to the server host (10.0.0.2):

cd /root/kubernetes-the-hard-way

scp \
  downloads/controller/etcd \
  downloads/client/etcdctl \
  units/etcd.service \
  root@server:~/

Next, establish an SSH session into the server node:

ssh root@server
Step 2: Install Binaries into the System Path

On server, place the etcd daemon and etcdctl CLI tool into the system executable path:

mv etcd etcdctl /usr/local/bin/
  • /usr/local/bin/: Standard directory for custom system binaries included in system-wide $PATH, enabling execution without requiring relative file paths.
Step 3: Configure Directories, Permissions, and TLS Certificates

Create the configuration and data storage directories, apply strict POSIX file permissions, and copy the required TLS credentials:

mkdir -p /etc/etcd /var/lib/etcd
chmod 700 /var/lib/etcd
cp ca.crt kube-api-server.key kube-api-server.crt /etc/etcd/
  • /etc/etcd/: Holds configuration files, root CA certificates, and private keypairs.
  • /var/lib/etcd: The physical storage path for WAL and database snapshot files.
  • chmod 700 /var/lib/etcd: Grants read, write, and execute permissions strictly to root (rwx------), shielding cluster state from unprivileged system users.
Step 4: Provision the systemd Unit File

Move the etcd.service configuration into the system manager path:

mv etcd.service /etc/systemd/system/

Inspect the parameter flags configured inside /etc/systemd/system/etcd.service:

[Unit]
Description=etcd
Documentation=https://github.com/etcd-io/etcd

[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
  --name server \
  --cert-file=/etc/etcd/kube-api-server.crt \
  --key-file=/etc/etcd/kube-api-server.key \
  --peer-cert-file=/etc/etcd/kube-api-server.crt \
  --peer-key-file=/etc/etcd/kube-api-server.key \
  --trusted-ca-file=/etc/etcd/ca.crt \
  --peer-trusted-ca-file=/etc/etcd/ca.crt \
  --peer-client-cert-auth \
  --client-cert-auth \
  --initial-advertise-peer-urls https://10.0.0.2:2380 \
  --listen-peer-urls https://10.0.0.2:2380 \
  --listen-client-urls https://127.0.0.1:2379,https://10.0.0.2:2379 \
  --advertise-client-urls https://10.0.0.2:2379 \
  --initial-cluster-token etcd-cluster-token \
  --initial-cluster server=https://10.0.0.2:2380 \
  --initial-cluster-state new \
  --data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
Unit Configuration Breakdown
Flag / Parameter Technical Purpose
--name server Assigns a unique node identifier within the etcd cluster topology.
**--cert-file / --key-file** Server TLS certificate and private key used to secure client connections on port 2379.
--trusted-ca-file Root Certificate Authority used to authenticate incoming client TLS connections.
--client-cert-auth Enforces Mutual TLS (mTLS); requires clients to present a valid TLS certificate.
**--peer-cert-file / --peer-key-file** TLS certificate pair used for peer-to-peer cluster consensus on port 2380.
--listen-client-urls Network sockets etcd binds to for client traffic (127.0.0.1:2379 and 10.0.0.2:2379).
--advertise-client-urls Endpoint address published to cluster clients for establishing API connections.
--listen-peer-urls Network socket bound for internal Raft consensus traffic between peer nodes (10.0.0.2:2380).
--initial-cluster Bootstrap configuration defining initial cluster members and peer URLs.
--data-dir Path to the persistent directory where WAL and database state files are stored.
Step 5: Start and Enable the etcd Service

Reload the systemd daemon to pick up the new unit configuration, enable automatic launch at boot, and start etcd:

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd

7.3 Real-World Troubleshooting & Pitfalls

The Error Symptom

Executing etcdctl member list with TLS flags fails with an authentication error:

rpc error: code = DeadlineExceeded desc = context deadline exceeded; ... authentication handshake failed: EOF

Conversely, running an unencrypted etcdctl member list without flags unexpectedly succeeds, displaying plaintext endpoints:

6702b0a34e2cfd39, started, controller, http://127.0.0.1:2380, http://127.0.0.1:2379, false
Underlying Root Cause

This issue occurs due to a Protocol Mismatch (HTTP vs. HTTPS/TLS) between the running etcd daemon and the client:

  1. The etcd.service unit was executed without TLS flags or was configured with plaintext http:// listeners.
  2. When etcdctl initiates an encrypted TLS handshake packet (TLS Client Hello) against [https://127.0.0.1:2379](https://127.0.0.1:2379), the plaintext HTTP listener on etcd fails to decode the encrypted binary stream.
  3. The server immediately closes the underlying TCP socket connection without returning a TLS alert, causing etcdctl to report an unexpected EOF (End of File).
Why This Matters for Kubernetes

The kube-apiserver mandates strict mTLS encryption when communicating with etcd. If etcd runs in unencrypted HTTP mode, kube-apiserver will fail to establish a database connection, halting control plane initialization in subsequent labs.

The Fix

Verify and update /etc/systemd/system/etcd.service to ensure all listener and advertise URLs explicitly specify the https:// protocol, and that certificate flags are set. Then reload systemd and restart the service:

# Apply updated systemd parameters
systemctl daemon-reload
systemctl restart etcd

Verification & Validation

1. Inspect Systemd Service Health

Verify that the etcd daemon process is active and running cleanly:

systemctl status etcd --no-pager

Expected Healthy Output:

● etcd.service - etcd
     Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2026-10-31 12:15:00 UTC; 10s ago
       Docs: https://github.com/etcd-io/etcd
   Main PID: 1420 (etcd)
      Tasks: 8 (limit: 4652)
     Memory: 24.1M
     CGroup: /system.slice/etcd.service
             └─1420 /usr/local/bin/etcd --name server ...
2. Verify Cluster Membership over mTLS

Execute etcdctl member list, providing the root CA and API server certificate pair to validate client-side mTLS communication:

etcdctl member list \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/etcd/ca.crt \
  --cert=/etc/etcd/kube-api-server.crt \
  --key=/etc/etcd/kube-api-server.key

Expected Healthy Output:

6702b0a34e2cfd39, started, server, https://10.0.0.2:2380, https://10.0.0.2:2379, false

The output confirms that member 6702b0a34e2cfd39 named server is active, listening on secure https:// endpoints, and ready to accept control plane transactions.


8. Bootstrapping Control Plane Controllers (docs/08-bootstrapping-kubernetes-controllers.md)

  • Control Plane Binary Setup: Installing kube-apiserver, kube-controller-manager, kube-scheduler, and kubectl on server.

  • Service Configuration:

  • Deploying kube-apiserver.service with encryption providers and PKI flags.

  • Configuring kube-controller-manager.service and kube-scheduler.service with dedicated kubeconfigs.

  • RBAC Authorization: Applying kube-apiserver-to-kubelet.yaml to grant the API server access to worker node Kubelet APIs.

  • Control Plane Verification: Validating API server initialization via kubectl cluster-info and HTTPS version endpoints.

8.1 Section Overview & Context

In this lab, you will assemble the core control plane of your Kubernetes cluster on the dedicated server host (10.0.0.2). Having previously initialized etcd as your distributed storage backend, you are now ready to deploy the three core management daemons that transform a single database into a functioning, self-healing container orchestrator:

  1. kube-apiserver: The central administrative hub and sole gatekeeper to etcd.
  2. kube-controller-manager: The background engine executing continuous state reconciliation loops.
  3. kube-scheduler: The decision engine evaluating resource availability and binding workloads to worker nodes.

8.2 Technical Understanding & Deep Dive (The “Why”)

Before launching systemd services, it is critical to understand how these control plane components interact securely and why specific flags and configurations are necessary.

The Kubernetes Control Plane Architecture

The control plane components form a Hub-and-Spoke topology centered around kube-apiserver:

                               +--------------------------------------------------+
                               |            CONTROL PLANE NODE (server)           |
                               |                     10.0.0.2                     |
                               |                                                  |
  kubectl (admin) ------------>+--> [ kube-apiserver ] <---> [ etcd (Port 2379) ] |
                               |          ^        ^                              |
                               |          |        |                              |
                               |          v        v                              |
                               |  [ controller ]  [ scheduler ]                   |
                               +----------+---------------------------------------+
                                          |
                                          | (HTTPS / Port 10250)
                                          v
                               [ worker nodes (node-0, node-1) ]
Deep Dive: Component Flags & Configurations
kube-apiserver Security & Authorization

The API Server serves as the single entry point for all cluster operations. It must validate requests, enforce authorization, sign service account tokens, and secure state storage:

  • --authorization-mode=Node,RBAC: Enables two authorization layers. Node Authorizer specifically restricts kubelet API requests so worker nodes can only read/modify their own resources. RBAC handles role-based access control for human users, automated scripts, and in-cluster ServiceAccounts.
  • --enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,ResourceQuota: Admission controllers act as inline interceptors after authentication and authorization. For example, NodeRestriction stops a compromised kubelet from altering another node’s labels, taints, or pod statuses.
  • --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml: Ensures that raw Kubernetes Secret data is encrypted using AES-CBC key encryption before being written to etcd, guarding against plain-text database compromises.
  • Service Account JWT Keys: --service-account-signing-key-file (private key) signs JSON Web Tokens issued to Pods, while --service-account-key-file (public key) allows kube-apiserver to verify incoming tokens.
kube-controller-manager State Management

The Controller Manager continuously compares the cluster’s actual state against the desired state stored in etcd:

  • --cluster-cidr=10.200.0.0/16: Reserves the master IP pool for all Pods across the entire cluster.
  • --allocate-node-cidrs=true: Tells the controller manager to automatically slice the /16 cluster range into distinct /24 subnets for each worker node (e.g., node-0 receives 10.200.0.0/24, node-1 receives 10.200.1.0/24).
  • **--cluster-signing-cert-file & --cluster-signing-key-file**: Used by the built-in CertificateSigningRequest (CSR) controller to automatically issue and renew TLS certificates for worker nodes.
kube-scheduler Declarative Configuration

Unlike older Kubernetes releases that used CLI flags, modern versions (such as Kubernetes v1.32) configure kube-scheduler using a declarative KubeSchedulerConfiguration file (kube-scheduler.yaml):

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
clientConnection:
  kubeconfig: "/var/lib/kubernetes/kube-scheduler.kubeconfig"
leaderElection:
  leaderElect: true

Using a YAML configuration allows for modular plugin configuration (filtering, scoring, reserving) while referencing kube-scheduler.kubeconfig to authenticate back to kube-apiserver over TLS.

API Server-to-Kubelet RBAC Authorization

When an administrator runs kubectl exec, kubectl logs, or kubectl top, the request flows through kube-apiserver directly to the kubelet running on a worker node:

$$\text{kubectl} \xrightarrow{\text{HTTPS :6443}} \text{kube-apiserver} \xrightarrow{\text{HTTPS :10250}} \text{kubelet}$$

Because kubelet operates with --authorization-mode=Webhook, it asks kube-apiserver to authorize incoming requests. The system:kube-apiserver-to-kubelet ClusterRole explicitly grants the kubernetes certificate common name (CN) permission to access kubelet endpoints (/exec, /log, /metrics, /stats).


8.3 Step-by-Step Implementation & Command Breakdown

Step 1: Copy Binaries and Units from jumpbox

Execute from /root/kubernetes-the-hard-way on the jumpbox (10.0.0.1) machine to transfer all necessary artifacts to server (10.0.0.2):

cd /root/kubernetes-the-hard-way

scp \
  downloads/controller/kube-apiserver \
  downloads/controller/kube-controller-manager \
  downloads/controller/kube-scheduler \
  downloads/client/kubectl \
  units/kube-apiserver.service \
  units/kube-controller-manager.service \
  units/kube-scheduler.service \
  configs/kube-scheduler.yaml \
  configs/kube-apiserver-to-kubelet.yaml \
  root@server:~/
Step 2: Organize Binaries, Certificates, and Service Units on server

SSH into the server machine:

ssh root@server

Create the standard directory structures and move files to their designated system locations:

# 1. Create configuration and state directories
mkdir -p /etc/kubernetes/config /var/lib/kubernetes/

# 2. Install executables into system execution path
mv kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/

# 3. Move TLS certificates, keys, and encryption settings into /var/lib/kubernetes/
mv ca.crt ca.key \
  kube-api-server.key kube-api-server.crt \
  service-accounts.key service-accounts.crt \
  encryption-config.yaml \
  /var/lib/kubernetes/

# 4. Configure kube-apiserver systemd service
mv kube-apiserver.service /etc/systemd/system/

# 5. Configure kube-controller-manager kubeconfig and unit
mv kube-controller-manager.kubeconfig /var/lib/kubernetes/
mv kube-controller-manager.service /etc/systemd/system/

# 6. Configure kube-scheduler config and unit
mv kube-scheduler.kubeconfig /var/lib/kubernetes/
mv kube-scheduler.yaml /etc/kubernetes/config/
mv kube-scheduler.service /etc/systemd/system/
File Placement Reference Matrix
Host Machine Placement Directory File Name(s) Function / Serving Port
server /usr/local/bin/ kube-apiserver, kube-controller-manager, kube-scheduler, kubectl Executable control plane binaries
server /var/lib/kubernetes/ ca.crt, ca.key, kube-api-server.crt, .key, service-accounts.crt, .key, encryption-config.yaml TLS credentials, JWT signing keys, and data encryption settings
server /var/lib/kubernetes/ kube-controller-manager.kubeconfig, kube-scheduler.kubeconfig Local component mTLS authentication files
server /etc/kubernetes/config/ kube-scheduler.yaml KubeSchedulerConfiguration declarative settings
server /etc/systemd/system/ kube-apiserver.service, kube-controller-manager.service, kube-scheduler.service Systemd service units managing daemon lifecycle
Ports 10.0.0.2:6443 kube-apiserver Main Kubernetes API Server HTTPS endpoint
Ports 127.0.0.1:10257 kube-controller-manager Local secure metrics and health endpoint
Ports 127.0.0.1:10259 kube-scheduler Local secure metrics and health endpoint
Step 3: Start and Enable Control Plane Services

Reload systemd to register the new unit files, then enable and launch all three daemons:

systemctl daemon-reload

systemctl enable kube-apiserver kube-controller-manager kube-scheduler

systemctl start kube-apiserver kube-controller-manager kube-scheduler

Note: Allow up to 10 seconds for kube-apiserver to complete its initialization, establish mTLS connections to etcd, and start accepting requests.

Step 4: Provision API Server-to-Kubelet RBAC Rules

To allow kube-apiserver to query kubelet instances on worker nodes for logs, metrics, and interactive execution, apply the ClusterRole and ClusterRoleBinding on server:

kubectl apply -f kube-apiserver-to-kubelet.yaml --kubeconfig admin.kubeconfig

Expected Output:

clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created

8.4 Real-World Troubleshooting & Pitfalls

Pitfall 1: Incorrect File Permissions on TLS Private Keys
  • Symptom: kube-apiserver continuously fails or restarts, and journalctl -u kube-apiserver displays permission denied errors when reading /var/lib/kubernetes/ca.key or kube-api-server.key.
  • Root Cause: Private key files were copied with open or incompatible file permissions (or incorrect ownership).
  • Fix: Enforce standard secure permissions on /var/lib/kubernetes/:
chmod 600 /var/lib/kubernetes/*.key
chmod 644 /var/lib/kubernetes/*.crt
Pitfall 2: etcd Unavailability or TLS Mismatch
  • Symptom: systemctl status kube-apiserver reports fatal connection failures or TLS handshake errors pointing to [https://127.0.0.1:2379](https://127.0.0.1:2379).
  • Root Cause: etcd is not actively running on the local host, or the certificates supplied to --etcd-cafile, --etcd-certfile, and --etcd-keyfile do not match etcd’s client credentials.
  • Fix: Check etcd health before launching control plane services:
systemctl is-active etcd
Pitfall 3: Systemd Unit File Syntax Errors or Missing Reloads
  • Symptom: Executing systemctl start kube-apiserver returns Unit kube-apiserver.service not found or fails silently.
  • Root Cause: The service unit file was placed in /etc/systemd/system/ without re-indexing systemd manager configuration.
  • Fix: Ensure the file exists at /etc/systemd/system/kube-apiserver.service and execute systemctl daemon-reload.

8.5 Verification & Validation

1. Verify Active Service States on server

Run systemctl is-active to confirm all three control plane components are active:

systemctl is-active kube-apiserver kube-controller-manager kube-scheduler

Expected Terminal Output:

active
active
active
2. Check Local Control Plane Health via kubectl

Run kubectl cluster-info on server using the administrator kubeconfig:

kubectl cluster-info --kubeconfig admin.kubeconfig

Expected Terminal Output:

Kubernetes control plane is running at https://127.0.0.1:6443
3. Remote Verification from jumpbox

Return to the jumpbox (10.0.0.1) machine to test external HTTPS access to the API Server version endpoint:

exit
curl --cacert ca.crt https://server.kubernetes.local:6443/version

Expected Terminal Output:

{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.3",
  "gitCommit": "32cc146f75aad04beaaa245a7157eb35063a9f99",
  "gitTreeState": "clean",
  "buildDate": "2025-03-11T19:52:21Z",
  "goVersion": "go1.23.6",
  "compiler": "gc",
  "platform": "linux/arm64"
}

8.6 Section Summary

You have successfully bootstrapped the Kubernetes control plane on the server host, bringing up kube-apiserver, kube-controller-manager, and kube-scheduler with full mTLS authentication and API-to-Kubelet RBAC authorization. With the cluster’s decision-making center operational, you are ready to proceed to Lab 09: Bootstrapping the Kubernetes Worker Nodes.


9. Bootstrapping Worker Nodes (docs/09-bootstrapping-kubernetes-workers.md)

  • Host Prerequisites: Installing runtime dependencies (socat, conntrack, ipset) and disabling system swap.

  • Container Runtime:

  • Deploying runc and containerd.

  • Configuring CNI bridge networking (10-bridge.conf, 99-loopback.conf) and setting kernel parameters (br-netfilter).

  • Kubelet & Kube-Proxy Setup: Installing unit files, applying per-node Pod CIDRs, and starting services via systemctl.

  • Node Verification: Querying cluster node status from the jumpbox using kubectl get nodes.

9.1 Section Overview & Context

In this lab, you will configure and bootstrap two Kubernetes worker nodes: node-0 (10.0.0.3) and node-1 (10.0.0.4). While the control plane manages cluster state and scheduling decisions, the worker nodes perform the heavy lifting: running containerized workloads inside Pods, managing network interfaces, and enforcing network proxy rules.

To turn a bare Linux host into a fully functional Kubernetes worker node, we must install and orchestrate four distinct runtime and networking layers:

+-----------------------------------------------------------------------------------+
|                            WORKER NODE (node-0 / node-1)                          |
|                                                                                   |
|  [ kubelet ]  <--- (CRI) ---> [ containerd ] <--- (OCI) ---> [ runc ]             |
|       |                           |                                 |             |
|       |                           v                                 v             |
|       |                   [ Pod Sandbox ]                   [ App Containers ]    |
|       v                           |                                               |
|  [ kube-proxy ]                   v                                               |
|       |                   [ CNI Plugins ]                                         |
|       v                   (bridge / loopback)                                     |
|  (iptables rules)                 |                                               |
+-------+---------------------------+-----------------------------------------------+
        |                           |
        v                           v
  [ Pod-to-Pod Traffic ]     [ cni0 Bridge ]
  • Container Runtime Interface (CRI) & OCI Runtime: containerd manages container lifecycles, image pulls, and local storage, while runc acts as the low-level Open Container Initiative (OCI) executor that interacts with Linux kernel namespaces and cgroups to spawn isolation boundaries.
  • Node Agent (kubelet): The primary node agent responsible for registering the node with kube-apiserver, monitoring host resources, and continuously reconciling local container states against assigned PodSpec manifests.
  • Container Network Interface (CNI): Executable binaries (bridge, loopback, host-local) that dynamically provision virtual ethernet (veth) pairs, configure bridge devices, and assign IP addresses to Pods.
  • Network Proxy (kube-proxy): The node-level network agent that updates host iptables rules to translate Kubernetes Service virtual IPs (ClusterIPs and NodePorts) into real Pod endpoint addresses.

9.2 Technical Understanding & Deep Dive (The “Why”)

Before executing installation commands, it is essential to understand why each kernel setting, configuration flag, and utility binary is required.

A. Pod CIDR Subnet Allocation

Kubernetes mandates that every Pod gets its own unique, routable IP address within the cluster network. To achieve this without IP address collisions or complex NAT layers on host-to-host Pod communication, each worker node is assigned a non-overlapping sub-block of the cluster-wide IP range (10.200.0.0/16):

  • node-0 is assigned 10.200.0.0/24 (Pod IPs: 10.200.0.1–10.200.0.254).
  • node-1 is assigned 10.200.1.0/24 (Pod IPs: 10.200.1.1–10.200.1.254).

During file templating on the jumpbox, we dynamically substitute the literal string SUBNET inside 10-bridge.conf and kubelet-config.yaml with each node’s allocated subnet.

B. Linux Kernel Requirements & Sysctl Parameters

  • Disabling Swap (swapoff -a): kubelet relies on deterministic resource accounting. If host swap memory is enabled, memory-constrained Pods can thrash swap disks instead of hitting hard limits, leading to unpredictable latency, broken scheduling guarantees, and delayed Out-Of-Memory (OOM) handler triggers.
  • Kernel Module br-netfilter: By default, Linux virtual bridge interfaces filter packet frames at Layer 2 (Data Link) without passing them through host Layer 3 (Network) packet filtering. Loading br-netfilter and setting net.bridge.bridge-nf-call-iptables = 1 forces bridged traffic to pass through the host’s iptables netfilter rules. This allows kube-proxy to intercept Pod-to-Service traffic passing across the cni0 bridge and apply Destination NAT (DNAT).

C. System Dependencies

  • socat: A socket relay tool that enables bidirectional data streams between standard streams or network sockets. kubelet depends on socat to execute streaming operations like kubectl port-forward, kubectl exec, and kubectl logs.
  • conntrack: Enables the Linux kernel to track stateful IPv4 connections. Required by kube-proxy to ensure NAT mappings remain persistent and consistent across multi-packet connection streams.
  • ipset: An administration tool for IP sets in the Linux kernel. Allows high-performance evaluation of dynamic IP rule tables without degrading system performance.

9.3 Step-by-Step Implementation & Command Breakdown

Step 3.1: Template and Distribute Configuration Files from jumpbox

Execute the following commands from /root/kubernetes-the-hard-way on the jumpbox (10.0.0.1).

  1. Extract Subnets & Substitute Configurations:
cd /root/kubernetes-the-hard-way

for HOST in node-0 node-1; do
  SUBNET=$(grep ${HOST} machines.txt | cut -d " " -f 4)
  sed "s|SUBNET|$SUBNET|g" configs/10-bridge.conf > 10-bridge.conf
  sed "s|SUBNET|$SUBNET|g" configs/kubelet-config.yaml > kubelet-config.yaml
  scp 10-bridge.conf kubelet-config.yaml root@${HOST}:~/
done
  1. Transfer Worker Binaries, Systemd Units, and Configurations:
for HOST in node-0 node-1; do
  scp \
    downloads/worker/* \
    downloads/client/kubectl \
    configs/99-loopback.conf \
    configs/containerd-config.toml \
    configs/kube-proxy-config.yaml \
    units/containerd.service \
    units/kubelet.service \
    units/kube-proxy.service \
    root@${HOST}:~/
done
  1. Transfer CNI Plugins:
for HOST in node-0 node-1; do
  ssh root@${HOST} "mkdir -p ~/cni-plugins"
  scp downloads/cni-plugins/* root@${HOST}:~/cni-plugins/
done

Step 3.2: Provision Worker Node 0 (node-0)

Log in to node-0 (10.0.0.3):

ssh root@node-0

Execute the full provisioning pipeline:

# 1. Install OS Dependencies
apt-get update && apt-get -y install socat conntrack ipset kmod

# 2. Disable Swap Memory
swapoff -a

# 3. Create Required Directory Tree
mkdir -p \
  /etc/cni/net.d \
  /opt/cni/bin \
  /var/lib/kubelet \
  /var/lib/kube-proxy \
  /var/lib/kubernetes \
  /var/run/kubernetes

# 4. Install Binaries to Path
mv crictl kube-proxy kubelet runc /usr/local/bin/
mv containerd containerd-shim-runc-v2 containerd-stress /bin/
mv cni-plugins/* /opt/cni/bin/

# 5. Configure CNI Network & Kernel Modules
mv 10-bridge.conf 99-loopback.conf /etc/cni/net.d/
modprobe br-netfilter
echo "br-netfilter" >> /etc/modules-load.d/modules.conf

echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.d/kubernetes.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.d/kubernetes.conf
sysctl -p /etc/sysctl.d/kubernetes.conf

# 6. Configure containerd Runtime
mkdir -p /etc/containerd/
mv containerd-config.toml /etc/containerd/config.toml
mv containerd.service /etc/systemd/system/

# 7. Configure kubelet
mv kubelet-config.yaml /var/lib/kubelet/
mv kubelet.service /etc/systemd/system/

# 8. Configure kube-proxy
mv kube-proxy-config.yaml /var/lib/kube-proxy/
mv kube-proxy.service /etc/systemd/system/

# 9. Reload Systemd and Launch Worker Services
systemctl daemon-reload
systemctl enable containerd kubelet kube-proxy
systemctl start containerd kubelet kube-proxy

Step 3.3: Provision Worker Node 1 (node-1)

Log out of node-0 and log into node-1 (10.0.0.4):

exit
ssh root@node-1

Run the identical provisioning block on node-1:

apt-get update && apt-get -y install socat conntrack ipset kmod
swapoff -a

mkdir -p \
  /etc/cni/net.d \
  /opt/cni/bin \
  /var/lib/kubelet \
  /var/lib/kube-proxy \
  /var/lib/kubernetes \
  /var/run/kubernetes

mv crictl kube-proxy kubelet runc /usr/local/bin/
mv containerd containerd-shim-runc-v2 containerd-stress /bin/
mv cni-plugins/* /opt/cni/bin/

mv 10-bridge.conf 99-loopback.conf /etc/cni/net.d/
modprobe br-netfilter
echo "br-netfilter" >> /etc/modules-load.d/modules.conf

echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.d/kubernetes.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.d/kubernetes.conf
sysctl -p /etc/sysctl.d/kubernetes.conf

mkdir -p /etc/containerd/
mv containerd-config.toml /etc/containerd/config.toml
mv containerd.service /etc/systemd/system/

mv kubelet-config.yaml /var/lib/kubelet/
mv kubelet.service /etc/systemd/system/

mv kube-proxy-config.yaml /var/lib/kube-proxy/
mv kube-proxy.service /etc/systemd/system/

systemctl daemon-reload
systemctl enable containerd kubelet kube-proxy
systemctl start containerd kubelet kube-proxy

Worker Node Architecture & Storage Matrix

Component / File Path Target Directory Purpose Key Parameters / Flags
kubelet /usr/local/bin/ Primary worker node agent --config=/var/lib/kubelet/kubelet-config.yaml, --kubeconfig
kube-proxy /usr/local/bin/ Service proxy & load balancer --config=/var/lib/kube-proxy/kube-proxy-config.yaml
containerd /bin/ High-level CRI runtime daemon Configured via /etc/containerd/config.toml (SystemdCgroup = true)
runc /usr/local/bin/ OCI container execution tool Spawns low-level cgroups & namespaces
CNI Binaries /opt/cni/bin/ Executables (bridge, loopback) Invoked dynamically by containerd on Pod lifecycle events
Port 10250/TCP 0.0.0.0:10250 Kubelet Server API Incoming HTTPS requests from API Server (exec, logs, attach)
Port 10256/TCP 0.0.0.0:10256 Kube-proxy Health Check Serves HTTP /healthz endpoints for node load balancers

9.4 Real-World Troubleshooting & Pitfalls (Proxmox LXC Specifics)

If you are running your Kubernetes nodes inside Proxmox LXC containers (sharing the hyproxmox host kernel), standard systemd initialization will encounter several container isolation restrictions. Below are the specific gotchas, root causes, and exact remediation steps.

Gotcha 1: modprobe: FATAL: Module br-netfilter not found & containerd Systemd Pre-Start Failure

  • Symptom: Running modprobe inside an LXC worker fails with FATAL: Module br-netfilter not found. Additionally, containerd.service enters a crash loop with ExecStartPre=/sbin/modprobe overlay returning status code 1/FAILURE.
  • Root Cause: LXC containers share the Proxmox host’s kernel directly and lack a standalone /lib/modules/$(uname -r) file tree. Unprivileged LXC containers cannot dynamically load kernel modules.
  • Fix:
  1. Load the modules directly on the Proxmox Host CLI (root@hyproxmox):
modprobe br_netfilter
modprobe overlay
  1. Comment out the ExecStartPre directive in /etc/systemd/system/containerd.service on both worker nodes (node-0 and node-1):
sed -i 's/^ExecStartPre=.*modprobe.*/# &/' /etc/systemd/system/containerd.service
systemctl daemon-reload
systemctl restart containerd

Gotcha 2: Kubelet Crash — Missing /dev/kmsg & OOM Adjustment Denials

  • Symptom: kubelet fails on startup. Inspecting journalctl -u kubelet -n 30 --no-pager displays:
Failed to create an oomWatcher ... err="open /dev/kmsg: no such file or directory"
failed to set "/proc/self/oom_score_adj" to "-999": permission denied
  • Root Cause: LXC containers do not expose a pseudo-device for /dev/kmsg (kernel log ring buffer). Furthermore, container security profiles prevent unprivileged processes from adjusting their process OOM scores to negative values.
  • Fix: Execute the following commands on both node-0 and node-1:
# 1. Create a persistent /dev/kmsg symlink service
cat <<EOF > /etc/systemd/system/kmsg-symlink.service
[Unit]
Description=Create /dev/kmsg symlink for Kubelet
Before=kubelet.service

[Service]
Type=oneshot
ExecStart=/usr/bin/ln -sf /dev/console /dev/kmsg

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now kmsg-symlink.service

# 2. Force Kubelet OOM Score Adjustment to 0
echo "oomScoreAdj: 0" >> /var/lib/kubelet/kubelet-config.yaml
systemctl restart kubelet

Gotcha 3: Kubelet Crash — Read-Only /proc/sys Kernel Parameter Writes

  • Symptom: kubelet logs indicate ContainerManager initialization errors when attempting to apply kernel settings (vm.overcommit_memory, kernel.panic, kernel.panic_on_oops).
  • Root Cause: Proxmox mounts /proc/sys as read-only inside LXC environments to protect overall host stability.
  • Fix:
  1. Apply sysctls globally on the Proxmox Host (root@hyproxmox):
sysctl -w vm.overcommit_memory=1
sysctl -w kernel.panic=10
sysctl -w kernel.panic_on_oops=1
  1. Configure kubelet on worker nodes to protect host kernel defaults and allow user-namespace operation by updating /var/lib/kubelet/kubelet-config.yaml:
cat <<EOF >> /var/lib/kubelet/kubelet-config.yaml
protectKernelDefaults: true
featureGates:
  KubeletInUserNamespace: true
EOF
systemctl restart kubelet

Gotcha 4: Kube-Proxy Failure — nf_conntrack_max Permission Denied & Invalid Syntax

  • Symptom: kube-proxy fails to start, outputting open /proc/sys/net/netfilter/nf_conntrack_max: permission denied. Appending conntrack: incorrectly can also trigger yaml: line 5: did not find expected key.
  • Root Cause: Write operations to netfilter connection tracking tables inside LXC are restricted. YAML indentation syntax errors prevent parsing the proxy configuration file.
  • Fix:
  1. Set conntrack limits on the Proxmox Host (root@hyproxmox):
sysctl -w net.netfilter.nf_conntrack_max=131072
  1. Overwrite /var/lib/kube-proxy/kube-proxy-config.yaml with a clean configuration instructing kube-proxy to skip kernel conntrack adjustments (maxPerCore: 0):
cat <<EOF > /var/lib/kube-proxy/kube-proxy-config.yaml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
  kubeconfig: "/var/lib/kube-proxy/kubeconfig"
mode: "iptables"
clusterCIDR: "10.200.0.0/16"
conntrack:
  maxPerCore: 0
EOF

systemctl restart kube-proxy

9.5 Verification & Validation

Step 5.1: Local Service Verification

Verify that all three core worker services report an active state on both node-0 and node-1:

systemctl is-active containerd kubelet kube-proxy

Expected Terminal Output:

active
active
active

Step 5.2: Cluster Node Registration Check

Log into the jumpbox (10.0.0.1) and query the Kubernetes API server running on server (10.0.0.2):

ssh root@server "kubectl get nodes --kubeconfig admin.kubeconfig"

Expected Terminal Output:

NAME     STATUS   ROLES    AGE     VERSION
node-0   Ready    <none>   1m      v1.32.3
node-1   Ready    <none>   10s     v1.32.3

Both worker nodes successfully report a Ready status, confirming that kubelet has registered with the API server, the CRI container runtime is responsive, and CNI networking plugin paths are properly linked.



10. Configuring Remote kubectl Access (docs/10-configuring-kubectl.md)

  • Admin Context Setup: Configuring local kubectl on the jumpbox with embedded admin credentials targeting [https://server.kubernetes.local:6443](https://server.kubernetes.local:6443).
  • Cluster Access Verification: Executing kubectl version and kubectl get nodes without specifying explicit config flags.

10.1 Section Overview & Context

In this section, you will configure kubectl on the jumpbox (10.0.0.1) to enable administrative access to the remote Kubernetes cluster.

Up to this point, worker nodes and control plane components have been configured with dedicated kubeconfig files tailored to their specific identities (e.g., node-0.kubeconfig, kube-scheduler.kubeconfig). Here, you will construct a centralized default configuration file located at ~/.kube/config on your administrative workstation. This setup allows you to execute cluster management commands directly from the jumpbox without manually passing credentials or configuration flags on every invocation.

+-----------------------------------------------------------------------+
|                             JUMPBOX                                   |
|                           (10.0.0.1)                                  |
|                                                                       |
|   ~/.kube/config                                                      |
|   +---------------------------------------------------------------+   |
|   | Context: kubernetes-the-hard-way                              |   |
|   |   ├── Cluster: https://server.kubernetes.local:6443 (Embedded CA) |   |
|   |   └── User: admin (CN=admin, O=system:masters Cert/Key)       |   |
|   +---------------------------------------------------------------+   |
|                                |                                      |
|                        kubectl command                                |
+--------------------------------|--------------------------------------+
                                 | Mutual TLS (Port 6443)
                                 v
+-----------------------------------------------------------------------+
|                        CONTROL PLANE (server)                         |
|                           (10.0.0.2)                                  |
|                                                                       |
|   kube-apiserver                                                      |
|   ├── Validates client cert against Root CA                           |
|   └── Authorizes 'O=system:masters' with cluster-admin rights         |
+-----------------------------------------------------------------------+

10.2 Technical Understanding & Deep Dive

Local vs. Remote Administration Endpoints

In Lab 05, the initial admin.kubeconfig was generated targeting [https://127.0.0.1:6443](https://127.0.0.1:6443) because it was intended for execution directly on the control plane server. If you attempt to use that configuration from the jumpbox, kubectl will attempt to connect to 127.0.0.1:6443 on the jumpbox itself—where no API server exists—resulting in a connection refused error.

To manage the cluster remotely, kubectl must target the domain server.kubernetes.local:6443, which resolves via /etc/hosts to the control plane IP (10.0.0.2).

Kubeconfig Structure & TLS Data Embedding

A kubeconfig file consists of three logical sections:

  1. Clusters: Maps cluster nicknames to API Server URLs and public CA certificates.
  2. Users: Maps user identities to authentication credentials (client certificates and private keys).
  3. Contexts: Links a specific User identity to a specific Cluster endpoint.

When configuring these entries, passing the --embed-certs=true flag instructs kubectl to read the raw contents of ca.crt, admin.crt, and admin.key, base64-encode them, and store them directly inside ~/.kube/config under certificate-authority-data, client-certificate-data, and client-key-data.

Why Embed Certs? Without embedding, kubectl relies on relative or absolute file paths to locate certificates. If you run kubectl from a different working directory or move the configuration file, commands will fail due to missing path references. Embedding turns ~/.kube/config into a portable, self-contained file.

RBAC & Superuser Authentication

The client certificate (admin.crt) contains specific Subject Distinguished Name fields:

  • Common Name (CN): admin
  • Organization (O): system:masters

When kube-apiserver receives an HTTPS request, it performs Mutual TLS (mTLS) verification using its trusted Root CA. Upon confirming the certificate is valid, it inspects the Organization field (O=system:masters). In Kubernetes, system:masters is a hardcoded break-glass group that automatically grants unrestricted administrative rights (cluster-admin) across the cluster, bypassing standard RBAC restrictions.


10.3 Step-by-Step Implementation & Command Breakdown

Perform all commands in this section from the jumpbox (10.0.0.1) inside the /root/kubernetes-the-hard-way directory.

Step 1: Verify Remote API Server Reachability

Before writing the configuration file, verify HTTPS reachability and TLS trust between the jumpbox and the API server:

cd /root/kubernetes-the-hard-way

curl --cacert ca.crt https://server.kubernetes.local:6443/version

Expected Output:

{
  "major": "1",
  "minor": "32",
  "gitVersion": "v1.32.3",
  "gitCommit": "32cc146f75aad04beaaa245a7157eb35063a9f99",
  "gitTreeState": "clean",
  "buildDate": "2025-03-11T19:52:21Z",
  "goVersion": "go1.23.6",
  "compiler": "gc",
  "platform": "linux/amd64"
}
  • --cacert ca.crt: Explicitly supplies the Certificate Authority certificate to validate the TLS identity presented by kube-apiserver.

Step 2: Generate the Default Administrative ~/.kube/config

Execute the following block to define the cluster endpoint, credentials, and default context profile:

{
  kubectl config set-cluster kubernetes-the-hard-way \
    --certificate-authority=ca.crt \
    --embed-certs=true \
    --server=https://server.kubernetes.local:6443

  kubectl config set-credentials admin \
    --client-certificate=admin.crt \
    --client-key=admin.key

  kubectl config set-context kubernetes-the-hard-way \
    --cluster=kubernetes-the-hard-way \
    --user=admin

  kubectl config use-context kubernetes-the-hard-way
}

Detailed Flag Breakdown:

Command / Flag Purpose
set-cluster kubernetes-the-hard-way Names the cluster entry inside the configuration file.
--certificate-authority=ca.crt Points to the Root CA certificate used to verify the server’s identity.
--embed-certs=true Inlines base64-encoded binary certificate and key data into the file.
--server=https://...:6443 Specifies the HTTPS listener address of kube-apiserver.
set-credentials admin Names the user profile storing client credentials.
--client-certificate=admin.crt Sets the client certificate containing CN=admin and O=system:masters.
--client-key=admin.key Sets the 4096-bit RSA private key used for TLS client authentication.
set-context ... Maps the admin user to the kubernetes-the-hard-way cluster endpoint.
use-context ... Sets current-context to kubernetes-the-hard-way in ~/.kube/config.

10.4 Troubleshooting & Gotchas

1. Host Resolution Failures

  • Symptom: curl: (6) Could not resolve host: server.kubernetes.local or unable to connect to server: dial tcp: lookup server.kubernetes.local.
  • Root Cause: Missing DNS mapping in /etc/hosts on the jumpbox.
  • Fix: Add the control plane IP mapping to /etc/hosts:
echo "10.0.0.2 server.kubernetes.local" >> /etc/hosts

2. Loopback Connection Errors

  • Symptom: The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
  • Root Cause: The kubeconfig file was generated using [https://127.0.0.1:6443](https://127.0.0.1:6443) rather than [https://server.kubernetes.local:6443](https://server.kubernetes.local:6443).
  • Fix: Re-run kubectl config set-cluster with --server=[https://server.kubernetes.local:6443](https://server.kubernetes.local:6443).

3. Broken Relative Paths

  • Symptom: error: unable to read client-cert admin.crt: no such file or directory when running kubectl commands outside /root/kubernetes-the-hard-way.
  • Root Cause: Omitting --embed-certs=true during kubectl config set-cluster or set-credentials.
  • Fix: Re-run the configuration commands ensuring --embed-certs=true is included, or manually re-generate the config while inside the directory containing the .crt and .key files.

10.5 Verification & Validation

Inspect the active configuration stored in ~/.kube/config:

kubectl config view

Confirm that kubectl can query the remote Kubernetes cluster version without passing explicit config flags:

kubectl version

Expected Output:

Client Version: v1.32.3
Kustomize Version: v5.5.0
Server Version: v1.32.3

Verify administrative control by querying all registered worker nodes:

kubectl get nodes

Expected Output:

NAME     STATUS   ROLES    AGE   VERSION
node-0   Ready    <none>   15m   v1.32.3
node-1   Ready    <none>   14m   v1.32.3

10.6 Section Summary

You have successfully configured remote administrative access on the jumpbox by building a self-contained ~/.kube/config file containing embedded mTLS credentials and Root CA data. With kubectl communicating securely with kube-apiserver and returning both worker nodes in a Ready state, the cluster control plane is fully operational and ready for network routing configuration.


11. Pod Network Routes (docs/11-pod-network-routes.md)

  • Routing Problem Statement: Addressing cross-node Pod communication without overlay networks.
  • Routing Table Provisioning: Adding static kernel routes (ip route add) across server, node-0, and node-1 mapping Pod subnets (10.200.x.0/24) to target node internal IPs.
  • Verification: Inspecting kernel routing tables across all cluster nodes (ip route).

11.1 Section Overview & Context

In the previous sections of this cluster build, you configured the Container Network Interface (CNI) bridge plugin across worker nodes and assigned each a dedicated Pod CIDR block (10.200.0.0/24 to node-0 and 10.200.1.0/24 to node-1). While each node can now allocate IP addresses to local Pods via its local cni0 bridge interface, packets destined for Pods on other nodes currently have nowhere to go.

This section covers the creation of static kernel network routes across all nodes in the cluster (server, node-0, and node-1). By mapping each worker node’s Pod CIDR subnet to its host IPv4 address, we establish flat, un-encapsulated Layer 3 routing. This step is the bridge between isolated single-node Pod networks and a fully connected multi-node Kubernetes cluster.


11.2 Technical Understanding & Deep Dive

The Kubernetes Networking Model: IP-Per-Pod

Kubernetes mandates two core networking constraints for all compliant clusters:

  1. Pod-to-Pod Communication: Any Pod on any node can communicate with any other Pod on any node without Network Address Translation (NAT).
  2. Node-to-Pod Communication: Any system process or agent on a node (such as the kubelet or kube-apiserver) can communicate directly with any Pod on any node.

Demystifying Cross-Node Pod Traffic Flow

When Pod A (10.200.0.2) on node-0 sends a packet to Pod B (10.200.1.2) on node-1:

  +-------------------------------------+         +-------------------------------------+
  |             WORKER NODE-0           |         |             WORKER NODE-1           |
  |              (10.0.0.3)             |         |              (10.0.0.4)             |
  |                                     |         |                                     |
  |  Pod A (10.200.0.2)                 |         |  Pod B (10.200.1.2)                 |
  |       │                             |         |       ▲                             |
  |       ▼                             |         |       │                             |
  |  cni0 Bridge (10.200.0.1/24)        |         |  cni0 Bridge (10.200.1.1/24)        |
  |       │                             |         |       ▲                             |
  |       ▼                             |         |       │                             |
  |  Linux Routing Table                |         |  Linux Routing Table                |
  |  [ 10.200.1.0/24 via 10.0.0.4 ]      |         |  [ 10.200.0.0/24 via 10.0.0.3 ]      |
  +-------┼-----------------------------+         +-------▲-----------------------------+
          │                                               │
          └───────────────────── eth0 ────────────────────┘
                       (Subnet Transit: 10.0.0.0/24)
  1. Local Egress: The packet leaves Pod A’s network namespace via a virtual ethernet pair (veth) and hits the cni0 bridge interface on node-0.
  2. Kernel Route Lookup: The Linux kernel on node-0 inspects the packet’s destination IP (10.200.1.2).
  • Without static routes: The kernel has no matching rule for 10.200.1.0/24. It forwards the packet to the default gateway (10.0.0.254) where it is dropped or rejected.
  • With static routes: The kernel matches the 10.200.1.0/24 entry and routes the packet out interface eth0 directly to next-hop gateway 10.0.0.4 (node-1).
  1. Ingress Delivery: node-1 receives the frame on eth0. Its kernel recognizes 10.200.1.2 as belonging to its local cni0 bridge network and forwards the packet straight into Pod B.

Why Control Plane Nodes (server) Need Routes

The control plane host (server) runs kube-apiserver, which needs to open direct TCP streams to Pod IPs when running operations like:

  • Executing commands inside containers (kubectl exec)
  • Streaming pod logs (kubectl logs)
  • Invoking API admission webhooks hosted inside cluster Pods

Without static routes on server, control plane commands directed at Pod IPs will fail with connection timeouts.


11.3 Step-by-Step Implementation & Command Breakdown

All commands are executed from the jumpbox administrative node (10.0.0.1) inside the /root/kubernetes-the-hard-way directory.

Step 1: Extract Network Variables

Extract the node IPv4 addresses and allocated Pod CIDR subnets from machines.txt:

cd /root/kubernetes-the-hard-way

{
  SERVER_IP=$(grep server machines.txt | cut -d " " -f 1)
  NODE_0_IP=$(grep node-0 machines.txt | cut -d " " -f 1)
  NODE_0_SUBNET=$(grep node-0 machines.txt | cut -d " " -f 4)
  NODE_1_IP=$(grep node-1 machines.txt | cut -d " " -f 1)
  NODE_1_SUBNET=$(grep node-1 machines.txt | cut -d " " -f 4)
}
Command Breakdown:
  • grep <host> machines.txt: Filters the inventory file for the target node record.
  • cut -d " " -f 1: Parses field 1, extracting the internal host IP (10.0.0.2, 10.0.0.3, or 10.0.0.4).
  • cut -d " " -f 4: Parses field 4, extracting the allocated Pod CIDR subnet (10.200.0.0/24 or 10.200.1.0/24).

Step 2: Provision Routes on the Control Plane Server (server)

Configure static routes on server to reach both worker node Pod CIDRs:

ssh root@server <<EOF
  ip route add ${NODE_0_SUBNET} via ${NODE_0_IP}
  ip route add ${NODE_1_SUBNET} via ${NODE_1_IP}
EOF
Flag & Argument Breakdown:
  • ip route add: Adds an entry to the Linux kernel IPv4 routing table.
  • ${NODE_0_SUBNET} (10.200.0.0/24): Specifies the destination IP CIDR block.
  • via ${NODE_0_IP} (via 10.0.0.3): Sets the Layer 3 next-hop gateway address. Traffic is forwarded over the primary network interface (eth0).

Step 3: Provision Routes on Worker Node 0 (node-0)

Configure node-0 to route traffic destined for node-1 Pods across the shared subnet:

ssh root@node-0 <<EOF
  ip route add ${NODE_1_SUBNET} via ${NODE_1_IP}
EOF

Step 4: Provision Routes on Worker Node 1 (node-1)

Configure node-1 to route traffic destined for node-0 Pods across the shared subnet:

ssh root@node-1 <<EOF
  ip route add ${NODE_0_SUBNET} via ${NODE_0_IP}
EOF

11.4 Real-World Troubleshooting & Pitfalls

Ephemeral Routes Lost Across Reboots

  • Symptom: Cross-node Pod communication and kubectl logs work initially, but fail suddenly after an LXC container or node reboot.
  • Root Cause: Executing ip route add updates the active kernel routing table stored in RAM, but does not write to persistent networking configuration files.
  • Fix: For persistent routing in Debian/Ubuntu-based environments, append static route declarations to /etc/network/interfaces or your Netplan configuration file (/etc/netplan/01-netcfg.yaml):
network:
  version: 2
  ethernets:
    eth0:
      routes:
        - to: 10.200.1.0/24
          via: 10.0.0.4

Proxmox LXC MAC/IP Filtering Denials

  • Symptom: Routes exist in ip route, but ping between Pod IPs on different nodes times out. Packet dumps (tcpdump -i eth0) show ARP requests going unanswered.
  • Root Cause: In Proxmox VE or hypervisor environments, MAC/IP anti-spoofing flags on virtual interfaces (e.g., veth903i0) may drop frames originating from IP ranges not explicitly assigned to the container interface (10.200.x.x).
  • Fix: Ensure IP Filter / MAC Filter options are disabled for the container virtual network interfaces in the Proxmox VE Firewall settings for the guest.

11.5 Verification & Validation

Execute ip route on each node via SSH to confirm that the static routes are active in the kernel.

1. Verify Control Plane (server) Routing Table

ssh root@server ip route

Expected Output:

default via 10.0.0.254 dev eth0 proto static 
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.2 
10.200.0.0/24 via 10.0.0.3 dev eth0 
10.200.1.0/24 via 10.0.0.4 dev eth0 

2. Verify Worker Node 0 (node-0) Routing Table

ssh root@node-0 ip route

Expected Output:

default via 10.0.0.254 dev eth0 proto static 
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.3 
10.200.1.0/24 via 10.0.0.4 dev eth0 

3. Verify Worker Node 1 (node-1) Routing Table

ssh root@node-1 ip route

Expected Output:

default via 10.0.0.254 dev eth0 proto static 
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.4 
10.200.0.0/24 via 10.0.0.3 dev eth0 

Cluster Routing Matrix Summary

Host Node IP Target Pod Subnet Next-Hop Gateway Status
server 10.0.0.2 10.200.0.0/24 (node-0) 10.0.0.3 Active
10.200.1.0/24 (node-1) 10.0.0.4 Active
node-0 10.0.0.3 10.200.1.0/24 (node-1) 10.0.0.4 Active
node-1 10.0.0.4 10.200.0.0/24 (node-0) 10.0.0.3 Active

11.6 Section Summary

By populating the kernel routing tables on server, node-0, and node-1, you have fulfilled the fundamental Kubernetes networking requirement of un-NATted, cross-node Pod routing. With the underlying network fabric complete, you are ready to validate end-to-end cluster functionality in the Smoke Test.


12. Smoke Test (docs/12-smoke-test.md)

  • Data Encryption: Creating a generic Secret and performing an etcdctl hexdump to verify the k8s:enc:aescbc:v1:key1 header.

  • Workload Management: Deploying an nginx workload and testing cluster primitives:

  • kubectl port-forward connection forwarding.

  • kubectl logs log stream retrieval.

  • kubectl exec container command execution.

  • Service Reachability: Exposing nginx as a NodePort service and testing HTTP access via node host IP and assigned port.

12.1 Section Overview & Context

With the control plane services (kube-apiserver, kube-controller-manager, kube-scheduler) operational, worker binaries (kubelet, kube-proxy, containerd) running, and cross-node networking established via custom CNI bridges and host routes, your cluster infrastructure is fully assembled.

However, a running set of systemd units does not guarantee a functional cluster. The Smoke Test is the ultimate end-to-end verification suite for your Kubernetes build. In this step, you will validate the core data plane and control plane mechanisms:

  • Data Encryption at Rest: Confirming that sensitive data stored via kube-apiserver is ciphered before hitting etcd.
  • Workload Scheduling & Execution: Ensuring the end-to-end container orchestration loop succeeds from deployment manifest to running runc container namespaces.
  • Control Plane-to-Node Streaming: Validating API proxy channels for port-forward, container log streaming (kubectl logs), and interactive execution (kubectl exec).
  • Service Discovery & NAT Routing: Verifying kube-proxy iptables rules and cross-node network packet delivery using NodePort services.
                  +-----------------------------------------------------------------------+
                  |                         SMOKE TEST VALIDATION                         |
                  +-----------------------------------------------------------------------+
                                                      |
    +-----------------------+-------------------------+-----------------------+-----------------------+
    |                       |                         |                       |                       |
    v                       v                         v                       v                       v
[Secret Encryption]    [Workload Scheduling]    [Stream Proxying]       [Log & Exec Hooks]      [NodePort NAT Routing]
`kube-apiserver`        ReplicaSet & Scheduler   `kubectl port-forward`  API Server -> Kubelet   `kube-proxy` iptables
-> AES-CBC -> `etcd`    -> CRI -> `runc`        -> `socat` -> Pod       mTLS Auth & Logs Stream  -> Host IP:Port -> Pod

12.2 Technical Understanding & Deep Dive (The “Why”)

1. Envelope Encryption at Rest

In previous labs, you configured kube-apiserver with an EncryptionConfiguration specifying the aescbc (AES-CBC with PKCS#7 padding) provider and an encryption key (key1).

When a client creates a Secret, kube-apiserver handles the payload in memory as plaintext, encrypts the raw JSON byte stream using the key, and writes the resulting ciphertext to etcd at /registry/secrets/<namespace>/<name>. Querying etcd directly via etcdctl bypasses the API server’s decryption layer, allowing you to prove that disk persistence is secure and plaintext data (e.g., mykey=mydata) is never exposed in raw database blocks.

2. Workload Scheduling & Namespace Provisioning

Executing kubectl create deployment triggers a sequence across the control plane and worker components:

  1. kube-apiserver validates and persists the Deployment manifest to etcd.
  2. kube-controller-manager detects the new deployment, creates a ReplicaSet, and spawns a target Pod object in the Unscheduled state (nodeName: "").
  3. kube-scheduler evaluates available compute nodes, assigns the Pod to a worker (e.g., node-0 at 10.0.0.3 or node-1 at 10.0.0.4), and updates the binding in the API server.
  4. kubelet on the assigned worker detects the Pod assignment via its API watch connection, instructs containerd over the Container Runtime Interface (CRI) to pull the image, and invokes runc to isolate the container inside Linux namespaces (net, pid, mnt, ipc).

3. Stream Proxying (port-forward, logs, exec)

Admin commands like port-forward, logs, and exec do not communicate with containers directly. Instead, they leverage secured API streams:

  Jumpbox (10.0.0.1)             Control Plane (10.0.0.2)           Worker Node (node-0 / node-1)
+--------------------+         +------------------------+         +-------------------------------+
| curl 127.0.0.1:8080|         |    kube-apiserver      |         |     kubelet (Port 10250)      |
|         │          |         |      (Port 6443)       |         |               │               |
|         ▼          |  HTTPS  |            │           |  HTTPS  |               ▼               |
|   kubectl port-----+─────────┼────────────┼───────────┼─────────┼──> socat -> Nginx Pod Port 80 |
|     forward        | (mTLS)  |            │           | (mTLS)  |         (10.200.x.x:80)       |
+--------------------+         +------------------------+         +-------------------------------+
  • port-forward: kubectl establishes an HTTP/2 SPDY connection to kube-apiserver (port 6443), which proxies HTTPS traffic to kubelet (port 10250) on the target worker node. kubelet spawns socat on the node to relay raw TCP sockets between the stream and the container’s virtual network interface.
  • **logs & exec**: kube-apiserver authenticates against kubelet using its client certificate (kube-api-server.crt), validated by the system:kube-apiserver-to-kubelet ClusterRole. kubelet reads stdout log files directly from /var/log/pods/... or invokes runc exec inside the target namespaces.

4. Service Exposure & NodePort Packet Flow

Exposing a workload via NodePort tests the network proxy implementation:

  1. kube-proxy watches the API server for new Service endpoints.
  2. Upon detecting a NodePort service, kube-proxy opens a high-range port (between 30000-32767) across every worker host and programs local iptables rules (PREROUTING and KUBE-SERVICES chains).
  3. Incoming traffic to <Worker-IP>:<NodePort> is intercepted by host iptables rules, which apply Destination NAT (DNAT) to rewrite the destination socket from the worker node’s IP to the Pod’s internal IP (10.200.x.x:80). If the target Pod resides on a different physical host, host static routes forward the packet over the bridge network (vmbr1).

12.3 Step-by-Step Implementation & Command Breakdown

All commands in this lab are executed from your administrative workstation (jumpbox at 10.0.0.1).

Step 1: Verify Secret Data Encryption at Rest

  1. Create a generic Secret containing a literal key-value pair:
kubectl create secret generic kubernetes-the-hard-way \
  --from-literal="mykey=mydata"
  • create secret generic: Creates an opaque Kubernetes Secret object named kubernetes-the-hard-way.
  • --from-literal="mykey=mydata": Sets the key mykey to store the value mydata.
  1. Inspect the raw key-value entry stored directly inside etcd on the control plane host (server at 10.0.0.2):
ssh root@server \
  'etcdctl get /registry/secrets/default/kubernetes-the-hard-way | hexdump -C'
  • etcdctl get /registry/secrets/default/...: Directly queries the key-value store database path for the secret in the default namespace.
  • hexdump -C: Converts raw binary database bytes into hexadecimal representations with ASCII translations.
Expected Terminal Output:
00000000  2f 72 65 67 69 73 74 72  79 2f 73 65 63 72 65 74  |/registry/secret|
00000010  73 2f 64 65 66 61 75 6c  74 2f 6b 75 62 65 72 6e  |s/default/kubern|
00000020  65 74 65 73 2d 74 68 65  2d 68 61 72 64 2d 77 61  |etes-the-hard-wa|
00000030  79 0a 6b 38 73 3a 65 6e  63 3a 61 65 73 63 62 63  |y.k8s:enc:aescbc|
00000040  3a 76 31 3a 6b 65 79 31  3a 4f 1b 80 d8 89 72 f4  |:v1:key1:O....r.|
00000050  60 8a 2c a0 76 1a e1 dc  98 d6 00 7a a4 2f f3 92  |`.,.v......z./..|
00000060  87 63 c9 22 f4 58 c8 27  b9 ff 2c 2e 1a b6 55 be  |.c.".X.'..,...U.|
00000070  d5 5c 4d 69 82 2f b7 e4  b3 b0 12 e1 58 c4 9c 77  |.\Mi./......X..w|
00000080  78 0c 1a 90 c9 c1 23 6c  73 8e 6e fd 8e 9c 3d 84  |x.....#ls.n...=.|
00000090  7d bf 69 81 ce c9 aa 38  be 3b dd 66 aa a3 33 27  |}.i....8.;.f..3'|
000000a0  df be 6d ac 1c 6d 8a 82  df b3 19 da 0f 93 94 1e  |..m..m..........|
000000b0  e0 7d 46 8d b5 14 d0 c5  97 e2 94 76 26 a8 cb 33  |.}F........v&..3|
000000c0  57 2a d0 27 a6 5a e1 76  a7 3f f0 b7 0a 7b ff 53  |W*.'.Z.v.?...{.S|
000000d0  cf c9 1a 18 5b 45 f8 b1  06 3b a9 45 02 76 23 61  |....[E...;.E.v#a|
000000e0  5e dc 86 cf 8e a4 d3 c9  5c 6a 6f e6 33 7b 5b 8f  |^.......\jo.3{[.|
000000f0  fb 8a 14 74 58 f9 49 2f  97 98 cc 5c d4 4a 10 1a  |...tX.I/...\.J..|
00000100  64 0a 79 21 68 a0 9e 7a  03 b7 19 e6 20 e4 1b ce  |d.y!h..z.... ...|
00000110  91 64 ce 90 d9 4f 86 ca  fb 45 2f d6 56 93 68 e1  |.d...O...E/.V.h.|
00000120  0b aa 8c a0 20 a6 97 fa  a1 de 07 6d 5b 4c 02 96  |.... ......m[L..|
00000130  31 70 20 83 16 f9 0a 22  5c 63 ad f1 ea 41 a7 1e  |1p ...."\c...A..|
00000140  29 1a d4 a4 e9 d7 0c 04  74 66 04 6d 73 d8 2e 3f  |).......tf.ms..?|
00000150  f0 b9 2f 77 bd 07 d7 7c  42 0a                    |../w...|B.|
0000015a
  • Header Verification: Note the prefix k8s:enc:aescbc:v1:key1 at offset 00000030. This confirms that kube-apiserver successfully processed the write using the aescbc encryption provider and key key1.
  • Ciphertext Verification: The string "mydata" does not appear anywhere in cleartext within the ASCII representation.

Step 2: Deploy Workloads & Inspect Scheduling

  1. Provision an nginx web server deployment:
kubectl create deployment nginx \
  --image=nginx:latest
  • create deployment nginx: Declares a Deployment controller managing 1 replica of nginx:latest.
  1. Inspect the provisioned Pod state:
kubectl get pods -l app=nginx
  • -l app=nginx: Filters the list of active pods matching the deployment label selector.
Expected Terminal Output:
NAME                     READY   STATUS    RESTARTS   AGE
nginx-56fcf95486-c8dnx   1/1     Running   0          8s

Step 3: Test Remote Port Forwarding

  1. Extract the active Pod name into an environment variable:
POD_NAME=$(kubectl get pods -l app=nginx \
  -o jsonpath="{.items[0].metadata.name}")
  • -o jsonpath="{.items[0].metadata.name}": Parses API output to return only the single string pod identifier.
  1. Establish a TCP tunnel forwarding local port 8080 on the jumpbox host to container port 80:
kubectl port-forward $POD_NAME 8080:80
Expected Terminal Output:
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
  1. Open a separate terminal session on jumpbox and dispatch an HTTP HEAD request to the local tunnel:
curl --head http://127.0.0.1:8080
Expected Terminal Output:
HTTP/1.1 200 OK
Server: nginx/1.27.4
Date: Sun, 06 Apr 2025 17:17:12 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Wed, 05 Feb 2025 11:06:32 GMT
Connection: keep-alive
ETag: "67a34638-267"
Accept-Ranges: bytes
  1. Return to the original terminal running the port-forward session and terminate the process using Ctrl+C.

Step 4: Verify Container Log Streaming and Exec Capabilities

  1. Fetch stdout/stderr logs generated by the container runtime:
kubectl logs $POD_NAME
Expected Terminal Output:
...
127.0.0.1 - - [06/Apr/2025:17:17:12 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.88.1" "-"
  • Note: The log entry records the HEAD HTTP request routed through the kubectl port-forward tunnel in Step 3.
  1. Execute commands directly inside the active Nginx container namespace:
kubectl exec -ti $POD_NAME -- nginx -v
  • **-t / -i**: Allocates an interactive pseudo-TTY session and keeps stdin open.
  • --: Signals the separator between kubectl options and the binary path (nginx -v) executed within the target container container.
Expected Terminal Output:
nginx version: nginx/1.27.4

Step 5: Expose Applications via NodePort Services

  1. Expose the nginx deployment port publicly using a NodePort service:
kubectl expose deployment nginx \
  --port 80 --type NodePort
  • --type NodePort: Allocates a cluster-wide high port across every worker node to proxy incoming traffic to target pods.

Note: Standard cloud provider LoadBalancer types require an external cloud controller manager integration, which is out of scope for a manual cluster build.

  1. Retrieve the dynamic nodePort assigned by kube-apiserver:
NODE_PORT=$(kubectl get svc nginx \
  --output=jsonpath='{range .spec.ports[0]}{.nodePort}')
  1. Query the exact host worker node where kube-scheduler placed the Pod:
NODE_NAME=$(kubectl get pods \
  -l app=nginx \
  -o jsonpath="{.items[0].spec.nodeName}")
  1. Dispatch an HTTP request targeting the assigned worker node host IP/name and the dynamic NodePort:
curl -I http://${NODE_NAME}:${NODE_PORT}
Expected Terminal Output:
HTTP/1.1 200 OK
Server: nginx/1.27.4
Date: Sun, 06 Apr 2025 17:18:36 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Wed, 05 Feb 2025 11:06:32 GMT
Connection: keep-alive
ETag: "67a34638-267"
Accept-Ranges: bytes

12.4 Real-World Troubleshooting & Pitfalls

LXC Container Runtime Failure (can't get final child's PID from pipe: EOF)

When running Kubernetes worker nodes inside Proxmox LXC containers (e.g., node-0 CT 903 at 10.0.0.3 and node-1 CT 904 at 10.0.0.4), deploying a Pod can result in the workload remaining stuck in ContainerCreating or FailedCreatePodSandBox.

Error Diagnostics

Checking events via kubectl describe pod -l app=nginx displays the following warning:

Events:
  Type     Reason                  Age               From               Message
  ----     ------                  ----              ----               -------
  Warning  FailedCreatePodSandBox  2m (x12 over 5m)  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox container task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF
Root Cause Analysis

This error occurs due to strict security isolation enforced by Proxmox host kernel defaults on unconfined LXC guests:

  1. When containerd triggers runc to isolate namespaces (pid, net, mnt), runc forks a child helper process to set up cgroup limits and /proc mounts.
  2. The host kernel’s default LXC AppArmor profiles and cgroup v2 device access lists block writes to /sys/fs/cgroup and /proc/sys inside the nested container.
  3. The Proxmox host terminates runc’s child process abruptly. runc encounters an immediate EOF when trying to read the child PID from its synchronization pipe.
Step-by-Step Fix
  1. Update the LXC guest security profiles directly on your Proxmox Host CLI (root@hyproxmox):
# Append full nested isolation overrides for CT 903 (node-0)
cat <<EOF >> /etc/pve/lxc/903.conf
lxc.apparmor.profile: unconfined
lxc.cgroup2.devices.allow: a
lxc.cap.drop:
lxc.mount.auto: proc:rw sys:rw
EOF

# Append full nested isolation overrides for CT 904 (node-1)
cat <<EOF >> /etc/pve/lxc/904.conf
lxc.apparmor.profile: unconfined
lxc.cgroup2.devices.allow: a
lxc.cap.drop:
lxc.mount.auto: proc:rw sys:rw
EOF
  1. Execute a cold reboot of both worker container instances from the Proxmox host to enforce kernel flags:
pct stop 903 && pct start 903
pct stop 904 && pct start 904
  1. Re-establish /dev/kmsg character device symlinks and restart worker services on node-0 and node-1:
ssh root@node-0 "ln -sf /dev/console /dev/kmsg && systemctl restart containerd kubelet kube-proxy"
ssh root@node-1 "ln -sf /dev/console /dev/kmsg && systemctl restart containerd kubelet kube-proxy"
  1. Delete the stuck Pod on jumpbox to force kube-controller-manager to recreate the sandbox cleanly:
kubectl delete pod -l app=nginx
kubectl get pods -l app=nginx -w

12.5 Verification & Validation Matrix

Use this checklist matrix to confirm that all layers of your custom Kubernetes installation operate in accordance with Kubernetes core specifications:

Sub-Test Component Validated Path / Protocol Tested Success Criteria
Data Encryption kube-apiserver & etcd Encryption at rest (aescbc) k8s:enc:aescbc:v1:key1 header present in raw etcdctl dump; payload encrypted
Workload Deployment controller-manager, scheduler, kubelet, containerd Pod scheduling & OCI container execution Pod transitions to 1/1 Running state
Port Forwarding kube-apiserver, kubelet, socat SPDY / HTTP2 stream proxying Returns HTTP/1.1 200 OK from [http://127.0.0.1:8080](http://127.0.0.1:8080)
Log Streaming kube-apiserver, kubelet RBAC mTLS HTTPS 10250 log streaming kubectl logs streams container stdout access entries
In-Container Exec kubelet, containerd-shim, runc CRI process execution Exec returns container binary output (nginx version: nginx/1.27.4)
NodePort Service kube-proxy, iptables, CNI bridge, host routes DNAT & cross-node packet routing Returns HTTP/1.1 200 OK via http://${NODE_NAME}:${NODE_PORT}

12.6 Section Summary

You have verified that secret data encryption at rest, workload scheduling, control plane stream proxying, container log aggregation, execution hooks, and NodePort service network routing function end-to-end across your custom-built cluster.


13. Cleanup (docs/13-cleanup.md)

  • Resource Decommissioning: Guidelines for tearing down the provisioned VM infrastructure.
  • Wrap-up: Resetting the environment and finalizing the lab state.

Delete all virtual machines created for the exercise

pct destroy 901 --purge
pct destroy 902 --purge
pct destroy 903 --purge
pct destroy 904 --purge

High Level Summary

Why “Kubernetes the Hard Way”?

“Kubernetes the Hard Way” is the gold standard for understanding the mechanics of Kubernetes because it strips away all automation abstractions. There are no managed control planes (EKS/GKE), no automated cluster lifecycle tools (kubeadm, k3s), no operator-driven CNI installers, and no automatic TLS cert managers (cert-manager).

By building the cluster manually, you configure every layer by hand:

  • Manual PKI Infrastructure: You generate a custom Certificate Authority (CA) using OpenSSL and create individual X.509 client/server certificates for every single component, mapping RBAC authorization directly through Certificate Common Name (CN) and Organization (O) fields.
  • Bare Binary Management: Every control plane process (etcd, kube-apiserver, kube-controller-manager, kube-scheduler) and worker process (containerd, kubelet, kube-proxy) runs directly as a bare-metal Linux systemd service rather than inside static pods or containers.
  • Kernel & OS Level Networking: Container networking is built directly on Linux network bridges, CNI plugins, sysctl kernel tuning (br-netfilter), and static OS routing tables (ip route), bypassing automated overlay networks (Flannel/Calico).
  • Explicit Encryption & Configuration: Secret encryption keys, API server authorization webhooks, and kubeconfig files are constructed and wired together line-by-line.

Hard Way vs. Automated & Managed Approaches

Feature / Dimension Managed Kubernetes (EKS / GKE / AKS) Automated Installers (kubeadm / k3s) Kubernetes the Hard Way
Control Plane Deployment Managed black-box service managed by Cloud Provider Automated as static pods or lightweight binaries Manual OS systemd services running raw binaries
PKI & Certificate Management Handled automatically; hidden from operator Generated automatically via CLI commands Manual OpenSSL CA & mTLS key generation per component
Network & CNI Provisioning Provider CNI (VPC-native) auto-configured Deployed via YAML manifests (Flannel, Calico, Cilium) Manual CNI bridge binaries, kernel modules, & ip route tables
Data Encryption at Rest Cloud KMS integrated with 1-click toggle Configurable via flag/manifest edits Hand-crafted AES-CBC encryption configuration file for etcd
Component Authentication Service Accounts & Provider IAM integration Auto-generated Kubeconfig files Hand-built Kubeconfigs with embedded mTLS certificates

Chapter-by-Chapter Bootstrapping Chain

The setup relies on a strict sequential dependency chain where each lab produces the exact configuration, binary, or credential required by the next.

1. Prerequisites ──► 02. Jumpbox Setup ──► 03. Compute & DNS ──► 04. CA & TLS Certificates
                                                                         │
2. etcd Cluster ◄── 06. Data Encryption ◄── 05. Kubeconfig Files ◄──────┘
       │
       ▼
3. Control Plane ──► 09. Worker Nodes ──► 10. Remote kubectl ──► 11. Pod Routes ──► 12. Smoke Test

Chapter 01: Prerequisites

  • What Happens: Defines the hardware and OS requirements—four Debian 12 (bookworm) virtual or physical instances (jumpbox, server, node-0, node-1).
  • Dependency Linkage: Establishes the exact machine topology and resource boundaries (CPUs, RAM, Disk) needed before provisioning administrative tools and remote connections.

Chapter 02: Set Up The Jumpbox

  • What Happens: Configures the jumpbox as the central administrative workstation. Installs OS utilities (curl, openssl, git), clones the tutorial repo, downloads all component binaries (etcd, kube-apiserver, kubelet, containerd, kubectl, CNI plugins), and installs kubectl locally.
  • Dependency Linkage: Centralizes binaries and tools on a single machine so they can be generated, staged, and distributed to cluster nodes via SCP without repeated internet downloads.

Chapter 03: Provisioning Compute Resources

  • What Happens: Maps out node attributes in machines.txt (IPs, FQDNs, Hostnames, Pod CIDRs), configures SSH keypairs for passwordless root access, sets system hostnames, and populates /etc/hosts across all nodes.
  • Dependency Linkage: Provides reliable hostname resolution and passwordless execution across all machines, which is strictly required for generating TLS certificates with valid Subject Alternative Names (SANs) in the next step.

Chapter 04: Provisioning a CA and Generating TLS Certificates

  • What Happens: Bootstraps a local PKI using OpenSSL. Creates a Root CA (ca.crt/ca.key) and generates signed mTLS client/server certificates for admin, node-0, node-1, kube-proxy, kube-scheduler, kube-controller-manager, kube-api-server, and service-accounts. Distributes certificates to respective nodes.
  • Dependency Linkage: Establishes the trust foundation. These TLS certificates serve as component credentials to construct authenticated kubeconfig files in Chapter 05.

Chapter 05: Generating Kubernetes Configuration Files for Authentication

  • What Happens: Uses kubectl config to embed the CA certificate and component-specific client keys into kubeconfig files for node-0, node-1, kube-proxy, kube-controller-manager, kube-scheduler, and admin. Distributes them to worker and control plane nodes.
  • Dependency Linkage: Translates raw TLS keys into structured API server authentication files. The components will use these configs to establish secure mTLS connections to kube-apiserver.

Chapter 06: Generating the Data Encryption Config and Key

  • What Happens: Generates a 32-byte AES random key and constructs encryption-config.yaml specifying the aescbc provider to encrypt Kubernetes Secret objects at rest. SCPs the file to the server.
  • Dependency Linkage: The API server requires this configuration file upon startup to handle Secret object creation, ensuring state data stored in etcd (Chapter 07) is encrypted.

Chapter 07: Bootstrapping the etcd Cluster

  • What Happens: Installs etcd and etcdctl on the server machine, configures TLS keys, creates etcd.service, and starts the single-node etcd cluster.
  • Dependency Linkage: etcd is the single source of truth and state storage for Kubernetes. The control plane (kube-apiserver) cannot start or function without a running etcd service listening on port 2379.

Chapter 08: Bootstrapping the Kubernetes Control Plane

  • What Happens: Installs kube-apiserver, kube-controller-manager, and kube-scheduler on server as systemd services. Starts all components and applies an RBAC ClusterRole (system:kube-apiserver-to-kubelet) permitting the API server to query Kubelet APIs for logs and exec commands.
  • Dependency Linkage: Brings the Kubernetes API server alive on port 6443. The control plane is now fully functional and waiting for worker nodes to register.

Chapter 09: Bootstrapping the Kubernetes Worker Nodes

  • What Happens: Installs container runtimes (runc, containerd), CNI plugins, kubelet, and kube-proxy on node-0 and node-1. Configures CNI bridge networks (10-bridge.conf), enables kernel modules (br-netfilter, sysctl bridge-nf-call-iptables), and starts worker systemd services.
  • Dependency Linkage: Kubelets read their respective node kubeconfig files and register with kube-apiserver. The cluster now shows nodes in Ready status, but cross-node pod routing is not yet configured.

Chapter 10: Configuring kubectl for Remote Access

  • What Happens: Configures ~/.kube/config on the jumpbox using the admin credentials to point to [https://server.kubernetes.local:6443](https://server.kubernetes.local:6443).
  • Dependency Linkage: Allows cluster administrators to run kubectl commands remotely from the jumpbox without SSHing directly into the server control plane node.

Chapter 11: Provisioning Pod Network Routes

  • What Happens: Sets up static IP routing entries (ip route add) on each node to route traffic destined for another node’s Pod CIDR (10.200.0.0/24 for node-0, 10.200.1.0/24 for node-1) via that node’s internal network IP address.
  • Dependency Linkage: Solves cross-node pod-to-pod networking at the Linux OS layer, enabling flat network communication required for multi-node deployments and services.

Chapter 12: Smoke Test

  • What Happens: Executes end-to-end operational validation:
  1. Creates a Secret and inspects raw bytes in etcd using etcdctl to verify k8s:enc:aescbc:v1:key1 encryption.
  2. Deploys an Nginx workload (kubectl create deployment).
  3. Tests port forwarding (kubectl port-forward).
  4. Fetches pod logs (kubectl logs) and executes interactive commands (kubectl exec).
  5. Exposes the deployment as a NodePort service and verifies external HTTP access.
  • Dependency Linkage: Validates that all bootstrapped layers—PKI, Authentication, Storage Encryption, Control Plane, Worker Runtimes, CNI, and OS Routing—work harmoniously as a healthy cluster.

Chapter 13: Cleaning Up

  • What Happens: Teardown phase to delete all virtual machines and clean up local configurations.

Final Cluster Architecture & Artifact Placement

Node & Network Topology

  • Admin Jumpbox (jumpbox): Manages cluster remotely via kubectl pointing to server.kubernetes.local:6443.
  • Control Plane (server): Runs etcd (ports 2379/2380), kube-apiserver (port 6443), kube-controller-manager, and kube-scheduler.
  • Worker Node 0 (node-0): Runs containerd, kubelet (port 10250), kube-proxy, CNI Bridge (10.200.0.0/24 Pod CIDR).
  • Worker Node 1 (node-1): Runs containerd, kubelet (port 10250), kube-proxy, CNI Bridge (10.200.1.0/24 Pod CIDR).
flowchart TB
    subgraph Jumpbox["Jumpbox Admin Host"]
        kubectl["kubectl CLI"]
        pki["Root CA & Cert Generator"]
    end

    subgraph ControlPlane["Control Plane Server (server)"]
        direction TB
        apiserver["kube-apiserver<br/>(Port 6443)"]
        etcd["etcd Key-Value Store<br/>(Ports 2379 / 2380)"]
        cm["kube-controller-manager"]
        sched["kube-scheduler"]

        cm -->|mTLS| apiserver
        sched -->|mTLS| apiserver
        apiserver <-->|mTLS / Port 2379| etcd
    end

    subgraph Node0["Worker Node 0 (node-0)"]
        kubelet0["kubelet<br/>(Port 10250)"]
        kproxy0["kube-proxy"]
        cruntime0["containerd / runc"]
        cni0["CNI Bridge<br/>(Pod CIDR: 10.200.0.0/24)"]

        kubelet0 --> cruntime0
        cruntime0 --> cni0
    end

    subgraph Node1["Worker Node 1 (node-1)"]
        kubelet1["kubelet<br/>(Port 10250)"]
        kproxy1["kube-proxy"]
        cruntime1["containerd / runc"]
        cni1["CNI Bridge<br/>(Pod CIDR: 10.200.1.0/24)"]

        kubelet1 --> cruntime1
        cruntime1 --> cni1
    end

    %% External & Admin Connections
    kubectl -->|mTLS / Port 6443| apiserver

    %% Worker Node Control Connections
    apiserver <-->|API / Metrics / Exec| kubelet0
    apiserver <-->|API / Metrics / Exec| kubelet1
    kproxy0 -->|Watch API| apiserver
    kproxy1 -->|Watch API| apiserver

    %% Cross-Node Pod Network Routes
    cni0 <===>|Static OS Kernel Routing 'ip route'| cni1

Comprehensive File & Artifact Distribution Map

Machine Component Path / Directory Deployed Binaries / Files Description & Role
jumpbox /usr/local/bin/ kubectl Local cluster administration CLI tool.
~/kubernetes-the-hard-way/ ca.crt, ca.key, ca.conf Root Certificate Authority files.
~/.kube/config Admin kubeconfig Embedded admin TLS certs configured to manage cluster remotely.
server /usr/local/bin/ etcd, etcdctl, kube-apiserver, kube-controller-manager, kube-scheduler, kubectl Core control plane executable binaries.
/etc/etcd/ ca.crt, kube-api-server.crt, kube-api-server.key TLS certificates securing etcd client/peer communication.
/var/lib/etcd/ member/ data directory Persistent state database directory for cluster object storage.
/var/lib/kubernetes/ ca.crt, ca.key, kube-api-server.crt, kube-api-server.key, service-accounts.crt, service-accounts.key, encryption-config.yaml, kube-controller-manager.kubeconfig, kube-scheduler.kubeconfig Control plane authentication keys, service account signing pairs, data encryption configuration, and component kubeconfigs.
/etc/kubernetes/config/ kube-scheduler.yaml Scheduler component configuration file.
/etc/systemd/system/ etcd.service, kube-apiserver.service, kube-controller-manager.service, kube-scheduler.service Systemd unit service files for process lifecycle control.
**node-0 & node-1** /usr/local/bin/ crictl, kube-proxy, kubelet, runc Worker runtime binaries and CRI management tools.
/bin/ containerd, containerd-shim-runc-v2, containerd-stress Container execution and shim management binaries.
/opt/cni/bin/ bridge, loopback, host-local, etc. CNI network plugin binaries for pod interface creation.
/etc/cni/net.d/ 10-bridge.conf, 99-loopback.conf CNI network subnet configuration mapped per node (10.200.x.0/24).
/etc/containerd/ config.toml Runtime configuration for containerd daemon.
/var/lib/kubelet/ ca.crt, kubelet.crt, kubelet.key, kubeconfig, kubelet-config.yaml Kubelet mTLS certificates, node identity configuration, and API server access kubeconfig.
/var/lib/kube-proxy/ kubeconfig, kube-proxy-config.yaml Kube-proxy network routing configuration and authentication file.
/etc/systemd/system/ containerd.service, kubelet.service, kube-proxy.service Systemd service files managing node daemons.

Conclusion & Key Takeaways

Building Kubernetes “the hard way” demystifies the black box of cloud-native infrastructure. Through this lab, we explicitly decoupled and configured every core architectural primitive:

  1. Explicit Identity & PKI: Certificate Subject fields (CN and O) are directly tied to Kubernetes RBAC authorization modes (such as system:nodes and system:node:<node-name>).
  2. State & Control Plane Separation: etcd manages cluster state, while systemd units for kube-apiserver, kube-controller-manager, and kube-scheduler drive declarative state convergence independently.
  3. Bare-Metal Networking: Without an overlay network (like Calico or Cilium), Pod-to-Pod cross-node communication relies directly on layer-3 static routing tables paired with local CNI bridge interfaces (br-netfilter).
  4. Data Protection: Secrets are not inherently secure in etcd unless explicit provider configurations (encryption-config.yaml) encrypt payloads at rest.

Mastering these components provides the foundational troubleshooting skills required to debug complex production cluster failures, network plugin issues, and control plane degradation.