Kubernetes The Hard Way: Bootstrapping a Bare-Metal Cluster from Scratch
Abstract
Modern managed Kubernetes distributions—such as kubeadm, k3s, EKS, and GKE—abstract away the intricate underlying mechanics that keep container orchestration systems running. While these tools streamline production deployments, they hide the essential relationships between control plane components, public key infrastructure (PKI), container runtime interfaces, and bare-metal routing.
This post documents the step-by-step journey of manually bootstrapping a fully functional Kubernetes v1.32 cluster on four bare-metal or virtualized Debian 12 nodes. By manually issuing TLS certificates, configuring etcd, assembling control plane binaries (kube-apiserver, kube-controller-manager, kube-scheduler), wiring worker runtimes (containerd, runc, CNI plugins), and managing Linux static routing tables, you will gain a deep, production-grade operational understanding of Kubernetes mechanics from the ground up.
1. Prerequisites (docs/01-prerequisites.md)
- System Requirements: Overview of the four Debian 12 (Bookworm) virtual or physical instances required for the lab topology:
| Name | Description | CPU | RAM | Storage |
|---|---|---|---|---|
jumpbox |
Administration host | 1 vCPU | 512 MB | 10 GB |
server |
Kubernetes control plane | 1 vCPU | 2 GB | 20 GB |
node-0 |
Kubernetes worker node | 1 vCPU | 2 GB | 20 GB |
node-1 |
Kubernetes worker node | 1 vCPU | 2 GB | 20 GB |
- OS Verification: Validating release metadata via
/etc/os-releaseacross all target instances.
This section covers the provisioning of a 4-node infrastructure layout using Proxmox LXC containers connected to a dedicated virtual bridge (vmbr2).
[ jumpbox (10.0.0.1) ] ──┐
[ server (10.0.0.2) ] ──┼──► [ Proxmox Bridge: vmbr2 ] (10.0.0.254/24)
[ node-0 (10.0.0.3) ] ──┤ │ (IPTables SNAT / NAT)
[ node-1 (10.0.0.4) ] ──┘ ▼
[ Proxmox Bridge: vmbr0 ] (192.168.1.x)
│
▼
[ Gateway Router / Internet ]
1.1 High-Level Node Architecture
| Hostname | CT ID | Role | vCPU | Memory | Disk | IP Address | Subnet |
|---|---|---|---|---|---|---|---|
| jumpbox | 901 |
Administration & CA Authority | 1 | 512 MB | 4 GB | 10.0.0.1 |
10.0.0.0/24 |
| server | 902 |
Kubernetes Control Plane (etcd, apiserver, etc.) |
1 | 1024 MB | 8 GB | 10.0.0.2 |
10.0.0.0/24 |
| node-0 | 903 |
Worker Node 0 (containerd, kubelet, kube-proxy) |
1 | 512 MB | 4 GB | 10.0.0.3 |
10.0.0.0/24 |
| node-1 | 904 |
Worker Node 1 (containerd, kubelet, kube-proxy) |
1 | 512 MB | 4 GB | 10.0.0.4 |
10.0.0.0/24 |
1.2 Technical Understanding & Deep Dive
Virtual Bridges vs. Physical Egress Routing
Linux bridge interfaces (vmbr0, vmbr2) act as virtual Layer 2 switches inside the kernel. When a container sends a packet to an external destination (e.g., 1.1.1.1), the kernel consults its host routing table to find the appropriate egress interface.
In a nested virtual network where physical NIC nic0 is enslaved under vmbr0, outbound packets exiting the host use vmbr0 as their master network device. Defining an IP masquerading rule targeting nic0 directly fails because outbound packets are attached to the vmbr0 logical interface before leaving the system.
+--------------------------------------------------------------------------+
| Proxmox Host (hyproxmox) |
| |
| [ LXC: jumpbox ] -- (veth) --> [ Bridge: vmbr2 ] (10.0.0.254) |
| | |
| (IP Forwarding) |
| v |
| [ LAN / WAN ] <-- (nic0) -- [ Bridge: vmbr0 ] (192.168.1.x) <--- NAT |
+--------------------------------------------------------------------------+
SNAT / IP Masquerading Mechanism
Because the 10.0.0.0/24 subnet consists of non-routable RFC 1918 private IP addresses, external routers on your local network or the wider internet do not know how to send return traffic to 10.0.0.1.
Source Network Address Translation (SNAT)—specifically MASQUERADE in iptables—rewrites the IP header of every outbound packet originating from 10.0.0.0/24. It changes the source address from the internal container IP (e.g., 10.0.0.1) to the host’s LAN IP on vmbr0 (e.g., 192.168.1.x). The host tracks these active connection states in its conntrack table so return traffic is routed seamlessly back to the originating container.
Reverse Path Filtering (rp_filter)
Linux includes a security feature called Reverse Path Filtering (rp_filter). The kernel checks whether an incoming packet on an interface arrives via the same path that would be used to reach the source IP. If asymmetric paths exist or multiple virtual bridges overlap on similar subnets, rp_filter drops packets as potential spoofed traffic. Setting net.ipv4.conf.all.rp_filter = 0 disables these strict checks on custom bridged setups.
1.3 Step-by-Step Implementation
Step 1: Provision LXC Containers via Proxmox CLI
Run the automated provisioning commands on your Proxmox host (root@hyproxmox):
# Extract the exact Ubuntu 24.04 LXC template path
TEMPLATE=$(pveam list local | grep ubuntu-24.04 | awk '{print $1}' | head -n 1)
STORAGE="local-lvm"
BRIDGE="vmbr2"
GATEWAY="10.0.0.254"
ROOT_PASS="ChangeMe123!"
# 1. jumpbox (ID: 901, IP: 10.0.0.1)
pct create 901 $TEMPLATE \
--hostname jumpbox \
--ostype ubuntu \
--cores 1 \
--memory 512 \
--swap 0 \
--rootfs ${STORAGE}:4 \
--net0 name=eth0,bridge=${BRIDGE},ip=10.0.0.1/24,gw=${GATEWAY} \
--unprivileged 1 \
--features nesting=1,keyctl=1 \
--password "$ROOT_PASS" \
--onboot 1
# 2. server (ID: 902, IP: 10.0.0.2)
pct create 902 $TEMPLATE \
--hostname server \
--ostype ubuntu \
--cores 1 \
--memory 1024 \
--swap 0 \
--rootfs ${STORAGE}:8 \
--net0 name=eth0,bridge=${BRIDGE},ip=10.0.0.2/24,gw=${GATEWAY} \
--unprivileged 1 \
--features nesting=1,keyctl=1 \
--password "$ROOT_PASS" \
--onboot 1
# 3. node-0 (ID: 903, IP: 10.0.0.3)
pct create 903 $TEMPLATE \
--hostname node-0 \
--ostype ubuntu \
--cores 1 \
--memory 512 \
--swap 0 \
--rootfs ${STORAGE}:4 \
--net0 name=eth0,bridge=${BRIDGE},ip=10.0.0.3/24,gw=${GATEWAY} \
--unprivileged 1 \
--features nesting=1,keyctl=1 \
--password "$ROOT_PASS" \
--onboot 1
# 4. node-1 (ID: 904, IP: 10.0.0.4)
pct create 904 $TEMPLATE \
--hostname node-1 \
--ostype ubuntu \
--cores 1 \
--memory 512 \
--swap 0 \
--rootfs ${STORAGE}:4 \
--net0 name=eth0,bridge=${BRIDGE},ip=10.0.0.4/24,gw=${GATEWAY} \
--unprivileged 1 \
--features nesting=1,keyctl=1 \
--password "$ROOT_PASS" \
--onboot 1
--unprivileged 1: Runs the container in an isolated user namespace for security.--features nesting=1,keyctl=1: Enables container nesting and keyring functionality required for running systemd services and container runtimes (such ascontainerd) inside LXCs.
Step 2: Start Containers and Configure Host NAT Rules
Start all instances concurrently:
pct start 901 && pct start 902 && pct start 903 && pct start 904
Apply runtime network rules on root@hyproxmox to allow IP forwarding and source translation over master bridge vmbr0:
# Clean up any misplaced rules pointing to the physical enslaved slave port
iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o nic0 -j MASQUERADE 2>/dev/null || true
# Append MASQUERADE rule targeting the active egress master bridge
iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE
# Permit local host traffic and forward traffic across the bridge
iptables -I INPUT -i vmbr2 -j ACCEPT
iptables -I FORWARD -i vmbr2 -j ACCEPT
iptables -I FORWARD -o vmbr2 -j ACCEPT
Step 3: Persist Network and Kernel Configurations
To ensure bridge forwarding and NAT rules persist across host reboots, append the following configurations on root@hyproxmox:
Update /etc/network/interfaces:
auto vmbr2
iface vmbr2 inet static
address 10.0.0.254/24
bridge-ports none
bridge-stp off
bridge-fd 0
post-up echo 1 > /proc/sys/net/ipv4/ip_forward
post-up iptables -t nat -A POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE
post-up iptables -I INPUT -i vmbr2 -j ACCEPT
post-up iptables -I FORWARD -i vmbr2 -j ACCEPT
post-up iptables -I FORWARD -o vmbr2 -j ACCEPT
post-down iptables -t nat -D POSTROUTING -s '10.0.0.0/24' -o vmbr0 -j MASQUERADE
post-down iptables -D INPUT -i vmbr2 -j ACCEPT
post-down iptables -D FORWARD -i vmbr2 -j ACCEPT
post-down iptables -D FORWARD -o vmbr2 -j ACCEPT
Create /etc/sysctl.d/99-k8s-lxc.conf:
net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.vmbr2.rp_filter = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0
Apply sysctl changes immediately:
sysctl --system
1.4 Verification & Validation
1. Verify LXC Status on Proxmox Host
Inspect the status of running containers:
pct list
VMID Status Lookups Name
901 running 10.0.0.1 jumpbox
902 running 10.0.0.2 server
903 running 10.0.0.3 node-0
904 running 10.0.0.4 node-1
2. Verify Packet Matching on NAT Rules
Confirm that outbound packets from 10.0.0.0/24 are hitting the correct MASQUERADE target:
iptables -t nat -L POSTROUTING -n -v | grep MASQUERADE
12 840 MASQUERADE all -- * vmbr0 10.0.0.0/24 0.0.0.0/0
3. Verify Local Bridge and Public WAN Reachability
Test ping connectivity from jumpbox (CT 901) to the host gateway (10.0.0.254) and a public address (1.1.1.1):
pct exec 901 -- ping -c 3 10.0.0.254
PING 10.0.0.254 (10.0.0.254) 56(84) bytes of data.
64 bytes from 10.0.0.254: icmp_seq=1 ttl=64 time=0.048 ms
64 bytes from 10.0.0.254: icmp_seq=2 ttl=64 time=0.039 ms
64 bytes from 10.0.0.254: icmp_seq=3 ttl=64 time=0.038 ms
--- 10.0.0.254 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2045ms
rtt min/avg/max/mdev = 0.038/0.041/0.048/0.005 ms
pct exec 901 -- ping -c 3 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=56 time=11.2 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=56 time=10.8 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=56 time=10.9 ms
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 10.841/10.980/11.214/0.165 ms
4. Verify HTTP and DNS Resolution
Perform a curl request to confirm external HTTPS connectivity and working DNS resolution:
pct exec 901 -- curl -I https://github.com
HTTP/2 200
server: GitHub.com
date: Sat, 31 Oct 2026 10:36:00 GMT
content-type: text/html; charset=utf-8
vary: X-PJAX, X-PJAX-Container, Accept-Encoding, Accept, X-Requested-With
2. Jumpbox Provisioning (docs/02-jumpbox.md)
- Admin Tooling: Installing standard utilities (
wget,curl,vim,openssl,git) on thejumpbox. - Repository Sync: Cloning the core repository and establishing the base working environment.
- Binary Staging: Staging architecture-specific binaries (
etcd,kubectl,kube-apiserver,containerd,runc,cni-plugins) indownloads/. - Local Client Tooling: Installing and verifying
kubectlon the administrative workstation.
Setting Up the Jumpbox
2.1 Section Overview & Context
Before bootstrapping a Kubernetes cluster from the ground up, you need a dedicated administrative environment—the Jumpbox.
In the broader architecture of Kubernetes The Hard Way, the jumpbox acts as your central command center and artifact repository. Rather than installing build tools, fetching binaries, or generating Public Key Infrastructure (PKI) certificates on individual cluster nodes, you perform all preparation work here.
+-----------------------------------+
| ADMIN JUMPBOX |
| 10.0.0.1 |
| |
| [Local Artifact Cache] |
| ├── downloads/client/ |
| ├── downloads/controller/ |
| └── downloads/worker/ |
+-----------------+-----------------+
|
+-------------------------+-------------------------+
| | |
v v v
+------------------+ +------------------+ +------------------+
| Control Plane | | Worker 0 | | Worker 1 |
| 10.0.0.10-12 | | 10.0.0.20 | | 10.0.0.21 |
+------------------+ +------------------+ +------------------+
High-Level Architecture Summary
Setting up the jumpbox accomplishes three primary goals:
- Centralized Administration: Provides a isolated, uniform environment (
10.0.0.1) to execute provisioning scripts, manage SSH keys, generate TLS certificates, and issue cluster-wide commands. - Local Artifact Caching: Downloads over 500 MB of core binaries (
etcd,containerd,kube-apiserver,kubelet,cni-plugins, etc.) once. These binaries are organized by node role (client,controller,worker), eliminating redundant external downloads when staging cluster nodes. - Management Tooling: Installs client tools such as
kubectllocally, enabling direct control-plane communication once the API server is online.
2.2 Technical Understanding & Deep Dive: The “Why”
Why Use a Dedicated Jumpbox?
In production environments, control plane instances and worker nodes should be locked down minimal operating system images with restricted external internet access. Staging all binaries, configuration templates, and cryptographic assets on a dedicated jumpbox allows you to maintain strict security boundaries while providing a single source of truth for cluster assets.
Architecture-Aware Downloading (dpkg --print-architecture)
Kubernetes components are distributed as compiled Go binaries tied to specific CPU architectures (e.g., amd64, arm64). By dynamically querying your system’s architecture using dpkg --print-architecture, the bootstrap process automatically selects the correct download manifest file (downloads-amd64.txt vs. downloads-arm64.txt). This avoids architecture mismatch errors during binary execution (such as Exec format error).
Binary Role Segregation
Instead of dumping all binaries into a single directory, we extract and group them into strict role-based directories:
| Target Directory | Binaries Included | Functional Role |
|---|---|---|
downloads/client/ |
kubectl, etcdctl |
CLI client tools used by administrators to interact with Kubernetes and etcd. |
downloads/controller/ |
etcd, kube-apiserver, kube-controller-manager, kube-scheduler |
Core control plane services responsible for cluster consensus, API exposure, state control loops, and Pod scheduling. |
downloads/worker/ |
kubelet, kube-proxy, containerd, crictl, runc |
Node-level runtime execution engines, container orchestration agents, and networking proxies. |
downloads/cni-plugins/ |
bridge, loopback, host-local, etc. |
Container Network Interface binaries that manage Pod networking, IP allocation, and virtual interfaces. |
Archive Extraction Strategy: Strip Components
When extracting multi-file distribution archives like etcd or containerd, the archive root often contains nested directory hierarchies (e.g., etcd-v3.6.0-rc.3-linux-amd64/etcd). Using tar --strip-components 1 removes leading directory paths during extraction, allowing binaries to land cleanly into targeted target directories without unwanted subfolder nesting.
2.3 Step-by-Step Implementation & Command Breakdown
All commands are executed on the jumpbox (10.0.0.1) as the root user to ensure consistent access permissions throughout the setup.
Step 1: Log In to the Jumpbox & Check Connectivity
Access your jumpbox container via Proxmox LXC enter or SSH:
# If accessing directly from Proxmox VE host:
pct enter 901
# Or via SSH from your workstation:
ssh root@10.0.0.1
Verify outbound network access:
ping -c 2 github.com
Step 2: Install Required Utilities
Update local repository metadata and install essential administrative tools:
apt-get update && apt-get -y install wget curl vim openssl git
wget/curl: Used to retrieve remote binaries, configuration files, and API endpoints.openssl: Essential for inspecting and generating PKI TLS certificates.git: Required to clone the source configuration repository.
Step 3: Clone the Kubernetes The Hard Way Repository
Clone the upstream repository to obtain setup scripts, systemd unit templates, and binary download manifests:
git clone --depth 1 https://github.com/kelseyhightower/kubernetes-the-hard-way.git
cd kubernetes-the-hard-way
--depth 1: Performs a shallow clone, downloading only the latest revision to save time and bandwidth.
Verify your working directory:
pwd
Expected Output:
/root/kubernetes-the-hard-way
Step 4: Download & Organize Kubernetes Binaries
Inspect your system architecture manifest list:
cat downloads-$(dpkg --print-architecture).txt
Download all cluster components into the downloads/ cache directory:
wget -q --show-progress \
--https-only \
--timestamping \
-P downloads \
-i downloads-$(dpkg --print-architecture).txt
Command Flag Breakdown:
-q --show-progress: Hides verbose output while rendering a clean progress bar.--https-only: Enforces secure HTTPS transport for binary verification.--timestamping(-N): Re-downloads files only if the remote file is newer than the local copy.-P downloads: Saves all retrieved files into thedownloadsdirectory.-i <file>: Reads target download URLs directly from the specified manifest file.
Verify the downloaded archives (~500 MB total):
ls -lh downloads
Extract binaries and route them into their target role directories:
{
ARCH=$(dpkg --print-architecture)
mkdir -p downloads/{client,cni-plugins,controller,worker}
# Worker runtime & CNI binaries
tar -xvf downloads/crictl-v1.32.0-linux-${ARCH}.tar.gz -C downloads/worker/
tar -xvf downloads/containerd-2.1.0-beta.0-linux-${ARCH}.tar.gz --strip-components 1 -C downloads/worker/
tar -xvf downloads/cni-plugins-linux-${ARCH}-v1.6.2.tgz -C downloads/cni-plugins/
# etcd binaries
tar -xvf downloads/etcd-v3.6.0-rc.3-linux-${ARCH}.tar.gz -C downloads/ \
--strip-components 1 \
etcd-v3.6.0-rc.3-linux-${ARCH}/etcdctl \
etcd-v3.6.0-rc.3-linux-${ARCH}/etcd
# Categorize binaries into role directories
mv downloads/{etcdctl,kubectl} downloads/client/
mv downloads/{etcd,kube-apiserver,kube-controller-manager,kube-scheduler} downloads/controller/
mv downloads/{kubelet,kube-proxy} downloads/worker/
mv downloads/runc.${ARCH} downloads/worker/runc
}
Clean up tarballs and grant execution permissions across all categorized binaries:
rm -rf downloads/*.gz downloads/*.tgz
chmod +x downloads/{client,cni-plugins,controller,worker}/*
Step 5: Install kubectl on the Jumpbox
Install kubectl into your system PATH to allow immediate administrative access to remote clusters:
cp downloads/client/kubectl /usr/local/bin/
chmod +x /usr/local/bin/kubectl
2.4 Verification & Validation
Verify that your local workspace is organized correctly and that the client tools are ready.
1. Validate Organized Directory Structure
Inspect the contents of the downloads/ directory:
ls -l downloads/*
Expected Output:
downloads/client:
total 162M
-rwxr-xr-x 1 root root 98M Mar 27 00:00 etcdctl
-rwxr-xr-x 1 root root 65M Mar 27 00:00 kubectl
downloads/cni-plugins:
total 72M
-rwxr-xr-x 1 root root 4.1M Mar 27 00:00 bandwidth
-rwxr-xr-x 1 root root 4.6M Mar 27 00:00 bridge
-rwxr-xr-x 1 root root 11M Mar 27 00:00 dhcp
-rwxr-xr-x 1 root root 4.3M Mar 27 00:00 firewall
-rwxr-xr-x 1 root root 3.7M Mar 27 00:00 host-device
-rwxr-xr-x 1 root root 3.6M Mar 27 00:00 host-local
-rwxr-xr-x 1 root root 4.3M Mar 27 00:00 ipvlan
-rwxr-xr-x 1 root root 3.8M Mar 27 00:00 loopback
-rwxr-xr-x 1 root root 4.4M Mar 27 00:00 macvlan
-rwxr-xr-x 1 root root 4.1M Mar 27 00:00 ptp
-rwxr-xr-x 1 root root 3.9M Mar 27 00:00 sbr
-rwxr-xr-x 1 root root 3.1M Mar 27 00:00 static
-rwxr-xr-x 1 root root 4.2M Mar 27 00:00 tap
-rwxr-xr-x 1 root root 3.8M Mar 27 00:00 tuning
-rwxr-xr-x 1 root root 4.3M Mar 27 00:00 vlan
-rwxr-xr-x 1 root root 4.7M Mar 27 00:00 vrf
downloads/controller:
total 252M
-rwxr-xr-x 1 root root 24M Mar 27 00:00 etcd
-rwxr-xr-x 1 root root 124M Mar 27 00:00 kube-apiserver
-rwxr-xr-x 1 root root 68M Mar 27 00:00 kube-controller-manager
-rwxr-xr-x 1 root root 36M Mar 27 00:00 kube-scheduler
downloads/worker:
total 165M
-rwxr-xr-x 1 root root 52M Mar 27 00:00 containerd
-rwxr-xr-x 1 root root 7.1M Mar 27 00:00 containerd-shim-runc-v2
-rwxr-xr-x 1 root root 14M Mar 27 00:00 crictl
-rwxr-xr-x 1 root root 24M Mar 27 00:00 kube-proxy
-rwxr-xr-x 1 root root 54M Mar 27 00:00 kubelet
-rwxr-xr-x 1 root root 14M Mar 27 00:00 runc
2. Verify kubectl Client Binary
Confirm that kubectl is installed globally and functioning:
kubectl version --client
Expected Output:
Client Version: v1.32.3
Kustomize Version: v5.5.0
The administrative jumpbox (10.0.0.1) is fully provisioned with essential utilities, repository templates, binary caching directories, and the kubectl client tool (v1.32.3).
3. Provisioning Compute Resources (docs/03-compute-resources.md)
- Machine Database: Creating
machines.txtto track IP address allocations, FQDNs, hostnames, and assigned Pod CIDRs (10.200.x.0/24). - SSH Connectivity: Enabling
rootSSH login across instances, generating SSH keypairs on thejumpbox, and distributing public keys. - Hostname Configuration: Setting FQDN hostnames using
hostnamectland generating host lookup tables across all node/etc/hostsfiles.
Before installing container runtimes, configuring control plane services, or issuing TLS certificates, you must establish a reliable compute and networking baseline. In this section, you will lay the groundwork for a 4-node cluster comprising one control plane node (server), two worker nodes (node-0, node-1), and a dedicated administrative node (jumpbox).
Installing Kubernetes from scratch requires total administrative control and predictable network routes across all participating systems. To achieve this without relying on external cloud providers or automated provisioning scripts, you will:
- Define an authoritative machine database tracking static IPs, Hostnames, FQDNs, and Pod CIDR allocations.
- Enable root administrative access and set up an SSH key-based trust model for non-interactive orchestration from the
jumpbox. - Configure system hostnames and establish a static local host lookup table across every node to ensure deterministic cluster-wide name resolution without an external DNS server.
+-----------------------------------+
| jumpbox (10.0.0.1) |
| (Orchestration & Administration) |
+-----------------+-----------------+
|
+-----------------------+-----------------------+
| | |
v v v
+-----------------------+ +-------------------+ +-------------------+
| server (10.0.0.2) | | node-0 (10.0.0.3)| | node-1 (10.0.0.4)|
| Control Plane / etcd | | Worker Node | | Worker Node |
| (No Pod Subnet) | | Subnet: | | Subnet: |
| | | 10.200.0.0/24 | | 10.200.1.0/24 |
+-----------------------+ +-------------------+ +-------------------+
3.1 TECHNICAL UNDERSTANDING & DEEP DIVE (THE “WHY”)
1. Machine Database & Pod Subnet Allocation
Kubernetes imposes a strict networking requirement: every Pod must receive a unique, routable IP address across the entire cluster. To enforce this without complex dynamic allocation during initial boot, each worker node is assigned a non-overlapping IP subnet—a Pod CIDR (e.g., 10.200.0.0/24).
- Node
node-0receives10.200.0.0/24, reserving IPs10.200.0.1through10.200.0.254exclusively for Pods running onnode-0. - Node
node-1receives10.200.1.0/24, reserving IPs10.200.1.1through10.200.1.254for Pods running onnode-1.
Storing these parameters in a structured file (machines.txt) creates a single source of truth. Automated shell loops (while read ...) parse this data dynamically throughout subsequent setup steps.
2. Root SSH Access & Passwordless Key Authentication
Bootstrapping a control plane requires creating system directories, installing systemd unit files, managing etcd datastores, and configuring kernel modules—actions that demand superuser (root) privileges. While disabling root SSH access is standard security practice, optimizing for clarity and operational speed during low-level cluster creation justifies using direct SSH key access from a trusted administrative jumpbox.
Using an unpassphrased RSA key pair allows shell scripts on the jumpbox to push files via scp and execute commands via ssh without stopping for password prompts.
3. Hostname Resolution without External DNS
Kubernetes components rely heavily on hostnames:
kubeletregisters itself to the cluster control plane using its local hostname.kube-apiservervalidates client identity by cross-referencing Hostnames and FQDNs against Subject Alternative Names (SANs) embedded inside TLS certificates.etcdnodes coordinate consensus using member hostnames.
Rather than introducing an external DNS server (which adds failure domain complexity to a bootstrap environment), local host resolution is enforced statically via /etc/hosts across all machines.
3.2 STEP-BY-STEP IMPLEMENTATION & COMMAND BREAKDOWN
Step 1: Create the Machine Database
On the jumpbox (10.0.0.1), navigate to your working directory and create the machines.txt source of truth file:
cat <<EOF > machines.txt
10.0.0.2 server.kubernetes.local server
10.0.0.3 node-0.kubernetes.local node-0 10.200.0.0/24
10.0.0.4 node-1.kubernetes.local node-1 10.200.1.0/24
EOF
Inspect the generated database:
cat machines.txt
10.0.0.2 server.kubernetes.local server
10.0.0.3 node-0.kubernetes.local node-0 10.200.0.0/24
10.0.0.4 node-1.kubernetes.local node-1 10.200.1.0/24
Schema Breakdown:
- Column 1 (
IPV4_ADDRESS): The primary host IP address accessible over the local network. - Column 2 (
FQDN): Fully Qualified Domain Name used for certificate SAN alignment. - Column 3 (
HOSTNAME): Short hostname assigned to the node operating system. - Column 4 (
POD_SUBNET): The dedicated/24CIDR range reserved for container allocation on worker nodes.
Step 2: Configure Root SSH Access on Cluster Nodes
Log into each node (server, node-0, node-1) via standard user SSH, elevate to root, and permit root logins in the OpenSSH daemon configuration.
- Elevate to root:
su - root
Note on
su -: The-(login shell) flag clears the caller’s environment and loads a clean environment withroot’s native$PATH,$HOME, and system variables.
- Enable root login in
/etc/ssh/sshd_config:
sed -i \
's/^#*PermitRootLogin.*/PermitRootLogin yes/' \
/etc/ssh/sshd_config
sed -i: Modifies the file in-place without writing a secondary output file.'s/^#*PermitRootLogin.*/PermitRootLogin yes/': Uses regular expressions to locate any variation of thePermitRootLoginkey (whether active or commented out with#) and replaces the line withPermitRootLogin yes.
- Restart the SSH daemon to apply changes:
systemctl restart sshd
Step 3: Generate and Distribute SSH Keys
From the jumpbox (10.0.0.1), generate a secure 4096-bit RSA key pair without a passphrase and distribute the public key to all nodes.
- Generate the SSH key pair on
jumpbox:
ssh-keygen -t rsa -b 4096 -f /root/.ssh/id_rsa -N ""
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/id_rsa
Your public key has been saved in /root/.ssh/id_rsa.pub
-t rsa -b 4096: Requests RSA key generation using a 4096-bit length.-f /root/.ssh/id_rsa: Sets the target path for the private key file.-N "": Configures an empty passphrase for unprompted script execution.
- Distribute the public key to each cluster machine:
while read IP FQDN HOST SUBNET; do
ssh-copy-id root@${IP}
done < machines.txt
Enter the
rootpassword for each node when prompted.ssh-copy-idappends/root/.ssh/id_rsa.pubinto the target host’s/root/.ssh/authorized_keysfile and sets correct permissions (600).
- Verify non-interactive execution across all machines:
while read IP FQDN HOST SUBNET; do
ssh -n root@${IP} hostname
done < machines.txt
server
node-0
node-1
- The
-nflag: Redirectsstdinfrom/dev/null. Without-n, thesshbinary consumes the remaining lines ofmachines.txtdirectly from standard input, causing the loop to exit after processing only the first entry.
Step 4: Configure Hostnames across the Cluster
Set hostnames on each target host dynamically from the jumpbox:
while read IP FQDN HOST SUBNET; do
CMD="sed -i 's/^127.0.1.1.*/127.0.1.1\t${FQDN} ${HOST}/' /etc/hosts"
ssh -n root@${IP} "$CMD"
ssh -n root@${IP} hostnamectl set-hostname ${HOST}
ssh -n root@${IP} systemctl restart systemd-hostnamed
done < machines.txt
Command Breakdown:
sed -i 's/^127.0.1.1...*/...': Updates local loopback definitions in/etc/hoststo match the target FQDN and short hostname, preventing administrative utilities likesudofrom raisingunable to resolve hosterrors.hostnamectl set-hostname ${HOST}: Interacts withsystemd-hostnamedto set the runtime kernel hostname parameter (sysctl kernel.hostname) and writes the configuration persistently to/etc/hostname.systemctl restart systemd-hostnamed: Forces background administrative daemons to adopt the updated hostname string immediately.
Step 5: Build and Distribute the Host Lookup Table
Create a consolidated static lookup table containing host mapping entries for all cluster nodes, then apply it locally and remotely.
- Generate the static host entries on
jumpbox:
echo "" > hosts
echo "# Kubernetes The Hard Way" >> hosts
while read IP FQDN HOST SUBNET; do
ENTRY="${IP} ${FQDN} ${HOST}"
echo $ENTRY >> hosts
done < machines.txt
- Inspect the generated file:
cat hosts
# Kubernetes The Hard Way
10.0.0.2 server.kubernetes.local server
10.0.0.3 node-0.kubernetes.local node-0
10.0.0.4 node-1.kubernetes.local node-1
- Append host entries to the local
/etc/hostsfile onjumpbox:
cat hosts >> /etc/hosts
- Distribute and append host entries to all cluster nodes:
while read IP FQDN HOST SUBNET; do
scp hosts root@${HOST}:~/
ssh -n root@${HOST} "cat hosts >> /etc/hosts"
done < machines.txt
3.3 REAL-WORLD TROUBLESHOOTING & PITFALLS
VERIFICATION & VALIDATION
Verify that hostnames are correctly configured and reachable using their short names from the jumpbox.
- Verify Fully Qualified Domain Name resolution on each node:
while read IP FQDN HOST SUBNET; do
ssh -n root@${IP} hostname --fqdn
done < machines.txt
server.kubernetes.local
node-0.kubernetes.local
node-1.kubernetes.local
- Verify hostname-based SSH connectivity from the
jumpboxwithout raw IP addresses:
for host in server node-0 node-1; do
ssh root@${host} hostname
done
server
node-0
node-1
- Verify cross-node host lookup entry in
/etc/hosts:
ssh root@node-0 "ping -c 1 server"
PING server.kubernetes.local (10.0.0.2) 56(84) bytes of data.
64 bytes from server.kubernetes.local (10.0.0.2): icmp_seq=1 ttl=64 time=0.185 ms
--- server.kubernetes.local ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.185/0.185/0.185/0.185 ms
4. PKI & Certificate Authority (docs/04-certificate-authority.md)
-
Certificate Authority Setup: Bootstrapping a self-signed root CA (
ca.key,ca.crt) using OpenSSL andca.conf. -
Component Certificates: Generating TLS client and server certificate pairs for:
-
Administrative identity:
admin -
Control plane services:
kube-api-server,kube-controller-manager,kube-scheduler,service-accounts -
Worker components:
kube-proxy, and node-specifickubeletcerts (node-0,node-1) -
Key Distribution: Copying target certificates and private keys to
/var/lib/kubelet/on workers and~/on the server.
Kubernetes operates on a Zero-Trust network architecture:
- no component trusts any other component by default.
- Every internal connection—from
kubectlissuing commands to the API server, to worker node daemons reporting pod statuses - is encrypted in transit and authenticated using Mutual TLS (mTLS).
In this section, you will establish the foundational Public Key Infrastructure (PKI) for your Kubernetes cluster from the jumpbox provisioning node (10.0.0.1). You will create a self-signed Root Certificate Authority (CA) and generate all client/server X.509 certificate pairs required by the Kubernetes control plane and worker nodes. Finally, you will distribute these sensitive security credentials to their respective host nodes (10.0.0.2, 10.0.0.3, and 10.0.0.4).
+---------------------------------------+
| JUMPBOX |
| (10.0.0.1) |
| Generates Root CA, component certs, |
| and stores client admin credentials |
+-------------------+-------------------+
|
+-------------------+-------------------+
| SCP | SCP
v v
+------------------------------------+ +------------------------------------+
| SERVER | | NODE-0 / NODE-1 |
| (10.0.0.2) | | (10.0.0.3 / 10.0.0.4) |
| Control Plane Host | | Worker Node Hosts |
| - ca.crt & ca.key | | - ca.crt |
| - kube-api-server.crt / .key | | - kubelet.crt (node-specific) |
| - service-accounts.crt / .key | | - kubelet.key (node-specific) |
+------------------------------------+ +------------------------------------+
4.1 Technical Understanding & Deep Dive (The “Why”)
Before executing commands, it is crucial to understand how Kubernetes uses digital certificates for identity, authorization, and encryption.
1. Mutual TLS (mTLS) vs. Standard TLS
In traditional web browsing, TLS is one-way: your browser verifies the server’s identity via a public certificate, but the server relies on passwords or cookies to identify you. In Kubernetes mTLS:
- Server Verification: The server presents a certificate to prove its identity to the client.
- Client Verification: The client presents its own certificate to prove its identity to the server.
- Common Trust Anchor: Both parties validate each other against the same Root Certificate Authority (
ca.crt).
2. X.509 Certificate Fields mapped to Kubernetes RBAC
Kubernetes does not maintain an internal user database or user table. Instead, the kube-apiserver extracts user identity directly from the subject fields of presented X.509 client certificates:
- Common Name (
CN): Interpreted by Kubernetes as the User Identity (e.g.,CN=adminorCN=system:node:node-0). - Organization (
O): Interpreted by Kubernetes as Group Membership (e.g.,O=system:mastersgrants cluster-wide administrative privileges;O=system:nodesidentifies worker node groups).
3. Subject Alternative Names (SANs)
When a client connects to an HTTPS service like kube-apiserver, TLS verification checks that the IP address or domain name requested matches the certificate. The Subject Alternative Names (SAN) field explicitly lists every valid IP (10.0.0.2, 127.0.0.1) and domain name (kubernetes, kubernetes.default, kubernetes.default.svc.cluster.local) through which kube-apiserver can be reached.
4. The Special Case: service-accounts Key Pair
Unlike the other certificate pairs generated in this section, service-accounts.key and service-accounts.crt are not used for TLS network encryption. Instead:
kube-controller-managerusesservice-accounts.key(RSA private key) to sign asymmetric JWT tokens issued to Service Accounts inside Pods.kube-apiserverusesservice-accounts.crt(public key) to verify those token signatures when Pods authenticate against the API.
5. Inventory of Generated File Extensions
| Extension | File Type | Purpose | Security Level |
|---|---|---|---|
.key |
Private Key | Secret key used to sign tokens, decrypt traffic, and prove identity ownership. | Strictly Secret |
.csr |
Certificate Signing Request | Temporary request containing identity attributes (CN, O, SANs) sent to the CA for signing. |
Temporary (Disposable) |
.crt |
Public Certificate | Signed public certificate containing the public key and identity attributes. Distributed freely for identity verification. | Public / Distributed |
4.2 Step-by-Step Implementation & Command Breakdown
All commands in this section are executed from the jumpbox (10.0.0.1) inside the /root/kubernetes-the-hard-way directory.
Step 1: Create the Certificate Authority (CA)
First, provision the Root CA private key and self-signed root certificate.
cd /root/kubernetes-the-hard-way
{
openssl genrsa -out ca.key 4096
openssl req -x509 -new -sha512 -noenc \
-key ca.key -days 3653 \
-config ca.conf \
-out ca.crt
}
Command & Flag Breakdown:
genrsa -out ca.key 4096: Generates an unencrypted 4096-bit RSA private key (ca.key).req -x509: Outputs a self-signed root certificate (ca.crt) rather than a signing request.-sha512: Specifies SHA-512 hashing for signature security.-noenc: Prevents encrypting the private key with a passphrase, allowing unattended component startup.-days 3653: Sets certificate validity to 10 years.-config ca.conf: References OpenSSL presets for CA subject fields.
Step 2: Generate Component Certificates and Keys
Define the list of required cluster identities and loop through them to generate private keys, signing requests, and signed certificates:
certs=(
"admin" "node-0" "node-1"
"kube-proxy" "kube-scheduler"
"kube-controller-manager"
"kube-api-server"
"service-accounts"
)
for i in ${certs[*]}; do
openssl genrsa -out "${i}.key" 4096
openssl req -new -key "${i}.key" -sha256 \
-config "ca.conf" -section ${i} \
-out "${i}.csr"
openssl x509 -req -days 3653 -in "${i}.csr" \
-copy_extensions copyall \
-sha256 -CA "ca.crt" \
-CAkey "ca.key" \
-CAcreateserial \
-out "${i}.crt"
done
Critical Loop Execution Details:
-section ${i}: Directs OpenSSL to read specific sections fromca.conf(e.g.,[node-0],[kube-api-server]) to populate exactCN,O, and SAN settings.-copy_extensions copyall: Mandatory flag. Copies SAN extension fields defined in the CSR into the final signed.crtfile. Omitting this drops SANs and causes TLS connection failures.-CAcreateserial: Createsca.srlto track issued serial numbers and prevent certificate collision.
Step 3: Verify Local Certificate Output
List all generated PKI files on the jumpbox:
ls -1 *.crt *.key *.csr
Expected Terminal Output:
admin.crt
admin.csr
admin.key
ca.crt
ca.key
kube-api-server.crt
kube-api-server.csr
kube-api-server.key
kube-controller-manager.crt
kube-controller-manager.csr
kube-controller-manager.key
kube-proxy.crt
kube-proxy.csr
kube-proxy.key
kube-scheduler.crt
kube-scheduler.csr
kube-scheduler.key
node-0.crt
node-0.csr
node-0.key
node-1.crt
node-1.csr
node-1.key
service-accounts.crt
service-accounts.csr
service-accounts.key
Step 4: Distribute Certificates to Worker Nodes
Transfer ca.crt and node-specific keypairs to worker nodes node-0 (10.0.0.3) and node-1 (10.0.0.4), placing them directly in the default kubelet directory /var/lib/kubelet/:
for host in node-0 node-1; do
ssh root@${host} mkdir -p /var/lib/kubelet/
scp ca.crt root@${host}:/var/lib/kubelet/
scp ${host}.crt \
root@${host}:/var/lib/kubelet/kubelet.crt
scp ${host}.key \
root@${host}:/var/lib/kubelet/kubelet.key
done
Step 5: Distribute Certificates to Control Plane Server
Transfer control plane certificates to the server host (10.0.0.2):
scp \
ca.key ca.crt \
kube-api-server.key kube-api-server.crt \
service-accounts.key service-accounts.crt \
root@server:~/
4.3 Summary of Host Placement & Roles
| Node Host | IP Address | Target Directory | Files Copied | Functionality |
|---|---|---|---|---|
server |
10.0.0.2 |
/root/ |
ca.crt, ca.key, kube-api-server.crt, .key, service-accounts.crt, .key |
Serves API endpoint on port 6443, authenticates mTLS requests, and signs/verifies Service Account JWTs. |
node-0 |
10.0.0.3 |
/var/lib/kubelet/ |
ca.crt, kubelet.crt (renamed from node-0.crt), kubelet.key (renamed from node-0.key) |
Authenticates node-0 (CN=system:node:node-0, O=system:nodes) to kube-apiserver and secures local endpoint (10250). |
node-1 |
10.0.0.4 |
/var/lib/kubelet/ |
ca.crt, kubelet.crt (renamed from node-1.crt), kubelet.key (renamed from node-1.key) |
Authenticates node-1 (CN=system:node:node-1, O=system:nodes) to kube-apiserver and secures local endpoint (10250). |
jumpbox |
10.0.0.1 |
/root/kubernetes-the-hard-way/ |
admin.*, kube-proxy.*, kube-scheduler.*, kube-controller-manager.* |
Retains remaining client keypairs to generate .kubeconfig authentication files in the next lab. |
4.4 Real-World Troubleshooting & Pitfalls
Gotcha 1: Missing SAN Extensions in Signed Certificates
- Symptom: Clients or nodes fail to connect to
kube-apiserver, logging errors like:x509: certificate is valid for 127.0.0.1, not 10.0.0.2orx509: certificate relies on legacy Common Name field, not SANs. - Root Cause: Executing
openssl x509without-copy_extensions copyallcauses OpenSSL to strip the SAN fields defined in the CSR when emitting the signed.crt. - Fix: Always verify certificate SANs using OpenSSL before deployment, ensuring
-copy_extensions copyallwas passed during generation.
Gotcha 2: Incorrect CN or O Mappings Leading to RBAC Failures
- Symptom: Worker nodes join the cluster but show
403 Forbiddenerrors inkubelet.logwhen trying to post node status. - Root Cause: Typos in
ca.confunder section blocks (e.g., settingCN=node-0instead ofCN=system:node:node-0, or omittingO=system:nodes). - Fix: Ensure exact prefixing for system components as specified by Kubernetes Node Authorizer specs.
4.5 Verification & Validation
Inspect the generated certificates on the jumpbox using openssl x509 to confirm identity fields (CN, O) and Subject Alternative Names (SANs).
1. Verify Administrative Client Certificate (admin.crt)
Check that admin.crt carries superuser administrative privileges (O=system:masters):
openssl x509 -in admin.crt -text -noout | grep -E "Subject:"
Expected Terminal Output:
Subject: C=US, ST=Washington, L=Seattle, O=system:masters, OU=Kubernetes The Hard Way, CN=admin
2. Verify API Server Certificate SANs (kube-api-server.crt)
Check that kube-api-server.crt contains all necessary IP addresses and hostnames:
openssl x509 -in kube-api-server.crt -text -noout | grep -A 1 "Subject Alternative Name"
Expected Terminal Output:
X509v3 Subject Alternative Name:
DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, IP Address:10.0.0.2, IP Address:127.0.0.1
3. Verify Remote Distribution on Control Plane (server)
Execute a remote check on server (10.0.0.2) to ensure files arrived safely:
ssh root@server "ls -l /root/*.crt /root/*.key"
Expected Terminal Output:
-rw-r--r-- 1 root root 2098 Oct 31 10:50 /root/ca.crt
-rw------- 1 root root 3243 Oct 31 10:50 /root/ca.key
-rw-r--r-- 1 root root 2212 Oct 31 10:50 /root/kube-api-server.crt
-rw------- 1 root root 3243 Oct 31 10:50 /root/kube-api-server.key
-rw-r--r-- 1 root root 2086 Oct 31 10:50 /root/service-accounts.crt
-rw------- 1 root root 3243 Oct 31 10:50 /root/service-accounts.key
Actual Terminal Output run:
# 04 certificate authority
root@jumpbox:~/kubernetes-the-hard-way# certs=(
"admin" "node-0" "node-1"
"kube-proxy" "kube-scheduler"
"kube-controller-manager"
"kube-api-server"
"service-accounts"
)
root@jumpbox:~/kubernetes-the-hard-way# for i in ${certs[*]}; do
openssl genrsa -out "${i}.key" 4096
openssl req -new -key "${i}.key" -sha256 \
-config "ca.conf" -section ${i} \
-out "${i}.csr"
openssl x509 -req -days 3653 -in "${i}.csr" \
-copy_extensions copyall \
-sha256 -CA "ca.crt" \
-CAkey "ca.key" \
-CAcreateserial \
-out "${i}.crt"
done
Certificate request self-signature ok
subject=CN = admin, O = system:masters
Certificate request self-signature ok
subject=CN = system:node:node-0, O = system:nodes, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:node:node-1, O = system:nodes, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:kube-proxy, O = system:node-proxier, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:kube-scheduler, O = system:system:kube-scheduler, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = system:kube-controller-manager, O = system:kube-controller-manager, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = kubernetes, C = US, ST = Washington, L = Seattle
Certificate request self-signature ok
subject=CN = service-accounts
root@jumpbox:~/kubernetes-the-hard-way#
root@jumpbox:~/kubernetes-the-hard-way# ls -1 *.crt *.key *.csr
admin.crt
admin.csr
admin.key
ca.crt
ca.key
kube-api-server.crt
kube-api-server.csr
kube-api-server.key
kube-controller-manager.crt
kube-controller-manager.csr
kube-controller-manager.key
kube-proxy.crt
kube-proxy.csr
kube-proxy.key
kube-scheduler.crt
kube-scheduler.csr
kube-scheduler.key
node-0.crt
node-0.csr
node-0.key
node-1.crt
node-1.csr
node-1.key
service-accounts.crt
service-accounts.csr
service-accounts.key
root@jumpbox:~/kubernetes-the-hard-way# for host in node-0 node-1; do
ssh root@${host} mkdir /var/lib/kubelet/
scp ca.crt root@${host}:/var/lib/kubelet/
scp ${host}.crt \
root@${host}:/var/lib/kubelet/kubelet.crt
scp ${host}.key \
root@${host}:/var/lib/kubelet/kubelet.key
done
ca.crt 100% 1899 13.6MB/s 00:00
node-0.crt 100% 2147 13.0MB/s 00:00
node-0.key 100% 3272 14.7MB/s 00:00
ca.crt 100% 1899 12.9MB/s 00:00
node-1.crt 100% 2147 14.1MB/s 00:00
node-1.key 100% 3272 18.4MB/s 00:00
root@jumpbox:~/kubernetes-the-hard-way# scp \
ca.key ca.crt \
kube-api-server.key kube-api-server.crt \
service-accounts.key service-accounts.crt \
root@server:~/
ca.key 100% 3272 14.9MB/s 00:00
ca.crt 100% 1899 19.4MB/s 00:00
kube-api-server.key 100% 3272 26.7MB/s 00:00
kube-api-server.crt 100% 2354 23.4MB/s 00:00
service-accounts.key 100% 3268 37.2MB/s 00:00
service-accounts.crt 100% 2004 23.8MB/s 00:00
root@jumpbox:~/kubernetes-the-hard-way#
4.6 Section Summary
You have successfully provisioned a Public Key Infrastructure (PKI) for the cluster, issued X.509 client/server certificates mapped to Kubernetes RBAC identities, and securely distributed credentials to control plane and worker nodes. In the next section, you will use the remaining client certificates on the jumpbox to generate authentication .kubeconfig files for kubectl, kube-proxy, kube-scheduler, and kube-controller-manager.
5. Kubernetes Configuration Files (docs/05-kubernetes-configuration-files.md)
-
Kubeconfig Generation: Packaging API server endpoints and embedded TLS credentials using
kubectl config: -
node-0.kubeconfigandnode-1.kubeconfig(using Node Authorizer credentials) -
kube-proxy.kubeconfig -
kube-controller-manager.kubeconfig -
kube-scheduler.kubeconfig -
admin.kubeconfig -
File Distribution: Pushing generated
.kubeconfigfiles to their respective destinations on worker nodes and control plane controllers.
5.1 Section Overview & Context
In a Kubernetes cluster built from scratch, components do not communicate over open, unauthenticated channels. Every core daemon—from the kubelet agents running on worker nodes to the kube-scheduler on the control plane—must present a verified identity when making API requests to kube-apiserver.
While TLS certificates (generated in the previous lab) provide cryptographic proof of identity, Kubernetes components require a standardized configuration wrapper to locate the API server, verify its server certificate, and attach client credentials. This manifest is a kubeconfig file.
In this section, you will generate and distribute self-contained kubeconfig files for all core system components and administrative users across your 4-node Proxmox LXC infrastructure:
+-----------------------------+
| JUMPBOX |
| (10.0.0.1) |
| Generates ALL .kubeconfigs |
+--------------+--------------+
|
+-------------------+-------------------+
| SCP | SCP
v v
+--------------------------+ +--------------------------+
| SERVER | | NODE-0 / NODE-1 |
| (10.0.0.2) | | (10.0.0.3 / 10.0.0.4) |
| /root/ | | /var/lib/kubelet/ |
| - admin.kubeconfig | | - kubeconfig (node-X) |
| - kube-controller- | | /var/lib/kube-proxy/ |
| manager.kubeconfig | | - kubeconfig |
| - kube-scheduler. | +--------------------------+
| kubeconfig |
+--------------------------+
5.2 Technical Understanding & Deep Dive (The “Why”)
Anatomy of a kubeconfig File
A kubeconfig file is a structured YAML document defined by the client.authentication.k8s.io/v1 API schema. It organises cluster access around three distinct structures:
clusters: Defines where to connect (API Server URL endpoint, e.g.,[https://server.kubernetes.local:6443](https://server.kubernetes.local:6443)) and how to verify the server (the Root CA certificate).users: Defines who is connecting (client public certificate data and private key data).contexts: Binds a specific User identity to a specific Cluster profile under a designated context name (e.g.,default). Settingcurrent-contextinstructskubectlor system daemons which identity to present by default.
apiVersion: v1
kind: Config
preferences: {}
clusters:
- cluster:
certificate-authority-data: <BASE64_ENCODED_CA_CERT>
server: https://server.kubernetes.local:6443
name: kubernetes-the-hard-way
users:
- name: system:node:node-0
user:
client-certificate-data: <BASE64_ENCODED_CLIENT_CERT>
client-key-data: <BASE64_ENCODED_CLIENT_KEY>
contexts:
- context:
cluster: kubernetes-the-hard-way
user: system:node:node-0
name: default
current-context: default
The Four Key kubectl config Subcommands
Instead of manually editing YAML files—which is prone to indentation and syntax errors—you construct kubeconfig files using the kubectl config command utility:
kubectl config set-cluster: Appends or updates an entry in theclustersarray.kubectl config set-credentials: Appends or updates authentication credentials in theusersarray.kubectl config set-context: Defines a logical mapping rule in thecontextsarray.kubectl config use-context: Sets thecurrent-contextfield in the root of the configuration document.
Why Use --embed-certs=true?
By default, passing --certificate-authority=ca.crt or --client-certificate=node-0.crt to kubectl config stores local filesystem path references inside the generated file. If you move that .kubeconfig file to a target node without also copying the associated .crt and .key files to the exact same directory path, authentication fails.
Setting --embed-certs=true instructs kubectl to read the target certificate and key files, base64-encode their raw contents, and write them directly into the YAML parameters:
certificate-authority-dataclient-certificate-dataclient-key-data
This renders every generated .kubeconfig file 100% standalone, self-contained, and portable.
Identity, RBAC, and the Node Authorizer
Kubernetes lacks a database of static user accounts. When a client presents a TLS certificate to kube-apiserver, the API server decrypts the client certificate and inspects its X.509 fields:
- Subject Common Name (
CN): Interpreted as the User. - Organization (
O): Interpreted as the Group(s).
The API server then passes these values to its configured authorization modules:
- Worker Nodes (
node-0.kubeconfig,node-1.kubeconfig): Signed withCN=system:node:node-0andO=system:nodes. Requests containing this credential are processed by the Node Authorizer. This specialized authorizer strictly enforces least privilege:node-0can only read/write Pod specs, secrets, and status updates assigned tonode-0. It is blocked from accessing resources belonging tonode-1. - System Components (
kube-proxy,kube-controller-manager,kube-scheduler): Configured with dedicated system users (CN=system:kube-proxy,CN=system:kube-scheduler). These match pre-defined cluster Role-Based Access Control (RBAC) system bindings (cluster-admin,system:kube-scheduler, etc.). - Admin User (
admin.kubeconfig): Signed withCN=adminandO=system:masters. Thesystem:mastersorganization group bypasses RBAC checks, granting full root-level control over the cluster.
Target Endpoint Differences: Public Domain vs. Local Loopback
Notice the difference in API server endpoints across the generated files:
- Component Configs (
server.kubernetes.local:6443): System daemons running across different nodes (node-0,node-1) communicate withkube-apiserverover the internal network using the cluster’s domain name or control plane IP address. - Admin Config (
127.0.0.1:6443): Administrative operations usingadmin.kubeconfigare typically performed directly on the control plane server host (serverat10.0.0.2) or via SSH tunnels. Pointing to127.0.0.1ensures administrative access remains local to the control plane, bypassing external ingress routes and network dependency failures.
5.3 Step-by-Step Implementation & Command Breakdown
All commands in this section must be executed from your jumpbox (10.0.0.1) inside the working directory where your CA and TLS certificates were generated (/root/kubernetes-the-hard-way).
Step 1: Generate Kubelet Configuration Files for Worker Nodes
Run the following loop to construct node-0.kubeconfig and node-1.kubeconfig:
for host in node-0 node-1; do
kubectl config set-cluster kubernetes-the-hard-way \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://server.kubernetes.local:6443 \
--kubeconfig=${host}.kubeconfig
kubectl config set-credentials system:node:${host} \
--client-certificate=${host}.crt \
--client-key=${host}.key \
--embed-certs=true \
--kubeconfig=${host}.kubeconfig
kubectl config set-context default \
--cluster=kubernetes-the-hard-way \
--user=system:node:${host} \
--kubeconfig=${host}.kubeconfig
kubectl config use-context default \
--kubeconfig=${host}.kubeconfig
done
Command Breakdown:
--server=[https://server.kubernetes.local:6443](https://server.kubernetes.local:6443): Directs worker node traffic to the control plane API server.--client-certificate=${host}.crt: Uses the node-specific TLS certificate (CN=system:node:node-0).--kubeconfig=${host}.kubeconfig: Writes the generated configuration into an isolated, node-specific file.
Step 2: Generate the kube-proxy Configuration File
Generate the configuration file for the network proxy daemon running on worker nodes:
{
kubectl config set-cluster kubernetes-the-hard-way \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://server.kubernetes.local:6443 \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-credentials system:kube-proxy \
--client-certificate=kube-proxy.crt \
--client-key=kube-proxy.key \
--embed-certs=true \
--kubeconfig=kube-proxy.kubeconfig
kubectl config set-context default \
--cluster=kubernetes-the-hard-way \
--user=system:kube-proxy \
--kubeconfig=kube-proxy.kubeconfig
kubectl config use-context default \
--kubeconfig=kube-proxy.kubeconfig
}
Step 3: Generate the kube-controller-manager Configuration File
Generate the configuration file for the background controller loops running on the control plane:
{
kubectl config set-cluster kubernetes-the-hard-way \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://server.kubernetes.local:6443 \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-credentials system:kube-controller-manager \
--client-certificate=kube-controller-manager.crt \
--client-key=kube-controller-manager.key \
--embed-certs=true \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config set-context default \
--cluster=kubernetes-the-hard-way \
--user=system:kube-controller-manager \
--kubeconfig=kube-controller-manager.kubeconfig
kubectl config use-context default \
--kubeconfig=kube-controller-manager.kubeconfig
}
Step 4: Generate the kube-scheduler Configuration File
Generate the configuration file for the cluster workload scheduler:
{
kubectl config set-cluster kubernetes-the-hard-way \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://server.kubernetes.local:6443 \
--kubeconfig=kube-scheduler.kubeconfig
kubectl config set-credentials system:kube-scheduler \
--client-certificate=kube-scheduler.crt \
--client-key=kube-scheduler.key \
--embed-certs=true \
--kubeconfig=kube-scheduler.kubeconfig
kubectl config set-context default \
--cluster=kubernetes-the-hard-way \
--user=system:kube-scheduler \
--kubeconfig=kube-scheduler.kubeconfig
kubectl config use-context default \
--kubeconfig=kube-scheduler.kubeconfig
}
Step 5: Generate the admin Configuration File
Generate the superuser administrative configuration file for cluster management:
{
kubectl config set-cluster kubernetes-the-hard-way \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://127.0.0.1:6443 \
--kubeconfig=admin.kubeconfig
kubectl config set-credentials admin \
--client-certificate=admin.crt \
--client-key=admin.key \
--embed-certs=true \
--kubeconfig=admin.kubeconfig
kubectl config set-context default \
--cluster=kubernetes-the-hard-way \
--user=admin \
--kubeconfig=admin.kubeconfig
kubectl config use-context default \
--kubeconfig=admin.kubeconfig
}
Step 6: Distribute Configuration Files
With all six kubeconfig files generated on the jumpbox, distribute them to their respective destinations using scp.
- Distribute to Worker Nodes (
node-0at10.0.0.3andnode-1at10.0.0.4):
for host in node-0 node-1; do
ssh root@${host} "mkdir -p /var/lib/{kube-proxy,kubelet}"
scp kube-proxy.kubeconfig \
root@${host}:/var/lib/kube-proxy/kubeconfig
scp ${host}.kubeconfig \
root@${host}:/var/lib/kubelet/kubeconfig
done
- Distribute to Control Plane Server (
serverat10.0.0.2):
scp admin.kubeconfig \
kube-controller-manager.kubeconfig \
kube-scheduler.kubeconfig \
root@server:~/
File Placement Matrix
| Target Host | Host IP | Target Path | Transferred File | Renamed To | Executing Service |
|---|---|---|---|---|---|
node-0 |
10.0.0.3 |
/var/lib/kubelet/ |
node-0.kubeconfig |
kubeconfig |
kubelet |
node-0 |
10.0.0.3 |
/var/lib/kube-proxy/ |
kube-proxy.kubeconfig |
kubeconfig |
kube-proxy |
node-1 |
10.0.0.4 |
/var/lib/kubelet/ |
node-1.kubeconfig |
kubeconfig |
kubelet |
node-1 |
10.0.0.4 |
/var/lib/kube-proxy/ |
kube-proxy.kubeconfig |
kubeconfig |
kube-proxy |
server |
10.0.0.2 |
/root/ |
admin.kubeconfig |
admin.kubeconfig |
Administrative CLI (kubectl) |
server |
10.0.0.2 |
/root/ |
kube-controller-manager.kubeconfig |
kube-controller-manager.kubeconfig |
kube-controller-manager |
server |
10.0.0.2 |
/root/ |
kube-scheduler.kubeconfig |
kube-scheduler.kubeconfig |
kube-scheduler |
5.4 Real-World Troubleshooting & Pitfalls
Gotcha 1: Omission of --embed-certs=true
- Symptom: Services start up on worker nodes (
node-0,node-1), but instantly fail during API handshake with file errors:error: failed to load kubeconfig: open ca.crt: no such file or directory - Root Cause: The
kubeconfigwas created without--embed-certs=true. It contains path references likecertificate-authority: ca.crtinstead of raw base64 data undercertificate-authority-data. - Fix: Regenerate all
kubeconfigfiles using the--embed-certs=trueflag before distribution.
Gotcha 2: Node Authorizer Rejection (403 Forbidden)
- Symptom:
kubeletfails to register withkube-apiserver, logging:Node "node-0" is unauthorized to modify node "node-0" - Root Cause: The user identity configured in
set-credentialsdid not strictly follow the formatsystem:node:<node-name>during certificate generation, or the context bound the wrong username. - Fix: Verify that the certificate Common Name (
CN) matchessystem:node:node-0and that the credential set inset-contextreferences this exact user string.
Gotcha 3: Secure Permission Denied on Target Directories
- Symptom:
scptransfers fail withNo such file or directoryorPermission denied. - Root Cause: Destination system paths (
/var/lib/kubelet,/var/lib/kube-proxy) do not exist on clean base OS installations. - Fix: Always ensure target directories are created beforehand using
ssh root@${host} "mkdir -p /var/lib/{kube-proxy,kubelet}".
Verification & Validation
1. Confirm Local Generation on Jumpbox
Verify that all six .kubeconfig files were generated successfully in your working directory:
ls -1 *.kubeconfig
Expected Output:
admin.kubeconfig
kube-controller-manager.kubeconfig
kube-proxy.kubeconfig
kube-scheduler.kubeconfig
node-0.kubeconfig
node-1.kubeconfig
2. Validate Embedded Data Integrity
Inspect the structure of a generated file (e.g., admin.kubeconfig) to confirm that certificates are embedded as raw base64 data rather than path references:
kubectl config view --kubeconfig=admin.kubeconfig
Expected Output:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: DATA+OMITTED
server: https://127.0.0.1:6443
name: kubernetes-the-hard-way
contexts:
- context:
cluster: kubernetes-the-hard-way
user: admin
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: admin
user:
client-certificate-data: DATA+OMITTED
client-key-data: DATA+OMITTED
3. Verify Remote File Distribution
Verify that files are in place on worker nodes (node-0 and node-1) and the control plane server (server):
ssh root@node-0 "ls -la /var/lib/kubelet/kubeconfig /var/lib/kube-proxy/kubeconfig"
ssh root@server "ls -la ~/admin.kubeconfig ~/kube-controller-manager.kubeconfig ~/kube-scheduler.kubeconfig"
Expected Output:
-rw------- 1 root root 6240 Oct 31 10:57 /var/lib/kube-proxy/kubeconfig
-rw------- 1 root root 6284 Oct 31 10:57 /var/lib/kubelet/kubeconfig
-rw------- 1 root root 6210 Oct 31 10:57 /root/admin.kubeconfig
-rw------- 1 root root 6320 Oct 31 10:57 /root/kube-controller-manager.kubeconfig
-rw------- 1 root root 6280 Oct 31 10:57 /root/kube-scheduler.kubeconfig
5.5 Section Summary
You have successfully generated self-contained Kubernetes configuration files for every system component and distributed them across your control plane and worker nodes. Each daemon now possesses a dedicated, portable identity manifest configured for mutual TLS authentication and strict role-based access control.
Next, you will generate the cluster-wide data encryption configuration and key to secure secret data at rest in etcd.
6. Data Encryption at Rest (docs/06-data-encryption-keys.md)
- Encryption Key Generation: Extracting a 32-byte Base64 key from
/dev/urandom. - Config Creation: Generating
encryption-config.yamlwith theaescbcprovider specification to secure Kubernetes Secrets insideetcd.
5.1 Section Overview & Context
In a production-grade Kubernetes cluster, securing persistent state is just as critical as securing network traffic. Kubernetes relies on etcd as its primary key-value datastore for all cluster state, configurations, and sensitive credentials like Secret objects.
By default, etcd stores data as unencrypted plaintext. If an attacker gains unauthorized access to etcd backups, storage snapshots, or the underlying filesystem on the control plane host (server at 10.0.0.2), they can read raw authentication tokens, private keys, and API credentials.
+-----------------------------------------------------------------------------------+
| JUMPBOX (10.0.0.1) |
| 1. Generate 256-bit Key --> 2. Render Config --> 3. SCP to Control Plane |
+-----------------------------------------------------------------------------------+
|
v
+-----------------------------------------------------------------------------------+
| CONTROL PLANE (10.0.0.2) |
| /var/lib/kubernetes/encryption-config.yaml |
| |
| +-------------------+ Encrypts +-----------------------------+ |
| | kube-apiserver | --------------------> | etcd (/var/lib/etcd) | |
| | (v1.32) | <-------------------- | (Ciphertext stored on disk) | |
| +-------------------+ Decrypts +-----------------------------+ |
+-----------------------------------------------------------------------------------+
In this step, you will generate a cryptographically secure 256-bit symmetric key on the jumpbox (10.0.0.1) and render an EncryptionConfiguration manifest. This manifest instructs kube-apiserver (v1.32) to transparently encrypt Secret resources in-flight using AES-CBC mode before persisting them into etcd.
5.2 Technical Understanding & Deep Dive (The “Why”)
Base64 Encoding vs. Cryptographic Encryption
A common misconception among Kubernetes operators is that Kubernetes Secret objects are encrypted by default because their values appear as encoded strings (e.g., dGVzdA==) when queried with kubectl get secret -o yaml.
Base64 is an encoding scheme designed for safe data transport over text protocols—it provides zero confidentiality. Anyone with read access to the cluster API or etcd can instantly decode Base64 strings using standard tools (echo "dGVzdA==" | base64 -d). True Encryption at Rest requires transforming plaintext data into unreadable ciphertext using a secret key and a cryptographic algorithm.
In-Flight Envelope Encryption via kube-apiserver
Kubernetes handles encryption at the API layer through kube-apiserver:
- Write Path: When a client posts a
Secretpayload tokube-apiserver, the API server intercepts the request, reads its activeEncryptionConfiguration, encrypts the payload in memory using the defined key, and writes the resulting ciphertext toetcd. - Read Path: When an authorized client requests a
Secret,kube-apiserverreads the ciphertext frometcd, decrypts it using the matching key, and returns the payload to the authenticated client over TLS.
Deconstructing encryption-config.yaml
The EncryptionConfiguration structure defines which resources are protected and which encryption algorithms (providers) are applied:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: <BASE64_ENCRYPTION_KEY>
- identity: {}
resources: Specifies the array of Kubernetes API resources to encrypt. Here, onlysecretsare targeted.providers: An ordered list of encryption providers evaluated sequentially:aescbc(AES-CBC with PKCS#7 padding): Placed first. All new write operations toetcdare encrypted using AES in Cipher Block Chaining mode withkey1.identity: {}: Placed second. Represents plaintext (no encryption). This allowskube-apiserverto read existing, unencrypted secrets stored inetcdprior to enabling encryption. Upon updating an unencrypted secret,kube-apiserverautomatically rewrites it using the primary provider (aescbc).
5.3 Step-by-Step Implementation & Command Breakdown
Perform all commands from the repository root directory on your jumpbox (10.0.0.1).
Step 1: Generate the Symmetric Encryption Key
Generate a 32-byte (256-bit) cryptographically secure random key and store it in an environment variable:
cd /root/kubernetes-the-hard-way
export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
Command Breakdown:
/dev/urandom: The Linux kernel entropy device providing non-blocking pseudo-random bytes.head -c 32: Extracts exactly 32 bytes (256 bits) from the entropy stream.| base64: Pipes the raw binary data intobase64to encode it as a 44-character ASCII string compatible with YAML configuration files.export ENCRYPTION_KEY=...: Assigns the value to the shell environment for tool substitution.
Step 2: Render encryption-config.yaml
Substitute the environment variable into the configuration template using envsubst:
envsubst < configs/encryption-config.yaml > encryption-config.yaml
Command Breakdown:
envsubst: Scans the input stream for references like$ENCRYPTION_KEYand replaces them with the actual environment string.< configs/encryption-config.yaml: Standard input redirection feeding the template file.> encryption-config.yaml: Standard output redirection writing the rendered manifest to disk.
Step 3: Inspect the Rendered Configuration
Verify that the template populated correctly with your Base64 string:
cat encryption-config.yaml
Realistic Terminal Output:
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: 4Xz8vK9mP2qR5wT8yU1vC3bN6mQ9xZ2aL5pK8jH3fG4=
- identity: {}
Step 4: Transfer the Manifest to the Control Plane
Copy the configuration file over SSH to the control plane host (server at 10.0.0.2):
scp encryption-config.yaml root@server:~/
Realistic Terminal Output:
encryption-config.yaml 100% 243 187.4KB/s 00:00
Placement & Component Matrix
| Host Machine | File Transferred | Destination Path | Target Component | Operational Purpose |
|---|---|---|---|---|
server (10.0.0.2) |
encryption-config.yaml |
/root/ (staged for /var/lib/kubernetes/) |
kube-apiserver |
Instructs API server to encrypt/decrypt Secret payloads stored in etcd. |
5.4 Real-World Troubleshooting & Pitfalls
Pitfall 1: envsubst: command not found
- Symptom: Executing
envsubstfails withbash: envsubst: command not found. - Root Cause: The GNU
gettext-baseutility package is not installed on thejumpboxLXC container. - Fix: Install the utility package via
apt:
apt-get update && apt-get install -y gettext-base
Pitfall 2: Blank secret: Field or Invalid Key Length
- Symptom: Inspecting
encryption-config.yamlshowssecret:(empty) or a key shorter than 44 characters. - Root Cause: Using
head -n 32instead ofhead -c 32. The-nflag attempts to read 32 lines instead of 32 bytes, which fails or produces wrong byte counts when handling raw binary data from/dev/urandom. - Fix: Re-export the key using
-c 32explicitly and re-render:
export ENCRYPTION_KEY=$(head -c 32 /dev/urandom | base64)
envsubst < configs/encryption-config.yaml > encryption-config.yaml
Pitfall 3: SCP Permission Denied on Host Transfer
- Symptom:
scp encryption-config.yaml root@server:~/' returnsPermission denied (publickey)`. - Root Cause: Missing SSH key setup between
jumpboxandserveror improper permissions on/root/.ssh/authorized_keysonserver. - Fix: Verify connectivity and key propagation:
ssh root@server "mkdir -p ~/.ssh && chmod 700 ~/.ssh"
scp encryption-config.yaml root@server:~/
5.5 Verification & Validation
Verify that the key generation and configuration deployment were executed accurately before proceeding to etcd bootstrapping.
1. Validate Secret String Character Count
A valid 256-bit Base64-encoded key must be exactly 44 characters long (including trailing = padding characters):
grep "secret:" encryption-config.yaml | awk '{print $2}' | wc -c
Expected Output:
45
(Note: wc -c counts 44 ASCII characters plus 1 trailing newline character).
2. Confirm File Presence on Control Plane
Query server (10.0.0.2) remotely via SSH to ensure the manifest was delivered cleanly:
ssh root@server "ls -la ~/encryption-config.yaml"
Expected Output:
-rw-r--r-- 1 root root 243 Oct 31 11:18 /root/encryption-config.yaml
5.6 Section Summary
You have generated a symmetric 256-bit encryption key and staged the EncryptionConfiguration manifest on the control plane host (server). This ensures kube-apiserver is equipped to secure Kubernetes secrets at rest once the control plane services are started.
7. Bootstrapping etcd (docs/07-bootstrapping-etcd.md)
- Etcd Installation: Installing
etcdandetcdctlbinaries on theservernode. - Configuration & Systemd: Configuring
/etc/etcd/certificates, deployingetcd.service, and starting the systemd unit. - Cluster Verification: Confirming cluster member health and status using
etcdctl member list.
Kubernetes components are entirely stateless. They do not store cluster state in local memory or on container disk images. Instead, all persistent state—every Pod definition, Service specification, Secret, ConfigMap, and Role-Based Access Control (RBAC) policy—is stored in etcd.
In this section, you will bootstrap a single-node etcd database cluster on your control plane server (server at 10.0.0.2).
+-----------------------------------------------------------------------+
| CONTROL PLANE (server) |
| |
| +---------------------+ +-----------------------+ |
| | kube-apiserver | <-- mTLS (2379) | etcd | |
| | (Stateless Gateway) | | (State Storage Engine| |
| +---------------------+ +-----------------------+ |
| | |
| WAL / Snapshot DB |
| (/var/lib/etcd: 0700) |
+-----------------------------------------------------------------------+
Bootstrapping etcd involves provisioning the executable binaries, configuring Mutual TLS (mTLS) authentication certificates, defining a dedicated systemd unit file, and hardening the persistent storage directory on disk.
7.1 Technical Understanding & Deep Dive: The “Why”
1. Raft Consensus and Persistent Data Storage
etcd is a strongly consistent, distributed key-value store implementing the Raft consensus algorithm. Raft ensures that all writes are committed to a Write-Ahead Log (WAL) before updating the main database file (db) located in /var/lib/etcd.
Even in a single-node setup, etcd maintains strict transaction logs to guarantee data durability across system reboots. Because etcd holds sensitive cluster secrets in plaintext or encrypted formats, directory permissions must be restricted to 0700 (rwx------), ensuring only the root user or dedicated etcd service accounts can inspect raw database files.
2. The Isolation of kube-apiserver
In a healthy Kubernetes architecture, no component communicates with etcd directly except the kube-apiserver. Controller managers, schedulers, and worker node kubelet daemons all direct their read and write requests to kube-apiserver, which acts as a secure, validating gateway in front of etcd.
3. Network Ports and Dual-Listener Architecture
etcd utilizes two distinct networking ports:
- Port
2379(Client Traffic): Used for client-to-etcd communication (e.g.,kube-apiserverquerying state oretcdctlperforming administrative tasks). - Port
2380(Peer Traffic): Used exclusively for node-to-node replication, leader election, and Raft consensus synchronization across cluster peers.
4. PKI and Mutual TLS (mTLS) Authentication
To prevent unauthorized access, etcd enforces mTLS on both client and peer interfaces. Under mTLS:
- The server presents its certificate (
kube-api-server.crt) to prove its identity to the client. - The client (
kube-apiserveroretcdctl) presents its own client certificate signed by the Certificate Authority (ca.crt). - Connection is granted only if both parties successfully validate each other’s certificates against the shared CA.
7.2 Step-by-Step Implementation & Command Breakdown
Step 1: Copy Binaries and Configuration to the Control Plane
From your management host (jumpbox at 10.0.0.1), transfer the pre-downloaded binaries and systemd service template to the server host (10.0.0.2):
cd /root/kubernetes-the-hard-way
scp \
downloads/controller/etcd \
downloads/client/etcdctl \
units/etcd.service \
root@server:~/
Next, establish an SSH session into the server node:
ssh root@server
Step 2: Install Binaries into the System Path
On server, place the etcd daemon and etcdctl CLI tool into the system executable path:
mv etcd etcdctl /usr/local/bin/
/usr/local/bin/: Standard directory for custom system binaries included in system-wide$PATH, enabling execution without requiring relative file paths.
Step 3: Configure Directories, Permissions, and TLS Certificates
Create the configuration and data storage directories, apply strict POSIX file permissions, and copy the required TLS credentials:
mkdir -p /etc/etcd /var/lib/etcd
chmod 700 /var/lib/etcd
cp ca.crt kube-api-server.key kube-api-server.crt /etc/etcd/
/etc/etcd/: Holds configuration files, root CA certificates, and private keypairs./var/lib/etcd: The physical storage path for WAL and database snapshot files.chmod 700 /var/lib/etcd: Grants read, write, and execute permissions strictly toroot(rwx------), shielding cluster state from unprivileged system users.
Step 4: Provision the systemd Unit File
Move the etcd.service configuration into the system manager path:
mv etcd.service /etc/systemd/system/
Inspect the parameter flags configured inside /etc/systemd/system/etcd.service:
[Unit]
Description=etcd
Documentation=https://github.com/etcd-io/etcd
[Service]
Type=notify
ExecStart=/usr/local/bin/etcd \
--name server \
--cert-file=/etc/etcd/kube-api-server.crt \
--key-file=/etc/etcd/kube-api-server.key \
--peer-cert-file=/etc/etcd/kube-api-server.crt \
--peer-key-file=/etc/etcd/kube-api-server.key \
--trusted-ca-file=/etc/etcd/ca.crt \
--peer-trusted-ca-file=/etc/etcd/ca.crt \
--peer-client-cert-auth \
--client-cert-auth \
--initial-advertise-peer-urls https://10.0.0.2:2380 \
--listen-peer-urls https://10.0.0.2:2380 \
--listen-client-urls https://127.0.0.1:2379,https://10.0.0.2:2379 \
--advertise-client-urls https://10.0.0.2:2379 \
--initial-cluster-token etcd-cluster-token \
--initial-cluster server=https://10.0.0.2:2380 \
--initial-cluster-state new \
--data-dir=/var/lib/etcd
Restart=on-failure
RestartSec=5
[Install]
WantedBy=multi-user.target
Unit Configuration Breakdown
| Flag / Parameter | Technical Purpose |
|---|---|
--name server |
Assigns a unique node identifier within the etcd cluster topology. |
**--cert-file / --key-file** |
Server TLS certificate and private key used to secure client connections on port 2379. |
--trusted-ca-file |
Root Certificate Authority used to authenticate incoming client TLS connections. |
--client-cert-auth |
Enforces Mutual TLS (mTLS); requires clients to present a valid TLS certificate. |
**--peer-cert-file / --peer-key-file** |
TLS certificate pair used for peer-to-peer cluster consensus on port 2380. |
--listen-client-urls |
Network sockets etcd binds to for client traffic (127.0.0.1:2379 and 10.0.0.2:2379). |
--advertise-client-urls |
Endpoint address published to cluster clients for establishing API connections. |
--listen-peer-urls |
Network socket bound for internal Raft consensus traffic between peer nodes (10.0.0.2:2380). |
--initial-cluster |
Bootstrap configuration defining initial cluster members and peer URLs. |
--data-dir |
Path to the persistent directory where WAL and database state files are stored. |
Step 5: Start and Enable the etcd Service
Reload the systemd daemon to pick up the new unit configuration, enable automatic launch at boot, and start etcd:
systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
7.3 Real-World Troubleshooting & Pitfalls
The Error Symptom
Executing etcdctl member list with TLS flags fails with an authentication error:
rpc error: code = DeadlineExceeded desc = context deadline exceeded; ... authentication handshake failed: EOF
Conversely, running an unencrypted etcdctl member list without flags unexpectedly succeeds, displaying plaintext endpoints:
6702b0a34e2cfd39, started, controller, http://127.0.0.1:2380, http://127.0.0.1:2379, false
Underlying Root Cause
This issue occurs due to a Protocol Mismatch (HTTP vs. HTTPS/TLS) between the running etcd daemon and the client:
- The
etcd.serviceunit was executed without TLS flags or was configured with plaintexthttp://listeners. - When
etcdctlinitiates an encrypted TLS handshake packet (TLS Client Hello) against[https://127.0.0.1:2379](https://127.0.0.1:2379), the plaintext HTTP listener onetcdfails to decode the encrypted binary stream. - The server immediately closes the underlying TCP socket connection without returning a TLS alert, causing
etcdctlto report an unexpectedEOF(End of File).
Why This Matters for Kubernetes
The kube-apiserver mandates strict mTLS encryption when communicating with etcd. If etcd runs in unencrypted HTTP mode, kube-apiserver will fail to establish a database connection, halting control plane initialization in subsequent labs.
The Fix
Verify and update /etc/systemd/system/etcd.service to ensure all listener and advertise URLs explicitly specify the https:// protocol, and that certificate flags are set. Then reload systemd and restart the service:
# Apply updated systemd parameters
systemctl daemon-reload
systemctl restart etcd
Verification & Validation
1. Inspect Systemd Service Health
Verify that the etcd daemon process is active and running cleanly:
systemctl status etcd --no-pager
Expected Healthy Output:
● etcd.service - etcd
Loaded: loaded (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2026-10-31 12:15:00 UTC; 10s ago
Docs: https://github.com/etcd-io/etcd
Main PID: 1420 (etcd)
Tasks: 8 (limit: 4652)
Memory: 24.1M
CGroup: /system.slice/etcd.service
└─1420 /usr/local/bin/etcd --name server ...
2. Verify Cluster Membership over mTLS
Execute etcdctl member list, providing the root CA and API server certificate pair to validate client-side mTLS communication:
etcdctl member list \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/etcd/ca.crt \
--cert=/etc/etcd/kube-api-server.crt \
--key=/etc/etcd/kube-api-server.key
Expected Healthy Output:
6702b0a34e2cfd39, started, server, https://10.0.0.2:2380, https://10.0.0.2:2379, false
The output confirms that member 6702b0a34e2cfd39 named server is active, listening on secure https:// endpoints, and ready to accept control plane transactions.
8. Bootstrapping Control Plane Controllers (docs/08-bootstrapping-kubernetes-controllers.md)
-
Control Plane Binary Setup: Installing
kube-apiserver,kube-controller-manager,kube-scheduler, andkubectlonserver. -
Service Configuration:
-
Deploying
kube-apiserver.servicewith encryption providers and PKI flags. -
Configuring
kube-controller-manager.serviceandkube-scheduler.servicewith dedicated kubeconfigs. -
RBAC Authorization: Applying
kube-apiserver-to-kubelet.yamlto grant the API server access to worker node Kubelet APIs. -
Control Plane Verification: Validating API server initialization via
kubectl cluster-infoand HTTPS version endpoints.
8.1 Section Overview & Context
In this lab, you will assemble the core control plane of your Kubernetes cluster on the dedicated server host (10.0.0.2). Having previously initialized etcd as your distributed storage backend, you are now ready to deploy the three core management daemons that transform a single database into a functioning, self-healing container orchestrator:
kube-apiserver: The central administrative hub and sole gatekeeper toetcd.kube-controller-manager: The background engine executing continuous state reconciliation loops.kube-scheduler: The decision engine evaluating resource availability and binding workloads to worker nodes.
8.2 Technical Understanding & Deep Dive (The “Why”)
Before launching systemd services, it is critical to understand how these control plane components interact securely and why specific flags and configurations are necessary.
The Kubernetes Control Plane Architecture
The control plane components form a Hub-and-Spoke topology centered around kube-apiserver:
+--------------------------------------------------+
| CONTROL PLANE NODE (server) |
| 10.0.0.2 |
| |
kubectl (admin) ------------>+--> [ kube-apiserver ] <---> [ etcd (Port 2379) ] |
| ^ ^ |
| | | |
| v v |
| [ controller ] [ scheduler ] |
+----------+---------------------------------------+
|
| (HTTPS / Port 10250)
v
[ worker nodes (node-0, node-1) ]
Deep Dive: Component Flags & Configurations
kube-apiserver Security & Authorization
The API Server serves as the single entry point for all cluster operations. It must validate requests, enforce authorization, sign service account tokens, and secure state storage:
--authorization-mode=Node,RBAC: Enables two authorization layers. Node Authorizer specifically restrictskubeletAPI requests so worker nodes can only read/modify their own resources. RBAC handles role-based access control for human users, automated scripts, and in-cluster ServiceAccounts.--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,ResourceQuota: Admission controllers act as inline interceptors after authentication and authorization. For example,NodeRestrictionstops a compromisedkubeletfrom altering another node’s labels, taints, or pod statuses.--encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml: Ensures that raw KubernetesSecretdata is encrypted using AES-CBC key encryption before being written toetcd, guarding against plain-text database compromises.- Service Account JWT Keys:
--service-account-signing-key-file(private key) signs JSON Web Tokens issued to Pods, while--service-account-key-file(public key) allowskube-apiserverto verify incoming tokens.
kube-controller-manager State Management
The Controller Manager continuously compares the cluster’s actual state against the desired state stored in etcd:
--cluster-cidr=10.200.0.0/16: Reserves the master IP pool for all Pods across the entire cluster.--allocate-node-cidrs=true: Tells the controller manager to automatically slice the/16cluster range into distinct/24subnets for each worker node (e.g.,node-0receives10.200.0.0/24,node-1receives10.200.1.0/24).- **
--cluster-signing-cert-file&--cluster-signing-key-file**: Used by the built-in CertificateSigningRequest (CSR) controller to automatically issue and renew TLS certificates for worker nodes.
kube-scheduler Declarative Configuration
Unlike older Kubernetes releases that used CLI flags, modern versions (such as Kubernetes v1.32) configure kube-scheduler using a declarative KubeSchedulerConfiguration file (kube-scheduler.yaml):
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
clientConnection:
kubeconfig: "/var/lib/kubernetes/kube-scheduler.kubeconfig"
leaderElection:
leaderElect: true
Using a YAML configuration allows for modular plugin configuration (filtering, scoring, reserving) while referencing kube-scheduler.kubeconfig to authenticate back to kube-apiserver over TLS.
API Server-to-Kubelet RBAC Authorization
When an administrator runs kubectl exec, kubectl logs, or kubectl top, the request flows through kube-apiserver directly to the kubelet running on a worker node:
$$\text{kubectl} \xrightarrow{\text{HTTPS :6443}} \text{kube-apiserver} \xrightarrow{\text{HTTPS :10250}} \text{kubelet}$$
Because kubelet operates with --authorization-mode=Webhook, it asks kube-apiserver to authorize incoming requests. The system:kube-apiserver-to-kubelet ClusterRole explicitly grants the kubernetes certificate common name (CN) permission to access kubelet endpoints (/exec, /log, /metrics, /stats).
8.3 Step-by-Step Implementation & Command Breakdown
Step 1: Copy Binaries and Units from jumpbox
Execute from /root/kubernetes-the-hard-way on the jumpbox (10.0.0.1) machine to transfer all necessary artifacts to server (10.0.0.2):
cd /root/kubernetes-the-hard-way
scp \
downloads/controller/kube-apiserver \
downloads/controller/kube-controller-manager \
downloads/controller/kube-scheduler \
downloads/client/kubectl \
units/kube-apiserver.service \
units/kube-controller-manager.service \
units/kube-scheduler.service \
configs/kube-scheduler.yaml \
configs/kube-apiserver-to-kubelet.yaml \
root@server:~/
Step 2: Organize Binaries, Certificates, and Service Units on server
SSH into the server machine:
ssh root@server
Create the standard directory structures and move files to their designated system locations:
# 1. Create configuration and state directories
mkdir -p /etc/kubernetes/config /var/lib/kubernetes/
# 2. Install executables into system execution path
mv kube-apiserver kube-controller-manager kube-scheduler kubectl /usr/local/bin/
# 3. Move TLS certificates, keys, and encryption settings into /var/lib/kubernetes/
mv ca.crt ca.key \
kube-api-server.key kube-api-server.crt \
service-accounts.key service-accounts.crt \
encryption-config.yaml \
/var/lib/kubernetes/
# 4. Configure kube-apiserver systemd service
mv kube-apiserver.service /etc/systemd/system/
# 5. Configure kube-controller-manager kubeconfig and unit
mv kube-controller-manager.kubeconfig /var/lib/kubernetes/
mv kube-controller-manager.service /etc/systemd/system/
# 6. Configure kube-scheduler config and unit
mv kube-scheduler.kubeconfig /var/lib/kubernetes/
mv kube-scheduler.yaml /etc/kubernetes/config/
mv kube-scheduler.service /etc/systemd/system/
File Placement Reference Matrix
| Host Machine | Placement Directory | File Name(s) | Function / Serving Port |
|---|---|---|---|
server |
/usr/local/bin/ |
kube-apiserver, kube-controller-manager, kube-scheduler, kubectl |
Executable control plane binaries |
server |
/var/lib/kubernetes/ |
ca.crt, ca.key, kube-api-server.crt, .key, service-accounts.crt, .key, encryption-config.yaml |
TLS credentials, JWT signing keys, and data encryption settings |
server |
/var/lib/kubernetes/ |
kube-controller-manager.kubeconfig, kube-scheduler.kubeconfig |
Local component mTLS authentication files |
server |
/etc/kubernetes/config/ |
kube-scheduler.yaml |
KubeSchedulerConfiguration declarative settings |
server |
/etc/systemd/system/ |
kube-apiserver.service, kube-controller-manager.service, kube-scheduler.service |
Systemd service units managing daemon lifecycle |
| Ports | 10.0.0.2:6443 |
kube-apiserver |
Main Kubernetes API Server HTTPS endpoint |
| Ports | 127.0.0.1:10257 |
kube-controller-manager |
Local secure metrics and health endpoint |
| Ports | 127.0.0.1:10259 |
kube-scheduler |
Local secure metrics and health endpoint |
Step 3: Start and Enable Control Plane Services
Reload systemd to register the new unit files, then enable and launch all three daemons:
systemctl daemon-reload
systemctl enable kube-apiserver kube-controller-manager kube-scheduler
systemctl start kube-apiserver kube-controller-manager kube-scheduler
Note: Allow up to 10 seconds for
kube-apiserverto complete its initialization, establish mTLS connections toetcd, and start accepting requests.
Step 4: Provision API Server-to-Kubelet RBAC Rules
To allow kube-apiserver to query kubelet instances on worker nodes for logs, metrics, and interactive execution, apply the ClusterRole and ClusterRoleBinding on server:
kubectl apply -f kube-apiserver-to-kubelet.yaml --kubeconfig admin.kubeconfig
Expected Output:
clusterrole.rbac.authorization.k8s.io/system:kube-apiserver-to-kubelet created
clusterrolebinding.rbac.authorization.k8s.io/system:kube-apiserver created
8.4 Real-World Troubleshooting & Pitfalls
Pitfall 1: Incorrect File Permissions on TLS Private Keys
- Symptom:
kube-apiservercontinuously fails or restarts, andjournalctl -u kube-apiserverdisplayspermission deniederrors when reading/var/lib/kubernetes/ca.keyorkube-api-server.key. - Root Cause: Private key files were copied with open or incompatible file permissions (or incorrect ownership).
- Fix: Enforce standard secure permissions on
/var/lib/kubernetes/:
chmod 600 /var/lib/kubernetes/*.key
chmod 644 /var/lib/kubernetes/*.crt
Pitfall 2: etcd Unavailability or TLS Mismatch
- Symptom:
systemctl status kube-apiserverreports fatal connection failures or TLS handshake errors pointing to[https://127.0.0.1:2379](https://127.0.0.1:2379). - Root Cause:
etcdis not actively running on the local host, or the certificates supplied to--etcd-cafile,--etcd-certfile, and--etcd-keyfiledo not matchetcd’s client credentials. - Fix: Check
etcdhealth before launching control plane services:
systemctl is-active etcd
Pitfall 3: Systemd Unit File Syntax Errors or Missing Reloads
- Symptom: Executing
systemctl start kube-apiserverreturnsUnit kube-apiserver.service not foundor fails silently. - Root Cause: The service unit file was placed in
/etc/systemd/system/without re-indexing systemd manager configuration. - Fix: Ensure the file exists at
/etc/systemd/system/kube-apiserver.serviceand executesystemctl daemon-reload.
8.5 Verification & Validation
1. Verify Active Service States on server
Run systemctl is-active to confirm all three control plane components are active:
systemctl is-active kube-apiserver kube-controller-manager kube-scheduler
Expected Terminal Output:
active
active
active
2. Check Local Control Plane Health via kubectl
Run kubectl cluster-info on server using the administrator kubeconfig:
kubectl cluster-info --kubeconfig admin.kubeconfig
Expected Terminal Output:
Kubernetes control plane is running at https://127.0.0.1:6443
3. Remote Verification from jumpbox
Return to the jumpbox (10.0.0.1) machine to test external HTTPS access to the API Server version endpoint:
exit
curl --cacert ca.crt https://server.kubernetes.local:6443/version
Expected Terminal Output:
{
"major": "1",
"minor": "32",
"gitVersion": "v1.32.3",
"gitCommit": "32cc146f75aad04beaaa245a7157eb35063a9f99",
"gitTreeState": "clean",
"buildDate": "2025-03-11T19:52:21Z",
"goVersion": "go1.23.6",
"compiler": "gc",
"platform": "linux/arm64"
}
8.6 Section Summary
You have successfully bootstrapped the Kubernetes control plane on the server host, bringing up kube-apiserver, kube-controller-manager, and kube-scheduler with full mTLS authentication and API-to-Kubelet RBAC authorization. With the cluster’s decision-making center operational, you are ready to proceed to Lab 09: Bootstrapping the Kubernetes Worker Nodes.
9. Bootstrapping Worker Nodes (docs/09-bootstrapping-kubernetes-workers.md)
-
Host Prerequisites: Installing runtime dependencies (
socat,conntrack,ipset) and disabling system swap. -
Container Runtime:
-
Deploying
runcandcontainerd. -
Configuring CNI bridge networking (
10-bridge.conf,99-loopback.conf) and setting kernel parameters (br-netfilter). -
Kubelet & Kube-Proxy Setup: Installing unit files, applying per-node Pod CIDRs, and starting services via
systemctl. -
Node Verification: Querying cluster node status from the
jumpboxusingkubectl get nodes.
9.1 Section Overview & Context
In this lab, you will configure and bootstrap two Kubernetes worker nodes: node-0 (10.0.0.3) and node-1 (10.0.0.4). While the control plane manages cluster state and scheduling decisions, the worker nodes perform the heavy lifting: running containerized workloads inside Pods, managing network interfaces, and enforcing network proxy rules.
To turn a bare Linux host into a fully functional Kubernetes worker node, we must install and orchestrate four distinct runtime and networking layers:
+-----------------------------------------------------------------------------------+
| WORKER NODE (node-0 / node-1) |
| |
| [ kubelet ] <--- (CRI) ---> [ containerd ] <--- (OCI) ---> [ runc ] |
| | | | |
| | v v |
| | [ Pod Sandbox ] [ App Containers ] |
| v | |
| [ kube-proxy ] v |
| | [ CNI Plugins ] |
| v (bridge / loopback) |
| (iptables rules) | |
+-------+---------------------------+-----------------------------------------------+
| |
v v
[ Pod-to-Pod Traffic ] [ cni0 Bridge ]
- Container Runtime Interface (CRI) & OCI Runtime:
containerdmanages container lifecycles, image pulls, and local storage, whileruncacts as the low-level Open Container Initiative (OCI) executor that interacts with Linux kernel namespaces and cgroups to spawn isolation boundaries. - Node Agent (
kubelet): The primary node agent responsible for registering the node withkube-apiserver, monitoring host resources, and continuously reconciling local container states against assignedPodSpecmanifests. - Container Network Interface (CNI): Executable binaries (
bridge,loopback,host-local) that dynamically provision virtual ethernet (veth) pairs, configure bridge devices, and assign IP addresses to Pods. - Network Proxy (
kube-proxy): The node-level network agent that updates hostiptablesrules to translate KubernetesServicevirtual IPs (ClusterIPs and NodePorts) into real Pod endpoint addresses.
9.2 Technical Understanding & Deep Dive (The “Why”)
Before executing installation commands, it is essential to understand why each kernel setting, configuration flag, and utility binary is required.
A. Pod CIDR Subnet Allocation
Kubernetes mandates that every Pod gets its own unique, routable IP address within the cluster network. To achieve this without IP address collisions or complex NAT layers on host-to-host Pod communication, each worker node is assigned a non-overlapping sub-block of the cluster-wide IP range (10.200.0.0/16):
node-0is assigned10.200.0.0/24(Pod IPs:10.200.0.1–10.200.0.254).node-1is assigned10.200.1.0/24(Pod IPs:10.200.1.1–10.200.1.254).
During file templating on the jumpbox, we dynamically substitute the literal string SUBNET inside 10-bridge.conf and kubelet-config.yaml with each node’s allocated subnet.
B. Linux Kernel Requirements & Sysctl Parameters
- Disabling Swap (
swapoff -a):kubeletrelies on deterministic resource accounting. If host swap memory is enabled, memory-constrained Pods can thrash swap disks instead of hitting hard limits, leading to unpredictable latency, broken scheduling guarantees, and delayed Out-Of-Memory (OOM) handler triggers. - Kernel Module
br-netfilter: By default, Linux virtual bridge interfaces filter packet frames at Layer 2 (Data Link) without passing them through host Layer 3 (Network) packet filtering. Loadingbr-netfilterand settingnet.bridge.bridge-nf-call-iptables = 1forces bridged traffic to pass through the host’siptablesnetfilter rules. This allowskube-proxyto intercept Pod-to-Service traffic passing across thecni0bridge and apply Destination NAT (DNAT).
C. System Dependencies
socat: A socket relay tool that enables bidirectional data streams between standard streams or network sockets.kubeletdepends onsocatto execute streaming operations likekubectl port-forward,kubectl exec, andkubectl logs.conntrack: Enables the Linux kernel to track stateful IPv4 connections. Required bykube-proxyto ensure NAT mappings remain persistent and consistent across multi-packet connection streams.ipset: An administration tool for IP sets in the Linux kernel. Allows high-performance evaluation of dynamic IP rule tables without degrading system performance.
9.3 Step-by-Step Implementation & Command Breakdown
Step 3.1: Template and Distribute Configuration Files from jumpbox
Execute the following commands from /root/kubernetes-the-hard-way on the jumpbox (10.0.0.1).
- Extract Subnets & Substitute Configurations:
cd /root/kubernetes-the-hard-way
for HOST in node-0 node-1; do
SUBNET=$(grep ${HOST} machines.txt | cut -d " " -f 4)
sed "s|SUBNET|$SUBNET|g" configs/10-bridge.conf > 10-bridge.conf
sed "s|SUBNET|$SUBNET|g" configs/kubelet-config.yaml > kubelet-config.yaml
scp 10-bridge.conf kubelet-config.yaml root@${HOST}:~/
done
- Transfer Worker Binaries, Systemd Units, and Configurations:
for HOST in node-0 node-1; do
scp \
downloads/worker/* \
downloads/client/kubectl \
configs/99-loopback.conf \
configs/containerd-config.toml \
configs/kube-proxy-config.yaml \
units/containerd.service \
units/kubelet.service \
units/kube-proxy.service \
root@${HOST}:~/
done
- Transfer CNI Plugins:
for HOST in node-0 node-1; do
ssh root@${HOST} "mkdir -p ~/cni-plugins"
scp downloads/cni-plugins/* root@${HOST}:~/cni-plugins/
done
Step 3.2: Provision Worker Node 0 (node-0)
Log in to node-0 (10.0.0.3):
ssh root@node-0
Execute the full provisioning pipeline:
# 1. Install OS Dependencies
apt-get update && apt-get -y install socat conntrack ipset kmod
# 2. Disable Swap Memory
swapoff -a
# 3. Create Required Directory Tree
mkdir -p \
/etc/cni/net.d \
/opt/cni/bin \
/var/lib/kubelet \
/var/lib/kube-proxy \
/var/lib/kubernetes \
/var/run/kubernetes
# 4. Install Binaries to Path
mv crictl kube-proxy kubelet runc /usr/local/bin/
mv containerd containerd-shim-runc-v2 containerd-stress /bin/
mv cni-plugins/* /opt/cni/bin/
# 5. Configure CNI Network & Kernel Modules
mv 10-bridge.conf 99-loopback.conf /etc/cni/net.d/
modprobe br-netfilter
echo "br-netfilter" >> /etc/modules-load.d/modules.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.d/kubernetes.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.d/kubernetes.conf
sysctl -p /etc/sysctl.d/kubernetes.conf
# 6. Configure containerd Runtime
mkdir -p /etc/containerd/
mv containerd-config.toml /etc/containerd/config.toml
mv containerd.service /etc/systemd/system/
# 7. Configure kubelet
mv kubelet-config.yaml /var/lib/kubelet/
mv kubelet.service /etc/systemd/system/
# 8. Configure kube-proxy
mv kube-proxy-config.yaml /var/lib/kube-proxy/
mv kube-proxy.service /etc/systemd/system/
# 9. Reload Systemd and Launch Worker Services
systemctl daemon-reload
systemctl enable containerd kubelet kube-proxy
systemctl start containerd kubelet kube-proxy
Step 3.3: Provision Worker Node 1 (node-1)
Log out of node-0 and log into node-1 (10.0.0.4):
exit
ssh root@node-1
Run the identical provisioning block on node-1:
apt-get update && apt-get -y install socat conntrack ipset kmod
swapoff -a
mkdir -p \
/etc/cni/net.d \
/opt/cni/bin \
/var/lib/kubelet \
/var/lib/kube-proxy \
/var/lib/kubernetes \
/var/run/kubernetes
mv crictl kube-proxy kubelet runc /usr/local/bin/
mv containerd containerd-shim-runc-v2 containerd-stress /bin/
mv cni-plugins/* /opt/cni/bin/
mv 10-bridge.conf 99-loopback.conf /etc/cni/net.d/
modprobe br-netfilter
echo "br-netfilter" >> /etc/modules-load.d/modules.conf
echo "net.bridge.bridge-nf-call-iptables = 1" >> /etc/sysctl.d/kubernetes.conf
echo "net.bridge.bridge-nf-call-ip6tables = 1" >> /etc/sysctl.d/kubernetes.conf
sysctl -p /etc/sysctl.d/kubernetes.conf
mkdir -p /etc/containerd/
mv containerd-config.toml /etc/containerd/config.toml
mv containerd.service /etc/systemd/system/
mv kubelet-config.yaml /var/lib/kubelet/
mv kubelet.service /etc/systemd/system/
mv kube-proxy-config.yaml /var/lib/kube-proxy/
mv kube-proxy.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable containerd kubelet kube-proxy
systemctl start containerd kubelet kube-proxy
Worker Node Architecture & Storage Matrix
| Component / File Path | Target Directory | Purpose | Key Parameters / Flags |
|---|---|---|---|
kubelet |
/usr/local/bin/ |
Primary worker node agent | --config=/var/lib/kubelet/kubelet-config.yaml, --kubeconfig |
kube-proxy |
/usr/local/bin/ |
Service proxy & load balancer | --config=/var/lib/kube-proxy/kube-proxy-config.yaml |
containerd |
/bin/ |
High-level CRI runtime daemon | Configured via /etc/containerd/config.toml (SystemdCgroup = true) |
runc |
/usr/local/bin/ |
OCI container execution tool | Spawns low-level cgroups & namespaces |
| CNI Binaries | /opt/cni/bin/ |
Executables (bridge, loopback) |
Invoked dynamically by containerd on Pod lifecycle events |
| Port 10250/TCP | 0.0.0.0:10250 |
Kubelet Server API | Incoming HTTPS requests from API Server (exec, logs, attach) |
| Port 10256/TCP | 0.0.0.0:10256 |
Kube-proxy Health Check | Serves HTTP /healthz endpoints for node load balancers |
9.4 Real-World Troubleshooting & Pitfalls (Proxmox LXC Specifics)
If you are running your Kubernetes nodes inside Proxmox LXC containers (sharing the hyproxmox host kernel), standard systemd initialization will encounter several container isolation restrictions. Below are the specific gotchas, root causes, and exact remediation steps.
Gotcha 1: modprobe: FATAL: Module br-netfilter not found & containerd Systemd Pre-Start Failure
- Symptom: Running
modprobeinside an LXC worker fails withFATAL: Module br-netfilter not found. Additionally,containerd.serviceenters a crash loop withExecStartPre=/sbin/modprobe overlayreturning status code1/FAILURE. - Root Cause: LXC containers share the Proxmox host’s kernel directly and lack a standalone
/lib/modules/$(uname -r)file tree. Unprivileged LXC containers cannot dynamically load kernel modules. - Fix:
- Load the modules directly on the Proxmox Host CLI (
root@hyproxmox):
modprobe br_netfilter
modprobe overlay
- Comment out the
ExecStartPredirective in/etc/systemd/system/containerd.serviceon both worker nodes (node-0andnode-1):
sed -i 's/^ExecStartPre=.*modprobe.*/# &/' /etc/systemd/system/containerd.service
systemctl daemon-reload
systemctl restart containerd
Gotcha 2: Kubelet Crash — Missing /dev/kmsg & OOM Adjustment Denials
- Symptom:
kubeletfails on startup. Inspectingjournalctl -u kubelet -n 30 --no-pagerdisplays:
Failed to create an oomWatcher ... err="open /dev/kmsg: no such file or directory"
failed to set "/proc/self/oom_score_adj" to "-999": permission denied
- Root Cause: LXC containers do not expose a pseudo-device for
/dev/kmsg(kernel log ring buffer). Furthermore, container security profiles prevent unprivileged processes from adjusting their process OOM scores to negative values. - Fix: Execute the following commands on both
node-0andnode-1:
# 1. Create a persistent /dev/kmsg symlink service
cat <<EOF > /etc/systemd/system/kmsg-symlink.service
[Unit]
Description=Create /dev/kmsg symlink for Kubelet
Before=kubelet.service
[Service]
Type=oneshot
ExecStart=/usr/bin/ln -sf /dev/console /dev/kmsg
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now kmsg-symlink.service
# 2. Force Kubelet OOM Score Adjustment to 0
echo "oomScoreAdj: 0" >> /var/lib/kubelet/kubelet-config.yaml
systemctl restart kubelet
Gotcha 3: Kubelet Crash — Read-Only /proc/sys Kernel Parameter Writes
- Symptom:
kubeletlogs indicateContainerManagerinitialization errors when attempting to apply kernel settings (vm.overcommit_memory,kernel.panic,kernel.panic_on_oops). - Root Cause: Proxmox mounts
/proc/sysas read-only inside LXC environments to protect overall host stability. - Fix:
- Apply sysctls globally on the Proxmox Host (
root@hyproxmox):
sysctl -w vm.overcommit_memory=1
sysctl -w kernel.panic=10
sysctl -w kernel.panic_on_oops=1
- Configure
kubeleton worker nodes to protect host kernel defaults and allow user-namespace operation by updating/var/lib/kubelet/kubelet-config.yaml:
cat <<EOF >> /var/lib/kubelet/kubelet-config.yaml
protectKernelDefaults: true
featureGates:
KubeletInUserNamespace: true
EOF
systemctl restart kubelet
Gotcha 4: Kube-Proxy Failure — nf_conntrack_max Permission Denied & Invalid Syntax
- Symptom:
kube-proxyfails to start, outputtingopen /proc/sys/net/netfilter/nf_conntrack_max: permission denied. Appendingconntrack:incorrectly can also triggeryaml: line 5: did not find expected key. - Root Cause: Write operations to netfilter connection tracking tables inside LXC are restricted. YAML indentation syntax errors prevent parsing the proxy configuration file.
- Fix:
- Set conntrack limits on the Proxmox Host (
root@hyproxmox):
sysctl -w net.netfilter.nf_conntrack_max=131072
- Overwrite
/var/lib/kube-proxy/kube-proxy-config.yamlwith a clean configuration instructingkube-proxyto skip kernel conntrack adjustments (maxPerCore: 0):
cat <<EOF > /var/lib/kube-proxy/kube-proxy-config.yaml
kind: KubeProxyConfiguration
apiVersion: kubeproxy.config.k8s.io/v1alpha1
clientConnection:
kubeconfig: "/var/lib/kube-proxy/kubeconfig"
mode: "iptables"
clusterCIDR: "10.200.0.0/16"
conntrack:
maxPerCore: 0
EOF
systemctl restart kube-proxy
9.5 Verification & Validation
Step 5.1: Local Service Verification
Verify that all three core worker services report an active state on both node-0 and node-1:
systemctl is-active containerd kubelet kube-proxy
Expected Terminal Output:
active
active
active
Step 5.2: Cluster Node Registration Check
Log into the jumpbox (10.0.0.1) and query the Kubernetes API server running on server (10.0.0.2):
ssh root@server "kubectl get nodes --kubeconfig admin.kubeconfig"
Expected Terminal Output:
NAME STATUS ROLES AGE VERSION
node-0 Ready <none> 1m v1.32.3
node-1 Ready <none> 10s v1.32.3
Both worker nodes successfully report a Ready status, confirming that kubelet has registered with the API server, the CRI container runtime is responsive, and CNI networking plugin paths are properly linked.
10. Configuring Remote kubectl Access (docs/10-configuring-kubectl.md)
- Admin Context Setup: Configuring local
kubectlon thejumpboxwith embedded admin credentials targeting[https://server.kubernetes.local:6443](https://server.kubernetes.local:6443). - Cluster Access Verification: Executing
kubectl versionandkubectl get nodeswithout specifying explicit config flags.
10.1 Section Overview & Context
In this section, you will configure kubectl on the jumpbox (10.0.0.1) to enable administrative access to the remote Kubernetes cluster.
Up to this point, worker nodes and control plane components have been configured with dedicated kubeconfig files tailored to their specific identities (e.g., node-0.kubeconfig, kube-scheduler.kubeconfig). Here, you will construct a centralized default configuration file located at ~/.kube/config on your administrative workstation. This setup allows you to execute cluster management commands directly from the jumpbox without manually passing credentials or configuration flags on every invocation.
+-----------------------------------------------------------------------+
| JUMPBOX |
| (10.0.0.1) |
| |
| ~/.kube/config |
| +---------------------------------------------------------------+ |
| | Context: kubernetes-the-hard-way | |
| | ├── Cluster: https://server.kubernetes.local:6443 (Embedded CA) | |
| | └── User: admin (CN=admin, O=system:masters Cert/Key) | |
| +---------------------------------------------------------------+ |
| | |
| kubectl command |
+--------------------------------|--------------------------------------+
| Mutual TLS (Port 6443)
v
+-----------------------------------------------------------------------+
| CONTROL PLANE (server) |
| (10.0.0.2) |
| |
| kube-apiserver |
| ├── Validates client cert against Root CA |
| └── Authorizes 'O=system:masters' with cluster-admin rights |
+-----------------------------------------------------------------------+
10.2 Technical Understanding & Deep Dive
Local vs. Remote Administration Endpoints
In Lab 05, the initial admin.kubeconfig was generated targeting [https://127.0.0.1:6443](https://127.0.0.1:6443) because it was intended for execution directly on the control plane server. If you attempt to use that configuration from the jumpbox, kubectl will attempt to connect to 127.0.0.1:6443 on the jumpbox itself—where no API server exists—resulting in a connection refused error.
To manage the cluster remotely, kubectl must target the domain server.kubernetes.local:6443, which resolves via /etc/hosts to the control plane IP (10.0.0.2).
Kubeconfig Structure & TLS Data Embedding
A kubeconfig file consists of three logical sections:
- Clusters: Maps cluster nicknames to API Server URLs and public CA certificates.
- Users: Maps user identities to authentication credentials (client certificates and private keys).
- Contexts: Links a specific User identity to a specific Cluster endpoint.
When configuring these entries, passing the --embed-certs=true flag instructs kubectl to read the raw contents of ca.crt, admin.crt, and admin.key, base64-encode them, and store them directly inside ~/.kube/config under certificate-authority-data, client-certificate-data, and client-key-data.
Why Embed Certs? Without embedding,
kubectlrelies on relative or absolute file paths to locate certificates. If you runkubectlfrom a different working directory or move the configuration file, commands will fail due to missing path references. Embedding turns~/.kube/configinto a portable, self-contained file.
RBAC & Superuser Authentication
The client certificate (admin.crt) contains specific Subject Distinguished Name fields:
- Common Name (
CN):admin - Organization (
O):system:masters
When kube-apiserver receives an HTTPS request, it performs Mutual TLS (mTLS) verification using its trusted Root CA. Upon confirming the certificate is valid, it inspects the Organization field (O=system:masters). In Kubernetes, system:masters is a hardcoded break-glass group that automatically grants unrestricted administrative rights (cluster-admin) across the cluster, bypassing standard RBAC restrictions.
10.3 Step-by-Step Implementation & Command Breakdown
Perform all commands in this section from the jumpbox (10.0.0.1) inside the /root/kubernetes-the-hard-way directory.
Step 1: Verify Remote API Server Reachability
Before writing the configuration file, verify HTTPS reachability and TLS trust between the jumpbox and the API server:
cd /root/kubernetes-the-hard-way
curl --cacert ca.crt https://server.kubernetes.local:6443/version
Expected Output:
{
"major": "1",
"minor": "32",
"gitVersion": "v1.32.3",
"gitCommit": "32cc146f75aad04beaaa245a7157eb35063a9f99",
"gitTreeState": "clean",
"buildDate": "2025-03-11T19:52:21Z",
"goVersion": "go1.23.6",
"compiler": "gc",
"platform": "linux/amd64"
}
--cacert ca.crt: Explicitly supplies the Certificate Authority certificate to validate the TLS identity presented bykube-apiserver.
Step 2: Generate the Default Administrative ~/.kube/config
Execute the following block to define the cluster endpoint, credentials, and default context profile:
{
kubectl config set-cluster kubernetes-the-hard-way \
--certificate-authority=ca.crt \
--embed-certs=true \
--server=https://server.kubernetes.local:6443
kubectl config set-credentials admin \
--client-certificate=admin.crt \
--client-key=admin.key
kubectl config set-context kubernetes-the-hard-way \
--cluster=kubernetes-the-hard-way \
--user=admin
kubectl config use-context kubernetes-the-hard-way
}
Detailed Flag Breakdown:
| Command / Flag | Purpose |
|---|---|
set-cluster kubernetes-the-hard-way |
Names the cluster entry inside the configuration file. |
--certificate-authority=ca.crt |
Points to the Root CA certificate used to verify the server’s identity. |
--embed-certs=true |
Inlines base64-encoded binary certificate and key data into the file. |
--server=https://...:6443 |
Specifies the HTTPS listener address of kube-apiserver. |
set-credentials admin |
Names the user profile storing client credentials. |
--client-certificate=admin.crt |
Sets the client certificate containing CN=admin and O=system:masters. |
--client-key=admin.key |
Sets the 4096-bit RSA private key used for TLS client authentication. |
set-context ... |
Maps the admin user to the kubernetes-the-hard-way cluster endpoint. |
use-context ... |
Sets current-context to kubernetes-the-hard-way in ~/.kube/config. |
10.4 Troubleshooting & Gotchas
1. Host Resolution Failures
- Symptom:
curl: (6) Could not resolve host: server.kubernetes.localorunable to connect to server: dial tcp: lookup server.kubernetes.local. - Root Cause: Missing DNS mapping in
/etc/hostson the jumpbox. - Fix: Add the control plane IP mapping to
/etc/hosts:
echo "10.0.0.2 server.kubernetes.local" >> /etc/hosts
2. Loopback Connection Errors
- Symptom:
The connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port? - Root Cause: The kubeconfig file was generated using
[https://127.0.0.1:6443](https://127.0.0.1:6443)rather than[https://server.kubernetes.local:6443](https://server.kubernetes.local:6443). - Fix: Re-run
kubectl config set-clusterwith--server=[https://server.kubernetes.local:6443](https://server.kubernetes.local:6443).
3. Broken Relative Paths
- Symptom:
error: unable to read client-cert admin.crt: no such file or directorywhen runningkubectlcommands outside/root/kubernetes-the-hard-way. - Root Cause: Omitting
--embed-certs=trueduringkubectl config set-clusterorset-credentials. - Fix: Re-run the configuration commands ensuring
--embed-certs=trueis included, or manually re-generate the config while inside the directory containing the.crtand.keyfiles.
10.5 Verification & Validation
Inspect the active configuration stored in ~/.kube/config:
kubectl config view
Confirm that kubectl can query the remote Kubernetes cluster version without passing explicit config flags:
kubectl version
Expected Output:
Client Version: v1.32.3
Kustomize Version: v5.5.0
Server Version: v1.32.3
Verify administrative control by querying all registered worker nodes:
kubectl get nodes
Expected Output:
NAME STATUS ROLES AGE VERSION
node-0 Ready <none> 15m v1.32.3
node-1 Ready <none> 14m v1.32.3
10.6 Section Summary
You have successfully configured remote administrative access on the jumpbox by building a self-contained ~/.kube/config file containing embedded mTLS credentials and Root CA data. With kubectl communicating securely with kube-apiserver and returning both worker nodes in a Ready state, the cluster control plane is fully operational and ready for network routing configuration.
11. Pod Network Routes (docs/11-pod-network-routes.md)
- Routing Problem Statement: Addressing cross-node Pod communication without overlay networks.
- Routing Table Provisioning: Adding static kernel routes (
ip route add) acrossserver,node-0, andnode-1mapping Pod subnets (10.200.x.0/24) to target node internal IPs. - Verification: Inspecting kernel routing tables across all cluster nodes (
ip route).
11.1 Section Overview & Context
In the previous sections of this cluster build, you configured the Container Network Interface (CNI) bridge plugin across worker nodes and assigned each a dedicated Pod CIDR block (10.200.0.0/24 to node-0 and 10.200.1.0/24 to node-1). While each node can now allocate IP addresses to local Pods via its local cni0 bridge interface, packets destined for Pods on other nodes currently have nowhere to go.
This section covers the creation of static kernel network routes across all nodes in the cluster (server, node-0, and node-1). By mapping each worker node’s Pod CIDR subnet to its host IPv4 address, we establish flat, un-encapsulated Layer 3 routing. This step is the bridge between isolated single-node Pod networks and a fully connected multi-node Kubernetes cluster.
11.2 Technical Understanding & Deep Dive
The Kubernetes Networking Model: IP-Per-Pod
Kubernetes mandates two core networking constraints for all compliant clusters:
- Pod-to-Pod Communication: Any Pod on any node can communicate with any other Pod on any node without Network Address Translation (NAT).
- Node-to-Pod Communication: Any system process or agent on a node (such as the
kubeletorkube-apiserver) can communicate directly with any Pod on any node.
Demystifying Cross-Node Pod Traffic Flow
When Pod A (10.200.0.2) on node-0 sends a packet to Pod B (10.200.1.2) on node-1:
+-------------------------------------+ +-------------------------------------+
| WORKER NODE-0 | | WORKER NODE-1 |
| (10.0.0.3) | | (10.0.0.4) |
| | | |
| Pod A (10.200.0.2) | | Pod B (10.200.1.2) |
| │ | | ▲ |
| ▼ | | │ |
| cni0 Bridge (10.200.0.1/24) | | cni0 Bridge (10.200.1.1/24) |
| │ | | ▲ |
| ▼ | | │ |
| Linux Routing Table | | Linux Routing Table |
| [ 10.200.1.0/24 via 10.0.0.4 ] | | [ 10.200.0.0/24 via 10.0.0.3 ] |
+-------┼-----------------------------+ +-------▲-----------------------------+
│ │
└───────────────────── eth0 ────────────────────┘
(Subnet Transit: 10.0.0.0/24)
- Local Egress: The packet leaves Pod A’s network namespace via a virtual ethernet pair (
veth) and hits thecni0bridge interface onnode-0. - Kernel Route Lookup: The Linux kernel on
node-0inspects the packet’s destination IP (10.200.1.2).
- Without static routes: The kernel has no matching rule for
10.200.1.0/24. It forwards the packet to the default gateway (10.0.0.254) where it is dropped or rejected. - With static routes: The kernel matches the
10.200.1.0/24entry and routes the packet out interfaceeth0directly to next-hop gateway10.0.0.4(node-1).
- Ingress Delivery:
node-1receives the frame oneth0. Its kernel recognizes10.200.1.2as belonging to its localcni0bridge network and forwards the packet straight into Pod B.
Why Control Plane Nodes (server) Need Routes
The control plane host (server) runs kube-apiserver, which needs to open direct TCP streams to Pod IPs when running operations like:
- Executing commands inside containers (
kubectl exec) - Streaming pod logs (
kubectl logs) - Invoking API admission webhooks hosted inside cluster Pods
Without static routes on server, control plane commands directed at Pod IPs will fail with connection timeouts.
11.3 Step-by-Step Implementation & Command Breakdown
All commands are executed from the jumpbox administrative node (10.0.0.1) inside the /root/kubernetes-the-hard-way directory.
Step 1: Extract Network Variables
Extract the node IPv4 addresses and allocated Pod CIDR subnets from machines.txt:
cd /root/kubernetes-the-hard-way
{
SERVER_IP=$(grep server machines.txt | cut -d " " -f 1)
NODE_0_IP=$(grep node-0 machines.txt | cut -d " " -f 1)
NODE_0_SUBNET=$(grep node-0 machines.txt | cut -d " " -f 4)
NODE_1_IP=$(grep node-1 machines.txt | cut -d " " -f 1)
NODE_1_SUBNET=$(grep node-1 machines.txt | cut -d " " -f 4)
}
Command Breakdown:
grep <host> machines.txt: Filters the inventory file for the target node record.cut -d " " -f 1: Parses field 1, extracting the internal host IP (10.0.0.2,10.0.0.3, or10.0.0.4).cut -d " " -f 4: Parses field 4, extracting the allocated Pod CIDR subnet (10.200.0.0/24or10.200.1.0/24).
Step 2: Provision Routes on the Control Plane Server (server)
Configure static routes on server to reach both worker node Pod CIDRs:
ssh root@server <<EOF
ip route add ${NODE_0_SUBNET} via ${NODE_0_IP}
ip route add ${NODE_1_SUBNET} via ${NODE_1_IP}
EOF
Flag & Argument Breakdown:
ip route add: Adds an entry to the Linux kernel IPv4 routing table.${NODE_0_SUBNET}(10.200.0.0/24): Specifies the destination IP CIDR block.via ${NODE_0_IP}(via 10.0.0.3): Sets the Layer 3 next-hop gateway address. Traffic is forwarded over the primary network interface (eth0).
Step 3: Provision Routes on Worker Node 0 (node-0)
Configure node-0 to route traffic destined for node-1 Pods across the shared subnet:
ssh root@node-0 <<EOF
ip route add ${NODE_1_SUBNET} via ${NODE_1_IP}
EOF
Step 4: Provision Routes on Worker Node 1 (node-1)
Configure node-1 to route traffic destined for node-0 Pods across the shared subnet:
ssh root@node-1 <<EOF
ip route add ${NODE_0_SUBNET} via ${NODE_0_IP}
EOF
11.4 Real-World Troubleshooting & Pitfalls
Ephemeral Routes Lost Across Reboots
- Symptom: Cross-node Pod communication and
kubectl logswork initially, but fail suddenly after an LXC container or node reboot. - Root Cause: Executing
ip route addupdates the active kernel routing table stored in RAM, but does not write to persistent networking configuration files. - Fix: For persistent routing in Debian/Ubuntu-based environments, append static route declarations to
/etc/network/interfacesor your Netplan configuration file (/etc/netplan/01-netcfg.yaml):
network:
version: 2
ethernets:
eth0:
routes:
- to: 10.200.1.0/24
via: 10.0.0.4
Proxmox LXC MAC/IP Filtering Denials
- Symptom: Routes exist in
ip route, butpingbetween Pod IPs on different nodes times out. Packet dumps (tcpdump -i eth0) show ARP requests going unanswered. - Root Cause: In Proxmox VE or hypervisor environments, MAC/IP anti-spoofing flags on virtual interfaces (e.g.,
veth903i0) may drop frames originating from IP ranges not explicitly assigned to the container interface (10.200.x.x). - Fix: Ensure IP Filter / MAC Filter options are disabled for the container virtual network interfaces in the Proxmox VE Firewall settings for the guest.
11.5 Verification & Validation
Execute ip route on each node via SSH to confirm that the static routes are active in the kernel.
1. Verify Control Plane (server) Routing Table
ssh root@server ip route
Expected Output:
default via 10.0.0.254 dev eth0 proto static
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.2
10.200.0.0/24 via 10.0.0.3 dev eth0
10.200.1.0/24 via 10.0.0.4 dev eth0
2. Verify Worker Node 0 (node-0) Routing Table
ssh root@node-0 ip route
Expected Output:
default via 10.0.0.254 dev eth0 proto static
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.3
10.200.1.0/24 via 10.0.0.4 dev eth0
3. Verify Worker Node 1 (node-1) Routing Table
ssh root@node-1 ip route
Expected Output:
default via 10.0.0.254 dev eth0 proto static
10.0.0.0/24 dev eth0 proto kernel scope link src 10.0.0.4
10.200.0.0/24 via 10.0.0.3 dev eth0
Cluster Routing Matrix Summary
| Host | Node IP | Target Pod Subnet | Next-Hop Gateway | Status |
|---|---|---|---|---|
server |
10.0.0.2 |
10.200.0.0/24 (node-0) |
10.0.0.3 |
Active |
10.200.1.0/24 (node-1) |
10.0.0.4 |
Active | ||
node-0 |
10.0.0.3 |
10.200.1.0/24 (node-1) |
10.0.0.4 |
Active |
node-1 |
10.0.0.4 |
10.200.0.0/24 (node-0) |
10.0.0.3 |
Active |
11.6 Section Summary
By populating the kernel routing tables on server, node-0, and node-1, you have fulfilled the fundamental Kubernetes networking requirement of un-NATted, cross-node Pod routing. With the underlying network fabric complete, you are ready to validate end-to-end cluster functionality in the Smoke Test.
12. Smoke Test (docs/12-smoke-test.md)
-
Data Encryption: Creating a generic Secret and performing an
etcdctlhexdump to verify thek8s:enc:aescbc:v1:key1header. -
Workload Management: Deploying an
nginxworkload and testing cluster primitives: -
kubectl port-forwardconnection forwarding. -
kubectl logslog stream retrieval. -
kubectl execcontainer command execution. -
Service Reachability: Exposing
nginxas aNodePortservice and testing HTTP access via node host IP and assigned port.
12.1 Section Overview & Context
With the control plane services (kube-apiserver, kube-controller-manager, kube-scheduler) operational, worker binaries (kubelet, kube-proxy, containerd) running, and cross-node networking established via custom CNI bridges and host routes, your cluster infrastructure is fully assembled.
However, a running set of systemd units does not guarantee a functional cluster. The Smoke Test is the ultimate end-to-end verification suite for your Kubernetes build. In this step, you will validate the core data plane and control plane mechanisms:
- Data Encryption at Rest: Confirming that sensitive data stored via
kube-apiserveris ciphered before hittingetcd. - Workload Scheduling & Execution: Ensuring the end-to-end container orchestration loop succeeds from deployment manifest to running
runccontainer namespaces. - Control Plane-to-Node Streaming: Validating API proxy channels for
port-forward, container log streaming (kubectl logs), and interactive execution (kubectl exec). - Service Discovery & NAT Routing: Verifying
kube-proxyiptablesrules and cross-node network packet delivery usingNodePortservices.
+-----------------------------------------------------------------------+
| SMOKE TEST VALIDATION |
+-----------------------------------------------------------------------+
|
+-----------------------+-------------------------+-----------------------+-----------------------+
| | | | |
v v v v v
[Secret Encryption] [Workload Scheduling] [Stream Proxying] [Log & Exec Hooks] [NodePort NAT Routing]
`kube-apiserver` ReplicaSet & Scheduler `kubectl port-forward` API Server -> Kubelet `kube-proxy` iptables
-> AES-CBC -> `etcd` -> CRI -> `runc` -> `socat` -> Pod mTLS Auth & Logs Stream -> Host IP:Port -> Pod
12.2 Technical Understanding & Deep Dive (The “Why”)
1. Envelope Encryption at Rest
In previous labs, you configured kube-apiserver with an EncryptionConfiguration specifying the aescbc (AES-CBC with PKCS#7 padding) provider and an encryption key (key1).
When a client creates a Secret, kube-apiserver handles the payload in memory as plaintext, encrypts the raw JSON byte stream using the key, and writes the resulting ciphertext to etcd at /registry/secrets/<namespace>/<name>. Querying etcd directly via etcdctl bypasses the API server’s decryption layer, allowing you to prove that disk persistence is secure and plaintext data (e.g., mykey=mydata) is never exposed in raw database blocks.
2. Workload Scheduling & Namespace Provisioning
Executing kubectl create deployment triggers a sequence across the control plane and worker components:
kube-apiservervalidates and persists theDeploymentmanifest toetcd.kube-controller-managerdetects the new deployment, creates aReplicaSet, and spawns a targetPodobject in theUnscheduledstate (nodeName: "").kube-schedulerevaluates available compute nodes, assigns the Pod to a worker (e.g.,node-0at10.0.0.3ornode-1at10.0.0.4), and updates the binding in the API server.kubeleton the assigned worker detects the Pod assignment via its API watch connection, instructscontainerdover the Container Runtime Interface (CRI) to pull the image, and invokesruncto isolate the container inside Linux namespaces (net,pid,mnt,ipc).
3. Stream Proxying (port-forward, logs, exec)
Admin commands like port-forward, logs, and exec do not communicate with containers directly. Instead, they leverage secured API streams:
Jumpbox (10.0.0.1) Control Plane (10.0.0.2) Worker Node (node-0 / node-1)
+--------------------+ +------------------------+ +-------------------------------+
| curl 127.0.0.1:8080| | kube-apiserver | | kubelet (Port 10250) |
| │ | | (Port 6443) | | │ |
| ▼ | HTTPS | │ | HTTPS | ▼ |
| kubectl port-----+─────────┼────────────┼───────────┼─────────┼──> socat -> Nginx Pod Port 80 |
| forward | (mTLS) | │ | (mTLS) | (10.200.x.x:80) |
+--------------------+ +------------------------+ +-------------------------------+
port-forward:kubectlestablishes an HTTP/2 SPDY connection tokube-apiserver(port6443), which proxies HTTPS traffic tokubelet(port10250) on the target worker node.kubeletspawnssocaton the node to relay raw TCP sockets between the stream and the container’s virtual network interface.- **
logs&exec**:kube-apiserverauthenticates againstkubeletusing its client certificate (kube-api-server.crt), validated by thesystem:kube-apiserver-to-kubeletClusterRole.kubeletreads stdout log files directly from/var/log/pods/...or invokesrunc execinside the target namespaces.
4. Service Exposure & NodePort Packet Flow
Exposing a workload via NodePort tests the network proxy implementation:
kube-proxywatches the API server for newServiceendpoints.- Upon detecting a
NodePortservice,kube-proxyopens a high-range port (between30000-32767) across every worker host and programs localiptablesrules (PREROUTINGandKUBE-SERVICESchains). - Incoming traffic to
<Worker-IP>:<NodePort>is intercepted by hostiptablesrules, which apply Destination NAT (DNAT) to rewrite the destination socket from the worker node’s IP to the Pod’s internal IP (10.200.x.x:80). If the target Pod resides on a different physical host, host static routes forward the packet over the bridge network (vmbr1).
12.3 Step-by-Step Implementation & Command Breakdown
All commands in this lab are executed from your administrative workstation (jumpbox at 10.0.0.1).
Step 1: Verify Secret Data Encryption at Rest
- Create a generic
Secretcontaining a literal key-value pair:
kubectl create secret generic kubernetes-the-hard-way \
--from-literal="mykey=mydata"
create secret generic: Creates an opaque KubernetesSecretobject namedkubernetes-the-hard-way.--from-literal="mykey=mydata": Sets the keymykeyto store the valuemydata.
- Inspect the raw key-value entry stored directly inside
etcdon the control plane host (serverat10.0.0.2):
ssh root@server \
'etcdctl get /registry/secrets/default/kubernetes-the-hard-way | hexdump -C'
etcdctl get /registry/secrets/default/...: Directly queries the key-value store database path for the secret in thedefaultnamespace.hexdump -C: Converts raw binary database bytes into hexadecimal representations with ASCII translations.
Expected Terminal Output:
00000000 2f 72 65 67 69 73 74 72 79 2f 73 65 63 72 65 74 |/registry/secret|
00000010 73 2f 64 65 66 61 75 6c 74 2f 6b 75 62 65 72 6e |s/default/kubern|
00000020 65 74 65 73 2d 74 68 65 2d 68 61 72 64 2d 77 61 |etes-the-hard-wa|
00000030 79 0a 6b 38 73 3a 65 6e 63 3a 61 65 73 63 62 63 |y.k8s:enc:aescbc|
00000040 3a 76 31 3a 6b 65 79 31 3a 4f 1b 80 d8 89 72 f4 |:v1:key1:O....r.|
00000050 60 8a 2c a0 76 1a e1 dc 98 d6 00 7a a4 2f f3 92 |`.,.v......z./..|
00000060 87 63 c9 22 f4 58 c8 27 b9 ff 2c 2e 1a b6 55 be |.c.".X.'..,...U.|
00000070 d5 5c 4d 69 82 2f b7 e4 b3 b0 12 e1 58 c4 9c 77 |.\Mi./......X..w|
00000080 78 0c 1a 90 c9 c1 23 6c 73 8e 6e fd 8e 9c 3d 84 |x.....#ls.n...=.|
00000090 7d bf 69 81 ce c9 aa 38 be 3b dd 66 aa a3 33 27 |}.i....8.;.f..3'|
000000a0 df be 6d ac 1c 6d 8a 82 df b3 19 da 0f 93 94 1e |..m..m..........|
000000b0 e0 7d 46 8d b5 14 d0 c5 97 e2 94 76 26 a8 cb 33 |.}F........v&..3|
000000c0 57 2a d0 27 a6 5a e1 76 a7 3f f0 b7 0a 7b ff 53 |W*.'.Z.v.?...{.S|
000000d0 cf c9 1a 18 5b 45 f8 b1 06 3b a9 45 02 76 23 61 |....[E...;.E.v#a|
000000e0 5e dc 86 cf 8e a4 d3 c9 5c 6a 6f e6 33 7b 5b 8f |^.......\jo.3{[.|
000000f0 fb 8a 14 74 58 f9 49 2f 97 98 cc 5c d4 4a 10 1a |...tX.I/...\.J..|
00000100 64 0a 79 21 68 a0 9e 7a 03 b7 19 e6 20 e4 1b ce |d.y!h..z.... ...|
00000110 91 64 ce 90 d9 4f 86 ca fb 45 2f d6 56 93 68 e1 |.d...O...E/.V.h.|
00000120 0b aa 8c a0 20 a6 97 fa a1 de 07 6d 5b 4c 02 96 |.... ......m[L..|
00000130 31 70 20 83 16 f9 0a 22 5c 63 ad f1 ea 41 a7 1e |1p ...."\c...A..|
00000140 29 1a d4 a4 e9 d7 0c 04 74 66 04 6d 73 d8 2e 3f |).......tf.ms..?|
00000150 f0 b9 2f 77 bd 07 d7 7c 42 0a |../w...|B.|
0000015a
- Header Verification: Note the prefix
k8s:enc:aescbc:v1:key1at offset00000030. This confirms thatkube-apiserversuccessfully processed the write using theaescbcencryption provider and keykey1. - Ciphertext Verification: The string
"mydata"does not appear anywhere in cleartext within the ASCII representation.
Step 2: Deploy Workloads & Inspect Scheduling
- Provision an
nginxweb server deployment:
kubectl create deployment nginx \
--image=nginx:latest
create deployment nginx: Declares a Deployment controller managing 1 replica ofnginx:latest.
- Inspect the provisioned Pod state:
kubectl get pods -l app=nginx
-l app=nginx: Filters the list of active pods matching the deployment label selector.
Expected Terminal Output:
NAME READY STATUS RESTARTS AGE
nginx-56fcf95486-c8dnx 1/1 Running 0 8s
Step 3: Test Remote Port Forwarding
- Extract the active Pod name into an environment variable:
POD_NAME=$(kubectl get pods -l app=nginx \
-o jsonpath="{.items[0].metadata.name}")
-o jsonpath="{.items[0].metadata.name}": Parses API output to return only the single string pod identifier.
- Establish a TCP tunnel forwarding local port
8080on thejumpboxhost to container port80:
kubectl port-forward $POD_NAME 8080:80
Expected Terminal Output:
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
- Open a separate terminal session on
jumpboxand dispatch an HTTPHEADrequest to the local tunnel:
curl --head http://127.0.0.1:8080
Expected Terminal Output:
HTTP/1.1 200 OK
Server: nginx/1.27.4
Date: Sun, 06 Apr 2025 17:17:12 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Wed, 05 Feb 2025 11:06:32 GMT
Connection: keep-alive
ETag: "67a34638-267"
Accept-Ranges: bytes
- Return to the original terminal running the
port-forwardsession and terminate the process usingCtrl+C.
Step 4: Verify Container Log Streaming and Exec Capabilities
- Fetch stdout/stderr logs generated by the container runtime:
kubectl logs $POD_NAME
Expected Terminal Output:
...
127.0.0.1 - - [06/Apr/2025:17:17:12 +0000] "HEAD / HTTP/1.1" 200 0 "-" "curl/7.88.1" "-"
- Note: The log entry records the
HEADHTTP request routed through thekubectl port-forwardtunnel in Step 3.
- Execute commands directly inside the active Nginx container namespace:
kubectl exec -ti $POD_NAME -- nginx -v
- **
-t/-i**: Allocates an interactive pseudo-TTY session and keepsstdinopen. --: Signals the separator betweenkubectloptions and the binary path (nginx -v) executed within the target container container.
Expected Terminal Output:
nginx version: nginx/1.27.4
Step 5: Expose Applications via NodePort Services
- Expose the
nginxdeployment port publicly using aNodePortservice:
kubectl expose deployment nginx \
--port 80 --type NodePort
--type NodePort: Allocates a cluster-wide high port across every worker node to proxy incoming traffic to target pods.
Note: Standard cloud provider
LoadBalancertypes require an external cloud controller manager integration, which is out of scope for a manual cluster build.
- Retrieve the dynamic
nodePortassigned bykube-apiserver:
NODE_PORT=$(kubectl get svc nginx \
--output=jsonpath='{range .spec.ports[0]}{.nodePort}')
- Query the exact host worker node where
kube-schedulerplaced the Pod:
NODE_NAME=$(kubectl get pods \
-l app=nginx \
-o jsonpath="{.items[0].spec.nodeName}")
- Dispatch an HTTP request targeting the assigned worker node host IP/name and the dynamic
NodePort:
curl -I http://${NODE_NAME}:${NODE_PORT}
Expected Terminal Output:
HTTP/1.1 200 OK
Server: nginx/1.27.4
Date: Sun, 06 Apr 2025 17:18:36 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Wed, 05 Feb 2025 11:06:32 GMT
Connection: keep-alive
ETag: "67a34638-267"
Accept-Ranges: bytes
12.4 Real-World Troubleshooting & Pitfalls
LXC Container Runtime Failure (can't get final child's PID from pipe: EOF)
When running Kubernetes worker nodes inside Proxmox LXC containers (e.g., node-0 CT 903 at 10.0.0.3 and node-1 CT 904 at 10.0.0.4), deploying a Pod can result in the workload remaining stuck in ContainerCreating or FailedCreatePodSandBox.
Error Diagnostics
Checking events via kubectl describe pod -l app=nginx displays the following warning:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 2m (x12 over 5m) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox container task: OCI runtime create failed: runc create failed: unable to start container process: can't get final child's PID from pipe: EOF
Root Cause Analysis
This error occurs due to strict security isolation enforced by Proxmox host kernel defaults on unconfined LXC guests:
- When
containerdtriggersruncto isolate namespaces (pid,net,mnt),runcforks a child helper process to set up cgroup limits and/procmounts. - The host kernel’s default LXC AppArmor profiles and cgroup v2 device access lists block writes to
/sys/fs/cgroupand/proc/sysinside the nested container. - The Proxmox host terminates
runc’s child process abruptly.runcencounters an immediateEOFwhen trying to read the child PID from its synchronization pipe.
Step-by-Step Fix
- Update the LXC guest security profiles directly on your Proxmox Host CLI (
root@hyproxmox):
# Append full nested isolation overrides for CT 903 (node-0)
cat <<EOF >> /etc/pve/lxc/903.conf
lxc.apparmor.profile: unconfined
lxc.cgroup2.devices.allow: a
lxc.cap.drop:
lxc.mount.auto: proc:rw sys:rw
EOF
# Append full nested isolation overrides for CT 904 (node-1)
cat <<EOF >> /etc/pve/lxc/904.conf
lxc.apparmor.profile: unconfined
lxc.cgroup2.devices.allow: a
lxc.cap.drop:
lxc.mount.auto: proc:rw sys:rw
EOF
- Execute a cold reboot of both worker container instances from the Proxmox host to enforce kernel flags:
pct stop 903 && pct start 903
pct stop 904 && pct start 904
- Re-establish
/dev/kmsgcharacter device symlinks and restart worker services onnode-0andnode-1:
ssh root@node-0 "ln -sf /dev/console /dev/kmsg && systemctl restart containerd kubelet kube-proxy"
ssh root@node-1 "ln -sf /dev/console /dev/kmsg && systemctl restart containerd kubelet kube-proxy"
- Delete the stuck Pod on
jumpboxto forcekube-controller-managerto recreate the sandbox cleanly:
kubectl delete pod -l app=nginx
kubectl get pods -l app=nginx -w
12.5 Verification & Validation Matrix
Use this checklist matrix to confirm that all layers of your custom Kubernetes installation operate in accordance with Kubernetes core specifications:
| Sub-Test | Component Validated | Path / Protocol Tested | Success Criteria |
|---|---|---|---|
| Data Encryption | kube-apiserver & etcd |
Encryption at rest (aescbc) |
k8s:enc:aescbc:v1:key1 header present in raw etcdctl dump; payload encrypted |
| Workload Deployment | controller-manager, scheduler, kubelet, containerd |
Pod scheduling & OCI container execution | Pod transitions to 1/1 Running state |
| Port Forwarding | kube-apiserver, kubelet, socat |
SPDY / HTTP2 stream proxying | Returns HTTP/1.1 200 OK from [http://127.0.0.1:8080](http://127.0.0.1:8080) |
| Log Streaming | kube-apiserver, kubelet RBAC |
mTLS HTTPS 10250 log streaming |
kubectl logs streams container stdout access entries |
| In-Container Exec | kubelet, containerd-shim, runc |
CRI process execution | Exec returns container binary output (nginx version: nginx/1.27.4) |
| NodePort Service | kube-proxy, iptables, CNI bridge, host routes |
DNAT & cross-node packet routing | Returns HTTP/1.1 200 OK via http://${NODE_NAME}:${NODE_PORT} |
12.6 Section Summary
You have verified that secret data encryption at rest, workload scheduling, control plane stream proxying, container log aggregation, execution hooks, and NodePort service network routing function end-to-end across your custom-built cluster.
13. Cleanup (docs/13-cleanup.md)
- Resource Decommissioning: Guidelines for tearing down the provisioned VM infrastructure.
- Wrap-up: Resetting the environment and finalizing the lab state.
Delete all virtual machines created for the exercise
pct destroy 901 --purge
pct destroy 902 --purge
pct destroy 903 --purge
pct destroy 904 --purge
High Level Summary
Why “Kubernetes the Hard Way”?
“Kubernetes the Hard Way” is the gold standard for understanding the mechanics of Kubernetes because it strips away all automation abstractions. There are no managed control planes (EKS/GKE), no automated cluster lifecycle tools (kubeadm, k3s), no operator-driven CNI installers, and no automatic TLS cert managers (cert-manager).
By building the cluster manually, you configure every layer by hand:
- Manual PKI Infrastructure: You generate a custom Certificate Authority (CA) using OpenSSL and create individual X.509 client/server certificates for every single component, mapping RBAC authorization directly through Certificate
Common Name(CN) andOrganization(O) fields. - Bare Binary Management: Every control plane process (
etcd,kube-apiserver,kube-controller-manager,kube-scheduler) and worker process (containerd,kubelet,kube-proxy) runs directly as a bare-metal Linux systemd service rather than inside static pods or containers. - Kernel & OS Level Networking: Container networking is built directly on Linux network bridges, CNI plugins, sysctl kernel tuning (
br-netfilter), and static OS routing tables (ip route), bypassing automated overlay networks (Flannel/Calico). - Explicit Encryption & Configuration: Secret encryption keys, API server authorization webhooks, and kubeconfig files are constructed and wired together line-by-line.
Hard Way vs. Automated & Managed Approaches
| Feature / Dimension | Managed Kubernetes (EKS / GKE / AKS) | Automated Installers (kubeadm / k3s) |
Kubernetes the Hard Way |
|---|---|---|---|
| Control Plane Deployment | Managed black-box service managed by Cloud Provider | Automated as static pods or lightweight binaries | Manual OS systemd services running raw binaries |
| PKI & Certificate Management | Handled automatically; hidden from operator | Generated automatically via CLI commands | Manual OpenSSL CA & mTLS key generation per component |
| Network & CNI Provisioning | Provider CNI (VPC-native) auto-configured | Deployed via YAML manifests (Flannel, Calico, Cilium) | Manual CNI bridge binaries, kernel modules, & ip route tables |
| Data Encryption at Rest | Cloud KMS integrated with 1-click toggle | Configurable via flag/manifest edits | Hand-crafted AES-CBC encryption configuration file for etcd |
| Component Authentication | Service Accounts & Provider IAM integration | Auto-generated Kubeconfig files | Hand-built Kubeconfigs with embedded mTLS certificates |
Chapter-by-Chapter Bootstrapping Chain
The setup relies on a strict sequential dependency chain where each lab produces the exact configuration, binary, or credential required by the next.
1. Prerequisites ──► 02. Jumpbox Setup ──► 03. Compute & DNS ──► 04. CA & TLS Certificates
│
2. etcd Cluster ◄── 06. Data Encryption ◄── 05. Kubeconfig Files ◄──────┘
│
▼
3. Control Plane ──► 09. Worker Nodes ──► 10. Remote kubectl ──► 11. Pod Routes ──► 12. Smoke Test
Chapter 01: Prerequisites
- What Happens: Defines the hardware and OS requirements—four Debian 12 (bookworm) virtual or physical instances (
jumpbox,server,node-0,node-1). - Dependency Linkage: Establishes the exact machine topology and resource boundaries (CPUs, RAM, Disk) needed before provisioning administrative tools and remote connections.
Chapter 02: Set Up The Jumpbox
- What Happens: Configures the
jumpboxas the central administrative workstation. Installs OS utilities (curl,openssl,git), clones the tutorial repo, downloads all component binaries (etcd,kube-apiserver,kubelet,containerd,kubectl, CNI plugins), and installskubectllocally. - Dependency Linkage: Centralizes binaries and tools on a single machine so they can be generated, staged, and distributed to cluster nodes via SCP without repeated internet downloads.
Chapter 03: Provisioning Compute Resources
- What Happens: Maps out node attributes in
machines.txt(IPs, FQDNs, Hostnames, Pod CIDRs), configures SSH keypairs for passwordlessrootaccess, sets system hostnames, and populates/etc/hostsacross all nodes. - Dependency Linkage: Provides reliable hostname resolution and passwordless execution across all machines, which is strictly required for generating TLS certificates with valid Subject Alternative Names (SANs) in the next step.
Chapter 04: Provisioning a CA and Generating TLS Certificates
- What Happens: Bootstraps a local PKI using OpenSSL. Creates a Root CA (
ca.crt/ca.key) and generates signed mTLS client/server certificates foradmin,node-0,node-1,kube-proxy,kube-scheduler,kube-controller-manager,kube-api-server, andservice-accounts. Distributes certificates to respective nodes. - Dependency Linkage: Establishes the trust foundation. These TLS certificates serve as component credentials to construct authenticated
kubeconfigfiles in Chapter 05.
Chapter 05: Generating Kubernetes Configuration Files for Authentication
- What Happens: Uses
kubectl configto embed the CA certificate and component-specific client keys intokubeconfigfiles fornode-0,node-1,kube-proxy,kube-controller-manager,kube-scheduler, andadmin. Distributes them to worker and control plane nodes. - Dependency Linkage: Translates raw TLS keys into structured API server authentication files. The components will use these configs to establish secure mTLS connections to
kube-apiserver.
Chapter 06: Generating the Data Encryption Config and Key
- What Happens: Generates a 32-byte AES random key and constructs
encryption-config.yamlspecifying theaescbcprovider to encrypt KubernetesSecretobjects at rest. SCPs the file to theserver. - Dependency Linkage: The API server requires this configuration file upon startup to handle
Secretobject creation, ensuring state data stored inetcd(Chapter 07) is encrypted.
Chapter 07: Bootstrapping the etcd Cluster
- What Happens: Installs
etcdandetcdctlon theservermachine, configures TLS keys, createsetcd.service, and starts the single-nodeetcdcluster. - Dependency Linkage:
etcdis the single source of truth and state storage for Kubernetes. The control plane (kube-apiserver) cannot start or function without a runningetcdservice listening on port2379.
Chapter 08: Bootstrapping the Kubernetes Control Plane
- What Happens: Installs
kube-apiserver,kube-controller-manager, andkube-scheduleronserveras systemd services. Starts all components and applies an RBAC ClusterRole (system:kube-apiserver-to-kubelet) permitting the API server to query Kubelet APIs for logs and exec commands. - Dependency Linkage: Brings the Kubernetes API server alive on port
6443. The control plane is now fully functional and waiting for worker nodes to register.
Chapter 09: Bootstrapping the Kubernetes Worker Nodes
- What Happens: Installs container runtimes (
runc,containerd), CNI plugins,kubelet, andkube-proxyonnode-0andnode-1. Configures CNI bridge networks (10-bridge.conf), enables kernel modules (br-netfilter, sysctlbridge-nf-call-iptables), and starts worker systemd services. - Dependency Linkage: Kubelets read their respective node
kubeconfigfiles and register withkube-apiserver. The cluster now shows nodes inReadystatus, but cross-node pod routing is not yet configured.
Chapter 10: Configuring kubectl for Remote Access
- What Happens: Configures
~/.kube/configon thejumpboxusing theadmincredentials to point to[https://server.kubernetes.local:6443](https://server.kubernetes.local:6443). - Dependency Linkage: Allows cluster administrators to run
kubectlcommands remotely from thejumpboxwithout SSHing directly into theservercontrol plane node.
Chapter 11: Provisioning Pod Network Routes
- What Happens: Sets up static IP routing entries (
ip route add) on each node to route traffic destined for another node’s Pod CIDR (10.200.0.0/24fornode-0,10.200.1.0/24fornode-1) via that node’s internal network IP address. - Dependency Linkage: Solves cross-node pod-to-pod networking at the Linux OS layer, enabling flat network communication required for multi-node deployments and services.
Chapter 12: Smoke Test
- What Happens: Executes end-to-end operational validation:
- Creates a
Secretand inspects raw bytes inetcdusingetcdctlto verifyk8s:enc:aescbc:v1:key1encryption. - Deploys an Nginx workload (
kubectl create deployment). - Tests port forwarding (
kubectl port-forward). - Fetches pod logs (
kubectl logs) and executes interactive commands (kubectl exec). - Exposes the deployment as a
NodePortservice and verifies external HTTP access.
- Dependency Linkage: Validates that all bootstrapped layers—PKI, Authentication, Storage Encryption, Control Plane, Worker Runtimes, CNI, and OS Routing—work harmoniously as a healthy cluster.
Chapter 13: Cleaning Up
- What Happens: Teardown phase to delete all virtual machines and clean up local configurations.
Final Cluster Architecture & Artifact Placement
Node & Network Topology
- Admin Jumpbox (
jumpbox): Manages cluster remotely viakubectlpointing toserver.kubernetes.local:6443. - Control Plane (
server): Runsetcd(ports 2379/2380),kube-apiserver(port 6443),kube-controller-manager, andkube-scheduler. - Worker Node 0 (
node-0): Runscontainerd,kubelet(port 10250),kube-proxy, CNI Bridge (10.200.0.0/24Pod CIDR). - Worker Node 1 (
node-1): Runscontainerd,kubelet(port 10250),kube-proxy, CNI Bridge (10.200.1.0/24Pod CIDR).
flowchart TB
subgraph Jumpbox["Jumpbox Admin Host"]
kubectl["kubectl CLI"]
pki["Root CA & Cert Generator"]
end
subgraph ControlPlane["Control Plane Server (server)"]
direction TB
apiserver["kube-apiserver<br/>(Port 6443)"]
etcd["etcd Key-Value Store<br/>(Ports 2379 / 2380)"]
cm["kube-controller-manager"]
sched["kube-scheduler"]
cm -->|mTLS| apiserver
sched -->|mTLS| apiserver
apiserver <-->|mTLS / Port 2379| etcd
end
subgraph Node0["Worker Node 0 (node-0)"]
kubelet0["kubelet<br/>(Port 10250)"]
kproxy0["kube-proxy"]
cruntime0["containerd / runc"]
cni0["CNI Bridge<br/>(Pod CIDR: 10.200.0.0/24)"]
kubelet0 --> cruntime0
cruntime0 --> cni0
end
subgraph Node1["Worker Node 1 (node-1)"]
kubelet1["kubelet<br/>(Port 10250)"]
kproxy1["kube-proxy"]
cruntime1["containerd / runc"]
cni1["CNI Bridge<br/>(Pod CIDR: 10.200.1.0/24)"]
kubelet1 --> cruntime1
cruntime1 --> cni1
end
%% External & Admin Connections
kubectl -->|mTLS / Port 6443| apiserver
%% Worker Node Control Connections
apiserver <-->|API / Metrics / Exec| kubelet0
apiserver <-->|API / Metrics / Exec| kubelet1
kproxy0 -->|Watch API| apiserver
kproxy1 -->|Watch API| apiserver
%% Cross-Node Pod Network Routes
cni0 <===>|Static OS Kernel Routing 'ip route'| cni1
Comprehensive File & Artifact Distribution Map
| Machine | Component Path / Directory | Deployed Binaries / Files | Description & Role |
|---|---|---|---|
jumpbox |
/usr/local/bin/ |
kubectl |
Local cluster administration CLI tool. |
~/kubernetes-the-hard-way/ |
ca.crt, ca.key, ca.conf |
Root Certificate Authority files. | |
~/.kube/config |
Admin kubeconfig |
Embedded admin TLS certs configured to manage cluster remotely. | |
server |
/usr/local/bin/ |
etcd, etcdctl, kube-apiserver, kube-controller-manager, kube-scheduler, kubectl |
Core control plane executable binaries. |
/etc/etcd/ |
ca.crt, kube-api-server.crt, kube-api-server.key |
TLS certificates securing etcd client/peer communication. |
|
/var/lib/etcd/ |
member/ data directory |
Persistent state database directory for cluster object storage. | |
/var/lib/kubernetes/ |
ca.crt, ca.key, kube-api-server.crt, kube-api-server.key, service-accounts.crt, service-accounts.key, encryption-config.yaml, kube-controller-manager.kubeconfig, kube-scheduler.kubeconfig |
Control plane authentication keys, service account signing pairs, data encryption configuration, and component kubeconfigs. | |
/etc/kubernetes/config/ |
kube-scheduler.yaml |
Scheduler component configuration file. | |
/etc/systemd/system/ |
etcd.service, kube-apiserver.service, kube-controller-manager.service, kube-scheduler.service |
Systemd unit service files for process lifecycle control. | |
**node-0 & node-1** |
/usr/local/bin/ |
crictl, kube-proxy, kubelet, runc |
Worker runtime binaries and CRI management tools. |
/bin/ |
containerd, containerd-shim-runc-v2, containerd-stress |
Container execution and shim management binaries. | |
/opt/cni/bin/ |
bridge, loopback, host-local, etc. |
CNI network plugin binaries for pod interface creation. | |
/etc/cni/net.d/ |
10-bridge.conf, 99-loopback.conf |
CNI network subnet configuration mapped per node (10.200.x.0/24). |
|
/etc/containerd/ |
config.toml |
Runtime configuration for containerd daemon. | |
/var/lib/kubelet/ |
ca.crt, kubelet.crt, kubelet.key, kubeconfig, kubelet-config.yaml |
Kubelet mTLS certificates, node identity configuration, and API server access kubeconfig. | |
/var/lib/kube-proxy/ |
kubeconfig, kube-proxy-config.yaml |
Kube-proxy network routing configuration and authentication file. | |
/etc/systemd/system/ |
containerd.service, kubelet.service, kube-proxy.service |
Systemd service files managing node daemons. |
Conclusion & Key Takeaways
Building Kubernetes “the hard way” demystifies the black box of cloud-native infrastructure. Through this lab, we explicitly decoupled and configured every core architectural primitive:
- Explicit Identity & PKI: Certificate Subject fields (
CNandO) are directly tied to Kubernetes RBAC authorization modes (such assystem:nodesandsystem:node:<node-name>). - State & Control Plane Separation:
etcdmanages cluster state, while systemd units forkube-apiserver,kube-controller-manager, andkube-schedulerdrive declarative state convergence independently. - Bare-Metal Networking: Without an overlay network (like Calico or Cilium), Pod-to-Pod cross-node communication relies directly on layer-3 static routing tables paired with local CNI bridge interfaces (
br-netfilter). - Data Protection: Secrets are not inherently secure in
etcdunless explicit provider configurations (encryption-config.yaml) encrypt payloads at rest.
Mastering these components provides the foundational troubleshooting skills required to debug complex production cluster failures, network plugin issues, and control plane degradation.