Domain 5Containers & RuntimesINTERMEDIATE

Mastering Docker: Architecture, Administration, and Production Troubleshooting for Linux Engineers

A comprehensive operational handbook covering Docker architecture, Linux kernel namespaces, cgroups, OverlayFS internals, CLI commands, multi-stage Dockerfiles, Compose workflows, and step-by-step production troubleshooting.

Domain: Domain 5: Containers & Runtimes•Updated: 2026-11-03
#linux#sysadmin#docker#containers#sre

1. Overview

Docker standardizes how applications are built, shipped, and executed across heterogeneous environments. By encapsulating an application along with its dependencies, runtime binaries, system libraries, and default configurations into an immutable image format, Docker guarantees that an application runs identically on a developer’s laptop, a continuous integration (CI) test runner, or a multi-node cloud production cluster. This consistency eliminates the classic “works on my machine” deployment failure caused by subtle environment drift between development and production Linux servers.

In modern Site Reliability Engineering (SRE) and Linux system administration, Docker serves several critical roles:

  • Microservice Containerization: Encapsulating isolated application components with independent dependency stacks.
  • Local Development Stacks: Provisioning reproducible local environments containing databases, caches, and web services matching production parameters.
  • CI/CD Build & Test Environments: Running ephemeral build pipelines and integration tests inside clean, disposable containers.
  • Kubernetes Runtime Foundation: Operating as the underlying engine or supplying core runtime components (such as containerd) managed via the Container Runtime Interface (CRI).

Docker vs. Hardware Virtualization (Virtual Machines)

A fundamental architectural distinction exists between hypervisor-based Virtual Machines (VMs) and kernel-level Linux containers.

+---------------------------------+     +---------------------------------+
|  App A   |  App B   |  App C    |     |  App A   |  App B   |  App C    |
+----------+----------+-----------+     +----------+----------+-----------+
| Guest OS | Guest OS | Guest OS  |     | Bins/Libs| Bins/Libs| Bins/Libs |
+----------+----------+-----------+     +---------------------------------+
|          Hypervisor             |     |         Docker Engine           |
+---------------------------------+     +---------------------------------+
|        Host Linux OS            |     |        Host Linux Kernel        |
+---------------------------------+     +---------------------------------+
|       Physical Hardware         |     |       Physical Hardware         |
+---------------------------------+     +---------------------------------+
     Hypervisor-Based VMs                      Docker Containers

Containers are not VMs; they share the host Linux kernel directly. While hypervisors virtualize physical hardware and require a dedicated guest operating system for every virtual instance, containers isolate user-space processes directly on the host kernel. Consequently, containers spin up in milliseconds, consume megabytes of RAM instead of gigabytes, and impose minimal I/O and CPU overhead compared to full hardware virtualization.

However, sharing the host kernel implies that container security relies on kernel-level isolation primitives. While strong enough for standard multi-tenant workloads, container isolation is not equivalent to hardware-enforced hypervisor boundaries across all threat models.

Simple Operational Example

To demonstrate Docker’s functional model, launching an isolated, published NGINX web server requires a single command:

docker run -d --name web-server -p 8080:80 nginx:1.25

When executed, Docker pulls the nginx:1.25 image from the registry (if not already cached), creates a network namespace, publishes port 8080 on the host to route to port 80 inside the container, attaches a writable filesystem layer, and executes NGINX as PID 1.


2. Architecture and Mental Model

To effectively operate and troubleshoot Docker in enterprise environments, system administrators must understand the execution stack and the underlying Linux kernel mechanisms that enforce container isolation and resource allocation.

The Docker Execution Stack

Modern Docker relies on a layered, modular architecture rather than a single monolithic binary. When an administrator executes a container, request execution flows through several specialized sub-components:

graph TD
    A[Docker CLI] -->|REST API over Unix Socket /var/run/docker.sock| B[Docker Daemon: dockerd]
    B -->|gRPC Requests| C[containerd]
    C -->|Spawns OCI Shims| D[runc]
    D -->|Configures Kernel Features| E[Linux Kernel]
    E -->|Namespaces| F[Process & Network Isolation]
    E -->|Cgroups| G[CPU & Memory Controls]
    E -->|OverlayFS| H[Stacked Filesystem Layers]
  1. Docker CLI (docker): The client binary that parses user commands and formats REST API requests.
  2. Docker Daemon (dockerd): The persistent background service listening on /var/run/docker.sock. It handles high-level user API calls, image management, volume orchestration, and network plumbing.
  3. containerd: An industry-standard, high-performance container daemon. It manages the complete container lifecycle—including image pulling, snapshot management, container execution, and supervisor tasks. (Note: Kubernetes nodes communicate directly with containerd or CRI-O via the CRI API, bypassing dockerd entirely).
  4. runc: The low-level OCI (Open Container Initiative) compliant reference runtime. It interacts directly with the Linux kernel to assemble namespaces and cgroups, starts the PID 1 container process, and then exits, leaving a lightweight shim process to supervise the container.

Linux Kernel Isolation Primitives

A container is not an abstract virtual machine; it is a standard Linux process (or process tree) restricted by kernel primitives.

1. Linux Namespaces (Resource Isolation)

Namespaces restrict what a process can see by providing isolated views of global system resources:

Namespace Subsystem Isolated Operational Impact & Function
PID Process IDs Processes inside have independent PID numbering starting at PID 1; host PIDs are hidden.
NET Networking Container receives dedicated virtual interfaces, loopback (lo), routing tables, and IP addresses.
MNT Mount points Provides a dedicated filesystem mount table (/ inside container is isolated from host root).
UTS Hostname & Domain Allows container to maintain an independent system hostname.
IPC Inter-Process Comm. Isolates System V IPC objects and POSIX message queues from other containers.
USER UID / GID Mappings Maps inside container root (UID 0) to an unprivileged host UID for rootless operation.

2. Control Groups / cgroups (Resource Constraints)

Control Groups enforce how much system resource a process can consume. The kernel measures and limits usage across key subsystems:

  • Memory: Sets hard thresholds (--memory). Exceeding allocated RAM triggers the Linux Out-Of-Memory (OOM) killer, terminating the container process (Exit Code 137).
  • CPU: Allocates CPU shares (--cpu-shares) or hard execution time caps (--cpus).
  • PIDs: Prevents fork-bomb attacks by capping total allowable process threads (--pids-limit).
  • BlkIO: Limits read/write bandwidth and I/O operations per second (IOPS) on block devices.

3. OverlayFS (Union Filesystem Storage)

Docker constructs container filesystems using stacked, read-only image layers topped by a thin, read-write container layer. On modern Linux distributions, this is implemented using the overlay2 storage driver located at /var/lib/docker/overlay2/.

+-------------------------------------------------+
| Writable Container Layer (Read-Write)           | <-- File modifications, whiteouts
+-------------------------------------------------+
| Layer N: App Code & Configs (Read-Only)         | <-- Image Layer
+-------------------------------------------------+
| Layer 1: Base Dependencies (Read-Only)          | <-- Image Layer
+-------------------------------------------------+
| Layer 0: Base OS (Ubuntu/Alpine) (Read-Only)    | <-- Image Layer
+-------------------------------------------------+
  • Copy-on-Write (CoW): When a process inside a container modifies an existing file originating from a lower read-only layer, OverlayFS copies the file up to the thin writable layer before applying modifications.
  • Whiteout Files: Deleting a file originating from an image layer creates a special “whiteout” character device file in the writable layer. This masks the lower layer file without deleting it from the underlying immutable image layer.
  • Ephemeral Nature: Deleting a container via docker rm permanently destroys its writable layer and all unmounted data changes.

3. Core Concepts and Terminology

Understanding key Docker terminology ensures precise communication and prevents administrative errors during deployment and maintenance.

Image vs. Container

  • Docker Image: A read-only template consisting of stacked filesystem layers and runtime metadata (entrypoints, environment variables, default commands). Built from a Dockerfile or pulled from a container registry.
  • Docker Container: A running or stopped stateful instance of an image. It combines the read-only image layers with an isolated process namespace, cgroup resource limits, network configuration, and a thin read-write filesystem layer.

Image Tags and Registries

Images are referenced using the canonical syntax [registry_host[:port]/][repository/]image_name:tag.

  • Tagging Strategy: Omitting a tag causes Docker to default to :latest. Production deployments must never rely on :latest because it leads to unpredictable image drift. Explicit, immutable version tags (e.g., nginx:1.25.3-alpine) must be pinned.

Storage Types

  • Bind Mounts: Maps an absolute directory path on the host system directly into a container path (-v /opt/configs:/etc/app). Subject to host permissions and path dependencies.
  • Named Volumes: Docker-managed persistent storage allocated under /var/lib/docker/volumes/. Preferred for production databases as they decouple data lifecycles from container lifecycles.
  • tmpfs Mounts: Ephemeral storage backed exclusively by host RAM. Used for high-speed temporary caching or keeping sensitive secrets out of persistent disk storage.

Key Glossary

Term Definition
Base Image The foundational image defined in the initial FROM directive of a Dockerfile.
Build Context The set of local files and directories sent from the Docker CLI to the daemon during an image build.
dockerinit / Tini A tiny init process used inside containers to harvest zombie processes and forward signals properly.
Dangling Image An unreferenced image layer marked as <none>:<none> resulting from building an image with an existing tag.
DOCKER_BUILDKIT The modern, high-performance image build engine featuring parallel stage execution and secret mounting.

4. Installation and Version Management

Prerequisites and Dependencies

Docker requires a 64-bit Linux distribution with kernel version 3.10 or higher (kernel 5.x+ recommended for full overlay2 and cgroups v2 capabilities).

Installing Docker CE on Ubuntu / Debian

Follow the official Docker repository installation workflow to guarantee receipt of security updates:

# 1. Remove legacy packages
sudo apt-get remove -y docker docker-engine docker.io containerd runc

# 2. Install repository prerequisites
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg lsb-release

# 3. Add Docker's official GPG key
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg

# 4. Set up the stable repository
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# 5. Install Docker Engine, containerd, and Docker Compose CLI plugin
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Installing Docker CE on RHEL / Rocky Linux / AlmaLinux

# 1. Install yum-utils
sudo dnf install -y yum-utils

# 2. Add the official repository
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

# 3. Install packages
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

# 4. Enable and start daemon
sudo systemctl enable --now docker

Validating Installation and Identifying Versions

Inspect installed version details across CLI and Daemon engines:

# Display client and server engine versions
docker version

# Display system-wide information including storage driver, cgroup version, and container counts
docker info

5. Configuration Files and Directory Structure

Docker’s operation on a Linux server relies on specific configuration files, sockets, and storage directories.

File and Directory Reference

File / Directory Path Purpose Important Details
/etc/docker/daemon.json Main daemon configuration file. Controls network bridges, storage drivers, logging options, DNS, and insecure registries.
/var/lib/docker/ Primary daemon storage directory. Holds images, overlay layers, volumes, and container states. Requires dedicated disk space.
/var/lib/docker/overlay2/ Storage location for OverlayFS image and container layers. Contains diff, work, and merged directories per layer.
/var/lib/docker/volumes/ Default path for Docker named volumes. Stores persistent container volume data managed by Docker.
/var/run/docker.sock Unix domain socket used for API communication. Default interface between CLI and dockerd. Secured by docker group permissions.
~/.docker/config.json Per-user CLI configuration file. Stores registry authentication credentials (e.g., docker login tokens) and CLI formatting settings.
/lib/systemd/system/docker.service Default systemd unit configuration. Defines process execution flags, file descriptor limits (LimitNOFILE=1048576), and startup dependencies.

Production /etc/docker/daemon.json Reference

Below is a production-hardened /etc/docker/daemon.json configuration specifying log rotation, cgroups driver management, and storage engine parameters:

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "50m",
    "max-file": "3"
  },
  "storage-driver": "overlay2",
  "dns": ["8.8.8.8", "1.1.1.1"],
  "live-restore": true,
  "userland-proxy": false,
  "no-new-privileges": true
}
  • live-restore: Keeps containers running during daemon updates or crashes, reducing service downtime.
  • log-opts: Prevents container log files from consuming all host disk space.
  • userland-proxy: false: Disables the inefficient docker-proxy process, routing container traffic directly using Linux iptables NAT rules.

After modifying /etc/docker/daemon.json, validate syntax and reload the service:

# Validate JSON syntax
jq . /etc/docker/daemon.json

# Reload systemd and apply configuration to dockerd
sudo systemctl reload docker

6. Service Management and Common Operations

Managing the Docker background daemon relies on systemd operational commands.

Daemon Management Commands

# Check detailed daemon operational status
sudo systemctl status docker

# Start the Docker daemon
sudo systemctl start docker

# Stop the Docker daemon
sudo systemctl stop docker

# Restart the daemon (briefly interrupts API connectivity unless live-restore is enabled)
sudo systemctl restart docker

# Enable automatic startup at host boot
sudo systemctl enable docker

Container Lifecycle State Changes

Understanding the distinction between lifecycle operations prevents unintended downtime:

  • Container Restart (docker restart <container>): Sends a SIGTERM signal to PID 1 inside the container, waits 10 seconds for graceful cleanup, and sends SIGKILL if the process has not exited. It then recreates the process namespace and restarts PID 1.
  • Application Reload: For applications supporting live reloading (such as NGINX or Apache), configuration updates should be reloaded without stopping the container namespace by executing the reload signal directly inside PID 1:
docker exec web-server nginx -s reload

7. Common Administrative Tasks and Workflows

Workflow 1: Building Optimized Multi-Stage Docker Images

Unoptimized Dockerfiles generate massive images containing build tools (compilers, git, dev libraries) that introduce security vulnerabilities and slow deployment speeds. Multi-stage builds solve this by separating the build environment from the runtime image.

Practical Multi-Stage Dockerfile (Go Application)

# Stage 1: Build Environment
FROM golang:1.22-alpine AS builder

# Set workspace
WORKDIR /app

# Optimize layer caching by copying dependency manifests first
COPY go.mod go.sum ./
RUN go mod download

# Copy source code and compile static binary
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-w -s" -o /app/server .

# Stage 2: Minimal Production Runtime
FROM alpine:3.19

# Security: Create non-root user and group
RUN addgroup -S appgroup && adduser -S appuser -G appgroup

WORKDIR /app

# Copy binary from builder stage
COPY --from=builder /app/server /app/server

# Adjust ownership
RUN chown -R appuser:appgroup /app

# Expose port and set security context
USER appuser
EXPOSE 8080

# Default execution entrypoint (exec form)
ENTRYPOINT ["/app/server"]

Workflow 2: Orchestrating Services with Docker Compose (v2)

Docker Compose defines multi-container architecture stacks using a declarative docker-compose.yml file.

Production docker-compose.yml Example

version: '3.8'

services:
  web:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8080:8080"
    environment:
      - DB_HOST=db
      - DB_PORT=5432
    depends_on:
      db:
        condition: service_healthy
    restart: always
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 512M

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: appdb
      POSTGRES_USER: dbuser
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U dbuser -d appdb"]
      interval: 5s
      timeout: 5s
      retries: 5
    restart: always

volumes:
  pgdata:
    driver: local

secrets:
  db_password:
    file: ./db_password.txt

Operational Commands for Compose

# Start stack in detached mode and rebuild changed images
docker compose up -d --build

# Inspect status of running services
docker compose ps

# View combined, colorized logs across all services in real time
docker compose logs -f --tail 100

# Tear down containers, networks, and ephemeral resources (preserving named volumes)
docker compose down

# Tear down containers AND remove named volumes (Destructive!)
docker compose down -v

8. CLI Command Reference and Cheatsheet

Image Management Commands

Command Purpose Key Flags & Arguments Practical Example Explanation
docker pull Downloads image from registry. -a (all tags) docker pull nginx:1.25-alpine Fetches specified tag to local cache.
docker images Lists cached local images. -a (all), -q (quiet IDs) docker images --filter "dangling=true" Lists images; filter helps spot untagged layers.
docker build Builds image from Dockerfile. -t (tag), --no-cache, -f (file) docker build -t app:1.0 -f Dockerfile . Compiles layers using current directory as context.
docker rmi Removes local images. -f (force) docker rmi app:1.0 Deletes image metadata and unused layers.
docker history Displays layer history of an image. --no-trunc docker history --no-trunc app:1.0 Shows layer sizes and Dockerfile execution steps.

Container Run and Lifecycle Commands

Command Purpose Key Flags & Arguments Practical Example Explanation
docker run Creates and starts a container. -d (detach), -p (port mapping), -v (volume mount), --name docker run -d --name web -p 80:80 -v /data:/app/data nginx:1.25 Instantiates isolated container running in background.
docker ps Lists active containers. -a (all including stopped), -q (IDs) docker ps -a --filter "status=exited" Displays runtime state, published ports, names.
docker stop Gracefully stops container. -t (timeout in seconds) docker stop -t 30 web Sends SIGTERM, waits timeout, then sends SIGKILL.
docker kill Forcefully kills container. -s (signal name/number) docker kill -s SIGKILL web Sends immediate raw kernel signal to PID 1.
docker rm Removes stopped container. -f (force running container) docker rm -f web Deletes container metadata and thin writable layer.

Diagnostics, Logs, and Inspection

Command Purpose Key Flags & Arguments Practical Example Explanation
docker logs Fetches container log output. -f (follow), --tail N, --since docker logs -f --tail 100 --since 10m web Tails stdout/stderr streams from container PID 1.
docker exec Runs command inside running container. -i (interactive), -t (tty), -u (user) docker exec -it -u root web /bin/sh Attaches process to existing container namespaces.
docker inspect Returns low-level JSON configuration. -f / --format (Go template) docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' web Queries exact IP address, mounts, environment, and state.
docker stats Streams live resource utilization. --no-stream docker stats web Reads cgroup metrics for CPU, RAM, Network, and I/O.
docker top Displays processes running inside container. Host ps flags docker top web aux Translates container PIDs to host process table.

9. Logs, Monitoring and Diagnostics

Inspecting Container Logs

Docker captures standard output (stdout) and standard error (stderr) emitted by PID 1 inside the container.

# Follow live log stream for a specific container
docker logs -f web-server

# View last 50 lines with timestamps
docker logs --tail 50 -t web-server

# View logs emitted since a relative time threshold
docker logs --since 15m web-server

Log Drivers and Rotation

By default, Docker uses the json-file log driver, storing raw log output in /var/lib/docker/containers/<container-id>/<container-id>-json.log. Unmanaged logging can quickly fill disk space.

Always configure global rotation via /etc/docker/daemon.json (as shown in Section 5) or apply per-container flags:

docker run -d --log-driver json-file --log-opt max-size=10m --log-opt max-file=3 nginx

Resource Utilization and Cgroup Inspection

# Stream real-time resource utilization across all active containers
docker stats

# Inspect processes running inside a container from host perspective
docker top web-server

To inspect raw cgroup limits directly via the kernel interface (cgroups v2 example):

# Query memory cap enforced by cgroup driver
cat /sys/fs/cgroup/docker/<container-id>/memory.max

# Check current memory usage
cat /sys/fs/cgroup/docker/<container-id>/memory.current

10. Troubleshooting Guide and Common Failure Scenarios

This section details systematic investigation workflows for common Docker operational failures.


Scenario 1: Cannot Connect to the Docker Daemon

Symptoms

User receives error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? or Permission denied while trying to connect to the Docker daemon socket.

Possible Causes

  1. The docker systemd service is stopped or crashed.
  2. The executing user lacks read/write permissions on /var/run/docker.sock.

Investigation and Diagnosis

  1. Check the systemd service state:
systemctl status docker
  1. Verify ownership and permissions on the Unix socket:
ls -la /var/run/docker.sock

Expected Output: srw-rw---- 1 root docker 0 Nov 3 10:00 /var/run/docker.sock 3. Verify user group membership:

id $USER

Resolution

  • If daemon is down, start it:
sudo systemctl start docker
  • If permissions issue, append user to docker group and re-authenticate:
sudo usermod -aG docker $USER
newgrp docker

Scenario 2: Container Exits Immediately Upon Startup

Symptoms

Executing docker run returns a container ID, but docker ps shows no running container. docker ps -a reveals state Exited (0) or Exited (1).

CONTAINER ID   IMAGE     COMMAND                  CREATED          STATUS                      PORTS     NAMES
a1b2c3d4e5f6   myapp     "/docker-entrypoint…"   5 seconds ago    Exited (1) 4 seconds ago              my-app

Possible Causes

  1. PID 1 Exited: Containers run only as long as PID 1 remains active. If the command runs in background mode (e.g., nginx without daemon off;) or completes an execution script, PID 1 exits immediately.
  2. Missing Configuration / Crash: Application threw an unhandled exception or failed to read environment flags on launch.

Investigation and Diagnosis

  1. Inspect container exit status and logs:
docker inspect my-app | jq -r '.[0].State | {Status: .Status, ExitCode: .ExitCode, Error: .Error}'
docker logs --tail 50 my-app
  1. Override container entrypoint with an interactive shell to debug image contents:
docker run --rm -it --entrypoint /bin/sh myapp

Inside shell, verify configuration files, environment variables, and binary execution manually.

Resolution

Ensure the application command runs in the foreground. For example, in NGINX Dockerfiles, pass daemon off;:

CMD ["nginx", "-g", "daemon off;"]

Scenario 3: Interpreting Container Exit Codes

When diagnosing exited containers via docker inspect CONTAINER | jq '.[0].State.ExitCode', use this reference guide to determine root cause:

Exit Code Signal / Meaning Root Cause & Diagnostic Workflow
0 Success / Clean Exit Container completed its execution normally (common for batch jobs or improperly backgrounded services).
1 Application Error Generic application crash. Check application stack traces via docker logs.
126 Command Cannot Execute Entrypoint/CMD file is not marked executable (chmod +x missing) or architecture mismatch.
127 Command Not Found Executable specified in ENTRYPOINT/CMD does not exist in the container $PATH.
137 SIGKILL (128 + 9) Process forcibly killed by Linux kernel. Check if OOMKilled: true via docker inspect or check host `dmesg
143 SIGTERM (128 + 15) Process gracefully terminated via docker stop.

Scenario 4: Host Port Binding Failure (bind: address already in use)

Symptoms

Executing docker run or docker compose up fails with: driver failed programming external connectivity on endpoint web: Error starting userland proxy: listen tcp4 0.0.0.0:8080: bind: address already in use.

Possible Causes

Another host process or Docker container is already listening on published host port 8080.

Investigation and Diagnosis

  1. Identify the process holding host port 8080:
sudo ss -tulpn | grep :8080
# OR
sudo lsof -i :8080
  1. Identify conflicting Docker containers:
docker ps --format "table {{.Names}}\t{{.Ports}}" | grep 8080

Resolution

  • Stop host service holding the port (sudo systemctl stop apache2 or docker stop <conflicting-container>).
  • Or re-map host port binding syntax (-p <host_port>:<container_port>):
docker run -d -p 8081:8080 myapp

Scenario 5: Container OOM (Out Of Memory) Killed

Symptoms

Container suddenly restarts or stops. docker ps -a shows exit code 137.

Investigation and Diagnosis

  1. Query container OOM state via inspect:
docker inspect web-app | jq -r '.[0].State | {OOMKilled: .OOMKilled, ExitCode: .ExitCode}'

*If OOMKilled: true*, the container breached its configured --memory limit. 2. Search kernel log buffer for OOM events:

sudo dmesg -T | grep -i "oom-killer"

Resolution

  • Increase memory ceiling in Dockerfile/Compose (memory: 1G).
  • Investigate memory leaks inside application code using runtime profiling tools.

Scenario 6: Filesystem Exhaustion (No space left on device)

Symptoms

Docker builds fail, containers fail to launch or write logs, emitting error: No space left on device.

Investigation and Diagnosis

  1. Check disk utilization across system filesystems:
df -h /var/lib/docker
  1. Run Docker disk usage breakdown:
docker system df -v

Identifies space consumed individually by images, containers, named volumes, and build caches.

Resolution (Safe Pruning Sequence)

Execute cleanup operations sequentially from safest to most aggressive:

# 1. Remove exited/stopped containers
docker container prune -f

# 2. Remove dangling image layers (<none>:<none>)
docker image prune -f

# 3. Remove unused BuildKit cache
docker builder prune -f

# 4. Remove all images not attached to an active container
docker image prune -a -f

# 5. WARNING: Remove unattached named volumes (Destructive to offline data!)
docker volume prune -f

11. Security and Best Practices

Securing containers requires implementing defense-in-depth principles across the host OS, daemon configuration, and Dockerfiles.

Security Controls Checklist

  1. Never Run Containers as Root: Default containers run as root inside the container namespace, exposing host risk if kernel escapes occur. Specify non-root users inside Dockerfiles:
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser
  1. Enforce Read-Only Root Filesystems: Prevent attackers from modifying application code or downloading binary payloads by mounting container root as read-only, using temporary tmpfs mounts for runtime file activity:
docker run -d --read-only --tmpfs /tmp --tmpfs /run nginx
  1. Secure the Docker Unix Socket: Access to /var/run/docker.sock grants effective root access to the host machine. Never expose /var/run/docker.sock to application containers unless absolutely required, and never expose it directly over an unencrypted TCP port.
  2. Resource Quotas (Prevent DoS Attacks): Always specify memory, CPU, and process ceiling limits on shared production hosts to block rogue process consumption:
docker run -d --memory="512m" --cpus="1.5" --pids-limit=100 myapp
  1. BuildKit Secret Mounting: Do not bake sensitive API credentials into image layers using ENV or ARG directives. Use BuildKit secret mounts, which expose credentials temporarily during build time without leaving remnants in image layers:
RUN --mount=type=secret,id=api_key \
    API_KEY=$(cat /run/secrets/api_key) ./build_script.sh

12. Performance and Resource Optimisation

Optimizing Docker environments focuses on minimizing image layer sizes, maximizing layer cache reusability, and reducing system runtime resource overhead.

Maximizing Layer Cache Efficiency

Docker executes Dockerfile instructions sequentially, caching the intermediate image layer for each instruction. If an instruction changes, Docker invalidates that layer’s cache and rebuilds all subsequent layers.

Anti-Pattern (Invalidates Cache Frequently)

FROM node:20-alpine
WORKDIR /app
# BAD: Copying all app files FIRST invalidates package install cache on every single line edit
COPY . .
RUN npm install
CMD ["node", "server.js"]

Optimized Pattern (Maximizes Cache Hits)

FROM node:20-alpine
WORKDIR /app
# GOOD: Copy dependency manifests ONLY; layer stays cached unless package.json changes
COPY package*.json ./
RUN npm install
# Copy remaining source code after heavy dependencies are installed
COPY . .
CMD ["node", "server.js"]

Layer Count Optimization

Combine package updating, installation, and cleanup tasks into single RUN instructions to prevent apt metadata cache from bloating image layers:

RUN apt-get update && apt-get install -y --no-install-recommends \
    curl \
    ca-certificates \
 && rm -rf /var/lib/apt/lists/*

13. Backup, Recovery and Maintenance

Backing Up Docker Named Volumes

Named volumes stored in /var/lib/docker/volumes/ should be backed up using disposable helper containers that mount the target volume alongside a host backup directory:

# Backup volume 'pgdata' to a compressed tar archive on the host
docker run --rm \
  -v pgdata:/volume-data:ro \
  -v $(pwd)/backups:/backup \
  alpine tar -czf /backup/pgdata_backup_$(date +%Y%m%d).tar.gz -C /volume-data .

Restoring Docker Named Volumes

# Create target volume
docker volume create pgdata_restored

# Restore archive contents into new volume
docker run --rm \
  -v pgdata_restored:/volume-data \
  -v $(pwd)/backups:/backup \
  alpine sh -c "cd /volume-data && tar -xzf /backup/pgdata_backup_20261103.tar.gz"

Exporting and Importing Images for Air-Gapped Environments

# Save image to tarball archive
docker save -o nginx_1.25.tar nginx:1.25

# Load image tarball archive into air-gapped system registry
docker load -i nginx_1.25.tar

14. Automation and Integration

Integrating Docker workflows into standard Linux maintenance routines guarantees host reliability.

Systemd Timer for Automated Docker Disk Cleanup

Create a daily systemd service and timer to automatically prune unused dangling resources.

/etc/systemd/system/docker-cleanup.service

[Unit]
Description=Automated Docker System Cleanup
After=docker.service
Requires=docker.service

[Service]
Type=oneshot
ExecStart=/usr/bin/docker system prune -f --filter "until=168h"

/etc/systemd/system/docker-cleanup.timer

[Unit]
Description=Run Docker Cleanup Daily

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Enable timer:

sudo systemctl daemon-reload
sudo systemctl enable --now docker-cleanup.timer

15. Practical Labs and Exercises

Lab 1: Multi-Stage Container Building

Objective: Build a production-ready, non-root microservice image under 20MB.

  1. Create directory lab1 and define a minimal Go service main.go:
package main
import ("fmt"; "net/http")
func main() {
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, "Lab 1 Operational")
    })
    http.ListenAndServe(":8080", nil)
}
  1. Write a multi-stage Dockerfile using golang:alpine as builder and scratch or alpine as production target. Ensure non-root execution.
  2. Build image: docker build -t lab1:v1 .
  3. Validate binary size and user execution:
docker images lab1:v1
docker run -d -p 8080:8080 --name lab1-test lab1:v1
docker exec lab1-test id

Expected Outcome: User UID is non-zero, image size is drastically smaller than standard base build.


Lab 2: Troubleshooting a Broken Startup Container

Objective: Diagnose and repair an application container failing on boot.

  1. Launch test broken container:
docker run -d --name broken-app nginx:alpine sh -c "nginx -g 'daemon off;'"
  1. Wait 10 seconds and check container state:
docker ps -a --filter "name=broken-app"
  1. Perform investigation:
  • Read exit code via docker inspect.
  • Check container log output via docker logs.
  • Identify missing options or configuration errors.
  1. Clean up broken instance:
docker rm broken-app

16. Quick Reference and Cheat Sheet

Important Files and Locations

File / Location Purpose
/etc/docker/daemon.json Central Docker engine daemon configuration.
/var/lib/docker/ Docker runtime root directory (images, layers, containers).
/var/run/docker.sock Daemon IPC socket interface.
~/.docker/config.json Registry user auth credentials.

Important Command Cheatsheet

Purpose Syntax Command
Build Image docker build -t name:tag .
Run Detached Container docker run -d --name app -p 8080:80 image:tag
Exec Shell docker exec -it app /bin/sh
Tail Container Logs docker logs -f --tail 100 app
Live Resource Stats docker stats
System Disk Cleanup docker system prune -a --volumes

Operational Troubleshooting Quick Guide

Symptom Initial Diagnostic Command Primary Suspect
Socket connection refused systemctl status docker Daemon stopped or user not in docker group.
Container exits immediately `docker inspect app jq ‘.[0].State’`
Exit code 137 `dmesg grep -i oom`
Port bind failure `ss -tulpn grep :`
Filesystem full docker system df -v Accumulation of unused dangling image layers and volumes.

17. Further Reading and References

  1. Official Docker Documentation — docs.docker.com
  2. Dockerfile Best Practices Reference — docs.docker.com/engine/reference/builder
  3. Docker Compose Documentation — docs.docker.com/compose
  4. Docker Storage & OverlayFS Driver Mechanics — docs.docker.com/storage
  5. Docker Container Networking Architecture — docs.docker.com/network