1. Overview
Docker standardizes how applications are built, shipped, and executed across heterogeneous environments. By encapsulating an application along with its dependencies, runtime binaries, system libraries, and default configurations into an immutable image format, Docker guarantees that an application runs identically on a developer’s laptop, a continuous integration (CI) test runner, or a multi-node cloud production cluster. This consistency eliminates the classic “works on my machine” deployment failure caused by subtle environment drift between development and production Linux servers.
In modern Site Reliability Engineering (SRE) and Linux system administration, Docker serves several critical roles:
- Microservice Containerization: Encapsulating isolated application components with independent dependency stacks.
- Local Development Stacks: Provisioning reproducible local environments containing databases, caches, and web services matching production parameters.
- CI/CD Build & Test Environments: Running ephemeral build pipelines and integration tests inside clean, disposable containers.
- Kubernetes Runtime Foundation: Operating as the underlying engine or supplying core runtime components (such as
containerd) managed via the Container Runtime Interface (CRI).
Docker vs. Hardware Virtualization (Virtual Machines)
A fundamental architectural distinction exists between hypervisor-based Virtual Machines (VMs) and kernel-level Linux containers.
+---------------------------------+ +---------------------------------+
| App A | App B | App C | | App A | App B | App C |
+----------+----------+-----------+ +----------+----------+-----------+
| Guest OS | Guest OS | Guest OS | | Bins/Libs| Bins/Libs| Bins/Libs |
+----------+----------+-----------+ +---------------------------------+
| Hypervisor | | Docker Engine |
+---------------------------------+ +---------------------------------+
| Host Linux OS | | Host Linux Kernel |
+---------------------------------+ +---------------------------------+
| Physical Hardware | | Physical Hardware |
+---------------------------------+ +---------------------------------+
Hypervisor-Based VMs Docker Containers
Containers are not VMs; they share the host Linux kernel directly. While hypervisors virtualize physical hardware and require a dedicated guest operating system for every virtual instance, containers isolate user-space processes directly on the host kernel. Consequently, containers spin up in milliseconds, consume megabytes of RAM instead of gigabytes, and impose minimal I/O and CPU overhead compared to full hardware virtualization.
However, sharing the host kernel implies that container security relies on kernel-level isolation primitives. While strong enough for standard multi-tenant workloads, container isolation is not equivalent to hardware-enforced hypervisor boundaries across all threat models.
Simple Operational Example
To demonstrate Docker’s functional model, launching an isolated, published NGINX web server requires a single command:
docker run -d --name web-server -p 8080:80 nginx:1.25
When executed, Docker pulls the nginx:1.25 image from the registry (if not already cached), creates a network namespace, publishes port 8080 on the host to route to port 80 inside the container, attaches a writable filesystem layer, and executes NGINX as PID 1.
2. Architecture and Mental Model
To effectively operate and troubleshoot Docker in enterprise environments, system administrators must understand the execution stack and the underlying Linux kernel mechanisms that enforce container isolation and resource allocation.
The Docker Execution Stack
Modern Docker relies on a layered, modular architecture rather than a single monolithic binary. When an administrator executes a container, request execution flows through several specialized sub-components:
graph TD
A[Docker CLI] -->|REST API over Unix Socket /var/run/docker.sock| B[Docker Daemon: dockerd]
B -->|gRPC Requests| C[containerd]
C -->|Spawns OCI Shims| D[runc]
D -->|Configures Kernel Features| E[Linux Kernel]
E -->|Namespaces| F[Process & Network Isolation]
E -->|Cgroups| G[CPU & Memory Controls]
E -->|OverlayFS| H[Stacked Filesystem Layers]
- Docker CLI (
docker): The client binary that parses user commands and formats REST API requests. - Docker Daemon (
dockerd): The persistent background service listening on/var/run/docker.sock. It handles high-level user API calls, image management, volume orchestration, and network plumbing. containerd: An industry-standard, high-performance container daemon. It manages the complete container lifecycle—including image pulling, snapshot management, container execution, and supervisor tasks. (Note: Kubernetes nodes communicate directly withcontainerdorCRI-Ovia the CRI API, bypassingdockerdentirely).runc: The low-level OCI (Open Container Initiative) compliant reference runtime. It interacts directly with the Linux kernel to assemble namespaces and cgroups, starts the PID 1 container process, and then exits, leaving a lightweight shim process to supervise the container.
Linux Kernel Isolation Primitives
A container is not an abstract virtual machine; it is a standard Linux process (or process tree) restricted by kernel primitives.
1. Linux Namespaces (Resource Isolation)
Namespaces restrict what a process can see by providing isolated views of global system resources:
| Namespace | Subsystem Isolated | Operational Impact & Function |
|---|---|---|
| PID | Process IDs | Processes inside have independent PID numbering starting at PID 1; host PIDs are hidden. |
| NET | Networking | Container receives dedicated virtual interfaces, loopback (lo), routing tables, and IP addresses. |
| MNT | Mount points | Provides a dedicated filesystem mount table (/ inside container is isolated from host root). |
| UTS | Hostname & Domain | Allows container to maintain an independent system hostname. |
| IPC | Inter-Process Comm. | Isolates System V IPC objects and POSIX message queues from other containers. |
| USER | UID / GID Mappings | Maps inside container root (UID 0) to an unprivileged host UID for rootless operation. |
2. Control Groups / cgroups (Resource Constraints)
Control Groups enforce how much system resource a process can consume. The kernel measures and limits usage across key subsystems:
- Memory: Sets hard thresholds (
--memory). Exceeding allocated RAM triggers the Linux Out-Of-Memory (OOM) killer, terminating the container process (Exit Code 137). - CPU: Allocates CPU shares (
--cpu-shares) or hard execution time caps (--cpus). - PIDs: Prevents fork-bomb attacks by capping total allowable process threads (
--pids-limit). - BlkIO: Limits read/write bandwidth and I/O operations per second (IOPS) on block devices.
3. OverlayFS (Union Filesystem Storage)
Docker constructs container filesystems using stacked, read-only image layers topped by a thin, read-write container layer. On modern Linux distributions, this is implemented using the overlay2 storage driver located at /var/lib/docker/overlay2/.
+-------------------------------------------------+
| Writable Container Layer (Read-Write) | <-- File modifications, whiteouts
+-------------------------------------------------+
| Layer N: App Code & Configs (Read-Only) | <-- Image Layer
+-------------------------------------------------+
| Layer 1: Base Dependencies (Read-Only) | <-- Image Layer
+-------------------------------------------------+
| Layer 0: Base OS (Ubuntu/Alpine) (Read-Only) | <-- Image Layer
+-------------------------------------------------+
- Copy-on-Write (CoW): When a process inside a container modifies an existing file originating from a lower read-only layer, OverlayFS copies the file up to the thin writable layer before applying modifications.
- Whiteout Files: Deleting a file originating from an image layer creates a special “whiteout” character device file in the writable layer. This masks the lower layer file without deleting it from the underlying immutable image layer.
- Ephemeral Nature: Deleting a container via
docker rmpermanently destroys its writable layer and all unmounted data changes.
3. Core Concepts and Terminology
Understanding key Docker terminology ensures precise communication and prevents administrative errors during deployment and maintenance.
Image vs. Container
- Docker Image: A read-only template consisting of stacked filesystem layers and runtime metadata (entrypoints, environment variables, default commands). Built from a
Dockerfileor pulled from a container registry. - Docker Container: A running or stopped stateful instance of an image. It combines the read-only image layers with an isolated process namespace, cgroup resource limits, network configuration, and a thin read-write filesystem layer.
Image Tags and Registries
Images are referenced using the canonical syntax [registry_host[:port]/][repository/]image_name:tag.
- Tagging Strategy: Omitting a tag causes Docker to default to
:latest. Production deployments must never rely on:latestbecause it leads to unpredictable image drift. Explicit, immutable version tags (e.g.,nginx:1.25.3-alpine) must be pinned.
Storage Types
- Bind Mounts: Maps an absolute directory path on the host system directly into a container path (
-v /opt/configs:/etc/app). Subject to host permissions and path dependencies. - Named Volumes: Docker-managed persistent storage allocated under
/var/lib/docker/volumes/. Preferred for production databases as they decouple data lifecycles from container lifecycles. tmpfsMounts: Ephemeral storage backed exclusively by host RAM. Used for high-speed temporary caching or keeping sensitive secrets out of persistent disk storage.
Key Glossary
| Term | Definition |
|---|---|
| Base Image | The foundational image defined in the initial FROM directive of a Dockerfile. |
| Build Context | The set of local files and directories sent from the Docker CLI to the daemon during an image build. |
dockerinit / Tini |
A tiny init process used inside containers to harvest zombie processes and forward signals properly. |
| Dangling Image | An unreferenced image layer marked as <none>:<none> resulting from building an image with an existing tag. |
| DOCKER_BUILDKIT | The modern, high-performance image build engine featuring parallel stage execution and secret mounting. |
4. Installation and Version Management
Prerequisites and Dependencies
Docker requires a 64-bit Linux distribution with kernel version 3.10 or higher (kernel 5.x+ recommended for full overlay2 and cgroups v2 capabilities).
Installing Docker CE on Ubuntu / Debian
Follow the official Docker repository installation workflow to guarantee receipt of security updates:
# 1. Remove legacy packages
sudo apt-get remove -y docker docker-engine docker.io containerd runc
# 2. Install repository prerequisites
sudo apt-get update
sudo apt-get install -y ca-certificates curl gnupg lsb-release
# 3. Add Docker's official GPG key
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
# 4. Set up the stable repository
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
# 5. Install Docker Engine, containerd, and Docker Compose CLI plugin
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Installing Docker CE on RHEL / Rocky Linux / AlmaLinux
# 1. Install yum-utils
sudo dnf install -y yum-utils
# 2. Add the official repository
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
# 3. Install packages
sudo dnf install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
# 4. Enable and start daemon
sudo systemctl enable --now docker
Validating Installation and Identifying Versions
Inspect installed version details across CLI and Daemon engines:
# Display client and server engine versions
docker version
# Display system-wide information including storage driver, cgroup version, and container counts
docker info
5. Configuration Files and Directory Structure
Docker’s operation on a Linux server relies on specific configuration files, sockets, and storage directories.
File and Directory Reference
| File / Directory Path | Purpose | Important Details |
|---|---|---|
/etc/docker/daemon.json |
Main daemon configuration file. | Controls network bridges, storage drivers, logging options, DNS, and insecure registries. |
/var/lib/docker/ |
Primary daemon storage directory. | Holds images, overlay layers, volumes, and container states. Requires dedicated disk space. |
/var/lib/docker/overlay2/ |
Storage location for OverlayFS image and container layers. | Contains diff, work, and merged directories per layer. |
/var/lib/docker/volumes/ |
Default path for Docker named volumes. | Stores persistent container volume data managed by Docker. |
/var/run/docker.sock |
Unix domain socket used for API communication. | Default interface between CLI and dockerd. Secured by docker group permissions. |
~/.docker/config.json |
Per-user CLI configuration file. | Stores registry authentication credentials (e.g., docker login tokens) and CLI formatting settings. |
/lib/systemd/system/docker.service |
Default systemd unit configuration. | Defines process execution flags, file descriptor limits (LimitNOFILE=1048576), and startup dependencies. |
Production /etc/docker/daemon.json Reference
Below is a production-hardened /etc/docker/daemon.json configuration specifying log rotation, cgroups driver management, and storage engine parameters:
{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "50m",
"max-file": "3"
},
"storage-driver": "overlay2",
"dns": ["8.8.8.8", "1.1.1.1"],
"live-restore": true,
"userland-proxy": false,
"no-new-privileges": true
}
live-restore: Keeps containers running during daemon updates or crashes, reducing service downtime.log-opts: Prevents container log files from consuming all host disk space.userland-proxy: false: Disables the inefficientdocker-proxyprocess, routing container traffic directly using LinuxiptablesNAT rules.
After modifying /etc/docker/daemon.json, validate syntax and reload the service:
# Validate JSON syntax
jq . /etc/docker/daemon.json
# Reload systemd and apply configuration to dockerd
sudo systemctl reload docker
6. Service Management and Common Operations
Managing the Docker background daemon relies on systemd operational commands.
Daemon Management Commands
# Check detailed daemon operational status
sudo systemctl status docker
# Start the Docker daemon
sudo systemctl start docker
# Stop the Docker daemon
sudo systemctl stop docker
# Restart the daemon (briefly interrupts API connectivity unless live-restore is enabled)
sudo systemctl restart docker
# Enable automatic startup at host boot
sudo systemctl enable docker
Container Lifecycle State Changes
Understanding the distinction between lifecycle operations prevents unintended downtime:
- Container Restart (
docker restart <container>): Sends aSIGTERMsignal to PID 1 inside the container, waits 10 seconds for graceful cleanup, and sendsSIGKILLif the process has not exited. It then recreates the process namespace and restarts PID 1. - Application Reload: For applications supporting live reloading (such as NGINX or Apache), configuration updates should be reloaded without stopping the container namespace by executing the reload signal directly inside PID 1:
docker exec web-server nginx -s reload
7. Common Administrative Tasks and Workflows
Workflow 1: Building Optimized Multi-Stage Docker Images
Unoptimized Dockerfiles generate massive images containing build tools (compilers, git, dev libraries) that introduce security vulnerabilities and slow deployment speeds. Multi-stage builds solve this by separating the build environment from the runtime image.
Practical Multi-Stage Dockerfile (Go Application)
# Stage 1: Build Environment
FROM golang:1.22-alpine AS builder
# Set workspace
WORKDIR /app
# Optimize layer caching by copying dependency manifests first
COPY go.mod go.sum ./
RUN go mod download
# Copy source code and compile static binary
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-w -s" -o /app/server .
# Stage 2: Minimal Production Runtime
FROM alpine:3.19
# Security: Create non-root user and group
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
WORKDIR /app
# Copy binary from builder stage
COPY --from=builder /app/server /app/server
# Adjust ownership
RUN chown -R appuser:appgroup /app
# Expose port and set security context
USER appuser
EXPOSE 8080
# Default execution entrypoint (exec form)
ENTRYPOINT ["/app/server"]
Workflow 2: Orchestrating Services with Docker Compose (v2)
Docker Compose defines multi-container architecture stacks using a declarative docker-compose.yml file.
Production docker-compose.yml Example
version: '3.8'
services:
web:
build:
context: .
dockerfile: Dockerfile
ports:
- "8080:8080"
environment:
- DB_HOST=db
- DB_PORT=5432
depends_on:
db:
condition: service_healthy
restart: always
deploy:
resources:
limits:
cpus: '1.0'
memory: 512M
db:
image: postgres:16-alpine
environment:
POSTGRES_DB: appdb
POSTGRES_USER: dbuser
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
secrets:
- db_password
volumes:
- pgdata:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U dbuser -d appdb"]
interval: 5s
timeout: 5s
retries: 5
restart: always
volumes:
pgdata:
driver: local
secrets:
db_password:
file: ./db_password.txt
Operational Commands for Compose
# Start stack in detached mode and rebuild changed images
docker compose up -d --build
# Inspect status of running services
docker compose ps
# View combined, colorized logs across all services in real time
docker compose logs -f --tail 100
# Tear down containers, networks, and ephemeral resources (preserving named volumes)
docker compose down
# Tear down containers AND remove named volumes (Destructive!)
docker compose down -v
8. CLI Command Reference and Cheatsheet
Image Management Commands
| Command | Purpose | Key Flags & Arguments | Practical Example | Explanation |
|---|---|---|---|---|
docker pull |
Downloads image from registry. | -a (all tags) |
docker pull nginx:1.25-alpine |
Fetches specified tag to local cache. |
docker images |
Lists cached local images. | -a (all), -q (quiet IDs) |
docker images --filter "dangling=true" |
Lists images; filter helps spot untagged layers. |
docker build |
Builds image from Dockerfile. | -t (tag), --no-cache, -f (file) |
docker build -t app:1.0 -f Dockerfile . |
Compiles layers using current directory as context. |
docker rmi |
Removes local images. | -f (force) |
docker rmi app:1.0 |
Deletes image metadata and unused layers. |
docker history |
Displays layer history of an image. | --no-trunc |
docker history --no-trunc app:1.0 |
Shows layer sizes and Dockerfile execution steps. |
Container Run and Lifecycle Commands
| Command | Purpose | Key Flags & Arguments | Practical Example | Explanation |
|---|---|---|---|---|
docker run |
Creates and starts a container. | -d (detach), -p (port mapping), -v (volume mount), --name |
docker run -d --name web -p 80:80 -v /data:/app/data nginx:1.25 |
Instantiates isolated container running in background. |
docker ps |
Lists active containers. | -a (all including stopped), -q (IDs) |
docker ps -a --filter "status=exited" |
Displays runtime state, published ports, names. |
docker stop |
Gracefully stops container. | -t (timeout in seconds) |
docker stop -t 30 web |
Sends SIGTERM, waits timeout, then sends SIGKILL. |
docker kill |
Forcefully kills container. | -s (signal name/number) |
docker kill -s SIGKILL web |
Sends immediate raw kernel signal to PID 1. |
docker rm |
Removes stopped container. | -f (force running container) |
docker rm -f web |
Deletes container metadata and thin writable layer. |
Diagnostics, Logs, and Inspection
| Command | Purpose | Key Flags & Arguments | Practical Example | Explanation |
|---|---|---|---|---|
docker logs |
Fetches container log output. | -f (follow), --tail N, --since |
docker logs -f --tail 100 --since 10m web |
Tails stdout/stderr streams from container PID 1. |
docker exec |
Runs command inside running container. | -i (interactive), -t (tty), -u (user) |
docker exec -it -u root web /bin/sh |
Attaches process to existing container namespaces. |
docker inspect |
Returns low-level JSON configuration. | -f / --format (Go template) |
docker inspect --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' web |
Queries exact IP address, mounts, environment, and state. |
docker stats |
Streams live resource utilization. | --no-stream |
docker stats web |
Reads cgroup metrics for CPU, RAM, Network, and I/O. |
docker top |
Displays processes running inside container. | Host ps flags |
docker top web aux |
Translates container PIDs to host process table. |
9. Logs, Monitoring and Diagnostics
Inspecting Container Logs
Docker captures standard output (stdout) and standard error (stderr) emitted by PID 1 inside the container.
# Follow live log stream for a specific container
docker logs -f web-server
# View last 50 lines with timestamps
docker logs --tail 50 -t web-server
# View logs emitted since a relative time threshold
docker logs --since 15m web-server
Log Drivers and Rotation
By default, Docker uses the json-file log driver, storing raw log output in /var/lib/docker/containers/<container-id>/<container-id>-json.log. Unmanaged logging can quickly fill disk space.
Always configure global rotation via /etc/docker/daemon.json (as shown in Section 5) or apply per-container flags:
docker run -d --log-driver json-file --log-opt max-size=10m --log-opt max-file=3 nginx
Resource Utilization and Cgroup Inspection
# Stream real-time resource utilization across all active containers
docker stats
# Inspect processes running inside a container from host perspective
docker top web-server
To inspect raw cgroup limits directly via the kernel interface (cgroups v2 example):
# Query memory cap enforced by cgroup driver
cat /sys/fs/cgroup/docker/<container-id>/memory.max
# Check current memory usage
cat /sys/fs/cgroup/docker/<container-id>/memory.current
10. Troubleshooting Guide and Common Failure Scenarios
This section details systematic investigation workflows for common Docker operational failures.
Scenario 1: Cannot Connect to the Docker Daemon
Symptoms
User receives error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? or Permission denied while trying to connect to the Docker daemon socket.
Possible Causes
- The
dockersystemd service is stopped or crashed. - The executing user lacks read/write permissions on
/var/run/docker.sock.
Investigation and Diagnosis
- Check the systemd service state:
systemctl status docker
- Verify ownership and permissions on the Unix socket:
ls -la /var/run/docker.sock
Expected Output: srw-rw---- 1 root docker 0 Nov 3 10:00 /var/run/docker.sock
3. Verify user group membership:
id $USER
Resolution
- If daemon is down, start it:
sudo systemctl start docker
- If permissions issue, append user to
dockergroup and re-authenticate:
sudo usermod -aG docker $USER
newgrp docker
Scenario 2: Container Exits Immediately Upon Startup
Symptoms
Executing docker run returns a container ID, but docker ps shows no running container. docker ps -a reveals state Exited (0) or Exited (1).
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a1b2c3d4e5f6 myapp "/docker-entrypoint…" 5 seconds ago Exited (1) 4 seconds ago my-app
Possible Causes
- PID 1 Exited: Containers run only as long as PID 1 remains active. If the command runs in background mode (e.g.,
nginxwithoutdaemon off;) or completes an execution script, PID 1 exits immediately. - Missing Configuration / Crash: Application threw an unhandled exception or failed to read environment flags on launch.
Investigation and Diagnosis
- Inspect container exit status and logs:
docker inspect my-app | jq -r '.[0].State | {Status: .Status, ExitCode: .ExitCode, Error: .Error}'
docker logs --tail 50 my-app
- Override container entrypoint with an interactive shell to debug image contents:
docker run --rm -it --entrypoint /bin/sh myapp
Inside shell, verify configuration files, environment variables, and binary execution manually.
Resolution
Ensure the application command runs in the foreground. For example, in NGINX Dockerfiles, pass daemon off;:
CMD ["nginx", "-g", "daemon off;"]
Scenario 3: Interpreting Container Exit Codes
When diagnosing exited containers via docker inspect CONTAINER | jq '.[0].State.ExitCode', use this reference guide to determine root cause:
| Exit Code | Signal / Meaning | Root Cause & Diagnostic Workflow |
|---|---|---|
| 0 | Success / Clean Exit | Container completed its execution normally (common for batch jobs or improperly backgrounded services). |
| 1 | Application Error | Generic application crash. Check application stack traces via docker logs. |
| 126 | Command Cannot Execute | Entrypoint/CMD file is not marked executable (chmod +x missing) or architecture mismatch. |
| 127 | Command Not Found | Executable specified in ENTRYPOINT/CMD does not exist in the container $PATH. |
| 137 | SIGKILL (128 + 9) |
Process forcibly killed by Linux kernel. Check if OOMKilled: true via docker inspect or check host `dmesg |
| 143 | SIGTERM (128 + 15) |
Process gracefully terminated via docker stop. |
Scenario 4: Host Port Binding Failure (bind: address already in use)
Symptoms
Executing docker run or docker compose up fails with:
driver failed programming external connectivity on endpoint web: Error starting userland proxy: listen tcp4 0.0.0.0:8080: bind: address already in use.
Possible Causes
Another host process or Docker container is already listening on published host port 8080.
Investigation and Diagnosis
- Identify the process holding host port 8080:
sudo ss -tulpn | grep :8080
# OR
sudo lsof -i :8080
- Identify conflicting Docker containers:
docker ps --format "table {{.Names}}\t{{.Ports}}" | grep 8080
Resolution
- Stop host service holding the port (
sudo systemctl stop apache2ordocker stop <conflicting-container>). - Or re-map host port binding syntax (
-p <host_port>:<container_port>):
docker run -d -p 8081:8080 myapp
Scenario 5: Container OOM (Out Of Memory) Killed
Symptoms
Container suddenly restarts or stops. docker ps -a shows exit code 137.
Investigation and Diagnosis
- Query container OOM state via inspect:
docker inspect web-app | jq -r '.[0].State | {OOMKilled: .OOMKilled, ExitCode: .ExitCode}'
*If OOMKilled: true*, the container breached its configured --memory limit.
2. Search kernel log buffer for OOM events:
sudo dmesg -T | grep -i "oom-killer"
Resolution
- Increase memory ceiling in Dockerfile/Compose (
memory: 1G). - Investigate memory leaks inside application code using runtime profiling tools.
Scenario 6: Filesystem Exhaustion (No space left on device)
Symptoms
Docker builds fail, containers fail to launch or write logs, emitting error: No space left on device.
Investigation and Diagnosis
- Check disk utilization across system filesystems:
df -h /var/lib/docker
- Run Docker disk usage breakdown:
docker system df -v
Identifies space consumed individually by images, containers, named volumes, and build caches.
Resolution (Safe Pruning Sequence)
Execute cleanup operations sequentially from safest to most aggressive:
# 1. Remove exited/stopped containers
docker container prune -f
# 2. Remove dangling image layers (<none>:<none>)
docker image prune -f
# 3. Remove unused BuildKit cache
docker builder prune -f
# 4. Remove all images not attached to an active container
docker image prune -a -f
# 5. WARNING: Remove unattached named volumes (Destructive to offline data!)
docker volume prune -f
11. Security and Best Practices
Securing containers requires implementing defense-in-depth principles across the host OS, daemon configuration, and Dockerfiles.
Security Controls Checklist
- Never Run Containers as Root:
Default containers run as
rootinside the container namespace, exposing host risk if kernel escapes occur. Specify non-root users inside Dockerfiles:
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser
- Enforce Read-Only Root Filesystems:
Prevent attackers from modifying application code or downloading binary payloads by mounting container root as read-only, using temporary
tmpfsmounts for runtime file activity:
docker run -d --read-only --tmpfs /tmp --tmpfs /run nginx
- Secure the Docker Unix Socket:
Access to
/var/run/docker.sockgrants effective root access to the host machine. Never expose/var/run/docker.sockto application containers unless absolutely required, and never expose it directly over an unencrypted TCP port. - Resource Quotas (Prevent DoS Attacks): Always specify memory, CPU, and process ceiling limits on shared production hosts to block rogue process consumption:
docker run -d --memory="512m" --cpus="1.5" --pids-limit=100 myapp
- BuildKit Secret Mounting:
Do not bake sensitive API credentials into image layers using
ENVorARGdirectives. Use BuildKit secret mounts, which expose credentials temporarily during build time without leaving remnants in image layers:
RUN --mount=type=secret,id=api_key \
API_KEY=$(cat /run/secrets/api_key) ./build_script.sh
12. Performance and Resource Optimisation
Optimizing Docker environments focuses on minimizing image layer sizes, maximizing layer cache reusability, and reducing system runtime resource overhead.
Maximizing Layer Cache Efficiency
Docker executes Dockerfile instructions sequentially, caching the intermediate image layer for each instruction. If an instruction changes, Docker invalidates that layer’s cache and rebuilds all subsequent layers.
Anti-Pattern (Invalidates Cache Frequently)
FROM node:20-alpine
WORKDIR /app
# BAD: Copying all app files FIRST invalidates package install cache on every single line edit
COPY . .
RUN npm install
CMD ["node", "server.js"]
Optimized Pattern (Maximizes Cache Hits)
FROM node:20-alpine
WORKDIR /app
# GOOD: Copy dependency manifests ONLY; layer stays cached unless package.json changes
COPY package*.json ./
RUN npm install
# Copy remaining source code after heavy dependencies are installed
COPY . .
CMD ["node", "server.js"]
Layer Count Optimization
Combine package updating, installation, and cleanup tasks into single RUN instructions to prevent apt metadata cache from bloating image layers:
RUN apt-get update && apt-get install -y --no-install-recommends \
curl \
ca-certificates \
&& rm -rf /var/lib/apt/lists/*
13. Backup, Recovery and Maintenance
Backing Up Docker Named Volumes
Named volumes stored in /var/lib/docker/volumes/ should be backed up using disposable helper containers that mount the target volume alongside a host backup directory:
# Backup volume 'pgdata' to a compressed tar archive on the host
docker run --rm \
-v pgdata:/volume-data:ro \
-v $(pwd)/backups:/backup \
alpine tar -czf /backup/pgdata_backup_$(date +%Y%m%d).tar.gz -C /volume-data .
Restoring Docker Named Volumes
# Create target volume
docker volume create pgdata_restored
# Restore archive contents into new volume
docker run --rm \
-v pgdata_restored:/volume-data \
-v $(pwd)/backups:/backup \
alpine sh -c "cd /volume-data && tar -xzf /backup/pgdata_backup_20261103.tar.gz"
Exporting and Importing Images for Air-Gapped Environments
# Save image to tarball archive
docker save -o nginx_1.25.tar nginx:1.25
# Load image tarball archive into air-gapped system registry
docker load -i nginx_1.25.tar
14. Automation and Integration
Integrating Docker workflows into standard Linux maintenance routines guarantees host reliability.
Systemd Timer for Automated Docker Disk Cleanup
Create a daily systemd service and timer to automatically prune unused dangling resources.
/etc/systemd/system/docker-cleanup.service
[Unit]
Description=Automated Docker System Cleanup
After=docker.service
Requires=docker.service
[Service]
Type=oneshot
ExecStart=/usr/bin/docker system prune -f --filter "until=168h"
/etc/systemd/system/docker-cleanup.timer
[Unit]
Description=Run Docker Cleanup Daily
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
Enable timer:
sudo systemctl daemon-reload
sudo systemctl enable --now docker-cleanup.timer
15. Practical Labs and Exercises
Lab 1: Multi-Stage Container Building
Objective: Build a production-ready, non-root microservice image under 20MB.
- Create directory
lab1and define a minimal Go servicemain.go:
package main
import ("fmt"; "net/http")
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Lab 1 Operational")
})
http.ListenAndServe(":8080", nil)
}
- Write a multi-stage
Dockerfileusinggolang:alpineas builder andscratchoralpineas production target. Ensure non-root execution. - Build image:
docker build -t lab1:v1 . - Validate binary size and user execution:
docker images lab1:v1
docker run -d -p 8080:8080 --name lab1-test lab1:v1
docker exec lab1-test id
Expected Outcome: User UID is non-zero, image size is drastically smaller than standard base build.
Lab 2: Troubleshooting a Broken Startup Container
Objective: Diagnose and repair an application container failing on boot.
- Launch test broken container:
docker run -d --name broken-app nginx:alpine sh -c "nginx -g 'daemon off;'"
- Wait 10 seconds and check container state:
docker ps -a --filter "name=broken-app"
- Perform investigation:
- Read exit code via
docker inspect. - Check container log output via
docker logs. - Identify missing options or configuration errors.
- Clean up broken instance:
docker rm broken-app
16. Quick Reference and Cheat Sheet
Important Files and Locations
| File / Location | Purpose |
|---|---|
/etc/docker/daemon.json |
Central Docker engine daemon configuration. |
/var/lib/docker/ |
Docker runtime root directory (images, layers, containers). |
/var/run/docker.sock |
Daemon IPC socket interface. |
~/.docker/config.json |
Registry user auth credentials. |
Important Command Cheatsheet
| Purpose | Syntax Command |
|---|---|
| Build Image | docker build -t name:tag . |
| Run Detached Container | docker run -d --name app -p 8080:80 image:tag |
| Exec Shell | docker exec -it app /bin/sh |
| Tail Container Logs | docker logs -f --tail 100 app |
| Live Resource Stats | docker stats |
| System Disk Cleanup | docker system prune -a --volumes |
Operational Troubleshooting Quick Guide
| Symptom | Initial Diagnostic Command | Primary Suspect |
|---|---|---|
| Socket connection refused | systemctl status docker |
Daemon stopped or user not in docker group. |
| Container exits immediately | `docker inspect app | jq ‘.[0].State’` |
| Exit code 137 | `dmesg | grep -i oom` |
| Port bind failure | `ss -tulpn | grep : |
| Filesystem full | docker system df -v |
Accumulation of unused dangling image layers and volumes. |
17. Further Reading and References
- Official Docker Documentation — docs.docker.com
- Dockerfile Best Practices Reference — docs.docker.com/engine/reference/builder
- Docker Compose Documentation — docs.docker.com/compose
- Docker Storage & OverlayFS Driver Mechanics — docs.docker.com/storage
- Docker Container Networking Architecture — docs.docker.com/network