Domain 5Containers & RuntimesINTERMEDIATE

Podman Architecture, Operations, and Troubleshooting Guide

A comprehensive Linux SysAdmin and SRE guide to Podman architecture, daemonless execution, rootless container isolation, Quadlet systemd integration, Buildah/Skopeo workflows, and step-by-step troubleshooting.

Domain: Domain 5: Containers & Runtimes•Updated: 2026-11-03
#linux#sysadmin#podman#containers#quadlet

1. Overview

Podman (the Pod Manager) is an open-source, daemonless container engine designed to develop, manage, and run Open Container Initiative (OCI) containers and container pods on Linux systems.

In modern production environments, Podman solves the fundamental security and architectural challenges posed by legacy daemon-based container engines. Standard container engines rely on a centralized, high-privilege daemon process (such as dockerd) running continuously as root. This daemon model introduces a single point of failure (if the daemon crashes, container management halts) and creates a significant security vulnerability: access to the daemon socket gives unprivileged users full root control over the host system.

Podman operates on a fork/exec model. When an administrator executes a podman command, the CLI directly interacts with the Linux kernel, storage drivers, and low-level OCI container runtimes without communicating with a background daemon. Furthermore, Podman natively supports rootless containers, allowing unprivileged users to build, run, and manage containers without root privileges or sudo rights.

+-----------------------------------------------------------------------+
|                             Docker Model                              |
|  [CLI User] ---> [/var/run/docker.sock] ---> [dockerd Daemon (ROOT)]  |
|                                                       |               |
|                                                 [containerd]          |
|                                                       |               |
|                                                 [runc / Container]    |
+-----------------------------------------------------------------------+

+-----------------------------------------------------------------------+
|                            Podman Model                               |
|  [User / CLI] ---> [Podman Process] ---> [conmon] ---> [crun / runc]  |
|                                                              |        |
|                                                     [Container]       |
|  (No daemon socket, direct fork/exec, runs fully unprivileged)        |
+-----------------------------------------------------------------------+

Common Use Cases

  • Hardened Server Environments: Deploying application containers on multi-tenant or enterprise Linux servers where running a root-level daemon socket is prohibited by security policy.
  • Systemd-Managed Workloads: Running production microservices directly managed by systemd using Podman’s native Quadlet generator, ensuring container lifecycle hooks match host startup, shutdown, and restart policies.
  • Kubernetes-Local Prototyping: Testing multi-container Kubernetes manifests locally using Podman pods and podman play kube prior to cluster deployment.
  • Unprivileged CI/CD Pipelines: Building and testing container images inside non-root build runners using Podman and Buildah without requiring Docker-in-Docker (dind) security trade-offs.

Podman vs Docker Comparison

Feature / Metric Docker Podman
Architecture Client-Server with long-running daemon (dockerd) Daemonless, direct fork/exec engine
Default Security Model Rootful daemon requiring docker group permissions Rootless by default using user namespaces (/etc/subuid)
Systemd Integration Requires third-party tools or complex wrapper units Native systemd integration via Quadlet .container units
Pod Abstraction Requires Docker Compose or Swarm for multi-container groups Native pod support sharing network/IPC/PID namespaces
Image Building Integrated docker build daemon engine Direct CLI or standalone specialized tool (buildah)
Image Inspection/Copy Requires pulling image into local daemon daemon storage Standalone tool (skopeo) inspects/copies remote registries
CLI Compatibility Native Docker CLI commands Docker-compatible CLI syntax (alias docker=podman)

Practical Example

To demonstrate Podman’s CLI compatibility and daemonless nature, deploy a detached Nginx web server mapped to host port 8080:

# Execute container creation as a standard unprivileged user
podman run -d --name web-server -p 8080:80 nginx:alpine

# Verify running containers directly via kernel process tree
podman ps

# Verify HTTP response from local host port
curl http://localhost:8080

2. Architecture and Mental Model

To effectively operate and troubleshoot Podman, an SRE must understand its daemonless lifecycle and modular architecture. Podman does not maintain an active server socket to track container states; instead, state management is delegated to tiny monitoring processes and kernel-level Linux features.

graph TD
    A[User CLI: podman run] --> B[Image Resolution & Storage Engine]
    B -->|Overlay / fuse-overlayfs| C[Root Filesystem Setup]
    A --> D[Network Engine: Netavark / pasta / slirp4netns]
    D -->|Network Namespace| E[Container Network Setup]
    A --> F[Spawn Monitoring Process: conmon]
    F --> G[OCI Runtime: crun / runc]
    G --> H[Container Instance: PID 1 Workload]
    F -->|Supervises & captures logs| H
    F -->|Writes exit status & stdout| I[Host Storage / Journald]

Key Architectural Components

  1. Podman CLI Engine: The entry point tool that parses user commands, reads configuration files (/etc/containers/containers.conf), manages image layers, and calls low-level libraries (such as c/storage and c/image).
  2. Conmon (Container Monitor): A lightweight C utility spawned by Podman for every container. conmon serves as the parent supervisor for the container process (PID 1). It attaches to standard I/O streams, holds open the TTY, records container exit codes, and writes log files even when the main podman CLI execution completes.
  3. OCI Runtime (crun / runc): The low-level binary invoked by conmon to set up Linux namespaces, cgroup limits, SELinux security contexts, and capabilities before executing the container payload. crun (written in C) is the default runtime on modern Linux distros due to low memory overhead and fast execution speed.
  4. Storage Engine (containers/storage): Handles layer creation and mount points. Rootful installations use native kernel overlay drivers. Rootless installations use unprivileged kernel overlay mounts (kernel 5.11+) or fuse-overlayfs fallback mechanisms.
  5. Networking Layer (Netavark & Pasta / Slirp4netns):
  • Rootful Networking: Managed by Netavark (the modern network stack replacing CNI), which configures bridge interfaces, iptables/nftables rules, and host networking directly.
  • Rootless Networking: Configured using user-space network abstractions like pasta (Pack A Subtle Tap Abstraction) or slirp4netns. Pasta passes network traffic directly into the user namespace without requiring root privilege or bridge interfaces.
  1. Systemd & Quadlet Generator: The architectural bridge linking Linux host init systems with container management. Quadlet acts as a systemd generator, parsing high-level .container or .pod files at system boot and translating them into standard systemd service unit files.

Step-by-Step Container Lifecycle Execution

When an administrator executes podman run -d --name app -p 8080:80 nginx:alpine:

  1. Image Resolution: Podman searches local storage (~/.local/share/containers/storage or /var/lib/containers/storage). If missing, it queries configured registries in /etc/containers/registries.conf and pulls the image layers.
  2. Namespace & Storage Provisioning: Podman creates a read-write container layer on top of the read-only image layers using overlay or fuse-overlayfs. It allocates UID/GID maps via /etc/subuid and /etc/subgid if running rootless.
  3. Network Allocation: Podman sets up a dedicated network namespace. In rootless mode, pasta or slirp4netns connects host port 8080 to container port 80 inside the namespace.
  4. Conmon Forking: Podman forks and executes conmon as a background process. conmon opens pipes for stdout/stderr logging and creates a sockets directory under /run/user/UID/containers/.
  5. Runtime Execution: conmon calls crun (or runc), which configures cgroups v2 resource boundaries, applies SELinux contexts (system_u:system_r:container_t:s0), drops unwanted Linux capabilities, and executes the entrypoint binary (nginx) as PID 1 inside the container.
  6. Completion & Exit Tracking: When the containerized process exits, conmon captures the exit code, closes log streams, updates container status in storage metadata, and terminates.

Ecosystem Tools: Buildah and Skopeo

Podman relies on two modular partner utilities to avoid monolithic daemon bloat:

  • Buildah: Dedicated tool for building OCI/Docker container images. While podman build wraps Buildah commands under the hood, standalone Buildah allows shell-scripting image creation layer-by-layer without requiring a Dockerfile.
  • Skopeo: Command-line tool designed to inspect, copy, sign, and transfer images between different container registries and local storage backends directly—without downloading image layers or executing containers.

3. Core Concepts and Terminology

Understanding Podman requires familiarity with core Linux namespace abstractions, user mapping mechanics, and container orchestration concepts.

Key Concepts Explained

  • Daemonless Execution: The design paradigm where container operations run as direct child processes of the invocation tool or host init system (systemd), eliminating background service dependencies.
  • Rootless Mode: Executing containers entirely under an unprivileged user account. Root inside the container maps to an unprivileged sub-UID on the host system, mitigating container breakout risks.
  • SubUID and SubGID Mapping: The Linux security allocation defined in /etc/subuid and /etc/subgid that grants an unprivileged user a range of non-overlapping subordinate UIDs and GIDs for container isolation.
  • Pod Abstraction: A group of one or more containers sharing the same infrastructure (infra) container, network namespace, IPC namespace, and optionally PID namespace. Modeled directly after Kubernetes pods.
  • Quadlet: Modern systemd integration mechanism introduced in Podman 4.4+. It uses intuitive configuration files (.container, .volume, .network, .pod) processed during systemctl daemon-reload to auto-generate fully managed systemd service units.
  • SELinux Labeling (:Z / :z): Security context flags appended to volume mount definitions. :Z assigns a private, unshared SELinux label to the volume directory; :z assigns a shared context accessible across multiple containers.

Glossary of Essential Terms

Term Definition Primary Impact
cgroup v2 Unified Linux kernel resource control hierarchy Enforces CPU, memory, and I/O limits for rootless and rootful containers.
conmon Container Monitor C utility Supervises container process state, TTY connection, and log capturing.
pasta Pack A Subtle Tap Abstraction network driver High-performance, unprivileged network driver replacing slirp4netns in modern Podman.
slirp4netns User-space network emulator Legacy network stack for rootless containers using TAP interfaces.
Buildah Modular container image builder Enables fine-grained image creation, scriptable builds, and mounting storage layers.
Skopeo Remote registry image management CLI Inspects remote manifests and transfers images across registries without pulling layers.
Infra Container Pause container inside a Pod Holds namespaces open so operational containers in the pod can stop/restart seamlessly.

4. Installation and Version Management

Prerequisites and Dependencies

Before installing Podman, ensure the target Linux host meets the following prerequisites:

  • Kernel version 5.4+ (5.11+ recommended for native unprivileged overlayfs).
  • cgroups v2 enabled (default on RHEL 9, Fedora, Debian 11+, Ubuntu 22.04+).
  • SubUID/SubGID mapping configuration packages installed (shadow-utils or uidmap).

Distribution-Specific Installation

RHEL / Fedora / CentOS Stream

# Install Podman, Buildah, Skopeo, and network plugins
sudo dnf install -y podman buildah skopeo netavark pasta

Debian / Ubuntu

# Update package index and install standard Podman tools
sudo apt update
sudo apt install -y podman buildah skopeo uidmap slirp4netns

Arch Linux

# Install container suite from official repositories
sudo pacman -S podman buildah skopeo netavark

Version and Installation Verification

Verify binary paths and inspect active runtime capabilities:

# Check client engine versions
podman version

# Comprehensive runtime system inspection
podman info

# Filter rootless operating status from system info
podman info | grep -i rootless

Critical File Locations

  • Binary Installation Directory: /usr/bin/podman, /usr/bin/conmon, /usr/bin/crun
  • System-Wide Configuration: /etc/containers/
  • Rootful Container Storage: /var/lib/containers/storage
  • Rootless User Container Storage: ~/.local/share/containers/storage
  • Default OCI Runtimes: /usr/bin/crun, /usr/bin/runc

5. Configuration Files and Directory Structure

Podman configuration uses a cascading configuration model. System-wide defaults residing under /etc/containers/ are overridden by user-specific configuration files located under ~/.config/containers/.

Configuration Directory Reference

File / Directory Purpose Important Details
/etc/containers/containers.conf System-wide engine defaults Defines default runtimes, network drivers, log drivers, and cgroup managers.
~/.config/containers/containers.conf User-level engine overrides Controls rootless settings, default environment variables, and image registries.
/etc/containers/storage.conf System storage configuration Configures graph drivers (overlay), storage paths, and mount options.
~/.config/containers/storage.conf User storage configuration Sets rootless storage drivers (e.g., fuse-overlayfs or overlay).
/etc/containers/registries.conf Registry search paths Specifies mirror locations, unqualified search registries, and insecure registries.
/etc/subuid & /etc/subgid User namespace mappings Allocates subordinate UID/GID ranges for unprivileged users.
/etc/containers/systemd/ System-level Quadlet directory Holds system-wide .container, .volume, .network unit files.
~/.config/containers/systemd/ User-level Quadlet directory Holds unprivileged user Quadlet unit files.
/run/user/UID/podman/podman.sock Rootless Docker API socket Exposes Docker-compatible UNIX domain socket for non-root users.
/run/podman/podman.sock Rootful Docker API socket Exposes system-wide Docker-compatible UNIX domain socket.

Important Configuration Directives

Below is a production-grade example of a user-level engine configuration file (~/.config/containers/containers.conf):

[engine]
# Define default low-level OCI runtime (crun is faster and consumes less memory)
runtime = "crun"

# Set network backend driver for rootless mode (pasta preferred over slirp4netns)
network_cmd_path = "/usr/bin/pasta"

# Set default log driver (journald integrates natively with host logging)
log_driver = "journald"

# Set stop signal timeout default in seconds
stop_timeout = 10

[containers]
# Automatically mount /etc/hosts from host into container
netns = "private"

# Default environment variables injected into all containers
env = [
    "TZ=UTC",
]

# Set default resource limits
pids_limit = 2048

Precedence and Validation

Configuration parameters follow a strict loading precedence:

  1. Hardcoded CLI flags (e.g., --runtime=crun).
  2. User configuration (~/.config/containers/containers.conf).
  3. System configuration (/etc/containers/containers.conf).
  4. Distribution vendor defaults (/usr/share/containers/containers.conf).

To validate the effective configuration after making changes:

# Synthesze and print final active configuration state
podman info --format json | jq '.host.ociRuntime, .store.graphDriverName'

6. Service Management and Common Operations

Because Podman lacks a background daemon, traditional service management focuses on controlling individual container states, systemd integration units, and optional socket listeners.

Rootful vs. Rootless Execution Modes

  • Rootful Execution: Commands run with sudo podman. Storage is written to /var/lib/containers/storage. Containers have direct access to host network interfaces, kernel modules, and low ports (< 1024).
  • Rootless Execution: Commands run as standard unprivileged users. Storage lives under ~/.local/share/containers/storage. Process privileges are restricted by user namespace boundaries.

Enabling the Docker API Socket (Compatibility Layer)

Tools like Ansible Docker modules, Docker CLI plugins, or CI/CD scripts that require access to /var/run/docker.sock can interact with Podman’s background UNIX socket service.

# For Rootful environments: Enable system-wide Podman socket
sudo systemctl enable --now podman.socket

# For Rootless environments: Enable user socket and set DOCKER_HOST environment variable
systemctl --user enable --now podman.socket
export DOCKER_HOST="unix:///run/user/$(id -u)/podman/podman.sock"

# Test API response using docker-compatible CLI or curl
curl --unix-socket $XDG_RUNTIME_DIR/podman/podman.sock http://localhost/_ping

Container Operations: Reload vs. Restart

  • Reload (podman exec ... / SIGHUP): Sends a reload signal to the PID 1 workload inside the container to re-read internal application configurations without destroying the container instance.
  • Restart (podman restart <name>): Instructs conmon to stop the active process tree (via SIGTERM followed by SIGKILL) and invoke the OCI runtime to re-initialize container namespaces from scratch.

Essential Service Management Commands

# Check running container status
podman ps

# Display all containers (including stopped and exited states)
podman ps -a

# Inspect host listening ports associated with container bindings
ss -tlnp | grep podman

# Check resource consumption across active containers in real-time
podman stats --no-stream

7. Common Administrative Tasks and Workflows

Task 1: Deploying a Pod with Shared Network and Persistent Storage

Objective

Deploy a multi-container pod containing an Nginx reverse proxy and a Redis backend sharing a private network namespace, persistent storage, and correct SELinux labeling.

Step-by-Step Procedure

  1. Create Host Data Directory:
mkdir -p ~/app-data/redis
  1. Create the Pod with Mapped Ports:
# Create pod named 'web-app' publishing port 8080 on the host
podman pod create --name web-app -p 8080:80
  1. Deploy the Backend Redis Container into the Pod:
# Attach Redis container to 'web-app' pod using SELinux :Z volume labeling
podman run -d --pod web-app --name redis-db \
  -v ~/app-data/redis:/data:Z \
  redis:alpine
  1. Deploy the Frontend Nginx Container into the Pod:
# Attach Nginx container to 'web-app' pod (shares localhost network with redis-db)
podman run -d --pod web-app --name nginx-frontend \
  nginx:alpine
  1. Validation:
# Inspect pod status and contained processes
podman pod ps
podman ps --filter pod=web-app
curl http://localhost:8080

Task 2: Converting Container Workloads to Declarative Systemd Services using Quadlet

Objective

Migrate an imperative container deployment to a declarative, systemd-managed daemon service that starts automatically on host boot.

Prerequisites

Enable user linger so user services start at boot without an active SSH session:

loginctl enable-linger $USER

Step-by-Step Procedure

  1. Create Quadlet Configuration Directory:
mkdir -p ~/.config/containers/systemd/
  1. Create a .container File (~/.config/containers/systemd/webapp.container):
[Unit]
Description=Production Web Application Container
After=network-online.target

[Container]
Image=docker.io/library/nginx:alpine
ContainerName=production-webapp
PublishPort=8080:80
Exec=nginx -g 'daemon off;'
Volume=%h/webapp-data:/usr/share/nginx/html:Z

[Service]
Restart=always

[Install]
WantedBy=default.target
  1. Reload Systemd and Start Service:
# Notify systemd generator to process Quadlet files and build unit files
systemctl --user daemon-reload

# Start and enable the generated service
systemctl --user enable --now webapp.service
  1. Validation:
# Inspect systemd unit status
systemctl --user status webapp.service

# Verify underlying Podman container status
podman ps --filter name=production-webapp

Task 3: Managing Container Images securely using Buildah and Skopeo

Objective

Inspect remote image metadata without downloading layers, copy images directly between registries, and execute a scriptable image build using Buildah.

Step-by-Step Procedure

  1. Inspect Remote Image Metadata with Skopeo:
# Query image labels and environment variables directly from registry
skopeo inspect docker://docker.io/library/alpine:latest
  1. Copy Image directly between Registries / Storage Backends:
# Copy remote image directly into local Podman container storage
skopeo copy docker://docker.io/library/alpine:latest containers-storage:alpine:local
  1. Build Custom Image using Buildah Shell Scripting:
# Create working container mount point
container=$(buildah from alpine)

# Mount root filesystem to host path
mountpoint=$(buildah mount $container)

# Modify container filesystem directly from host shell
echo "hello from buildah" > $mountpoint/var/www/index.html
buildah config --entrypoint "cat /var/www/index.html" $container

# Commit and save new image layer
buildah commit $container custom-app:1.0
buildah unmount $container
buildah rm $container
  1. Validation:
# Verify image presence in Podman storage engine
podman images | grep custom-app
podman run --rm custom-app:1.0

Task 4: Configuring Automated Container Updates (podman auto-update)

Objective

Configure automated registry updates and zero-downtime container rollouts using Podman auto-update policies.

Step-by-Step Procedure

  1. Deploy Container with Auto-Update Label:
podman run -d --name auto-nginx \
  --label "io.containers.autoupdate=registry" \
  -p 8081:80 registry.access.redhat.com/ubi9/nginx-120
  1. Trigger Dry-Run Update Scan:
# Check if remote registry contains newer image digest without restarting
podman auto-update --dry-run
  1. Perform Live Update Execution:
# Pull new image layers and restart affected container workloads automatically
podman auto-update
  1. Enable Systemd Auto-Update Timer:
# Enable native timer for periodic update scans (default: daily)
systemctl --user enable --now podman-auto-update.timer

8. CLI Command Reference and Cheatsheet

1. Engine Information and Runtime Diagnostics

Command Purpose Important Flags / Arguments Practical Example Explanation
podman info Inspect engine host environment --format json `podman info grep -i rootless`
podman version Print Podman binary versions --format podman version Displays client API, engine runtime, and Go compiler version details.
podman system df Display disk space usage None podman system df Shows space consumed by images, active containers, and volumes.
podman system reset DESTRUCTIVE: Reset system state -f, --force podman system reset Completely wipes all local container storage, networks, and configurations.

2. Container Lifecycle Operations

Command Purpose Important Flags / Arguments Practical Example Explanation
podman run Create and start container -d (detach), -p (port), -v (volume), --name podman run -d --name web -p 8080:80 nginx:alpine Provisions namespaces, storage, networking, and executes PID 1 workload.
podman ps List containers -a (all), -q (quiet), --filter podman ps -a --filter status=exited Filters and displays active or terminated container processes.
podman exec Run command in container -it (interactive tty), -u (user) podman exec -it web sh Spawns a secondary process inside an existing container namespace.
podman stop Graceful stop container -t, --time (timeout sec) podman stop -t 10 web Sends SIGTERM to container PID 1, followed by SIGKILL after timeout.
podman rm Remove container instance -f, --force (kill running), -v (remove volumes) podman rm -f web Deletes container metadata and read-write storage layer.

3. Image and Registry Management

Command Purpose Important Flags / Arguments Practical Example Explanation
podman pull Pull image from registry --quiet podman pull nginx:alpine Downloads remote layers into local containers/storage.
podman images List local images -a (all), --format podman images Displays repositories, tags, image IDs, and layer sizes.
podman rmi Remove local image -f (force) podman rmi nginx:alpine Deletes image layers from local host storage engine.
podman build Build container image -t (tag), -f (file) podman build -t myapp:1.0 . Executes Containerfile instructions using underlying Buildah engine.

4. Pod Management

Command Purpose Important Flags / Arguments Practical Example Explanation
podman pod create Create new empty pod --name, -p (port map) podman pod create --name mypod -p 8080:80 Initializes infra container and shared network namespace.
podman pod ps List pods --ctr-status podman pod ps Displays active pods, infra container IDs, and container counts.
podman pod rm Remove pod instance -f, --force podman pod rm mypod Deletes pod infra container and all associated workloads.

5. Compose and Kubernetes Interoperability

Command Purpose Important Flags / Arguments Practical Example Explanation
podman compose Manage multi-container stack up -d, down, ps podman compose up -d Executes docker-compose.yml stacks via Podman backend.
podman play kube Deploy Kubernetes YAML --configmap podman play kube deployment.yaml Reads K8s Pod/Deployment manifests and creates local Podman pods.
podman generate kube Export K8s manifest --service podman generate kube webapp > pod.yaml Translates existing Podman containers/pods into K8s YAML specs.

9. Logs, Monitoring and Diagnostics

Effective observation of Podman workloads requires combining standard CLI utilities with host-level system diagnostics.

Log Inspection and Journald Integration

By default, Podman uses journald or k8s-file as log drivers.

# View container logs directly using Podman CLI
podman logs --tail 50 -f web-server

# View container logs with timestamps
podman logs --timestamps web-server

# If log_driver is journald, query host systemd journal directly
journalctl --user -u container-web-server.service -n 100 --no-pager

Process, Port, and Resource Inspection

# Inspect processes running inside a container (ps on host)
podman top web-server aux

# Extract specific JSON runtime state parameters using jq
podman inspect web-server | jq '.[0] | {Status: .State.Status, ExitCode: .State.ExitCode, OOMKilled: .State.OOMKilled}'

# Inspect volume mount points and host paths
podman inspect web-server | jq '.[0].Mounts'

# Verify mapped network port bindings
podman port web-server

System Health and Metric Monitoring

  • Systemd Cgroup Monitoring: Use systemd-cgtop to monitor CPU, memory, and I/O consumption by container cgroups in real-time.
  • Audit SELinux Denials: Query audit logs for AVC (Access Vector Cache) security violations triggered by container mount operations:
ausearch -m avc -ts recent 2>/dev/null | tail -n 15

10. Troubleshooting Guide and Common Failure Scenarios

This section outlines diagnostic workflows for common production failure scenarios.


Scenario 1: “Cannot Connect to Docker Daemon” Error

Symptoms

Executing a CLI command, script, or docker-compose wrapper fails with the error message: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Possible Causes

  1. A build script or third-party tool explicitly targets /var/run/docker.sock expecting standard Docker.
  2. Podman operates daemonlessly and does not automatically listen on a UNIX socket unless instructed.

Investigation and Diagnosis

  1. Verify if Podman CLI functions standalone:
podman ps
  1. Check if the Podman API socket service is active:
systemctl --user status podman.socket
  1. Inspect DOCKER_HOST environment variable:
echo $DOCKER_HOST

Root Cause Identification

The client tool is configured to communicate exclusively over a socket path that has not been initialized.

Resolution

  • Option A (Shell Alias): For scripts requiring CLI syntax, set an alias:
alias docker=podman
  • Option B (Socket Activation): Start the Podman compatibility socket:
# Enable user socket
systemctl --user enable --now podman.socket

# Export variable in ~/.bashrc
export DOCKER_HOST="unix:///run/user/$(id -u)/podman/podman.sock"

Verification

Execute docker version or query the socket directly:

curl --unix-socket $XDG_RUNTIME_DIR/podman/podman.sock http://localhost/_ping

Prevention

Standardize deployment manifests on native Podman commands or Quadlet units instead of legacy Docker daemon scripts.


Scenario 2: SELinux “Permission Denied” on Volume Mounts

Symptoms

A container fails to read or write to a host bind-mounted directory, throwing Permission denied errors inside application logs. Host file permissions (chmod 777) appear completely open.

Possible Causes

SELinux Mandatory Access Control (MAC) on RHEL/Fedora blocks container process access to files carrying standard host SELinux labels (e.g., user_home_t).

Investigation and Diagnosis

  1. Inspect audit logs for AVC denials:
ausearch -m avc -ts recent
  1. Check file security context of the host mount directory:
ls -Z /path/to/host/dir

Root Cause Identification

An AVC rejection matching scontext=system_u:system_r:container_t confirms SELinux label isolation is blocking file access.

Resolution

Append SELinux relabeling flags to the volume volume flag:

  • Use :Z to grant private unshared container access:
podman run -v /host/data:/container/data:Z myapp
  • Use :z to grant shared access across multiple containers:
podman run -v /host/data:/container/data:z myapp

Verification

Verify container can read/write to mount:

podman run --rm -v /host/data:/data:Z alpine touch /data/testfile

Prevention

Ensure all Quadlet unit definitions and automated scripts explicitly specify :Z or :z mount parameters on host bind volumes.


Scenario 3: Rootless Volume UID/GID Permission Mismatch

Symptoms

Container process fails with Operation not permitted or cannot write to mapped storage, even when SELinux is disabled. Host ls -l shows files owned by arbitrary high numeric UIDs (e.g., 100999).

Possible Causes

Rootless Podman maps internal container UIDs to subordinate ranges defined in /etc/subuid. Container root (UID 0) maps to the host user’s actual UID, but internal non-root users (e.g., nginx UID 101) map to subordinate host UIDs (e.g., 100100).

Investigation and Diagnosis

  1. Check host user subordinate mapping:
cat /etc/subuid /etc/subgid
  1. Inspect target user UID defined inside container configuration:
podman inspect myapp | jq '.[0].Config.User'

Root Cause Identification

Directory ownership on the host belongs to host UID 1000, while the container process executes as internal UID 1000 (which maps to host UID 100999).

Resolution

  • Fix 1 (Host Permission Adjustment using podman unshare): Execute directory ownership modification within the user namespace:
podman unshare chown -R 1000:1000 /host/data
  • Fix 2 (Keep Host User ID Flag): Pass --userns=keep-id so container UID 1000 directly matches host UID 1000:
podman run -v /host/data:/data --userns=keep-id myapp

Verification

Confirm directory write access:

podman run --rm --userns=keep-id -v /host/data:/data myapp touch /data/test.txt

Prevention

Standardize UID definitions in custom container images and use podman unshare when prepping volume directories on host filesystems.


Scenario 4: Container Exits Immediately Upon Startup

Symptoms

Running podman run completes instantly, but podman ps shows no active containers. podman ps -a shows container state as Exited (1) or Exited (137).

Possible Causes

  1. The containerized entrypoint binary terminated immediately (missing foreground execution parameter).
  2. Container encountered out-of-memory (OOM) killer constraints.
  3. Missing essential environment variables or privileges.

Investigation and Diagnosis

  1. Check exit code and OOM status:
podman inspect NAME | jq '.[0] | {ExitCode: .State.ExitCode, OOMKilled: .State.OOMKilled}'
  1. Retrieve raw logs:
podman logs --tail 50 NAME
  1. Test container interactively with an alternative shell entrypoint:
podman run -it --entrypoint sh IMAGE

Root Cause Identification

Reviewing standard error output in logs reveals missing runtime dependencies or entrypoint application termination.

Resolution

Adjust application execution flags to maintain foreground processing (e.g., nginx -g 'daemon off;') or allocate increased memory boundaries.

Verification

Verify sustained container execution:

podman ps --filter name=NAME

Scenario 5: Port Binding Failure (Privileged Port or Address In Use)

Symptoms

Container fails to start, outputting Error: rootlessport listen tcp 0.0.0.0:80: bind: permission denied or address already in use.

Possible Causes

  1. Standard Linux security prevents non-root users from binding low privileged host ports (< 1024).
  2. Another host service is actively bound to the requested port.

Investigation and Diagnosis

  1. Check if target host port is currently occupied:
ss -tlnp | grep :80
  1. Inspect unprivileged port start sysctl setting:
cat /proc/sys/net/ipv4/ip_unprivileged_port_start

Root Cause Identification

Rootless container attempt to bind host port 80 fails because host ip_unprivileged_port_start defaults to 1024.

Resolution

  • Option A (Use Unprivileged High Port): Map host port 8080 to container port 80:
podman run -p 8080:80 nginx:alpine
  • Option B (Lower Kernel Port Boundary): Adjust kernel parameters to permit unprivileged low port binding:
sudo sysctl -w net.ipv4.ip_unprivileged_port_start=80
echo "net.ipv4.ip_unprivileged_port_start=80" | sudo tee /etc/sysctl.d/99-podman.conf

Verification

Confirm listener initialization:

podman run -d -p 80:80 nginx:alpine && ss -tlnp | grep :80

Scenario 6: Rootless Network Isolation / Cannot Reach External LAN

Symptoms

Containerized applications cannot resolve external DNS names or reach host network interfaces.

Possible Causes

Rootless networking uses user-space tap abstractions (pasta or slirp4netns). Unlike rootful bridge interfaces, user networks do not automatically inherit raw host interface routing tables.

Investigation and Diagnosis

  1. Test outbound ICMP and DNS connectivity inside container:
podman run --rm alpine ping -c 2 1.1.1.1
podman run --rm alpine nslookup google.com
  1. Verify active net driver in podman info:
podman info | grep networkBackend

Root Cause Identification

slirp4netns network stack overhead or invalid network configuration in ~/.config/cni/net.d/.

Resolution

Switch network backend driver to modern pasta or pass host network mode explicitly if isolation is not required:

# Force container to share host network namespace directly
podman run --network host myapp

Verification

Verify DNS query resolution:

podman run --rm --network host alpine nslookup google.com

Scenario 7: podman compose Execution Failure

Symptoms

Running podman compose up fails with command not found, or returns syntax errors on supported compose keys.

Possible Causes

  1. podman-compose binary or Compose v2 plugin is missing.
  2. Environment defaults to legacy docker-compose engine binaries.

Investigation and Diagnosis

  1. Verify installed Compose provider version:
podman compose version
  1. Inspect environment variables governing Compose routing:
echo $COMPOSE_PROVIDER

Root Cause Identification

Missing Compose plugin engine binaries or invalid provider mapping.

Resolution

Install podman-compose plugin and export provider settings:

# Install package
sudo dnf install -y podman-compose

# Force provider export in environment
export COMPOSE_PROVIDER=podman

Verification

podman compose up -d && podman compose ps

Scenario 8: Disk Space Exhaustion (“No Space Left on Device”)

Symptoms

Image pulls, builds, or container execution fail with error writing layer: No space left on device.

Possible Causes

Dangling images, unreferenced layer caches, or abandoned volume files consume full disk allocations under ~/.local/share/containers/.

Investigation and Diagnosis

  1. Audit Podman storage utilization:
podman system df
  1. Inspect host mount points:
df -h ~/.local/share/containers/

Root Cause Identification

Accumulation of unreferenced dangling container layers.

Resolution

Execute comprehensive system storage pruning:

# Prune all stopped containers, unreferenced networks, and unused images
podman system prune -a --volumes

Verification

Verify freed disk capacity:

podman system df

Scenario 9: Systemd Service Fails to Start Container on Host Boot

Symptoms

A user systemd unit or Quadlet service works during active SSH sessions, but fails to launch automatically after a server reboot.

Possible Causes

By default, systemd terminates user session daemons and background tasks when the unprivileged user logs out.

Investigation and Diagnosis

  1. Query systemd user linger status:
ls /var/lib/systemd/linger/$USER
  1. Check user service logs post-reboot:
journalctl --user -u container-myapp.service

Root Cause Identification

User account lacks session linger activation.

Resolution

Enable persistent session linger for the unprivileged administrative user:

loginctl enable-linger $USER

Verification

Verify linger file existence:

loginctl show-user $USER | grep Linger

Symptoms

Pulling an image fails with Error: short-name "nginx" did not match any image or x509: certificate signed by unknown authority.

Possible Causes

  1. Missing unqualified search registries in /etc/containers/registries.conf.
  2. Self-signed TLS certificate missing from host trust store.
  3. Invalid registry authentication credentials.

Investigation and Diagnosis

  1. Test pull using fully qualified image path:
podman pull docker.io/library/nginx:alpine
  1. Inspect search registries configuration:
cat /etc/containers/registries.conf

Root Cause Identification

Short name resolution policy requires explicit registry prefixing or configuration rules.

Resolution

  • Update /etc/containers/registries.conf search configurations:
unqualified-search-registries = ["docker.io", "quay.io", "registry.access.redhat.com"]
  • Perform registry authentication login:
podman login registry.example.com

Verification

Verify short-name image resolution:

podman pull alpine:latest

11. Security and Best Practices

Rootless Isolation Principles

Always operate workloads rootless whenever possible. Running rootless ensures that even if a container process suffers a total application breakout, the attacker gains access only to the unprivileged host user account, preventing kernel tampering or hardware control.

Mandatory Security Controls

  • SELinux Relabeling: Always use explicit :Z or :z volume flags. Never set SELinux to permissive on production container hosts.
  • Drop Unnecessary Linux Capabilities: Restrict runtime capabilities to absolute minimums:
podman run -d --cap-drop=ALL --cap-add=NET_BIND_SERVICE nginx:alpine
  • Read-Only Root Filesystems: Enforce read-only container root structures to block unauthorized binary drops:
podman run -d --read-only --tmpfs /tmp --tmpfs /run nginx:alpine
  • Secret Management: Avoid storing production credentials inside environment variables or image layers. Use Podman native secrets management:
# Create secret from file input
podman secret create db_password ./pass.txt

# Mount secret into target container
podman run -d --secret db_password myapp

12. Performance and Resource Optimisation

Network Driver Optimization

Driver Use Case Latency / Throughput Resource Overhead
Pasta Default Rootless (Podman 5+) High throughput, near-native host speed Low CPU usage
Slirp4netns Legacy Rootless Moderate throughput, higher latency Higher user-space CPU usage
Host Network High-Performance Workloads Zero overhead (shares host namespace) Low isolation

Storage Performance: Native Overlay vs. Fuse-Overlayfs

On modern Linux kernels (5.11+), Podman defaults to native unprivileged overlay mounts. On older kernels lacking unprivileged overlay support, Podman falls back to fuse-overlayfs. To verify active graph driver parameters:

podman info | grep -i graphdrivername

Ensure native overlay is active for optimal I/O throughput.

Cgroup v2 Resource Limitations

Enforce granular memory, CPU, and process limitations on container workloads:

podman run -d --name constrained-app \
  --memory=512m \
  --cpus=1.5 \
  --pids-limit=200 \
  myapp:1.0

13. Backup, Recovery and Maintenance

Volume and Configuration Backup Workflows

1. Exporting Named Volumes

# Export volume data into compressed tarball archive
podman volume export my-data-volume > /backups/my-data-volume.tar

2. Restoring Named Volumes

# Create target volume
podman volume create my-restored-volume

# Import tarball content into restored volume
podman volume import my-restored-volume /backups/my-data-volume.tar

3. Backing Up Quadlet Declarative Files

Backup configuration state by archiving user and system Quadlet directories:

tar -czvf quadlet-backup.tar.gz ~/.config/containers/systemd/ /etc/containers/systemd/

Maintenance and Pruning Protocols

Establish scheduled maintenance jobs to prevent storage exhaustion:

# Weekly cron or timer job to prune unused images and volumes
podman system prune -a --force --volumes

14. Automation and Integration

Ansible Automation Integration

Automate Podman deployments using Ansible’s official containers.podman collection:

- name: Deploy Production Quadlet Container Service
  hosts: webservers
  tasks:
    - name: Ensure Quadlet user directory exists
      ansible.builtin.file:
        path: ~/.config/containers/systemd
        state: directory
        mode: '0755'

    - name: Copy Quadlet file definition
      ansible.builtin.copy:
        src: webapp.container
        dest: ~/.config/containers/systemd/webapp.container
        mode: '0644'
      notify: Reload Systemd

  handlers:
    - name: Reload Systemd
      ansible.builtin.systemd:
        daemon_reload: true
        scope: user
        name: webapp.service
        state: restarted

Kubernetes Manifest Prototyping

Use podman play kube to test production Kubernetes pod definitions on local developer workstations prior to cluster deployment:

# Generate Kubernetes Pod YAML from live Podman workload
podman generate kube web-app > deployment.yaml

# Tear down local pod
podman pod rm -f web-app

# Re-deploy pod directly from Kubernetes YAML definition
podman play kube deployment.yaml

15. Practical Labs and Exercises

Lab 1 (Beginner): Deploying an Unprivileged Nginx Web Server

Objective

Deploy an unprivileged, detached Nginx container using custom port mapping and host data volume binding.

Instructions

  1. Create a directory named ~/lab1-html on the host and write an index.html file inside it containing “Podman Lab 1”.
  2. Launch an unprivileged Nginx container named lab1-web mapping host port 8085 to container port 80.
  3. Bind mount ~/lab1-html to /usr/share/nginx/html with correct SELinux flags.

Command Solution

mkdir -p ~/lab1-html
echo "Podman Lab 1" > ~/lab1-html/index.html
podman run -d --name lab1-web -p 8085:80 -v ~/lab1-html:/usr/share/nginx/html:Z nginx:alpine

Validation

curl http://localhost:8085

Lab 2 (Intermediate): Building a Multi-Container Pod

Objective

Build a Pod named analytics-pod containing a Python Flask app container and a Redis container sharing network space.

Instructions

  1. Create a pod named analytics-pod exposing host port 5000.
  2. Deploy a Redis database container into analytics-pod.
  3. Deploy an application container into analytics-pod that queries localhost:6379.

Command Solution

podman pod create --name analytics-pod -p 5000:5000
podman run -d --pod analytics-pod --name redis-backend redis:alpine
podman run -d --pod analytics-pod --name app-frontend training/webapp

Validation

podman pod ps
curl http://localhost:5000

Lab 3 (Advanced/Troubleshooting): Declarative Quadlet Service Deployment

Objective

Deploy a Quadlet container unit under an unprivileged user, configure systemd lingering, and resolve volume permission errors.

Instructions

  1. Enable user linger for your user account.
  2. Create ~/.config/containers/systemd/db.container running PostgreSQL 15.
  3. Mount host directory ~/db-data to /var/lib/postgresql/data with private SELinux flags (:Z).
  4. Enable and verify service activation via systemctl --user.

Command Solution

loginctl enable-linger $USER
mkdir -p ~/.config/containers/systemd ~/db-data

cat << 'EOF' > ~/.config/containers/systemd/db.container
[Unit]
Description=PostgreSQL Database Service

[Container]
Image=docker.io/library/postgres:15-alpine
ContainerName=postgres-db
Environment=POSTGRES_PASSWORD=secretpass
Volume=%h/db-data:/var/lib/postgresql/data:Z

[Service]
Restart=always

[Install]
WantedBy=default.target
EOF

systemctl --user daemon-reload
systemctl --user enable --now db.service

Validation

systemctl --user status db.service
podman ps --filter name=postgres-db

16. Quick Reference and Cheat Sheet

Important Configuration Files

File / Path Purpose
/etc/containers/containers.conf System-wide engine configuration defaults
~/.config/containers/containers.conf User rootless engine configuration overrides
/etc/containers/storage.conf Storage driver configuration (overlay, paths)
/etc/containers/registries.conf Registry search paths and security policies
/etc/subuid & /etc/subgid Subordinate UID/GID user mapping tables
~/.config/containers/systemd/ User Quadlet declarative systemd file location

Important Log Locations

Log Target Command / Location Purpose
Podman Standard Logs podman logs <container> Direct output stream of container PID 1
Systemd User Journal journalctl --user -u container-<name> Quadlet service execution logs
SELinux Audit Log /var/log/audit/audit.log Security context and AVC denial records

Common Administrative Commands

Task Syntax Command
Check Rootless Status `podman info
Run Detached Container podman run -d --name <name> -p <hport>:<cport> -v /host:/cont:Z <image>
View Active Containers podman ps
Shell into Container podman exec -it <name> sh
Create Container Pod podman pod create --name <podname> -p 8080:80
Enable Docker Socket systemctl --user enable --now podman.socket
Reload Quadlet Units systemctl --user daemon-reload
Prune Unused Storage podman system prune -a --volumes

Troubleshooting Quick Reference

Symptom Initial Diagnostic Command Primary Fix
Docker Daemon Socket Missing systemctl --user status podman.socket Enable socket or set alias docker=podman
Volume Permission Denied ausearch -m avc -ts recent Append :Z or :z to volume flag
UID Permission Mismatch cat /etc/subuid Run podman unshare chown or --userns=keep-id
privileged Port Bind Failure sysctl net.ipv4.ip_unprivileged_port_start Use port > 1024 or adjust sysctl
User Unit Dies on Logout loginctl show-user $USER Enable linger: loginctl enable-linger $USER

17. Further Reading and References

  1. Official Documentation & Manuals:
  1. Container Infrastructure Standards:
  1. Interactive Practice Labs: