Domain 10Linux & SREINTERMEDIATE

Mastering systemd: Architecture, Operations, Unit Hardening, and Production Troubleshooting

A comprehensive operational handbook on systemd for Linux SysAdmins and SREs, detailing architecture, unit lifecycle, service control, timer automation, sandboxing, journald diagnostics, and step-by-step production troubleshooting workflows.

Domain: Domain 10: Linux Internals, SRE & Troubleshooting•Updated: 2026-11-03
#linux#sysadmin#systemd#sre#troubleshooting

1. Overview

systemd is the standard init system and system/service manager for modern Linux distributions. Operating as Process ID 1 (PID 1), systemd is the first user-space process executed by the Linux kernel during boot. It remains active until system shutdown, supervising system initialization, daemon lifecycle, dependency resolution, hardware event handling, resource control, and centralized logging.

Problem Statement and Evolution

Prior to systemd, traditional Linux init systems like SysVinit relied on shell scripts located in /etc/init.d/ executed sequentially based on runlevels. SysVinit suffered from severe limitations in cloud and modern enterprise infrastructure:

  • Sequential Boot Delays: Services were forced to start one after another, leading to slow boot times on multi-core systems.
  • Fragile Process Tracking: SysVinit relied on daemon-written PID files (/var/run/*.pid). If a process crashed without cleaning up its PID file or spawned child worker processes, SysVinit lost track of active processes, leaving orphan processes behind.
  • Ad-hoc Dependency Handling: Scripts used crude numerical ordering or manual sleep commands to handle initialization order, resulting in race conditions during startup.

systemd addresses these architectural weaknesses by leveraging modern Linux kernel mechanisms—specifically cgroups (Control Groups), D-Bus IPC, and socket activation.

Key Operational Advantages and Limitations

Feature Category Advantages Limitations / Trade-offs
Process Tracking Tracks processes using kernel cgroups; child processes cannot escape supervision even if they fork or clear their PID. Complex monolithic scope compared to simple UNIX “do one thing well” philosophy.
Startup Speed Resolves dependencies dynamically and launches independent units concurrently in parallel. Cascading failures can occur rapidly if strict dependencies (Requires=) are misconfigured.
Unified Logging Captures stdout/stderr from all supervised processes into a structured binary journal (journald). Binary log files require dedicated tools (journalctl) to query.
Declarative Config Replaces fragile shell scripts with standard declarative unit files. Requires learning systemd-specific syntax, directives, and lifecycle states.

Practical Purpose Example

To start an Nginx web server and ensure it launches automatically when the server boots, SysVinit required executing shell initialization scripts and manually creating runlevel symlinks. Under systemd, this administrative intent is declared with two distinct operations:

# Immediately launch the Nginx service in the current session
sudo systemctl start nginx

# Create filesystem symlinks to auto-start Nginx during boot target evaluation
sudo systemctl enable nginx

Systemd keeps these operations explicit: start affects the current runtime state, while enable persistence settings across reboots.


2. Architecture and Mental Model

To effectively operate and troubleshoot systemd, system administrators must understand how PID 1 interacts with the kernel, system d-bus, unit state engines, and cgroups v2.

High-Level Architecture Diagram

graph TD
    Kernel[Linux Kernel] -->|Spawns PID 1| PID1[systemd PID 1 Core Manager]
    
    subgraph "Systemd Core Subsystems"
        PID1 --> SocketEngine[Socket Activation Engine]
        PID1 --> DependencyEngine[Dependency & State Machine Engine]
        PID1 --> CgroupEngine[cgroups v2 Resource Controller]
    end
    
    subgraph "Unit Hierarchy"
        DependencyEngine --> DefaultTarget[default.target / multi-user.target]
        DefaultTarget --> WebService[nginx.service]
        DefaultTarget --> DBService[postgresql.service]
        DefaultTarget --> TimerUnit[backup.timer]
    end

    subgraph "Auxiliary Daemons"
        PID1 <--> Journald[systemd-journald.service]
        PID1 <--> Logind[systemd-logind.service]
        PID1 <--> Udevd[systemd-udevd.service]
    end

    Journald -->|Captures stdout/stderr| WebService
    Journald -->|Captures stdout/stderr| DBService

Core Subsystems and Auxiliary Daemons

  • PID 1 (systemd core): Manages the global unit dependency graph, executes process lifecycles, traps system signals, and supervises system state transitions.
  • systemd-journald: Intercepts standard output, standard error, kernel log messages, syslog events, and audit traces into structured binary journal files.
  • systemd-logind: Supervises user session logins, seat allocation, power management (suspend, reboot), and user-level systemd instances.
  • systemd-udevd: Listens for kernel uevent signals when hardware devices are added, removed, or state-changed, dynamically creating device nodes in /dev/.
  • systemd-resolved & systemd-timesyncd: Light-weight network name resolution and Network Time Protocol (NTP) synchronization services.

Boot Sequence and Dependency Resolution Lifecycle

When the machine powers on, systemd executes the following lifecycle:

+-----------------------------------------------------------------------------------+
| 1. Kernel Hand-off                                                                |
|    Kernel completes initial hardware setup, mounts rootfs, and executes PID 1.    |
+-----------------------------------------------------------------------------------+
                                          |
                                          v
+-----------------------------------------------------------------------------------+
| 2. Target Evaluation                                                              |
|    PID 1 loads default.target (usually a symlink to multi-user.target).           |
+-----------------------------------------------------------------------------------+
                                          |
                                          v
+-----------------------------------------------------------------------------------+
| 3. Dependency Graph Build & Ordering                                             |
|    Resolves Wants=, Requires=, After=, and Before= directives to build activation |
|    sequence.                                                                      |
+-----------------------------------------------------------------------------------+
                                          |
                                          v
+-----------------------------------------------------------------------------------+
| 4. Parallel Process Execution                                                    |
|    Executes ExecStart= for independent units concurrently once ordering criteria |
|    are met.                                                                       |
+-----------------------------------------------------------------------------------+
                                          |
                                          v
+-----------------------------------------------------------------------------------+
| 5. Steady State Supervision & Monitoring                                          |
|    Monitors cgroup processes; triggers Restart= rules on crash; routes logs to   |
|    journald.                                                                      |
+-----------------------------------------------------------------------------------+

3. Core Concepts and Terminology

Unit Types Overview

Everything managed by systemd is encapsulated in a Unit. The primary unit file extensions and their operational purposes are detailed below:

Unit Extension Primary Purpose Common Practical Example
.service Manages background daemons, applications, or one-time execution scripts. nginx.service, sshd.service
.socket Encapsulates IPC or network sockets for socket-based service activation. docker.socket, systemd-journald.socket
.timer Manages scheduled execution of paired .service units (replacing cron). logrotate.timer, fstrim.timer
.target Groups multiple units to establish system boot states or operational runlevels. multi-user.target, graphical.target
.mount Manages static filesystem mount points in system storage. tmp.mount, data-backup.mount
.automount Dynamically mounts filesystems upon direct access to the target path. proc-sys-fs-binfmt_misc.automount
.slice Sets up cgroup v2 sub-trees for resource allocation and limits. system.slice, user.slice
.scope Wraps externally started processes launched outside of systemd. session-1.scope
.path Triggers service execution based on filesystem file/directory changes. certbot-renew.path

Key Terminology Glossary

  • Drop-in Directory: A folder named <unit-name>.service.d/ containing partial .conf files used to extend or override default unit directives without altering vendor-packaged unit files.
  • Socket Activation: A pattern where systemd opens network or UNIX sockets during boot and passes the listening file descriptor to the service process when a client connects.
  • Control Group (cgroup v2): A Linux kernel feature used by systemd to group processes into hierarchical structures to monitor execution and enforce limits on CPU, memory, and I/O.
  • Transient Unit: Dynamically generated units created at runtime via systemd-run that evaporate upon system reboot.
  • Failed State: A state assigned to a unit when its monitored main process crashes, exits with an error code, hits timing thresholds, or encounters failed strict dependencies.

4. Installation and Version Management

Systemd is deeply integrated into major enterprise Linux distributions (including RHEL, Rocky Linux, AlmaLinux, Fedora, Debian, Ubuntu, openSUSE, and Arch Linux).

Verifying Installed Version and Build Capabilities

To check the installed systemd version, architecture features, and active build flags, run:

systemctl --version

Example Command Output:

systemd 252 (252.22-1.el9)
+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PKGCONFIG +PWQUALITY +BZIP2 +LZ4 +XZ +ZSTD +CHATTR +KMOD +LIBSECCOMP +HAVE_BLKID +HAVE_CLR +HAVE_TZDATA

Installation File Paths and Locations

The core binaries, configuration generators, and helper tools for systemd are distributed across standard binary directories:

  • /usr/lib/systemd/systemd: The main PID 1 system manager executable.
  • /usr/bin/systemctl: Command-line management client.
  • /usr/bin/journalctl: Log query utility.
  • /usr/lib/systemd/: System helper daemons (e.g., systemd-journald, systemd-udevd).

5. Configuration Files and Directory Structure

Configuration File Precedence

Systemd unit configuration follows a strict order of precedence. Higher-priority locations override settings found in lower-priority locations:

Order Path Usage & Ownership Override Behavior
1 (Highest) /etc/systemd/system/ System Administrator local units and overrides. Completely overrides unit files with identical names in vendor locations.
2 /etc/systemd/system/<unit>.service.d/*.conf Drop-in configuration overrides. Merges directives into the base unit without altering the base file.
3 /run/systemd/system/ Runtime transient units generated at runtime. Supersedes packaged defaults until reboot.
4 (Lowest) /usr/lib/systemd/system/ (or /lib/systemd/system/) Distribution vendor package defaults. Do not edit directly; changes will be overwritten by package updates.
User Space ~/.config/systemd/user/ Per-user custom unit files. Managed by the unprivileged user instance (systemctl --user).

Unit File Structure and Key Directives

A standard service unit file is divided into three distinct section headers:

[Unit]
Description=My Application Daemon
Documentation=man:myapp(8) https://docs.example.com
After=network.target mariadb.service
Wants=mariadb.service

[Service]
Type=simple
User=deploy
Group=deploy
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/server --config /etc/myapp/config.yaml
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
Environment="PORT=8080"
EnvironmentFile=-/etc/myapp/env

[Install]
WantedBy=multi-user.target

Directive Breakdown

[Unit] Section Directives
  • Description=: Descriptive string detailing the service purpose displayed in status reports.
  • After=: Ensures the unit starts after the specified units have completed initialization. (Ordering directive only—does not enforce strict dependency).
  • Before=: Ensures the unit starts before the specified units start.
  • Wants=: Weak dependency. Systemd attempts to start listed units, but failure of those units does not prevent this unit from starting.
  • Requires=: Strict dependency. If any listed unit fails to start or is stopped, this unit will fail or be deactivated immediately.
[Service] Section Directives
  • Type=: Defines how systemd manages process lifecycle state:

  • simple: Default. Systemd considers the service active immediately after ExecStart forks the main process.

  • forking: Suitable for legacy daemons that fork into the background. Systemd tracks the child PID reported by the parent process.

  • oneshot: Executed once and exits. Often paired with RemainAfterExit=yes so the unit remains marked active after the process completes.

  • notify: Similar to simple, but the daemon sends a notification signal via sd_notify() to systemd when initialization is complete.

  • ExecStart=: Full path to the executable command and arguments.

  • User= / Group=: Unix identity under which the process executes.

  • Restart=: Restart policy (no, on-success, on-failure, always).

  • RestartSec=: Time delay before retrying execution after a crash.

  • Environment= / EnvironmentFile=: Defines inline environment variables or paths to key-value files.

[Install] Section Directives
  • WantedBy=: Defines target units that create a dependency symlink when systemctl enable is executed.

Modifying Units via Drop-In Files and Systemd Edit

Instead of modifying upstream vendor files directly in /usr/lib/systemd/system/, administrators should use drop-in files. Running:

sudo systemctl edit nginx

Creates /etc/systemd/system/nginx.service.d/override.conf.

Crucial Rule — The Empty Value Reset Trap: When overriding single-value directives that accept multiple entries or executable lists (such as ExecStart=), you must first provide an empty assignment to clear the upstream definition before assigning the new value:

[Service]
# Clear existing vendor ExecStart directive first
ExecStart=
# Set new custom binary command line
ExecStart=/usr/local/bin/custom-nginx -g 'daemon off;'

After creating or modifying unit files, reload systemd configuration definitions:

sudo systemctl daemon-reload

6. Service Management and Common Operations

Core Service Control Commands

Administrators manage running services using systemctl actions:

# Start a service immediately
sudo systemctl start nginx.service

# Stop a running service
sudo systemctl stop nginx.service

# Restart a service (stops and starts main PID)
sudo systemctl restart nginx.service

# Reload configuration without dropping client socket connections (if supported)
sudo systemctl reload nginx.service

# Display process status, main PID, cgroup tree, and recent logs
systemctl status nginx.service

Enablement vs Runtime State

enable and start perform fundamentally different administrative operations:

# Enable service to start automatically during boot targets
sudo systemctl enable nginx

# Disable auto-start on boot (removes symlink under /etc/systemd/system/*.wants/)
sudo systemctl disable nginx

# Enable and start concurrently in a single command
sudo systemctl enable --now nginx

Inspecting Runtime States

# Verify if a unit is currently executing (returns active or inactive)
systemctl is-active nginx

# Verify if a unit is enabled to start on boot
systemctl is-enabled nginx

Unprivileged User Service Instances (--user)

Systemd allows non-privileged users to manage user-scoped daemons, local timers, and personal background tasks without root access.

  • User Unit Path: ~/.config/systemd/user/
  • User Management Flag: --user
# Start a user-managed application daemon
systemctl --user start custom-agent.service

# Enable a user service upon user login
systemctl --user enable custom-agent.service

# Inspect status of user units
systemctl --user status custom-agent.service

Note on Persistent User Services (Linger): By default, the systemd user instance starts when the user logs in and terminates when all user sessions log out. To allow non-root user services to start on system boot and stay running after logout, enable session lingering:

loginctl enable-linger deploy

7. Common Administrative Tasks and Workflows

Workflow 1: Writing and Deploying a Custom Python API Daemon

  1. Create dedicated service user:
sudo useradd -r -s /bin/false appuser
  1. Create unit file at /etc/systemd/system/api-server.service:
[Unit]
Description=Custom Python REST API Service
After=network.target

[Service]
Type=simple
User=appuser
Group=appuser
WorkingDirectory=/opt/api-server
ExecStart=/usr/bin/python3 /opt/api-server/main.py
Restart=on-failure
RestartSec=3s
Environment="PORT=5000"

[Install]
WantedBy=multi-user.target
  1. Reload daemon engine, enable, start, and verify:
sudo systemctl daemon-reload
sudo systemctl enable --now api-server.service
systemctl status api-server.service

Workflow 2: Creating a Drop-In Override for Resource Limits

To adjust MemoryMax and environment settings for api-server.service without altering the base unit file:

  1. Execute edit command:
sudo systemctl edit api-server.service
  1. Add configuration lines into the editor:
[Service]
Environment="PORT=8000" "NODE_ENV=production"
MemoryMax=1G
MemoryHigh=800M
  1. Save, reload, and verify effective settings:
sudo systemctl daemon-reload
sudo systemctl restart api-server.service
systemctl show api-server.service -p MemoryMax,Environment

Workflow 3: Setting Up a systemd Timer to Replace Cron

A systemd timer requires two matching files in /etc/systemd/system/: a .service unit and a .timer unit.

  1. Create service unit (/etc/systemd/system/db-backup.service):
[Unit]
Description=Database Nightly Backup Execution

[Service]
Type=oneshot
ExecStart=/usr/local/bin/db-backup.sh
  1. Create timer unit (/etc/systemd/system/db-backup.timer):
[Unit]
Description=Trigger Database Nightly Backup

[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=15m
Persistent=true

[Install]
WantedBy=timers.target
  1. Enable and verify schedule:
sudo systemctl daemon-reload
sudo systemctl enable --now db-backup.timer
systemctl list-timers --all

8. CLI Command Reference and Cheatsheet

Service Control Commands

Task Command Description
Start Service systemctl start <unit> Starts a service immediately.
Stop Service systemctl stop <unit> Stops a running service.
Restart Service systemctl restart <unit> Restarts a service.
Reload Service systemctl reload <unit> Asynchronously reloads service configuration.
View Status systemctl status <unit> Shows running state, PID, cgroups, and recent logs.
Enable Boot Start systemctl enable <unit> Creates symlinks for startup at boot.
Disable Boot Start systemctl disable <unit> Removes enablement symlinks.
Check Active State systemctl is-active <unit> Prints active or inactive (useful for scripts).
Check Enabled State systemctl is-enabled <unit> Prints enabled or disabled.

Unit Listing and Inspection Commands

Task Command Description
List Loaded Services systemctl list-units --type=service Lists all active and loaded service units.
List Failed Units systemctl --failed Displays all failed units across services, mounts, and timers.
List Explicit Failed systemctl list-units --state=failed Explicit filter for failed units.
Reset Failed State systemctl reset-failed [<unit>] Clears failed flag on units after applying fixes.
List Installed Files systemctl list-unit-files Displays enablement state of all installed unit files.
List Active Timers systemctl list-timers --all Shows active and inactive timer schedules.
Show All Properties systemctl show <unit> Output low-level internal key-value unit variables.
Inspect Property systemctl show <unit> -p ExecMainStatus,Result Output targeted runtime property values.

Unit Management and Validation

Task Command Description
View Consolidated Unit systemctl cat <unit> Shows base unit contents along with drop-in overrides.
Create Drop-in systemctl edit <unit> Opens editor for drop-in overrides in /etc/systemd/system/.
Edit Base Unit systemctl edit --full <unit> Creates full unit copy in /etc/systemd/system/ to edit.
Reload Configuration systemctl daemon-reload Rescans filesystem for unit changes and updates dependency graph.
Validate Unit Syntax systemd-analyze verify <file.service> Parses unit file for syntax errors and directive typos.
Mask Unit systemctl mask <unit> Symlinks unit to /dev/null to block all manual and automatic starts.
Unmask Unit systemctl unmask <unit> Removes mask symlink, restoring standard start capability.

Journal Log Query Commands

Task Command Description
Query Unit Logs journalctl -u <unit> Shows log entries for a specific unit.
Follow Logs Live journalctl -u <unit> -f Tail logs in real time.
Filter Since Time journalctl -u <unit> --since today Filter entries generated since midnight.
Filter Current Boot journalctl -b Show entries generated during the current boot session.
Filter Priority Error journalctl -p err -b Show error-level (and higher) logs for current boot.
Check Disk Usage journalctl --disk-usage Displays filesystem storage consumed by binary logs.

Boot Analysis and Performance Commands

Task Command Description
View Boot Time systemd-analyze Displays kernel vs user-space boot timing total.
View Boot Blame systemd-analyze blame Prints units sorted by initialization duration.
Critical Chain Tree systemd-analyze critical-chain Displays time-critical dependency boot chain.
View Default Target systemctl get-default Shows target unit evaluated on boot.
Switch Target systemctl isolate rescue.target Switches system to rescue state without rebooting.

9. Logs, Monitoring and Diagnostics

Journald Logging Architecture

systemd-journald captures stdout and stderr outputs directly from supervised process file descriptors. Binary journal files reside in:

  • /run/log/journal/: Volatile storage (cleared on reboot).
  • /var/log/journal/: Persistent storage.
# Query recent lines without pagination truncation
journalctl -u nginx.service -n 100 --no-pager

Advanced Journal Filtering Examples

# Filter logs between specific timestamp bounds
journalctl -u postgresql.service --since "2026-11-01 08:00:00" --until "2026-11-03 12:00:00"

# Filter logs by executable PID
journalctl _PID=4521

# Filter by syslog priority level (0: emerg, 1: alert, 2: crit, 3: err, 4: warning)
journalctl -u api-server.service -p err..warning

Resource and Process Monitoring Tools

To monitor resource consumption mapped to unit slice hierarchies:

# Interactive top-like view of CPU, Memory, and IO per slice/unit
systemd-cgtop

# Display running processes in hierarchical cgroup tree view
systemd-cgls

10. Troubleshooting Guide and Common Failure Scenarios

Triage Philosophy

Follow this step-by-step diagnostic workflow when investigating failures:

                       +-------------------------------+
                       |  0. High-Level System Triage  |
                       |    systemctl --failed         |
                       +-------------------------------+
                                       |
                                       v
                       +-------------------------------+
                       |  1. Inspect Unit Status & Logs|
                       |    systemctl status <unit>    |
                       |    journalctl -u <unit> -n 50 |
                       +-------------------------------+
                                       |
                                       v
                       +-------------------------------+
                       |  2. Inspect Config & Context  |
                       |    systemctl cat <unit>       |
                       |    systemctl list-dependencies|
                       +-------------------------------+
                                       |
                                       v
                       +-------------------------------+
                       |  3. Environment Validation    |
                       |    Test ExecStart as user     |
                       |    Check sandbox flags        |
                       +-------------------------------+
                                       |
                                       v
                       +-------------------------------+
                       |  4. Apply Fix & Clear State   |
                       |    systemctl daemon-reload    |
                       |    systemctl reset-failed     |
                       |    systemctl restart <unit>   |
                       +-------------------------------+

Scenario 1: Unknown System Failure — Global Triage

Symptoms

System alerts indicate a background problem, but the affected component is unknown.

Possible Causes

  • Service crash, broken mount point, or failed scheduled timer task.

Investigation and Diagnosis

  1. Run a system-wide query to identify failed units:
systemctl --failed
  1. Filter explicitly by unit state:
systemctl list-units --state=failed

Root Cause Identification

Review the output to pinpoint the exact failing unit name.

Resolution

Address the underlying issue on the specific unit, then clear the failure state:

sudo systemctl reset-failed broken-service.service

Verification

systemctl is-failed broken-service.service
# Expected output: inactive

Prevention

Implement automated alert monitoring tracking systemctl --failed via Node Exporter or custom scripts.


Scenario 2: Service Runs But Immediately Exits (inactive (dead))

Symptoms

Running systemctl start myservice reports success, but systemctl status myservice immediately shows inactive (dead) or failed.

Possible Causes

  1. Type Mismatch: Type=simple configured for a legacy application that automatically forks into the background.
  2. Oneshot Execution: Type=oneshot configured without RemainAfterExit=yes.
  3. Application Wrapper Exit: Wrapper script spawned a process in background and exited immediately.

Investigation and Diagnosis

  1. Inspect the service status and exit codes:
systemctl status myservice.service
  1. Inspect the effective unit file:
systemctl cat myservice.service

Root Cause Identification

  • If the binary forks into background while Type=simple, systemd treats the parent exit as service termination.
  • If it is a setup script with Type=oneshot, systemd marks it inactive after execution finishes unless RemainAfterExit=yes is set.

Resolution

Modify Type= in a drop-in override:

[Service]
# For background-forking daemons:
Type=forking

# OR for setup scripts:
Type=oneshot
RemainAfterExit=yes

Verification

sudo systemctl daemon-reload
sudo systemctl restart myservice.service
systemctl status myservice.service

Prevention

Match unit Type= definitions to the process foreground/backgrounding behavior.


Scenario 3: Service Restart Loop (start-limit-hit)

Symptoms

Service fails repeatedly in a quick loop, then logs: Failed to start My Service: Unit is entered failed state, followed by start-limit-hit.

Possible Causes

The service crashed repeatedly within a short window, exceeding systemd’s safety rate limit (StartLimitBurst).

Investigation and Diagnosis

  1. Inspect full log trace:
journalctl -u myservice.service -b --no-pager -n 100
  1. Check unit restart limit configuration:
systemctl show myservice.service -p StartLimitBurst,StartLimitIntervalSec,RestartSec

Root Cause Identification

Identify the underlying error in the logs causing the crash (e.g., missing database connection, bad configuration file syntax, or invalid permissions).

Resolution

  1. Fix the underlying application error identified in logs.
  2. Reset the systemd rate-limit counter:
sudo systemctl reset-failed myservice.service
  1. (Optional) Tune restart parameters in a drop-in file:
[Service]
RestartSec=10s

Verification

sudo systemctl restart myservice.service
systemctl status myservice.service

Prevention

Configure adequate RestartSec= delays so services do not trigger rate limits during transient network failures.


Scenario 4: Permission Denied / Sandboxing Directives Block Service Startup

Symptoms

The application starts fine when run manually from shell (sudo -u appuser /opt/myapp/bin/server), but fails under systemctl start with permission errors or missing file errors.

Possible Causes

Systemd security sandboxing directives (ProtectSystem=, PrivateTmp=, ReadOnlyPaths=, NoNewPrivileges=) are restricting filesystem access.

Investigation and Diagnosis

  1. Search unit file for active sandboxing directives:
systemctl cat myservice.service | grep -E 'Protect|Private|ReadOnly|ReadWrite|NoNew'
  1. Evaluate systemd security score:
systemd-analyze security myservice.service
  1. Compare PrivateTmp= behavior: If PrivateTmp=yes, files created in /tmp are placed inside a private isolated namespace directory (/tmp/systemd-private-*-myservice.service-*/tmp), invisible to other processes.

Root Cause Identification

ProtectSystem=strict or ProtectSystem=full mounts path locations (like /etc, /var, or app root) as read-only, causing write operations to fail.

Resolution

Grant explicit write permissions for required paths via drop-in file:

[Service]
ReadWritePaths=/var/log/myapp /opt/myapp/data

Verification

sudo systemctl daemon-reload
sudo systemctl restart myservice.service
systemctl status myservice.service

Prevention

Audit sandboxing options before updating units, testing new directives in non-production environments.


Scenario 5: Service Fails After Unit File Editing (“Unit changes ignored”)

Symptoms

Configuration edits made to /etc/systemd/system/myservice.service do not take effect when restarting the service.

Possible Causes

Systemd has not reloaded its in-memory unit file configuration graph from disk.

Investigation and Diagnosis

  1. Inspect service status for reload warnings:
systemctl status myservice.service

Warning message: Warning: The unit file, source configuration file or drop-ins of myservice.service changed on disk. Run 'systemctl daemon-reload' to reload units.

Root Cause Identification

Systemd caches unit configurations in memory. Edits on disk are ignored until explicitly reloaded.

Resolution

Execute daemon reload:

sudo systemctl daemon-reload
sudo systemctl restart myservice.service

Verification

Verify effective active configuration parameters:

systemctl cat myservice.service

Prevention

Always run systemctl daemon-reload after editing unit files.


11. Security and Best Practices

Sandboxing Directives Reference

Systemd provides built-in process isolation directives that restrict filesystem access, capability escalations, and system modification options.

[Service]
# Execute as dedicated unprivileged user
User=appuser
Group=appuser

# Prevents process and children from escalating privileges via setuid/setgid bits
NoNewPrivileges=yes

# Mounts /usr, /boot, and /etc read-only for the service
ProtectSystem=strict

# Makes /home, /root, and /run/user invisible (empty) to the service
ProtectHome=yes

# Provides isolated /tmp and /var/tmp directories unique to this unit
PrivateTmp=yes

# Restricts write access strictly to listed directory paths
ReadWritePaths=/var/log/myapp /var/lib/myapp

# Restricts kernel module loading and kernel tunable changes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes

Evaluating Service Exposure

Run systemd-analyze security to analyze security settings across active services:

systemd-analyze security nginx.service

Example Output:

NAME          OVERALL EXPOSURE / DETAILS
nginx.service 2.1 OK 😃

Lower exposure scores indicate stronger sandboxing protections.


12. Performance and Resource Optimisation

Managing Resource Controls with cgroups v2

Systemd translates unit file directives directly into kernel cgroups v2 configuration values:

[Service]
# Restrict CPU utilization (200% = 2 full CPU cores)
CPUQuota=200%

# Soft memory limit; triggers memory reclaim pressure when exceeded
MemoryHigh=1.5G

# Hard memory limit; triggers OOM killer when reached
MemoryMax=2G

# Maximum process and thread count to prevent fork bombs
TasksMax=512

# Relative I/O weight balance (1-10000)
IOWeight=100

Boot Performance Optimization

To analyze boot time performance and identify slow services:

# Print total boot duration (kernel vs user space)
systemd-analyze

# List services sorted by initialization time
systemd-analyze blame

# Print critical path chain of blocking service dependencies
systemd-analyze critical-chain

13. Backup, Recovery and Maintenance

Backing Up Configuration

Backup local systemd custom configurations and drop-ins regularly:

sudo tar -czvf systemd-custom-backup.tar.gz /etc/systemd/system/

Journal Log Storage Maintenance

To manage journal disk usage and prune log files:

# View current disk space consumed by journal files
journalctl --disk-usage

# Prune log data down to a maximum size
sudo journalctl --vacuum-size=500M

# Prune log data older than 14 days
sudo journalctl --vacuum-time=14d

Disaster Recovery: Booting into Maintenance Targets

If a misconfiguration prevents boot completion, switch to maintenance targets from the kernel command line in GRUB by appending systemd.unit=:

  • Rescue Target: systemd.unit=rescue.target (Mounts local filesystems and prompts for root password).
  • Emergency Target: systemd.unit=emergency.target (Mounts root read-only with minimal services).

14. Automation and Integration

Ansible Automation Example

Deploying systemd service overrides using Ansible:

- name: Deploy Custom Systemd Drop-in Override
  ansible.builtin.file:
    path: /etc/systemd/system/api-server.service.d
    state: directory
    owner: root
    group: root
    mode: '0755'

- name: Write Override Configuration File
  ansible.builtin.copy:
    dest: /etc/systemd/system/api-server.service.d/override.conf
    content: |
      [Service]
      RestartSec=10s
      MemoryMax=2G
    owner: root
    group: root
    mode: '0644'
  notify:
    - Reload Systemd Daemon
    - Restart API Service

handlers:
  - name: Reload Systemd Daemon
    ansible.builtin.systemd:
      daemon_reload: true

  - name: Restart API Service
    ansible.builtin.systemd:
      name: api-server.service
      state: restarted

15. Practical Labs and Exercises

Lab 1 (Beginner): Deploy a Custom Python Web Daemon

  • Objective: Create, register, enable, and start a web service.
  • Prerequisites: Linux server with root or sudo access.
  • Instructions:
  1. Write a script /usr/local/bin/dummy-web.py:
#!/usr/bin/env python3
import http.server, socketserver
Handler = http.server.SimpleHTTPRequestHandler
with socketserver.TCPServer(("", 8050), Handler) as httpd:
    httpd.serve_forever()
  1. Make executable: sudo chmod +x /usr/local/bin/dummy-web.py
  2. Create /etc/systemd/system/dummy-web.service with Type=simple, ExecStart=/usr/local/bin/dummy-web.py, and WantedBy=multi-user.target.
  3. Reload daemon, enable --now, and test endpoint curl http://localhost:8050.
  • Validation: Confirm status reports active (running).

Lab 2 (Intermediate): Build a Scheduled Maintenance Timer

  • Objective: Replace a cron job with a native systemd timer.
  • Instructions:
  1. Create /etc/systemd/system/cleanup.service (Type=oneshot, ExecStart=/usr/bin/find /tmp -type f -mtime +7 -delete).
  2. Create /etc/systemd/system/cleanup.timer with OnCalendar=daily and Persistent=true.
  3. Enable and start cleanup.timer.
  • Validation: Run systemctl list-timers and verify next execution schedule.

Lab 3 (Troubleshooting): Fix a Crashing Sandboxed Service

  • Objective: Diagnose and fix a service failing due to sandboxing and type misconfiguration.
  • Instructions:
  1. Deploy a broken unit /etc/systemd/system/broken.service:
[Unit]
Description=Broken Lab Unit

[Service]
Type=forking
ExecStart=/usr/bin/python3 -m http.server 9090
ProtectSystem=strict
ReadWritePaths=/tmp
  1. Run sudo systemctl daemon-reload && sudo systemctl start broken.service.
  2. Note startup failure. Diagnose using systemctl status broken and journalctl -u broken.
  3. Identify root cause: Python module runs in foreground, but Type=forking expects backgrounding.
  4. Fix unit file by changing Type=simple.
  5. Reload daemon and restart service.
  • Validation: Service reaches active (running) state.

16. Quick Reference and Cheat Sheet

Important Configuration Directories

Path Purpose
/etc/systemd/system/ Administrator unit files and local custom overrides.
/etc/systemd/system/<unit>.d/ Drop-in override directory for modifying specific service options.
/usr/lib/systemd/system/ Package-installed distribution default units (do not edit directly).
~/.config/systemd/user/ User-space unprivileged unit file definitions.

Log Locations

Log Path / Utility Purpose
journalctl -u <unit> Queries systemd binary logs for a targeted unit.
/var/log/journal/ Persistent system journal storage directory.

Common Administrative Commands

Task Command
Start Service Now systemctl start <unit>
Enable Service at Boot systemctl enable <unit>
Start & Enable Combined systemctl enable --now <unit>
Reload Systemd Engine systemctl daemon-reload
View System Failures systemctl --failed
Follow Live Logs journalctl -u <unit> -f

Troubleshooting Quick Reference

Symptom Initial Diagnostic Command Primary Action
Service failed systemctl status <unit> Inspect exit code and last 10 log lines.
Immediate exit (inactive) systemctl cat <unit> Verify Type= matches process foreground/fork behavior.
Sandboxing error `systemctl cat grep Protect`
Changes ignored systemctl status <unit> Run systemctl daemon-reload.

Important System Targets

Target Unit SysVinit Runlevel Equivalent Purpose
poweroff.target 0 Shuts down and powers off system.
rescue.target 1 Single-user rescue maintenance shell.
multi-user.target 3 Multi-user non-graphical command line environment.
graphical.target 5 Multi-user graphical desktop environment.
reboot.target 6 Restarts system.

17. Further Reading and References

  1. systemd.service(5) Manual Page — Service unit configuration format and directives: man7.org/linux/man-pages/man5/systemd.service.5.html
  2. systemctl(1) Manual Page — Controlling systemd service manager: man7.org/linux/man-pages/man1/systemctl.1.html
  3. journalctl(1) Manual Page — Querying systemd journal logs: man7.org/linux/man-pages/man1/journalctl.1.html
  4. systemd.timer(5) Manual Page — Systemd timer units and calendar rules: man7.org/linux/man-pages/man5/systemd.timer.5.html
  5. Freedesktop.org Official systemd Documentation — Core architecture specs and API documentation: freedesktop.org/wiki/Software/systemd