Domain 10Linux & SREINTERMEDIATE

(Darkar) Fixing an Interrupted Package Upgrade: DPKG Lifecycle and Systemd Service Recovery

A step-by-step technical deep dive into diagnosing and recovering from an interrupted Debian package upgrade (install ok unpacked state) and restoring systemd service health.

Domain: Domain 10: Linux Internals, SRE & Troubleshooting•Updated: 2026-11-09
#Linux#DPKG#APT#Systemd#Package Management#Troubleshooting#SRE

1. Introduction

In Linux system administration and Site Reliability Engineering (SRE), software deployment relies heavily on low-level package managers such as dpkg (on Debian/Ubuntu systems) or rpm (on Red Hat systems). Package management operations are designed to be transactional and reliable; however, unexpected disruptions—such as an interrupted SSH session, a power loss, an Out-Of-Memory (OOM) process termination, or a forced reboot during an upgrade—can leave software packages in an incomplete, unconfigured state.

When a package installation or upgrade is interrupted halfway through, binaries may exist on disk while system services remain non-functional, disabled, or unmonitored. This creates a challenging operational state where the system appears to have installed the software, but systemd unit files have not been properly hooked into target targets or started.

This article walks through a practical troubleshooting challenge (SadServers scenario: “Dakar”). We will examine how Debian package states function under the hood, how systemd services interact with maintainer scripts (postinst), how to diagnose an install ok unpacked status safely, and how to recover package state without risking system corruption or creating unmanaged service overrides.


2. Problem Description and Environment

Scenario Overview

  • Scenario Name: Dakar: The half-installed package
  • Difficulty: Medium
  • Category: Package Management & Service Fix

Problem Statement

The local health agent (checkout-agent) for a checkout platform stopped working after an automated or manual package upgrade was interrupted mid-execution. Although the binary executable was extracted to disk, its accompanying systemd service unit was left inactive and non-functional.

Operational Constraints & Success Criteria

  1. Restore Package & Service State: The checkout-agent package must be restored to a fully installed state (install ok installed), and its systemd service (checkout-agent.service) must be currently active (running) and enabled across system reboots.
  2. No Database Destruction: The /var/lib/dpkg package status database must not be wiped or manually altered.
  3. No Unmanaged Overrides: The issue must not be bypassed by creating a hand-written systemd service unit override file (e.g., in /etc/systemd/system/). The systemd service unit under /lib/systemd/system/ must remain owned by the original Debian package.
  4. Validation Test: Verification is performed via the scenario evaluation script /home/admin/agent/check.sh.

Environment Details

  • Operating System: Linux (Debian/Ubuntu-based distribution)
  • Target Package: checkout-agent
  • Target Service: checkout-agent.service
  • Evaluation Script: /home/admin/agent/check.sh

3. Initial Investigation and Hypotheses

Before modifying system state or executing corrective actions, a systematic SRE investigation starts by establishing what is known, what is unknown, and what hypotheses account for the failure.

Initial Knowledge State

  • Known: The binary exists on disk, but the agent service is down. A package upgrade operation was interrupted.
  • Unknown: Exact internal package status in /var/lib/dpkg/status; exact reason why the service is stopped (failed execution vs. disabled unit); whether dependencies were left broken.

Formulated Hypotheses

Hypothesis Supporting Evidence Evidence Against Diagnostic Step
H1: Unpacked but Unconfigured Package Problem description mentions an interrupted package upgrade. None at initial state. Query dpkg -s checkout-agent to check status.
H2: Broken or Crashing Service Binary Service is not running. Binary was deployed via package upgrade. Inspect systemctl status checkout-agent and journal logs.
H3: Unmet Package Dependencies Package upgrade failed mid-transit. No dependency errors reported yet. Execute dpkg --audit or apt-get check.
H4: Missing or Overwritten Unit File Service stopped working after upgrade. Package unpacks standard files to filesystem. Check systemd unit file presence under /lib/systemd/system/.

4. Troubleshooting Process

Step 1: Inspect Package Status

Objective: Determine the exact status of the checkout-agent package within the low-level dpkg database.

Command Executed:

dpkg -s checkout-agent

Command Explanation:

  • dpkg: The primary low-level package manager for Debian-based systems.
  • -s (--status): Queries /var/lib/dpkg/status for record headers matching the requested package name.

Output Received:

Package: checkout-agent
Status: install ok unpacked
Priority: optional
Section: admin
Maintainer: Sofia Operations <ops@example.invalid>
Architecture: all
Version: 1.0
Description: Checkout API health agent
 A small local health agent used by the checkout platform.

Analysis & Decision: The header Status: install ok unpacked provides critical diagnostic evidence. In dpkg terminology:

  • install: Desired action is installation.
  • ok: Package manager error status is clear (no error flag on the database entry itself).
  • unpacked: Files from the .deb archive were extracted into the root filesystem, but the maintainer script (postinst) never completed.

This confirms Hypothesis H1: The package update was cut off after unpacking files, leaving the configuration phase unexecuted.


Step 2: Audit Pending DPKG Operations

Objective: Scan the local system package database for unconfigured, half-installed, or trigger-pending packages.

Command Executed:

dpkg --audit

Output Received:

dpkg: error: unable to check lock file for dpkg database directory /var/lib/dpkg: Permission denied

Analysis & Decision: The command failed with a Permission denied error because inspecting lock files under /var/lib/dpkg/ requires superuser (root) privileges. This reinforces an operational rule: while package status queries (dpkg -s) can run as an unprivileged user, database verification and execution commands require sudo privileges.


Step 3: Evaluate Systemd Service Operational State

Objective: Check if systemd recognizes the service unit and determine its current state (active, inactive, enabled, or disabled).

Command Executed:

systemctl status checkout-agent

Command Explanation:

  • systemctl: The control utility for managing the systemd system and service manager.
  • status: Displays current runtime details, execution status, enablement status, and recent log snippets for a specified unit.

Output Received:

○ checkout-agent.service - Checkout API health agent
     Loaded: loaded (/usr/lib/systemd/system/checkout-agent.service; disabled; preset: enabled)
     Active: inactive (dead)

Analysis & Decision:

  • Loaded: The unit file exists at /usr/lib/systemd/system/checkout-agent.service (symlinked from /lib/systemd/system/).
  • Enablement: disabled (Preset state is enabled).
  • Active: inactive (dead).

This output confirms that the unit file was unpacked to disk, but because postinst was interrupted, systemd was never instructed to enable or start the unit.


Step 4: Attempt Package Configuration Without Elevation (Negative Test)

Objective: Attempt package configuration to confirm required privileges.

Command Executed:

dpkg --configure checkout-agent

Output Received:

dpkg: error: requested operation requires superuser privilege

Analysis & Decision: Modifying the /var/lib/dpkg/ database or running package maintainer scripts requires root privileges. The step must be run using sudo.


Step 5: Execute Package Configuration with Root Elevation

Objective: Complete the configuration phase for checkout-agent by executing its maintainer postinst script.

Command Executed:

sudo dpkg --configure checkout-agent

Command Explanation:

  • sudo: Elevates execution privileges to superuser (root).
  • dpkg --configure <package>: Unpacks no new files; instead, it executes /var/lib/dpkg/info/<package>.postinst configure to finalize package setup.

Output Received:

Setting up checkout-agent (1.0) ...
Created symlink '/etc/systemd/system/multi-user.target.wants/checkout-agent.service' → '/usr/lib/systemd/system/checkout-agent.service'.

Analysis & Decision: The output shows that running --configure executed the maintainer script, which invoked systemd helpers (such as deb-systemd-helper / systemctl). This automatically created the startup symlink under /etc/systemd/system/multi-user.target.wants/, enabling the service to auto-start on boot.


Step 6: Verify Package and Service State Post-Remediation

Objective: Confirm that the package status transitioned to installed and that the systemd service is actively running.

Command 1 Executed:

dpkg -s checkout-agent

Output 1 Received:

Package: checkout-agent
Status: install ok installed
Priority: optional
Section: admin
Maintainer: Sofia Operations <ops@example.invalid>
Architecture: all
Version: 1.0
Description: Checkout API health agent
 A small local health agent used by the checkout platform.

Command 2 Executed:

systemctl status checkout-agent

Output 2 Received:

● checkout-agent.service - Checkout API health agent
     Loaded: loaded (/usr/lib/systemd/system/checkout-agent.service; enabled; preset: enabled)
     Active: active (running) since Mon 2026-11-09 04:41:20 UTC; 21s ago
   Invocation: 5f46788049134eb59b7aab2c90f98902
   Main PID: 1018 (checkout-agent)
      Tasks: 2 (limit: 501)
     Memory: 464K (peak: 1.7M)
        CPU: 10ms
     CGroup: /system.slice/checkout-agent.service
             ├─1018 /bin/sh /usr/local/bin/checkout-agent
             └─1019 sleep 3600

Nov 09 04:41:20 i-0b77c9594bbd685cd systemd[1]: Started checkout-agent.service - Checkout API health agent.

Command 3 Executed (Redundant Safeguard):

sudo systemctl enable --now checkout-agent

Analysis & Decision:

  • Package Status: Successfully transitioned from install ok unpacked to install ok installed.
  • Service Status: Active (running), enabled (multi-user.target.wants symlink created), and executing under PID 1018.

Step 7: Inspect Package File Ownership and Usrmerge Nuance

Objective: Verify that the systemd unit file is formally tracked by dpkg.

Command Executed:

dpkg -S /usr/lib/systemd/system/checkout-agent.service

Output Received:

dpkg-query: no path found matching pattern /usr/lib/systemd/system/checkout-agent.service

Technical Analysis: Why did dpkg -S report no path found when querying /usr/lib/systemd/system/? On modern Linux distributions implementing usrmerge, root directories /lib, /bin, and /sbin are symbolic links pointing to their /usr counterparts (e.g., /lib $\rightarrow$ /usr/lib).

However, dpkg maintains literal path indexing inside /var/lib/dpkg/info/<package>.list. If the original package archive specified its target installation path as /lib/systemd/system/checkout-agent.service, dpkg-query looks for that literal path string in its index.

Querying with the literal legacy path confirms ownership:

dpkg -S /lib/systemd/system/checkout-agent.service
# Result: checkout-agent: /lib/systemd/system/checkout-agent.service

Step 8: Final Scenario Validation Test

Objective: Execute the scenario evaluation script to confirm all test criteria pass.

Command Executed:

/home/admin/agent/check.sh

Output Received:

OK

Verification succeeded.


5. Root Cause Analysis

+-------------------------------------------------------------------------+
|                       DEBIAN PACKAGE LIFECYCLE                          |
+-------------------------------------------------------------------------+
|                                                                         |
|   1. Extract Archive (.deb)                                             |
|      Files written to disk (/usr/bin, /lib/systemd/system/...)          |
|      DPKG Status updated -> "install ok unpacked"                       |
|                                                                         |
|                            [ INTERRUPTED ]                              |
|         (Power loss, OOM kill, SIGINT, dropped connection)              |
|                                     |                                   |
|                                     v                                   |
|   ===================================================================   |
|   CURRENT FAULT STATE:                                                  |
|   - Files exist on filesystem                                           |
|   - /var/lib/dpkg/status lists package as "unpacked"                    |
|   - postinst script was NEVER executed                                  |
|   - systemd service unit remains disabled and inactive                  |
|   ===================================================================   |
|                                     |                                   |
|                                     | (Remediation: dpkg --configure)   |
|                                     v                                   |
|   2. Execute Maintainer Script                                          |
|      Runs /var/lib/dpkg/info/checkout-agent.postinst configure        |
|      - Registers systemd unit (deb-systemd-helper)                      |
|      - Creates symlink under /etc/systemd/system/multi-user.target.wants|
|      - Reloads systemd daemon & starts service                          |
|                                                                         |
|   3. Completion                                                         |
|      DPKG Status updated -> "install ok installed"                      |
|      Service State -> Active (running) & Enabled                        |
+-------------------------------------------------------------------------+

Direct Root Cause

The root cause of the failure was an interrupted package installation lifecycle. The dpkg operation extracted the archive’s binaries and unit files into the filesystem but terminated before executing the package’s postinst (post-installation) maintainer script.

Conceptual Mechanics

Package installation on Debian-based systems follows a strict two-phase state machine:

  1. Unpacking Phase: dpkg unpacks binaries, configuration files, and systemd units from the .deb archive to disk. Upon completing file extraction, dpkg writes Status: install ok unpacked into /var/lib/dpkg/status.
  2. Configuration Phase: dpkg invokes the package’s maintainer shell script located at /var/lib/dpkg/info/<package>.postinst configure. This script is responsible for registering system users, running systemctl daemon-reload, enabling unit files via deb-systemd-helper, and starting background daemons.

Because the original transaction was interrupted between Step 1 and Step 2, the system was left with valid binaries on disk but no active systemd enablement symlinks or running processes.


6. Solution and Remediation

Standard Fix Applied

To resolve the issue cleanly without corrupting the dpkg database or introducing non-standard unit files, execute package configuration directly:

sudo dpkg --configure checkout-agent

Why This Solution is Optimal

  • Preserves Package Integrity: Uses standard Debian package lifecycle management without bypassing dpkg tracking.
  • Executes Vendor Setup: Triggers the native postinst script created by the package maintainers.
  • Ensures Boot Persistence: Creates the required systemd target symlinks (/etc/systemd/system/multi-user.target.wants/) naturally, guaranteeing the service starts upon reboot.
  • Zero Technical Debt: Avoids hand-written override files under /etc/systemd/system/ that could cause configuration drift during future upgrades.

7. Verification and Validation

Proper SRE methodology requires verifying not just that a remediation command succeeded, but that the system as a whole is healthy and compliant.

Verification Checklist

  1. Verify DPKG Status:
dpkg -s checkout-agent | grep "Status:"

Expected Result: Status: install ok installed. 2. Verify Systemd Service Status and Enablement:

systemctl is-enabled checkout-agent
systemctl is-active checkout-agent

Expected Output: enabled and active. 3. Verify File Ownership:

dpkg -S /lib/systemd/system/checkout-agent.service

Expected Output: Owned by checkout-agent. 4. Run Scenario Audit Script:

/home/admin/agent/check.sh

Expected Output: OK.


8. Alternative Troubleshooting Approaches

Depending on the operational context, several alternative methods exist for investigating and repairing broken package states.

Comparison of Recovery Approaches

Approach Command Mechanism Pros Cons Best Used When
A. System-wide DPKG Recovery sudo dpkg --configure -a Configures all currently unpacked/unconfigured packages system-wide. Resolves all pending package breaks in one command. May configure packages you were not intending to touch immediately. Batch recovery after an unexpected reboot mid-upgrade.
B. APT Dependency Fix sudo apt-get install -f Analyzes dependency graphs, fixes missing dependencies, and runs dpkg --configure -a. Resolves broken dependencies automatically alongside configuration. Requires access to APT repositories if packages are missing. An upgrade broke due to missing uninstalled dependencies.
C. Maintainer Script Inspection cat /var/lib/dpkg/info/checkout-agent.postinst Directly reads the shell script that runs during configuration. Provides complete visibility into exact setup steps and commands. Read-only inspection; does not resolve the state on its own. dpkg --configure fails with a script error and requires debugging.
D. Reinstallation via DPKG sudo dpkg -i package.deb Re-extracts archive and re-runs configuration from scratch. Overwrites corrupted binaries or files on disk. Requires having the original .deb file available on disk. Files on disk were corrupted during the power loss/interrupt.

9. Linux Command Deep Dive

Command 1: dpkg (Debian Package Manager)

dpkg is the foundational package management software on Debian and derivative distributions (Ubuntu, Linux Mint, Pop!_OS). Unlike apt or apt-get, dpkg does not automatically download files from remote repositories or resolve complex dependency trees; it operates strictly on local .deb files and the local database under /var/lib/dpkg/.

Syntax

dpkg [options] action [package|file]

Frequently Used Troubleshooting Options

Option / Flag Purpose Example Usage
-s, --status Displays status details for an installed or unpacked package. dpkg -s checkout-agent
-l, --list Lists matching packages with 2-character status codes. dpkg -l 'checkout*'
--configure Configures an unpacked package by running its postinst script. sudo dpkg --configure checkout-agent
--configure -a Configures all unpacked and unconfigured packages. sudo dpkg --configure -a
--audit, -C Searches for packages installed incomplete on the system. sudo dpkg --audit
-L, --listfiles Lists all files installed on the system by a package. dpkg -L checkout-agent
-S, --search Searches for which package owns a specific file path on disk. dpkg -S /usr/bin/python3

Decoding dpkg -l Status Codes

When running dpkg -l, the output presents a two-letter status code prefix for every package:

  • First character (Desired State):

  • i = Install

  • r = Remove

  • p = Purge

  • Second character (Current Status):

  • n = Not installed

  • i = Installed (install ok installed)

  • U = Unpacked (install ok unpacked)

  • F = Half-configured (install ok half-configured)

  • H = Half-installed

  • W = Triggers-awaited

  • t = Triggers-pending


Command 2: systemctl (Systemd System Control)

systemctl is the central management tool for inspecting and controlling the state of the systemd init system and service manager.

Syntax

systemctl [command] [unit]

Useful Options for Service Diagnostics

Command / Flag Purpose Example Usage
status Shows status, PID, memory, CGroup, and recent log tail. systemctl status checkout-agent
is-active Returns exit code 0 if service is currently running. systemctl is-active checkout-agent
is-enabled Checks if service is set to start on boot. systemctl is-enabled checkout-agent
enable --now Enables service for boot and starts it immediately. sudo systemctl enable --now checkout-agent
cat Displays the raw unit file contents and drop-in overrides. systemctl cat checkout-agent.service
daemon-reload Reloads systemd manager configuration (scans for new/changed units). sudo systemctl daemon-reload

10. Underlying Linux Concepts and Architecture

1. DPKG Database Architecture (/var/lib/dpkg/)

The dpkg database does not rely on a complex relational database engine. Instead, it consists of plain-text status records and flat control files located in /var/lib/dpkg/:

  • /var/lib/dpkg/status: Main text file listing every package known to the system, its architecture, version, dependencies, and state (installed, unpacked, etc.).
  • /var/lib/dpkg/info/: Directory containing maintainer scripts and file index lists:
  • <package>.list: Text file listing every file path installed by the package.
  • <package>.postinst: Script executed after unpacking during configuration.
  • <package>.prerm: Script executed before package removal.
  • <package>.postrm: Script executed after package removal.

2. Merged /usr Filesystem (usrmerge)

In traditional Linux layouts, core binaries lived in /bin and /sbin, while system libraries lived in /lib. Modern Linux distributions merge these directories into /usr:

  • /bin $\rightarrow$ /usr/bin
  • /sbin $\rightarrow$ /usr/sbin
  • /lib $\rightarrow$ /usr/lib

Implication for System Administrators

System tools like systemctl resolve /lib/systemd/system/foo.service to /usr/lib/systemd/system/foo.service transparently. However, low-level database index tools like dpkg -S look up exact string matches against /var/lib/dpkg/info/*.list. If a package was built specifying /lib/..., searching for /usr/lib/... will fail unless searched using the precise path string used during packaging.


11. Real-World Sysadmin and SRE Applications

Production Risk Analysis

Interrupted package installations in production environments introduce significant risks:

  • Silent Failures: If a daemon package is updated via automated unattended upgrades, an interrupted postinst might leave the binary updated on disk while the running daemon process is dead or executing obsolete code.
  • Database Lockouts: A locked /var/lib/dpkg/lock-frontend prevents future automated security patches or CI/CD deployments from executing.

Observability & Alerting

In an enterprise SRE environment, broken package states should be monitored proactively:

  1. Host Auditing: Run scheduled node checks or configuration management queries for unconfigured packages:
dpkg-query -W -f='${Package} ${Status}\n' | grep -v 'ok installed'
  1. Systemd Unit Monitoring: Configure Prometheus node_exporter with the systemd collector enabled to alert whenever a system unit state is failed or unexpected inactive.

Incident Prevention & Automation Rules

  • Idempotent Playbooks: Ensure Ansible or Chef routines include tasks that execute dpkg --configure -a prior to running package upgrades.
  • Avoid Hand-written Overrides: Never manually edit unit files in /lib/systemd/system/ or write ad-hoc scripts to start packaged binaries. Always rely on dpkg configuration or /etc/systemd/system/ drop-in files (.conf) to preserve maintainability.

12. Further Learning and Experiments

Safe Hands-on Experiments

Experiment 1: Inspect Package Control Scripts

  1. Open terminal access on a Debian/Ubuntu test environment.
  2. Locate the post-installation script for any common package:
cat /var/lib/dpkg/info/cron.postinst
  1. Observe how maintainer scripts invoke deb-systemd-helper or systemctl to handle service enablement.

Experiment 2: Query DPKG Status Programmatically

Execute dpkg-query with custom formatted fields to list all packages currently in an incomplete state:

dpkg-query -f '${binary:Package}\t${db:Status-Abbrev}\t${Status}\n' -W | grep -v 'ii'

13. Key Takeaways

  1. Understand Package States: install ok unpacked means files are on disk, but package configuration scripts (postinst) have not completed.
  2. Standard Remediation: Recover interrupted Debian package upgrades safely using sudo dpkg --configure <package> or sudo dpkg --configure -a.
  3. Respect Packaging Ownership: Do not manually invent custom unit overrides or wipe /var/lib/dpkg when a package configuration is simply pending.
  4. Be Aware of Usrmerge: dpkg -S searches for literal path strings as listed in .list files. If searching /usr/lib/... fails on a merged system, search /lib/....
  5. Always Verify Persistence: Confirm that services are not only active (running) right now, but also enabled under systemd to ensure survival across reboots.

14. Quick Troubleshooting Reference / Runbook

                  INTERRUPTED DPKG PACKAGE RECOVERY RUNBOOK
                  ========================================

1. DIAGNOSE PACKAGE STATE
   ----------------------
   $ dpkg -s <package-name>
   -> Look for: "Status: install ok unpacked"

2. CHECK FOR PENDING SYSTEM-WIDE UNCONFIGURED PACKAGES
   ---------------------------------------------------
   $ sudo dpkg --audit

3. DIAGNOSE SERVICE STATE
   ----------------------
   $ systemctl status <service-name>
   -> Look for: "Loaded: loaded", "Active: inactive (dead)", "disabled"

4. EXECUTE RECOVERY
   ----------------
   # Single package recovery:
   $ sudo dpkg --configure <package-name>

   # All pending packages recovery:
   $ sudo dpkg --configure -a

5. VERIFY SERVICE HEALTH AND ENABLEMENT
   ------------------------------------
   $ systemctl is-enabled <service-name>  # Should return: enabled
   $ systemctl is-active <service-name>   # Should return: active

6. VALIDATE SYSTEM END-TO-END
   --------------------------
   $ dpkg -s <package-name> | grep Status # Should return: install ok installed