1. Overview
Supervisord is a lightweight client/server process control system designed for UNIX-like operating systems. It allows system administrators and Site Reliability Engineers (SREs) to monitor, manage, and control a collection of long-running subprocesses from a single centralized interface. Unlike traditional operating system init systems that focus on boot-time service initialization across the entire machine, Supervisord is engineered to manage application-level process trees in user space.
What Problem Supervisord Solves
In complex application environments, applications often consist of multiple background workers, API servers, queue processors, and scheduled tasks. Operating systems historically required root privileges to write SysVinit scripts or systemd unit files for every background task. Supervisord solves this problem by providing:
- Centralized Subprocess Lifecycle Management: Automatically starting, stopping, restarting, and monitoring application processes without requiring root privileges for day-to-day operations.
- Automatic Process Recovery: Monitoring managed processes and automatically restarting them if they crash or exit unexpectedly.
- Streamlined Log Redirection: Capturing standard output (
stdout) and standard error (stderr) streams from subprocesses and writing them directly to configurable log files with built-in log rotation. - Unified Control Interfaces: Offering command-line (
supervisorctl), XML-RPC, and web-based interfaces for process control.
Common Use Cases
Supervisord is widely deployed across modern Linux infrastructures:
- Python-Centric Application Stacks: Managing web application servers like Gunicorn/Uvicorn alongside task queues like Celery workers and Beat schedulers.
- Containerized Environments (Docker/Podman): Serving as the primary process manager (
PID 1) inside containers that must execute multiple concurrent services (e.g., Nginx alongside a background worker). - Multi-Tenant / Legacy Server Stacks: Running background processes for specific non-root application users without granting full
sudoaccess to system-wide init daemons.
Advantages and Limitations
| Aspect | Advantages | Limitations |
|---|---|---|
| Deployment | Simple Python installation via pip or distribution package managers (apt, dnf). |
Dependent on a Python runtime environment. |
| Privileges | Can run entirely in unprivileged user space without root access. | Cannot manage OS kernel parameters or low-level boot hardware drivers. |
| Logging | Native stream interception (stdout/stderr) with automatic size-based log rotation. |
Does not natively integrate with journald binary logs without wrapper configuration. |
| Control | Provides standard CLI, web UI, and XML-RPC API. | Lacks native multi-node cluster orchestration or high-availability failover. |
Supervisord vs. Systemd
On modern Linux distributions, systemd is the default system init daemon responsible for OS boot initialization, hardware events, cgroup resource constraints, and service dependency ordering. Supervisord operates higher in the stack as an application supervisor.
+-------------------------------------------------------------------+
| Linux Kernel & Boot |
+-------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------+
| systemd (PID 1 System Init) |
+-------------------------------------------------------------------+
|
v
+-------------------------------------------------------------------+
| supervisord.service (Daemon) |
+-------------------------------------------------------------------+
| |
v v
+---------------------+ +---------------------+
| Gunicorn App Worker| | Celery Queue Worker |
+---------------------+ +---------------------+
While greenfield Linux deployments typically leverage native systemd unit files, Supervisord remains essential in Docker sidecar containers, legacy application stacks, and environments where developers require self-service process control without root privileges.
2. Architecture and Mental Model
Supervisord functions through a client/server model operating entirely in user space. Understanding its internal components, socket communications, and process state machine is essential for diagnosing production issues.
flowchart TD
subgraph Client Space
CLI["supervisorctl (CLI Tool)"]
WEB["Web Browser / HTTP Client"]
end
subgraph Supervisord Daemon [supervisord Process]
RPC["XML-RPC Server / HTTP Listener\n(unix:// or inet://)"]
ENGINE["Process Management Engine\n(State Machine & Signal Handler)"]
LOGOUT["Log Manager\n(stdout/stderr capture & rotation)"]
end
subgraph Managed Subprocesses
P1["Program A (e.g., Gunicorn)\n[PID: 1042]"]
P2["Program B (e.g., Celery Worker)\n[PID: 1043]"]
P3["Program C (e.g., Redis)\n[PID: 1044]"]
end
CLI -->|XML-RPC over UNIX Socket| RPC
WEB -->|HTTP over TCP Port 9001| RPC
RPC --> ENGINE
ENGINE -->|fork / exec / kill| P1
ENGINE -->|fork / exec / kill| P2
ENGINE -->|fork / exec / kill| P3
P1 -->|stdout / stderr pipes| LOGOUT
P2 -->|stdout / stderr pipes| LOGOUT
P3 -->|stdout / stderr pipes| LOGOUT
Main Components and Responsibilities
supervisord(Daemon Server):
- The primary daemon process that reads configuration files on startup, forks child subprocesses, captures file descriptors, monitors state changes, handles signals, and executes log rotation.
- It runs continuously in the background and operates as the parent process to all managed programs.
supervisorctl(CLI Control Client):
- A command-line client used by system administrators to query status, stop, start, restart, and reload managed programs.
- It does not modify sub-processes directly; instead, it sends XML-RPC requests to the
supervisordserver daemon.
- HTTP Server Interface:
supervisordembeds an HTTP server configured via[unix_http_server](UNIX domain socket) or[inet_http_server](TCP socket).- This HTTP server exposes an XML-RPC API used by
supervisorctlas well as an optional web control UI.
- XML-RPC Interface (
[rpcinterface:supervisor]):
- A standard protocol interface translating HTTP/socket payloads into internal control directives for process execution.
Subprocess Lifecycle and State Machine
Supervisord tracks every managed process through a formal state machine. Troubleshooting process failures requires understanding these states:
+-----------------------------------+
| STOPPED |
+-----------------------------------+
| (start request / autostart)
v
+-----------------------------------+
| STARTING |
+-----------------------------------+
/ \
(exits before startsecs) (runs > startsecs)
/ \
v v
+------------------+ +-------------------+
| BACKOFF | | RUNNING |
+------------------+ +-------------------+
/ \ / \
(retries < startretries) (retries >= max) (stop request) (exit code)
/ \ / \
v v v v
[STARTING] +-------+ +----------+ +--------+
| FATAL | | STOPPING | | EXITED |
+-------+ +----------+ +--------+
STOPPED(0): The process is stopped or was never started.STARTING(10): A start request was issued; the process is starting and executing its startup sequence.RUNNING(20): The process has started successfully and remained alive for at leaststartsecsseconds.BACKOFF(30): The process started but exited prematurely beforestartsecselapsed. Supervisord automatically enters a delay before attempting a restart.STOPPING(40): A stop request was issued; Supervisord has sent a termination signal (stopsignal, e.g.,SIGTERM) and is waitingstopwaitsecsfor the process to exit clean.EXITED(100): The process exited after reachingRUNNINGstate. Depending onautorestart, Supervisord will decide whether to restart it.FATAL(200): The process failed to start successfully after exhaustingstartretriesattempts from theBACKOFFstate. Manual intervention is required to clear this state.UNKNOWN(1000): The process is in an invalid state due to an internal daemon error.
3. Core Concepts and Terminology
-
Program (
[program:x]): A named block in the configuration defining a specific executable command to run, along with runtime settings like execution directory, environment variables, user permissions, and auto-restart rules. -
Group (
[group:x]): A logical collection of programs defined under[group:group_name]. Grouping allows operations on multiple processes simultaneously (e.g.,supervisorctl start workers:*). -
Process Count (
numprocs): A parameter within a program block allowing Supervisord to spawn multiple identical instances of a command (e.g., launching 4 parallel worker instances) using variable substitution likeprocess_name=%(program_name)s_%(process_num)02d. -
Foreground Requirement: Subprocesses managed by Supervisord must run in the foreground and must not daemonize/detach. If a managed command forks into the background, Supervisord loses track of its PID and falsely marks the process as
EXITED. -
Autostart (
autostart=true|false): Specifies whether Supervisord should automatically launch the process when the Supervisord daemon itself boots up. -
Autorestart (
autorestart=true|false|unexpected): Defines restart behavior on exit: -
true: Unconditionally restarts the process whenever it exits. -
false: Never restarts the process regardless of exit code. -
unexpected: Restarts the process only if it exits with an exit code not specified inexitcodes(defaults to code0). -
Startsecs: The minimum duration (in seconds) a process must stay running after spawn to be considered in the
RUNNINGstate. -
Startretries: The number of consecutive failed startup attempts Supervisord will make before declaring a process
FATAL.
4. Installation and Version Management
Prerequisites and Package Manager Installation
Supervisord is available in standard package repositories across major Linux distributions:
Debian / Ubuntu
sudo apt update
sudo apt install -y supervisor
Distribution Default Paths: Main config at /etc/supervisor/supervisord.conf, drop-ins in /etc/supervisor/conf.d/*.conf, logs in /var/log/supervisor/.
RHEL / Rocky Linux / AlmaLinux / CentOS
sudo dnf install -y epel-release
sudo dnf install -y supervisor
Distribution Default Paths: Main config at /etc/supervisord.conf, drop-ins in /etc/supervisord.d/*.ini, logs in /var/log/supervisor/.
Universal Installation via Python Pip
In isolated environments or container images lacking OS packages, Supervisord can be installed via PyPI:
pip install supervisor
Verifying Installation
To verify the installed version and check daemon availability:
# Check CLI binary version
supervisord -v
# Check client version
supervisorctl version
# Verify systemd service unit status (distro package installs)
systemctl status supervisor
5. Configuration Files and Directory Structure
Supervisord uses a standard Windows-INI syntax format (Python ConfigParser).
Path Reference Table
| Path / Location | Purpose | Distribution / Context |
|---|---|---|
/etc/supervisor/supervisord.conf |
Primary configuration file | Debian / Ubuntu default |
/etc/supervisord.conf |
Primary configuration file | RHEL / CentOS / Fedora default |
/etc/supervisor/conf.d/*.conf |
Per-application drop-in configuration files | Debian / Ubuntu pattern |
/etc/supervisord.d/*.ini |
Per-application drop-in configuration files | RHEL / CentOS pattern |
/var/log/supervisor/ |
Default directory for daemon and subprocess logs | Standard across Linux distributions |
/var/run/supervisor.sock or /tmp/supervisor.sock |
UNIX domain socket file for XML-RPC | Global runtime socket location |
/var/run/supervisord.pid |
PID tracking file for main daemon | Daemon runtime process lock |
Key Configuration Sections
1. [supervisord] (Global Daemon Configuration)
Defines global parameters for the master daemon.
[supervisord]
logfile=/var/log/supervisor/supervisord.log ; Main log file path
logfile_maxbytes=50MB ; Rotate log after reaching this size
logfile_backups=10 ; Number of rotated backups to keep
loglevel=info ; Log level (debug, info, warn, error, critical)
pidfile=/var/run/supervisord.pid ; PID file location
nodaemon=false ; Run in foreground if set to true (used in Docker)
minfds=10240 ; Minimum system file descriptors required at startup
minprocs=200 ; Minimum process slots required at startup
user=root ; Default user starting daemon
2. [unix_http_server] (Local Socket Server)
Enables an HTTP server over a UNIX domain socket for supervisorctl control.
[unix_http_server]
file=/var/run/supervisor.sock ; Socket file path
chmod=0700 ; Socket permission bits (restricted to owner)
chown=root:root ; Socket owner user and group
username=admin ; Optional socket HTTP basic auth username
password=SecretPassword123 ; Optional socket HTTP basic auth password
3. [inet_http_server] (Network Control Server & Web UI)
Optional section enabling TCP listening for remote Web UI and remote RPC access.
[inet_http_server]
port=127.0.0.1:9001 ; Network interface and port
username=admin ; HTTP basic auth username
password=SecretPassword123 ; HTTP basic auth password (cleartext or {SHA} hash)
4. [supervisorctl] (Client Configuration)
Tells supervisorctl how to connect to the daemon.
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; Point to UNIX socket
; serverurl=http://127.0.0.1:9001 ; Alternatively, point to TCP server
username=admin ; Matches server basic auth user
password=SecretPassword123 ; Matches server basic auth password
5. [program:x] (Subprocess Definition)
Defines an individual application process managed by Supervisord.
# /etc/supervisor/conf.d/myapp.conf
[program:myapp]
command=/opt/myapp/venv/bin/gunicorn -w 4 app:app ; Executable path & args (required)
directory=/opt/myapp ; Working directory before execution
user=deploy ; Run subprocess under unprivileged user
autostart=true ; Start automatically when daemon starts
autorestart=unexpected ; Restart only on unexpected exit codes
startsecs=5 ; Seconds program must run to be considered stable
startretries=3 ; Consecutive launch retry attempts before FATAL
stopsignal=QUIT ; Signal used to stop the process (default: TERM)
stopwaitsecs=10 ; Seconds to wait before SIGKILL if stopsignal fails
stdout_logfile=/var/log/supervisor/myapp.log ; Path for stdout log redirection
stdout_logfile_maxbytes=10MB ; Max log file size before rotation
stdout_logfile_backups=5 ; Rotated file retention count
stderr_logfile=/var/log/supervisor/myapp.err ; Path for stderr log redirection
stderr_logfile_maxbytes=10MB
stderr_logfile_backups=5
environment=PATH="/opt/myapp/venv/bin",DJANGO_SETTINGS_MODULE="app.settings" ; Custom env vars
6. [group:x] (Process Grouping)
Groups multiple program definitions into a unified control unit.
[group:workers]
programs=celery-worker,celery-beat ; Comma-separated program names
priority=999 ; Shutdown/startup relative priority
7. [include] (Configuration Inclusions)
Allows splitting process configurations into modular files.
[include]
files = /etc/supervisor/conf.d/*.conf ; Wildcard path to drop-in configs
6. Service Management and Common Operations
Managing the Main Supervisord Daemon
When Supervisord is managed by systemd, control the underlying daemon using standard systemctl commands:
# Start the supervisord service daemon
sudo systemctl start supervisor
# Stop the supervisord service daemon (stops all managed child processes)
sudo systemctl stop supervisor
# Restart the supervisord daemon
sudo systemctl restart supervisor
# Check status of the supervisord service daemon
sudo systemctl status supervisor
If starting the daemon manually without systemd:
# Start supervisord directly specifying a explicit configuration file
supervisord -c /etc/supervisor/supervisord.conf
Reloading Configuration Changes: reread vs. update vs. restart
When updating program configuration files, a common operational mistake is executing supervisorctl restart all or restarting the master daemon. This disrupts uninterrupted services.
Supervisord provides a two-phase reload mechanism to apply changes safely:
+-------------------+ supervisorctl reread +-------------------+
| Config Files On | -----------------------------> | Updated Configs |
| Disk Edited | | Read into Memory |
+-------------------+ +-------------------+
|
| supervisorctl update
v
+-------------------+
| Affected Programs |
| Restarted/Added |
+-------------------+
- **Phase 1:
supervisorctl reread**
- Scans configuration directories (
conf.d/*.conf) for edits, additions, or deletions. - Loads the updated configuration into memory without restarting or disrupting any running process.
- **Phase 2:
supervisorctl update**
- Applies the newly loaded configuration.
- Starts newly added programs, stops removed programs, and restarts only those existing programs whose configuration definitions were altered. Unmodified programs remain running without interruption.
7. Common Administrative Tasks and Workflows
Task 1: Adding a New Application Service
Objective: Deploy a new Python FastAPI application service named api-server under Supervisord control.
- Create a drop-in configuration file
/etc/supervisor/conf.d/api-server.conf:
[program:api-server]
command=/opt/api/venv/bin/uvicorn main:app --host 127.0.0.1 --port 8000
directory=/opt/api
user=www-data
autostart=true
autorestart=unexpected
stdout_logfile=/var/log/supervisor/api-server.log
stderr_logfile=/var/log/supervisor/api-server.err
environment=PATH="/opt/api/venv/bin",PRODUCTION="true"
- Verify directory and permission prerequisites:
sudo chown -R www-data:www-data /opt/api
sudo touch /var/log/supervisor/api-server.log /var/log/supervisor/api-server.err
sudo chown www-data:www-data /var/log/supervisor/api-server.*
- Load and start the new process without affecting other services:
sudo supervisorctl reread
sudo supervisorctl update
- Confirm operational status:
sudo supervisorctl status api-server
Task 2: Managing Process Groups
Objective: Control a multi-component stack (e.g., Celery background task workers) collectively.
- Configure
/etc/supervisor/conf.d/celery.conf:
[program:celery-worker]
command=/opt/app/venv/bin/celery -A tasks worker --loglevel=info
directory=/opt/app
user=deploy
[program:celery-beat]
command=/opt/app/venv/bin/celery -A tasks beat --loglevel=info
directory=/opt/app
user=deploy
[group:celery-stack]
programs=celery-worker,celery-beat
- Apply changes:
sudo supervisorctl reread && sudo supervisorctl update
- Manage the group as a single unit using wildcards (
:*):
# Restart all programs in the celery-stack group
sudo supervisorctl restart celery-stack:*
# Stop all programs in the celery-stack group
sudo supervisorctl stop celery-stack:*
8. CLI Command Reference and Cheatsheet
Process Status and Control Commands
| Command | Purpose | Flags / Arguments | Practical Example | Description & Notes |
|---|---|---|---|---|
supervisorctl status |
Display current state of managed processes | [name] (optional specific program) |
supervisorctl status |
Shows state, PID, and uptime for all processes. |
supervisorctl start |
Start stopped process(es) | <name> or <group>:* or all |
supervisorctl start myapp |
Moves process from STOPPED to STARTING. |
supervisorctl stop |
Gracefully stop running process(es) | <name> or <group>:* or all |
supervisorctl stop workers:* |
Sends stopsignal (SIGTERM) to process group. |
supervisorctl restart |
Stop and immediately start process(es) | <name> or <group>:* or all |
supervisorctl restart all |
Performs sequential stop and start. |
supervisorctl pid |
Retrieve the OS PID of a process | <name> or all |
supervisorctl pid myapp |
Returns the raw PID for monitoring scripts. |
Configuration and Daemon Control Commands
| Command | Purpose | Flags / Arguments | Practical Example | Description & Notes |
|---|---|---|---|---|
supervisorctl reread |
Reload config files from disk into memory | None | supervisorctl reread |
Reads changes without restarting running processes. |
supervisorctl update |
Apply memory configuration changes | None | supervisorctl update |
Restarts altered programs and launches new ones. |
supervisorctl clear |
Clear process log files | <name> or all |
supervisorctl clear myapp |
Empties standard log files for specified program. |
supervisorctl shutdown |
Stop supervisord daemon and all child processes |
None | supervisorctl shutdown |
Terminates daemon completely. Destructive. |
supervisord |
Start main daemon executable | -c <config_path>, -n (foreground) |
supervisord -c /etc/supervisor/supervisord.conf |
Launches server daemon manually. |
9. Logs, Monitoring and Diagnostics
Log Architecture
Supervisord maintains two distinct layers of log output:
- Daemon Activity Log (
/var/log/supervisor/supervisord.log): Records system daemon lifecycle events, configuration reloads, process state transitions, socket connection events, and internal warnings. - Subprocess Application Logs (
stdout_logfile&stderr_logfile): Stores stdout and stderr outputs generated by individual child processes.
+-------------------------------+
| supervisord Master Daemon |
+-------------------------------+
|
+----------------------+----------------------+
| |
v v
+---------------------------+ +---------------------------+
| Daemon Activity Log | | Child Process Log Pipes |
| (/var/log/supervisor/ | | (Configured per program) |
| supervisord.log) [1.2.2] | +---------------------------+
+---------------------------+ |
+------------------+------------------+
| |
v v
+---------------------------+ +---------------------------+
| stdout_logfile [1, 3] | | stderr_logfile [3] |
| (/var/log/supervisor/ | | (/var/log/supervisor/ |
| myapp.log) | | myapp.err) |
+---------------------------+ +---------------------------+
Reading and Tailing Logs with supervisorctl
Supervisord allows real-time log inspection without needing shell path references:
# View the last 100 lines of standard output for 'myapp'
supervisorctl tail -100 myapp
# View the last 100 lines of standard error output for 'myapp'
supervisorctl tail -100 myapp stderr
# Follow standard output in real time (press Ctrl+C to exit)
supervisorctl tail -f myapp
System Diagnostics and Resource Inspection
When investigating resource saturation caused by a managed process, correlate supervisorctl PID outputs with standard system diagnostic commands:
# 1. Obtain the OS Process ID of the target program
PID=$(supervisorctl pid myapp)
# 2. Inspect CPU, Memory, and thread execution using ps
ps aux -q $PID
# 3. Check open file descriptors and sockets associated with the process
lsof -p $PID
# 4. Monitor real-time I/O performance
pidstat -p $PID 1 5
10. Troubleshooting Guide and Common Failure Scenarios
Problem 1: Program Stuck in FATAL or BACKOFF State
Symptoms
supervisorctl status displays the application in a FATAL or BACKOFF state:
myapp FATAL Exited too quickly (process log may have details)
Possible Causes
- Binary or Script Error: Executable path is incorrect, missing, or missing execute (
+x) permissions. - Missing Dependencies / Environment: Required Python virtual environment, dynamic library, or environment variable is missing.
- Daemonization Failure: The target application forks into the background rather than running in the foreground.
- Immediate Exit: The process executes successfully and exits cleanly (exit code
0) beforestartsecselapses.
Investigation and Diagnosis
- Step 1: Inspect standard error log output
- Command:
supervisorctl tail myapp stderr - Expected Result: Tracebacks, missing file errors, or library import failures.
- Interpretation: If a “No such file or directory” or “ModuleNotFoundError” is visible, the runtime path or virtual environment path is invalid.
- Step 2: Manually execute command outside Supervisord
- Objective: Reproduce the execution context as the configured user.
- Command:
sudo -u deploy bash -c 'export PATH="/opt/myapp/venv/bin:$PATH"; /opt/myapp/venv/bin/gunicorn -w 4 app:app'
- Interpretation: If execution fails directly in the shell, solve the missing dependency or code defect first. If execution succeeds in shell but fails in Supervisord, inspect environment variable definitions.
Root Cause Identification
If the command succeeds in shell but fails under Supervisord, the cause is either missing shell profile environment variables or the application detaching into a background daemon.
Resolution
- For environment issues, explicitly set environment variables and absolute paths in
/etc/supervisor/conf.d/myapp.conf:
command=/opt/myapp/venv/bin/gunicorn --stdfd 1 app:app
environment=PATH="/opt/myapp/venv/bin",PYTHONPATH="/opt/myapp"
- If the application exits too quickly on clean completion, set
startsecs=0or adjustautorestart=unexpected.
Verification
sudo supervisorctl reread && sudo supervisorctl update
sudo supervisorctl start myapp
sudo supervisorctl status myapp
Confirm state changes to RUNNING and remains stable past startsecs duration.
Prevention
Always test commands with absolute paths using sudo -u <user> prior to writing Supervisord configuration files. Ensure target web frameworks (e.g., Gunicorn, Nginx, Redis) are configured with foreground execution flags.
Problem 2: unix:///var/run/supervisor.sock: no such file
Symptoms
Executing supervisorctl status yields the following error:
error: <class 'FileNotFoundError'>, [Errno 2] No such file or directory: file: /var/run/supervisor.sock
Possible Causes
- The
supervisordmaster daemon process is not running. - The socket file path specified in
[supervisorctl]does not match the socket file path defined under[unix_http_server]. - System cleanup tasks deleted socket files located in
/tmpacross system reboots.
Investigation and Diagnosis
- Step 1: Verify daemon process existence
- Command:
ps aux | grep supervisordorsystemctl status supervisor - Output Analysis: If no
supervisordprocess is listed, the daemon is stopped.
- Step 2: Compare configuration socket paths
- Command:
grep -E "file|serverurl" /etc/supervisor/supervisord.conf - Output Analysis: Ensure the path under
[unix_http_server] file=matches[supervisorctl] serverurl=unix://...exactly.
Root Cause Identification
If the process is running but socket file is missing in /tmp, systemd tmpfiles.d or /tmp periodic cleanup wiped the socket file.
Resolution
- Correct configuration mismatches to point to persistent runtime paths like
/var/run/supervisor.sock:
[unix_http_server]
file=/var/run/supervisor.sock
[supervisorctl]
serverurl=unix:///var/run/supervisor.sock
- Start or restart the main daemon:
sudo systemctl restart supervisor
Verification
sudo supervisorctl status
Verify supervisorctl successfully connects and lists managed programs.
Prevention
Avoid locating UNIX socket files (.sock) or PID files in /tmp. Always use standard persistent runtime storage directories like /var/run/ or /run/.
Problem 3: Configuration Changes Not Applied
Symptoms
Configuration parameters (e.g., updated environment variables or command arguments) are saved to /etc/supervisor/conf.d/myapp.conf, but running supervisorctl restart myapp continues executing the old configuration definition.
Possible Causes
Editing files on disk updates the file system, but does not reload configuration directives into supervisord daemon memory. supervisorctl restart only restarts the binary using the existing in-memory configuration.
Investigation and Diagnosis
- Step 1: Query unapplied configuration changes
- Command:
sudo supervisorctl reread - Expected Output:
myapp: available
- Interpretation: A status of
availableorchangedindicates Supervisord detected file modifications that have not yet been updated in memory.
Root Cause Identification
Executing restart instead of the two-step reload workflow (reread -> update).
Resolution
Apply configuration updates using the appropriate operational sequence:
sudo supervisorctl reread
sudo supervisorctl update
Verification
sudo supervisorctl status myapp
Check that uptime reset appropriately, indicating update applied the change.
Prevention
Incorporate supervisorctl reread && supervisorctl update into CI/CD deployment pipelines rather than using supervisorctl restart.
11. Security and Best Practices
Socket and File Permissions
- Restrict Socket Access: Grant socket access only to administrative users. Set
chmod=0700orchmod=0770in[unix_http_server]to prevent unprivileged users from sending signals to managed processes. - Set Socket Ownership: Use explicit user/group assignment, such as
chown=root:sudo.
[unix_http_server]
file=/var/run/supervisor.sock
chmod=0770
chown=root:sudo
Securing the HTTP Web UI
- Avoid Binding to 0.0.0.0: Never bind
[inet_http_server]to all network interfaces (0.0.0.0) without firewall controls and strong authentication. Bind to127.0.0.1and route connections over an SSH tunnel or encrypted reverse proxy (Nginx/Envoy). - Hashed Passwords: Use SHA-1 password hashes instead of cleartext passwords in configuration files:
echo -n "MySecretPassword" | sha1sum
# Output: 7b21848ac9af35ff03b6e8b4e78265004e8705a2
In configuration:
[inet_http_server]
port=127.0.0.1:9001
username=admin
password={SHA}7b21848ac9af35ff03b6e8b4e78265004e8705a2
Principle of Least Privilege
- Drop Child Privileges: Always run subprocesses under dedicated unprivileged application service accounts using the
user=parameter.
[program:background-worker]
command=/usr/bin/python3 worker.py
user=appuser ; Run subprocess as unprivileged system user
12. Performance and Resource Optimisation
Open File Descriptors and Subprocess Limits
Supervisord daemons managing large pools of workers can quickly hit kernel descriptor limits. Adjust descriptor settings inside [supervisord]:
[supervisord]
minfds=65536 ; Set file descriptor limits for heavy concurrent I/O
minprocs=4096 ; Expand maximum child process allocation slots
Log File Saturation Prevention
Uncapped output writing can exhaust disk storage. Always define strict rotation constraints for every managed program:
[program:heavy-logger]
command=/usr/local/bin/app
stdout_logfile_maxbytes=20MB
stdout_logfile_backups=5
stderr_logfile_maxbytes=20MB
stderr_logfile_backups=5
For high-throughput logging where log rotation overhead is handled by external log shippers (e.g., Vector or Fluentbit), stream stdout directly to /dev/stdout:
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
13. Backup, Recovery and Maintenance
Backing Up Configuration States
To ensure fast disaster recovery, back up the configuration directory structure:
# Archive configuration tree
tar -czvf supervisor-config-backup.tar.gz /etc/supervisor/
Disaster Recovery and State Rebuild Procedure
If a server experiences hardware failure, restore service state on a clean host:
- Install Supervisord package via package manager.
- Restore configuration tree to
/etc/supervisor/. - Ensure runtime paths, service users, and log directories exist:
sudo mkdir -p /var/log/supervisor /var/run/
sudo systemctl restart supervisor
- Confirm process restoration state:
sudo supervisorctl status
14. Automation and Integration
Using Supervisord inside Docker Containers
In Docker containers running multiple background components, Supervisord can serve as PID 1. Set nodaemon=true to keep the master process running in the foreground:
FROM python:3.11-slim
RUN apt-get update && apt-get install -y supervisor && rm -rf /var/lib/apt/lists/*
COPY supervisord.conf /etc/supervisor/supervisord.conf
COPY conf.d/ /etc/supervisor/conf.d/
COPY . /app
WORKDIR /app
# Run Supervisord in foreground mode as PID 1
CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/supervisord.conf"]
Systemd Integration Service Unit
When installed via Linux distribution package managers, Supervisord is managed as a standard systemd unit (/lib/systemd/system/supervisor.service):
[Unit]
Description=Supervisor process control system for UNIX
Documentation=http://supervisord.org
After=network.target
[Service]
ExecStart=/usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf
ExecStop=/usr/bin/supervisorctl shutdown
ExecReload=/usr/bin/supervisorctl reread && /usr/bin/supervisorctl update
KillMode=process
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.target
15. Practical Labs and Exercises
Lab 1: Deploying a Web Application Under Supervisord Control
Difficulty: Beginner
Objective: Install Supervisord, configure a Python HTTP server application running as an unprivileged user, and verify log redirection.
- Install Supervisord on your host system.
- Create an unprivileged user
labuser:
sudo useradd -m -s /bin/bash labuser
- Create a test directory and a simple Python server:
sudo mkdir -p /var/www/labapp
echo "Hello from Supervisord" | sudo tee /var/www/labapp/index.html
- Create configuration
/etc/supervisor/conf.d/labapp.conf:
[program:labapp]
command=/usr/bin/python3 -m http.server 8080
directory=/var/www/labapp
user=labuser
autostart=true
autorestart=unexpected
stdout_logfile=/var/log/supervisor/labapp.log
stderr_logfile=/var/log/supervisor/labapp.err
- Validation:
Execute
supervisorctl rereadandsupervisorctl update. Confirm withcurl [http://127.0.0.1:8080](http://127.0.0.1:8080)and verify/var/log/supervisor/labapp.logcontains access entries.
Lab 2: Troubleshooting a Broken Process Stuck in FATAL
Difficulty: Intermediate (Troubleshooting)
Objective: Diagnose and repair an application failing to launch due to path and privilege misconfigurations.
- Deploy the broken configuration
/etc/supervisor/conf.d/broken_app.conf:
[program:broken_app]
command=/usr/bin/python3 -c "import non_existent_module"
directory=/tmp
user=nobody
autostart=true
autorestart=unexpected
startsecs=3
stdout_logfile=/var/log/supervisor/broken.log
stderr_logfile=/var/log/supervisor/broken.err
- Load configuration:
sudo supervisorctl reread && sudo supervisorctl update. - Run
supervisorctl statusand observe state transitioning toBACKOFFthenFATAL. - Task:
- Inspect standard error using
supervisorctl tail broken_app stderr. - Identify missing dependencies or errors in output.
- Update
command=to/usr/bin/python3 -c "import time; time.sleep(300)". - Re-apply configuration changes via
rereadandupdate.
- Validation: Confirm
supervisorctl status broken_appdisplays stateRUNNING.
16. Quick Reference and Cheat Sheet
Important Configuration Files
| Path | Purpose |
|---|---|
/etc/supervisor/supervisord.conf |
Primary configuration file (Debian/Ubuntu) |
/etc/supervisord.conf |
Primary configuration file (RHEL/CentOS) |
/etc/supervisor/conf.d/*.conf |
Drop-in program configuration directory (Debian/Ubuntu) |
/etc/supervisord.d/*.ini |
Drop-in program configuration directory (RHEL/CentOS) |
Important Log Locations
| Log Path | Description |
|---|---|
/var/log/supervisor/supervisord.log |
Master daemon operational log |
/var/log/supervisor/<program>.log |
Program standard output (stdout) log path |
/var/log/supervisor/<program>.err |
Program standard error (stderr) log path |
Common Administrative Commands
| Task | Command |
|---|---|
| View process status | supervisorctl status |
| Load configuration changes into memory | supervisorctl reread |
| Apply configuration changes safely | supervisorctl update |
| Start/Stop specific application | supervisorctl start <app> / supervisorctl stop <app> |
| Follow application standard log output | supervisorctl tail -f <app> |
| Shutdown master daemon process | supervisorctl shutdown |
Troubleshooting Quick Reference
| Symptom | Initial Diagnostic Command | Primary Root Causes |
|---|---|---|
Process stuck in FATAL or BACKOFF |
supervisorctl tail <app> stderr |
Missing executable path, missing dependency, process forking into background. |
Socket file error (FileNotFoundError) |
systemctl status supervisor |
Daemon stopped, or socket file path mismatch between daemon and client configs. |
| Edited configs not taking effect | supervisorctl reread |
Executed restart instead of reread followed by update. |
| Permission denied on logs or socket | ls -la /var/log/supervisor/ |
Log directory or socket file permissions restricted for configured user=. |
Ports and Sockets
| Protocol / Interface | Default Path / Address | Purpose |
|---|---|---|
| UNIX Domain Socket | /var/run/supervisor.sock |
Local XML-RPC control interface for supervisorctl |
| TCP Network Socket | 127.0.0.1:9001 |
Remote XML-RPC API and Web UI control interface |
17. Further Reading and References
- Official Supervisor Introduction Guide: https://supervisord.org/introduction
- Official Supervisor Configuration Reference: https://supervisord.org/configuration
- Official Subprocess Documentation & State Machine: https://supervisord.org/subprocess
- Running Supervisor and Command-Line Interface: https://supervisord.org/running