1. Overview
Gunicorn (Green Unicorn) is a Web Server Gateway Interface (WSGI) HTTP server for Python applications designed around a pre-fork worker model. Originally ported from Ruby’s Unicorn project, Gunicorn sits between a web server or reverse proxy (such as Nginx or Apache) and a Python web application (such as Django, Flask, Bottle, or Pyramid). Its primary responsibility is to execute Python application code concurrently, manage worker process lifecycles, and translate HTTP requests passed from the proxy into standard WSGI calls.
In production Linux infrastructure, Gunicorn solves the concurrency and process management challenges inherent to Python web execution. Plain Python interpreters execute code single-threaded within a single process due to the Global Interpreter Lock (GIL). Gunicorn bypasses this limitation by spawning multiple worker processes to handle concurrent client requests across available CPU cores.
+-----------------+ HTTP/HTTPS +-------------------+ Unix Socket / TCP +-------------------+
| Web Client | -------------------> | Nginx Proxy | -------------------------> | Gunicorn Master |
+-----------------+ +-------------------+ +-------------------+
|
+----------------+----------------+
| |
+--------------+ +--------------+
| Worker Proc 1| | Worker Proc 2|
+--------------+ +--------------+
| |
+--------------+ +--------------+
| WSGI App Code| | WSGI App Code|
+--------------+ +--------------+
Common Use Cases
- WSGI Application Serving: Hosting production Python web frameworks including Django, Flask, Pyramid, and Bottle.
- Asynchronous & Threaded Workloads: Running I/O-heavy web services using thread pools or greenlet event loops.
- ASGI Framework Adapter: Serving ASGI applications (e.g., FastAPI, Starlette, Django ASGI) using integrated ASGI worker classes like
uvicorn.workers.UvicornWorker.
Key Advantages
- Pre-fork Model: Simple process management; worker crashes do not bring down the entire application server.
- Low Resource Overhead: Fast process creation and minimal memory footprint compared to full application servers.
- Zero-Downtime Reloads: Supports graceful worker restarts on code or configuration changes without dropping active TCP sockets.
- Broad Compatibility: Native implementation of PEP 3333 (WSGI) with optional worker engines (
sync,gthread,gevent,eventlet,uvicorn).
Limitations
- Static File Handling: Lacks optimized I/O mechanisms (like
sendfile) for serving static assets efficiently; must rely on Nginx or CDNs. - TLS/SSL Termination: While Gunicorn supports native TLS, terminating TLS at the application level adds unnecessary CPU load to Python processes; TLS should be offloaded to Nginx or a Load Balancer.
- Direct Public Exposure: Gunicorn is susceptible to Slowloris and denial-of-service (DoS) attacks if directly exposed to public client traffic without a buffering reverse proxy.
2. Architecture and Mental Model
Gunicorn operates using a UNIX pre-fork model. A single Master process manages one or more Worker processes that handle incoming HTTP requests. Understanding the division of duties between the Master and Worker processes is essential for reasoning about performance bottlenecks, memory leaks, and service failures.
graph TD
Client[Web Client] -->|HTTP / HTTPS| Proxy[Nginx Reverse Proxy]
Proxy -->|Unix Socket / Local TCP| Master[Gunicorn Master Process]
subgraph Gunicorn Process Group
Master -->|Manages / Reaps| W1[Worker Process 1]
Master -->|Manages / Reaps| W2[Worker Process 2]
Master -->|Manages / Reaps| W3[Worker Process 3]
end
W1 -->|WSGI Call| App1[Python WSGI App]
W2 -->|WSGI Call| App2[Python WSGI App]
W3 -->|WSGI Call| App3[Python WSGI App]
App1 -->|Queries| DB[(Database / Cache)]
App2 -->|Queries| DB
App3 -->|Queries| DB
The Master Process
The Master process does not accept connections or handle HTTP requests directly from clients. It runs an execution loop that manages the pool of worker processes:
- Socket Binding: Binds to configured network interfaces (TCP) or UNIX domain sockets before spawning workers.
- Process Lifecycle Control: Spawns workers at startup and continuously monitors worker health. If a worker process exits unexpectedly or becomes uncommunicative, the Master reaps it and immediately forks a new replacement.
- Signal Handling: Listens for standard POSIX signals to execute operational tasks:
SIGHUP: Reloads configuration files and gracefully restarts all worker processes without closing listening sockets.SIGTERM/SIGINT: Triggers immediate worker shutdown.SIGQUIT: Initiates graceful worker shutdown (allows workers to finish handling active requests up to--graceful-timeout).SIGTTIN/SIGTTOU: Dynamically increments or decrements the number of active worker processes.
Worker Processes
Worker processes perform the actual application work:
- Connection Handling: Accept incoming client socket connections delegated by the OS kernel from the master’s bound socket.
- Request Parsing: Read raw HTTP headers and request bodies sent by Nginx.
- WSGI Execution: Translate the HTTP request into a Python WSGI environment dictionary (
environ) and pass it to the WSGI application entry point (module:callable). - Response Streaming: Receive response status, headers, and body iterables returned by Python application code and stream them back over the socket to Nginx.
Complete Request Lifecycle
- Client Request: A browser initiates a TLS request to
[https://example.com/api/v1/resource](https://example.com/api/v1/resource). - Proxy Ingestion: Nginx handles TLS decryption, validates headers, and maps the request location to a backend proxy block.
- Socket Forwarding: Nginx forwards the raw HTTP stream across a UNIX domain socket (
unix:/run/gunicorn.sock) or loopback TCP port (127.0.0.1:8000). - Worker Pick-up: An idle Gunicorn worker process waiting on
accept()accepts the connection. - App Callable Invocation: The worker executes the configured Python entry point (e.g.,
myproject.wsgi:application). - Data Processing: Python application logic executes database queries, Redis cache lookups, or external API calls.
- Streaming Response: The worker streams the HTTP response code (e.g.,
200 OK) and payload back through the socket to Nginx. - Client Delivery: Nginx buffers and returns the response payload to the web client.
3. Core Concepts and Terminology
WSGI (PEP 3333)
The Web Server Gateway Interface is a standardized interface specification between web servers and Python web application frameworks. A WSGI application is defined as a standard Python callable (a function or class with a __call__ method) accepting two positional parameters:
environ: A Python dictionary containing request headers, HTTP methods, paths, and server environment variables.start_response: A callback function used to begin sending response status codes and headers back to the server.
Application Entry Point Syntax
Gunicorn identifies application callables using the module_name:callable_name convention:
myproject.wsgi:application: For Django projects, loadingapplicationfrommyproject/wsgi.py.app:appormain:app: For Flask applications, loading theappinstance fromapp.pyormain.py.
Pre-fork Engine vs Async Workers
- Sync Workers (
sync): The default worker model. Each worker process processes exactly one request at a time synchronously. If all sync workers are busy waiting on database queries or slow external API calls, incoming requests queue up or timeout. - Threaded Workers (
gthread): Spawns a pool of execution threads within each worker process. This allows a single worker process to handle multiple concurrent I/O-bound requests without requiring greenlet monkey-patching. - Greenlet Async Workers (
gevent/eventlet): Utilizes cooperative multitasking (greenlets). I/O operations yield control back to an event loop, allowing thousands of concurrent connections within a single worker process. Requires application code and third-party libraries to be compatible with greenlet patching. - ASGI Workers (
uvicorn.workers.UvicornWorker): Enables Gunicorn to serve Python ASGI applications (Asynchronous Server Gateway Interface) like FastAPI or Starlette.
Preloading (--preload)
By default, Gunicorn forks worker processes first, and then each worker independently imports the Python WSGI application code. When --preload (or preload = True) is set, the Master process imports the application into memory before forking workers. Workers inherit the pre-loaded memory via Linux Copy-on-Write (CoW), significantly reducing initial RAM usage.
Warning on Preloading: If application modules initialize stateful connections (such as database pools, Redis clients, or file descriptors) during module import, those connections will be shared across all forked workers, leading to corrupted sockets and broken database transactions.
4. Installation and Version Management
Prerequisites
Gunicorn requires Python 3.7+ and a working installation of pip.
Production Installation Methods
1. Python Virtual Environment (Recommended)
Installing Gunicorn directly inside your application’s Python virtual environment guarantees isolated dependencies and prevents system-level package conflicts.
# Navigate to application directory
cd /opt/myapp
# Create virtual environment if not present
python3 -m venv venv
# Activate virtual environment
source venv/bin/python
# Upgrade pip and install Gunicorn along with async dependencies
/opt/myapp/venv/bin/pip install --upgrade pip
/opt/myapp/venv/bin/pip install gunicorn gevent uvicorn
2. System Package Managers (Ubuntu/Debian & RHEL/Fedora)
# Ubuntu / Debian
sudo apt update
sudo apt install gunicorn python3-gunicorn
# RHEL / Rocky Linux / AlmaLinux
sudo dnf install gunicorn
Identifying Installed Version
# Check version using specific virtualenv binary
/opt/myapp/venv/bin/gunicorn --version
# Check system installation
gunicorn --version
Expected Output:
gunicorn (version 21.2.0)
Installation File Layout
- Executable Binary:
/opt/myapp/venv/bin/gunicorn - Python Libraries:
/opt/myapp/venv/lib/python3.x/site-packages/gunicorn/ - Systemd Unit File Directory:
/etc/systemd/system/
5. Configuration Files and Directory Structure
Configuration Architecture
Gunicorn supports configuration through three distinct mechanisms, evaluated in order of precedence:
- Command-Line Arguments (Highest Precedence)
- Python Configuration File (
-c/--config) - Framework Defaults (Lowest Precedence)
Using a dedicated Python file (gunicorn.conf.py) is recommended for production environments because it supports dynamic logic, environment evaluation, and post-fork lifecycle hooks.
System Layout Reference
| File / Directory | Purpose | Important Details |
|---|---|---|
/opt/myapp/gunicorn.conf.py |
Main application configuration file. | Written in standard Python; supports dynamic values and lifecycle hooks. |
/etc/systemd/system/gunicorn.service |
Systemd service definition. | Manages daemon execution, process restart policies, environment files, and permissions. |
/run/gunicorn/gunicorn.sock |
UNIX Domain Socket file. | Used for zero-overhead local communication between Nginx and Gunicorn. |
/run/gunicorn/gunicorn.pid |
Process ID file. | Contains the PID of the active Gunicorn Master process. |
/var/log/gunicorn/access.log |
Application access logs. | Stores standard HTTP access records when stdout redirection is not used. |
/var/log/gunicorn/error.log |
Gunicorn error and debug logs. | Captures Python tracebacks, worker lifecycle events, and process timeouts. |
Production gunicorn.conf.py Example
import multiprocessing
import os
# Server Socket Binding
bind = "unix:/run/gunicorn/gunicorn.sock"
# Worker Processes & Concurrency
# Formula: (2 * CPUs) + 1 for sync workers
workers = multiprocessing.cpu_count() * 2 + 1
worker_class = "sync"
threads = 1
# Process Management & Safety
timeout = 30
graceful_timeout = 30
keepalive = 2
# Worker Recycling (Mitigates memory leaks)
max_requests = 1000
max_requests_jitter = 50
# Security & Privilege Separation
user = "deploy"
group = "deploy"
umask = 0o007
# Logging Configuration
accesslog = "-" # Output access logs to stdout (captured by journald)
errorlog = "-" # Output error logs to stderr (captured by journald)
loglevel = "info"
capture_output = True
# Process Naming
proc_name = "myapp_gunicorn"
# Preloading Code (Use with caution)
preload_app = False
# Post-fork lifecycle hooks (Fixes connection pool cloning when preload=True)
def post_fork(server, worker):
server.log.info("Worker spawned (pid: %s)", worker.pid)
# Force database connection reset post-fork if necessary
Validating Configuration Files
To verify configuration syntax without starting the server, run Gunicorn with the --check-config flag:
/opt/myapp/venv/bin/gunicorn --config /opt/myapp/gunicorn.conf.py --check-config myapp.wsgi:application
6. Service Management and Common Operations
Systemd Integration
For production Linux servers, Gunicorn should be managed as a persistent systemd service.
Example /etc/systemd/system/gunicorn.service
[Unit]
Description=Gunicorn daemon for Application Service
After=network.target
[Service]
Type=notify
User=deploy
Group=deploy
WorkingDirectory=/opt/myapp
RuntimeDirectory=gunicorn
ExecStart=/opt/myapp/venv/bin/gunicorn \
--config /opt/myapp/gunicorn.conf.py \
myproject.wsgi:application
ExecReload=/bin/kill -s HUP $MAINPID
KillMode=mixed
TimeoutStopSec=5
PrivateTmp=true
Restart=on-failure
RestartSec=5s
[Install]
WantedBy=multi-user.target
Essential Systemctl Commands
# Reload systemd configuration after modifying unit file
sudo systemctl daemon-reload
# Enable Gunicorn to start automatically on system boot
sudo systemctl enable gunicorn
# Start the Gunicorn service
sudo systemctl start gunicorn
# Check active service status, PID, and recent log outputs
sudo systemctl status gunicorn
# Gracefully reload workers (Zero-Downtime deployment)
sudo systemctl reload gunicorn
# Hard restart of Master process and all workers
sudo systemctl restart gunicorn
# Stop the Gunicorn service
sudo systemctl stop gunicorn
Reload vs. Restart Operation
| Operation | System Mechanism | Process Behavior | Impact on Traffic |
|---|---|---|---|
Reload (systemctl reload) |
Sends SIGHUP signal to Master PID. |
Master reads updated config/code, spawns new workers, and gracefully terminates old workers as they finish active requests. | Zero Downtime. Listening sockets remain open in kernel buffer. |
Restart (systemctl restart) |
Sends SIGTERM followed by new execution. |
Master process dies completely; socket is closed; new Master process initializes and imports code. | Brief Downtime. Incoming requests may fail with 502 Bad Gateway until new socket is bound. |
7. Common Administrative Tasks and Workflows
Task 1: Zero-Downtime Application Deployment
Objective: Deploy updated Python code to a production server without dropping incoming user requests.
- Pull updated application code:
cd /opt/myapp
git pull origin main
- Update dependencies inside virtualenv:
/opt/myapp/venv/bin/pip install -r requirements.txt
- Execute database migrations:
/opt/myapp/venv/bin/python manage.py migrate
- Collect static assets:
/opt/myapp/venv/bin/python manage.py collectstatic --noinput
- Issue Graceful Reload to Gunicorn:
sudo systemctl reload gunicorn
- Verify graceful reload execution in system journal:
sudo journalctl -u gunicorn -n 20 --no-pager
Expected Log Output:
[INFO] Handling signal: hup
[INFO] Hang up: Master
[INFO] Booting worker with pid: 142051
[INFO] Worker exiting (pid: 141890)
Task 2: Configuring Nginx Reverse Proxy over UNIX Socket
Objective: Wire Nginx to forward HTTP traffic securely to a local Gunicorn UNIX socket.
- Create Nginx server block at
/etc/nginx/sites-available/myapp.conf:
upstream gunicorn_backend {
server unix:/run/gunicorn/gunicorn.sock fail_timeout=0;
}
server {
listen 80;
server_name app.example.com;
client_max_body_size 4G;
location /static/ {
alias /opt/myapp/staticfiles/;
expires 30d;
add_header Cache-Control "public, no-transform";
}
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_redirect off;
if (!-f $request_filename) {
proxy_pass http://gunicorn_backend;
break;
}
}
error_page 500 502 503 504 /500.html;
location = /500.html {
root /opt/myapp/templates/;
}
}
- Enable site configuration and test Nginx syntax:
sudo ln -s /etc/nginx/sites-available/myapp.conf /etc/nginx/sites-enabled/
sudo nginx -t
- Reload Nginx to apply changes:
sudo systemctl reload nginx
8. CLI Command Reference and Cheatsheet
Execution & Binding Commands
| Command | Purpose | Flags / Arguments | Example | Explanation |
|---|---|---|---|---|
gunicorn |
Start WSGI server | module:callable |
gunicorn app:app |
Starts Gunicorn with default bind (127.0.0.1:8000). |
gunicorn --bind |
Bind to address/socket | -b, --bind |
gunicorn -b unix:/run/gunicorn.sock app:app |
Binds to UNIX domain socket for Nginx communication. |
gunicorn --config |
Specify config file | -c, --config |
gunicorn -c gunicorn.conf.py app:app |
Loads configuration settings from a Python file. |
gunicorn --chdir |
Change directory | --chdir |
gunicorn --chdir /opt/myapp myapp.wsgi:application |
Switches working directory prior to importing application. |
Worker & Concurrency Flags
| Command / Flag | Purpose | Options / Syntax | Example | Explanation |
|---|---|---|---|---|
-w, --workers |
Set worker count | Integer ($2 \times \text{CPUs} + 1$) | gunicorn -w 4 app:app |
Spawns specified number of worker processes. |
--worker-class |
Set execution engine | sync, gthread, gevent, uvicorn.workers.UvicornWorker |
gunicorn --worker-class gthread app:app |
Changes worker concurrency model. |
--threads |
Threads per worker | Integer | gunicorn -w 2 --threads 4 app:app |
Sets thread count per worker when using gthread. |
-t, --timeout |
Worker kill timeout | Seconds (Default: 30) | gunicorn --timeout 60 app:app |
Aborts worker if silent longer than timeout seconds. |
--max-requests |
Recycle workers | Integer | gunicorn --max-requests 1000 app:app |
Restarts worker after processing N requests to limit memory leaks. |
--max-requests-jitter |
Randomize worker recycling | Integer | gunicorn --max-requests-jitter 50 app:app |
Adds random delta to max-requests to prevent all workers restarting at once. |
--preload |
Preload application | Flag | gunicorn --preload app:app |
Imports code into Master before forking workers to save RAM. |
System Inspection Commands
| Task | Command | Target / Context | Expected Output / Notes |
|---|---|---|---|
| View active sockets | `ss -tlnp | grep 8000` | TCP Listeners |
| Inspect socket file | ls -la /run/gunicorn/gunicorn.sock |
File Permissions | Verifies socket existence and user/group access. |
| View process tree | `ps aux | grep gunicorn` | Process table |
| Direct app test | curl -v --unix-socket /run/gunicorn/gunicorn.sock http://localhost/ |
Bypass Proxy | Directly tests Gunicorn HTTP response via UNIX socket. |
9. Logs, Monitoring and Diagnostics
Log Configuration & Redirects
Systemd environments function best when services stream logs directly to standard output streams (stdout and stderr), allowing systemd-journald to manage rotation, indexing, and storage.
In gunicorn.conf.py:
accesslog = "-" # Redirect access logs to stdout
errorlog = "-" # Redirect error logs to stderr
loglevel = "info"
capture_output = True # Capture print statements and stdout from Python code
Reading Systemd Logs with journalctl
# Stream live Gunicorn log output
sudo journalctl -u gunicorn -f
# View last 100 log lines without line truncation
sudo journalctl -u gunicorn -n 100 --no-pager
# Filter logs for errors and critical failure messages
sudo journalctl -u gunicorn -p err..crit --since "1 hour ago"
Interpreting Common Log Patterns
1. Worker Timeout Log
[2026-11-03 14:22:10 +0000] [142051] [CRITICAL] WORKER TIMEOUT (pid:142099)
[2026-11-03 14:22:10 +0000] [142099] [INFO] Worker exiting (pid: 142099)
[2026-11-03 14:22:11 +0000] [142301] [INFO] Booting worker with pid: 142301
Interpretation: A worker process took longer than --timeout (e.g., 30s) to return a response. The Master process killed worker 142099 with SIGKILL and immediately spawned replacement worker 142301.
2. Segfault / Native Crash Log
[2026-11-03 14:25:01 +0000] [142051] [WARNING] Worker (pid:142301) was killed by signal 11
[2026-11-03 14:25:01 +0000] [142410] [INFO] Booting worker with pid: 142410
Interpretation: Signal 11 indicates a Segmentation Fault. A native C extension used by Python (e.g., OpenCV, NumPy, database driver) attempted to access invalid memory.
10. Troubleshooting Guide and Common Failure Scenarios
Problem 1: Gunicorn Fails to Start — Import Error
Symptoms
systemctl status gunicorn displays failed state with exit code 1/FAILURE. Running journalctl -u gunicorn shows a Python traceback ending in ModuleNotFoundError or ImportError.
ImportError: No module named 'myproject'
Possible Causes
- Invalid module path in Gunicorn startup parameters (
myproject.wsgi:application). - Executable running under global Python instead of project virtualenv.
WorkingDirectorydirective in systemd unit points to incorrect path.- Missing environment variables (
PYTHONPATHorDJANGO_SETTINGS_MODULE).
Investigation and Diagnosis
- Verify directory structure and paths: Check if the module specified in the command exists relative to the working directory.
- Reproduce import manually using virtualenv: Execute Python directly using the application service user context:
cd /opt/myapp
/opt/myapp/venv/bin/python -c "from myproject.wsgi import application"
If this fails: The Python environment is missing dependencies or PYTHONPATH is inaccurate.
If this succeeds: The issue is unique to systemd environment variables or paths.
Root Cause Identification
If manual execution fails with ModuleNotFoundError, check missing installed packages via pip list inside the virtualenv. If manual execution succeeds, compare systemd unit settings (WorkingDirectory, ExecStart) against manual paths.
Resolution
Update /etc/systemd/system/gunicorn.service to explicitly set WorkingDirectory and use absolute paths for virtualenv binaries:
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/venv/bin/gunicorn --config /opt/myapp/gunicorn.conf.py myproject.wsgi:application
Verification
sudo systemctl daemon-reload
sudo systemctl restart gunicorn
sudo systemctl status gunicorn
Prevention
Always deploy using virtual environments with explicit absolute binary paths in unit files.
Problem 2: App Imports Fine but Crashes on First Request
Symptoms
Gunicorn starts up successfully without error logs, but the first incoming HTTP request returns 500 Internal Server Error to the client, and journalctl reports a traceback.
psycopg2.OperationalError: could not connect to server: Connection refused
Possible Causes
- Database server (PostgreSQL/MySQL) is unavailable or unreachable.
- Redis cache server unreachable.
- Missing database credentials or secret keys in systemd service environment.
- Unapplied database migrations.
Investigation and Diagnosis
- Reproduce application import:
cd /opt/myapp && /opt/myapp/venv/bin/python -c "from myapp.wsgi import application"
Result: Succeeds because module loading does not open database sockets immediately. 2. Issue local HTTP request to trigger view logic:
curl -v http://127.0.0.1:8000/
- Inspect traceback in system journal:
sudo journalctl -u gunicorn -n 50 --no-pager
Root Cause Identification
Review the exception raised during HTTP request handling in journalctl. If the traceback points to psycopg2 or redis, verify network reachability to database/cache hosts.
Resolution
- Ensure systemd unit loads environment secrets via
EnvironmentFile:
EnvironmentFile=/opt/myapp/.env
- Verify local or remote database service state:
sudo systemctl status postgresql
Verification
curl -i http://127.0.0.1:8000/health
Expected Output: HTTP/1.1 200 OK
Prevention
Implement dedicated health-check endpoints (/health) that explicitly validate database and cache connectivity during CI/CD smoke tests.
Problem 3: 502 Bad Gateway from Nginx
Symptoms
Browsing to the site returns 502 Bad Gateway from Nginx. Nginx error logs (/var/log/nginx/error.log) report:
2026/11/03 14:30:00 [error] 1410#1410: *1 connect() to unix:/run/gunicorn/gunicorn.sock failed (2: No such file or directory)
or
2026/11/03 14:30:00 [error] 1410#1410: *1 connect() to unix:/run/gunicorn/gunicorn.sock failed (13: Permission denied)
Possible Causes
- Gunicorn service is stopped or crashing.
- Socket path mismatch between Nginx
proxy_passand Gunicornbindconfiguration. - Insufficient filesystem permissions on UNIX socket file or socket parent directory.
Investigation and Diagnosis
- Check Gunicorn service status:
sudo systemctl status gunicorn
If Gunicorn is down, start it. 2. Verify socket file existence and path:
ls -la /run/gunicorn/gunicorn.sock
Compare this path with proxy_pass in /etc/nginx/sites-enabled/myapp.conf.
3. Inspect socket ownership and permissions:
ls -ld /run/gunicorn
ls -l /run/gunicorn/gunicorn.sock
Root Cause Identification
- If
lsreturnsNo such file or directory: Gunicorn failed to bind or directory/run/gunicorndoes not exist. - If socket ownership is
deploy:deploywith mode0700(srwx------), Nginx user (www-dataornginx) cannot read/write to socket, raisingPermission denied.
Resolution
- Configure systemd unit to generate socket runtime directory automatically:
RuntimeDirectory=gunicorn
- Set permissive umask and group ownership in
gunicorn.conf.py:
bind = "unix:/run/gunicorn/gunicorn.sock"
umask = 0o007 # Socket created with 770 permissions
- Add Nginx execution user (
www-data/nginx) to Gunicorn group (deploy):
sudo usermod -aG deploy www-data
Verification
sudo systemctl restart gunicorn
sudo systemctl reload nginx
curl -I http://localhost/
Prevention
Use systemd RuntimeDirectory directives to manage runtime socket directories automatically across system reboots.
Problem 4: Worker Timeout (WORKER TIMEOUT in logs)
Symptoms
Gunicorn error logs contain repeated CRITICAL WORKER TIMEOUT messages. Clients experience intermittent 504 Gateway Timeout or connection reset errors.
Possible Causes
- Long-running database queries blocking synchronous worker threads.
- Unhandled synchronous external HTTP requests (e.g., third-party API call without request timeouts).
- High concurrency saturation: sync worker pool exhausted by slow clients.
Investigation and Diagnosis
- Check worker log tracebacks:
sudo journalctl -u gunicorn -p crit --no-pager
- Inspect active worker stack traces:
Send
SIGUSR1to Gunicorn Master to log worker stack traces:
sudo kill -usr1 $(cat /run/gunicorn/gunicorn.pid)
sudo journalctl -u gunicorn -n 100 --no-pager
Root Cause Identification
If stack traces show workers blocked inside requests.get() or psycopg2.execute(), the root cause is external I/O blocking synchronous execution.
Resolution
- Enforce strict timeouts on all external outbound HTTP calls inside application code (e.g.,
requests.get(url, timeout=5)). - If application handles long-lived requests (such as file downloads or long-polling), switch worker class from
synctogthread:
worker_class = "gthread"
workers = 2
threads = 4
timeout = 120
- Offload long-running tasks to an asynchronous background task queue (e.g., Celery or RQ).
Verification
Monitor system journal for 30 minutes following configuration update to verify absence of WORKER TIMEOUT entries.
Prevention
Never issue unbounded network calls inside synchronous WSGI view handlers.
Problem 5: Workers Repeatedly Crashing (Segfault / OOM)
Symptoms
Workers exit silently or log Worker was killed by signal 9 (SIGKILL) or signal 11 (SIGSEGV). Service performance degraded.
Possible Causes
- Signal 9 (OOM Killer): The Linux kernel Out-Of-Memory killer terminated the worker process due to RAM exhaustion.
- Signal 11 (Segfault): Native C library error (e.g., memory corruption in binary extensions).
Investigation and Diagnosis
- Check kernel messages for OOM invocation:
sudo dmesg -T | grep -i oom
Look for: Killed process <pid> (gunicorn) total-vm:... anon-rss:...
2. Inspect memory usage across worker processes:
ps aux --sort=-%mem | grep gunicorn
Root Cause Identification
- If
dmesgshows OOM Killer entries: Application memory leak or over-allocated worker count for physical server RAM. - If error log shows signal 11: Binary extension incompatibility.
Resolution
- For OOM Memory Leaks: Enable automatic worker recycling in
gunicorn.conf.py:
max_requests = 1000
max_requests_jitter = 50
Effect: Workers restart automatically after processing 1,000 requests, releasing accumulated leaked memory back to the OS.
2. Reduce overall worker count: Decrease workers to fit within physical RAM boundaries.
Verification
sudo systemctl restart gunicorn
free -h
Prevention
Always define max_requests and max_requests_jitter in production Gunicorn deployments.
Problem 6: Address Already in Use (Errno 98)
Symptoms
Gunicorn fails to start. Log output displays:
[ERROR] Connection in use: ('127.0.0.1', 8000)
OSError: [Errno 98] Address already in use
Possible Causes
- Another instance of Gunicorn or a secondary service (e.g., Uvicorn, Django dev server) is already bound to port 8000.
- An orphaned Gunicorn Master process survived systemd service stop.
Investigation and Diagnosis
Identify process occupying target port:
sudo ss -tlnp | grep 8000
Sample Output:
LISTEN 0 128 127.0.0.1:8000 users:(("gunicorn",pid=99412,fd=6))
Root Cause Identification
An orphaned Gunicorn Master process (PID 99412) is holding socket binding outside of systemd tracking.
Resolution
- Attempt graceful shutdown of orphaned Master process using PID:
sudo kill -TERM 99412
- If process fails to exit, forcibly kill:
sudo kill -9 99412
- Restart systemd service:
sudo systemctl start gunicorn
Verification
sudo ss -tlnp | grep 8000
sudo systemctl status gunicorn
Prevention
Avoid running manual Gunicorn CLI commands on production servers managed by systemd.
Problem 7: Permission Denied on Unix Socket (SELinux Enforcement)
Symptoms
Nginx raises 502 Bad Gateway. Nginx logs report (13: Permission denied) while connecting to socket. standard Linux file permissions (ls -la) show 0777 permissions on /run/gunicorn/gunicorn.sock.
Possible Causes
Security-Enhanced Linux (SELinux) is operating in Enforcing mode and blocking Nginx (httpd_t domain) from accessing files created under non-standard security contexts.
Investigation and Diagnosis
- Check SELinux status:
sestatus
- Search audit logs for AVC denials related to Nginx and socket:
sudo ausearch -m avc -ts recent | grep nginx
Look for: denied { write } for pid=... comm="nginx" path="/run/gunicorn/gunicorn.sock" dev="tmpfs" scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:var_run_t:s0
Root Cause Identification
SELinux policy prevents Nginx process (httpd_t) from interacting with socket files labeled with default context (var_run_t or unconfined_t).
Resolution
Apply correct SELinux context type (httpd_sys_rw_content_t) to runtime socket directory:
# Apply context to socket directory
sudo semanage fcontext -a -t httpd_sys_rw_content_t "/run/gunicorn(/.*)?"
# Relabel filesystem paths
sudo restorecon -R -v /run/gunicorn
Verification
ls -Z /run/gunicorn/gunicorn.sock
Expected Output:
system_u:object_r:httpd_sys_rw_content_t:s0 /run/gunicorn/gunicorn.sock
Perform client HTTP request through Nginx to confirm resolution.
Prevention
Label application runtime directories using standard SELinux types during automated server provisioning.
Problem 8: Missing Environment Variables in Application
Symptoms
Application crashes during runtime due to KeyError when attempting to access os.environ['DATABASE_URL'] or os.environ['SECRET_KEY']. Variables function properly when tested inside user interactive login shell.
Possible Causes
Systemd services execute inside isolated non-interactive environments and do not load user shell configuration files (~/.bashrc, ~/.bash_profile, /etc/profile).
Investigation and Diagnosis
Inspect active systemd environment for running Gunicorn service:
sudo systemctl show gunicorn -p Environment
Root Cause Identification
Environment variables set in admin shell profiles are absent from systemd service environment table.
Resolution
Define required environment variables explicitly inside /etc/systemd/system/gunicorn.service or reference an environment file:
Option A: EnvironmentFile Directive (Recommended)
Create /opt/myapp/.env (permissions 0600, owned by deploy):
DJANGO_SETTINGS_MODULE=myproject.settings.production
DATABASE_URL=postgres://user:pass@localhost:5432/dbname
SECRET_KEY=super-secret-key-value
Update /etc/systemd/system/gunicorn.service:
[Service]
EnvironmentFile=/opt/myapp/.env
Option B: raw_env in gunicorn.conf.py
raw_env = [
"DJANGO_SETTINGS_MODULE=myproject.settings.production",
"SECRET_KEY=super-secret-key-value",
]
Verification
sudo systemctl daemon-reload
sudo systemctl restart gunicorn
sudo systemctl show gunicorn -p Environment
Prevention
Never rely on shell profile files for production service environment variables.
Problem 9: Worker Class Mismatch / Poor Concurrency under Load
Symptoms
Application experiences severe performance degradation under moderate client traffic. Web pages stall or fail with timeouts. CPU utilization remains low while worker processes saturate.
Possible Causes
Application utilizes default synchronous worker class (sync) for workloads involving long-polling, WebSockets, or slow I/O operations. A single client making a slow request blocks an entire worker process from servicing other users.
Investigation and Diagnosis
- Inspect running Gunicorn command flags:
ps aux | grep gunicorn
- Verify if
worker_classis unset or explicitly assigned tosync.
Root Cause Identification
sync workers process only one connection at a time. Under slow I/O workloads, all workers become blocked simultaneously.
Resolution
Match Gunicorn worker class to application concurrency profile:
For I/O Bound WSGI Applications (Django / Flask)
Switch to gthread to process multi-threaded requests per worker process:
# gunicorn.conf.py
worker_class = "gthread"
workers = 2
threads = 4
For ASGI Applications (FastAPI / Starlette)
Switch to UvicornWorker:
# gunicorn.conf.py
worker_class = "uvicorn.workers.UvicornWorker"
workers = 3
Verification
Execute benchmark load test (e.g., using hey or ab) to confirm request throughput:
hey -n 1000 -c 50 http://app.example.com/
Prevention
Evaluate request lifetime profiles (I/O vs CPU bound) prior to selecting worker class.
Problem 10: Preload Footgun: Shared State & Stale DB Connections After Fork (--preload)
Symptoms
Random InterfaceError: connection already closed or database connection corruption errors occurring across worker processes following server startup. Errors clear temporarily when workers are manually restarted.
Possible Causes
The application was started with --preload (or preload_app = True in config). Database or cache connection objects were instantiated during Python module import in the Master process prior to calling fork(). Child workers inherited duplicated file descriptors referencing identical underlying network sockets.
Investigation and Diagnosis
- Check if preload flag is enabled in Gunicorn settings:
grep -i "preload" /opt/myapp/gunicorn.conf.py /etc/systemd/system/gunicorn.service
- Inspect traceback in logs:
psycopg2.DatabaseError: SSL error: decryption failed or bad record mac
Root Cause Identification
Worker processes spawned via fork() share the same open TCP database socket created during Master preload. Concurrent operations across workers scramble socket communications.
Resolution
Method A: Disable Preloading (Simple Fix)
In gunicorn.conf.py:
preload_app = False
Method B: Re-initialize Connections in post_fork Hook
If preloading is required for memory savings, close inherited sockets inside post_fork in gunicorn.conf.py:
preload_app = True
def post_fork(server, worker):
# Force Django to close stale connections inherited from Master
try:
from django.db import connections
connections.close_all()
server.log.info("Closed inherited DB connections for worker %s", worker.pid)
except ImportError:
pass
Verification
sudo systemctl restart gunicorn
sudo journalctl -u gunicorn -n 20 --no-pager
Verify workers initialize independent database connections cleanly.
Prevention
Avoid creating stateful network client objects at root module level during application import.
11. Security and Best Practices
Privilege Minimization
- Unprivileged Daemon User: Never run Gunicorn as
root. Execute under a dedicated service user account with restricted filesystem access (User=deploy,Group=deploy). - Umask Restrictions: Enforce strict file creation umasks (
umask = 0o007or0o027) to restrict access to socket files and logs.
Network Exposure Controls
- Bind Safety: Never expose Gunicorn directly to public internet interfaces (
0.0.0.0) without a reverse proxy. Bind exclusively to loopback interfaces (127.0.0.1) or local UNIX domain sockets (unix:/run/gunicorn/gunicorn.sock). - Trusted Proxies (
forwarded_allow_ips): Configure Gunicorn to accept forwarded headers (such asX-Forwarded-For) only from designated proxy IP addresses to prevent HTTP header spoofing:
forwarded_allow_ips = "127.0.0.1, ::1"
Header Security
Ensure Nginx strips unsafe client headers and supplies validated proxy headers to Gunicorn:
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
12. Performance and Resource Optimisation
Worker Count Formulas
Synchronous Workers (sync)
For CPU-bound WSGI workloads, compute worker process count using the standard baseline formula:
$$\text{Workers} = (2 \times \text{CPU Cores}) + 1$$
Example: On a server with 4 CPU cores: $(2 \times 4) + 1 = 9 \text{ Workers}$.
Threaded Workers (gthread)
For mixed I/O-bound workloads (applications accessing remote databases or APIs):
$$\text{Workers} = \text{CPU Cores} + 1$$
$$\text{Threads per Worker} = 2 \text{ to } 4$$
Example: On a 4-core server: 5 workers, 4 threads per worker = 20 concurrent request handlers.
Memory Recycling Configuration
To mitigate gradual memory accumulation caused by Python application memory leaks, configure worker recycling parameters:
# Recycle worker process after handling 1000 requests
max_requests = 1000
# Add random jitter between 0 and 50 to prevent simultaneous worker restarts
max_requests_jitter = 50
13. Backup, Recovery and Maintenance
Essential Backup Artifacts
Maintain version-controlled backups of the following configuration files:
- Gunicorn Configuration:
/opt/myapp/gunicorn.conf.py - Systemd Service Unit:
/etc/systemd/system/gunicorn.service - Nginx Site Configuration:
/etc/nginx/sites-available/myapp.conf - Environment Variable Configuration:
/opt/myapp/.env(Store securely in password manager or vault)
Upgrade and Patching Procedure
- Test application compatibility with upgraded Gunicorn version in staging environment.
- Upgrade binary inside production virtualenv:
/opt/myapp/venv/bin/pip install --upgrade gunicorn
- Perform dry-run configuration check:
/opt/myapp/venv/bin/gunicorn --config /opt/myapp/gunicorn.conf.py --check-config myapp.wsgi:application
- Reload service cleanly:
sudo systemctl reload gunicorn
14. Automation and Integration
Ansible Deployment Example
The following Ansible task block automates the installation and configuration of Gunicorn systemd services:
- name: Deploy Gunicorn configuration
template:
src: gunicorn.conf.py.j2
dest: /opt/myapp/gunicorn.conf.py
owner: deploy
group: deploy
mode: '0644'
notify: reload gunicorn
- name: Deploy Systemd service file
template:
src: gunicorn.service.j2
dest: /etc/systemd/system/gunicorn.service
owner: root
group: root
mode: '0644'
notify:
- reload systemd
- restart gunicorn
- name: Ensure Gunicorn service is enabled and started
systemd:
name: gunicorn
enabled: yes
state: started
15. Practical Labs and Exercises
Lab 1: Deploy a Flask Application with Gunicorn and Systemd
Difficulty: Beginner Objective: Deploy a basic Flask application served by Gunicorn under systemd process control.
- Setup Directory & Virtual Environment:
sudo mkdir -p /opt/flaskapp
sudo chown -R $USER:$USER /opt/flaskapp
cd /opt/flaskapp
python3 -m venv venv
./venv/bin/pip install flask gunicorn
- Create Sample Flask App (
app.py):
from flask import Flask
app = Flask(__name__)
@app.route("/")
def index():
return "Gunicorn Lab 1: Success\n"
- Test Manual Gunicorn Execution:
./venv/bin/gunicorn --bind 127.0.0.1:8000 app:app
In a separate terminal, test response:
curl http://127.0.0.1:8000/
Terminate manual execution with Ctrl+C.
4. Create Systemd Service (/etc/systemd/system/flaskapp.service):
[Unit]
Description=Flask App Lab 1
After=network.target
[Service]
User=ubuntu
WorkingDirectory=/opt/flaskapp
ExecStart=/opt/flaskapp/venv/bin/gunicorn --workers 2 --bind 127.0.0.1:8000 app:app
Restart=on-failure
[Install]
WantedBy=multi-user.target
- Start and Validate Service:
sudo systemctl daemon-reload
sudo systemctl start flaskapp
sudo systemctl status flaskapp
curl http://127.0.0.1:8000/
Lab 2: Troubleshooting a Broken Gunicorn Socket Connection
Difficulty: Intermediate
Objective: Diagnose and repair a simulated 502 Bad Gateway error caused by UNIX socket permission mismatches.
- Simulate Broken State: Stop Gunicorn, create socket manually with restrictive permissions:
sudo systemctl stop gunicorn
sudo touch /run/gunicorn/gunicorn.sock
sudo chmod 0600 /run/gunicorn/gunicorn.sock
sudo chown root:root /run/gunicorn/gunicorn.sock
- Diagnose Failure: Attempt to curl site through Nginx:
curl -i http://localhost/
Observe: HTTP/1.1 502 Bad Gateway
3. Inspect Nginx Logs:
sudo tail -n 5 /var/log/nginx/error.log
Identify: Permission denied while connecting to upstream
4. Apply Fix:
Remove invalid socket file, update permissions in gunicorn.conf.py (umask = 0o007), and restart service:
sudo rm /run/gunicorn/gunicorn.sock
sudo systemctl restart gunicorn
- Validate Recovery:
curl -i http://localhost/
Observe: HTTP/1.1 200 OK
16. Quick Reference and Cheat Sheet
Important Configuration Files
| Path / File | Purpose |
|---|---|
/opt/myapp/gunicorn.conf.py |
Main Python configuration file. |
/etc/systemd/system/gunicorn.service |
Systemd service definition. |
/etc/nginx/sites-available/myapp.conf |
Nginx reverse proxy configuration. |
Important Log Locations
| Log Target | Purpose / Syntax |
|---|---|
journalctl -u gunicorn -f |
Stream live systemd service logs. |
/var/log/nginx/error.log |
Nginx proxy error log (captures 502 upstream errors). |
accesslog = "-" |
Stream Gunicorn access logs to stdout. |
Common Administrative Commands
| Task | Command |
|---|---|
| Start Gunicorn service | sudo systemctl start gunicorn |
| Stop Gunicorn service | sudo systemctl stop gunicorn |
| Graceful Zero-Downtime Reload | sudo systemctl reload gunicorn |
| Direct Signal Reload | sudo kill -HUP $(cat /run/gunicorn/gunicorn.pid) |
| Inspect active workers | `ps aux |
| Test app import | python -c "from myapp.wsgi import application" |
Troubleshooting Quick Reference
| Symptom | Initial Diagnostic Command | Primary Root Cause |
|---|---|---|
ModuleNotFoundError on boot |
journalctl -u gunicorn -n 50 |
Missing dependencies or invalid working directory. |
502 Bad Gateway |
ls -la /run/gunicorn/gunicorn.sock |
Gunicorn is down or socket permissions deny access. |
WORKER TIMEOUT |
journalctl -u gunicorn -p crit |
Slow database query or blocking external API call. |
Errno 98 Address already in use |
`ss -tlnp | grep 8000` |
Socket Permission Denied under SELinux |
ausearch -m avc -ts recent |
Missing httpd_sys_rw_content_t context label. |
Important Ports and Protocols
| Default Port / Endpoint | Protocol | Purpose |
|---|---|---|
127.0.0.1:8000 |
TCP / HTTP | Default Gunicorn local TCP listener. |
unix:/run/gunicorn/gunicorn.sock |
UNIX Socket | Recommended local high-performance IPC connection. |
17. Further Reading and References
- Official Gunicorn Documentation
- Gunicorn Configuration Settings Reference
- Deploying Gunicorn in Production
- Django Documentation: How to Deploy with Gunicorn
- PEP 3333 – Python Web Server Gateway Interface Specification v1.0.1