1. Overview
Nginx (pronounced “engine-x”) is a high-performance, asynchronous HTTP server, reverse proxy, and Layer 4/7 load balancer. Originally written by Igor Sysoev to address the C10K problem (handling 10,000 concurrent client connections on a single server), Nginx achieves extreme scalability by discarding the traditional process-per-request or thread-per-request model in favor of an event-driven, non-blocking architecture.
Primary Purpose and Use Cases
In modern Linux production environments, Nginx acts as the front-line gateway for network traffic. It handles high volumes of concurrent HTTP/HTTPS connections and efficiently distributes requests. Common production use cases include:
- Static Asset Delivery: Serving static files (HTML, CSS, JavaScript, media) directly from disk with minimal memory overhead and kernel-level optimizations like
sendfile. - Reverse Proxying: Terminating client connections and forwarding requests to upstream application servers such as Gunicorn (Python), uWSGI, Node.js, PHP-FPM, or Puma (Ruby).
- Layer 7 Load Balancing: Distributing incoming HTTP traffic across pools of backend servers using algorithms like round-robin, IP hash, or least connections.
- TLS/SSL Termination: Offloading CPU-intensive TLS handshakes, certificate management, and SNI routing from backend application servers.
- Layer 4 TCP/UDP Proxying: Proxying non-HTTP protocols (e.g., database connections, LDAP, custom TCP/UDP services) via the
streammodule context. - Edge Offloading: Handling gzip compression, request rate limiting, response caching, and client header sanitization before requests reach application code.
Key Advantages and Limitations
| Architectural Dimension | Nginx | Traditional Threaded Servers (e.g., Apache mpm_prefork) |
|---|---|---|
| Concurrency Model | Asynchronous, non-blocking event loop handled by fixed worker processes. | Synchronous process or thread spawned per connection. |
| Resource Footprint | Extremely low memory footprint; RAM usage remains predictable under thousands of concurrent connections. | Memory consumption scales linearly with active connections, risking OOM thrashing. |
| Configuration Model | Plain-text declarative block syntax parsed into memory at startup/reload. | Text-based configuration files; historically supported per-directory .htaccess overrides. |
| Extensibility | Compiled C modules or dynamically loaded C/Lua/NJS scripts. | Dynamic module loading at runtime with broad directory-level runtime overrides. |
Limitations: Nginx worker processes do not parse per-directory runtime configuration files (like Apache .htaccess). All configuration changes require modifying server configuration files and triggering a graceful configuration reload. Furthermore, CPU-bound tasks within Nginx (such as heavy inline transformation scripts or unbuffered disk I/O) can block an entire worker process event loop if not properly offloaded to thread pools.
Practical Example
Consider a web application receiving 5,000 requests per second. Using Nginx as a reverse proxy, Nginx handles all client TCP connections, terminates TLS certificates, serves static images directly from disk, caches common API responses, and routes dynamic requests over persistent HTTP/1.1 or Unix domain sockets to a backend pool of Gunicorn application processes.
Client Requests (HTTPS) ──► [ Nginx Frontend ] ──┬──► Static Assets (/var/www/static)
├──► Microservice A (Unix Socket)
└──► Upstream App Pool (TCP :8000)
2. Architecture and Mental Model
To diagnose production issues effectively, an SRE or SysAdmin must understand Nginx’s process hierarchy, event loop mechanics, and request handling lifecycle.
Process Hierarchy: Master and Worker Architecture
Nginx runs as a multi-process architecture consisting of a single Master Process and multiple Worker Processes:
┌─────────────────────────┐
│ Master Process │
│ (Runs as root/UID 0) │
└───────────┬─────────────┘
│
┌─────────────────────────┼─────────────────────────┐
▼ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Worker Process │ │ Worker Process │ │ Worker Process │
│(www-data / nginx)│ │(www-data / nginx)│ │(www-data / nginx)│
└─────────────────┘ └─────────────────┘ └─────────────────┘
- Master Process: Runs with elevated privileges (typically
root). Its sole responsibilities are reading and validating configuration files, binding to privileged ports (such as 80 and 443), managing worker processes, and processing signals (such asHUPfor reload orQUITfor graceful shutdown). The master process does not process client requests directly. - Worker Processes: Run unprivileged as a non-root system user (such as
www-dataon Debian/Ubuntu ornginxon RHEL/Rocky). Workers handle network connections, read files from disk, execute proxy operations, and write log entries. Each worker process operates a single-threaded non-blocking event loop (using Linuxepoll) capable of handling thousands of multiplexed connections simultaneously. - Cache Manager & Loader (Optional): Auxiliary background processes spawned when HTTP caching (
proxy_cache_path) is configured to manage disk cache expiration and index cache keys into shared memory.
Complete Request Lifecycle Flow
sequenceDiagram
autonumber
actor Client
participant Master as Master Process (root)
participant Worker as Worker Process (www-data)
participant Kernel as OS Kernel (epoll)
participant Backend as Upstream Backend
Note over Master: Binds ports 80/443 & spawns Workers
Client->>Kernel: TCP SYN Connection Request
Kernel->>Worker: epoll notification (Accept connection)
Worker->>Client: TCP 3-Way Handshake completed
opt TLS Connection
Client->>Worker: TLS Client Hello (SNI)
Worker->>Client: Selects Cert & completes TLS Handshake
end
Client->>Worker: HTTP Request Header
Note over Worker: Server Block Match (listen + server_name)
Note over Worker: Location Block Match (Exact -> Regex -> Prefix)
alt Static File Handler
Worker->>Worker: Read filesystem (root / alias / try_files)
else Proxy Handler
Worker->>Backend: Forward Request (proxy_pass over TCP/Socket)
Backend-->>Worker: HTTP Response Body
end
Worker->>Client: Send HTTP Response Body
Worker->>Worker: Write access_log & error_log
- TCP Connection Acceptance: The Linux kernel accepts the TCP connection and notifies the worker processes via an
epollevent. One worker accepts the file descriptor. - TLS Handshake & SNI: If HTTPS is used, the worker negotiates TLS. It inspects the Server Name Indication (SNI) extension in the Client Hello to select the appropriate TLS certificate before parsing full HTTP request headers.
- Server Block Matching: Nginx matches the request to a specific
server { }block by comparing the socket IP/port (listen) and the HTTPHostheader (or SNI) against definedserver_namedirectives. - Location Block Matching: Once a server block is chosen, Nginx evaluates the request URI against all defined
locationdirectives inside that server block. - Handler Execution: The chosen location’s handler executes:
- Static file handler checks paths using
root,alias, ortry_files. - Reverse proxy handler transforms headers and proxies traffic to an
upstreampool usingproxy_pass. - Redirect handler generates an immediate HTTP response using
returnorrewrite.
- Logging & Connection Cleanup: The worker sends the response payload, logs entry details to
access_log, logs errors toerror_log, and keeps the connection open (HTTP Keep-Alive) or closes the socket.
3. Core Concepts and Terminology
Understanding Nginx syntax and semantics requires mastering its core building blocks:
Directives and Context Hierarchy
Nginx configuration is organized hierarchically into nested contexts using braces {}. Directives outside any block exist in the main (global) context.
maincontext: Global settings (e.g.,user,worker_processes,error_log,pid).eventscontext: Connection-handling settings (e.g.,worker_connections,use epoll).httpcontext: Web and HTTP proxy rules (e.g.,include,sendfile,keepalive_timeout).servercontext: Virtual host definitions configured withinhttporstreamcontexts.locationcontext: URI path routing rules configured inside an HTTPserverblock.upstreamcontext: Pools of backend servers used for load balancing configured insidehttporstream.streamcontext: Top-level context for Layer 4 TCP/UDP proxying, parallel tohttp.
Location Matching Algorithm and Priorities
Nginx matches request URIs against location blocks using strict rules. Priority is not strictly determined by the order of appearance in the configuration file; specific modifier syntax overrides block placement.
The evaluation sequence occurs as follows:
- Exact Match (
=): Matches the URI exactly. Processing stops immediately if matched. - Preferential Prefix (
^~): If the longest matching prefix location has^~, Nginx stops searching for regular expressions. - Standard Prefix (no modifier): Nginx searches all prefix locations and remembers the longest match.
- Regular Expression (
~case-sensitive,~*case-insensitive): Nginx checks regular expressions in the order they appear in the config file. The first matching regex wins. - Fallback: If no regex matches, Nginx uses the longest prefix match remembered from step 3.
| Modifier | Match Type | Priority | Description / Example |
|---|---|---|---|
= |
Exact match | 1 (Highest) | location = /favicon.ico — Matches /favicon.ico only. |
^~ |
Preferential prefix match | 2 | location ^~ /static/ — Longest prefix match; skips regex search. |
~ |
Case-sensitive regex | 3 | location ~ \.php$ — Matches URIs ending in .php. |
~* |
Case-insensitive regex | 3 | `location ~* .(jpg |
| (None) | Standard prefix match | 4 (Lowest) | location /api/ — Matches any URI starting with /api/. |
Operational Warning: A broad regex block defined early in a configuration file can hijack requests intended for prefix locations located further down. Always test URI resolution logic thoroughly.
Server Block Selection (server_name Logic)
When an HTTP request arrives, Nginx determines the correct server block using the following order of precedence based on the Host header:
- Exact hostname match (e.g.,
server_name api.example.com;). - Longest wildcard match starting with an asterisk (e.g.,
server_name *.example.com;). - Longest wildcard match ending with an asterisk (e.g.,
server_name example.*;). - First matching regular expression in order of configuration appearance (e.g.,
server_name ~^api\.;). - The
default_serverdefined for thatlistenport. If nodefault_serverflag is explicitly set, the firstserverblock parsed for that port becomes the default fallback.
4. Installation and Version Management
Package Repositories and Installation
Linux distributions provide Nginx via default repositories, but official Nginx repositories ensure access to supported stable and mainline releases.
Debian / Ubuntu Installation
# Install prerequisites
sudo apt-get update
sudo apt-get install -y curl gnupg2 ca-certificates lsb-release debian-archive-keyring
# Import official Nginx signing key
curl -fsSL https://nginx.org/keys/nginx_signing.key | sudo gpg --dearmor -o /usr/share/keyrings/nginx-archive-keyring.gpg
# Add official APT repository (Ubuntu example)
echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] http://nginx.org/packages/ubuntu $(lsb_release -cs) nginx" | sudo tee /etc/apt/sources.list.d/nginx.list
# Install Nginx
sudo apt-get update
sudo apt-get install -y nginx
RHEL / Rocky Linux / AlmaLinux Installation
# Install yum-utils
sudo dnf install -y yum-utils
# Create repo file
cat << 'EOF' | sudo tee /etc/yum.repos.d/nginx.repo
[nginx-stable]
name=nginx static repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
EOF
# Install Nginx
sudo dnf install -y nginx
Identifying Installed Version and Modules
To inspect version details, compile flags, and enabled modules, use the -v and -V flags:
# Print version only
nginx -v
# Output:
# nginx version: nginx/1.30.6
# Print version, compiler details, and configure arguments (modules)
nginx -V
Versioning Strategy: Mainline vs. Stable
Nginx maintains two active release branches:
- Mainline Branch: Receives new features, performance improvements, and bug fixes. Recommended for general production deployment to access recent protocol enhancements.
- Stable Branch: Receives critical security fixes and major bug fixes only; no new feature additions.
5. Configuration Files and Directory Structure
Configuration Hierarchy Reference
| File / Directory | Purpose | Important Details |
|---|---|---|
/etc/nginx/nginx.conf |
Global primary configuration file. | Sets master process parameters, worker settings, global HTTP directives, and includes secondary files. |
/etc/nginx/conf.d/*.conf |
Distribution standard configuration directory (RHEL/CentOS/Rocky & Official Nginx builds). | Files ending in .conf are automatically included into the main http context. |
/etc/nginx/sites-available/ |
Storage for virtual host configurations (Debian/Ubuntu convention). | Contains configuration files for sites that are available but not necessarily active. |
/etc/nginx/sites-enabled/ |
Active virtual host directory (Debian/Ubuntu convention). | Contains symbolic links pointing back to /etc/nginx/sites-available/. |
/etc/nginx/snippets/ |
Modular shared configuration fragments. | Stores reusable snippets like SSL options or proxy headers included across multiple sites. |
/var/log/nginx/access.log |
Default HTTP request log file. | Records details for all incoming HTTP requests. |
/var/log/nginx/error.log |
Default operational and diagnostic error log file. | Records startup failures, worker errors, application proxy failures, and crash dumps. |
/var/www/html/ or /usr/share/nginx/html/ |
Default document root directories. | Stores default static files served by placeholder site blocks. |
/run/nginx.pid or /var/run/nginx.pid |
Process ID file. | Contains the numeric PID of the running master process. |
Main nginx.conf Structure Example
Below is an operational, annotated /etc/nginx/nginx.conf file:
# User and process settings
user www-data; # Worker process user identity
worker_processes auto; # Auto-detect available CPU cores
pid /run/nginx.pid; # PID file location
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 2048; # Max connections per worker process
use epoll; # Efficient Linux event notification mechanism
multi_accept on; # Accept all new connections at once
}
http {
# Basic I/O & Performance directives
sendfile on; # Use kernel zero-copy sendfile()
tcp_nopush on; # Send headers in one TCP packet
tcp_nodelay on; # Disable Nagle's algorithm for low latency
keepalive_timeout 65; # HTTP Keep-Alive client timeout
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Global security parameters
server_tokens off; # Hide Nginx version from server headers
client_max_body_size 16M; # Maximum allowed client upload payload
# Logging Formats and Paths
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" '
'rt=$request_time uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log warn;
# Include modular virtual host configurations
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
# Stream context for Layer 4 TCP/UDP proxying
stream {
include /etc/nginx/stream.d/*.conf;
}
6. Service Management and Common Operations
Managing the Nginx daemon correctly prevents unintended site downtime during configuration changes.
Service Control Commands
| Task | Command | Description & Behaviour |
|---|---|---|
| Check Status | systemctl status nginx |
Displays active service state, main PID, memory usage, and recent systemd log entries. |
| Validate Configuration | sudo nginx -t |
Tests configuration syntax and checks that referenced path files exist. |
| Dump Effective Config | sudo nginx -T |
Validates configuration and prints the entire merged configuration to stdout. |
| Reload Service | sudo systemctl reload nginx |
Re-reads configuration gracefully without dropping active client connections. |
| Restart Service | sudo systemctl restart nginx |
Stops master and worker processes completely and restarts them; causes brief connection drop. |
| Enable on Boot | sudo systemctl enable nginx |
Configures systemd to start Nginx automatically during system startup. |
Graceful Reload Mechanics vs. Full Restart
- Graceful Reload (
systemctl reload nginxornginx -s reload):
- Master process re-parses and validates the configuration file.
- If valid, the master process spawns a new set of worker processes with the new configuration.
- Master sends a
QUITsignal to old worker processes, instructing them to stop accepting new connections while completing existing requests. - Once existing connections finish, old workers terminate. Zero downtime is achieved.
- Full Restart (
systemctl restart nginx): The master process immediately terminates all worker processes and stops listener sockets. Active HTTP connections are interrupted before the daemon re-initializes.
Operational Rule: Always execute
nginx -tbefore executingsystemctl reload nginx. If a syntax error is present, runningnginx -tcatches the error immediately. Runningsystemctl reload nginxwith invalid syntax will fail, and while running workers remain active, automated deployment scripts may halt or fail unexpectedly.
7. Common Administrative Tasks and Workflows
Task 1: Enabling and Disabling Virtual Hosts (Debian/Ubuntu Model)
- Objective: Safely publish a new site configuration using symbolic links.
- Step 1: Create configuration file in
/etc/nginx/sites-available/app.example.com.conf. - Step 2: Enable site by creating a symlink:
sudo ln -s /etc/nginx/sites-available/app.example.com.conf /etc/nginx/sites-enabled/
- Step 3: Validate and reload:
sudo nginx -t && sudo systemctl reload nginx
- Step 4: Disable site (when retiring):
sudo rm /etc/nginx/sites-enabled/app.example.com.conf
sudo nginx -t && sudo systemctl reload nginx
(Removing the symlink in sites-enabled/ leaves the original configuration file intact in sites-available/).
Task 2: Configuring an HTTP Reverse Proxy with Custom Headers
- Objective: Pass HTTP traffic to a Python Gunicorn app listening on
127.0.0.1:8000while retaining original client connection details.
# /etc/nginx/conf.d/app_proxy.conf
upstream gunicorn_backend {
server 127.0.0.1:8000 max_fails=3 fail_timeout=10s;
keepalive 32; # Maintain pool of persistent connections to backend
}
server {
listen 80;
server_name app.example.com;
location / {
proxy_pass http://gunicorn_backend;
# Preserve client headers for backend logging
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Enable HTTP/1.1 for keepalive upstream connections
proxy_http_version 1.1;
proxy_set_header Connection "";
# Timeouts
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
}
}
Task 3: Deploying a Single Page Application (SPA) Fallback Route
- Objective: Serve static assets directly and fallback all client-side routing paths (e.g.,
/dashboard,/settings) to/index.html.
server {
listen 80;
server_name spa.example.com;
root /var/www/spa-app;
index index.html;
location / {
# Check literal URI file, directory, or fall back to /index.html
try_files $uri $uri/ /index.html;
}
# Static media caching
location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg)$ {
expires 30d;
add_header Cache-Control "public, no-transform";
}
}
Task 4: Configuring Layer 4 TCP Load Balancing (Stream Context)
- Objective: Load balance raw TCP connections across two MySQL/MariaDB database read-replicas.
# Add to /etc/nginx/nginx.conf or /etc/nginx/stream.d/db_proxy.conf
stream {
upstream mysql_read_pool {
server 192.168.1.50:3306 max_fails=2 fail_timeout=5s;
server 192.168.1.51:3306 max_fails=2 fail_timeout=5s;
}
server {
listen 3306;
proxy_pass mysql_read_pool;
proxy_connect_timeout 5s;
proxy_timeout 1m;
}
}
8. CLI Command Reference and Cheatsheet
Process & Service Controls
| Command | Purpose | Flags / Arguments | Practical Example | Explanation |
|---|---|---|---|---|
nginx |
Direct executable invoker. | -t (test), -T (dump), -s (signal), -c (config path). |
sudo nginx -t |
Tests all active configuration files for syntax and structural errors. |
nginx |
Send process signal. | -s stop, -s quit, -s reload, -s reopen. |
sudo nginx -s reload |
Signals master process to reload configuration gracefully. |
nginx |
Specify custom config file. | -c <file_path> |
sudo nginx -c /tmp/test_nginx.conf -t |
Validates a specific configuration file without changing production. |
systemctl |
Manage systemd service. | status, start, stop, restart, reload. |
sudo systemctl status nginx |
Checks active execution status and runtime logs of Nginx service. |
Diagnostic & Inspection Commands
| Command | Purpose | Flags / Arguments | Practical Example | Explanation |
|---|---|---|---|---|
nginx -T |
Dump effective config. | -T |
`sudo nginx -T | grep -n ‘server_name’` |
ss |
Socket Statistics. | -t (tcp), -l (listen), -n (numeric), -p (processes). |
`sudo ss -tlnp | grep nginx` |
ps |
Process Status. | aux |
`ps aux | grep nginx` |
curl |
HTTP Client inspection. | -v (verbose), -I (headers only), -H (custom header). |
curl -vI -H "Host: api.example.com" [http://127.0.0.1/](http://127.0.0.1/) |
Bypasses local DNS resolution to test specific server_name blocks directly against localhost. |
openssl |
TLS/SSL diagnostics. | s_client -connect -servername |
openssl s_client -connect example.com:443 -servername example.com |
Tests TLS handshake, verifies certificate chains, and checks expiration dates. |
9. Logs, Monitoring and Diagnostics
Log File Locations and Formats
Nginx produces two primary categories of logs:
- Access Logs (
/var/log/nginx/access.log): Records details of every HTTP request handled by the server. - Error Logs (
/var/log/nginx/error.log): Records operational events, syntax errors, proxy backend connection failures, and file permission denied errors.
Configuring Diagnostic Upstream Log Formats
Standard access logs omit upstream backend latency metrics. Augmenting the log_format directive in nginx.conf provides visibility into backend performance:
log_format debug_upstream '$remote_addr - $remote_user [$time_local] "$request" '
'status=$status body_bytes=$body_bytes_sent '
'request_time=$request_time '
'upstream_addr="$upstream_addr" '
'upstream_status="$upstream_status" '
'upstream_response_time="$upstream_response_time" '
'upstream_connect_time="$upstream_connect_time"';
Real-Time Monitoring and Diagnostic Workflows
# Follow live error log output
sudo tail -f /var/log/nginx/error.log
# Follow live access log output with filtering for HTTP 5xx errors
sudo tail -f /var/log/nginx/access.log | grep --line-buffered "HTTP/1.1\" 5"
# Inspect Nginx systemd journal logs since last boot
sudo journalctl -u nginx -b --no-pager -n 100
# Analyze top requested URIs from access log
sudo awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -n 10
10. Troubleshooting Guide and Common Failure Scenarios
Problem 1: Nginx Service Fails to Start or Reload
Symptoms
Running systemctl reload nginx or systemctl start nginx returns a non-zero exit code: Job for nginx.service failed. Active requests fail or configuration updates are ignored.
Possible Causes
- Syntax errors in configuration files (missing semicolons, unmatched braces).
- Duplicate
listendirectives binding the same IP/port combination twice. - Invalid path references inside
includedirectives. - Unprivileged user attempting to bind to ports < 1024.
Investigation and Diagnosis
- Objective: Locate the exact syntax error file and line number.
- Command:
sudo nginx -t - Output Interpretation: If errors exist, Nginx outputs the exact file path and line number:
nginx: [emerg] invalid number of arguments in "listen" directive in /etc/nginx/conf.d/app.conf:3
nginx: configuration file /etc/nginx/nginx.conf test failed
- Objective: Inspect systemd failure reason if
nginx -tpasses.
- Command:
sudo journalctl -u nginx -n 50 --no-pager - Output Interpretation: Look for OS-level bind errors or file lock issues.
Root Cause Identification
Confirm whether the issue stems from configuration syntax errors, missing include files, or socket binding conflicts.
Resolution
- Correct the syntax error on the indicated file and line number.
- Remove duplicate
listendirectives. - Ensure referenced directories exist (
sudo mkdir -p /var/log/nginx/custom).
Verification
Execute sudo nginx -t. Ensure it returns: syntax is ok and test is successful. Then run sudo systemctl reload nginx.
Prevention
Integrate nginx -t into CI/CD deployment pipelines prior to copying configuration files into production.
Problem 2: Reload Succeeded but Configuration Behavior Changed Unexpectedly
Symptoms
nginx -t and systemctl reload nginx succeed, but the website continues to serve old configuration responses or wrong routing rules.
Possible Causes
- Modifying a configuration file that is not included in the active
nginx.conf. - Running multiple conflicting Nginx master processes.
- Testing against a container or host whose config mounts point elsewhere.
- Downstream browser caching or CDN edge caching.
Investigation and Diagnosis
- Objective: Inspect the effective merged configuration loaded in memory.
- Command:
sudo nginx -T | grep -n 'your-directive' - Output Interpretation: If your directive does not appear in the output of
nginx -T, your edited file is not included in the master config path.
- Objective: Verify which binary and configuration file the active master process is using.
- Command:
ps aux | grep nginx - Output Interpretation: Ensure the master process PID uses
/usr/sbin/nginx -c /etc/nginx/nginx.conf.
Root Cause Identification
The active Nginx instance is either executing a different configuration file tree or your changes are overridden by another included server block.
Resolution
- Ensure the target configuration file is symlinked properly from
sites-availabletosites-enabled(Debian/Ubuntu) or placed in/etc/nginx/conf.d/(RHEL/Rocky). - Terminate rogue duplicate Nginx processes (
sudo pkill -9 nginx && sudo systemctl start nginx).
Verification
Test locally using curl:
curl -vI -H "Host: actual.domain.com" http://127.0.0.1/
Prevention
Use nginx -T to verify merged settings during deployment audits.
Problem 3: 403 Forbidden Errors and Worker Process Permissions
Symptoms
Clients receive HTTP 403 Forbidden. The Nginx error log contains: (13: Permission denied) while reading upstream or stat() "/var/www/app/index.html" failed (13: Permission denied).
Possible Causes
- The directory path or target file lacks read/execute permissions for the unprivileged Nginx worker user (
www-dataornginx). - Missing
indexfile whenautoindexis set tooff. - SELinux or AppArmor blocking Nginx worker access.
Investigation and Diagnosis
- Objective: Identify the active worker user identity.
- Command:
grep -E '^(user|group)' /etc/nginx/nginx.conf
- Objective: Test file traversal and read permissions as the unprivileged worker user.
- Command:
sudo -u www-data test -r /var/www/app/index.html && echo "READ OK" - Output Interpretation: If it returns nothing or permission denied, the worker user cannot read the file or traverse parent directories.
Crucial SRE Insight: Testing permissions as
root(e.g.cat /var/www/app/index.html) will succeed even when Nginx workers fail with 403. Always test permissions usingsudo -u <worker_user>.
- Objective: Inspect SELinux denial audits (on RHEL/Rocky Linux).
- Command:
sudo ausearch -m avc -ts recentorsudo grep nginx /var/log/audit/audit.log
Root Cause Identification
Permissions are restricted on one of the parent path directories, or SELinux context labels are incorrect.
Resolution
- Grant execute permissions on all parent directories down to the file path:
sudo chmod +x /var /var/www /var/www/app
sudo chmod 644 /var/www/app/index.html
- Fix SELinux web content contexts:
sudo restorecon -Rv /var/www/app
sudo setsebool -P httpd_can_network_connect 1
Verification
Execute:
sudo -u www-data ls -la /var/www/app/
curl -I http://localhost/index.html
Problem 4: 502 Bad Gateway / 504 Gateway Timeout Errors
Symptoms
Clients receive HTTP 502 (Bad Gateway) or HTTP 504 (Gateway Timeout). Nginx error.log reports connect() failed (111: Connection refused), no live upstreams, or upstream timed out.
Possible Causes
- The upstream backend service (Gunicorn, PHP-FPM, Node.js) is stopped or crashing.
- Upstream app is bound to
127.0.0.1:8000but Nginx is configured for a Unix socket or wrong IP/port. - Upstream backend queue is saturated, exceeding Nginx connection/read timeouts.
Investigation and Diagnosis
- Objective: Check if the backend application process is listening.
- Command:
sudo ss -tlnp | grep 8000
- Objective: Test the backend directly from the Nginx host bypassing Nginx.
- Command:
curl -v [http://127.0.0.1:8000/health](http://127.0.0.1:8000/health) - Output Interpretation: If connection fails with
Connection refused, the upstream service is down.
- Objective: Inspect Nginx error log for specific proxy errors.
- Command:
sudo tail -n 20 /var/log/nginx/error.log
Resolution
- Restart upstream backend service (
sudo systemctl restart gunicorn). - If
504 Gateway Timeoutoccurs during heavy queries, increase upstream timeouts innginx.conf:
proxy_connect_timeout 90s;
proxy_read_timeout 90s;
proxy_send_timeout 90s;
Verification
Run curl -v [http://127.0.0.1:8000/](http://127.0.0.1:8000/) followed by an end-to-end HTTPS request.
Problem 5: Unix Domain Socket Permission & Path Failures
Symptoms
HTTP 502 Bad Gateway. Nginx error.log reports connect() to unix:/run/app.sock failed (2: No such file or directory) or (13: Permission denied) while connecting to upstream.
Possible Causes
- Unix socket file does not exist because the upstream daemon failed to create it.
- Socket file permissions (e.g.,
0600) restrict access to the socket owner, blocking the Nginx worker user.
Investigation and Diagnosis
- Objective: Check socket file existence and permission flags.
- Command:
ls -la /run/app.sock - Output Interpretation: Verify permissions show
srw-rw----and group ownership matches the Nginx worker user group (e.g.,www-data).
Resolution
- Align upstream socket owner settings (e.g., in Gunicorn or PHP-FPM pool configuration):
# Example PHP-FPM www.conf / Gunicorn config
listen.owner = www-data
listen.group = www-data
listen.mode = 0660
- Restart backend service to recreate socket file.
Verification
Run sudo -u www-data test -w /run/app.sock && echo "SOCKET WRITABLE".
Problem 6: Root vs. Alias Directive Confusion
Symptoms
Static asset requests return HTTP 404 Not Found. The paths shown in error.log contain duplicate directory segments (e.g., /data/images/images/logo.png).
Root Cause Analysis & Behavioral Rules
rootdirective: Appends the full request URI to the root path.
location /images/ {
root /data;
}
# Request GET /images/logo.png -> Resolves to /data/images/logo.png
aliasdirective: Replaces the matched location URI path with the alias path.
location /images/ {
alias /data/;
}
# Request GET /images/logo.png -> Resolves to /data/logo.png
Operational Warning: When using
aliasinside a location ending with a trailing slash (/images/), thealiasdirective MUST also end with a trailing slash (/data/). Trailing slash mismatches cause malformed path resolution.
Resolution
Adjust root or alias directives based on target directory structure:
# Correct use of alias
location /assets/ {
alias /var/www/static-files/;
}
Problem 7: Wrong Virtual Host or Default Server Served
Symptoms
Requests for [https://api.example.com](https://api.example.com) serve content from a completely different site hosted on the same server.
Possible Causes
- No
server_namedirective matchesapi.example.com. - Request falls through to the default server for port 80/443.
- Broad regex
server_namematches before exact host rules.
Investigation and Diagnosis
- Objective: Identify all configured
listensockets andserver_namedirectives.
- Command:
sudo nginx -T | grep -E 'server_name|listen|default_server'
- Objective: Simulate Host header requests against localhost.
- Command:
curl -v -H "Host: api.example.com" [http://127.0.0.1/](http://127.0.0.1/)
Resolution
- Add explicit
server_namedirectives. - Designate an explicit catch-all fallback block to drop unhandled hosts:
server {
listen 80 default_server;
listen 443 ssl default_server;
ssl_certificate /etc/ssl/certs/fallback.crt;
ssl_certificate_key /etc/ssl/private/fallback.key;
server_name _;
return 444; # Special Nginx status code: Close connection without response
}
Problem 8: 413 Request Entity Too Large
Symptoms
Users attempting file uploads receive HTTP 413 Request Entity Too Large.
Possible Causes
Client request body size exceeds the limit defined by client_max_body_size (default is 1 Megabyte).
Resolution
Increase client_max_body_size in the http, server, or location block:
http {
client_max_body_size 64M;
}
Validate and reload: sudo nginx -t && sudo systemctl reload nginx.
Problem 9: Incomplete TLS/SSL Certificate Chain Warnings
Symptoms
Modern browsers show NET::ERR_CERT_AUTHORITY_INVALID or mobile clients fail TLS verification, even though basic SSL tests pass.
Possible Causes
The ssl_certificate file contains only the server leaf certificate, omitting intermediate Certificate Authority (CA) certificates.
Investigation and Diagnosis
- Objective: Inspect full certificate chain validation.
- Command:
openssl s_client -connect example.com:443 -servername example.com - Output Interpretation: Look for
Verify return code: 21 (unable to verify the first certificate).
Resolution
Concatenate your leaf certificate and intermediate CA certificates into a single full-chain file:
cat domain_leaf.crt intermediate.crt root.crt > /etc/ssl/certs/fullchain.crt
Update configuration:
ssl_certificate /etc/ssl/certs/fullchain.crt;
ssl_certificate_key /etc/ssl/private/domain.key;
11. Security and Best Practices
Hardening Guidelines
- Disable Server Identity Tokens: Suppress Nginx version disclosure in HTTP header outputs.
server_tokens off;
- Restrict Unprivileged User Identity: Ensure Nginx worker processes run under a dedicated, unprivileged system account (
www-dataornginx) with no shell access (/nologin). - Configure Secure TLS Settings: Enforce TLS 1.2 and TLS 1.3 while disabling outdated SSLv3, TLS 1.0, and TLS 1.1 protocols.
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
- Implement Rate Limiting: Prevent HTTP request flood attacks and brute-force attempts using
limit_req_zone.
# Define rate limit zone in http context
limit_req_zone $binary_remote_addr zone=login_limit:10m rate=5r/s;
server {
location /login {
limit_req zone=login_limit burst=10 nodelay;
proxy_pass http://backend;
}
}
- Set Security Response Headers: Inject defensive HTTP headers to mitigate cross-site scripting (XSS) and clickjacking.
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Content-Security-Policy "default-src 'self' http: https:;" always;
12. Performance and Resource Optimisation
Tuning File Descriptors and Connections
Linux kernels limit maximum open file descriptors per process by default. Because Nginx handles connections as file descriptors, tune worker_rlimit_nofile and connection limits:
# /etc/nginx/nginx.conf
# Increase worker open file descriptor limit
worker_rlimit_nofile 65535;
events {
worker_connections 8192;
use epoll;
}
Ensure the kernel fs.file-max setting supports process-level limits:
sysctl -w fs.file-max=2097152
High-Performance File Delivery Settings
Enable sendfile and TCP packet optimizations to stream static files directly from kernel buffer cache to network sockets without copying data into application memory space:
http {
sendfile on;
tcp_nopush on; # Works with sendfile to send HTTP header and file start in one packet
tcp_nodelay on; # Disables Nagle algorithm for real-time interactive data
# Gzip Compression Tuning
gzip on;
gzip_comp_level 5;
gzip_min_length 256;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml+rss;
}
13. Backup, Recovery and Maintenance
Backup Strategy
Maintain backups of configuration directories and custom certificates:
# Archive configuration files and certificates
sudo tar -czvf /backups/nginx_config_$(date +%F).tar.gz /etc/nginx /etc/ssl
Version Upgrade and Rollback Workflow
sequenceDiagram
autonumber
actor Admin as SRE / SysAdmin
participant Package as Package Manager (apt/dnf)
participant Nginx as Nginx Master Process
Admin->>Admin: Create config backup tarball
Admin->>Package: Upgrade nginx package
Package->>Nginx: Replaces binary on disk
Admin->>Nginx: Send USR2 signal (Spawn new master & workers)
Note over Nginx: New Master runs alongside Old Master
Admin->>Nginx: Send WINCH signal (Gracefully stop old workers)
alt Verification Success
Admin->>Nginx: Send QUIT signal (Terminate old Master)
else Verification Failure / Rollback
Admin->>Nginx: Send HUP signal to Old Master (Restore original workers)
Admin->>Nginx: Send TERM signal to New Master
end
Executing Zero-Downtime Binary Upgrades (Manual Method)
- Replace binary via package manager (
apt-get install --only-upgrade nginx). - Send
USR2signal to master process to spawn new master and new workers using the new binary:
sudo kill -USR2 $(cat /run/nginx.pid)
- Send
WINCHsignal to old master to gracefully shut down old workers:
sudo kill -WINCH $(cat /run/nginx.pid.oldbin)
- Verify application health. If successful, send
QUITto old master:
sudo kill -QUIT $(cat /run/nginx.pid.oldbin)
14. Automation and Integration
Ansible Configuration Management Automation
Below is a production Ansible task snippet to deploy and validate Nginx:
---
- name: Deploy and Manage Nginx Web Server
hosts: webservers
become: true
tasks:
- name: Install Nginx package
ansible.builtin.package:
name: nginx
state: present
- name: Copy Nginx primary configuration
ansible.builtin.template:
src: templates/nginx.conf.j2
dest: /etc/nginx/nginx.conf
owner: root
group: root
mode: '0644'
notify: Validate and Reload Nginx
handlers:
- name: Validate and Reload Nginx
ansible.builtin.command: nginx -t
changed_when: false
failed_when: false
notify: Reload Systemctl Nginx
- name: Reload Systemctl Nginx
ansible.builtin.systemd:
name: nginx
state: reloaded
15. Practical Labs and Exercises
Exercise 1 (Beginner): Deploying a Multi-Site Web Host
- Objective: Configure two separate websites (
site-a.localandsite-b.local) on a single Linux server listening on port 80. - Environment Setup:
- Create directories:
/var/www/site-aand/var/www/site-b. - Populate index files:
echo "Site A" > /var/www/site-a/index.htmlandecho "Site B" > /var/www/site-b/index.html.
- Instructions:
- Create
/etc/nginx/conf.d/site-a.confwithserver_name site-a.local;pointing root to/var/www/site-a. - Create
/etc/nginx/conf.d/site-b.confwithserver_name site-b.local;pointing root to/var/www/site-b. - Validate syntax (
nginx -t) and reload Nginx (systemctl reload nginx).
- Validation:
curl -H "Host: site-a.local" http://127.0.0.1/ # Expected output: Site A
curl -H "Host: site-b.local" http://127.0.0.1/ # Expected output: Site B
Exercise 2 (Troubleshooting): Investigating Permission Denied (403) Failures
- Objective: Diagnose and resolve an intentionally broken file permission state.
- Environment Setup:
sudo mkdir -p /var/www/secure-app
echo "Secret App" | sudo tee /var/www/secure-app/index.html
sudo chmod 700 /var/www/secure-app # Restricted parent dir!
- Instructions:
- Create a server block pointing to
/var/www/secure-app. - Reload Nginx and request
http://localhost/. Observe the 403 Forbidden error. - Inspect
/var/log/nginx/error.logto confirm permission failure. - Test read access using
sudo -u www-data test -r /var/www/secure-app/index.html. - Fix directory traversal permissions (
sudo chmod 755 /var/www/secure-app).
- Validation:
Run
curl -I http://localhost/and verify HTTP 200 OK.
16. Quick Reference and Cheat Sheet
Important Configuration Files
| File / Directory | Purpose |
|---|---|
/etc/nginx/nginx.conf |
Global primary configuration file. |
/etc/nginx/conf.d/*.conf |
HTTP virtual host directory (RHEL/CentOS/Rocky & official packages). |
/etc/nginx/sites-available/ |
Available site configurations (Debian/Ubuntu). |
/etc/nginx/sites-enabled/ |
Active site symlinks (Debian/Ubuntu). |
/etc/nginx/mime.types |
File extension to MIME-type mappings. |
Important Log Locations
| Log Path | Purpose |
|---|---|
/var/log/nginx/access.log |
Records all processed HTTP client requests. |
/var/log/nginx/error.log |
Operational error log (syntax, worker, proxy errors). |
Common Commands
| Task | Command |
|---|---|
| Test configuration syntax | sudo nginx -t |
| Test and dump merged config | sudo nginx -T |
| Graceful reload | sudo systemctl reload nginx |
| Check active ports/listeners | `sudo ss -tlnp |
| Inspect active worker user | `ps aux |
| Test Host header routing locally | curl -vI -H "Host: example.com" [http://127.0.0.1/](http://127.0.0.1/) |
Troubleshooting Quick Reference
| Symptom | Initial Investigation Step |
|---|---|
| Nginx service won’t reload | Run sudo nginx -t to locate syntax errors. |
| Config changes ignored | Run sudo nginx -T to verify if file is included. |
| 403 Forbidden | Test path access using sudo -u www-data test -r <file>. |
| 502 Bad Gateway | Test upstream port using curl -v [http://127.0.0.1](http://127.0.0.1):<backend_port>. |
| 413 Entity Too Large | Increase client_max_body_size in HTTP/Server context. |
Important Ports and Protocols
| Port | Protocol | Purpose |
|---|---|---|
| 80 | TCP | Default HTTP unencrypted web traffic. |
| 443 | TCP | Default HTTPS TLS-encrypted web traffic. |
| 3306 / custom | TCP | Layer 4 stream proxying (databases/custom protocols). |
17. Further Reading and References
- Source 1: Nginx Guide — Concept, Architecture, Directives, and Core References (
Nginx guide).
- Official Documentation: nginx.org/en/docs
- Virtual Host / Server Names Reference: nginx.org/en/docs/http/server_names
- Reverse Proxy Module Guide: nginx.org/en/docs/http/ngx_http_proxy_module
- Beginner’s Configuration Guide: nginx.org/en/docs/beginners_guide
- Source 2: Nginx Operational Troubleshooting Guide (
Nginx troubleshooting). - Source 3: Nginx Command and Operations Cheatsheet (
Nginx cheatsheet).