Domain 7Web & Application LayerINTERMEDIATE

Nginx Master Handbook: Architecture, Operations, and Systematic Troubleshooting

A comprehensive guide to Nginx architecture, event-driven concurrency, core directives, HTTP and Layer 4 proxying, systematic troubleshooting, and operational best practices.

Domain: Domain 7: Web & Application Layer•Updated: 2026-11-03
#linux#sysadmin#nginx#webserver#sre

1. Overview

Nginx (pronounced “engine-x”) is a high-performance, asynchronous HTTP server, reverse proxy, and Layer 4/7 load balancer. Originally written by Igor Sysoev to address the C10K problem (handling 10,000 concurrent client connections on a single server), Nginx achieves extreme scalability by discarding the traditional process-per-request or thread-per-request model in favor of an event-driven, non-blocking architecture.

Primary Purpose and Use Cases

In modern Linux production environments, Nginx acts as the front-line gateway for network traffic. It handles high volumes of concurrent HTTP/HTTPS connections and efficiently distributes requests. Common production use cases include:

  • Static Asset Delivery: Serving static files (HTML, CSS, JavaScript, media) directly from disk with minimal memory overhead and kernel-level optimizations like sendfile.
  • Reverse Proxying: Terminating client connections and forwarding requests to upstream application servers such as Gunicorn (Python), uWSGI, Node.js, PHP-FPM, or Puma (Ruby).
  • Layer 7 Load Balancing: Distributing incoming HTTP traffic across pools of backend servers using algorithms like round-robin, IP hash, or least connections.
  • TLS/SSL Termination: Offloading CPU-intensive TLS handshakes, certificate management, and SNI routing from backend application servers.
  • Layer 4 TCP/UDP Proxying: Proxying non-HTTP protocols (e.g., database connections, LDAP, custom TCP/UDP services) via the stream module context.
  • Edge Offloading: Handling gzip compression, request rate limiting, response caching, and client header sanitization before requests reach application code.

Key Advantages and Limitations

Architectural Dimension Nginx Traditional Threaded Servers (e.g., Apache mpm_prefork)
Concurrency Model Asynchronous, non-blocking event loop handled by fixed worker processes. Synchronous process or thread spawned per connection.
Resource Footprint Extremely low memory footprint; RAM usage remains predictable under thousands of concurrent connections. Memory consumption scales linearly with active connections, risking OOM thrashing.
Configuration Model Plain-text declarative block syntax parsed into memory at startup/reload. Text-based configuration files; historically supported per-directory .htaccess overrides.
Extensibility Compiled C modules or dynamically loaded C/Lua/NJS scripts. Dynamic module loading at runtime with broad directory-level runtime overrides.

Limitations: Nginx worker processes do not parse per-directory runtime configuration files (like Apache .htaccess). All configuration changes require modifying server configuration files and triggering a graceful configuration reload. Furthermore, CPU-bound tasks within Nginx (such as heavy inline transformation scripts or unbuffered disk I/O) can block an entire worker process event loop if not properly offloaded to thread pools.

Practical Example

Consider a web application receiving 5,000 requests per second. Using Nginx as a reverse proxy, Nginx handles all client TCP connections, terminates TLS certificates, serves static images directly from disk, caches common API responses, and routes dynamic requests over persistent HTTP/1.1 or Unix domain sockets to a backend pool of Gunicorn application processes.

Client Requests (HTTPS) ──► [ Nginx Frontend ] ──┬──► Static Assets (/var/www/static)
                                                ├──► Microservice A (Unix Socket)
                                                └──► Upstream App Pool (TCP :8000)

2. Architecture and Mental Model

To diagnose production issues effectively, an SRE or SysAdmin must understand Nginx’s process hierarchy, event loop mechanics, and request handling lifecycle.

Process Hierarchy: Master and Worker Architecture

Nginx runs as a multi-process architecture consisting of a single Master Process and multiple Worker Processes:

                       ┌─────────────────────────┐
                       │     Master Process      │
                       │   (Runs as root/UID 0)  │
                       └───────────┬─────────────┘
                                   │
         ┌─────────────────────────┼─────────────────────────┐
         ▼                         ▼                         ▼
┌─────────────────┐       ┌─────────────────┐       ┌─────────────────┐
│ Worker Process  │       │ Worker Process  │       │ Worker Process  │
│(www-data / nginx)│       │(www-data / nginx)│       │(www-data / nginx)│
└─────────────────┘       └─────────────────┘       └─────────────────┘
  • Master Process: Runs with elevated privileges (typically root). Its sole responsibilities are reading and validating configuration files, binding to privileged ports (such as 80 and 443), managing worker processes, and processing signals (such as HUP for reload or QUIT for graceful shutdown). The master process does not process client requests directly.
  • Worker Processes: Run unprivileged as a non-root system user (such as www-data on Debian/Ubuntu or nginx on RHEL/Rocky). Workers handle network connections, read files from disk, execute proxy operations, and write log entries. Each worker process operates a single-threaded non-blocking event loop (using Linux epoll) capable of handling thousands of multiplexed connections simultaneously.
  • Cache Manager & Loader (Optional): Auxiliary background processes spawned when HTTP caching (proxy_cache_path) is configured to manage disk cache expiration and index cache keys into shared memory.

Complete Request Lifecycle Flow

sequenceDiagram
    autonumber
    actor Client
    participant Master as Master Process (root)
    participant Worker as Worker Process (www-data)
    participant Kernel as OS Kernel (epoll)
    participant Backend as Upstream Backend

    Note over Master: Binds ports 80/443 & spawns Workers
    Client->>Kernel: TCP SYN Connection Request
    Kernel->>Worker: epoll notification (Accept connection)
    Worker->>Client: TCP 3-Way Handshake completed
    opt TLS Connection
        Client->>Worker: TLS Client Hello (SNI)
        Worker->>Client: Selects Cert & completes TLS Handshake
    end
    Client->>Worker: HTTP Request Header
    Note over Worker: Server Block Match (listen + server_name)
    Note over Worker: Location Block Match (Exact -> Regex -> Prefix)
    alt Static File Handler
        Worker->>Worker: Read filesystem (root / alias / try_files)
    else Proxy Handler
        Worker->>Backend: Forward Request (proxy_pass over TCP/Socket)
        Backend-->>Worker: HTTP Response Body
    end
    Worker->>Client: Send HTTP Response Body
    Worker->>Worker: Write access_log & error_log
  1. TCP Connection Acceptance: The Linux kernel accepts the TCP connection and notifies the worker processes via an epoll event. One worker accepts the file descriptor.
  2. TLS Handshake & SNI: If HTTPS is used, the worker negotiates TLS. It inspects the Server Name Indication (SNI) extension in the Client Hello to select the appropriate TLS certificate before parsing full HTTP request headers.
  3. Server Block Matching: Nginx matches the request to a specific server { } block by comparing the socket IP/port (listen) and the HTTP Host header (or SNI) against defined server_name directives.
  4. Location Block Matching: Once a server block is chosen, Nginx evaluates the request URI against all defined location directives inside that server block.
  5. Handler Execution: The chosen location’s handler executes:
  • Static file handler checks paths using root, alias, or try_files.
  • Reverse proxy handler transforms headers and proxies traffic to an upstream pool using proxy_pass.
  • Redirect handler generates an immediate HTTP response using return or rewrite.
  1. Logging & Connection Cleanup: The worker sends the response payload, logs entry details to access_log, logs errors to error_log, and keeps the connection open (HTTP Keep-Alive) or closes the socket.

3. Core Concepts and Terminology

Understanding Nginx syntax and semantics requires mastering its core building blocks:

Directives and Context Hierarchy

Nginx configuration is organized hierarchically into nested contexts using braces {}. Directives outside any block exist in the main (global) context.

  • main context: Global settings (e.g., user, worker_processes, error_log, pid).
  • events context: Connection-handling settings (e.g., worker_connections, use epoll).
  • http context: Web and HTTP proxy rules (e.g., include, sendfile, keepalive_timeout).
  • server context: Virtual host definitions configured within http or stream contexts.
  • location context: URI path routing rules configured inside an HTTP server block.
  • upstream context: Pools of backend servers used for load balancing configured inside http or stream.
  • stream context: Top-level context for Layer 4 TCP/UDP proxying, parallel to http.

Location Matching Algorithm and Priorities

Nginx matches request URIs against location blocks using strict rules. Priority is not strictly determined by the order of appearance in the configuration file; specific modifier syntax overrides block placement.

The evaluation sequence occurs as follows:

  1. Exact Match (=): Matches the URI exactly. Processing stops immediately if matched.
  2. Preferential Prefix (^~): If the longest matching prefix location has ^~, Nginx stops searching for regular expressions.
  3. Standard Prefix (no modifier): Nginx searches all prefix locations and remembers the longest match.
  4. Regular Expression (~ case-sensitive, ~* case-insensitive): Nginx checks regular expressions in the order they appear in the config file. The first matching regex wins.
  5. Fallback: If no regex matches, Nginx uses the longest prefix match remembered from step 3.
Modifier Match Type Priority Description / Example
= Exact match 1 (Highest) location = /favicon.ico — Matches /favicon.ico only.
^~ Preferential prefix match 2 location ^~ /static/ — Longest prefix match; skips regex search.
~ Case-sensitive regex 3 location ~ \.php$ — Matches URIs ending in .php.
~* Case-insensitive regex 3 `location ~* .(jpg
(None) Standard prefix match 4 (Lowest) location /api/ — Matches any URI starting with /api/.

Operational Warning: A broad regex block defined early in a configuration file can hijack requests intended for prefix locations located further down. Always test URI resolution logic thoroughly.

Server Block Selection (server_name Logic)

When an HTTP request arrives, Nginx determines the correct server block using the following order of precedence based on the Host header:

  1. Exact hostname match (e.g., server_name api.example.com;).
  2. Longest wildcard match starting with an asterisk (e.g., server_name *.example.com;).
  3. Longest wildcard match ending with an asterisk (e.g., server_name example.*;).
  4. First matching regular expression in order of configuration appearance (e.g., server_name ~^api\.;).
  5. The default_server defined for that listen port. If no default_server flag is explicitly set, the first server block parsed for that port becomes the default fallback.

4. Installation and Version Management

Package Repositories and Installation

Linux distributions provide Nginx via default repositories, but official Nginx repositories ensure access to supported stable and mainline releases.

Debian / Ubuntu Installation

# Install prerequisites
sudo apt-get update
sudo apt-get install -y curl gnupg2 ca-certificates lsb-release debian-archive-keyring

# Import official Nginx signing key
curl -fsSL https://nginx.org/keys/nginx_signing.key | sudo gpg --dearmor -o /usr/share/keyrings/nginx-archive-keyring.gpg

# Add official APT repository (Ubuntu example)
echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] http://nginx.org/packages/ubuntu $(lsb_release -cs) nginx" | sudo tee /etc/apt/sources.list.d/nginx.list

# Install Nginx
sudo apt-get update
sudo apt-get install -y nginx

RHEL / Rocky Linux / AlmaLinux Installation

# Install yum-utils
sudo dnf install -y yum-utils

# Create repo file
cat << 'EOF' | sudo tee /etc/yum.repos.d/nginx.repo
[nginx-stable]
name=nginx static repo
baseurl=http://nginx.org/packages/centos/$releasever/$basearch/
gpgcheck=1
enabled=1
gpgkey=https://nginx.org/keys/nginx_signing.key
module_hotfixes=true
EOF

# Install Nginx
sudo dnf install -y nginx

Identifying Installed Version and Modules

To inspect version details, compile flags, and enabled modules, use the -v and -V flags:

# Print version only
nginx -v

# Output:
# nginx version: nginx/1.30.6

# Print version, compiler details, and configure arguments (modules)
nginx -V

Versioning Strategy: Mainline vs. Stable

Nginx maintains two active release branches:

  • Mainline Branch: Receives new features, performance improvements, and bug fixes. Recommended for general production deployment to access recent protocol enhancements.
  • Stable Branch: Receives critical security fixes and major bug fixes only; no new feature additions.

5. Configuration Files and Directory Structure

Configuration Hierarchy Reference

File / Directory Purpose Important Details
/etc/nginx/nginx.conf Global primary configuration file. Sets master process parameters, worker settings, global HTTP directives, and includes secondary files.
/etc/nginx/conf.d/*.conf Distribution standard configuration directory (RHEL/CentOS/Rocky & Official Nginx builds). Files ending in .conf are automatically included into the main http context.
/etc/nginx/sites-available/ Storage for virtual host configurations (Debian/Ubuntu convention). Contains configuration files for sites that are available but not necessarily active.
/etc/nginx/sites-enabled/ Active virtual host directory (Debian/Ubuntu convention). Contains symbolic links pointing back to /etc/nginx/sites-available/.
/etc/nginx/snippets/ Modular shared configuration fragments. Stores reusable snippets like SSL options or proxy headers included across multiple sites.
/var/log/nginx/access.log Default HTTP request log file. Records details for all incoming HTTP requests.
/var/log/nginx/error.log Default operational and diagnostic error log file. Records startup failures, worker errors, application proxy failures, and crash dumps.
/var/www/html/ or /usr/share/nginx/html/ Default document root directories. Stores default static files served by placeholder site blocks.
/run/nginx.pid or /var/run/nginx.pid Process ID file. Contains the numeric PID of the running master process.

Main nginx.conf Structure Example

Below is an operational, annotated /etc/nginx/nginx.conf file:

# User and process settings
user www-data;                      # Worker process user identity
worker_processes auto;              # Auto-detect available CPU cores
pid /run/nginx.pid;                 # PID file location
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections 2048;        # Max connections per worker process
    use epoll;                      # Efficient Linux event notification mechanism
    multi_accept on;                # Accept all new connections at once
}

http {
    # Basic I/O & Performance directives
    sendfile on;                    # Use kernel zero-copy sendfile()
    tcp_nopush on;                  # Send headers in one TCP packet
    tcp_nodelay on;                 # Disable Nagle's algorithm for low latency
    keepalive_timeout 65;           # HTTP Keep-Alive client timeout
    types_hash_max_size 2048;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    # Global security parameters
    server_tokens off;              # Hide Nginx version from server headers
    client_max_body_size 16M;       # Maximum allowed client upload payload

    # Logging Formats and Paths
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for" '
                    'rt=$request_time uct="$upstream_connect_time" uht="$upstream_header_time" urt="$upstream_response_time"';

    access_log /var/log/nginx/access.log main;
    error_log /var/log/nginx/error.log warn;

    # Include modular virtual host configurations
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

# Stream context for Layer 4 TCP/UDP proxying
stream {
    include /etc/nginx/stream.d/*.conf;
}

6. Service Management and Common Operations

Managing the Nginx daemon correctly prevents unintended site downtime during configuration changes.

Service Control Commands

Task Command Description & Behaviour
Check Status systemctl status nginx Displays active service state, main PID, memory usage, and recent systemd log entries.
Validate Configuration sudo nginx -t Tests configuration syntax and checks that referenced path files exist.
Dump Effective Config sudo nginx -T Validates configuration and prints the entire merged configuration to stdout.
Reload Service sudo systemctl reload nginx Re-reads configuration gracefully without dropping active client connections.
Restart Service sudo systemctl restart nginx Stops master and worker processes completely and restarts them; causes brief connection drop.
Enable on Boot sudo systemctl enable nginx Configures systemd to start Nginx automatically during system startup.

Graceful Reload Mechanics vs. Full Restart

  • Graceful Reload (systemctl reload nginx or nginx -s reload):
  1. Master process re-parses and validates the configuration file.
  2. If valid, the master process spawns a new set of worker processes with the new configuration.
  3. Master sends a QUIT signal to old worker processes, instructing them to stop accepting new connections while completing existing requests.
  4. Once existing connections finish, old workers terminate. Zero downtime is achieved.
  • Full Restart (systemctl restart nginx): The master process immediately terminates all worker processes and stops listener sockets. Active HTTP connections are interrupted before the daemon re-initializes.

Operational Rule: Always execute nginx -t before executing systemctl reload nginx. If a syntax error is present, running nginx -t catches the error immediately. Running systemctl reload nginx with invalid syntax will fail, and while running workers remain active, automated deployment scripts may halt or fail unexpectedly.


7. Common Administrative Tasks and Workflows

Task 1: Enabling and Disabling Virtual Hosts (Debian/Ubuntu Model)

  • Objective: Safely publish a new site configuration using symbolic links.
  • Step 1: Create configuration file in /etc/nginx/sites-available/app.example.com.conf.
  • Step 2: Enable site by creating a symlink:
sudo ln -s /etc/nginx/sites-available/app.example.com.conf /etc/nginx/sites-enabled/
  • Step 3: Validate and reload:
sudo nginx -t && sudo systemctl reload nginx
  • Step 4: Disable site (when retiring):
sudo rm /etc/nginx/sites-enabled/app.example.com.conf
sudo nginx -t && sudo systemctl reload nginx

(Removing the symlink in sites-enabled/ leaves the original configuration file intact in sites-available/).

Task 2: Configuring an HTTP Reverse Proxy with Custom Headers

  • Objective: Pass HTTP traffic to a Python Gunicorn app listening on 127.0.0.1:8000 while retaining original client connection details.
# /etc/nginx/conf.d/app_proxy.conf

upstream gunicorn_backend {
    server 127.0.0.1:8000 max_fails=3 fail_timeout=10s;
    keepalive 32; # Maintain pool of persistent connections to backend
}

server {
    listen 80;
    server_name app.example.com;

    location / {
        proxy_pass http://gunicorn_backend;

        # Preserve client headers for backend logging
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # Enable HTTP/1.1 for keepalive upstream connections
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        # Timeouts
        proxy_connect_timeout 60s;
        proxy_read_timeout 60s;
        proxy_send_timeout 60s;
    }
}

Task 3: Deploying a Single Page Application (SPA) Fallback Route

  • Objective: Serve static assets directly and fallback all client-side routing paths (e.g., /dashboard, /settings) to /index.html.
server {
    listen 80;
    server_name spa.example.com;
    root /var/www/spa-app;
    index index.html;

    location / {
        # Check literal URI file, directory, or fall back to /index.html
        try_files $uri $uri/ /index.html;
    }

    # Static media caching
    location ~* \.(js|css|png|jpg|jpeg|gif|ico|svg)$ {
        expires 30d;
        add_header Cache-Control "public, no-transform";
    }
}

Task 4: Configuring Layer 4 TCP Load Balancing (Stream Context)

  • Objective: Load balance raw TCP connections across two MySQL/MariaDB database read-replicas.
# Add to /etc/nginx/nginx.conf or /etc/nginx/stream.d/db_proxy.conf
stream {
    upstream mysql_read_pool {
        server 192.168.1.50:3306 max_fails=2 fail_timeout=5s;
        server 192.168.1.51:3306 max_fails=2 fail_timeout=5s;
    }

    server {
        listen 3306;
        proxy_pass mysql_read_pool;
        proxy_connect_timeout 5s;
        proxy_timeout 1m;
    }
}

8. CLI Command Reference and Cheatsheet

Process & Service Controls

Command Purpose Flags / Arguments Practical Example Explanation
nginx Direct executable invoker. -t (test), -T (dump), -s (signal), -c (config path). sudo nginx -t Tests all active configuration files for syntax and structural errors.
nginx Send process signal. -s stop, -s quit, -s reload, -s reopen. sudo nginx -s reload Signals master process to reload configuration gracefully.
nginx Specify custom config file. -c <file_path> sudo nginx -c /tmp/test_nginx.conf -t Validates a specific configuration file without changing production.
systemctl Manage systemd service. status, start, stop, restart, reload. sudo systemctl status nginx Checks active execution status and runtime logs of Nginx service.

Diagnostic & Inspection Commands

Command Purpose Flags / Arguments Practical Example Explanation
nginx -T Dump effective config. -T `sudo nginx -T grep -n ‘server_name’`
ss Socket Statistics. -t (tcp), -l (listen), -n (numeric), -p (processes). `sudo ss -tlnp grep nginx`
ps Process Status. aux `ps aux grep nginx`
curl HTTP Client inspection. -v (verbose), -I (headers only), -H (custom header). curl -vI -H "Host: api.example.com" [http://127.0.0.1/](http://127.0.0.1/) Bypasses local DNS resolution to test specific server_name blocks directly against localhost.
openssl TLS/SSL diagnostics. s_client -connect -servername openssl s_client -connect example.com:443 -servername example.com Tests TLS handshake, verifies certificate chains, and checks expiration dates.

9. Logs, Monitoring and Diagnostics

Log File Locations and Formats

Nginx produces two primary categories of logs:

  1. Access Logs (/var/log/nginx/access.log): Records details of every HTTP request handled by the server.
  2. Error Logs (/var/log/nginx/error.log): Records operational events, syntax errors, proxy backend connection failures, and file permission denied errors.

Configuring Diagnostic Upstream Log Formats

Standard access logs omit upstream backend latency metrics. Augmenting the log_format directive in nginx.conf provides visibility into backend performance:

log_format debug_upstream '$remote_addr - $remote_user [$time_local] "$request" '
                          'status=$status body_bytes=$body_bytes_sent '
                          'request_time=$request_time '
                          'upstream_addr="$upstream_addr" '
                          'upstream_status="$upstream_status" '
                          'upstream_response_time="$upstream_response_time" '
                          'upstream_connect_time="$upstream_connect_time"';

Real-Time Monitoring and Diagnostic Workflows

# Follow live error log output
sudo tail -f /var/log/nginx/error.log

# Follow live access log output with filtering for HTTP 5xx errors
sudo tail -f /var/log/nginx/access.log | grep --line-buffered "HTTP/1.1\" 5"

# Inspect Nginx systemd journal logs since last boot
sudo journalctl -u nginx -b --no-pager -n 100

# Analyze top requested URIs from access log
sudo awk '{print $7}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -n 10

10. Troubleshooting Guide and Common Failure Scenarios


Problem 1: Nginx Service Fails to Start or Reload

Symptoms

Running systemctl reload nginx or systemctl start nginx returns a non-zero exit code: Job for nginx.service failed. Active requests fail or configuration updates are ignored.

Possible Causes

  • Syntax errors in configuration files (missing semicolons, unmatched braces).
  • Duplicate listen directives binding the same IP/port combination twice.
  • Invalid path references inside include directives.
  • Unprivileged user attempting to bind to ports < 1024.

Investigation and Diagnosis

  1. Objective: Locate the exact syntax error file and line number.
  • Command: sudo nginx -t
  • Output Interpretation: If errors exist, Nginx outputs the exact file path and line number:
nginx: [emerg] invalid number of arguments in "listen" directive in /etc/nginx/conf.d/app.conf:3
nginx: configuration file /etc/nginx/nginx.conf test failed
  1. Objective: Inspect systemd failure reason if nginx -t passes.
  • Command: sudo journalctl -u nginx -n 50 --no-pager
  • Output Interpretation: Look for OS-level bind errors or file lock issues.

Root Cause Identification

Confirm whether the issue stems from configuration syntax errors, missing include files, or socket binding conflicts.

Resolution

  • Correct the syntax error on the indicated file and line number.
  • Remove duplicate listen directives.
  • Ensure referenced directories exist (sudo mkdir -p /var/log/nginx/custom).

Verification

Execute sudo nginx -t. Ensure it returns: syntax is ok and test is successful. Then run sudo systemctl reload nginx.

Prevention

Integrate nginx -t into CI/CD deployment pipelines prior to copying configuration files into production.


Problem 2: Reload Succeeded but Configuration Behavior Changed Unexpectedly

Symptoms

nginx -t and systemctl reload nginx succeed, but the website continues to serve old configuration responses or wrong routing rules.

Possible Causes

  • Modifying a configuration file that is not included in the active nginx.conf.
  • Running multiple conflicting Nginx master processes.
  • Testing against a container or host whose config mounts point elsewhere.
  • Downstream browser caching or CDN edge caching.

Investigation and Diagnosis

  1. Objective: Inspect the effective merged configuration loaded in memory.
  • Command: sudo nginx -T | grep -n 'your-directive'
  • Output Interpretation: If your directive does not appear in the output of nginx -T, your edited file is not included in the master config path.
  1. Objective: Verify which binary and configuration file the active master process is using.
  • Command: ps aux | grep nginx
  • Output Interpretation: Ensure the master process PID uses /usr/sbin/nginx -c /etc/nginx/nginx.conf.

Root Cause Identification

The active Nginx instance is either executing a different configuration file tree or your changes are overridden by another included server block.

Resolution

  • Ensure the target configuration file is symlinked properly from sites-available to sites-enabled (Debian/Ubuntu) or placed in /etc/nginx/conf.d/ (RHEL/Rocky).
  • Terminate rogue duplicate Nginx processes (sudo pkill -9 nginx && sudo systemctl start nginx).

Verification

Test locally using curl:

curl -vI -H "Host: actual.domain.com" http://127.0.0.1/

Prevention

Use nginx -T to verify merged settings during deployment audits.


Problem 3: 403 Forbidden Errors and Worker Process Permissions

Symptoms

Clients receive HTTP 403 Forbidden. The Nginx error log contains: (13: Permission denied) while reading upstream or stat() "/var/www/app/index.html" failed (13: Permission denied).

Possible Causes

  • The directory path or target file lacks read/execute permissions for the unprivileged Nginx worker user (www-data or nginx).
  • Missing index file when autoindex is set to off.
  • SELinux or AppArmor blocking Nginx worker access.

Investigation and Diagnosis

  1. Objective: Identify the active worker user identity.
  • Command: grep -E '^(user|group)' /etc/nginx/nginx.conf
  1. Objective: Test file traversal and read permissions as the unprivileged worker user.
  • Command: sudo -u www-data test -r /var/www/app/index.html && echo "READ OK"
  • Output Interpretation: If it returns nothing or permission denied, the worker user cannot read the file or traverse parent directories.

Crucial SRE Insight: Testing permissions as root (e.g. cat /var/www/app/index.html) will succeed even when Nginx workers fail with 403. Always test permissions using sudo -u <worker_user>.

  1. Objective: Inspect SELinux denial audits (on RHEL/Rocky Linux).
  • Command: sudo ausearch -m avc -ts recent or sudo grep nginx /var/log/audit/audit.log

Root Cause Identification

Permissions are restricted on one of the parent path directories, or SELinux context labels are incorrect.

Resolution

  • Grant execute permissions on all parent directories down to the file path:
sudo chmod +x /var /var/www /var/www/app
sudo chmod 644 /var/www/app/index.html
  • Fix SELinux web content contexts:
sudo restorecon -Rv /var/www/app
sudo setsebool -P httpd_can_network_connect 1

Verification

Execute:

sudo -u www-data ls -la /var/www/app/
curl -I http://localhost/index.html

Problem 4: 502 Bad Gateway / 504 Gateway Timeout Errors

Symptoms

Clients receive HTTP 502 (Bad Gateway) or HTTP 504 (Gateway Timeout). Nginx error.log reports connect() failed (111: Connection refused), no live upstreams, or upstream timed out.

Possible Causes

  • The upstream backend service (Gunicorn, PHP-FPM, Node.js) is stopped or crashing.
  • Upstream app is bound to 127.0.0.1:8000 but Nginx is configured for a Unix socket or wrong IP/port.
  • Upstream backend queue is saturated, exceeding Nginx connection/read timeouts.

Investigation and Diagnosis

  1. Objective: Check if the backend application process is listening.
  • Command: sudo ss -tlnp | grep 8000
  1. Objective: Test the backend directly from the Nginx host bypassing Nginx.
  • Command: curl -v [http://127.0.0.1:8000/health](http://127.0.0.1:8000/health)
  • Output Interpretation: If connection fails with Connection refused, the upstream service is down.
  1. Objective: Inspect Nginx error log for specific proxy errors.
  • Command: sudo tail -n 20 /var/log/nginx/error.log

Resolution

  • Restart upstream backend service (sudo systemctl restart gunicorn).
  • If 504 Gateway Timeout occurs during heavy queries, increase upstream timeouts in nginx.conf:
proxy_connect_timeout 90s;
proxy_read_timeout 90s;
proxy_send_timeout 90s;

Verification

Run curl -v [http://127.0.0.1:8000/](http://127.0.0.1:8000/) followed by an end-to-end HTTPS request.


Problem 5: Unix Domain Socket Permission & Path Failures

Symptoms

HTTP 502 Bad Gateway. Nginx error.log reports connect() to unix:/run/app.sock failed (2: No such file or directory) or (13: Permission denied) while connecting to upstream.

Possible Causes

  • Unix socket file does not exist because the upstream daemon failed to create it.
  • Socket file permissions (e.g., 0600) restrict access to the socket owner, blocking the Nginx worker user.

Investigation and Diagnosis

  1. Objective: Check socket file existence and permission flags.
  • Command: ls -la /run/app.sock
  • Output Interpretation: Verify permissions show srw-rw---- and group ownership matches the Nginx worker user group (e.g., www-data).

Resolution

  • Align upstream socket owner settings (e.g., in Gunicorn or PHP-FPM pool configuration):
# Example PHP-FPM www.conf / Gunicorn config
listen.owner = www-data
listen.group = www-data
listen.mode = 0660
  • Restart backend service to recreate socket file.

Verification

Run sudo -u www-data test -w /run/app.sock && echo "SOCKET WRITABLE".


Problem 6: Root vs. Alias Directive Confusion

Symptoms

Static asset requests return HTTP 404 Not Found. The paths shown in error.log contain duplicate directory segments (e.g., /data/images/images/logo.png).

Root Cause Analysis & Behavioral Rules

  • root directive: Appends the full request URI to the root path.
location /images/ {
    root /data;
}
# Request GET /images/logo.png -> Resolves to /data/images/logo.png
  • alias directive: Replaces the matched location URI path with the alias path.
location /images/ {
    alias /data/;
}
# Request GET /images/logo.png -> Resolves to /data/logo.png

Operational Warning: When using alias inside a location ending with a trailing slash (/images/), the alias directive MUST also end with a trailing slash (/data/). Trailing slash mismatches cause malformed path resolution.

Resolution

Adjust root or alias directives based on target directory structure:

# Correct use of alias
location /assets/ {
    alias /var/www/static-files/;
}

Problem 7: Wrong Virtual Host or Default Server Served

Symptoms

Requests for [https://api.example.com](https://api.example.com) serve content from a completely different site hosted on the same server.

Possible Causes

  • No server_name directive matches api.example.com.
  • Request falls through to the default server for port 80/443.
  • Broad regex server_name matches before exact host rules.

Investigation and Diagnosis

  1. Objective: Identify all configured listen sockets and server_name directives.
  • Command: sudo nginx -T | grep -E 'server_name|listen|default_server'
  1. Objective: Simulate Host header requests against localhost.
  • Command: curl -v -H "Host: api.example.com" [http://127.0.0.1/](http://127.0.0.1/)

Resolution

  • Add explicit server_name directives.
  • Designate an explicit catch-all fallback block to drop unhandled hosts:
server {
    listen 80 default_server;
    listen 443 ssl default_server;
    ssl_certificate /etc/ssl/certs/fallback.crt;
    ssl_certificate_key /etc/ssl/private/fallback.key;
    server_name _;
    return 444; # Special Nginx status code: Close connection without response
}

Problem 8: 413 Request Entity Too Large

Symptoms

Users attempting file uploads receive HTTP 413 Request Entity Too Large.

Possible Causes

Client request body size exceeds the limit defined by client_max_body_size (default is 1 Megabyte).

Resolution

Increase client_max_body_size in the http, server, or location block:

http {
    client_max_body_size 64M;
}

Validate and reload: sudo nginx -t && sudo systemctl reload nginx.


Problem 9: Incomplete TLS/SSL Certificate Chain Warnings

Symptoms

Modern browsers show NET::ERR_CERT_AUTHORITY_INVALID or mobile clients fail TLS verification, even though basic SSL tests pass.

Possible Causes

The ssl_certificate file contains only the server leaf certificate, omitting intermediate Certificate Authority (CA) certificates.

Investigation and Diagnosis

  1. Objective: Inspect full certificate chain validation.
  • Command: openssl s_client -connect example.com:443 -servername example.com
  • Output Interpretation: Look for Verify return code: 21 (unable to verify the first certificate).

Resolution

Concatenate your leaf certificate and intermediate CA certificates into a single full-chain file:

cat domain_leaf.crt intermediate.crt root.crt > /etc/ssl/certs/fullchain.crt

Update configuration:

ssl_certificate /etc/ssl/certs/fullchain.crt;
ssl_certificate_key /etc/ssl/private/domain.key;

11. Security and Best Practices

Hardening Guidelines

  1. Disable Server Identity Tokens: Suppress Nginx version disclosure in HTTP header outputs.
server_tokens off;
  1. Restrict Unprivileged User Identity: Ensure Nginx worker processes run under a dedicated, unprivileged system account (www-data or nginx) with no shell access (/nologin).
  2. Configure Secure TLS Settings: Enforce TLS 1.2 and TLS 1.3 while disabling outdated SSLv3, TLS 1.0, and TLS 1.1 protocols.
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
  1. Implement Rate Limiting: Prevent HTTP request flood attacks and brute-force attempts using limit_req_zone.
# Define rate limit zone in http context
limit_req_zone $binary_remote_addr zone=login_limit:10m rate=5r/s;

server {
    location /login {
        limit_req zone=login_limit burst=10 nodelay;
        proxy_pass http://backend;
    }
}
  1. Set Security Response Headers: Inject defensive HTTP headers to mitigate cross-site scripting (XSS) and clickjacking.
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-Content-Type-Options "nosniff" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Content-Security-Policy "default-src 'self' http: https:;" always;

12. Performance and Resource Optimisation

Tuning File Descriptors and Connections

Linux kernels limit maximum open file descriptors per process by default. Because Nginx handles connections as file descriptors, tune worker_rlimit_nofile and connection limits:

# /etc/nginx/nginx.conf

# Increase worker open file descriptor limit
worker_rlimit_nofile 65535;

events {
    worker_connections 8192;
    use epoll;
}

Ensure the kernel fs.file-max setting supports process-level limits:

sysctl -w fs.file-max=2097152

High-Performance File Delivery Settings

Enable sendfile and TCP packet optimizations to stream static files directly from kernel buffer cache to network sockets without copying data into application memory space:

http {
    sendfile on;
    tcp_nopush on; # Works with sendfile to send HTTP header and file start in one packet
    tcp_nodelay on; # Disables Nagle algorithm for real-time interactive data

    # Gzip Compression Tuning
    gzip on;
    gzip_comp_level 5;
    gzip_min_length 256;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml+rss;
}

13. Backup, Recovery and Maintenance

Backup Strategy

Maintain backups of configuration directories and custom certificates:

# Archive configuration files and certificates
sudo tar -czvf /backups/nginx_config_$(date +%F).tar.gz /etc/nginx /etc/ssl

Version Upgrade and Rollback Workflow

sequenceDiagram
    autonumber
    actor Admin as SRE / SysAdmin
    participant Package as Package Manager (apt/dnf)
    participant Nginx as Nginx Master Process

    Admin->>Admin: Create config backup tarball
    Admin->>Package: Upgrade nginx package
    Package->>Nginx: Replaces binary on disk
    Admin->>Nginx: Send USR2 signal (Spawn new master & workers)
    Note over Nginx: New Master runs alongside Old Master
    Admin->>Nginx: Send WINCH signal (Gracefully stop old workers)
    alt Verification Success
        Admin->>Nginx: Send QUIT signal (Terminate old Master)
    else Verification Failure / Rollback
        Admin->>Nginx: Send HUP signal to Old Master (Restore original workers)
        Admin->>Nginx: Send TERM signal to New Master
    end

Executing Zero-Downtime Binary Upgrades (Manual Method)

  1. Replace binary via package manager (apt-get install --only-upgrade nginx).
  2. Send USR2 signal to master process to spawn new master and new workers using the new binary:
sudo kill -USR2 $(cat /run/nginx.pid)
  1. Send WINCH signal to old master to gracefully shut down old workers:
sudo kill -WINCH $(cat /run/nginx.pid.oldbin)
  1. Verify application health. If successful, send QUIT to old master:
sudo kill -QUIT $(cat /run/nginx.pid.oldbin)

14. Automation and Integration

Ansible Configuration Management Automation

Below is a production Ansible task snippet to deploy and validate Nginx:

---
- name: Deploy and Manage Nginx Web Server
  hosts: webservers
  become: true
  tasks:
    - name: Install Nginx package
      ansible.builtin.package:
        name: nginx
        state: present

    - name: Copy Nginx primary configuration
      ansible.builtin.template:
        src: templates/nginx.conf.j2
        dest: /etc/nginx/nginx.conf
        owner: root
        group: root
        mode: '0644'
      notify: Validate and Reload Nginx

  handlers:
    - name: Validate and Reload Nginx
      ansible.builtin.command: nginx -t
      changed_when: false
      failed_when: false
      notify: Reload Systemctl Nginx

    - name: Reload Systemctl Nginx
      ansible.builtin.systemd:
        name: nginx
        state: reloaded

15. Practical Labs and Exercises

Exercise 1 (Beginner): Deploying a Multi-Site Web Host

  • Objective: Configure two separate websites (site-a.local and site-b.local) on a single Linux server listening on port 80.
  • Environment Setup:
  1. Create directories: /var/www/site-a and /var/www/site-b.
  2. Populate index files: echo "Site A" > /var/www/site-a/index.html and echo "Site B" > /var/www/site-b/index.html.
  • Instructions:
  1. Create /etc/nginx/conf.d/site-a.conf with server_name site-a.local; pointing root to /var/www/site-a.
  2. Create /etc/nginx/conf.d/site-b.conf with server_name site-b.local; pointing root to /var/www/site-b.
  3. Validate syntax (nginx -t) and reload Nginx (systemctl reload nginx).
  • Validation:
curl -H "Host: site-a.local" http://127.0.0.1/ # Expected output: Site A
curl -H "Host: site-b.local" http://127.0.0.1/ # Expected output: Site B

Exercise 2 (Troubleshooting): Investigating Permission Denied (403) Failures

  • Objective: Diagnose and resolve an intentionally broken file permission state.
  • Environment Setup:
sudo mkdir -p /var/www/secure-app
echo "Secret App" | sudo tee /var/www/secure-app/index.html
sudo chmod 700 /var/www/secure-app  # Restricted parent dir!
  • Instructions:
  1. Create a server block pointing to /var/www/secure-app.
  2. Reload Nginx and request http://localhost/. Observe the 403 Forbidden error.
  3. Inspect /var/log/nginx/error.log to confirm permission failure.
  4. Test read access using sudo -u www-data test -r /var/www/secure-app/index.html.
  5. Fix directory traversal permissions (sudo chmod 755 /var/www/secure-app).
  • Validation: Run curl -I http://localhost/ and verify HTTP 200 OK.

16. Quick Reference and Cheat Sheet

Important Configuration Files

File / Directory Purpose
/etc/nginx/nginx.conf Global primary configuration file.
/etc/nginx/conf.d/*.conf HTTP virtual host directory (RHEL/CentOS/Rocky & official packages).
/etc/nginx/sites-available/ Available site configurations (Debian/Ubuntu).
/etc/nginx/sites-enabled/ Active site symlinks (Debian/Ubuntu).
/etc/nginx/mime.types File extension to MIME-type mappings.

Important Log Locations

Log Path Purpose
/var/log/nginx/access.log Records all processed HTTP client requests.
/var/log/nginx/error.log Operational error log (syntax, worker, proxy errors).

Common Commands

Task Command
Test configuration syntax sudo nginx -t
Test and dump merged config sudo nginx -T
Graceful reload sudo systemctl reload nginx
Check active ports/listeners `sudo ss -tlnp
Inspect active worker user `ps aux
Test Host header routing locally curl -vI -H "Host: example.com" [http://127.0.0.1/](http://127.0.0.1/)

Troubleshooting Quick Reference

Symptom Initial Investigation Step
Nginx service won’t reload Run sudo nginx -t to locate syntax errors.
Config changes ignored Run sudo nginx -T to verify if file is included.
403 Forbidden Test path access using sudo -u www-data test -r <file>.
502 Bad Gateway Test upstream port using curl -v [http://127.0.0.1](http://127.0.0.1):<backend_port>.
413 Entity Too Large Increase client_max_body_size in HTTP/Server context.

Important Ports and Protocols

Port Protocol Purpose
80 TCP Default HTTP unencrypted web traffic.
443 TCP Default HTTPS TLS-encrypted web traffic.
3306 / custom TCP Layer 4 stream proxying (databases/custom protocols).

17. Further Reading and References

  1. Source 1: Nginx Guide — Concept, Architecture, Directives, and Core References (Nginx guide).
  1. Source 2: Nginx Operational Troubleshooting Guide (Nginx troubleshooting).
  2. Source 3: Nginx Command and Operations Cheatsheet (Nginx cheatsheet).