Domain 10Linux & SREINTERMEDIATE

Prometheus Monitoring & Observability Handbook: Architecture, Configuration, and Production Operations

A comprehensive Linux SysAdmin and SRE guide to Prometheus architecture, metrics collection, PromQL querying, alert routing, operational management, and systematic troubleshooting.

Domain: Domain 10: Linux Internals, SRE & Troubleshooting•Updated: 2026-11-03
#linux#sysadmin#prometheus#monitoring#observability

1. Overview

Prometheus is an open-source monitoring system and time-series database designed specifically for modern infrastructure and cloud-native environments. It addresses key operational questions in production environments, such as identifying real-time error rates, detecting instances running out of disk space, or determining whether latency spiked immediately following a application deployment.

Applications and host exporters expose operational metrics in a standardized, line-oriented text format over HTTP. Prometheus periodically pulls (scrapes) these metrics on a configured time interval and stores the sampled metrics as multi-dimensional time series identified by metric names and key-value labels.

[ Application / Node Exporter ] 
       │ (Exposes HTTP /metrics)
       ▼
[ Prometheus Server (Scrapes & Stores TSDB) ]
       │                                   │
       ▼                                   ▼
[ Alertmanager ] ──► Notifications   [ Grafana ] ──► Dashboards & Exploration

The Modern Observability Stack

In production environments, Prometheus functions as the telemetry collection and alerting engine within a multi-tiered monitoring stack:

  • Exporters & Applications: Expose operational status, host metrics, and runtime metrics via HTTP endpoints. (Source: Reference 1)
  • Prometheus Server: Periodically scrapes target endpoints, evaluates alerting and recording rules, and persists time-series data to local disk. (Source: Reference 1)
  • Grafana: Connects to Prometheus as an analytical data source to render visual dashboards, charts, and interactive tables. (Source: Reference 1)
  • Alertmanager: Receives fired alerts from Prometheus, deduplicates them, groups related events, manages silences, and routes notifications to downstream channels like Slack, PagerDuty, or email. (Source: Reference 1)

Core Strengths and Operational Limitations

Dimension Prometheus Strengths Operational Limitations
Data Collection Active pull model simplifies target state verification and prevents unmonitored agent proliferation. Requires direct network reachability or proxies to scrape target endpoints.
Data Model Rich multi-dimensional data model using key-value labels for dynamic slicing and aggregation. High-cardinality data (e.g., user IDs or IP addresses as labels) degrades TSDB performance and fills disk space rapidly.
Query Engine PromQL enables complex mathematical calculations, rate calculations, and quantile aggregations. Not designed for event logging, text searching, or distributed tracing.
Architecture Autonomous single-binary server design without complex cluster dependencies. Out-of-the-box local TSDB is intended for short-to-medium term retention (~15 days), requiring remote write integrations for long-term historical storage.

2. Architecture and Mental Model

To effectively operate and troubleshoot Prometheus, system administrators must understand its core architecture, data collection engine, and internal lifecycle.

High-Level Architecture and Data Flow

Prometheus operates primarily on an active pull model. Rather than waiting for external agents to push metrics, Prometheus actively reaches out to configured target endpoints at regular intervals (scrape_interval).

flowchart TD
    subgraph Targets ["Monitored Targets"]
        APP["Instrumented App (/metrics)"]
        NODE["Node Exporter (Port 9100)"]
        PUSH["Pushgateway (Batch Jobs)"]
    end

    subgraph Server ["Prometheus Server"]
        SD["Service Discovery Engine\n(File SD, K8s, Static)"]
        SCRAPER["Scrape Engine\n(Interval Pull)"]
        RELABEL["Relabeling Engine\n(Target & Metric Filters)"]
        TSDB[("TSDB Storage\n(Local Disk / WAL)")]
        RULES["Rule Engine\n(PromQL / Alerting)"]
        API["Web UI & HTTP API\n(Port 9090)"]
    end

    subgraph Consumers ["Visualization & Alerting"]
        AM["Alertmanager\n(Port 9093)"]
        GRAFANA["Grafana UI\n(Port 3000)"]
    end

    SD -->|Target Lists| SCRAPER
    SCRAPER -->|HTTP Scrape| APP
    SCRAPER -->|HTTP Scrape| NODE
    SCRAPER -->|HTTP Scrape| PUSH
    SCRAPER --> RELABEL
    RELABEL --> TSDB
    RULES -->|Evaluate PromQL| TSDB
    RULES -->|Fire Alerts| AM
    GRAFANA -->|PromQL Queries| API
    API -->|Read TSDB| TSDB

Main Components and Responsibilities

  1. Scrape Engine: Reads configured jobs and dynamic target lists, initiates HTTP GET requests to target metric paths (defaulting to /metrics), parses text metric formats, and records execution metrics like scrape_duration_seconds and target reachability (up).
  2. TSDB (Time Series Database): Stores raw sample data (timestamp and float64 value pairs) efficiently on disk inside two-hour blocks with Write-Ahead Logging (WAL) to prevent data loss across crashes.
  3. Service Discovery (SD): Interacts with infrastructure providers (Kubernetes API, cloud providers, file-based discovery) to automatically detect added or removed scrape targets.
  4. Relabeling Engine: Applies transformation rules before ingestion to rewrite metric labels, filter noisy endpoints, or drop high-cardinality labels.
  5. Rule Engine: Periodically executes PromQL expressions for Recording Rules (pre-computing expensive queries) and Alerting Rules (evaluating failure conditions).
  6. Web UI and API: Serves built-in diagnostic pages (/targets, /graph, /status), accepts configuration reloads (/-/reload), and handles PromQL queries from Grafana or CLI utilities.

Network Ports & Communication Map

  • Port 9090: Prometheus Server Web UI and REST API. (Source: Reference 1, 3)
  • Port 9100: node_exporter default metric port. (Source: Reference 2, 3)
  • Port 9093: Alertmanager Web UI and API. (Source: Reference 2)
  • Port 3000: Grafana Web Interface. (Source: Reference 1, 3)

3. Core Concepts and Terminology

Understanding Prometheus requires familiarity with its fundamental data structures, metric types, and PromQL execution mechanics.

Metric Types

Prometheus supports four primary metric types in its client libraries:

  1. Counter: A cumulative metric that only increases or resets to zero upon server restart. Used for measuring total requests, error counts, or completed tasks. Always apply rate() or irate() functions when graphing counters in PromQL.
  2. Gauge: A metric representing a single numerical value that can arbitrarily go up or down. Used for current memory usage, CPU temperature, queue depth, or disk availability (node_memory_MemAvailable_bytes).
  3. Histogram: Samples observations (such as request durations or response sizes) and counts them in configurable bucket ranges. Also provides a total sum of observed values and total count. Essential for calculating percentiles (e.g., 99th percentile latency) using histogram_quantile().
  4. Summary: Similar to histograms, summaries calculate client-side quantiles over a sliding time window. While accurate on individual instances, summary quantiles cannot be aggregated across multiple server instances in PromQL.

Metric Syntax and Label Cardinality

A Prometheus metric consists of a metric name, key-value pair labels, and a numerical payload value:

http_requests_total{method="GET", handler="/api/v1/users", status="200"} 1043
  • Labels: Dimensions attached to time series that allow granular PromQL filtering, grouping, and multi-dimensional analysis. (Source: Reference 1)
  • Label Cardinality: The total number of unique time series generated by combining all label key-value pairs. High cardinality occurs when unbounded data (such as user IDs, UUIDs, container IP addresses, or raw request URLs) is added as labels. This causes exponential TSDB growth, high memory utilization, scrape timeouts, and database slowdowns.

Core Terminology Glossary

Term Definition Operational Relevance
Scrape Interval Frequency at which Prometheus fetches metrics from targets. Configured globally or per job (scrape_interval: 15s).
Target An individual HTTP endpoint scraped by Prometheus. Tracked via the up metric (1 = healthy, 0 = down).
Pushgateway An intermediary metric cache for short-lived batch jobs. Short-lived jobs push metrics to Pushgateway, which Prometheus scrapes.
Recording Rule Pre-computed PromQL expressions evaluated continuously. Reduces computational overhead when rendering complex Grafana dashboards.
Alerting Rule PromQL expression evaluated to trigger operational alerts. Evaluated on schedule; firing alerts are pushed directly to Alertmanager.
Relabeling Label modification step applied before storing time series. Used to rewrite metric paths, drop high-cardinality labels, or map dynamic discovery metadata.

4. Installation and Version Management

Prometheus is distributed as a single static Go binary, making installation straightforward on Linux distributions.

Installing Prometheus on Enterprise Linux (RHEL/Rocky/AlmaLinux & Ubuntu/Debian)

# 1. Create dedicated system user and directories (Linux Best Practice)
sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus

# 2. Download latest stable binary package
ARCH="linux-amd64"
VERSION="2.54.1" # Example stable release
wget https://github.com/prometheus/prometheus/releases/download/v${VERSION}/prometheus-${VERSION}.${ARCH}.tar.gz

# 3. Extract and distribute binaries and assets
tar -xvf prometheus-${VERSION}.${ARCH}.tar.gz
cd prometheus-${VERSION}.${ARCH}
sudo cp prometheus promtool /usr/local/bin/
sudo cp -r consoles console_libraries /etc/prometheus/
sudo cp prometheus.yml /etc/prometheus/prometheus.yml

# 4. Set strict file ownership
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool

Method 2: Installing via System Package Managers

On Ubuntu/Debian systems:

sudo apt-get update
sudo apt-get install -y prometheus promtool

On Enterprise Linux (RHEL / EPEL):

sudo dnf install -y epel-release
sudo dnf install -y prometheus2

Verifying Binary Installation and Software Version

To confirm successful installation and inspect built-in features, execute:

# Verify Prometheus server binary version
prometheus --version

# Verify promtool CLI utility version
promtool --version

Default Filesystem Layout

  • Binary executables: /usr/local/bin/prometheus and /usr/local/bin/promtool (or /usr/bin/ for package installs).
  • Configuration directory: /etc/prometheus/.
  • TSDB Storage directory: /var/lib/prometheus/.

5. Configuration Files and Directory Structure

Prometheus uses YAML syntax for its configuration files. Formatting is strictly space-indented (tabs are invalid and cause startup failures).

Comprehensive Path Reference

File / Directory Purpose Important Details
/etc/prometheus/prometheus.yml Main configuration file. Defines global scrape intervals, rule files, alertmanager targets, and scrape jobs.
/etc/prometheus/rules/*.yml Alerting and recording rule files. Grouped YAML definitions for alert conditions and pre-computed metrics.
/var/lib/prometheus/ Primary TSDB storage directory. Contains block directories, chunk files, and wal/ (Write-Ahead Log).
/etc/systemd/system/prometheus.service Systemd service definition unit file. Defines daemon flags, startup parameters, user context, and resource limits.
/usr/local/bin/promtool CLI verification and debugging tool. Used to validate config syntax (promtool check config) before reloading services.

Main Configuration Structure (/etc/prometheus/prometheus.yml)

The standard configuration file divides settings into clear operational blocks:

# Global defaults applied across all scrape jobs unless overridden
global:
  scrape_interval:     15s # How frequently Prometheus scrapes targets (Default: 1m)
  evaluation_interval: 15s # How frequently Prometheus evaluates rules
  scrape_timeout:      10s # Scrape request timeout threshold

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - 'localhost:9093' # Alertmanager endpoint

# Rule file paths loaded into the rule evaluation engine
rule_files:
  - "/etc/prometheus/rules/*.yml"

# Scrape targets configuration
scrape_configs:
  # Internal self-monitoring job
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Linux host infrastructure monitoring job
  - job_name: 'node_exporter'
    scrape_interval: 10s
    static_configs:
      - targets: ['192.168.1.10:9100', '192.168.1.11:9100']

  # Custom application monitoring job
  - job_name: 'backend_api'
    metrics_path: '/metrics'
    scheme: 'http'
    static_configs:
      - targets: ['app1.example.com:8080']
``` *(Source: Reference 1, 3)*

### Configuration Precedence and Modification Mechanics

1. **CLI Flags Overwrite Configuration Defaults**: Flags passed in `ExecStart` inside the systemd service file (such as storage retention or network flags) take precedence over general daemon settings.
2. **Configuration Validation**: ALWAYS validate configuration syntax prior to applying changes using `promtool check config` to prevent service crashes.
3. **Dynamic Reloading**: Prometheus supports zero-downtime configuration reloads without restarting the process.

---

# 6. Service Management and Common Operations

Prometheus is managed as a standard background service using Systemd.

### Creating the Systemd Service Unit (`/etc/systemd/system/prometheus.service`)

Create the systemd service file to control execution parameters and flags:

```ini
[Unit]
Description=Prometheus Time Series Database & Monitoring Engine
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/ \
  --storage.tsdb.retention.time=15d \
  --web.enable-lifecycle \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries

ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target
``` *(Source: Reference 1)*

> **Security & Operational Note**: The `--web.enable-lifecycle` flag enables the HTTP POST `/-/reload` endpoint. Ensure the web port is protected by network firewalls or reverse proxy authentication if this flag is enabled.

### Common Service Management Commands

```bash
# Reload systemd configuration manager after editing service files
sudo systemctl daemon-reload

# Enable Prometheus service to start automatically on system boot
sudo systemctl enable prometheus

# Start the Prometheus monitoring daemon
sudo systemctl start prometheus

# Inspect execution status, PID, and recent log outputs
systemctl status prometheus

# Check that Prometheus is actively listening on network port 9090
ss -tlnp | grep 9090
``` *(Source: Reference 3)*

### Reloading Configuration vs. Restarting Service

| Operation | Command | Impact | Operational Guidance |
|---|---|---|---|
| **Hot Reload** | `curl -X POST http://localhost:9090/-/reload` or `systemctl reload prometheus` | **Zero Downtime**. Re-reads config and rule files without interrupting metric scraping or TSDB memory locks. | Use for adding target endpoints, modifying alert thresholds, or adjusting scrape intervals. Requires `--web.enable-lifecycle`. |
| **Full Restart** | `systemctl restart prometheus` | **Brief Disruption**. Stops execution, flushes TSDB state, closes sockets, and restarts process. | Required when modifying command-line flags (e.g., retention periods, TSDB paths, or listening ports). |

---

# 7. Common Administrative Tasks and Workflows

### Workflow 1: Adding a New Target to Prometheus Scrape Configurations

#### Objective
Add a fleet of Linux servers running `node_exporter` (port 9100) to Prometheus scraping.

#### Procedure
1. Edit `/etc/prometheus/prometheus.yml`:
   ```yaml
   scrape_configs:
     - job_name: 'node_infrastructure'
       scrape_interval: 15s
       static_configs:
         - targets:
             - '10.0.1.50:9100'
             - '10.0.1.51:9100'
           labels:
             environment: 'production'
   ``` *(Source: Reference 3)*
2. Validate syntax using `promtool`:
   ```bash
   promtool check config /etc/prometheus/prometheus.yml
   ``` *(Source: Reference 2, 3)*
3. Trigger dynamic configuration reload:
   ```bash
   curl -X POST http://localhost:9090/-/reload
   ``` *(Source: Reference 1, 3)*
4. Confirm target registration via HTTP API:
   ```bash
   curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .discoveredLabels.job, health: .health, instance: .discoveredLabels.__address__}'
   ``` *(Source: Reference 3)*

### Workflow 2: Configuring an Alerting Rule for Node Outages

#### Objective
Detect when host targets go offline (`up == 0`) for more than 5 minutes and route notifications to Alertmanager.

#### Procedure
1. Create rule file `/etc/prometheus/rules/host_alerts.yml`:
   ```yaml
   groups:
     - name: host_availability_alerts
       rules:
         - alert: InstanceDown
           expr: up == 0
           for: 5m
           labels:
             severity: critical
           annotations:
             summary: "Prometheus target system is offline"
             description: "Target instance {{ $labels.instance }} of job {{ $labels.job }} has been unreachable for more than 5 minutes."
   ``` *(Source: Reference 3)*
2. Validate rule syntax:
   ```bash
   promtool check rules /etc/prometheus/rules/host_alerts.yml
   ``` *(Source: Reference 2, 3)*
3. Verify that `/etc/prometheus/prometheus.yml` includes the rule file path:
   ```yaml
   rule_files:
     - "/etc/prometheus/rules/*.yml"
   ``` *(Source: Reference 1)*
4. Reload configuration and verify alerts tab in UI (`http://localhost:9090/alerts`).

---

# 8. CLI Command Reference and Cheatsheet

### `promtool` Diagnostic Utility Reference

`promtool` is the official command-line verification and debugging tool shipped with Prometheus.

| Command Syntax | Purpose | Key Arguments / Flags | Practical Example | Operational Explanation |
|---|---|---|---|---|
| `promtool check config` | Validates configuration syntax. | `<config-file-path>` | `promtool check config /etc/prometheus/prometheus.yml` | Inspects YAML structure, target block definitions, and referenced paths. Exits with zero on success. |
| `promtool check rules` | Validates rule syntax. | `<rule-file-path>` | `promtool check rules /etc/prometheus/rules/*.yml` | Validates syntax of alerting and recording rules, verifying required fields like `expr`. |
| `promtool query instant` | Executes instant PromQL query via HTTP API. | `<server-url> <promql-expression>` | `promtool query instant http://localhost:9090 'up'` | Tests query execution directly from the command line without opening a browser. |
| `promtool query range` | Executes range PromQL query | `<url> <expr> --start --end --step` | `promtool query range http://localhost:9090 'up' --start=1h` | Retrieves historical data points for a given PromQL query over time. |

### Diagnostic HTTP API Endpoints

Prometheus exposes operational HTTP API endpoints useful for automation and troubleshooting script workflows:

| Action / Endpoint | HTTP Method | Command Example | Expected Result / Purpose |
|---|---|---|---|
| **Hot Reload** | `POST` | `curl -X POST http://localhost:9090/-/reload` | Triggers zero-downtime configuration reload. |
| **Instant Query** | `GET` | `curl -s 'http://localhost:9090/api/v1/query?query=up' \| jq .` | Returns instant query execution results in JSON format. |
| **Scrape Targets State** | `GET` | `curl -s 'http://localhost:9090/api/v1/targets' \| jq '.data.activeTargets[].health'` | Dumps target scrape states, health metrics, and scrape error strings. |
| **TSDB Storage Flags** | `GET` | `curl -s 'http://localhost:9090/api/v1/status/flags' \| jq .` | Returns runtime command-line flags loaded into memory. |

### Essential PromQL Production Queries

```promql
# 1. Target Availability Check (1 = Up, 0 = Down)
up

# 2. Per-second rate of HTTP requests over 5-minute window
rate(http_requests_total[5m])

# 3. 99th Latency Percentile Calculation across services (Histogram)
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

# 4. Available RAM Ratio on Linux Host Nodes
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes

# 5. Overall CPU Usage Percentage across Nodes (Excluding Idle Mode)
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# 6. Aggregate HTTP 5xx Server Error Rate by Job
sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)

9. Logs, Monitoring and Diagnostics

Monitoring the health of the Prometheus server itself is essential to ensure operational continuity of your telemetry system.

Inspecting Service Logs

Prometheus outputs standard log records to stderr, which Systemd captures in the system journal.

# Tail Prometheus logs in real time
journalctl -u prometheus -f

# Inspect last 50 log lines without line truncation
journalctl -u prometheus -n 50 --no-pager

# Filter logs for errors or warnings
journalctl -u prometheus -p err..warning --since "1 hour ago"

Key Diagnostic Metrics (Prometheus Self-Monitoring)

Prometheus exposes its internal execution metrics at its own /metrics endpoint. Key internal metrics include:

Metric Name Type Diagnostic Focus & Meaning
prometheus_tsdb_storage_blocks_bytes Gauge Total disk storage used by TSDB blocks. Rapid growth indicates high cardinality or excessive retention.
prometheus_rule_evaluation_failures_total Counter Cumulative count of rule evaluation failures. Non-zero value indicates syntax or resource limits in PromQL rules.
scrape_duration_seconds Gauge/Histogram Duration of target scrape execution. High durations (>10s) indicate slow target response or oversized payloads.
prometheus_target_scrapes_exceeded_sample_limit_total Counter Count of target scrapes dropped due to exceeding sample_limit thresholds. Direct indicator of high-cardinality explosions.
ALERTS & ALERTS_FOR_STATE Gauge Tracks active alerts and pending durations within the Prometheus rule engine.

10. Troubleshooting Guide and Common Failure Scenarios

This section details step-by-step diagnostic workflows for common production issues.


Scenario 1: Target down (up == 0)

Symptoms

Grafana displays missing metric graphs or alerts fire indicating target endpoints are offline. Running up == 0 in PromQL returns active instance series.

Possible Causes

  1. Target service or node_exporter daemon is crashed or stopped.
  2. Firewall rules (iptables, ufw, cloud security groups) block port 9100/8080 from the Prometheus host IP.
  3. Target exporter is bound to 127.0.0.1 instead of 0.0.0.0.
  4. Incorrect metrics_path or host/port specified in prometheus.yml.

Investigation and Diagnosis

  1. Open Prometheus Web UI and navigate to Status -> Targets (http://localhost:9090/targets). Inspect the Error column:
  • connection refused: Daemon not running or wrong port.
  • context deadline exceeded: Firewall drop or slow network endpoint.
  • 404 Not Found: Incorrect metrics_path configured.
  1. Execute diagnostic CLI probes directly from the Prometheus server node:
# Test TCP socket network reachability
nc -zv target-host 9100

# Test HTTP metric collection directly
curl -i http://target-host:9100/metrics
  1. Log into target node and verify listening bindings:
ss -tlnp | grep 9100

Resolution

  • If bound to 127.0.0.1, update target service command-line arguments to listen on 0.0.0.0:9100.
  • Adjust security groups or local Linux firewall (sudo ufw allow 9100/tcp or sudo firewall-cmd --add-port=9100/tcp --permanent).
  • Correct configuration path or target IP in /etc/prometheus/prometheus.yml and trigger reload.

Verification

Run promtool query instant http://localhost:9090 'up{instance="target-host:9100"}' and verify value equals 1.


Scenario 2: Grafana Dashboard Shows “No Data”

Symptoms

Grafana panels render empty charts with “No data” warnings, while the underlying infrastructure is known to be operational.

Possible Causes

  1. Prometheus data source URL in Grafana is misconfigured or points to an unreachable endpoint.
  2. Query error in Grafana panel (typo in metric name, incorrect label filters, case sensitivity).
  3. Prometheus target scraping is down or failing at the source.
  4. PromQL query uses rate() on a Gauge metric instead of a Counter.
[ Grafana Dashboard "No Data" ]
              │
              ▼
    Is Prometheus Up? ──(No)──► Fix Prometheus / Targets Scrape
              │ (Yes)
              ▼
   Test Query in Prometheus /graph
              │
      ┌───────┴───────┐
   (Works)         (Fails)
      │               │
      ▼               ▼
Fix Grafana       Fix PromQL / Metric Name
Data Source       or Label Selectors

Investigation and Diagnosis

  1. Separate Grafana configuration from Prometheus target health.
  2. Copy the PromQL string from Grafana and execute it directly in the Prometheus Web UI (http://localhost:9090/graph) or Grafana Explore view.
  3. If Prometheus returns data but Grafana does not:
  • Navigate to Grafana Connections -> Data Sources -> Prometheus.
  • Test URL endpoint (e.g., http://prometheus-server:9090) and press Save & Test.
  1. If Prometheus UI returns empty series:
  • Verify metric existence using metric name search: {__name__=~"http_requests.*"}.
  • Check metric type: ensure counters use rate(metric[5m]) while gauges do not.

Resolution

  • Update Grafana Data Source settings with the correct Prometheus HTTP URL.
  • Fix PromQL syntax typos or case-sensitivity errors in label matchers.

Verification

Confirm panel renders data points in Grafana after saving panel settings.


Scenario 3: Scrape Timeout / Context Deadline Exceeded

Symptoms

Target status in Prometheus UI shows context deadline exceeded error. Target scrape_duration_seconds approaches or exceeds global scrape_timeout (default 10s).

Possible Causes

  1. Target application metric endpoint is performing expensive, slow calculations per request.
  2. Label cardinality explosion generates megabytes of metric payload text.
  3. High network latency or packet loss between Prometheus and target.

Investigation and Diagnosis

  1. Measure raw metrics payload size and response duration using curl:
curl -w "\nTime: %{time_total}s\nSize: %{size_download} bytes\n" -o /tmp/metrics.txt http://target:8080/metrics
  1. Inspect the file size of /tmp/metrics.txt. Files larger than 5-10 MB indicate a high-cardinality issue.
  2. Search target output for dynamic or unbounded labels (e.g., user IDs, session tokens, exact timestamps).

Resolution

  • Application Level: Remove high-cardinality labels from client instrumentation.
  • Prometheus Configuration: Drop noisy metrics before ingestion using metric_relabel_configs:
scrape_configs:
  - job_name: 'backend_api'
    scrape_timeout: 20s # Temporarily increase scrape timeout
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'http_request_duration_seconds_bucket'
        action: drop
``` *(Source: Reference 2)*

Verification

Monitor scrape_duration_seconds{job="backend_api"} in Prometheus UI and verify execution times fall safely below timeout thresholds.


Scenario 4: PromQL Query Returns Empty or Unexpected Results

Symptoms

PromQL queries yield no data points or show flat lines when metrics are expected to change.

Possible Causes

  1. Using rate() or increase() on a Gauge metric instead of a Counter.
  2. Filtering with strict label matchers that do not match available label values.
  3. Recording rule name differs from raw metric name.

Investigation and Diagnosis

  1. Check metric type in Prometheus UI or raw /metrics text output.
  2. Verify label values using promtool:
promtool query labels http://localhost:9090 'environment'
  1. Test query incrementally: start with bare metric name {__name__="http_requests_total"}, then append label selectors one by one.

Resolution

  • Remove rate() from Gauge metrics (gauges represent instant values; rate() requires monotonically increasing counters).
  • Align label key-value pairs with values present in /targets metadata.

Scenario 5: Config Reload Fails or Service Fails to Start

Symptoms

curl -X POST localhost:9090/-/reload returns 404 Not Found or HTTP 500 error, or systemctl start prometheus fails.

Possible Causes

  1. YAML indentation error or tab characters inside /etc/prometheus/prometheus.yml.
  2. Duplicate alert rule names across included rule files.
  3. --web.enable-lifecycle flag is missing from Systemd service startup options.

Investigation and Diagnosis

  1. Run syntax inspection commands using promtool:
promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/*.yml
  1. Inspect journal logs for exact line parsing errors:
journalctl -u prometheus -e --no-pager

Resolution

  • Fix YAML syntax errors (replace all tabs with spaces, fix indentation).
  • Append --web.enable-lifecycle flag to systemd startup options if /-/reload returns 404.
  • Run systemctl daemon-reload && systemctl restart prometheus.

Scenario 6: Disk Full / TSDB Storage Exhaustion

Symptoms

Host partition containing /var/lib/prometheus reaches 100% disk usage. Prometheus crashes or reports write errors in system logs.

Possible Causes

  1. Retention time (--storage.tsdb.retention.time) is configured too high for available storage.
  2. High cardinality metric ingestion spikes TSDB block storage requirements.

Investigation and Diagnosis

  1. Inspect TSDB block disk consumption:
du -sh /var/lib/prometheus/*
  1. Execute PromQL query to find memory and block byte allocations:
prometheus_tsdb_storage_blocks_bytes

Resolution

  1. Reduce Retention Period: Update retention setting in systemd unit file (e.g., --storage.tsdb.retention.time=7d).
  2. Set Storage Size Cap: Add size limit flag to prevent disk exhaustion (e.g., --storage.tsdb.retention.size=50GB).
  3. Clean old data or expand storage volume. Restart service and monitor Write-Ahead Log (WAL) replay in system logs.

11. Security and Best Practices

  1. Principle of Least Privilege: Run Prometheus under a unprivileged dedicated service user (prometheus) with read-only permissions on configuration files and write permissions restricted strictly to /var/lib/prometheus.
  2. API Protection: Do not expose raw Port 9090 directly to the public internet. Use a reverse proxy (e.g., NGINX or Caddy) with TLS termination and Basic Authentication for external access.
  3. Cardinality Governance: Enforce strict code review rules against placing dynamic values (e.g., IP addresses, email addresses, database auto-increment IDs) as metric labels.
  4. Scrape Network Isolation: Restrict target /metrics HTTP access using firewall rules so endpoints only respond to requests originating from internal Prometheus scraper IPs.

12. Performance and Resource Optimisation

Managing High-Cardinality Metrics

High cardinality is the leading cause of memory spikes and TSDB degradation in Prometheus.

Use promtool to analyze cardinality on live instances:

promtool tsdb analyze /var/lib/prometheus/

To drop high-cardinality label keys during scraping, configure metric_relabel_configs in target jobs:

scrape_configs:
  - job_name: 'app_metrics'
    static_configs:
      - targets: ['localhost:8080']
    metric_relabel_configs:
      # Strips specific unwanted high-cardinality labels
      - regex: 'user_id|session_token|client_ip'
        action: labeldrop
``` *(Source: Reference 2)*

### Recording Rules Optimization

Precompute computationally expensive PromQL expressions (such as complex histogram quantile aggregations) using Recording Rules to reduce Grafana dashboard rendering latency.

Example `/etc/prometheus/rules/recording_rules.yml`:
```yaml
groups:
  - name: web_performance_rules
    rules:
      - record: job:http_request_latency_99th:quantile
        expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
``` *(Source: Reference 3)*

---

# 13. Backup, Recovery and Maintenance

### Backing Up Prometheus Configuration and Rules

Configuration files should be managed in Git version control and backed up regularly:

```bash
# Backup configuration and alerting rules to archive
tar -czvf prometheus_config_backup_$(date +%F).tar.gz /etc/prometheus/
``` *(Source: Reference 1)*

### Creating TSDB Storage Snapshots

Prometheus supports consistent disk snapshots via its Admin API without taking the server offline:

```bash
# 1. Enable Admin API flag in service startup: --web.enable-admin-api
# 2. Trigger TSDB snapshot via API call
curl -X POST http://localhost:9090/api/v1/admin/tsdb/snapshot

# 3. Output returns snapshot location directory name:
# {"status":"success","data":{"name":"20261103T120000Z-7a8b9c"}}

# 4. Copy snapshot directory from TSDB data path to backup storage
sudo cp -r /var/lib/prometheus/snapshots/20261103T120000Z-7a8b9c /mnt/backups/

Crash Recovery and WAL Replay

If the host crashes or power cuts out unexpectedly, Prometheus reads its Write-Ahead Log (WAL) located at /var/lib/prometheus/wal upon startup. During this period, memory usage spikes and port 9090 may delay responding. Monitor system logs (journalctl -u prometheus -f) to track recovery progress before taking manual intervention steps.


14. Automation and Integration

Dynamic Service Discovery with File-Based SD (file_sd)

Rather than editing prometheus.yml and triggering service reloads every time infrastructure changes, use file-based service discovery (file_sd_configs).

In prometheus.yml:

scrape_configs:
  - job_name: 'dynamic_nodes'
    file_sd_configs:
      - files:
          - '/etc/prometheus/targets/*.json'
        refresh_interval: 1m
``` *(Source: Reference 1, 2)*

#### Target file `/etc/prometheus/targets/nodes.json`:
```json
[
  {
    "targets": ["10.0.2.10:9100", "10.0.2.11:9100"],
    "labels": {
      "environment": "production",
      "datacenter": "us-east-1"
    }
  }
]

Prometheus automatically monitors /etc/prometheus/targets/*.json for changes and updates target lists in real time without requiring explicit HTTP reloads.


15. Practical Labs and Exercises

Lab 1: Deploying node_exporter and Building a Host Metric Scrape Job

  • Objective: Install node_exporter on a target host, configure scraping, and query metrics in Prometheus.
  • Prerequisites: Functional Prometheus server instance running on Port 9090.
  • Instructions:
  1. Download and extract node_exporter on target host:
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar -xvf node_exporter-1.8.2.linux-amd64.tar.gz
sudo cp node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/
  1. Create service unit /etc/systemd/system/node_exporter.service and start daemon:
[Unit]
Description=Node Exporter
[Service]
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload && sudo systemctl enable --now node_exporter
  1. Update /etc/prometheus/prometheus.yml to include target localhost:9100.
  2. Reload Prometheus configuration: curl -X POST http://localhost:9090/-/reload.
  • Validation: Run up{job="node"} in Prometheus Web UI /graph and verify result value equals 1.

Lab 2: Authoring Alerting Rules and Verifying Syntax

  • Objective: Create an alert rule that triggers when target CPU utilization exceeds a specified threshold.
  • Instructions:
  1. Create file /etc/prometheus/rules/cpu_alerts.yml:
groups:
  - name: cpu_alerts
    rules:
      - alert: HighCpuUsage
        expr: 100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Host CPU utilization high"
``` *(Source: Reference 3)*
  1. Test file syntax using promtool check rules /etc/prometheus/rules/cpu_alerts.yml.
  • Validation: Confirm promtool returns SUCCESS.

16. Quick Reference and Cheat Sheet

Important Configuration Files

File / Location Purpose
/etc/prometheus/prometheus.yml Primary system configuration file.
/etc/prometheus/rules/*.yml Alerting and recording rule files.
/var/lib/prometheus/ Time-series database storage directory.

Important Log Locations

Log Source Access Command
Systemd Service Journal journalctl -u prometheus -n 50 --no-pager
Real-time Tail journalctl -u prometheus -f

Common Administrative Commands

Operational Task Command / Action
Validate Main Config promtool check config /etc/prometheus/prometheus.yml
Validate Rules File promtool check rules /etc/prometheus/rules/alerts.yml
Hot Reload Config curl -X POST http://localhost:9090/-/reload
Check Listening Ports `ss -tlnp
Check Service Status systemctl status prometheus

Troubleshooting Quick Reference

Symptom Initial Diagnostic Command / Probe
Target down (up == 0) curl -i http://target:9100/metrics or check /targets UI
Grafana “No Data” Run PromQL in Grafana Explore view & check data source URL
Slow Scrapes / Timeout Check target metric payload size (curl) & scrape_duration_seconds
Config Reload Error promtool check config /etc/prometheus/prometheus.yml

Important Network Ports

Port Protocol Usage / Purpose
9090 HTTP Prometheus Web UI and REST API
9100 HTTP node_exporter metric server
9093 HTTP Alertmanager Web UI and API
3000 HTTP Grafana Web Interface

17. Further Reading and References

  1. Official Prometheus Documentation: prometheus.io/docs (Source: Reference 1)
  2. PromQL Querying Basics: prometheus.io/docs/prometheus/latest/querying/basics/ (Source: Reference 1)
  3. Grafana Prometheus Data Source Guide: grafana.com/docs/grafana/latest/datasources/prometheus/ (Source: Reference 1)
  4. Prometheus Node Exporter GitHub Repository: github.com/prometheus/node_exporter (Source: Reference 1)
  5. Alertmanager Official Documentation: prometheus.io/docs/alerting/latest/alertmanager/ (Source: Reference 1)