1. Overview
Prometheus is an open-source monitoring system and time-series database designed specifically for modern infrastructure and cloud-native environments. It addresses key operational questions in production environments, such as identifying real-time error rates, detecting instances running out of disk space, or determining whether latency spiked immediately following a application deployment.
Applications and host exporters expose operational metrics in a standardized, line-oriented text format over HTTP. Prometheus periodically pulls (scrapes) these metrics on a configured time interval and stores the sampled metrics as multi-dimensional time series identified by metric names and key-value labels.
[ Application / Node Exporter ]
│ (Exposes HTTP /metrics)
▼
[ Prometheus Server (Scrapes & Stores TSDB) ]
│ │
▼ ▼
[ Alertmanager ] ──► Notifications [ Grafana ] ──► Dashboards & Exploration
The Modern Observability Stack
In production environments, Prometheus functions as the telemetry collection and alerting engine within a multi-tiered monitoring stack:
- Exporters & Applications: Expose operational status, host metrics, and runtime metrics via HTTP endpoints. (Source: Reference 1)
- Prometheus Server: Periodically scrapes target endpoints, evaluates alerting and recording rules, and persists time-series data to local disk. (Source: Reference 1)
- Grafana: Connects to Prometheus as an analytical data source to render visual dashboards, charts, and interactive tables. (Source: Reference 1)
- Alertmanager: Receives fired alerts from Prometheus, deduplicates them, groups related events, manages silences, and routes notifications to downstream channels like Slack, PagerDuty, or email. (Source: Reference 1)
Core Strengths and Operational Limitations
| Dimension | Prometheus Strengths | Operational Limitations |
|---|---|---|
| Data Collection | Active pull model simplifies target state verification and prevents unmonitored agent proliferation. | Requires direct network reachability or proxies to scrape target endpoints. |
| Data Model | Rich multi-dimensional data model using key-value labels for dynamic slicing and aggregation. | High-cardinality data (e.g., user IDs or IP addresses as labels) degrades TSDB performance and fills disk space rapidly. |
| Query Engine | PromQL enables complex mathematical calculations, rate calculations, and quantile aggregations. | Not designed for event logging, text searching, or distributed tracing. |
| Architecture | Autonomous single-binary server design without complex cluster dependencies. | Out-of-the-box local TSDB is intended for short-to-medium term retention (~15 days), requiring remote write integrations for long-term historical storage. |
2. Architecture and Mental Model
To effectively operate and troubleshoot Prometheus, system administrators must understand its core architecture, data collection engine, and internal lifecycle.
High-Level Architecture and Data Flow
Prometheus operates primarily on an active pull model. Rather than waiting for external agents to push metrics, Prometheus actively reaches out to configured target endpoints at regular intervals (scrape_interval).
flowchart TD
subgraph Targets ["Monitored Targets"]
APP["Instrumented App (/metrics)"]
NODE["Node Exporter (Port 9100)"]
PUSH["Pushgateway (Batch Jobs)"]
end
subgraph Server ["Prometheus Server"]
SD["Service Discovery Engine\n(File SD, K8s, Static)"]
SCRAPER["Scrape Engine\n(Interval Pull)"]
RELABEL["Relabeling Engine\n(Target & Metric Filters)"]
TSDB[("TSDB Storage\n(Local Disk / WAL)")]
RULES["Rule Engine\n(PromQL / Alerting)"]
API["Web UI & HTTP API\n(Port 9090)"]
end
subgraph Consumers ["Visualization & Alerting"]
AM["Alertmanager\n(Port 9093)"]
GRAFANA["Grafana UI\n(Port 3000)"]
end
SD -->|Target Lists| SCRAPER
SCRAPER -->|HTTP Scrape| APP
SCRAPER -->|HTTP Scrape| NODE
SCRAPER -->|HTTP Scrape| PUSH
SCRAPER --> RELABEL
RELABEL --> TSDB
RULES -->|Evaluate PromQL| TSDB
RULES -->|Fire Alerts| AM
GRAFANA -->|PromQL Queries| API
API -->|Read TSDB| TSDB
Main Components and Responsibilities
- Scrape Engine: Reads configured jobs and dynamic target lists, initiates HTTP
GETrequests to target metric paths (defaulting to/metrics), parses text metric formats, and records execution metrics likescrape_duration_secondsand target reachability (up). - TSDB (Time Series Database): Stores raw sample data (timestamp and float64 value pairs) efficiently on disk inside two-hour blocks with Write-Ahead Logging (WAL) to prevent data loss across crashes.
- Service Discovery (SD): Interacts with infrastructure providers (Kubernetes API, cloud providers, file-based discovery) to automatically detect added or removed scrape targets.
- Relabeling Engine: Applies transformation rules before ingestion to rewrite metric labels, filter noisy endpoints, or drop high-cardinality labels.
- Rule Engine: Periodically executes PromQL expressions for Recording Rules (pre-computing expensive queries) and Alerting Rules (evaluating failure conditions).
- Web UI and API: Serves built-in diagnostic pages (
/targets,/graph,/status), accepts configuration reloads (/-/reload), and handles PromQL queries from Grafana or CLI utilities.
Network Ports & Communication Map
- Port 9090: Prometheus Server Web UI and REST API. (Source: Reference 1, 3)
- Port 9100:
node_exporterdefault metric port. (Source: Reference 2, 3) - Port 9093: Alertmanager Web UI and API. (Source: Reference 2)
- Port 3000: Grafana Web Interface. (Source: Reference 1, 3)
3. Core Concepts and Terminology
Understanding Prometheus requires familiarity with its fundamental data structures, metric types, and PromQL execution mechanics.
Metric Types
Prometheus supports four primary metric types in its client libraries:
- Counter: A cumulative metric that only increases or resets to zero upon server restart. Used for measuring total requests, error counts, or completed tasks. Always apply
rate()orirate()functions when graphing counters in PromQL. - Gauge: A metric representing a single numerical value that can arbitrarily go up or down. Used for current memory usage, CPU temperature, queue depth, or disk availability (
node_memory_MemAvailable_bytes). - Histogram: Samples observations (such as request durations or response sizes) and counts them in configurable bucket ranges. Also provides a total sum of observed values and total count. Essential for calculating percentiles (e.g., 99th percentile latency) using
histogram_quantile(). - Summary: Similar to histograms, summaries calculate client-side quantiles over a sliding time window. While accurate on individual instances, summary quantiles cannot be aggregated across multiple server instances in PromQL.
Metric Syntax and Label Cardinality
A Prometheus metric consists of a metric name, key-value pair labels, and a numerical payload value:
http_requests_total{method="GET", handler="/api/v1/users", status="200"} 1043
- Labels: Dimensions attached to time series that allow granular PromQL filtering, grouping, and multi-dimensional analysis. (Source: Reference 1)
- Label Cardinality: The total number of unique time series generated by combining all label key-value pairs. High cardinality occurs when unbounded data (such as user IDs, UUIDs, container IP addresses, or raw request URLs) is added as labels. This causes exponential TSDB growth, high memory utilization, scrape timeouts, and database slowdowns.
Core Terminology Glossary
| Term | Definition | Operational Relevance |
|---|---|---|
| Scrape Interval | Frequency at which Prometheus fetches metrics from targets. | Configured globally or per job (scrape_interval: 15s). |
| Target | An individual HTTP endpoint scraped by Prometheus. | Tracked via the up metric (1 = healthy, 0 = down). |
| Pushgateway | An intermediary metric cache for short-lived batch jobs. | Short-lived jobs push metrics to Pushgateway, which Prometheus scrapes. |
| Recording Rule | Pre-computed PromQL expressions evaluated continuously. | Reduces computational overhead when rendering complex Grafana dashboards. |
| Alerting Rule | PromQL expression evaluated to trigger operational alerts. | Evaluated on schedule; firing alerts are pushed directly to Alertmanager. |
| Relabeling | Label modification step applied before storing time series. | Used to rewrite metric paths, drop high-cardinality labels, or map dynamic discovery metadata. |
4. Installation and Version Management
Prometheus is distributed as a single static Go binary, making installation straightforward on Linux distributions.
Installing Prometheus on Enterprise Linux (RHEL/Rocky/AlmaLinux & Ubuntu/Debian)
Method 1: Installing via Official Standalone Binary Releases (Recommended for Control)
# 1. Create dedicated system user and directories (Linux Best Practice)
sudo useradd --no-create-home --shell /bin/false prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus
# 2. Download latest stable binary package
ARCH="linux-amd64"
VERSION="2.54.1" # Example stable release
wget https://github.com/prometheus/prometheus/releases/download/v${VERSION}/prometheus-${VERSION}.${ARCH}.tar.gz
# 3. Extract and distribute binaries and assets
tar -xvf prometheus-${VERSION}.${ARCH}.tar.gz
cd prometheus-${VERSION}.${ARCH}
sudo cp prometheus promtool /usr/local/bin/
sudo cp -r consoles console_libraries /etc/prometheus/
sudo cp prometheus.yml /etc/prometheus/prometheus.yml
# 4. Set strict file ownership
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool
Method 2: Installing via System Package Managers
On Ubuntu/Debian systems:
sudo apt-get update
sudo apt-get install -y prometheus promtool
On Enterprise Linux (RHEL / EPEL):
sudo dnf install -y epel-release
sudo dnf install -y prometheus2
Verifying Binary Installation and Software Version
To confirm successful installation and inspect built-in features, execute:
# Verify Prometheus server binary version
prometheus --version
# Verify promtool CLI utility version
promtool --version
Default Filesystem Layout
- Binary executables:
/usr/local/bin/prometheusand/usr/local/bin/promtool(or/usr/bin/for package installs). - Configuration directory:
/etc/prometheus/. - TSDB Storage directory:
/var/lib/prometheus/.
5. Configuration Files and Directory Structure
Prometheus uses YAML syntax for its configuration files. Formatting is strictly space-indented (tabs are invalid and cause startup failures).
Comprehensive Path Reference
| File / Directory | Purpose | Important Details |
|---|---|---|
/etc/prometheus/prometheus.yml |
Main configuration file. | Defines global scrape intervals, rule files, alertmanager targets, and scrape jobs. |
/etc/prometheus/rules/*.yml |
Alerting and recording rule files. | Grouped YAML definitions for alert conditions and pre-computed metrics. |
/var/lib/prometheus/ |
Primary TSDB storage directory. | Contains block directories, chunk files, and wal/ (Write-Ahead Log). |
/etc/systemd/system/prometheus.service |
Systemd service definition unit file. | Defines daemon flags, startup parameters, user context, and resource limits. |
/usr/local/bin/promtool |
CLI verification and debugging tool. | Used to validate config syntax (promtool check config) before reloading services. |
Main Configuration Structure (/etc/prometheus/prometheus.yml)
The standard configuration file divides settings into clear operational blocks:
# Global defaults applied across all scrape jobs unless overridden
global:
scrape_interval: 15s # How frequently Prometheus scrapes targets (Default: 1m)
evaluation_interval: 15s # How frequently Prometheus evaluates rules
scrape_timeout: 10s # Scrape request timeout threshold
# Alertmanager configuration
alerting:
alertmanagers:
- static_configs:
- targets:
- 'localhost:9093' # Alertmanager endpoint
# Rule file paths loaded into the rule evaluation engine
rule_files:
- "/etc/prometheus/rules/*.yml"
# Scrape targets configuration
scrape_configs:
# Internal self-monitoring job
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
# Linux host infrastructure monitoring job
- job_name: 'node_exporter'
scrape_interval: 10s
static_configs:
- targets: ['192.168.1.10:9100', '192.168.1.11:9100']
# Custom application monitoring job
- job_name: 'backend_api'
metrics_path: '/metrics'
scheme: 'http'
static_configs:
- targets: ['app1.example.com:8080']
``` *(Source: Reference 1, 3)*
### Configuration Precedence and Modification Mechanics
1. **CLI Flags Overwrite Configuration Defaults**: Flags passed in `ExecStart` inside the systemd service file (such as storage retention or network flags) take precedence over general daemon settings.
2. **Configuration Validation**: ALWAYS validate configuration syntax prior to applying changes using `promtool check config` to prevent service crashes.
3. **Dynamic Reloading**: Prometheus supports zero-downtime configuration reloads without restarting the process.
---
# 6. Service Management and Common Operations
Prometheus is managed as a standard background service using Systemd.
### Creating the Systemd Service Unit (`/etc/systemd/system/prometheus.service`)
Create the systemd service file to control execution parameters and flags:
```ini
[Unit]
Description=Prometheus Time Series Database & Monitoring Engine
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus/ \
--storage.tsdb.retention.time=15d \
--web.enable-lifecycle \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=5s
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
``` *(Source: Reference 1)*
> **Security & Operational Note**: The `--web.enable-lifecycle` flag enables the HTTP POST `/-/reload` endpoint. Ensure the web port is protected by network firewalls or reverse proxy authentication if this flag is enabled.
### Common Service Management Commands
```bash
# Reload systemd configuration manager after editing service files
sudo systemctl daemon-reload
# Enable Prometheus service to start automatically on system boot
sudo systemctl enable prometheus
# Start the Prometheus monitoring daemon
sudo systemctl start prometheus
# Inspect execution status, PID, and recent log outputs
systemctl status prometheus
# Check that Prometheus is actively listening on network port 9090
ss -tlnp | grep 9090
``` *(Source: Reference 3)*
### Reloading Configuration vs. Restarting Service
| Operation | Command | Impact | Operational Guidance |
|---|---|---|---|
| **Hot Reload** | `curl -X POST http://localhost:9090/-/reload` or `systemctl reload prometheus` | **Zero Downtime**. Re-reads config and rule files without interrupting metric scraping or TSDB memory locks. | Use for adding target endpoints, modifying alert thresholds, or adjusting scrape intervals. Requires `--web.enable-lifecycle`. |
| **Full Restart** | `systemctl restart prometheus` | **Brief Disruption**. Stops execution, flushes TSDB state, closes sockets, and restarts process. | Required when modifying command-line flags (e.g., retention periods, TSDB paths, or listening ports). |
---
# 7. Common Administrative Tasks and Workflows
### Workflow 1: Adding a New Target to Prometheus Scrape Configurations
#### Objective
Add a fleet of Linux servers running `node_exporter` (port 9100) to Prometheus scraping.
#### Procedure
1. Edit `/etc/prometheus/prometheus.yml`:
```yaml
scrape_configs:
- job_name: 'node_infrastructure'
scrape_interval: 15s
static_configs:
- targets:
- '10.0.1.50:9100'
- '10.0.1.51:9100'
labels:
environment: 'production'
``` *(Source: Reference 3)*
2. Validate syntax using `promtool`:
```bash
promtool check config /etc/prometheus/prometheus.yml
``` *(Source: Reference 2, 3)*
3. Trigger dynamic configuration reload:
```bash
curl -X POST http://localhost:9090/-/reload
``` *(Source: Reference 1, 3)*
4. Confirm target registration via HTTP API:
```bash
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .discoveredLabels.job, health: .health, instance: .discoveredLabels.__address__}'
``` *(Source: Reference 3)*
### Workflow 2: Configuring an Alerting Rule for Node Outages
#### Objective
Detect when host targets go offline (`up == 0`) for more than 5 minutes and route notifications to Alertmanager.
#### Procedure
1. Create rule file `/etc/prometheus/rules/host_alerts.yml`:
```yaml
groups:
- name: host_availability_alerts
rules:
- alert: InstanceDown
expr: up == 0
for: 5m
labels:
severity: critical
annotations:
summary: "Prometheus target system is offline"
description: "Target instance {{ $labels.instance }} of job {{ $labels.job }} has been unreachable for more than 5 minutes."
``` *(Source: Reference 3)*
2. Validate rule syntax:
```bash
promtool check rules /etc/prometheus/rules/host_alerts.yml
``` *(Source: Reference 2, 3)*
3. Verify that `/etc/prometheus/prometheus.yml` includes the rule file path:
```yaml
rule_files:
- "/etc/prometheus/rules/*.yml"
``` *(Source: Reference 1)*
4. Reload configuration and verify alerts tab in UI (`http://localhost:9090/alerts`).
---
# 8. CLI Command Reference and Cheatsheet
### `promtool` Diagnostic Utility Reference
`promtool` is the official command-line verification and debugging tool shipped with Prometheus.
| Command Syntax | Purpose | Key Arguments / Flags | Practical Example | Operational Explanation |
|---|---|---|---|---|
| `promtool check config` | Validates configuration syntax. | `<config-file-path>` | `promtool check config /etc/prometheus/prometheus.yml` | Inspects YAML structure, target block definitions, and referenced paths. Exits with zero on success. |
| `promtool check rules` | Validates rule syntax. | `<rule-file-path>` | `promtool check rules /etc/prometheus/rules/*.yml` | Validates syntax of alerting and recording rules, verifying required fields like `expr`. |
| `promtool query instant` | Executes instant PromQL query via HTTP API. | `<server-url> <promql-expression>` | `promtool query instant http://localhost:9090 'up'` | Tests query execution directly from the command line without opening a browser. |
| `promtool query range` | Executes range PromQL query | `<url> <expr> --start --end --step` | `promtool query range http://localhost:9090 'up' --start=1h` | Retrieves historical data points for a given PromQL query over time. |
### Diagnostic HTTP API Endpoints
Prometheus exposes operational HTTP API endpoints useful for automation and troubleshooting script workflows:
| Action / Endpoint | HTTP Method | Command Example | Expected Result / Purpose |
|---|---|---|---|
| **Hot Reload** | `POST` | `curl -X POST http://localhost:9090/-/reload` | Triggers zero-downtime configuration reload. |
| **Instant Query** | `GET` | `curl -s 'http://localhost:9090/api/v1/query?query=up' \| jq .` | Returns instant query execution results in JSON format. |
| **Scrape Targets State** | `GET` | `curl -s 'http://localhost:9090/api/v1/targets' \| jq '.data.activeTargets[].health'` | Dumps target scrape states, health metrics, and scrape error strings. |
| **TSDB Storage Flags** | `GET` | `curl -s 'http://localhost:9090/api/v1/status/flags' \| jq .` | Returns runtime command-line flags loaded into memory. |
### Essential PromQL Production Queries
```promql
# 1. Target Availability Check (1 = Up, 0 = Down)
up
# 2. Per-second rate of HTTP requests over 5-minute window
rate(http_requests_total[5m])
# 3. 99th Latency Percentile Calculation across services (Histogram)
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
# 4. Available RAM Ratio on Linux Host Nodes
node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
# 5. Overall CPU Usage Percentage across Nodes (Excluding Idle Mode)
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
# 6. Aggregate HTTP 5xx Server Error Rate by Job
sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
9. Logs, Monitoring and Diagnostics
Monitoring the health of the Prometheus server itself is essential to ensure operational continuity of your telemetry system.
Inspecting Service Logs
Prometheus outputs standard log records to stderr, which Systemd captures in the system journal.
# Tail Prometheus logs in real time
journalctl -u prometheus -f
# Inspect last 50 log lines without line truncation
journalctl -u prometheus -n 50 --no-pager
# Filter logs for errors or warnings
journalctl -u prometheus -p err..warning --since "1 hour ago"
Key Diagnostic Metrics (Prometheus Self-Monitoring)
Prometheus exposes its internal execution metrics at its own /metrics endpoint. Key internal metrics include:
| Metric Name | Type | Diagnostic Focus & Meaning |
|---|---|---|
prometheus_tsdb_storage_blocks_bytes |
Gauge | Total disk storage used by TSDB blocks. Rapid growth indicates high cardinality or excessive retention. |
prometheus_rule_evaluation_failures_total |
Counter | Cumulative count of rule evaluation failures. Non-zero value indicates syntax or resource limits in PromQL rules. |
scrape_duration_seconds |
Gauge/Histogram | Duration of target scrape execution. High durations (>10s) indicate slow target response or oversized payloads. |
prometheus_target_scrapes_exceeded_sample_limit_total |
Counter | Count of target scrapes dropped due to exceeding sample_limit thresholds. Direct indicator of high-cardinality explosions. |
ALERTS & ALERTS_FOR_STATE |
Gauge | Tracks active alerts and pending durations within the Prometheus rule engine. |
10. Troubleshooting Guide and Common Failure Scenarios
This section details step-by-step diagnostic workflows for common production issues.
Scenario 1: Target down (up == 0)
Symptoms
Grafana displays missing metric graphs or alerts fire indicating target endpoints are offline. Running up == 0 in PromQL returns active instance series.
Possible Causes
- Target service or
node_exporterdaemon is crashed or stopped. - Firewall rules (iptables, ufw, cloud security groups) block port 9100/8080 from the Prometheus host IP.
- Target exporter is bound to
127.0.0.1instead of0.0.0.0. - Incorrect
metrics_pathor host/port specified inprometheus.yml.
Investigation and Diagnosis
- Open Prometheus Web UI and navigate to
Status -> Targets(http://localhost:9090/targets). Inspect the Error column:
connection refused: Daemon not running or wrong port.context deadline exceeded: Firewall drop or slow network endpoint.404 Not Found: Incorrectmetrics_pathconfigured.
- Execute diagnostic CLI probes directly from the Prometheus server node:
# Test TCP socket network reachability
nc -zv target-host 9100
# Test HTTP metric collection directly
curl -i http://target-host:9100/metrics
- Log into target node and verify listening bindings:
ss -tlnp | grep 9100
Resolution
- If bound to
127.0.0.1, update target service command-line arguments to listen on0.0.0.0:9100. - Adjust security groups or local Linux firewall (
sudo ufw allow 9100/tcporsudo firewall-cmd --add-port=9100/tcp --permanent). - Correct configuration path or target IP in
/etc/prometheus/prometheus.ymland trigger reload.
Verification
Run promtool query instant http://localhost:9090 'up{instance="target-host:9100"}' and verify value equals 1.
Scenario 2: Grafana Dashboard Shows “No Data”
Symptoms
Grafana panels render empty charts with “No data” warnings, while the underlying infrastructure is known to be operational.
Possible Causes
- Prometheus data source URL in Grafana is misconfigured or points to an unreachable endpoint.
- Query error in Grafana panel (typo in metric name, incorrect label filters, case sensitivity).
- Prometheus target scraping is down or failing at the source.
- PromQL query uses
rate()on a Gauge metric instead of a Counter.
[ Grafana Dashboard "No Data" ]
│
▼
Is Prometheus Up? ──(No)──► Fix Prometheus / Targets Scrape
│ (Yes)
▼
Test Query in Prometheus /graph
│
┌───────┴───────┐
(Works) (Fails)
│ │
▼ ▼
Fix Grafana Fix PromQL / Metric Name
Data Source or Label Selectors
Investigation and Diagnosis
- Separate Grafana configuration from Prometheus target health.
- Copy the PromQL string from Grafana and execute it directly in the Prometheus Web UI (
http://localhost:9090/graph) or Grafana Explore view. - If Prometheus returns data but Grafana does not:
- Navigate to Grafana
Connections -> Data Sources -> Prometheus. - Test URL endpoint (e.g.,
http://prometheus-server:9090) and press Save & Test.
- If Prometheus UI returns empty series:
- Verify metric existence using metric name search:
{__name__=~"http_requests.*"}. - Check metric type: ensure counters use
rate(metric[5m])while gauges do not.
Resolution
- Update Grafana Data Source settings with the correct Prometheus HTTP URL.
- Fix PromQL syntax typos or case-sensitivity errors in label matchers.
Verification
Confirm panel renders data points in Grafana after saving panel settings.
Scenario 3: Scrape Timeout / Context Deadline Exceeded
Symptoms
Target status in Prometheus UI shows context deadline exceeded error. Target scrape_duration_seconds approaches or exceeds global scrape_timeout (default 10s).
Possible Causes
- Target application metric endpoint is performing expensive, slow calculations per request.
- Label cardinality explosion generates megabytes of metric payload text.
- High network latency or packet loss between Prometheus and target.
Investigation and Diagnosis
- Measure raw metrics payload size and response duration using
curl:
curl -w "\nTime: %{time_total}s\nSize: %{size_download} bytes\n" -o /tmp/metrics.txt http://target:8080/metrics
- Inspect the file size of
/tmp/metrics.txt. Files larger than 5-10 MB indicate a high-cardinality issue. - Search target output for dynamic or unbounded labels (e.g., user IDs, session tokens, exact timestamps).
Resolution
- Application Level: Remove high-cardinality labels from client instrumentation.
- Prometheus Configuration: Drop noisy metrics before ingestion using
metric_relabel_configs:
scrape_configs:
- job_name: 'backend_api'
scrape_timeout: 20s # Temporarily increase scrape timeout
metric_relabel_configs:
- source_labels: [__name__]
regex: 'http_request_duration_seconds_bucket'
action: drop
``` *(Source: Reference 2)*
Verification
Monitor scrape_duration_seconds{job="backend_api"} in Prometheus UI and verify execution times fall safely below timeout thresholds.
Scenario 4: PromQL Query Returns Empty or Unexpected Results
Symptoms
PromQL queries yield no data points or show flat lines when metrics are expected to change.
Possible Causes
- Using
rate()orincrease()on a Gauge metric instead of a Counter. - Filtering with strict label matchers that do not match available label values.
- Recording rule name differs from raw metric name.
Investigation and Diagnosis
- Check metric type in Prometheus UI or raw
/metricstext output. - Verify label values using
promtool:
promtool query labels http://localhost:9090 'environment'
- Test query incrementally: start with bare metric name
{__name__="http_requests_total"}, then append label selectors one by one.
Resolution
- Remove
rate()from Gauge metrics (gauges represent instant values;rate()requires monotonically increasing counters). - Align label key-value pairs with values present in
/targetsmetadata.
Scenario 5: Config Reload Fails or Service Fails to Start
Symptoms
curl -X POST localhost:9090/-/reload returns 404 Not Found or HTTP 500 error, or systemctl start prometheus fails.
Possible Causes
- YAML indentation error or tab characters inside
/etc/prometheus/prometheus.yml. - Duplicate alert rule names across included rule files.
--web.enable-lifecycleflag is missing from Systemd service startup options.
Investigation and Diagnosis
- Run syntax inspection commands using
promtool:
promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/*.yml
- Inspect journal logs for exact line parsing errors:
journalctl -u prometheus -e --no-pager
Resolution
- Fix YAML syntax errors (replace all tabs with spaces, fix indentation).
- Append
--web.enable-lifecycleflag to systemd startup options if/-/reloadreturns 404. - Run
systemctl daemon-reload && systemctl restart prometheus.
Scenario 6: Disk Full / TSDB Storage Exhaustion
Symptoms
Host partition containing /var/lib/prometheus reaches 100% disk usage. Prometheus crashes or reports write errors in system logs.
Possible Causes
- Retention time (
--storage.tsdb.retention.time) is configured too high for available storage. - High cardinality metric ingestion spikes TSDB block storage requirements.
Investigation and Diagnosis
- Inspect TSDB block disk consumption:
du -sh /var/lib/prometheus/*
- Execute PromQL query to find memory and block byte allocations:
prometheus_tsdb_storage_blocks_bytes
Resolution
- Reduce Retention Period: Update retention setting in systemd unit file (e.g.,
--storage.tsdb.retention.time=7d). - Set Storage Size Cap: Add size limit flag to prevent disk exhaustion (e.g.,
--storage.tsdb.retention.size=50GB). - Clean old data or expand storage volume. Restart service and monitor Write-Ahead Log (WAL) replay in system logs.
11. Security and Best Practices
- Principle of Least Privilege: Run Prometheus under a unprivileged dedicated service user (
prometheus) with read-only permissions on configuration files and write permissions restricted strictly to/var/lib/prometheus. - API Protection: Do not expose raw Port 9090 directly to the public internet. Use a reverse proxy (e.g., NGINX or Caddy) with TLS termination and Basic Authentication for external access.
- Cardinality Governance: Enforce strict code review rules against placing dynamic values (e.g., IP addresses, email addresses, database auto-increment IDs) as metric labels.
- Scrape Network Isolation: Restrict target
/metricsHTTP access using firewall rules so endpoints only respond to requests originating from internal Prometheus scraper IPs.
12. Performance and Resource Optimisation
Managing High-Cardinality Metrics
High cardinality is the leading cause of memory spikes and TSDB degradation in Prometheus.
Use promtool to analyze cardinality on live instances:
promtool tsdb analyze /var/lib/prometheus/
To drop high-cardinality label keys during scraping, configure metric_relabel_configs in target jobs:
scrape_configs:
- job_name: 'app_metrics'
static_configs:
- targets: ['localhost:8080']
metric_relabel_configs:
# Strips specific unwanted high-cardinality labels
- regex: 'user_id|session_token|client_ip'
action: labeldrop
``` *(Source: Reference 2)*
### Recording Rules Optimization
Precompute computationally expensive PromQL expressions (such as complex histogram quantile aggregations) using Recording Rules to reduce Grafana dashboard rendering latency.
Example `/etc/prometheus/rules/recording_rules.yml`:
```yaml
groups:
- name: web_performance_rules
rules:
- record: job:http_request_latency_99th:quantile
expr: histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, job))
``` *(Source: Reference 3)*
---
# 13. Backup, Recovery and Maintenance
### Backing Up Prometheus Configuration and Rules
Configuration files should be managed in Git version control and backed up regularly:
```bash
# Backup configuration and alerting rules to archive
tar -czvf prometheus_config_backup_$(date +%F).tar.gz /etc/prometheus/
``` *(Source: Reference 1)*
### Creating TSDB Storage Snapshots
Prometheus supports consistent disk snapshots via its Admin API without taking the server offline:
```bash
# 1. Enable Admin API flag in service startup: --web.enable-admin-api
# 2. Trigger TSDB snapshot via API call
curl -X POST http://localhost:9090/api/v1/admin/tsdb/snapshot
# 3. Output returns snapshot location directory name:
# {"status":"success","data":{"name":"20261103T120000Z-7a8b9c"}}
# 4. Copy snapshot directory from TSDB data path to backup storage
sudo cp -r /var/lib/prometheus/snapshots/20261103T120000Z-7a8b9c /mnt/backups/
Crash Recovery and WAL Replay
If the host crashes or power cuts out unexpectedly, Prometheus reads its Write-Ahead Log (WAL) located at /var/lib/prometheus/wal upon startup. During this period, memory usage spikes and port 9090 may delay responding. Monitor system logs (journalctl -u prometheus -f) to track recovery progress before taking manual intervention steps.
14. Automation and Integration
Dynamic Service Discovery with File-Based SD (file_sd)
Rather than editing prometheus.yml and triggering service reloads every time infrastructure changes, use file-based service discovery (file_sd_configs).
In prometheus.yml:
scrape_configs:
- job_name: 'dynamic_nodes'
file_sd_configs:
- files:
- '/etc/prometheus/targets/*.json'
refresh_interval: 1m
``` *(Source: Reference 1, 2)*
#### Target file `/etc/prometheus/targets/nodes.json`:
```json
[
{
"targets": ["10.0.2.10:9100", "10.0.2.11:9100"],
"labels": {
"environment": "production",
"datacenter": "us-east-1"
}
}
]
Prometheus automatically monitors /etc/prometheus/targets/*.json for changes and updates target lists in real time without requiring explicit HTTP reloads.
15. Practical Labs and Exercises
Lab 1: Deploying node_exporter and Building a Host Metric Scrape Job
- Objective: Install
node_exporteron a target host, configure scraping, and query metrics in Prometheus. - Prerequisites: Functional Prometheus server instance running on Port 9090.
- Instructions:
- Download and extract
node_exporteron target host:
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar -xvf node_exporter-1.8.2.linux-amd64.tar.gz
sudo cp node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/
- Create service unit
/etc/systemd/system/node_exporter.serviceand start daemon:
[Unit]
Description=Node Exporter
[Service]
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload && sudo systemctl enable --now node_exporter
- Update
/etc/prometheus/prometheus.ymlto include targetlocalhost:9100. - Reload Prometheus configuration:
curl -X POST http://localhost:9090/-/reload.
- Validation: Run
up{job="node"}in Prometheus Web UI/graphand verify result value equals1.
Lab 2: Authoring Alerting Rules and Verifying Syntax
- Objective: Create an alert rule that triggers when target CPU utilization exceeds a specified threshold.
- Instructions:
- Create file
/etc/prometheus/rules/cpu_alerts.yml:
groups:
- name: cpu_alerts
rules:
- alert: HighCpuUsage
expr: 100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
for: 2m
labels:
severity: warning
annotations:
summary: "Host CPU utilization high"
``` *(Source: Reference 3)*
- Test file syntax using
promtool check rules /etc/prometheus/rules/cpu_alerts.yml.
- Validation: Confirm
promtoolreturnsSUCCESS.
16. Quick Reference and Cheat Sheet
Important Configuration Files
| File / Location | Purpose |
|---|---|
/etc/prometheus/prometheus.yml |
Primary system configuration file. |
/etc/prometheus/rules/*.yml |
Alerting and recording rule files. |
/var/lib/prometheus/ |
Time-series database storage directory. |
Important Log Locations
| Log Source | Access Command |
|---|---|
| Systemd Service Journal | journalctl -u prometheus -n 50 --no-pager |
| Real-time Tail | journalctl -u prometheus -f |
Common Administrative Commands
| Operational Task | Command / Action |
|---|---|
| Validate Main Config | promtool check config /etc/prometheus/prometheus.yml |
| Validate Rules File | promtool check rules /etc/prometheus/rules/alerts.yml |
| Hot Reload Config | curl -X POST http://localhost:9090/-/reload |
| Check Listening Ports | `ss -tlnp |
| Check Service Status | systemctl status prometheus |
Troubleshooting Quick Reference
| Symptom | Initial Diagnostic Command / Probe |
|---|---|
Target down (up == 0) |
curl -i http://target:9100/metrics or check /targets UI |
| Grafana “No Data” | Run PromQL in Grafana Explore view & check data source URL |
| Slow Scrapes / Timeout | Check target metric payload size (curl) & scrape_duration_seconds |
| Config Reload Error | promtool check config /etc/prometheus/prometheus.yml |
Important Network Ports
| Port | Protocol | Usage / Purpose |
|---|---|---|
| 9090 | HTTP | Prometheus Web UI and REST API |
| 9100 | HTTP | node_exporter metric server |
| 9093 | HTTP | Alertmanager Web UI and API |
| 3000 | HTTP | Grafana Web Interface |
17. Further Reading and References
- Official Prometheus Documentation: prometheus.io/docs (Source: Reference 1)
- PromQL Querying Basics: prometheus.io/docs/prometheus/latest/querying/basics/ (Source: Reference 1)
- Grafana Prometheus Data Source Guide: grafana.com/docs/grafana/latest/datasources/prometheus/ (Source: Reference 1)
- Prometheus Node Exporter GitHub Repository: github.com/prometheus/node_exporter (Source: Reference 1)
- Alertmanager Official Documentation: prometheus.io/docs/alerting/latest/alertmanager/ (Source: Reference 1)