Design an Anomaly Detection System like PagerDuty
Table of Contents
- Requirements Clarification
- Back-of-Envelope Calculations
- System Interface Design
- High-Level Architecture
- Core Component Design
- Data Model
- Key Design Decisions
- Scalability Strategy
- Curveball Scenarios
- Failure Modes and Mitigations
1. Requirements Clarification
1.1 Functional Requirements
Metric Collection & Monitoring
- Collect metrics from multiple sources: agents, APIs, integrations (Datadog, Prometheus, New Relic, custom)
- Support numeric metrics, logs, traces, and synthetic checks
- Configurable collection intervals (15s, 30s, 60s, 300s)
- Metric tagging and dimension support
Threshold-Based Alerting
- Static thresholds: absolute values, percentage changes
- Multi-window thresholds: “if CPU > 90% for 5 min AND > 80% for 15 min”
- Baseline thresholds: compare against historical baseline
- Composite conditions: AND/OR logic across multiple metrics
ML-Based Anomaly Detection
- Automatic baseline learning per metric
- Seasonal pattern detection (daily, weekly, monthly cycles)
- Change point detection for sudden shifts
- Forecasting-based anomaly detection
- Noise reduction and signal smoothing
Incident Management
- Incident creation from alerts with automatic deduplication
- Incident lifecycle: triggered → acknowledged → resolved
- Incident grouping: related alerts grouped into single incident
- Incident timeline with all events and actions
- Post-incident review and root cause analysis
On-Call Scheduling
- Rotating schedules with customizable patterns
- Escalation policies: multi-level escalation with timeouts
- Override and swap management
- Timezone-aware scheduling
- Integration with calendar systems
Notification & Communication
- Multi-channel notifications: push, SMS, email, voice call, Slack, Teams
- Escalation: if not acknowledged within timeout, escalate to next level
- Incident bridge calls: conference bridge for incident response
- Status updates and stakeholder notifications
Integrations
- 200+ integrations: monitoring tools, ticketing systems, chat platforms
- Webhook support for custom integrations
- Events API for programmatic incident management
- Change management integration (correlate incidents with deployments)
1.2 Non-Functional Requirements
| Requirement | Target | Rationale |
|---|---|---|
| Alert-to-Notification Latency | < 30 seconds (p99) | Critical for incident response |
| Metric Ingestion Throughput | 10M+ metrics/sec | Enterprise monitoring scale |
| Anomaly Detection Latency | < 60 seconds | Near-real-time detection |
| System Availability | 99.99% | Mission-critical for operations |
| Notification Delivery | 99.9% within 30 sec | On-call reliability |
| Data Retention | 1 year metrics, 2 years incidents | Compliance and analysis |
| Multi-Tenancy | Strong isolation | Enterprise security |
| Scalability | 100K+ customers, 10M+ metrics | Enterprise market |
2. Back-of-Envelope Calculations
2.1 Assumption Sets
| Parameter | Conservative | Realistic | Aggressive |
|---|---|---|---|
| Customers | 10K | 100K | 500K |
| Metrics/customer | 1,000 | 5,000 | 20,000 |
| Collection interval | 60s | 30s | 15s |
| Total metric points/sec | 167K | 1.67M | 6.67M |
| Peak metric points/sec (3×) | 500K | 5M | 20M |
| Avg metric payload | 200 bytes | 300 bytes | 500 bytes |
| Daily metric volume | 14.4 GB | 432 GB | 2.88 TB |
| Monthly metric volume | 432 GB | 12.96 TB | 86.4 TB |
2.2 Alert Volume Estimates
| Parameter | Realistic | Notes |
|---|---|---|
| Alerts/day | 10M | 0.6% of metric points trigger alerts |
| Incidents/day | 500K | 5% of alerts become incidents (after dedup) |
| Active incidents | 50K | Concurrent incidents at any time |
| Notifications/day | 50M | 10 notifications per incident (escalations) |
| SMS notifications | 5M | 10% of notifications are SMS |
| Voice calls | 500K | 1% of notifications are voice |
2.3 Storage Calculations
Metric Data Storage:
| Retention | Realistic Monthly | Compression | Physical |
|---|---|---|---|
| Raw (15 days) | 12.96 TB | 10:1 (columnar) | 1.3 TB |
| Downsampled 1min (90 days) | 64.8 TB | 10:1 | 6.5 TB |
| Downsampled 1hr (1 year) | 777.6 TB | 10:1 | 78 TB |
| Downsampled 1day (2 years) | 9.33 PB | 10:1 | 933 TB |
Incident Data Storage:
- 500K incidents/day × 1 KB/incident = 500 MB/day
- 2 years retention = 365 GB
Schedule Data Storage:
- 100K customers × 10 schedules/customer × 1 KB = 1 GB
- Minimal growth
2.4 Bandwidth Calculations
Metric Ingestion:
- 1.67M metrics/sec × 300 bytes = 500 MB/s average
- Peak: 1.5 GB/s
Notification Delivery:
- 50M notifications/day = 580 notifications/sec average
- SMS: 58/sec, Voice: 6/sec, Push: 400/sec, Email: 100/sec, Slack: 16/sec
Internal Data Movement:
- Metrics → Anomaly Detection: 500 MB/s
- Anomaly Detection → Alert Engine: 10 MB/s (filtered)
- Alerts → Incident Manager: 5 MB/s (deduplicated)
2.5 Compute Estimates
Metric Ingestion:
- 5M peak metrics/sec
- Each server handles ~100K metrics/sec
- Need ~50 ingestion servers
Anomaly Detection:
- 1.67M metrics/sec to analyze
- Each ML model evaluation: ~1ms per metric
- Need ~1,670 ML evaluation instances
Alert Processing:
- 10M alerts/day = 115 alerts/sec average
- Each server handles ~100 alerts/sec
- Need ~2 alert processing servers (with redundancy: 6)
Notification Delivery:
- 580 notifications/sec
- Each server handles ~50 notifications/sec
- Need ~12 notification servers
3. System Interface Design
3.1 Events API (V2)
POST /v2/enqueue
Content-Type: application/json
{
"routing_key": "abc123def456",
"event_action": "trigger",
"payload": {
"summary": "High CPU on web-server-01",
"severity": "critical",
"source": "web-server-01",
"timestamp": "2026-08-28T10:30:00Z",
"component": "web-server",
"group": "web-infrastructure",
"class": "performance",
"custom_details": {
"cpu_percent": 95.5,
"memory_percent": 78.2,
"hostname": "web-server-01"
}
},
"dedup_key": "web-server-01-cpu-high",
"images": [{
"src": "https://example.com/cpu-graph.png",
"alt": "CPU Usage Graph"
}],
"links": [{
"href": "https://grafana.example.com/d/cpu",
"text": "View in Grafana"
}]
}
Response 202 Accepted:
{
"status": "success",
"message": "Event processed"
}
3.2 Metrics API
POST /v2/metrics
Content-Type: application/json
{
"metrics": [
{
"name": "cpu.utilization",
"timestamp": 1724839800,
"value": 95.5,
"type": "gauge",
"tags": {
"host": "web-server-01",
"region": "us-east-1",
"env": "production"
}
},
{
"name": "memory.usage",
"timestamp": 1724839800,
"value": 4096,
"type": "gauge",
"tags": {
"host": "web-server-01",
"region": "us-east-1"
}
}
]
}
Response 204 No Content
3.3 Anomaly Detection Configuration
POST /v2/anomaly-detection/rules
{
"name": "CPU Anomaly Detection",
"metric": "cpu.utilization",
"tags": { "env": "production" },
"algorithm": "seasonal_decomposition",
"parameters": {
"sensitivity": 0.85,
"min_duration_seconds": 300,
"seasonal_periods": [1440, 10080],
"learning_period_days": 14
},
"notification_policy": "critical-escalation",
"enabled": true
}
Response 201 Created:
{
"id": "anomaly-rule-abc123",
"status": "active",
"created_at": "2026-08-28T10:30:00Z"
}
3.4 On-Call Schedule API
POST /v2/schedules
{
"name": "Platform Team On-Call",
"timezone": "America/New_York",
"description": "Primary on-call rotation for platform team",
"overrides": [
{
"user_id": "PABC123",
"start": "2026-09-01T00:00:00Z",
"end": "2026-09-08T00:00:00Z"
}
],
"rotation": {
"type": "weekly",
"start": "2026-09-01T00:00:00Z",
"users": [
{ "user_id": "PABC123", "name": "Alice" },
{ "user_id": "PDEF456", "name": "Bob" },
{ "user_id": "PGHI789", "name": "Charlie" }
]
}
}
GET /v2/schedules/{scheduleId}/users?time=2026-08-28T10:00:00Z
Response 200:
{
"users": [
{
"user_id": "PABC123",
"name": "Alice",
"email": "alice@example.com",
"phone_numbers": [
{ "number": "+1-555-01001", "type": "mobile" }
]
}
]
}
3.5 Escalation Policy API
POST /v2/escalation_policies
{
"name": "Critical Infrastructure Escalation",
"num_loops": 3,
"escalation_rules": [
{
"escalation_delay_in_minutes": 5,
"targets": [
{ "target_id": "schedule-abc123", "target_type": "schedule" }
]
},
{
"escalation_delay_in_minutes": 10,
"targets": [
{ "target_id": "PDEF456", "target_type": "user" }
]
},
{
"escalation_delay_in_minutes": 15,
"targets": [
{ "target_id": "PGHI789", "target_type": "user" }
]
}
]
}
3.6 Incident API
GET /v2/incidents?status=triggered&time_range=last_24h
Response 200:
{
"incidents": [
{
"id": "PINC123",
"status": "triggered",
"created_at": "2026-08-28T10:30:00Z",
"title": "High CPU on web-server-01",
"priority": "P1",
"urgency": "high",
"assignments": [
{
"assignee": { "id": "PABC123", "summary": "Alice" },
"at": "2026-08-28T10:30:05Z"
}
],
"escalation_policy": { "id": "PEPOL123" },
"alert_count": 3,
"first_trigger_log_entry": {
"id": "PLE123",
"created_at": "2026-08-28T10:30:00Z",
"kind": "trigger"
}
}
]
}
POST /v2/incidents/{incidentId}/manage
{
"type": "acknowledge"
}
4. High-Level Architecture
graph TB
subgraph "Data Sources"
AGENTS[Monitoring Agents]
INTEGRATIONS[200+ Integrations]
API[Events API]
METRICS[Metrics API]
SYNTHETIC[Synthetic Checks]
end
subgraph "Ingestion Layer"
API_GW[API Gateway]
AUTH[Auth Service]
RATE_LIMIT[Rate Limiter]
KAFKA[Kafka Cluster]
end
subgraph "Processing Layer"
subgraph "Metric Processing"
METRIC_INGEST[Metric Ingestion]
METRIC_STORE[Time-Series DB]
DOWNSAMPLE[Downsampler]
end
subgraph "Anomaly Detection"
ANOMALY_ENGINE[Anomaly Detection Engine]
ML_MODELS[ML Model Store]
BASELINE[Baseline Learner]
end
subgraph "Alert Processing"
THRESHOLD[Threshold Evaluator]
ANOMALY_ALERT[Anomaly Alert Generator]
DEDUP[Alert Deduplicator]
GROUPER[Alert Grouping Engine]
end
end
subgraph "Incident Management"
INCIDENT_SVC[Incident Service]
ESCALATION[Escalation Engine]
SCHEDULE_SVC[Schedule Service]
NOTIFICATION[Notification Service]
end
subgraph "Notification Channels"
PUSH[Push Notifications]
SMS[SMS Gateway]
EMAIL[Email Service]
VOICE[Voice Call Service]
SLACK[Slack Integration]
TEAMS[Teams Integration]
end
subgraph "Storage"
TSDB[Time-Series DB]
INCIDENT_DB[Incident DB]
SCHEDULE_DB[Schedule DB]
USER_DB[User DB]
CONFIG_DB[Config DB]
end
subgraph "Dashboard & API"
WEB_UI[Web Dashboard]
MOBILE_APP[Mobile App]
REST_API[REST API]
GRAPHQL[GraphQL API]
end
AGENTS --> API_GW
INTEGRATIONS --> API_GW
API --> API_GW
METRICS --> API_GW
SYNTHETIC --> API_GW
API_GW --> AUTH
AUTH --> RATE_LIMIT
RATE_LIMIT --> KAFKA
KAFKA --> METRIC_INGEST
METRIC_INGEST --> METRIC_STORE
METRIC_STORE --> DOWNSAMPLE
DOWNSAMPLE --> METRIC_STORE
METRIC_STORE --> ANOMALY_ENGINE
ANOMALY_ENGINE --> ML_MODELS
ANOMALY_ENGINE --> BASELINE
BASELINE --> ML_MODELS
METRIC_STORE --> THRESHOLD
ANOMALY_ENGINE --> ANOMALY_ALERT
THRESHOLD --> DEDUP
ANOMALY_ALERT --> DEDUP
DEDUP --> GROUPER
GROUPER --> INCIDENT_SVC
INCIDENT_SVC --> ESCALATION
ESCALATION --> SCHEDULE_SVC
ESCALATION --> NOTIFICATION
SCHEDULE_SVC --> SCHEDULE_DB
INCIDENT_SVC --> INCIDENT_DB
METRIC_STORE --> TSDB
NOTIFICATION --> PUSH
NOTIFICATION --> SMS
NOTIFICATION --> EMAIL
NOTIFICATION --> VOICE
NOTIFICATION --> SLACK
NOTIFICATION --> TEAMS
WEB_UI --> REST_API
MOBILE_APP --> REST_API
REST_API --> INCIDENT_SVC
REST_API --> SCHEDULE_SVC
REST_API --> METRIC_STORE
GRAPHQL --> INCIDENT_SVC
GRAPHQL --> METRIC_STORE
5. Core Component Design
5.1 Metric Ingestion Pipeline
Design Goals: Maximum throughput, low latency, exactly-once processing
graph LR
subgraph "Ingestion"
CLIENT[Client] --> GW[API Gateway]
GW --> VALIDATE[Schema Validator]
VALIDATE --> KAFKA[Kafka Topic: raw-metrics]
end
subgraph "Processing"
KAFKA --> CONSUME[Consumer Group]
CONSUME --> PARSE[Parse & Normalize]
PARSE --> TAG[Tag Indexing]
TAG --> STORE[Time-Series DB Write]
end
subgraph "Fan-out"
STORE --> THRESHOLD[Threshold Check]
STORE --> ANOMALY[Anomaly Detection]
STORE --> DOWNSAMPLE[Downsampling]
end
Ingestion Design:
- Batch writes: Accumulate metrics in memory (up to 1000 or 100ms), write in batch
- Schema validation: Lightweight validation at ingestion (type check, required fields)
- Tag normalization: Tags indexed for fast querying
- Time-series DB: Write-optimized columnar storage (e.g., VictoriaMetrics, TimescaleDB)
Metric Types:
| Type | Description | Storage | Example |
|---|---|---|---|
| Gauge | Point-in-time value | Latest value + history | CPU%, memory |
| Counter | Monotonically increasing | Rate calculation | Requests/sec |
| Histogram | Distribution of values | Buckets + percentiles | Response time |
| Set | Unique values | HyperLogLog | Unique users |
5.2 Anomaly Detection Engine
Multi-Algorithm Architecture:
graph TB
METRIC[Time-Series Metric] --> ROUTE{Algorithm Router}
ROUTE -->|Seasonal| SEASONAL[Seasonal Decomposition]
ROUTE -->|Trending| TREND[Trend-Based Detection]
ROUTE -->|Spike| SPIKE[Spike Detection]
ROUTE -->|Change| CHANGE[Change Point Detection]
ROUTE -->|Custom| CUSTOM[Custom ML Model]
SEASONAL --> SCORE1[Anomaly Score]
TREND --> SCORE2[Anomaly Score]
SPIKE --> SCORE3[Anomaly Score]
CHANGE --> SCORE4[Anomaly Score]
CUSTOM --> SCORE5[Anomaly Score]
SCORE1 --> ENSEMBLE[Ensemble Scorer]
SCORE2 --> ENSEMBLE
SCORE3 --> ENSEMBLE
SCORE4 --> ENSEMBLE
SCORE5 --> ENSEMBLE
ENSEMBLE --> FINAL{Final Score > Threshold?}
FINAL -->|Yes| ALERT[Generate Anomaly Alert]
FINAL -->|No| NORMAL[Normal]
Algorithm Details:
| Algorithm | Use Case | Input | Output | Latency |
|---|---|---|---|---|
| Seasonal Decomposition (STL) | Metrics with daily/weekly patterns | 14+ days history | Anomaly score per point | 10ms |
| Holt-Winters Exponential Smoothing | Trending metrics | 7+ days history | Forecast + confidence interval | 5ms |
| Isolation Forest | Multi-dimensional anomalies | Recent window + features | Anomaly probability | 2ms |
| Change Point Detection (PELT) | Sudden shifts | Recent window | Change point locations | 3ms |
| Median Absolute Deviation | Simple spike detection | Recent window | Deviation score | 1ms |
| Custom ML (LSTM/Autoencoder) | Complex patterns | Trained model | Reconstruction error | 50ms |
Seasonal Decomposition (STL) Deep Dive:
Input: Time series Y(t) for t = 1, 2, ..., n
Step 1: Decompose Y(t) = Trend(t) + Seasonal(t) + Residual(t)
- Trend: Low-pass filtered series (moving average)
- Seasonal: Repeating pattern (daily, weekly)
- Residual: What's left after removing trend and seasonal
Step 2: Compute anomaly score
- score(t) = |Residual(t)| / MAD(Residual)
- MAD = Median Absolute Deviation of residuals
Step 3: Classify
- If score(t) > threshold → anomaly
- Threshold configurable (default: 3.0 MAD)
Step 4: Post-processing
- Minimum duration filter (avoid noise)
- Hysteresis (avoid flapping)
Model Training Pipeline:
graph LR
subgraph "Training"
RAW[Historical Metrics] --> FEATURE[Feature Engineering]
FEATURE --> TRAIN[Model Training]
TRAIN --> EVAL[Evaluation]
EVAL --> DEPLOY[Deploy Model]
end
subgraph "Inference"
LIVE[Live Metrics] --> PREP[Preprocessing]
PREP --> MODEL[Trained Model]
MODEL --> SCORE[Anomaly Score]
end
subgraph "Feedback"
SCORE --> FEEDBACK[User Feedback]
FEEDBACK --> RETRAIN[Retraining Trigger]
RETRAIN --> TRAIN
end
5.3 Alert Processing Pipeline
Deduplication Strategy:
graph TD
ALERT[Incoming Alert] --> CHECK{Dedup Key Exists?}
CHECK -->|No| NEW[Create New Incident]
CHECK -->|Yes| ACTIVE{Incident Active?}
ACTIVE -->|Yes| APPEND[Append to Existing Incident]
ACTIVE -->|No| REOPEN{Auto-Reopen?}
REOPEN -->|Yes| REOPEN_INC[Reopen Incident]
REOPEN -->|No| NEW2[Create New Incident]
NEW --> GROUP{Grouping Rules?}
APPEND --> GROUP
REOPEN_INC --> GROUP
NEW2 --> GROUP
GROUP -->|Yes| MERGE{Merge with Existing?}
GROUP -->|No| ESCALATE[Send to Escalation]
MERGE -->|Yes| MERGE_INC[Merge Incidents]
MERGE -->|No| ESCALATE
MERGE_INC --> ESCALATE
ESCALATE --> NOTIFY[Notify On-Call]
Deduplication Logic:
- Dedup key: Customer-provided or auto-generated from (source + summary)
- Dedup window: 1 hour (alerts with same key within 1 hour → same incident)
- Auto-reopen: If alert re-triggers within 24 hours of resolution → reopen
- Grouping: Alerts with same group tag → same incident
Alert Grouping:
| Grouping Type | Criteria | Example |
|---|---|---|
| Source-based | Same source | All alerts from web-server-01 |
| Tag-based | Same tags | All alerts with env=production |
| Time-based | Same time window | All alerts within 5 min |
| Composite | Multiple criteria | Same source + same severity |
5.4 Escalation Engine
Escalation Flow:
sequenceDiagram
participant INC as Incident
participant ESC as Escalation Engine
participant SCHED as Schedule Service
participant NOTIFY as Notification Service
participant USER as On-Call User
INC->>ESC: Incident created
ESC->>SCHED: Get on-call user for schedule
SCHED-->>ESC: User Alice (primary)
ESC->>NOTIFY: Send notification to Alice
NOTIFY->>USER: Push + SMS to Alice
alt Alice acknowledges within 5 min
USER->>ESC: Acknowledge
ESC->>INC: Incident acknowledged
else Alice does not respond
ESC->>SCHED: Get next escalation target
SCHED-->>ESC: User Bob (secondary)
ESC->>NOTIFY: Send notification to Bob
NOTIFY->>USER: Voice call to Bob
end
Escalation Policy Evaluation:
Input: Incident, Escalation Policy
Step 1: Resolve schedule → current on-call user(s)
- Evaluate rotation rules
- Check overrides and swaps
- Handle timezone conversions
Step 2: Send notifications to Level 1 targets
- Push notification (immediate)
- SMS (immediate)
- Email (immediate)
Step 3: Wait for acknowledgment (configurable timeout, default 5 min)
- If acknowledged → stop escalation
- If not acknowledged → proceed to next level
Step 4: Escalate to next level
- Voice call (if configured)
- Notify next target(s)
Step 5: Repeat until acknowledged or max loops reached
5.5 On-Call Schedule Service
Schedule Evaluation:
graph TB
subgraph "Schedule Definition"
ROTATION[Rotation Rules]
OVERRIDES[Overrides]
SWAPS[Swaps]
HOLIDAYS[Holiday Calendar]
end
subgraph "Evaluation Engine"
EVAL[Evaluation Engine]
CACHE[Schedule Cache]
end
subgraph "Output"
ONCALL[On-Call User(s)]
NEXT[NEXT On-Call User]
SCHEDULE[Full Schedule View]
end
ROTATION --> EVAL
OVERRIDES --> EVAL
SWAPS --> EVAL
HOLIDAYS --> EVAL
EVAL --> CACHE
CACHE --> ONCALL
CACHE --> NEXT
CACHE --> SCHEDULE
Schedule Types:
| Type | Description | Example |
|---|---|---|
| Fixed rotation | Users rotate in order | Alice → Bob → Charlie → Alice |
| Weighted rotation | Users rotate with weights | Alice(2) → Bob(1) → Charlie(1) |
| Follow-the-sun | Geographic rotation | US → EU → APAC |
| Layered | Multiple layers active | Primary + Secondary simultaneously |
| Custom | User-defined rules | Business hours only, weekends off |
Schedule Caching:
- Pre-compute on-call assignments for next 30 days
- Cache invalidated on schedule change
- Evaluation: O(1) lookup from cache
5.6 Notification Service
Multi-Channel Delivery:
graph TB
subgraph "Notification Service"
QUEUE[Notification Queue]
ROUTER[Channel Router]
RETRY[Retry Manager]
end
subgraph "Channels"
PUSH[Push Service]
SMS[SMS Gateway]
EMAIL[Email Service]
VOICE[Voice Service]
SLACK[Slack Webhook]
TEAMS[Teams Webhook]
end
subgraph "Delivery Tracking"
TRACK[Delivery Tracker]
ACK[Acknowledgment Handler]
end
QUEUE --> ROUTER
ROUTER --> PUSH
ROUTER --> SMS
ROUTER --> EMAIL
ROUTER --> VOICE
ROUTER --> SLACK
ROUTER --> TEAMS
PUSH --> RETRY
SMS --> RETRY
EMAIL --> RETRY
VOICE --> RETRY
RETRY --> TRACK
TRACK --> ACK
Channel Priority and Fallback:
| Priority | Channel | Timeout | Retry |
|---|---|---|---|
| 1 | Push notification | 30 sec | 3 attempts |
| 2 | SMS | 60 sec | 3 attempts |
| 3 | 120 sec | 2 attempts | |
| 4 | Voice call | 180 sec | 2 attempts |
| 5 | Slack/Teams | 30 sec | 2 attempts |
Notification Throttling:
- Per-user: Max 10 notifications per hour (configurable)
- Per-incident: Max 20 notifications total
- Quiet hours: Configurable per-user quiet hours (no non-critical notifications)
6. Data Model
6.1 Metric Schema
-- Time-series metric data (stored in TSDB, shown as relational for clarity)
CREATE TABLE metrics (
metric_id BIGSERIAL,
customer_id UUID NOT NULL,
metric_name VARCHAR(256) NOT NULL,
timestamp BIGINT NOT NULL, -- Unix timestamp in milliseconds
value DOUBLE PRECISION NOT NULL,
metric_type VARCHAR(16) NOT NULL, -- gauge, counter, histogram, set
tags JSONB NOT NULL, -- {"host": "web-01", "region": "us-east"}
tags_hash VARCHAR(64) GENERATED ALWAYS AS (
md5(tags::text)
) STORED,
PRIMARY KEY (customer_id, metric_name, tags_hash, timestamp)
);
-- Metric metadata
CREATE TABLE metric_metadata (
customer_id UUID NOT NULL,
metric_name VARCHAR(256) NOT NULL,
display_name VARCHAR(256),
unit VARCHAR(32),
description TEXT,
tags JSONB,
created_at TIMESTAMP NOT NULL,
last_seen TIMESTAMP NOT NULL,
is_active BOOLEAN DEFAULT TRUE,
PRIMARY KEY (customer_id, metric_name)
);
-- Tag index for fast querying
CREATE TABLE metric_tags (
customer_id UUID NOT NULL,
tag_key VARCHAR(128) NOT NULL,
tag_value VARCHAR(256) NOT NULL,
metric_count BIGINT DEFAULT 0,
PRIMARY KEY (customer_id, tag_key, tag_value)
);
-- Indexes
CREATE INDEX idx_metrics_query ON metrics(customer_id, metric_name, tags_hash, timestamp DESC);
CREATE INDEX idx_metrics_time ON metrics(timestamp DESC);
6.2 Incident Schema
CREATE TABLE incidents (
incident_id VARCHAR(10) PRIMARY KEY, -- PINC123
customer_id UUID NOT NULL,
status VARCHAR(16) NOT NULL, -- triggered, acknowledged, resolved
urgency VARCHAR(8) NOT NULL, -- high, low
title VARCHAR(1024) NOT NULL,
priority VARCHAR(4), -- P1, P2, P3, P4
created_at TIMESTAMP NOT NULL,
acknowledged_at TIMESTAMP,
resolved_at TIMESTAMP,
last_status_change TIMESTAMP NOT NULL,
escalation_policy_id VARCHAR(10),
service_id VARCHAR(10),
alert_count INT DEFAULT 0,
group VARCHAR(256),
dedup_key VARCHAR(256),
custom_details JSONB,
timeline JSONB, -- Array of timeline events
tags JSONB
);
CREATE TABLE incident_alerts (
alert_id VARCHAR(10) PRIMARY KEY,
incident_id VARCHAR(10) REFERENCES incidents(incident_id),
status VARCHAR(16) NOT NULL, -- triggered, acknowledged, resolved, merged
title VARCHAR(1024),
detail TEXT,
severity VARCHAR(16),
source VARCHAR(256),
created_at TIMESTAMP NOT NULL,
acknowledged_at TIMESTAMP,
resolved_at TIMESTAMP,
dedup_key VARCHAR(256),
custom_details JSONB
);
CREATE TABLE incident_assignments (
assignment_id BIGSERIAL PRIMARY KEY,
incident_id VARCHAR(10) REFERENCES incidents(incident_id),
user_id VARCHAR(10) NOT NULL,
assigned_at TIMESTAMP NOT NULL,
type VARCHAR(16) NOT NULL, -- assigned, acknowledged
is_active BOOLEAN DEFAULT TRUE
);
CREATE TABLE incident_log_entries (
log_entry_id VARCHAR(10) PRIMARY KEY,
incident_id VARCHAR(10) REFERENCES incidents(incident_id),
kind VARCHAR(32) NOT NULL, -- trigger, acknowledge, resolve, escalation
created_at TIMESTAMP NOT NULL,
user_id VARCHAR(10),
world_write BOOLEAN DEFAULT FALSE,
body TEXT,
dedup_key VARCHAR(256),
data JSONB
);
-- Indexes
CREATE INDEX idx_incidents_customer ON incidents(customer_id, status, created_at DESC);
CREATE INDEX idx_incidents_alert ON incident_alerts(incident_id);
CREATE INDEX idx_incident_assignments ON incident_assignments(incident_id, is_active);
CREATE INDEX idx_incident_logs ON incident_log_entries(incident_id, created_at DESC);
6.3 Schedule Schema
CREATE TABLE schedules (
schedule_id VARCHAR(10) PRIMARY KEY, -- PSCH123
customer_id UUID NOT NULL,
name VARCHAR(256) NOT NULL,
description TEXT,
timezone VARCHAR(64) NOT NULL DEFAULT 'UTC',
type VARCHAR(16) NOT NULL, -- rotation, calendar, embedded
created_at TIMESTAMP NOT NULL,
updated_at TIMESTAMP NOT NULL,
final_schedule JSONB, -- Computed schedule
overrides JSONB, -- Override entries
restrictions JSONB, -- Time restrictions
starts_at TIMESTAMP,
ends_at TIMESTAMP
);
CREATE TABLE schedule_rotations (
rotation_id BIGSERIAL PRIMARY KEY,
schedule_id VARCHAR(10) REFERENCES schedules(schedule_id),
name VARCHAR(256),
type VARCHAR(16) NOT NULL, -- round_robin, single
start TIMESTAMP NOT NULL,
end TIMESTAMP,
operation VARCHAR(16) NOT NULL, -- add, remove
rotation_length_seconds BIGINT NOT NULL,
users JSONB NOT NULL, -- Array of user objects with weights
is_primary BOOLEAN DEFAULT TRUE
);
CREATE TABLE schedule_overrides (
override_id BIGSERIAL PRIMARY KEY,
schedule_id VARCHAR(10) REFERENCES schedules(schedule_id),
user_id VARCHAR(10) NOT NULL,
start TIMESTAMP NOT NULL,
end TIMESTAMP NOT NULL,
created_by VARCHAR(10),
created_at TIMESTAMP NOT NULL
);
-- Pre-computed on-call assignments (cached)
CREATE TABLE schedule_assignments (
schedule_id VARCHAR(10) REFERENCES schedules(schedule_id),
user_id VARCHAR(10) NOT NULL,
start TIMESTAMP NOT NULL,
end TIMESTAMP NOT NULL,
is_primary BOOLEAN DEFAULT TRUE,
PRIMARY KEY (schedule_id, user_id, start)
);
CREATE INDEX idx_schedule_assignments_time ON schedule_assignments(schedule_id, start, end);
6.4 Escalation Policy Schema
CREATE TABLE escalation_policies (
policy_id VARCHAR(10) PRIMARY KEY, -- PEPOL123
customer_id UUID NOT NULL,
name VARCHAR(256) NOT NULL,
num_loops INT DEFAULT 1,
created_at TIMESTAMP NOT NULL,
updated_at TIMESTAMP NOT NULL
);
CREATE TABLE escalation_rules (
rule_id BIGSERIAL PRIMARY KEY,
policy_id VARCHAR(10) REFERENCES escalation_policies(policy_id),
position INT NOT NULL,
escalation_delay_minutes INT NOT NULL,
targets JSONB NOT NULL, -- Array of target objects
created_at TIMESTAMP NOT NULL
);
CREATE TABLE services (
service_id VARCHAR(10) PRIMARY KEY,
customer_id UUID NOT NULL,
name VARCHAR(256) NOT NULL,
description TEXT,
escalation_policy_id VARCHAR(10) REFERENCES escalation_policies(policy_id),
alert_creation_type VARCHAR(16) DEFAULT 'all',
auto_resolve_timeout INT DEFAULT 0,
ack_timeout INT DEFAULT 0,
created_at TIMESTAMP NOT NULL
);
6.5 Anomaly Detection Configuration
CREATE TABLE anomaly_rules (
rule_id UUID PRIMARY KEY,
customer_id UUID NOT NULL,
name VARCHAR(256) NOT NULL,
metric_name VARCHAR(256) NOT NULL,
tags_filter JSONB,
algorithm VARCHAR(64) NOT NULL,
parameters JSONB NOT NULL,
sensitivity DOUBLE PRECISION DEFAULT 0.85,
min_duration_seconds INT DEFAULT 300,
learning_period_days INT DEFAULT 14,
enabled BOOLEAN DEFAULT TRUE,
created_at TIMESTAMP NOT NULL,
updated_at TIMESTAMP NOT NULL
);
CREATE TABLE anomaly_models (
model_id UUID PRIMARY KEY,
rule_id UUID REFERENCES anomaly_rules(rule_id),
metric_name VARCHAR(256) NOT NULL,
tags_hash VARCHAR(64) NOT NULL,
model_type VARCHAR(64) NOT NULL,
model_data BYTEA, -- Serialized model
training_data_start TIMESTAMP NOT NULL,
training_data_end TIMESTAMP NOT NULL,
accuracy DOUBLE PRECISION,
created_at TIMESTAMP NOT NULL,
updated_at TIMESTAMP NOT NULL,
is_active BOOLEAN DEFAULT TRUE,
PRIMARY KEY (model_id)
);
CREATE TABLE anomaly_events (
event_id UUID PRIMARY KEY,
rule_id UUID REFERENCES anomaly_rules(rule_id),
metric_name VARCHAR(256) NOT NULL,
timestamp TIMESTAMP NOT NULL,
actual_value DOUBLE PRECISION,
expected_value DOUBLE PRECISION,
anomaly_score DOUBLE PRECISION NOT NULL,
severity VARCHAR(16),
duration_seconds INT,
is_resolved BOOLEAN DEFAULT FALSE,
resolved_at TIMESTAMP,
created_at TIMESTAMP NOT NULL
);
CREATE INDEX idx_anomaly_events_rule ON anomaly_events(rule_id, timestamp DESC);
CREATE INDEX idx_anomaly_events_metric ON anomaly_events(metric_name, timestamp DESC);
7. Key Design Decisions
7.1 Alerting: Push vs Pull
Decision: Push-based alerting with pull-based fallback
| Approach | Pros | Cons | Our Choice |
|---|---|---|---|
| Push | Low latency, real-time | Complex, requires persistent connections | Primary |
| Pull | Simple, reliable | Higher latency, polling overhead | Fallback |
Rationale: Push provides sub-30-second alert-to-notification latency. Pull used for:
- Dashboard polling (15-second interval)
- Mobile app background sync
- Integration webhooks (retry on failure)
7.2 Anomaly Detection: Rule-Based vs ML-Based
Decision: Hybrid approach with automatic algorithm selection
| Approach | Pros | Cons | Our Choice |
|---|---|---|---|
| Rule-based (thresholds) | Simple, predictable, no training | Manual tuning, doesn’t adapt | For simple metrics |
| ML-based | Automatic, adapts to patterns | Complex, needs training data, false positives | For complex metrics |
| Hybrid | Best of both | More complex | Default |
Algorithm Selection Logic:
Input: Metric time series
Step 1: Analyze metric characteristics
- Has seasonal pattern? → STL decomposition
- Has trend? → Holt-Winters
- Stable baseline? → MAD-based
- Sudden shifts? → Change point detection
- Multi-dimensional? → Isolation Forest
Step 2: Train selected algorithm
- Use 14 days of historical data
- Validate against known anomalies (if available)
Step 3: Deploy and monitor
- Track false positive rate
- Auto-retrain if performance degrades
7.3 Incident Grouping: Aggressive vs Conservative
Decision: Configurable grouping with conservative defaults
| Approach | Pros | Cons |
|---|---|---|
| Aggressive | Fewer incidents, less noise | May hide unrelated issues |
| Conservative | Accurate, no missed issues | More incidents, more noise |
| Configurable | Customer choice | Requires customer tuning |
Default Grouping Rules:
- Same source + same severity + within 5 minutes → group
- Same service + within 15 minutes → group
- Customer can customize grouping rules per service
7.4 Notification: All-at-Once vs Escalation
Decision: Escalation-based with configurable parallel notification
| Approach | Pros | Cons |
|---|---|---|
| All-at-once | Fastest response | Notification fatigue, confusion |
| Escalation | Ordered, clear ownership | Slower if primary doesn’t respond |
| Hybrid | Fast + ordered | Complex |
Default Behavior:
- Level 1: Push + SMS simultaneously
- Level 2 (after timeout): Voice call + SMS
- Level 3 (after timeout): Notify manager + all team members
7.5 Data Retention: Raw vs Downsampled
Decision: Tiered retention with automatic downsampling
| Tier | Retention | Resolution | Storage |
|---|---|---|---|
| Raw | 15 days | Collection interval (15s-5min) | High |
| 1-minute | 90 days | 1-minute averages | Medium |
| 1-hour | 1 year | 1-hour averages | Low |
| 1-day | 2 years | 1-day averages | Minimal |
Downsampling Strategy:
- Aggregate: avg, min, max, sum, count per window
- Preserve: percentiles (p50, p95, p99) for latency metrics
- Delete raw data after 15 days (configurable)
8. Scalability Strategy
8.1 Multi-Region Deployment
graph TB
subgraph "Primary Region (US-East)"
US_API[API Gateway]
US_KAFKA[Kafka Cluster]
US_TSDB[Time-Series DB]
US_INCIDENT[Incident Service]
US_SCHEDULE[Schedule Service]
end
subgraph "Secondary Region (EU-West)"
EU_API[API Gateway]
EU_KAFKA[Kafka Cluster]
EU_TSDB[Time-Series DB]
EU_INCIDENT[Incident Service]
EU_SCHEDULE[Schedule Service]
end
subgraph "Tertiary Region (APAC)"
AP_API[API Gateway]
AP_KAFKA[Kafka Cluster]
AP_TSDB[Time-Series DB]
AP_INCIDENT[Incident Service]
AP_SCHEDULE[Schedule Service]
end
GLOBAL_DNS[Global DNS] --> US_API
GLOBAL_DNS --> EU_API
GLOBAL_DNS --> AP_API
US_TSDB <-->|Async Replication| EU_TSDB
EU_TSDB <-->|Async Replication| AP_TSDB
Customer Sharding:
- Customers assigned to regions via consistent hashing
- Each customer’s data stays in one primary region
- Cross-region replication for disaster recovery
8.2 Auto-Scaling Triggers
| Component | Scale-Up Trigger | Scale-Down Trigger | Min | Max |
|---|---|---|---|---|
| API Gateway | CPU > 70% for 1 min | CPU < 30% for 5 min | 10 | 100 |
| Kafka Brokers | Disk > 70% | Disk < 40% | 6 | 50 |
| TSDB Nodes | Write latency > 100ms | Write latency < 20ms | 4 | 40 |
| Anomaly Engine | Queue depth > 10K | Queue depth < 1K | 4 | 50 |
| Alert Processor | Queue depth > 1K | Queue depth < 100 | 2 | 20 |
| Notification | Queue depth > 5K | Queue depth < 500 | 4 | 30 |
8.3 Metric Partitioning Strategy
Time-Series DB Partitioning:
- Partition by
customer_id(hash) →metric_name→timestamp(range) - Each partition: ~10 GB per month for average customer
- Hot partitions (recent data) on SSD, cold partitions on HDD
Tag Indexing:
- Inverted index: tag_key + tag_value → list of (customer_id, metric_name)
- Enables fast queries: “show all CPU metrics for env=production”
9. Curveball Scenarios
9.1 Curveball 1: A Cascading Failure Generates 100K Simultaneous Alerts
Scenario: A major cloud provider outage causes all customer services to fail simultaneously. 100K alerts fire within 1 minute across 10K customers.
Impact Analysis:
- Alert processing pipeline overwhelmed
- Notification system flooded (millions of notifications)
- On-call engineers receive hundreds of alerts
- Incident management UI becomes unresponsive
Response Strategy:
| Layer | Action | Timeline |
|---|---|---|
| Ingestion | Rate limiting per customer. Queue excess alerts for processing. | Immediate |
| Deduplication | Aggressive deduplication: same source + same time window → single incident. | Immediate |
| Grouping | Automatic mega-grouping: all alerts from same cloud provider → single incident per customer. | 1-2 min |
| Notification | Notification throttling: max 1 notification per customer per minute during surge. | Immediate |
| Escalation | Pause escalation during surge. Resume after surge subsides. | 5 min |
| UI | Read-only mode for incident list. Disable non-essential features. | 1 min |
| Post-Surge | Process queued alerts. Send summary notifications. | 15-30 min |
Prevention:
- Surge detection: Alert rate > 10× normal → activate surge mode
- Pre-defined grouping rules for known cascade patterns
- Notification rate limiting with customer-configurable thresholds
9.2 Curveball 2: An On-Call Engineer’s Phone is Unreachable for 24 Hours
Scenario: The primary on-call engineer loses phone service (dead battery, lost phone, no coverage). Critical incidents are not being acknowledged.
Impact Analysis:
- Escalation policy should handle this, but what if all levels are unreachable?
- Critical incidents may go unacknowledged
- Business impact depends on incident severity
Response Strategy:
| Phase | Action | Timeline |
|---|---|---|
| Detection | Escalation engine detects no acknowledgment after all levels exhausted. | 15-30 min |
| Escalation | Final escalation: notify all team members, manager, and executive sponsor. | 30-60 min |
| Auto-Assignment | If configured: auto-assign to backup on-call or team lead. | 1 hr |
| Bridge Call | Auto-create conference bridge and invite all available team members. | 1 hr |
| Resolution | Team collaborates to resolve incident. | Variable |
| Post-Incident | Review escalation policy. Add backup contacts. | Post-incident |
Prevention:
- Multi-contact requirement: Each on-call user must have at least 2 contact methods
- Health check: Periodic check-in with on-call users (daily ping)
- Backup on-call: Required backup for each schedule
- Executive escalation: Final escalation to management for unacknowledged P1 incidents
9.3 Curveball 3: A False Positive Storm from ML Anomaly Detection
Scenario: A system deployment changes the behavior of 1,000 metrics. The ML anomaly detection model hasn’t learned the new patterns yet, generating 50,000 false positive alerts in 1 hour.
Impact Analysis:
- Alert fatigue for on-call engineers
- Real incidents may be missed in the noise
- Trust in anomaly detection eroded
Response Strategy:
| Phase | Action | Timeline |
|---|---|---|
| Detection | Anomaly detection self-monitoring: false positive rate > 10% → alert. | 15 min |
| Mitigation | Automatic sensitivity reduction for affected metrics. Pause anomaly alerts for metrics with > 50% false positive rate. | 15-30 min |
| Communication | Notify customers of increased false positive rate and mitigation actions. | 30 min |
| Retraining | Trigger immediate model retraining with recent data. | 1-2 hrs |
| Validation | Validate retrained models against recent data before re-enabling. | 2-4 hrs |
| Gradual Re-enable | Re-enable anomaly detection gradually with increased thresholds. | 4-8 hrs |
Prevention:
- Change correlation: Correlate anomaly spikes with deployment events
- Grace period: 24-hour grace period after deployments (reduced sensitivity)
- Self-monitoring: Track false positive rate per model, auto-adjust sensitivity
- Feedback loop: User feedback on false positives triggers immediate retraining
9.4 Curveball 4: SMS Gateway Provider Goes Down
Scenario: The primary SMS gateway provider experiences a global outage. All SMS notifications fail for 4 hours.
Impact Analysis:
- 10% of notifications are SMS-only for some users
- Critical alerts may not reach on-call engineers
- Escalation chains broken
Response Strategy:
| Phase | Action | Timeline |
|---|---|---|
| Detection | SMS delivery failure rate > 5% → alert. | 5 min |
| Failover | Switch to backup provider. | 5-10 min |
| Fallback | For users with SMS-only: escalate to voice call immediately. | 10-15 min |
| Notification | Notify customers of SMS degradation and fallback behavior. | 15 min |
| Recovery | Primary provider restored → gradual traffic shift back. | 4+ hrs |
| Post-Incident | Add third SMS provider. Require multi-channel for critical alerts. | Post-incident |
Prevention:
- Multi-provider: At least 2 SMS providers with automatic failover
- Multi-channel requirement: Critical alerts require at least 2 delivery channels
- Delivery confirmation: Track delivery status, retry with alternative channel on failure
9.5 Curveball 5: A Customer Accidentally Creates an Infinite Escalation Loop
Scenario: A customer misconfigures their escalation policy, creating a circular reference that causes infinite notification loops. The system sends millions of notifications in minutes.
Impact Analysis:
- Notification system overwhelmed
- SMS/voice costs spike
- Other customers affected
Response Strategy:
| Phase | Action | Timeline |
|---|---|---|
| Detection | Notification rate anomaly: single customer > 1000 notifications/min → alert. | 1-2 min |
| Containment | Automatic circuit breaker: pause notifications for the customer. | 2-3 min |
| Assessment | Identify the misconfigured escalation policy. | 5 min |
| Remediation | Disable the problematic escalation policy. Notify customer. | 5-10 min |
| Recovery | Resume notifications with validation. | 10-15 min |
| Prevention | Add escalation policy validation: detect circular references before saving. | Post-incident |
Prevention:
- Policy validation: Validate escalation policies for circular references on save
- Notification budget: Per-customer notification budget with hard limits
- Rate limiting: Max notifications per customer per hour (configurable, default 1000)
- Cost monitoring: Real-time notification cost monitoring with alerts
10. Failure Modes and Mitigations
10.1 Failure Mode Matrix
| Failure Mode | Probability | Impact | Detection | Mitigation | RTO | Data Loss |
|---|---|---|---|---|---|---|
| Metric ingestion failure | Medium | Medium | Write error rate | Multi-region failover, client retry | 30 sec | Client buffer |
| Kafka broker failure | Low | High | Broker health monitoring | Auto-replace, ISR management | 1 min | 0 |
| TSDB node failure | Low | Medium | Query latency monitoring | Multi-AZ replication, failover | 30 sec | 0 |
| Anomaly engine failure | Low | Low | Queue depth monitoring | Fallback to threshold-only | 1 min | 0 |
| Alert processing failure | Low | High | Alert queue depth | Redundant processing, retry | 30 sec | 0 |
| Notification delivery failure | Medium | High | Delivery failure rate | Multi-channel fallback | 1 min | 0 |
| Schedule service failure | Low | Medium | Query latency monitoring | Cached schedules, fallback | 1 min | 0 |
| Escalation engine failure | Low | Critical | Escalation timeout monitoring | Manual escalation, backup contacts | 5 min | 0 |
| Database corruption | Very Low | Critical | Checksum validation | Backup restoration | 1 hr | 0 |
| DDoS attack | Medium | Medium | Traffic anomaly detection | Rate limiting, WAF | 5 min | 0 |
| Third-party integration failure | Medium | Low | Integration health monitoring | Graceful degradation | 5 min | 0 |
10.2 Notification Delivery Reliability
graph TB
subgraph "Notification Pipeline"
QUEUE[Notification Queue]
ROUTER[Channel Router]
DELIVER[Delivery Engine]
TRACK[Delivery Tracker]
end
subgraph "Delivery Channels"
PUSH[Push]
SMS1[SMS Provider A]
SMS2[SMS Provider B]
EMAIL[Email]
VOICE[Voice]
end
subgraph "Fallback Logic"
RETRY[Retry Manager]
ESCALATE[Escalation]
end
QUEUE --> ROUTER
ROUTER --> DELIVER
DELIVER --> PUSH
DELIVER --> SMS1
DELIVER --> EMAIL
DELIVER --> VOICE
SMS1 -->|Fail| SMS2
PUSH -->|Fail| RETRY
SMS1 -->|Fail| RETRY
SMS2 -->|Fail| RETRY
EMAIL -->|Fail| RETRY
VOICE -->|Fail| RETRY
RETRY --> TRACK
TRACK -->|All failed| ESCALATE
ESCALATE --> DELIVER
Delivery Guarantee:
- At-least-once: Notifications delivered at least once (idempotent delivery)
- Multi-channel: Critical alerts sent via multiple channels simultaneously
- Retry: Exponential backoff (1s, 5s, 30s, 5 min), max 5 attempts
- Fallback: If primary channel fails, try alternative channels
- Confirmation: Delivery confirmation tracked, used for escalation decisions
10.3 Incident Management Resilience
Incident Service HA:
graph TB
subgraph "Incident Service HA"
subgraph "AZ-1a"
INC1[Incident Service Primary]
DB1[Incident DB Primary]
end
subgraph "AZ-1b"
INC2[Incident Service Standby]
DB2[Incident DB Replica]
end
subgraph "AZ-1c"
INC3[Incident Service Standby]
DB3[Incident DB Replica]
end
end
LB[Load Balancer] --> INC1
LB --> INC2
LB --> INC3
INC1 --> DB1
INC2 --> DB2
INC3 --> DB3
DB1 <-->|Sync Replication| DB2
DB2 <-->|Sync Replication| DB3
HEALTH[Health Checker] --> INC1
HEALTH --> INC2
HEALTH --> INC3
Key Principles:
- Quorum-based writes: Incident state changes require 2-of-3 AZ acknowledgment
- Automatic leader election: If primary fails, standby takes over within 30 seconds
- Idempotent operations: All incident operations are idempotent (safe to retry)
- Event sourcing: All incident changes stored as events, state derived from events
10.4 Disaster Recovery Plan
| Scenario | Strategy | RTO | RPO |
|---|---|---|---|
| AZ failure | Cross-AZ failover (automatic) | 1 min | 0 |
| Region failure | Multi-region failover (automatic for metrics, manual for incidents) | 5 min | 0 |
| Global failure | Not designed for (requires geographic distribution at customer level) | N/A | N/A |
| Database corruption | Backup restoration + event replay | 30 min | 0 |
| Notification provider failure | Multi-provider failover | 5 min | 0 |
| ML model corruption | Fallback to threshold-only detection | 1 min | 0 |
Appendix: Glossary
| Term | Definition |
|---|---|
| STL | Seasonal and Trend decomposition using Loess |
| MAD | Median Absolute Deviation |
| PELT | Pruned Exact Linear Time (change point detection algorithm) |
| HLL | HyperLogLog - Probabilistic distinct counting |
| ISR | In-Sync Replicas - Kafka replication mechanism |
| TSDB | Time-Series Database |
| MTTR | Mean Time To Resolution |
| MTTA | Mean Time To Acknowledge |
| SLA | Service Level Agreement |
| SLO | Service Level Objective |
| SLI | Service Level Indicator |
| Runbook | Documented procedures for incident response |
| Postmortem | Post-incident review document |
| Blameless | Postmortem culture that focuses on process, not people |