Analytics & TelemetryQuestion #6CORE

PagerDuty: Anomaly Detection, Alerting & Incident Management

Mission-critical incident escalation engine featuring telemetry deduplication, anomaly detection, on-call scheduling state machines, and multi-channel failover alerting.

Category: Category 4: Analytics & Telemetry•Updated: 2026-08-30
#Analytics#Anomaly Detection#Alerting#On-Call#Incident Mgmt

Design an Anomaly Detection System like PagerDuty

Table of Contents

  1. Requirements Clarification
  2. Back-of-Envelope Calculations
  3. System Interface Design
  4. High-Level Architecture
  5. Core Component Design
  6. Data Model
  7. Key Design Decisions
  8. Scalability Strategy
  9. Curveball Scenarios
  10. Failure Modes and Mitigations

1. Requirements Clarification

1.1 Functional Requirements

Metric Collection & Monitoring

  • Collect metrics from multiple sources: agents, APIs, integrations (Datadog, Prometheus, New Relic, custom)
  • Support numeric metrics, logs, traces, and synthetic checks
  • Configurable collection intervals (15s, 30s, 60s, 300s)
  • Metric tagging and dimension support

Threshold-Based Alerting

  • Static thresholds: absolute values, percentage changes
  • Multi-window thresholds: “if CPU > 90% for 5 min AND > 80% for 15 min”
  • Baseline thresholds: compare against historical baseline
  • Composite conditions: AND/OR logic across multiple metrics

ML-Based Anomaly Detection

  • Automatic baseline learning per metric
  • Seasonal pattern detection (daily, weekly, monthly cycles)
  • Change point detection for sudden shifts
  • Forecasting-based anomaly detection
  • Noise reduction and signal smoothing

Incident Management

  • Incident creation from alerts with automatic deduplication
  • Incident lifecycle: triggered → acknowledged → resolved
  • Incident grouping: related alerts grouped into single incident
  • Incident timeline with all events and actions
  • Post-incident review and root cause analysis

On-Call Scheduling

  • Rotating schedules with customizable patterns
  • Escalation policies: multi-level escalation with timeouts
  • Override and swap management
  • Timezone-aware scheduling
  • Integration with calendar systems

Notification & Communication

  • Multi-channel notifications: push, SMS, email, voice call, Slack, Teams
  • Escalation: if not acknowledged within timeout, escalate to next level
  • Incident bridge calls: conference bridge for incident response
  • Status updates and stakeholder notifications

Integrations

  • 200+ integrations: monitoring tools, ticketing systems, chat platforms
  • Webhook support for custom integrations
  • Events API for programmatic incident management
  • Change management integration (correlate incidents with deployments)

1.2 Non-Functional Requirements

Requirement Target Rationale
Alert-to-Notification Latency < 30 seconds (p99) Critical for incident response
Metric Ingestion Throughput 10M+ metrics/sec Enterprise monitoring scale
Anomaly Detection Latency < 60 seconds Near-real-time detection
System Availability 99.99% Mission-critical for operations
Notification Delivery 99.9% within 30 sec On-call reliability
Data Retention 1 year metrics, 2 years incidents Compliance and analysis
Multi-Tenancy Strong isolation Enterprise security
Scalability 100K+ customers, 10M+ metrics Enterprise market

2. Back-of-Envelope Calculations

2.1 Assumption Sets

Parameter Conservative Realistic Aggressive
Customers 10K 100K 500K
Metrics/customer 1,000 5,000 20,000
Collection interval 60s 30s 15s
Total metric points/sec 167K 1.67M 6.67M
Peak metric points/sec (3×) 500K 5M 20M
Avg metric payload 200 bytes 300 bytes 500 bytes
Daily metric volume 14.4 GB 432 GB 2.88 TB
Monthly metric volume 432 GB 12.96 TB 86.4 TB

2.2 Alert Volume Estimates

Parameter Realistic Notes
Alerts/day 10M 0.6% of metric points trigger alerts
Incidents/day 500K 5% of alerts become incidents (after dedup)
Active incidents 50K Concurrent incidents at any time
Notifications/day 50M 10 notifications per incident (escalations)
SMS notifications 5M 10% of notifications are SMS
Voice calls 500K 1% of notifications are voice

2.3 Storage Calculations

Metric Data Storage:

Retention Realistic Monthly Compression Physical
Raw (15 days) 12.96 TB 10:1 (columnar) 1.3 TB
Downsampled 1min (90 days) 64.8 TB 10:1 6.5 TB
Downsampled 1hr (1 year) 777.6 TB 10:1 78 TB
Downsampled 1day (2 years) 9.33 PB 10:1 933 TB

Incident Data Storage:

  • 500K incidents/day × 1 KB/incident = 500 MB/day
  • 2 years retention = 365 GB

Schedule Data Storage:

  • 100K customers × 10 schedules/customer × 1 KB = 1 GB
  • Minimal growth

2.4 Bandwidth Calculations

Metric Ingestion:

  • 1.67M metrics/sec × 300 bytes = 500 MB/s average
  • Peak: 1.5 GB/s

Notification Delivery:

  • 50M notifications/day = 580 notifications/sec average
  • SMS: 58/sec, Voice: 6/sec, Push: 400/sec, Email: 100/sec, Slack: 16/sec

Internal Data Movement:

  • Metrics → Anomaly Detection: 500 MB/s
  • Anomaly Detection → Alert Engine: 10 MB/s (filtered)
  • Alerts → Incident Manager: 5 MB/s (deduplicated)

2.5 Compute Estimates

Metric Ingestion:

  • 5M peak metrics/sec
  • Each server handles ~100K metrics/sec
  • Need ~50 ingestion servers

Anomaly Detection:

  • 1.67M metrics/sec to analyze
  • Each ML model evaluation: ~1ms per metric
  • Need ~1,670 ML evaluation instances

Alert Processing:

  • 10M alerts/day = 115 alerts/sec average
  • Each server handles ~100 alerts/sec
  • Need ~2 alert processing servers (with redundancy: 6)

Notification Delivery:

  • 580 notifications/sec
  • Each server handles ~50 notifications/sec
  • Need ~12 notification servers

3. System Interface Design

3.1 Events API (V2)

POST /v2/enqueue
Content-Type: application/json

{
  "routing_key": "abc123def456",
  "event_action": "trigger",
  "payload": {
    "summary": "High CPU on web-server-01",
    "severity": "critical",
    "source": "web-server-01",
    "timestamp": "2026-08-28T10:30:00Z",
    "component": "web-server",
    "group": "web-infrastructure",
    "class": "performance",
    "custom_details": {
      "cpu_percent": 95.5,
      "memory_percent": 78.2,
      "hostname": "web-server-01"
    }
  },
  "dedup_key": "web-server-01-cpu-high",
  "images": [{
    "src": "https://example.com/cpu-graph.png",
    "alt": "CPU Usage Graph"
  }],
  "links": [{
    "href": "https://grafana.example.com/d/cpu",
    "text": "View in Grafana"
  }]
}

Response 202 Accepted:
{
  "status": "success",
  "message": "Event processed"
}

3.2 Metrics API

POST /v2/metrics
Content-Type: application/json

{
  "metrics": [
    {
      "name": "cpu.utilization",
      "timestamp": 1724839800,
      "value": 95.5,
      "type": "gauge",
      "tags": {
        "host": "web-server-01",
        "region": "us-east-1",
        "env": "production"
      }
    },
    {
      "name": "memory.usage",
      "timestamp": 1724839800,
      "value": 4096,
      "type": "gauge",
      "tags": {
        "host": "web-server-01",
        "region": "us-east-1"
      }
    }
  ]
}

Response 204 No Content

3.3 Anomaly Detection Configuration

POST /v2/anomaly-detection/rules

{
  "name": "CPU Anomaly Detection",
  "metric": "cpu.utilization",
  "tags": { "env": "production" },
  "algorithm": "seasonal_decomposition",
  "parameters": {
    "sensitivity": 0.85,
    "min_duration_seconds": 300,
    "seasonal_periods": [1440, 10080],
    "learning_period_days": 14
  },
  "notification_policy": "critical-escalation",
  "enabled": true
}

Response 201 Created:
{
  "id": "anomaly-rule-abc123",
  "status": "active",
  "created_at": "2026-08-28T10:30:00Z"
}

3.4 On-Call Schedule API

POST /v2/schedules

{
  "name": "Platform Team On-Call",
  "timezone": "America/New_York",
  "description": "Primary on-call rotation for platform team",
  "overrides": [
    {
      "user_id": "PABC123",
      "start": "2026-09-01T00:00:00Z",
      "end": "2026-09-08T00:00:00Z"
    }
  ],
  "rotation": {
    "type": "weekly",
    "start": "2026-09-01T00:00:00Z",
    "users": [
      { "user_id": "PABC123", "name": "Alice" },
      { "user_id": "PDEF456", "name": "Bob" },
      { "user_id": "PGHI789", "name": "Charlie" }
    ]
  }
}

GET /v2/schedules/{scheduleId}/users?time=2026-08-28T10:00:00Z

Response 200:
{
  "users": [
    {
      "user_id": "PABC123",
      "name": "Alice",
      "email": "alice@example.com",
      "phone_numbers": [
        { "number": "+1-555-01001", "type": "mobile" }
      ]
    }
  ]
}

3.5 Escalation Policy API

POST /v2/escalation_policies

{
  "name": "Critical Infrastructure Escalation",
  "num_loops": 3,
  "escalation_rules": [
    {
      "escalation_delay_in_minutes": 5,
      "targets": [
        { "target_id": "schedule-abc123", "target_type": "schedule" }
      ]
    },
    {
      "escalation_delay_in_minutes": 10,
      "targets": [
        { "target_id": "PDEF456", "target_type": "user" }
      ]
    },
    {
      "escalation_delay_in_minutes": 15,
      "targets": [
        { "target_id": "PGHI789", "target_type": "user" }
      ]
    }
  ]
}

3.6 Incident API

GET /v2/incidents?status=triggered&time_range=last_24h

Response 200:
{
  "incidents": [
    {
      "id": "PINC123",
      "status": "triggered",
      "created_at": "2026-08-28T10:30:00Z",
      "title": "High CPU on web-server-01",
      "priority": "P1",
      "urgency": "high",
      "assignments": [
        {
          "assignee": { "id": "PABC123", "summary": "Alice" },
          "at": "2026-08-28T10:30:05Z"
        }
      ],
      "escalation_policy": { "id": "PEPOL123" },
      "alert_count": 3,
      "first_trigger_log_entry": {
        "id": "PLE123",
        "created_at": "2026-08-28T10:30:00Z",
        "kind": "trigger"
      }
    }
  ]
}

POST /v2/incidents/{incidentId}/manage
{
  "type": "acknowledge"
}

4. High-Level Architecture

graph TB
    subgraph "Data Sources"
        AGENTS[Monitoring Agents]
        INTEGRATIONS[200+ Integrations]
        API[Events API]
        METRICS[Metrics API]
        SYNTHETIC[Synthetic Checks]
    end

    subgraph "Ingestion Layer"
        API_GW[API Gateway]
        AUTH[Auth Service]
        RATE_LIMIT[Rate Limiter]
        KAFKA[Kafka Cluster]
    end

    subgraph "Processing Layer"
        subgraph "Metric Processing"
            METRIC_INGEST[Metric Ingestion]
            METRIC_STORE[Time-Series DB]
            DOWNSAMPLE[Downsampler]
        end

        subgraph "Anomaly Detection"
            ANOMALY_ENGINE[Anomaly Detection Engine]
            ML_MODELS[ML Model Store]
            BASELINE[Baseline Learner]
        end

        subgraph "Alert Processing"
            THRESHOLD[Threshold Evaluator]
            ANOMALY_ALERT[Anomaly Alert Generator]
            DEDUP[Alert Deduplicator]
            GROUPER[Alert Grouping Engine]
        end
    end

    subgraph "Incident Management"
        INCIDENT_SVC[Incident Service]
        ESCALATION[Escalation Engine]
        SCHEDULE_SVC[Schedule Service]
        NOTIFICATION[Notification Service]
    end

    subgraph "Notification Channels"
        PUSH[Push Notifications]
        SMS[SMS Gateway]
        EMAIL[Email Service]
        VOICE[Voice Call Service]
        SLACK[Slack Integration]
        TEAMS[Teams Integration]
    end

    subgraph "Storage"
        TSDB[Time-Series DB]
        INCIDENT_DB[Incident DB]
        SCHEDULE_DB[Schedule DB]
        USER_DB[User DB]
        CONFIG_DB[Config DB]
    end

    subgraph "Dashboard & API"
        WEB_UI[Web Dashboard]
        MOBILE_APP[Mobile App]
        REST_API[REST API]
        GRAPHQL[GraphQL API]
    end

    AGENTS --> API_GW
    INTEGRATIONS --> API_GW
    API --> API_GW
    METRICS --> API_GW
    SYNTHETIC --> API_GW

    API_GW --> AUTH
    AUTH --> RATE_LIMIT
    RATE_LIMIT --> KAFKA

    KAFKA --> METRIC_INGEST
    METRIC_INGEST --> METRIC_STORE
    METRIC_STORE --> DOWNSAMPLE
    DOWNSAMPLE --> METRIC_STORE

    METRIC_STORE --> ANOMALY_ENGINE
    ANOMALY_ENGINE --> ML_MODELS
    ANOMALY_ENGINE --> BASELINE
    BASELINE --> ML_MODELS

    METRIC_STORE --> THRESHOLD
    ANOMALY_ENGINE --> ANOMALY_ALERT

    THRESHOLD --> DEDUP
    ANOMALY_ALERT --> DEDUP
    DEDUP --> GROUPER
    GROUPER --> INCIDENT_SVC

    INCIDENT_SVC --> ESCALATION
    ESCALATION --> SCHEDULE_SVC
    ESCALATION --> NOTIFICATION

    SCHEDULE_SVC --> SCHEDULE_DB
    INCIDENT_SVC --> INCIDENT_DB
    METRIC_STORE --> TSDB

    NOTIFICATION --> PUSH
    NOTIFICATION --> SMS
    NOTIFICATION --> EMAIL
    NOTIFICATION --> VOICE
    NOTIFICATION --> SLACK
    NOTIFICATION --> TEAMS

    WEB_UI --> REST_API
    MOBILE_APP --> REST_API
    REST_API --> INCIDENT_SVC
    REST_API --> SCHEDULE_SVC
    REST_API --> METRIC_STORE
    GRAPHQL --> INCIDENT_SVC
    GRAPHQL --> METRIC_STORE

5. Core Component Design

5.1 Metric Ingestion Pipeline

Design Goals: Maximum throughput, low latency, exactly-once processing

graph LR
    subgraph "Ingestion"
        CLIENT[Client] --> GW[API Gateway]
        GW --> VALIDATE[Schema Validator]
        VALIDATE --> KAFKA[Kafka Topic: raw-metrics]
    end

    subgraph "Processing"
        KAFKA --> CONSUME[Consumer Group]
        CONSUME --> PARSE[Parse & Normalize]
        PARSE --> TAG[Tag Indexing]
        TAG --> STORE[Time-Series DB Write]
    end

    subgraph "Fan-out"
        STORE --> THRESHOLD[Threshold Check]
        STORE --> ANOMALY[Anomaly Detection]
        STORE --> DOWNSAMPLE[Downsampling]
    end

Ingestion Design:

  • Batch writes: Accumulate metrics in memory (up to 1000 or 100ms), write in batch
  • Schema validation: Lightweight validation at ingestion (type check, required fields)
  • Tag normalization: Tags indexed for fast querying
  • Time-series DB: Write-optimized columnar storage (e.g., VictoriaMetrics, TimescaleDB)

Metric Types:

Type Description Storage Example
Gauge Point-in-time value Latest value + history CPU%, memory
Counter Monotonically increasing Rate calculation Requests/sec
Histogram Distribution of values Buckets + percentiles Response time
Set Unique values HyperLogLog Unique users

5.2 Anomaly Detection Engine

Multi-Algorithm Architecture:

graph TB
    METRIC[Time-Series Metric] --> ROUTE{Algorithm Router}

    ROUTE -->|Seasonal| SEASONAL[Seasonal Decomposition]
    ROUTE -->|Trending| TREND[Trend-Based Detection]
    ROUTE -->|Spike| SPIKE[Spike Detection]
    ROUTE -->|Change| CHANGE[Change Point Detection]
    ROUTE -->|Custom| CUSTOM[Custom ML Model]

    SEASONAL --> SCORE1[Anomaly Score]
    TREND --> SCORE2[Anomaly Score]
    SPIKE --> SCORE3[Anomaly Score]
    CHANGE --> SCORE4[Anomaly Score]
    CUSTOM --> SCORE5[Anomaly Score]

    SCORE1 --> ENSEMBLE[Ensemble Scorer]
    SCORE2 --> ENSEMBLE
    SCORE3 --> ENSEMBLE
    SCORE4 --> ENSEMBLE
    SCORE5 --> ENSEMBLE

    ENSEMBLE --> FINAL{Final Score > Threshold?}
    FINAL -->|Yes| ALERT[Generate Anomaly Alert]
    FINAL -->|No| NORMAL[Normal]

Algorithm Details:

Algorithm Use Case Input Output Latency
Seasonal Decomposition (STL) Metrics with daily/weekly patterns 14+ days history Anomaly score per point 10ms
Holt-Winters Exponential Smoothing Trending metrics 7+ days history Forecast + confidence interval 5ms
Isolation Forest Multi-dimensional anomalies Recent window + features Anomaly probability 2ms
Change Point Detection (PELT) Sudden shifts Recent window Change point locations 3ms
Median Absolute Deviation Simple spike detection Recent window Deviation score 1ms
Custom ML (LSTM/Autoencoder) Complex patterns Trained model Reconstruction error 50ms

Seasonal Decomposition (STL) Deep Dive:

Input: Time series Y(t) for t = 1, 2, ..., n

Step 1: Decompose Y(t) = Trend(t) + Seasonal(t) + Residual(t)
  - Trend: Low-pass filtered series (moving average)
  - Seasonal: Repeating pattern (daily, weekly)
  - Residual: What's left after removing trend and seasonal

Step 2: Compute anomaly score
  - score(t) = |Residual(t)| / MAD(Residual)
  - MAD = Median Absolute Deviation of residuals

Step 3: Classify
  - If score(t) > threshold → anomaly
  - Threshold configurable (default: 3.0 MAD)

Step 4: Post-processing
  - Minimum duration filter (avoid noise)
  - Hysteresis (avoid flapping)

Model Training Pipeline:

graph LR
    subgraph "Training"
        RAW[Historical Metrics] --> FEATURE[Feature Engineering]
        FEATURE --> TRAIN[Model Training]
        TRAIN --> EVAL[Evaluation]
        EVAL --> DEPLOY[Deploy Model]
    end

    subgraph "Inference"
        LIVE[Live Metrics] --> PREP[Preprocessing]
        PREP --> MODEL[Trained Model]
        MODEL --> SCORE[Anomaly Score]
    end

    subgraph "Feedback"
        SCORE --> FEEDBACK[User Feedback]
        FEEDBACK --> RETRAIN[Retraining Trigger]
        RETRAIN --> TRAIN
    end

5.3 Alert Processing Pipeline

Deduplication Strategy:

graph TD
    ALERT[Incoming Alert] --> CHECK{Dedup Key Exists?}

    CHECK -->|No| NEW[Create New Incident]
    CHECK -->|Yes| ACTIVE{Incident Active?}

    ACTIVE -->|Yes| APPEND[Append to Existing Incident]
    ACTIVE -->|No| REOPEN{Auto-Reopen?}

    REOPEN -->|Yes| REOPEN_INC[Reopen Incident]
    REOPEN -->|No| NEW2[Create New Incident]

    NEW --> GROUP{Grouping Rules?}
    APPEND --> GROUP
    REOPEN_INC --> GROUP
    NEW2 --> GROUP

    GROUP -->|Yes| MERGE{Merge with Existing?}
    GROUP -->|No| ESCALATE[Send to Escalation]

    MERGE -->|Yes| MERGE_INC[Merge Incidents]
    MERGE -->|No| ESCALATE

    MERGE_INC --> ESCALATE
    ESCALATE --> NOTIFY[Notify On-Call]

Deduplication Logic:

  • Dedup key: Customer-provided or auto-generated from (source + summary)
  • Dedup window: 1 hour (alerts with same key within 1 hour → same incident)
  • Auto-reopen: If alert re-triggers within 24 hours of resolution → reopen
  • Grouping: Alerts with same group tag → same incident

Alert Grouping:

Grouping Type Criteria Example
Source-based Same source All alerts from web-server-01
Tag-based Same tags All alerts with env=production
Time-based Same time window All alerts within 5 min
Composite Multiple criteria Same source + same severity

5.4 Escalation Engine

Escalation Flow:

sequenceDiagram
    participant INC as Incident
    participant ESC as Escalation Engine
    participant SCHED as Schedule Service
    participant NOTIFY as Notification Service
    participant USER as On-Call User

    INC->>ESC: Incident created
    ESC->>SCHED: Get on-call user for schedule
    SCHED-->>ESC: User Alice (primary)
    ESC->>NOTIFY: Send notification to Alice
    NOTIFY->>USER: Push + SMS to Alice

    alt Alice acknowledges within 5 min
        USER->>ESC: Acknowledge
        ESC->>INC: Incident acknowledged
    else Alice does not respond
        ESC->>SCHED: Get next escalation target
        SCHED-->>ESC: User Bob (secondary)
        ESC->>NOTIFY: Send notification to Bob
        NOTIFY->>USER: Voice call to Bob
    end

Escalation Policy Evaluation:

Input: Incident, Escalation Policy

Step 1: Resolve schedule → current on-call user(s)
  - Evaluate rotation rules
  - Check overrides and swaps
  - Handle timezone conversions

Step 2: Send notifications to Level 1 targets
  - Push notification (immediate)
  - SMS (immediate)
  - Email (immediate)

Step 3: Wait for acknowledgment (configurable timeout, default 5 min)
  - If acknowledged → stop escalation
  - If not acknowledged → proceed to next level

Step 4: Escalate to next level
  - Voice call (if configured)
  - Notify next target(s)

Step 5: Repeat until acknowledged or max loops reached

5.5 On-Call Schedule Service

Schedule Evaluation:

graph TB
    subgraph "Schedule Definition"
        ROTATION[Rotation Rules]
        OVERRIDES[Overrides]
        SWAPS[Swaps]
        HOLIDAYS[Holiday Calendar]
    end

    subgraph "Evaluation Engine"
        EVAL[Evaluation Engine]
        CACHE[Schedule Cache]
    end

    subgraph "Output"
        ONCALL[On-Call User(s)]
        NEXT[NEXT On-Call User]
        SCHEDULE[Full Schedule View]
    end

    ROTATION --> EVAL
    OVERRIDES --> EVAL
    SWAPS --> EVAL
    HOLIDAYS --> EVAL

    EVAL --> CACHE
    CACHE --> ONCALL
    CACHE --> NEXT
    CACHE --> SCHEDULE

Schedule Types:

Type Description Example
Fixed rotation Users rotate in order Alice → Bob → Charlie → Alice
Weighted rotation Users rotate with weights Alice(2) → Bob(1) → Charlie(1)
Follow-the-sun Geographic rotation US → EU → APAC
Layered Multiple layers active Primary + Secondary simultaneously
Custom User-defined rules Business hours only, weekends off

Schedule Caching:

  • Pre-compute on-call assignments for next 30 days
  • Cache invalidated on schedule change
  • Evaluation: O(1) lookup from cache

5.6 Notification Service

Multi-Channel Delivery:

graph TB
    subgraph "Notification Service"
        QUEUE[Notification Queue]
        ROUTER[Channel Router]
        RETRY[Retry Manager]
    end

    subgraph "Channels"
        PUSH[Push Service]
        SMS[SMS Gateway]
        EMAIL[Email Service]
        VOICE[Voice Service]
        SLACK[Slack Webhook]
        TEAMS[Teams Webhook]
    end

    subgraph "Delivery Tracking"
        TRACK[Delivery Tracker]
        ACK[Acknowledgment Handler]
    end

    QUEUE --> ROUTER
    ROUTER --> PUSH
    ROUTER --> SMS
    ROUTER --> EMAIL
    ROUTER --> VOICE
    ROUTER --> SLACK
    ROUTER --> TEAMS

    PUSH --> RETRY
    SMS --> RETRY
    EMAIL --> RETRY
    VOICE --> RETRY

    RETRY --> TRACK
    TRACK --> ACK

Channel Priority and Fallback:

Priority Channel Timeout Retry
1 Push notification 30 sec 3 attempts
2 SMS 60 sec 3 attempts
3 Email 120 sec 2 attempts
4 Voice call 180 sec 2 attempts
5 Slack/Teams 30 sec 2 attempts

Notification Throttling:

  • Per-user: Max 10 notifications per hour (configurable)
  • Per-incident: Max 20 notifications total
  • Quiet hours: Configurable per-user quiet hours (no non-critical notifications)

6. Data Model

6.1 Metric Schema

-- Time-series metric data (stored in TSDB, shown as relational for clarity)
CREATE TABLE metrics (
    metric_id         BIGSERIAL,
    customer_id       UUID NOT NULL,
    metric_name       VARCHAR(256) NOT NULL,
    timestamp         BIGINT NOT NULL,     -- Unix timestamp in milliseconds
    value             DOUBLE PRECISION NOT NULL,
    metric_type       VARCHAR(16) NOT NULL, -- gauge, counter, histogram, set
    tags              JSONB NOT NULL,       -- {"host": "web-01", "region": "us-east"}
    tags_hash         VARCHAR(64) GENERATED ALWAYS AS (
        md5(tags::text)
    ) STORED,
    PRIMARY KEY (customer_id, metric_name, tags_hash, timestamp)
);

-- Metric metadata
CREATE TABLE metric_metadata (
    customer_id       UUID NOT NULL,
    metric_name       VARCHAR(256) NOT NULL,
    display_name      VARCHAR(256),
    unit              VARCHAR(32),
    description       TEXT,
    tags              JSONB,
    created_at        TIMESTAMP NOT NULL,
    last_seen         TIMESTAMP NOT NULL,
    is_active         BOOLEAN DEFAULT TRUE,
    PRIMARY KEY (customer_id, metric_name)
);

-- Tag index for fast querying
CREATE TABLE metric_tags (
    customer_id       UUID NOT NULL,
    tag_key           VARCHAR(128) NOT NULL,
    tag_value         VARCHAR(256) NOT NULL,
    metric_count      BIGINT DEFAULT 0,
    PRIMARY KEY (customer_id, tag_key, tag_value)
);

-- Indexes
CREATE INDEX idx_metrics_query ON metrics(customer_id, metric_name, tags_hash, timestamp DESC);
CREATE INDEX idx_metrics_time ON metrics(timestamp DESC);

6.2 Incident Schema

CREATE TABLE incidents (
    incident_id       VARCHAR(10) PRIMARY KEY,  -- PINC123
    customer_id     UUID NOT NULL,
    status            VARCHAR(16) NOT NULL,     -- triggered, acknowledged, resolved
    urgency           VARCHAR(8) NOT NULL,             -- high, low
    title             VARCHAR(1024) NOT NULL,
    priority          VARCHAR(4),               -- P1, P2, P3, P4
    created_at        TIMESTAMP NOT NULL,
    acknowledged_at   TIMESTAMP,
    resolved_at       TIMESTAMP,
    last_status_change TIMESTAMP NOT NULL,
    escalation_policy_id VARCHAR(10),
    service_id        VARCHAR(10),
    alert_count       INT DEFAULT 0,
    group             VARCHAR(256),
    dedup_key         VARCHAR(256),
    custom_details    JSONB,
    timeline          JSONB,                    -- Array of timeline events
    tags              JSONB
);

CREATE TABLE incident_alerts (
    alert_id          VARCHAR(10) PRIMARY KEY,
    incident_id       VARCHAR(10) REFERENCES incidents(incident_id),
    status            VARCHAR(16) NOT NULL,     -- triggered, acknowledged, resolved, merged
    title             VARCHAR(1024),
    detail            TEXT,
    severity          VARCHAR(16),
    source            VARCHAR(256),
    created_at        TIMESTAMP NOT NULL,
    acknowledged_at   TIMESTAMP,
    resolved_at       TIMESTAMP,
    dedup_key         VARCHAR(256),
    custom_details    JSONB
);

CREATE TABLE incident_assignments (
    assignment_id     BIGSERIAL PRIMARY KEY,
    incident_id       VARCHAR(10) REFERENCES incidents(incident_id),
    user_id           VARCHAR(10) NOT NULL,
    assigned_at       TIMESTAMP NOT NULL,
    type              VARCHAR(16) NOT NULL,  -- assigned, acknowledged
    is_active         BOOLEAN DEFAULT TRUE
);

CREATE TABLE incident_log_entries (
    log_entry_id      VARCHAR(10) PRIMARY KEY,
    incident_id       VARCHAR(10) REFERENCES incidents(incident_id),
    kind              VARCHAR(32) NOT NULL,      -- trigger, acknowledge, resolve, escalation
    created_at        TIMESTAMP NOT NULL,
    user_id           VARCHAR(10),
    world_write     BOOLEAN DEFAULT FALSE,
    body              TEXT,
    dedup_key         VARCHAR(256),
    data              JSONB
);

-- Indexes
CREATE INDEX idx_incidents_customer ON incidents(customer_id, status, created_at DESC);
CREATE INDEX idx_incidents_alert ON incident_alerts(incident_id);
CREATE INDEX idx_incident_assignments ON incident_assignments(incident_id, is_active);
CREATE INDEX idx_incident_logs ON incident_log_entries(incident_id, created_at DESC);

6.3 Schedule Schema

CREATE TABLE schedules (
    schedule_id       VARCHAR(10) PRIMARY KEY,  -- PSCH123
    customer_id       UUID NOT NULL,
    name              VARCHAR(256) NOT NULL,
    description       TEXT,
    timezone          VARCHAR(64) NOT NULL DEFAULT 'UTC',
    type              VARCHAR(16) NOT NULL,      -- rotation, calendar, embedded
    created_at        TIMESTAMP NOT NULL,
    updated_at        TIMESTAMP NOT NULL,
    final_schedule    JSONB,                     -- Computed schedule
    overrides         JSONB,                     -- Override entries
    restrictions      JSONB,                     -- Time restrictions
    starts_at         TIMESTAMP,
    ends_at           TIMESTAMP
);

CREATE TABLE schedule_rotations (
    rotation_id       BIGSERIAL PRIMARY KEY,
    schedule_id       VARCHAR(10) REFERENCES schedules(schedule_id),
    name              VARCHAR(256),
    type              VARCHAR(16) NOT NULL,      -- round_robin, single
    start             TIMESTAMP NOT NULL,
    end               TIMESTAMP,
    operation         VARCHAR(16) NOT NULL,      -- add, remove
    rotation_length_seconds BIGINT NOT NULL,
    users             JSONB NOT NULL,   -- Array of user objects with weights
    is_primary        BOOLEAN DEFAULT TRUE
);

CREATE TABLE schedule_overrides (
    override_id       BIGSERIAL PRIMARY KEY,
    schedule_id       VARCHAR(10) REFERENCES schedules(schedule_id),
    user_id           VARCHAR(10) NOT NULL,
    start             TIMESTAMP NOT NULL,
    end               TIMESTAMP NOT NULL,
    created_by        VARCHAR(10),
    created_at        TIMESTAMP NOT NULL
);

-- Pre-computed on-call assignments (cached)
CREATE TABLE schedule_assignments (
    schedule_id       VARCHAR(10) REFERENCES schedules(schedule_id),
    user_id           VARCHAR(10) NOT NULL,
    start             TIMESTAMP NOT NULL,
    end               TIMESTAMP NOT NULL,
    is_primary        BOOLEAN DEFAULT TRUE,
    PRIMARY KEY (schedule_id, user_id, start)
);

CREATE INDEX idx_schedule_assignments_time ON schedule_assignments(schedule_id, start, end);

6.4 Escalation Policy Schema

CREATE TABLE escalation_policies (
    policy_id         VARCHAR(10) PRIMARY KEY,  -- PEPOL123
    customer_id     UUID NOT NULL,
    name            VARCHAR(256) NOT NULL,
    num_loops       INT DEFAULT 1,
    created_at      TIMESTAMP NOT NULL,
    updated_at      TIMESTAMP NOT NULL
);

CREATE TABLE escalation_rules (
    rule_id           BIGSERIAL PRIMARY KEY,
    policy_id         VARCHAR(10) REFERENCES escalation_policies(policy_id),
    position        INT NOT NULL,
    escalation_delay_minutes INT NOT NULL,
    targets           JSONB NOT NULL,  -- Array of target objects
    created_at      TIMESTAMP NOT NULL
);

CREATE TABLE services (
    service_id      VARCHAR(10) PRIMARY KEY,
    customer_id     UUID NOT NULL,
    name            VARCHAR(256) NOT NULL,
    description     TEXT,
    escalation_policy_id VARCHAR(10) REFERENCES escalation_policies(policy_id),
    alert_creation_type VARCHAR(16) DEFAULT 'all',
    auto_resolve_timeout INT DEFAULT 0,
    ack_timeout       INT DEFAULT 0,
    created_at      TIMESTAMP NOT NULL
);

6.5 Anomaly Detection Configuration

CREATE TABLE anomaly_rules (
    rule_id           UUID PRIMARY KEY,
    customer_id       UUID NOT NULL,
    name              VARCHAR(256) NOT NULL,
    metric_name       VARCHAR(256) NOT NULL,
    tags_filter       JSONB,
    algorithm         VARCHAR(64) NOT NULL,
    parameters        JSONB NOT NULL,
    sensitivity       DOUBLE PRECISION DEFAULT 0.85,
    min_duration_seconds INT DEFAULT 300,
    learning_period_days INT DEFAULT 14,
    enabled             BOOLEAN DEFAULT TRUE,
    created_at        TIMESTAMP NOT NULL,
    updated_at        TIMESTAMP NOT NULL
);

CREATE TABLE anomaly_models (
    model_id          UUID PRIMARY KEY,
    rule_id           UUID REFERENCES anomaly_rules(rule_id),
    metric_name       VARCHAR(256) NOT NULL,
    tags_hash         VARCHAR(64) NOT NULL,
    model_type        VARCHAR(64) NOT NULL,
    model_data        BYTEA,                    -- Serialized model
    training_data_start TIMESTAMP NOT NULL,
    training_data_end TIMESTAMP NOT NULL,
    accuracy          DOUBLE PRECISION,
    created_at        TIMESTAMP NOT NULL,
    updated_at        TIMESTAMP NOT NULL,
    is_active         BOOLEAN DEFAULT TRUE,
    PRIMARY KEY (model_id)
);

CREATE TABLE anomaly_events (
    event_id          UUID PRIMARY KEY,
    rule_id           UUID REFERENCES anomaly_rules(rule_id),
    metric_name       VARCHAR(256) NOT NULL,
    timestamp         TIMESTAMP NOT NULL,
    actual_value      DOUBLE PRECISION,
    expected_value    DOUBLE PRECISION,
    anomaly_score     DOUBLE PRECISION NOT NULL,
    severity          VARCHAR(16),
    duration_seconds  INT,
    is_resolved       BOOLEAN DEFAULT FALSE,
    resolved_at     TIMESTAMP,
    created_at        TIMESTAMP NOT NULL
);

CREATE INDEX idx_anomaly_events_rule ON anomaly_events(rule_id, timestamp DESC);
CREATE INDEX idx_anomaly_events_metric ON anomaly_events(metric_name, timestamp DESC);

7. Key Design Decisions

7.1 Alerting: Push vs Pull

Decision: Push-based alerting with pull-based fallback

Approach Pros Cons Our Choice
Push Low latency, real-time Complex, requires persistent connections Primary
Pull Simple, reliable Higher latency, polling overhead Fallback

Rationale: Push provides sub-30-second alert-to-notification latency. Pull used for:

  • Dashboard polling (15-second interval)
  • Mobile app background sync
  • Integration webhooks (retry on failure)

7.2 Anomaly Detection: Rule-Based vs ML-Based

Decision: Hybrid approach with automatic algorithm selection

Approach Pros Cons Our Choice
Rule-based (thresholds) Simple, predictable, no training Manual tuning, doesn’t adapt For simple metrics
ML-based Automatic, adapts to patterns Complex, needs training data, false positives For complex metrics
Hybrid Best of both More complex Default

Algorithm Selection Logic:

Input: Metric time series

Step 1: Analyze metric characteristics
  - Has seasonal pattern? → STL decomposition
  - Has trend? → Holt-Winters
  - Stable baseline? → MAD-based
  - Sudden shifts? → Change point detection
  - Multi-dimensional? → Isolation Forest

Step 2: Train selected algorithm
  - Use 14 days of historical data
  - Validate against known anomalies (if available)

Step 3: Deploy and monitor
  - Track false positive rate
  - Auto-retrain if performance degrades

7.3 Incident Grouping: Aggressive vs Conservative

Decision: Configurable grouping with conservative defaults

Approach Pros Cons
Aggressive Fewer incidents, less noise May hide unrelated issues
Conservative Accurate, no missed issues More incidents, more noise
Configurable Customer choice Requires customer tuning

Default Grouping Rules:

  • Same source + same severity + within 5 minutes → group
  • Same service + within 15 minutes → group
  • Customer can customize grouping rules per service

7.4 Notification: All-at-Once vs Escalation

Decision: Escalation-based with configurable parallel notification

Approach Pros Cons
All-at-once Fastest response Notification fatigue, confusion
Escalation Ordered, clear ownership Slower if primary doesn’t respond
Hybrid Fast + ordered Complex

Default Behavior:

  • Level 1: Push + SMS simultaneously
  • Level 2 (after timeout): Voice call + SMS
  • Level 3 (after timeout): Notify manager + all team members

7.5 Data Retention: Raw vs Downsampled

Decision: Tiered retention with automatic downsampling

Tier Retention Resolution Storage
Raw 15 days Collection interval (15s-5min) High
1-minute 90 days 1-minute averages Medium
1-hour 1 year 1-hour averages Low
1-day 2 years 1-day averages Minimal

Downsampling Strategy:

  • Aggregate: avg, min, max, sum, count per window
  • Preserve: percentiles (p50, p95, p99) for latency metrics
  • Delete raw data after 15 days (configurable)

8. Scalability Strategy

8.1 Multi-Region Deployment

graph TB
    subgraph "Primary Region (US-East)"
        US_API[API Gateway]
        US_KAFKA[Kafka Cluster]
        US_TSDB[Time-Series DB]
        US_INCIDENT[Incident Service]
        US_SCHEDULE[Schedule Service]
    end

    subgraph "Secondary Region (EU-West)"
        EU_API[API Gateway]
        EU_KAFKA[Kafka Cluster]
        EU_TSDB[Time-Series DB]
        EU_INCIDENT[Incident Service]
        EU_SCHEDULE[Schedule Service]
    end

    subgraph "Tertiary Region (APAC)"
        AP_API[API Gateway]
        AP_KAFKA[Kafka Cluster]
        AP_TSDB[Time-Series DB]
        AP_INCIDENT[Incident Service]
        AP_SCHEDULE[Schedule Service]
    end

    GLOBAL_DNS[Global DNS] --> US_API
    GLOBAL_DNS --> EU_API
    GLOBAL_DNS --> AP_API

    US_TSDB <-->|Async Replication| EU_TSDB
    EU_TSDB <-->|Async Replication| AP_TSDB

Customer Sharding:

  • Customers assigned to regions via consistent hashing
  • Each customer’s data stays in one primary region
  • Cross-region replication for disaster recovery

8.2 Auto-Scaling Triggers

Component Scale-Up Trigger Scale-Down Trigger Min Max
API Gateway CPU > 70% for 1 min CPU < 30% for 5 min 10 100
Kafka Brokers Disk > 70% Disk < 40% 6 50
TSDB Nodes Write latency > 100ms Write latency < 20ms 4 40
Anomaly Engine Queue depth > 10K Queue depth < 1K 4 50
Alert Processor Queue depth > 1K Queue depth < 100 2 20
Notification Queue depth > 5K Queue depth < 500 4 30

8.3 Metric Partitioning Strategy

Time-Series DB Partitioning:

  • Partition by customer_id (hash) → metric_name → timestamp (range)
  • Each partition: ~10 GB per month for average customer
  • Hot partitions (recent data) on SSD, cold partitions on HDD

Tag Indexing:

  • Inverted index: tag_key + tag_value → list of (customer_id, metric_name)
  • Enables fast queries: “show all CPU metrics for env=production”

9. Curveball Scenarios

9.1 Curveball 1: A Cascading Failure Generates 100K Simultaneous Alerts

Scenario: A major cloud provider outage causes all customer services to fail simultaneously. 100K alerts fire within 1 minute across 10K customers.

Impact Analysis:

  • Alert processing pipeline overwhelmed
  • Notification system flooded (millions of notifications)
  • On-call engineers receive hundreds of alerts
  • Incident management UI becomes unresponsive

Response Strategy:

Layer Action Timeline
Ingestion Rate limiting per customer. Queue excess alerts for processing. Immediate
Deduplication Aggressive deduplication: same source + same time window → single incident. Immediate
Grouping Automatic mega-grouping: all alerts from same cloud provider → single incident per customer. 1-2 min
Notification Notification throttling: max 1 notification per customer per minute during surge. Immediate
Escalation Pause escalation during surge. Resume after surge subsides. 5 min
UI Read-only mode for incident list. Disable non-essential features. 1 min
Post-Surge Process queued alerts. Send summary notifications. 15-30 min

Prevention:

  • Surge detection: Alert rate > 10× normal → activate surge mode
  • Pre-defined grouping rules for known cascade patterns
  • Notification rate limiting with customer-configurable thresholds

9.2 Curveball 2: An On-Call Engineer’s Phone is Unreachable for 24 Hours

Scenario: The primary on-call engineer loses phone service (dead battery, lost phone, no coverage). Critical incidents are not being acknowledged.

Impact Analysis:

  • Escalation policy should handle this, but what if all levels are unreachable?
  • Critical incidents may go unacknowledged
  • Business impact depends on incident severity

Response Strategy:

Phase Action Timeline
Detection Escalation engine detects no acknowledgment after all levels exhausted. 15-30 min
Escalation Final escalation: notify all team members, manager, and executive sponsor. 30-60 min
Auto-Assignment If configured: auto-assign to backup on-call or team lead. 1 hr
Bridge Call Auto-create conference bridge and invite all available team members. 1 hr
Resolution Team collaborates to resolve incident. Variable
Post-Incident Review escalation policy. Add backup contacts. Post-incident

Prevention:

  • Multi-contact requirement: Each on-call user must have at least 2 contact methods
  • Health check: Periodic check-in with on-call users (daily ping)
  • Backup on-call: Required backup for each schedule
  • Executive escalation: Final escalation to management for unacknowledged P1 incidents

9.3 Curveball 3: A False Positive Storm from ML Anomaly Detection

Scenario: A system deployment changes the behavior of 1,000 metrics. The ML anomaly detection model hasn’t learned the new patterns yet, generating 50,000 false positive alerts in 1 hour.

Impact Analysis:

  • Alert fatigue for on-call engineers
  • Real incidents may be missed in the noise
  • Trust in anomaly detection eroded

Response Strategy:

Phase Action Timeline
Detection Anomaly detection self-monitoring: false positive rate > 10% → alert. 15 min
Mitigation Automatic sensitivity reduction for affected metrics. Pause anomaly alerts for metrics with > 50% false positive rate. 15-30 min
Communication Notify customers of increased false positive rate and mitigation actions. 30 min
Retraining Trigger immediate model retraining with recent data. 1-2 hrs
Validation Validate retrained models against recent data before re-enabling. 2-4 hrs
Gradual Re-enable Re-enable anomaly detection gradually with increased thresholds. 4-8 hrs

Prevention:

  • Change correlation: Correlate anomaly spikes with deployment events
  • Grace period: 24-hour grace period after deployments (reduced sensitivity)
  • Self-monitoring: Track false positive rate per model, auto-adjust sensitivity
  • Feedback loop: User feedback on false positives triggers immediate retraining

9.4 Curveball 4: SMS Gateway Provider Goes Down

Scenario: The primary SMS gateway provider experiences a global outage. All SMS notifications fail for 4 hours.

Impact Analysis:

  • 10% of notifications are SMS-only for some users
  • Critical alerts may not reach on-call engineers
  • Escalation chains broken

Response Strategy:

Phase Action Timeline
Detection SMS delivery failure rate > 5% → alert. 5 min
Failover Switch to backup provider. 5-10 min
Fallback For users with SMS-only: escalate to voice call immediately. 10-15 min
Notification Notify customers of SMS degradation and fallback behavior. 15 min
Recovery Primary provider restored → gradual traffic shift back. 4+ hrs
Post-Incident Add third SMS provider. Require multi-channel for critical alerts. Post-incident

Prevention:

  • Multi-provider: At least 2 SMS providers with automatic failover
  • Multi-channel requirement: Critical alerts require at least 2 delivery channels
  • Delivery confirmation: Track delivery status, retry with alternative channel on failure

9.5 Curveball 5: A Customer Accidentally Creates an Infinite Escalation Loop

Scenario: A customer misconfigures their escalation policy, creating a circular reference that causes infinite notification loops. The system sends millions of notifications in minutes.

Impact Analysis:

  • Notification system overwhelmed
  • SMS/voice costs spike
  • Other customers affected

Response Strategy:

Phase Action Timeline
Detection Notification rate anomaly: single customer > 1000 notifications/min → alert. 1-2 min
Containment Automatic circuit breaker: pause notifications for the customer. 2-3 min
Assessment Identify the misconfigured escalation policy. 5 min
Remediation Disable the problematic escalation policy. Notify customer. 5-10 min
Recovery Resume notifications with validation. 10-15 min
Prevention Add escalation policy validation: detect circular references before saving. Post-incident

Prevention:

  • Policy validation: Validate escalation policies for circular references on save
  • Notification budget: Per-customer notification budget with hard limits
  • Rate limiting: Max notifications per customer per hour (configurable, default 1000)
  • Cost monitoring: Real-time notification cost monitoring with alerts

10. Failure Modes and Mitigations

10.1 Failure Mode Matrix

Failure Mode Probability Impact Detection Mitigation RTO Data Loss
Metric ingestion failure Medium Medium Write error rate Multi-region failover, client retry 30 sec Client buffer
Kafka broker failure Low High Broker health monitoring Auto-replace, ISR management 1 min 0
TSDB node failure Low Medium Query latency monitoring Multi-AZ replication, failover 30 sec 0
Anomaly engine failure Low Low Queue depth monitoring Fallback to threshold-only 1 min 0
Alert processing failure Low High Alert queue depth Redundant processing, retry 30 sec 0
Notification delivery failure Medium High Delivery failure rate Multi-channel fallback 1 min 0
Schedule service failure Low Medium Query latency monitoring Cached schedules, fallback 1 min 0
Escalation engine failure Low Critical Escalation timeout monitoring Manual escalation, backup contacts 5 min 0
Database corruption Very Low Critical Checksum validation Backup restoration 1 hr 0
DDoS attack Medium Medium Traffic anomaly detection Rate limiting, WAF 5 min 0
Third-party integration failure Medium Low Integration health monitoring Graceful degradation 5 min 0

10.2 Notification Delivery Reliability

graph TB
    subgraph "Notification Pipeline"
        QUEUE[Notification Queue]
        ROUTER[Channel Router]
        DELIVER[Delivery Engine]
        TRACK[Delivery Tracker]
    end

    subgraph "Delivery Channels"
        PUSH[Push]
        SMS1[SMS Provider A]
        SMS2[SMS Provider B]
        EMAIL[Email]
        VOICE[Voice]
    end

    subgraph "Fallback Logic"
        RETRY[Retry Manager]
        ESCALATE[Escalation]
    end

    QUEUE --> ROUTER
    ROUTER --> DELIVER
    DELIVER --> PUSH
    DELIVER --> SMS1
    DELIVER --> EMAIL
    DELIVER --> VOICE

    SMS1 -->|Fail| SMS2
    PUSH -->|Fail| RETRY
    SMS1 -->|Fail| RETRY
    SMS2 -->|Fail| RETRY
    EMAIL -->|Fail| RETRY
    VOICE -->|Fail| RETRY

    RETRY --> TRACK
    TRACK -->|All failed| ESCALATE
    ESCALATE --> DELIVER

Delivery Guarantee:

  • At-least-once: Notifications delivered at least once (idempotent delivery)
  • Multi-channel: Critical alerts sent via multiple channels simultaneously
  • Retry: Exponential backoff (1s, 5s, 30s, 5 min), max 5 attempts
  • Fallback: If primary channel fails, try alternative channels
  • Confirmation: Delivery confirmation tracked, used for escalation decisions

10.3 Incident Management Resilience

Incident Service HA:

graph TB
    subgraph "Incident Service HA"
        subgraph "AZ-1a"
            INC1[Incident Service Primary]
            DB1[Incident DB Primary]
        end
        subgraph "AZ-1b"
            INC2[Incident Service Standby]
            DB2[Incident DB Replica]
        end
        subgraph "AZ-1c"
            INC3[Incident Service Standby]
            DB3[Incident DB Replica]
        end
    end

    LB[Load Balancer] --> INC1
    LB --> INC2
    LB --> INC3

    INC1 --> DB1
    INC2 --> DB2
    INC3 --> DB3

    DB1 <-->|Sync Replication| DB2
    DB2 <-->|Sync Replication| DB3

    HEALTH[Health Checker] --> INC1
    HEALTH --> INC2
    HEALTH --> INC3

Key Principles:

  • Quorum-based writes: Incident state changes require 2-of-3 AZ acknowledgment
  • Automatic leader election: If primary fails, standby takes over within 30 seconds
  • Idempotent operations: All incident operations are idempotent (safe to retry)
  • Event sourcing: All incident changes stored as events, state derived from events

10.4 Disaster Recovery Plan

Scenario Strategy RTO RPO
AZ failure Cross-AZ failover (automatic) 1 min 0
Region failure Multi-region failover (automatic for metrics, manual for incidents) 5 min 0
Global failure Not designed for (requires geographic distribution at customer level) N/A N/A
Database corruption Backup restoration + event replay 30 min 0
Notification provider failure Multi-provider failover 5 min 0
ML model corruption Fallback to threshold-only detection 1 min 0

Appendix: Glossary

Term Definition
STL Seasonal and Trend decomposition using Loess
MAD Median Absolute Deviation
PELT Pruned Exact Linear Time (change point detection algorithm)
HLL HyperLogLog - Probabilistic distinct counting
ISR In-Sync Replicas - Kafka replication mechanism
TSDB Time-Series Database
MTTR Mean Time To Resolution
MTTA Mean Time To Acknowledge
SLA Service Level Agreement
SLO Service Level Objective
SLI Service Level Indicator
Runbook Documented procedures for incident response
Postmortem Post-incident review document
Blameless Postmortem culture that focuses on process, not people