Video & StreamingQuestion #18ADVANCED

ESPN Live: Ultra-Low Latency Live Streaming, Transcoding & DVR

Broadcast-grade live media delivery system handling raw RTMP/SRT stream ingest, hardware chunked ABR transcoding, global CDN multi-origin distribution, and instant rolling DVR playback.

Category: Category 12: Streaming & Video Platforms•Updated: 2026-08-30
#Video#Live Ingest#Transcoding#CDN#Low-latency#DVR

Design a Live-Streaming Broadcast App like ESPN

Scope: A professional-grade live sports broadcasting platform delivering low-latency, high-reliability live streams with DVR/rewind, multi-camera support, and real-time stats overlays to millions of concurrent viewers.


Table of Contents

  1. Requirements Clarification
  2. Back-of-Envelope Calculations
  3. System Interface Design
  4. High-Level Architecture
  5. Core Component Design
  6. Data Model
  7. Key Design Decisions
  8. Scalability Strategy
  9. Curveball Scenarios
  10. Failure Modes and Mitigations

1. Requirements Clarification

1.1 Functional Requirements

  • Live Ingest: Accept live video feeds from broadcast trucks, studios, and remote venues via multiple protocols (RTMP, SRT, RTP/RTSP, HLS ingest)
  • Transcoding Pipeline: Real-time transcoding into multiple bitrates/resolutions (SD, HD, Full HD, 4K) with adaptive bitrate ladders
  • Multi-Camera Support: Deliver 2-8 simultaneous camera angles per event; viewers can switch angles in real-time
  • DVR / Time-Shift Rewind: Allow viewers to rewind up to 3 hours back within a live broadcast
  • Real-Time Stats Overlay: Superimpose live scores, player stats, and graphics onto the video stream
  • VOD Archive: Automatically archive completed broadcasts for on-demand replay
  • Multi-Platform Delivery: Support web browsers, iOS, Android, smart TVs, set-top boxes, and gaming consoles
  • Geo-Rights Management: Enforce regional licensing restrictions per broadcast
  • Simulcast / Multi-Event: Support broadcasting multiple events simultaneously (e.g., NFL Sunday with 14 concurrent games)
  • Closed Captioning & Multi-Language Audio: Support CC and multiple audio tracks (English, Spanish, etc.)
  • Push Notifications: Alert subscribers about game start, score changes, and breaking events

1.2 Non-Functional Requirements

Requirement Target Rationale
Live Latency 3-10 seconds (HLS LL / WebRTC optional) Sports fans need near-real-time; no spoilers from social media
Availability 99.99% uptime for live events A dropped stream during a championship is catastrophic
Concurrent Viewers 50M+ peak (Super Bowl scale) Must handle black-event traffic spikes
Stream Start Time < 2 seconds from click to first frame Viewer retention is critical
Rebuffer Ratio < 1% of total watch time Quality of experience metric
Transcoding Delay < 3 seconds end-to-end Pipeline must keep up with live ingest
Geo-Blocking Accuracy 99.9% enforcement Legal compliance for regional rights
Data Consistency Eventual consistency for stats; strong for billing Stats can lag slightly; billing must be exact
Scalability Horizontal scale for all stateless components Traffic is bursty and unpredictable

2. Back-of-Envelope Calculations

2.1 Assumption Sets

We consider three scenarios to bound our design:

Parameter Conservative (Weekday) Typical (Weekend) Black Event (Super Bowl)
Concurrent Viewers 500K 5M 50M
Live Events Concurrent 10 30 14 (NFL Sunday)
Avg Watch Duration 45 min 2 hours 3.5 hours
Avg Bitrate 3 Mbps 3.5 Mbps 4 Mbps
Multi-Camera Angles 2 3 4
Transcoding Profiles per Angle 5 5 5

2.2 Bandwidth Estimates

Outbound CDN Bandwidth:

Scenario Calculation Total Bandwidth
Weekday 500K × 3 Mbps 1.5 Tbps
Weekend 5M × 3.5 Mbps 17.5 Tbps
Super Bowl 50M × 4 Mbps 200 Tbps

Note: 200 Tbps is at the scale of major CDN backbone capacity. This requires pre-provisioning with CDN providers months in advance.

Ingest Bandwidth (per event, assuming 4K source at 50 Mbps):

Scenario Calculation Total Ingest
Weekday 10 × 50 Mbps 500 Mbps
Weekend 30 × 50 Mbps 1.5 Gbps
Super Bowl 14 × 50 Mbps 700 Mbps

2.3 Storage Estimates

DVR Window Storage (3-hour rewind window, per stream):

  • Per angle: 3 hours × 3 Mbps × 3600 sec = 4.05 GB
  • Per event (4 angles): 4 × 4.05 GB = 16.2 GB
  • 30 concurrent events: 30 × 16.2 GB = 486 GB hot storage

VOD Archive (daily):

  • 50 events/day × 3 hours × 3 Mbps × 4 angles = 606 GB/day raw
  • With 5 transcoding profiles: 606 GB × 5 = 3 TB/day
  • Monthly: ~90 TB/month

Total Annual Storage: ~1.1 PB/year (with compression and dedup, ~600 TB effective)

2.4 QPS Estimates

Operation Weekday Weekend Super Bowl
Stream Start Requests 10K/s 100K/s 1M/s
API Calls (metadata, stats) 50K/s 500K/s 5M/s
DVR Seek Requests 5K/s 50K/s 500K/s
Camera Switch Requests 2K/s 20K/s 200K/s
Auth Token Requests 20K/s 200K/s 2M/s
Chat / Social (if enabled) 100K/s 1M/s 10M/s

2.5 Transcoding Compute

  • Each transcoding profile: ~2-4 vCPU cores (hardware-accelerated with NVENC/QSV)
  • Per angle: 5 profiles × 3 cores = 15 cores
  • Per event (4 angles): 60 cores
  • 30 events: 1,800 GPU-accelerated cores minimum

3. System Interface Design

3.1 Public APIs

# --- Ingest APIs (Internal / Partner-Facing) ---

POST /api/v1/ingest/session
  Create a new ingest session; returns ingest URL and credentials
  Request:  { event_id, protocol, resolution, bitrate, camera_angle }
  Response: { session_id, ingest_url, stream_key, expires_at }

POST /api/v1/ingest/session/{id}/heartbeat
  Keep-alive for active ingest sessions
  Response: { status: "active" | "expired" }

DELETE /api/v1/ingest/session/{id}
  Terminate an ingest session gracefully

# --- Viewer APIs ---

GET /api/v1/events
  List available live and upcoming events
  Query: ?status=live&sport=nfl&geo=US
  Response: { events: [{ id, title, sport, start_time, status, camera_angles, geo_rights }] }

GET /api/v1/events/{id}
  Get event details
  Response: { id, title, description, start_time, end_time, status,
              camera_angles: [{ id, label, thumbnail }],
              stats_feed_url, chat_enabled, geo_rights }

GET /api/v1/events/{id}/stream/manifest
  Get HLS/DASH manifest for a specific camera angle
  Query: ?angle=main&format=hll&drm=widevine
  Response: HLS/DASH manifest (m3u8 / mpd)

GET /api/v1/events/{id}/stream/manifest/segment/{seq}
  Get a specific media segment (served by CDN)

POST /api/v1/events/{id}/dvr/seek
  Seek to a position in the DVR window
  Request: { position_seconds, angle_id }
  Response: { manifest_url, position }

GET /api/v1/events/{id}/stats
  Get real-time event statistics
  Response: { score, period, clock, possessions, player_stats, timeline }

GET /api/v1/events/{id}/archive
  Get VOD archive info for completed events
  Response: { archive_id, duration, available_from, manifest_url }

# --- User / Subscription APIs ---

POST /api/v1/auth/token
  Authenticate user; returns JWT + entitlements
  Request: { username, password } or { device_token }
  Response: { access_token, refresh_token, entitlements: ["live", "dvr", "4k"] }

GET /api/v1/user/entitlements
  Check user's subscription tier and geo rights
  Response: { tier, channels, geo_allowed, expires_at }

POST /api/v1/notifications/subscribe
  Subscribe to event notifications
  Request: { event_id, types: ["start", "score_change", "end"] }

# --- Admin / Operations APIs ---

POST /api/v1/admin/events
  Create/schedule a new broadcast event

PUT /api/v1/admin/events/{id}/go-live
  Manually trigger event to go live

GET /api/v1/admin/ingest/health
  Health status of all active ingest sessions

GET /api/v1/admin/analytics/realtime
  Real-time viewer counts, bitrate distribution, error rates

3.2 Internal Service APIs (gRPC)

// Transcoding Service
service TranscoderService {
  rpc StartTranscode(TranscodeRequest) returns (stream TranscodeStatus);
  rpc StopTranscode(StopRequest) returns (Empty);
  rpc GetPipelineHealth(HealthRequest) returns (PipelineHealth);
}

// DVR Service
service DVRService {
  rpc OpenWindow(DVROpenRequest) returns (DVRWindow);
  rpc WriteSegment(DVRWriteRequest) returns (Empty);
  rpc Seek(DVRSeekRequest) returns (DVRManifest);
  rpc CloseWindow(DVRCloseRequest) returns (Empty);
}

// Stats Overlay Service
service StatsOverlayService {
  rpc UpdateOverlay(OverlayUpdate) returns (Empty);
  rpc GetOverlayState(EventId) returns (OverlayState);
}

// Geo-Rights Service
service GeoRightsService {
  rpc CheckAccess(AccessRequest) returns (AccessResponse);
  rpc UpdateRights(RightsUpdate) returns (Empty);
}

4. High-Level Architecture

flowchart TB
    subgraph INGEST["📡 Ingest Layer"]
        BT[Broadcast Truck<br/>SRT / RTP]
        STUDIO[Studio Feed<br/>RTMP / SDI]
        REMOTE[Remote Venue<br/>SRT / HLS Ingest]
        IG[Ingest Gateway<br/>Protocol Adapter]
    end

    subgraph PROCESSING["⚙️ Processing Layer"]
        TC[Transcoding Cluster<br/>GPU-Accelerated]
        QC[Quality Checker<br/>PSNR / VMAF]
        STATS[Stats Ingestion<br/>Real-Time Feed]
        OVERLAY[Graphics Overlay<br/>Server-Side Rendering]
    end

    subgraph STORAGE["💾 Storage Layer"]
        DVR[DVR Buffer<br/>Object Storage + Index]
        VOD[VOD Archive<br/>Cold + Warm Storage]
        MANIFEST[Manifest Store<br/>Redis + CDN Origin]
    end

    subgraph DISTRIBUTION["🌐 Distribution Layer"]
        ORIGIN[Origin Server<br/>Edge Logic]
        CDN[CDN Edge POPs<br/>Global Distribution]
        EDGE[Edge Cache<br/>LL-HLS / DASH]
    end

    subgraph CLIENTS["📱 Client Layer"]
        WEB[Web Browser]
        MOBILE[iOS / Android]
        TV[Smart TV / STB]
        CONSOLE[Gaming Console]
    end

    subgraph SERVICES["🔧 Support Services"]
        AUTH[Auth Service<br/>JWT / OAuth]
        GEO[Geo-Rights Engine<br/>IP + Subscriber DB]
        NOTIF[Notification Service<br/>Push / SMS]
        ANALYTICS[Analytics Pipeline<br/>Kafka + Flink]
        ADMIN[Admin Dashboard<br/>Operations Console]
    end

    BT --> IG
    STUDIO --> IG
    REMOTE --> IG
    IG --> TC
    TC --> QC
    QC -->|Pass| OVERLAY
    QC -->|Fail| TC
    OVERLAY --> DVR
    OVERLAY --> ORIGIN
    STATS --> OVERLAY
    DVR --> ORIGIN
    VOD --> ORIGIN
    ORIGIN --> CDN
    CDN --> EDGE
    EDGE --> WEB
    EDGE --> MOBILE
    EDGE --> TV
    EDGE --> CONSOLE
    ORIGIN -.-> MANIFEST
    GEO -.-> ORIGIN
    AUTH -.-> ORIGIN
    NOTIF -.-> CLIENTS
    ANALYTICS -.-> ORIGIN
    ADMIN -.-> PROCESSING

5. Core Component Design

5.1 Ingest Gateway

Purpose: Accept incoming live feeds from diverse sources and normalize them into a unified internal format.

Supported Protocols:

Protocol Use Case Latency Reliability
SRT (Secure Reliable Transport) Primary for broadcast trucks 200-500ms FEC + ARQ, handles 20% packet loss
RTP/RTSP Legacy broadcast equipment 300-800ms UDP-based, needs RTCP feedback
RTMP Studios, backup feeds 1-3s TCP-based, widely supported
HLS Ingest Remote contributors, backup 5-10s HTTP-based, firewall-friendly

Design Details:

  • Protocol Adapters: Each protocol has a dedicated adapter microservice that normalizes to an internal raw stream format
  • Redundant Ingest: Each event accepts 2-3 simultaneous feeds (primary + backup); a failover controller selects the best feed
  • Health Monitoring: Continuous RTCP/SRT stats collection; automatic failover on quality degradation
  • Capacity: Each ingest node handles ~50 concurrent streams; scale horizontally

Failover Logic:

Primary Feed Quality Score = f(packet_loss, jitter, latency, resolution)
If Primary Score < Threshold for > 3 seconds:
    Switch to Backup Feed
    Notify Transcoding Pipeline of source change
    Log failover event for post-mortem

5.2 Transcoding Pipeline

Purpose: Convert incoming high-bitrate source streams into multiple delivery-optimized profiles in real-time.

Architecture:

flowchart LR
    SRC[Source Stream<br/>4K @ 50Mbps]
    
    subgraph ENCODE["Encoding Cluster"]
        E1[Encoder Node 1<br/>NVENC H.264]
        E2[Encoder Node 2<br/>NVENC H.265]
        E3[Encoder Node 3<br/>AV1 (select)]
    end
    
    subgraph PROFILES["Bitrate Ladder"]
        P1[4K @ 15Mbps]
        P2[1080p @ 8Mbps]
        P3[1080p @ 5Mbps]
        P4[720p @ 3Mbps]
        P5[480p @ 1.5Mbps]
        P6[360p @ 800Kbps]
    end
    
    subgraph AUDIO["Audio Processing"]
        A1[EN Track]
        A2[ES Track]
        A3[CC Track]
    end
    
    SRC --> ENCODE
    ENCODE --> PROFILES
    ENCODE --> AUDIO

Key Design Choices:

  • Hardware Acceleration: NVIDIA NVENC (H.264/H.265) for primary encoding; Intel QSV as fallback; AV1 via software for premium content
  • Segment Duration: 2-6 second segments for LL-HLS; 6-10 second for standard HLS
  • Keyframe Alignment: All bitrate variants share synchronized keyframes for seamless switching
  • Pipeline Parallelism: Each camera angle runs an independent pipeline; profiles within an angle share the decode stage

Transcoding Delay Budget:

Stage Budget
Ingest decode 200ms
GPU encode (per profile) 500ms
Segment packaging 300ms
Network to origin 500ms
Total Pipeline Delay ~2 seconds

5.3 DVR / Time-Shift Rewind

Purpose: Enable viewers to rewind up to 3 hours within a live broadcast.

Architecture:

flowchart TB
    SEG[Incoming Segments<br/>from Transcoder]
    
    WRITER[DVR Writer Service]
    INDEX[DVR Index Service<br/>Redis]
    
    subgraph OBJ["Object Storage (Hot)"]
        S1[Segment 1]
        S2[Segment 2]
        S3[...]
        S4[Segment N]
    end
    
    MANIFEST_GEN[Manifest Generator]
    CLIENT[Viewer Client]
    
    SEG --> WRITER
    WRITER --> OBJ
    WRITER --> INDEX
    CLIENT -->|Seek Request| MANIFEST_GEN
    MANIFEST_GEN --> INDEX
    MANIFEST_GEN --> OBJ
    MANIFEST_GEN -->|Custom m3u8| CLIENT

Implementation Details:

  • Sliding Window: A 3-hour sliding window of segments is maintained in hot object storage (SSD-backed)
  • Segment Indexing: Each segment is indexed by (event_id, angle_id, sequence_number, timestamp) in Redis
  • Dynamic Manifest Generation: On seek request, a custom HLS manifest is generated containing only the segments from the seek point forward
  • Eviction Policy: Segments older than 3 hours are moved to warm storage or deleted (depending on archive policy)
  • Storage per Event: ~16 GB/event (4 angles × 3 Mbps × 3 hours)

Seek Latency: < 500ms (manifest generation + first segment fetch from CDN)

5.4 Multi-Camera Support

Purpose: Deliver multiple simultaneous camera angles per event.

Design:

  • Each camera angle is treated as an independent stream through the entire pipeline
  • Ingest → Transcode → DVR → CDN is replicated per angle
  • Client maintains a single session but can switch angles by requesting a different manifest
  • Angle switching is near-instant (< 1 second) because:
    • All angles share synchronized keyframes (aligned to the same clock)
    • CDN caches all angles; switching is a manifest swap, not a reconnection

Synchronization Strategy:

All camera angles are timestamped using PTP (Precision Time Protocol)
from the broadcast truck's master clock.
The transcoding pipeline preserves these timestamps.
The client uses the timestamps to maintain sync across angles.
When switching angles, the client requests the segment matching
the current playback timestamp in the new angle's manifest.

5.5 CDN Distribution

Purpose: Deliver video segments to viewers with minimal latency and maximum reliability.

Multi-Tier CDN Strategy:

flowchart TB
    ORIGIN[Origin Server<br/>Primary + Backup]
    
    subgraph TIER1["Tier 1: Regional POPs (50-100)"]
        R1[Regional POP 1]
        R2[Regional POP 2]
        R3[...]
    end
    
    subgraph TIER2["Tier 2: Edge POPs (1000-5000)"]
        E1[Edge POP 1]
        E2[Edge POP 2]
        E3[...]
    end
    
    subgraph TIER3["Tier 3: Last-Mile (ISP-level)"]
        L1[ISP Cache 1]
        L2[ISP Cache 2]
    end
    
    CLIENTS[Viewers]
    
    ORIGIN --> TIER1
    TIER1 --> TIER2
    TIER2 --> TIER3
    TIER3 --> CLIENTS

CDN Features:

  • Pre-positioning: For known events (Super Bowl, World Cup), segments are pre-cached at edge POPs
  • LL-HLS (Low-Latency HLS): Uses Loader and Rendition tags for block-based delivery; achieves 3-5 second latency
  • Origin Shield: Intermediate caching layer between origin and edge to reduce origin load
  • Geo-Redirect: Route viewers to the nearest POP with available capacity
  • Multi-CDN Failover: Primary CDN (e.g., Akamai) with fallback to secondary (e.g., CloudFront)

Cache Strategy:

Content TTL Strategy
Live segments (current) 10 seconds Short TTL, always fresh
Live segments (DVR window) 3 hours Medium TTL, pinned
VOD segments 30 days Long TTL, evergreen
Manifests 2 seconds Very short, regenerated frequently
Static assets 1 year Long TTL, versioned

5.6 Real-Time Stats Overlay

Purpose: Superimpose live scores, player stats, and graphics onto the video stream.

Two Approaches:

Approach Server-Side Rendering Client-Side Overlay
How it works Graphics composited into video during transcoding Client renders overlay on top of video
Latency Adds 200-500ms to pipeline Near-zero additional latency
Consistency Identical for all viewers Varies by client implementation
Screenshot-safe Yes (baked into video) No (overlay may not appear)
Flexibility Low (fixed templates) High (user-customizable)

Recommended: Hybrid approach — server-side for broadcast-quality graphics (score bugs, replays), client-side for personalized stats.

Stats Data Pipeline:

Sports Data Provider (e.g., Sportradar)
    → Kafka Topic: sports_events
    → Flink Stream Processor (enrich, validate)
    → Redis (low-latency read for overlay service)
    → Overlay Service (server-side) / WebSocket (client-side)

6. Data Model

6.1 Core Entities

-- Events
CREATE TABLE events (
    event_id          BIGINT PRIMARY KEY,
    title             VARCHAR(500) NOT NULL,
    sport             VARCHAR(50) NOT NULL,
    league            VARCHAR(100),
    scheduled_start   TIMESTAMP NOT NULL,
    scheduled_end     TIMESTAMP,
    actual_start      TIMESTAMP,
    actual_end        TIMESTAMP,
    status            ENUM('scheduled', 'pre_show', 'live', 'post_show', 'archived', 'cancelled'),
    geo_rights        JSONB,              -- { "US": true, "UK": false, ... }
    metadata          JSONB,              -- teams, venue, etc.
    created_at        TIMESTAMP DEFAULT NOW(),
    updated_at        TIMESTAMP DEFAULT NOW()
);

-- Camera Angles
CREATE TABLE camera_angles (
    angle_id          BIGINT PRIMARY KEY,
    event_id          BIGINT REFERENCES events(event_id),
    label             VARCHAR(100) NOT NULL,  -- "Main", "Sideline", "SkyCam"
    order_index       INT NOT NULL,
    is_default        BOOLEAN DEFAULT false,
    thumbnail_url     VARCHAR(500),
    created_at        TIMESTAMP DEFAULT NOW()
);

-- Ingest Sessions
CREATE TABLE ingest_sessions (
    session_id        UUID PRIMARY KEY,
    event_id          BIGINT REFERENCES events(event_id),
    angle_id          BIGINT REFERENCES camera_angles(angle_id),
    protocol          VARCHAR(20) NOT NULL,   -- 'SRT', 'RTMP', 'RTP'
    ingest_url        VARCHAR(500) NOT NULL,
    stream_key        VARCHAR(100) NOT NULL,
    source_ip         INET,
    status            ENUM('active', 'idle', 'failed', 'ended'),
    quality_score     FLOAT,                  -- 0.0 to 1.0
    started_at        TIMESTAMP,
    ended_at          TIMESTAMP,
    expires_at        TIMESTAMP NOT NULL
);

-- Transcoding Profiles
CREATE TABLE transcoding_profiles (
    profile_id        BIGINT PRIMARY KEY,
    name              VARCHAR(50) NOT NULL,   -- '4K', '1080p_High', '720p'
    codec             VARCHAR(20) NOT NULL,   -- 'H.264', 'H.265', 'AV1'
    resolution        VARCHAR(20) NOT NULL,   -- '3840x2160', '1920x1080'
    video_bitrate     INT NOT NULL,           -- bps
    audio_bitrate     INT NOT NULL,           -- bps
    framerate    DECIMAL(4,1) NOT NULL,
    segment_duration  INT NOT NULL,           -- seconds
    is_ll_hls         BOOLEAN DEFAULT false
);

-- Stream Segments (for DVR indexing)
CREATE TABLE stream_segments (
    segment_id        BIGINT PRIMARY KEY,
    event_id          BIGINT NOT NULL,
    angle_id          BIGINT NOT NULL,
    profile_id        BIGINT NOT NULL,
    sequence_number   BIGINT NOT NULL,
    start_time        TIMESTAMP NOT NULL,
    duration          DECIMAL(6,3) NOT NULL,
    storage_path      VARCHAR(500) NOT NULL,
    cdn_url           VARCHAR(500),
    size_bytes        BIGINT,
    created_at        TIMESTAMP DEFAULT NOW(),
    UNIQUE(event_id, angle_id, profile_id, sequence_number)
);

-- Users and Entitlements
CREATE TABLE users (
    user_id           BIGINT PRIMARY KEY,
    email             VARCHAR(255) UNIQUE NOT NULL,
    display_name      VARCHAR(100),
    subscription_tier ENUM('free', 'basic', 'premium', 'ultimate'),
    subscription_end  TIMESTAMP,
    geo_region        VARCHAR(10),
    created_at        TIMESTAMP DEFAULT NOW()
);

CREATE TABLE user_entitlements (
    entitlement_id    BIGINT PRIMARY KEY,
    user_id           BIGINT REFERENCES users(user_id),
    channel_id        BIGINT,                -- specific channel/package
    event_id          BIGINT,                -- specific event (PPV)
    rights            JSONB,                 -- { "live": true, "dvr": true, "4k": true }
    granted_at        TIMESTAMP DEFAULT NOW(),
    expires_at        TIMESTAMP
);

-- Analytics Events
CREATE TABLE analytics_events (
    event_id          BIGINT PRIMARY KEY,
    user_id           BIGINT,
    session_id        UUID NOT NULL,
    action            VARCHAR(50) NOT NULL,  -- 'stream_start', 'seek', 'angle_switch', 'rebuffer'
    context           JSONB,                 -- details about the action
    timestamp         TIMESTAMP DEFAULT NOW()
);

6.2 Redis Data Structures

# DVR Segment Index
# Key: dvr:{event_id}:{angle_id}:{profile_id}
# Type: Sorted Set (score = timestamp)
ZADD dvr:1001:1:10 1709280000.000 "seg_00001"
ZADD dvr:1001:1:10 1709280006.000 "seg_00002"

# Live Event Status
# Key: event:live:{event_id}
# Type: Hash
HSET event:live:1001 status "live" viewer_count "5000000" started_at "1709280000"

# Ingest Health
# Key: ingest:health:{session_id}
# Type: Hash
HSET ingest:health:abc-123 quality_score "0.95" packet_loss "0.01" latency_ms "350"

# Rate Limiting
# Key: ratelimit:{user_id}:{action}
# Type: String with TTL
SETEX ratelimit:100:stream_start 60 "5"

7. Key Design Decisions

7.1 Streaming Protocol: LL-HLS vs WebRTC vs CMAF

Protocol Latency Scalability Browser Support Complexity
LL-HLS 3-5s Excellent (HTTP-based) All modern browsers Low
WebRTC 500ms-2s Hard (P2P / SFU needed) All modern browsers High
CMAF / DASH 3-10s Excellent Limited (needs JS) Medium
MSE + FLV 2-4s Good Chrome/Edge only Medium

Decision: LL-HLS as primary, WebRTC as optional low-latency mode for premium subscribers.

Rationale:

  • LL-HLS provides the best balance of latency, scalability, and compatibility
  • HTTP-based means it works through any firewall and leverages existing CDN infrastructure
  • WebRTC added as a premium feature for latency-sensitive viewers (betting markets, etc.)

7.2 Transcoding: Hardware vs Software

Decision: Hardware-accelerated (NVENC) as primary, software (x264/x265) for quality-critical content.

Rationale:

  • NVENC provides 10-20x throughput improvement over software encoding at comparable quality
  • At Super Bowl scale (200 Tbps output), software encoding would require ~50,000 CPU cores
  • Hardware encoding reduces cost by ~80% and power consumption by ~70%
  • Software encoding reserved for highlight reels and VOD where quality matters more than latency

7.3 DVR Storage: Object Storage vs Block Storage

Decision: Object storage (S3-compatible) with SSD tier for hot DVR window.

Rationale:

  • Object storage scales to petabytes and provides durability (11 nines)
  • SSD tier ensures sub-100ms segment retrieval for DVR seeks
  • Block storage would require complex volume management and doesn’t scale as gracefully
  • Hot segments stay in SSD tier; older segments transition to standard storage

7.4 Multi-CDN Strategy

Decision: Primary + Secondary CDN with client-side failover.

Rationale:

  • No single CDN covers all geographies optimally
  • Client-side failover (via manifest redirect) is faster than DNS-based failover
  • Cost optimization: route traffic to the cheapest CDN per region while maintaining quality
  • Black events require pre-negotiated capacity with multiple providers

7.5 Geo-Rights Enforcement

Decision: Edge-enforced with origin validation (defense in depth).

Rationale:

  • CDN edge can block based on IP geo-location (fast, low-latency)
  • Origin performs token-based validation as a second layer
  • Token contains signed geo-entitlements; prevents CDN bypass attacks
  • Legal compliance requires both technical and contractual enforcement

8. Scalability Strategy

8.1 Horizontal Scaling Plan

Component Scaling Trigger Min Max Scale Method
Ingest Gateway CPU > 70% or stream count 10 200 K8s HPA
Transcoders Queue depth > 100 segments 50 2000 GPU pool autoscaler
Origin Servers RPS > 50K/node 20 500 K8s HPA
Auth Service Latency > 100ms 10 100 K8s HPA
DVR Writers Write queue depth 5 50 Fixed per event
Analytics Kafka lag > 1000 5 50 Flink parallelism

8.2 Black Event Preparation

For known mega-events (Super Bowl, World Cup Final):

  1. T-90 days: Reserve CDN capacity; pre-provision transcoder GPU clusters
  2. T-30 days: Load test with synthetic traffic at 150% expected peak
  3. T-7 days: Pre-position static assets and common segments at edge POPs
  4. T-1 day: Warm all caches; verify failover paths; run dry-run ingest
  5. T-1 hour: Scale all services to peak capacity; enable enhanced monitoring
  6. During event: Real-time dashboard with per-POP health; on-call war room
  7. Post-event: Gradual scale-down over 24-48 hours

8.3 Database Scaling

  • Primary DB: PostgreSQL with read replicas (1 writer, 8+ readers)
  • Sharding: Events sharded by sport or region; users sharded by user_id hash
  • Caching: Redis cluster for hot data (live event status, DVR indexes, auth tokens)
  • Analytics: Write-to-Kafka → Flink → ClickHouse for real-time analytics queries

8.4 CDN Scaling

  • Edge POP Count: 1,000+ global POPs for standard operation; 2,000+ for black events
  • Capacity Reservation: Pre-purchase bandwidth guarantees with CDN providers
  • Traffic Shaping: QoS policies prioritize live segments over VOD during peak
  • Protocol Optimization: HTTP/3 (QUIC) for reduced connection setup time

9. Curveball Scenarios

Curveball 1: “The Super Bowl stream drops 3 minutes before the end — 80 million viewers are affected. What happens and how do you recover?”

Impact Assessment:

  • 80M concurrent viewers lose stream during the most critical moment
  • Social media explosion; brand damage; potential regulatory scrutiny
  • Revenue impact from advertiser complaints

Root Cause Analysis Framework:

Possible Failure Points:
├── Ingest Layer
│   ├── Broadcast truck SRT link failure
│   ├── Primary feed corruption
│   └── Ingest gateway overload
├── Transcoding Layer
│   ├── GPU cluster failure
│   ├── Pipeline backlog overflow
│   └── Storage write bottleneck
├── Distribution Layer
│   ├── Origin server exhaustion
│   ├── CDN POP failure (regional)
│   └── BGP routing issue
└── Network Layer
    ├── Backbone congestion
    ├── DDoS attack
    └── DNS resolution failure

Recovery Plan:

  1. Immediate (0-30 seconds):

    • Automated failover to backup ingest feed (should trigger within 3 seconds)
    • If transcoding failed: activate cold-standby transcoder cluster in a different region
    • If CDN failed: client-side multi-CDN failover redirects to secondary CDN
  2. Short-term (30 sec - 5 min):

    • Origin servers scale up automatically (pre-provisioned capacity for black events)
    • CDN re-routes traffic away from failed POPs
    • Push notification to affected users: “We’re restoring your stream”
  3. Communication:

    • Status page update within 60 seconds
    • Social media response within 5 minutes
    • Post-event transparency report within 48 hours

Prevention for Future:

  • N+2 redundancy at every layer for black events (not just N+1)
  • Chaos engineering: Monthly failure injection tests on the full pipeline
  • Geographic diversity: Ingest, transcode, and origin in at least 3 regions
  • Circuit breakers: Automatic isolation of failing components to prevent cascade

Curveball 2: “A major league wants to add AR (Augmented Reality) first-down lines and real-time player tracking overlays. How does this change the architecture?”

Architecture Changes:

  1. New Data Pipeline:

    Camera Feeds → Computer Vision Engine → Player Positions (60fps)
        → AR Rendering Engine → Overlay Graphics
        → Compositing with Video → Transcoded Output
  2. Processing Requirements:

    • Computer vision: ~200 GPU cores per field (object detection + tracking)
    • AR rendering: Real-time 3D engine (Unreal/Unity) generating overlay frames
    • Compositing: Additional 500ms in the transcoding pipeline
  3. Latency Impact:

    • AR pipeline adds 800-1500ms end-to-end
    • Must be absorbed within the existing 3-second budget
    • Solution: Run AR pipeline in parallel with transcoding; composite at the segment level
  4. Client-Side Alternative:

    • Send player position data as a separate WebSocket stream
    • Client renders AR overlay locally using the position data
    • Reduces server load; enables user-customizable AR
    • Trade-off: Requires client-side 3D rendering capability
  5. Storage Impact:

    • Player tracking data: ~50 KB/sec per player × 22 players = ~1.1 MB/sec
    • 3-hour game: ~12 GB of tracking data per event
    • Store alongside video for replay with AR reconstruction

Recommendation: Start with client-side AR (lower risk, more flexible), add server-side compositing for broadcast-quality output.


Curveball 3: “Regulators require that all live broadcasts in the EU be stored for 90 days for compliance. How does this affect storage costs and architecture?”

Storage Impact Calculation:

Per Event Storage (90-day retention):
├── Source (4K @ 50 Mbps): 3 hrs × 50 Mbps × 3600 = 6.75 GB
├── Transcoded (5 profiles): 6.75 GB × 5 = 33.75 GB
├── Multi-angle (4 angles): 33.75 GB × 4 = 135 GB/event
└── Metadata + Indexes: ~1 GB/event

Daily Events (EU-only): ~20 events/day
Daily Storage: 20 × 135 GB = 2.7 TB/day
90-Day Retention: 2.7 TB × 90 = 243 TB

Cost Analysis (AWS pricing estimates):

Storage Tier Use Case Cost/TB/Month Monthly Cost
S3 Standard 0-7 days (hot) $23 $5,589
S3 IA 7-30 days (warm) $12.5 $2,925
S3 Glacier Deep 30-90 days (cold) $4 $972
Total ~$9,486/month

Architecture Changes:

  1. Lifecycle Policy:

    Day 0-7:   S3 Standard (frequent access for DVR + replays)
    Day 7-30:  S3 Infrequent Access (replay requests)
    Day 30-90: S3 Glacier Deep Archive (compliance only)
  2. Compliance Service:

    • Immutable storage (S3 Object Lock) to prevent deletion
    • Audit logging for all access attempts
    • Automated retention policy enforcement
    • Export capability for regulatory requests
  3. Index Preservation:

    • Maintain segment indexes in a compressed format
    • Enable fast retrieval from Glacier (3-5 hour restore for deep archive)
    • Keep a manifest database for quick event lookup
  4. Legal Considerations:

    • GDPR compliance: ensure personal data (chat, user info) is handled separately
    • Right to be forgotten: video content vs. personal data distinction
    • Cross-border data transfer: store EU content in EU regions only

Curveball 4: “A DDoS attack targets your origin servers during a championship final. How do you protect the service?”

Defense-in-Depth Strategy:

  1. Perimeter Defense:

    • CDN absorbs the majority of attack traffic (CDN providers handle 10+ Tbps attacks)
    • DDoS protection service (e.g., AWS Shield Advanced, Akamai Prolexic)
    • Rate limiting at the CDN edge per IP and per session
  2. Origin Protection:

    • Origin is behind a WAF (Web Application Firewall) with strict allow-listing
    • Only CDN edge IPs can reach origin (IP allow-list)
    • Token-based authentication on all origin requests
  3. Application-Level:

    • JWT token validation prevents unauthorized manifest requests
    • Rate limiting per user token (not just IP)
    • Bot detection via behavioral analysis
  4. Capacity Absorption:

    • Pre-provisioned excess capacity for black events absorbs attack overhead
    • Auto-scaling triggers at lower thresholds during known attack patterns
  5. Incident Response:

    • Automated attack detection via anomaly detection on traffic patterns
    • One-click activation of enhanced DDoS mitigation rules
    • Communication plan for stakeholders

Curveball 5: “Viewers complain that the stream is 30 seconds behind Twitter. How do you reduce latency further?”

Current State Analysis:

  • LL-HLS typical latency: 3-5 seconds
  • If viewers report 30 seconds, something is wrong:
    • CDN caching too aggressive (long TTL on live segments)
    • Client buffer too large
    • Network congestion adding delay

Optimization Steps:

  1. Immediate Fixes:

    • Reduce segment duration from 6s to 2s
    • Enable LL-HLS Part segments (0.5-1s parts within a segment)
    • Reduce client buffer to 2 segments minimum
    • Verify CDN TTL is set to segment duration (not longer)
  2. Protocol Upgrade:

    • Deploy WebRTC low-latency mode as an option
    • WebRTC achieves 500ms-2 second latency
    • Use SFU (Selective Forwarding Unit) architecture for scalability
  3. Network Optimization:

    • HTTP/3 (QUIC) for faster connection setup and 0-RTT resumption
    • Prefetch next segment while playing current
    • Edge POPs closer to viewers
  4. Target Latency Breakdown:

Component Current Optimized WebRTC Mode
Ingest → Transcode 500ms 300ms 200ms
Transcode → Origin 500ms 300ms 200ms
Origin → CDN Edge 200ms 100ms 100ms
CDN Edge → Client 300ms 200ms 100ms
Client Buffer 2000ms 1000ms 500ms
Total ~4s ~2s ~1.1s
  1. Trade-off Warning: Lower latency increases rebuffer risk. Must balance with adaptive bitrate quality.

10. Failure Modes and Mitigations

10.1 Failure Mode Matrix

Component Failure Mode Probability Impact Mitigation RTO
Ingest Primary feed loss Medium High N+1 backup feed; auto-failover in < 3s < 3s
Ingest All feeds lost Low Critical Pre-recorded content fallback; “technical difficulty” screen < 10s
Transcode GPU node failure Medium Medium Redundant encoders per profile; auto-reassign < 5s
Transcode Pipeline backlog Low High Drop oldest segments; alert ops; scale up < 30s
DVR Storage write failure Low Medium Dual-write to redundant storage; queue and retry < 10s
CDN POP failure Medium Medium Multi-CDN failover; traffic reroute < 15s
CDN Origin overload Low High Origin shield caching; request queuing; rate limit < 30s
Auth Token service down Low High Cached tokens with extended TTL; graceful degradation < 60s
Geo Geo-DB stale Low Medium Cache with short TTL; fallback to allow < 5s
Stats Data feed loss Medium Low Last-known-state hold; “data delayed” indicator N/A
DB Primary DB failover Low High Automated failover to replica; read-only mode < 30s
Network BGP hijack Very Low Critical Multi-homed network; BGP monitoring; rapid reroute < 60s

10.2 Cascading Failure Prevention

flowchart LR
    subgraph CIRCUIT_BREAKERS["Circuit Breakers"]
        CB1[Ingest → Transcode<br/>Break if queue > 1000]
        CB2[Transcode → Origin<br/>Break if latency > 5s]
        CB3[Origin → CDN<br/>Break if error rate > 5%]
        CB4[Client → Origin<br/>Break if 5xx > 10%]
    end
    
    subgraph FALLBACKS["Fallback Strategies"]
        FB1[Backup Ingest Feed]
        FB2[Pre-Cached Segments]
        FB3[Secondary CDN]
        FB4[Static Error Page]
    end
    
    CB1 --> FB1
    CB2 --> FB2
    CB3 --> FB3
    CB4 --> FB4

10.3 Monitoring and Alerting

Key Metrics:

Metric Warning Threshold Critical Threshold Action
End-to-end latency > 8s > 15s Scale transcoders; check ingest
Rebuffer ratio > 2% > 5% Check CDN health; reduce bitrate
Ingest packet loss > 1% > 5% Switch to backup feed
Transcode queue depth > 100 > 500 Scale encoder cluster
Origin 5xx rate > 1% > 5% Scale origin; enable circuit breaker
CDN cache hit ratio < 95% < 85% Check TTL; warm caches
Concurrent viewers — — Info only; capacity planning
DVR storage usage > 80% > 95% Evict older segments; scale storage

Observability Stack:

  • Metrics: Prometheus + Grafana (real-time dashboards)
  • Logging: ELK Stack (Elasticsearch, Logstash, Kibana)
  • Tracing: Jaeger / OpenTelemetry (end-to-end request tracing)
  • Alerting: PagerDuty integration with severity-based escalation

Appendix: Technology Stack Summary

Layer Technology Choices
Ingest SRT (Haivision), RTMP (NGINX RTMP), GStreamer
Transcoding NVIDIA NVENC, FFmpeg, Intel QSV, AWS MediaConvert
Packaging Bento4, MP4Box, LL-HLS packager
Origin NGINX, Apache Traffic Server, custom Go service
CDN Akamai (primary), CloudFront (secondary), Cloudflare (tertiary)
Storage AWS S3 (hot + warm), S3 Glacier (archive), Redis (indexes)
Database PostgreSQL (primary), Redis (cache), ClickHouse (analytics)
Streaming LL-HLS (primary), WebRTC (low-latency mode), DASH (fallback)
Messaging Apache Kafka (events), Redis Pub/Sub (real-time)
Stream Processing Apache Flink (analytics), Kafka Streams (lightweight)
Auth Auth0 / Keycloak, JWT tokens
Monitoring Prometheus, Grafana, ELK, Jaeger
Infrastructure Kubernetes, Terraform, Helm