Design a Live-Streaming Broadcast App like ESPN
Scope: A professional-grade live sports broadcasting platform delivering low-latency, high-reliability live streams with DVR/rewind, multi-camera support, and real-time stats overlays to millions of concurrent viewers.
Table of Contents
- Requirements Clarification
- Back-of-Envelope Calculations
- System Interface Design
- High-Level Architecture
- Core Component Design
- Data Model
- Key Design Decisions
- Scalability Strategy
- Curveball Scenarios
- Failure Modes and Mitigations
1. Requirements Clarification
1.1 Functional Requirements
- Live Ingest: Accept live video feeds from broadcast trucks, studios, and remote venues via multiple protocols (RTMP, SRT, RTP/RTSP, HLS ingest)
- Transcoding Pipeline: Real-time transcoding into multiple bitrates/resolutions (SD, HD, Full HD, 4K) with adaptive bitrate ladders
- Multi-Camera Support: Deliver 2-8 simultaneous camera angles per event; viewers can switch angles in real-time
- DVR / Time-Shift Rewind: Allow viewers to rewind up to 3 hours back within a live broadcast
- Real-Time Stats Overlay: Superimpose live scores, player stats, and graphics onto the video stream
- VOD Archive: Automatically archive completed broadcasts for on-demand replay
- Multi-Platform Delivery: Support web browsers, iOS, Android, smart TVs, set-top boxes, and gaming consoles
- Geo-Rights Management: Enforce regional licensing restrictions per broadcast
- Simulcast / Multi-Event: Support broadcasting multiple events simultaneously (e.g., NFL Sunday with 14 concurrent games)
- Closed Captioning & Multi-Language Audio: Support CC and multiple audio tracks (English, Spanish, etc.)
- Push Notifications: Alert subscribers about game start, score changes, and breaking events
1.2 Non-Functional Requirements
| Requirement | Target | Rationale |
|---|---|---|
| Live Latency | 3-10 seconds (HLS LL / WebRTC optional) | Sports fans need near-real-time; no spoilers from social media |
| Availability | 99.99% uptime for live events | A dropped stream during a championship is catastrophic |
| Concurrent Viewers | 50M+ peak (Super Bowl scale) | Must handle black-event traffic spikes |
| Stream Start Time | < 2 seconds from click to first frame | Viewer retention is critical |
| Rebuffer Ratio | < 1% of total watch time | Quality of experience metric |
| Transcoding Delay | < 3 seconds end-to-end | Pipeline must keep up with live ingest |
| Geo-Blocking Accuracy | 99.9% enforcement | Legal compliance for regional rights |
| Data Consistency | Eventual consistency for stats; strong for billing | Stats can lag slightly; billing must be exact |
| Scalability | Horizontal scale for all stateless components | Traffic is bursty and unpredictable |
2. Back-of-Envelope Calculations
2.1 Assumption Sets
We consider three scenarios to bound our design:
| Parameter | Conservative (Weekday) | Typical (Weekend) | Black Event (Super Bowl) |
|---|---|---|---|
| Concurrent Viewers | 500K | 5M | 50M |
| Live Events Concurrent | 10 | 30 | 14 (NFL Sunday) |
| Avg Watch Duration | 45 min | 2 hours | 3.5 hours |
| Avg Bitrate | 3 Mbps | 3.5 Mbps | 4 Mbps |
| Multi-Camera Angles | 2 | 3 | 4 |
| Transcoding Profiles per Angle | 5 | 5 | 5 |
2.2 Bandwidth Estimates
Outbound CDN Bandwidth:
| Scenario | Calculation | Total Bandwidth |
|---|---|---|
| Weekday | 500K × 3 Mbps | 1.5 Tbps |
| Weekend | 5M × 3.5 Mbps | 17.5 Tbps |
| Super Bowl | 50M × 4 Mbps | 200 Tbps |
Note: 200 Tbps is at the scale of major CDN backbone capacity. This requires pre-provisioning with CDN providers months in advance.
Ingest Bandwidth (per event, assuming 4K source at 50 Mbps):
| Scenario | Calculation | Total Ingest |
|---|---|---|
| Weekday | 10 × 50 Mbps | 500 Mbps |
| Weekend | 30 × 50 Mbps | 1.5 Gbps |
| Super Bowl | 14 × 50 Mbps | 700 Mbps |
2.3 Storage Estimates
DVR Window Storage (3-hour rewind window, per stream):
- Per angle: 3 hours × 3 Mbps × 3600 sec = 4.05 GB
- Per event (4 angles): 4 × 4.05 GB = 16.2 GB
- 30 concurrent events: 30 × 16.2 GB = 486 GB hot storage
VOD Archive (daily):
- 50 events/day × 3 hours × 3 Mbps × 4 angles = 606 GB/day raw
- With 5 transcoding profiles: 606 GB × 5 = 3 TB/day
- Monthly: ~90 TB/month
Total Annual Storage: ~1.1 PB/year (with compression and dedup, ~600 TB effective)
2.4 QPS Estimates
| Operation | Weekday | Weekend | Super Bowl |
|---|---|---|---|
| Stream Start Requests | 10K/s | 100K/s | 1M/s |
| API Calls (metadata, stats) | 50K/s | 500K/s | 5M/s |
| DVR Seek Requests | 5K/s | 50K/s | 500K/s |
| Camera Switch Requests | 2K/s | 20K/s | 200K/s |
| Auth Token Requests | 20K/s | 200K/s | 2M/s |
| Chat / Social (if enabled) | 100K/s | 1M/s | 10M/s |
2.5 Transcoding Compute
- Each transcoding profile: ~2-4 vCPU cores (hardware-accelerated with NVENC/QSV)
- Per angle: 5 profiles × 3 cores = 15 cores
- Per event (4 angles): 60 cores
- 30 events: 1,800 GPU-accelerated cores minimum
3. System Interface Design
3.1 Public APIs
# --- Ingest APIs (Internal / Partner-Facing) ---
POST /api/v1/ingest/session
Create a new ingest session; returns ingest URL and credentials
Request: { event_id, protocol, resolution, bitrate, camera_angle }
Response: { session_id, ingest_url, stream_key, expires_at }
POST /api/v1/ingest/session/{id}/heartbeat
Keep-alive for active ingest sessions
Response: { status: "active" | "expired" }
DELETE /api/v1/ingest/session/{id}
Terminate an ingest session gracefully
# --- Viewer APIs ---
GET /api/v1/events
List available live and upcoming events
Query: ?status=live&sport=nfl&geo=US
Response: { events: [{ id, title, sport, start_time, status, camera_angles, geo_rights }] }
GET /api/v1/events/{id}
Get event details
Response: { id, title, description, start_time, end_time, status,
camera_angles: [{ id, label, thumbnail }],
stats_feed_url, chat_enabled, geo_rights }
GET /api/v1/events/{id}/stream/manifest
Get HLS/DASH manifest for a specific camera angle
Query: ?angle=main&format=hll&drm=widevine
Response: HLS/DASH manifest (m3u8 / mpd)
GET /api/v1/events/{id}/stream/manifest/segment/{seq}
Get a specific media segment (served by CDN)
POST /api/v1/events/{id}/dvr/seek
Seek to a position in the DVR window
Request: { position_seconds, angle_id }
Response: { manifest_url, position }
GET /api/v1/events/{id}/stats
Get real-time event statistics
Response: { score, period, clock, possessions, player_stats, timeline }
GET /api/v1/events/{id}/archive
Get VOD archive info for completed events
Response: { archive_id, duration, available_from, manifest_url }
# --- User / Subscription APIs ---
POST /api/v1/auth/token
Authenticate user; returns JWT + entitlements
Request: { username, password } or { device_token }
Response: { access_token, refresh_token, entitlements: ["live", "dvr", "4k"] }
GET /api/v1/user/entitlements
Check user's subscription tier and geo rights
Response: { tier, channels, geo_allowed, expires_at }
POST /api/v1/notifications/subscribe
Subscribe to event notifications
Request: { event_id, types: ["start", "score_change", "end"] }
# --- Admin / Operations APIs ---
POST /api/v1/admin/events
Create/schedule a new broadcast event
PUT /api/v1/admin/events/{id}/go-live
Manually trigger event to go live
GET /api/v1/admin/ingest/health
Health status of all active ingest sessions
GET /api/v1/admin/analytics/realtime
Real-time viewer counts, bitrate distribution, error rates
3.2 Internal Service APIs (gRPC)
// Transcoding Service
service TranscoderService {
rpc StartTranscode(TranscodeRequest) returns (stream TranscodeStatus);
rpc StopTranscode(StopRequest) returns (Empty);
rpc GetPipelineHealth(HealthRequest) returns (PipelineHealth);
}
// DVR Service
service DVRService {
rpc OpenWindow(DVROpenRequest) returns (DVRWindow);
rpc WriteSegment(DVRWriteRequest) returns (Empty);
rpc Seek(DVRSeekRequest) returns (DVRManifest);
rpc CloseWindow(DVRCloseRequest) returns (Empty);
}
// Stats Overlay Service
service StatsOverlayService {
rpc UpdateOverlay(OverlayUpdate) returns (Empty);
rpc GetOverlayState(EventId) returns (OverlayState);
}
// Geo-Rights Service
service GeoRightsService {
rpc CheckAccess(AccessRequest) returns (AccessResponse);
rpc UpdateRights(RightsUpdate) returns (Empty);
}
4. High-Level Architecture
flowchart TB
subgraph INGEST["📡 Ingest Layer"]
BT[Broadcast Truck<br/>SRT / RTP]
STUDIO[Studio Feed<br/>RTMP / SDI]
REMOTE[Remote Venue<br/>SRT / HLS Ingest]
IG[Ingest Gateway<br/>Protocol Adapter]
end
subgraph PROCESSING["⚙️ Processing Layer"]
TC[Transcoding Cluster<br/>GPU-Accelerated]
QC[Quality Checker<br/>PSNR / VMAF]
STATS[Stats Ingestion<br/>Real-Time Feed]
OVERLAY[Graphics Overlay<br/>Server-Side Rendering]
end
subgraph STORAGE["💾 Storage Layer"]
DVR[DVR Buffer<br/>Object Storage + Index]
VOD[VOD Archive<br/>Cold + Warm Storage]
MANIFEST[Manifest Store<br/>Redis + CDN Origin]
end
subgraph DISTRIBUTION["🌐 Distribution Layer"]
ORIGIN[Origin Server<br/>Edge Logic]
CDN[CDN Edge POPs<br/>Global Distribution]
EDGE[Edge Cache<br/>LL-HLS / DASH]
end
subgraph CLIENTS["📱 Client Layer"]
WEB[Web Browser]
MOBILE[iOS / Android]
TV[Smart TV / STB]
CONSOLE[Gaming Console]
end
subgraph SERVICES["🔧 Support Services"]
AUTH[Auth Service<br/>JWT / OAuth]
GEO[Geo-Rights Engine<br/>IP + Subscriber DB]
NOTIF[Notification Service<br/>Push / SMS]
ANALYTICS[Analytics Pipeline<br/>Kafka + Flink]
ADMIN[Admin Dashboard<br/>Operations Console]
end
BT --> IG
STUDIO --> IG
REMOTE --> IG
IG --> TC
TC --> QC
QC -->|Pass| OVERLAY
QC -->|Fail| TC
OVERLAY --> DVR
OVERLAY --> ORIGIN
STATS --> OVERLAY
DVR --> ORIGIN
VOD --> ORIGIN
ORIGIN --> CDN
CDN --> EDGE
EDGE --> WEB
EDGE --> MOBILE
EDGE --> TV
EDGE --> CONSOLE
ORIGIN -.-> MANIFEST
GEO -.-> ORIGIN
AUTH -.-> ORIGIN
NOTIF -.-> CLIENTS
ANALYTICS -.-> ORIGIN
ADMIN -.-> PROCESSING
5. Core Component Design
5.1 Ingest Gateway
Purpose: Accept incoming live feeds from diverse sources and normalize them into a unified internal format.
Supported Protocols:
| Protocol | Use Case | Latency | Reliability |
|---|---|---|---|
| SRT (Secure Reliable Transport) | Primary for broadcast trucks | 200-500ms | FEC + ARQ, handles 20% packet loss |
| RTP/RTSP | Legacy broadcast equipment | 300-800ms | UDP-based, needs RTCP feedback |
| RTMP | Studios, backup feeds | 1-3s | TCP-based, widely supported |
| HLS Ingest | Remote contributors, backup | 5-10s | HTTP-based, firewall-friendly |
Design Details:
- Protocol Adapters: Each protocol has a dedicated adapter microservice that normalizes to an internal raw stream format
- Redundant Ingest: Each event accepts 2-3 simultaneous feeds (primary + backup); a failover controller selects the best feed
- Health Monitoring: Continuous RTCP/SRT stats collection; automatic failover on quality degradation
- Capacity: Each ingest node handles ~50 concurrent streams; scale horizontally
Failover Logic:
Primary Feed Quality Score = f(packet_loss, jitter, latency, resolution)
If Primary Score < Threshold for > 3 seconds:
Switch to Backup Feed
Notify Transcoding Pipeline of source change
Log failover event for post-mortem
5.2 Transcoding Pipeline
Purpose: Convert incoming high-bitrate source streams into multiple delivery-optimized profiles in real-time.
Architecture:
flowchart LR
SRC[Source Stream<br/>4K @ 50Mbps]
subgraph ENCODE["Encoding Cluster"]
E1[Encoder Node 1<br/>NVENC H.264]
E2[Encoder Node 2<br/>NVENC H.265]
E3[Encoder Node 3<br/>AV1 (select)]
end
subgraph PROFILES["Bitrate Ladder"]
P1[4K @ 15Mbps]
P2[1080p @ 8Mbps]
P3[1080p @ 5Mbps]
P4[720p @ 3Mbps]
P5[480p @ 1.5Mbps]
P6[360p @ 800Kbps]
end
subgraph AUDIO["Audio Processing"]
A1[EN Track]
A2[ES Track]
A3[CC Track]
end
SRC --> ENCODE
ENCODE --> PROFILES
ENCODE --> AUDIO
Key Design Choices:
- Hardware Acceleration: NVIDIA NVENC (H.264/H.265) for primary encoding; Intel QSV as fallback; AV1 via software for premium content
- Segment Duration: 2-6 second segments for LL-HLS; 6-10 second for standard HLS
- Keyframe Alignment: All bitrate variants share synchronized keyframes for seamless switching
- Pipeline Parallelism: Each camera angle runs an independent pipeline; profiles within an angle share the decode stage
Transcoding Delay Budget:
| Stage | Budget |
|---|---|
| Ingest decode | 200ms |
| GPU encode (per profile) | 500ms |
| Segment packaging | 300ms |
| Network to origin | 500ms |
| Total Pipeline Delay | ~2 seconds |
5.3 DVR / Time-Shift Rewind
Purpose: Enable viewers to rewind up to 3 hours within a live broadcast.
Architecture:
flowchart TB
SEG[Incoming Segments<br/>from Transcoder]
WRITER[DVR Writer Service]
INDEX[DVR Index Service<br/>Redis]
subgraph OBJ["Object Storage (Hot)"]
S1[Segment 1]
S2[Segment 2]
S3[...]
S4[Segment N]
end
MANIFEST_GEN[Manifest Generator]
CLIENT[Viewer Client]
SEG --> WRITER
WRITER --> OBJ
WRITER --> INDEX
CLIENT -->|Seek Request| MANIFEST_GEN
MANIFEST_GEN --> INDEX
MANIFEST_GEN --> OBJ
MANIFEST_GEN -->|Custom m3u8| CLIENT
Implementation Details:
- Sliding Window: A 3-hour sliding window of segments is maintained in hot object storage (SSD-backed)
- Segment Indexing: Each segment is indexed by
(event_id, angle_id, sequence_number, timestamp)in Redis - Dynamic Manifest Generation: On seek request, a custom HLS manifest is generated containing only the segments from the seek point forward
- Eviction Policy: Segments older than 3 hours are moved to warm storage or deleted (depending on archive policy)
- Storage per Event: ~16 GB/event (4 angles × 3 Mbps × 3 hours)
Seek Latency: < 500ms (manifest generation + first segment fetch from CDN)
5.4 Multi-Camera Support
Purpose: Deliver multiple simultaneous camera angles per event.
Design:
- Each camera angle is treated as an independent stream through the entire pipeline
- Ingest → Transcode → DVR → CDN is replicated per angle
- Client maintains a single session but can switch angles by requesting a different manifest
- Angle switching is near-instant (< 1 second) because:
- All angles share synchronized keyframes (aligned to the same clock)
- CDN caches all angles; switching is a manifest swap, not a reconnection
Synchronization Strategy:
All camera angles are timestamped using PTP (Precision Time Protocol)
from the broadcast truck's master clock.
The transcoding pipeline preserves these timestamps.
The client uses the timestamps to maintain sync across angles.
When switching angles, the client requests the segment matching
the current playback timestamp in the new angle's manifest.
5.5 CDN Distribution
Purpose: Deliver video segments to viewers with minimal latency and maximum reliability.
Multi-Tier CDN Strategy:
flowchart TB
ORIGIN[Origin Server<br/>Primary + Backup]
subgraph TIER1["Tier 1: Regional POPs (50-100)"]
R1[Regional POP 1]
R2[Regional POP 2]
R3[...]
end
subgraph TIER2["Tier 2: Edge POPs (1000-5000)"]
E1[Edge POP 1]
E2[Edge POP 2]
E3[...]
end
subgraph TIER3["Tier 3: Last-Mile (ISP-level)"]
L1[ISP Cache 1]
L2[ISP Cache 2]
end
CLIENTS[Viewers]
ORIGIN --> TIER1
TIER1 --> TIER2
TIER2 --> TIER3
TIER3 --> CLIENTS
CDN Features:
- Pre-positioning: For known events (Super Bowl, World Cup), segments are pre-cached at edge POPs
- LL-HLS (Low-Latency HLS): Uses
LoaderandRenditiontags for block-based delivery; achieves 3-5 second latency - Origin Shield: Intermediate caching layer between origin and edge to reduce origin load
- Geo-Redirect: Route viewers to the nearest POP with available capacity
- Multi-CDN Failover: Primary CDN (e.g., Akamai) with fallback to secondary (e.g., CloudFront)
Cache Strategy:
| Content | TTL | Strategy |
|---|---|---|
| Live segments (current) | 10 seconds | Short TTL, always fresh |
| Live segments (DVR window) | 3 hours | Medium TTL, pinned |
| VOD segments | 30 days | Long TTL, evergreen |
| Manifests | 2 seconds | Very short, regenerated frequently |
| Static assets | 1 year | Long TTL, versioned |
5.6 Real-Time Stats Overlay
Purpose: Superimpose live scores, player stats, and graphics onto the video stream.
Two Approaches:
| Approach | Server-Side Rendering | Client-Side Overlay |
|---|---|---|
| How it works | Graphics composited into video during transcoding | Client renders overlay on top of video |
| Latency | Adds 200-500ms to pipeline | Near-zero additional latency |
| Consistency | Identical for all viewers | Varies by client implementation |
| Screenshot-safe | Yes (baked into video) | No (overlay may not appear) |
| Flexibility | Low (fixed templates) | High (user-customizable) |
Recommended: Hybrid approach — server-side for broadcast-quality graphics (score bugs, replays), client-side for personalized stats.
Stats Data Pipeline:
Sports Data Provider (e.g., Sportradar)
→ Kafka Topic: sports_events
→ Flink Stream Processor (enrich, validate)
→ Redis (low-latency read for overlay service)
→ Overlay Service (server-side) / WebSocket (client-side)
6. Data Model
6.1 Core Entities
-- Events
CREATE TABLE events (
event_id BIGINT PRIMARY KEY,
title VARCHAR(500) NOT NULL,
sport VARCHAR(50) NOT NULL,
league VARCHAR(100),
scheduled_start TIMESTAMP NOT NULL,
scheduled_end TIMESTAMP,
actual_start TIMESTAMP,
actual_end TIMESTAMP,
status ENUM('scheduled', 'pre_show', 'live', 'post_show', 'archived', 'cancelled'),
geo_rights JSONB, -- { "US": true, "UK": false, ... }
metadata JSONB, -- teams, venue, etc.
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
-- Camera Angles
CREATE TABLE camera_angles (
angle_id BIGINT PRIMARY KEY,
event_id BIGINT REFERENCES events(event_id),
label VARCHAR(100) NOT NULL, -- "Main", "Sideline", "SkyCam"
order_index INT NOT NULL,
is_default BOOLEAN DEFAULT false,
thumbnail_url VARCHAR(500),
created_at TIMESTAMP DEFAULT NOW()
);
-- Ingest Sessions
CREATE TABLE ingest_sessions (
session_id UUID PRIMARY KEY,
event_id BIGINT REFERENCES events(event_id),
angle_id BIGINT REFERENCES camera_angles(angle_id),
protocol VARCHAR(20) NOT NULL, -- 'SRT', 'RTMP', 'RTP'
ingest_url VARCHAR(500) NOT NULL,
stream_key VARCHAR(100) NOT NULL,
source_ip INET,
status ENUM('active', 'idle', 'failed', 'ended'),
quality_score FLOAT, -- 0.0 to 1.0
started_at TIMESTAMP,
ended_at TIMESTAMP,
expires_at TIMESTAMP NOT NULL
);
-- Transcoding Profiles
CREATE TABLE transcoding_profiles (
profile_id BIGINT PRIMARY KEY,
name VARCHAR(50) NOT NULL, -- '4K', '1080p_High', '720p'
codec VARCHAR(20) NOT NULL, -- 'H.264', 'H.265', 'AV1'
resolution VARCHAR(20) NOT NULL, -- '3840x2160', '1920x1080'
video_bitrate INT NOT NULL, -- bps
audio_bitrate INT NOT NULL, -- bps
framerate DECIMAL(4,1) NOT NULL,
segment_duration INT NOT NULL, -- seconds
is_ll_hls BOOLEAN DEFAULT false
);
-- Stream Segments (for DVR indexing)
CREATE TABLE stream_segments (
segment_id BIGINT PRIMARY KEY,
event_id BIGINT NOT NULL,
angle_id BIGINT NOT NULL,
profile_id BIGINT NOT NULL,
sequence_number BIGINT NOT NULL,
start_time TIMESTAMP NOT NULL,
duration DECIMAL(6,3) NOT NULL,
storage_path VARCHAR(500) NOT NULL,
cdn_url VARCHAR(500),
size_bytes BIGINT,
created_at TIMESTAMP DEFAULT NOW(),
UNIQUE(event_id, angle_id, profile_id, sequence_number)
);
-- Users and Entitlements
CREATE TABLE users (
user_id BIGINT PRIMARY KEY,
email VARCHAR(255) UNIQUE NOT NULL,
display_name VARCHAR(100),
subscription_tier ENUM('free', 'basic', 'premium', 'ultimate'),
subscription_end TIMESTAMP,
geo_region VARCHAR(10),
created_at TIMESTAMP DEFAULT NOW()
);
CREATE TABLE user_entitlements (
entitlement_id BIGINT PRIMARY KEY,
user_id BIGINT REFERENCES users(user_id),
channel_id BIGINT, -- specific channel/package
event_id BIGINT, -- specific event (PPV)
rights JSONB, -- { "live": true, "dvr": true, "4k": true }
granted_at TIMESTAMP DEFAULT NOW(),
expires_at TIMESTAMP
);
-- Analytics Events
CREATE TABLE analytics_events (
event_id BIGINT PRIMARY KEY,
user_id BIGINT,
session_id UUID NOT NULL,
action VARCHAR(50) NOT NULL, -- 'stream_start', 'seek', 'angle_switch', 'rebuffer'
context JSONB, -- details about the action
timestamp TIMESTAMP DEFAULT NOW()
);
6.2 Redis Data Structures
# DVR Segment Index
# Key: dvr:{event_id}:{angle_id}:{profile_id}
# Type: Sorted Set (score = timestamp)
ZADD dvr:1001:1:10 1709280000.000 "seg_00001"
ZADD dvr:1001:1:10 1709280006.000 "seg_00002"
# Live Event Status
# Key: event:live:{event_id}
# Type: Hash
HSET event:live:1001 status "live" viewer_count "5000000" started_at "1709280000"
# Ingest Health
# Key: ingest:health:{session_id}
# Type: Hash
HSET ingest:health:abc-123 quality_score "0.95" packet_loss "0.01" latency_ms "350"
# Rate Limiting
# Key: ratelimit:{user_id}:{action}
# Type: String with TTL
SETEX ratelimit:100:stream_start 60 "5"
7. Key Design Decisions
7.1 Streaming Protocol: LL-HLS vs WebRTC vs CMAF
| Protocol | Latency | Scalability | Browser Support | Complexity |
|---|---|---|---|---|
| LL-HLS | 3-5s | Excellent (HTTP-based) | All modern browsers | Low |
| WebRTC | 500ms-2s | Hard (P2P / SFU needed) | All modern browsers | High |
| CMAF / DASH | 3-10s | Excellent | Limited (needs JS) | Medium |
| MSE + FLV | 2-4s | Good | Chrome/Edge only | Medium |
Decision: LL-HLS as primary, WebRTC as optional low-latency mode for premium subscribers.
Rationale:
- LL-HLS provides the best balance of latency, scalability, and compatibility
- HTTP-based means it works through any firewall and leverages existing CDN infrastructure
- WebRTC added as a premium feature for latency-sensitive viewers (betting markets, etc.)
7.2 Transcoding: Hardware vs Software
Decision: Hardware-accelerated (NVENC) as primary, software (x264/x265) for quality-critical content.
Rationale:
- NVENC provides 10-20x throughput improvement over software encoding at comparable quality
- At Super Bowl scale (200 Tbps output), software encoding would require ~50,000 CPU cores
- Hardware encoding reduces cost by ~80% and power consumption by ~70%
- Software encoding reserved for highlight reels and VOD where quality matters more than latency
7.3 DVR Storage: Object Storage vs Block Storage
Decision: Object storage (S3-compatible) with SSD tier for hot DVR window.
Rationale:
- Object storage scales to petabytes and provides durability (11 nines)
- SSD tier ensures sub-100ms segment retrieval for DVR seeks
- Block storage would require complex volume management and doesn’t scale as gracefully
- Hot segments stay in SSD tier; older segments transition to standard storage
7.4 Multi-CDN Strategy
Decision: Primary + Secondary CDN with client-side failover.
Rationale:
- No single CDN covers all geographies optimally
- Client-side failover (via manifest redirect) is faster than DNS-based failover
- Cost optimization: route traffic to the cheapest CDN per region while maintaining quality
- Black events require pre-negotiated capacity with multiple providers
7.5 Geo-Rights Enforcement
Decision: Edge-enforced with origin validation (defense in depth).
Rationale:
- CDN edge can block based on IP geo-location (fast, low-latency)
- Origin performs token-based validation as a second layer
- Token contains signed geo-entitlements; prevents CDN bypass attacks
- Legal compliance requires both technical and contractual enforcement
8. Scalability Strategy
8.1 Horizontal Scaling Plan
| Component | Scaling Trigger | Min | Max | Scale Method |
|---|---|---|---|---|
| Ingest Gateway | CPU > 70% or stream count | 10 | 200 | K8s HPA |
| Transcoders | Queue depth > 100 segments | 50 | 2000 | GPU pool autoscaler |
| Origin Servers | RPS > 50K/node | 20 | 500 | K8s HPA |
| Auth Service | Latency > 100ms | 10 | 100 | K8s HPA |
| DVR Writers | Write queue depth | 5 | 50 | Fixed per event |
| Analytics | Kafka lag > 1000 | 5 | 50 | Flink parallelism |
8.2 Black Event Preparation
For known mega-events (Super Bowl, World Cup Final):
- T-90 days: Reserve CDN capacity; pre-provision transcoder GPU clusters
- T-30 days: Load test with synthetic traffic at 150% expected peak
- T-7 days: Pre-position static assets and common segments at edge POPs
- T-1 day: Warm all caches; verify failover paths; run dry-run ingest
- T-1 hour: Scale all services to peak capacity; enable enhanced monitoring
- During event: Real-time dashboard with per-POP health; on-call war room
- Post-event: Gradual scale-down over 24-48 hours
8.3 Database Scaling
- Primary DB: PostgreSQL with read replicas (1 writer, 8+ readers)
- Sharding: Events sharded by
sportorregion; users sharded byuser_idhash - Caching: Redis cluster for hot data (live event status, DVR indexes, auth tokens)
- Analytics: Write-to-Kafka → Flink → ClickHouse for real-time analytics queries
8.4 CDN Scaling
- Edge POP Count: 1,000+ global POPs for standard operation; 2,000+ for black events
- Capacity Reservation: Pre-purchase bandwidth guarantees with CDN providers
- Traffic Shaping: QoS policies prioritize live segments over VOD during peak
- Protocol Optimization: HTTP/3 (QUIC) for reduced connection setup time
9. Curveball Scenarios
Curveball 1: “The Super Bowl stream drops 3 minutes before the end — 80 million viewers are affected. What happens and how do you recover?”
Impact Assessment:
- 80M concurrent viewers lose stream during the most critical moment
- Social media explosion; brand damage; potential regulatory scrutiny
- Revenue impact from advertiser complaints
Root Cause Analysis Framework:
Possible Failure Points:
├── Ingest Layer
│ ├── Broadcast truck SRT link failure
│ ├── Primary feed corruption
│ └── Ingest gateway overload
├── Transcoding Layer
│ ├── GPU cluster failure
│ ├── Pipeline backlog overflow
│ └── Storage write bottleneck
├── Distribution Layer
│ ├── Origin server exhaustion
│ ├── CDN POP failure (regional)
│ └── BGP routing issue
└── Network Layer
├── Backbone congestion
├── DDoS attack
└── DNS resolution failure
Recovery Plan:
-
Immediate (0-30 seconds):
- Automated failover to backup ingest feed (should trigger within 3 seconds)
- If transcoding failed: activate cold-standby transcoder cluster in a different region
- If CDN failed: client-side multi-CDN failover redirects to secondary CDN
-
Short-term (30 sec - 5 min):
- Origin servers scale up automatically (pre-provisioned capacity for black events)
- CDN re-routes traffic away from failed POPs
- Push notification to affected users: “We’re restoring your stream”
-
Communication:
- Status page update within 60 seconds
- Social media response within 5 minutes
- Post-event transparency report within 48 hours
Prevention for Future:
- N+2 redundancy at every layer for black events (not just N+1)
- Chaos engineering: Monthly failure injection tests on the full pipeline
- Geographic diversity: Ingest, transcode, and origin in at least 3 regions
- Circuit breakers: Automatic isolation of failing components to prevent cascade
Curveball 2: “A major league wants to add AR (Augmented Reality) first-down lines and real-time player tracking overlays. How does this change the architecture?”
Architecture Changes:
-
New Data Pipeline:
Camera Feeds → Computer Vision Engine → Player Positions (60fps) → AR Rendering Engine → Overlay Graphics → Compositing with Video → Transcoded Output -
Processing Requirements:
- Computer vision: ~200 GPU cores per field (object detection + tracking)
- AR rendering: Real-time 3D engine (Unreal/Unity) generating overlay frames
- Compositing: Additional 500ms in the transcoding pipeline
-
Latency Impact:
- AR pipeline adds 800-1500ms end-to-end
- Must be absorbed within the existing 3-second budget
- Solution: Run AR pipeline in parallel with transcoding; composite at the segment level
-
Client-Side Alternative:
- Send player position data as a separate WebSocket stream
- Client renders AR overlay locally using the position data
- Reduces server load; enables user-customizable AR
- Trade-off: Requires client-side 3D rendering capability
-
Storage Impact:
- Player tracking data: ~50 KB/sec per player × 22 players = ~1.1 MB/sec
- 3-hour game: ~12 GB of tracking data per event
- Store alongside video for replay with AR reconstruction
Recommendation: Start with client-side AR (lower risk, more flexible), add server-side compositing for broadcast-quality output.
Curveball 3: “Regulators require that all live broadcasts in the EU be stored for 90 days for compliance. How does this affect storage costs and architecture?”
Storage Impact Calculation:
Per Event Storage (90-day retention):
├── Source (4K @ 50 Mbps): 3 hrs × 50 Mbps × 3600 = 6.75 GB
├── Transcoded (5 profiles): 6.75 GB × 5 = 33.75 GB
├── Multi-angle (4 angles): 33.75 GB × 4 = 135 GB/event
└── Metadata + Indexes: ~1 GB/event
Daily Events (EU-only): ~20 events/day
Daily Storage: 20 × 135 GB = 2.7 TB/day
90-Day Retention: 2.7 TB × 90 = 243 TB
Cost Analysis (AWS pricing estimates):
| Storage Tier | Use Case | Cost/TB/Month | Monthly Cost |
|---|---|---|---|
| S3 Standard | 0-7 days (hot) | $23 | $5,589 |
| S3 IA | 7-30 days (warm) | $12.5 | $2,925 |
| S3 Glacier Deep | 30-90 days (cold) | $4 | $972 |
| Total | ~$9,486/month |
Architecture Changes:
-
Lifecycle Policy:
Day 0-7: S3 Standard (frequent access for DVR + replays) Day 7-30: S3 Infrequent Access (replay requests) Day 30-90: S3 Glacier Deep Archive (compliance only) -
Compliance Service:
- Immutable storage (S3 Object Lock) to prevent deletion
- Audit logging for all access attempts
- Automated retention policy enforcement
- Export capability for regulatory requests
-
Index Preservation:
- Maintain segment indexes in a compressed format
- Enable fast retrieval from Glacier (3-5 hour restore for deep archive)
- Keep a manifest database for quick event lookup
-
Legal Considerations:
- GDPR compliance: ensure personal data (chat, user info) is handled separately
- Right to be forgotten: video content vs. personal data distinction
- Cross-border data transfer: store EU content in EU regions only
Curveball 4: “A DDoS attack targets your origin servers during a championship final. How do you protect the service?”
Defense-in-Depth Strategy:
-
Perimeter Defense:
- CDN absorbs the majority of attack traffic (CDN providers handle 10+ Tbps attacks)
- DDoS protection service (e.g., AWS Shield Advanced, Akamai Prolexic)
- Rate limiting at the CDN edge per IP and per session
-
Origin Protection:
- Origin is behind a WAF (Web Application Firewall) with strict allow-listing
- Only CDN edge IPs can reach origin (IP allow-list)
- Token-based authentication on all origin requests
-
Application-Level:
- JWT token validation prevents unauthorized manifest requests
- Rate limiting per user token (not just IP)
- Bot detection via behavioral analysis
-
Capacity Absorption:
- Pre-provisioned excess capacity for black events absorbs attack overhead
- Auto-scaling triggers at lower thresholds during known attack patterns
-
Incident Response:
- Automated attack detection via anomaly detection on traffic patterns
- One-click activation of enhanced DDoS mitigation rules
- Communication plan for stakeholders
Curveball 5: “Viewers complain that the stream is 30 seconds behind Twitter. How do you reduce latency further?”
Current State Analysis:
- LL-HLS typical latency: 3-5 seconds
- If viewers report 30 seconds, something is wrong:
- CDN caching too aggressive (long TTL on live segments)
- Client buffer too large
- Network congestion adding delay
Optimization Steps:
-
Immediate Fixes:
- Reduce segment duration from 6s to 2s
- Enable LL-HLS
Partsegments (0.5-1s parts within a segment) - Reduce client buffer to 2 segments minimum
- Verify CDN TTL is set to segment duration (not longer)
-
Protocol Upgrade:
- Deploy WebRTC low-latency mode as an option
- WebRTC achieves 500ms-2 second latency
- Use SFU (Selective Forwarding Unit) architecture for scalability
-
Network Optimization:
- HTTP/3 (QUIC) for faster connection setup and 0-RTT resumption
- Prefetch next segment while playing current
- Edge POPs closer to viewers
-
Target Latency Breakdown:
| Component | Current | Optimized | WebRTC Mode |
|---|---|---|---|
| Ingest → Transcode | 500ms | 300ms | 200ms |
| Transcode → Origin | 500ms | 300ms | 200ms |
| Origin → CDN Edge | 200ms | 100ms | 100ms |
| CDN Edge → Client | 300ms | 200ms | 100ms |
| Client Buffer | 2000ms | 1000ms | 500ms |
| Total | ~4s | ~2s | ~1.1s |
- Trade-off Warning: Lower latency increases rebuffer risk. Must balance with adaptive bitrate quality.
10. Failure Modes and Mitigations
10.1 Failure Mode Matrix
| Component | Failure Mode | Probability | Impact | Mitigation | RTO |
|---|---|---|---|---|---|
| Ingest | Primary feed loss | Medium | High | N+1 backup feed; auto-failover in < 3s | < 3s |
| Ingest | All feeds lost | Low | Critical | Pre-recorded content fallback; “technical difficulty” screen | < 10s |
| Transcode | GPU node failure | Medium | Medium | Redundant encoders per profile; auto-reassign | < 5s |
| Transcode | Pipeline backlog | Low | High | Drop oldest segments; alert ops; scale up | < 30s |
| DVR | Storage write failure | Low | Medium | Dual-write to redundant storage; queue and retry | < 10s |
| CDN | POP failure | Medium | Medium | Multi-CDN failover; traffic reroute | < 15s |
| CDN | Origin overload | Low | High | Origin shield caching; request queuing; rate limit | < 30s |
| Auth | Token service down | Low | High | Cached tokens with extended TTL; graceful degradation | < 60s |
| Geo | Geo-DB stale | Low | Medium | Cache with short TTL; fallback to allow | < 5s |
| Stats | Data feed loss | Medium | Low | Last-known-state hold; “data delayed” indicator | N/A |
| DB | Primary DB failover | Low | High | Automated failover to replica; read-only mode | < 30s |
| Network | BGP hijack | Very Low | Critical | Multi-homed network; BGP monitoring; rapid reroute | < 60s |
10.2 Cascading Failure Prevention
flowchart LR
subgraph CIRCUIT_BREAKERS["Circuit Breakers"]
CB1[Ingest → Transcode<br/>Break if queue > 1000]
CB2[Transcode → Origin<br/>Break if latency > 5s]
CB3[Origin → CDN<br/>Break if error rate > 5%]
CB4[Client → Origin<br/>Break if 5xx > 10%]
end
subgraph FALLBACKS["Fallback Strategies"]
FB1[Backup Ingest Feed]
FB2[Pre-Cached Segments]
FB3[Secondary CDN]
FB4[Static Error Page]
end
CB1 --> FB1
CB2 --> FB2
CB3 --> FB3
CB4 --> FB4
10.3 Monitoring and Alerting
Key Metrics:
| Metric | Warning Threshold | Critical Threshold | Action |
|---|---|---|---|
| End-to-end latency | > 8s | > 15s | Scale transcoders; check ingest |
| Rebuffer ratio | > 2% | > 5% | Check CDN health; reduce bitrate |
| Ingest packet loss | > 1% | > 5% | Switch to backup feed |
| Transcode queue depth | > 100 | > 500 | Scale encoder cluster |
| Origin 5xx rate | > 1% | > 5% | Scale origin; enable circuit breaker |
| CDN cache hit ratio | < 95% | < 85% | Check TTL; warm caches |
| Concurrent viewers | — | — | Info only; capacity planning |
| DVR storage usage | > 80% | > 95% | Evict older segments; scale storage |
Observability Stack:
- Metrics: Prometheus + Grafana (real-time dashboards)
- Logging: ELK Stack (Elasticsearch, Logstash, Kibana)
- Tracing: Jaeger / OpenTelemetry (end-to-end request tracing)
- Alerting: PagerDuty integration with severity-based escalation
Appendix: Technology Stack Summary
| Layer | Technology Choices |
|---|---|
| Ingest | SRT (Haivision), RTMP (NGINX RTMP), GStreamer |
| Transcoding | NVIDIA NVENC, FFmpeg, Intel QSV, AWS MediaConvert |
| Packaging | Bento4, MP4Box, LL-HLS packager |
| Origin | NGINX, Apache Traffic Server, custom Go service |
| CDN | Akamai (primary), CloudFront (secondary), Cloudflare (tertiary) |
| Storage | AWS S3 (hot + warm), S3 Glacier (archive), Redis (indexes) |
| Database | PostgreSQL (primary), Redis (cache), ClickHouse (analytics) |
| Streaming | LL-HLS (primary), WebRTC (low-latency mode), DASH (fallback) |
| Messaging | Apache Kafka (events), Redis Pub/Sub (real-time) |
| Stream Processing | Apache Flink (analytics), Kafka Streams (lightweight) |
| Auth | Auth0 / Keycloak, JWT tokens |
| Monitoring | Prometheus, Grafana, ELK, Jaeger |
| Infrastructure | Kubernetes, Terraform, Helm |