Design an On-Demand Streaming App like Netflix
Scope: A global video-on-demand streaming platform delivering personalized, high-quality content to 200M+ subscribers with adaptive bitrate streaming, intelligent recommendations, DRM protection, and an always-on prefetching engine.
Table of Contents
- Requirements Clarification
- Back-of-Envelope Calculations
- System Interface Design
- High-Level Architecture
- Core Component Design
- Data Model
- Key Design Decisions
- Scalability Strategy
- Curveball Scenarios
- Failure Modes and Mitigations
1. Requirements Clarification
1.1 Functional Requirements
- Video On-Demand Playback: Stream movies and TV shows with play, pause, seek, and resume
- Multi-Bitrate Adaptive Streaming: Automatically adjust quality based on network conditions (HLS / DASH)
- Recommendation Engine: Personalized content recommendations based on viewing history, preferences, and collaborative filtering
- User Profiles: Multiple profiles per account (up to 5); individual watchlists and recommendations per profile
- Continue Watching: Resume playback across devices from the last watched position
- Download for Offline: Allow premium users to download content for offline viewing (with DRM)
- DRM Protection: Content encryption and license management (Widevine, FairPlay, PlayReady)
- Search and Discovery: Full-text search, genre browsing, trending, and curated collections
- Subtitles and Audio Tracks: Multi-language subtitles and dubbed audio tracks
- Social Features: Shared watchlists, ratings, and reviews (optional)
- Parental Controls: Content rating restrictions and PIN-protected access
- Subscription Management: Tier management (Basic, Standard, Premium), billing, and auto-renewal
- Content Upload and Processing: Internal pipeline for ingesting, encoding, and publishing new titles
- A/B Testing Framework: Test UI changes, recommendation algorithms, and thumbnails
1.2 Non-Functional Requirements
| Requirement | Target | Rationale |
|---|---|---|
| Availability | 99.95% uptime | Downtime directly impacts subscriber retention |
| Startup Time | < 3 seconds from tap to first frame | Critical for user experience |
| Rebuffer Ratio | < 0.5% of watch time | Industry benchmark for premium streaming |
| Concurrent Streams | 200M+ subscribers, ~40M concurrent | Peak evening hours in largest markets |
| Content Catalog | 15,000+ titles, growing 50-100/month | Competitive content library |
| Recommendation Accuracy | > 80% click-through on top recommendations | Drives engagement and retention |
| Search Latency | < 200ms p95 | Instant search experience |
| Geo-Availability | 190+ countries with localized catalogs | Global reach with regional licensing |
| Data Privacy | GDPR, CCPA compliant | Legal requirement |
| DRM Compliance | Widevine L1, FairPlay, PlayReady | Required for premium content distribution |
2. Back-of-Envelope Calculations
2.1 Assumption Sets
| Parameter | Conservative | Current Scale | Future (3 years) |
|---|---|---|---|
| Subscribers | 50M | 200M | 500M |
| Concurrent Viewers (peak) | 10M | 40M | 100M |
| Avg Daily Watch Time | 90 min | 120 min | 150 min |
| Avg Bitrate | 3 Mbps | 4 Mbps (HD default) | 5 Mbps (4K growth) |
| Titles in Catalog | 5,000 | 15,000 | 25,000 |
| Avg Title Duration | 90 min (movie) / 40 min (episode) | Same | Same |
| Encoding Profiles per Title | 8 | 12 | 15 |
| Languages per Title | 3 | 6 | 10 |
2.2 Bandwidth Estimates
Daily Outbound Traffic:
| Scenario | Calculation | Daily Traffic |
|---|---|---|
| Conservative | 10M × 90 min × 3 Mbps | 1.94 PB/day |
| Current | 40M × 120 min × 4 Mbps | 12.8 PB/day |
| Future | 100M × 150 min × 5 Mbps | 46.9 PB/day |
Peak Hour Bandwidth:
| Scenario | Calculation | Peak Bandwidth |
|---|---|---|
| Conservative | 10M × 3 Mbps | 30 Tbps |
| Current | 40M × 4 Mbps | 160 Tbps |
| Future | 100M × 5 Mbps | 500 Tbps |
Note: 160 Tbps sustained is massive. Netflix actually uses ~3-5% of global internet bandwidth. This requires a dedicated CDN (Open Connect) rather than relying on public CDNs.
2.3 Storage Estimates
Content Library Storage:
Per Title Storage (movie, 90 min):
├── Source (4K ProRes): 90 min × 500 Mbps = 337.5 GB
├── Encoding Profiles (12 profiles):
│ ├── 4K HDR @ 25 Mbps: 90 × 25 Mbps = 28.1 GB
│ ├── 4K @ 15 Mbps: 90 × 15 Mbps = 16.9 GB
│ ├── 1080p HDR @ 10 Mbps: 90 × 10 Mbps = 11.3 GB
│ ├── 1080p @ 6 Mbps: 90 × 6 Mbps = 6.75 GB
│ ├── 720p @ 3 Mbps: 90 × 3 Mbps = 3.37 GB
│ ├── 480p @ 1.5 Mbps: 90 × 1.5 Mbps = 1.69 GB
│ ├── 360p @ 800 Kbps: 90 × 0.8 Mbps = 0.84 GB
│ └── Audio-only @ 128 Kbps: 90 × 0.128 Mbps = 0.13 GB
│ Total Encoded: ~69 GB
├── Audio Tracks (6 languages × 320 Kbps): 0.6 GB
├── Subtitle Files (10 languages): 50 MB
├── Metadata, Thumbnails, Posters: 100 MB
└── Total per Title: ~408 GB (source) + ~70 GB (encoded) = ~478 GB
15,000 Titles:
├── Source Archive: 15,000 × 408 GB = 6.1 PB
├── Encoded (delivery): 15,000 × 70 GB = 1.05 PB
└── Total Origin Storage: ~7.2 PB
CDN Storage (replicated across POPs):
- Each POP stores a subset of the catalog based on regional demand
- Top 1,000 titles at each POP: 1,000 × 70 GB = 70 TB per POP
- 2,000 POPs × 70 TB = 140 PB total CDN storage (with overlap)
User Data Storage:
- Watch history, preferences, profiles: ~1 KB/user → 200M × 1 KB = 200 GB
- Playback positions: ~100 bytes/title/user → 200M × 50 titles × 100 B = 1 TB
- Recommendation model data: ~50 GB
- Search index: ~500 GB
2.4 QPS Estimates
| Operation | Off-Peak | Peak Evening | Black Event (New Release) |
|---|---|---|---|
| Playback Start | 50K/s | 200K/s | 2M/s |
| Segment Requests | 5M/s | 20M/s | 200M/s |
| API Calls (browse, search) | 100K/s | 500K/s | 5M/s |
| Recommendation Requests | 20K/s | 100K/s | 500K/s |
| Auth Token Requests | 10K/s | 50K/s | 500K/s |
| Search Queries | 50K/s | 200K/s | 2M/s |
| Profile Switch | 5K/s | 20K/s | 100K/s |
2.5 Encoding Compute
Batch Encoding (new title):
- Per profile: ~2-4x real-time on GPU (NVENC)
- 90-minute movie × 12 profiles × 3x real-time = 36 hours of GPU time
- With 100 GPU encoders: ~21 minutes per title
- 100 new titles/month: ~35 GPU-hours/month (trivial at scale)
Real-Time Processing (live events, if added):
- Same as ESPN design; not primary focus for VOD
3. System Interface Design
3.1 Public APIs
# --- Content Catalog APIs ---
GET /api/v1/browse/home
Get personalized home page content rows
Headers: Authorization: Bearer <token>
Response: {
rows: [
{
title: "Continue Watching",
items: [{ id, title, image_url, progress_percent, resume_point }]
},
{
title: "Recommended for You",
items: [{ id, title, image_url, match_score, description }]
},
...
]
}
GET /api/v1/browse/genre/{genre_id}
Browse titles by genre
Query: ?sort=relevance&offset=0&limit=25
Response: { titles: [...], total_count, has_more }
GET /api/v1/search
Full-text search across titles
Query: ?q=action+movie&types=movie,series&offset=0&limit=25
Response: { results: [...], suggestions: [...], facets: [...] }
GET /api/v1/titles/{id}
Get title details
Response: {
id, title, description, type, year, rating, duration,
cast, crew, genres, maturity_rating,
audio_tracks: [{ language, codec }],
subtitle_tracks: [{ language, url }],
seasons: [{ season_number, episodes: [...] }],
availability: { available_from, available_until, geo_available }
}
# --- Playback APIs ---
GET /api/v1/titles/{id}/manifest
Get streaming manifest (HLS/DASH)
Headers: Authorization: Bearer <token>, DRM-Header: <license_info>
Query: ?device=smart_tv&codec=h265&hdr=true
Response: HLS/DASH manifest with encrypted segment URLs
GET /api/v1/titles/{id}/segments/{segment_id}
Get encrypted media segment (served by CDN)
Response: Encrypted MP4 segment
POST /api/v1/playback/session
Create a playback session
Request: { title_id, episode_id, device_id, profile_id }
Response: { session_id, manifest_url, drm_license_url, expires_at }
POST /api/v1/playback/session/{id}/progress
Update playback progress
Request: { position_seconds, is_completed }
GET /api/v1/playback/session/{id}/resume
Get resume point for a title
Response: { position_seconds, last_watched_at }
# --- DRM APIs ---
POST /api/v1/drm/license
Request DRM license for content decryption
Request: { content_id, device_info, hw_id, challenge }
Response: { license: <encrypted_license>, expires_in }
# --- User / Profile APIs ---
GET /api/v1/user/profiles
List user profiles
Response: { profiles: [{ id, name, avatar, maturity_rating, is_child }] }
POST /api/v1/user/profiles
Create a new profile
PUT /api/v1/user/profiles/{id}
Update profile settings
GET /api/v1/user/profiles/{id}/watchlist
Get profile's watchlist
POST /api/v1/user/profiles/{id}/watchlist
Add title to watchlist
DELETE /api/v1/user/profiles/{id}/watchlist/{title_id}
Remove title from watchlist
GET /api/v1/user/profiles/{id}/history
Get watch history
POST /api/v1/user/profiles/{id}/rating
Rate a title
Request: { title_id, rating: 1-5 }
# --- Download APIs ---
POST /api/v1/downloads/{title_id}
Initiate offline download
Request: { profile_id, quality: "high" | "low" }
Response: { download_id, estimated_size, expires_at }
GET /api/v1/downloads/{id}/status
Check download progress
Response: { status: "downloading" | "completed" | "failed", progress_percent }
DELETE /api/v1/downloads/{id}
Remove downloaded content
# --- Subscription APIs ---
GET /api/v1/subscription
Get current subscription details
Response: { tier, max_streams, max_quality, renewal_date, auto_renew }
POST /api/v1/subscription/update
Change subscription tier
Request: { tier: "basic" | "standard" | "premium" }
3.2 Internal Service APIs (gRPC)
// Recommendation Service
service RecommendationService {
rpc GetHomeRow(HomeRowRequest) returns (HomeRowResponse);
rpc GetSimilarTitles(SimilarRequest) returns (SimilarResponse);
rpc UpdateUserModel(UserEvent) returns (Empty);
rpc BatchScoreTitles(BatchScoreRequest) returns (BatchScoreResponse);
}
// Encoding Service
service EncodingService {
rpc SubmitEncodingJob(EncodingJobRequest) returns (JobId);
rpc GetJobStatus(JobId) returns (JobStatus);
rpc CancelJob(JobId) returns (Empty);
}
// DRM Service
service DRMService {
rpc GenerateKey(KeyRequest) returns (KeyResponse);
rpc IssueLicense(LicenseRequest) returns (LicenseResponse);
rpc RevokeLicense(RevokeRequest) returns (Empty);
}
// Prefetch Service
service PrefetchService {
rpc PredictDemand(DemandRequest) returns (DemandPrediction);
rpc ScheduleReplication(ReplicationRequest) returns (Empty);
rpc GetPOPStatus(POPId) returns (POPStatus);
}
// Search Service
service SearchService {
rpc Search(SearchRequest) returns (SearchResponse);
rpc IndexTitle(IndexRequest) returns (Empty);
rpc RemoveTitle(RemoveRequest) returns (Empty);
}
4. High-Level Architecture
flowchart TB
subgraph CONTENT["🎬 Content Pipeline"]
UPLOAD[Content Ingestion<br/>FTP / Aspera]
ENCODE[Encoding Cluster<br/>Multi-Bitrate + Multi-Codec]
QC[Quality Assurance<br/>Automated + Manual]
PACK[Packaging<br/>HLS / DASH + DRM]
META[Metadata Enrichment<br/>Cast, Genres, Descriptions]
end
subgraph STORAGE["💾 Storage Layer"]
ORIGIN_STORE[Origin Storage<br/>Encrypted Content]
CDN_POps[CDN POPs<br/>Open Connect Appliances]
SEARCH_IDX[Search Index<br/>Elasticsearch Cluster]
REC_MODEL[Recommendation Models<br/>ML Model Store]
end
subgraph PLATFORM["🔧 Platform Services"]
AUTH[Auth Service<br/>OAuth 2.0 / JWT]
USER[User Service<br/>Profiles, Preferences]
REC[Recommendation Engine<br/>Real-Time + Batch]
SEARCH[Search Service<br/>Full-Text + Faceted]
DRM[DRM License Server<br/>Widevine / FairPlay / PlayReady]
BILLING[Billing Service<br/>Subscription Management]
ANALYTICS[Analytics Pipeline<br/>Event Processing]
PREFETCH[Prefetch Engine<br/>Demand Prediction]
end
subgraph API_GATEWAY["🌐 API Gateway"]
GW[API Gateway<br/>Rate Limiting + Routing]
end
subgraph CLIENTS["📱 Client Apps"]
WEB[Web App<br/>Chrome, Safari, Firefox]
MOBILE[iOS / Android App]
TV[Smart TV Apps<br/>Roku, Apple TV, Fire TV, Tizen, webOS]
CONSOLE[PS5 / Xbox]
end
UPLOAD --> ENCODE
ENCODE --> QC
QC -->|Pass| PACK
QC -->|Fail| ENCODE
PACK --> ORIGIN_STORE
META --> ORIGIN_STORE
ORIGIN_STORE --> CDN_POps
PREFETCH --> CDN_POps
CLIENTS --> GW
GW --> AUTH
GW --> USER
GW --> REC
GW --> SEARCH
GW --> DRM
GW --> BILLING
SEARCH --> SEARCH_IDX
REC --> REC_MODEL
CDN_POps --> CLIENTS
DRM --> CLIENTS
ANALYTICS --> REC
ANALYTICS --> PREFETCH
5. Core Component Design
5.1 Video Encoding Pipeline
Purpose: Convert source content into multiple delivery-optimized profiles with different bitrates, resolutions, codecs, and HDR formats.
Encoding Ladder:
| Profile | Resolution | Codec | Video Bitrate | Audio | Use Case |
|---|---|---|---|---|---|
| 4K HDR Dolby Vision | 3840×2160 | H.265 10-bit | 20-25 Mbps | AC-4 5.1 | Premium 4K TVs |
| 4K HDR | 3840×2160 | H.265 10-bit | 15-20 Mbps | E-AC3 5.1 | Standard 4K TVs |
| 4K SDR | 3840×2160 | H.265 | 12-15 Mbps | AAC 5.1 | 4K non-HDR |
| 1080p HDR | 1920×1080 | H.265 | 8-10 Mbps | E-AC3 5.1 | HDR mid-tier |
| 1080p | 1920×1080 | H.264 | 5-6 Mbps | AAC 5.1 | Standard HD |
| 720p | 1280×720 | H.264 | 2.5-3 Mbps | AAC Stereo | Good quality |
| 480p | 854×480 | H.264 | 1-1.5 Mbps | AAC Stereo | Mobile / low bandwidth |
| 360p | 640×360 | H.264 | 500-800 Kbps | AAC Stereo | Very low bandwidth |
| Audio Only | — | — | — | AAC 128 Kbps | Data-saving mode |
Pipeline Architecture:
flowchart LR
SRC[Source File<br/>4K ProRes / DNxHR]
subgraph ANALYSIS["Content Analysis"]
SCENE[Scene Detection]
COMPLEX[Complexity Analysis]
COLOR[Color Space Analysis]
end
subgraph ENCODE["Parallel Encoding"]
E1[Encoder 1<br/>4K Profiles]
E2[Encoder 2<br/>HD Profiles]
E3[Encoder 3<br/>SD Profiles]
E4[Encoder 4<br/>Audio Extraction]
end
subgraph POST["Post-Processing"]
DRM_ENC[DRM Encryption]
MANIFEST[Manifest Generation]
THUMB[Thumbnail Generation]
end
SRC --> ANALYSIS
ANALYSIS --> E1
ANALYSIS --> E2
ANALYSIS --> E3
ANALYSIS --> E4
E1 --> POST
E2 --> POST
E3 --> POST
E4 --> POST
POST -->|All Profiles| OUT[Ready for Distribution]
Key Design Decisions:
- Per-Title Encoding: Each title is encoded independently with parameters tuned to its content complexity (scene detection determines keyframe placement)
- VBR (Variable Bitrate): Used instead of CBR to maintain quality while reducing average bitrate by 15-20%
- Hardware Acceleration: NVENC for H.264/H.265; software x264/x265 for quality-critical profiles
- Encoding Time: A 90-minute movie takes 2-4 hours with 50 GPU encoders
- Quality Metrics: VMAF (Video Multimethod Assessment Fusion) target > 90 for all profiles
5.2 CDN and POP Architecture
Purpose: Deliver content to viewers with minimal latency and maximum throughput using a dedicated CDN infrastructure.
Open Connect Appliance (OCA) Design:
flowchart TB
subgraph ORIGIN["Origin Data Centers (3-5)"]
O1[Origin DC 1<br/>Full Catalog]
O2[Origin DC 2<br/>Full Catalog]
O3[Origin DC 3<br/>Full Catalog]
end
subgraph REGIONAL["Regional Distribution Centers (20-50)"]
R1[RDC 1<br/>Regional Subset]
R2[RDC 2<br/>Regional Subset]
end
subgraph POPS["Edge POPs (2000+)"]
P1[POP 1<br/>Top 1000 Titles]
P2[POP 2<br/>Top 1000 Titles]
P3[POP 3<br/>Top 1000 Titles]
Pn[POP N<br/>Top 1000 Titles]
end
subgraph ISPS["ISP Peering Points"]
I1[ISP 1 Cache]
I2[ISP 2 Cache]
end
VIEWERS[Viewers]
O1 --> R1
O2 --> R2
R1 --> P1
R2 --> P2
P1 --> I1
P2 --> I2
I1 --> VIEWERS
I2 --> VIEWERS
OCA Specifications:
| Component | Specification |
|---|---|
| Storage | 100+ TB per appliance (top titles) |
| Network | 40-100 Gbps uplink per POP |
| Cache Hit Ratio | > 95% for top content |
| Replication | Background sync from RDC; priority sync for new releases |
Content Placement Strategy:
- Global Top 100: Replicated to every POP immediately
- Regional Top 1,000: Replicated based on regional popularity
- Long Tail: Stored at RDC level; fetched on-demand
- New Releases: Priority replication to all POPs within 1-2 hours
- Predictive Prefetch: ML model predicts upcoming demand and pre-positions content
Prefetch Algorithm:
Demand Score = w1 × Trending_Score + w2 × Release_Proximity
+ w3 × Day_of_Week_Pattern + w4 × Geo_Popularity
+ w5 × Marketing_Campaign_Boost
If Demand Score > Threshold:
Schedule replication to relevant POPs
Priority = Score / Current_POP_Availability
5.3 Recommendation Engine
Purpose: Deliver personalized content recommendations that drive engagement and retention.
Multi-Stage Architecture:
flowchart TB
subgraph STAGE1["Stage 1: Candidate Generation"]
CF[Collaborative Filtering<br/>User-Item Matrix]
CONTENT[Content-Based<br/>Genre, Cast, Director]
TRENDING[Trending<br/>Global + Regional]
CONTINUE[Continue Watching<br/>Resume Points]
end
subgraph STAGE2["Stage 2: Feature Extraction"]
USER_FEAT[User Features<br/>Watch History, Demographics]
ITEM_FEAT[Item Features<br/>Genre, Rating, Freshness]
CONTEXT[Context Features<br/>Time, Device, Network]
end
subgraph STAGE3["Stage 3: Ranking"]
MODEL[Deep Learning Ranker<br/>Wide & Deep / Two-Tower]
DIVERSITY[Diversity Filter<br/>MMR Algorithm]
BUSINESS[Business Rules<br/>Promotion, Geo-Rights]
end
subgraph STAGE4["Stage 4: Presentation"]
ROWS[Row Generation<br/>Curated Rows]
THUMBNAILS[Dynamic Thumbnails<br/>A/B Tested]
end
CF --> STAGE2
CONTENT --> STAGE2
TRENDING --> STAGE2
CONTINUE --> STAGE2
STAGE2 --> MODEL
MODEL --> DIVERSITY
DIVERSITY --> BUSINESS
BUSINESS --> ROWS
ROWS --> THUMBNAILS
Algorithm Details:
-
Candidate Generation:
- Collaborative filtering using matrix factorization (ALS) on 200M users × 15K items
- Content-based filtering using title embeddings (genre, cast, director, plot similarity)
- Real-time candidates from recent watch activity
-
Ranking Model:
- Two-tower neural network: user tower + item tower → similarity score
- Trained on implicit feedback (watch, complete, skip, search)
- Features: 200+ user features, 100+ item features, 50+ context features
- Training: Daily batch retraining + hourly incremental updates
-
Diversity and Exploration:
- MMR (Maximal Marginal Relevance) to avoid同质化 recommendations
- Epsilon-greedy exploration: 5% of recommendations are exploratory
- Category diversity: ensure multiple genres in top 50
-
Dynamic Thumbnails:
- Multiple thumbnail variants per title
- User-specific thumbnail selection based on predicted click-through rate
- A/B testing framework for continuous optimization
Scale:
- Model inference: < 50ms per request (batched for home page)
- Training data: 10B+ events/day
- Model size: ~500 MB (two-tower embeddings)
- Update frequency: Daily full retrain + hourly incremental
5.4 Adaptive Bitrate Streaming (ABR)
Purpose: Automatically adjust video quality based on network conditions to minimize rebuffering while maximizing quality.
Manifest Structure (HLS Example):
#EXTM3U
#EXT-X-VERSION:7
#EXT-X-STREAM-INF:BANDWIDTH=25000000,RESOLUTION=3840x2160,HDR="DV",CODECS="hvc1.2.4.L153.b0"
4k_hdr_dv/master.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=15000000,RESOLUTION=3840x2160,HDR="PQ",CODECS="hvc1.2.4.L153.b0"
4k_hdr/master.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=8000000,RESOLUTION=1920x1080,CODECS="hvc1.1.6.L120.b0"
1080p_h265/master.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=6000000,RESOLUTION=1920x1080,CODECS="avc1.640028"
1080p/master.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=3000000,RESOLUTION=1280x720,CODECS="avc1.4d401f"
720p/master.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1500000,RESOLUTION=854x480,CODECS="avc1.4d401e"
480p/master.m3u8
Client-Side ABR Algorithm:
// Simplified ABR decision logic
function selectBitrate(buffer_health, bandwidth_estimate, previous_bitrate) {
// Buffer-based component
if (buffer_health < 2 seconds) {
return downgrade(previous_bitrate);
}
// Bandwidth-based component
estimated_throughput = exponential_moving_average(bandwidth_samples);
target_bitrate = estimated_throughput * 0.8; // 80% safety margin
// Smoothness constraint
max_change = previous_bitrate * 0.5; // Don't change more than one level
selected = closest_profile(target_bitrate, max_change);
// Minimum quality floor
selected = max(selected, MIN_BITRATE);
return selected;
}
Key ABR Parameters:
| Parameter | Value | Rationale |
|---|---|---|
| Buffer Target | 30 seconds | Balance between startup time and rebuffer resilience |
| Buffer Low Watermark | 5 seconds | Trigger downgrade below this |
| Bandwidth Estimation Window | 30 seconds | Smooth out transient fluctuations |
| Switch Hysteresis | 1 level per decision | Prevent oscillation between profiles |
| Startup Profile | 720p | Fast startup with good quality |
5.5 DRM System
Purpose: Protect content from unauthorized copying and distribution.
Three-DRM Approach:
| DRM System | Platforms | Key Level |
|---|---|---|
| Widevine | Android, Chrome, Linux | L1 (hardware-secure) |
| FairPlay Streaming | iOS, macOS, Safari | Hardware-secure |
| PlayReady | Windows, Xbox, Edge | Hardware-secure |
Encryption and License Flow:
sequenceDiagram
participant Client
participant CDN
participant LicenseServer
participant KeyServer
Client->>CDN: Request manifest
CDN-->>Client: Return manifest with encrypted segment URLs
Client->>CDN: Request encrypted segment
CDN-->>Client: Return encrypted segment (AES-128)
Client->>LicenseServer: Request license (with challenge)
LicenseServer->>KeyServer: Validate entitlements
KeyServer-->>LicenseServer: Return content key
LicenseServer-->>Client: Return encrypted license
Client->>Client: Decrypt license with device key
Client->>Client: Decrypt and play segment
Key Design Decisions:
- CENC (Common Encryption): Single encryption format (AES-128 CBC) compatible with all three DRM systems
- Key Rotation: New key per title (not per segment) to balance security and performance
- License Caching: Client caches licenses for offline playback (with expiration)
- Offline Downloads: Encrypted content stored on device; license checked periodically
- Hardware Security: L1 Widevine and equivalent required for HD/4K playback
5.6 Search System
Purpose: Enable fast, relevant search across the entire content catalog.
Architecture:
Client Query → API Gateway → Search Service → Elasticsearch Cluster
↓
Results Ranking
↓
Enrich with Metadata
↓
Return Results
Index Design:
{
"mappings": {
"properties": {
"title": { "type": "text", "analyzer": "title_analyzer" },
"description": { "type": "text", "analyzer": "content_analyzer" },
"cast": { "type": "keyword", "index": true },
"director": { "type": "keyword", "index": true },
"genres": { "type": "keyword", "index": true },
"year": { "type": "integer", "index": true },
"rating": { "type": "float", "index": true },
"type": { "type": "keyword", "index": true },
"popularity_score": { "type": "float", "index": true },
"geo_available": { "type": "keyword", "index": true }
}
}
}
Search Features:
- Typo Tolerance: Fuzzy matching with edit distance ≤ 2
- Autocomplete: Prefix Tree (Trie) with popularity weighting
- Faceted Search: Filter by genre, year, type, rating
- Personalized Ranking: Boost results based on user preferences
- Synonym Expansion: “star wars” → “Star Wars”, “SW”, “George Lucas”
Performance:
- Index size: ~500 GB for 15K titles with full metadata
- Query latency: < 50ms p95, < 200ms p99
- Throughput: 50K queries/second per cluster node
6. Data Model
6.1 Core Entities
-- Titles (Movies and Series)
CREATE TABLE titles (
title_id BIGINT PRIMARY KEY,
slug VARCHAR(200) UNIQUE NOT NULL,
title VARCHAR(500) NOT NULL,
type ENUM('movie', 'series', 'documentary', 'special') NOT NULL,
description TEXT,
year INT NOT NULL,
maturity_rating VARCHAR(10) NOT NULL, -- 'G', 'PG', 'PG-13', 'R', 'TV-MA'
duration_seconds INT, -- For movies
genres VARCHAR(100)[], -- Array of genre tags
cast JSONB, -- [{ name, role, image_url }]
crew JSONB, -- [{ name, role }]
poster_url VARCHAR(500),
backdrop_url VARCHAR(500),
trailer_url VARCHAR(500),
avg_rating DECIMAL(3,2),
view_count BIGINT DEFAULT 0,
is_original BOOLEAN DEFAULT false,
available_from TIMESTAMP NOT NULL,
available_until TIMESTAMP,
geo_availability JSONB, -- { "US": true, "JP": false, ... }
metadata JSONB,
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
-- Seasons and Episodes (for series)
CREATE TABLE seasons (
season_id BIGINT PRIMARY KEY,
title_id BIGINT REFERENCES titles(title_id),
season_number INT NOT NULL,
title VARCHAR(500),
description TEXT,
poster_url VARCHAR(500),
episode_count INT,
UNIQUE(title_id, season_number)
);
CREATE TABLE episodes (
episode_id BIGINT PRIMARY KEY,
season_id BIGINT REFERENCES seasons(season_id),
episode_number INT NOT NULL,
title VARCHAR(500) NOT NULL,
description TEXT,
duration_seconds INT NOT NULL,
poster_url VARCHAR(500),
available_from TIMESTAMP NOT NULL,
UNIQUE(season_id, episode_number)
);
-- Encoding Profiles and Assets
CREATE TABLE encoding_profiles (
profile_id BIGINT PRIMARY KEY,
title_id BIGINT REFERENCES titles(title_id),
content_type ENUM('movie', 'episode') NOT NULL,
content_id BIGINT NOT NULL, -- title_id or episode_id
resolution VARCHAR(20) NOT NULL,
codec VARCHAR(20) NOT NULL,
video_bitrate INT NOT NULL,
audio_bitrate INT NOT NULL,
hdr_format VARCHAR(20), -- 'HDR10', 'Dolby Vision', 'HLG'
segment_duration INT NOT NULL,
total_size_bytes BIGINT,
manifest_url VARCHAR(500) NOT NULL,
drm_key_id UUID NOT NULL,
status ENUM('encoding', 'ready', 'failed') DEFAULT 'encoding',
vmaf_score DECIMAL(5,2),
created_at TIMESTAMP DEFAULT NOW()
);
-- Users and Profiles
CREATE TABLE users (
user_id BIGINT PRIMARY KEY,
email VARCHAR(255) UNIQUE NOT NULL,
password_hash VARCHAR(255) NOT NULL,
subscription_tier ENUM('basic', 'standard', 'premium') NOT NULL,
subscription_end TIMESTAMP,
auto_renew BOOLEAN DEFAULT true,
country VARCHAR(10) NOT NULL,
language VARCHAR(10) DEFAULT 'en',
created_at TIMESTAMP DEFAULT NOW(),
updated_at TIMESTAMP DEFAULT NOW()
);
CREATE TABLE user_profiles (
profile_id BIGINT PRIMARY KEY,
user_id BIGINT REFERENCES users(user_id),
name VARCHAR(100) NOT NULL,
avatar_url VARCHAR(500),
maturity_rating VARCHAR(10) DEFAULT 'PG-13',
is_child BOOLEAN DEFAULT false,
language VARCHAR(10),
subtitle_language VARCHAR(10),
audio_language VARCHAR(10),
autoplay BOOLEAN DEFAULT true,
created_at TIMESTAMP DEFAULT NOW()
);
-- Watch History and Progress
CREATE TABLE watch_history (
history_id BIGINT PRIMARY KEY,
profile_id BIGINT REFERENCES user_profiles(profile_id),
title_id BIGINT REFERENCES titles(title_id),
episode_id BIGINT,
position_seconds INT NOT NULL DEFAULT 0,
is_completed BOOLEAN DEFAULT false,
last_watched_at TIMESTAMP DEFAULT NOW(),
device_id VARCHAR(100),
UNIQUE(profile_id, title_id, episode_id)
);
-- Watchlist
CREATE TABLE watchlist (
watchlist_id BIGINT PRIMARY KEY,
profile_id BIGINT REFERENCES user_profiles(profile_id),
title_id BIGINT REFERENCES titles(title_id),
added_at TIMESTAMP DEFAULT NOW(),
UNIQUE(profile_id, title_id)
);
-- Ratings
CREATE TABLE ratings (
rating_id BIGINT PRIMARY KEY,
profile_id BIGINT REFERENCES user_profiles(profile_id),
title_id BIGINT REFERENCES titles(title_id),
rating INT NOT NULL CHECK (rating BETWEEN 1 AND 5),
created_at TIMESTAMP DEFAULT NOW(),
UNIQUE(profile_id, title_id)
);
-- Recommendation Events (for ML training)
CREATE TABLE rec_events (
event_id BIGINT PRIMARY KEY,
profile_id BIGINT,
title_id BIGINT,
event_type VARCHAR(50) NOT NULL, -- 'impression', 'click', 'play', 'complete', 'skip'
context JSONB, -- { position, row_name, device, ... }
timestamp TIMESTAMP DEFAULT NOW()
);
-- Downloads
CREATE TABLE downloads (
download_id UUID PRIMARY KEY,
profile_id BIGINT REFERENCES user_profiles(profile_id),
title_id BIGINT REFERENCES titles(title_id),
episode_id BIGINT,
quality VARCHAR(20) NOT NULL, -- 'high', 'low'
size_bytes BIGINT,
status ENUM('pending', 'downloading', 'completed', 'failed', 'expired'),
expires_at TIMESTAMP,
drm_license_expires TIMESTAMP,
created_at TIMESTAMP DEFAULT NOW()
);
6.2 Redis Data Structures
# User Session Cache
# Key: session:{session_id}
# Type: Hash
HSET session:abc-123 user_id "100" profile_id "5" device "ios" expires "1709300000"
# Playback Progress (Real-Time)
# Key: progress:{profile_id}:{title_id}
# Type: String
SET progress:5:1001 "3600" EX 86400
# Trending Titles
# Key: trending:{country}:{timeframe}
# Type: Sorted Set (score = view count)
ZADD trending:US:24h 1500000 "title_1001"
ZADD trending:US:24h 1200000 "title_1002"
# Recommendation Cache
# Key: rec:{profile_id}:home
# Type: String (JSON)
SET rec:5:home '{"rows":[...]}' EX 300
# Rate Limiting
# Key: ratelimit:{user_id}:{endpoint}
# Type: String with TTL
SETEX ratelimit:100:search 60 "100"
# A/B Test Assignment
# Key: abtest:{test_name}:{user_id}
# Type: String
SET abtest:thumbnail_v2:100 "variant_b" EX 2592000
7. Key Design Decisions
7.1 Dedicated CDN vs Public CDN
Decision: Dedicated CDN (Open Connect Appliances) deployed at ISP points of presence.
Rationale:
- At 160 Tbps scale, public CDN costs would be prohibitive ($100M+/month)
- OCAs placed inside ISP networks reduce transit costs and improve performance
- Better cache hit ratios (>95%) due to dedicated content placement
- Full control over content replication and prefetching
- Public CDN used only as fallback for uncovered geographies
Cost Comparison:
| Approach | Cost/TB | Monthly Cost (12.8 PB/day) |
|---|---|---|
| Public CDN (AWS CloudFront) | ~$40/TB | ~$15.4M/month |
| Dedicated CDN (OCA) | ~$5/TB (amortized) | ~$1.9M/month |
| Savings | ~$13.5M/month |
7.2 Encoding Strategy: Per-Title vs Global Ladder
Decision: Per-title encoding with content-aware parameters.
Rationale:
- Action movies need higher bitrates for fast-motion scenes
- Dialogue-heavy dramas can use lower bitrates
- Scene detection optimizes keyframe placement for each title
- Content-aware encoding saves 15-25% bandwidth at equal quality
- Trade-off: More complex pipeline; longer encoding time per title
7.3 Recommendation: Two-Tower vs Sequential Models
Decision: Two-tower neural network for ranking (not transformer/sequential).
Rationale:
- Two-tower enables efficient candidate retrieval (user embedding dot product with item embeddings)
- Sequential models (Transformers) are too slow for real-time inference at 200M user scale
- Two-tower achieves 95% of sequential model accuracy at 100x inference speed
- Sequential patterns captured via engineered features (recent watch history as input)
7.4 DRM: CENC Multi-DRM vs Single DRM
Decision: CENC (Common Encryption) with three DRM systems.
Rationale:
- Single encryption pass produces content compatible with Widevine, FairPlay, and PlayReady
- Eliminates need for separate encoding per DRM system
- Industry standard; supported by all major players
- License servers are separate per DRM but content is shared
7.5 Data Storage: Polyglot Persistence
Decision: Multiple specialized databases rather than a single monolithic database.
| Data Type | Database | Rationale |
|---|---|---|
| User data, titles | PostgreSQL | ACID transactions, complex queries |
| Cache, sessions | Redis | Sub-millisecond reads, in-memory |
| Search | Elasticsearch | Full-text search, faceting |
| Analytics events | Kafka + ClickHouse | High-write throughput, analytical queries |
| Recommendation models | MLflow / S3 | Model versioning, large binary storage |
| CDN metadata | DynamoDB | Global distribution, low-latency reads |
8. Scalability Strategy
8.1 Horizontal Scaling Plan
| Component | Scaling Trigger | Min | Max | Scale Method |
|---|---|---|---|---|
| API Gateway | CPU > 60% or RPS > 100K/node | 20 | 200 | K8s HPA |
| Auth Service | Latency > 50ms | 10 | 100 | K8s HPA |
| Recommendation | Inference queue > 1000 | 20 | 200 | GPU autoscaler |
| Search | Query latency > 100ms | 5 | 50 | ES cluster scaling |
| DRM License | RPS > 10K/node | 5 | 50 | K8s HPA |
| Analytics Ingestion | Kafka lag > 5000 | 10 | 100 | Flink parallelism |
| CDN POPs | Cache miss > 10% | 2,000 | 5,000 | Physical deployment |
8.2 Content Distribution Scaling
POP Deployment Strategy:
- Tier 1 Markets (US, UK, Japan, etc.): 100+ POPs each, full top-1000 catalog
- Tier 2 Markets (Brazil, India, etc.): 50+ POPs each, top-500 catalog
- Tier 3 Markets: 10-20 POPs, top-200 catalog + on-demand fetch
Replication Pipeline:
New Title Published
→ Encode all profiles (2-4 hours)
→ Push to Origin DCs (immediate)
→ Push to RDCs (within 1 hour)
→ Push to Tier 1 POPs (within 2 hours)
→ Push to Tier 2 POPs (within 4 hours)
→ Push to Tier 3 POPs (within 8 hours)
→ Long tail available on-demand from RDC
8.3 Database Scaling
- PostgreSQL: Primary + 10 read replicas; connection pooling via PgBouncer
- Sharding: Users sharded by
user_idhash (10 shards); titles not sharded (small enough) - Redis: Cluster mode with 12 nodes (3 masters, 9 replicas); automatic failover
- Elasticsearch: 50-node cluster; 3 replicas per shard; daily index rotation
- ClickHouse: 20-node cluster for analytics; materialized views for common queries
8.4 Recommendation System Scaling
- Model Serving: TensorFlow Serving with GPU acceleration; batch inference for home page
- Embedding Cache: Redis cluster with pre-computed user embeddings (updated hourly)
- Item Embeddings: Stored in Elasticsearch alongside title metadata
- Real-Time Features: Kafka Streams for real-time feature computation
- Training Pipeline: Daily full retrain on 10B events; hourly incremental updates
9. Curveball Scenarios
Curveball 1: “A highly anticipated original series drops all 10 episodes at once. Traffic spikes 10x within the first hour. The system starts degrading. What do you do?”
Impact Assessment:
- 10x traffic spike: 400M segment requests/second at peak
- CDN POPs running out of capacity
- Recommendation service overloaded
- Database read replicas falling behind
Immediate Response:
-
CDN Layer (0-5 minutes):
- Emergency content replication to all available POPs
- Increase POP cache TTL temporarily (reduce origin fetches)
- Activate backup CDN for overflow traffic
- Enable HTTP/2 multiplexing and HTTP/3 where available
-
Application Layer (5-15 minutes):
- Scale API gateway to maximum pre-provisioned capacity
- Cache recommendation results longer (5 min → 30 min TTL)
- Simplify home page: reduce number of recommendation rows
- Enable static fallback for browse pages
-
Database Layer (15-30 minutes):
- Add emergency read replicas
- Redirect non-critical reads to cached data
- Disable analytics collection temporarily (reduce write load)
-
Client-Side:
- Push app update with simplified UI for the event
- Increase client-side caching of metadata
- Show “high demand” message to manage expectations
Prevention for Future:
- Pre-warm caches 24 hours before known drops
- Pre-scale infrastructure based on predicted demand
- Load test with 15x expected traffic before major releases
- Progressive rollout: Release episodes staggered if single-drop causes issues
Curveball 2: “A zero-day vulnerability is discovered in your DRM implementation. Content could be ripped and distributed. How do you respond?”
Impact Assessment:
- Premium content at risk of mass piracy
- Revenue impact from subscription cancellations
- Legal liability from content owners
Response Plan:
-
Immediate (0-1 hour):
- Rotate all DRM keys for affected content
- Push emergency client update with patched DRM module
- For offline downloads: invalidate existing licenses server-side
- Notify content owners per contractual obligations
-
Short-Term (1-24 hours):
- Deploy server-side watermarking as additional protection
- Monitor piracy sites for leaked content
- Issue DMCA takedown notices for discovered leaks
- Communicate transparently with subscribers
-
Medium-Term (1-7 days):
- Complete audit of DRM implementation
- Implement additional protection layers:
- Per-device key binding
- Visible and invisible watermarking
- Playback environment attestation
- Engage third-party security firm for penetration testing
-
Long-Term:
- Implement FIDO2-based device attestation
- Move to hardware-backed key storage (Secure Enclave, TEE)
- Establish bug bounty program for DRM research
Architectural Improvements:
Current: Content Key → DRM Module → Decrypted Video
Improved: Content Key + Device Key + Session Key → DRM Module + Attestation → Decrypted Video
Curveball 3: “You need to expand to a new market (e.g., India) where average bandwidth is 5 Mbps and data costs are high. How do you adapt?”
Adaptations:
-
Encoding Ladder Adjustment:
- Add more low-bitrate profiles (200 Kbps, 300 Kbps, 500 Kbps)
- Optimize H.265 encoding for better compression at low bitrates
- Add audio-only mode for data-conscious users
- Default to lower quality profile; let ABR upgrade
-
Data-Saving Features:
- Wi-Fi-only download option
- Data usage indicator in-app
- Compression mode: reduce metadata and image sizes
- Adaptive thumbnail loading (low-res first, high-res on demand)
-
Pricing and Packaging:
- Mobile-only plan with data cap
- Lower-tier subscription with SD-only streaming
- Partner with telecoms for zero-rated data
-
Content Localization:
- Local language dubbing and subtitles
- Regional content acquisition
- Local payment methods (UPI, wallets)
-
Infrastructure:
- Deploy POPs in partnership with local ISPs
- Optimize for high-latency, low-bandwidth networks
- Client-side optimizations for older devices
Bandwidth Impact:
| Profile | Original | India-Optimized | Savings |
|---|---|---|---|
| Default | 5 Mbps (1080p) | 1.5 Mbps (480p) | 70% |
| Low | 1.5 Mbps (480p) | 500 Kbps (360p) | 67% |
| Ultra-Low | — | 200 Kbps (240p) | New |
| Audio Only | — | 128 Kbps | New |
Curveball 4: “Your recommendation engine starts showing biased or inappropriate content to certain user groups. How do you detect and fix this?”
Detection:
-
Automated Monitoring:
- Diversity metrics: genre distribution, content rating distribution per demographic
- Fairness metrics: equal exposure across protected groups
- Anomaly detection: sudden shifts in recommendation patterns
-
Human Review:
- Regular audit of recommendation rows by diverse review team
- User feedback channel for reporting inappropriate recommendations
- Red team exercises to test for bias
Mitigation:
-
Algorithmic Fixes:
- Add fairness constraints to the ranking model
- Implement minimum diversity thresholds per row
- Debias training data by oversampling underrepresented groups
-
Business Rules:
- Content filtering rules for sensitive categories
- Age-appropriate filtering for child profiles
- Cultural sensitivity rules per region
-
Transparency:
- “Why am I seeing this?” explanation feature
- User controls to adjust recommendation preferences
- Regular public reporting on fairness metrics
Architecture Addition:
Ranking Model → Fairness Filter → Diversity Filter → Business Rules → Final Results
↑ ↑
Fairness Metrics Diversity Metrics
Curveball 5: “A major ISP in your largest market decides to throttle your traffic unless you pay for prioritization. How do you respond?”
Response Strategy:
-
Technical:
- Deploy more OCAs within the ISP’s network (bypass throttling point)
- Negotiate direct peering agreements
- Use protocol obfuscation (HTTPS already in place)
- Multi-ISP redundancy: route traffic through alternative ISPs
-
Business:
- Public communication about net neutrality concerns
- Legal review of throttling practices
- Customer support preparation for quality complaints
- Explore regulatory complaints if throttling is discriminatory
-
Product:
- Offer lower-quality streaming option for affected users
- Transparent communication about quality changes
- Data usage optimization to reduce ISP costs
-
Long-Term:
- Build relationships with multiple ISPs
- Invest in last-mile infrastructure partnerships
- Advocate for net neutrality regulations
10. Failure Modes and Mitigations
10.1 Failure Mode Matrix
| Component | Failure Mode | Probability | Impact | Mitigation | RTO |
|---|---|---|---|---|---|
| CDN POP | POP failure | Medium | Medium | Multi-POP failover; client retries | < 30s |
| CDN | Regional outage | Low | High | Geographic failover; backup CDN | < 2 min |
| Origin | Origin DC failure | Low | High | Multi-origin active-active; DNS failover | < 1 min |
| Encoding | Encoding failure | Medium | Low | Retry with different encoder; alert ops | < 1 hr |
| Auth | Auth service down | Low | High | Cached tokens; extended TTL; graceful degradation | < 5 min |
| DRM | License server down | Low | High | Cached licenses; offline grace period | < 5 min |
| Recommendation | Rec service slow | Medium | Medium | Cached recommendations; fallback to trending | < 30s |
| Search | ES cluster down | Low | Medium | Cached search results; degraded search | < 1 min |
| Database | Primary DB failover | Low | High | Automated failover; read replicas | < 30s |
| Client | App crash | Medium | Low | Crash recovery; resume from last position | < 5s |
| Network | ISP outage | Medium | Medium | Multi-ISP routing; CDN failover | < 1 min |
| Content | Missing segments | Low | Medium | Regenerate from source; alert ops | < 1 hr |
10.2 Cascading Failure Prevention
flowchart LR
subgraph PROTECTIONS["Protection Layers"]
P1[Client Retry<br/>Exponential Backoff]
P2[CDN Failover<br/>Multi-POP + Multi-CDN]
P3[API Rate Limiting<br/>Per-User + Global]
P4[Circuit Breakers<br/>Per-Service]
P5[Database Connection Pool<br/>With Timeout]
P6[Cache Fallback<br/>Stale Data OK]
end
P1 --> P2
P2 --> P3
P3 --> P4
P4 --> P5
P5 --> P6
10.3 Monitoring and Alerting
Key Metrics:
| Metric | Warning | Critical | Action |
|---|---|---|---|
| Rebuffer Ratio | > 1% | > 3% | Check CDN; reduce bitrate |
| Startup Time | > 5s | > 10s | Check CDN cache; origin health |
| API Latency p95 | > 200ms | > 500ms | Scale services; check DB |
| CDN Cache Hit | < 90% | < 80% | Warm caches; check replication |
| Encoding Queue | > 50 titles | > 200 titles | Scale encoders |
| DRM License Fail | > 0.1% | > 1% | Check license server |
| Search Latency | > 200ms | > 500ms | Scale ES cluster |
| Rec Latency | > 100ms | > 500ms | Cache more; scale GPU |
| Error Rate 5xx | > 0.5% | > 2% | Investigate; scale up |
| Subscriber Churn | > 2%/month | > 5%/month | Business review |
Observability Stack:
- Metrics: Prometheus + Grafana
- Logging: ELK Stack
- Tracing: Jaeger / OpenTelemetry
- Alerting: PagerDuty with severity-based escalation
- Dashboards: Real-time QoE dashboard, business metrics dashboard, infrastructure health dashboard
Appendix: Technology Stack Summary
| Layer | Technology Choices |
|---|---|
| Encoding | FFmpeg, NVIDIA NVENC, Intel QSV, x264/x265 |
| Packaging | Bento4, MP4Box, AWS MediaPackage |
| CDN | Open Connect Appliances (dedicated), CloudFront (fallback) |
| Origin | NGINX, Apache Traffic Server |
| Storage | AWS S3 (content), Redis (cache), PostgreSQL (metadata) |
| Search | Elasticsearch |
| Analytics | Apache Kafka, Apache Flink, ClickHouse |
| ML/Rec | TensorFlow, PyTorch, MLflow, Redis (embeddings) |
| DRM | Widevine, FairPlay, PlayReady (CENC) |
| Auth | Auth0 / Keycloak, JWT |
| API | GraphQL (client), REST (internal), gRPC (service-to-service) |
| Monitoring | Prometheus, Grafana, ELK, Jaeger |
| Infrastructure | Kubernetes, Terraform, Helm |
| Streaming | HLS (primary), DASH (fallback), CMAF |