Video & StreamingQuestion #20ADVANCED

Twitch: Live Video Ingest, Real-Time Chat & Clip Processing

Interactive live streaming platform orchestrating low-latency HLS/WebRTC video distribution, IRC-scale high-throughput room chat fanout, instant video clipping, and creator monetization events.

Category: Category 12: Streaming & Video Platforms•Updated: 2026-08-30
#Video#Live Social#Chat Integration#Clips#Monetization

Design a Live-Streaming Social Media App like Twitch

Scope: A user-generated live-streaming social platform enabling millions of creators to broadcast to engaged audiences with real-time chat, clip creation, subscription monetization, and community features.


Table of Contents

  1. Requirements Clarification
  2. Back-of-Envelope Calculations
  3. System Interface Design
  4. High-Level Architecture
  5. Core Component Design
  6. Data Model
  7. Key Design Decisions
  8. Scalability Strategy
  9. Curveball Scenarios
  10. Failure Modes and Mitigations

1. Requirements Clarification

1.1 Functional Requirements

  • User-Generated Live Streaming: Any authenticated user can start a live stream from desktop, mobile, or broadcasting software (OBS, Streamlabs)
  • Real-Time Chat: Per-stream chat with messages, emotes, badges, and moderation tools
  • Viewer Scaling: Support streams from 0 to 100K+ concurrent viewers
  • Clip Creation: Viewers and streamers can create short clips (5-60 seconds) from live or past broadcasts
  • Subscription / Monetization: Tiered subscriptions (Tier 1/2/3), bits/cheers, donations, ads, and channel points
  • Low-Latency Mode: Optional sub-2-second latency mode for interactive streams
  • Video Quality Options: 160p to 1080p60, including source quality for premium subscribers
  • Categories and Tags: Organize streams by game, category, and custom tags
  • Followers and Notifications: Follow channels; get notified when followed streamers go live
  • Raids and Hosts: Streamers can raid (redirect viewers) to other channels
  • VOD / Past Broadcasts: Automatic archival of streams for later viewing
  • Moderation Tools: AutoMod, banned words, slow mode, subscriber-only mode, VIP system
  • Extensions and Overlays: Browser-based interactive overlays and stream extensions
  • Multi-Platform Clients: Web, desktop (Windows/Mac), mobile (iOS/Android), and console apps
  • Creator Dashboard: Analytics, schedule management, revenue reports, and community tools
  • Team / Organization Support: Multi-channel organizations with shared resources

1.2 Non-Functional Requirements

Requirement Target Rationale
Live Latency (Standard) 3-10 seconds Balance between latency and reliability
Live Latency (Low-Latency Mode) < 2 seconds For interactive gaming and betting
Chat Latency < 500ms end-to-end Real-time conversation feel
Availability 99.9% uptime Downtime loses viewers and creator trust
Concurrent Streamers 50K+ simultaneous streams Platform scale
Concurrent Viewers 20M+ peak Major events and popular streamers
Chat Throughput 100K messages/second per top stream Hype trains and raids
Stream Start Time < 3 seconds from click to first frame Viewer retention
Clip Creation Latency < 5 seconds from trigger to shareable clip Capitalize on moment
Scalability Horizontal scale for all components User base grows organically
Data Consistency Eventual for chat; strong for billing Chat can lose messages; billing must be exact

2. Back-of-Envelope Calculations

2.1 Assumption Sets

Parameter Conservative Current Scale Peak Event
Concurrent Streamers 5K 50K 100K
Concurrent Viewers 2M 20M 50M
Avg Viewers per Stream 400 400 500
Avg Stream Duration 4 hours 6 hours 8 hours
Avg Bitrate 2.5 Mbps 3.5 Mbps 4 Mbps
Chat Messages per Viewer per Minute 0.5 1 3
Clip Creation Rate 1K/hour 10K/hour 50K/hour
Subscription Holders 5M 30M —

2.2 Bandwidth Estimates

Outbound CDN Bandwidth:

Scenario Calculation Total Bandwidth
Conservative 2M × 2.5 Mbps 5 Tbps
Current 20M × 3.5 Mbps 70 Tbps
Peak Event 50M × 4 Mbps 200 Tbps

Ingest Bandwidth (per streamer, assuming 1080p60 at 6 Mbps):

Scenario Calculation Total Ingest
Conservative 5K × 6 Mbps 30 Gbps
Current 50K × 6 Mbps 300 Gbps
Peak Event 100K × 6 Mbps 600 Gbps

2.3 Storage Estimates

VOD Archive (daily):

Per Stream (6 hours avg @ 3.5 Mbps):
├── Source: 6 hrs × 3.5 Mbps × 3600 = 9.45 GB
├── Transcoded (6 profiles): 9.45 GB × 6 = 56.7 GB
└── Total per Stream: ~66 GB

Daily Streams (50K streamers × 1 stream/day):
├── Daily Storage: 50K × 66 GB = 3.3 PB/day
└── Monthly Storage: ~100 PB/month

Retention Policy:
├── 14 days (free streamers): 100 PB × 14/30 = 47 PB
├── 60 days (affiliates): 40% of streams × 60/30 = 80 PB
├── 365 days (partners): 10% of streams × 365/30 = 233 PB
└── Total Active Storage: ~360 PB

Clip Storage:

  • Per clip: 30 seconds × 3.5 Mbps × 6 profiles = ~45 MB
  • 0MB
  • 10K clips/hour × 24 hours = 240K clips/day
  • Daily clip storage: 240K × 45 MB = 10.8 TB/day
  • Monthly: ~324 TB/month

Chat Message Storage:

  • Per message: ~200 bytes
  • 20M viewers × 1 msg/min × 60 min × 6 hours = 7.2B messages/day
  • Daily: 7.2B × 200 B = 1.44 TB/day
  • Retention: 30 days = 43 TB

2.4 QPS Estimates

Operation Off-Peak Peak Major Event
Stream Start Requests 50K/s 200K/s 2M/s
Segment Requests 5M/s 20M/s 200M/s
Chat Messages (total) 500K/s 5M/s 50M/s
Chat Messages (top stream) 5K/s 50K/s 500K/s
API Calls (browse, follow) 100K/s 500K/s 5M/s
Clip Creation 1/s 10/s 50/s
Subscription Purchases 10/s 100/s 500/s
Notification Delivery 100K/s 1M/s 10M/s
Auth Token Requests 20K/s 100K/s 1M/s

2.5 Transcoding Compute

  • Per stream: 6 profiles × 2 cores (NVENC) = 12 cores
  • 50K concurrent streams: 600K GPU-accelerated cores
  • Peak (100K streams): 1.2M cores

3. System Interface Design

3.1 Public APIs

# --- Stream Management APIs ---

POST /api/v2/streams
  Start a new live stream
  Headers: Authorization: Bearer <token>
  Request: {
    user_id, game_id, title, language, is_mature: false,
    latency_mode: "standard" | "low" | "ultra-low",
    tags: ["english", "competitive"]
  }
  Response: { stream_id, ingest_url, stream_key, goes_live_at }

DELETE /api/v2/streams/{id}
  End a live stream

GET /api/v2/streams
  List live streams
  Query: ?game=1&language=en&first=20&after=cursor
  Response: {
    data: [{
      id, user_id, user_login, user_name, game_id, game_name,
      title, viewer_count, started_at, language, is_mature,
      thumbnail_url, tag_ids, latency_mode
    }],
    pagination: { cursor, has_more }
  }

GET /api/v2/streams/{id}
  Get stream details
  Response: { id, user_id, game_id, title, viewer_count, started_at, ... }

GET /api/v2/streams/{id}/manifest
  Get streaming manifest
  Query: ?format=hls&quality=source
  Response: HLS manifest

# --- Chat APIs (WebSocket) ---

WS /ws/chat
  WebSocket connection for real-time chat
  Messages:
  → JOIN: { channel_id, auth_token }
  ← CHAT_MESSAGE: { id, user_id, username, message, badges, color, timestamp }
  ← SYSTEM_MSG: { type: "sub" | "raid" | "bits" | "host", data: {...} }
  ← ROOM_STATE: { slow_mode: 30, sub_only: false, followers_only: 10 }
  → SEND_MESSAGE: { channel_id, message }
  → EMOTE: { channel_id, emote_id }

# --- Clip APIs ---

POST /api/v2/clips
  Create a clip from a live or past broadcast
  Request: {
    broadcast_id, broadcaster_id,
    has_delay: false,
    duration_seconds: 30,
    game_id, language
  }
  Response: { clip_id, url, embed_url, created_at, duration }

GET /api/v2/clips
  List clips
  Query: ?broadcaster_id=123&first=20
  Response: { data: [{ id, url, title, viewer_count, created_at, duration }] }

GET /api/v2/clips/{id}
  Get clip details and manifest

# --- User / Channel APIs ---

GET /api/v2/users
  Get user information
  Query: ?login=ninja&first=1
  Response: { data: [{ id, login, display_name, description, profile_image_url, offline_image_url, type, broadcaster_type, created_at }] }

GET /api/v2/users/{id}/follows
  Get users followed by this user
  Query: ?first=20&after=cursor

PUT /api/v2/users/follows
  Follow a user
  Request: { from_id, to_id }

GET /api/v2/channels/{id}
  Get channel information
  Response: { id, user_id, game_id, title, delay_seconds, language, ... }

# --- Subscription / Monetization APIs ---

GET /api/v2/subscriptions
  Get subscription information
  Headers: Authorization: Bearer <token>
  Query: ?broadcaster_id=123&user_id=456
  Response: { data: [{ user_id, user_name, broadcaster_id, tier: "1000", is_gift: false, tenor: "monthly" }] }

POST /api/v2/subscriptions
  Subscribe to a channel
  Request: { broadcaster_id, tier: "1000", gift: false }
  Response: { subscription_id, tier, starts_at, ends_at }

POST /api/v2/bits/cheer
  Cheer (send bits) in a channel
  Request: { broadcaster_id, bits: 100 }
  Response: { transaction_id, bits, message }

POST /api/v2/ads/schedule
  Schedule a commercial break
  Request: { length_seconds: 30, delay_seconds: 0 }
  Response: { ad_session_id, starts_at, length_seconds }

# --- Analytics APIs ---

GET /api/v2/analytics/streams
  Get stream analytics
  Query: ?started_at=...&ended_at=...&first=100
  Response: { data: [{ period, average_viewers, peak_viewers, total_viewers, estimated_minutes_watched }] }

GET /api/v2/analytics/entitlements
  Get subscription revenue analytics
  Response: { data: [{ period, subscriptions, revenue, churn }] }

# --- Moderation APIs ---

POST /api/v2/moderation/bans
  Ban a user from a channel
  Request: { broadcaster_id, user_id, duration_seconds, reason }

POST /api/v2/moderation/blocks
  Block a user (they can't chat with you)

GET /api/v2/moderation/chat/clear
  Clear all chat messages

PUT /api/v2/moderation/chat/slow
  Set slow mode
  Request: { broadcaster_id, seconds: 30 }

3.2 Internal Service APIs (gRPC)

// Ingest Service
service IngestService {
  rpc CreateIngest(IngestRequest) returns (IngestResponse);
  rpc HealthCheck(HealthRequest) returns (IngestHealth);
  rpc Failover(FailoverRequest) returns (Empty);
}

// Chat Service
service ChatService {
  rpc SendMessage(ChatMessage) returns (MessageAck);
  rpc GetRoomState(RoomId) returns (RoomState);
  rpc GetRecentMessages(RecentRequest) returns (stream ChatMessage);
  rpc ModerateMessage(ModerationRequest) returns (ModerationResult);
}

// Clip Service
service ClipService {
  rpc CreateClip(ClipRequest) returns (ClipResponse);
  rpc GetClipStatus(ClipId) returns (ClipStatus);
  rpc DeleteClip(ClipId) returns (Empty);
}

// Notification Service
service NotificationService {
  rpc SendNotification(NotificationRequest) returns (Empty);
  rpc SubscribeToChannel(SubscriptionRequest) returns (Empty);
  rpc Unsubscribe(UnsubscriptionRequest) returns (Empty);
}

// Monetization Service
service MonetizationService {
  rpc ProcessSubscription(SubscriptionRequest) returns (SubscriptionResponse);
  rpc ProcessBits(BitsRequest) returns (BitsResponse);
  rpc ProcessAdRevenue(AdRevenueRequest) returns (RevenueResponse);
  rpc GetPayout(PayoutRequest) returns (PayoutResponse);
}

4. High-Level Architecture

flowchart TB
    subgraph STREAMERS["🎮 Streamers"]
        OBS[OBS / Streamlabs<br/>RTMP Ingest]
        MOBILE[Mobile App<br/>RTMP / SRT]
        CONSOLE[Console App<br/>RTMP]
        INGEST_CLI[Ingest Gateway<br/>RTMP Server]
    end

    subgraph PROCESSING["⚙️ Processing Layer"]
        TC[Transcoding Cluster<br/>GPU-Accelerated]
        QC[Quality Monitor<br/>Health + Stats]
        CLIP_ENGINE[Clip Engine<br/>Segment Capture]
        VOD_ARCHIVE[VOD Archiver<br/>Stream Recording]
    end

    subgraph DISTRIBUTION["🌐 Distribution Layer"]
        ORIGIN[Origin Server]
        CDN[CDN Edge POPs]
        LL_ORIGIN[Low-Latency Origin<br/>WebRTC SFU]
    end

    subgraph CHAT["💬 Chat System"]
        CHAT_SERVER[Chat Server Cluster<br/>WebSocket]
        AUTOMOD[AutoMod Service<br/>ML Moderation]
        CHAT_DB[Chat Storage<br/>Time-Series DB]
        EMOTE[Emote Service<br/>Custom Emotes]
    end

    subgraph SERVICES["🔧 Platform Services"]
        AUTH[Auth Service<br/>OAuth 2.0]
        USER[User Service<br/>Profiles + Follows]
        NOTIF[Notification Service<br/>Push + Email]
        MONETIZE[Monetization Service<br/>Subs + Bits + Ads]
        ANALYTICS[Analytics Pipeline<br/>Kafka + Flink]
        SEARCH[Search Service<br/>Discover Streams]
        REC[Recommendation Service<br/>Stream Discovery]
    end

    subgraph VIEWERS["📱 Viewer Clients"]
        WEB[Web Browser]
        DESKTOP[Desktop App]
        MOBILE_V[iOS / Android]
        TV[Smart TV]
    end

    OBS --> INGEST_CLI
    MOBILE --> INGEST_CLI
    CONSOLE --> INGEST_CLI
    INGEST_CLI --> TC
    TC --> QC
    TC --> ORIGIN
    TC --> VOD_ARCHIVE
    QC --> CLIP_ENGINE
    CLIP_ENGINE --> ORIGIN
    ORIGIN --> CDN
    CDN --> VIEWERS
    LL_ORIGIN --> VIEWERS
    VIEWERS --> CHAT_SERVER
    CHAT_SERVER --> AUTOMOD
    CHAT_SERVER --> CHAT_DB
    CHAT_SERVER --> EMOTE
    VIEWERS --> AUTH
    VIEWERS --> USER
    VIEWERS --> NOTIF
    VIEWERS --> MONETIZE
    VIEWERS --> SEARCH
    VIEWERS --> REC
    ANALYTICS --> REC
    ANALYTICS --> NOTIF

5. Core Component Design

5.1 User-Generated Ingest System

Purpose: Accept live streams from millions of users using various broadcasting software and devices.

Ingest Architecture:

flowchart TB
    subgraph SOURCES["Stream Sources"]
        OBS[OBS Studio]
        STREAMLABS[Streamlabs]
        MOBILE[Mobile App]
        CONSOLE[Console]
        REMOTE[Remote Producer]
    end
    
    subgraph INGEST["Ingest Infrastructure"]
        subgraph RTMP["RTMP Servers (Primary)"]
            R1[RTMP Server 1]
            R2[RTMP Server 2]
            R3[RTMP Server N]
        end
        subgraph SRT["SRT Servers (Backup)"]
            S1[SRT Server 1]
            S2[SRT Server 2]
        end
    end
    
    subgraph LOAD_BALANCE["Ingest Load Balancer"]
        LB[Global Load Balancer<br/>Geo-Aware + Health-Based]
    end
    
    SOURCES --> LB
    LB --> RTMP
    LB --> SRT
    RTMP --> PROCESS[Processing Pipeline]
    SRT --> PROCESS

Key Design Decisions:

  • RTMP as Primary Protocol: Most broadcasting software supports RTMP; simple, widely adopted
  • Multiple Ingest Endpoints: Each streamer gets a unique ingest URL with their stream key
  • Geo-Distributed Ingest: Ingest servers in 20+ regions; streamers connect to nearest
  • Stream Key Security: One-time stream keys; rotated on each new stream
  • Health Monitoring: Continuous monitoring of ingest health (bitrate, resolution, frame rate, keyframe interval)

Ingest Validation:

On Stream Start:
1. Validate stream key (active, not expired)
2. Check user entitlements (can they stream?)
3. Verify stream parameters (resolution, bitrate within limits)
4. Assign transcoding resources
5. Initialize chat room
6. Update stream directory
7. Notify followers

On Stream Health Check (every 5 seconds):
1. Verify bitrate within expected range
2. Check for audio/video sync
3. Monitor packet loss
4. If unhealthy: warn streamer → auto-reconnect → failover

5.2 Real-Time Chat System

Purpose: Handle millions of concurrent chat connections with sub-500ms latency and robust moderation.

Architecture:

flowchart TB
    subgraph CLIENTS["Chat Clients"]
        C1[Viewer 1]
        C2[Viewer 2]
        C3[Viewer N]
        MOD[Moderator]
        BROADCASTER[Streamer]
    end
    
    subgraph CHAT_CLUSTER["Chat Server Cluster"]
        subgraph SHARD1["Shard 1<br/>Channels A-M"]
            CS1[Chat Server 1]
            CS2[Chat Server 2]
        end
        subgraph SHARD2["Shard 2<br/>Channels N-Z"]
            CS3[Chat Server 3]
            CS4[Chat Server 4]
        end
    end
    
    subgraph SUPPORT["Support Services"]
        AUTOMOD[AutoMod<br/>ML-Based Filtering]
        RATE_LIMIT[Rate Limiter<br/>Per-User + Per-Room]
        EMOTE[Emote Resolver<br/>Global + Channel]
        BADGE[Badge Service<br/>Sub, Mod, VIP]
    end
    
    subgraph STORAGE["Chat Storage"]
        TSDB[Time-Series DB<br/>Recent Messages]
        ARCHIVE[Chat Archive<br/>Long-Term Storage]
    end
    
    C1 --> CS1
    C2 --> CS1
    C3 --> CS3
    MOD --> CS1
    BROADCASTER --> CS1
    CS1 --> AUTOMOD
    CS1 --> RATE_LIMIT
    CS1 --> EMOTE
    CS1 --> BADGE
    CS1 --> TSDB
    TSDB --> ARCHIVE

Design Details:

  • Sharding by Channel: Each channel’s chat is handled by a dedicated shard; scales horizontally
  • WebSocket Connections: Persistent connections for real-time bidirectional communication
  • Message Ordering: Sequence numbers ensure total order within a channel
  • Reconnection Handling: Clients can reconnect and catch up with missed messages
  • Rate Limiting: Per-user rate limits (configurable per channel); slow mode enforced server-side

Chat Message Flow:

Client → WebSocket → Chat Server → Rate Limiter → AutoMod → Badge/Emote Enrichment
    → Broadcast to Connected Clients → Time-Series DB → Archive

Scale:

  • Top stream: 100K concurrent chat connections, 50K messages/second
  • Average stream: 100 connections, 5 messages/second
  • Total chat connections: 20M+ concurrent
  • Chat server capacity: ~50K connections per node

AutoMod Design:

Message → Tokenization → ML Classification → Decision
                                    ↓
                    Spam / Hate / Self-Harm / Sexual / Other
                                    ↓
                    Block / Hold for Review / Allow
  • ML Model: Fine-tuned transformer model for toxic content detection
  • Latency: < 50ms per message
  • False Positive Handling: Moderators can release held messages; model learns from corrections
  • Custom Rules: Per-channel banned words, permitted terms, and regex patterns

5.3 Clip Creation Engine

Purpose: Allow viewers and streamers to create shareable short clips from live or past broadcasts.

Architecture:

flowchart TB
    subgraph TRIGGERS["Clip Triggers"]
        VIEWER[Viewer Clicks<br/>\"Clip That!\" Button]
        AUTO[Auto-Clip<br/>ML-Detected Moments]
        STREAMER[Streamer<br/>Manual Clip]
    end
    
    subgraph CAPTURE["Capture Engine"]
        BUFFER[Ring Buffer<br/>60s Rolling Buffer]
        SEGMENT[Segment Selector<br/>Start + End Points]
    end
    
    subgraph PROCESSING["Clip Processing"]
        EXTRACT[Segment Extraction<br/>from VOD Storage]
        TRANSCODE[Clip Transcoding<br/>Multiple Profiles]
        THUMB[Thumbnail Generation<br/>Auto-Select Best Frame]
        WATERMARK[Watermark<br/>Channel Branding]
    end
    
    subgraph OUTPUT["Clip Output"]
        CDN_CLIP[CDN Distribution]
        SHARE[Share URLs<br/>Social Media Embeds]
        FEED[Clip Feed<br/>Channel + Global]
    end
    
    VIEWER --> CAPTURE
    AUTO --> CAPTURE
    STREAMER --> CAPTURE
    CAPTURE --> PROCESSING
    PROCESSING --> OUTPUT
    OUTPUT --> CDN_CLIP
    OUTPUT --> SHARE
    OUTPUT --> FEED

Design Details:

  • Ring Buffer: Each live stream maintains a 60-second rolling buffer of encoded segments
  • Instant Clip: When triggered, the system marks the start and end points in the buffer
  • Async Processing: Clip transcoding happens asynchronously; viewer gets a preview immediately
  • Multi-Profile: Clips are transcoded into multiple profiles for adaptive streaming
  • Thumbnail Auto-Select: Computer vision selects the most visually interesting frame

Clip Processing Pipeline:

Clip Request (duration=30s, offset=-15s from now)
    → Locate segments in ring buffer / VOD storage
    → Extract raw segments
    → Transcode to 4 profiles (1080p, 720p, 480p, 360p)
    → Generate thumbnail (CV-based frame selection)
    → Apply watermark (channel logo)
    → Upload to CDN
    → Generate share URL and embed code
    → Notify creator
    → Add to clip feed

Processing Time: 3-5 seconds for a 30-second clip

5.4 Subscription and Monetization System

Purpose: Enable creators to monetize their content through subscriptions, bits, ads, and donations.

Monetization Tiers:

Revenue Stream Description Platform Cut Creator Cut
Subscriptions Monthly recurring (Tier 1/2/3) 50% (standard) 50% (70% for top partners)
Bits / Cheers One-time virtual currency tips 1% processing 99% (after processing)
Ads Pre-roll, mid-roll, banner ads Varies by deal Varies by deal
Gift Subs Users gift subs to others 50% 50%
Channel Points Custom rewards (non-monetary) N/A N/A

Subscription Architecture:

flowchart TB
    subgraph PURCHASE["Purchase Flow"]
        USER[User Initiates<br/>Subscription]
        PAYMENT[Payment Processor<br/>Stripe / PayPal]
        ENTITLEMENT[Entitlement Service<br/>Grant Benefits]
        NOTIF[Notification<br/>Thank You + Benefits]
    end
    
    subgraph BENEFITS["Subscriber Benefits"]
        NO_ADS[Ad-Free Viewing]
        EMOTES[Subscriber-Only Emotes]
        BADGE[Subscriber Badge in Chat]
        SUB_CHAT[Subscriber-Only Chat Mode]
        HQ[Higher Quality Streaming]
    end
    
    subgraph RECURRING["Recurring Management"]
        RENEW[Auto-Renewal<br/>Monthly Billing]
        CHURN[Churn Prevention<br/>Reminders + Offers]
        CANCEL[Cancellation Flow<br/>Retention Offers]
    end
    
    USER --> PAYMENT
    PAYMENT --> ENTITLEMENT
    ENTITLEMENT --> NOTIF
    ENTITLEMENT --> BENEFITS
    ENTITLEMENT --> RENEW
    RENEW --> CHURN
    RENEW --> CANCEL

Key Design Decisions:

  • Entitlement Service: Centralized service managing all subscriber benefits across the platform
  • Real-Time Benefits: Subscriber status checked on every chat message and stream access
  • Revenue Split: Configurable per creator tier; automated payout processing
  • Fraud Prevention: Payment fraud detection, sub account abuse detection

5.5 Low-Latency Streaming Mode

Purpose: Provide sub-2-second latency for interactive streams (gaming, live events, betting).

Architecture Comparison:

flowchart LR
    subgraph STANDARD["Standard Mode<br/>3-10s Latency"]
        S1[RTMP Ingest]
        S2[HLS Transcoding]
        S3[CDN Distribution]
        S4[HTTP Client]
    end
    
    subgraph LOW_LATENCY["Low-Latency Mode<br/>< 2s Latency"]
        L1[SRT / WebRTC Ingest]
        L2[WebRTC SFU<br/>Selective Forwarding]
        L3[Edge Compute<br/>Near Viewer]
        L4[WebRTC Client]
    end
    
    S1 --> S2 --> S3 --> S4
    L1 --> L2 --> L3 --> L4

Low-Latency Mode Design:

  • Protocol: WebRTC for both ingest and delivery
  • SFU Architecture: Selective Forwarding Unit forwards streams without re-encoding
  • Edge Deployment: SFUs deployed at edge locations for minimal network hop count
  • Adaptive Quality: Client-side bitrate adaptation via WebRTC bandwidth estimation
  • Trade-offs:
    • Lower reliability than HLS (no segment caching)
    • Higher bandwidth cost (no CDN caching benefit)
    • Limited to WebRTC-capable clients
    • Maximum ~10K concurrent viewers per SFU instance

Latency Breakdown:

Component Standard Mode Low-Latency Mode
Ingest 1-2s (RTMP) 200-500ms (SRT/WebRTC)
Processing 1-2s (transcode) 100-200ms (SFU forward)
Distribution 1-3s (CDN) 200-500ms (edge)
Client Buffer 1-3s 200-500ms
Total 4-10s 0.7-1.7s

5.6 Discovery and Recommendation

Purpose: Help viewers find streams and help streamers find audiences.

Discovery Features:

  1. Browse Directory: Filter by game, category, language, tags
  2. Recommended Channels: Personalized stream recommendations
  3. Followed Streamers: Quick access to followed channels’ live status
  4. Trending: Most-watched streams globally and per region
  5. Search: Full-text search across stream titles, channel names, and categories

Recommendation Algorithm:

Score = w1 × Relevance_Score + w2 × Viewer_Count_Boost 
      + w3 × Freshness_Boost + w4 × Follow_Signal
      + w5 × Category_Affinity + w6 × Language_Match

Relevance_Score = collaborative_filtering(user, stream)
Category_Affinity = user's historical watch time in category / total watch time

Real-Time Updates:

  • Stream directory updated every 30 seconds
  • Viewer counts updated every 10 seconds
  • Live status propagated within 5 seconds of stream start

6. Data Model

6.1 Core Entities

-- Users
CREATE TABLE users (
    user_id           BIGINT PRIMARY KEY,
    login             VARCHAR(25) UNIQUE NOT NULL,
    display_name      VARCHAR(100) NOT NULL,
    email             VARCHAR(255) UNIQUE,
    password_hash     VARCHAR(255),
    description       TEXT,
    profile_image_url VARCHAR(500),
    offline_image_url VARCHAR(500),
    type              ENUM('user', 'affiliate', 'partner') DEFAULT 'user',
    created_at        TIMESTAMP DEFAULT NOW(),
    updated_at        TIMESTAMP DEFAULT NOW()
);

-- Channels
CREATE TABLE channels (
    channel_id        BIGINT PRIMARY KEY,
    user_id           BIGINT REFERENCES users(user_id) UNIQUE,
    game_id           BIGINT,
    title             VARCHAR(140) NOT NULL,
    language          VARCHAR(10) DEFAULT 'en',
    delay_seconds     INT DEFAULT 0,
    is_mature         BOOLEAN DEFAULT false,
    can_edit_delay    BOOLEAN DEFAULT false,
    created_at        TIMESTAMP DEFAULT NOW(),
    updated_at        TIMESTAMP DEFAULT NOW()
);

-- Streams (Active and Historical)
CREATE TABLE streams (
    stream_id         BIGINT PRIMARY KEY,
    user_id           BIGINT REFERENCES users(user_id),
    game_id           BIGINT,
    title             VARCHAR(140) NOT NULL,
    viewer_count      INT DEFAULT 0,
    started_at        TIMESTAMP NOT NULL,
    ended_at          TIMESTAMP,
    is_live           BOOLEAN DEFAULT true,
    language          VARCHAR(10) DEFAULT 'en',
    is_mature         BOOLEAN DEFAULT false,
    latency_mode      ENUM('standard', 'low', 'ultra-low') DEFAULT 'standard',
    tag_ids           UUID[],
    thumbnail_url     VARCHAR(500),
    preview_url       VARCHAR(500),
    avg_fps           DECIMAL(5,2),
    peak_viewers      INT DEFAULT 0,
    created_at        TIMESTAMP DEFAULT NOW()
);

-- Games and Categories
CREATE TABLE games (
    game_id           BIGINT PRIMARY KEY,
    name              VARCHAR(100) NOT NULL,
    box_art_url       VARCHAR(500),
    viewer_count      INT DEFAULT 0,
    created_at        TIMESTAMP DEFAULT NOW()
);

-- Follows
CREATE TABLE follows (
    follow_id         BIGINT PRIMARY KEY,
    from_user_id      BIGINT REFERENCES users(user_id),
    to_user_id        BIGINT REFERENCES users(user_id),
    followed_at       TIMESTAMP DEFAULT NOW(),
    UNIQUE(from_user_id, to_user_id)
);

-- Subscriptions
CREATE TABLE subscriptions (
    subscription_id   BIGINT PRIMARY KEY,
    user_id           BIGINT REFERENCES users(user_id),
    broadcaster_id    BIGINT REFERENCES users(user_id),
    tier              VARCHAR(10) NOT NULL,  -- '1000', '2000', '3000'
    is_gift           BOOLEAN DEFAULT false,
    gift_from_id      BIGINT,
    tenor             ENUM('monthly', 'yearly') DEFAULT 'monthly',
    starts_at         TIMESTAMP DEFAULT NOW(),
    ends_at           TIMESTAMP,
    cancel_at_period_end BOOLEAN DEFAULT false,
    created_at        TIMESTAMP DEFAULT NOW()
);

-- Bits / Cheers Transactions
CREATE TABLE bits_transactions (
    transaction_id    BIGINT PRIMARY KEY,
    user_id           BIGINT REFERENCES users(user_id),
    broadcaster_id    BIGINT REFERENCES users(user_id),
    stream_id         BIGINT REFERENCES streams(stream_id),
    bits              INT NOT NULL,
    message           VARCHAR(500),
    created_at        TIMESTAMP DEFAULT NOW()
);

-- Clips
CREATE TABLE clips (
    clip_id           UUID PRIMARY KEY,
    broadcast_id      BIGINT REFERENCES streams(stream_id),
    broadcaster_id    BIGINT REFERENCES users(user_id),
    viewer_id         BIGINT REFERENCES users(user_id),  -- Who created the clip
    title             VARCHAR(50) DEFAULT '',
    duration_seconds  DECIMAL(5,2) NOT NULL,
    video_url         VARCHAR(500) NOT NULL,
    embed_url         VARCHAR(500) NOT NULL,
    thumbnail_url     VARCHAR(500),
    view_count        INT DEFAULT 0,
    game_id           BIGINT,
    language          VARCHAR(10),
    created_at        TIMESTAMP DEFAULT NOW()
);

-- Chat Messages (Time-Series)
CREATE TABLE chat_messages (
    message_id        BIGINT PRIMARY KEY,
    channel_id        BIGINT NOT NULL,
    user_id           BIGINT NOT NULL,
    message           TEXT NOT NULL,
    message_type      ENUM('text', 'emote', 'system', 'cheer', 'sub') DEFAULT 'text',
    badges            JSONB,
    color             VARCHAR(7),
    is_action         BOOLEAN DEFAULT false,
    moderation_status ENUM('allowed', 'blocked', 'held') DEFAULT 'allowed',
    created_at        TIMESTAMP DEFAULT NOW(),
    INDEX idx_channel_time (channel_id, created_at DESC)
);

-- Emotes
CREATE TABLE emotes (
    emote_id          BIGINT PRIMARY KEY,
    name              VARCHAR(50) NOT NULL,
    image_url_1x      VARCHAR(500) NOT NULL,
    image_url_2x      VARCHAR(500),
    image_url_4x      VARCHAR(500),
    type              ENUM('global', 'channel', 'sub', 'prime') NOT NULL,
    owner_id          BIGINT,  -- NULL for global emotes
    channel_id        BIGINT,  -- NULL for global emotes
    created_at        TIMESTAMP DEFAULT NOW()
);

-- Channel Points Rewards
CREATE TABLE channel_rewards (
    reward_id         UUID PRIMARY KEY,
    channel_id        BIGINT NOT NULL,
    title             VARCHAR(50) NOT NULL,
    cost              INT NOT NULL,
    prompt            VARCHAR(200),
    is_enabled        BOOLEAN DEFAULT true,
    is_user_input_required BOOLEAN DEFAULT false,
    max_per_stream    INT,
    global_cooldown   INT,  -- seconds
    background_color  VARCHAR(7),
    created_at        TIMESTAMP DEFAULT NOW()
);

-- Analytics Events
CREATE TABLE analytics_events (
    event_id          BIGINT PRIMARY KEY,
    stream_id         BIGINT,
    user_id           BIGINT,
    event_type        VARCHAR(50) NOT NULL,
    event_data        JSONB,
    timestamp         TIMESTAMP DEFAULT NOW()
);

6.2 Redis Data Structures

# Live Stream Status
# Key: stream:live:{user_id}
# Type: Hash
HSET stream:live:1001 stream_id "5001" viewer_count "50000" started_at "1709280000" game_id "21218" title "Live!"

# Chat Room State
# Key: chat:room:{channel_id}
# Type: Hash
HSET chat:room:1001 slow_mode "30" sub_only "false" followers_only "0" unique_chat "false"

# Chat Rate Limiting
# Key: chat:rate:{channel_id}:{user_id}
# Type: String with TTL
SETEX chat:rate:1001:500 30 "5"

# Viewer Count (Real-Time)
# Key: viewers:{stream_id}
# Type: String (atomic counter)
INCR viewers:5001

# Follow Count
# Key:
# Key: follows:count:{user_id}
# Type: String
INCR follows:count:1001

# Trending Streams
# Key: trending:global
# Type: Sorted Set (score = viewer_count)
ZADD trending:global 50000 "stream_5001"
ZADD trending:global 30000 "stream_5002"

# Stream Directory (by game)
# Key: directory:game:{game_id}
# Type: Sorted Set (score = viewer_count)
ZADD directory:game:21218 50000 "stream_5001"

# Notification Queue
# Key: notif:queue:{user_id}
# Type: List
RPUSH notif:queue:1001 '{"type":"followed_live","channel_id":2001}'

7. Key Design Decisions

7.1 Streaming Protocol: RTMP + HLS vs WebRTC-Only

Decision: RTMP ingest + HLS delivery as standard; WebRTC as optional low-latency mode.

Rationale:

  • RTMP is universally supported by broadcasting software (OBS, Streamlabs, etc.)
  • HLS provides reliable, cacheable delivery at scale
  • WebRTC added as optional mode for latency-sensitive use cases
  • Hybrid approach balances compatibility, reliability, and latency

Protocol Matrix:

Mode Ingest Delivery Latency Max Viewers Use Case
Standard RTMP HLS 3-10s Unlimited Default for all streams
Low-Latency SRT LL-HLS 1-3s ~100K Interactive streams
Ultra-Low WebRTC WebRTC < 1s ~10K Betting, live interaction

7.2 Chat Architecture: WebSocket vs Server-Sent Events

Decision: WebSocket for bidirectional chat.

Rationale:

  • Chat requires bidirectional communication (send messages + receive messages)
  • WebSocket provides full-duplex communication over a single connection
  • Server-Sent Events (SSE) is unidirectional (server-to-client only)
  • WebSocket connection overhead is acceptable for the interactivity required

Connection Management:

Per Chat Server Node:
├── Max Connections: 50,000
├── Memory per Connection: ~50 KB
├── Total Memory: 50K × 50 KB = 2.5 GB
└── CPU per Connection: ~0.1% at idle, ~1% at peak chat

For 20M concurrent viewers:
├── Chat Servers Needed: 20M / 50K = 400 nodes
└── Total Infrastructure: ~400 chat server nodes

7.3 Clip Storage: Separate from VOD vs Shared

Decision: Clips stored separately from VOD with their own CDN distribution.

Rationale:

  • Clips have different access patterns (viral, bursty) vs VOD (gradual, long-tail)
  • Separate storage allows independent lifecycle management
  • Clips can be optimized for social sharing (shorter manifests, embed-friendly)
  • VOD retention policies don’t apply to clips (clips persist after VOD expires)

7.4 Monetization: Platform vs Third-Party

Decision: Platform-native monetization with integrated payment processing.

Rationale:

  • Integrated payments provide better user experience (no redirect)
  • Platform can enforce revenue splits automatically
  • Better fraud detection and chargeback handling
  • Third-party donations (PayPal, Patreon) allowed as supplements

7.5 Moderation: AI-First vs Human-First

Decision: AI-first (AutoMod) with human oversight.

Rationale:

  • At scale (50M messages/second peak), human moderation is impossible
  • ML models can process messages in < 50ms
  • Human moderators review held messages and train the model
  • Per-channel custom rules allow streamer-specific moderation

8. Scalability Strategy

8.1 Horizontal Scaling Plan

Component Scaling Trigger Min Max Scale Method
Ingest Servers CPU > 70% or streams > 100/node 50 500 K8s HPA
Transcoders Queue depth > 50 streams 200 5000 GPU pool autoscaler
Chat Servers Connections > 40K/node 100 1000 K8s HPA
Origin Servers RPS > 50K/node 50 500 K8s HPA
AutoMod Message queue > 10K 10 100 K8s HPA
Clip Engine Processing queue > 100 5 50 K8s HPA
Auth Service Latency > 100ms 10 100 K8s HPA

8.2 Chat Scaling Strategy

Sharding Strategy:

Channel IDs are hashed to determine the chat shard:
shard_id = hash(channel_id) % num_shards

Each shard handles a subset of channels:
├── Shard 1: Channels 0-9999
├── Shard 2: Channels 10000-19999
├── ...
└── Shard N: Channels (N-1)*10000 - N*10000

Benefits:
├── Even distribution of load
├── Independent scaling per shard
├── No cross-shard communication needed
└── Easy to add/remove shards

Connection Scaling:

  • Each chat server node: 50K connections
  • Top stream (100K viewers): Spans 2+ chat server nodes within a shard
  • Load balancing within shard distributes connections evenly

8.3 Transcoding Scaling

GPU Cluster Management:

Transcoding Demand = Active Streams × Profiles per Stream

Current: 50K streams × 6 profiles = 300K encoding tasks
Peak: 100K streams × 6 profiles = 600K encoding tasks

GPU Requirements:
├── Per GPU (A100): ~50 concurrent H.264 encodes
├── Current: 300K / 50 = 6,000 GPUs
└── Peak: 600K / 50 = 12,000 GPUs

Auto-Scaling:
├── Scale up: When queue depth > 50 streams
├── Scale down: When queue depth < 10 streams for 15 minutes
└── Warm pool: Maintain 10% excess capacity for rapid scale-up

8.4 CDN Scaling

Content Distribution:

  • Live Segments: Short TTL (10-30 seconds); edge POPs fetch from origin frequently
  • VOD Segments: Long TTL (30 days); cached at edge for repeated viewing
  • Clips: Medium TTL (7 days); pre-cached at popular POPs
  • Static Assets: Long TTL (1 year); versioned URLs

POP Deployment:

  • 2,000+ edge POPs globally
  • Priority POPs in gaming-heavy regions (North America, Europe, Southeast Asia)
  • ISP-level peering for reduced last-mile latency

8.5 Database Scaling

  • Primary DB: PostgreSQL with read replicas (1 writer, 10+ readers)
  • Sharding: Users sharded by user_id hash; streams not sharded (time-partitioned)
  • Chat Storage: Time-series database (TimescaleDB / Cassandra) partitioned by channel and time
  • Redis Cluster: 20+ nodes for caching, rate limiting, and real-time counters
  • Analytics: Kafka → Flink → ClickHouse for real-time analytics

9. Curveball Scenarios

Curveball 1: “A top streamer with 500K concurrent viewers gets raided by another top streamer with 300K viewers. The chat system collapses under 800K concurrent users sending 100K messages/second. What happens?”

Impact Assessment:

  • Chat server shard handling the channel receives 8x normal load
  • Message delivery latency spikes from 200ms to 10+ seconds
  • WebSocket connections drop due to timeout
  • AutoMod can’t keep up; toxic content slips through
  • Viewer experience degrades; users leave the stream

Immediate Response:

  1. Auto-Scaling (0-30 seconds):

    • Chat shard detects load spike; triggers emergency scale-up
    • New chat server nodes join the shard within 30 seconds
    • Connections redistributed across new nodes
  2. Load Shedding (30-60 seconds):

    • If scale-up insufficient: enable aggressive rate limiting
    • Slow mode automatically activated (60-second delay between messages)
    • Non-essential features disabled (emote animations, badge updates)
  3. Message Queue Buffering:

    • Messages queued in Kafka if chat servers can’t process fast enough
    • Processed at reduced rate once capacity recovers
    • Oldest messages dropped if queue exceeds capacity
  4. Client-Side Adaptation:

    • Client detects high latency; reduces message send frequency
    • Show “chat is moving fast” indicator
    • Buffer messages locally; send when capacity recovers

Prevention for Future:

  • Predictive Scaling: Monitor raid patterns; pre-scale before raid completes
  • Shard Splitting: Top channels get dedicated shards with more resources
  • Hierarchical Rate Limiting: Global → Shard → Channel → User rate limits
  • Chat Threading: Separate hype train messages from regular chat

Curveball 2: “A streamer broadcasts copyrighted content (e.g., a movie or music). The content owner issues a DMCA takedown. How do you handle this at scale?”

Impact Assessment:

  • Legal liability for the platform
  • Potential loss of content distribution agreements
  • Streamer account suspension needed

Response Plan:

  1. Immediate (0-5 minutes):

    • Automated DMCA processing system receives the takedown notice
    • Stream is terminated immediately (ingest session killed)
    • VOD archive for the stream is blocked
    • Streamer receives notification and warning
  2. Content Removal (5-30 minutes):

    • All clips from the stream are identified and removed
    • VOD segments are flagged for deletion
    • Stream is removed from the directory and search results
  3. Streamer Action (30-60 minutes):

    • First offense: Warning + stream termination
    • Second offense: 7-day suspension
    • Third offense: Permanent ban
    • Repeat offenders flagged in the system
  4. Proactive Detection:

    • Audio Fingerprinting: Compare stream audio against copyrighted database (Audible Magic)
    • Video Fingerprinting: Compare stream video against known copyrighted content
    • Real-Time Scanning: Sample stream every 30 seconds; compare against fingerprint database
    • Automated Takedown: If match found, automatically terminate stream and notify streamer

Architecture Addition:

Live Stream → Fingerprint Extractor → Copyright Database Match
                                    ↓
                    Match Found → Terminate Stream + Notify Streamer + Log DMCA
                    No Match → Continue Streaming

Scale:

  • Fingerprint comparison: < 100ms per sample
  • Sampling rate: Every 30 seconds per stream
  • 50K streams × 2 samples/minute = 100K comparisons/minute

Curveball 3: “Your platform needs to support interactive streaming where viewers can influence the game in real-time (e.g., spawn enemies, change weather). How does this change the architecture?”

Architecture Changes:

  1. Bidirectional Communication Channel:

    Viewer → Action Request → Action Service → Game Engine → Stream Update
       ↑                                              ↓
       └───────────── Action Result ←─────────────────┘
  2. Low-Latency Requirement:

    • End-to-end latency must be < 1 second for meaningful interaction
    • Requires ultra-low latency streaming mode (WebRTC)
    • Action requests processed in real-time via WebSocket
  3. Action Service Design:

    • Validates viewer permissions (subscriber-only actions, channel point costs)
    • Rate limits actions per viewer and globally
    • Queues actions if the game can’t process them fast enough
    • Logs all actions for replay and analytics
  4. State Synchronization:

    • Game state broadcast to all viewers via WebSocket
    • State updates include: player position, game events, viewer actions
    • Conflict resolution for concurrent actions
  5. Scalability Concerns:

    • Each interactive stream limited to ~10K concurrent viewers (WebRTC constraint)
    • Action processing: ~1K actions/second per stream
    • State broadcast: ~100 state updates/second to all viewers

New Components:

Interactive Stream Architecture:
├── Action Ingestion Service (WebSocket)
├── Action Validation Service
├── Game State Manager
├── State Broadcast Service (WebSocket)
├── Action Queue (Kafka)
└── Replay Service (for VOD with interactive elements)

Trade-offs:

  • Interactive mode limited to smaller audiences
  • Higher infrastructure cost per viewer
  • Requires game developer integration
  • Not all content types support interactivity

Curveball 4: “A coordinated bot attack floods your platform with fake streams and spam chat messages. How do you detect and mitigate?”

Detection:

  1. Fake Stream Detection:

    • Analyze stream content: black screen, static image, looped video
    • Check viewer engagement: fake streams have low chat activity relative to viewer count
    • Monitor stream metadata: duplicate titles, suspicious patterns
    • ML model: classify streams as genuine vs. fake based on features
  2. Bot Chat Detection:

    • Message pattern analysis: repetitive messages, identical timing
    • Account behavior: new accounts, no watch history, rapid message sending
    • Network analysis: same IP sending from multiple accounts
    • ML model: classify messages as bot-generated vs. human

Mitigation:

  1. Automated Response:

    • Fake streams: Automatically terminate and suspend streamer account
    • Bot messages: AutoMod blocks and reports accounts
    • Rate limiting: Aggressive rate limits on suspicious accounts
  2. Account Verification:

    • Phone verification for new accounts
    • CAPTCHA for chat in high-traffic channels
    • Watch time requirements before chatting
  3. Infrastructure Protection:

    • DDoS protection at the network level
    • Rate limiting at the API gateway
    • Connection limits per IP address
  4. Human Review:

    • Trust and Safety team reviews flagged accounts
    • Appeal process for false positives
    • Regular audit of detection model accuracy

Architecture Addition:

Stream/Message → Behavior Analyzer → Suspicion Score
                                    ↓
                    Score > High Threshold → Auto-Block + Flag for Review
                    Score > Medium → Enhanced Monitoring
                    Score < Medium → Allow

Curveball 5: “You need to add a ‘Watch Together’ feature where friends can watch a stream同步 and chat in a private group. How do you design this?”

Feature Design:

  1. Synchronized Playback:

    • Group leader controls playback (play, pause, seek)
    • All members receive playback commands via WebSocket
    • Drift correction: periodic sync to ensure all viewers are aligned
  2. Private Chat Channel:

    • Separate from the stream’s public chat
    • Only group members can see and send messages
    • End-to-end encryption option for privacy
  3. Group Management:

    • Create group (up to 50 members)
    • Invite via link or user search
    • Kick/ban members
    • Group chat history

Architecture:

flowchart TB
    subgraph GROUP["Watch Together Group"]
        LEADER[Group Leader<br/>Controls Playback]
        MEMBER1[Member 1]
        MEMBER2[Member 2]
        MEMBERn[Member N]
    end
    
    subgraph SYNC["Sync Service"]
        PLAYBACK[Playback Sync<br/>WebSocket]
        CHAT[Private Chat<br/>WebSocket]
        DRIFT[Drift Correction<br/>Periodic Sync]
    end
    
    subgraph STREAM["Stream"]
        STREAM_VIDEO[Stream Video]
        PUBLIC_CHAT[Public Chat]
    end
    
    LEADER --> PLAYBACK
    MEMBER1 --> PLAYBACK
    MEMBER2 --> PLAYBACK
    MEMBERn --> PLAYBACK
    PLAYBACK --> DRIFT
    LEADER --> CHAT
    MEMBER1 --> CHAT
    MEMBER2 --> CHAT
    MEMBERn --> CHAT
    GROUP --> STREAM_VIDEO
    GROUP --> PUBLIC_CHAT

Technical Details:

  • Sync Protocol: Leader sends playback commands; followers adjust their players
  • Drift Tolerance: ±500ms; auto-correct if drift exceeds threshold
  • Network Handling: Handle member join/leave gracefully; leader election if leader leaves
  • Scale: 50 members per group; unlimited groups

10. Failure Modes and Mitigations

10.1 Failure Mode Matrix

Component Failure Mode Probability Impact Mitigation RTO
Ingest Ingest server failure Medium Medium Multi-region ingest; auto-failover < 10s
Ingest Streamer connection drop High Low Auto-reconnect; stream key validation < 30s
Transcode GPU node failure Medium Medium Redundant encoders; auto-reassign < 5s
Transcode Pipeline backlog Low High Drop oldest segments; scale up < 30s
Chat Chat server crash Medium High Shard redundancy; auto-reconnect < 15s
Chat Chat shard overload Low High Emergency rate limiting; scale up < 30s
CDN POP failure Medium Medium Multi-POP failover < 30s
CDN Origin overload Low High Origin shield; request queuing < 30s
AutoMod ML model degradation Low Medium Fallback to rule-based; retrain model < 5 min
Clip Clip engine failure Low Low Queue clips; process when recovered < 5 min
Payment Payment processor down Low High Backup payment processor; queue transactions < 5 min
Auth Auth service down Low High Cached tokens; extended TTL < 5 min
Database Primary DB failover Low High Automated failover; read replicas < 30s
Notification Push service down Medium Low Queue notifications; retry < 1 min

10.2 Cascading Failure Prevention

flowchart LR
    subgraph BREAKERS["Circuit Breakers"]
        CB1[Ingest → Transcode<br/>Break if queue > 500]
        CB2[Transcode → Origin<br/>Break if latency > 5s]
        CB3[Chat → AutoMod<br/>Break if latency > 200ms]
        CB4[Client → CDN<br/>Break if 5xx > 10%]
    end
    
    subgraph FALLBACKS["Fallbacks"]
        FB1[Backup Ingest Region]
        FB2[Cached Stream Segments]
        FB3[Rule-Based Moderation]
        FB4[Secondary CDN]
    end
    
    CB1 --> FB1
    CB2 --> FB2
    CB3 --> FB3
    CB4 --> FB4

10.3 Monitoring and Alerting

Key Metrics:

Metric Warning Critical Action
Stream Latency > 10s > 20s Check transcode pipeline
Chat Latency > 1s > 5s Scale chat servers
Chat Message Loss > 0.1% > 1% Check chat shard health
Transcode Queue > 50 streams > 200 streams Scale GPU cluster
CDN Cache Hit < 90% < 80% Check TTL; warm caches
AutoMod Latency > 100ms > 500ms Scale AutoMod; fallback to rules
Clip Processing Time > 10s > 30s Scale clip engine
Payment Failure Rate > 1% > 5% Check payment processor
Active Streamer Count — — Info; capacity planning
Concurrent Viewer Count — — Info; CDN scaling

Observability Stack:

  • Metrics: Prometheus + Grafana
  • Logging: ELK Stack
  • Tracing: Jaeger / OpenTelemetry
  • Alerting: PagerDuty with severity-based escalation
  • Dashboards: Stream health dashboard, chat health dashboard, business metrics dashboard

Appendix: Technology Stack Summary

Layer Technology Choices
Ingest NGINX RTMP, SRT (Haivision), GStreamer
Transcoding NVIDIA NVENC, FFmpeg, Intel QSV
Packaging Bento4, MP4Box, LL-HLS packager
Origin NGINX, Apache Traffic Server
CDN Akamai / CloudFront (primary), Cloudflare (secondary)
WebRTC mediasoup, Janus, or Custom SFU
Chat WebSocket (custom Go/Node.js servers), Redis Pub/Sub
Storage AWS S3 (VOD + clips), Redis (cache), PostgreSQL (metadata)
Chat Storage TimescaleDB / Cassandra (time-series)
Search Elasticsearch
Analytics Apache Kafka, Apache Flink, ClickHouse
ML/AutoMod TensorFlow, PyTorch, fine-tuned transformer
Auth Auth0 / Keycloak, OAuth 2.0, JWT
Payments Stripe, PayPal, Apple Pay, Google Pay
Notifications Firebase Cloud Messaging, APNs, SES
Monitoring Prometheus, Grafana, ELK, Jaeger
Infrastructure Kubernetes, Terraform, Helm
Streaming HLS (standard), LL-HLS (low-latency), WebRTC (ultra-low)