Communication & ChatQuestion #10ADVANCED

WhatsApp: Real-Time Messaging, E2EE & Presence Engine

Massive-concurrency messaging platform leveraging persistent connection clusters, end-to-end Signal encryption, distributed presence registries, and decentralized group fanout.

Category: Category 7: Communication & Chat•Updated: 2026-08-30
#Chat#Real-time Messaging#E2EE#Signal Protocol#Presence#Groups

Design a Chat Application like WhatsApp

Author: System Design Expert | Date: 2026-08-28 Scope: Real-time messaging with presence, read receipts, media sharing, group chats, end-to-end encryption, and offline message delivery.


Table of Contents

  1. Requirements Clarification
  2. Back-of-Envelope Calculations
  3. System Interface Design
  4. High-Level Architecture
  5. Core Component Design
  6. Data Model
  7. Key Design Decisions
  8. Scalability Strategy
  9. Curveball Scenarios
  10. Failure Modes and Mitigations

1. Requirements Clarification

1.1 Functional Requirements

ID Requirement Priority
FR-01 User Registration: Phone number-based registration with SMS/voice OTP verification P0
FR-02 Contact Discovery: Sync phone contacts, discover which contacts are on the platform P0
FR-03 One-on-One Messaging: Send/receive text messages in real-time P0
FR-04 End-to-End Encryption (E2EE): All messages encrypted client-side; server cannot decrypt P0
FR-05 Message Delivery Status: Delivered (single check), read (double blue check) receipts P0
FR-06 Presence & Typing Indicators: Online/offline status, “typing…” indicator P0
FR-07 Offline Message Delivery: Store and forward messages when recipient is offline P0
FR-08 Media Sharing: Images, videos, audio (voice notes), documents, GIFs, stickers P1
FR-09 Group Chats: Create groups, add/remove members, admin controls, group info P1
FR-10 Message History Sync: Sync chat history across multiple devices P1
FR-11 Status/Stories: Ephemeral media/text posts visible for 24 hours P1
FR-12 Call Signaling: Voice and video call setup (signaling only; media via WebRTC P2P) P1
FR-13 Message Search: Search through message history by keyword P2
FR-14 Message Reactions & Replies: React to messages with emojis, reply to specific messages P1
FR-15 Block/Mute: Block users, mute conversations P1
FR-16 Chat Backup & Restore: Cloud backup of chat history P2

1.2 Non-Functional Requirements

ID Requirement Target
NFR-01 Message Latency P95 < 500ms end-to-end (online-to-online)
NFR-02 Availability 99.95% uptime for messaging path
NFR-03 Throughput Support 2B+ MAU, 100B+ messages/day
NFR-04 Security E2EE (Signal Protocol), TLS 1.3 for transport, zero-knowledge architecture
NFR-05 Privacy Metadata minimization, no message content stored in plaintext on server
NFR-06 Offline Support Messages stored for 30 days; delivered when recipient comes online
NFR-07 Battery Efficiency Minimal battery drain on mobile devices (efficient keepalive, batching)
NFR-08 Data Efficiency Compressed messages, delta sync, adaptive media quality
NFR-09 Consistency Messages delivered in order per conversation; at-least-once delivery
NFR-10 Scalability Handle 10M+ concurrent WebSocket connections per region

2. Back-of-Envelope Calculations

2.1 Assumption Sets

Parameter Conservative (A) Base (B) Aggressive (C)
Monthly Active Users (MAU) 500M 2B 3B
Daily Active Users (DAU) 300M 1.2B 1.8B
Messages / DAU 20 30 40
Avg. text message size 200 bytes 300 bytes 400 bytes
Media messages / total 10% 15% 20%
Avg. media message size 500 KB 800 KB 1 MB
Concurrent connections 50M 200M 300M
Avg. group size 5 8 10
Group messages / total 15% 20% 25%
Fan-out multiplier (groups) 1.5x 2x 2.5x

2.2 Traffic Estimates (Base Case B)

Daily messages = DAU × messages/DAU = 1.2B × 30 = 36B messages/day

After group fan-out: 36B × 2x = 72B message deliveries/day

Steady-state message QPS:
  72B / 86,400s ≈ 833,333 message deliveries/sec

Peak QPS (3x): ~2.5M message deliveries/sec

Text message bandwidth:
  72B × 300 bytes = 21.6 TB/day text
  72B × 0.15 × 800 KB = 8.6 TB/day media
  Total: ~30 TB/day message data

2.3 Connection Estimates

Concurrent WebSocket connections: 200M
  Each connection: ~2 KB memory (buffers + state)
  Total connection memory: 200M × 2 KB = 400 GB

Connection keepalive:
  Ping/pong every 30 seconds
  200M / 30s ≈ 6.7M ping/pong frames/sec (bidirectional)

2.4 Storage Estimates

Component Calculation (Base) Size
Offline messages (30-day retention) 36B/day × 30 days × 300 bytes ~324 TB
Media files (90-day retention) 5.4B media/day × 90 days × 800 KB ~389 PB
User profiles 2B × 1 KB ~2 GB
Contact lists 2B × 100 contacts × 100 bytes ~20 GB
Group metadata 500M groups × 2 KB ~1 TB
Message metadata (encrypted) 36B/day × 30 days × 100 bytes ~108 TB
Total ~490 TB+ (media dominates)

2.5 Bandwidth Estimates

Outbound message bandwidth:
  Text: 21.6 TB/day = 250 MB/s average
  Media: 8.6 TB/day = 100 MB/s average
  Total: ~350 MB/s sustained

With CDN for media: 80% media served from CDN edge
  Origin egress: 350 MB/s × 0.2 (media) + 250 MB/s (text) ≈ 290 MB/s

3. System Interface Design

3.1 Client-Server Protocol (WebSocket + REST)

REST APIs (Registration, Discovery, Sync)

POST   /api/v1/auth/request-otp              — Request SMS/voice OTP
POST   /api/v1/auth/verify-otp                — Verify OTP, register/login
POST   /api/v1/auth/register-device           — Register new device
DELETE /api/v1/auth/unregister-device         — Unregister device

GET    /api/v1/contacts/upload                — Upload contact hash list
POST   /api/v1/contacts/upload                — Blobs of PHA hashes
GET    /api/v1/contacts/status                — Get online status of contacts

GET    /api/v1/messages/pending               — Fetch pending (offline) messages
POST   /api/v1/messages/ack                   — Acknowledge message receipt
GET    /api/v1/messages/history?chatId={id}&cursor={c}  — Fetch history

GET    /api/v1/groups/{groupId}               — Group info
POST   /api/v1/groups                          — Create group
POST   /api/v1/groups/{groupId}/participants   — Add participants
DELETE /api/v1/groups/{groupId}/participants/{userId}  — Remove participant

POST   /api/v1/media/upload                   — Upload media (returns URL)
GET    /api/v1/media/{mediaId}                 — Download media (CDN redirect)

GET    /api/v1/status/list                     — List available statuses
POST   /api/v1/status/upload                   — Upload status
POST   /api/v1/status/view/{statusId}          — Record status view

WebSocket Protocol (Real-Time Messaging)

Client → Server Frames:
  { "type": "MESSAGE", "chatId": "...", "payload": "<encrypted>", "timestamp": ..., "messageId": "..." }
  { "type": "TYPING", "chatId": "...", "isTyping": true/false }
  { "type": "READ_RECEIPT", "chatId": "...", "messageId": "..." }
  { "type": "DELIVERY_RECEIPT", "chatId": "...", "messageId": "..." }
  { "type": "PRESENCE", "status": "online/offline" }
  { "type": "PONG", "timestamp": ... }
  { "type": "REACTION", "chatId": "...", "messageId": "...", "emoji": "..." }

Server → Client Frames:
  { "type": "MESSAGE", "chatId": "...", "senderId": "...", "payload": "<encrypted>", "timestamp": ..., "messageId": "..." }
  { "type": "TYPING", "chatId": "...", "userId": "...", "isTyping": true/false }
  { "type": "READ_RECEIPT", "chatId": "...", "messageId": "...", "readerId": "..." }
  { "type": "DELIVERY_RECEIPT", "chatId": "...", "messageId": "..." }
  { "type": "PRESENCE", "userId": "...", "status": "online/offline", "lastSeen": ... }
  { "type": "PING", "timestamp": ... }
  { "type": "GROUP_UPDATE", "groupId": "...", "change": "..." }
  { "type": "DEVICE_MESSAGE", "payload": "<encrypted>" }  — Cross-device sync

3.2 Internal gRPC Interfaces

// Message Delivery Service
service MessageDeliveryService {
  rpc Deliver(DeliverRequest) returns (DeliverResponse);
  rpc StoreOffline(StoreOfflineRequest) returns (StoreOfflineResponse);
  rpc GetPending(GetPendingRequest) returns (stream GetPendingResponse);
}

// Presence Service
service PresenceService {
  rpc UpdatePresence(PresenceRequest) returns (PresenceResponse);
  rpc GetPresence(GetPresenceRequest) returns (GetPresenceResponse);
  rpc SubscribePresence(SubscribePresenceRequest) returns (stream PresenceUpdate);
}

// Contact Discovery Service
service ContactDiscoveryService {
  rpc UploadHashes(UploadHashesRequest) returns (UploadHashesResponse);
  rpc LookupMatches(LookupMatchesRequest) returns (LookupMatchesResponse);
}

// Key Management Service
service KeyManagementService {
  rpc UploadPublicKey(UploadPublicKeyRequest) returns (UploadPublicKeyResponse);
  rpc GetPublicKey(GetPublicKeyRequest) returns (GetPublicKeyResponse);
  rpc UploadPreKeys(UploadPreKeysRequest) returns (UploadPreKeysResponse);
  rpc GetPreKeys(GetPreKeysRequest) returns (GetPreKeysResponse);
}

4. High-Level Architecture

flowchart TB
    subgraph Clients["Client Layer"]
        WA[WhatsApp Android]
        WB[WhatsApp iOS]
        WW[WhatsApp Web/Desktop]
        WC[WhatsApp Business]
    end

    subgraph Edge["Edge Layer"]
        CDN[CDN<br/>(Media, Static Assets)]
        LB[Global Load Balancer<br/>(Geo-routing)]
    end

    subgraph Connection["Connection Layer"]
        subgraph ConnPool["Connection Managers"]
            CM1[Connection Manager Pool 1]
            CM2[Connection Manager Pool 2]
            CM3[Connection Manager Pool N]
        end
        Router[Connection Router<br/>(User → CM mapping)]
    end

    subgraph Services["Service Layer"]
        subgraph MsgSvc["Message Service"]
        end
        subgraph PresSvc["Presence Service"]
        end
        subgraph KeySvc["Key Management Service"]
        end
        subgraph ContactSvc["Contact Discovery Service"]
        end
        subgraph GroupSvc["Group Service"]
        end
        subgraph MediaSvc["Media Service"]
        end
        subgraph StatusSvc["Status Service"]
        end
        subgraph NotifSvc["Push Notification Service"]
        end
        subgraph AuthSvc["Auth Service"]
        end
    end

    subgraph Messaging["Messaging Backbone"]
        K[Apache Kafka<br/>(Message Events)]
        PQ[Priority Queue<br/>(Offline Messages)]
    end

    subgraph Data["Data Layer"]
        subgraph Databases["Databases"]
            Cassandra[(Cassandra<br/>Messages, Metadata)]
            Scylla[(ScyllaDB<br/>Offline Message Store)]
        end
        subgraph Cache["Cache"]
            Redis1[Redis Cluster 1<br/>(Presence, Sessions)]
            Redis2[Redis Cluster 2<br/>(Typing, Read Receipts)]
        end
        subgraph Storage["Object Storage"]
            S3[S3 / OSS<br/>(Encrypted Media)]
        end
        subgraph KeyStore["Key Store"]
            HBase[(HBase<br/>E2EE Keys, PreKeys)]
        end
    end

    subgraph Infra["Infrastructure"]
        Monitor[Monitoring & Alerting]
        Trace[Distributed Tracing]
        Security[Security & Compliance]
    end

    WA --> LB
    WB --> LB
    WW --> LB
    WC --> LB

    LB --> CM1
    LB --> CM2
    LB --> CM3

    CM1 --> Router
    CM2 --> Router
    CM3 --> Router

    Router --> MsgSvc
    Router --> PresSvc
    Router --> KeySvc
    Router --> ContactSvc
    Router --> GroupSvc
    Router --> MediaSvc
    Router --> StatusSvc

    MsgSvc --> K
    PresSvc --> Redis1
    PresSvc --> Redis2

    K --> PQ
    K --> NotifSvc

    MsgSvc --> Cassandra
    MsgSvc --> Scylla
    PresSvc --> Redis1
    KeySvc --> HBase
    MediaSvc --> S3
    GroupSvc --> Cassandra
    ContactSvc --> Cassandra
    AuthSvc --> Cassandra
    StatusSvc --> S3

5. Core Component Design

5.1 Connection Manager (WebSocket Gateway)

Purpose: Maintain persistent WebSocket connections with clients; route messages to/from services.

Design:

  • Protocol: WebSocket over TLS 1.3; fallback to long-polling for restrictive networks
  • Connection lifecycle:
    Connect → Authenticate (JWT/token) → Subscribe to user's message queue → Active
    Active ↔ Send/Receive frames
    Idle (>5 min no activity) → Throttle keepalive
    Disconnect → Mark user offline → Drain pending messages
  • User-to-Connection mapping: Redis-based consistent hashing
    • user:{userId} → connectionManager:{cmId}:connection:{connId}
    • TTL-based; refreshed on each ping/pong
  • Connection pooling: Each CM instance handles ~500K concurrent connections
    • 200M connections / 500K per CM = 400 CM instances
  • Keepalive strategy:
    • Server sends PING every 30 seconds
    • Client must respond with PONG within 10 seconds
    • After 3 missed pongs, connection considered dead
  • Battery optimization:
    • Adaptive keepalive: Increase interval when device is on battery + idle
    • Message batching: Coalesce multiple messages into single frame when possible

Connection Manager Memory Budget:

Per connection:
  - Socket buffer: 64 KB (send) + 64 KB (receive)
  - Session state: 2 KB
  - Frame buffers: 16 KB
  Total per connection: ~146 KB

400 CMs × 500K connections × 146 KB = ~29 TB total connection memory

5.2 Message Service

Purpose: Core message routing, delivery, and offline storage.

Message Flow (Online-to-Online):

sequenceDiagram
    participant A as Sender Device
    participant CMA as CM (Sender)
    participant MS as Message Service
    participant K as Kafka
    participant CMB as CM (Recipient)
    participant B as Recipient Device

    A->>CMA: WebSocket: MESSAGE (encrypted payload)
    CMA->>MS: Deliver message
    MS->>MS: Validate, assign messageId, timestamp
    MS->>K: Publish to user-{recipientId} topic
    MS-->>CMA: ACK
    CMA-->>A: Delivery receipt (single check)

    K->>MS: Consume from user-{recipientId}
    MS->>MS: Lookup recipient connection
    MS->>CMB: Push message
    CMB->>B: WebSocket: MESSAGE
    B->>CMB: DELIVERY_RECEIPT
    CMB->>MS: Ack delivery
    MS->>K: Publish delivery event
    K->>MS: Consume delivery event
    MS->>CMA: Forward delivery receipt
    CMA->>A: Delivery receipt update

Message Flow (Online-to-Offline):

1. Sender sends message → Message Service receives
2. Message Service checks recipient presence (Redis)
3. Recipient offline → Store in ScyllaDB (offline message store)
4. Publish event to Kafka for push notification trigger
5. Push Notification Service sends FCM/APNs notification
6. When recipient comes online:
   a. Client fetches pending messages via REST API
   b. OR Connection Manager pushes queued messages on reconnect
7. Messages deleted from offline store after delivery + grace period

Message Ordering:

  • Per-conversation ordering guaranteed via sequence numbers
  • Each message carries (chatId, senderId, sequenceNumber) tuple
  • Client-side deduplication using messageId
  • Server-side: Kafka partition key = chatId ensures per-conversation ordering

Message Deduplication:

  • Idempotent delivery via messageId (UUID v4 generated by sender)
  • Server tracks delivered message IDs in a sliding window (Redis Bloom filter)
  • Duplicate messages silently dropped

5.3 End-to-End Encryption (Signal Protocol Implementation)

Purpose: Ensure only sender and recipient can read message contents.

Protocol: Double Ratchet Algorithm (Signal Protocol)

Key Exchange Flow:
1. Each user generates:
   - Identity Key pair (long-term, stored on device)
   - Signed Pre-Key (signed by Identity Key, uploaded to server)
   - 100 One-Time Pre-Keys (uploaded to server)
   - Reserved Pre-Key bundle

2. When User A messages User B for the first time:
   a. A fetches B's Signed Pre-Key + one One-Time Pre-Key from server
   b. A performs X3DH (Extended Triple Diffie-Hellman):
      Shared Secret = DH(A.identityPriv, B.signedPreKeyPub)
                   ⊕ DH(A.identityPriv, B.oneTimePreKeyPub)
                   ⊕ DH(A.oneTimePreKeyPriv, B.signedPreKeyPub)
   c. Shared Secret → SKDF → Initial Messaging Key + Chain Key
   d. Double Ratchet takes over for subsequent messages

3. Each message:
   - Ratchet forward (chain key → message key)
   - Encrypt payload with message key (AES-256-GCM)
   - Include ratchet info for remote key synchronization

Server’s Role:

  • Stores pre-keys (encrypted, server cannot decrypt messages)
  • Routes encrypted blobs transparently
  • Never sees plaintext message content
  • Key bundles deleted after use (one-time pre-keys)

Key Storage:

HBase Table: signal_prekeys
  Row Key: userId
  Columns:
    signed_prekey: {keyId, publicKey, signature}
    one_time_prekeys: [{keyId, publicKey}, ...]  (100 keys)
    identity_key: {publicKey}
    device_id: int

Multi-Device Support:

  • Primary device holds the root key
  • Secondary devices receive encrypted key bundles via primary
  • Each device has its own DH key pair; messages encrypted to all active devices

5.4 Presence Service

Purpose: Track user online/offline status and last seen timestamp.

Design:

  • Storage: Redis Cluster with geo-partitioning
    Key: presence:{userId}
    Value: { "status": "online/offline", "lastSeen": timestamp, "cmId": "..." }
    TTL: 2 minutes (auto-expire → offline)
  • Status transitions:
    Connected → Online (publish to subscribers)
    Idle > 2 min → Away (optional)
    Disconnected → Offline (publish to subscribers)
  • Privacy settings:
    • “Everyone can see my last seen”
    • “My contacts only”
    • “Nobody”
    • Enforced at query time, not storage time
  • Subscription model:
    • Users subscribe to presence of their contacts
    • Stored in Redis: presence_subscribers:{userId} → Set(contactId1, contactId2, ...)
    • On status change, publish to all subscribers via pub/sub

Typing Indicator:

Key: typing:{chatId}:{userId}
Value: true
TTL: 5 seconds

When client sends typing frame:
  1. Set typing:{chatId}:{userId} = true, TTL 5s
  2. Publish to chat subscribers via Redis pub/sub
  3. Recipients show "typing..." indicator

TTL expiry → typing stops automatically

5.5 Read Receipts Service

Purpose: Track message delivery and read status.

Design:

  • Delivery receipt (single gray check):

    • Generated when message reaches recipient’s device (Connection Manager)
    • Stored in Redis: receipt:delivered:{messageId} → {userId, timestamp}
    • Forwarded to sender via Kafka event
  • Read receipt (double blue check):

    • Generated when recipient’s app brings conversation to foreground
    • Client sends READ_RECEIPT frame → Message Service → Kafka → Sender’s CM → Sender
    • Stored in Cassandra for history: read_receipts(messageId, readerId, timestamp)
  • Privacy:

    • Users can disable read receipts in settings
    • When disabled, read events not sent; sender sees single check only
    • Group chats: Show count of members who read (“Read by 3/5”)

5.6 Contact Discovery Service

Purpose: Let users find which of their phone contacts are on WhatsApp, without exposing privacy.

Design (Privacy-Preserving):

1. Client side:
   a. Read phone contact list (user-granted permission)
   b. Hash each phone number: PHA = SHA-256(phoneNumber)
   c. Upload PHA hashes in batches (blobs) to server

2. Server side:
   a. Store registered user PHAs in Cassandra
   b. Compute intersection: uploaded PHAs ∩ registered PHAs
   c. Return matching user IDs (with profile info)

3. Privacy guarantees:
   - Server sees PHAs, not raw phone numbers
   - PHA blobs deleted after processing
   - Rate limited: Max N contact uploads per day

PHA Upload Protocol:

POST /api/v1/contacts/upload
Content-Type: application/octet-stream

Body: Encrypted blob containing PHA hashes
  - Encrypted with server's public key
  - Contains: [PHA1, PHA2, ..., PHAN]
  - Max 500 PHAs per request

5.7 Group Chat Service

Purpose: Manage group creation, membership, and message fan-out.

Design:

  • Group model:

    {
      "groupId": "gid-abc123",
      "name": "Team Project",
      "creatorId": "user-001",
      "admins": ["user-001", "user-002"],
      "participants": ["user-001", "user-002", "user-003", ...],
      "participantCount": 45,
      "createdAt": "2026-08-01T10:00:00Z",
      "settings": {
        "onlyAdminsCanSend": false,
        "approvalRequired": false,
        "description": "Project discussion group"
      }
    }
  • Max group size: 1,024 participants (standard), 2,000 (community)

  • Message fan-out:

    • When user sends message to group, Message Service fans out to all participants
    • Fan-out happens asynchronously via Kafka
    • Each participant receives the message on their personal topic
    • Offline participants get stored + push notification
  • Group E2EE:

    • Each group has its own ratchet state per participant
    • Sender encrypts message once per recipient (N encryptions for N participants)
    • For large groups, use broadcast channels (optimized encryption)
  • Group events:

    • Member added/removed → System message in group chat
    • Name changed → System message
    • Admin changed → System message
    • All events go through same message pipeline (encrypted system messages)

5.8 Media Service

Purpose: Handle upload, storage, and delivery of media files.

Design:

  • Upload flow:

    1. Client compresses/optimizes media (client-side)
    2. Client encrypts media with session key (E2EE for media too)
    3. Client uploads encrypted blob to Media Service
    4. Media Service stores in S3/OSS with metadata
    5. Media Service returns media URL + metadata to Message Service
    6. Message Service sends message with media reference to recipient
    7. Recipient downloads from CDN (cached encrypted blob)
    8. Recipient decrypts with session key
  • Media optimization:

    • Images: WebP format, adaptive quality based on network
    • Videos: H.264, multiple bitrate versions (client selects)
    • Audio: Opus codec for voice notes
    • Documents: Stored as-is, size limit 100 MB
  • Storage lifecycle:

    • Media retained for 90 days (configurable)
    • After 90 days, media deleted; message shows “Media expired”
    • “Keep in Chat” option downloads to device permanently
  • CDN integration:

    • Encrypted media blobs cached at CDN edge
    • Cache key includes media ID + encryption version
    • TTL: 24 hours at edge, 7 days at origin shield

5.9 Push Notification Service

Purpose: Notify offline users of new messages via FCM (Android) and APNs (iOS).

Design:

  • Trigger: Kafka consumer subscribes to message.delivered events where recipient is offline
  • Notification content:
    • With E2EE: Notification shows sender name + “Message” (no content)
    • If user enables “Show message preview”: Content encrypted with notification key (separate from message key)
  • Provider integration:
    • Android: Firebase Cloud Messaging (FCM)
    • iOS: Apple Push Notification Service (APNs)
  • Batching: Coalesce multiple messages into single notification (“3 new messages from Alice”)
  • Silent notifications: Data-only notifications to wake app without showing UI

5.10 Status/Stories Service

Purpose: Ephemeral content visible for 24 hours.

Design:

  • Upload: Similar to media service but with 24-hour TTL
  • Viewer list: Track who viewed each status (stored in Redis, TTL 24h)
  • Delivery: Push to all contacts when status is posted
  • Expiration: Background job deletes expired statuses and viewer lists
  • Privacy: Status visibility settings (Everyone, My Contacts, Only Share With, My Except)

6. Data Model

6.1 Cassandra Schema (Messages, Metadata)

-- User Profiles
CREATE TABLE user_profiles (
    user_id TEXT PRIMARY KEY,
    phone_number TEXT,
    display_name TEXT,
    profile_picture_url TEXT,
    about_text TEXT,
    privacy_settings FROZEN<PrivacySettings>,
    created_at TIMESTAMP,
    updated_at TIMESTAMP
) WITH COMPACT STORAGE;

TYPE PrivacySettings (
    last_seen TEXT,
    profile_photo TEXT,
    status TEXT,
    read_receipts BOOLEAN,
    group_add TEXT
);

-- Messages (partitioned by chatId for per-conversation ordering)
CREATE TABLE messages (
    chat_id TEXT,
    sequence_num BIGINT,
    message_id TEXT,
    sender_id TEXT,
    timestamp BIGINT,
    encrypted_payload BLOB,
    message_type TEXT,  -- 'text', 'image', 'video', 'audio', 'document', 'system'
    media_metadata FROZEN<MediaMetadata>,
    ttl_days INT,
    PRIMARY KEY (chat_id, sequence_num)
) WITH CLUSTERING ORDER BY (sequence_num ASC)
  AND gc_grace_seconds = 86400;

TYPE MediaMetadata (
    media_id TEXT,
    media_url TEXT,
    thumbnail_url TEXT,
    mime_type TEXT,
    file_size BIGINT,
    width INT,
    height INT,
    duration INT
);

-- Offline Message Store (partitioned by recipient for efficient fetch)
CREATE TABLE offline_messages (
    recipient_id TEXT,
    timestamp BIGINT,
    message_id TEXT,
    chat_id TEXT,
    sender_id TEXT,
    encrypted_payload BLOB,
    message_type TEXT,
    media_metadata FROZEN<MediaMetadata>,
    delivered BOOLEAN,
    PRIMARY KEY (recipient_id, timestamp, message_id)
) WITH CLUSTERING ORDER BY (timestamp ASC);

-- Chat Metadata
CREATE TABLE chat_metadata (
    chat_id TEXT PRIMARY KEY,
    chat_type TEXT,  -- 'direct', 'group'
    participant_ids SET<TEXT>,
    created_at TIMESTAMP,
    updated_at TIMESTAMP,
    last_message_timestamp BIGINT,
    last_message_preview TEXT,
    settings FROZEN<ChatSettings>
);

TYPE ChatSettings (
    name TEXT,
    description TEXT,
    group_icon_url TEXT,
    only_admins_can_send BOOLEAN,
    admins SET<TEXT>
);

-- Read Receipts
CREATE TABLE read_receipts (
    message_id TEXT,
    reader_id TEXT,
    read_at TIMESTAMP,
    PRIMARY KEY (message_id, reader_id)
);

-- Delivery Receipts
CREATE TABLE delivery_receipts (
    message_id TEXT PRIMARY KEY,
    recipients MAP<TEXT, TIMESTAMP>
) WITH COMPACT STORAGE;

-- Contact Relationships
CREATE TABLE contacts (
    user_id TEXT,
    contact_id TEXT,
    status TEXT,  -- 'active', 'blocked', 'muted'
    muting_until TIMESTAMP,
    starred BOOLEAN,
    unread_count INT,
    PRIMARY KEY (user_id, contact_id)
);

-- Group Memberships
CREATE TABLE group_memberships (
    group_id TEXT,
    user_id TEXT,
    role TEXT,  -- 'admin', 'member'
    joined_at TIMESTAMP,
    primary_key_list TEXT,  -- E2EE key for group
    PRIMARY KEY (group_id, user_id)
);

6.2 Redis Data Structures

# Presence
SET presence:{userId} '{"status":"online","lastSeen":1724839200,"cmId":"cm-42"}' EX 120

# Presence Subscribers
SADD presence_subscribers:{userId} contactId1 contactId2 contactId3

# Typing Indicator
SET typing:{chatId}:{userId} "true" EX 5

# User-to-Connection Mapping
SET conn:{userId} '{"cmId":"cm-42","connId":"conn-abc"}' EX 60

# Message Deduplication (Bloom Filter)
BF.ADD dedup:{userId} messageId1 messageId2 messageId3

# Read Receipts (short-term cache)
HSET receipts:read:{chatId} messageId1 userId1 messageId2 userId2

# Rate Limiting
INCR rate:{userId}:message
EXPIRE rate:{userId}:message 60

# Push Notification Queue
LPUSH push_queue:{userId} '{"senderId":"...","chatId":"...","preview":"..."}'

6.3 HBase Schema (E2EE Keys)

Table: signal_prekeys

Row Key: {userId}_{deviceId}

Column Families:
  id:  identity_key_pub, identity_key_sig
  spk: signed_prekey_{keyId}  →  {publicKey, signature, timestamp}
  otp: one_time_prekey_{keyId} →  {publicKey, timestamp}
  info: next_otp_key_id, uploaded_at

6.4 Kafka Topics

Topic: message.{recipientId}          — Per-user message delivery (partitioned by chatId)
  Partition key: chatId
  Value: {messageId, senderId, chatId, encryptedPayload, timestamp, messageType}

Topic: message.delivery               — Delivery receipt events
  Value: {messageId, recipientId, deliveredAt}

Topic: message.read                   — Read receipt events
  Value: {messageId, readerId, readAt}

Topic: presence.change                — Presence status changes
  Value: {userId, status, lastSeen, cmId}

Topic: group.event                    — Group membership changes
  Value: {groupId, eventType, userId, timestamp}

Topic: push.notification              — Push notification triggers
  Value: {userId, senderId, chatId, preview, timestamp}

Topic: media.upload                   — Media upload events
  Value: {mediaId, userId, url, mimeType, size}

7. Key Design Decisions

7.1 Trade-off Analysis

Decision Option A Option B Choice Rationale
Transport Protocol HTTP/REST (polling) WebSocket (persistent) WebSocket Real-time messaging requires push capability. Polling wastes bandwidth and battery.
Message Store MySQL Cassandra / ScyllaDB Cassandra Write-heavy, high throughput, wide row per conversation. Cassandra’s partition model fits perfectly.
Offline Store Same DB as messages Separate ScyllaDB cluster Separate ScyllaDB Offline messages have different access patterns (bulk fetch on reconnect). Separation prevents interference.
E2EE Protocol Custom encryption Signal Protocol (Double Ratchet) Signal Protocol Battle-tested, open-source, forward secrecy + future secrecy. No reason to reinvent.
Presence Storage Database Redis Redis Presence is ephemeral, high-frequency updates. Redis sub-ms latency is essential.
Contact Discovery Plain phone numbers PHA (Privacy-Enhanced Address) PHA hashes Privacy requirement. Server should not see raw phone numbers.
Group Fan-out Sync fan-out Async via Kafka Async Kafka Sync fan-out to 1000+ members would block sender. Async ensures sender gets fast ACK.
Media Storage Database BLOB Object Storage (S3) S3 + CDN Media is large, infrequently accessed after delivery. Object storage + CDN is cost-effective.
Push Notifications In-house FCM / APNs FCM / APNs OS-level integration required for reliable delivery. No viable alternative.
Message Ordering Global sequence Per-conversation sequence Per-conversation Global ordering impossible at scale. Per-conversation ordering is what users perceive.

7.2 E2EE Architecture Implications

The E2EE requirement fundamentally shapes the architecture:

  1. Server is a blind relay: Cannot inspect, filter, or search message content
  2. Spam/abuse moderation: Must use client-side signals (user reports, metadata patterns) rather than content analysis
  3. Backup/restore: Encrypted backups only; server cannot restore without user’s key
  4. Multi-device: Requires key distribution protocol; primary device is trust anchor
  5. Legal compliance: Cannot comply with lawful intercept for message content (only metadata)

8. Scalability Strategy

8.1 Connection Scaling

Target: 200M concurrent connections

Connection Manager sizing:
  - Per CM capacity: 500K connections
  - Required CMs: 200M / 500K = 400 instances
  - Per region: ~100 CMs (4 regions)

CM hardware:
  - 64 vCPU, 256 GB RAM
  - 500K connections × 146 KB = ~73 GB/connection memory
  - Headroom: 256 GB / 73 GB = 3.5x

Auto-scaling:
  - Scale CM pool based on connection count
  - Target utilization: 60% (300K connections per CM)
  - Scale-up threshold: > 400K connections per CM
  - Scale-down threshold: < 200K connections per CM

8.2 Message Throughput Scaling

Target: 833K message deliveries/sec steady-state

Kafka sizing:
  - Per partition throughput: ~50K msg/sec
  - User topics: 2B users (one topic per user is too many)
  - Solution: Topic hierarchy
    - 10,000 aggregate topics (user hash mod 10,000)
    - Each topic: 64 partitions
    - Total partitions: 640,000
    - Per partition: 833K / 640K ≈ 1.3K msg/sec (well within capacity)

Consumer groups:
  - Message Service consumers: 640K partitions / 4 consumers per partition = 256K consumer instances
  - Actually: Over-provision with 4x = 1M consumer instances across all CMs
  - Each CM consumes for its connected users only (filtered consumption)

8.3 Database Scaling

Cassandra (Messages):
  - Write rate: 36B messages/day = 416K msg/sec
  - Partition key: chatId
  - Hot partition risk: Popular group chats
  - Mitigation: Sub-partitioning for groups > 500 members
  - Cluster: 500 nodes, 3 replicas
  - Per node write capacity: 416K / (500 × 3) ≈ 280 writes/sec (trivial)

ScyllaDB (Offline Messages):
  - Write rate: Same as message rate (all messages stored)
  - Read pattern: Bursty (user reconnects, fetches all pending)
  - Partition key: recipientId
  - Cluster: 300 nodes, 2 replicas
  - TTL-based auto-expiry (30 days)

Redis (Presence):
  - 2B users × 1 entry = 2B keys
  - Cluster: 1,000 shards across 500 nodes
  - Per shard: 2M keys (well within capacity)

8.4 Multi-Region Deployment

Region Strategy: Active-Active with user affinity

Users assigned to home region based on:
  - Registration location (SIM country code)
  - Persistent assignment (stuck to home region for data consistency)

Cross-region messaging:
  - Sender in US, recipient in EU
  - Message routed via Kafka cross-region replication
  - Latency: ~100ms additional for cross-region

Regional data sovereignty:
  - User data stays in home region
  - Complies with GDPR, local data residency laws

8.5 Media Scaling

Media CDN:
  - Global CDN with 2,000+ edge locations
  - Origin shield layers: Edge → Regional → Origin
  - Cache hit rate target: 95%+ at edge
  - Bandwidth: 8.6 TB/day media × 0.95 CDN hit = 0.43 TB/day origin

Media storage lifecycle:
  Hot (0-7 days):   SSD storage, CDN cached
  Warm (7-30 days):  Standard storage, CDN cached
  Cold (30-90 days): Infrequent access storage
  Expired (90+ days): Deleted

9. Curveball Scenarios

Curveball 1: “A viral message is forwarded to 100,000 groups simultaneously. How do you handle the fan-out?”

Analysis:

  • A single message fanning out to 100K groups × avg 8 members = 800K deliveries
  • This is a fan-out amplification problem
  • The bottleneck is the Message Service writing to 800K user queues

Response:

  1. Detection:

    • Monitor fan-out ratio per message (deliveries / 1)
    • Alert when fan-out > 10K
  2. Handling:

    • Async fan-out with rate limiting:
      • Group fan-out batches: 1,000 groups per batch
      • Each batch processed by dedicated fan-out workers
      • Rate limit: Max 50K fan-out operations/sec per message
    • Priority queuing:
      • Direct messages: Highest priority
      • Small groups (< 50): High priority
      • Large groups (> 500): Normal priority
      • Broadcast channels: Lower priority
    • Sender experience:
      • Sender gets immediate ACK (message accepted)
      • Delivery receipts arrive asynchronously
      • No blocking on fan-out completion
  3. Prevention:

    • Forward limits: Max N forwards per 24 hours (WhatsApp already does this)
    • “Forwarded many times” label for widely forwarded messages
    • Rate limiting on forward actions per user
  4. Infrastructure:

    • Dedicated Kafka topic for high-fan-out messages
    • Separate consumer pool for fan-out processing
    • Auto-scale fan-out workers based on queue depth

Curveball 2: “A zero-day vulnerability is discovered in the Signal Protocol implementation. How do you respond?”

Analysis:

  • E2EE protocol compromise is the highest-severity scenario
  • All message confidentiality potentially breached
  • Must balance security response with service continuity

Response:

  1. Immediate actions (0-1 hour):

    • Security team: Assess scope and severity
    • Key rotation: Force ratchet reset for all active sessions
    • Pre-key refresh: Invalidate all one-time pre-keys; force regeneration
    • Client update: Emergency OTA update with patched protocol
  2. User communication:

    • In-app notification about security update
    • Transparent communication without causing panic
    • Clear instructions for manual key reset
  3. Technical response:

    • Server-side: Delete all stored pre-keys; force re-upload on next client connection
    • Session reset: On next message send, trigger fresh X3DH key exchange
    • Backward compatibility: Patched clients can communicate with unpatched clients using fallback mode (with warning)
  4. Long-term:

    • Independent security audit of protocol implementation
    • Bug bounty program expansion
    • Formal verification of critical crypto code paths
    • Consider multi-protocol support for redundancy

Curveball 3: “A government demands backdoor access to messages for national security. How do you handle it?”

Analysis:

  • Fundamental conflict between E2EE architecture and backdoor requirements
  • A backdoor for “bad actors” is a backdoor for everyone
  • Legal, ethical, and technical dimensions

Response:

  1. Technical reality:

    • Cannot comply with message content backdoor: E2EE by design prevents server-side access
    • Any backdoor weakens security for all users
    • Historical evidence: Backdoors are always exploitable beyond intended use
  2. What can be provided (metadata, not content):

    • User registration data (phone number, registration date)
    • Contact list metadata (who communicated with whom, when)
    • Group membership lists
    • Last seen timestamps
    • Cannot provide: Message content, media content, call content
  3. Legal process:

    • Require valid legal orders (warrants, court orders)
    • Legal team review before any data disclosure
    • User notification where legally permitted
    • Transparency reports published regularly
  4. Architecture defense:

    • Minimize metadata retention (auto-delete after N days)
    • Encrypt metadata at rest where possible
    • Design for “plausible deniability” where legally feasible

Curveball 4: “The app is banned in a country with 200M users. How do you handle the transition?”

Analysis:

  • 200M users lose access overnight
  • Data retention and deletion requirements
  • Business and ethical considerations

Response:

  1. Immediate impact assessment:

    • 200M users × 30 messages/day = 6B messages/day lost
    • Revenue impact from ads/business features
    • User migration patterns (to competing apps)
  2. User data handling:

    • Data export: Allow users to export chat history before cutoff
    • Grace period: 30-day wind-down for data export
    • Data deletion: Comply with local data deletion requirements
    • Encrypted backups: User-controlled; server cannot access
  3. Technical measures:

    • Geo-blocking at CDN and API Gateway level
    • SIM-based detection for registration blocking
    • Existing sessions: Allow to expire naturally or force disconnect
  4. Alternative access (if ethically/legal permissible):

    • Web-only access (no app store listing)
    • Desktop client only
    • Evaluate legal risks before enabling any workaround

Curveball 5: “A bug causes messages to be delivered out of order. Users are confused. How do you fix it?”

Analysis:

  • Message ordering is critical for conversation experience
  • Out-of-order delivery breaks conversation coherence
  • Must fix both the bug and the user-facing experience

Response:

  1. Root cause analysis:

    • Likely causes: Kafka partition reassignment, consumer rebalancing, multi-path routing
    • Check: Is ordering broken per-conversation or globally?
    • Verify: Sequence numbers still assigned correctly?
  2. Immediate mitigation:

    • Client-side reordering: Client buffers messages for 2 seconds, sorts by sequence number before display
    • Server-side fix: Ensure single consumer per chatId partition
    • Kafka partition
    • Rollback: If caused by recent deployment, rollback immediately
  3. User communication:

    • In-app message: “You may see messages out of order temporarily”
    • Status page update with ETA
  4. Long-term prevention:

    • Sequence number validation: Server rejects out-of-order sequence numbers
    • Consumer assignment stability: Sticky partition assignment for consumers
    • Testing: Chaos engineering tests for ordering under failure scenarios
    • Monitoring: Real-time ordering violation detection and alerting

Curveball 6: “How do you detect and prevent child abuse material (CAM) without breaking E2EE?”

Analysis:

  • Legal requirement in many jurisdictions
  • Fundamental tension with E2EE promise
  • Must find technical solution that doesn’t compromise all users’ privacy

Response:

  1. Client-side scanning (controversial but technically feasible):

    • PhotoDNA / perceptual hashing: Generate hash of images on device
    • Hash comparison: Compare against known CAM hash database (stored on device)
    • Reporting: If match found, flag to server (hash only, not image)
    • Privacy: Hash database updated via signed OTA updates; auditable
  2. User reporting (primary mechanism):

    • “Report” button on every message and contact
    • Reported content reviewed by trained moderators (user must consent to decrypt for review)
    • Pattern detection: Multiple reports from different users on same content
  3. Metadata analysis:

    • Behavioral patterns indicative of abuse (without seeing content)
    • Network analysis: Known bad actors, coordinated behavior
    • Volume anomalies: Unusual message volumes to/from certain accounts
  4. Legal compliance:

    • Work with law enforcement on specific investigations
    • Court-ordered client-side search tools (targeted, not mass surveillance)
    • Transparency about capabilities and limitations

10. Failure Modes and Mitigations

10.1 Failure Mode Matrix

Component Failure Mode Impact Detection Mitigation RTO
Connection Manager Process crash Users disconnected, messages queued Health check, connection drop spike Auto-restart, reconnect to another CM < 30s
Connection Manager Memory exhaustion New connections rejected Memory > 90% alert Connection draining, scale up < 2min
Connection Manager Network partition Regional disconnect Cross-AZ health checks Traffic failover to healthy AZ < 1min
Message Service Kafka consumer lag Message delivery delay Consumer lag metric Auto-scale consumers, add partitions < 1min
Message Service Kafka broker failure Message loss risk Broker down alert ISR replication, auto-failover < 30s
Message Service Duplicate delivery User sees duplicate messages Client dedup report Bloom filter dedup, idempotent delivery Auto
Presence Service Redis node failure Presence stale for shard Node down alert Redis Cluster failover, rehash < 30s
Presence Service Redis memory full New presence writes fail Memory alert Maxmemory policy, scale shards Auto
Offline Store ScyllaDB node failure Offline messages unavailable Node down alert Replica failover, repair < 1min
Offline Store TTL expiry bug Messages lost prematurely Message count anomaly TTL validation job, alert on anomaly Manual
Key Management HBase region server down Key lookup failures Region offline alert Region server restart, split < 2min
Key Management Pre-key exhaustion New sessions fail Pre-key count < threshold Auto-refill pre-keys on client connect Auto
Media Service S3 outage Media upload/download fail HTTP 5xx rate Fallback to secondary storage, queue < 5min
Media Service CDN cache poisoning Wrong media served Content hash mismatch Cache invalidation, short TTL < 1min
Push Notifications FCM/APNs outage No push for offline users Delivery rate drop Retry queue, fallback to SMS (critical) < 5min
Push Notifications Token expiration Push fails for users FCM registration error Token refresh on next app open Auto
Contact Discovery PHA collision Wrong contact matches User report Extended PHA, collision detection Manual

10.2 Cascading Failure Prevention

flowchart LR
    subgraph Patterns["Resilience Patterns"]
        CB[Circuit Breaker<br/>Per-service, per-dependency]
        B[Bulkhead<br/>Connection pools isolated]
        R[Retry with Backoff<br/>Exponential + jitter]
        T[Timeout<br/>Strict per-call timeouts]
        Q[Queue-Based Leveling<br/>Kafka as buffer]
        D[Graceful Degradation<br/>Drop non-critical features]
        F[Feature Flags<br/>Kill switches per feature]
    end

    CB -->|Prevents| CF[Cascading Failures]
    B -->|Contains| CF
    R -->|Handles| TF[Transient Failures]
    T -->|Prevents| RH[Thread/Connection Exhaustion]
    Q -->|Absorbs| SP[Traffic Spikes]
    D -->|Preserves| CP[Core Messaging]
    F -->|Enables| QR[Quick Response]

Degradation Modes (priority order):

  1. Normal: All features active
  2. Elevated load: Disable typing indicators, reduce presence update frequency
  3. High load: Disable read receipts, disable status/stories, reduce push notification frequency
  4. Critical load: Text-only messaging, disable media, disable group fan-out (queue only)
  5. Emergency: Direct messages only, all groups paused

10.3 Disaster Recovery Plan

Scenario RPO RTO Procedure
AZ failure 0 < 1 min Traffic shifts to remaining AZs; CM auto-reconnect
Region failure < 1 min < 5 min DNS failover; users reconnect to nearest healthy region
Kafka cluster failure 0 < 2 min Kafka controller election; consumers reconnect
Cassandra cluster failure < 1 min < 5 min Replica promotion; repair on recovery
Redis cluster failure Lost presence < 1 min Presence rebuilds from connections; typing/receipts lost (acceptable)
CDN failure N/A < 5 min Media served from origin; increased latency
Complete outage Last replicated < 15 min Full region failover; gradual service restoration

10.4 Monitoring and Alerting

Golden Signals:

  • Latency: P50, P95, P99 message delivery time (client-to-client)
  • Traffic: Messages/sec, connections/sec, active connections
  • Errors: Message delivery failure rate, connection drop rate, encryption errors
  • Saturation: CM connection utilization, Kafka consumer lag, DB write latency

Messaging-Specific Metrics:

  • Message delivery success rate (target: > 99.9%)
  • Message delivery latency P95 (target: < 500ms)
  • Offline message queue depth
  • Push notification delivery rate
  • E2EE key exchange success rate
  • Group fan-out latency
  • Contact discovery match rate

Business Metrics:

  • Daily active users (DAU)
  • Messages per DAU
  • Average session duration
  • User retention (D1, D7, D30)
  • Churn rate

Alerting Strategy:

  • Page: Message delivery failure > 1%, connection drop rate > 5%, Kafka lag > 100K
  • Ticket: Latency P99 > 2s, push delivery rate < 95%, media upload failure > 2%
  • Dashboard: Real-time metrics in Grafana; SLO burn rate monitoring

Appendix A: Technology Stack Summary

Layer Technology Rationale
Client Native (Kotlin/Swift), Web (React) Performance, OS integration, push notifications
Connection WebSocket (Netty / custom) Persistent connections, low latency, binary frames
API Gateway Custom (Go) Lightweight, high throughput, WebSocket-native
Services Go, Java Go for connection-heavy services; Java for complex business logic
Message Queue Apache Kafka High throughput, durability, replayability, partition ordering
Message Store Apache Cassandra Write-optimized, horizontal scaling, per-partition ordering
Offline Store ScyllaDB C++-based Cassandra compat, higher throughput, lower latency
Cache Redis Cluster Sub-ms latency, pub/sub, rich data structures
Key Store HBase Wide-column, scalable key-value for pre-key storage
Object Storage S3 / OSS Scalable media storage, CDN integration
CDN CloudFront / Cloudflare Global edge caching for media
Push FCM (Android), APNs (iOS) OS-level push delivery
Container Orchestration Kubernetes Auto-scaling, self-healing, multi-tenant
Monitoring Prometheus + Grafana + Jaeger Metrics, dashboards, distributed tracing
Crypto libsignal (Signal Protocol) Battle-tested E2EE implementation

Appendix B: Capacity Planning Summary

Metric Steady State Peak (3x) Headroom Target
Concurrent Connections 200M 300M 400M
Message Deliveries/sec 833K 2.5M 3M
WebSocket Frames/sec 5M 15M 20M
Kafka Throughput 1M msg/sec 3M msg/sec 5M msg/sec
Cassandra Writes/sec 416K 1.2M 2M
Redis Ops/sec 2M 6M 10M
CDN Bandwidth 100 MB/s 300 MB/s 500 MB/s

Appendix C: Security Architecture Summary

Control Implementation
Transport Security TLS 1.3 for all client-server communication
Message Encryption Signal Protocol (Double Ratchet) — AES-256-GCM
Key Exchange X3DH with pre-keys
Forward Secrecy Yes (ratchet advances per message)
Future Secrecy Yes (ratchet reset on each new key exchange)
Deniability Yes (no long-term keys can prove message origin)
Authentication Phone number + OTP; device registration with key attestation
Integrity HMAC on all protocol frames; signature verification on keys
Metadata Protection Minimal metadata retention; encrypted at rest
Access Control RBAC for internal systems; mTLS for service-to-service
Audit Immutable audit logs for admin actions
Compliance GDPR, CCPA, SOC 2, ISO 27001