Design a Chat Application like WhatsApp
Author: System Design Expert | Date: 2026-08-28 Scope: Real-time messaging with presence, read receipts, media sharing, group chats, end-to-end encryption, and offline message delivery.
Table of Contents
- Requirements Clarification
- Back-of-Envelope Calculations
- System Interface Design
- High-Level Architecture
- Core Component Design
- Data Model
- Key Design Decisions
- Scalability Strategy
- Curveball Scenarios
- Failure Modes and Mitigations
1. Requirements Clarification
1.1 Functional Requirements
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | User Registration: Phone number-based registration with SMS/voice OTP verification | P0 |
| FR-02 | Contact Discovery: Sync phone contacts, discover which contacts are on the platform | P0 |
| FR-03 | One-on-One Messaging: Send/receive text messages in real-time | P0 |
| FR-04 | End-to-End Encryption (E2EE): All messages encrypted client-side; server cannot decrypt | P0 |
| FR-05 | Message Delivery Status: Delivered (single check), read (double blue check) receipts | P0 |
| FR-06 | Presence & Typing Indicators: Online/offline status, “typing…” indicator | P0 |
| FR-07 | Offline Message Delivery: Store and forward messages when recipient is offline | P0 |
| FR-08 | Media Sharing: Images, videos, audio (voice notes), documents, GIFs, stickers | P1 |
| FR-09 | Group Chats: Create groups, add/remove members, admin controls, group info | P1 |
| FR-10 | Message History Sync: Sync chat history across multiple devices | P1 |
| FR-11 | Status/Stories: Ephemeral media/text posts visible for 24 hours | P1 |
| FR-12 | Call Signaling: Voice and video call setup (signaling only; media via WebRTC P2P) | P1 |
| FR-13 | Message Search: Search through message history by keyword | P2 |
| FR-14 | Message Reactions & Replies: React to messages with emojis, reply to specific messages | P1 |
| FR-15 | Block/Mute: Block users, mute conversations | P1 |
| FR-16 | Chat Backup & Restore: Cloud backup of chat history | P2 |
1.2 Non-Functional Requirements
| ID | Requirement | Target |
|---|---|---|
| NFR-01 | Message Latency | P95 < 500ms end-to-end (online-to-online) |
| NFR-02 | Availability | 99.95% uptime for messaging path |
| NFR-03 | Throughput | Support 2B+ MAU, 100B+ messages/day |
| NFR-04 | Security | E2EE (Signal Protocol), TLS 1.3 for transport, zero-knowledge architecture |
| NFR-05 | Privacy | Metadata minimization, no message content stored in plaintext on server |
| NFR-06 | Offline Support | Messages stored for 30 days; delivered when recipient comes online |
| NFR-07 | Battery Efficiency | Minimal battery drain on mobile devices (efficient keepalive, batching) |
| NFR-08 | Data Efficiency | Compressed messages, delta sync, adaptive media quality |
| NFR-09 | Consistency | Messages delivered in order per conversation; at-least-once delivery |
| NFR-10 | Scalability | Handle 10M+ concurrent WebSocket connections per region |
2. Back-of-Envelope Calculations
2.1 Assumption Sets
| Parameter | Conservative (A) | Base (B) | Aggressive (C) |
|---|---|---|---|
| Monthly Active Users (MAU) | 500M | 2B | 3B |
| Daily Active Users (DAU) | 300M | 1.2B | 1.8B |
| Messages / DAU | 20 | 30 | 40 |
| Avg. text message size | 200 bytes | 300 bytes | 400 bytes |
| Media messages / total | 10% | 15% | 20% |
| Avg. media message size | 500 KB | 800 KB | 1 MB |
| Concurrent connections | 50M | 200M | 300M |
| Avg. group size | 5 | 8 | 10 |
| Group messages / total | 15% | 20% | 25% |
| Fan-out multiplier (groups) | 1.5x | 2x | 2.5x |
2.2 Traffic Estimates (Base Case B)
Daily messages = DAU × messages/DAU = 1.2B × 30 = 36B messages/day
After group fan-out: 36B × 2x = 72B message deliveries/day
Steady-state message QPS:
72B / 86,400s ≈ 833,333 message deliveries/sec
Peak QPS (3x): ~2.5M message deliveries/sec
Text message bandwidth:
72B × 300 bytes = 21.6 TB/day text
72B × 0.15 × 800 KB = 8.6 TB/day media
Total: ~30 TB/day message data
2.3 Connection Estimates
Concurrent WebSocket connections: 200M
Each connection: ~2 KB memory (buffers + state)
Total connection memory: 200M × 2 KB = 400 GB
Connection keepalive:
Ping/pong every 30 seconds
200M / 30s ≈ 6.7M ping/pong frames/sec (bidirectional)
2.4 Storage Estimates
| Component | Calculation (Base) | Size |
|---|---|---|
| Offline messages (30-day retention) | 36B/day × 30 days × 300 bytes | ~324 TB |
| Media files (90-day retention) | 5.4B media/day × 90 days × 800 KB | ~389 PB |
| User profiles | 2B × 1 KB | ~2 GB |
| Contact lists | 2B × 100 contacts × 100 bytes | ~20 GB |
| Group metadata | 500M groups × 2 KB | ~1 TB |
| Message metadata (encrypted) | 36B/day × 30 days × 100 bytes | ~108 TB |
| Total | ~490 TB+ (media dominates) |
2.5 Bandwidth Estimates
Outbound message bandwidth:
Text: 21.6 TB/day = 250 MB/s average
Media: 8.6 TB/day = 100 MB/s average
Total: ~350 MB/s sustained
With CDN for media: 80% media served from CDN edge
Origin egress: 350 MB/s × 0.2 (media) + 250 MB/s (text) ≈ 290 MB/s
3. System Interface Design
3.1 Client-Server Protocol (WebSocket + REST)
REST APIs (Registration, Discovery, Sync)
POST /api/v1/auth/request-otp — Request SMS/voice OTP
POST /api/v1/auth/verify-otp — Verify OTP, register/login
POST /api/v1/auth/register-device — Register new device
DELETE /api/v1/auth/unregister-device — Unregister device
GET /api/v1/contacts/upload — Upload contact hash list
POST /api/v1/contacts/upload — Blobs of PHA hashes
GET /api/v1/contacts/status — Get online status of contacts
GET /api/v1/messages/pending — Fetch pending (offline) messages
POST /api/v1/messages/ack — Acknowledge message receipt
GET /api/v1/messages/history?chatId={id}&cursor={c} — Fetch history
GET /api/v1/groups/{groupId} — Group info
POST /api/v1/groups — Create group
POST /api/v1/groups/{groupId}/participants — Add participants
DELETE /api/v1/groups/{groupId}/participants/{userId} — Remove participant
POST /api/v1/media/upload — Upload media (returns URL)
GET /api/v1/media/{mediaId} — Download media (CDN redirect)
GET /api/v1/status/list — List available statuses
POST /api/v1/status/upload — Upload status
POST /api/v1/status/view/{statusId} — Record status view
WebSocket Protocol (Real-Time Messaging)
Client → Server Frames:
{ "type": "MESSAGE", "chatId": "...", "payload": "<encrypted>", "timestamp": ..., "messageId": "..." }
{ "type": "TYPING", "chatId": "...", "isTyping": true/false }
{ "type": "READ_RECEIPT", "chatId": "...", "messageId": "..." }
{ "type": "DELIVERY_RECEIPT", "chatId": "...", "messageId": "..." }
{ "type": "PRESENCE", "status": "online/offline" }
{ "type": "PONG", "timestamp": ... }
{ "type": "REACTION", "chatId": "...", "messageId": "...", "emoji": "..." }
Server → Client Frames:
{ "type": "MESSAGE", "chatId": "...", "senderId": "...", "payload": "<encrypted>", "timestamp": ..., "messageId": "..." }
{ "type": "TYPING", "chatId": "...", "userId": "...", "isTyping": true/false }
{ "type": "READ_RECEIPT", "chatId": "...", "messageId": "...", "readerId": "..." }
{ "type": "DELIVERY_RECEIPT", "chatId": "...", "messageId": "..." }
{ "type": "PRESENCE", "userId": "...", "status": "online/offline", "lastSeen": ... }
{ "type": "PING", "timestamp": ... }
{ "type": "GROUP_UPDATE", "groupId": "...", "change": "..." }
{ "type": "DEVICE_MESSAGE", "payload": "<encrypted>" } — Cross-device sync
3.2 Internal gRPC Interfaces
// Message Delivery Service
service MessageDeliveryService {
rpc Deliver(DeliverRequest) returns (DeliverResponse);
rpc StoreOffline(StoreOfflineRequest) returns (StoreOfflineResponse);
rpc GetPending(GetPendingRequest) returns (stream GetPendingResponse);
}
// Presence Service
service PresenceService {
rpc UpdatePresence(PresenceRequest) returns (PresenceResponse);
rpc GetPresence(GetPresenceRequest) returns (GetPresenceResponse);
rpc SubscribePresence(SubscribePresenceRequest) returns (stream PresenceUpdate);
}
// Contact Discovery Service
service ContactDiscoveryService {
rpc UploadHashes(UploadHashesRequest) returns (UploadHashesResponse);
rpc LookupMatches(LookupMatchesRequest) returns (LookupMatchesResponse);
}
// Key Management Service
service KeyManagementService {
rpc UploadPublicKey(UploadPublicKeyRequest) returns (UploadPublicKeyResponse);
rpc GetPublicKey(GetPublicKeyRequest) returns (GetPublicKeyResponse);
rpc UploadPreKeys(UploadPreKeysRequest) returns (UploadPreKeysResponse);
rpc GetPreKeys(GetPreKeysRequest) returns (GetPreKeysResponse);
}
4. High-Level Architecture
flowchart TB
subgraph Clients["Client Layer"]
WA[WhatsApp Android]
WB[WhatsApp iOS]
WW[WhatsApp Web/Desktop]
WC[WhatsApp Business]
end
subgraph Edge["Edge Layer"]
CDN[CDN<br/>(Media, Static Assets)]
LB[Global Load Balancer<br/>(Geo-routing)]
end
subgraph Connection["Connection Layer"]
subgraph ConnPool["Connection Managers"]
CM1[Connection Manager Pool 1]
CM2[Connection Manager Pool 2]
CM3[Connection Manager Pool N]
end
Router[Connection Router<br/>(User → CM mapping)]
end
subgraph Services["Service Layer"]
subgraph MsgSvc["Message Service"]
end
subgraph PresSvc["Presence Service"]
end
subgraph KeySvc["Key Management Service"]
end
subgraph ContactSvc["Contact Discovery Service"]
end
subgraph GroupSvc["Group Service"]
end
subgraph MediaSvc["Media Service"]
end
subgraph StatusSvc["Status Service"]
end
subgraph NotifSvc["Push Notification Service"]
end
subgraph AuthSvc["Auth Service"]
end
end
subgraph Messaging["Messaging Backbone"]
K[Apache Kafka<br/>(Message Events)]
PQ[Priority Queue<br/>(Offline Messages)]
end
subgraph Data["Data Layer"]
subgraph Databases["Databases"]
Cassandra[(Cassandra<br/>Messages, Metadata)]
Scylla[(ScyllaDB<br/>Offline Message Store)]
end
subgraph Cache["Cache"]
Redis1[Redis Cluster 1<br/>(Presence, Sessions)]
Redis2[Redis Cluster 2<br/>(Typing, Read Receipts)]
end
subgraph Storage["Object Storage"]
S3[S3 / OSS<br/>(Encrypted Media)]
end
subgraph KeyStore["Key Store"]
HBase[(HBase<br/>E2EE Keys, PreKeys)]
end
end
subgraph Infra["Infrastructure"]
Monitor[Monitoring & Alerting]
Trace[Distributed Tracing]
Security[Security & Compliance]
end
WA --> LB
WB --> LB
WW --> LB
WC --> LB
LB --> CM1
LB --> CM2
LB --> CM3
CM1 --> Router
CM2 --> Router
CM3 --> Router
Router --> MsgSvc
Router --> PresSvc
Router --> KeySvc
Router --> ContactSvc
Router --> GroupSvc
Router --> MediaSvc
Router --> StatusSvc
MsgSvc --> K
PresSvc --> Redis1
PresSvc --> Redis2
K --> PQ
K --> NotifSvc
MsgSvc --> Cassandra
MsgSvc --> Scylla
PresSvc --> Redis1
KeySvc --> HBase
MediaSvc --> S3
GroupSvc --> Cassandra
ContactSvc --> Cassandra
AuthSvc --> Cassandra
StatusSvc --> S3
5. Core Component Design
5.1 Connection Manager (WebSocket Gateway)
Purpose: Maintain persistent WebSocket connections with clients; route messages to/from services.
Design:
- Protocol: WebSocket over TLS 1.3; fallback to long-polling for restrictive networks
- Connection lifecycle:
Connect → Authenticate (JWT/token) → Subscribe to user's message queue → Active Active ↔ Send/Receive frames Idle (>5 min no activity) → Throttle keepalive Disconnect → Mark user offline → Drain pending messages - User-to-Connection mapping: Redis-based consistent hashing
user:{userId} → connectionManager:{cmId}:connection:{connId}- TTL-based; refreshed on each ping/pong
- Connection pooling: Each CM instance handles ~500K concurrent connections
- 200M connections / 500K per CM = 400 CM instances
- Keepalive strategy:
- Server sends PING every 30 seconds
- Client must respond with PONG within 10 seconds
- After 3 missed pongs, connection considered dead
- Battery optimization:
- Adaptive keepalive: Increase interval when device is on battery + idle
- Message batching: Coalesce multiple messages into single frame when possible
Connection Manager Memory Budget:
Per connection:
- Socket buffer: 64 KB (send) + 64 KB (receive)
- Session state: 2 KB
- Frame buffers: 16 KB
Total per connection: ~146 KB
400 CMs × 500K connections × 146 KB = ~29 TB total connection memory
5.2 Message Service
Purpose: Core message routing, delivery, and offline storage.
Message Flow (Online-to-Online):
sequenceDiagram
participant A as Sender Device
participant CMA as CM (Sender)
participant MS as Message Service
participant K as Kafka
participant CMB as CM (Recipient)
participant B as Recipient Device
A->>CMA: WebSocket: MESSAGE (encrypted payload)
CMA->>MS: Deliver message
MS->>MS: Validate, assign messageId, timestamp
MS->>K: Publish to user-{recipientId} topic
MS-->>CMA: ACK
CMA-->>A: Delivery receipt (single check)
K->>MS: Consume from user-{recipientId}
MS->>MS: Lookup recipient connection
MS->>CMB: Push message
CMB->>B: WebSocket: MESSAGE
B->>CMB: DELIVERY_RECEIPT
CMB->>MS: Ack delivery
MS->>K: Publish delivery event
K->>MS: Consume delivery event
MS->>CMA: Forward delivery receipt
CMA->>A: Delivery receipt update
Message Flow (Online-to-Offline):
1. Sender sends message → Message Service receives
2. Message Service checks recipient presence (Redis)
3. Recipient offline → Store in ScyllaDB (offline message store)
4. Publish event to Kafka for push notification trigger
5. Push Notification Service sends FCM/APNs notification
6. When recipient comes online:
a. Client fetches pending messages via REST API
b. OR Connection Manager pushes queued messages on reconnect
7. Messages deleted from offline store after delivery + grace period
Message Ordering:
- Per-conversation ordering guaranteed via sequence numbers
- Each message carries
(chatId, senderId, sequenceNumber)tuple - Client-side deduplication using
messageId - Server-side: Kafka partition key =
chatIdensures per-conversation ordering
Message Deduplication:
- Idempotent delivery via
messageId(UUID v4 generated by sender) - Server tracks delivered message IDs in a sliding window (Redis Bloom filter)
- Duplicate messages silently dropped
5.3 End-to-End Encryption (Signal Protocol Implementation)
Purpose: Ensure only sender and recipient can read message contents.
Protocol: Double Ratchet Algorithm (Signal Protocol)
Key Exchange Flow:
1. Each user generates:
- Identity Key pair (long-term, stored on device)
- Signed Pre-Key (signed by Identity Key, uploaded to server)
- 100 One-Time Pre-Keys (uploaded to server)
- Reserved Pre-Key bundle
2. When User A messages User B for the first time:
a. A fetches B's Signed Pre-Key + one One-Time Pre-Key from server
b. A performs X3DH (Extended Triple Diffie-Hellman):
Shared Secret = DH(A.identityPriv, B.signedPreKeyPub)
⊕ DH(A.identityPriv, B.oneTimePreKeyPub)
⊕ DH(A.oneTimePreKeyPriv, B.signedPreKeyPub)
c. Shared Secret → SKDF → Initial Messaging Key + Chain Key
d. Double Ratchet takes over for subsequent messages
3. Each message:
- Ratchet forward (chain key → message key)
- Encrypt payload with message key (AES-256-GCM)
- Include ratchet info for remote key synchronization
Server’s Role:
- Stores pre-keys (encrypted, server cannot decrypt messages)
- Routes encrypted blobs transparently
- Never sees plaintext message content
- Key bundles deleted after use (one-time pre-keys)
Key Storage:
HBase Table: signal_prekeys
Row Key: userId
Columns:
signed_prekey: {keyId, publicKey, signature}
one_time_prekeys: [{keyId, publicKey}, ...] (100 keys)
identity_key: {publicKey}
device_id: int
Multi-Device Support:
- Primary device holds the root key
- Secondary devices receive encrypted key bundles via primary
- Each device has its own DH key pair; messages encrypted to all active devices
5.4 Presence Service
Purpose: Track user online/offline status and last seen timestamp.
Design:
- Storage: Redis Cluster with geo-partitioning
Key: presence:{userId} Value: { "status": "online/offline", "lastSeen": timestamp, "cmId": "..." } TTL: 2 minutes (auto-expire → offline) - Status transitions:
Connected → Online (publish to subscribers) Idle > 2 min → Away (optional) Disconnected → Offline (publish to subscribers) - Privacy settings:
- “Everyone can see my last seen”
- “My contacts only”
- “Nobody”
- Enforced at query time, not storage time
- Subscription model:
- Users subscribe to presence of their contacts
- Stored in Redis:
presence_subscribers:{userId} → Set(contactId1, contactId2, ...) - On status change, publish to all subscribers via pub/sub
Typing Indicator:
Key: typing:{chatId}:{userId}
Value: true
TTL: 5 seconds
When client sends typing frame:
1. Set typing:{chatId}:{userId} = true, TTL 5s
2. Publish to chat subscribers via Redis pub/sub
3. Recipients show "typing..." indicator
TTL expiry → typing stops automatically
5.5 Read Receipts Service
Purpose: Track message delivery and read status.
Design:
-
Delivery receipt (single gray check):
- Generated when message reaches recipient’s device (Connection Manager)
- Stored in Redis:
receipt:delivered:{messageId} → {userId, timestamp} - Forwarded to sender via Kafka event
-
Read receipt (double blue check):
- Generated when recipient’s app brings conversation to foreground
- Client sends
READ_RECEIPTframe → Message Service → Kafka → Sender’s CM → Sender - Stored in Cassandra for history:
read_receipts(messageId, readerId, timestamp)
-
Privacy:
- Users can disable read receipts in settings
- When disabled, read events not sent; sender sees single check only
- Group chats: Show count of members who read (“Read by 3/5”)
5.6 Contact Discovery Service
Purpose: Let users find which of their phone contacts are on WhatsApp, without exposing privacy.
Design (Privacy-Preserving):
1. Client side:
a. Read phone contact list (user-granted permission)
b. Hash each phone number: PHA = SHA-256(phoneNumber)
c. Upload PHA hashes in batches (blobs) to server
2. Server side:
a. Store registered user PHAs in Cassandra
b. Compute intersection: uploaded PHAs ∩ registered PHAs
c. Return matching user IDs (with profile info)
3. Privacy guarantees:
- Server sees PHAs, not raw phone numbers
- PHA blobs deleted after processing
- Rate limited: Max N contact uploads per day
PHA Upload Protocol:
POST /api/v1/contacts/upload
Content-Type: application/octet-stream
Body: Encrypted blob containing PHA hashes
- Encrypted with server's public key
- Contains: [PHA1, PHA2, ..., PHAN]
- Max 500 PHAs per request
5.7 Group Chat Service
Purpose: Manage group creation, membership, and message fan-out.
Design:
-
Group model:
{ "groupId": "gid-abc123", "name": "Team Project", "creatorId": "user-001", "admins": ["user-001", "user-002"], "participants": ["user-001", "user-002", "user-003", ...], "participantCount": 45, "createdAt": "2026-08-01T10:00:00Z", "settings": { "onlyAdminsCanSend": false, "approvalRequired": false, "description": "Project discussion group" } } -
Max group size: 1,024 participants (standard), 2,000 (community)
-
Message fan-out:
- When user sends message to group, Message Service fans out to all participants
- Fan-out happens asynchronously via Kafka
- Each participant receives the message on their personal topic
- Offline participants get stored + push notification
-
Group E2EE:
- Each group has its own ratchet state per participant
- Sender encrypts message once per recipient (N encryptions for N participants)
- For large groups, use broadcast channels (optimized encryption)
-
Group events:
- Member added/removed → System message in group chat
- Name changed → System message
- Admin changed → System message
- All events go through same message pipeline (encrypted system messages)
5.8 Media Service
Purpose: Handle upload, storage, and delivery of media files.
Design:
-
Upload flow:
1. Client compresses/optimizes media (client-side) 2. Client encrypts media with session key (E2EE for media too) 3. Client uploads encrypted blob to Media Service 4. Media Service stores in S3/OSS with metadata 5. Media Service returns media URL + metadata to Message Service 6. Message Service sends message with media reference to recipient 7. Recipient downloads from CDN (cached encrypted blob) 8. Recipient decrypts with session key -
Media optimization:
- Images: WebP format, adaptive quality based on network
- Videos: H.264, multiple bitrate versions (client selects)
- Audio: Opus codec for voice notes
- Documents: Stored as-is, size limit 100 MB
-
Storage lifecycle:
- Media retained for 90 days (configurable)
- After 90 days, media deleted; message shows “Media expired”
- “Keep in Chat” option downloads to device permanently
-
CDN integration:
- Encrypted media blobs cached at CDN edge
- Cache key includes media ID + encryption version
- TTL: 24 hours at edge, 7 days at origin shield
5.9 Push Notification Service
Purpose: Notify offline users of new messages via FCM (Android) and APNs (iOS).
Design:
- Trigger: Kafka consumer subscribes to
message.deliveredevents where recipient is offline - Notification content:
- With E2EE: Notification shows sender name + “Message” (no content)
- If user enables “Show message preview”: Content encrypted with notification key (separate from message key)
- Provider integration:
- Android: Firebase Cloud Messaging (FCM)
- iOS: Apple Push Notification Service (APNs)
- Batching: Coalesce multiple messages into single notification (“3 new messages from Alice”)
- Silent notifications: Data-only notifications to wake app without showing UI
5.10 Status/Stories Service
Purpose: Ephemeral content visible for 24 hours.
Design:
- Upload: Similar to media service but with 24-hour TTL
- Viewer list: Track who viewed each status (stored in Redis, TTL 24h)
- Delivery: Push to all contacts when status is posted
- Expiration: Background job deletes expired statuses and viewer lists
- Privacy: Status visibility settings (Everyone, My Contacts, Only Share With, My Except)
6. Data Model
6.1 Cassandra Schema (Messages, Metadata)
-- User Profiles
CREATE TABLE user_profiles (
user_id TEXT PRIMARY KEY,
phone_number TEXT,
display_name TEXT,
profile_picture_url TEXT,
about_text TEXT,
privacy_settings FROZEN<PrivacySettings>,
created_at TIMESTAMP,
updated_at TIMESTAMP
) WITH COMPACT STORAGE;
TYPE PrivacySettings (
last_seen TEXT,
profile_photo TEXT,
status TEXT,
read_receipts BOOLEAN,
group_add TEXT
);
-- Messages (partitioned by chatId for per-conversation ordering)
CREATE TABLE messages (
chat_id TEXT,
sequence_num BIGINT,
message_id TEXT,
sender_id TEXT,
timestamp BIGINT,
encrypted_payload BLOB,
message_type TEXT, -- 'text', 'image', 'video', 'audio', 'document', 'system'
media_metadata FROZEN<MediaMetadata>,
ttl_days INT,
PRIMARY KEY (chat_id, sequence_num)
) WITH CLUSTERING ORDER BY (sequence_num ASC)
AND gc_grace_seconds = 86400;
TYPE MediaMetadata (
media_id TEXT,
media_url TEXT,
thumbnail_url TEXT,
mime_type TEXT,
file_size BIGINT,
width INT,
height INT,
duration INT
);
-- Offline Message Store (partitioned by recipient for efficient fetch)
CREATE TABLE offline_messages (
recipient_id TEXT,
timestamp BIGINT,
message_id TEXT,
chat_id TEXT,
sender_id TEXT,
encrypted_payload BLOB,
message_type TEXT,
media_metadata FROZEN<MediaMetadata>,
delivered BOOLEAN,
PRIMARY KEY (recipient_id, timestamp, message_id)
) WITH CLUSTERING ORDER BY (timestamp ASC);
-- Chat Metadata
CREATE TABLE chat_metadata (
chat_id TEXT PRIMARY KEY,
chat_type TEXT, -- 'direct', 'group'
participant_ids SET<TEXT>,
created_at TIMESTAMP,
updated_at TIMESTAMP,
last_message_timestamp BIGINT,
last_message_preview TEXT,
settings FROZEN<ChatSettings>
);
TYPE ChatSettings (
name TEXT,
description TEXT,
group_icon_url TEXT,
only_admins_can_send BOOLEAN,
admins SET<TEXT>
);
-- Read Receipts
CREATE TABLE read_receipts (
message_id TEXT,
reader_id TEXT,
read_at TIMESTAMP,
PRIMARY KEY (message_id, reader_id)
);
-- Delivery Receipts
CREATE TABLE delivery_receipts (
message_id TEXT PRIMARY KEY,
recipients MAP<TEXT, TIMESTAMP>
) WITH COMPACT STORAGE;
-- Contact Relationships
CREATE TABLE contacts (
user_id TEXT,
contact_id TEXT,
status TEXT, -- 'active', 'blocked', 'muted'
muting_until TIMESTAMP,
starred BOOLEAN,
unread_count INT,
PRIMARY KEY (user_id, contact_id)
);
-- Group Memberships
CREATE TABLE group_memberships (
group_id TEXT,
user_id TEXT,
role TEXT, -- 'admin', 'member'
joined_at TIMESTAMP,
primary_key_list TEXT, -- E2EE key for group
PRIMARY KEY (group_id, user_id)
);
6.2 Redis Data Structures
# Presence
SET presence:{userId} '{"status":"online","lastSeen":1724839200,"cmId":"cm-42"}' EX 120
# Presence Subscribers
SADD presence_subscribers:{userId} contactId1 contactId2 contactId3
# Typing Indicator
SET typing:{chatId}:{userId} "true" EX 5
# User-to-Connection Mapping
SET conn:{userId} '{"cmId":"cm-42","connId":"conn-abc"}' EX 60
# Message Deduplication (Bloom Filter)
BF.ADD dedup:{userId} messageId1 messageId2 messageId3
# Read Receipts (short-term cache)
HSET receipts:read:{chatId} messageId1 userId1 messageId2 userId2
# Rate Limiting
INCR rate:{userId}:message
EXPIRE rate:{userId}:message 60
# Push Notification Queue
LPUSH push_queue:{userId} '{"senderId":"...","chatId":"...","preview":"..."}'
6.3 HBase Schema (E2EE Keys)
Table: signal_prekeys
Row Key: {userId}_{deviceId}
Column Families:
id: identity_key_pub, identity_key_sig
spk: signed_prekey_{keyId} → {publicKey, signature, timestamp}
otp: one_time_prekey_{keyId} → {publicKey, timestamp}
info: next_otp_key_id, uploaded_at
6.4 Kafka Topics
Topic: message.{recipientId} — Per-user message delivery (partitioned by chatId)
Partition key: chatId
Value: {messageId, senderId, chatId, encryptedPayload, timestamp, messageType}
Topic: message.delivery — Delivery receipt events
Value: {messageId, recipientId, deliveredAt}
Topic: message.read — Read receipt events
Value: {messageId, readerId, readAt}
Topic: presence.change — Presence status changes
Value: {userId, status, lastSeen, cmId}
Topic: group.event — Group membership changes
Value: {groupId, eventType, userId, timestamp}
Topic: push.notification — Push notification triggers
Value: {userId, senderId, chatId, preview, timestamp}
Topic: media.upload — Media upload events
Value: {mediaId, userId, url, mimeType, size}
7. Key Design Decisions
7.1 Trade-off Analysis
| Decision | Option A | Option B | Choice | Rationale |
|---|---|---|---|---|
| Transport Protocol | HTTP/REST (polling) | WebSocket (persistent) | WebSocket | Real-time messaging requires push capability. Polling wastes bandwidth and battery. |
| Message Store | MySQL | Cassandra / ScyllaDB | Cassandra | Write-heavy, high throughput, wide row per conversation. Cassandra’s partition model fits perfectly. |
| Offline Store | Same DB as messages | Separate ScyllaDB cluster | Separate ScyllaDB | Offline messages have different access patterns (bulk fetch on reconnect). Separation prevents interference. |
| E2EE Protocol | Custom encryption | Signal Protocol (Double Ratchet) | Signal Protocol | Battle-tested, open-source, forward secrecy + future secrecy. No reason to reinvent. |
| Presence Storage | Database | Redis | Redis | Presence is ephemeral, high-frequency updates. Redis sub-ms latency is essential. |
| Contact Discovery | Plain phone numbers | PHA (Privacy-Enhanced Address) | PHA hashes | Privacy requirement. Server should not see raw phone numbers. |
| Group Fan-out | Sync fan-out | Async via Kafka | Async Kafka | Sync fan-out to 1000+ members would block sender. Async ensures sender gets fast ACK. |
| Media Storage | Database BLOB | Object Storage (S3) | S3 + CDN | Media is large, infrequently accessed after delivery. Object storage + CDN is cost-effective. |
| Push Notifications | In-house | FCM / APNs | FCM / APNs | OS-level integration required for reliable delivery. No viable alternative. |
| Message Ordering | Global sequence | Per-conversation sequence | Per-conversation | Global ordering impossible at scale. Per-conversation ordering is what users perceive. |
7.2 E2EE Architecture Implications
The E2EE requirement fundamentally shapes the architecture:
- Server is a blind relay: Cannot inspect, filter, or search message content
- Spam/abuse moderation: Must use client-side signals (user reports, metadata patterns) rather than content analysis
- Backup/restore: Encrypted backups only; server cannot restore without user’s key
- Multi-device: Requires key distribution protocol; primary device is trust anchor
- Legal compliance: Cannot comply with lawful intercept for message content (only metadata)
8. Scalability Strategy
8.1 Connection Scaling
Target: 200M concurrent connections
Connection Manager sizing:
- Per CM capacity: 500K connections
- Required CMs: 200M / 500K = 400 instances
- Per region: ~100 CMs (4 regions)
CM hardware:
- 64 vCPU, 256 GB RAM
- 500K connections × 146 KB = ~73 GB/connection memory
- Headroom: 256 GB / 73 GB = 3.5x
Auto-scaling:
- Scale CM pool based on connection count
- Target utilization: 60% (300K connections per CM)
- Scale-up threshold: > 400K connections per CM
- Scale-down threshold: < 200K connections per CM
8.2 Message Throughput Scaling
Target: 833K message deliveries/sec steady-state
Kafka sizing:
- Per partition throughput: ~50K msg/sec
- User topics: 2B users (one topic per user is too many)
- Solution: Topic hierarchy
- 10,000 aggregate topics (user hash mod 10,000)
- Each topic: 64 partitions
- Total partitions: 640,000
- Per partition: 833K / 640K ≈ 1.3K msg/sec (well within capacity)
Consumer groups:
- Message Service consumers: 640K partitions / 4 consumers per partition = 256K consumer instances
- Actually: Over-provision with 4x = 1M consumer instances across all CMs
- Each CM consumes for its connected users only (filtered consumption)
8.3 Database Scaling
Cassandra (Messages):
- Write rate: 36B messages/day = 416K msg/sec
- Partition key: chatId
- Hot partition risk: Popular group chats
- Mitigation: Sub-partitioning for groups > 500 members
- Cluster: 500 nodes, 3 replicas
- Per node write capacity: 416K / (500 × 3) ≈ 280 writes/sec (trivial)
ScyllaDB (Offline Messages):
- Write rate: Same as message rate (all messages stored)
- Read pattern: Bursty (user reconnects, fetches all pending)
- Partition key: recipientId
- Cluster: 300 nodes, 2 replicas
- TTL-based auto-expiry (30 days)
Redis (Presence):
- 2B users × 1 entry = 2B keys
- Cluster: 1,000 shards across 500 nodes
- Per shard: 2M keys (well within capacity)
8.4 Multi-Region Deployment
Region Strategy: Active-Active with user affinity
Users assigned to home region based on:
- Registration location (SIM country code)
- Persistent assignment (stuck to home region for data consistency)
Cross-region messaging:
- Sender in US, recipient in EU
- Message routed via Kafka cross-region replication
- Latency: ~100ms additional for cross-region
Regional data sovereignty:
- User data stays in home region
- Complies with GDPR, local data residency laws
8.5 Media Scaling
Media CDN:
- Global CDN with 2,000+ edge locations
- Origin shield layers: Edge → Regional → Origin
- Cache hit rate target: 95%+ at edge
- Bandwidth: 8.6 TB/day media × 0.95 CDN hit = 0.43 TB/day origin
Media storage lifecycle:
Hot (0-7 days): SSD storage, CDN cached
Warm (7-30 days): Standard storage, CDN cached
Cold (30-90 days): Infrequent access storage
Expired (90+ days): Deleted
9. Curveball Scenarios
Curveball 1: “A viral message is forwarded to 100,000 groups simultaneously. How do you handle the fan-out?”
Analysis:
- A single message fanning out to 100K groups × avg 8 members = 800K deliveries
- This is a fan-out amplification problem
- The bottleneck is the Message Service writing to 800K user queues
Response:
-
Detection:
- Monitor fan-out ratio per message (deliveries / 1)
- Alert when fan-out > 10K
-
Handling:
- Async fan-out with rate limiting:
- Group fan-out batches: 1,000 groups per batch
- Each batch processed by dedicated fan-out workers
- Rate limit: Max 50K fan-out operations/sec per message
- Priority queuing:
- Direct messages: Highest priority
- Small groups (< 50): High priority
- Large groups (> 500): Normal priority
- Broadcast channels: Lower priority
- Sender experience:
- Sender gets immediate ACK (message accepted)
- Delivery receipts arrive asynchronously
- No blocking on fan-out completion
- Async fan-out with rate limiting:
-
Prevention:
- Forward limits: Max N forwards per 24 hours (WhatsApp already does this)
- “Forwarded many times” label for widely forwarded messages
- Rate limiting on forward actions per user
-
Infrastructure:
- Dedicated Kafka topic for high-fan-out messages
- Separate consumer pool for fan-out processing
- Auto-scale fan-out workers based on queue depth
Curveball 2: “A zero-day vulnerability is discovered in the Signal Protocol implementation. How do you respond?”
Analysis:
- E2EE protocol compromise is the highest-severity scenario
- All message confidentiality potentially breached
- Must balance security response with service continuity
Response:
-
Immediate actions (0-1 hour):
- Security team: Assess scope and severity
- Key rotation: Force ratchet reset for all active sessions
- Pre-key refresh: Invalidate all one-time pre-keys; force regeneration
- Client update: Emergency OTA update with patched protocol
-
User communication:
- In-app notification about security update
- Transparent communication without causing panic
- Clear instructions for manual key reset
-
Technical response:
- Server-side: Delete all stored pre-keys; force re-upload on next client connection
- Session reset: On next message send, trigger fresh X3DH key exchange
- Backward compatibility: Patched clients can communicate with unpatched clients using fallback mode (with warning)
-
Long-term:
- Independent security audit of protocol implementation
- Bug bounty program expansion
- Formal verification of critical crypto code paths
- Consider multi-protocol support for redundancy
Curveball 3: “A government demands backdoor access to messages for national security. How do you handle it?”
Analysis:
- Fundamental conflict between E2EE architecture and backdoor requirements
- A backdoor for “bad actors” is a backdoor for everyone
- Legal, ethical, and technical dimensions
Response:
-
Technical reality:
- Cannot comply with message content backdoor: E2EE by design prevents server-side access
- Any backdoor weakens security for all users
- Historical evidence: Backdoors are always exploitable beyond intended use
-
What can be provided (metadata, not content):
- User registration data (phone number, registration date)
- Contact list metadata (who communicated with whom, when)
- Group membership lists
- Last seen timestamps
- Cannot provide: Message content, media content, call content
-
Legal process:
- Require valid legal orders (warrants, court orders)
- Legal team review before any data disclosure
- User notification where legally permitted
- Transparency reports published regularly
-
Architecture defense:
- Minimize metadata retention (auto-delete after N days)
- Encrypt metadata at rest where possible
- Design for “plausible deniability” where legally feasible
Curveball 4: “The app is banned in a country with 200M users. How do you handle the transition?”
Analysis:
- 200M users lose access overnight
- Data retention and deletion requirements
- Business and ethical considerations
Response:
-
Immediate impact assessment:
- 200M users × 30 messages/day = 6B messages/day lost
- Revenue impact from ads/business features
- User migration patterns (to competing apps)
-
User data handling:
- Data export: Allow users to export chat history before cutoff
- Grace period: 30-day wind-down for data export
- Data deletion: Comply with local data deletion requirements
- Encrypted backups: User-controlled; server cannot access
-
Technical measures:
- Geo-blocking at CDN and API Gateway level
- SIM-based detection for registration blocking
- Existing sessions: Allow to expire naturally or force disconnect
-
Alternative access (if ethically/legal permissible):
- Web-only access (no app store listing)
- Desktop client only
- Evaluate legal risks before enabling any workaround
Curveball 5: “A bug causes messages to be delivered out of order. Users are confused. How do you fix it?”
Analysis:
- Message ordering is critical for conversation experience
- Out-of-order delivery breaks conversation coherence
- Must fix both the bug and the user-facing experience
Response:
-
Root cause analysis:
- Likely causes: Kafka partition reassignment, consumer rebalancing, multi-path routing
- Check: Is ordering broken per-conversation or globally?
- Verify: Sequence numbers still assigned correctly?
-
Immediate mitigation:
- Client-side reordering: Client buffers messages for 2 seconds, sorts by sequence number before display
- Server-side fix: Ensure single consumer per chatId partition
- Kafka partition
- Rollback: If caused by recent deployment, rollback immediately
-
User communication:
- In-app message: “You may see messages out of order temporarily”
- Status page update with ETA
-
Long-term prevention:
- Sequence number validation: Server rejects out-of-order sequence numbers
- Consumer assignment stability: Sticky partition assignment for consumers
- Testing: Chaos engineering tests for ordering under failure scenarios
- Monitoring: Real-time ordering violation detection and alerting
Curveball 6: “How do you detect and prevent child abuse material (CAM) without breaking E2EE?”
Analysis:
- Legal requirement in many jurisdictions
- Fundamental tension with E2EE promise
- Must find technical solution that doesn’t compromise all users’ privacy
Response:
-
Client-side scanning (controversial but technically feasible):
- PhotoDNA / perceptual hashing: Generate hash of images on device
- Hash comparison: Compare against known CAM hash database (stored on device)
- Reporting: If match found, flag to server (hash only, not image)
- Privacy: Hash database updated via signed OTA updates; auditable
-
User reporting (primary mechanism):
- “Report” button on every message and contact
- Reported content reviewed by trained moderators (user must consent to decrypt for review)
- Pattern detection: Multiple reports from different users on same content
-
Metadata analysis:
- Behavioral patterns indicative of abuse (without seeing content)
- Network analysis: Known bad actors, coordinated behavior
- Volume anomalies: Unusual message volumes to/from certain accounts
-
Legal compliance:
- Work with law enforcement on specific investigations
- Court-ordered client-side search tools (targeted, not mass surveillance)
- Transparency about capabilities and limitations
10. Failure Modes and Mitigations
10.1 Failure Mode Matrix
| Component | Failure Mode | Impact | Detection | Mitigation | RTO |
|---|---|---|---|---|---|
| Connection Manager | Process crash | Users disconnected, messages queued | Health check, connection drop spike | Auto-restart, reconnect to another CM | < 30s |
| Connection Manager | Memory exhaustion | New connections rejected | Memory > 90% alert | Connection draining, scale up | < 2min |
| Connection Manager | Network partition | Regional disconnect | Cross-AZ health checks | Traffic failover to healthy AZ | < 1min |
| Message Service | Kafka consumer lag | Message delivery delay | Consumer lag metric | Auto-scale consumers, add partitions | < 1min |
| Message Service | Kafka broker failure | Message loss risk | Broker down alert | ISR replication, auto-failover | < 30s |
| Message Service | Duplicate delivery | User sees duplicate messages | Client dedup report | Bloom filter dedup, idempotent delivery | Auto |
| Presence Service | Redis node failure | Presence stale for shard | Node down alert | Redis Cluster failover, rehash | < 30s |
| Presence Service | Redis memory full | New presence writes fail | Memory alert | Maxmemory policy, scale shards | Auto |
| Offline Store | ScyllaDB node failure | Offline messages unavailable | Node down alert | Replica failover, repair | < 1min |
| Offline Store | TTL expiry bug | Messages lost prematurely | Message count anomaly | TTL validation job, alert on anomaly | Manual |
| Key Management | HBase region server down | Key lookup failures | Region offline alert | Region server restart, split | < 2min |
| Key Management | Pre-key exhaustion | New sessions fail | Pre-key count < threshold | Auto-refill pre-keys on client connect | Auto |
| Media Service | S3 outage | Media upload/download fail | HTTP 5xx rate | Fallback to secondary storage, queue | < 5min |
| Media Service | CDN cache poisoning | Wrong media served | Content hash mismatch | Cache invalidation, short TTL | < 1min |
| Push Notifications | FCM/APNs outage | No push for offline users | Delivery rate drop | Retry queue, fallback to SMS (critical) | < 5min |
| Push Notifications | Token expiration | Push fails for users | FCM registration error | Token refresh on next app open | Auto |
| Contact Discovery | PHA collision | Wrong contact matches | User report | Extended PHA, collision detection | Manual |
10.2 Cascading Failure Prevention
flowchart LR
subgraph Patterns["Resilience Patterns"]
CB[Circuit Breaker<br/>Per-service, per-dependency]
B[Bulkhead<br/>Connection pools isolated]
R[Retry with Backoff<br/>Exponential + jitter]
T[Timeout<br/>Strict per-call timeouts]
Q[Queue-Based Leveling<br/>Kafka as buffer]
D[Graceful Degradation<br/>Drop non-critical features]
F[Feature Flags<br/>Kill switches per feature]
end
CB -->|Prevents| CF[Cascading Failures]
B -->|Contains| CF
R -->|Handles| TF[Transient Failures]
T -->|Prevents| RH[Thread/Connection Exhaustion]
Q -->|Absorbs| SP[Traffic Spikes]
D -->|Preserves| CP[Core Messaging]
F -->|Enables| QR[Quick Response]
Degradation Modes (priority order):
- Normal: All features active
- Elevated load: Disable typing indicators, reduce presence update frequency
- High load: Disable read receipts, disable status/stories, reduce push notification frequency
- Critical load: Text-only messaging, disable media, disable group fan-out (queue only)
- Emergency: Direct messages only, all groups paused
10.3 Disaster Recovery Plan
| Scenario | RPO | RTO | Procedure |
|---|---|---|---|
| AZ failure | 0 | < 1 min | Traffic shifts to remaining AZs; CM auto-reconnect |
| Region failure | < 1 min | < 5 min | DNS failover; users reconnect to nearest healthy region |
| Kafka cluster failure | 0 | < 2 min | Kafka controller election; consumers reconnect |
| Cassandra cluster failure | < 1 min | < 5 min | Replica promotion; repair on recovery |
| Redis cluster failure | Lost presence | < 1 min | Presence rebuilds from connections; typing/receipts lost (acceptable) |
| CDN failure | N/A | < 5 min | Media served from origin; increased latency |
| Complete outage | Last replicated | < 15 min | Full region failover; gradual service restoration |
10.4 Monitoring and Alerting
Golden Signals:
- Latency: P50, P95, P99 message delivery time (client-to-client)
- Traffic: Messages/sec, connections/sec, active connections
- Errors: Message delivery failure rate, connection drop rate, encryption errors
- Saturation: CM connection utilization, Kafka consumer lag, DB write latency
Messaging-Specific Metrics:
- Message delivery success rate (target: > 99.9%)
- Message delivery latency P95 (target: < 500ms)
- Offline message queue depth
- Push notification delivery rate
- E2EE key exchange success rate
- Group fan-out latency
- Contact discovery match rate
Business Metrics:
- Daily active users (DAU)
- Messages per DAU
- Average session duration
- User retention (D1, D7, D30)
- Churn rate
Alerting Strategy:
- Page: Message delivery failure > 1%, connection drop rate > 5%, Kafka lag > 100K
- Ticket: Latency P99 > 2s, push delivery rate < 95%, media upload failure > 2%
- Dashboard: Real-time metrics in Grafana; SLO burn rate monitoring
Appendix A: Technology Stack Summary
| Layer | Technology | Rationale |
|---|---|---|
| Client | Native (Kotlin/Swift), Web (React) | Performance, OS integration, push notifications |
| Connection | WebSocket (Netty / custom) | Persistent connections, low latency, binary frames |
| API Gateway | Custom (Go) | Lightweight, high throughput, WebSocket-native |
| Services | Go, Java | Go for connection-heavy services; Java for complex business logic |
| Message Queue | Apache Kafka | High throughput, durability, replayability, partition ordering |
| Message Store | Apache Cassandra | Write-optimized, horizontal scaling, per-partition ordering |
| Offline Store | ScyllaDB | C++-based Cassandra compat, higher throughput, lower latency |
| Cache | Redis Cluster | Sub-ms latency, pub/sub, rich data structures |
| Key Store | HBase | Wide-column, scalable key-value for pre-key storage |
| Object Storage | S3 / OSS | Scalable media storage, CDN integration |
| CDN | CloudFront / Cloudflare | Global edge caching for media |
| Push | FCM (Android), APNs (iOS) | OS-level push delivery |
| Container Orchestration | Kubernetes | Auto-scaling, self-healing, multi-tenant |
| Monitoring | Prometheus + Grafana + Jaeger | Metrics, dashboards, distributed tracing |
| Crypto | libsignal (Signal Protocol) | Battle-tested E2EE implementation |
Appendix B: Capacity Planning Summary
| Metric | Steady State | Peak (3x) | Headroom Target |
|---|---|---|---|
| Concurrent Connections | 200M | 300M | 400M |
| Message Deliveries/sec | 833K | 2.5M | 3M |
| WebSocket Frames/sec | 5M | 15M | 20M |
| Kafka Throughput | 1M msg/sec | 3M msg/sec | 5M msg/sec |
| Cassandra Writes/sec | 416K | 1.2M | 2M |
| Redis Ops/sec | 2M | 6M | 10M |
| CDN Bandwidth | 100 MB/s | 300 MB/s | 500 MB/s |
Appendix C: Security Architecture Summary
| Control | Implementation |
|---|---|
| Transport Security | TLS 1.3 for all client-server communication |
| Message Encryption | Signal Protocol (Double Ratchet) — AES-256-GCM |
| Key Exchange | X3DH with pre-keys |
| Forward Secrecy | Yes (ratchet advances per message) |
| Future Secrecy | Yes (ratchet reset on each new key exchange) |
| Deniability | Yes (no long-term keys can prove message origin) |
| Authentication | Phone number + OTP; device registration with key attestation |
| Integrity | HMAC on all protocol frames; signature verification on keys |
| Metadata Protection | Minimal metadata retention; encrypted at rest |
| Access Control | RBAC for internal systems; mTLS for service-to-service |
| Audit | Immutable audit logs for admin actions |
| Compliance | GDPR, CCPA, SOC 2, ISO 27001 |