System Design: Social Media App Like Instagram
Author: System Design Document | Date: 2026-08-28 Target Scale: 500M+ MAU, 1B+ DAU (Instagram-scale) Document Level: FAANG Interview / Production-Ready Thinking
Table of Contents
- Requirements Clarification
- Back-of-Envelope Calculations
- System Interface Design
- High-Level Architecture
- Core Component Design
- Data Model
- Key Design Decisions
- Scalability Strategy
- Curveball Scenarios
- Failure Modes and Mitigations
1. Requirements Clarification
1.1 Functional Requirements
Core Features (MVP)
-
User Authentication & Profiles
- Sign up / login (email, phone, OAuth)
- Profile management (bio, profile picture, username)
- Private / public account toggle
-
Media Posting
- Upload photos (single or carousel up to 10)
- Upload short videos (up to 60s for MVP; Reels up to 90s)
- Apply filters and basic edits (brightness, contrast, crop)
- Add captions, hashtags, location tags, and mentions (@user)
-
Feed
- Home Feed (Chronological + Algorithmic): Posts from followed users, ranked by relevance
- Explore Feed: Discovery feed of recommended content from non-followed users
- Pull-to-refresh and infinite scroll pagination
-
Social Graph
- Follow / unfollow users (asymmetric relationship)
- Mutual follow indicator
- Suggested follows based on social graph and interests
-
Interactions
- Like / unlike posts
- Comment / reply to comments (nested, up to 2 levels)
- Share posts (to DM or story)
- Save / bookmark posts
-
Direct Messaging (DM)
- 1:1 and group messaging
- Text, photo, video, GIF support
- Read receipts and typing indicators
- Disappearing messages (vanish mode)
-
Stories
- 24-hour ephemeral content (photos + short videos)
- Viewers list, reply to stories
- Story highlights (pinned to profile)
-
Notifications
- Push notifications for likes, comments, follows, mentions, DMs
- In-app notification center
Advanced Features (Post-MVP)
- Reels: Vertical short-form video (TikTok-style), up to 90s
- Live Streaming: Real-time video broadcasting with live comments
- Shopping: Product tags, checkout integration
- IGTV / Long-form Video: Videos up to 60 minutes
- Collaborative Posts: Co-authored posts with multiple creators
- Close Friends List: Restricted story/feed sharing
- Archiving: Archive posts without deleting
1.2 Non-Functional Requirements
| Requirement | Target Metric | Rationale |
|---|---|---|
| Availability | 99.95% (26 min downtime/month) | Social media is expected to be always-on |
| Read Latency (Feed) | P99 < 200ms | Feed is the most frequent user action |
| Write Latency (Post) | P99 < 2s (upload + process) | Users tolerate slightly longer for uploads |
| Media Delivery | P99 < 1s first byte (CDN) | Visual-first app; slow images = churn |
| Throughput | 100K+ QPS (peak read), 5K+ QPS (peak write) | See Section 2 for derivation |
| Data Durability | 99.9999999% (11 nines) | User photos are irreplaceable |
| Consistency | Eventual for feed/likes; Strong for auth/DM | Feed can be eventually consistent; DM needs ordering |
| Scalability | Linear horizontal scaling for all services | Must handle 10x growth without re-architecture |
| Security | GDPR/CCPA compliant, encrypted at rest and in transit | Regulatory + Cost Efficiency |
1.3 Assumptions
| Assumption | Value | Reasoning |
|---|---|---|
| MAU | 500 million | Instagram-scale |
| DAU | 300 million (60% of MAU) | Industry standard engagement |
| Posts per DAU per day | 2 posts | Conservative estimate |
| Feed views per DAU per day | 100 views (scrolls) | Heavy usage pattern |
| Avg. post size (compressed) | 500 KB (photo), 5 MB (video) | After compression + CDN optimization |
| Avg. followers per user | 150 | Power-law distribution; median is lower |
| Top 1% users | 10K+ followers each | Influencers / celebrities |
| Top 0.1% users | 1M+ followers each | Mega-influencers |
| Media retention | Permanent (unless deleted) | Unlike Stories (24h) |
| Geographic distribution | Global, with concentration in NA, EU, LATAM, SEA | Instagram’s actual distribution |
| Read:Write ratio | 100:1 (feed reads vs. posts) | Social media is read-heavy |
2. Back-of-Envelope Calculations
2.1 Storage Estimates
Daily Media Ingestion
Daily posts = DAU × posts_per_dau = 300M × 2 = 600M posts/day
Photo posts (70%): 420M × 500 KB = 210 TB/day
Video posts (30%): 180M × 5 MB = 900 TB/day
Total raw media/day = ~1,110 TB/day
Annual Storage (with redundancy)
Raw storage/year = 1,110 TB × 365 = ~405 PB/year
With 3x replication (durability): 405 × 3 = ~1,215 PB/year
With thumbnails, multiple resolutions, and metadata (~2x overhead):
Total storage/year ≈ 2,400 PB (~2.4 EB/year)
Key Insight: At Instagram scale, storage is measured in exabytes per year. This drives the need for tiered storage (hot/warm/cold) and aggressive compression.
Metadata Storage
Per post metadata: ~2 KB (JSON: author, caption, tags, timestamps, etc.)
600M posts/day × 2 KB = 1.2 TB/day metadata
Annual: ~440 TB/year (negligible compared to media)
Social Graph Storage
500M users × 150 avg followers × 8 bytes (two user IDs) = 600 GB
But with power-law distribution and fan-out cache, actual stored edges:
~75B edges × 16 bytes = ~1.2 TB (still manageable)
2.2 Bandwidth Estimates
Daily Egress
Feed views/day = DAU × views_per_dau = 300M × 100 = 30B views/day
Each view loads ~20 images (feed scroll):
30B × 20 × 500 KB = 300,000 TB/day = 300 EB/day of image egress
But with CDN caching (80% hit rate on popular content):
Actual origin egress = 300 EB × 20% = 60 EB/day
CDN edge egress = 300 EB/day
Key Insight: CDN caching is NOT optional at this scale. Without it, origin bandwidth requirements are physically impossible to sustain.
Peak Bandwidth
Peak hour = 20% of daily traffic in 1 hour
Peak image egress = 300 EB / 24 × 20% = 2.5 EB/hour = 700 TB/s
This is why aggressive CDN caching, image optimization, and lazy loading are critical.
2.3 QPS Estimates
| Operation | Calculation | QPS (Avg) | QPS (Peak) |
|---|---|---|---|
| Feed Read | 30B views/day ÷ 86,400s | ~347K | ~700K |
| Post Write | 600M posts/day ÷ 86,400s | ~7K | ~14K |
| Like | 3B likes/day ÷ 86,400s | ~35K | ~70K |
| Comment | 600M comments/day ÷ 86,400s | ~7K | ~14K |
| Follow/Unfollow | 30M/day ÷ 86,400s | ~350 | ~700 |
| DM Send | 3B messages/day ÷ 86,400s | ~35K | ~70K |
| Profile View | 3B views/day ÷ 86,400s | ~35K | ~70K |
| Auth/Login | 300M logins/day ÷ 86,400s | ~3.5K | ~7K |
Total estimated peak QPS: ~1M+ across all services combined.
2.4 Infrastructure Estimates
Assuming each app server handles 1,000 QPS (read-heavy, cached):
Feed Service servers = 700K peak QPS / 1,000 = 700 instances
Media Service servers = 14K peak QPS / 500 (write-heavy) = 28 instances
Like Service servers = 70K peak QPS / 1,000 = 70 instances
User Service servers = 70K peak QPS / 1,000 = 70 instances
Total app servers: ~1,000+ instances (excluding CDN, DB, cache layers)
Database instances (sharded):
- 500M users / 10M per shard = 50 user shards
- 600M posts/day / 10M per shard = 60 media shards
- Each shard = 1 primary + 2 replicas = 3 instances
- Total DB instances: (50 + 60) × 3 = 330 instances
Redis cache clusters:
- Feed cache: 200 nodes (for fan-out cache)
- Session cache: 100 nodes
- Rate limiting: 50 nodes
3. System Interface Design
3.1 RESTful API Definitions
User Service
POST /api/v1/users/register # Register new user
POST /api/v1/users/login # Login, returns JWT
GET /api/v1/users/{user_id} # Get user profile
PUT /api/v1/users/{user_id} # Update profile
GET /api/v1/users/{user_id}/followers # List followers (paginated)
GET /api/v1/users/{user_id}/following # List following (paginated)
POST /api/v1/users/{user_id}/follow # Follow a user
DELETE /api/v1/users/{user_id}/follow # Unfollow a user
GET /api/v1/users/suggestions # Suggested users to follow
Media / Post Service
POST /api/v1/media/upload # Upload media (multipart)
GET /api/v1/media/{media_id} # Get media metadata
DELETE /api/v1/media/{media_id} # Delete media
POST /api/v1/posts # Create a post
GET /api/v1/posts/{post_id} # Get post details
PUT /api/v1/posts/{post_id} # Update post (caption)
DELETE /api/v1/posts/{post_id} # Delete post
GET /api/v1/posts/{post_id}/comments # Get comments
POST /api/v1/posts/{post_id}/comments # Add comment
GET /api/v1/posts/{post_id}/likes # Get likers (paginated)
POST /api/v1/posts/{post_id}/like # Like a post
DELETE /api/v1/posts/{post_id}/like # Unlike a post
POST /api/v1/posts/{post_id}/save # Save/bookmark a post
DELETE /api/v1/posts/{post_id}/save # Unsave a post
Feed Service
GET /api/v1/feed/home # Home feed (algorithmic)
GET /api/v1/feed/home?mode=chronological # Chronological feed
GET /api/v1/feed/explore # Explore/discovery feed
GET /api/v1/feed/tagged/{hashtag} # Hashtag feed
GET /api/v1/feed/nearby # Location-based feed
POST /api/v1/feed/refresh # Pull-to-refresh
Stories Service
POST /api/v1/stories # Create a story
GET /api/v1/stories/{user_id} # Get user's active stories
POST /api/v1/stories/{story_id}/view # Mark story as viewed
POST /api/v1/stories/{story_id}/reply # Reply to a story
GET /api/v1/stories/highlights/{user_id} # Get story highlights
POST /api/v1/stories/highlights # Create story highlight
Direct Messaging Service
POST /api/v1/dm/conversations # Create conversation
GET /api/v1/dm/conversations # List conversations
GET /api/v1/dm/conversations/{conv_id} # Get conversation details
POST /api/v1/dm/conversations/{conv_id}/messages # Send message
GET /api/v1/dm/conversations/{conv_id}/messages # Get messages
POST /api/v1/dm/conversations/{conv_id}/read # Mark as read
WS /ws/dm # WebSocket for real-time DM
Notification Service
GET /api/v1/notifications # Get notifications (paginated)
POST /api/v1/notifications/{id}/read # Mark as read
POST /api/v1/notifications/read-all # Mark all as read
POST /api/v1/notifications/push/register # Register push token
3.2 Key API Response Examples
Feed Item Response
{
"data": {
"post_id": "p_8f3k29d",
"author": {
"user_id": "u_12345",
"username": "traveler_jane",
"profile_pic_url": "https://cdn.instagram.com/profiles/u_12345.jpg",
"is_verified": true,
"is_following": false
},
"media": [
{
"media_id": "m_abc123",
"type": "image",
"url": "https://cdn.instagram.com/media/m_abc123_1080.jpg",
"thumbnail_url": "https://cdn.instagram.com/media/m_abc123_300.jpg",
"width": 1080,
"height": 1350
}
],
"caption": "Beautiful sunset at Bali 🌅 #travel #bali",
"like_count": 15234,
"comment_count": 342,
"is_liked": false,
"is_saved": false,
"created_at": "2026-08-28T14:30:00Z",
"location": {
"name": "Ubud, Bali",
"lat": -8.5069,
"lng": 115.2625
}
},
"pagination": {
"next_cursor": "eyJwb3N0X2lkIjogInBfOGYzazI5ZCJ9",
"has_more": true
}
}
4. High-Level Architecture
flowchart TB
subgraph Client["Client Layer"]
iOS[iOS App]
Android[Android App]
Web[Web Client]
end
subgraph Edge["Edge / CDN Layer"]
CDN[CDN<br/>CloudFront / Akamai]
WAF[WAF / DDoS Protection]
LB[Global Load Balancer<br/>L7]
end
subgraph Gateway["API Gateway Layer"]
GW[API Gateway<br/>Kong / Custom]
Auth[Auth Middleware<br/>JWT Validation]
Rate[Rate Limiter<br/>Token Bucket]
end
subgraph Services["Microservices Layer"]
subgraph Core["Core Services"]
FeedSvc[Feed Service]
MediaSvc[Media Service]
UserSvc[User Service]
GraphSvc[Social Graph Service]
end
subgraph Engagement["Engagement Services"]
LikeSvc[Like Service]
CommentSvc[Comment Service]
NotifSvc[Notification Service]
end
subgraph Realtime["Real-time Services"]
DMSvc[DM Service]
StorySvc[Stories Service]
LiveSvc[Live Service]
end
subgraph Discovery["Discovery Services"]
ExploreSvc[Explore Service]
SuggestSvc[Suggestion Service]
SearchSvc[Search Service]
end
end
subgraph Async["Async / Messaging Layer"]
Kafka[Kafka Cluster<br/>Event Bus]
MQ[Message Queue<br/>RabbitMQ / SQS]
end
subgraph Cache["Cache Layer"]
Redis[Redis Cluster<br/>Session, Feed Cache, Rate Limit]
Memcached[Memcached<br/>Hot Data]
end
subgraph Storage["Data Storage Layer"]
subgraph DB["Databases"]
UserDB[(User DB<br/>MySQL Sharded)]
PostDB[(Post DB<br/>MySQL Sharded)]
GraphDB[(Graph DB<br/>Cassandra)]
NotifDB[(Notification DB<br/>MongoDB)]
end
subgraph ObjectStore["Object Storage"]
S3[S3 / OSS<br/>Media Storage]
ThumbS3[Thumbnail Store]
end
subgraph SearchStore["Search & Analytics"]
ES[Elasticsearch<br/>Search Index]
Clickhouse[ClickHouse<br/>Analytics]
end
end
subgraph Processing["Processing Layer"]
ImgProc[Image Processing<br/>Resize, Filter, Compress]
VidProc[Video Processing<br/>Transcode, Thumbnail]
ML[ML Pipeline<br/>Ranking, Moderation]
Moderation[Content Moderation<br/>AI + Human]
end
subgraph Infra["Infrastructure"]
K8s[Kubernetes Cluster]
Monitor[Monitoring<br/>Prometheus + Grafana]
Log[Logging<br/>ELK Stack]
Trace[Distributed Tracing<br/>Jaeger]
end
iOS --> CDN
Android --> CDN
Web --> CDN
CDN --> WAF
WAF --> LB
LB --> GW
GW --> Auth
Auth --> Rate
Rate --> FeedSvc
Rate --> MediaSvc
Rate --> UserSvc
Rate --> GraphSvc
Rate --> LikeSvc
Rate --> CommentSvc
Rate --> NotifSvc
Rate --> DMSvc
Rate --> StorySvc
Rate --> ExploreSvc
Rate --> SuggestSvc
Rate --> SearchSvc
FeedSvc --> Redis
FeedSvc --> Kafka
MediaSvc --> S3
MediaSvc --> ThumbS3
MediaSvc --> Kafka
UserSvc --> UserDB
GraphSvc --> GraphDB
LikeSvc --> Redis
LikeSvc --> Kafka
CommentSvc --> PostDB
NotifSvc --> NotifDB
NotifSvc --> Kafka
DMSvc --> Redis
DMSvc --> Kafka
StorySvc --> S3
StorySvc --> Kafka
ExploreSvc --> ES
SuggestSvc --> GraphDB
SearchSvc --> ES
Kafka --> ImgProc
Kafka --> VidProc
Kafka --> ML
Kafka --> Moderation
ImgProc --> S3
VidProc --> S3
ML --> Clickhouse
Moderation --> PostDB
FeedSvc --> PostDB
MediaSvc --> PostDB
CommentSvc --> Redis
K8s -.-> Services
Monitor -.-> Services
Log -.-> Services
Trace -.-> Services
5. Core Component Design
5.1 Feed Service
The feed is the most critical component of Instagram. It determines user engagement and retention.
Feed Generation Strategies
| Strategy | How It Works | Pros | Cons | Best For |
|---|---|---|---|---|
| Fan-out on Write (Push) | When a user posts, push to all followers’ feed caches | O(1) read latency | O(N) write cost; explodes for celebrities | Users with < 10K followers |
| Pull on Read (Pull) | When user opens feed, query posts from followed users | O(1) write cost | O(N) read cost; slow for many follows | Users with many follows |
| Hybrid (Push + Pull) | Push for normal users; pull for celebrities; merge at read time | Balanced read/write | Complex merging logic | Production choice |
| Pre-computed + Real-time | Pre-rank feed periodically; inject real-time posts at read | Best UX, fresh content | Storage overhead, stale pre-computed data | Instagram-scale |
Recommended: Hybrid Feed with Pre-computation
┌─────────────────────────────────────────────────────────────┐
│ Feed Generation Pipeline │
├─────────────────────────────────────────────────────────────┤
│ │
│ 1. CANDIDATE COLLECTION │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ Pushed │ │ Pulled │ │ Explore │ │
│ │ Feed │ │ Feed │ │ Candidates│ │
│ │ (Cache) │ │ (Query) │ │ (ML) │ │
│ └────┬─────┘ └────┬─────┘ └────┬─────┘ │
│ └──────────────┴─────────────┘ │
│ ▼ │
│ 2. RANKING (ML Model) │
│ ┌─────────────────────────────────┐ │
│ │ Features: recency, relationship,│ │
│ │ media type, predicted like, │ │
│ │ predicted comment, diversity │ │
│ └──────────────┬──────────────────┘ │
│ ▼ │
│ 3. FILTERING & RE-RANKING │
│ ┌─────────────────────────────────┐ │
│ │ Remove: hidden posts, muted, │ │
│ │ duplicate creators, sensitive │ │
│ │ Re-rank: diversity, freshness │ │
│ └──────────────┬──────────────────┘ │
│ ▼ │
│ 4. CACHING │
│ ┌─────────────────────────────────┐ │
│ │ Redis: feed_{user_id}_{cursor} │ │
│ │ TTL: 5 minutes (hot users) │ │
│ │ TTL: 30 minutes (cold users) │ │
│ └─────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘
Fan-out Threshold Decision
Threshold = 10,000 followers
User has < 10K followers:
→ Fan-out on write (push to follower caches)
→ Write cost: O(followers) = O(10K) max
→ Read cost: O(1) = cache lookup
User has >= 10K followers:
→ No fan-out on write
→ Pull on read (query their posts at feed time)
→ Write cost: O(1) = just store the post
→ Read cost: O(1) = their posts are cached individually
Feed Ranking Features
| Feature Category | Features | Weight |
|---|---|---|
| Recency | Post age (exponential decay) | High |
| Relationship | Mutual follow, DM frequency, profile visits, tag frequency | High |
| Interest | Past engagement with this creator, media type preference | Medium |
| Predicted Affinity | ML-predicted like probability, comment probability, share probability | High |
| Diversity | Creator diversity (avoid same creator spam), media type diversity | Medium |
| Social Proof | Like count, comment count (velocity, not absolute) | Low |
5.2 Media Service
Handles upload, processing, storage, and delivery of all media content.
Upload Pipeline
sequenceDiagram
participant Client
participant GW as API Gateway
participant MediaSvc as Media Service
participant Presign as Presign Service
participant S3 as Object Storage
participant Kafka
participant ImgProc as Image Processor
participant VidProc as Video Processor
participant Moderation as Content Moderation
participant DB as Post DB
Client->>GW: POST /media/upload (request)
GW->>MediaSvc: Validate request
MediaSvc->>Presign: Generate presigned URL
Presign->>S3: Create presigned PUT URL
Presign-->>MediaSvc: Presigned URL + media_id
MediaSvc-->>Client: Return presigned URL
Client->>S3: PUT media (direct upload)
S3-->>Client: 200 OK
Client->>GW: POST /posts (create post with media_id)
GW->>MediaSvc: Create post
MediaSvc->>DB: Store post metadata
MediaSvc->>Kafka: Emit POST_CREATED event
Kafka->>ImgProc: Process image
ImgProc->>S3: Store thumbnails (multiple sizes)
ImgProc->>Moderation: Check content
Kafka->>VidProc: Process video (if applicable)
VidProc->>S3: Store transcoded versions
VidProc->>Moderation: Check content
Moderation->>DB: Update moderation status
Media Processing Pipeline
Original Upload
│
▼
┌─────────────────┐
│ Validation │ ← Format, size, aspect ratio check
└────────┬────────┘
▼
┌─────────────────┐
│ Image Processing│
│ (if photo) │
│ │
│ • Resize: │
│ - 1080px │ ← Full resolution
│ - 640px │ ← Feed display
│ - 300px │ ← Thumbnail
│ - 150px │ ← Grid view
│ • Compress: │
│ - JPEG quality 85% │
│ - WebP format │ ← Modern browsers
│ • EXIF strip │ ← Privacy
└────────┬────────┘
▼
┌─────────────────┐
│ Video Processing│
│ (if video) │
│ │
│ • Transcode: │
│ - 1080p H.264 │ ← High quality
│ - 720p H.264 │ ← Standard
│ - 480p H.264 │ ← Low bandwidth
│ - H.265/HEVC │ ← Modern devices
│ • Thumbnail: │
│ - Keyframe extraction │
│ - 3 thumbnails │
│ • Audio: │
│ - AAC 128kbps │
└────────┬────────┘
▼
┌─────────────────┐
│ Content │
│ Moderation │
│ │
│ • NSFW detection│ ← AI model
│ • Copyright │ ← Audio fingerprint
│ • Policy │ ← Text/caption scan
└────────┬────────┘
▼
Store in S3 + Update DB
CDN Strategy
Tier 1: Edge CDN (CloudFront / Akamai)
- 200+ PoPs globally
- Caches: thumbnails, profile pics, popular media
- TTL: 24h for media, 1h for profile pics
- Hit rate target: 85%+
Tier 2: Regional CDN
- Regional caching for less popular content
- TTL: 1 week
- Fallback to origin if edge miss
Tier 3: Origin (S3)
- Single source of truth
- All processed media versions stored here
5.3 User Service
Authentication Flow
sequenceDiagram
participant Client
participant AuthSvc as Auth Service
participant Redis as Redis (Session)
participant UserDB as User DB
participant Push as Push Notification
Client->>AuthSvc: POST /login (email + password)
AuthSvc->>UserDB: Query user by email
UserDB-->>AuthSvc: User record + hashed password
AuthSvc->>AuthSvc: Verify password (bcrypt)
AuthSvc->>AuthSvc: Generate JWT (access + refresh)
AuthSvc->>Redis: Store session
AuthSvc-->>Client: JWT tokens + user info
Note over Client,Push: Subsequent requests
Client->>AuthSvc: Request with Authorization: Bearer <JWT>
AuthSvc->>AuthSvc: Validate JWT signature
AuthSvc->>Redis: Check session active
Redis-->>AuthSvc: Session valid
AuthSvc-->>Client: 200 OK
JWT Token Design
Access Token:
- TTL: 15 minutes
- Contains: user_id, scope, issued_at
- Signed with RS256 (asymmetric)
- Stored in memory (not localStorage for XSS protection)
Refresh Token:
- TTL: 30 days
- Stored in HTTP-only cookie
- Rotated on each use (prevent token theft)
- Revoked on logout / password change
5.4 Social Graph Service
Manages follow relationships, suggestions, and graph traversals.
Graph Storage Design
Follow relationships stored in Cassandra (wide-column):
Table: follows
Partition Key: user_id (follower)
Clustering Key: followed_user_id
Columns: followed_user_id, timestamp, status
Allows:
- O(1) lookup: "Who does user X follow?"
- O(1) lookup: "Does A follow B?" (via secondary index)
Table: followers
Partition Key: user_id (followed)
Clustering Key: follower_user_id
Columns: follower_user_id, timestamp
Allows:
- O(1) lookup: "Who follows user X?"
- Fan-out on write uses this table
Suggestion Algorithm
Suggestion Score = w1 × Friend-of-Friends
+ w2 × Mutual Followers
+ w3 × Profile View Velocity
+ w4 × Search Query Match
+ w5 × Location Proximity
+ w6 × Account Similarity (interests)
Candidate Pool:
1. Friends of friends (2-hop neighbors) — highest weight
2. Users with high mutual follower count
3. Users recently viewed by the target user
4. Users searched for by the target user
5. Users in same location with similar interests
Filter:
- Exclude already followed
- Exclude blocked/muted
- Exclude inactive accounts (>90 days)
- Limit to 50 suggestions per request
5.5 Notification Service
Notification Types and Delivery
| Type | Trigger | Delivery Channel | Priority |
|---|---|---|---|
| Like | User likes your post | Push + In-app | Low |
| Comment | User comments on your post | Push + In-app | Medium |
| Follow | New follower | Push + In-app | Medium |
| Mention | @mentioned in post/comment | Push + In-app | High |
| DM | New direct message | Push + In-app + Badge | High |
| Story View | Someone views your story | In-app only | Low |
| Live Start | Followed user goes live | Push | High |
| Trending | Your post is trending | Push + In-app | Medium |
Notification Pipeline
flowchart LR
A[Event: Like/Comment/Follow] --> B[Kafka Topic: notifications]
B --> C[Notification Worker]
C --> D{Deduplication}
D -->|New| E[Write to MongoDB]
D -->|Duplicate| F[Drop]
E --> G[Batch Aggregation]
G --> H[Push Notification Service]
H --> I[FCM / APNs]
I --> J[User Device]
G --> K[In-App Notification Cache]
K --> L[Redis: user_notifications_{id}]
Notification Deduplication and Batching
Problem: If 100 people like a post in 1 minute, don't send 100 push notifications.
Solution: Batch + Deduplicate
1. Write each notification to Kafka
2. Worker consumes with 30-second window
3. Aggregate: "John, Jane, and 98 others liked your photo"
4. Send ONE push notification per aggregation window
5. Store individual notifications in MongoDB for in-app detail view
Deduplication Key: {actor_id, target_user_id, post_id, action_type}
5.6 Direct Messaging (DM) Service
Real-time Architecture
WebSocket Connection Pool
│
▼
┌─────────────────┐
│ Connection │
│ Manager │
│ (per-user WS │
│ connections) │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Message Router │
│ │
│ • Route to │
│ conversation │
│ • Persist │
│ to DB │
│ • Fan-out to │
│ recipients │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Redis Pub/Sub │
│ (in-memory │
│ message bus) │
└────────┬────────┘
│
▼
┌─────────────────┐
│ Recipient WS │
│ Connections │
└─────────────────┘
Message Ordering and Reliability
Guarantees:
- Per-conversation FIFO ordering (sequence numbers)
- At-least-once delivery (ack + retry)
- Message persistence (even if recipient offline)
Message Schema:
{
"message_id": "uuid",
"conversation_id": "conv_123",
"sender_id": "u_456",
"type": "text" | "image" | "video" | "gif",
"content": "...",
"timestamp": 1690598400000,
"sequence_num": 42,
"status": "sent" | "delivered" | "read",
"ttl": null | "2026-09-28T00:00:00Z" // for disappearing messages
}
5.7 Stories Service
Ephemeral Content Lifecycle
1. User creates story → Upload to S3 with 24h TTL metadata
2. Fan-out to followers' "available stories" list in Redis
3. When follower views:
- Record view in Redis Set: story_views_{story_id}
- Remove from follower's "unviewed" list
- Show progress bar + viewer position
4. After 24 hours:
- Cron job marks story as expired
- Remove from all followers' lists
- Media remains in S3 (for highlights) but hidden from feed
5. Story Highlights:
- User can pin stories to profile
- Stored separately, no TTL
Redis Data Structures for Stories
Available stories for a user:
ZSET: stories_available_{user_id}
Score: story_expiry_timestamp
Value: story_id
Viewed stories for a user:
SET: stories_viewed_{user_id}
Value: story_id
Story viewers:
SET: story_viewers_{story_id}
Value: viewer_user_id
Story viewer order (for progress bar):
ZSET: story_viewer_order_{story_id}
Score: view_timestamp
Value: viewer_user_id
5.8 Explore / Discovery Service
Recommendation Pipeline
Candidate Generation (Retrieval) → Ranking → Filtering → Serving
1. CANDIDATE GENERATION (Broad Retrieval)
Sources:
a) Collaborative Filtering: Users similar to you liked these posts
b) Content-Based: Posts similar to what you engaged with
c) Trending: Posts with high engagement velocity
d) Social: Posts from friends-of-friends
e) Location-Based: Popular posts near you
Output: ~1000 candidate posts
2. RANKING (ML Model)
Features:
- User engagement history with similar content
- Post engagement velocity (likes/min, comments/min)
- Creator relevance to user
- Media type preference
- Recency
Model: Gradient Boosted Trees (XGBoost) or Two-Tower DNN
Output: Ranked list of ~100 posts
3. FILTERING
- Remove already seen posts
- Remove from blocked/muted users
- Ensure diversity (max 2 posts per creator)
- Ensure media type diversity
4. SERVING
- Cache ranked results in Redis (TTL: 10 minutes)
- Serve via CDN for media assets
6. Data Model
6.1 Relational Schema (MySQL - Sharded)
Users Table
CREATE TABLE users (
user_id BIGINT UNSIGNED NOT NULL, -- Snowflake ID
username VARCHAR(30) NOT NULL UNIQUE,
email VARCHAR(255) NOT NULL UNIQUE,
phone VARCHAR(20) UNIQUE,
password_hash VARCHAR(255) NOT NULL,
full_name VARCHAR(100),
bio VARCHAR(150),
profile_pic_url VARCHAR(500),
is_private BOOLEAN DEFAULT FALSE,
is_verified BOOLEAN DEFAULT FALSE,
is_business_account BOOLEAN DEFAULT FALSE,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
last_login_at TIMESTAMP NULL,
status ENUM('active', 'suspended', 'deleted') DEFAULT 'active',
PRIMARY KEY (user_id),
INDEX idx_username (username),
INDEX idx_email (email),
INDEX idx_status (status)
) ENGINE=InnoDB;
Posts Table
CREATE TABLE posts (
post_id BIGINT UNSIGNED NOT NULL, -- Snowflake ID
user_id BIGINT UNSIGNED NOT NULL,
caption TEXT,
media_type ENUM('image', 'video', 'carousel') NOT NULL,
media_count INT DEFAULT 1,
location_id BIGINT UNSIGNED NULL,
is_deleted BOOLEAN DEFAULT FALSE,
moderation_status ENUM('pending', 'approved', 'rejected', 'reviewing') DEFAULT 'pending',
like_count INT UNSIGNED DEFAULT 0,
comment_count INT UNSIGNED DEFAULT 0,
share_count INT UNSIGNED DEFAULT 0,
view_count BIGINT UNSIGNED DEFAULT 0, -- For video/reels
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
PRIMARY KEY (post_id),
INDEX idx_user_id (user_id),
INDEX idx_created_at (created_at DESC),
INDEX idx_moderation (moderation_status),
INDEX idx_user_created (user_id, created_at DESC)
) ENGINE=InnoDB;
Media Table
CREATE TABLE media (
media_id BIGINT UNSIGNED NOT NULL,
post_id BIGINT UNSIGNED NOT NULL,
media_type ENUM('image', 'video') NOT NULL,
original_url VARCHAR(1000) NOT NULL,
thumbnail_url VARCHAR(1000),
width INT,
height INT,
duration_sec INT NULL, -- For video
file_size_bytes BIGINT UNSIGNED,
format VARCHAR(10), -- jpeg, mp4, etc.
display_order INT DEFAULT 0, -- For carousel
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (media_id),
INDEX idx_post_id (post_id),
FOREIGN KEY (post_id) REFERENCES posts(post_id) ON DELETE CASCADE
) ENGINE=InnoDB;
Hashtags Table
CREATE TABLE post_hashtags (
post_id BIGINT UNSIGNED NOT NULL,
hashtag VARCHAR(30) NOT NULL,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (post_id, hashtag),
INDEX idx_hashtag (hashtag),
FOREIGN KEY (post_id) REFERENCES posts(post_id) ON DELETE CASCADE
) ENGINE=InnoDB;
CREATE TABLE hashtags (
hashtag VARCHAR(30) NOT NULL UNIQUE,
post_count INT UNSIGNED DEFAULT 0,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (hashtag)
) ENGINE=InnoDB;
6.2 NoSQL Schema (Cassandra - Social Graph)
-- Follow relationships (fan-out on write)
CREATE TABLE follows (
follower_id UUID,
following_id UUID,
created_at TIMESTAMP,
status TEXT, -- 'pending', 'active'
PRIMARY KEY (follower_id, following_id)
) WITH CLUSTERING ORDER BY (following_id ASC);
-- Followers (for notification fan-out)
CREATE TABLE followers (
user_id UUID,
follower_id UUID,
created_at TIMESTAMP,
PRIMARY KEY (user_id, follower_id)
) WITH CLUSTERING ORDER BY (follower_id ASC);
-- User timeline (pushed feed items)
CREATE TABLE user_timeline (
user_id UUID,
post_id UUID,
author_id UUID,
created_at TIMESTAMP,
like_count INT,
media_type TEXT,
PRIMARY KEY (user_id, created_at, post_id)
) WITH CLUSTERING ORDER BY (created_at DESC);
6.3 MongoDB Schema (Notifications)
{
"_id": "ObjectId",
"notification_id": "n_abc123",
"recipient_id": "u_789",
"actor_id": "u_456",
"type": "like",
"post_id": "p_xyz789",
"message": "traveler_jane liked your photo",
"actor_username": "traveler_jane",
"actor_profile_pic": "https://cdn.../u_456.jpg",
"post_thumbnail": "https://cdn.../p_xyz789_thumb.jpg",
"is_read": false,
"created_at": "2026-08-28T14:30:00Z",
"aggregated_with": ["n_def456", "n_ghi789"], // For batched notifications
"expires_at": "2026-11-28T14:30:00Z" // 90-day retention
}
6.4 Elasticsearch Index (Search)
{
"mappings": {
"properties": {
"user_id": { "type": "long" },
"username": {
"type": "text",
"analyzer": "username_analyzer",
"fields": {
"keyword": { "type": "keyword" }
}
},
"full_name": { "type": "text" },
"bio": { "type": "text" },
"post_id": { "type": "long" },
"caption": { "type": "text" },
"hashtags": { "type": "keyword" },
"mentions": { "type": "keyword" },
"location": { "type": "geo_point" },
"media_type": { "type": "keyword" },
"like_count": { "type": "integer" },
"created_at": { "type": "date" },
"is_verified": { "type": "boolean" },
"follower_count": { "type": "integer" }
}
}
}
6.5 Redis Data Structures
# Session Storage
SET session:{session_id} "{user_id, expires_at, device_info}" EX 86400
# Feed Cache
ZSET feed:{user_id}:home
Score: ranking_score (higher = more relevant)
Value: post_id
# Like Count (Atomic Counter)
INCR like_count:{post_id}
GET like_count:{post_id}
# Like Membership (Has user liked?)
SADD likers:{post_id} {user_id}
SISMEMBER likers:{post_id} {user_id}
# Rate Limiting
INCR rate_limit:{user_id}:{endpoint}:{minute}
EXPIRE rate_limit:{user_id}:{endpoint}:{minute} 60
# User Online Status
SET online:{user_id} "1" EX 300 (5-minute TTL, heartbeat refreshes)
# DM Conversation Unread Count
HINCRBY dm_unread:{conv_id}:{user_id} unread_count 1
# Story Availability
ZADD stories_available:{user_id} {expiry_timestamp} {story_id}
# Trending Hashtags
ZSET trending:hashtags
Score: engagement_velocity (likes + comments per minute)
Value: hashtag_name
7. Key Design Decisions
7.1 Feed: Push vs. Pull
| Aspect | Push (Fan-out on Write) | Pull (Compute on Read) | Hybrid (Our Choice) |
|---|---|---|---|
| Read Latency | O(1) — instant | O(N) — slow for many follows | O(1) — cached |
| Write Latency | O(followers) — slow for celebrities | O(1) — instant | O(1) for celebrities, O(followers) for others |
| Storage | High — duplicate feed per follower | Low — single source of truth | Medium — selective caching |
| Consistency | Eventual (fan-out may lag) | Strong (always fresh) | Eventual (acceptable) |
| Celebrity Problem | ❌ Explodes at 1M+ followers | ✅ Handles gracefully | ✅ Threshold-based switch |
Decision: Hybrid with a threshold of 10K followers. Below threshold: push. Above threshold: pull. This gives O(1) reads for 99.9% of users while avoiding the celebrity fan-out explosion.
7.2 Database: SQL vs. NoSQL
| Component | Choice | Rationale |
|---|---|---|
| User Data | MySQL (sharded) | ACID needed for auth, billing, profile consistency |
| Posts | MySQL (sharded) | Relational integrity with media, comments, likes |
| Social Graph | Cassandra | Massive write throughput for follows, O(1) lookups, horizontal scaling |
| Notifications | MongoDB | Schema flexibility, high write volume, eventual consistency OK |
| Feed Cache | Redis | Sub-millisecond reads, sorted sets for ranking |
| Search | Elasticsearch | Full-text search, faceting, geo-search |
| Media Metadata | MySQL + S3 | Relational for queries, object storage for blobs |
Decision: Polyglot persistence. No single database can handle all access patterns optimally. Each component uses the best-fit storage engine.
7.3 Media Storage: S3 vs. Custom
| Aspect | S3 / Cloud Object Store | Custom Distributed FS |
|---|---|---|
| Durability | 11 nines (built-in) | Must build yourself |
| Scalability | Unlimited | Limited by cluster size |
| Cost | $0.023/GB/month | Lower at massive scale |
| CDN Integration | Native (CloudFront) | Requires integration |
| Operations | Managed service | Team overhead |
Decision: S3 (or equivalent cloud object store) for MVP and growth phase. Transition to custom storage only when cost savings justify the engineering overhead (typically at 10+ EB scale).
7.4 ID Generation: UUID vs. Snowflake
| Aspect | UUID v4 | Snowflake | Auto-Increment |
|---|---|---|---|
| Predictability | Random | Time-ordered | Sequential |
| DB Friendly | Poor (random writes) | Good (time-ordered) | Best |
| Sharding | Natural | Range-based | Difficult |
| Info Leakage | None | Timestamp embedded | Sequence visible |
| Length | 36 chars (string) | 19 chars (int64) | Variable |
Decision: Snowflake IDs. They are:
- Time-ordered (good for DB indexing and range queries)
- Distributed (no central coordinator bottleneck)
- Compact (64-bit integer, fits in BIGINT)
- Embed timestamp (useful for debugging and routing)
Snowflake ID Structure (64 bits):
┌────────────┬─────────┬─────────┬──────────┐
│ Timestamp │ Worker │ Datacenter │ Sequence │
│ 41 bits │ 5 bits │ 5 bits │ 12 bits │
│ (ms since │ (machine│ (region) │ (per-ms │
│ custom │ ID) │ │ counter)│
│ epoch) │ │ │ │
└────────────┴─────────┴─────────┴──────────┘
7.5 API Style: REST vs. GraphQL
| Aspect | REST | GraphQL | Hybrid |
|---|---|---|---|
| Over-fetching | Common | Solved | Partial |
| Caching | Native (HTTP cache) | Complex (need Relay) | Native for core |
| Learning Curve | Low | High | Medium |
| Mobile Efficiency | Poor (multiple calls) | Excellent (single call) | Good |
| CDN Friendly | Yes | No (POST only) | Yes for core |
Decision: REST for public APIs and CDN-cacheable endpoints (feed, media). GraphQL for mobile app’s complex queries (profile + feed + suggestions in one call). This hybrid approach gives the best of both worlds.
7.6 Consistency Model
| Data Type | Consistency Level | Rationale |
|---|---|---|
| Auth / Sessions | Strong | Security-critical; must be consistent |
| User Profile | Strong | Users expect immediate updates |
| Posts | Strong (eventual to readers) | Writer sees immediately; readers see after propagation |
| Likes / Comments | Eventual | Count can be slightly stale; no harm |
| Feed | Eventual | Stale feed is acceptable; freshness < availability |
| DM Messages | Strong (per conversation) | Message ordering is critical |
| Notifications | Eventual | Delayed notification is acceptable |
| Stories | Eventual | 24h TTL makes eventual consistency fine |
8. Scalability Strategy
8.1 Horizontal Scaling by Component
| Component | Scaling Strategy | Shard Key | Max Before Re-shard |
|---|---|---|---|
| User Service | Consistent hashing on user_id | user_id | 10M users/shard |
| Post Service | Range partitioning by post_id (time-ordered) | post_id (embedded timestamp) | 100M posts/shard |
| Feed Service | Stateless; scale instances behind LB | N/A (state in Redis) | Unlimited |
| Like Service | Hash partitioning on post_id | post_id | 50M likes/shard |
| Comment Service | Hash partitioning on post_id | post_id | 10M comments/shard |
| Graph Service | Consistent hashing on user_id | user_id | 5M users/shard |
| DM Service | Hash partitioning on conversation_id | conversation_id | 10M convos/shard |
| Notification Service | Hash partitioning on recipient_id | recipient_id | 10M users/shard |
8.2 Caching Strategy
Cache Hierarchy:
Level 0: Client Cache (App Memory)
- Profile pics, recent feed items
- TTL: 5 minutes
- Size: ~50 MB per device
Level 1: CDN Edge Cache
- Media files (images, videos, thumbnails)
- Static assets (JS, CSS)
- TTL: 24 hours (media), 1 hour (profile pics)
- Hit rate target: 85%+
Level 2: API Response Cache (Redis)
- Feed responses
- Profile data
- Post details (hot posts)
- TTL: 1-5 minutes
- Hit rate target: 60%+
Level 3: Database Read Replicas
- Read-heavy queries routed to replicas
- Write-heavy queries go to primary
- Replication lag: < 1 second
Level 4: Primary Database
- Source of truth
- All writes go here
8.3 Database Sharding Strategy
Phase 1: Single Shard (0 - 10M users)
- Single MySQL instance with read replicas
- Simple, easy to manage
Phase 2: Vertical Split (10M - 50M users)
- Separate databases for users, posts, comments
- Each with own read replicas
Phase 3: Horizontal Sharding (50M - 500M users)
- Shard by user_id (consistent hashing)
- 50 shards, each with 1 primary + 2 replicas
- Proxy layer (Vitess / ShardingSphere) for routing
Phase 4: Multi-Region (500M+ users)
- Active-active multi-region deployment
- Each region has full shard set
- Cross-region replication for disaster recovery
- User routed to nearest region
8.4 Auto-Scaling Triggers
CPU Utilization > 70% for 2 minutes → Scale up
Memory Utilization > 80% for 1 minute → Scale up
Request Latency P99 > 500ms for 30 seconds → Scale up
Queue Depth > 1000 messages → Scale up consumers
Connection Count > 80% of max → Scale up
Scale Down:
CPU Utilization < 30% for 10 minutes → Scale down
Min instances: 3 (availability)
Max instances: 100 (cost control)
Cooldown: 5 minutes between scale events
9. Curveball Scenarios
Curveball 1: “What if reads are 100x writes?”
Context: This is actually Instagram’s real ratio. Feed reads vastly outnumber post writes.
Impact Analysis:
- Feed read QPS: ~700K peak
- Post write QPS: ~14K peak
- Ratio: ~50:1 (even higher during peak hours)
Design Adaptations:
-
Read-Optimized Architecture
- Heavy caching at every layer (CDN → Redis → DB replicas)
- Feed pre-computation: rank feeds in background, serve from cache
- Read replicas: 10:1 replica-to-primary ratio for hot data
-
Write-Optimized Paths
- Async processing: accept write immediately, process in background
- Eventual consistency: likes/comments update asynchronously
- Batch writes: aggregate notifications, batch DB writes
-
CDN as Primary Delivery
- 85%+ of all reads served from CDN edge
- Origin only handles cache misses and writes
- Predictive preloading: preload likely-to-be-viewed content to CDN
-
Feed Cache Invalidation Strategy
Problem: How to invalidate feed cache when new posts arrive? Solution: Version-based cache with soft TTL - Each user's feed has a version number - New post from followed user → increment version - Cache key: feed:{user_id}:{version} - Old cache entries expire naturally (TTL) - No explicit invalidation needed
Result: The system is already designed for this ratio. The hybrid feed model, aggressive caching, and CDN strategy all optimize for read-heavy workloads.
Curveball 2: “What if we need to support Reels / Short-Form Video?”
Context: Reels is Instagram’s answer to TikTok. Short-form vertical video with music, effects, and algorithmic discovery.
New Requirements:
- Video upload up to 90 seconds
- Auto-play in feed (muted by default)
- Music/audio licensing and attribution
- Advanced video effects and filters
- Separate Reels feed (algorithmic, not social graph)
- Higher bandwidth requirements
Architecture Changes:
-
Video Processing Pipeline Enhancement
New processing steps: - Audio extraction and fingerprinting (for copyright) - Multiple bitrate laddering (288p, 360p, 480p, 720p, 1080p) - HLS/DASH adaptive streaming manifest generation - Thumbnail generation (3-5 keyframes) - Audio normalization and loudness matching - Effect/filter rendering (AR effects, transitions) -
Storage Impact
Per Reel storage: - Original: 50 MB - Transcoded versions (5 bitrates): 20 MB × 5 = 100 MB - Thumbnails: 5 × 50 KB = 250 KB - Audio track: 2 MB - Metadata + manifests: 100 KB Total per Reel: ~152 MB If 30% of posts become Reels: 180M Reels/day × 152 MB = 27.4 TB/day (vs. 900 TB for full video) Actually LESS than full video due to shorter duration -
New Service: Reels Service
Responsibilities: - Reels feed generation (separate from main feed) - Music library management and licensing - Effect/filter catalog - Reels-specific ranking algorithm - Audio trend detection Ranking features (different from main feed): - Completion rate (did user watch till end?) - Rewatch rate - Share rate - Audio trend score - Effect novelty score -
Bandwidth Impact
Reels auto-play in feed: - Each Reel preview: 2-3 seconds at 480p = ~1 MB - User scrolls 100 items/day, 30% are Reels - 30 Reels × 1 MB = 30 MB/user/day for Reels previews - 300M DAU × 30 MB = 9 EB/day for Reels previews alone Mitigation: - WiFi-only auto-play (option) - Lower bitrate for preview (288p) - Aggressive CDN caching of popular Reels - Preload next 2-3 Reels during watch -
Music Licensing Integration
New components: - Music Catalog Service (licensed tracks) - Audio Fingerprinting (Shazam-like matching) - Royalty Tracking (per-play counting) - Takedown Pipeline (DMCA compliance) Flow: User selects music → Check license → Embed track ID → On play → Increment play count → Monthly royalty settlement
Curveball 3: “What if we need real-time collaboration (co-authored posts)?”
Context: Multiple creators collaborate on a single post, sharing ownership, credits, and analytics.
New Requirements:
- Invite collaborators to a post
- Real-time co-editing (caption, media selection)
- Shared ownership and attribution
- Combined follower reach
- Split analytics and monetization
Architecture Changes:
-
Data Model Extensions
CREATE TABLE post_collaborators ( post_id BIGINT UNSIGNED NOT NULL, user_id BIGINT UNSIGNED NOT NULL, role ENUM('owner', 'collaborator', 'editor') NOT NULL, invited_by BIGINT UNSIGNED NOT NULL, status ENUM('pending', 'accepted', 'rejected') DEFAULT 'pending', added_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (post_id, user_id), FOREIGN KEY (post_id) REFERENCES posts(post_id) ON DELETE CASCADE, FOREIGN KEY (user_id) REFERENCES users(user_id) ); -
Real-Time Co-Editing
Technology: WebSocket + Operational Transformation (OT) Flow: 1. Owner creates draft post 2. Invites collaborators via WebSocket channel: collab_{post_id} 3. Each editor connects to the channel 4. Edges as OT operations (insert, delete, format) 5. Server merges operations and broadcasts to all editors 6. On publish: all collaborators notified Conflict Resolution: - Last-write-wins for simple fields (caption) - OT for rich text editing - Media additions are append-only (no conflicts) -
Feed and Discovery Impact
Problem: A collaborative post appears in ALL collaborators' feeds. Solution: - Post is associated with multiple user_ids - Fan-out to ALL collaborators' followers - Feed shows all collaborators' profile pics - Clicking any collaborator navigates to their profile Storage Impact: - post_users table maps posts to all associated users - Feed fan-out multiplies by number of collaborators - Mitigation: Cap at 5 collaborators per post -
Analytics Splitting
Each collaborator gets: - Their share of impressions (based on whose followers saw it) - Combined engagement metrics - Attribution breakdown Implementation: - impression_events table records (post_id, viewer_id, source_user_id) - source_user_id = whose feed the viewer was scrolling - Aggregated per collaborator for their analytics dashboard
Curveball 4: “What if a celebrity posts and gets 10M likes in an hour?”
Context: A mega-influencer (e.g., Selena Gomez with 400M followers) posts a photo.
Impact Analysis:
Fan-out cost: 400M followers × cache write = 400M writes
If each write takes 1ms: 400 seconds = 6.7 minutes of fan-out
During this time, followers see stale feeds
Mitigations:
-
Threshold-Based Fan-out (Already Designed)
- Users with > 10K followers use pull model
- No fan-out explosion for celebrities
- Their posts are cached individually with high replication
-
Like Counter Handling
Problem: 10M likes in 1 hour = 2,778 likes/second Single DB row update bottleneck Solution: Counter Sharding - Split like count across 100 virtual shards - Each shard handles ~28 likes/second (trivial) - Read: sum all shards (100 reads, parallelized) - Display: approximate count (good enough for UI) Redis Implementation: SET like_count:{post_id}:shard_001 123456 SET like_count:{post_id}:shard_002 234567 ... GETSUM like_count:{post_id}:shard_* → total -
Comment Handling
Problem: 1M comments in 1 hour = 278 comments/second Solution: - Comments stored in Cassandra (high write throughput) - Paginated display (show top 50 + "load more") - Top comments pre-sorted by engagement - New comments appear after a short delay (eventual consistency) -
Notification Storm Prevention
Problem: If the celebrity follows you back, you get a notification. If 1M people follow them simultaneously... Solution: - Rate-limit notifications per user (max 100/hour) - Batch notifications aggressively - Deprioritize low-importance notifications during storms - Queue overflow → discard lowest priority
Curveball 5: “What if we need to support live video streaming?”
Context: Instagram Live allows real-time video broadcasting with live comments and reactions.
Architecture Additions:
-
Streaming Infrastructure
Ingest → Processing → Distribution → Playback Ingest: - RTMP ingest servers (auto-scaled) - Each stream gets dedicated ingest instance - Geo-distributed ingest points Processing: - Real-time transcoding (multiple bitrates) - Low-latency HLS or WebRTC - Audio processing (noise reduction, normalization) Distribution: - CDN with live streaming optimization - Edge caching of recent segments - Adaptive bitrate switching Playback: - HLS for broad compatibility - WebRTC for lowest latency (newer browsers) - DASH for Android -
Live Comments Service
Requirements: - Sub-second latency for comments - Handle 10K+ concurrent commenters - Comment moderation in real-time - Pinned comments, emoji reactions Architecture: - WebSocket connection per viewer - Comments published to Kafka topic: live_{stream_id} - Moderation worker filters toxic content - Clean comments broadcast to all viewers via WebSocket - Comment history stored in Cassandra for replay -
Scalability
Concurrent streams: 100K (peak) Viewers per stream: 1M (top stream) Total concurrent viewers: 100M WebSocket connections: 100M concurrent Connection managers: 100M / 10K per instance = 10,000 instances Mitigation: - Connection multiplexing (one WS per device, not per stream) - Comment relay via server-sent events (SSE) for non-interactive viewers - Only active commenters get full WS connection
10. Failure Modes and Mitigations
10.1 Failure Mode Matrix
| Failure Mode | Probability | Impact | Detection | Mitigation |
|---|---|---|---|---|
| CDN outage | Low | Critical | CDN health checks, error rate spike | Multi-CDN failover, direct origin fallback |
| Database primary failure | Medium | Critical | Replication lag, write failures | Automatic failover to replica, read-only mode |
| Redis cluster partition | Medium | High | Cache miss rate spike, latency increase | Local cache fallback, degraded mode |
| Kafka broker failure | Low | High | Consumer lag, message loss | Multi-broker replication, exactly-once semantics |
| S3 outage | Very Low | Critical | Upload failures, media 404s | Multi-region replication, CDN cache sustains reads |
| ML model degradation | Medium | Medium | Engagement drop, A/B test failure | Model rollback, rule-based fallback |
| DDoS attack | Medium | Critical | Latency spike, resource exhaustion | WAF, rate limiting, auto-scaling, scrubbing center |
| Data breach | Low | Critical | Security monitoring, anomaly detection | Encryption, access controls, incident response |
| Cascade failure | Low | Critical | Cross-service error rate correlation | Circuit breakers, bulkheads, graceful degradation |
| Clock skew | Medium | Low | Duplicate IDs, ordering issues | NTP synchronization, logical clocks |
10.2 Detailed Mitigation Strategies
Circuit Breaker Pattern
Every inter-service call has a circuit breaker:
States:
CLOSED → Normal operation, requests flow through
OPEN → Failure threshold reached, requests fail fast
HALF-OPEN → Test requests allowed, monitor success rate
Configuration:
Failure threshold: 5 failures in 10 seconds → OPEN
Recovery timeout: 30 seconds → HALF-OPEN
Success threshold: 5 successes in HALF-OPEN → CLOSED
Failure in HALF-OPEN → OPEN
Fallback behaviors:
Feed Service: Return cached feed (stale but available)
Like Service: Queue like for later, return optimistic count
Notification Service: Queue notification, deliver when recovered
Search Service: Return empty results with "try again" message
Bulkhead Pattern
Isolate failures by partitioning resources:
Thread Pool Bulkheads:
Feed pool: 200 threads
Media pool: 100 threads
User pool: 50 threads
Notification pool: 50 threads
If Media pool is exhausted (video processing backlog),
Feed and User pools remain unaffected.
Connection Pool Bulkheads:
DB pool per service:
User DB: 50 connections
Post DB: 100 connections
Graph DB: 50 connections
If Post DB is slow, User DB connections are unaffected.
Graceful Degradation
When components fail, degrade gracefully instead of crashing:
Priority 1 (Always Available):
- User authentication
- Feed reading (cached)
- Profile viewing
Priority 2 (Degrade if Needed):
- Post creation (queue if processing is down)
- Likes/comments (accept but delay processing)
- Notifications (batch and delay)
Priority 3 (Disable if Needed):
- Explore feed (return empty or cached)
- Suggestions (return empty)
- Stories (disable creation, allow viewing cached)
- Live streaming (disable new streams)
User Communication:
- Show "Some features may be slow" banner
- Retry buttons for failed actions
- Offline mode for cached content
Disaster Recovery
RTO (Recovery Time Objective): 1 hour
RPO (Recovery Point Objective): 5 minutes
Strategy:
1. Multi-region active-passive deployment
- Primary region: handles all traffic
- Secondary region: warm standby, replicated data
- Failover: DNS switch + traffic reroute (15 minutes)
2. Data replication
- MySQL: Cross-region replication (5-minute lag)
- S3: Cross-region replication (continuous)
- Redis: Cross-region async replication
- Kafka: MirrorMaker for cross-region topic replication
3. Regular DR drills
- Monthly failover test (secondary region)
- Quarterly full DR exercise
- Annual chaos engineering game day
Chaos Engineering
Regularly inject failures to validate resilience:
Tests:
- Kill random service instances → Verify auto-recovery
- Simulate network partition → Verify split-brain handling
- Inject latency (500ms-5s) → Verify timeout handling
- Fill disk space → Verify graceful degradation
- Corrupt cache entries → Verify cache miss handling
- Simulate clock skew → Verify ID generation correctness
Tools:
- Chaos Monkey (random instance termination)
- Latency Monkey (random latency injection)
- Custom fault injection framework
10.3 Monitoring and Alerting
Key Metrics
Availability:
- Service uptime percentage (per service)
- Error rate (5xx responses / total requests)
- Circuit breaker state (per service pair)
Performance:
- Request latency (P50, P95, P99) per endpoint
- Throughput (QPS) per service
- Cache hit rate (CDN, Redis, local)
- Database query latency (P99)
Infrastructure:
- CPU, memory, disk, network utilization
- Connection pool usage
- Queue depth (Kafka consumer lag)
- Active WebSocket connections
Business:
- Daily active users (DAU)
- Posts created per hour
- Feed views per hour
- Upload success rate
- Time to first byte (TTFB) for media
Alerting Thresholds
| Metric | Warning | Critical | Action |
|---|---|---|---|
| Error Rate | > 1% for 5 min | > 5% for 2 min | Page on-call engineer |
| P99 Latency | > 500ms for 5 min | > 2s for 2 min | Auto-scale + page |
| Cache Hit Rate | < 50% for 10 min | < 30% for 5 min | Investigate cache invalidation |
| DB Replication Lag | > 5s for 5 min | > 30s for 2 min | Failover to replica |
| Kafka Consumer Lag | > 10K messages | > 100K messages | Scale consumers |
| CDN Error Rate | > 0.1% for 5 min | > 1% for 2 min | Switch to backup CDN |
Appendix A: Technology Stack Summary
| Layer | Technology | Alternative |
|---|---|---|
| Frontend | React Native (mobile), React (web) | Flutter, Swift/Kotlin native |
| API Gateway | Kong / Custom Go gateway | AWS API Gateway, Envoy |
| Backend Services | Go (performance), Java (complexity) | Python, Rust |
| Message Queue | Apache Kafka | RabbitMQ, AWS SQS |
| Relational DB | MySQL (Vitess sharding) | PostgreSQL (Citus) |
| NoSQL DB | Cassandra (graph), MongoDB (notifications) | DynamoDB, ScyllaDB |
| Cache | Redis Cluster | Memcached, Dragonfly |
| Search | Elasticsearch | OpenSearch, Meilisearch |
| Object Storage | AWS S3 | GCS, Azure Blob, MinIO |
| CDN | CloudFront + Akamai (multi-CDN) | Fastly, Cloudflare |
| Container Orchestration | Kubernetes | ECS, Nomad |
| Service Mesh | Istio | Linkerd, Consul |
| Monitoring | Prometheus + Grafana + ELK | Datadog, New Relic |
| CI/CD | GitLab CI / GitHub Actions | Jenkins, ArgoCD |
| Infrastructure as Code | Terraform | Pulumi, CloudFormation |
Appendix B: Cost Estimates at Scale
| Component | Monthly Cost (500M MAU) | Notes |
|---|---|---|
| CDN Egress | $15-25M | 300 EB/month at $0.05/GB |
| Object Storage | $5-8M | 2.4 EB at $0.023/GB |
| Compute (K8s) | $3-5M | ~10,000 instances |
| Databases | $2-3M | Managed DB instances |
| Cache (Redis) | $0.5-1M | Large Redis clusters |
| Search (ES) | $0.5-1M | Large ES clusters |
| ML Inference | $1-2M | GPU instances for ranking |
| Monitoring/Logging | $0.5-1M | Data ingestion and storage |
| Total | $27.5-46M/month | ~$330-550M/year |
Cost Optimization Levers:
- Image compression (WebP/AVIF) → 30-50% bandwidth savings
- CDN multi-provider negotiation → 20-30% cost reduction
- Tiered storage (hot/warm/cold) → 40% storage savings
- Spot instances for batch processing → 60-70% compute savings
- Data lifecycle policies → Auto-delete old logs, compress old data
Appendix C: Security Considerations
| Threat | Mitigation |
|---|---|
| Account Takeover | MFA, device fingerprinting, anomaly detection |
| DDoS | WAF, rate limiting, CDN absorption, auto-scaling |
| Data Breach | Encryption at rest (AES-256), in transit (TLS 1.3), key rotation |
| Content Abuse | AI moderation + human review + user reporting |
| CSRF/XSS | SameSite cookies, CSP headers, input sanitization |
| API Abuse | Rate limiting per user/IP, API key rotation, anomaly detection |
| Insider Threat | RBAC, audit logging, least privilege access |
| GDPR Compliance | Data export/delete APIs, consent management, DPA |
| Child Safety | Age verification, content filtering, reporting mechanisms |
Summary
This design document covers a production-ready, billion-user-scale social media platform inspired by Instagram. The key architectural principles are:
- Read-optimized by default — Feed is the most frequent operation; cache aggressively at every layer
- Hybrid feed model — Push for normal users, pull for celebrities; avoids the fan-out explosion
- Polyglot persistence — Each data type uses the optimal storage engine
- Event-driven architecture — Kafka decouples services and enables async processing
- Graceful degradation — Circuit breakers, bulkheads, and priority-based fallbacks
- Horizontal scalability — Every component scales linearly with sharding and stateless design
- Multi-layer caching — CDN → Redis → DB replicas → Primary DB
The system is designed to handle 500M+ MAU with 99.95% availability, sub-200ms feed latency, and linear cost scaling. Each curveball scenario demonstrates how the architecture adapts to new requirements without fundamental re-architecture.
Document Version: 1.0 | Last Updated: 2026-08-28