System Design: Emailing Service (Gmail-like)
Author: System Design Document | Date: 2026-08-28 Scope: End-to-end design of a cloud-scale email service supporting billions of users, trillions of messages, with real-time delivery, search, spam protection, and collaboration features.
Table of Contents
- Requirements Clarification
- Back-of-Envelope Calculations
- System Interface Design
- High-Level Architecture
- Core Component Design
- Data Model
- Key Design Decisions
- Scalability Strategy
- Curveball Scenarios
- Failure Modes and Mitigations
1. Requirements Clarification
1.1 Functional Requirements
| ID | Requirement | Priority | Notes |
|---|---|---|---|
| FR-01 | Send email — Compose, attach files, send to single/multiple recipients (To, CC, BCC) | P0 | Support inline images, HTML + plain text |
| FR-02 | Receive email — Ingest via SMTP, deliver to recipient mailbox in real-time | P0 | Support bounce handling, delivery receipts |
| FR-03 | Mailbox management — Inbox, Sent, Drafts, Trash, Spam, custom labels/folders | P0 | Labels (not folders) — an email can have multiple labels |
| FR-04 | Full-text search — Search by sender, recipient, subject, body, date range, labels, attachments | P0 | Sub-second latency for personal mailbox |
| FR-05 | Spam & phishing detection — Classify incoming mail as spam/ham/phishing | P0 | ML-based + rule-based hybrid |
| FR-06 | Attachment handling — Upload, store, preview, download attachments | P0 | Support common formats (PDF, images, Office docs) |
| FR-07 | Push notifications — Real-time notification of new mail to web/mobile clients | P1 | WebSocket / HTTP long-polling fallback |
| FR-08 | Contact management — Address book, auto-complete, contact groups | P1 | Sync with global directory (enterprise) |
| FR-09 | Compose & Drafts — Rich text editor, auto-save drafts, scheduled send | P1 | Offline support for mobile |
| FR-10 | IMAP/SMTP client access — Standard protocol support for third-party clients | P1 | IMAP IDLE for push, SMTP AUTH for send |
| FR-11 | Threading — Group related emails into conversation threads | P1 | By subject + participant heuristics |
| FR-12 | Quota management — Per-user storage quota enforcement | P1 | Soft/hard limits, cleanup suggestions |
| FR-13 | Encryption — TLS in transit, optional E2E encryption (S/MIME, PGP) | P1 | DMARC, DKIM, SPF for sender verification |
| FR-14 | Calendar integration — Detect events in email, surface in calendar | P2 | Optional |
| FR-15 | Smart features — Smart compose, smart reply, priority inbox, snooze | P2 | ML-powered |
1.2 Non-Functional Requirements
| ID | Requirement | Target | Rationale |
|---|---|---|---|
| NFR-01 | Availability | 99.99% (52.6 min downtime/year) | Email is a critical business service |
| NFR-02 | Delivery latency | P99 < 10 seconds for same-domain; < 30 seconds cross-domain | User expectation for “instant” delivery |
| NFR-03 | Search latency | P99 < 500ms for personal mailbox queries | Competitive with Gmail experience |
| NFR-04 | Scalability | 1B+ users, 100B+ messages, 1M+ QPS peak | Global-scale service |
| NFR-05 | Storage efficiency | < 2 bytes/byte overhead (compression + dedup) | Cost control at petabyte scale |
| NFR-06 | Durability | 99.9999999% (11 nines) — no message loss | Email is a legal record |
| NFR-07 | Security | Zero-trust architecture, DLP, anti-phishing | Compliance (GDPR, HIPAA for enterprise) |
| NFR-08 | Consistency | Eventual consistency for mailbox state; strong consistency for send | Balance latency vs correctness |
| NFR-09 | Internationalization | UTF-8 everywhere, RTL languages, timezone-aware | Global user base |
2. Back-of-Envelope Calculations
2.1 Assumption Sets
We reason from first principles with three assumption sets to bound the design space:
| Parameter | Conservative (A) | Base (B) | Aggressive (C) |
|---|---|---|---|
| Active users | 100M | 1B | 5B |
| Emails/user/day | 50 | 30 | 20 |
| Avg email size (with headers) | 100 KB | 50 KB | 30 KB |
| Avg attachment size | 500 KB | 200 KB | 100 KB |
| Attachment ratio | 20% | 15% | 10% |
| Ingestion rate (emails/sec) | 58K | 104K | 231K |
| Total daily emails | 5B | 3B | 1B |
Note: Set B (Base) is our primary design target. Sets A and C bound the extremes.
2.2 Storage Estimates (Base Case — 1B users)
Daily email volume: 1B users × 30 emails/user = 30B emails/day
Email body + headers: 30B × 50 KB = 1.5 PB/day
Attachments: 30B × 15% × 200 KB = 900 TB/day
Total daily ingest: ~2.4 PB/day
Annual storage (raw): 2.4 PB/day × 365 = 876 PB/year
With 3x replication: 876 × 3 = 2.63 EB/year
With compression (2:1): ~1.3 EB/year effective
Storage breakdown per user (1 year):
| Component | Size/user | Notes |
|---|---|---|
| Email bodies | ~550 MB | 30 emails/day × 50 KB × 365 |
| Attachments | ~11 GB | 30 × 15% × 200 KB × 365 |
| Index data | ~200 MB | Inverted index, metadata |
| Metadata | ~50 MB | Labels, threads, flags |
| Total/user | ~11.8 GB | Before replication |
2.3 Bandwidth Estimates
Inbound SMTP: 104K emails/sec × 50 KB = 5.2 GB/s (41.6 Gbps)
Outbound SMTP: 104K emails/sec × 50 KB = 5.2 GB/s (41.6 Gbps)
Attachment CDN: 104K × 15% × 200 KB = 3.1 GB/s (24.8 Gbps)
API traffic: ~200K req/s × 5 KB avg = 1 GB/s (8 Gbps)
Internal replication: ~10 GB/s (80 Gbps)
Total network: ~20 GB/s (160 Gbps) per major region
2.4 QPS Estimates
| Operation | QPS (Base) | Read/Write | Notes |
|---|---|---|---|
| SMTP ingest | 104K | Write | Peak may be 3-5x during business hours |
| IMAP fetch | 200K | Read | Polling + IDLE clients |
| Web API reads | 500K | Read | Inbox load, message view |
| Search queries | 50K | Read | Full-text search |
| Compose/send | 30K | Write | Draft save + send |
| Push notifications | 100K | Push | WebSocket connections |
| Spam scanning | 104K | Read+Write | Every inbound email |
| Total | ~1.1M QPS | ~70% reads | Per region |
2.5 Infrastructure Sizing (Per Region)
Assuming 4 regions (US, EU, APAC, LATAM), each handling ~25% of traffic:
SMTP Ingestion Servers: 104K QPS / 25% / 1K req/s per server = ~104 servers
Web API Servers: 500K QPS / 25% / 500 req/s per server = ~400 servers
Search Cluster: 50K QPS / 25% / 500 queries/s per node = ~40 nodes
Mailbox Storage Nodes: 250M users / 500K users per node = ~500 nodes
Object Storage (attachments): Shared regional cluster
Message Queue Brokers: ~50 brokers (Kafka, 3 replicas)
Cache (Redis/Memcached): ~200 nodes (for session, hot data)
Spam ML Inference: ~50 GPU/TPU nodes
3. System Interface Design
3.1 REST API Definitions
3.1.1 Mailbox Operations
GET /api/v1/users/{userId}/mailbox/labels
List all labels for a user
Response: { labels: [{ id, name, messageCount, unreadCount }] }
POST /api/v1/users/{userId}/mailbox/labels
Create a new label
Body: { name, color?, parentLabelId? }
GET /api/v1/users/{userId}/mailbox/messages?label={labelId}&page={n}&limit={limit}&sort={date|relevance}
List messages with pagination
Response: { messages: [{ id, snippet, from, to, subject, date, labels, starred, unread }], nextPageToken }
GET /api/v1/users/{userId}/mailbox/messages/{messageId}
Get full message content
Response: { id, headers, bodyHtml, bodyPlain, attachments, labels, threadId }
PUT /api/v1/users/{userId}/mailbox/messages/{messageId}/labels
Modify labels on a message
Body: { add: [labelId], remove: [labelId] }
DELETE /api/v1/users/{userId}/mailbox/messages/{messageId}
Delete (move to trash) a message
3.1.2 Compose & Send
POST /api/v1/users/{userId}/compose
Create a new draft
Body: { to: [], cc: [], bcc: [], subject, bodyHtml?, bodyPlain?, attachments: [] }
Response: { draftId }
PUT /api/v1/users/{userId}/compose/{draftId}
Update a draft
POST /api/v1/users/{userId}/compose/{draftId}/send
Send a draft
Body: { scheduledTime? } // optional for scheduled send
Response: { messageId, status }
POST /api/v1/users/{userId}/send
Send email directly (no draft)
Body: { to: [], cc: [], bcc: [], subject, bodyHtml?, bodyPlain?, attachments: [] }
Response: { messageId, status }
3.1.3 Search
GET /api/v1/users/{userId}/search?q={query}&label={labelId}&dateRange={start,end}&page={n}
Full-text search across mailbox
Query syntax: from:xxx to:xxx subject:xxx has:attachment after:2024-01-01 before:2024-12-31
Response: { results: [{ id, snippet, from, subject, date, labels }], totalHits, nextPageToken }
3.1.4 Attachments
POST /api/v1/users/{userId}/attachments/upload
Upload attachment (multipart)
Response: { attachmentId, url, size, mimeType }
GET /api/v1/users/{userId}/attachments/{attachmentId}/download
Download attachment
Response: Binary stream
GET /api/v1/users/{userId}/attachments/{attachmentId}/preview
Generate preview (thumbnail/first page)
Response: Image URL or embedded HTML
3.1.5 Contacts
GET /api/v1/users/{userId}/contacts
List contacts
Response: { contacts: [{ id, name, emails: [], phoneNumbers: [], groups: [] }] }
POST /api/v1/users/{userId}/contacts
Create contact
Body: { name, emails: [{address, type}], phoneNumbers: [{number, type}], groups: [] }
GET /api/v1/users/{userId}/contacts/suggest?q={query}
Auto-complete contact search
Response: { suggestions: [{ id, name, email }] }
3.1.6 Notifications (WebSocket)
WS /api/v1/users/{userId}/notifications
Real-time push channel
Events:
- NEW_MESSAGE: { messageId, labelId, from, subject, snippet }
- LABEL_CHANGED: { messageId, added: [], removed: [] }
- DRAFT_SAVED: { draftId }
- QUOTA_WARNING: { usedPercent, totalBytes }
3.2 SMTP Protocol Interface
Standard SMTP (RFC 5321) with extensions:
Supported ESMTP extensions:
- STARTTLS (RFC 3207) — Mandatory TLS upgrade
- AUTH (RFC 4954) — PLAIN, LOGIN, XOAUTH2
- 8BITMIME (RFC 6152) — 8-bit data support
- SIZE (RFC 1870) — Maximum message size declaration
- DSN (RFC 3461) — Delivery Status Notifications
- SMTPUTF8 (RFC 6531) — Internationalized email addresses
Connection flow:
Client -> EHLO/HELO -> STARTTLS -> AUTH XOAUTH2 -> MAIL FROM -> RCPT TO -> DATA -> . -> 250 OK
3.3 IMAP Protocol Interface
Standard IMAP4rev1 (RFC 3501) with extensions:
Supported IMAP extensions:
- IDLE (RFC 2177) — Server push for new mail
- CONDSTORE (RFC 4466) — Mod-sequence for efficient sync
- UNSELECT (RFC 3691) — Deselect without expunge
- MOVE (RFC 6851) — Atomic message move
- LIST-EXTENDED (RFC 5890) — Subscribe status
- XLIST (RFC 5890) — Mailbox listing with attributes
Connection flow:
Client -> LOGIN -> SELECT INBOX -> FETCH/SEARCH/STORE -> IDLE (wait for updates)
4. High-Level Architecture
flowchart TB
subgraph Clients["Clients"]
Web[Web Client<br/>React/PWA]
Mobile[Mobile App<br/>iOS/Android]
ThirdParty[Third-Party Clients<br/>IMAP/SMTP]
end
subgraph Edge["Edge Layer"]
CDN[CDN / WAF<br/>Static Assets + Attachments]
LB[Global Load Balancer<br/>GSLB + L7 LB]
Auth[Auth Gateway<br/>OAuth2 / SSO]
end
subgraph API["API Layer"]
WebAPI[Web API Gateway<br/>REST + GraphQL]
SMTPGW[SMTP Gateway<br/>RFC 5321]
IMAPGW[IMAP Gateway<br/>RFC 3501]
WS[WebSocket Gateway<br/>Push Notifications]
end
subgraph Services["Service Layer"]
ComposeSvc[Compose Service]
SendSvc[Send Service]
MailboxSvc[Mailbox Service]
SearchSvc[Search Service]
SpamSvc[Spam & Security Service]
ContactSvc[Contact Service]
AttachSvc[Attachment Service]
NotifySvc[Notification Service]
ThreadSvc[Threading Service]
QuotaSvc[Quota Service]
end
subgraph Pipeline["Ingestion Pipeline"]
MQ[(Message Queue<br/>Kafka)]
SpamFilter[Spam Filter<br/>ML + Rules]
VirusScan[Virus Scanner<br/>ClamAV / Custom]
DKIMVerify[DKIM/SPF/DMARC<br/>Verifier]
Indexer[Indexer<br/>Search + Metadata]
end
subgraph Storage["Storage Layer"]
MsgStore[(Message Store<br/>Distributed KV / Cassandra)]
AttachStore[(Object Store<br/>S3-compatible)]
SearchIdx[(Search Index<br/>Elasticsearch / Solr)]
Cache[(Cache Layer<br/>Redis Cluster)]
ContactDB[(Contact DB<br/>PostgreSQL)]
UserDB[(User DB<br/>PostgreSQL)]
MetadataDB[(Metadata DB<br/>Cassandra)]
end
subgraph ML["ML Infrastructure"]
SpamML[Spam Classifier<br/>Deep Learning]
PhishML[Phishing Detector<br/>URL + Content Analysis]
SmartML[Smart Features<br/>Compose / Reply / Priority]
FeatureStore[Feature Store]
end
subgraph Infra["Infrastructure"]
Monitor[Monitoring<br/>Prometheus + Grafana]
Log[Logging<br/>ELK Stack]
Config[Config Management<br/>Consul / etcd]
CI[CI/CD Pipeline]
end
%% Client flows
Web --> LB
Mobile --> LB
ThirdParty --> LB
LB --> Auth
Auth --> WebAPI
Auth --> SMTPGW
Auth --> IMAPGW
Auth --> WS
%% API to Services
WebAPI --> ComposeSvc
WebAPI --> MailboxSvc
WebAPI --> SearchSvc
WebAPI --> ContactSvc
WebAPI --> AttachSvc
SMTPGW --> MQ
IMAPGW --> MailboxSvc
%% Service interactions
ComposeSvc --> SendSvc
SendSvc --> MQ
SendSvc --> DKIMVerify
%% Ingestion pipeline
MQ --> SpamFilter
SpamFilter --> VirusScan
VirusScan --> DKIMVerify
DKIMVerify --> Indexer
Indexer --> MsgStore
Indexer --> SearchIdx
Indexer --> MetadataDB
%% Spam/ML
SpamFilter --> SpamML
SpamFilter --> PhishML
SpamML --> FeatureStore
PhishML --> FeatureStore
%% Services to Storage
MailboxSvc --> MsgStore
MailboxSvc --> MetadataDB
MailboxSvc --> Cache
SearchSvc --> SearchIdx
AttachSvc --> AttachStore
ContactSvc --> ContactDB
NotifySvc --> WS
QuotaSvc --> UserDB
%% Smart features
ComposeSvc --> SmartML
SmartML --> FeatureStore
%% Monitoring
Monitor -.-> Services
Log -.-> Services
Config -.-> Services
4.1 Architecture Principles
- Layered decomposition — Clear separation between edge, API, service, pipeline, and storage layers
- Async-first ingestion — All email ingestion flows through Kafka for decoupling and replayability
- Shared-nothing services — Each service is independently deployable and scalable
- Write-once, read-many — Messages are immutable; updates create new versions
- Region-local with cross-region replication — Data residency compliance + disaster recovery
5. Core Component Design
5.1 SMTP Ingestion Pipeline
The SMTP pipeline is the entry point for all inbound email. It must handle massive concurrent connections while performing security checks.
flowchart LR
subgraph SMTP["SMTP Gateway Cluster"]
Conn[Connection Handler<br/>Accept TCP 25/587]
Proto[Protocol Parser<br/>SMTP State Machine]
Auth[Auth Module<br/>XOAUTH2 / PLAIN]
end
subgraph Validation["Validation Chain"]
SPF[SPF Check<br/>Sender IP verification]
DKIM[DKIM Verify<br/>Cryptographic signature]
DMARC[DMARC Policy<br/>Alignment check]
Rate[Rate Limiter<br/>Per-sender throttling]
end
subgraph Queue["Buffering"]
Kafka[Kafka Topics<br/>raw_email, spam_suspect, quarantine]
end
Conn --> Proto --> Auth --> SPF --> DKIM --> DMARC --> Rate --> Kafka
Design Details
Connection Handling:
- Each SMTP gateway node handles ~10K concurrent connections using async I/O (epoll/io_uring)
- TLS termination at the gateway (TLS 1.3 preferred)
- Connection pooling for upstream services
Protocol Parsing:
- State machine implementing RFC 5321
- Streaming parser — no full buffering until DATA command
- Header/body separation at parse time
Validation Chain (pipeline pattern):
| Stage | Purpose | Failure Action | Latency Budget |
|---|---|---|---|
| SPF | Verify sender IP authorized by domain DNS | Soft fail (log + flag) | < 50ms |
| DKIM | Verify cryptographic signature integrity | Soft fail (log + flag) | < 100ms |
| DMARC | Enforce domain policy (none/quarantine/reject) | Per-policy action | < 20ms |
| Rate Limit | Prevent abuse per sender IP/domain | 451 Temporary failure | < 5ms |
Kafka Topic Design:
Topic: raw_email
Partition key: recipient_domain (for domain-level ordering)
Retention: 7 days
Replication: 3
Schema: { messageId, envelopeFrom, envelopeTo[], rawHeaders, bodyRef, timestamp, sourceIp }
Topic: spam_suspect
Partition key: messageId
For messages flagged for deeper ML analysis
Topic: delivery_queue
Partition key: recipientUserId
For final delivery to mailbox
5.2 Mailbox Storage and Indexing
The mailbox is the core data structure. Each user’s mailbox is a collection of messages with labels, flags, and metadata.
flowchart TB
subgraph WritePath["Write Path (Ingest)"]
Ingest[Ingestion Service]
Dedup[Deduplication Check<br/>Message-ID + Content Hash]
Store[Message Store<br/>Append-only KV]
MetaWrite[Metadata Writer<br/>Labels, Flags, Counters]
IndexWrite[Index Writer<br/>Search + Lookup]
end
subgraph ReadPath["Read Path (Fetch)"]
Cache[Cache Layer<br/>Hot messages]
StoreR[Message Store<br/>Range scan]
MetaRead[Metadata Reader<br/>Labels, Counters]
Merge[Merge & Project<br/>Final response]
end
Ingest --> Dedup --> Store --> MetaWrite --> IndexWrite
Cache --> StoreR --> MetaRead --> Merge
Storage Architecture
Message Store (Distributed KV — Cassandra/ScyllaDB):
Keyspace: messages
Primary key: (userId, messageId)
Columns: {
userId: UUID,
messageId: UUID, // Generated or from Message-ID header
rawMessage: blob, // Compressed raw RFC 2822 message
headers: map<text, text>, // Parsed headers
bodyPlain: text, // Plain text body
bodyHtml: text, // HTML body
size: bigint, // Total size in bytes
receivedTime: timestamp,
labels: set<text>, // Label IDs
flags: set<text>, // Seen, Answered, Flagged, Deleted, Draft
threadId: UUID, // Conversation thread
attachmentRefs: list<text>,// References to object store
checksum: text, // SHA-256 for integrity
version: bigint // For optimistic concurrency
}
Label Index (for fast label-based queries):
Keyspace: label_index
Primary key: (userId, labelId, receivedTime DESC)
Columns: {
userId: UUID,
labelId: text,
receivedTime: timestamp,
messageId: UUID,
snippet: text, // First 200 chars of body
from: text,
subject: text
}
Counters (for unread/message counts):
Keyspace: mailbox_counters
Primary key: (userId, labelId)
Columns: {
userId: UUID,
labelId: text,
totalMessages: counter,
unreadMessages: counter,
totalSize: counter
}
Indexing Strategy
| Index Type | Purpose | Technology | Update Frequency |
|---|---|---|---|
| Primary index | userId + messageId lookup | Cassandra PK | On write |
| Label index | List messages by label | Cassandra secondary table | On write |
| Thread index | Group by conversation | Cassandra (threadId) | On write |
| Full-text index | Search body/subject | Elasticsearch | Async (near-real-time) |
| Date index | Date-range queries | Cassandra (receivedTime) | On write |
| Sender index | “from:xxx” queries | Elasticsearch | Async |
5.3 Spam and Phishing Detection
A multi-layered defense system combining rule-based filters, statistical models, and deep learning.
flowchart TB
subgraph Layer1["Layer 1: Reputation & Rules (Fast Path)"]
IPRep[IP Reputation<br/>Blocklist + Score]
DomainRep[Domain Reputation<br/>Age, History, Blacklists]
HeaderRules[Header Rules<br/>Suspicious patterns]
RateCheck[Rate Analysis<br/>Volume anomalies]
end
subgraph Layer2["Layer 2: Content Analysis (Medium Path)"]
Bayesian[Bayesian Filter<br/>Word frequency]
URLScan[URL Analysis<br/>Shortener, Redirect, Malicious]
AttachScan[Attachment Scan<br/>HEIC, Macro, Executable]
SPFResult[SPF/DKIM/DMARC<br/>Results scoring]
end
subgraph Layer3["Layer 3: ML Deep Analysis (Slow Path)"]
DLModel[Deep Learning Model<br/>BERT-based classifier]
GraphML[Graph Analysis<br/>Sender-recipient patterns]
PhishML[Phishing Detector<br/>Visual + URL similarity]
end
subgraph Actions["Actions"]
HAM[Deliver to Inbox]
SPAM[Move to Spam Folder]
PHISH[Quarantine + Alert]
QUARANTINE[Hold for Review]
end
IPRep --> Bayesian --> DLModel --> HAM
IPRep --> Bayesian --> DLModel --> SPAM
DomainRep --> Bayesian
HeaderRules --> Bayesian
RateCheck --> Bayesian
URLScan --> DLModel
AttachScan --> DLModel
SPFResult --> Bayesian
DLModel --> PHISH
PhishML --> PHISH
GraphML --> QUARANTINE
Spam Scoring Pipeline
Each email receives a composite spam score (0.0 to 1.0):
spam_score = w1 * reputation_score + w2 * content_score + w3 * ml_score + w4 * graph_score
Where:
reputation_score: Based on sender IP/domain reputation (0-1)
content_score: Bayesian + rule-based content analysis (0-1)
ml_score: Deep learning model prediction (0-1)
graph_score: Anomaly in sender-recipient graph (0-1)
Weights (tunable):
w1 = 0.25, w2 = 0.30, w3 = 0.35, w4 = 0.10
Decision thresholds:
spam_score < 0.3: HAM — Deliver to inbox
0.3 <= score < 0.7: SPAM — Move to spam folder
0.7 <= score < 0.9: PHISH — Quarantine, notify user
score >= 0.9: BLOCK — Reject outright
Phishing Detection Specifics
| Technique | Description | Detection Rate |
|---|---|---|
| URL similarity | Compare URLs to known brand domains (Levenshtein, visual similarity) | ~95% |
| Logo/image analysis | CNN-based comparison of logos against brand database | ~90% |
| Domain age check | Newly registered domains flagged | ~80% |
| Impersonation detection | Display name vs actual sender mismatch | ~85% |
| Behavioral analysis | Unusual sender patterns for known contacts | ~75% |
| URL sandboxing | Execute URLs in sandbox, analyze behavior | ~98% (for queued URLs) |
5.4 Full-Text Search at Scale
Search is one of Gmail’s killer features. It requires indexing billions of messages with sub-second response times.
flowchart TB
subgraph Indexing["Indexing Pipeline"]
Stream[Kafka Stream<br/>New/Updated Messages]
Parse[Message Parser<br/>Extract searchable fields]
NLP[NLP Pipeline<br/>Tokenization, Stemming, NER]
Chunk[Chunking<br/>Split large bodies]
Bulk[Bulk Indexer<br/>Batch to ES]
end
subgraph SearchCluster["Search Cluster (Elasticsearch)"]
Master[Master Nodes<br/>Cluster coordination]
Data[Data Nodes<br/>Shards + Replicas]
Coords[Coordinating Nodes<br/>Query routing]
end
subgraph Query["Query Processing"]
Parser[Query Parser<br/>Gmail query syntax]
Rewrite[Query Rewriter<br/>Synonyms, corrections]
Exec[Query Executor<br/>Distributed search]
Rank[Ranker<br/>BM25 + ML ranking]
Cache[Result Cache<br/>Hot queries]
end
Stream --> Parse --> NLP --> Chunk --> Bulk --> Data
Parser --> Rewrite --> Exec --> Rank
Coords --> Data
Cache --> Exec
Index Schema (Elasticsearch)
{
"mappings": {
"properties": {
"messageId": { "type": "keyword" },
"userId": { "type": "keyword" },
"from": {
"type": "object",
"properties": {
"address": { "type": "keyword" },
"name": { "type": "text", "analyzer": "email_analyzer" }
}
},
"to": {
"type": "nested",
"properties": {
"address": { "type": "keyword" },
"name": { "type": "text", "analyzer": "email_analyzer" }
}
},
"subject": { "type": "text", "analyzer": "subject_analyzer", "fields": { "keyword": { "type": "keyword" } } },
"body": { "type": "text", "analyzer": "body_analyzer" },
"date": { "type": "date" },
"labels": { "type": "keyword" },
"threadId": { "type": "keyword" },
"hasAttachment": { "type": "boolean" },
"attachmentNames": { "type": "text", "analyzer": "attachment_analyzer" },
"size": { "type": "long" }
}
},
"settings": {
"number_of_shards": 100,
"number_of_replicas": 2,
"refresh_interval": "5s"
}
}
Custom Analyzers
| Analyzer | Purpose | Components |
|---|---|---|
email_analyzer |
Parse email addresses and names | Email tokenizer, lowercase, stop words |
subject_analyzer |
Index email subjects | Edge n-gram (for prefix search), synonym filter |
body_analyzer |
Index email body text | ICU tokenizer, stemmer, stop words, synonym filter |
attachment_analyzer |
Index attachment filenames | Keyword tokenizer, lowercase |
Query Syntax Support
from:john@example.com
to:jane@example.com
subject:meeting
has:attachment
larger:5M
before:2024/06/01
after:2024/01/01
label:work -label:spam
is:unread
is:starred
filename:pdf
in:inbox
Search Performance Optimization
- User-scoped routing — Each user’s messages routed to specific shards (hash(userId) % num_shards)
- Result caching — LRU cache for frequent queries (same user + same query)
- Snippet pre-computation — Store searchable snippet at index time
- Pagination optimization — Search-after (scroll cursor) instead of deep offset
- Async indexing — Index updates batched and applied every 5 seconds (near-real-time)
5.5 Label / Category Management
Gmail uses labels instead of folders — a message can have multiple labels simultaneously.
flowchart LR
subgraph Labels["Label System"]
System[System Labels<br/>INBOX, SENT, DRAFTS,<br/>TRASH, SPAM, IMPORTANT,<br/>CATEGORY_PERSONAL,<br/>CATEGORY_PROMOTIONS,<br/>CATEGORY_SOCIAL,<br/>CATEGORY_UPDATES,<br/>CATEGORY_FORUMS]
User[User Labels<br/>Custom created]
Auto[Auto-Labels<br/>ML-categorized]
end
subgraph Storage["Label Storage"]
LabelMeta[(Label Metadata<br/>Name, Color, Hierarchy)]
LabelIdx[(Label-Message Index<br/>userId + labelId -> messageIds)]
Counters[(Counters<br/>Per-label message/unread counts)]
end
System --> LabelMeta
User --> LabelMeta
Auto --> LabelMeta
LabelMeta --> LabelIdx
LabelIdx --> Counters
Label Operations
| Operation | Implementation | Latency |
|---|---|---|
| Add label to message | Write to label_index + increment counter | < 10ms |
| Remove label from message | Delete from label_index + decrement counter | < 10ms |
| List messages by label | Scan label_index by (userId, labelId, time DESC) | < 50ms |
| Get unread count | Read counter from mailbox_counters | < 5ms |
| Create label | Write to label_metadata | < 10ms |
| Delete label | Cascade delete from label_index + metadata | < 100ms |
Auto-Categorization (ML)
Input features:
- Sender domain reputation
- Message content (NLP embeddings)
- Historical user interaction (opened, deleted, marked spam)
- Time of day patterns
- Recipient list characteristics
Model: Multi-class classifier (Personal, Promotions, Social, Updates, Forums)
Architecture: LightGBM / BERT-small
Training: Online learning with user feedback loop
Update frequency: Daily batch + hourly incremental
5.6 Compose Service
flowchart TB
subgraph Editor["Rich Text Editor"]
RTE[Rich Text Editor<br/>Quill / Custom]
InlineImg[Inline Image Handler]
Template[Template Engine<br/>Signatures, Canned responses]
end
subgraph Draft["Draft Management"]
AutoSave[Auto-Save<br/>Debounced, every 30s]
Versioning[Version Control<br/>Keep last 10 versions]
Recovery[Recovery<br/>Restore from version]
end
subgraph Send["Send Preparation"]
Validate[Validation<br/>Recipient check, size check]
Encrypt[Encryption<br/>S/MIME if configured]
Queue[Queue for Send<br/>Async via Kafka]
end
RTE --> AutoSave --> Versioning
InlineImg --> RTE
Template --> RTE
AutoSave --> Draft
Draft --> Validate --> Encrypt --> Queue
Auto-Save Strategy
Client-side:
- Debounce: Save after 30s of inactivity
- Throttle: Save at most every 5s during active typing
- Local storage: Keep draft in localStorage as fallback
- Conflict resolution: Last-write-wins with timestamp
Server-side:
- Store in drafts table (same schema as messages, with isDraft=true)
- TTL: 90 days for inactive drafts
- Cleanup: Async job removes drafts older than TTL
5.7 Attachment Handling
flowchart TB
subgraph Upload["Upload Path"]
Client[Client]
Multipart[Multipart Upload<br/>For large files]
VirusScan[Anti-Virus Scan<br/>On-write]
Dedup[Content Dedup<br/>SHA-256 check]
ObjStore[(Object Store<br/>S3-compatible)]
end
subgraph Download["Download Path"]
CDN[CDN Edge<br/>Cached attachments]
Cache[Attachment Cache<br/>Hot attachments]
ObjStoreR[(Object Store)]
end
subgraph Preview["Preview Generation"]
PDF[PDF Renderer<br/>First page thumbnail]
Img[Image Resizer<br/>Thumbnail generation]
Doc[Doc Converter<br/>Office -> HTML preview]
Video[Video Thumbnail<br/>First frame capture]
end
Client --> Multipart --> VirusScan --> Dedup --> ObjStore
CDN --> Cache --> ObjStoreR
ObjStore --> PDF
ObjStore --> Img
ObjStore --> Doc
ObjStore --> Video
Attachment Storage Design
| Aspect | Design | Rationale |
|---|---|---|
| Storage | S3-compatible object store (Ceph / MinIO) | Scalable, cheap, durable |
| Deduplication | Content-addressable (SHA-256 as key) | Save storage for common attachments |
| Lifecycle | Hot (SSD) → Warm (HDD) → Cold (Glacier) | Cost optimization |
| Max size | 25 MB per attachment (standard), 50 MB (enterprise) | Balance UX vs infrastructure |
| Virus scanning | On-write scan + periodic re-scan | Security |
| Preview | Async generation on upload | Non-blocking upload |
Content Deduplication
Upload flow:
1. Client computes SHA-256 of attachment
2. Check if hash exists in dedup_index (Redis)
3. If exists: create reference only (no storage write)
4. If not: upload to object store, record hash in dedup_index
Storage savings estimate:
- Common attachments (logos, signatures, stock images): ~30% dedup rate
- Enterprise environments: ~40% dedup rate (shared documents)
5.8 Push Notifications
flowchart TB
subgraph Events["Event Sources"]
NewMail[New Mail Delivered]
LabelChange[Label Modified]
DraftSaved[Draft Auto-Saved]
QuotaWarn[Quota Warning]
end
subgraph Gateway["WebSocket Gateway"]
ConnMgr[Connection Manager<br/>Active connections map]
Router[Event Router<br/>userId -> connections]
Heartbeat[Heartbeat<br/>Keep-alive every 30s]
Reconnect[Reconnect Handler<br/>Exponential backoff]
end
subgraph Clients["Client Connections"]
WebWS[Web Client<br/>WebSocket]
MobileWS[Mobile Client<br/>WebSocket / APNs / FCM]
IMAPIdle[IMAP Clients<br/>IDLE command]
end
NewMail --> Router
LabelChange --> Router
DraftSaved --> Router
QuotaWarn --> Router
Router --> ConnMgr --> WebWS
ConnMgr --> MobileWS
ConnMgr --> IMAPIdle
Heartbeat -.-> ConnMgr
Notification Delivery Strategy
| Client Type | Protocol | Fallback | Reconnection |
|---|---|---|---|
| Web browser | WebSocket | HTTP long-polling (every 30s) | Exponential backoff (1s, 2s, 4s, … 60s max) |
| Mobile (foreground) | WebSocket | N/A | Same as web |
| Mobile (background) | APNs (iOS) / FCM (Android) | Polling on next foreground | OS-managed |
| IMAP clients | IMAP IDLE | CONDSTORE + periodic CHECK | Client-managed |
Connection Scaling
WebSocket Gateway sizing:
- Each node handles ~50K concurrent connections
- Connection state stored in Redis (userId -> nodeId mapping)
- Sticky sessions via LB (user always reconnects to same node)
- Graceful drain on node restart (redirect connections)
For 1B users, ~100M concurrent connections:
Nodes needed: 100M / 50K = 2,000 gateway nodes
5.9 Contact Management
flowchart TB
subgraph Sources["Contact Sources"]
Manual[Manual Entry<br/>User-created]
Auto[Auto-Extracted<br/>From sent/received emails]
Global[Global Directory<br/>Enterprise LDAP/AD]
Import[Import<br/>CSV, vCard, Gmail]
end
subgraph Storage["Contact Storage"]
ContactStore[(Contact Store<br/>PostgreSQL)]
ContactIdx[(Contact Index<br/>Search + Auto-complete)]
Merge[(Merge Engine<br/>Duplicate detection)]
end
subgraph Features["Features"]
Autocomplete[Auto-complete<br/>Type-ahead in To/CC/BCC]
Groups[Contact Groups<br/>Aliases for distributions]
Avatar[Avatar Service<br/>Gravatar / Corporate photo]
Presence[Presence Indicator<br/>Online/Busy/Away]
end
Manual --> ContactStore
Auto --> Merge --> ContactStore
Global --> ContactStore
Import --> Merge --> ContactStore
ContactStore --> ContactIdx --> Autocomplete
ContactStore --> Groups
ContactStore --> Avatar
Auto-Contact Extraction
Process:
1. On each sent/received email, extract all addresses from From, To, CC, BCC
2. Check if address exists in user's contact store
3. If not, create "auto-contact" with:
- Email address
- Display name (from headers)
- Last contact date
- Contact frequency counter
- Source: "auto-extracted"
4. If exists, update last contact date and frequency
Merge rules:
- Same email address = same contact
- Same name + different addresses = potential merge (user confirmation)
- Frequency threshold: Show in auto-complete after 2+ interactions
6. Data Model
6.1 Entity Relationship Overview
erDiagram
USER ||--o{ MESSAGE : owns
USER ||--o{ LABEL : creates
USER ||--o{ CONTACT : manages
USER ||--o{ DRAFT : writes
MESSAGE ||--o{ ATTACHMENT : contains
MESSAGE }o--|| THREAD : belongs-to
MESSAGE }o--o{ LABEL : labeled-with
MESSAGE ||--o{ RECIPIENT : addressed-to
CONTACT }o--o{ CONTACT_GROUP : member-of
USER {
UUID userId PK
VARCHAR email
VARCHAR displayName
VARCHAR passwordHash
BIGINT storageUsed
BIGINT storageQuota
TIMESTAMP createdAt
TIMESTAMP lastLogin
JSON settings
}
MESSAGE {
UUID messageId PK
UUID userId FK
UUID threadId FK
VARCHAR messageIdHeader
MAP headers
TEXT bodyPlain
TEXT bodyHtml
BIGINT size
TIMESTAMP receivedTime
SET flags
TEXT checksum
BIGINT version
}
LABEL {
UUID labelId PK
UUID userId FK
VARCHAR name
VARCHAR color
UUID parentLabelId
BOOLEAN systemLabel
BOOLEAN showInList
TIMESTAMP createdAt
}
LABEL_MESSAGE {
UUID userId FK
VARCHAR labelId FK
UUID messageId FK
TIMESTAMP receivedTime
TEXT snippet
TEXT from
TEXT subject
}
ATTACHMENT {
UUID attachmentId PK
UUID messageId FK
VARCHAR contentHash
VARCHAR fileName
VARCHAR mimeType
BIGINT size
VARCHAR storagePath
TIMESTAMP uploadedAt
}
THREAD {
UUID threadId PK
UUID userId FK
VARCHAR subject
UUID lastMessageId
TIMESTAMP lastActivity
BIGINT messageCount
BIGINT participantCount
}
CONTACT {
UUID contactId PK
UUID userId FK
VARCHAR name
JSON emails
JSON phoneNumbers
JSON addresses
VARCHAR avatarUrl
BOOLEAN isAutoExtracted
TIMESTAMP lastContact
BIGINT contactFrequency
}
DRAFT {
UUID draftId PK
UUID userId FK
JSON recipients
VARCHAR subject
TEXT bodyHtml
TEXT bodyPlain
JSON attachmentRefs
TIMESTAMP lastModified
TIMESTAMP scheduledSendTime
}
6.2 Message Headers Storage
{
"messageId": "msg_abc123",
"headers": {
"From": "John Doe <john@example.com>",
"To": "Jane Smith <jane@example.com>",
"Cc": "Team <team@example.com>",
"Subject": "Q4 Planning Meeting",
"Date": "2026-08-28T10:30:00Z",
"Message-ID": "<unique-id@example.com>",
"In-Reply-To": "<parent-id@example.com>",
"References": "<grandparent-id@example.com> <parent-id@example.com>",
"Content-Type": "multipart/alternative",
"X-Priority": "1",
"X-Mailer": "Gmail-Clone/1.0",
"DKIM-Signature": "v=1; a=rsa-sha256; ...",
"Received-SPF": "pass (example.com: domain designates ...)",
"Authentication-Results": "example.com; dkim=pass; spf=pass; dmarc=pass"
}
}
6.3 Thread Construction Algorithm
Thread grouping rules:
1. Same Message-ID References chain (strongest signal)
2. Same normalized subject + overlapping participants
3. Reply detection: In-Reply-To header matches another message's Message-ID
Normalization:
- Strip "Re:", "Fw:", "Rw:" prefixes
- Trim whitespace
- Case-insensitive comparison
Thread ID generation:
- If In-Reply-To exists and references a known message: inherit threadId
- Else if subject matches existing thread (within 30 days): join thread
- Else: create new thread with new threadId
7. Key Design Decisions
7.1 Labels vs Folders
| Aspect | Labels (Gmail-style) | Folders (Traditional) |
|---|---|---|
| Model | Many-to-many (message ↔ label) | One-to-one (message → folder) |
| Flexibility | High — organize multiple ways | Low — single categorization |
| Complexity | Higher (multiple indexes) | Lower (single parent pointer) |
| User mental model | Set-based thinking | Hierarchical thinking |
| Storage overhead | Higher (label_message junction) | Lower |
Decision: Labels — More flexible, matches modern user expectations. The storage overhead is acceptable given the indexing benefits.
7.2 Message Storage: Raw vs Parsed
| Aspect | Store Raw | Store Parsed | Store Both |
|---|---|---|---|
| Fidelity | Perfect (RFC 2822) | Lossy | Perfect + convenient |
| Read performance | Slow (parse on read) | Fast | Fast |
| Storage | Compact | Larger | Largest |
| Forward compatibility | Best | Risk of loss | Best + convenient |
Decision: Store Both — Raw for fidelity and forwarding; parsed for fast reads and search indexing. Raw stored compressed; parsed fields denormalized for query performance.
7.3 Search: Elasticsearch vs Self-Built
| Aspect | Elasticsearch | Self-Built (LSM-based) |
|---|---|---|
| Maturity | Battle-tested | Greenfield risk |
| Features | Rich query DSL, analyzers | Build from scratch |
| Operations | Well-documented, community | Full ownership burden |
| Customization | Limited deep customization | Full control |
| Resource usage | Heavy (JVM) | Potentially lighter |
Decision: Elasticsearch — Time-to-market and feature completeness outweigh customization needs. Can migrate to self-built if scale demands it.
7.4 Consistency Model
| Operation | Consistency Level | Rationale |
|---|---|---|
| Send email | Strong (synchronous) | Sender must know success/failure |
| Receive email | Eventual (async via Kafka) | Pipeline allows slight delay |
| Label changes | Strong (synchronous) | User expects immediate feedback |
| Search indexing | Eventual (near-real-time, 5s) | Search can be slightly stale |
| Unread counters | Eventual (async counter updates) | Approximate counts acceptable |
| Spam classification | Strong (before delivery) | Must classify before user sees |
7.5 Attachment Storage: Inline vs External
| Aspect | Inline (in message store) | External (object store) |
|---|---|---|
| Simplicity | Simpler (single store) | More complex (two stores) |
| Scalability | Limited by DB size | Virtually unlimited |
| CDN caching | Hard | Easy |
| Deduplication | Hard | Natural (content-addressable) |
| Cost | Higher (DB storage) | Lower (object storage) |
Decision: External object store — Attachments are large, rarely modified, and benefit from CDN caching and deduplication. Message store keeps only references.
8. Scalability Strategy
8.1 Horizontal Scaling Dimensions
flowchart LR
subgraph Scale["Scaling Levers"]
S1[SMTP Gateways<br/>Scale by connection count]
S2[API Servers<br/>Scale by CPU/memory]
S3[Service Instances<br/>Scale by business logic load]
S4[Storage Shards<br/>Scale by data volume]
S5[Search Shards<br/>Scale by query load]
S6[Cache Nodes<br/>Scale by hit rate needs]
S7[WebSocket Gateways<br/>Scale by connection count]
end
subgraph Metrics["Scaling Metrics"]
M1[Active connections]
M2[CPU / Memory utilization]
M3[Request latency P99]
M4[Storage utilization %]
M5[Query latency P99]
M6[Cache hit rate]
end
S1 <-.-> M1
S2 <-.-> M2
S3 <-.-> M3
S4 <-.-> M4
S5 <-.-> M5
S6 <-.-> M6
S7 <-.-> M1
8.2 Sharding Strategy
Primary shard key: userId
Shard assignment: shard_id = hash(userId) % num_shards
Benefits:
- All of a user's data on the same shard (messages, labels, threads)
- Simple routing — no cross-shard queries for user operations
- Predictable load distribution (assuming uniform userId distribution)
Anti-patterns avoided:
- NOT sharding by domain (uneven distribution — gmail.com >> others)
- NOT sharding by time (hot partition on current data)
Shard count planning:
| Scale | Users | Shards | Data/shard | Nodes/shard |
|---|---|---|---|---|
| Startup | 1M | 100 | ~11 GB | 1 |
| Growth | 100M | 1,000 | ~11 GB | 2 |
| Scale | 1B | 10,000 | ~11 GB | 3 |
| Mega | 5B | 50,000 | ~11 GB | 5 |
8.3 Caching Strategy
flowchart TB
subgraph CacheLayers["Cache Hierarchy"]
L1[L1: Client Cache<br/>Browser localStorage / Mobile cache]
L2[L2: CDN Cache<br/>Static assets, public attachments]
L3[L3: API Response Cache<br/>Redis — hot mailbox state]
L4[L4: Service-Level Cache<br/>In-process — counters, metadata]
end
L1 --> L2 --> L3 --> L4
| Cache Layer | Content | TTL | Eviction | Hit Rate Target |
|---|---|---|---|---|
| Client | Drafts, recent messages | Session / 1 hour | LRU | 40% |
| CDN | Attachments, static assets | 24 hours | LRU | 80% |
| Redis | Inbox preview, counters, labels | 5 minutes | LRU + TTL | 60% |
| In-process | User settings, quota | 1 minute | LRU | 90% |
8.4 Database Scaling
Cassandra/ScyllaDB scaling:
Read scaling:
- Add nodes → more replicas → more read capacity
- Token-aware routing → local reads preferred
- Read repair → consistency without write amplification
Write scaling:
- Add nodes → more partitions → more parallel writes
- Batch writes → reduce round trips
- Async indexing → decouple search index from write path
Compaction strategy:
- SizeTieredCompaction for write-heavy workloads
- LeveledCompaction for read-heavy workloads
- Hybrid: SizeTiered for message data, Leveled for index data
8.5 Cross-Region Replication
Primary region: User's home region (based on signup / preference)
Replica regions: 2 additional regions for DR
Replication strategy:
- Synchronous: Within primary region (3 replicas)
- Asynchronous: Cross-region (eventual, < 1 minute lag)
- Failover: Automatic promotion of replica on primary failure
- Data residency: EU user data stays in EU regions (GDPR)
Conflict resolution:
- Last-write-wins with vector clocks
- User-facing operations: strong consistency within region
- Cross-region reads: eventually consistent
9. Curveball Scenarios
Curveball 1: “What if we need to support 1GB attachments?”
Problem: Standard email attachments are limited to ~25-50 MB. Supporting 1GB attachments breaks the SMTP pipeline, storage model, and user experience.
Analysis:
| Concern | Impact | Mitigation |
|---|---|---|
| SMTP size limits | RFC 5321 allows SIZE extension, but 1GB over SMTP is impractical | Bypass SMTP for large files |
| Memory buffering | 1GB in memory per connection = OOM risk | Streaming, never buffer fully |
| Storage cost | 1GB × 104K emails/sec = 104 TB/sec if all large | Quota limits, tiered pricing |
| Transfer time | 1GB at 10 Mbps = 13 minutes | Resumable uploads, background transfer |
| Virus scanning | Scanning 1GB files at scale | Streaming scan, sample-based for known-safe types |
Solution: Hybrid Attachment Model
Upload flow for large attachments (>25 MB):
1. Client detects large file
2. Instead of embedding in email, upload to "Smart Attachment" service:
a. Initiate multipart upload (100 MB parts)
b. Each part virus-scanned on arrival
c. Content-addressed storage (SHA-256)
d. Deduplication check per part
3. Service returns a shareable link + metadata card
4. Email contains a lightweight "attachment card":
{
"type": "smart_attachment",
"name": "presentation.pptx",
"size": 1073741824,
"mimeType": "application/vnd.ms-powerpoint",
"downloadUrl": "https://attachments.example.com/d/abc123",
"previewUrl": "https://attachments.example.com/p/abc123",
"expiresAt": "2027-08-28T00:00:00Z"
}
}
5. Recipient sees a card in the email body
6. Clicking downloads from CDN (not SMTP)
Benefits:
- Email stays small (fast delivery)
- Large files use optimized transfer (multipart, resumable)
- CDN serves downloads (not mail servers)
- Deduplication saves storage (same 1GB file uploaded by 1000 users = 1 copy)
- Expiration policies manage storage growth
Infrastructure changes:
- New "Smart Attachment" microservice
- Multipart upload support (S3-compatible)
- Streaming virus scanner
- CDN integration for downloads
- Link expiration + cleanup cron job
Curveball 2: “What if we need real-time collaboration on emails?”
Problem: Multiple users simultaneously editing a draft email (like Google Docs). Requires operational transformation or CRDTs for conflict resolution.
Analysis:
| Concern | Impact | Approach |
|---|---|---|
| Conflict resolution | Concurrent edits must merge correctly | OT or CRDT |
| Latency | Collaborators expect < 100ms sync | WebSocket, edge computing |
| Authorization | Who can edit? Granular permissions | RBAC on draft level |
| Presence | Show who’s editing, cursor positions | Real-time presence channel |
| History | Undo/redo across collaborators | Operation log |
Solution: CRDT-Based Collaborative Editing
Architecture:
Client A Collaborative Engine Client B
(Editor) <--> (WebSocket + CRDT) <--> (Editor)
|
Operation Log
(Append-only)
CRDT Choice: Yjs (Yjs-XML for rich text)
- Conflict-free by design
- No central coordinator needed for merge
- Supports rich text (bold, italic, links, images)
- Awareness protocol for presence
Implementation:
1. Draft becomes "collaborative" when shared:
POST /api/v1/users/{userId}/compose/{draftId}/share
Body: { collaborators: [userId1, userId2], permissions: { userId1: "edit", userId2: "view" } }
2. Each collaborator connects via WebSocket:
WS /api/v1/collaborate/{draftId}?token={authToken}
3. Engine maintains:
- Document state (Yjs document)
- Awareness (cursor positions, user presence)
- Operation log (for replay, audit)
4. Edit flow:
a. User types → local CRDT update
b. Delta sent to engine via WebSocket
c. Engine broadcasts delta to all collaborators
d. Each client applies delta to local CRDT
e. Conflict-free merge guaranteed by CRDT properties
5. Auto-save:
- Engine persists document state every 5 seconds
- On disconnect, last state saved
- Reconnection replays missed operations from log
Performance considerations:
- Typical draft: < 100 KB → CRDT overhead negligible
- Max collaborators per draft: 10 (practical limit)
- Operation rate: ~10 ops/sec per user (typing speed)
- Engine capacity: ~1,000 concurrent collaborative sessions per node
Limitations:
- Only applies to drafts, not sent messages
- Attachment collaboration not supported (link sharing instead)
- Recipient list collaboration: separate conflict resolution (last-write-wins)
Curveball 3: “What about spam at scale? A coordinated botnet sends 10M spam emails/second targeting our users.”
Problem: A sophisticated botnet launches a coordinated spam campaign at 10M emails/second — 100x normal traffic. This could overwhelm the ingestion pipeline, waste storage, and degrade service for legitimate users.
Analysis:
| Attack Vector | Characteristics | Detection Difficulty |
|---|---|---|
| Volume flood | Massive quantity from many IPs | Easy (rate anomaly) |
| Spoofed senders | Fake From addresses | Medium (SPF/DKIM) |
| Content variation | Slightly different content per email | Hard (ML needed) |
| Legitimate-looking | Well-formed, proper headers | Hardest (behavioral) |
| Slow drip | Below rate limits, over time | Hard (pattern analysis) |
Multi-Layer Defense Strategy:
flowchart TB
subgraph L0["Layer 0: Network-Level (Before SMTP)"]
GeoBlock[Geo-IP Blocking<br/>Block known bad regions]
IPBlock[IP Blocklist<br/>Real-time threat intel]
ConnLimit[Connection Rate Limit<br/>Per-IP max connections]
end
subgraph L1["Layer 1: SMTP-Level (During Handshake)"]
EHLOCheck[EHLO Validation<br/>Bot detection patterns]
TLSReq[Mandatory TLS<br/>Reject non-TLS]
AuthReq[Mandatory Auth<br/>For relay]
Greylist[Greylisting<br/>First-time senders delayed]
end
subgraph L2["Layer 2: Envelope-Level (Before Content)"]
SenderRep[Sender Reputation<br/>Real-time score]
RecipientCheck[Recipient Validation<br/>Reject invalid recipients]
BurstDetect[Burst Detection<br/>Per-sender rate limit]
end
subgraph L3["Layer 3: Content-Level (After Receipt)"]
FastSpam[Fast Spam Filter<br/>Rules + lightweight ML]
ContentBlock[Content Blocking<br/>Known spam patterns]
URLBlock[URL Blocklist<br/>Known malicious URLs]
end
subgraph L4["Layer 4: Behavioral (Post-Delivery)"]
UserFeedback[User Feedback Loop<br/>Mark as spam training]
PatternDetect[Pattern Detection<br/>Cross-user correlation]
AutoRemediate[Auto-Remediation<br/>Bulk delete, block sender]
end
L0 --> L1 --> L2 --> L3 --> L4
Detailed Response Plan:
Phase 1: Immediate Response (0-5 minutes)
1. Anomaly detection triggers alert:
- Ingestion rate > 5x baseline → Auto-scale + alert
- Spam score distribution shifts → Tighten thresholds
2. Automated countermeasures:
a. Dynamic rate limiting:
- Per-IP: 100 emails/hour (from 1000)
- Per-domain: 10K emails/hour (from 100K)
- New domains: 100 emails/hour (from 1000)
b. Reputation threshold tightening:
- Minimum sender reputation: 0.7 (from 0.3)
- New domain minimum: 0.9 (from 0.5)
c. Greylisting activation:
- All first-time senders: 5-minute delay
- Bots typically don't retry → filtered out
Phase 2: Active Mitigation (5-30 minutes)
3. ML model emergency update:
a. Extract features from detected spam batch
b. Train incremental model update (online learning)
c. Deploy updated model to all spam filter nodes
4. Blocklist propagation:
a. Extract sender IPs, domains, patterns
b. Push to all SMTP gateways (hot reload)
c. Push to DNS-based blocklists (DNSBL)
5. Storage protection:
a. Spam messages NOT stored in user mailbox
b. Spam stored in ephemeral quarantine (24h TTL)
c. If spam volume > threshold, discard without storage
Phase 3: Recovery & Hardening (30 min - 24 hours)
6. User communication:
a. Notify affected users of spam wave
b. Auto-clean spam folders
c. Provide "report spam" one-click action
7. Post-incident analysis:
a. Root cause analysis
b. Update detection rules
c. Retrain ML models on new data
d. Update rate limit policies
Storage impact analysis:
Normal: 104K emails/sec × 50 KB = 5.2 GB/s
Attack: 10M emails/sec × 50 KB = 500 GB/s
Without mitigation: 500 GB/s × 3600s = 1.8 TB/hour of spam
With Layer 0-2 filtering: ~90% rejected before content receipt
With Layer 3 filtering: ~95% of remaining caught
Storage impact: 500 GB/s × 10% × 5% = 250 MB/s = 900 GB/hour
Quarantine TTL 24h: Max 21.6 TB (acceptable with auto-cleanup)
Curveball 4: “What if a user’s mailbox has 10 million messages? How do we keep it performant?”
Problem: Power users, enterprise accounts, or archived mailboxes can accumulate millions of messages. Naive implementations would degrade linearly.
Solution: Tiered Access Pattern
Access tiers:
Tier 1: Hot (last 30 days, ~1,500 messages)
- Stored in memory-optimized storage (Redis + SSD)
- Sub-10ms access
- Full indexing
Tier 2: Warm (30 days - 1 year, ~10,000 messages)
- Stored in SSD-backed Cassandra
- < 50ms access
- Full indexing
Tier 3: Cold (1+ years, rest)
- Stored in HDD-backed storage
- < 200ms access
- Search index only (no full body cache)
Tier 4: Archive (>5 years)
- Stored in object storage (cold tier)
- < 2s access
- Metadata index only
UI implications:
- Default view: Tier 1 + Tier 2
- "Show older" button: Loads Tier 3
- Archive search: Explicit search with "include archive" toggle
Pagination optimization:
- Use search-after cursor (not offset-based)
- Pre-fetch next page in background
- Virtual scrolling for web client
Counter caching:
- Unread count: Cached counter (updated async)
- Total count: Approximate (Cassandra counter)
- Label counts: Materialized view, refreshed every 60s
Curveball 5: “How do we handle a zero-day phishing attack that bypasses all filters?”
Problem: A novel phishing technique evades all detection layers. Thousands of users receive and interact with malicious emails before detection.
Solution: Rapid Response + Retroactive Protection
Detection (beyond inbound filtering):
1. Client-side detection:
- Browser extension flags suspicious links on hover
- Mobile app sandbox for opened attachments
- Real-time URL reputation check on click
2. Behavioral analytics:
- Unusual click patterns across users
- Rapid credential submission after email open
- Cross-correlation with threat intelligence feeds
3. User reporting:
- One-click "Report phishing" button
- Aggregated reports trigger investigation
Response:
1. Retroactive scanning (within 1 hour of detection):
a. Identify all users who received the phishing email
b. Auto-move to spam folder
c. Send warning notification to affected users
d. Invalidate any tokens/credentials potentially compromised
2. Emergency blocklist:
a. Block sender domain, IP, URLs immediately
b. Push to all gateways (hot reload, < 30 seconds)
c. Block similar patterns (domain similarity, URL structure)
3. User protection:
a. Force password reset for users who clicked
b. Enable enhanced monitoring for 30 days
c. Push security awareness notification
Prevention improvements:
a. Update ML models with new phishing samples
b. Add new detection rules
c. Enhance URL sandboxing for new techniques
d. Update client-side detection heuristics
10. Failure Modes and Mitigations
10.1 Failure Mode Matrix
| Component | Failure Mode | Impact | Detection | Mitigation | RTO | RPO |
|---|---|---|---|---|---|---|
| SMTP Gateway | Node crash | Partial ingest loss | Health check timeout | Auto-replace + Kafka replay | 1 min | 0 (Kafka buffered) |
| SMTP Gateway | Region outage | Regional ingest loss | Cross-region health check | Failover to other region | 5 min | 0 |
| Message Queue | Broker failure | Pipeline stall | Consumer lag alert | Broker failover + replica promotion | 30 sec | 0 (replicated) |
| Message Queue | Topic corruption | Data loss | Checksum mismatch | Replay from SMTP gateways | 5 min | 5 min |
| Message Store | Node failure | Read/write degradation | Consistency check | Replica read + auto-repair | 1 min | 0 (3 replicas) |
| Message Store | Data corruption | Message loss | Checksum verification | Restore from backup + replay | 10 min | 0 |
| Search Index | Shard failure | Search degradation | Query error rate | Replica shard promotion | 2 min | 5 min (index lag) |
| Search Index | Index corruption | Wrong results | Result validation | Reindex from message store | 30 min | 0 |
| Spam Filter | Model degradation | Spam leak / false positive | Spam complaint rate | Model rollback + retrain | 5 min | N/A |
| Spam Filter | Service outage | No spam filtering | Latency spike | Bypass to rules-only mode | 1 min | N/A |
| Cache | Node failure | Cache miss surge | Hit rate drop | Peer failover + repopulation | 30 sec | N/A |
| Cache | Cache stampede | Backend overload | Latency spike | Lock-based single flight | 1 min | N/A |
| Object Store | Disk failure | Attachment loss | Checksum mismatch | Erasure coding recovery | 5 min | 0 |
| WebSocket GW | Connection drop | Notification loss | Heartbeat failure | Auto-reconnect + catch-up | 30 sec | N/A |
| Database | Split brain | Data inconsistency | Quorum check | Fence minority + heal | 2 min | 0 |
10.2 Disaster Recovery
DR Strategy: Active-Active Multi-Region
Region A (Primary for users in zone A)
Region B (Primary for users in zone B)
Region C (Standby + DR for both)
Cross-region replication:
- Async replication with < 1 minute lag
- Conflict resolution: Last-write-wins with vector clocks
- Data residency: EU data never leaves EU regions
DR Testing:
- Chaos engineering: Weekly random node kills
- Region failover drill: Quarterly full region failover
- Data recovery drill: Monthly restore from backup
Backup Strategy:
- Message store: Continuous WAL shipping + daily snapshots
- Search index: Daily full snapshot + hourly incremental
- Object store: Versioning + cross-region replication
- Metadata: Continuous replication + daily backup
10.3 Circuit Breaker Patterns
Circuit breaker configuration for critical paths:
SMTP → Kafka:
- Failure threshold: 5 consecutive failures
- Recovery timeout: 30 seconds
- Fallback: Buffer locally, retry on recovery
Service → Database:
- Failure threshold: 10% error rate over 60 seconds
- Recovery timeout: 15 seconds
- Fallback: Serve from cache (stale OK)
Service → Search:
- Failure threshold: 50% error rate over 30 seconds
- Recovery timeout: 10 seconds
- Fallback: Database-based search (slower, limited)
Service → Spam Filter:
- Failure threshold: 3 consecutive failures
- Recovery timeout: 5 seconds
- Fallback: Allow through with "unclassified" flag
10.4 Graceful Degradation
When under extreme load or partial failure:
Priority 1 (Always available):
- Send email (core function)
- Receive email (core function)
- Basic inbox view (last 100 messages)
Priority 2 (Degraded if needed):
- Full-text search (fallback to subject/sender search)
- Attachment preview (fallback to download-only)
- Push notifications (fallback to polling)
Priority 3 (Disabled if needed):
- Smart compose/reply
- Auto-categorization
- Contact suggestions
- Thread grouping (show flat list)
Communication:
- Status page updated in real-time
- In-app banners for degraded features
- Email notification for major outages
Appendix A: Technology Stack Summary
| Layer | Technology | Alternatives | Rationale |
|---|---|---|---|
| API Gateway | Kong / Envoy | AWS API Gateway, NGINX | Open source, extensible |
| SMTP Server | Custom (Go/Rust) | Postfix, Exim | Performance, customizability |
| IMAP Server | Custom (Go/Rust) | Dovecot | Performance, integration |
| Message Queue | Apache Kafka | RabbitMQ, Pulsar | Throughput, replayability |
| Message Store | ScyllaDB / Cassandra | DynamoDB, CockroachDB | Write throughput, scalability |
| Search | Elasticsearch | Solr, Meilisearch | Maturity, query DSL |
| Object Store | Ceph / MinIO | AWS S3, GCS | Self-hosted, S3-compatible |
| Cache | Redis Cluster | Memcached, Dragonfly | Rich data structures, pub/sub |
| User DB | PostgreSQL | MySQL, CockroachDB | ACID, JSON support |
| ML Platform | TensorFlow / PyTorch | XGBoost, ONNX | Flexibility, ecosystem |
| WebSocket | Custom (Go) | Socket.IO, WS | Lightweight, no overhead |
| Monitoring | Prometheus + Grafana | Datadog, New Relic | Open source, flexible |
| Logging | ELK Stack | Loki, Splunk | Full-text log search |
| Service Mesh | Istio / Linkerd | Consul Connect | mTLS, observability |
Appendix B: Key Metrics and SLOs
| Metric | SLO | Measurement | Alert Threshold |
|---|---|---|---|
| Email delivery latency (P99) | < 10 seconds | End-to-end timestamp | > 15 seconds for 5 min |
| Inbox load latency (P99) | < 500ms | API response time | > 1s for 5 min |
| Search latency (P99) | < 500ms | Query response time | > 1s for 5 min |
| Availability | 99.99% | Uptime / total time | < 99.9% for 1 hour |
| Spam catch rate | > 99.9% | Spam caught / total spam | < 99% for 1 hour |
| Spam false positive rate | < 0.1% | False positives / total ham | > 0.5% for 1 hour |
| Message durability | 99.9999999% | Messages lost / total | Any loss |
| Search freshness | < 5 seconds | Index lag | > 30 seconds |
| Push notification latency | < 1 second | Event to delivery | > 3 seconds for 5 min |
| Attachment upload success | 99.9% | Successful / total uploads | < 99% for 5 min |
Appendix C: Security Considerations
| Threat | Mitigation |
|---|---|
| Email spoofing | SPF, DKIM, DMARC enforcement |
| Phishing | ML detection, URL sandboxing, user education |
| Malware | Multi-engine virus scanning, attachment sandboxing |
| Data breach | Encryption at rest (AES-256), encryption in transit (TLS 1.3) |
| Account takeover | MFA, anomaly detection, secure password policy |
| DDoS | Rate limiting, CDN absorption, auto-scaling |
| Data leakage | DLP policies, audit logging, access controls |
| Insider threat | RBAC, least privilege, audit trails |
| Compliance | GDPR data subject requests, retention policies, audit logs |
Document Version: 1.0 | Last Updated: 2026-08-28