Mail & SearchQuestion #17ADVANCED

Gmail: Mail Ingestion, Spam Filtering & Full-Text Search

Global email infrastructure combining RFC-compliant SMTP/IMAP gateway clusters, ML spam detection pipelines, inverted-index full-text search, and multi-label indexing.

Category: Category 11: Communication & Search•Updated: 2026-08-30
#Email#SMTP/IMAP#Spam Filter#Full-text Search#Labels

System Design: Emailing Service (Gmail-like)

Author: System Design Document | Date: 2026-08-28 Scope: End-to-end design of a cloud-scale email service supporting billions of users, trillions of messages, with real-time delivery, search, spam protection, and collaboration features.


Table of Contents

  1. Requirements Clarification
  2. Back-of-Envelope Calculations
  3. System Interface Design
  4. High-Level Architecture
  5. Core Component Design
  6. Data Model
  7. Key Design Decisions
  8. Scalability Strategy
  9. Curveball Scenarios
  10. Failure Modes and Mitigations

1. Requirements Clarification

1.1 Functional Requirements

ID Requirement Priority Notes
FR-01 Send email — Compose, attach files, send to single/multiple recipients (To, CC, BCC) P0 Support inline images, HTML + plain text
FR-02 Receive email — Ingest via SMTP, deliver to recipient mailbox in real-time P0 Support bounce handling, delivery receipts
FR-03 Mailbox management — Inbox, Sent, Drafts, Trash, Spam, custom labels/folders P0 Labels (not folders) — an email can have multiple labels
FR-04 Full-text search — Search by sender, recipient, subject, body, date range, labels, attachments P0 Sub-second latency for personal mailbox
FR-05 Spam & phishing detection — Classify incoming mail as spam/ham/phishing P0 ML-based + rule-based hybrid
FR-06 Attachment handling — Upload, store, preview, download attachments P0 Support common formats (PDF, images, Office docs)
FR-07 Push notifications — Real-time notification of new mail to web/mobile clients P1 WebSocket / HTTP long-polling fallback
FR-08 Contact management — Address book, auto-complete, contact groups P1 Sync with global directory (enterprise)
FR-09 Compose & Drafts — Rich text editor, auto-save drafts, scheduled send P1 Offline support for mobile
FR-10 IMAP/SMTP client access — Standard protocol support for third-party clients P1 IMAP IDLE for push, SMTP AUTH for send
FR-11 Threading — Group related emails into conversation threads P1 By subject + participant heuristics
FR-12 Quota management — Per-user storage quota enforcement P1 Soft/hard limits, cleanup suggestions
FR-13 Encryption — TLS in transit, optional E2E encryption (S/MIME, PGP) P1 DMARC, DKIM, SPF for sender verification
FR-14 Calendar integration — Detect events in email, surface in calendar P2 Optional
FR-15 Smart features — Smart compose, smart reply, priority inbox, snooze P2 ML-powered

1.2 Non-Functional Requirements

ID Requirement Target Rationale
NFR-01 Availability 99.99% (52.6 min downtime/year) Email is a critical business service
NFR-02 Delivery latency P99 < 10 seconds for same-domain; < 30 seconds cross-domain User expectation for “instant” delivery
NFR-03 Search latency P99 < 500ms for personal mailbox queries Competitive with Gmail experience
NFR-04 Scalability 1B+ users, 100B+ messages, 1M+ QPS peak Global-scale service
NFR-05 Storage efficiency < 2 bytes/byte overhead (compression + dedup) Cost control at petabyte scale
NFR-06 Durability 99.9999999% (11 nines) — no message loss Email is a legal record
NFR-07 Security Zero-trust architecture, DLP, anti-phishing Compliance (GDPR, HIPAA for enterprise)
NFR-08 Consistency Eventual consistency for mailbox state; strong consistency for send Balance latency vs correctness
NFR-09 Internationalization UTF-8 everywhere, RTL languages, timezone-aware Global user base

2. Back-of-Envelope Calculations

2.1 Assumption Sets

We reason from first principles with three assumption sets to bound the design space:

Parameter Conservative (A) Base (B) Aggressive (C)
Active users 100M 1B 5B
Emails/user/day 50 30 20
Avg email size (with headers) 100 KB 50 KB 30 KB
Avg attachment size 500 KB 200 KB 100 KB
Attachment ratio 20% 15% 10%
Ingestion rate (emails/sec) 58K 104K 231K
Total daily emails 5B 3B 1B

Note: Set B (Base) is our primary design target. Sets A and C bound the extremes.

2.2 Storage Estimates (Base Case — 1B users)

Daily email volume:     1B users × 30 emails/user = 30B emails/day
Email body + headers:   30B × 50 KB = 1.5 PB/day
Attachments:            30B × 15% × 200 KB = 900 TB/day
Total daily ingest:     ~2.4 PB/day

Annual storage (raw):   2.4 PB/day × 365 = 876 PB/year
With 3x replication:    876 × 3 = 2.63 EB/year
With compression (2:1): ~1.3 EB/year effective

Storage breakdown per user (1 year):

Component Size/user Notes
Email bodies ~550 MB 30 emails/day × 50 KB × 365
Attachments ~11 GB 30 × 15% × 200 KB × 365
Index data ~200 MB Inverted index, metadata
Metadata ~50 MB Labels, threads, flags
Total/user ~11.8 GB Before replication

2.3 Bandwidth Estimates

Inbound SMTP:     104K emails/sec × 50 KB = 5.2 GB/s (41.6 Gbps)
Outbound SMTP:    104K emails/sec × 50 KB = 5.2 GB/s (41.6 Gbps)
Attachment CDN:   104K × 15% × 200 KB = 3.1 GB/s (24.8 Gbps)
API traffic:      ~200K req/s × 5 KB avg = 1 GB/s (8 Gbps)
Internal replication: ~10 GB/s (80 Gbps)

Total network:    ~20 GB/s (160 Gbps) per major region

2.4 QPS Estimates

Operation QPS (Base) Read/Write Notes
SMTP ingest 104K Write Peak may be 3-5x during business hours
IMAP fetch 200K Read Polling + IDLE clients
Web API reads 500K Read Inbox load, message view
Search queries 50K Read Full-text search
Compose/send 30K Write Draft save + send
Push notifications 100K Push WebSocket connections
Spam scanning 104K Read+Write Every inbound email
Total ~1.1M QPS ~70% reads Per region

2.5 Infrastructure Sizing (Per Region)

Assuming 4 regions (US, EU, APAC, LATAM), each handling ~25% of traffic:

SMTP Ingestion Servers:    104K QPS / 25% / 1K req/s per server = ~104 servers
Web API Servers:           500K QPS / 25% / 500 req/s per server = ~400 servers
Search Cluster:            50K QPS / 25% / 500 queries/s per node = ~40 nodes
Mailbox Storage Nodes:     250M users / 500K users per node = ~500 nodes
Object Storage (attachments): Shared regional cluster
Message Queue Brokers:     ~50 brokers (Kafka, 3 replicas)
Cache (Redis/Memcached):   ~200 nodes (for session, hot data)
Spam ML Inference:         ~50 GPU/TPU nodes

3. System Interface Design

3.1 REST API Definitions

3.1.1 Mailbox Operations

GET    /api/v1/users/{userId}/mailbox/labels
       List all labels for a user
       Response: { labels: [{ id, name, messageCount, unreadCount }] }

POST   /api/v1/users/{userId}/mailbox/labels
       Create a new label
       Body: { name, color?, parentLabelId? }

GET    /api/v1/users/{userId}/mailbox/messages?label={labelId}&page={n}&limit={limit}&sort={date|relevance}
       List messages with pagination
       Response: { messages: [{ id, snippet, from, to, subject, date, labels, starred, unread }], nextPageToken }

GET    /api/v1/users/{userId}/mailbox/messages/{messageId}
       Get full message content
       Response: { id, headers, bodyHtml, bodyPlain, attachments, labels, threadId }

PUT    /api/v1/users/{userId}/mailbox/messages/{messageId}/labels
       Modify labels on a message
       Body: { add: [labelId], remove: [labelId] }

DELETE /api/v1/users/{userId}/mailbox/messages/{messageId}
       Delete (move to trash) a message

3.1.2 Compose & Send

POST   /api/v1/users/{userId}/compose
       Create a new draft
       Body: { to: [], cc: [], bcc: [], subject, bodyHtml?, bodyPlain?, attachments: [] }
       Response: { draftId }

PUT    /api/v1/users/{userId}/compose/{draftId}
       Update a draft

POST   /api/v1/users/{userId}/compose/{draftId}/send
       Send a draft
       Body: { scheduledTime? }  // optional for scheduled send
       Response: { messageId, status }

POST   /api/v1/users/{userId}/send
       Send email directly (no draft)
       Body: { to: [], cc: [], bcc: [], subject, bodyHtml?, bodyPlain?, attachments: [] }
       Response: { messageId, status }
GET    /api/v1/users/{userId}/search?q={query}&label={labelId}&dateRange={start,end}&page={n}
       Full-text search across mailbox
       Query syntax: from:xxx to:xxx subject:xxx has:attachment after:2024-01-01 before:2024-12-31
       Response: { results: [{ id, snippet, from, subject, date, labels }], totalHits, nextPageToken }

3.1.4 Attachments

POST   /api/v1/users/{userId}/attachments/upload
       Upload attachment (multipart)
       Response: { attachmentId, url, size, mimeType }

GET    /api/v1/users/{userId}/attachments/{attachmentId}/download
       Download attachment
       Response: Binary stream

GET    /api/v1/users/{userId}/attachments/{attachmentId}/preview
       Generate preview (thumbnail/first page)
       Response: Image URL or embedded HTML

3.1.5 Contacts

GET    /api/v1/users/{userId}/contacts
       List contacts
       Response: { contacts: [{ id, name, emails: [], phoneNumbers: [], groups: [] }] }

POST   /api/v1/users/{userId}/contacts
       Create contact
       Body: { name, emails: [{address, type}], phoneNumbers: [{number, type}], groups: [] }

GET    /api/v1/users/{userId}/contacts/suggest?q={query}
       Auto-complete contact search
       Response: { suggestions: [{ id, name, email }] }

3.1.6 Notifications (WebSocket)

WS     /api/v1/users/{userId}/notifications
       Real-time push channel
       Events:
         - NEW_MESSAGE: { messageId, labelId, from, subject, snippet }
         - LABEL_CHANGED: { messageId, added: [], removed: [] }
         - DRAFT_SAVED: { draftId }
         - QUOTA_WARNING: { usedPercent, totalBytes }

3.2 SMTP Protocol Interface

Standard SMTP (RFC 5321) with extensions:

Supported ESMTP extensions:
  - STARTTLS (RFC 3207) — Mandatory TLS upgrade
  - AUTH (RFC 4954) — PLAIN, LOGIN, XOAUTH2
  - 8BITMIME (RFC 6152) — 8-bit data support
  - SIZE (RFC 1870) — Maximum message size declaration
  - DSN (RFC 3461) — Delivery Status Notifications
  - SMTPUTF8 (RFC 6531) — Internationalized email addresses

Connection flow:
  Client -> EHLO/HELO -> STARTTLS -> AUTH XOAUTH2 -> MAIL FROM -> RCPT TO -> DATA -> . -> 250 OK

3.3 IMAP Protocol Interface

Standard IMAP4rev1 (RFC 3501) with extensions:

Supported IMAP extensions:
  - IDLE (RFC 2177) — Server push for new mail
  - CONDSTORE (RFC 4466) — Mod-sequence for efficient sync
  - UNSELECT (RFC 3691) — Deselect without expunge
  - MOVE (RFC 6851) — Atomic message move
  - LIST-EXTENDED (RFC 5890) — Subscribe status
  - XLIST (RFC 5890) — Mailbox listing with attributes

Connection flow:
  Client -> LOGIN -> SELECT INBOX -> FETCH/SEARCH/STORE -> IDLE (wait for updates)

4. High-Level Architecture

flowchart TB
    subgraph Clients["Clients"]
        Web[Web Client<br/>React/PWA]
        Mobile[Mobile App<br/>iOS/Android]
        ThirdParty[Third-Party Clients<br/>IMAP/SMTP]
    end

    subgraph Edge["Edge Layer"]
        CDN[CDN / WAF<br/>Static Assets + Attachments]
        LB[Global Load Balancer<br/>GSLB + L7 LB]
        Auth[Auth Gateway<br/>OAuth2 / SSO]
    end

    subgraph API["API Layer"]
        WebAPI[Web API Gateway<br/>REST + GraphQL]
        SMTPGW[SMTP Gateway<br/>RFC 5321]
        IMAPGW[IMAP Gateway<br/>RFC 3501]
        WS[WebSocket Gateway<br/>Push Notifications]
    end

    subgraph Services["Service Layer"]
        ComposeSvc[Compose Service]
        SendSvc[Send Service]
        MailboxSvc[Mailbox Service]
        SearchSvc[Search Service]
        SpamSvc[Spam & Security Service]
        ContactSvc[Contact Service]
        AttachSvc[Attachment Service]
        NotifySvc[Notification Service]
        ThreadSvc[Threading Service]
        QuotaSvc[Quota Service]
    end

    subgraph Pipeline["Ingestion Pipeline"]
        MQ[(Message Queue<br/>Kafka)]
        SpamFilter[Spam Filter<br/>ML + Rules]
        VirusScan[Virus Scanner<br/>ClamAV / Custom]
        DKIMVerify[DKIM/SPF/DMARC<br/>Verifier]
        Indexer[Indexer<br/>Search + Metadata]
    end

    subgraph Storage["Storage Layer"]
        MsgStore[(Message Store<br/>Distributed KV / Cassandra)]
        AttachStore[(Object Store<br/>S3-compatible)]
        SearchIdx[(Search Index<br/>Elasticsearch / Solr)]
        Cache[(Cache Layer<br/>Redis Cluster)]
        ContactDB[(Contact DB<br/>PostgreSQL)]
        UserDB[(User DB<br/>PostgreSQL)]
        MetadataDB[(Metadata DB<br/>Cassandra)]
    end

    subgraph ML["ML Infrastructure"]
        SpamML[Spam Classifier<br/>Deep Learning]
        PhishML[Phishing Detector<br/>URL + Content Analysis]
        SmartML[Smart Features<br/>Compose / Reply / Priority]
        FeatureStore[Feature Store]
    end

    subgraph Infra["Infrastructure"]
        Monitor[Monitoring<br/>Prometheus + Grafana]
        Log[Logging<br/>ELK Stack]
        Config[Config Management<br/>Consul / etcd]
        CI[CI/CD Pipeline]
    end

    %% Client flows
    Web --> LB
    Mobile --> LB
    ThirdParty --> LB

    LB --> Auth
    Auth --> WebAPI
    Auth --> SMTPGW
    Auth --> IMAPGW
    Auth --> WS

    %% API to Services
    WebAPI --> ComposeSvc
    WebAPI --> MailboxSvc
    WebAPI --> SearchSvc
    WebAPI --> ContactSvc
    WebAPI --> AttachSvc

    SMTPGW --> MQ
    IMAPGW --> MailboxSvc

    %% Service interactions
    ComposeSvc --> SendSvc
    SendSvc --> MQ
    SendSvc --> DKIMVerify

    %% Ingestion pipeline
    MQ --> SpamFilter
    SpamFilter --> VirusScan
    VirusScan --> DKIMVerify
    DKIMVerify --> Indexer
    Indexer --> MsgStore
    Indexer --> SearchIdx
    Indexer --> MetadataDB

    %% Spam/ML
    SpamFilter --> SpamML
    SpamFilter --> PhishML
    SpamML --> FeatureStore
    PhishML --> FeatureStore

    %% Services to Storage
    MailboxSvc --> MsgStore
    MailboxSvc --> MetadataDB
    MailboxSvc --> Cache
    SearchSvc --> SearchIdx
    AttachSvc --> AttachStore
    ContactSvc --> ContactDB
    NotifySvc --> WS
    QuotaSvc --> UserDB

    %% Smart features
    ComposeSvc --> SmartML
    SmartML --> FeatureStore

    %% Monitoring
    Monitor -.-> Services
    Log -.-> Services
    Config -.-> Services

4.1 Architecture Principles

  1. Layered decomposition — Clear separation between edge, API, service, pipeline, and storage layers
  2. Async-first ingestion — All email ingestion flows through Kafka for decoupling and replayability
  3. Shared-nothing services — Each service is independently deployable and scalable
  4. Write-once, read-many — Messages are immutable; updates create new versions
  5. Region-local with cross-region replication — Data residency compliance + disaster recovery

5. Core Component Design

5.1 SMTP Ingestion Pipeline

The SMTP pipeline is the entry point for all inbound email. It must handle massive concurrent connections while performing security checks.

flowchart LR
    subgraph SMTP["SMTP Gateway Cluster"]
        Conn[Connection Handler<br/>Accept TCP 25/587]
        Proto[Protocol Parser<br/>SMTP State Machine]
        Auth[Auth Module<br/>XOAUTH2 / PLAIN]
    end

    subgraph Validation["Validation Chain"]
        SPF[SPF Check<br/>Sender IP verification]
        DKIM[DKIM Verify<br/>Cryptographic signature]
        DMARC[DMARC Policy<br/>Alignment check]
        Rate[Rate Limiter<br/>Per-sender throttling]
    end

    subgraph Queue["Buffering"]
        Kafka[Kafka Topics<br/>raw_email, spam_suspect, quarantine]
    end

    Conn --> Proto --> Auth --> SPF --> DKIM --> DMARC --> Rate --> Kafka

Design Details

Connection Handling:

  • Each SMTP gateway node handles ~10K concurrent connections using async I/O (epoll/io_uring)
  • TLS termination at the gateway (TLS 1.3 preferred)
  • Connection pooling for upstream services

Protocol Parsing:

  • State machine implementing RFC 5321
  • Streaming parser — no full buffering until DATA command
  • Header/body separation at parse time

Validation Chain (pipeline pattern):

Stage Purpose Failure Action Latency Budget
SPF Verify sender IP authorized by domain DNS Soft fail (log + flag) < 50ms
DKIM Verify cryptographic signature integrity Soft fail (log + flag) < 100ms
DMARC Enforce domain policy (none/quarantine/reject) Per-policy action < 20ms
Rate Limit Prevent abuse per sender IP/domain 451 Temporary failure < 5ms

Kafka Topic Design:

Topic: raw_email
  Partition key: recipient_domain (for domain-level ordering)
  Retention: 7 days
  Replication: 3
  Schema: { messageId, envelopeFrom, envelopeTo[], rawHeaders, bodyRef, timestamp, sourceIp }

Topic: spam_suspect
  Partition key: messageId
  For messages flagged for deeper ML analysis

Topic: delivery_queue
  Partition key: recipientUserId
  For final delivery to mailbox

5.2 Mailbox Storage and Indexing

The mailbox is the core data structure. Each user’s mailbox is a collection of messages with labels, flags, and metadata.

flowchart TB
    subgraph WritePath["Write Path (Ingest)"]
        Ingest[Ingestion Service]
        Dedup[Deduplication Check<br/>Message-ID + Content Hash]
        Store[Message Store<br/>Append-only KV]
        MetaWrite[Metadata Writer<br/>Labels, Flags, Counters]
        IndexWrite[Index Writer<br/>Search + Lookup]
    end

    subgraph ReadPath["Read Path (Fetch)"]
        Cache[Cache Layer<br/>Hot messages]
        StoreR[Message Store<br/>Range scan]
        MetaRead[Metadata Reader<br/>Labels, Counters]
        Merge[Merge & Project<br/>Final response]
    end

    Ingest --> Dedup --> Store --> MetaWrite --> IndexWrite
    Cache --> StoreR --> MetaRead --> Merge

Storage Architecture

Message Store (Distributed KV — Cassandra/ScyllaDB):

Keyspace: messages
  Primary key: (userId, messageId)
  Columns: {
    userId: UUID,
    messageId: UUID,          // Generated or from Message-ID header
    rawMessage: blob,         // Compressed raw RFC 2822 message
    headers: map<text, text>, // Parsed headers
    bodyPlain: text,          // Plain text body
    bodyHtml: text,           // HTML body
    size: bigint,             // Total size in bytes
    receivedTime: timestamp,
    labels: set<text>,        // Label IDs
    flags: set<text>,         // Seen, Answered, Flagged, Deleted, Draft
    threadId: UUID,           // Conversation thread
    attachmentRefs: list<text>,// References to object store
    checksum: text,           // SHA-256 for integrity
    version: bigint           // For optimistic concurrency
  }

Label Index (for fast label-based queries):

Keyspace: label_index
  Primary key: (userId, labelId, receivedTime DESC)
  Columns: {
    userId: UUID,
    labelId: text,
    receivedTime: timestamp,
    messageId: UUID,
    snippet: text,            // First 200 chars of body
    from: text,
    subject: text
  }

Counters (for unread/message counts):

Keyspace: mailbox_counters
  Primary key: (userId, labelId)
  Columns: {
    userId: UUID,
    labelId: text,
    totalMessages: counter,
    unreadMessages: counter,
    totalSize: counter
  }

Indexing Strategy

Index Type Purpose Technology Update Frequency
Primary index userId + messageId lookup Cassandra PK On write
Label index List messages by label Cassandra secondary table On write
Thread index Group by conversation Cassandra (threadId) On write
Full-text index Search body/subject Elasticsearch Async (near-real-time)
Date index Date-range queries Cassandra (receivedTime) On write
Sender index “from:xxx” queries Elasticsearch Async

5.3 Spam and Phishing Detection

A multi-layered defense system combining rule-based filters, statistical models, and deep learning.

flowchart TB
    subgraph Layer1["Layer 1: Reputation & Rules (Fast Path)"]
        IPRep[IP Reputation<br/>Blocklist + Score]
        DomainRep[Domain Reputation<br/>Age, History, Blacklists]
        HeaderRules[Header Rules<br/>Suspicious patterns]
        RateCheck[Rate Analysis<br/>Volume anomalies]
    end

    subgraph Layer2["Layer 2: Content Analysis (Medium Path)"]
        Bayesian[Bayesian Filter<br/>Word frequency]
        URLScan[URL Analysis<br/>Shortener, Redirect, Malicious]
        AttachScan[Attachment Scan<br/>HEIC, Macro, Executable]
        SPFResult[SPF/DKIM/DMARC<br/>Results scoring]
    end

    subgraph Layer3["Layer 3: ML Deep Analysis (Slow Path)"]
        DLModel[Deep Learning Model<br/>BERT-based classifier]
        GraphML[Graph Analysis<br/>Sender-recipient patterns]
        PhishML[Phishing Detector<br/>Visual + URL similarity]
    end

    subgraph Actions["Actions"]
        HAM[Deliver to Inbox]
        SPAM[Move to Spam Folder]
        PHISH[Quarantine + Alert]
        QUARANTINE[Hold for Review]
    end

    IPRep --> Bayesian --> DLModel --> HAM
    IPRep --> Bayesian --> DLModel --> SPAM
    DomainRep --> Bayesian
    HeaderRules --> Bayesian
    RateCheck --> Bayesian
    URLScan --> DLModel
    AttachScan --> DLModel
    SPFResult --> Bayesian
    DLModel --> PHISH
    PhishML --> PHISH
    GraphML --> QUARANTINE

Spam Scoring Pipeline

Each email receives a composite spam score (0.0 to 1.0):

spam_score = w1 * reputation_score + w2 * content_score + w3 * ml_score + w4 * graph_score

Where:
  reputation_score: Based on sender IP/domain reputation (0-1)
  content_score: Bayesian + rule-based content analysis (0-1)
  ml_score: Deep learning model prediction (0-1)
  graph_score: Anomaly in sender-recipient graph (0-1)

Weights (tunable):
  w1 = 0.25, w2 = 0.30, w3 = 0.35, w4 = 0.10

Decision thresholds:
  spam_score < 0.3:  HAM — Deliver to inbox
  0.3 <= score < 0.7: SPAM — Move to spam folder
  0.7 <= score < 0.9: PHISH — Quarantine, notify user
  score >= 0.9:       BLOCK — Reject outright

Phishing Detection Specifics

Technique Description Detection Rate
URL similarity Compare URLs to known brand domains (Levenshtein, visual similarity) ~95%
Logo/image analysis CNN-based comparison of logos against brand database ~90%
Domain age check Newly registered domains flagged ~80%
Impersonation detection Display name vs actual sender mismatch ~85%
Behavioral analysis Unusual sender patterns for known contacts ~75%
URL sandboxing Execute URLs in sandbox, analyze behavior ~98% (for queued URLs)

5.4 Full-Text Search at Scale

Search is one of Gmail’s killer features. It requires indexing billions of messages with sub-second response times.

flowchart TB
    subgraph Indexing["Indexing Pipeline"]
        Stream[Kafka Stream<br/>New/Updated Messages]
        Parse[Message Parser<br/>Extract searchable fields]
        NLP[NLP Pipeline<br/>Tokenization, Stemming, NER]
        Chunk[Chunking<br/>Split large bodies]
        Bulk[Bulk Indexer<br/>Batch to ES]
    end

    subgraph SearchCluster["Search Cluster (Elasticsearch)"]
        Master[Master Nodes<br/>Cluster coordination]
        Data[Data Nodes<br/>Shards + Replicas]
        Coords[Coordinating Nodes<br/>Query routing]
    end

    subgraph Query["Query Processing"]
        Parser[Query Parser<br/>Gmail query syntax]
        Rewrite[Query Rewriter<br/>Synonyms, corrections]
        Exec[Query Executor<br/>Distributed search]
        Rank[Ranker<br/>BM25 + ML ranking]
        Cache[Result Cache<br/>Hot queries]
    end

    Stream --> Parse --> NLP --> Chunk --> Bulk --> Data
    Parser --> Rewrite --> Exec --> Rank
    Coords --> Data
    Cache --> Exec

Index Schema (Elasticsearch)

{
  "mappings": {
    "properties": {
      "messageId": { "type": "keyword" },
      "userId": { "type": "keyword" },
      "from": {
        "type": "object",
        "properties": {
          "address": { "type": "keyword" },
          "name": { "type": "text", "analyzer": "email_analyzer" }
        }
      },
      "to": {
        "type": "nested",
        "properties": {
          "address": { "type": "keyword" },
          "name": { "type": "text", "analyzer": "email_analyzer" }
        }
      },
      "subject": { "type": "text", "analyzer": "subject_analyzer", "fields": { "keyword": { "type": "keyword" } } },
      "body": { "type": "text", "analyzer": "body_analyzer" },
      "date": { "type": "date" },
      "labels": { "type": "keyword" },
      "threadId": { "type": "keyword" },
      "hasAttachment": { "type": "boolean" },
      "attachmentNames": { "type": "text", "analyzer": "attachment_analyzer" },
      "size": { "type": "long" }
    }
  },
  "settings": {
    "number_of_shards": 100,
    "number_of_replicas": 2,
    "refresh_interval": "5s"
  }
}

Custom Analyzers

Analyzer Purpose Components
email_analyzer Parse email addresses and names Email tokenizer, lowercase, stop words
subject_analyzer Index email subjects Edge n-gram (for prefix search), synonym filter
body_analyzer Index email body text ICU tokenizer, stemmer, stop words, synonym filter
attachment_analyzer Index attachment filenames Keyword tokenizer, lowercase

Query Syntax Support

from:john@example.com
to:jane@example.com
subject:meeting
has:attachment
larger:5M
before:2024/06/01
after:2024/01/01
label:work -label:spam
is:unread
is:starred
filename:pdf
in:inbox

Search Performance Optimization

  • User-scoped routing — Each user’s messages routed to specific shards (hash(userId) % num_shards)
  • Result caching — LRU cache for frequent queries (same user + same query)
  • Snippet pre-computation — Store searchable snippet at index time
  • Pagination optimization — Search-after (scroll cursor) instead of deep offset
  • Async indexing — Index updates batched and applied every 5 seconds (near-real-time)

5.5 Label / Category Management

Gmail uses labels instead of folders — a message can have multiple labels simultaneously.

flowchart LR
    subgraph Labels["Label System"]
        System[System Labels<br/>INBOX, SENT, DRAFTS,<br/>TRASH, SPAM, IMPORTANT,<br/>CATEGORY_PERSONAL,<br/>CATEGORY_PROMOTIONS,<br/>CATEGORY_SOCIAL,<br/>CATEGORY_UPDATES,<br/>CATEGORY_FORUMS]
        User[User Labels<br/>Custom created]
        Auto[Auto-Labels<br/>ML-categorized]
    end

    subgraph Storage["Label Storage"]
        LabelMeta[(Label Metadata<br/>Name, Color, Hierarchy)]
        LabelIdx[(Label-Message Index<br/>userId + labelId -> messageIds)]
        Counters[(Counters<br/>Per-label message/unread counts)]
    end

    System --> LabelMeta
    User --> LabelMeta
    Auto --> LabelMeta
    LabelMeta --> LabelIdx
    LabelIdx --> Counters

Label Operations

Operation Implementation Latency
Add label to message Write to label_index + increment counter < 10ms
Remove label from message Delete from label_index + decrement counter < 10ms
List messages by label Scan label_index by (userId, labelId, time DESC) < 50ms
Get unread count Read counter from mailbox_counters < 5ms
Create label Write to label_metadata < 10ms
Delete label Cascade delete from label_index + metadata < 100ms

Auto-Categorization (ML)

Input features:
  - Sender domain reputation
  - Message content (NLP embeddings)
  - Historical user interaction (opened, deleted, marked spam)
  - Time of day patterns
  - Recipient list characteristics

Model: Multi-class classifier (Personal, Promotions, Social, Updates, Forums)
  Architecture: LightGBM / BERT-small
  Training: Online learning with user feedback loop
  Update frequency: Daily batch + hourly incremental

5.6 Compose Service

flowchart TB
    subgraph Editor["Rich Text Editor"]
        RTE[Rich Text Editor<br/>Quill / Custom]
        InlineImg[Inline Image Handler]
        Template[Template Engine<br/>Signatures, Canned responses]
    end

    subgraph Draft["Draft Management"]
        AutoSave[Auto-Save<br/>Debounced, every 30s]
        Versioning[Version Control<br/>Keep last 10 versions]
        Recovery[Recovery<br/>Restore from version]
    end

    subgraph Send["Send Preparation"]
        Validate[Validation<br/>Recipient check, size check]
        Encrypt[Encryption<br/>S/MIME if configured]
        Queue[Queue for Send<br/>Async via Kafka]
    end

    RTE --> AutoSave --> Versioning
    InlineImg --> RTE
    Template --> RTE
    AutoSave --> Draft
    Draft --> Validate --> Encrypt --> Queue

Auto-Save Strategy

Client-side:
  - Debounce: Save after 30s of inactivity
  - Throttle: Save at most every 5s during active typing
  - Local storage: Keep draft in localStorage as fallback
  - Conflict resolution: Last-write-wins with timestamp

Server-side:
  - Store in drafts table (same schema as messages, with isDraft=true)
  - TTL: 90 days for inactive drafts
  - Cleanup: Async job removes drafts older than TTL

5.7 Attachment Handling

flowchart TB
    subgraph Upload["Upload Path"]
        Client[Client]
        Multipart[Multipart Upload<br/>For large files]
        VirusScan[Anti-Virus Scan<br/>On-write]
        Dedup[Content Dedup<br/>SHA-256 check]
        ObjStore[(Object Store<br/>S3-compatible)]
    end

    subgraph Download["Download Path"]
        CDN[CDN Edge<br/>Cached attachments]
        Cache[Attachment Cache<br/>Hot attachments]
        ObjStoreR[(Object Store)]
    end

    subgraph Preview["Preview Generation"]
        PDF[PDF Renderer<br/>First page thumbnail]
        Img[Image Resizer<br/>Thumbnail generation]
        Doc[Doc Converter<br/>Office -> HTML preview]
        Video[Video Thumbnail<br/>First frame capture]
    end

    Client --> Multipart --> VirusScan --> Dedup --> ObjStore
    CDN --> Cache --> ObjStoreR
    ObjStore --> PDF
    ObjStore --> Img
    ObjStore --> Doc
    ObjStore --> Video

Attachment Storage Design

Aspect Design Rationale
Storage S3-compatible object store (Ceph / MinIO) Scalable, cheap, durable
Deduplication Content-addressable (SHA-256 as key) Save storage for common attachments
Lifecycle Hot (SSD) → Warm (HDD) → Cold (Glacier) Cost optimization
Max size 25 MB per attachment (standard), 50 MB (enterprise) Balance UX vs infrastructure
Virus scanning On-write scan + periodic re-scan Security
Preview Async generation on upload Non-blocking upload

Content Deduplication

Upload flow:
  1. Client computes SHA-256 of attachment
  2. Check if hash exists in dedup_index (Redis)
  3. If exists: create reference only (no storage write)
  4. If not: upload to object store, record hash in dedup_index

Storage savings estimate:
  - Common attachments (logos, signatures, stock images): ~30% dedup rate
  - Enterprise environments: ~40% dedup rate (shared documents)

5.8 Push Notifications

flowchart TB
    subgraph Events["Event Sources"]
        NewMail[New Mail Delivered]
        LabelChange[Label Modified]
        DraftSaved[Draft Auto-Saved]
        QuotaWarn[Quota Warning]
    end

    subgraph Gateway["WebSocket Gateway"]
        ConnMgr[Connection Manager<br/>Active connections map]
        Router[Event Router<br/>userId -> connections]
        Heartbeat[Heartbeat<br/>Keep-alive every 30s]
        Reconnect[Reconnect Handler<br/>Exponential backoff]
    end

    subgraph Clients["Client Connections"]
        WebWS[Web Client<br/>WebSocket]
        MobileWS[Mobile Client<br/>WebSocket / APNs / FCM]
        IMAPIdle[IMAP Clients<br/>IDLE command]
    end

    NewMail --> Router
    LabelChange --> Router
    DraftSaved --> Router
    QuotaWarn --> Router
    Router --> ConnMgr --> WebWS
    ConnMgr --> MobileWS
    ConnMgr --> IMAPIdle
    Heartbeat -.-> ConnMgr

Notification Delivery Strategy

Client Type Protocol Fallback Reconnection
Web browser WebSocket HTTP long-polling (every 30s) Exponential backoff (1s, 2s, 4s, … 60s max)
Mobile (foreground) WebSocket N/A Same as web
Mobile (background) APNs (iOS) / FCM (Android) Polling on next foreground OS-managed
IMAP clients IMAP IDLE CONDSTORE + periodic CHECK Client-managed

Connection Scaling

WebSocket Gateway sizing:
  - Each node handles ~50K concurrent connections
  - Connection state stored in Redis (userId -> nodeId mapping)
  - Sticky sessions via LB (user always reconnects to same node)
  - Graceful drain on node restart (redirect connections)

For 1B users, ~100M concurrent connections:
  Nodes needed: 100M / 50K = 2,000 gateway nodes

5.9 Contact Management

flowchart TB
    subgraph Sources["Contact Sources"]
        Manual[Manual Entry<br/>User-created]
        Auto[Auto-Extracted<br/>From sent/received emails]
        Global[Global Directory<br/>Enterprise LDAP/AD]
        Import[Import<br/>CSV, vCard, Gmail]
    end

    subgraph Storage["Contact Storage"]
        ContactStore[(Contact Store<br/>PostgreSQL)]
        ContactIdx[(Contact Index<br/>Search + Auto-complete)]
        Merge[(Merge Engine<br/>Duplicate detection)]
    end

    subgraph Features["Features"]
        Autocomplete[Auto-complete<br/>Type-ahead in To/CC/BCC]
        Groups[Contact Groups<br/>Aliases for distributions]
        Avatar[Avatar Service<br/>Gravatar / Corporate photo]
        Presence[Presence Indicator<br/>Online/Busy/Away]
    end

    Manual --> ContactStore
    Auto --> Merge --> ContactStore
    Global --> ContactStore
    Import --> Merge --> ContactStore
    ContactStore --> ContactIdx --> Autocomplete
    ContactStore --> Groups
    ContactStore --> Avatar

Auto-Contact Extraction

Process:
  1. On each sent/received email, extract all addresses from From, To, CC, BCC
  2. Check if address exists in user's contact store
  3. If not, create "auto-contact" with:
     - Email address
     - Display name (from headers)
     - Last contact date
     - Contact frequency counter
     - Source: "auto-extracted"
  4. If exists, update last contact date and frequency

Merge rules:
  - Same email address = same contact
  - Same name + different addresses = potential merge (user confirmation)
  - Frequency threshold: Show in auto-complete after 2+ interactions

6. Data Model

6.1 Entity Relationship Overview

erDiagram
    USER ||--o{ MESSAGE : owns
    USER ||--o{ LABEL : creates
    USER ||--o{ CONTACT : manages
    USER ||--o{ DRAFT : writes
    MESSAGE ||--o{ ATTACHMENT : contains
    MESSAGE }o--|| THREAD : belongs-to
    MESSAGE }o--o{ LABEL : labeled-with
    MESSAGE ||--o{ RECIPIENT : addressed-to
    CONTACT }o--o{ CONTACT_GROUP : member-of

    USER {
        UUID userId PK
        VARCHAR email
        VARCHAR displayName
        VARCHAR passwordHash
        BIGINT storageUsed
        BIGINT storageQuota
        TIMESTAMP createdAt
        TIMESTAMP lastLogin
        JSON settings
    }

    MESSAGE {
        UUID messageId PK
        UUID userId FK
        UUID threadId FK
        VARCHAR messageIdHeader
        MAP headers
        TEXT bodyPlain
        TEXT bodyHtml
        BIGINT size
        TIMESTAMP receivedTime
        SET flags
        TEXT checksum
        BIGINT version
    }

    LABEL {
        UUID labelId PK
        UUID userId FK
        VARCHAR name
        VARCHAR color
        UUID parentLabelId
        BOOLEAN systemLabel
        BOOLEAN showInList
        TIMESTAMP createdAt
    }

    LABEL_MESSAGE {
        UUID userId FK
        VARCHAR labelId FK
        UUID messageId FK
        TIMESTAMP receivedTime
        TEXT snippet
        TEXT from
        TEXT subject
    }

    ATTACHMENT {
        UUID attachmentId PK
        UUID messageId FK
        VARCHAR contentHash
        VARCHAR fileName
        VARCHAR mimeType
        BIGINT size
        VARCHAR storagePath
        TIMESTAMP uploadedAt
    }

    THREAD {
        UUID threadId PK
        UUID userId FK
        VARCHAR subject
        UUID lastMessageId
        TIMESTAMP lastActivity
        BIGINT messageCount
        BIGINT participantCount
    }

    CONTACT {
        UUID contactId PK
        UUID userId FK
        VARCHAR name
        JSON emails
        JSON phoneNumbers
        JSON addresses
        VARCHAR avatarUrl
        BOOLEAN isAutoExtracted
        TIMESTAMP lastContact
        BIGINT contactFrequency
    }

    DRAFT {
        UUID draftId PK
        UUID userId FK
        JSON recipients
        VARCHAR subject
        TEXT bodyHtml
        TEXT bodyPlain
        JSON attachmentRefs
        TIMESTAMP lastModified
        TIMESTAMP scheduledSendTime
    }

6.2 Message Headers Storage

{
  "messageId": "msg_abc123",
  "headers": {
    "From": "John Doe <john@example.com>",
    "To": "Jane Smith <jane@example.com>",
    "Cc": "Team <team@example.com>",
    "Subject": "Q4 Planning Meeting",
    "Date": "2026-08-28T10:30:00Z",
    "Message-ID": "<unique-id@example.com>",
    "In-Reply-To": "<parent-id@example.com>",
    "References": "<grandparent-id@example.com> <parent-id@example.com>",
    "Content-Type": "multipart/alternative",
    "X-Priority": "1",
    "X-Mailer": "Gmail-Clone/1.0",
    "DKIM-Signature": "v=1; a=rsa-sha256; ...",
    "Received-SPF": "pass (example.com: domain designates ...)",
    "Authentication-Results": "example.com; dkim=pass; spf=pass; dmarc=pass"
  }
}

6.3 Thread Construction Algorithm

Thread grouping rules:
  1. Same Message-ID References chain (strongest signal)
  2. Same normalized subject + overlapping participants
  3. Reply detection: In-Reply-To header matches another message's Message-ID

Normalization:
  - Strip "Re:", "Fw:", "Rw:" prefixes
  - Trim whitespace
  - Case-insensitive comparison

Thread ID generation:
  - If In-Reply-To exists and references a known message: inherit threadId
  - Else if subject matches existing thread (within 30 days): join thread
  - Else: create new thread with new threadId

7. Key Design Decisions

7.1 Labels vs Folders

Aspect Labels (Gmail-style) Folders (Traditional)
Model Many-to-many (message ↔ label) One-to-one (message → folder)
Flexibility High — organize multiple ways Low — single categorization
Complexity Higher (multiple indexes) Lower (single parent pointer)
User mental model Set-based thinking Hierarchical thinking
Storage overhead Higher (label_message junction) Lower

Decision: Labels — More flexible, matches modern user expectations. The storage overhead is acceptable given the indexing benefits.

7.2 Message Storage: Raw vs Parsed

Aspect Store Raw Store Parsed Store Both
Fidelity Perfect (RFC 2822) Lossy Perfect + convenient
Read performance Slow (parse on read) Fast Fast
Storage Compact Larger Largest
Forward compatibility Best Risk of loss Best + convenient

Decision: Store Both — Raw for fidelity and forwarding; parsed for fast reads and search indexing. Raw stored compressed; parsed fields denormalized for query performance.

7.3 Search: Elasticsearch vs Self-Built

Aspect Elasticsearch Self-Built (LSM-based)
Maturity Battle-tested Greenfield risk
Features Rich query DSL, analyzers Build from scratch
Operations Well-documented, community Full ownership burden
Customization Limited deep customization Full control
Resource usage Heavy (JVM) Potentially lighter

Decision: Elasticsearch — Time-to-market and feature completeness outweigh customization needs. Can migrate to self-built if scale demands it.

7.4 Consistency Model

Operation Consistency Level Rationale
Send email Strong (synchronous) Sender must know success/failure
Receive email Eventual (async via Kafka) Pipeline allows slight delay
Label changes Strong (synchronous) User expects immediate feedback
Search indexing Eventual (near-real-time, 5s) Search can be slightly stale
Unread counters Eventual (async counter updates) Approximate counts acceptable
Spam classification Strong (before delivery) Must classify before user sees

7.5 Attachment Storage: Inline vs External

Aspect Inline (in message store) External (object store)
Simplicity Simpler (single store) More complex (two stores)
Scalability Limited by DB size Virtually unlimited
CDN caching Hard Easy
Deduplication Hard Natural (content-addressable)
Cost Higher (DB storage) Lower (object storage)

Decision: External object store — Attachments are large, rarely modified, and benefit from CDN caching and deduplication. Message store keeps only references.


8. Scalability Strategy

8.1 Horizontal Scaling Dimensions

flowchart LR
    subgraph Scale["Scaling Levers"]
        S1[SMTP Gateways<br/>Scale by connection count]
        S2[API Servers<br/>Scale by CPU/memory]
        S3[Service Instances<br/>Scale by business logic load]
        S4[Storage Shards<br/>Scale by data volume]
        S5[Search Shards<br/>Scale by query load]
        S6[Cache Nodes<br/>Scale by hit rate needs]
        S7[WebSocket Gateways<br/>Scale by connection count]
    end

    subgraph Metrics["Scaling Metrics"]
        M1[Active connections]
        M2[CPU / Memory utilization]
        M3[Request latency P99]
        M4[Storage utilization %]
        M5[Query latency P99]
        M6[Cache hit rate]
    end

    S1 <-.-> M1
    S2 <-.-> M2
    S3 <-.-> M3
    S4 <-.-> M4
    S5 <-.-> M5
    S6 <-.-> M6
    S7 <-.-> M1

8.2 Sharding Strategy

Primary shard key: userId

Shard assignment: shard_id = hash(userId) % num_shards

Benefits:
  - All of a user's data on the same shard (messages, labels, threads)
  - Simple routing — no cross-shard queries for user operations
  - Predictable load distribution (assuming uniform userId distribution)

Anti-patterns avoided:
  - NOT sharding by domain (uneven distribution — gmail.com >> others)
  - NOT sharding by time (hot partition on current data)

Shard count planning:

Scale Users Shards Data/shard Nodes/shard
Startup 1M 100 ~11 GB 1
Growth 100M 1,000 ~11 GB 2
Scale 1B 10,000 ~11 GB 3
Mega 5B 50,000 ~11 GB 5

8.3 Caching Strategy

flowchart TB
    subgraph CacheLayers["Cache Hierarchy"]
        L1[L1: Client Cache<br/>Browser localStorage / Mobile cache]
        L2[L2: CDN Cache<br/>Static assets, public attachments]
        L3[L3: API Response Cache<br/>Redis — hot mailbox state]
        L4[L4: Service-Level Cache<br/>In-process — counters, metadata]
    end

    L1 --> L2 --> L3 --> L4
Cache Layer Content TTL Eviction Hit Rate Target
Client Drafts, recent messages Session / 1 hour LRU 40%
CDN Attachments, static assets 24 hours LRU 80%
Redis Inbox preview, counters, labels 5 minutes LRU + TTL 60%
In-process User settings, quota 1 minute LRU 90%

8.4 Database Scaling

Cassandra/ScyllaDB scaling:

Read scaling:
  - Add nodes → more replicas → more read capacity
  - Token-aware routing → local reads preferred
  - Read repair → consistency without write amplification

Write scaling:
  - Add nodes → more partitions → more parallel writes
  - Batch writes → reduce round trips
  - Async indexing → decouple search index from write path

Compaction strategy:
  - SizeTieredCompaction for write-heavy workloads
  - LeveledCompaction for read-heavy workloads
  - Hybrid: SizeTiered for message data, Leveled for index data

8.5 Cross-Region Replication

Primary region: User's home region (based on signup / preference)
Replica regions: 2 additional regions for DR

Replication strategy:
  - Synchronous: Within primary region (3 replicas)
  - Asynchronous: Cross-region (eventual, < 1 minute lag)
  - Failover: Automatic promotion of replica on primary failure
  - Data residency: EU user data stays in EU regions (GDPR)

Conflict resolution:
  - Last-write-wins with vector clocks
  - User-facing operations: strong consistency within region
  - Cross-region reads: eventually consistent

9. Curveball Scenarios

Curveball 1: “What if we need to support 1GB attachments?”

Problem: Standard email attachments are limited to ~25-50 MB. Supporting 1GB attachments breaks the SMTP pipeline, storage model, and user experience.

Analysis:

Concern Impact Mitigation
SMTP size limits RFC 5321 allows SIZE extension, but 1GB over SMTP is impractical Bypass SMTP for large files
Memory buffering 1GB in memory per connection = OOM risk Streaming, never buffer fully
Storage cost 1GB × 104K emails/sec = 104 TB/sec if all large Quota limits, tiered pricing
Transfer time 1GB at 10 Mbps = 13 minutes Resumable uploads, background transfer
Virus scanning Scanning 1GB files at scale Streaming scan, sample-based for known-safe types

Solution: Hybrid Attachment Model

Upload flow for large attachments (>25 MB):

  1. Client detects large file
  2. Instead of embedding in email, upload to "Smart Attachment" service:
     a. Initiate multipart upload (100 MB parts)
     b. Each part virus-scanned on arrival
     c. Content-addressed storage (SHA-256)
     d. Deduplication check per part
  3. Service returns a shareable link + metadata card
  4. Email contains a lightweight "attachment card":
     {
       "type": "smart_attachment",
       "name": "presentation.pptx",
       "size": 1073741824,
       "mimeType": "application/vnd.ms-powerpoint",
       "downloadUrl": "https://attachments.example.com/d/abc123",
       "previewUrl": "https://attachments.example.com/p/abc123",
       "expiresAt": "2027-08-28T00:00:00Z"
     }
     }
  5. Recipient sees a card in the email body
  6. Clicking downloads from CDN (not SMTP)

Benefits:
  - Email stays small (fast delivery)
  - Large files use optimized transfer (multipart, resumable)
  - CDN serves downloads (not mail servers)
  - Deduplication saves storage (same 1GB file uploaded by 1000 users = 1 copy)
  - Expiration policies manage storage growth

Infrastructure changes:
  - New "Smart Attachment" microservice
  - Multipart upload support (S3-compatible)
  - Streaming virus scanner
  - CDN integration for downloads
  - Link expiration + cleanup cron job

Curveball 2: “What if we need real-time collaboration on emails?”

Problem: Multiple users simultaneously editing a draft email (like Google Docs). Requires operational transformation or CRDTs for conflict resolution.

Analysis:

Concern Impact Approach
Conflict resolution Concurrent edits must merge correctly OT or CRDT
Latency Collaborators expect < 100ms sync WebSocket, edge computing
Authorization Who can edit? Granular permissions RBAC on draft level
Presence Show who’s editing, cursor positions Real-time presence channel
History Undo/redo across collaborators Operation log

Solution: CRDT-Based Collaborative Editing

Architecture:

  Client A          Collaborative Engine          Client B
  (Editor)    <-->  (WebSocket + CRDT)    <-->   (Editor)
                       |
                  Operation Log
                  (Append-only)

CRDT Choice: Yjs (Yjs-XML for rich text)
  - Conflict-free by design
  - No central coordinator needed for merge
  - Supports rich text (bold, italic, links, images)
  - Awareness protocol for presence

Implementation:

  1. Draft becomes "collaborative" when shared:
     POST /api/v1/users/{userId}/compose/{draftId}/share
     Body: { collaborators: [userId1, userId2], permissions: { userId1: "edit", userId2: "view" } }

  2. Each collaborator connects via WebSocket:
     WS /api/v1/collaborate/{draftId}?token={authToken}

  3. Engine maintains:
     - Document state (Yjs document)
     - Awareness (cursor positions, user presence)
     - Operation log (for replay, audit)

  4. Edit flow:
     a. User types → local CRDT update
     b. Delta sent to engine via WebSocket
     c. Engine broadcasts delta to all collaborators
     d. Each client applies delta to local CRDT
     e. Conflict-free merge guaranteed by CRDT properties

  5. Auto-save:
     - Engine persists document state every 5 seconds
     - On disconnect, last state saved
     - Reconnection replays missed operations from log

Performance considerations:
  - Typical draft: < 100 KB → CRDT overhead negligible
  - Max collaborators per draft: 10 (practical limit)
  - Operation rate: ~10 ops/sec per user (typing speed)
  - Engine capacity: ~1,000 concurrent collaborative sessions per node

Limitations:
  - Only applies to drafts, not sent messages
  - Attachment collaboration not supported (link sharing instead)
  - Recipient list collaboration: separate conflict resolution (last-write-wins)

Curveball 3: “What about spam at scale? A coordinated botnet sends 10M spam emails/second targeting our users.”

Problem: A sophisticated botnet launches a coordinated spam campaign at 10M emails/second — 100x normal traffic. This could overwhelm the ingestion pipeline, waste storage, and degrade service for legitimate users.

Analysis:

Attack Vector Characteristics Detection Difficulty
Volume flood Massive quantity from many IPs Easy (rate anomaly)
Spoofed senders Fake From addresses Medium (SPF/DKIM)
Content variation Slightly different content per email Hard (ML needed)
Legitimate-looking Well-formed, proper headers Hardest (behavioral)
Slow drip Below rate limits, over time Hard (pattern analysis)

Multi-Layer Defense Strategy:

flowchart TB
    subgraph L0["Layer 0: Network-Level (Before SMTP)"]
        GeoBlock[Geo-IP Blocking<br/>Block known bad regions]
        IPBlock[IP Blocklist<br/>Real-time threat intel]
        ConnLimit[Connection Rate Limit<br/>Per-IP max connections]
    end

    subgraph L1["Layer 1: SMTP-Level (During Handshake)"]
        EHLOCheck[EHLO Validation<br/>Bot detection patterns]
        TLSReq[Mandatory TLS<br/>Reject non-TLS]
        AuthReq[Mandatory Auth<br/>For relay]
        Greylist[Greylisting<br/>First-time senders delayed]
    end

    subgraph L2["Layer 2: Envelope-Level (Before Content)"]
        SenderRep[Sender Reputation<br/>Real-time score]
        RecipientCheck[Recipient Validation<br/>Reject invalid recipients]
        BurstDetect[Burst Detection<br/>Per-sender rate limit]
    end

    subgraph L3["Layer 3: Content-Level (After Receipt)"]
        FastSpam[Fast Spam Filter<br/>Rules + lightweight ML]
        ContentBlock[Content Blocking<br/>Known spam patterns]
        URLBlock[URL Blocklist<br/>Known malicious URLs]
    end

    subgraph L4["Layer 4: Behavioral (Post-Delivery)"]
        UserFeedback[User Feedback Loop<br/>Mark as spam training]
        PatternDetect[Pattern Detection<br/>Cross-user correlation]
        AutoRemediate[Auto-Remediation<br/>Bulk delete, block sender]
    end

    L0 --> L1 --> L2 --> L3 --> L4

Detailed Response Plan:

Phase 1: Immediate Response (0-5 minutes)

  1. Anomaly detection triggers alert:
     - Ingestion rate > 5x baseline → Auto-scale + alert
     - Spam score distribution shifts → Tighten thresholds

  2. Automated countermeasures:
     a. Dynamic rate limiting:
        - Per-IP: 100 emails/hour (from 1000)
        - Per-domain: 10K emails/hour (from 100K)
        - New domains: 100 emails/hour (from 1000)

     b. Reputation threshold tightening:
        - Minimum sender reputation: 0.7 (from 0.3)
        - New domain minimum: 0.9 (from 0.5)

     c. Greylisting activation:
        - All first-time senders: 5-minute delay
        - Bots typically don't retry → filtered out

Phase 2: Active Mitigation (5-30 minutes)

  3. ML model emergency update:
     a. Extract features from detected spam batch
     b. Train incremental model update (online learning)
     c. Deploy updated model to all spam filter nodes

  4. Blocklist propagation:
     a. Extract sender IPs, domains, patterns
     b. Push to all SMTP gateways (hot reload)
     c. Push to DNS-based blocklists (DNSBL)

  5. Storage protection:
     a. Spam messages NOT stored in user mailbox
     b. Spam stored in ephemeral quarantine (24h TTL)
     c. If spam volume > threshold, discard without storage

Phase 3: Recovery & Hardening (30 min - 24 hours)

  6. User communication:
     a. Notify affected users of spam wave
     b. Auto-clean spam folders
     c. Provide "report spam" one-click action

  7. Post-incident analysis:
     a. Root cause analysis
     b. Update detection rules
     c. Retrain ML models on new data
     d. Update rate limit policies

Storage impact analysis:
  Normal: 104K emails/sec × 50 KB = 5.2 GB/s
  Attack: 10M emails/sec × 50 KB = 500 GB/s

  Without mitigation: 500 GB/s × 3600s = 1.8 TB/hour of spam
  With Layer 0-2 filtering: ~90% rejected before content receipt
  With Layer 3 filtering: ~95% of remaining caught
  Storage impact: 500 GB/s × 10% × 5% = 250 MB/s = 900 GB/hour
  Quarantine TTL 24h: Max 21.6 TB (acceptable with auto-cleanup)

Curveball 4: “What if a user’s mailbox has 10 million messages? How do we keep it performant?”

Problem: Power users, enterprise accounts, or archived mailboxes can accumulate millions of messages. Naive implementations would degrade linearly.

Solution: Tiered Access Pattern

Access tiers:

  Tier 1: Hot (last 30 days, ~1,500 messages)
    - Stored in memory-optimized storage (Redis + SSD)
    - Sub-10ms access
    - Full indexing

  Tier 2: Warm (30 days - 1 year, ~10,000 messages)
    - Stored in SSD-backed Cassandra
    - < 50ms access
    - Full indexing

  Tier 3: Cold (1+ years, rest)
    - Stored in HDD-backed storage
    - < 200ms access
    - Search index only (no full body cache)

  Tier 4: Archive (>5 years)
    - Stored in object storage (cold tier)
    - < 2s access
    - Metadata index only

UI implications:
  - Default view: Tier 1 + Tier 2
  - "Show older" button: Loads Tier 3
  - Archive search: Explicit search with "include archive" toggle

Pagination optimization:
  - Use search-after cursor (not offset-based)
  - Pre-fetch next page in background
  - Virtual scrolling for web client

Counter caching:
  - Unread count: Cached counter (updated async)
  - Total count: Approximate (Cassandra counter)
  - Label counts: Materialized view, refreshed every 60s

Curveball 5: “How do we handle a zero-day phishing attack that bypasses all filters?”

Problem: A novel phishing technique evades all detection layers. Thousands of users receive and interact with malicious emails before detection.

Solution: Rapid Response + Retroactive Protection

Detection (beyond inbound filtering):

  1. Client-side detection:
     - Browser extension flags suspicious links on hover
     - Mobile app sandbox for opened attachments
     - Real-time URL reputation check on click

  2. Behavioral analytics:
     - Unusual click patterns across users
     - Rapid credential submission after email open
     - Cross-correlation with threat intelligence feeds

  3. User reporting:
     - One-click "Report phishing" button
     - Aggregated reports trigger investigation

Response:

  1. Retroactive scanning (within 1 hour of detection):
     a. Identify all users who received the phishing email
     b. Auto-move to spam folder
     c. Send warning notification to affected users
     d. Invalidate any tokens/credentials potentially compromised

  2. Emergency blocklist:
     a. Block sender domain, IP, URLs immediately
     b. Push to all gateways (hot reload, < 30 seconds)
     c. Block similar patterns (domain similarity, URL structure)

  3. User protection:
     a. Force password reset for users who clicked
     b. Enable enhanced monitoring for 30 days
     c. Push security awareness notification

Prevention improvements:
  a. Update ML models with new phishing samples
  b. Add new detection rules
  c. Enhance URL sandboxing for new techniques
  d. Update client-side detection heuristics

10. Failure Modes and Mitigations

10.1 Failure Mode Matrix

Component Failure Mode Impact Detection Mitigation RTO RPO
SMTP Gateway Node crash Partial ingest loss Health check timeout Auto-replace + Kafka replay 1 min 0 (Kafka buffered)
SMTP Gateway Region outage Regional ingest loss Cross-region health check Failover to other region 5 min 0
Message Queue Broker failure Pipeline stall Consumer lag alert Broker failover + replica promotion 30 sec 0 (replicated)
Message Queue Topic corruption Data loss Checksum mismatch Replay from SMTP gateways 5 min 5 min
Message Store Node failure Read/write degradation Consistency check Replica read + auto-repair 1 min 0 (3 replicas)
Message Store Data corruption Message loss Checksum verification Restore from backup + replay 10 min 0
Search Index Shard failure Search degradation Query error rate Replica shard promotion 2 min 5 min (index lag)
Search Index Index corruption Wrong results Result validation Reindex from message store 30 min 0
Spam Filter Model degradation Spam leak / false positive Spam complaint rate Model rollback + retrain 5 min N/A
Spam Filter Service outage No spam filtering Latency spike Bypass to rules-only mode 1 min N/A
Cache Node failure Cache miss surge Hit rate drop Peer failover + repopulation 30 sec N/A
Cache Cache stampede Backend overload Latency spike Lock-based single flight 1 min N/A
Object Store Disk failure Attachment loss Checksum mismatch Erasure coding recovery 5 min 0
WebSocket GW Connection drop Notification loss Heartbeat failure Auto-reconnect + catch-up 30 sec N/A
Database Split brain Data inconsistency Quorum check Fence minority + heal 2 min 0

10.2 Disaster Recovery

DR Strategy: Active-Active Multi-Region

  Region A (Primary for users in zone A)
  Region B (Primary for users in zone B)
  Region C (Standby + DR for both)

  Cross-region replication:
    - Async replication with < 1 minute lag
    - Conflict resolution: Last-write-wins with vector clocks
    - Data residency: EU data never leaves EU regions

  DR Testing:
    - Chaos engineering: Weekly random node kills
    - Region failover drill: Quarterly full region failover
    - Data recovery drill: Monthly restore from backup

  Backup Strategy:
    - Message store: Continuous WAL shipping + daily snapshots
    - Search index: Daily full snapshot + hourly incremental
    - Object store: Versioning + cross-region replication
    - Metadata: Continuous replication + daily backup

10.3 Circuit Breaker Patterns

Circuit breaker configuration for critical paths:

  SMTP → Kafka:
    - Failure threshold: 5 consecutive failures
    - Recovery timeout: 30 seconds
    - Fallback: Buffer locally, retry on recovery

  Service → Database:
    - Failure threshold: 10% error rate over 60 seconds
    - Recovery timeout: 15 seconds
    - Fallback: Serve from cache (stale OK)

  Service → Search:
    - Failure threshold: 50% error rate over 30 seconds
    - Recovery timeout: 10 seconds
    - Fallback: Database-based search (slower, limited)

  Service → Spam Filter:
    - Failure threshold: 3 consecutive failures
    - Recovery timeout: 5 seconds
    - Fallback: Allow through with "unclassified" flag

10.4 Graceful Degradation

When under extreme load or partial failure:

  Priority 1 (Always available):
    - Send email (core function)
    - Receive email (core function)
    - Basic inbox view (last 100 messages)

  Priority 2 (Degraded if needed):
    - Full-text search (fallback to subject/sender search)
    - Attachment preview (fallback to download-only)
    - Push notifications (fallback to polling)

  Priority 3 (Disabled if needed):
    - Smart compose/reply
    - Auto-categorization
    - Contact suggestions
    - Thread grouping (show flat list)

  Communication:
    - Status page updated in real-time
    - In-app banners for degraded features
    - Email notification for major outages

Appendix A: Technology Stack Summary

Layer Technology Alternatives Rationale
API Gateway Kong / Envoy AWS API Gateway, NGINX Open source, extensible
SMTP Server Custom (Go/Rust) Postfix, Exim Performance, customizability
IMAP Server Custom (Go/Rust) Dovecot Performance, integration
Message Queue Apache Kafka RabbitMQ, Pulsar Throughput, replayability
Message Store ScyllaDB / Cassandra DynamoDB, CockroachDB Write throughput, scalability
Search Elasticsearch Solr, Meilisearch Maturity, query DSL
Object Store Ceph / MinIO AWS S3, GCS Self-hosted, S3-compatible
Cache Redis Cluster Memcached, Dragonfly Rich data structures, pub/sub
User DB PostgreSQL MySQL, CockroachDB ACID, JSON support
ML Platform TensorFlow / PyTorch XGBoost, ONNX Flexibility, ecosystem
WebSocket Custom (Go) Socket.IO, WS Lightweight, no overhead
Monitoring Prometheus + Grafana Datadog, New Relic Open source, flexible
Logging ELK Stack Loki, Splunk Full-text log search
Service Mesh Istio / Linkerd Consul Connect mTLS, observability

Appendix B: Key Metrics and SLOs

Metric SLO Measurement Alert Threshold
Email delivery latency (P99) < 10 seconds End-to-end timestamp > 15 seconds for 5 min
Inbox load latency (P99) < 500ms API response time > 1s for 5 min
Search latency (P99) < 500ms Query response time > 1s for 5 min
Availability 99.99% Uptime / total time < 99.9% for 1 hour
Spam catch rate > 99.9% Spam caught / total spam < 99% for 1 hour
Spam false positive rate < 0.1% False positives / total ham > 0.5% for 1 hour
Message durability 99.9999999% Messages lost / total Any loss
Search freshness < 5 seconds Index lag > 30 seconds
Push notification latency < 1 second Event to delivery > 3 seconds for 5 min
Attachment upload success 99.9% Successful / total uploads < 99% for 5 min

Appendix C: Security Considerations

Threat Mitigation
Email spoofing SPF, DKIM, DMARC enforcement
Phishing ML detection, URL sandboxing, user education
Malware Multi-engine virus scanning, attachment sandboxing
Data breach Encryption at rest (AES-256), encryption in transit (TLS 1.3)
Account takeover MFA, anomaly detection, secure password policy
DDoS Rate limiting, CDN absorption, auto-scaling
Data leakage DLP policies, audit logging, access controls
Insider threat RBAC, least privilege, audit trails
Compliance GDPR data subject requests, retention policies, audit logs

Document Version: 1.0 | Last Updated: 2026-08-28