Collaborative StorageQuestion #15ADVANCED

Google Docs: Real-Time Collaborative Editing via OT & CRDT

Multi-user concurrent document editor using Operational Transformation and CRDT engines, dual-tree character positioning, presence broadcasting, and delta-compressed revision histories.

Category: Category 10: Collaborative File Storage•Updated: 2026-08-30
#File Sharing#OT/CRDT#Real-time Collaboration#Presence#Versioning

Design a Documentation System like Google Docs

Author: System Design Expert | Date: 2026-08-28 Scope: Real-time collaborative document editing at Google scale


Table of Contents

  1. Requirements Clarification
  2. Back-of-Envelope Calculations
  3. System Interface Design
  4. High-Level Architecture
  5. Core Component Design
  6. Data Model
  7. Key Design Decisions
  8. Scalability Strategy
  9. Curveball Scenarios
  10. Failure Modes and Mitigations

1. Requirements Clarification

1.1 Functional Requirements

ID Requirement Priority
FR-01 Real-time collaborative editing — Multiple users edit the same document simultaneously with changes visible within ~200ms Must Have
FR-02 Cursor presence — See other users’ cursors, selections, and avatars in real-time Must Have
FR-03 Conflict resolution — Concurrent edits from multiple users must be merged correctly without data loss Must Have
FR-04 Version history — Automatic save with full version history; users can view, compare, and restore any version Must Have
FR-05 Offline editing — Users can edit documents offline; changes sync when connection is restored Must Have
FR-06 Document permissions — Granular access control (viewer, commenter, editor, owner) with shareable links Must Have
FR-07 Comments and suggestions — Inline comments, suggestion mode, comment threads with resolution Must Have
FR-08 Rich text formatting — Bold, italic, headings, lists, tables, images, code blocks, equations Must Have
FR-09 Document search — Full-text search within a document and across all documents Should Have
FR-10 Export/Import — Export to PDF, DOCX, HTML, ODT; import from DOCX, ODT, HTML Should Have
FR-11 Templates — Pre-built document templates for common use cases Should Have
FR-12 Revision comparison — Side-by-side diff view between any two versions Should Have
FR-13 Activity timeline — See who did what and when (edit history timeline) Could Have
FR-14 Integration — Embed charts, draw diagrams, link to other services Could Have

1.2 Non-Functional Requirements

ID Requirement Target
NFR-01 Edit latency P99 < 200ms for edit propagation to co-editers
NFR-02 Availability 99.95% uptime (26 min downtime/month)
NFR-03 Durability 99.999999999% (11 nines) — same as S3
NFR-04 Concurrency Support 100+ concurrent editors per document
NFR-05 Document size Support documents up to 100MB (text + media)
NFR-06 Scalability 1B+ documents, 100M+ monthly active users
NFR-07 Consistency Strong consistency for document state within a session; eventual consistency across replicas
NFR-08 Security AES-256 encryption at rest, TLS 1.3 in transit, SOC 2 compliance
NFR-09 Data residency Support geo-fencing for enterprise customers
NFR-10 Auto-save Changes persisted within 1 second of edit

2. Back-of-Envelope Calculations

2.1 Assumption Sets

We reason from first principles with three assumption sets to bound our estimates:

Parameter Conservative Base Case Aggressive
Monthly Active Users (MAU) 50M 100M 500M
Documents per user (avg) 20 50 100
Total documents 1B 5B 50B
Avg document size (text) 50 KB 100 KB 200 KB
Avg document size (with media) 200 KB 500 KB 1 MB
Edits per active session 500 1,000 5,000
Avg session duration 15 min 30 min 60 min
Concurrent editors per doc 3 5 10
Edit operation size 50 bytes 100 bytes 200 bytes

2.2 Storage Estimates

Base Case Calculation:
├── Raw document storage
│   ├── Text content:        5B docs × 100 KB = 500 TB
│   └── Media attachments:   5B docs × 400 KB = 2 PB
│   └── Subtotal:            2.5 PB
│
├── Version history (keep 10 versions avg)
│   └── 2.5 PB × 10 = 25 PB
│
├── Metadata (permissions, comments, etc.)
│   └── 5B docs × 10 KB = 50 TB
│
├── Index (search, full-text)
│   └── 2.5 PB × 2x = 5 PB
│
└── Total with 3x replication: ~90 PB

2.3 Bandwidth Estimates

Base Case — Edit Traffic:
├── Daily active users:      100M MAU × 20% = 20M DAU
├── Sessions per day:        20M × 3 sessions = 60M sessions
├── Edit operations/day:     60M × 1,000 = 60B operations
├── Edit payload/day:        60B × 100 bytes = 6 TB/day
├── Avg QPS (edits):         60B / 86,400s ≈ 694K QPS
├── Peak QPS (3x avg):       ~2M QPS
│
└── Document Load Traffic:
    ├── Doc loads/day:       60M sessions × 2 docs = 120M loads
    ├── Load payload/day:    120M × 500 KB = 60 TB/day
    └── Peak load QPS:       ~5K QPS (but high bandwidth burst)

2.4 Infrastructure Estimates

Edit Operation Servers:
├── Per server capacity:     5,000 ops/sec (WebSocket connections)
├── Servers needed:          2M QPS / 5K = 400 servers
└── With 3x redundancy:      1,200 servers

Document Storage Servers:
├── Per server capacity:     10 TB usable (3.6 TB × 3 drives)
├── Servers needed:          90 PB / 10 TB = 9,000 servers
└── With replication:        Already factored in

Database Nodes:
├── Metadata DB:             50 TB → ~200 nodes (250 GB/node)
└── Search Index:            5 PB → ~500 nodes (10 TB/node)

3. System Interface Design

3.1 REST API (Document Management)

# Document CRUD
POST   /api/v1/documents                    # Create document
GET    /api/v1/documents/{docId}             # Get document metadata
PUT    /api/v1/documents/{docId}             # Update metadata
DELETE /api/v1/documents/{docId}             # Delete document
GET    /api/v1/documents/{docId}/content     # Get full document content

# Version History
GET    /api/v1/documents/{docId}/versions                    # List versions
GET    /api/v1/documents/{docId}/versions/{versionId}        # Get specific version
POST   /api/v1/documents/{docId}/versions/{versionId}/restore # Restore version
GET    /api/v1/documents/{docId}/versions/{v1}/diff/{v2}     # Compare versions

# Permissions
GET    /api/v1/documents/{docId}/permissions                 # List permissions
POST   /api/v1/documents/{docId}/permissions                 # Grant permission
DELETE /api/v1/documents/{docId}/permissions/{userId}        # Revoke permission

# Comments
GET    /api/v1/documents/{docId}/comments                    # List comments
POST   /api/v1/documents/{docId}/comments                     # Add comment
PUT    /api/v1/documents/{docId}/comments/{commentId}         # Update comment
DELETE /api/v1/documents/{docId}/comments/{commentId}         # Delete comment

# Export/Import
POST   /api/v1/documents/{docId}/export?format=pdf             # Export
POST   /api/v1/documents/import                                # Import

# Search
GET    /api/v1/search?q={query}&scope={document|global}        # Search

3.2 WebSocket Protocol (Real-time Collaboration)

# Connection
WS CONNECT wss://docs.example.com/ws/{docId}?token={authToken}

# Client → Server Messages
{
  "type": "EDIT",
  "payload": {
    "operation": { "type": "insert", "position": 42, "text": "Hello" },
    "version": 1507,
    "clientId": "client-abc-123",
    "timestamp": 1724812345678
  }
}

{
  "type": "CURSOR_UPDATE",
  "payload": {
    "clientId": "client-abc-123",
    "position": { "start": 42, "end": 47 },
    "scrollY": 350
  }
}

{
  "type": "PRESENCE_JOIN",
  "payload": {
    "clientId": "client-abc-123",
    "userId": "user-456",
    "userName": "Alice",
    "avatarUrl": "https://...",
    "color": "#FF5722"
  }
}

{
  "type": "PRESENCE_LEAVE",
  "payload": { "clientId": "client-abc-123" }
}

{
  "type": "COMMENT_ADD",
  "payload": {
    "range": { "start": 100, "end": 110 },
    "text": "Please review this section",
    "timestamp": 1724812345678
  }
}

# Server → Client Messages
{
  "type": "EDIT_ACK",
  "payload": {
    "version": 1508,
    "clientId": "client-abc-123"
  }
}

{
  "type": "REMOTE_EDIT",
  "payload": {
    "operation": { "type": "delete", "position": 50, "length": 5 },
    "version": 1508,
    "clientId": "client-def-456",
    "userId": "user-789"
  }
}

{
  "type": "PRESENCE_UPDATE",
  "payload": {
    "clients": [
      { "clientId": "client-def-456", "userId": "user-789",
        "userName": "Bob", "cursor": { "start": 50, "end": 55 }, "color": "#2196F3" }
    ]
  }
}

{
  "type": "STATE_SYNC",
  "payload": {
    "documentContent": "...",
    "version": 1508,
    "operations": [ /* ops since client's last version */ ]
  }
}

{
  "type": "CONFLICT_RESOLUTION",
  "payload": {
    "conflictingOps": [ /* ops that need re-base */ ],
    "baseVersion": 1505
  }
}

3.3 gRPC Internal Services

service DocumentService {
  rpc GetDocument(GetDocumentRequest) returns (Document);
  rpc SaveDocument(SaveDocumentRequest) returns (SaveResponse);
  rpc ApplyOperation(ApplyOperationRequest) returns (OperationResult);
  rpc GetVersionHistory(GetVersionHistoryRequest) returns (VersionList);
}

service PresenceService {
  rpc UpdatePresence(PresenceUpdateRequest) returns (PresenceState);
  rpc GetActiveUsers(GetActiveUsersRequest) returns (UserList);
  rpc BroadcastPresence(BroadcastRequest) returns (stream PresenceUpdate);
}

service PermissionService {
  rpc CheckPermission(CheckPermissionRequest) returns (PermissionResult);
  rpc GrantPermission(GrantPermissionRequest) returns (PermissionResult);
}

4. High-Level Architecture

graph TB
    subgraph Client["Client Layer"]
        C1[User A - Browser]
        C2[User B - Browser]
        C3[User C - Mobile App]
    end

    subgraph Edge["Edge / CDN Layer"]
        CDN[CDN / Static Assets]
        WAF[WAF / DDoS Protection]
        LB[Global Load Balancer]
    end

    subgraph Gateway["API Gateway Layer"]
        GW[API Gateway]
        WS_GW[WebSocket Gateway]
        AUTH[Auth Service<br/>OAuth 2.0 / JWT]
    end

    subgraph Core["Core Services"]
        subgraph Collab["Collaboration Services"]
            OT[OT / CRDT Engine]
            PRESENCE[Presence Service]
            DOC_STATE[Document State Manager]
        end

        subgraph DocMgmt["Document Management"]
            DOC_API[Document API Service]
            VERSION[Version History Service]
            EXPORT[Export Service]
            SEARCH[Search Service]
        end

        subgraph Meta["Metadata Services"]
            PERM[Permission Service]
            COMMENT[Comment Service]
            NOTIF[Notification Service]
        end
    end

    subgraph Storage["Storage Layer"]
        REDIS[(Redis Cluster<br/>Op Log / Presence)]
        DYNAMO[(DynamoDB / Spanner<br/>Metadata)]
        S3[(S3 / GCS<br/>Document Content)]
        ES[(Elasticsearch<br/>Full-Text Index)]
        VERSION_S3[(S3 / GCS<br/>Version Snapshots)]
    end

    subgraph Infra["Infrastructure"]
        MQ[Kafka / Pub-Sub<br/>Event Bus]
        MONITOR[Monitoring & Alerting]
        CACHE[Memcached / Redis<br/>Read Cache]
    end

    C1 -->|WebSocket| WAF
    C2 -->|WebSocket| WAF
    C3 -->|HTTPS| WAF

    WAF --> LB
    LB --> GW
    LB --> WS_GW

    GW --> AUTH
    GW --> DOC_API
    GW --> VERSION
    GW --> EXPORT
    GW --> SEARCH
    GW --> PERM
    GW --> COMMENT

    WS_GW --> AUTH
    WS_GW --> OT
    WS_GW --> PRESENCE

    OT --> DOC_STATE
    OT --> REDIS
    DOC_STATE --> S3
    DOC_STATE --> DYNAMO

    PRESENCE --> REDIS

    DOC_API --> DYNAMO
    DOC_API --> S3
    DOC_API --> PERM

    VERSION --> VERSION_S3
    VERSION --> DYNAMO

    SEARCH --> ES

    DOC_STATE -->|Events| MQ
    COMMENT -->|Events| MQ
    MQ --> NOTIF
    MQ --> ES
    MQ --> VERSION

    DOC_API --> CACHE
    SEARCH --> CACHE

5. Core Component Design

5.1 Operational Transformation (OT) Engine

The OT engine is the heart of real-time collaboration. It ensures that concurrent edits from multiple users are correctly merged without conflicts.

How OT Works

User A: "Hello World"
User B: "Hello World"

Time T1: User A inserts " Beautiful" at position 6
         → "Hello Beautiful World"

Time T2: User B deletes "World" at position 6 (length 5)
         → "Hello "

Without OT: Both operations conflict!
With OT: Operation B is transformed against Operation A:
         - B's delete was at position 6, length 5 ("World")
         - A inserted 11 chars at position 6
         - Transform B: delete now at position 6+11=17, length 5
         - Apply A first: "Hello Beautiful World"
         - Apply transformed B: "Hello Beautiful "
         ✓ Both users see consistent result

OT Algorithm

function Transform(opA, opB):
    """
    Transform opA against opB.
    Returns (opA', opB') such that:
    Apply(Apply(doc, opA), opB') == Apply(Apply(doc, opB), opA')
    """
    opA_prime = copy(opA)
    opB_prime = copy(opB)

    if opA.type == INSERT and opB.type == INSERT:
        if opA.position < opB.position or
           (opA.position == opB.position and opA.clientId < opB.clientId):
            opB_prime.position += len(opA.text)
        else:
            opA_prime.position += len(opB.text)

    elif opA.type == INSERT and opB.type == DELETE:
        if opA.position > opB.position:
            opA_prime.position -= opB.length
            if opA.position <= opB.position + opB.length:
                opA_prime.text = opA.text[opA.position - opB.position:]
                opA_prime.position = opB.position

    elif opA.type == DELETE and opB.type == INSERT:
        if opB.position > opA.position:
            opB_prime.position -= opA.length

    elif opA.type == DELETE and opB.type == DELETE:
        # Handle overlapping deletes
        if opA.position < opB.position:
            if opA.position + opA.length > opB.position:
                overlap = opA.position + opA.length - opB.position
                opB_prime.length -= overlap
                opB_prime.position = opA.position
            else:
                opB_prime.position -= opA.length
        # ... symmetric cases

    return (opA_prime, opB_prime)

OT Server Architecture

graph LR
    subgraph OTServer["OT Server (per document)"]
        OPLOG[Operation Log<br/>Ordered Op Queue]
        TRANSFORM[Transform Engine]
        BROADCAST[Broadcast Manager]
        STATE[Document State<br/>In-Memory]
    end

    C1[Client 1] -->|Op v5| TRANSFORM
    C2[Client 2] -->|Op v5| TRANSFORM
    C3[Client 3] -->|Op v6| TRANSFORM

    TRANSFORM -->|Transform against ops v5-v7| OPLOG
    OPLOG -->|Ack v8| TRANSFORM
    TRANSFORM -->|Broadcast transformed ops| BROADCAST
    BROADCAST -->|Remote Op v8| C1
    BROADCAST -->|Remote Op v8| C2
    BROADCAST -->|Remote Op v8| C3

    STATE -->|Periodic snapshot| S3[(S3)]
    OPLOG -->|Async persist| REDIS[(Redis)]

Key OT Properties

Property Description Implementation
Convergence All clients reach the same document state Guaranteed by OT transform function correctness
Intention Preservation User’s intended edit is preserved Transform adjusts position, not intent
Causality Operations applied in causal order Version vectors + server-side ordering
Idempotency Re-applying same op has no effect Client ID + sequence number dedup

5.2 CRDT Alternative (Yjs / Automerge)

While OT is used by Google Docs, CRDTs offer an alternative approach:

graph TD
    subgraph CRDT["CRDT Approach (Yjs)"]
        subgraph YText["Y.Text Data Structure"]
            ITEM1[Item: id=(1,A) 'H']
            ITEM2[Item: id=(2,A) 'e']
            ITEM3[Item: id=(3,B) 'l']
            ITEM4[Item: id=(4,A) 'l']
            ITEM5[Item: id=(5,B) 'o']
        end

        subgraph YState["State Vector"]
            SV["{A: 4, B: 5}"]
        end

        subgraph YUpdate["Update Mechanism"]
            INC[Incremental Update]
            MERGE[State Merge]
        end
    end

    ITEM1 --> ITEM2 --> ITEM3 --> ITEM4 --> ITEM5
    SV --> MERGE
    INC --> MERGE

OT vs CRDT Comparison

Aspect OT CRDT (Yjs)
Central authority Required (server orders ops) Not required (peer-to-peer)
Offline support Limited (needs sync) Excellent (merge on reconnect)
Complexity High (transform matrix) Medium (data structure complexity)
Storage overhead Low (ops are small) Higher (metadata per character)
Undo/Redo Complex Built-in support
Network efficiency Better (smaller ops) Worse (more metadata)
Production maturity Proven (Google Docs, Etherpad) Growing (Codemirror, Tldraw)

5.3 Presence Service

Tracks real-time cursor positions, selections, and user information.

classDiagram
    class PresenceService {
        +Map~docId, DocumentPresence~ documents
        +Map~clientId, ClientState~ clients
        +join(docId, clientId, userInfo)
        +leave(clientId)
        +updateCursor(clientId, position)
        +getActiveUsers(docId)
        +broadcastPresence(docId, updates)
        +cleanupStaleClients(timeout)
    }

    class DocumentPresence {
        +String docId
        +Set~ClientState~ activeClients
        +Map~clientId, ClientState~ clientMap
        +lastUpdated: Timestamp
    }

    class ClientState {
        +String clientId
        +String userId
        +String userName
        +String avatarUrl
        +String color
        +CursorPos cursor
        +SelectionRange selection
        +int scrollY
        +Timestamp lastHeartbeat
    }

    class CursorPos {
        +int start
        +int end
        +Timestamp updatedAt
    }

    PresenceService --> DocumentPresence
    DocumentPresence --> ClientState
    ClientState --> CursorPos

Presence Update Flow

1. Client moves cursor
2. Client sends CURSOR_UPDATE over WebSocket (throttled to 50ms)
3. Presence Service updates Redis: HSET presence:{docId} {clientId} '{cursor}'
4. Presence Service broadcasts to all connected clients for that doc
5. Each client renders remote cursors with user's color and avatar
6. If no heartbeat for 30s, client marked as inactive
7. If no heartbeat for 60s, client removed from presence

5.4 Document State Manager

Manages the authoritative document state and persists changes.

sequenceDiagram
    participant C as Client
    participant WS as WebSocket Gateway
    participant OT as OT Engine
    participant DSM as Doc State Manager
    participant REDIS as Redis (Op Log)
    participant S3 as Object Storage
    participant DB as Metadata DB

    C->>WS: EDIT operation
    WS->>OT: Apply operation
    OT->>OT: Transform against pending ops
    OT->>DSM: Apply transformed op
    DSM->>REDIS: Append op to op log
    DSM->>OT: Ack with new version
    OT->>WS: Broadcast to other clients
    WS->>C: Remote edit notification

    Note over DSM,S3: Async persistence
    DSM->>DSM: Batch ops (1s window)
    DSM->>S3: Save document snapshot
    DSM->>DB: Update metadata

5.5 Version History Service

graph TD
    subgraph VersionService["Version History Service"]
        AUTO[Auto-Save Trigger<br/>Every 60s or 50 ops]
        MANUAL[Manual Save Trigger<br/>User-initiated]
        SNAPSHOT[Snapshot Creator]
        COMPRESS[Delta Compressor]
        STORE[Version Store]
        RESTORE[Version Restorer]
        DIFF[Diff Engine]
    end

    AUTO --> SNAPSHOT
    MANUAL --> SNAPSHOT
    SNAPSHOT --> COMPRESS
    COMPRESS --> STORE
    STORE -->|S3 Object| S3BUCKET[(S3: versions/{docId}/{versionId})]

    RESTORE --> STORE
    RESTORE -->|Reconstruct| DOC[Document Content]

    DIFF -->|Compare v1 vs v2| DIFFRESULT[Diff Result]

    subgraph Retention["Retention Policy"]
        KEEP[Keep last 100 versions]
        ARCHIVE[Archive older versions]
        DELETE[Delete after 7 years]
    end

    STORE --> KEEP
    KEEP --> ARCHIVE
    ARCHIVE --> DELETE

Version Storage Strategy

Version Type Frequency Retention Storage
Auto-save Every 60s or 50 ops Last 100 versions Delta-encoded in S3
Named versions User-created Permanent Full snapshot in S3
Milestone Daily (auto) 1 year Full snapshot in S3
Archive Older than retention 7 years Cold storage (Glacier)

5.6 Permission Service

graph TD
    subgraph PermissionModel["Permission Model"]
        OWNER[Owner<br/>Full control + transfer]
        EDITOR[Editor<br/>Edit + comment]
        COMMENTER[Commenter<br/>View + comment]
        VIEWER[Viewer<br/>Read-only]
    end

    subgraph PermissionTypes["Permission Types"]
        USER[User-specific<br/>email/userId]
        GROUP[Group-based<br/>org/team]
        LINK[Link-based<br/>anyone with link]
        DOMAIN[Domain-based<br/>@company.com]
    end

    subgraph PermissionCheck["Permission Check Flow"]
        REQ[Request] --> LOOKUP[Lookup ACL]
        LOOKUP --> CACHE{In Cache?}
        CACHE -->|Yes| RETURN[Return Result]
        CACHE -->|No| DB[Query Permission DB]
        DB --> EVAL[Evaluate Hierarchy]
        EVAL -->|Owner > Editor > Commenter > Viewer| RETURN
        RETURN --> UPDATE_CACHE[Update Cache]
    end

    OWNER --> EDITOR --> COMMENTER --> VIEWER

5.7 Offline Editing Support

sequenceDiagram
    participant C as Client
    participant LOCAL as Local Storage<br/>IndexedDB
    participant NET as Network Layer
    participant S as Server

    Note over C,S: Offline Phase
    C->>LOCAL: Edit operation (queued)
    C->>LOCAL: Edit operation (queued)
    C->>LOCAL: Edit operation (queued)
    Note over LOCAL: Ops stored with local timestamps

    Note over C,S: Reconnection Phase
    NET->>S: Check server version
    S->>NET: Server version = 1508
    NET->>C: Server state since v1500

    C->>C: Merge local ops (v1501-v1505) with server ops
    C->>C: Apply OT transform for conflicts
    C->>S: Send reconciled ops
    S->>C: Ack with new version
    C->>LOCAL: Clear synced ops

Offline Storage Design

IndexedDB Schema:
├── pending_ops
│   ├── docId: "doc-123"
│   ├── ops: [op1, op2, op3, ...]
│   ├── baseVersion: 1500
│   └── createdAt: timestamp
│
├── document_cache
│   ├── docId: "doc-123"
│   ├── content: "..."
│   ├── version: 1500
│   └── cachedAt: timestamp
│
└── sync_status
    ├── docId: "doc-123"
    ├── status: "pending" | "syncing" | "synced" | "conflict"
    └── lastSync: timestamp

5.8 Search Service

graph TD
    subgraph SearchPipeline["Search Pipeline"]
        DOC[Document Content] --> TOKENIZE[Tokenizer]
        TOKENIZE --> STEM[Stemmer / Lemmatizer]
        STEM --> FILTER[Stop Word Filter]
        FILTER --> INDEX[Inverted Index]
    end

    subgraph Elasticsearch["Elasticsearch Cluster"]
        INDEX --> SHARD1[Shard 1]
        INDEX --> SHARD2[Shard 2]
        INDEX --> SHARD3[Shard 3]
    end

    subgraph SearchQuery["Search Query"]
        Q[User Query] --> PARSE[Query Parser]
        PARSE --> SCORE[Scoring Engine]
        SCORE -->|BM25 Algorithm| RANK[Rank Results]
        RANK --> FILTER_PERM[Permission Filter]
        FILTER_PERM --> RESULTS[Search Results]
    end

    SHARD1 --> SCORE
    SHARD2 --> SCORE
    SHARD3 --> SCORE

6. Data Model

6.1 Relational Schema (Metadata)

-- Documents table
CREATE TABLE documents (
    doc_id          UUID PRIMARY KEY,
    title           VARCHAR(500) NOT NULL,
    owner_id        UUID NOT NULL REFERENCES users(user_id),
    created_at      TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at      TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    version         BIGINT NOT NULL DEFAULT 0,
    status          VARCHAR(20) DEFAULT 'active',  -- active, trashed, archived
    size_bytes      BIGINT DEFAULT 0,
    content_type    VARCHAR(50) DEFAULT 'document',
    space_id        UUID REFERENCES spaces(space_id),
    metadata        JSONB DEFAULT '{}',
    CONSTRAINT valid_status CHECK (status IN ('active', 'trashed', 'archived'))
);

-- Users table
CREATE TABLE users (
    user_id         UUID PRIMARY KEY,
    email           VARCHAR(255) UNIQUE NOT NULL,
    display_name    VARCHAR(200),
    avatar_url      TEXT,
    org_id          UUID REFERENCES organizations(org_id),
    created_at      TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    is_active       BOOLEAN DEFAULT true
);

-- Document permissions (ACL)
CREATE TABLE document_permissions (
    permission_id   UUID PRIMARY KEY,
    doc_id          UUID NOT NULL REFERENCES documents(doc_id) ON DELETE CASCADE,
    grantee_type    VARCHAR(20) NOT NULL,  -- user, group, domain, link
    grantee_id      VARCHAR(500) NOT NULL,
    role            VARCHAR(20) NOT NULL,  -- owner, editor, commenter, viewer
    granted_by      UUID REFERENCES users(user_id),
    granted_at      TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    link_token      VARCHAR(64),  -- For link-based sharing
    expires_at      TIMESTAMP WITH TIME ZONE,
    UNIQUE(doc_id, grantee_type, grantee_id),
    CONSTRAINT valid_role CHECK (role IN ('owner', 'editor', 'commenter', 'viewer'))
);

-- Operation log (for OT)
CREATE TABLE operation_log (
    op_id           BIGSERIAL PRIMARY KEY,
    doc_id          UUID NOT NULL REFERENCES documents(doc_id),
    client_id       VARCHAR(64) NOT NULL,
    user_id         UUID REFERENCES users(user_id),
    operation       JSONB NOT NULL,  -- {type, position, text, ...}
    version         BIGINT NOT NULL,
    base_version    BIGINT NOT NULL,
    created_at      TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    INDEX idx_op_doc_version (doc_id, version)
);

-- Document versions
CREATE TABLE document_versions (
    version_id      UUID PRIMARY KEY,
    doc_id          UUID NOT NULL REFERENCES documents(doc_id),
    version_number  BIGINT NOT NULL,
    snapshot_key    VARCHAR(500),  -- S3 key for snapshot
    delta_key       VARCHAR(500),  -- S3 key for delta
    created_by      UUID REFERENCES users(user_id),
    created_at      TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    is_named        BOOLEAN DEFAULT false,
    name            VARCHAR(200),
    size_bytes      BIGINT,
    UNIQUE(doc_id, version_number)
);

-- Comments
CREATE TABLE comments (
    comment_id      UUID PRIMARY KEY,
    doc_id          UUID NOT NULL REFERENCES documents(doc_id),
    version         BIGINT NOT NULL,
    user_id         UUID REFERENCES users(user_id),
    range_start     INTEGER,
    range_end       INTEGER,
    text            TEXT NOT NULL,
    parent_id       UUID REFERENCES comments(comment_id),
    status          VARCHAR(20) DEFAULT 'open',  -- open, resolved
    created_at      TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    updated_at      TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

-- Indexes
CREATE INDEX idx_docs_owner ON documents(owner_id);
CREATE INDEX idx_docs_space ON documents(space_id);
CREATE INDEX idx_docs_updated ON documents(updated_at DESC);
CREATE INDEX idx_perms_doc ON document_permissions(doc_id);
CREATE INDEX idx_perms_grantee ON document_permissions(grantee_type, grantee_id);
CREATE INDEX idx_versions_doc ON document_versions(doc_id, version_number DESC);
CREATE INDEX idx_comments_doc ON comments(doc_id, version);
CREATE INDEX idx_comments_doc ON comments(doc_id, version);
CREATE INDEX idx_comments_status ON comments(doc_id, status);

6.2 NoSQL Schema (Redis - Presence & Op Log)

# Presence data (TTL: 60s, refreshed by heartbeat)
HSET presence:doc:{docId}
    client:{clientId} '{"userId":"u1","cursor":{"start":42,"end":47},"color":"#FF5722"}'
EXPIRE presence:doc:{docId} 60

# Operation log (sorted set, version as score)
ZADD oplog:doc:{docId}
    {version} '{"clientId":"c1","op":{"type":"insert","pos":42,"text":"Hi"},"ts":1724812345}'

# Document state cache
SET doc:state:{docId} '{"version":1508,"content":"...","checksum":"abc123"}' EX 300

# Active editors count
INCR active:doc:{docId}
EXPIRE active:doc:{docId} 120

6.3 Object Storage Layout (S3)

s3://docs-storage/
├── documents/
│   └── {docId}/
│       ├── content.json              # Current document content
│       ├── metadata.json             # Document metadata
│       └── media/
│           ├── image-001.png
│           └── image-002.jpg
│
├── versions/
│   └── {docId}/
│       ├── {versionId}-snapshot.json  # Full snapshot
│       └── {versionId}-delta.json     # Delta from previous
│
├── exports/
│   └── {docId}/
│       ├── {timestamp}-export.pdf
│       └── {timestamp}-export.docx
│
└── templates/
    └── {templateId}/
        └── template.json

7. Key Design Decisions

7.1 OT vs CRDT

Factor Decision: OT Rationale
Network model Client-server (centralized) Simpler conflict resolution, single source of truth
Storage efficiency Lower overhead Operations are small (50-100 bytes) vs CRDT metadata
Maturity Battle-tested Google Docs, Etherpad, Quill use OT successfully
Offline support Acceptable with sync queue CRDT better for offline, but OT sufficient for our use case
Undo/Redo Requires extra logic CRDT has built-in undo, but OT can implement with op stacking

Decision: OT with server-side coordination — Best fit for Google Docs’ centralized model with strong consistency requirements.

7.2 WebSocket vs HTTP Long-Polling

Factor WebSocket HTTP Long-Polling
Latency ~10-50ms ~200-500ms
Bandwidth Lower (single connection) Higher (headers per request)
Server load Higher (persistent connections) Lower (stateless)
Firewall compatibility Sometimes blocked Always works
Scalability Connection-heavy Request-heavy

Decision: WebSocket primary, HTTP long-polling fallback — WebSocket for real-time editing, fallback for restrictive networks.

7.3 Document Storage Format

Format Pros Cons
JSON Human-readable, easy to parse Larger size, no schema enforcement
Protocol Buffers Compact, fast, schema-enforced Binary, not human-readable
Custom binary Most compact Hardest to debug, no tooling

Decision: JSON for content, Protocol Buffers for operations — JSON for readability and debugging, Protobuf for high-frequency operation transport.

7.4 Version History Strategy

Strategy Storage Restore Speed Use Case
Full snapshots High Fast Named versions, milestones
Delta-only Low Slow (replay all ops) Auto-save versions
Hybrid Medium Fast Best of both worlds

Decision: Hybrid approach — Full snapshot every 100 auto-saves + delta between saves. Named versions always full snapshots.

7.5 Sharding Strategy

Sharding Key: doc_id (UUID)

Why doc_id?
├── Even distribution (UUID is random)
├── All operations for a doc go to the same shard
├── No cross-shard transactions needed
└── Easy to add/remove shards

Shard Assignment:
├── Consistent hashing with virtual nodes
├── 1024 virtual nodes per physical shard
└── Minimize data movement on rebalance

7.6 Consistency Model

Operation Consistency Level Rationale
Edit operations Strong (within session) Co-editors must see consistent state
Document metadata Eventual Title/permissions can lag slightly
Search index Eventual Search can be seconds behind
Version history Strong Must not lose any version
Presence Eventual Cursor position can be slightly stale

8. Scalability Strategy

8.1 Horizontal Scaling

graph TD
    subgraph Layer1["Layer 1: Stateless Services (Auto-scale)"]
        API[API Gateway Instances]
        WS[WebSocket Gateway Instances]
        AUTH[Auth Service Instances]
    end

    subgraph Layer2["Layer 2: Sharded Services"]
        OT1[OT Engine Shard 1]
        OT2[OT Engine Shard 2]
        OT3[OT Engine Shard N]
    end

    subgraph Layer3["Layer 3: Distributed Storage"]
        REDIS1[Redis Cluster Shard 1]
        REDIS2[Redis Cluster Shard 2]
        S3[S3 / GCS (infinite scale)]
        ES1[ES Shard 1]
        ES2[ES Shard 2]
    end

    API --> OT1
    API --> OT2
    API --> OT3
    WS --> OT1
    WS --> OT2
    WS --> OT3

    OT1 --> REDIS1
    OT2 --> REDIS2
    OT3 --> S3
    OT1 --> ES1
    OT2 --> ES2

8.2 Scaling Dimensions

Dimension Strategy Trigger
WebSocket connections Add WS gateway instances Connection count > 80% capacity
OT engines Add shards via consistent hashing Ops/sec per shard > 10K
Storage S3 auto-scales No action needed
Redis Add cluster nodes Memory > 70% or connections > 80%
Search Add ES nodes/shards Index size > 1TB per node
Database Read replicas + sharding Read QPS > 10K or write latency > 100ms

8.3 Caching Strategy

Cache Hierarchy:
├── L1: Browser Cache (Service Worker)
│   └── Document content, static assets
│   └── TTL: Session-based, invalidated on edit
│
├── L2: CDN Cache
│   └── Static assets, exported PDFs
│   └── TTL: 24h for static, 1h for exports
│
├── L3: Application Cache (Redis)
│   └── Document metadata, permissions
│   └── TTL: 5 min, invalidated on change
│
└── L4: Database Read Replicas
    └── Read-heavy queries
    └── Replication lag: < 1s

8.4 Database Sharding

Sharding Strategy:
├── Primary key: doc_id (UUID)
├── Shard count: 64 initial, scalable to 256
├── Shard key hash: CRC32(doc_id) % shard_count
└── Rebalancing: Online, via consistent hashing

Cross-shard operations:
├── Search: Aggregated across shards (Elasticsearch handles this)
├── User's documents: Materialized view in user_index table
└── Permissions: Denormalized, replicated to relevant shards

9. Curveball Scenarios

Curveball 1: “Two users simultaneously delete the same paragraph — what happens?”

Scenario: Alice and Bob both select and delete the same paragraph “Hello World” at positions 10-20.

Analysis:

Initial state: "The [Hello World] is here"
                    ^--^ positions 10-20

Alice's op: DELETE position=10, length=11, version=100
Bob's op:   DELETE position=10, length=11, version=100

Server receives Alice's op first:
├── Apply Alice's DELETE: "The is here"
├── Ack version 101 to Alice
└── Broadcast to Bob

Server receives Bob's op:
├── Transform Bob's DELETE against Alice's DELETE (v101):
│   ├── Both delete at same position, same length
│   ├── Bob's delete overlaps completely with Alice's
│   └── Transformed: DELETE position=10, length=0 (no-op)
├── Apply no-op: "The is here" (unchanged)
├── Ack version 102 to Bob
└── Result: Both see "The is here" ✓

Resolution: The OT transform function detects overlapping deletes and converts the second delete to a no-op. Both users see the same result — the paragraph is deleted once.

Key insight: OT’s transform matrix must handle all 9 combinations (insert/insert, insert/delete, insert/retain, delete/insert, delete/delete, delete/retain, retain/insert, retain/delete, retain/retain). The delete/delete case is critical for this scenario.


Curveball 2: “A user’s internet drops mid-edit. They reconnect 5 minutes later. How do we reconcile?”

Scenario: Alice edits a document for 5 minutes offline, making 200 changes. Meanwhile, Bob and Charlie made 150 changes online.

Reconciliation Process:

sequenceDiagram
    participant A as Alice (Client)
    participant S as Server
    participant B as Bob (Client)
    participant C as Charlie (Client)

    Note over A,C: 5 minutes of divergence
    A->>A: 200 offline ops (v1001-v1200)
    B->>S: 100 ops (v1001-v1100)
    C->>S: 50 ops (v1101-v1150)
    Note over S: Server at version 1150

    Note over A,S: Alice reconnects
    A->>S: Sync request (last known v1000)
    S->>A: Server ops v1001-v1150 (150 ops)
    A->>A: Apply server ops to local doc
    A->>A: Transform local ops against server ops
    A->>S: Send transformed local ops (200 ops)
    S->>S: Apply transformed ops (v1151-v1350)
    S->>A: Ack v1350
    S->>B: Broadcast new ops
    S->>C: Broadcast new ops

    Note over A,C: All converge to v1350

Detailed Steps:

  1. Detect reconnection: Client detects network restoration via WebSocket reconnect
  2. Fetch server state: Client requests all operations since its last known version (v1000)
  3. Apply server ops locally: Client applies the 150 server operations to its local document
  4. Transform local ops: Each of Alice’s 200 offline ops is transformed against the 150 server ops
  5. Send transformed ops: Alice sends her 200 transformed ops to the server
  6. Server applies: Server applies Alice’s ops, advancing from v1150 to v1350
  7. Broadcast: Server broadcasts these ops to Bob and Charlie
  8. Convergence: All three clients now have the same document at v1350

Conflict handling during transform:

  • If Alice and Bob both edited the same text, OT preserves both intentions where possible
  • If completely overlapping, server-order wins (first received)
  • Alice’s client shows a conflict notification if her intention was significantly altered

Optimization: For large divergence (200 ops), use state-based sync instead of operation-based:

  • Server sends full document state at v1150
  • Client diffs its local state against server state
  • Only send the delta operations

Curveball 3: “A document goes viral — 10,000 concurrent editors. How do we handle this?”

Scenario: A shared document (e.g., a company-wide town hall Q&A) suddenly gets 10,000 concurrent editors.

Problem Analysis:

Normal doc: 5 concurrent editors, ~50 ops/sec
Viral doc:  10,000 concurrent editors, ~100,000 ops/sec

Bottlenecks:
├── Single OT engine instance: Can't handle 100K ops/sec
├── WebSocket connections: 10K connections to one server
├── Broadcast fan-out: Each op sent to 10K clients = 1B messages/sec
└── Redis op log: Single key contention

Multi-Layer Solution:

graph TD
    subgraph Normal["Normal Mode (≤100 editors)"]
        OT1[Single OT Engine]
        WS1[Single WS Server]
        REDIS1[Single Redis Key]
    end

    subgraph Viral["Viral Mode (>100 editors)"]
        subgraph Sharding["Document Sharding"]
            SHARD_A[OT Shard A<br/>Editors 1-2500]
            SHARD_B[OT Shard B<br/>Editors 2501-5000]
            SHARD_C[OT Shard C<br/>Editors 5001-7500]
            SHARD_D[OT Shard D<br/>Editors 7501-10000]
        end

        MERGER[Document Merger<br/>Merge shard states]
        BROADCASTER[Broadcast Fan-out<br/>Pub-Sub]

        subgraph WS["WebSocket Cluster"]
            WS_A[WS Server A]
            WS_B[WS Server B]
            WS_C[WS Server C]
            WS_D[WS Server D]
        end
    end

    SHARD_A --> MERGER
    SHARD_B --> MERGER
    SHARD_C --> MERGER
    SHARD_D --> MERGER
    MERGER --> BROADCASTER
    BROADCASTER --> WS_A
    BROADCASTER --> WS_B
    BROADCASTER --> WS_C
    BROADCASTER --> WS_D

Detailed Strategy:

  1. Auto-detect viral mode: Monitor concurrent editor count; trigger at >100 editors
  2. Document sharding: Split the document into logical sections, each with its own OT engine
    • Shard by document sections (paragraphs, pages)
    • Each shard handles 2,500 editors and ~25K ops/sec
  3. Cross-shard coordination: A merger service periodically merges shard states
    • Merge interval: 100ms
    • Cross-shard ops transformed against merge state
  4. WebSocket clustering: Distribute connections across multiple WS servers
    • Each WS server handles 2,500 connections
    • Pub-Sub for cross-server broadcast
  5. Broadcast optimization:
    • Throttle: Limit broadcast to 20 ops/sec per client (batch)
    • Selective: Only send ops relevant to the client’s viewport
    • Compress: Use delta encoding for broadcast messages
  6. Read-only fallback: If >50,000 editors, switch non-active users to read-only mode
  7. Graceful degradation:
    • Increase sync interval from 100ms to 500ms
    • Disable cursor presence for non-active users
    • Queue non-critical operations

Capacity planning:

Metric Normal Viral Solution
Concurrent editors 5 10,000 Shard into 4 OT engines
Ops/sec 50 100,000 25K per shard
WS connections 5 10,000 4 WS servers × 2,500
Broadcast msgs/sec 250 1B Throttle + batch + selective

Curveball 4: “How do we prevent a malicious user from spamming delete operations to destroy a document?”

Threat Model:

  • Attacker has editor permissions
  • Sends 10,000 delete operations per second
  • Goal: Destroy document content or DoS other editors

Multi-Layer Protection:

graph TD
    subgraph Layer1["Layer 1: Rate Limiting"]
        RL1[Per-user rate limit: 100 ops/sec]
        RL2[Per-document rate limit: 1000 ops/sec]
        RL3[Burst limit: 200 ops in 1 sec]
    end

    subgraph Layer2["Layer 2: Operation Validation"]
        VAL1[Operation size limit: 10KB max]
        VAL2[Delete limit: max 10% of doc per op]
        VAL3[Version gap check: reject if >100 versions behind]
    end

    subgraph Layer3["Layer 3: Anomaly Detection"]
        AD1[ML-based anomaly detection]
        AD2[Pattern recognition: spam detection]
        AD3[Behavioral analysis: human-like typing]
    end

    subgraph Layer4["Layer 4: Recovery"]
        REC1[Instant rollback from version history]
        REC2[Auto-revert suspicious bulk deletes]
        REC3[Admin override capability]
    end

    Layer1 --> Layer2 --> Layer3 --> Layer4

Implementation Details:

Layer Mechanism Threshold Action
Rate Limit Token bucket per user 100 ops/sec Queue excess, drop if overflow
Rate Limit Token bucket per doc 1,000 ops/sec Throttle all users proportionally
Validation Delete size check Max 10% of doc Reject oversized deletes
Validation Version freshness Within 100 versions Reject stale operations
Anomaly Typing pattern Human-like cadence Flag non-human patterns
Anomaly Delete ratio <50% of ops are deletes Flag aggressive deletion
Recovery Auto-rollback >10% content deleted in 10s Auto-restore from last safe version
Recovery Admin alert Any anomaly detected Notify document owner

Challenge: 500ms RTT means each edit takes 500ms round-trip. With OT requiring server acknowledgment, typing feels extremely laggy.

Solution: Local-First with Optimistic Execution

sequenceDiagram
    participant C as Client (High Latency)
    participant S as Server

    Note over C: Local-first execution
    C->>C: User types 'H' → Apply locally immediately (0ms)
    C->>S: Send op to server (500ms RTT)
    S->>S: Apply op, assign version
    S->>C: Ack with version (500ms)
    C->>C: Confirm op (already applied locally)

    Note over C: Remote ops handling
    S->>C: Remote op from User B (500ms)
    C->>C: Transform against local pending ops
    C->>C: Apply transformed remote op
    C->>S: Ack remote op

    Note over C: Conflict resolution
    C->>C: Local op conflicts with remote
    C->>C: OT transform resolves conflict
    C->>C: Display resolved state

Optimizations for High Latency:

Optimization Description Impact
Local-first execution Apply edits locally before server ack 0ms perceived latency
Operation batching Batch multiple keystrokes into one op Reduce round-trips by 10x
Predictive transform Predict remote ops and pre-transform Reduce conflict resolution time
Compression Compress operation payloads Reduce transmission time
Connection pooling Reuse WebSocket connections Reduce connection setup time
Edge caching Cache at edge locations closer to user Reduce effective latency
Graceful degradation Show “syncing” indicator during conflict Better UX during conflicts

Typing experience comparison:

Scenario Without Optimization With Local-First
Typing latency 500ms per character 0ms (local)
Remote edit visibility 500ms delay 500ms delay (unavoidable)
Conflict resolution Server-side only Client-side pre-transform
Perceived responsiveness Poor Excellent

10. Failure Modes and Mitigations

10.1 Failure Mode Matrix

Failure Mode Probability Impact Detection Mitigation RTO RPO
OT Engine Crash Medium High Health check timeout Auto-restart + failover to replica 30s 0 (ops in Redis)
Redis Cluster Failure Low Critical Connection errors Multi-AZ Redis, async replication 1min 1s
S3 Outage Very Low High API errors Multi-region replication, local cache 5min 0
WebSocket Gateway Down Medium Medium Connection drops Auto-reconnect, fallback to polling 10s 0
Database Corruption Very Low Critical Checksum mismatch Point-in-time recovery, backups 15min 5min
Network Partition Low High Latency spike Split-brain protocol, quorum 30s 0
DDoS Attack Medium High Traffic anomaly Rate limiting, WAF, auto-scale 1min 0
OT Transform Bug Low Critical Divergence detection Rollback to last known good version 1min 1 op
Memory Leak (OT Engine) Medium Medium Memory usage alert Auto-restart, memory limits 30s 0
Clock Skew Low Medium Timestamp anomaly Use logical clocks (Lamport) N/A N/A

10.2 Detailed Mitigation Strategies

OT Engine Failure

Problem: OT engine process crashes while processing operations

Mitigation:
├── Primary: Replica OT engine in same AZ
│   └── Hot standby with Redis op log as shared state
│   └── Failover in < 30 seconds
│
├── Secondary: Multi-AZ deployment
│   └── Active-passive across availability zones
│   └── RTO: 1 minute
│
├── Data protection:
│   └── All ops persisted to Redis before ack
│   └── Redis has AOF + RDB persistence
│   └── RPO: 0 (no op loss)
│
└── Client handling:
    └── WebSocket disconnect → auto-reconnect
    └── On reconnect: sync from last acked version
    └── Pending ops re-sent with idempotency check

Network Partition (Split-Brain)

graph TD
    subgraph Before["Before Partition"]
        S1[Server A] <-->|Sync| S2[Server B]
        C1[Client 1] --> S1
        C2[Client 2] --> S2
    end

    subgraph During["During Partition"]
        S1_INDEPENDENT[Server A<br/>Independent]
        S2_INDEPENDENT[Server B<br/>Independent]
        C1 --> S1_INDEPENDENT
        C2 --> S2_INDEPENDENT
        Note1[Both serve edits independently]
    end

    subgraph After["After Healing"]
        MERGE[Merge Engine]
        S1_INDEPENDENT -->|Ops during partition| MERGE
        S2_INDEPENDENT -->|Ops during partition| MERGE
        MERGE -->|Transform + Merge| MERGED[Merged State]
        MERGED -->|Sync| S1_HEALED[Server A]
        MERGED -->|Sync| S2_HEALED[Server B]
    end

Split-Brain Resolution Protocol:

  1. Detection: Heartbeat between servers fails after 3 consecutive misses
  2. Independent operation: Both sides continue serving edits independently
  3. Partition healing: When connectivity restored:
    • Both sides exchange operation logs during partition window
    • OT transform resolves all conflicts
    • Merge produces single consistent state
    • Both sides sync to merged state
  4. Client notification: Clients receive merged operations and converge

OT Transform Bug (Divergence Detection)

Problem: A bug in the OT transform function causes clients to diverge

Detection:
├── Checksum verification:
│   └── Server maintains document checksum (SHA-256)
│   └── Clients periodically send their checksum
│   └── Mismatch → divergence detected
│
├── Version reconciliation:
│   └── Server tracks expected version per client
│   └── Client reports its current version
│   └── Gap > threshold → potential divergence
│
└── Statistical monitoring:
    └── Track transform success rate
    └── Alert on >0.1% transform failures

Recovery:
├── Immediate: Force sync from server authoritative state
├── Client-side: Discard local state, re-download document
├── Server-side: Rollback to last verified checkpoint
└── Long-term: Deploy fix, replay ops from checkpoint

10.3 Disaster Recovery

DR Strategy:
├── Backup frequency: Every 5 minutes (incremental)
├── Backup retention: 7 years (compliance)
├── Backup location: Multi-region (primary + 2 secondary)
├── RTO: 15 minutes (full system recovery)
└── RPO: 5 minutes (max data loss)

DR Test Schedule:
├── Monthly: Automated failover test (non-production)
├── Quarterly: Full DR drill (production, off-hours)
└── Annually: Tabletop exercise with leadership

10.4 Monitoring and Alerting

Key Metrics:
├── Latency
│   ├── Edit propagation P50/P95/P99
│   ├── Document load time
│   └── WebSocket connection time
│
├── Throughput
│   ├── Operations per second (global + per shard)
│   ├── Active concurrent editors
│   └── WebSocket connections count
│
├── Reliability
│   ├── OT transform success rate
│   ├── Divergence detection count
│   ├── Failed operations count
│   └── Client reconnect rate
│
├── Storage
│   ├── Document count
│   ├── Storage usage (by type)
│   └── Version history size
│
└── Business
    ├── Active documents (24h)
    ├── New documents created
    └── Export count

Alert Thresholds:
├── Critical: P99 latency > 500ms for 5 min
├── Critical: OT transform failure rate > 1%
├── Critical: Divergence detected
├── Warning: P95 latency > 300ms for 10 min
├── Warning: WebSocket reconnect rate > 10%
└── Warning: Storage usage > 80% capacity

Appendix A: Glossary

Term Definition
OT Operational Transformation — algorithm for reconciling concurrent edits
CRDT Conflict-free Replicated Data Type — data structure for distributed collaboration
Convergence Property that all replicas reach the same state given the same operations
Intention Preservation Property that a user’s intended edit is preserved after transformation
Presence Real-time awareness of other users’ cursor positions and activities
Sharding Horizontal partitioning of data across multiple servers
Consistent Hashing Hashing algorithm that minimizes data movement on cluster changes
RTO Recovery Time Objective — max acceptable downtime
RPO Recovery Point Objective — max acceptable data loss

Appendix B: References


Document Version: 1.0 | Last Updated: 2026-08-28