Design a Documentation System like Google Docs
Author: System Design Expert | Date: 2026-08-28 Scope: Real-time collaborative document editing at Google scale
Table of Contents
- Requirements Clarification
- Back-of-Envelope Calculations
- System Interface Design
- High-Level Architecture
- Core Component Design
- Data Model
- Key Design Decisions
- Scalability Strategy
- Curveball Scenarios
- Failure Modes and Mitigations
1. Requirements Clarification
1.1 Functional Requirements
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | Real-time collaborative editing — Multiple users edit the same document simultaneously with changes visible within ~200ms | Must Have |
| FR-02 | Cursor presence — See other users’ cursors, selections, and avatars in real-time | Must Have |
| FR-03 | Conflict resolution — Concurrent edits from multiple users must be merged correctly without data loss | Must Have |
| FR-04 | Version history — Automatic save with full version history; users can view, compare, and restore any version | Must Have |
| FR-05 | Offline editing — Users can edit documents offline; changes sync when connection is restored | Must Have |
| FR-06 | Document permissions — Granular access control (viewer, commenter, editor, owner) with shareable links | Must Have |
| FR-07 | Comments and suggestions — Inline comments, suggestion mode, comment threads with resolution | Must Have |
| FR-08 | Rich text formatting — Bold, italic, headings, lists, tables, images, code blocks, equations | Must Have |
| FR-09 | Document search — Full-text search within a document and across all documents | Should Have |
| FR-10 | Export/Import — Export to PDF, DOCX, HTML, ODT; import from DOCX, ODT, HTML | Should Have |
| FR-11 | Templates — Pre-built document templates for common use cases | Should Have |
| FR-12 | Revision comparison — Side-by-side diff view between any two versions | Should Have |
| FR-13 | Activity timeline — See who did what and when (edit history timeline) | Could Have |
| FR-14 | Integration — Embed charts, draw diagrams, link to other services | Could Have |
1.2 Non-Functional Requirements
| ID | Requirement | Target |
|---|---|---|
| NFR-01 | Edit latency | P99 < 200ms for edit propagation to co-editers |
| NFR-02 | Availability | 99.95% uptime (26 min downtime/month) |
| NFR-03 | Durability | 99.999999999% (11 nines) — same as S3 |
| NFR-04 | Concurrency | Support 100+ concurrent editors per document |
| NFR-05 | Document size | Support documents up to 100MB (text + media) |
| NFR-06 | Scalability | 1B+ documents, 100M+ monthly active users |
| NFR-07 | Consistency | Strong consistency for document state within a session; eventual consistency across replicas |
| NFR-08 | Security | AES-256 encryption at rest, TLS 1.3 in transit, SOC 2 compliance |
| NFR-09 | Data residency | Support geo-fencing for enterprise customers |
| NFR-10 | Auto-save | Changes persisted within 1 second of edit |
2. Back-of-Envelope Calculations
2.1 Assumption Sets
We reason from first principles with three assumption sets to bound our estimates:
| Parameter | Conservative | Base Case | Aggressive |
|---|---|---|---|
| Monthly Active Users (MAU) | 50M | 100M | 500M |
| Documents per user (avg) | 20 | 50 | 100 |
| Total documents | 1B | 5B | 50B |
| Avg document size (text) | 50 KB | 100 KB | 200 KB |
| Avg document size (with media) | 200 KB | 500 KB | 1 MB |
| Edits per active session | 500 | 1,000 | 5,000 |
| Avg session duration | 15 min | 30 min | 60 min |
| Concurrent editors per doc | 3 | 5 | 10 |
| Edit operation size | 50 bytes | 100 bytes | 200 bytes |
2.2 Storage Estimates
Base Case Calculation:
├── Raw document storage
│ ├── Text content: 5B docs × 100 KB = 500 TB
│ └── Media attachments: 5B docs × 400 KB = 2 PB
│ └── Subtotal: 2.5 PB
│
├── Version history (keep 10 versions avg)
│ └── 2.5 PB × 10 = 25 PB
│
├── Metadata (permissions, comments, etc.)
│ └── 5B docs × 10 KB = 50 TB
│
├── Index (search, full-text)
│ └── 2.5 PB × 2x = 5 PB
│
└── Total with 3x replication: ~90 PB
2.3 Bandwidth Estimates
Base Case — Edit Traffic:
├── Daily active users: 100M MAU × 20% = 20M DAU
├── Sessions per day: 20M × 3 sessions = 60M sessions
├── Edit operations/day: 60M × 1,000 = 60B operations
├── Edit payload/day: 60B × 100 bytes = 6 TB/day
├── Avg QPS (edits): 60B / 86,400s ≈ 694K QPS
├── Peak QPS (3x avg): ~2M QPS
│
└── Document Load Traffic:
├── Doc loads/day: 60M sessions × 2 docs = 120M loads
├── Load payload/day: 120M × 500 KB = 60 TB/day
└── Peak load QPS: ~5K QPS (but high bandwidth burst)
2.4 Infrastructure Estimates
Edit Operation Servers:
├── Per server capacity: 5,000 ops/sec (WebSocket connections)
├── Servers needed: 2M QPS / 5K = 400 servers
└── With 3x redundancy: 1,200 servers
Document Storage Servers:
├── Per server capacity: 10 TB usable (3.6 TB × 3 drives)
├── Servers needed: 90 PB / 10 TB = 9,000 servers
└── With replication: Already factored in
Database Nodes:
├── Metadata DB: 50 TB → ~200 nodes (250 GB/node)
└── Search Index: 5 PB → ~500 nodes (10 TB/node)
3. System Interface Design
3.1 REST API (Document Management)
# Document CRUD
POST /api/v1/documents # Create document
GET /api/v1/documents/{docId} # Get document metadata
PUT /api/v1/documents/{docId} # Update metadata
DELETE /api/v1/documents/{docId} # Delete document
GET /api/v1/documents/{docId}/content # Get full document content
# Version History
GET /api/v1/documents/{docId}/versions # List versions
GET /api/v1/documents/{docId}/versions/{versionId} # Get specific version
POST /api/v1/documents/{docId}/versions/{versionId}/restore # Restore version
GET /api/v1/documents/{docId}/versions/{v1}/diff/{v2} # Compare versions
# Permissions
GET /api/v1/documents/{docId}/permissions # List permissions
POST /api/v1/documents/{docId}/permissions # Grant permission
DELETE /api/v1/documents/{docId}/permissions/{userId} # Revoke permission
# Comments
GET /api/v1/documents/{docId}/comments # List comments
POST /api/v1/documents/{docId}/comments # Add comment
PUT /api/v1/documents/{docId}/comments/{commentId} # Update comment
DELETE /api/v1/documents/{docId}/comments/{commentId} # Delete comment
# Export/Import
POST /api/v1/documents/{docId}/export?format=pdf # Export
POST /api/v1/documents/import # Import
# Search
GET /api/v1/search?q={query}&scope={document|global} # Search
3.2 WebSocket Protocol (Real-time Collaboration)
# Connection
WS CONNECT wss://docs.example.com/ws/{docId}?token={authToken}
# Client → Server Messages
{
"type": "EDIT",
"payload": {
"operation": { "type": "insert", "position": 42, "text": "Hello" },
"version": 1507,
"clientId": "client-abc-123",
"timestamp": 1724812345678
}
}
{
"type": "CURSOR_UPDATE",
"payload": {
"clientId": "client-abc-123",
"position": { "start": 42, "end": 47 },
"scrollY": 350
}
}
{
"type": "PRESENCE_JOIN",
"payload": {
"clientId": "client-abc-123",
"userId": "user-456",
"userName": "Alice",
"avatarUrl": "https://...",
"color": "#FF5722"
}
}
{
"type": "PRESENCE_LEAVE",
"payload": { "clientId": "client-abc-123" }
}
{
"type": "COMMENT_ADD",
"payload": {
"range": { "start": 100, "end": 110 },
"text": "Please review this section",
"timestamp": 1724812345678
}
}
# Server → Client Messages
{
"type": "EDIT_ACK",
"payload": {
"version": 1508,
"clientId": "client-abc-123"
}
}
{
"type": "REMOTE_EDIT",
"payload": {
"operation": { "type": "delete", "position": 50, "length": 5 },
"version": 1508,
"clientId": "client-def-456",
"userId": "user-789"
}
}
{
"type": "PRESENCE_UPDATE",
"payload": {
"clients": [
{ "clientId": "client-def-456", "userId": "user-789",
"userName": "Bob", "cursor": { "start": 50, "end": 55 }, "color": "#2196F3" }
]
}
}
{
"type": "STATE_SYNC",
"payload": {
"documentContent": "...",
"version": 1508,
"operations": [ /* ops since client's last version */ ]
}
}
{
"type": "CONFLICT_RESOLUTION",
"payload": {
"conflictingOps": [ /* ops that need re-base */ ],
"baseVersion": 1505
}
}
3.3 gRPC Internal Services
service DocumentService {
rpc GetDocument(GetDocumentRequest) returns (Document);
rpc SaveDocument(SaveDocumentRequest) returns (SaveResponse);
rpc ApplyOperation(ApplyOperationRequest) returns (OperationResult);
rpc GetVersionHistory(GetVersionHistoryRequest) returns (VersionList);
}
service PresenceService {
rpc UpdatePresence(PresenceUpdateRequest) returns (PresenceState);
rpc GetActiveUsers(GetActiveUsersRequest) returns (UserList);
rpc BroadcastPresence(BroadcastRequest) returns (stream PresenceUpdate);
}
service PermissionService {
rpc CheckPermission(CheckPermissionRequest) returns (PermissionResult);
rpc GrantPermission(GrantPermissionRequest) returns (PermissionResult);
}
4. High-Level Architecture
graph TB
subgraph Client["Client Layer"]
C1[User A - Browser]
C2[User B - Browser]
C3[User C - Mobile App]
end
subgraph Edge["Edge / CDN Layer"]
CDN[CDN / Static Assets]
WAF[WAF / DDoS Protection]
LB[Global Load Balancer]
end
subgraph Gateway["API Gateway Layer"]
GW[API Gateway]
WS_GW[WebSocket Gateway]
AUTH[Auth Service<br/>OAuth 2.0 / JWT]
end
subgraph Core["Core Services"]
subgraph Collab["Collaboration Services"]
OT[OT / CRDT Engine]
PRESENCE[Presence Service]
DOC_STATE[Document State Manager]
end
subgraph DocMgmt["Document Management"]
DOC_API[Document API Service]
VERSION[Version History Service]
EXPORT[Export Service]
SEARCH[Search Service]
end
subgraph Meta["Metadata Services"]
PERM[Permission Service]
COMMENT[Comment Service]
NOTIF[Notification Service]
end
end
subgraph Storage["Storage Layer"]
REDIS[(Redis Cluster<br/>Op Log / Presence)]
DYNAMO[(DynamoDB / Spanner<br/>Metadata)]
S3[(S3 / GCS<br/>Document Content)]
ES[(Elasticsearch<br/>Full-Text Index)]
VERSION_S3[(S3 / GCS<br/>Version Snapshots)]
end
subgraph Infra["Infrastructure"]
MQ[Kafka / Pub-Sub<br/>Event Bus]
MONITOR[Monitoring & Alerting]
CACHE[Memcached / Redis<br/>Read Cache]
end
C1 -->|WebSocket| WAF
C2 -->|WebSocket| WAF
C3 -->|HTTPS| WAF
WAF --> LB
LB --> GW
LB --> WS_GW
GW --> AUTH
GW --> DOC_API
GW --> VERSION
GW --> EXPORT
GW --> SEARCH
GW --> PERM
GW --> COMMENT
WS_GW --> AUTH
WS_GW --> OT
WS_GW --> PRESENCE
OT --> DOC_STATE
OT --> REDIS
DOC_STATE --> S3
DOC_STATE --> DYNAMO
PRESENCE --> REDIS
DOC_API --> DYNAMO
DOC_API --> S3
DOC_API --> PERM
VERSION --> VERSION_S3
VERSION --> DYNAMO
SEARCH --> ES
DOC_STATE -->|Events| MQ
COMMENT -->|Events| MQ
MQ --> NOTIF
MQ --> ES
MQ --> VERSION
DOC_API --> CACHE
SEARCH --> CACHE
5. Core Component Design
5.1 Operational Transformation (OT) Engine
The OT engine is the heart of real-time collaboration. It ensures that concurrent edits from multiple users are correctly merged without conflicts.
How OT Works
User A: "Hello World"
User B: "Hello World"
Time T1: User A inserts " Beautiful" at position 6
→ "Hello Beautiful World"
Time T2: User B deletes "World" at position 6 (length 5)
→ "Hello "
Without OT: Both operations conflict!
With OT: Operation B is transformed against Operation A:
- B's delete was at position 6, length 5 ("World")
- A inserted 11 chars at position 6
- Transform B: delete now at position 6+11=17, length 5
- Apply A first: "Hello Beautiful World"
- Apply transformed B: "Hello Beautiful "
✓ Both users see consistent result
OT Algorithm
function Transform(opA, opB):
"""
Transform opA against opB.
Returns (opA', opB') such that:
Apply(Apply(doc, opA), opB') == Apply(Apply(doc, opB), opA')
"""
opA_prime = copy(opA)
opB_prime = copy(opB)
if opA.type == INSERT and opB.type == INSERT:
if opA.position < opB.position or
(opA.position == opB.position and opA.clientId < opB.clientId):
opB_prime.position += len(opA.text)
else:
opA_prime.position += len(opB.text)
elif opA.type == INSERT and opB.type == DELETE:
if opA.position > opB.position:
opA_prime.position -= opB.length
if opA.position <= opB.position + opB.length:
opA_prime.text = opA.text[opA.position - opB.position:]
opA_prime.position = opB.position
elif opA.type == DELETE and opB.type == INSERT:
if opB.position > opA.position:
opB_prime.position -= opA.length
elif opA.type == DELETE and opB.type == DELETE:
# Handle overlapping deletes
if opA.position < opB.position:
if opA.position + opA.length > opB.position:
overlap = opA.position + opA.length - opB.position
opB_prime.length -= overlap
opB_prime.position = opA.position
else:
opB_prime.position -= opA.length
# ... symmetric cases
return (opA_prime, opB_prime)
OT Server Architecture
graph LR
subgraph OTServer["OT Server (per document)"]
OPLOG[Operation Log<br/>Ordered Op Queue]
TRANSFORM[Transform Engine]
BROADCAST[Broadcast Manager]
STATE[Document State<br/>In-Memory]
end
C1[Client 1] -->|Op v5| TRANSFORM
C2[Client 2] -->|Op v5| TRANSFORM
C3[Client 3] -->|Op v6| TRANSFORM
TRANSFORM -->|Transform against ops v5-v7| OPLOG
OPLOG -->|Ack v8| TRANSFORM
TRANSFORM -->|Broadcast transformed ops| BROADCAST
BROADCAST -->|Remote Op v8| C1
BROADCAST -->|Remote Op v8| C2
BROADCAST -->|Remote Op v8| C3
STATE -->|Periodic snapshot| S3[(S3)]
OPLOG -->|Async persist| REDIS[(Redis)]
Key OT Properties
| Property | Description | Implementation |
|---|---|---|
| Convergence | All clients reach the same document state | Guaranteed by OT transform function correctness |
| Intention Preservation | User’s intended edit is preserved | Transform adjusts position, not intent |
| Causality | Operations applied in causal order | Version vectors + server-side ordering |
| Idempotency | Re-applying same op has no effect | Client ID + sequence number dedup |
5.2 CRDT Alternative (Yjs / Automerge)
While OT is used by Google Docs, CRDTs offer an alternative approach:
graph TD
subgraph CRDT["CRDT Approach (Yjs)"]
subgraph YText["Y.Text Data Structure"]
ITEM1[Item: id=(1,A) 'H']
ITEM2[Item: id=(2,A) 'e']
ITEM3[Item: id=(3,B) 'l']
ITEM4[Item: id=(4,A) 'l']
ITEM5[Item: id=(5,B) 'o']
end
subgraph YState["State Vector"]
SV["{A: 4, B: 5}"]
end
subgraph YUpdate["Update Mechanism"]
INC[Incremental Update]
MERGE[State Merge]
end
end
ITEM1 --> ITEM2 --> ITEM3 --> ITEM4 --> ITEM5
SV --> MERGE
INC --> MERGE
OT vs CRDT Comparison
| Aspect | OT | CRDT (Yjs) |
|---|---|---|
| Central authority | Required (server orders ops) | Not required (peer-to-peer) |
| Offline support | Limited (needs sync) | Excellent (merge on reconnect) |
| Complexity | High (transform matrix) | Medium (data structure complexity) |
| Storage overhead | Low (ops are small) | Higher (metadata per character) |
| Undo/Redo | Complex | Built-in support |
| Network efficiency | Better (smaller ops) | Worse (more metadata) |
| Production maturity | Proven (Google Docs, Etherpad) | Growing (Codemirror, Tldraw) |
5.3 Presence Service
Tracks real-time cursor positions, selections, and user information.
classDiagram
class PresenceService {
+Map~docId, DocumentPresence~ documents
+Map~clientId, ClientState~ clients
+join(docId, clientId, userInfo)
+leave(clientId)
+updateCursor(clientId, position)
+getActiveUsers(docId)
+broadcastPresence(docId, updates)
+cleanupStaleClients(timeout)
}
class DocumentPresence {
+String docId
+Set~ClientState~ activeClients
+Map~clientId, ClientState~ clientMap
+lastUpdated: Timestamp
}
class ClientState {
+String clientId
+String userId
+String userName
+String avatarUrl
+String color
+CursorPos cursor
+SelectionRange selection
+int scrollY
+Timestamp lastHeartbeat
}
class CursorPos {
+int start
+int end
+Timestamp updatedAt
}
PresenceService --> DocumentPresence
DocumentPresence --> ClientState
ClientState --> CursorPos
Presence Update Flow
1. Client moves cursor
2. Client sends CURSOR_UPDATE over WebSocket (throttled to 50ms)
3. Presence Service updates Redis: HSET presence:{docId} {clientId} '{cursor}'
4. Presence Service broadcasts to all connected clients for that doc
5. Each client renders remote cursors with user's color and avatar
6. If no heartbeat for 30s, client marked as inactive
7. If no heartbeat for 60s, client removed from presence
5.4 Document State Manager
Manages the authoritative document state and persists changes.
sequenceDiagram
participant C as Client
participant WS as WebSocket Gateway
participant OT as OT Engine
participant DSM as Doc State Manager
participant REDIS as Redis (Op Log)
participant S3 as Object Storage
participant DB as Metadata DB
C->>WS: EDIT operation
WS->>OT: Apply operation
OT->>OT: Transform against pending ops
OT->>DSM: Apply transformed op
DSM->>REDIS: Append op to op log
DSM->>OT: Ack with new version
OT->>WS: Broadcast to other clients
WS->>C: Remote edit notification
Note over DSM,S3: Async persistence
DSM->>DSM: Batch ops (1s window)
DSM->>S3: Save document snapshot
DSM->>DB: Update metadata
5.5 Version History Service
graph TD
subgraph VersionService["Version History Service"]
AUTO[Auto-Save Trigger<br/>Every 60s or 50 ops]
MANUAL[Manual Save Trigger<br/>User-initiated]
SNAPSHOT[Snapshot Creator]
COMPRESS[Delta Compressor]
STORE[Version Store]
RESTORE[Version Restorer]
DIFF[Diff Engine]
end
AUTO --> SNAPSHOT
MANUAL --> SNAPSHOT
SNAPSHOT --> COMPRESS
COMPRESS --> STORE
STORE -->|S3 Object| S3BUCKET[(S3: versions/{docId}/{versionId})]
RESTORE --> STORE
RESTORE -->|Reconstruct| DOC[Document Content]
DIFF -->|Compare v1 vs v2| DIFFRESULT[Diff Result]
subgraph Retention["Retention Policy"]
KEEP[Keep last 100 versions]
ARCHIVE[Archive older versions]
DELETE[Delete after 7 years]
end
STORE --> KEEP
KEEP --> ARCHIVE
ARCHIVE --> DELETE
Version Storage Strategy
| Version Type | Frequency | Retention | Storage |
|---|---|---|---|
| Auto-save | Every 60s or 50 ops | Last 100 versions | Delta-encoded in S3 |
| Named versions | User-created | Permanent | Full snapshot in S3 |
| Milestone | Daily (auto) | 1 year | Full snapshot in S3 |
| Archive | Older than retention | 7 years | Cold storage (Glacier) |
5.6 Permission Service
graph TD
subgraph PermissionModel["Permission Model"]
OWNER[Owner<br/>Full control + transfer]
EDITOR[Editor<br/>Edit + comment]
COMMENTER[Commenter<br/>View + comment]
VIEWER[Viewer<br/>Read-only]
end
subgraph PermissionTypes["Permission Types"]
USER[User-specific<br/>email/userId]
GROUP[Group-based<br/>org/team]
LINK[Link-based<br/>anyone with link]
DOMAIN[Domain-based<br/>@company.com]
end
subgraph PermissionCheck["Permission Check Flow"]
REQ[Request] --> LOOKUP[Lookup ACL]
LOOKUP --> CACHE{In Cache?}
CACHE -->|Yes| RETURN[Return Result]
CACHE -->|No| DB[Query Permission DB]
DB --> EVAL[Evaluate Hierarchy]
EVAL -->|Owner > Editor > Commenter > Viewer| RETURN
RETURN --> UPDATE_CACHE[Update Cache]
end
OWNER --> EDITOR --> COMMENTER --> VIEWER
5.7 Offline Editing Support
sequenceDiagram
participant C as Client
participant LOCAL as Local Storage<br/>IndexedDB
participant NET as Network Layer
participant S as Server
Note over C,S: Offline Phase
C->>LOCAL: Edit operation (queued)
C->>LOCAL: Edit operation (queued)
C->>LOCAL: Edit operation (queued)
Note over LOCAL: Ops stored with local timestamps
Note over C,S: Reconnection Phase
NET->>S: Check server version
S->>NET: Server version = 1508
NET->>C: Server state since v1500
C->>C: Merge local ops (v1501-v1505) with server ops
C->>C: Apply OT transform for conflicts
C->>S: Send reconciled ops
S->>C: Ack with new version
C->>LOCAL: Clear synced ops
Offline Storage Design
IndexedDB Schema:
├── pending_ops
│ ├── docId: "doc-123"
│ ├── ops: [op1, op2, op3, ...]
│ ├── baseVersion: 1500
│ └── createdAt: timestamp
│
├── document_cache
│ ├── docId: "doc-123"
│ ├── content: "..."
│ ├── version: 1500
│ └── cachedAt: timestamp
│
└── sync_status
├── docId: "doc-123"
├── status: "pending" | "syncing" | "synced" | "conflict"
└── lastSync: timestamp
5.8 Search Service
graph TD
subgraph SearchPipeline["Search Pipeline"]
DOC[Document Content] --> TOKENIZE[Tokenizer]
TOKENIZE --> STEM[Stemmer / Lemmatizer]
STEM --> FILTER[Stop Word Filter]
FILTER --> INDEX[Inverted Index]
end
subgraph Elasticsearch["Elasticsearch Cluster"]
INDEX --> SHARD1[Shard 1]
INDEX --> SHARD2[Shard 2]
INDEX --> SHARD3[Shard 3]
end
subgraph SearchQuery["Search Query"]
Q[User Query] --> PARSE[Query Parser]
PARSE --> SCORE[Scoring Engine]
SCORE -->|BM25 Algorithm| RANK[Rank Results]
RANK --> FILTER_PERM[Permission Filter]
FILTER_PERM --> RESULTS[Search Results]
end
SHARD1 --> SCORE
SHARD2 --> SCORE
SHARD3 --> SCORE
6. Data Model
6.1 Relational Schema (Metadata)
-- Documents table
CREATE TABLE documents (
doc_id UUID PRIMARY KEY,
title VARCHAR(500) NOT NULL,
owner_id UUID NOT NULL REFERENCES users(user_id),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
version BIGINT NOT NULL DEFAULT 0,
status VARCHAR(20) DEFAULT 'active', -- active, trashed, archived
size_bytes BIGINT DEFAULT 0,
content_type VARCHAR(50) DEFAULT 'document',
space_id UUID REFERENCES spaces(space_id),
metadata JSONB DEFAULT '{}',
CONSTRAINT valid_status CHECK (status IN ('active', 'trashed', 'archived'))
);
-- Users table
CREATE TABLE users (
user_id UUID PRIMARY KEY,
email VARCHAR(255) UNIQUE NOT NULL,
display_name VARCHAR(200),
avatar_url TEXT,
org_id UUID REFERENCES organizations(org_id),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
is_active BOOLEAN DEFAULT true
);
-- Document permissions (ACL)
CREATE TABLE document_permissions (
permission_id UUID PRIMARY KEY,
doc_id UUID NOT NULL REFERENCES documents(doc_id) ON DELETE CASCADE,
grantee_type VARCHAR(20) NOT NULL, -- user, group, domain, link
grantee_id VARCHAR(500) NOT NULL,
role VARCHAR(20) NOT NULL, -- owner, editor, commenter, viewer
granted_by UUID REFERENCES users(user_id),
granted_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
link_token VARCHAR(64), -- For link-based sharing
expires_at TIMESTAMP WITH TIME ZONE,
UNIQUE(doc_id, grantee_type, grantee_id),
CONSTRAINT valid_role CHECK (role IN ('owner', 'editor', 'commenter', 'viewer'))
);
-- Operation log (for OT)
CREATE TABLE operation_log (
op_id BIGSERIAL PRIMARY KEY,
doc_id UUID NOT NULL REFERENCES documents(doc_id),
client_id VARCHAR(64) NOT NULL,
user_id UUID REFERENCES users(user_id),
operation JSONB NOT NULL, -- {type, position, text, ...}
version BIGINT NOT NULL,
base_version BIGINT NOT NULL,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
INDEX idx_op_doc_version (doc_id, version)
);
-- Document versions
CREATE TABLE document_versions (
version_id UUID PRIMARY KEY,
doc_id UUID NOT NULL REFERENCES documents(doc_id),
version_number BIGINT NOT NULL,
snapshot_key VARCHAR(500), -- S3 key for snapshot
delta_key VARCHAR(500), -- S3 key for delta
created_by UUID REFERENCES users(user_id),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
is_named BOOLEAN DEFAULT false,
name VARCHAR(200),
size_bytes BIGINT,
UNIQUE(doc_id, version_number)
);
-- Comments
CREATE TABLE comments (
comment_id UUID PRIMARY KEY,
doc_id UUID NOT NULL REFERENCES documents(doc_id),
version BIGINT NOT NULL,
user_id UUID REFERENCES users(user_id),
range_start INTEGER,
range_end INTEGER,
text TEXT NOT NULL,
parent_id UUID REFERENCES comments(comment_id),
status VARCHAR(20) DEFAULT 'open', -- open, resolved
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
-- Indexes
CREATE INDEX idx_docs_owner ON documents(owner_id);
CREATE INDEX idx_docs_space ON documents(space_id);
CREATE INDEX idx_docs_updated ON documents(updated_at DESC);
CREATE INDEX idx_perms_doc ON document_permissions(doc_id);
CREATE INDEX idx_perms_grantee ON document_permissions(grantee_type, grantee_id);
CREATE INDEX idx_versions_doc ON document_versions(doc_id, version_number DESC);
CREATE INDEX idx_comments_doc ON comments(doc_id, version);
CREATE INDEX idx_comments_doc ON comments(doc_id, version);
CREATE INDEX idx_comments_status ON comments(doc_id, status);
6.2 NoSQL Schema (Redis - Presence & Op Log)
# Presence data (TTL: 60s, refreshed by heartbeat)
HSET presence:doc:{docId}
client:{clientId} '{"userId":"u1","cursor":{"start":42,"end":47},"color":"#FF5722"}'
EXPIRE presence:doc:{docId} 60
# Operation log (sorted set, version as score)
ZADD oplog:doc:{docId}
{version} '{"clientId":"c1","op":{"type":"insert","pos":42,"text":"Hi"},"ts":1724812345}'
# Document state cache
SET doc:state:{docId} '{"version":1508,"content":"...","checksum":"abc123"}' EX 300
# Active editors count
INCR active:doc:{docId}
EXPIRE active:doc:{docId} 120
6.3 Object Storage Layout (S3)
s3://docs-storage/
├── documents/
│ └── {docId}/
│ ├── content.json # Current document content
│ ├── metadata.json # Document metadata
│ └── media/
│ ├── image-001.png
│ └── image-002.jpg
│
├── versions/
│ └── {docId}/
│ ├── {versionId}-snapshot.json # Full snapshot
│ └── {versionId}-delta.json # Delta from previous
│
├── exports/
│ └── {docId}/
│ ├── {timestamp}-export.pdf
│ └── {timestamp}-export.docx
│
└── templates/
└── {templateId}/
└── template.json
7. Key Design Decisions
7.1 OT vs CRDT
| Factor | Decision: OT | Rationale |
|---|---|---|
| Network model | Client-server (centralized) | Simpler conflict resolution, single source of truth |
| Storage efficiency | Lower overhead | Operations are small (50-100 bytes) vs CRDT metadata |
| Maturity | Battle-tested | Google Docs, Etherpad, Quill use OT successfully |
| Offline support | Acceptable with sync queue | CRDT better for offline, but OT sufficient for our use case |
| Undo/Redo | Requires extra logic | CRDT has built-in undo, but OT can implement with op stacking |
Decision: OT with server-side coordination — Best fit for Google Docs’ centralized model with strong consistency requirements.
7.2 WebSocket vs HTTP Long-Polling
| Factor | WebSocket | HTTP Long-Polling |
|---|---|---|
| Latency | ~10-50ms | ~200-500ms |
| Bandwidth | Lower (single connection) | Higher (headers per request) |
| Server load | Higher (persistent connections) | Lower (stateless) |
| Firewall compatibility | Sometimes blocked | Always works |
| Scalability | Connection-heavy | Request-heavy |
Decision: WebSocket primary, HTTP long-polling fallback — WebSocket for real-time editing, fallback for restrictive networks.
7.3 Document Storage Format
| Format | Pros | Cons |
|---|---|---|
| JSON | Human-readable, easy to parse | Larger size, no schema enforcement |
| Protocol Buffers | Compact, fast, schema-enforced | Binary, not human-readable |
| Custom binary | Most compact | Hardest to debug, no tooling |
Decision: JSON for content, Protocol Buffers for operations — JSON for readability and debugging, Protobuf for high-frequency operation transport.
7.4 Version History Strategy
| Strategy | Storage | Restore Speed | Use Case |
|---|---|---|---|
| Full snapshots | High | Fast | Named versions, milestones |
| Delta-only | Low | Slow (replay all ops) | Auto-save versions |
| Hybrid | Medium | Fast | Best of both worlds |
Decision: Hybrid approach — Full snapshot every 100 auto-saves + delta between saves. Named versions always full snapshots.
7.5 Sharding Strategy
Sharding Key: doc_id (UUID)
Why doc_id?
├── Even distribution (UUID is random)
├── All operations for a doc go to the same shard
├── No cross-shard transactions needed
└── Easy to add/remove shards
Shard Assignment:
├── Consistent hashing with virtual nodes
├── 1024 virtual nodes per physical shard
└── Minimize data movement on rebalance
7.6 Consistency Model
| Operation | Consistency Level | Rationale |
|---|---|---|
| Edit operations | Strong (within session) | Co-editors must see consistent state |
| Document metadata | Eventual | Title/permissions can lag slightly |
| Search index | Eventual | Search can be seconds behind |
| Version history | Strong | Must not lose any version |
| Presence | Eventual | Cursor position can be slightly stale |
8. Scalability Strategy
8.1 Horizontal Scaling
graph TD
subgraph Layer1["Layer 1: Stateless Services (Auto-scale)"]
API[API Gateway Instances]
WS[WebSocket Gateway Instances]
AUTH[Auth Service Instances]
end
subgraph Layer2["Layer 2: Sharded Services"]
OT1[OT Engine Shard 1]
OT2[OT Engine Shard 2]
OT3[OT Engine Shard N]
end
subgraph Layer3["Layer 3: Distributed Storage"]
REDIS1[Redis Cluster Shard 1]
REDIS2[Redis Cluster Shard 2]
S3[S3 / GCS (infinite scale)]
ES1[ES Shard 1]
ES2[ES Shard 2]
end
API --> OT1
API --> OT2
API --> OT3
WS --> OT1
WS --> OT2
WS --> OT3
OT1 --> REDIS1
OT2 --> REDIS2
OT3 --> S3
OT1 --> ES1
OT2 --> ES2
8.2 Scaling Dimensions
| Dimension | Strategy | Trigger |
|---|---|---|
| WebSocket connections | Add WS gateway instances | Connection count > 80% capacity |
| OT engines | Add shards via consistent hashing | Ops/sec per shard > 10K |
| Storage | S3 auto-scales | No action needed |
| Redis | Add cluster nodes | Memory > 70% or connections > 80% |
| Search | Add ES nodes/shards | Index size > 1TB per node |
| Database | Read replicas + sharding | Read QPS > 10K or write latency > 100ms |
8.3 Caching Strategy
Cache Hierarchy:
├── L1: Browser Cache (Service Worker)
│ └── Document content, static assets
│ └── TTL: Session-based, invalidated on edit
│
├── L2: CDN Cache
│ └── Static assets, exported PDFs
│ └── TTL: 24h for static, 1h for exports
│
├── L3: Application Cache (Redis)
│ └── Document metadata, permissions
│ └── TTL: 5 min, invalidated on change
│
└── L4: Database Read Replicas
└── Read-heavy queries
└── Replication lag: < 1s
8.4 Database Sharding
Sharding Strategy:
├── Primary key: doc_id (UUID)
├── Shard count: 64 initial, scalable to 256
├── Shard key hash: CRC32(doc_id) % shard_count
└── Rebalancing: Online, via consistent hashing
Cross-shard operations:
├── Search: Aggregated across shards (Elasticsearch handles this)
├── User's documents: Materialized view in user_index table
└── Permissions: Denormalized, replicated to relevant shards
9. Curveball Scenarios
Curveball 1: “Two users simultaneously delete the same paragraph — what happens?”
Scenario: Alice and Bob both select and delete the same paragraph “Hello World” at positions 10-20.
Analysis:
Initial state: "The [Hello World] is here"
^--^ positions 10-20
Alice's op: DELETE position=10, length=11, version=100
Bob's op: DELETE position=10, length=11, version=100
Server receives Alice's op first:
├── Apply Alice's DELETE: "The is here"
├── Ack version 101 to Alice
└── Broadcast to Bob
Server receives Bob's op:
├── Transform Bob's DELETE against Alice's DELETE (v101):
│ ├── Both delete at same position, same length
│ ├── Bob's delete overlaps completely with Alice's
│ └── Transformed: DELETE position=10, length=0 (no-op)
├── Apply no-op: "The is here" (unchanged)
├── Ack version 102 to Bob
└── Result: Both see "The is here" ✓
Resolution: The OT transform function detects overlapping deletes and converts the second delete to a no-op. Both users see the same result — the paragraph is deleted once.
Key insight: OT’s transform matrix must handle all 9 combinations (insert/insert, insert/delete, insert/retain, delete/insert, delete/delete, delete/retain, retain/insert, retain/delete, retain/retain). The delete/delete case is critical for this scenario.
Curveball 2: “A user’s internet drops mid-edit. They reconnect 5 minutes later. How do we reconcile?”
Scenario: Alice edits a document for 5 minutes offline, making 200 changes. Meanwhile, Bob and Charlie made 150 changes online.
Reconciliation Process:
sequenceDiagram
participant A as Alice (Client)
participant S as Server
participant B as Bob (Client)
participant C as Charlie (Client)
Note over A,C: 5 minutes of divergence
A->>A: 200 offline ops (v1001-v1200)
B->>S: 100 ops (v1001-v1100)
C->>S: 50 ops (v1101-v1150)
Note over S: Server at version 1150
Note over A,S: Alice reconnects
A->>S: Sync request (last known v1000)
S->>A: Server ops v1001-v1150 (150 ops)
A->>A: Apply server ops to local doc
A->>A: Transform local ops against server ops
A->>S: Send transformed local ops (200 ops)
S->>S: Apply transformed ops (v1151-v1350)
S->>A: Ack v1350
S->>B: Broadcast new ops
S->>C: Broadcast new ops
Note over A,C: All converge to v1350
Detailed Steps:
- Detect reconnection: Client detects network restoration via WebSocket reconnect
- Fetch server state: Client requests all operations since its last known version (v1000)
- Apply server ops locally: Client applies the 150 server operations to its local document
- Transform local ops: Each of Alice’s 200 offline ops is transformed against the 150 server ops
- Send transformed ops: Alice sends her 200 transformed ops to the server
- Server applies: Server applies Alice’s ops, advancing from v1150 to v1350
- Broadcast: Server broadcasts these ops to Bob and Charlie
- Convergence: All three clients now have the same document at v1350
Conflict handling during transform:
- If Alice and Bob both edited the same text, OT preserves both intentions where possible
- If completely overlapping, server-order wins (first received)
- Alice’s client shows a conflict notification if her intention was significantly altered
Optimization: For large divergence (200 ops), use state-based sync instead of operation-based:
- Server sends full document state at v1150
- Client diffs its local state against server state
- Only send the delta operations
Curveball 3: “A document goes viral — 10,000 concurrent editors. How do we handle this?”
Scenario: A shared document (e.g., a company-wide town hall Q&A) suddenly gets 10,000 concurrent editors.
Problem Analysis:
Normal doc: 5 concurrent editors, ~50 ops/sec
Viral doc: 10,000 concurrent editors, ~100,000 ops/sec
Bottlenecks:
├── Single OT engine instance: Can't handle 100K ops/sec
├── WebSocket connections: 10K connections to one server
├── Broadcast fan-out: Each op sent to 10K clients = 1B messages/sec
└── Redis op log: Single key contention
Multi-Layer Solution:
graph TD
subgraph Normal["Normal Mode (≤100 editors)"]
OT1[Single OT Engine]
WS1[Single WS Server]
REDIS1[Single Redis Key]
end
subgraph Viral["Viral Mode (>100 editors)"]
subgraph Sharding["Document Sharding"]
SHARD_A[OT Shard A<br/>Editors 1-2500]
SHARD_B[OT Shard B<br/>Editors 2501-5000]
SHARD_C[OT Shard C<br/>Editors 5001-7500]
SHARD_D[OT Shard D<br/>Editors 7501-10000]
end
MERGER[Document Merger<br/>Merge shard states]
BROADCASTER[Broadcast Fan-out<br/>Pub-Sub]
subgraph WS["WebSocket Cluster"]
WS_A[WS Server A]
WS_B[WS Server B]
WS_C[WS Server C]
WS_D[WS Server D]
end
end
SHARD_A --> MERGER
SHARD_B --> MERGER
SHARD_C --> MERGER
SHARD_D --> MERGER
MERGER --> BROADCASTER
BROADCASTER --> WS_A
BROADCASTER --> WS_B
BROADCASTER --> WS_C
BROADCASTER --> WS_D
Detailed Strategy:
- Auto-detect viral mode: Monitor concurrent editor count; trigger at >100 editors
- Document sharding: Split the document into logical sections, each with its own OT engine
- Shard by document sections (paragraphs, pages)
- Each shard handles 2,500 editors and ~25K ops/sec
- Cross-shard coordination: A merger service periodically merges shard states
- Merge interval: 100ms
- Cross-shard ops transformed against merge state
- WebSocket clustering: Distribute connections across multiple WS servers
- Each WS server handles 2,500 connections
- Pub-Sub for cross-server broadcast
- Broadcast optimization:
- Throttle: Limit broadcast to 20 ops/sec per client (batch)
- Selective: Only send ops relevant to the client’s viewport
- Compress: Use delta encoding for broadcast messages
- Read-only fallback: If >50,000 editors, switch non-active users to read-only mode
- Graceful degradation:
- Increase sync interval from 100ms to 500ms
- Disable cursor presence for non-active users
- Queue non-critical operations
Capacity planning:
| Metric | Normal | Viral | Solution |
|---|---|---|---|
| Concurrent editors | 5 | 10,000 | Shard into 4 OT engines |
| Ops/sec | 50 | 100,000 | 25K per shard |
| WS connections | 5 | 10,000 | 4 WS servers × 2,500 |
| Broadcast msgs/sec | 250 | 1B | Throttle + batch + selective |
Curveball 4: “How do we prevent a malicious user from spamming delete operations to destroy a document?”
Threat Model:
- Attacker has editor permissions
- Sends 10,000 delete operations per second
- Goal: Destroy document content or DoS other editors
Multi-Layer Protection:
graph TD
subgraph Layer1["Layer 1: Rate Limiting"]
RL1[Per-user rate limit: 100 ops/sec]
RL2[Per-document rate limit: 1000 ops/sec]
RL3[Burst limit: 200 ops in 1 sec]
end
subgraph Layer2["Layer 2: Operation Validation"]
VAL1[Operation size limit: 10KB max]
VAL2[Delete limit: max 10% of doc per op]
VAL3[Version gap check: reject if >100 versions behind]
end
subgraph Layer3["Layer 3: Anomaly Detection"]
AD1[ML-based anomaly detection]
AD2[Pattern recognition: spam detection]
AD3[Behavioral analysis: human-like typing]
end
subgraph Layer4["Layer 4: Recovery"]
REC1[Instant rollback from version history]
REC2[Auto-revert suspicious bulk deletes]
REC3[Admin override capability]
end
Layer1 --> Layer2 --> Layer3 --> Layer4
Implementation Details:
| Layer | Mechanism | Threshold | Action |
|---|---|---|---|
| Rate Limit | Token bucket per user | 100 ops/sec | Queue excess, drop if overflow |
| Rate Limit | Token bucket per doc | 1,000 ops/sec | Throttle all users proportionally |
| Validation | Delete size check | Max 10% of doc | Reject oversized deletes |
| Validation | Version freshness | Within 100 versions | Reject stale operations |
| Anomaly | Typing pattern | Human-like cadence | Flag non-human patterns |
| Anomaly | Delete ratio | <50% of ops are deletes | Flag aggressive deletion |
| Recovery | Auto-rollback | >10% content deleted in 10s | Auto-restore from last safe version |
| Recovery | Admin alert | Any anomaly detected | Notify document owner |
Curveball 5: “How do we support real-time collaboration over a high-latency connection (e.g., 500ms RTT satellite link)?”
Challenge: 500ms RTT means each edit takes 500ms round-trip. With OT requiring server acknowledgment, typing feels extremely laggy.
Solution: Local-First with Optimistic Execution
sequenceDiagram
participant C as Client (High Latency)
participant S as Server
Note over C: Local-first execution
C->>C: User types 'H' → Apply locally immediately (0ms)
C->>S: Send op to server (500ms RTT)
S->>S: Apply op, assign version
S->>C: Ack with version (500ms)
C->>C: Confirm op (already applied locally)
Note over C: Remote ops handling
S->>C: Remote op from User B (500ms)
C->>C: Transform against local pending ops
C->>C: Apply transformed remote op
C->>S: Ack remote op
Note over C: Conflict resolution
C->>C: Local op conflicts with remote
C->>C: OT transform resolves conflict
C->>C: Display resolved state
Optimizations for High Latency:
| Optimization | Description | Impact |
|---|---|---|
| Local-first execution | Apply edits locally before server ack | 0ms perceived latency |
| Operation batching | Batch multiple keystrokes into one op | Reduce round-trips by 10x |
| Predictive transform | Predict remote ops and pre-transform | Reduce conflict resolution time |
| Compression | Compress operation payloads | Reduce transmission time |
| Connection pooling | Reuse WebSocket connections | Reduce connection setup time |
| Edge caching | Cache at edge locations closer to user | Reduce effective latency |
| Graceful degradation | Show “syncing” indicator during conflict | Better UX during conflicts |
Typing experience comparison:
| Scenario | Without Optimization | With Local-First |
|---|---|---|
| Typing latency | 500ms per character | 0ms (local) |
| Remote edit visibility | 500ms delay | 500ms delay (unavoidable) |
| Conflict resolution | Server-side only | Client-side pre-transform |
| Perceived responsiveness | Poor | Excellent |
10. Failure Modes and Mitigations
10.1 Failure Mode Matrix
| Failure Mode | Probability | Impact | Detection | Mitigation | RTO | RPO |
|---|---|---|---|---|---|---|
| OT Engine Crash | Medium | High | Health check timeout | Auto-restart + failover to replica | 30s | 0 (ops in Redis) |
| Redis Cluster Failure | Low | Critical | Connection errors | Multi-AZ Redis, async replication | 1min | 1s |
| S3 Outage | Very Low | High | API errors | Multi-region replication, local cache | 5min | 0 |
| WebSocket Gateway Down | Medium | Medium | Connection drops | Auto-reconnect, fallback to polling | 10s | 0 |
| Database Corruption | Very Low | Critical | Checksum mismatch | Point-in-time recovery, backups | 15min | 5min |
| Network Partition | Low | High | Latency spike | Split-brain protocol, quorum | 30s | 0 |
| DDoS Attack | Medium | High | Traffic anomaly | Rate limiting, WAF, auto-scale | 1min | 0 |
| OT Transform Bug | Low | Critical | Divergence detection | Rollback to last known good version | 1min | 1 op |
| Memory Leak (OT Engine) | Medium | Medium | Memory usage alert | Auto-restart, memory limits | 30s | 0 |
| Clock Skew | Low | Medium | Timestamp anomaly | Use logical clocks (Lamport) | N/A | N/A |
10.2 Detailed Mitigation Strategies
OT Engine Failure
Problem: OT engine process crashes while processing operations
Mitigation:
├── Primary: Replica OT engine in same AZ
│ └── Hot standby with Redis op log as shared state
│ └── Failover in < 30 seconds
│
├── Secondary: Multi-AZ deployment
│ └── Active-passive across availability zones
│ └── RTO: 1 minute
│
├── Data protection:
│ └── All ops persisted to Redis before ack
│ └── Redis has AOF + RDB persistence
│ └── RPO: 0 (no op loss)
│
└── Client handling:
└── WebSocket disconnect → auto-reconnect
└── On reconnect: sync from last acked version
└── Pending ops re-sent with idempotency check
Network Partition (Split-Brain)
graph TD
subgraph Before["Before Partition"]
S1[Server A] <-->|Sync| S2[Server B]
C1[Client 1] --> S1
C2[Client 2] --> S2
end
subgraph During["During Partition"]
S1_INDEPENDENT[Server A<br/>Independent]
S2_INDEPENDENT[Server B<br/>Independent]
C1 --> S1_INDEPENDENT
C2 --> S2_INDEPENDENT
Note1[Both serve edits independently]
end
subgraph After["After Healing"]
MERGE[Merge Engine]
S1_INDEPENDENT -->|Ops during partition| MERGE
S2_INDEPENDENT -->|Ops during partition| MERGE
MERGE -->|Transform + Merge| MERGED[Merged State]
MERGED -->|Sync| S1_HEALED[Server A]
MERGED -->|Sync| S2_HEALED[Server B]
end
Split-Brain Resolution Protocol:
- Detection: Heartbeat between servers fails after 3 consecutive misses
- Independent operation: Both sides continue serving edits independently
- Partition healing: When connectivity restored:
- Both sides exchange operation logs during partition window
- OT transform resolves all conflicts
- Merge produces single consistent state
- Both sides sync to merged state
- Client notification: Clients receive merged operations and converge
OT Transform Bug (Divergence Detection)
Problem: A bug in the OT transform function causes clients to diverge
Detection:
├── Checksum verification:
│ └── Server maintains document checksum (SHA-256)
│ └── Clients periodically send their checksum
│ └── Mismatch → divergence detected
│
├── Version reconciliation:
│ └── Server tracks expected version per client
│ └── Client reports its current version
│ └── Gap > threshold → potential divergence
│
└── Statistical monitoring:
└── Track transform success rate
└── Alert on >0.1% transform failures
Recovery:
├── Immediate: Force sync from server authoritative state
├── Client-side: Discard local state, re-download document
├── Server-side: Rollback to last verified checkpoint
└── Long-term: Deploy fix, replay ops from checkpoint
10.3 Disaster Recovery
DR Strategy:
├── Backup frequency: Every 5 minutes (incremental)
├── Backup retention: 7 years (compliance)
├── Backup location: Multi-region (primary + 2 secondary)
├── RTO: 15 minutes (full system recovery)
└── RPO: 5 minutes (max data loss)
DR Test Schedule:
├── Monthly: Automated failover test (non-production)
├── Quarterly: Full DR drill (production, off-hours)
└── Annually: Tabletop exercise with leadership
10.4 Monitoring and Alerting
Key Metrics:
├── Latency
│ ├── Edit propagation P50/P95/P99
│ ├── Document load time
│ └── WebSocket connection time
│
├── Throughput
│ ├── Operations per second (global + per shard)
│ ├── Active concurrent editors
│ └── WebSocket connections count
│
├── Reliability
│ ├── OT transform success rate
│ ├── Divergence detection count
│ ├── Failed operations count
│ └── Client reconnect rate
│
├── Storage
│ ├── Document count
│ ├── Storage usage (by type)
│ └── Version history size
│
└── Business
├── Active documents (24h)
├── New documents created
└── Export count
Alert Thresholds:
├── Critical: P99 latency > 500ms for 5 min
├── Critical: OT transform failure rate > 1%
├── Critical: Divergence detected
├── Warning: P95 latency > 300ms for 10 min
├── Warning: WebSocket reconnect rate > 10%
└── Warning: Storage usage > 80% capacity
Appendix A: Glossary
| Term | Definition |
|---|---|
| OT | Operational Transformation — algorithm for reconciling concurrent edits |
| CRDT | Conflict-free Replicated Data Type — data structure for distributed collaboration |
| Convergence | Property that all replicas reach the same state given the same operations |
| Intention Preservation | Property that a user’s intended edit is preserved after transformation |
| Presence | Real-time awareness of other users’ cursor positions and activities |
| Sharding | Horizontal partitioning of data across multiple servers |
| Consistent Hashing | Hashing algorithm that minimizes data movement on cluster changes |
| RTO | Recovery Time Objective — max acceptable downtime |
| RPO | Recovery Point Objective — max acceptable data loss |
Appendix B: References
- “Designing Data-Intensive Applications” by Martin Kleppmann
- Google Docs architecture: https://research.google/pubs/pub38228/
- Yjs CRDT: https://docs.yjs.dev/
- Automerge CRDT: https://automerge.org/
- Etherpad OT implementation: https://github.com/ether/etherpad-lite
Document Version: 1.0 | Last Updated: 2026-08-28