Design a Turn-Based Online Gaming Service like Chess.com
Author: System Design Expert | Date: 2026-08-28 Scope: End-to-end architecture for a turn-based multiplayer chess platform supporting millions of concurrent players.
Table of Contents
- Requirements Clarification
- Back-of-Envelope Calculations
- System Interface Design
- High-Level Architecture
- Core Component Design
- Data Model
- Key Design Decisions
- Scalability Strategy
- Curveball Scenarios
- Failure Modes and Mitigations
1. Requirements Clarification
1.1 Functional Requirements
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | User registration, authentication, and profile management | P0 |
| FR-02 | Real-time matchmaking with skill-based pairing (ELO rating) | P0 |
| FR-03 | Turn-based game engine with move validation (legal/illegal move detection) | P0 |
| FR-04 | Multiple time controls: Bullet (1+0), Blitz (3+2, 5+0), Rapid (10+0, 15+10), Classical (30+0, 60+0) | P0 |
| FR-05 | Game state persistence and recovery (survive disconnects) | P0 |
| FR-06 | Game replay with PGN export/import | P1 |
| FR-07 | Spectator mode — live watch ongoing games | P1 |
| FR-08 | ELO rating system with provisional rating handling | P1 |
| FR-09 | Puzzle/challenge system with daily puzzles | P2 |
| FR-10 | Leaderboards and statistics (win/loss/draw, streaks, opening book) | P2 |
| FR-11 | Chat system (in-game + global + private) | P2 |
| FR-12 | Game analysis engine integration (post-game engine evaluation) | P2 |
| FR-13 | Tournament support (Swiss, knockout, round-robin) | P3 |
| FR-14 | Opening book and endgame tablebase integration | P3 |
1.2 Non-Functional Requirements
| ID | Requirement | Target |
|---|---|---|
| NFR-01 | Move latency (server round-trip) | < 200ms p95 |
| NFR-02 | Matchmaking wait time | < 10s p95 for Blitz, < 30s for Rapid |
| NFR-03 | Availability | 99.95% (game servers), 99.9% (API) |
| NFR-04 | Concurrency | 5M DAU, 500K concurrent games |
| NFR-05 | Data durability | Zero game state loss (replicated writes) |
| NFR-06 | Anti-cheat | Engine detection, move time anomaly detection |
| NFR-07 | Scalability | Horizontal scale to 10M DAU |
| NFR-08 | Session recovery | Reconnect within 30s restores exact game state |
| NFR-09 | Storage retention: 90 days hot, 1 year warm, indefinite cold (PGN only) |
2. Back-of-Envelope Calculations
2.1 Assumption Sets
We reason from first principles with three assumption sets (Conservative / Base / Aggressive):
| Parameter | Conservative | Base | Aggressive |
|---|---|---|---|
| DAU | 1M | 5M | 10M |
| Concurrent games | 100K | 500K | 1M |
| Avg moves per game | 40 | 60 | 80 |
| Avg game duration (Blitz) | 5 min | 8 min | 10 min |
| Avg game duration (Rapid) | 15 min | 20 min | 25 min |
| Game mix (Blitz / Rapid / Classical) | 60% / 30% / 10% | 50% / 35% / 15% | 40% / 40% / 20% |
| Avg moves per second (peak) | 0.5 | 1.0 | 1.5 |
| Move message size | 50 bytes | 80 bytes | 100 bytes |
| Spectator ratio (spectators / live game) | 2:1 | 3:1 | 5:1 |
2.2 QPS Estimates (Base Case)
Concurrent games: 500K
Avg moves/sec per game: 1.0
Move QPS (gameplay): 500K × 1.0 = 500,000 QPS
Each move = 1 write (game state) + 1 notify (opponent) + N notifies (spectators)
Spectator QPS: 500K × 3 × 1.0 = 1,500,000 QPS (fan-out)
Matchmaking QPS:
Games completed per hour = 500K / (avg 8min) × 60 = 3.75M/hr
Matchmaking requests: ~3.75M/hr = ~1,042 QPS
API QPS (non-gameplay):
Profile fetch, leaderboard, stats: ~50K QPS
Auth: ~5K QPS
Total estimated QPS: ~2.1M (peak ~4M with burst)
2.3 Storage Estimates
Per game storage:
- Game metadata: ~1 KB
- Move list (60 moves × 20 bytes): ~1.2 KB
- PGN notation: ~2 KB
- Engine analysis (optional): ~5 KB
Total per game: ~9.2 KB
Games per day (Base):
500K concurrent / 8min avg × 60min × 24hr = 90M games/day
Daily storage: 90M × 9.2 KB = ~828 GB/day
Retention:
Hot (90 days): 828 GB × 90 = ~75 TB
Warm (1 year): 828 GB × 365 = ~302 TB
Cold (PGN only, 2KB/game): 90M × 2KB × 365 = ~65 TB/year
User profiles: 50M users × 5 KB = ~250 MB (negligible)
2.4 Bandwidth Estimates
Per move (client ↔ server):
Request: 80 bytes
Response: 200 bytes (state delta + clock update)
Round-trip: 280 bytes
Per game (60 moves): 16.8 KB
Per second (500K games, 1 move/sec): 140 MB/s outbound (spectators included):
500K games × 4 clients (2 players + 3 spectators avg) × 200B × 1 move/s
= 400 MB/s outbound
Total network: ~1 Gbps per game server region (with headroom)
3. System Interface Design
3.1 REST API (Public / Client-Facing)
# Authentication
POST /api/v1/auth/register
POST /api/v1/auth/login
POST /api/v1/auth/refresh
POST /api/v1/auth/logout
# User Profile
GET /api/v1/users/{userId}
PUT /api/v1/users/{userId}/profile
GET /api/v1/users/{userId}/stats
GET /api/v1/users/{userId}/games?status=completed&limit=50
# Matchmaking
POST /api/v1/matchmaking/request { "variant": "standard", "timeControl": "5+0", "rated": true }
DELETE /api/v1/matchmaking/request/{requestId}
# Games
GET /api/v1/games/{gameId}
GET /api/v1/games/{gameId}/pgn
GET /api/v1/games/{gameId}/analysis
POST /api/v1/games/{gameId}/claim-draw
POST /api/v1/games/{gameId}/resign
# Spectator
GET /api/v1/spectate/{gameId}
GET /api/v1/live-games?userId={userId}
# Leaderboards
GET /api/v1/leaderboards?variant=standard&timeControl=blitz&limit=100
# Puzzles
GET /api/v1/puzzles/daily
GET /api/v1/puzzles?difficulty=medium&limit=20
POST /api/v1/puzzles/{puzzleId}/attempt
3.2 WebSocket Protocol (Gameplay)
# Connection
WS wss://game.chess.com/ws/v1?token={jwt}
# Client → Server Messages
{
"type": "MOVE",
"gameId": "g_abc123",
"move": "e2e4", // SAN or UCI notation
"timestamp": 1700000000123,
"signature": "..." client-side HMAC for anti-cheat
}
{
"type": "RESIGN",
"gameId": "g_abc123"
}
{
"type": "CLAIM_DRAW",
"gameId": "g_abc123"
}
{
"type": "CHAT",
"gameId": "g_abc123",
"message": "Good game!"
}
# Server → Client Messages
{
"type": "MOVE_ACK",
"gameId": "g_abc123",
"move": "e2e4",
"fen": "rnbqkbnr/pppppppp/8/8/4P3/8/PPPP1PPP/RNBQKBNR b KQkq e3 0 1",
"whiteTime": 300000,
"blackTime": 299800,
"moveNumber": 1
}
{
"type": "GAME_START",
"gameId": "g_abc123",
"fen": "startpos",
"white": { "userId": "u_1", "username": "PlayerA", "rating": 1500 },
"black": { "userId": "u_2", "username": "PlayerB", "rating": 1490 },
"timeControl": "5+0",
"increment": 0,
"rated": true
}
{
"type": "GAME_END",
"gameId": "g_abc123",
"result": "1-0",
"reason": "resign",
"whiteRatingDelta": +12,
"blackRatingDelta": -12
}
{
"type": "OPPONENT_DISCONNECTED",
"gameId": "g_abc123",
"gracePeriodMs": 30000
}
{
"type": "SPECTATOR_UPDATE",
"gameId": "g_abc123",
"fen": "...",
"moveNumber": 15,
"viewerCount": 42
}
3.3 Internal gRPC APIs
service GameEngineService {
rpc CreateGame(CreateGameRequest) returns (Game);
rpc PlayMove(PlayMoveRequest) returns (MoveResult);
rpc GetGameState(GetGameStateRequest) returns (GameState);
rpc EndGame(EndGameRequest) returns (GameResult);
rpc ValidateMove(ValidateMoveRequest) returns (ValidationResult);
}
service MatchmakingService {
rpc FindMatch(FindMatchRequest) returns (MatchResult);
rpc CancelSearch(CancelSearchRequest) returns (Empty);
rpc GetQueueStatus(QueueStatusRequest) returns (QueueStatus);
}
service RatingService {
rpc UpdateRating(RatingUpdateRequest) returns (RatingResult);
rpc GetRating(GetRatingRequest) returns (Rating);
}
4. High-Level Architecture
flowchart TB
subgraph Client["Client Layer"]
Web["Web Client<br/>(React)"]
Mobile["Mobile App<br/>(iOS/Android)"]
Desktop["Desktop App<br/>(Electron)"]
end
subgraph Edge["Edge / CDN Layer"]
CDN["CDN / WAF"]
GSLB["Global Load Balancer"]
WSS["WebSocket Proxy<br/>(nginx / Envoy)"]
end
subgraph API["API Gateway Layer"]
GW["API Gateway<br/>(Kong / APISIX)"]
Auth["Auth Filter"]
Rate["Rate Limiter"]
end
subgraph Services["Microservices"]
AuthService["Auth Service"]
UserService["User Service"]
MatchService["Matchmaking Service"]
GameAPI["Game API Service"]
SpectatorSvc["Spectator Service"]
RatingSvc["Rating Service"]
PuzzleSvc["Puzzle Service"]
ChatSvc["Chat Service"]
LeaderSvc["Leaderboard Service"]
AntiCheat["Anti-Cheat Service"]
AnalysisSvc["Analysis Service"]
TournamentSvc["Tournament Service"]
end
subgraph GameServers["Game Server Cluster"]
GS1["Game Server Pod 1"]
GS2["Game Server Pod 2"]
GS3["Game Server Pod N"]
GS_LB["Game Server LB"]
end
subgraph Infra["Infrastructure"]
Redis["Redis Cluster<br/>(Game State Cache)"]
Kafka["Kafka Cluster<br/>(Event Bus)"]
PG["PostgreSQL<br/>(Users, Metadata)"]
TSDB["TimeSeries DB<br/>(Metrics)"]
ES["Elasticsearch<br/>(Search, Logs)"]
S3["Object Storage<br/>(PGN, Analysis)"]
Memcached["Memcached<br/>(Leaderboard Cache)"]
end
Web --> CDN
Mobile --> CDN
Desktop --> CDN
CDN --> GSLB
GSLB --> GW
GSLB --> WSS
GW --> Auth
Auth --> Rate
Rate --> AuthService
Rate --> UserService
Rate --> MatchService
Rate --> GameAPI
Rate --> RatingSvc
Rate --> PuzzleSvc
Rate --> LeaderSvc
WSS --> GS_LB
GS_LB --> GS1
GS_LB --> GS2
GS_LB --> GS3
GS1 --> Redis
GS2 --> Redis
GS3 --> Redis
GS1 --> Kafka
GS2 --> Kafka
GS3 --> Kafka
MatchService --> Redis
MatchService --> Kafka
GameAPI --> PG
UserService --> PG
AuthService --> PG
RatingSvc --> PG
LeaderSvc --> Memcached
Kafka --> RatingSvc
Kafka --> AntiCheat
Kafka --> AnalysisSvc
Kafka --> SpectatorSvc
Kafka --> LeaderSvc
AntiCheat --> PG
AnalysisSvc --> S3
SpectatorSvc --> Redis
5. Core Component Design
5.1 Game Server (The Heart)
The Game Server is a stateful, long-lived process managing active game sessions.
Architecture
flowchart LR
subgraph GameServer["Game Server Pod"]
subgraph Shards["Game Shards (by hash)"]
S1["Shard 0"]
S2["Shard 1"]
SN["Shard N"]
end
Validator["Move Validator<br/>(Stockfish-lite)"]
Clock["Clock Manager"]
StateMgr["State Manager"]
end
S1 --> Validator
S2 --> Validator
SN --> Validator
S1 --> Clock
S2 --> Clock
SN --> Clock
S1 --> StateMgr
S2 --> StateMgr
SN --> StateMgr
Key Design Points
- Sharding Strategy: Games are assigned to shards via
hash(gameId) % N_SHARDS. Each shard manages ~5K concurrent games. - Move Validation Pipeline:
- Receive move from client
- Verify it’s the player’s turn
- Validate move legality using embedded chess engine (Stockfish-lite or equivalent)
- Check for special rules (castling rights, en passant, 50-move rule, threefold repetition)
- Update FEN state
- Deduct time from clock
- Persist to Redis (write-through)
- Publish to Kafka (async)
- Notify opponent via WebSocket
- Clock Management:
- Server-authoritative clock (prevents client clock manipulation)
- Increment applied per move (Fischer timer)
- Delay applied before countdown starts (Bronstein timer)
- Time checks on every move; flag on timeout
- State Persistence:
- Primary: Redis (in-memory, < 1ms read/write)
- Backup: Async write to PostgreSQL every 10 moves or on game end
- Redis Cluster with AOF + RDB for durability
Move Validation Deep Dive
Input: FEN + UCI move (e.g., "e2e4")
Output: VALID / ILLEGAL / GAME_ENDING
Steps:
1. Parse FEN → board state (64 squares, side to move, castling, EP, halfmove, fullmove)
2. Parse UCI → from_square, to_square, promotion_piece
3. Validate piece at from_square belongs to side to move
4. Validate move pattern for piece type:
- Pawn: forward 1/2, diagonal capture, EP, promotion
- Knight: L-shape (8 possible)
- Bishop: diagonal ray (max 7)
- Rook: orthogonal ray (max 7)
- Queen: diagonal + orthogonal
- King: 1 square any direction, castling
5. Validate destination not occupied by own piece
6. Simulate move → check if own king is in check
7. If check, verify no legal response exists → CHECKMATE
8. If no legal moves and not in check → STALEMATE
9. Check draw conditions: 50-move, threefold, insufficient material
10. Return result + new FEN
5.2 Matchmaking Service
ELO-Based Matchmaking Algorithm
flowchart TD
A[Player requests match] --> B[Join wait queue<br/>by rating bucket]
B --> C{Wait time < threshold?}
C -->|Yes| D[Expand search window<br/>±25 → ±50 → ±100]
C -->|No| E[Timeout → notify player]
D --> F{Found match?}
F -->|Yes| G[Create game<br/>Assign to Game Server]
F -->|No| D
G --> H[Notify both players<br/>via WebSocket]
H --> I[Game starts]
Rating Bucket Design
Bucket width: 25 ELO points
Search expansion:
0-10s: ±25 (exact bucket)
10-20s: ±50 (adjacent buckets)
20-30s: ±100 (wider range)
30s+: ±200 (last resort)
Queue data structure: Redis Sorted Set
Key: mm:queue:standard:5+0
Score: rating
Value: playerId:timestamp
For each matchmaking request:
1. ZADD mm:queue:standard:5+0 {rating} {playerId}:{ts}
2. ZRANGEBYSCORE mm:queue:standard:5+0 {rating-25} {rating+25}
3. If found → ZREM both, create game
4. If not → expand range, retry with backoff
ELO Rating Calculation
Expected score: E_A = 1 / (1 + 10^((R_B - R_A) / 400))
Rating update: R_A' = R_A + K × (S_A - E_A)
K-factor:
< 10 games: K = 40 (provisional)
< 30 games: K = 24
≥ 30 games: K = 16
≥ 2400 rating: K = 10 (grandmaster stability)
S_A = 1 (win), 0.5 (draw), 0 (loss)
5.3 Spectator Service
flowchart LR
subgraph Spectator["Spectator System"]
Pub["Kafka Topic: game.moves"]
Sub["Spectator Subscribers"]
Cache["Redis Pub/Sub<br/>per gameId"]
WS["WebSocket Fan-out"]
end
Pub --> Sub
Sub --> Cache
Cache --> WS
- Game Server publishes every move to Kafka topic
game.moves - Spectator Service consumes from Kafka, caches latest state in Redis
- Spectators connect via WebSocket, subscribe to
spectate:{gameId}channel - Redis Pub/Sub fans out updates to all connected spectators
- Viewer count tracked in Redis (INCR/DECR on join/leave)
5.4 Anti-Cheat Service
Detection Strategies
| Strategy | Method | Signal |
|---|---|---|
| Move accuracy | Compare moves to Stockfish top-3 | >95% accuracy over 20+ moves → flag |
| Move time pattern | Analyze time per move distribution | Engine-like: fast easy moves, deep calc on complex |
| Response latency | Time between opponent move and response | Sub-100ms consistently → suspicious |
| Opening book match | Compare to known opening databases | Unusual depth for player’s rating |
| Behavioral analysis | ML model on move sequences | Anomaly score > threshold → review |
| Client integrity | Check for known engine processes | OS-level process scan (desktop client) |
Enforcement Pipeline
1. Real-time scoring (per game, async via Kafka)
2. Accumulate suspicion score across games
3. Threshold levels:
- Low: Silent monitoring, collect more data
- Medium: Restrict to unrated games
- High: Manual review queue
- Critical: Immediate suspension + notification
4. Appeal process with game review dashboard
5.5 Game Replay System
- PGN (Portable Game Notation): Standard format for chess game records
- Every move stored with: move number, SAN notation, comment (optional), evaluation (if analyzed)
- Replay playback: Client-side FEN reconstruction from move list
- Server provides: full move list, initial FEN, time control, result, annotations
PGN Example:
[Event "Rated Blitz Game"]
[Site "Chess.com"]
[Date "2026.08.28"]
[White "PlayerA"]
[Black "PlayerB"]
[Result "1-0"]
[WhiteElo "1500"]
[BlackElo "1490"]
[TimeControl "300+2"]
1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. O-O ... {+0.3} 1-0
5.6 Chat Service
- In-game chat: WebSocket channel per game, rate-limited (5 msg/10s)
- Private messages: Stored in PostgreSQL, delivered via WebSocket
- Global chat: Redis-backed pub/sub, moderated, keyword filtering
- All messages logged for abuse detection
6. Data Model
6.1 PostgreSQL Schema
-- Users
CREATE TABLE users (
user_id BIGSERIAL PRIMARY KEY,
username VARCHAR(64) UNIQUE NOT NULL,
email VARCHAR(255) UNIQUE NOT NULL,
password_hash VARCHAR(255) NOT NULL,
display_name VARCHAR(128),
avatar_url VARCHAR(512),
country_code CHAR(2),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
is_active BOOLEAN DEFAULT TRUE
);
-- Ratings (one row per variant + time control)
CREATE TABLE ratings (
rating_id BIGSERIAL PRIMARY KEY,
user_id BIGINT REFERENCES users(user_id) ON DELETE CASCADE,
variant VARCHAR(32) NOT NULL DEFAULT 'standard', -- standard, chess960, etc.
time_control VARCHAR(16) NOT NULL, -- bullet, blitz, rapid, classical
rating INTEGER NOT NULL DEFAULT 1500,
max_rating INTEGER NOT NULL DEFAULT 1500,
games_played INTEGER NOT NULL DEFAULT 0,
wins INTEGER NOT NULL DEFAULT 0,
losses INTEGER NOT NULL DEFAULT 0,
draws INTEGER NOT NULL DEFAULT 0,
is_provisional BOOLEAN DEFAULT TRUE,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
UNIQUE(user_id, variant, time_control)
);
-- Games
CREATE TABLE games (
game_id VARCHAR(32) PRIMARY KEY, -- UUID-like
white_id BIGINT REFERENCES users(user_id),
black_id BIGINT REFERENCES users(user_id),
variant VARCHAR(32) NOT NULL DEFAULT 'standard',
time_control VARCHAR(16) NOT NULL,
base_time INTEGER NOT NULL, -- seconds
increment INTEGER NOT NULL DEFAULT 0,
rated BOOLEAN NOT NULL DEFAULT TRUE,
status VARCHAR(16) NOT NULL DEFAULT 'playing', -- playing, completed, abandoned
result VARCHAR(8), -- 1-0, 0-1, 1/2-1/2
end_reason VARCHAR(32), -- checkmate, resignation, timeout, draw_agreed, etc.
initial_fen VARCHAR(256) DEFAULT 'startpos',
move_count INTEGER NOT NULL DEFAULT 0,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
ended_at TIMESTAMP WITH TIME ZONE,
server_pod VARCHAR(64), -- which game server handled it
INDEX idx_games_white (white_id),
INDEX idx_games_black (black_id),
INDEX idx_games_status (status),
INDEX idx_games_created (created_at)
);
-- Moves (partitioned by month)
CREATE TABLE moves (
move_id BIGSERIAL,
game_id VARCHAR(32) REFERENCES games(game_id) ON DELETE CASCADE,
move_number INTEGER NOT NULL,
mover_id BIGINT NOT NULL, -- white or black
uci_move VARCHAR(6) NOT NULL, -- e2e4
san_move VARCHAR(8) NOT NULL, -- e4
fen_after VARCHAR(256) NOT NULL,
white_time INTEGER NOT NULL, -- ms remaining
black_time INTEGER NOT NULL,
move_time_ms INTEGER NOT NULL, -- time taken for this move
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
PRIMARY KEY (game_id, move_number)
) PARTITION BY RANGE (created_at);
-- Create monthly partitions
CREATE TABLE moves_y2026m08 PARTITION OF moves
FOR VALUES FROM ('2026-08-01') TO ('2026-09-01');
-- Matchmaking queue (materialized view of Redis state)
CREATE TABLE matchmaking_stats (
day DATE PRIMARY KEY,
variant VARCHAR(32),
time_control VARCHAR(16),
avg_wait_ms INTEGER,
total_matches INTEGER,
cancelled INTEGER
);
-- Puzzles
CREATE TABLE puzzles (
puzzle_id BIGSERIAL PRIMARY KEY,
fen VARCHAR(256) NOT NULL,
solution VARCHAR(256) NOT NULL, -- UCI moves
difficulty INTEGER NOT NULL, -- 1-10
theme VARCHAR(64), -- fork, pin, skewer, etc.
source_game VARCHAR(32),
rating INTEGER,
solve_count INTEGER DEFAULT 0,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
-- User puzzle progress
CREATE TABLE puzzle_progress (
user_id BIGINT REFERENCES users(user_id),
puzzle_id INTEGER REFERENCES puzzles(puzzle_id),
solved BOOLEAN DEFAULT FALSE,
attempts INTEGER DEFAULT 0,
best_time_ms INTEGER,
solved_at TIMESTAMP WITH TIME ZONE,
PRIMARY KEY (user_id, puzzle_id)
);
-- Anti-cheat flags
CREATE TABLE anticheat_flags (
flag_id BIGSERIAL PRIMARY KEY,
user_id BIGINT REFERENCES users(user_id),
game_id VARCHAR(32),
flag_type VARCHAR(32) NOT NULL, -- engine_suspect, move_time_anomaly, etc.
confidence FLOAT NOT NULL, -- 0.0 - 1.0
details JSONB,
status VARCHAR(16) DEFAULT 'pending', -- pending, reviewed, confirmed, dismissed
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
6.2 Redis Data Structures
# Game state (hot)
Key: game:{gameId}
Type: Hash
Fields: fen, whiteTime, blackTime, status, moveNumber, whiteId, blackId, turn
TTL: None (active games), 24h (completed)
# Game move history (hot)
Key: game:{gameId}:moves
Type: List
Value: [uci_move1, uci_move2, ...]
# Spectator count
Key: game:{gameId}:spectators
Type: Set
Value: {wsConnectionId1, wsConnectionId2, ...}
# Matchmaking queue
Key: mm:queue:{variant}:{timeControl}
Type: Sorted Set
Score: rating
Value: {playerId}:{timestamp}
# Player session
Key: session:{playerId}
Type: Hash
Fields: gameId, color, wsConnectionId, lastActive
TTL: 60s (renewed on each move)
# Rate limiting
Key: ratelimit:{userId}:{endpoint}
Type: Sorted Set
Score: timestamp
Value: request_id
7. Key Design Decisions
7.1 Server-Authoritative Game State
| Aspect | Decision | Rationale |
|---|---|---|
| Move validation | Server-side only | Prevents client-side cheating; client is dumb terminal |
| Clock authority | Server clock | Prevents clock manipulation; client clock is display-only |
| State storage | Redis primary, PG secondary | Redis for speed (<1ms), PG for durability and analytics |
| Move notation | UCI internal, SAN external | UCI is unambiguous for engine; SAN is human-readable |
7.2 WebSocket vs HTTP for Gameplay
| Factor | WebSocket | HTTP (SSE + POST) |
|---|---|---|
| Latency | Lower (persistent conn) | Higher (connection setup) |
| Bidirectional | Native | SSE one-way + POST |
| Proxy support | Complex (stateful) | Simple (stateless) |
| Reconnection | Manual | Browser handles SSE |
| Decision | WebSocket | — |
Rationale: Turn-based chess has low message frequency but requires low-latency delivery and bidirectional communication (move + ack + chat + notifications). WebSocket is the natural fit.
7.3 Game Server Sharding
| Strategy | Pros | Cons |
|---|---|---|
| Hash-based (gameId) | Even distribution, no rebalancing needed | Player’s games may be on different servers |
| Player-based (userId) | All games for a player on one server | Hot players cause imbalance |
| Decision | Hash-based by gameId | Acceptable trade-off |
7.4 ELO vs Glicko Rating System
| System | Pros | Cons |
|---|---|---|
| ELO | Simple, well-understood, fast | No rating deviation, ignores inactivity |
| Glicko | Rating deviation (RD), volatility | More complex, harder to explain |
| Decision | ELO with K-factor adjustments | Simpler to implement and explain; Glicko can be added later |
7.5 Move Persistence Strategy
Write path:
Client → Game Server → Redis (sync, blocking) → Kafka (async) → PostgreSQL (async consumer)
Read path (active game):
Client → Game Server → Redis (memory, <1ms)
Read path (historical):
Client → Game API → PostgreSQL (indexed, <50ms)
Rationale: Active games need sub-millisecond state access. Redis provides this. PostgreSQL is the source of truth for completed games and analytics. Kafka bridges the two with at-least-once delivery.
8. Scalability Strategy
8.1 Horizontal Scaling Dimensions
flowchart TD
subgraph Scale["Scaling Levers"]
A["More Game Server Pods<br/>(stateless shard assignment)"]
B["Redis Cluster Shards<br/>(key-based partitioning)"]
C["Kafka Partitions<br/>(per game server)"]
D["PostgreSQL Read Replicas<br/>(analytics, history)"]
E["CDN Edge Nodes<br/>(static assets)"]
end
A -->|"Handles more concurrent games"| B
B -->|"Stores more game states"| C
C -->|"Processes more events"| D
8.2 Capacity Planning
| Component | Per-Instance Capacity | Instances Needed (500K games) |
|---|---|---|
| Game Server Pod | 5K concurrent games | 100 pods |
| Redis Node | 1M keys, 10K ops/sec | 20 nodes (cluster) |
| Kafka Broker | 50K msg/sec/partition | 10 brokers, 50 partitions |
| PostgreSQL | 5K writes/sec (primary) | 1 primary + 4 replicas |
| API Gateway | 10K RPS | 20 instances |
8.3 Auto-Scaling Triggers
Game Server Pods:
- CPU > 70% for 2 min → scale up
- Active games per pod > 4.5K → scale up
- Scale down: CPU < 30% for 5 min
Redis Cluster:
- Memory > 75% → add shard
- Latency p99 > 5ms → investigate
Kafka:
- Consumer lag > 10K → add consumers
- Partition utilization > 80% → add partitions
8.4 Multi-Region Deployment
Primary Region (US-East):
- All game servers, primary DB, primary Redis
- Handles 70% of traffic
Secondary Region (EU-West):
- Game servers, read replicas, Redis replica
- Handles 20% of traffic
- Failover capable (RTO < 5 min)
Tertiary Region (APAC):
- Game servers, read replicas
- Handles 10% of traffic
- Reduces latency for APAC players
Cross-region sync:
- PostgreSQL: Logical replication
- Redis: Redis Cluster cross-region replication
- Kafka: MirrorMaker 2
9. Curveball Scenarios
Curveball 1: “A grandmaster challenges 100 bots simultaneously — how do you detect and handle engine-assisted play at scale?”
Analysis: This is a multi-layered anti-cheat problem. A single player playing against many opponents simultaneously is itself suspicious, but the real challenge is detecting engine assistance when the player appears to be playing legitimately.
Detection Layers:
-
Concurrency Detection:
- Track active games per user in real-time (Redis counter)
- Flag if user has > 5 simultaneous rated games (unusual for humans)
- Cross-reference move timing across games
-
Move Correlation Analysis:
- For each game, compute move accuracy vs. Stockfish depth 20
- If accuracy > 95% across multiple concurrent games → high suspicion
- Engine moves are consistent; human moves vary
-
Temporal Pattern Analysis:
- Engine: near-instant response on tactical positions, longer on positional
- Human: variable response time, often longer on tactical positions
- ML model trained on verified human vs. engine move times
-
Cross-Game Move Reuse:
- If same opening lines appear identically across concurrent games → engine book
- Humans vary their play; engines reproduce exact lines
Response:
Immediate (real-time):
- Silent flag, increase monitoring
- Force unrated mode for new games
- Challenge verification: send puzzle mid-game
Short-term (within 1 hour):
- Accumulate evidence across all active games
- If confidence > 80%: suspend account, notify user
Long-term:
- Add to permanent suspicion database
- Revoke rating gains from flagged games
- Publish anonymized detection stats (transparency)
System Changes:
- Add real-time move accuracy pipeline (Kafka → Stockfish batch eval → scoring)
- Add concurrent game limit enforcement in Game Server
- Add ML model serving for behavioral analysis
Curveball 2: “Your Redis cluster fails during peak hours. 500K active games lose state. What happens and how do you recover?”
Analysis: This is a catastrophic failure scenario. The impact depends on the failure mode and our persistence strategy.
Impact Assessment:
| Failure Mode | Impact | Recovery |
|---|---|---|
| Single Redis node failure | Games on that shard lost (5% of games) | Redis failover, restore from AOF |
| Redis cluster partition | Split-brain, some games inaccessible | Quorum-based recovery |
| Full Redis cluster failure | All active games lose hot state | Restore from PG + Kafka replay |
Recovery Strategy:
Phase 1: Immediate (0-2 min)
- Circuit breaker: Pause new game creation
- Redirect clients to maintenance page
- Activate Redis failover (sentinel / cluster manager)
Phase 2: State Recovery (2-10 min)
Option A: Redis AOF replay (if AOF intact)
- Restore from AOF logs
- Games lost: Only moves after last AOF fsync (typically < 1 sec of data)
Option B: Kafka replay (if AOF corrupted)
- Replay game.moves topic for last N minutes
- Reconstruct game states from move sequences
- Games lost: None (Kafka retention > game duration)
Option C: PostgreSQL restore (last resort)
- Restore from last PG sync (every 10 moves)
- Games lost: Up to 10 moves of progress per game
- Notify affected players, offer rematch
Phase 3: Validation (10-15 min)
- Verify all restored games have consistent state
- Cross-check FEN against move history
- Flag any inconsistent games for manual review
Phase 4: Resume (15-20 min)
- Gradually resume new game creation
- Monitor error rates closely
- Notify players of resolution
Prevention Measures:
- Redis Cluster with 3 replicas per shard (majority quorum)
- AOF with
everysecfsync policy - Kafka as write-ahead log (dual-write: Redis sync + Kafka async)
- Regular disaster recovery drills
- Game state checkpoint to S3 every 60 seconds
Curveball 3: “You need to support a new chess variant (Chess960, Atomic, Horde) without disrupting existing services. How do you design for extensibility?”
Analysis: Chess variants differ in rules, board size, piece types, and win conditions. The system must support variant-specific logic without modifying the core game engine for each new variant.
Design Approach: Strategy Pattern + Plugin Architecture
classDiagram
class GameEngine {
+playMove(gameId, move, variant) MoveResult
+getGameState(gameId) GameState
+endGame(gameId) GameResult
}
class VariantHandler {
<<interface>>
+validateMove(board, move) ValidationResult
+applyMove(board, move) Board
+isCheckmate(board) bool
+isStalemate(board) bool
+isDraw(board) bool
+getLegalMoves(board) Move[]
+serializeBoard(board) string
+deserializeBoard(fen) Board
}
class StandardHandler {
+validateMove() ValidationResult
+applyMove() Board
...
}
class Chess960Handler {
+validateMove() ValidationResult
+applyMove() Board
+castlingRules() different
}
class AtomicHandler {
+validateMove() ValidationResult
+applyMove() Board
+explosionLogic() new
}
class HordeHandler {
+validateMove() ValidationResult
+applyMove() Board
+asymmetricSetup() new
}
GameEngine --> VariantHandler
VariantHandler <|-- StandardHandler
VariantHandler <|-- Chess960Handler
VariantHandler <|-- AtomicHandler
VariantHandler <|-- HordeHandler
Implementation Details:
-
Variant Registry:
Key: variant:registry:{variantName} Value: { "handler": "Chess960Handler", "boardSize": "8x8", "pieceTypes": ["P","N","B","R","Q","K"], "initialFen": "rnbqkbnr...", "specialRules": ["chess960_castling"], "engineConfig": { "depth": 20, "threads": 1 } } -
Game Server Routing:
- Game creation includes
variantfield - Game Server loads appropriate
VariantHandlerat game start - Move validation delegates to the handler
- FEN format extended with variant-specific annotations
- Game creation includes
-
Client Support:
- Client receives variant config on game start
- UI renders board/pieces according to variant rules
- Move input validation uses variant-specific rules
-
Backward Compatibility:
- All existing games use
StandardHandler - New variants opt-in via
variantparameter - Matchmaking queues are variant-scoped
- Ratings are variant-scoped (separate ELO per variant)
- All existing games use
New Variant Onboarding Checklist:
- Implement
VariantHandlerinterface - Add variant to registry
- Create matchmaking queue for variant
- Add rating bucket for variant
- Update client UI for variant rendering
- Add engine support (Stockfish supports most variants)
- Load test with variant-specific patterns
- Update documentation
Curveball 4: “A DDoS attack targets your WebSocket endpoints. Legitimate players can’t connect. How do you protect the service?”
Analysis: WebSocket DDoS is particularly challenging because:
- Connections are long-lived (resource-intensive)
- Traditional rate limiting doesn’t apply well
- Attackers can mimic legitimate connection patterns
Defense Layers:
flowchart TD
A[Attack Traffic] --> B[WAF / Cloudflare]
B --> C{Legitimate?}
C -->|No| D[Block / Challenge]
C -->|Yes| E[WebSocket Proxy]
E --> F[Connection Rate Limiter]
F --> G{Within limits?}
G -->|No| H[CAPTCHA Challenge]
G -->|Yes| I[Game Server]
H -->|Passed| F
H -->|Failed| D
Specific Measures:
-
Edge Layer:
- WAF rules to block known attack patterns
- Geographic blocking (if attack from specific regions)
- IP reputation scoring
-
Connection Limits:
- Max connections per IP: 5
- Max connections per user: 3
- Connection rate limit: 10 new connections/min per IP
-
Challenge Mechanisms:
- JavaScript challenge before WebSocket upgrade
- CAPTCHA for suspicious IPs
- Token-based connection validation (pre-authenticated JWT)
-
Resource Protection:
- Connection timeout: 30s without game assignment
- Memory limits per connection
- Auto-scale WebSocket proxies to absorb traffic
-
Game Server Protection:
- Accept only connections from trusted WebSocket proxies
- Internal network only (no direct external access)
- Connection draining during attack
Curveball 5: “You need to add real-time game analysis (engine evaluation bar) during live games. This requires running Stockfish on every move. How do you handle the compute load?”
Analysis: Running a chess engine on every move for 500K concurrent games is computationally intensive. Stockfish at depth 20 can evaluate ~1M nodes/sec per core.
Compute Requirements:
500K games × 1 move/sec = 500K evaluations/sec
Stockfish depth 15: ~500K nodes/eval
Total: 500K × 500K = 250B nodes/sec
At 1M nodes/sec/core: 250,000 CPU cores needed
This is infeasible for real-time analysis on ALL games.
Practical Solution: Tiered Analysis
| Tier | Games Covered | Depth | Latency | Use Case |
|---|---|---|---|---|
| T0: None | 450K (90%) | — | — | No live analysis |
| T1: Lightweight | 40K (8%) | Depth 10 | < 100ms | Basic eval bar |
| T2: Standard | 8K (1.6%) | Depth 15 | < 500ms | Detailed analysis |
| T3: Premium | 2K (0.4%) | Depth 20 | < 2s | Grandmaster games, tournaments |
Architecture:
flowchart LR
GS["Game Server"] --> Kafka
Kafka["Kafka: game.moves"] --> Sub["Analysis Consumer Pool"]
Sub --> Pool["Stockfish Worker Pool<br/>(Kubernetes Jobs)"]
Pool --> Redis["Redis: game:{id}:eval"]
Redis --> Client["Client (polling / WS)"]
Sub --> Filter{"Tier Check"}
Filter -->|T0| Drop["Skip"]
Filter -->|T1-T3| Pool
Key Design Points:
- Analysis is async and best-effort (never blocks gameplay)
- Workers are auto-scaled Kubernetes Jobs
- Results cached in Redis with TTL = next move
- Client polls or receives eval via WebSocket (if subscribed)
- Premium users / featured games get higher priority
- Post-game analysis runs at full depth (no real-time constraint)
10. Failure Modes and Mitigations
10.1 Failure Mode Matrix
| Component | Failure Mode | Impact | Detection | Mitigation | RTO |
|---|---|---|---|---|---|
| Game Server Pod | Crash / OOM | Games on pod lost | Health check timeout | Auto-replace pod, restore from Redis | < 30s |
| Game Server Pod | Network partition | Games unreachable | Heartbeat miss | Client reconnect to new pod | < 30s |
| Redis Node | Disk failure | Shard data lost | Sentinel alert | Failover to replica | < 10s |
| Redis Cluster | Split-brain | Inconsistent reads | Quorum check | Pause writes, resolve partition | < 1 min |
| Kafka | Broker failure | Event loss | Consumer lag spike | Replica leadership transfer | < 30s |
| Kafka | Topic corruption | Data loss | CRC check failure | Restore from backup | < 5 min |
| PostgreSQL | Primary failure | Writes blocked | Connection errors | Patroni failover | < 30s |
| PostgreSQL | Replication lag | Stale reads | Lag monitoring | Route reads to primary | < 1 min |
| WebSocket Proxy | Memory leak | Connection drops | Memory > 90% | Rolling restart | < 1 min |
| API Gateway | Config error | 5xx errors | Error rate spike | Rollback config | < 1 min |
| CDN | Cache poison | Wrong content | Hash mismatch | Purge cache | < 5 min |
| Client | Network loss | Disconnected game | Ping timeout | Grace period + reconnect | 30s |
10.2 Critical Path: Game Move Flow Failure Analysis
flowchart TD
A[Client sends move] --> B{WS connected?}
B -->|No| C[Reconnect flow]
B -->|Yes| D{Game Server alive?}
D -->|No| E[Failover to replica]
D -->|Yes| F{Redis reachable?}
F -->|No| G[Queue move, retry]
F -->|Yes| H[Validate + Apply move]
H --> I{Valid?}
I -->|No| J[Reject move, notify client]
I -->|Yes| K[Persist to Redis]
K --> L{Success?}
L -->|No| M[Retry with backup Redis]
L -->|Yes| N[Notify opponent]
N --> O{Success?}
O -->|No| P[Queue notification, retry]
O -->|Yes| Q[Publish to Kafka]
Q --> R[Done]
C --> B
E --> D
G --> F
M --> K
P --> N
10.3 Disaster Recovery Plan
RPO (Recovery Point Objective):
- Game state: 0 (Redis AOF + Kafka WAL)
- User data: 5 min (PG replication lag)
- Analytics: 1 hour (batch processing)
RTO (Recovery Time Objective):
- Game servers: 30 seconds (auto-replace)
- Redis: 10 seconds (sentinel failover)
- PostgreSQL: 30 seconds (Patroni)
- Full region failover: 5 minutes
Backup Strategy:
- PostgreSQL: Daily full + WAL archiving (continuous)
- Redis: RDB snapshot every 15 min + AOF
- Kafka: Retention 7 days (replayable)
- S3: Versioned PGN archives
10.4 Monitoring and Alerting
| Metric | Warning | Critical | Action |
|---|---|---|---|
| Game server CPU | > 70% | > 90% | Auto-scale |
| Game server memory | > 80% | > 95% | Restart pod |
| Redis latency p99 | > 2ms | > 10ms | Investigate, scale |
| Redis memory | > 75% | > 90% | Add shard |
| Kafka consumer lag | > 5K | > 50K | Add consumers |
| PG replication lag | > 5s | > 30s | Route to primary |
| WS connection errors | > 1% | > 5% | Check proxy health |
| Move validation failures | > 0.1% | > 1% | Check engine health |
| Matchmaking wait p95 | > 15s | > 30s | Scale match service |
| API error rate | > 0.5% | > 2% | Rollback / scale |
Appendix A: Glossary
| Term | Definition |
|---|---|
| FEN | Forsyth-Edwards Notation — standard chess position encoding |
| UCI | Universal Chess Interface — move notation (e2e4) |
| SAN | Standard Algebraic Notation — human-readable (e4, Nf3) |
| PGN | Portable Game Notation — complete game record format |
| ELO | Rating system for player skill estimation |
| Fischer Timer | Clock that adds increment after each move |
| Bronstein Timer | Clock with delay before countdown starts |
| Flag | Losing on time (clock reaches zero) |
Appendix B: Technology Stack Summary
| Layer | Technology | Rationale |
|---|---|---|
| Client | React / React Native / Electron | Cross-platform, rich UI |
| WebSocket | uWebSockets.js / Go | High-performance WS server |
| Game Engine | Rust / C++ (Stockfish) | Performance-critical validation |
| API Gateway | Kong / APISIX | Mature, extensible |
| Microservices | Go / Java | Performance + ecosystem |
| Game State | Redis Cluster | Sub-millisecond access |
| Event Bus | Apache Kafka | Durable, scalable streaming |
| Database | PostgreSQL | ACID, JSONB, partitioning |
| Cache | Memcached | Leaderboard, hot data |
| Object Storage | S3 / MinIO | PGN archives, analysis |
| Search | Elasticsearch | Game search, logs |
| Orchestration | Kubernetes | Auto-scaling, self-healing |
| Monitoring | Prometheus + Grafana | Metrics, alerting |
| Tracing | Jaeger | Distributed tracing |