Design a Real-Time Gaming Service like WarCraft
Author: System Design Expert | Date: 2026-08-28 Scope: End-to-end architecture for a real-time multiplayer game service supporting persistent worlds, real-time combat, and thousands of concurrent players per zone.
Table of Contents
- Requirements Clarification
- Back-of-Envelope Calculations
- System Interface Design
- High-Level Architecture
- Core Component Design
- Data Model
- Key Design Decisions
- Scalability Strategy
- Curveball Scenarios
- Failure Modes and Mitigations
1. Requirements Clarification
1.1 Functional Requirements
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | Persistent world with zones/instances | P0 |
| FR-02 | Real-time player movement and action synchronization | P0 |
| FR-03 | Real-time combat system (melee, ranged, abilities) | P0 |
| FR-04 | Server-authoritative game state with anti-cheat | P0 |
| FR-05 | Zone-based world partitioning (sharding) | P0 |
| FR-06 | Entity interpolation and extrapolation for smooth movement | P0 |
| FR-07 | Party/group system with cross-zone support | P1 |
| FR-08 | Guild/clan system with persistent data | P1 |
| FR-09 | Quest system with state tracking | P1 |
| FR-10 | Inventory and item management | P1 |
| FR-11 | Player vs Environment (PvE) dungeons and raids | P1 |
| FR-12 | Player vs Player (PvP) arenas and battlegrounds | P1 |
| FR-13 | Chat system (say, party, guild, trade, whisper) | P1 |
| FR-14 | Auction house / player trading | P2 |
| FR-15 | Character progression (levels, skills, talents) | P2 |
| FR-16 | World events and dynamic content | P2 |
| FR-17 | Cross-realm / cross-server play | P2 |
| FR-18 | Spectator / broadcast system for PvP | P3 |
1.2 Non-Functional Requirements
| ID | Requirement | Target |
|---|---|---|
| NFR-01 | Game tick rate | 20 Hz (50ms tick interval) |
| NFR-02 | Client-to-server latency | < 100ms p95 (same region) |
| NFR-03 | Motion smoothness | Jitter < 16ms (60fps equivalent) |
| NFR-04 | Zone capacity | 1,000 players per zone |
| NFR-05 | World capacity | 50,000 players per realm/server |
| NFR-06 | Availability | 99.9% (planned maintenance windows only) |
| NFR-07 | State consistency | Strong consistency within zone, eventual across zones |
| NFR-08 | Anti-cheat | Real-time detection, zero-tolerance policy |
| NFR-09 | Save frequency | Auto-save every 60s, immediate on zone transition |
| NFR-10 | Scalability | Support 1M concurrent players globally |
| NFR-11 | Lag compensation | Sub-50ms compensation window |
| NFR-12 | Data durability | Zero character data loss |
2. Back-of-Envelope Calculations
2.1 Assumption Sets
Three assumption sets for capacity planning:
| Parameter | Conservative | Base | Aggressive |
|---|---|---|---|
| Concurrent players | 100K | 500K | 1M |
| Players per zone | 500 | 1,000 | 2,000 |
| Zones per realm | 20 | 50 | 100 |
| Realms/servers | 10 | 50 | 100 |
| Game tick rate | 20 Hz | 20 Hz | 20 Hz |
| Entities per player (visible) | 50 | 100 | 150 |
| Packet size per tick (outbound) | 200 bytes | 400 bytes | 600 bytes |
| Packet size per tick (inbound) | 50 bytes | 80 bytes | 100 bytes |
| Avg session duration | 1 hour | 2 hours | 3 hours |
| Character data size | 50 KB | 100 KB | 200 KB |
2.2 Network Bandwidth (Base Case)
Per player per tick:
Inbound (client → server): 80 bytes × 20 Hz = 1,600 B/s = 12.8 Kbps
Outbound (server → client): 400 bytes × 20 Hz = 8,000 B/s = 64 Kbps
Per zone (1,000 players):
Inbound: 12.8 Kbps × 1,000 = 12.8 Mbps
Outbound: 64 Kbps × 1,000 = 64 Mbps
Total: 76.8 Mbps per zone
Per realm (50 zones):
Total: 76.8 Mbps × 50 = 3.84 Gbps
Global (50 realms):
Total: 3.84 Gbps × 50 = 192 Gbps
With headroom (3×): ~600 Gbps backbone capacity needed
2.3 Server Compute (Base Case)
Game tick processing per zone:
Players: 1,000
Entities per player: 100
Total entity updates: 1,000 × 100 = 100,000 entities/tick
At 20 Hz: 2,000,000 entity updates/sec
Per entity update:
- Position interpolation: ~100 CPU cycles
- Collision detection: ~500 CPU cycles
- Ability processing: ~1,000 CPU cycles (when active)
- Visibility check: ~200 CPU cycles
Average: ~500 CPU cycles/entity
Total CPU per zone: 2M × 500 cycles = 1B cycles/sec
At 3 GHz: ~333M instructions/sec ≈ 0.33 cores (theoretical minimum)
Realistic (with overhead, networking, I/O):
~8-16 CPU cores per zone server
~32 GB RAM per zone server
Total zone servers: 50 zones × 1 server = 50 servers per realm
Total globally: 50 realms × 50 servers = 2,500 zone servers
2.4 Storage Estimates
Per character:
- Core data (level, stats, position): 5 KB
- Inventory (100 slots × 50 bytes): 5 KB
- Quest progress (50 active × 200 bytes): 10 KB
- Skill tree: 2 KB
- Guild data: 1 KB
- Social (friends, ignore): 2 KB
- Achievement progress: 5 KB
Total: ~30 KB per character
Active characters: 500K players × 5 chars/player = 2.5M characters
Character storage: 2.5M × 30 KB = ~75 GB
World state (persistent):
- NPC positions, spawns: ~500 MB per realm
- Dynamic world objects: ~100 MB per realm
- Total: ~30 GB for 50 realms
Transaction log (inventory changes, combat events):
- 500K players × 100 transactions/hr × 200 bytes = 1 GB/hr
- Daily: 24 GB
- Retention (30 days): 720 GB
Total storage: ~1 TB (with backups and headroom: 5 TB)
2.5 QPS Estimates
Game tick QPS:
500K players × 20 Hz = 10,000,000 QPS (tick processing)
Zone transition QPS:
500K players × 0.1 transitions/hr = 50K/hr = ~14 QPS
Inventory transaction QPS:
500K × 100/hr = 50M/hr = ~13,889 QPS
Chat message QPS:
500K × 10 msg/min = 83M/hr = ~23,148 QPS
Auth/login QPS:
500K sessions / 2hr avg = 250K logins/hr = ~69 QPS
Total non-tick QPS: ~50,000 QPS
Total with ticks: ~10,050,000 QPS
3. System Interface Design
3.1 Client-Server Binary Protocol
Real-time games use custom binary protocols for efficiency. We define a length-prefixed binary protocol over TCP/UDP.
Packet Format:
┌─────────┬──────────┬──────────┬───────────┬──────────┐
│ Version │ SeqNo │ Type │ Length │ Payload │
│ 1B │ 4B │ 2B │ 2B │ VarLen │
└─────────┴──────────┴──────────┴───────────┴──────────┘
Version: Protocol version (1 byte)
SeqNo: Sequence number for ordering (4 bytes, unsigned)
Type: Message type (2 bytes, enum)
Length: Payload length in bytes (2 bytes)
Payload: Type-specific data
3.2 Message Types
# === Client → Server ===
MSG_CLIENT_TICK (0x0100):
{
clientId: uint64,
position: Vector3 { x: float32, y: float32, z: float32 },
rotation: float32 (yaw),
cameraTarget: Vector3,
inputState: BitFlags {
moveForward, moveBackward, moveLeft, moveRight,
sprint, jump, crouch, attack, ability1-8, interact
},
selectedTarget: uint64 (entityId, 0 if none),
clientTimestamp: uint64 (ms since epoch),
pingEstimate: uint16 (ms)
}
MSG_ABILITY_CAST (0x0101):
{
abilityId: uint16,
targetId: uint64,
targetPos: Vector3,
castTime: uint32 (ms),
predictionId: uint32 (client-side prediction ID)
}
MSG_INTERACT (0x0102):
{
entityId: uint64,
action: uint8 (talk, loot, pickup, use, trade)
}
MSG_CHAT (0x0103):
{
channel: uint8 (say, party, guild, trade, whisper),
targetId: uint64 (for whisper, 0 otherwise),
message: string (max 256 bytes)
}
MSG_ZONE_REQUEST (0x0104):
{
targetZoneId: uint16,
targetPos: Vector3,
reason: uint8 (portal, teleport, death, manual)
}
MSG_INVENTORY_ACTION (0x0105):
{
action: uint8 (equip, unequip, use, drop, move, split),
sourceSlot: uint8,
targetSlot: uint8,
quantity: uint16
}
# === Server → Client ===
MSG_WORLD_STATE (0x0200):
{
serverTick: uint64,
playerState: {
position: Vector3,
rotation: float32,
health: uint16,
mana: uint16,
speed: float32,
buffs: Buff[] { id: uint16, duration: uint32, stacks: uint8 },
debuffs: Debuff[]
},
entities: EntityUpdate[] {
entityId: uint64,
position: Vector3,
rotation: float32,
health: uint16,
state: uint8 (idle, moving, attacking, casting, dead),
interpolationId: uint16
},
newEntities: EntitySpawn[] {
entityId: uint64,
entityType: uint16 (player, npc, creature, object),
templateId: uint32,
position: Vector3,
name: string
},
removedEntities: uint64[] (entityIds to despawn),
effects: VFX[] { effectId: uint16, position: Vector3, duration: uint32 }
}
MSG_COMBAT_EVENT (0x0201):
{
attackerId: uint64,
defenderId: uint64,
abilityId: uint16,
damage: uint16,
damageType: uint8 (physical, fire, frost, nature, holy, shadow),
critical: bool,
blocked: uint16,
absorbed: uint16,
healed: uint16
}
MSG_ZONE_DATA (0x0202):
{
zoneId: uint16,
zoneName: string,
initialPos: Vector3,
entities: EntitySpawn[],
npcs: NPCSpawn[],
objects: WorldObjectSpawn[],
weather: uint8,
timeOfDay: uint16 (0-2400)
}
MSG_INVENTORY_UPDATE (0x0203):
{
slot: uint8,
item: Item { itemId: uint32, quantity: uint16, enchantments: Enchant[] },
action: uint8 (added, removed, updated, equipped)
}
MSG_CHAT_BROADCAST (0x0204):
{
channel: uint8,
senderId: uint64,
senderName: string,
message: string,
timestamp: uint64
}
MSG_ERROR (0x02FF):
{
errorCode: uint16,
errorMessage: string,
correlationId: uint32
}
3.3 Internal Service APIs (gRPC)
service ZoneService {
rpc EnterZone(EnterZoneRequest) returns (ZoneEntry);
rpc LeaveZone(LeaveZoneRequest) returns (ZoneExit);
rpc GetZoneState(ZoneStateRequest) returns (ZoneState);
rpc TransferPlayer(TransferRequest) returns (TransferResult);
}
service CharacterService {
rpc LoadCharacter(LoadCharacterRequest) returns (Character);
rpc SaveCharacter(SaveCharacterRequest) returns (SaveResult);
rpc DeleteCharacter(DeleteCharacterRequest) returns (Empty);
rpc ListCharacters(ListCharactersRequest) returns (CharacterList);
}
service AuthService {
rpc Authenticate(AuthRequest) returns (AuthResponse);
rpc ValidateToken(TokenRequest) returns (TokenStatus);
rpc RevokeToken(RevokeRequest) returns (Empty);
}
service MatchmakingService {
rpc QueuePlayer(QueueRequest) returns (QueueStatus);
rpc LeaveQueue(LeaveQueueRequest) returns (Empty);
rpc CreateInstance(CreateInstanceRequest) returns (InstanceInfo);
}
service EconomyService {
rpc BuyItem(BuyItemRequest) returns (TransactionResult);
rpc SellItem(SellItemRequest) returns (TransactionResult);
rpc AuctionList(AuctionListRequest) returns (AuctionResult);
rpc AuctionBid(AuctionBidRequest) returns (BidResult);
}
service GuildService {
rpc CreateGuild(CreateGuildRequest) returns (GuildInfo);
rpc JoinGuild(JoinGuildRequest) returns (JoinResult);
rpc LeaveGuild(LeaveGuildRequest) returns (Empty);
rpc GetGuildInfo(GuildInfoRequest) returns (GuildInfo);
}
4. High-Level Architecture
flowchart TB
subgraph Client["Client Layer"]
GameClient["Game Client<br/>(Unity / Unreal)"]
Launcher["Game Launcher"]
end
subgraph Edge["Edge Network"]
CDN["CDN / WAF"]
GameLB["Game Load Balancer<br/>(L4 TCP/UDP)"]
AuthLB["Auth Load Balancer<br/>(L7 HTTPS)"]
end
subgraph AuthZone["Authentication Zone"]
AuthSvc["Auth Service"]
TokenSvc["Token Service<br/>(JWT / Session)"]
AccountSvc["Account Service"]
end
subgraph Gateway["Game Gateway"]
GW1["Game Gateway 1"]
GW2["Game Gateway 2"]
GWN["Game Gateway N"]
end
subgraph Realm["Realm Server Cluster"]
subgraph WorldMgr["World Management"]
WorldSvc["World Service"]
ZoneMgr["Zone Manager"]
CrossZone["Cross-Zone Router"]
end
subgraph Zones["Zone Servers"]
subgraph Zone1["Zone Server 1<br/>(Elwynn Forest)"]
Z1Engine["Game Engine"]
Z1Net["Network Handler"]
Z1AI["AI / NPC Manager"]
Z1Combat["Combat System"]
end
subgraph Zone2["Zone Server 2<br/>(Stormwind)"]
Z2Engine["Game Engine"]
Z2Net["Network Handler"]
Z2AI["AI / NPC Manager"]
Z2Combat["Combat System"]
end
subgraph ZoneN["Zone Server N<br/>(Dungeon Instance)"]
ZNEngine["Game Engine"]
ZNNet["Network Handler"]
ZNAI["AI / NPC Manager"]
ZNCombat["Combat System"]
end
end
end
subgraph Services["Support Services"]
CharSvc["Character Service"]
InvSvc["Inventory Service"]
QuestSvc["Quest Service"]
GuildSvc["Guild Service"]
AuctionSvc["Auction House Service"]
ChatSvc["Chat Service"]
MMService["Matchmaking Service"]
AntiCheatSvc["Anti-Cheat Service"]
LogSvc["Logging Service"]
end
subgraph Data["Data Layer"]
RedisGame["Redis Cluster<br/>(Hot Game State)"]
RedisCache["Redis Cache<br/>(Sessions, Leaderboards)"]
PGMain["PostgreSQL<br/>(Characters, Accounts)"]
PGShard1["PG Shard 1<br/>(Inventory)"]
PGShard2["PG Shard 2<br/>(Quests, Guilds)"]
Mongo["MongoDB<br/>(Chat, Logs, Events)"]
Kafka["Kafka Cluster<br/>(Event Streaming)"]
TSDB["TimeSeries DB<br/>(Metrics)"]
S3["Object Storage<br/>(Backups, Assets)"]
end
GameClient --> Launcher
Launcher --> CDN
GameClient --> GameLB
CDN --> AuthLB
AuthLB --> AuthSvc
AuthSvc --> TokenSvc
AuthSvc --> AccountSvc
GameLB --> GW1
GameLB --> GW2
GameLB --> GWN
GW1 --> WorldSvc
GW2 --> WorldSvc
GWN --> WorldSvc
WorldSvc --> ZoneMgr
ZoneMgr --> Zone1
ZoneMgr --> Zone2
ZoneMgr --> ZoneN
Z1Engine --> RedisGame
Z2Engine --> RedisGame
ZNEngine --> RedisGame
Z1Engine --> Kafka
Z2Engine --> Kafka
ZNEngine --> Kafka
Zone1 --> CrossZone
Zone2 --> CrossZone
CrossZone --> Zone1
CrossZone --> Zone2
CharSvc --> PGMain
InvSvc --> PGShard1
QuestSvc --> PGShard2
GuildSvc --> PGShard2
AuctionSvc --> PGShard1
ChatSvc --> Mongo
Kafka --> AntiCheatSvc
Kafka --> LogSvc
Kafka --> CharSvc
Kafka --> InvSvc
AuthSvc --> RedisCache
MMService --> RedisCache
5. Core Component Design
5.1 Game Gateway
The Game Gateway is the single entry point for all game client connections.
Responsibilities
- Connection management (TCP/UDP multiplexing)
- Protocol validation and parsing
- Player authentication (token verification)
- Zone routing (direct to correct zone server)
- Connection state tracking
- Rate limiting and flood protection
Architecture
flowchart LR
subgraph Gateway["Game Gateway"]
ConnMgr["Connection Manager"]
Parser["Protocol Parser"]
Router["Zone Router"]
RateLimiter["Rate Limiter"]
AuthChecker["Auth Checker"]
end
Client["Game Client"] --> ConnMgr
ConnMgr --> Parser
Parser --> AuthChecker
AuthChecker --> RateLimiter
RateLimiter --> Router
Router --> ZoneServer["Zone Server"]
Connection Model
Phase 1: TCP Handshake + Auth
Client → Gateway: CONNECT + auth_token
Gateway → Auth Service: Validate token
Auth Service → Gateway: Valid / Invalid
Gateway → Client: AUTH_OK (sessionId, serverTick)
Phase 2: Zone Assignment
Gateway → World Service: Get zone for player
World Service → Gateway: ZoneId, ZoneServerAddress
Gateway → Client: ROUTE_TO_ZONE (zoneServerAddr)
Phase 3: Direct Connection
Client → Zone Server: DIRECT_CONNECT (sessionId)
Zone Server → Client: WELCOME (zoneData, initialEntities)
Phase 4: Game Loop
Client ↔ Zone Server: Tick packets at 20 Hz
5.2 Zone Server (The Core Game Loop)
The Zone Server runs the authoritative game simulation for a specific zone.
Game Loop Architecture
flowchart TD
subgraph Tick["Game Tick (50ms cycle)"]
Input["1. Collect Inputs<br/>(buffered client packets)"]
Predict["2. Client Prediction<br/>(server-side replay)"]
Simulate["3. Physics Simulation<br/>(movement, collision)"]
Combat["4. Combat Resolution<br/>(abilities, damage)"]
AI["5. AI / NPC Updates<br/>(pathfinding, behavior)"]
Visibility["6. Visibility Culling<br/>(what each player sees)"]
Serialize["7. State Serialization<br/>(delta encoding)"]
Broadcast["8. Network Broadcast<br/>(send to clients)"]
Save["9. Async Persistence<br/>(checkpoint to Redis)"]
end
Input --> Predict
Predict --> Simulate
Simulate --> Combat
Combat --> AI
AI --> Visibility
Visibility --> Serialize
Serialize --> Broadcast
Broadcast --> Save
Detailed Tick Breakdown
| Phase | Duration | Description |
|---|---|---|
| 1. Input Collection | 5ms | Gather all client packets received since last tick; discard out-of-order |
| 2. Prediction Reconciliation | 5ms | Compare client-predicted positions with server-authoritative; correct discrepancies |
| 3. Physics Simulation | 10ms | Move entities, resolve collisions, apply gravity, terrain checks |
| 4. Combat Resolution | 10ms | Process ability casts, calculate damage, apply buffs/debuffs, check deaths |
| 5. AI Updates | 10ms | NPC behavior trees, pathfinding, aggro management, spawn/despawn logic |
| 6. Visibility Culling | 3ms | Determine which entities each player can see (distance + occlusion) |
| 7. Serialization | 4ms | Delta-encode state changes, compress packets |
| 8. Broadcast | 2ms | Send packets to connected clients |
| 9. Async Persistence | 1ms | Write checkpoint to Redis (non-blocking) |
| Total | 50ms | 20 Hz tick rate |
Entity Management
Entity Types:
- Player (controlled by client, authoritative on server)
- NPC (AI-controlled, static or scripted behavior)
- Creature (AI-controlled, combat-capable)
- WorldObject (interactive objects: chests, doors, portals)
- Projectile (abilities, arrows, spells)
- Effect (visual effects, AoE zones)
Entity Data Structure:
{
entityId: uint64 (unique, monotonic),
entityType: uint16,
templateId: uint32 (references asset/template),
position: Vector3 { x: float32, y: float32, z: float32 },
rotation: float32 (yaw),
velocity: Vector3,
health: uint16,
maxHealth: uint16,
state: uint8 (idle, moving, attacking, casting, dead, stunned),
owner: uint64 (playerId if player-controlled, 0 otherwise),
buffs: Buff[],
debuffs: Debuff[],
lastUpdateTick: uint64,
interpolationId: uint16 (for client interpolation)
}
5.3 Real-Time State Synchronization
The Core Challenge
Client A (ping 50ms) ────────┐
├──→ Server (authoritative) ←──┐
Client B (ping 80ms) ────────┘ └── Client C (ping 60ms)
At 20 Hz with variable latency, the server sees stale client state. The solution combines three techniques:
Technique 1: Client-Side Prediction
Client sends:
{ position: predicted_pos, input: current_input, clientTick: T }
Server receives at tick T+delay:
1. Replay from last known good state at tick T
2. Apply client's input from tick T
3. Compare result with client's predicted position
4. If within tolerance (±0.5m): accept
5. If outside tolerance: correct and send correction
Client receives correction:
1. If current prediction matches server: continue
2. If mismatch: snap to server position + rewind prediction
Technique 2: Server-Side Reconciliation
flowchart LR
subgraph Server["Server Reconciliation"]
A["Receive client packet<br/>(tick T, position P)"]
B["Find last authoritative<br/>state at tick T"]
C["Replay inputs from T<br/>to current tick"]
D["Compare replayed<br/>position with P"]
E{Within tolerance?}
F["Accept position"]
G["Send correction<br/>to client"]
end
A --> B --> C --> D --> E
E -->|Yes| F
E -->|No| G
Technique 3: Entity Interpolation
Client receives entity positions at 20 Hz but renders at 60+ Hz.
Interpolation:
For each visible entity:
prevPos = last received position
currPos = current received position
prevTick = tick of prevPos
currTick = tick of currPos
renderTick = current render tick
alpha = (renderTick - prevTick) / (currTick - prevTick)
alpha = clamp(alpha, 0, 1)
renderedPos = lerp(prevPos, currPos, alpha)
Extrapolation (for fast-moving entities):
If no update received for N ticks:
velocity = (currPos - prevPos) / (currTick - prevTick)
extrapolatedPos = currPos + velocity × (renderTick - currTick)
If extrapolation time > threshold: stop extrapolating, freeze
5.4 Lag Compensation
Hit Detection with Lag Compensation
flowchart TD
A[Attacker fires at target] --> B[Server receives attack packet]
B --> C["Rewind simulation to<br/>attacker's client tick"]
C --> D["Check if target was<br/>in line of sight at that tick"]
D --> E{Valid hit?}
E -->|Yes| F["Apply damage at<br/>current server tick"]
E -->|No| G["Reject hit<br/>(target was behind wall)"]
F --> H["Forward simulation<br/>to current tick"]
G --> H
Implementation Details
History Buffer:
- Each entity maintains position history for last 200ms (4 ticks at 20Hz)
- History entries: { tick, position, rotation, state }
- Memory: 1000 entities × 4 entries × 32 bytes = 128 KB per zone
Hit Validation:
1. Attacker sends: { targetId, targetPos_at_fire_time, fireTick }
2. Server rewinds target to fireTick using history buffer
3. Server checks line of sight from attacker position to rewound target
4. Server checks if target was in valid hitbox
5. If valid: apply damage at current tick
6. If invalid: reject (potential peeker's advantage abuse)
Anti-Abuse:
- Maximum rewind window: 200ms (prevents extreme lag exploitation)
- Clients with high ping (>150ms) get reduced compensation
- Repeated invalid hits → anti-cheat flag
5.5 World Partitioning and Zone Sharding
Zone Design
flowchart TD
subgraph Realm["Realm Server"]
subgraph Continent["Continent"]
Z1["Zone 1:<br/>Starting Area<br/>(1000 players)"]
Z2["Zone 2:<br/>Capital City<br/>(1000 players)"]
Z3["Zone 3:<br/>Wilderness<br/>(1000 players)"]
Z4["Zone 4:<br/>High-Level Zone<br/>(1000 players)"]
end
subgraph Instances["Instances"]
D1["Dungeon 1<br/>(5-10 players)"]
D2["Dungeon 2<br/>(5-10 players)"]
R1["Raid Instance<br/>(20-40 players)"]
A1["Arena Instance<br/>(5v5)"]
B1["Battleground<br/>(15v15)"]
end
Z1 <--> Z2
Z2 <--> Z3
Z3 <--> Z4
Z1 --> D1
Z3 --> D2
Z4 --> R1
end
Zone Transition Flow
1. Player approaches zone border
2. Client sends ZONE_REQUEST to current zone server
3. Current zone server validates request
4. Zone Manager allocates slot in target zone
5. Character state saved (immediate persist)
6. Target zone server loads player data
7. Player removed from current zone
8. Player added to target zone
9. Zone data sent to client (full state)
10. Client transitions (loading screen if needed)
11. Player appears in new zone
Total transition time: < 2 seconds
Cross-Zone Communication
Party members in different zones:
- Chat: Routed through Chat Service (Kafka-based)
- Party invites: Routed through Cross-Zone Router
- Shared objectives: Tracked in World Service
Implementation:
Cross-Zone Router maintains:
- Player → Zone mapping (Redis)
- Party → Player mapping
- Message routing table
When Player A (Zone 1) sends party message:
1. Zone 1 → Cross-Zone Router: party_msg(partyId, from=A, msg)
2. Cross-Zone Router looks up party members' zones
3. Routes to Zone 2, Zone 3, etc.
4. Each zone server delivers to respective player
5.6 Anti-Cheat System
Multi-Layer Defense
flowchart TD
subgraph Layers["Anti-Cheat Layers"]
L1["Layer 1: Client Integrity"]
L2["Layer 2: Input Validation"]
L3["Layer 3: Behavior Analysis"]
L4["Layer 4: Server Authority"]
L5["Layer 5: Post-Game Review"]
end
L1 --> L2 --> L3 --> L4 --> L5
subgraph L1Detail["Client Integrity"]
CI1["Binary integrity check<br/>(hash verification)"]
CI2["Memory scan<br/>(known cheat signatures)"]
CI3["Anti-debugging<br/>(debugger detection)"]
CI4["Anti-injection<br/>(DLL/process scan)"]
end
subgraph L2Detail["Input Validation"]
IV1["Speed check<br/>(max movement speed)"]
IV2["Ability cooldown check"]
IV3["Line of sight check"]
IV4["Range validation"]
end
subgraph L3Detail["Behavior Analysis"]
BA1["Aim pattern analysis<br/>(human vs bot)"]
BA2["Reaction time analysis"]
BA3["Movement pattern analysis"]
BA4["Statistical anomaly detection"]
end
subgraph L4Detail["Server Authority"]
SA1["All damage calculated server-side"]
SA2["All position validation server-side"]
SA3["All inventory changes server-side"]
SA4["All combat resolution server-side"]
end
subgraph L5Detail["Post-Game Review"]
PR1["Combat log analysis"]
PR2["Win rate anomaly detection"]
PR3["Manual review queue"]
PR4["Machine learning classification"]
end
L1Detail -.-> L1
L2Detail -.-> L2
L3Detail -.-> L3
L4Detail -.-> L4
L5Detail -.-> L5
Real-Time Detection Pipeline
Per-tick checks (every 50ms):
1. Speed check: |currentPos - lastPos| <= maxSpeed × deltaTime
2. Terrain check: position is on valid terrain
3. Input rate: client sending ≤ 25 packets/sec (20 Hz + 25% tolerance)
Per-action checks:
1. Ability cooldown: lastCastTime + cooldown <= currentTime
2. Resource check: mana >= abilityCost
3. Target validity: target exists, in range, in line of sight
4. State check: player not dead, not stunned, not in combat lock
Per-session analysis (async):
1. Movement smoothness: human movement has micro-variations
2. Aim precision: perfect aim over 100+ shots → flag
3. Reaction time: sub-100ms consistently → flag
4. Play pattern: 24/7 play without breaks → flag
5.7 Combat System
Ability Resolution Pipeline
flowchart TD
A[Client casts ability] --> B["Server receives<br/>ABILITY_CAST"]
B --> C{Cooldown check}
C -->|On CD| D[Reject: cooldown]
C -->|Ready| E{Resource check}
E -->|Insufficient| F[Reject: not enough mana]
E -->|Sufficient| G{Target validation}
G -->|Invalid| H[Reject: invalid target]
G -->|Valid| I[Apply ability effects]
I --> J["Calculate damage<br/>(attack vs defense)"]
J --> K["Apply modifiers<br/>(crit, dodge, block, resist)"]
K --> L["Apply damage/heal"]
L --> M["Check death conditions"]
M --> N["Broadcast combat event<br/>to all affected players"]
N --> O["Start cooldown"]
O --> P["Publish to Kafka<br/>(logging, analytics)"]
D --> Q[Send error to client]
F --> Q
H --> Q
Damage Calculation
Base Damage = attacker.attack × ability.multiplier
Modifiers:
- Critical hit: × 2.0 (if crit roll succeeds)
- Defense reduction: defender.defense × 0.5
- Resistance: type-specific (fire resist, etc.)
- Buffs: +flat or +% from active buffs
- Debuffs: -flat or -% on target
Final Damage = max(1, Base Damage - Defense Reduction) × Crit × Resistance
Damage Types:
Physical: Reduced by armor
Magic (Fire/Frost/Nature/Holy/Shadow): Reduced by spell resistance
True Damage: Ignored by all defenses
5.8 Matchmaking Service (Dungeons, Raids, PvP)
Queue System
flowchart TD
A[Player queues for dungeon] --> B["Join queue with<br/>role, level, rating"]
B --> C["Wait for group<br/>completion"]
C --> D{Group formed?}
D -->|Yes| E["Create instance"]
D -->|No| F{Timeout?}
F -->|No| C
F -->|Yes| G["Notify: no match found"]
E --> H["Assign players to<br/>instance"]
H --> I["Teleport all players<br/>to instance"]
I --> J["Instance begins"]
Group Formation Algorithm
Queue entry:
{
playerId: uint64,
role: uint8 (tank, healer, dps),
level: uint16,
rating: uint16 (for PvP),
queueTime: uint64,
flexibility: uint8 (main role, off role, any)
}
Matching algorithm:
1. Sort queue by queueTime (oldest first)
2. For each player, find compatible group:
- Role balance: 1 tank, 1-2 healers, rest DPS
- Level range: ±3 levels
- Rating range: ±100 (PvP)
- Latency compatibility: same region preferred
3. Score each potential group:
- Time waited (higher = better)
- Role match quality
- Rating similarity
- Latency similarity
4. Select highest-scoring group
5. Create instance, assign players
6. Remove from queue
6. Data Model
6.1 PostgreSQL Schema
-- Accounts
CREATE TABLE accounts (
account_id BIGSERIAL PRIMARY KEY,
username VARCHAR(64) UNIQUE NOT NULL,
email VARCHAR(255) UNIQUE NOT NULL,
password_hash VARCHAR(255) NOT NULL,
totp_secret VARCHAR(64),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
last_login TIMESTAMP WITH TIME ZONE,
is_active BOOLEAN DEFAULT TRUE,
ban_reason TEXT,
banned_at TIMESTAMP WITH TIME ZONE
);
-- Characters
CREATE TABLE characters (
character_id BIGSERIAL PRIMARY KEY,
account_id BIGINT REFERENCES accounts(account_id) ON DELETE CASCADE,
realm_id INTEGER NOT NULL,
name VARCHAR(32) NOT NULL,
race VARCHAR(32) NOT NULL,
class VARCHAR(32) NOT NULL,
gender CHAR(1) NOT NULL, -- M/F
level INTEGER NOT NULL DEFAULT 1,
experience BIGINT NOT NULL DEFAULT 0,
health INTEGER NOT NULL,
max_health INTEGER NOT NULL,
mana INTEGER NOT NULL,
max_mana INTEGER NOT NULL,
strength INTEGER NOT NULL DEFAULT 10,
agility INTEGER NOT NULL DEFAULT 10,
stamina INTEGER NOT NULL DEFAULT 10,
intellect INTEGER NOT NULL DEFAULT 10,
zone_id SMALLINT NOT NULL,
position_x REAL NOT NULL,
position_y REAL NOT NULL,
position_z REAL NOT NULL,
guild_id BIGINT,
is_online BOOLEAN DEFAULT FALSE,
last_login TIMESTAMP WITH TIME ZONE,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
deleted_at TIMESTAMP WITH TIME ZONE,
UNIQUE(account_id, name, realm_id)
);
CREATE INDEX idx_characters_account ON characters(account_id);
CREATE INDEX idx_characters_realm ON characters(realm_id);
CREATE INDEX idx_characters_online ON characters(is_online) WHERE is_online = TRUE;
-- Inventory (sharded by character_id)
CREATE TABLE inventory (
inventory_id BIGSERIAL PRIMARY KEY,
character_id BIGINT NOT NULL REFERENCES characters(character_id) ON DELETE CASCADE,
slot SMALLINT NOT NULL, -- 0-99
item_template_id INTEGER NOT NULL,
quantity INTEGER NOT NULL DEFAULT 1,
durability INTEGER,
enchantments JSONB, -- [{enchantId, duration}]
socket_gems JSONB, -- [{socketIndex, gemId}]
equipped BOOLEAN DEFAULT FALSE,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
UNIQUE(character_id, slot)
);
CREATE INDEX idx_inventory_character ON inventory(character_id);
-- Item templates
CREATE TABLE item_templates (
template_id INTEGER PRIMARY KEY,
name VARCHAR(128) NOT NULL,
item_type VARCHAR(32) NOT NULL, -- weapon, armor, consumable, quest
item_class VARCHAR(32), -- sword, shield, potion, etc.
quality SMALLINT NOT NULL, -- 0=poor, 1=common, 2=uncommon, 3=rare, 4=epic, 5=legendary
level_req INTEGER NOT NULL DEFAULT 1,
class_req VARCHAR(32),
stats JSONB, -- {strength: 10, agility: 5}
description TEXT,
icon_id INTEGER,
value INTEGER NOT NULL DEFAULT 0 -- copper
);
-- Quests
CREATE TABLE quest_templates (
quest_id INTEGER PRIMARY KEY,
name VARCHAR(128) NOT NULL,
description TEXT,
zone_id SMALLINT,
level_req INTEGER,
prerequisites JSONB, -- [questId1, questId2]
objectives JSONB, -- [{type: 'kill', target: 'wolf', count: 10}]
rewards JSONB, -- {xp: 1000, gold: 500, items: [{templateId, qty}]}
repeatable BOOLEAN DEFAULT FALSE
);
CREATE TABLE quest_progress (
character_id BIGINT REFERENCES characters(character_id) ON DELETE CASCADE,
quest_id INTEGER REFERENCES quest_templates(quest_id),
status VARCHAR(16) NOT NULL, -- active, completed, failed
objectives JSONB, -- Current objective progress
started_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
completed_at TIMESTAMP WITH TIME ZONE,
PRIMARY KEY (character_id, quest_id)
);
-- Skills / Talents
CREATE TABLE skill_templates (
skill_id INTEGER PRIMARY KEY,
name VARCHAR(64) NOT NULL,
class VARCHAR(32), -- NULL = all classes
description TEXT,
icon_id INTEGER,
cast_time INTEGER NOT NULL DEFAULT 0, -- ms
cooldown INTEGER NOT NULL DEFAULT 0, -- ms
range REAL NOT NULL,
mana_cost INTEGER NOT NULL DEFAULT 0,
damage_formula TEXT, -- "attack * 1.5 + intellect * 0.5"
effect_type VARCHAR(32), -- damage, heal, buff, debuff, cc
max_level INTEGER NOT NULL DEFAULT 1
);
CREATE TABLE character_skills (
character_id BIGINT REFERENCES characters(character_id) ON DELETE CASCADE,
skill_id INTEGER REFERENCES skill_templates(skill_id),
level INTEGER NOT NULL DEFAULT 1,
points_spent INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (character_id, skill_id)
);
-- Guilds
CREATE TABLE guilds (
guild_id BIGSERIAL PRIMARY KEY,
name VARCHAR(64) UNIQUE NOT NULL,
realm_id INTEGER NOT NULL,
leader_id BIGINT REFERENCES characters(character_id),
motto VARCHAR(256),
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
level INTEGER NOT NULL DEFAULT 1,
experience BIGINT NOT NULL DEFAULT 0
);
CREATE TABLE guild_members (
guild_id BIGINT REFERENCES guilds(guild_id) ON DELETE CASCADE,
character_id BIGINT REFERENCES characters(character_id) ON DELETE CASCADE,
rank VARCHAR(32) NOT NULL, -- leader, officer, member
joined_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
PRIMARY KEY (guild_id, character_id)
);
-- Auction House
CREATE TABLE auctions (
auction_id BIGSERIAL PRIMARY KEY,
seller_id BIGINT REFERENCES characters(character_id),
item_template_id INTEGER REFERENCES item_templates(template_id),
quantity INTEGER NOT NULL,
buyout_price INTEGER, -- copper (NULL = no buyout)
starting_bid INTEGER NOT NULL, -- copper
current_bid INTEGER NOT NULL,
bidder_id BIGINT REFERENCES characters(character_id),
expires_at TIMESTAMP WITH TIME ZONE NOT NULL,
status VARCHAR(16) DEFAULT 'active', -- active, sold, expired, cancelled
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
CREATE INDEX idx_auctions_status ON auctions(status, expires_at);
CREATE INDEX idx_auctions_item ON auctions(item_template_id);
-- PvP Ratings
CREATE TABLE pvp_ratings (
character_id BIGINT PRIMARY KEY REFERENCES characters(character_id) ON DELETE CASCADE,
bracket VARCHAR(32) NOT NULL, -- arena_2v2, arena_3v3, arena_5v5, rated_bgg
rating INTEGER NOT NULL DEFAULT 1500,
wins INTEGER NOT NULL DEFAULT 0,
losses INTEGER NOT NULL DEFAULT 0,
draws INTEGER NOT NULL DEFAULT 0,
weekly_wins INTEGER NOT NULL DEFAULT 0,
season_wins INTEGER NOT NULL DEFAULT 0,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
-- Anti-cheat flags
CREATE TABLE anticheat_flags (
flag_id BIGSERIAL PRIMARY KEY,
account_id BIGINT REFERENCES accounts(account_id),
character_id BIGINT REFERENCES characters(character_id),
flag_type VARCHAR(32) NOT NULL, -- speed_hack, aimbot, memory_edit, etc.
severity VARCHAR(16) NOT NULL, -- low, medium, high, critical
evidence JSONB,
auto_action VARCHAR(32), -- none, warn, kick, temp_ban, perm_ban
reviewed_by VARCHAR(64),
reviewed_at TIMESTAMP WITH TIME ZONE,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
6.2 Redis Data Structures
# Hot game state per zone
Key: zone:{realmId}:{zoneId}:entities
Type: Hash
Fields: {entityId}: {position, rotation, health, state, owner}
TTL: None (active zone)
# Player session
Key: session:{characterId}
Type: Hash
Fields: realmId, zoneId, gatewayConnId, zoneServerConnId, lastTick, position
TTL: 120s (renewed every tick)
# Player → Zone mapping (for cross-zone routing)
Key: player:zone:{characterId}
Type: String
Value: {realmId}:{zoneId}
TTL: 300s
# Zone player count
Key: zone:count:{realmId}:{zoneId}
Type: String
Value: player_count
TTL: None
# Entity position history (for lag compensation)
Key: entity:history:{entityId}
Type: List
Value: [{tick, position, rotation}, ...] (last 4 entries)
TTL: 1s
# Matchmaking queues
Key: mm:queue:{instanceType}
Type: Sorted Set
Score: queue_timestamp
Value: {characterId}:{role}:{level}:{rating}
# Rate limiting
Key: ratelimit:{accountId}:{action}
Type: Sorted Set
Score: timestamp
Value: request_id
# Chat channels
Key: chat:channel:{channelId}
Type: List
Value: [{senderId, message, timestamp}, ...] (last 100 messages)
TTL: 5min
# Leaderboards
Key: leaderboard:{type}:{realmId}
Type: Sorted Set
Score: rating / score
Value: characterId
6.3 MongoDB Schema (Chat, Logs, Events)
// Chat messages
{
_id: ObjectId,
channelId: "party_12345",
channelType: "party",
senderId: 98765,
senderName: "PlayerA",
message: "Need a healer!",
timestamp: ISODate("2026-08-28T10:30:00Z"),
realmId: 1
}
// Combat logs
{
_id: ObjectId,
realmId: 1,
zoneId: 42,
serverTick: 12345678,
timestamp: ISODate("2026-08-28T10:30:00Z"),
events: [
{
type: "damage",
attackerId: 111,
defenderId: 222,
abilityId: 101,
damage: 1500,
damageType: "physical",
critical: true
}
]
}
// World events
{
_id: ObjectId,
eventId: "world_boss_spawn",
realmId: 1,
zoneId: 15,
position: { x: 1234.5, y: 567.8, z: 12.3 },
startTime: ISODate("2026-08-28T12:00:00Z"),
endTime: ISODate("2026-08-28T12:30:00Z"),
participants: [111, 222, 333],
outcome: "defeated"
}
7. Key Design Decisions
7.1 TCP vs UDP for Game Transport
| Factor | TCP | UDP (with reliability layer) |
|---|---|---|
| Ordering | Guaranteed | Manual (sequence numbers) |
| Reliability | Guaranteed | Selective (only critical packets) |
| Latency | Head-of-line blocking | No blocking |
| Congestion control | Built-in | Custom |
| NAT traversal | Easier | Harder |
| Decision | — | UDP with custom reliability |
Rationale: Real-time games cannot tolerate head-of-line blocking. A lost packet shouldn’t delay all subsequent packets. We implement:
- Unreliable channel: Position updates, camera (latest wins, old packets discarded)
- Reliable channel: Combat events, inventory changes, chat (ordered, retransmitted)
- Sequenced channel: State updates (ordered but not retransmitted)
7.2 Server Authority Model
| Aspect | Client Authority | Server Authority | Hybrid |
|---|---|---|---|
| Anti-cheat | Weak | Strong | Medium |
| Latency feel | Best | Good (with prediction) | Good |
| Complexity | Low | High | Medium |
| Fairness | Poor | Excellent | Good |
| Decision | — | — | Server Authority + Client Prediction |
Rationale: Server authority is non-negotiable for competitive integrity. Client prediction makes the experience feel responsive despite server authority.
7.3 Zone Sharding Strategy
| Strategy | Pros | Cons |
|---|---|---|
| Static zones | Simple, predictable load | Hot zones overflow, cold zones waste resources |
| Dynamic sharding | Optimal resource use | Complex migration, state transfer |
| Decision | — | Static zones with overflow instances |
Implementation:
- Popular zones (capital cities) get multiple instances
- Players can choose instance or auto-assign to least populated
- Cross-realm technology merges low-population realms
- Dungeon/raid instances are always dynamic
7.4 Save Strategy
| Strategy | Pros | Cons |
|---|---|---|
| Save on every change | Zero data loss | High I/O, performance impact |
| Periodic save | Low I/O | Data loss on crash |
| Decision | — | Hybrid: periodic + event-triggered |
Implementation:
- Auto-save every 60 seconds (async, non-blocking)
- Immediate save on: zone transition, death, logout, inventory change
- Save to Kafka first (WAL), then PostgreSQL
- Redis holds hot state between saves
7.5 Database Sharding
Sharding key: character_id
Shard 1: character_id % N = 0 (inventory, equipment)
Shard 2: character_id % N = 1 (quests, achievements)
...
Benefits:
- Even distribution
- Character data co-located
- Horizontal scalability
Challenges:
- Cross-shard transactions (avoided by design)
- Global queries (use search index / materialized views)
8. Scalability Strategy
8.1 Scaling Dimensions
flowchart TD
subgraph Scale["Scaling Levers"]
A["More Zone Servers<br/>(add zone instances)"]
B["More Realms<br/>(horizontal world scaling)"]
C["Database Sharding<br/>(more PG shards)"]
D["Redis Cluster Expansion<br/>(more shards)"]
E["Kafka Partitions<br/>(more throughput)"]
F["Gateway Instances<br/>(more connections)"]
end
A -->|"Handles more players per zone"| B
B -->|"Handles more total players"| C
C -->|"Handles more data"| D
8.2 Capacity Planning
| Component | Per-Instance Capacity | Instances (500K players) |
|---|---|---|
| Zone Server | 1,000 players | 500 zone servers |
| Realm Server | 50,000 players | 10 realm servers |
| Game Gateway | 50,000 connections | 10 gateways |
| Redis Node | 10M keys, 100K ops/sec | 50 nodes (cluster) |
| Kafka Broker | 100K msg/sec | 20 brokers, 200 partitions |
| PostgreSQL | 10K writes/sec | 1 primary + 10 shards + 5 replicas |
| MongoDB | 50K writes/sec | 3-node replica set |
8.3 Auto-Scaling
Zone Servers:
- Player count > 800 → create overflow instance
- Player count < 200 → merge with adjacent instance
- CPU > 80% for 1 min → scale up
Gateways:
- Connection count > 40K → add gateway
- CPU > 70% → add gateway
Database:
- Write latency > 10ms → add shard
- Connection pool > 80% → add replica
8.4 Multi-Region Deployment
Region 1 (US-East) — Primary:
- 20 realms, 100K players
- Primary databases, primary Redis
- Full game services
Region 2 (EU-West):
- 15 realms, 75K players
- Independent databases
- Full game services
Region 3 (APAC):
- 10 realms, 50K players
- Independent databases
- Full game services
Region 4 (US-West):
- 5 realms, 25K players
- Shared DB with US-East (low latency)
Cross-region:
- No cross-region gameplay (latency too high)
- Cross-region leaderboards (aggregated)
- Cross-region chat (optional, with latency warning)
8.5 Cross-Realm Technology
When realm population drops below threshold:
1. Select compatible realms (same patch, same language)
2. Merge player directories (cross-realm character list)
3. Enable cross-realm groups
4. Enable cross-realm PvP queues
5. Keep inventory/quest data separate per realm
Implementation:
- Cross-realm directory service (Redis-backed)
- Player appears with realm tag: "PlayerA [Realm1]"
- Group formation validates cross-realm compatibility
- Combat remains within same realm (data isolation)
9. Curveball Scenarios
Curveball 1: “A zone server experiences a sudden spike in entity count (10× normal) due to a world event. The tick rate drops from 20 Hz to 5 Hz. Players experience severe lag. How do you handle this?”
Analysis: World events (world boss spawns, invasions, natural disasters) can dramatically increase entity counts. A zone designed for 1,000 players and 10,000 entities suddenly handles 100,000 entities.
Immediate Response:
Phase 1: Graceful Degradation (0-5 seconds)
1. Reduce tick rate target: 20 Hz → 10 Hz → 5 Hz
2. Reduce simulation fidelity:
- Skip non-critical AI updates
- Reduce physics precision
- Disable decorative entity updates
3. Reduce network bandwidth:
- Increase entity update interval (every 2 ticks → every 4 ticks)
- Reduce position precision (float digits)
- Stop sending non-visible entity updates
4. Notify clients: "High population event — reduced update rate"
Phase 2: Load Shedding (5-30 seconds)
1. Freeze non-essential systems:
- Pause chat processing
- Pause auction house updates
- Pause quest progress tracking
2. Entity culling:
- Despawn non-essential NPCs temporarily
- Reduce creature spawn rates
- Merge nearby entities visually
3. Client-side compensation:
- Increase interpolation window
- Enable client-side entity LOD (level of detail)
Phase 3: Horizontal Scaling (30-120 seconds)
1. Spawn overflow zone instance
2. Begin migrating players to overflow:
- New players → overflow instance
- Gradual migration of existing players
3. Split entity processing:
- Dedicated AI server for NPC processing
- Dedicated physics server for collision
Prevention:
1. Predictive scaling:
- World event schedule known in advance
- Pre-scale zone servers 10 minutes before event
- Reserve capacity for expected spikes
2. Entity budget system:
- Each zone has max entity budget
- World events reserve budget in advance
- Regular spawns reduced when event active
3. Tiered simulation:
- Critical entities (players, combat targets): full simulation
- Important entities (nearby NPCs): reduced simulation
- Background entities (distant NPCs): minimal simulation
4. Client-side assistance:
- Clients run simplified AI for distant NPCs
- Server validates critical actions only
- Reduces server compute by 30-50%
Curveball 2: “A sophisticated cheat modifies the client’s memory to grant unlimited mana and bypass ability cooldowns. The cheat sends valid-looking packets. How do you detect and prevent this?”
Analysis: Memory modification cheats are among the most dangerous because they don’t modify network packets — they modify the client’s internal state, which then generates “valid” packets from the client’s perspective.
Detection Layers:
Layer 1: Server-Side Validation (Primary Defense)
┌─────────────────────────────────────────────┐
│ Every ability cast is validated server-side: │
│ │
│ 1. Check actual mana (server-authoritative) │
│ → Client claims 1000 mana, server says │
│ player has 50 mana → REJECT │
│ │
│ 2. Check actual cooldown (server-side timer) │
│ → Client casts fireball, server cooldown │
│ not expired → REJECT │
│ │
│ 3. Check resource costs server-side │
│ → All costs validated against server │
│ state, not client claims │
└─────────────────────────────────────────────┘
Layer 2: State Consistency Checks
┌─────────────────────────────────────────────┐
│ Periodic state reconciliation: │
│ │
│ 1. Server sends periodic state snapshot │
│ → Client must acknowledge │
│ 2. If client state diverges significantly │
│ → Flag for review │
│ 3. Mana regeneration rate check │
│ → If player gains mana faster than max │
│ regen rate → impossible → flag │
└─────────────────────────────────────────────┘
Layer 3: Client Integrity Verification
┌─────────────────────────────────────────────┐
│ Anti-tamper measures: │
│ │
│ 1. Memory integrity checks (periodic) │
│ → Hash critical memory regions │
│ → Compare with known good values │
│ 2. Anti-debugging │
│ → Detect attached debuggers │
│ → Detect memory scanning tools │
│ 3. Code signature verification │
│ → Verify client binary hasn't been │
│ modified │
│ 4. Obfuscation │
│ → Critical values stored obfuscated │
│ → Mana, cooldowns not in plain memory │
└─────────────────────────────────────────────┘
Layer 4: Behavioral Analysis
┌─────────────────────────────────────────────┐
│ Statistical detection: │
│ │
│ 1. Ability usage rate analysis │
│ → If player casts abilities faster than │
│ possible with cooldowns → flag │
│ 2. Damage output analysis │
│ → If DPS exceeds theoretical maximum │
│ for class/level/gear → flag │
│ 3. Resource consumption analysis │
│ → If mana never depletes during combat │
│ → impossible → flag │
└─────────────────────────────────────────────┘
Response Protocol:
Detection → Action Matrix:
Confidence Level Action
─────────────────────────────────────────────
Low (30-50%) Silent monitoring, collect evidence
Medium (50-70%) Kick from game, require relaunch
High (70-90%) Temporary ban (1-7 days), review queue
Critical (90%+) Permanent ban, hardware ID ban, IP flag
Evidence Collection:
- Record all packets from suspected client
- Record server-side state at time of detection
- Record behavioral metrics (DPS, ability rate, etc.)
- Store for manual review and ML training
Key Principle: The server is the ultimate authority. The client is an untrusted input device. No client claim about game state is ever trusted — everything is validated against server-authoritative state.
Curveball 3: “You need to perform a zero-downtime game patch that changes the combat formula. Active players are mid-combat. How do you roll out the patch without disrupting gameplay?”
Analysis: Game patches that change core mechanics (combat formulas, ability balances, economy) cannot simply restart servers — active players would be disconnected mid-combat. The patch must be rolled out seamlessly.
Rollout Strategy:
flowchart TD
A["Prepare patch<br/>(new binary, config)"] --> B["Deploy to canary<br/>realm (1 realm)"]
B --> C{"Canary stable?<br/>30 min observation"}
C -->|No| D["Rollback canary"]
C -->|Yes| E["Batch 1:<br/>10% of realms"]
E --> F{"Batch 1 stable?<br/>15 min observation"}
F -->|No| G["Rollback batch 1<br/>Investigate"]
F -->|Yes| H["Batch 2:<br/>50% of realms"]
H --> I{"Batch 2 stable?<br/>15 min observation"}
I -->|No| J["Rollback batch 2"]
I -->|Yes| K["Batch 3:<br/>Remaining realms"]
K --> L["Patch complete"]
Technical Implementation:
Step 1: Configuration-Based Combat Formulas
- Combat formulas are NOT hardcoded
- Stored in configuration files / database
- Zone servers load formulas at startup
- Formulas can be updated without binary patch
Example config:
{
"damage_formula": {
"physical": "max(1, attacker.attack * ability.multiplier - defender.armor * 0.5)",
"magic": "max(1, attacker.spellPower * ability.multiplier * (1 - defender.resistance))",
"crit_chance": "attacker.critRating / 100",
"crit_multiplier": 2.0
}
}
Step 2: Hot-Reload Configuration
- Zone servers watch for config changes
- On change: validate new config, apply atomically
- Active combat uses current formula (no mid-fight change)
- New combat encounters use new formula
Step 3: Binary Patch (if code changes needed)
- Blue-green deployment for zone servers:
a. Deploy new version alongside old version
b. New version handles new connections only
c. Old version continues serving existing players
d. As players log out, they reconnect to new version
e. When old version has 0 players, shut it down
- Graceful transition:
- Players mid-combat: continue on old version
- Combat ends → player returned to zone on new version
- No mid-combat disruption
Step 4: Client Patch
- Client patch required for UI / asset changes
- Launcher enforces patch before login
- Grace period: old clients can play for N hours
- After grace period: old clients rejected
Edge Cases:
Q: What if a player is in a 40-minute raid when the patch deploys?
A: They continue on the old version. After the raid, they're
transferred to the new version. The raid instance runs to completion.
Q: What if the new formula causes a bug?
A: Canary detection catches it. Rollback reverts config to previous
version. New config is rejected, old config reloaded.
Q: What about cross-version compatibility?
A: Players on different versions cannot group together.
Cross-version play is blocked until all players are on the same version.
Curveball 4: “Two zone servers lose connectivity to each other (network partition). Players in both zones are in the same party. What happens and how do you handle it?”
Analysis: Network partitions between zone servers break cross-zone communication. Party members in different zones lose the ability to chat, coordinate, and see each other’s status.
Impact Assessment:
| Feature | Impact | Severity |
|---|---|---|
| Party chat | Broken | Medium |
| Party status visibility | Stale | Low |
| Cross-zone quests | Blocked | Medium |
| Group formation | Blocked | Low |
| Active combat | Unaffected (same zone) | None |
| Zone transition | Blocked across partition | High |
Handling Strategy:
Phase 1: Detection (0-5 seconds)
- Heartbeat between zone servers fails
- Cross-Zone Router detects partition
- Affected parties flagged
Phase 2: Graceful Degradation (5-30 seconds)
- Party chat: Queue messages, deliver when partition heals
- Party status: Show "connection unstable" indicator
- Zone transitions across partition: Block with message
- Active cross-zone quests: Pause, resume later
Phase 3: Player Communication (30-60 seconds)
- Notify affected players: "Network issue — some features limited"
- Provide ETA if known
- Suggest alternatives (stay in current zone)
Phase 4: Recovery (partition heals)
- Replay queued messages
- Resume paused quests
- Restore full functionality
- Log incident for analysis
Phase 5: If partition persists > 5 minutes
- Consider merging affected zones (if geographically close)
- Offer teleport options to reunite parties
- Escalate to network team
Prevention:
1. Redundant network paths between zone servers
2. Zone servers co-located in same availability zone
3. Cross-zone communication via Kafka (survives transient failures)
4. Automatic failover to backup network path
5. Regular network partition drills
Curveball 5: “You discover that your entity interpolation causes ‘rubber-banding’ for players with high ping (>150ms). How do you fix this without changing the tick rate?”
Analysis: Rubber-banding occurs when the client’s predicted position diverges significantly from the server’s authoritative position, causing the client to be “snapped back” repeatedly. This is especially problematic for high-ping players.
Root Cause Analysis:
High-ping player scenario:
- Client predicts movement for 150ms (round-trip time)
- Server receives stale input, corrects position
- Client sees correction, snaps back
- Client predicts again, diverges again
- Cycle repeats → rubber-banding
Key factors:
1. Prediction horizon too long (150ms of prediction)
2. Server correction too aggressive (full snap)
3. No smoothing on corrections
4. Client input buffering too short
Solutions:
Solution 1: Extended Client Input Buffering
- Server accepts inputs with timestamps up to 300ms in the past
- Server replays from last agreed state, not current state
- Reduces correction magnitude significantly
Solution 2: Smooth Correction (No Hard Snap)
- Instead of snapping to server position:
correctionSpeed = divergence / smoothingTime
clientPosition += correctionSpeed * deltaTime
- Smoothing time: 200-500ms (configurable by ping)
- High-ping players get longer smoothing
Solution 3: Server-Side Client Prediction
- Server maintains predicted state for each client
- Server compares prediction with actual
- Server sends correction deltas, not absolute positions
- Smaller deltas = less rubber-banding
Solution 4: Adaptive Tick Rate per Client
- Low ping (< 50ms): 20 Hz updates
- Medium ping (50-100ms): 15 Hz updates
- High ping (100-150ms): 10 Hz updates
- Very high ping (> 150ms): 5 Hz + extrapolation hints
- Reduces correction frequency for high-ping players
Solution 5: Position Smoothing on Client
- Client receives server position
- Instead of using directly: smoothPosition = lerp(smoothPosition, serverPosition, alpha)
- Alpha decreases with higher ping
- Creates illusion of smooth movement
Combined Approach:
1. Extended input buffering (server)
2. Smooth corrections (client)
3. Adaptive update rate (server)
4. Position smoothing (client)
5. Extrapolation hints for fast movers (server)
Implementation Details:
Server-side changes:
- Input buffer: 300ms (was 100ms)
- Correction smoothing: send delta, not absolute
- Adaptive tick: per-client update rate based on ping
Client-side changes:
- Position smoothing: lerp with ping-based alpha
- Extended prediction: use server hints for extrapolation
- Visual compensation: slight FOV increase for high-ping players
Testing:
- Simulate various ping levels (50ms, 100ms, 150ms, 200ms)
- Measure rubber-banding frequency
- Measure position accuracy
- Player feedback on movement feel
10. Failure Modes and Mitigations
10.1 Failure Mode Matrix
| Component | Failure Mode | Impact | Detection | Mitigation | RTO |
|---|---|---|---|---|---|
| Zone Server | Crash | Zone goes dark, players disconnected | Health check, heartbeat | Auto-restart, restore from Redis | < 30s |
| Zone Server | Tick rate drop | Lag, rubber-banding | Tick rate monitoring | Graceful degradation, load shed | < 10s |
| Zone Server | Memory leak | Gradual degradation, OOM crash | Memory monitoring | Rolling restart, memory limits | < 5 min |
| Game Gateway | Connection flood | New connections rejected | Connection rate alert | Rate limiting, auto-scale | < 1 min |
| Redis | Node failure | Hot state lost for shard | Sentinel alert | Failover to replica | < 10s |
| Redis | Cluster partition | Split-brain, data inconsistency | Quorum check | Pause writes, resolve | < 1 min |
| PostgreSQL | Primary failure | Character saves blocked | Connection errors | Patroni failover | < 30s |
| PostgreSQL | Replication lag | Stale character data | Lag monitoring | Route to primary | < 1 min |
| Kafka | Broker failure | Event loss | Consumer lag | Replica failover | < 30s |
| Network | Zone partition | Cross-zone features broken | Heartbeat miss | Queue messages, notify | Variable |
| Network | DDoS attack | Service degradation | Traffic anomaly | WAF, rate limiting, scrubbing | < 5 min |
| Client | Cheat detected | Fairness compromised | Anti-cheat flags | Kick, ban, review | Immediate |
| Database | Disk full | Writes fail | Disk usage alert | Emergency cleanup, expand | < 5 min |
| Config | Bad deploy | Game-breaking bug | Error rate spike | Rollback, canary detection | < 2 min |
10.2 Critical Path: Game Tick Failure Analysis
flowchart TD
A["Tick starts"] --> B{"Input collection<br/>complete?"}
B -->|Timeout| C["Use last known inputs<br/>continue tick"]
B -->|Yes| D["Physics simulation"]
D --> E{"Physics complete<br/>in time?"}
E -->|No| F["Reduce simulation<br/>precision, continue"]
E -->|Yes| G["Combat resolution"]
G --> H{"Combat complete<br/>in time?"}
H -->|No| I["Defer non-critical<br/>combat, continue"]
H -->|Yes| J["Network broadcast"]
J --> K{"Broadcast complete<br/>in time?"}
K -->|No| L["Drop non-critical<br/>updates, continue"]
K -->|Yes| M["Tick complete"]
C --> D
F --> G
I --> G
L --> M
M --> N{"Tick took<br/>> 50ms?"}
N -->|Yes| O["Alert: tick overrun<br/>log details"]
N -->|No| P["Normal operation"]
10.3 Disaster Recovery Plan
RPO (Recovery Point Objective):
- Game state: 50ms (last tick in Redis)
- Character data: 60s (auto-save interval)
- Inventory: 0 (immediate save on changes)
- Chat: 5 min (Kafka retention)
RTO (Recovery Time Objective):
- Zone server: 30 seconds (auto-restart + Redis restore)
- Game gateway: 1 minute (auto-scale replacement)
- Redis cluster: 10 seconds (sentinel failover)
- PostgreSQL: 30 seconds (Patroni failover)
- Full realm: 2 minutes (cascade restart)
- Full region: 10 minutes (cross-region failover)
Backup Strategy:
- PostgreSQL: Continuous WAL archiving + daily full backup
- Redis: AOF everysec + RDB every 15 min
- Kafka: 7-day retention (replayable)
- MongoDB: Daily snapshot + oplog
- S3: Versioned backups of all critical data
Recovery Procedures:
1. Zone server crash:
a. Kubernetes auto-restarts pod
b. New pod connects to Redis, loads zone state
c. Players reconnect automatically (client reconnect logic)
d. Missing players receive "server restart" notification
2. Redis failure:
a. Sentinel promotes replica to master
b. Zone servers reconnect to new master
c. If no replica: restore from AOF / Kafka replay
d. Players may experience brief disconnect
3. PostgreSQL failure:
a. Patroni promotes replica to primary
b. Services reconnect to new primary
c. Character saves resume
d. If data loss: restore from WAL archive
4. Full region failure:
a. DNS failover to secondary region
b. Players reconnect to secondary region
c. Data synced from cross-region replication
d. Estimated downtime: 5-10 minutes
10.4 Monitoring and Alerting
| Metric | Warning | Critical | Action |
|---|---|---|---|
| Tick rate | < 18 Hz | < 10 Hz | Scale zone server |
| Tick duration p99 | > 45ms | > 60ms | Investigate, optimize |
| Player count per zone | > 800 | > 950 | Create overflow instance |
| Server CPU | > 70% | > 90% | Auto-scale |
| Server memory | > 80% | > 95% | Restart, investigate leak |
| Network latency (internal) | > 5ms | > 20ms | Network investigation |
| Redis latency p99 | > 1ms | > 5ms | Scale Redis |
| Redis memory | > 75% | > 90% | Add shard |
| PG write latency | > 5ms | > 20ms | Scale DB, check queries |
| PG replication lag | > 5s | > 30s | Route to primary |
| Kafka consumer lag | > 10K | > 100K | Add consumers |
| Connection error rate | > 0.5% | > 2% | Check gateway health |
| Anti-cheat flags/min | > 10 | > 50 | Investigate attack |
| Packet loss rate | > 0.1% | > 1% | Network investigation |
| Client disconnect rate | > 1%/min | > 5%/min | Service health check |
10.5 Chaos Engineering Plan
Regular chaos experiments:
1. Kill random zone server → verify auto-restart + player reconnect
2. Kill Redis node → verify failover + state recovery
3. Simulate network partition → verify graceful degradation
4. Inject 200ms latency → verify rubber-banding mitigation
5. Simulate DDoS → verify rate limiting + auto-scale
6. Kill PostgreSQL primary → verify Patroni failover
7. Corrupt Redis data → verify Kafka replay recovery
8. Deploy bad config → verify canary detection + rollback
Frequency: Weekly automated, monthly full-scale drill
Appendix A: Glossary
| Term | Definition |
|---|---|
| Tick | One iteration of the game loop (50ms at 20 Hz) |
| Entity | Any object in the game world (player, NPC, projectile, effect) |
| Interpolation | Smoothing entity movement between received positions |
| Extrapolation | Predicting entity position when no update received |
| Rubber-banding | Client position snapped back by server correction |
| Client Prediction | Client predicts outcome before server confirmation |
| Server Reconciliation | Server corrects client prediction discrepancies |
| Lag Compensation | Rewinding simulation to validate actions at client’s perceived time |
| Zone | A partitioned area of the game world managed by one server |
| Realm | A complete game world with multiple zones |
| Instance | A private copy of a zone (dungeons, raids, arenas) |
| Delta Encoding | Sending only changes since last update, not full state |
| Head-of-Line Blocking | TCP behavior where lost packet blocks all subsequent packets |
| FOW | Forward-Only Write — write-once, append-only data model |
Appendix B: Technology Stack Summary
| Layer | Technology | Rationale |
|---|---|---|
| Game Client | Unity / Unreal Engine | Mature game engines, cross-platform |
| Game Server | C++ / Rust | Performance-critical, low-level control |
| Network | Custom UDP protocol | No head-of-line blocking, selective reliability |
| Gateway | Go / Rust | High-performance networking |
| Microservices | Go / Java | Performance + ecosystem |
| Game State | Redis Cluster | Sub-millisecond access, in-memory |
| Event Bus | Apache Kafka | Durable, scalable streaming |
| Primary DB | PostgreSQL | ACID, mature, JSONB support |
| Document DB | MongoDB | Chat, logs, flexible schema |
| Cache | Redis | Sessions, leaderboards, hot data |
| Object Storage | S3 / MinIO | Backups, assets |
| Orchestration | Kubernetes | Auto-scaling, self-healing |
| Monitoring | Prometheus + Grafana | Metrics, alerting |
| Tracing | Jaeger | Distributed tracing |
| Log Aggregation | ELK Stack | Centralized logging |
| CI/CD | Jenkins / GitLab CI | Automated deployment |
Appendix C: Protocol Wire Format Details
Transport: UDP (custom reliability layer)
Packet Header (10 bytes):
Byte 0: Protocol version (1)
Bytes 1-4: Sequence number (uint32, big-endian)
Bytes 5-6: Message type (uint16, big-endian)
Bytes 7-8: Payload length (uint16, big-endian)
Byte 9: Flags (reliable, ack, fragment)
Reliability Layer:
- Reliable packets: Assigned seq number, retransmitted if no ACK
- Unreliable packets: Fire-and-forget, no retransmission
- Sequenced packets: Ordered delivery, no retransmission
- ACK packets: Acknowledge received seq numbers
- NAK packets: Request retransmission of missing seq numbers
Fragmentation:
- Max payload: 1400 bytes (MTU safe)
- Large messages split into fragments
- Fragment header: {fragmentId, fragmentIndex, totalFragments}
- Reassembly on receiving end
Encryption:
- DTLS 1.3 for transport encryption
- AES-256-GCM for payload encryption
- Key exchange via SRTP