Real-Time GamingQuestion #14ADVANCED

World of Warcraft: Real-Time Multiplayer Sync, Lag Compensation & Sharding

Massive multiplayer real-time virtual world architecture featuring tick-rate game loops, client-side dead reckoning, spatial entity grid interest management, and seamless zone sharding.

Category: Category 9: Real-Time Gaming•Updated: 2026-08-30
#Gaming#Real-time Sync#Lag Compensation#Zone Sharding#Dead Reckoning

Design a Real-Time Gaming Service like WarCraft

Author: System Design Expert | Date: 2026-08-28 Scope: End-to-end architecture for a real-time multiplayer game service supporting persistent worlds, real-time combat, and thousands of concurrent players per zone.


Table of Contents

  1. Requirements Clarification
  2. Back-of-Envelope Calculations
  3. System Interface Design
  4. High-Level Architecture
  5. Core Component Design
  6. Data Model
  7. Key Design Decisions
  8. Scalability Strategy
  9. Curveball Scenarios
  10. Failure Modes and Mitigations

1. Requirements Clarification

1.1 Functional Requirements

ID Requirement Priority
FR-01 Persistent world with zones/instances P0
FR-02 Real-time player movement and action synchronization P0
FR-03 Real-time combat system (melee, ranged, abilities) P0
FR-04 Server-authoritative game state with anti-cheat P0
FR-05 Zone-based world partitioning (sharding) P0
FR-06 Entity interpolation and extrapolation for smooth movement P0
FR-07 Party/group system with cross-zone support P1
FR-08 Guild/clan system with persistent data P1
FR-09 Quest system with state tracking P1
FR-10 Inventory and item management P1
FR-11 Player vs Environment (PvE) dungeons and raids P1
FR-12 Player vs Player (PvP) arenas and battlegrounds P1
FR-13 Chat system (say, party, guild, trade, whisper) P1
FR-14 Auction house / player trading P2
FR-15 Character progression (levels, skills, talents) P2
FR-16 World events and dynamic content P2
FR-17 Cross-realm / cross-server play P2
FR-18 Spectator / broadcast system for PvP P3

1.2 Non-Functional Requirements

ID Requirement Target
NFR-01 Game tick rate 20 Hz (50ms tick interval)
NFR-02 Client-to-server latency < 100ms p95 (same region)
NFR-03 Motion smoothness Jitter < 16ms (60fps equivalent)
NFR-04 Zone capacity 1,000 players per zone
NFR-05 World capacity 50,000 players per realm/server
NFR-06 Availability 99.9% (planned maintenance windows only)
NFR-07 State consistency Strong consistency within zone, eventual across zones
NFR-08 Anti-cheat Real-time detection, zero-tolerance policy
NFR-09 Save frequency Auto-save every 60s, immediate on zone transition
NFR-10 Scalability Support 1M concurrent players globally
NFR-11 Lag compensation Sub-50ms compensation window
NFR-12 Data durability Zero character data loss

2. Back-of-Envelope Calculations

2.1 Assumption Sets

Three assumption sets for capacity planning:

Parameter Conservative Base Aggressive
Concurrent players 100K 500K 1M
Players per zone 500 1,000 2,000
Zones per realm 20 50 100
Realms/servers 10 50 100
Game tick rate 20 Hz 20 Hz 20 Hz
Entities per player (visible) 50 100 150
Packet size per tick (outbound) 200 bytes 400 bytes 600 bytes
Packet size per tick (inbound) 50 bytes 80 bytes 100 bytes
Avg session duration 1 hour 2 hours 3 hours
Character data size 50 KB 100 KB 200 KB

2.2 Network Bandwidth (Base Case)

Per player per tick:
  Inbound (client → server):  80 bytes × 20 Hz = 1,600 B/s = 12.8 Kbps
  Outbound (server → client): 400 bytes × 20 Hz = 8,000 B/s = 64 Kbps

Per zone (1,000 players):
  Inbound:  12.8 Kbps × 1,000 = 12.8 Mbps
  Outbound: 64 Kbps × 1,000 = 64 Mbps
  Total:    76.8 Mbps per zone

Per realm (50 zones):
  Total: 76.8 Mbps × 50 = 3.84 Gbps

Global (50 realms):
  Total: 3.84 Gbps × 50 = 192 Gbps

With headroom (3×): ~600 Gbps backbone capacity needed

2.3 Server Compute (Base Case)

Game tick processing per zone:
  Players: 1,000
  Entities per player: 100
  Total entity updates: 1,000 × 100 = 100,000 entities/tick
  At 20 Hz: 2,000,000 entity updates/sec

Per entity update:
  - Position interpolation: ~100 CPU cycles
  - Collision detection: ~500 CPU cycles
  - Ability processing: ~1,000 CPU cycles (when active)
  - Visibility check: ~200 CPU cycles
  Average: ~500 CPU cycles/entity

Total CPU per zone: 2M × 500 cycles = 1B cycles/sec
At 3 GHz: ~333M instructions/sec ≈ 0.33 cores (theoretical minimum)

Realistic (with overhead, networking, I/O):
  ~8-16 CPU cores per zone server
  ~32 GB RAM per zone server

Total zone servers: 50 zones × 1 server = 50 servers per realm
Total globally: 50 realms × 50 servers = 2,500 zone servers

2.4 Storage Estimates

Per character:
  - Core data (level, stats, position): 5 KB
  - Inventory (100 slots × 50 bytes): 5 KB
  - Quest progress (50 active × 200 bytes): 10 KB
  - Skill tree: 2 KB
  - Guild data: 1 KB
  - Social (friends, ignore): 2 KB
  - Achievement progress: 5 KB
  Total: ~30 KB per character

Active characters: 500K players × 5 chars/player = 2.5M characters
Character storage: 2.5M × 30 KB = ~75 GB

World state (persistent):
  - NPC positions, spawns: ~500 MB per realm
  - Dynamic world objects: ~100 MB per realm
  - Total: ~30 GB for 50 realms

Transaction log (inventory changes, combat events):
  - 500K players × 100 transactions/hr × 200 bytes = 1 GB/hr
  - Daily: 24 GB
  - Retention (30 days): 720 GB

Total storage: ~1 TB (with backups and headroom: 5 TB)

2.5 QPS Estimates

Game tick QPS:
  500K players × 20 Hz = 10,000,000 QPS (tick processing)

Zone transition QPS:
  500K players × 0.1 transitions/hr = 50K/hr = ~14 QPS

Inventory transaction QPS:
  500K × 100/hr = 50M/hr = ~13,889 QPS

Chat message QPS:
  500K × 10 msg/min = 83M/hr = ~23,148 QPS

Auth/login QPS:
  500K sessions / 2hr avg = 250K logins/hr = ~69 QPS

Total non-tick QPS: ~50,000 QPS
Total with ticks: ~10,050,000 QPS

3. System Interface Design

3.1 Client-Server Binary Protocol

Real-time games use custom binary protocols for efficiency. We define a length-prefixed binary protocol over TCP/UDP.

Packet Format:
┌─────────┬──────────┬──────────┬───────────┬──────────┐
│ Version │  SeqNo   │  Type    │  Length   │  Payload │
│  1B     │   4B     │   2B     │   2B      │  VarLen  │
└─────────┴──────────┴──────────┴───────────┴──────────┘

Version: Protocol version (1 byte)
SeqNo:   Sequence number for ordering (4 bytes, unsigned)
Type:    Message type (2 bytes, enum)
Length:  Payload length in bytes (2 bytes)
Payload: Type-specific data

3.2 Message Types

# === Client → Server ===

MSG_CLIENT_TICK (0x0100):
  {
    clientId:        uint64,
    position:        Vector3 { x: float32, y: float32, z: float32 },
    rotation:        float32 (yaw),
    cameraTarget:    Vector3,
    inputState:      BitFlags {
      moveForward, moveBackward, moveLeft, moveRight,
      sprint, jump, crouch, attack, ability1-8, interact
    },
    selectedTarget:  uint64 (entityId, 0 if none),
    clientTimestamp: uint64 (ms since epoch),
    pingEstimate:    uint16 (ms)
  }

MSG_ABILITY_CAST (0x0101):
  {
    abilityId:   uint16,
    targetId:    uint64,
    targetPos:   Vector3,
    castTime:    uint32 (ms),
    predictionId: uint32 (client-side prediction ID)
  }

MSG_INTERACT (0x0102):
  {
    entityId:  uint64,
    action:    uint8 (talk, loot, pickup, use, trade)
  }

MSG_CHAT (0x0103):
  {
    channel:   uint8 (say, party, guild, trade, whisper),
    targetId:  uint64 (for whisper, 0 otherwise),
    message:   string (max 256 bytes)
  }

MSG_ZONE_REQUEST (0x0104):
  {
    targetZoneId: uint16,
    targetPos:    Vector3,
    reason:       uint8 (portal, teleport, death, manual)
  }

MSG_INVENTORY_ACTION (0x0105):
  {
    action:     uint8 (equip, unequip, use, drop, move, split),
    sourceSlot: uint8,
    targetSlot: uint8,
    quantity:   uint16
  }

# === Server → Client ===

MSG_WORLD_STATE (0x0200):
  {
    serverTick:      uint64,
    playerState:     {
      position:    Vector3,
      rotation:    float32,
      health:      uint16,
      mana:        uint16,
      speed:       float32,
      buffs:       Buff[] { id: uint16, duration: uint32, stacks: uint8 },
      debuffs:     Debuff[]
    },
    entities:        EntityUpdate[] {
      entityId:    uint64,
      position:    Vector3,
      rotation:    float32,
      health:      uint16,
      state:       uint8 (idle, moving, attacking, casting, dead),
      interpolationId: uint16
    },
    newEntities:     EntitySpawn[] {
      entityId:    uint64,
      entityType:  uint16 (player, npc, creature, object),
      templateId:  uint32,
      position:    Vector3,
      name:        string
    },
    removedEntities: uint64[] (entityIds to despawn),
    effects:        VFX[] { effectId: uint16, position: Vector3, duration: uint32 }
  }

MSG_COMBAT_EVENT (0x0201):
  {
    attackerId:  uint64,
    defenderId:  uint64,
    abilityId:   uint16,
    damage:      uint16,
    damageType:  uint8 (physical, fire, frost, nature, holy, shadow),
    critical:    bool,
    blocked:     uint16,
    absorbed:    uint16,
    healed:      uint16
  }

MSG_ZONE_DATA (0x0202):
  {
    zoneId:        uint16,
    zoneName:      string,
    initialPos:    Vector3,
    entities:      EntitySpawn[],
    npcs:          NPCSpawn[],
    objects:       WorldObjectSpawn[],
    weather:       uint8,
    timeOfDay:     uint16 (0-2400)
  }

MSG_INVENTORY_UPDATE (0x0203):
  {
    slot:      uint8,
    item:      Item { itemId: uint32, quantity: uint16, enchantments: Enchant[] },
    action:    uint8 (added, removed, updated, equipped)
  }

MSG_CHAT_BROADCAST (0x0204):
  {
    channel:   uint8,
    senderId:  uint64,
    senderName: string,
    message:   string,
    timestamp: uint64
  }

MSG_ERROR (0x02FF):
  {
    errorCode:   uint16,
    errorMessage: string,
    correlationId: uint32
  }

3.3 Internal Service APIs (gRPC)

service ZoneService {
    rpc EnterZone(EnterZoneRequest) returns (ZoneEntry);
    rpc LeaveZone(LeaveZoneRequest) returns (ZoneExit);
    rpc GetZoneState(ZoneStateRequest) returns (ZoneState);
    rpc TransferPlayer(TransferRequest) returns (TransferResult);
}

service CharacterService {
    rpc LoadCharacter(LoadCharacterRequest) returns (Character);
    rpc SaveCharacter(SaveCharacterRequest) returns (SaveResult);
    rpc DeleteCharacter(DeleteCharacterRequest) returns (Empty);
    rpc ListCharacters(ListCharactersRequest) returns (CharacterList);
}

service AuthService {
    rpc Authenticate(AuthRequest) returns (AuthResponse);
    rpc ValidateToken(TokenRequest) returns (TokenStatus);
    rpc RevokeToken(RevokeRequest) returns (Empty);
}

service MatchmakingService {
    rpc QueuePlayer(QueueRequest) returns (QueueStatus);
    rpc LeaveQueue(LeaveQueueRequest) returns (Empty);
    rpc CreateInstance(CreateInstanceRequest) returns (InstanceInfo);
}

service EconomyService {
    rpc BuyItem(BuyItemRequest) returns (TransactionResult);
    rpc SellItem(SellItemRequest) returns (TransactionResult);
    rpc AuctionList(AuctionListRequest) returns (AuctionResult);
    rpc AuctionBid(AuctionBidRequest) returns (BidResult);
}

service GuildService {
    rpc CreateGuild(CreateGuildRequest) returns (GuildInfo);
    rpc JoinGuild(JoinGuildRequest) returns (JoinResult);
    rpc LeaveGuild(LeaveGuildRequest) returns (Empty);
    rpc GetGuildInfo(GuildInfoRequest) returns (GuildInfo);
}

4. High-Level Architecture

flowchart TB
    subgraph Client["Client Layer"]
        GameClient["Game Client<br/>(Unity / Unreal)"]
        Launcher["Game Launcher"]
    end

    subgraph Edge["Edge Network"]
        CDN["CDN / WAF"]
        GameLB["Game Load Balancer<br/>(L4 TCP/UDP)"]
        AuthLB["Auth Load Balancer<br/>(L7 HTTPS)"]
    end

    subgraph AuthZone["Authentication Zone"]
        AuthSvc["Auth Service"]
        TokenSvc["Token Service<br/>(JWT / Session)"]
        AccountSvc["Account Service"]
    end

    subgraph Gateway["Game Gateway"]
        GW1["Game Gateway 1"]
        GW2["Game Gateway 2"]
        GWN["Game Gateway N"]
    end

    subgraph Realm["Realm Server Cluster"]
        subgraph WorldMgr["World Management"]
            WorldSvc["World Service"]
            ZoneMgr["Zone Manager"]
            CrossZone["Cross-Zone Router"]
        end

        subgraph Zones["Zone Servers"]
            subgraph Zone1["Zone Server 1<br/>(Elwynn Forest)"]
                Z1Engine["Game Engine"]
                Z1Net["Network Handler"]
                Z1AI["AI / NPC Manager"]
                Z1Combat["Combat System"]
            end
            subgraph Zone2["Zone Server 2<br/>(Stormwind)"]
                Z2Engine["Game Engine"]
                Z2Net["Network Handler"]
                Z2AI["AI / NPC Manager"]
                Z2Combat["Combat System"]
            end
            subgraph ZoneN["Zone Server N<br/>(Dungeon Instance)"]
                ZNEngine["Game Engine"]
                ZNNet["Network Handler"]
                ZNAI["AI / NPC Manager"]
                ZNCombat["Combat System"]
            end
        end
    end

    subgraph Services["Support Services"]
        CharSvc["Character Service"]
        InvSvc["Inventory Service"]
        QuestSvc["Quest Service"]
        GuildSvc["Guild Service"]
        AuctionSvc["Auction House Service"]
        ChatSvc["Chat Service"]
        MMService["Matchmaking Service"]
        AntiCheatSvc["Anti-Cheat Service"]
        LogSvc["Logging Service"]
    end

    subgraph Data["Data Layer"]
        RedisGame["Redis Cluster<br/>(Hot Game State)"]
        RedisCache["Redis Cache<br/>(Sessions, Leaderboards)"]
        PGMain["PostgreSQL<br/>(Characters, Accounts)"]
        PGShard1["PG Shard 1<br/>(Inventory)"]
        PGShard2["PG Shard 2<br/>(Quests, Guilds)"]
        Mongo["MongoDB<br/>(Chat, Logs, Events)"]
        Kafka["Kafka Cluster<br/>(Event Streaming)"]
        TSDB["TimeSeries DB<br/>(Metrics)"]
        S3["Object Storage<br/>(Backups, Assets)"]
    end

    GameClient --> Launcher
    Launcher --> CDN
    GameClient --> GameLB
    CDN --> AuthLB

    AuthLB --> AuthSvc
    AuthSvc --> TokenSvc
    AuthSvc --> AccountSvc

    GameLB --> GW1
    GameLB --> GW2
    GameLB --> GWN

    GW1 --> WorldSvc
    GW2 --> WorldSvc
    GWN --> WorldSvc

    WorldSvc --> ZoneMgr
    ZoneMgr --> Zone1
    ZoneMgr --> Zone2
    ZoneMgr --> ZoneN

    Z1Engine --> RedisGame
    Z2Engine --> RedisGame
    ZNEngine --> RedisGame

    Z1Engine --> Kafka
    Z2Engine --> Kafka
    ZNEngine --> Kafka

    Zone1 --> CrossZone
    Zone2 --> CrossZone

    CrossZone --> Zone1
    CrossZone --> Zone2

    CharSvc --> PGMain
    InvSvc --> PGShard1
    QuestSvc --> PGShard2
    GuildSvc --> PGShard2
    AuctionSvc --> PGShard1
    ChatSvc --> Mongo

    Kafka --> AntiCheatSvc
    Kafka --> LogSvc
    Kafka --> CharSvc
    Kafka --> InvSvc

    AuthSvc --> RedisCache
    MMService --> RedisCache

5. Core Component Design

5.1 Game Gateway

The Game Gateway is the single entry point for all game client connections.

Responsibilities

  • Connection management (TCP/UDP multiplexing)
  • Protocol validation and parsing
  • Player authentication (token verification)
  • Zone routing (direct to correct zone server)
  • Connection state tracking
  • Rate limiting and flood protection

Architecture

flowchart LR
    subgraph Gateway["Game Gateway"]
        ConnMgr["Connection Manager"]
        Parser["Protocol Parser"]
        Router["Zone Router"]
        RateLimiter["Rate Limiter"]
        AuthChecker["Auth Checker"]
    end

    Client["Game Client"] --> ConnMgr
    ConnMgr --> Parser
    Parser --> AuthChecker
    AuthChecker --> RateLimiter
    RateLimiter --> Router
    Router --> ZoneServer["Zone Server"]

Connection Model

Phase 1: TCP Handshake + Auth
  Client → Gateway: CONNECT + auth_token
  Gateway → Auth Service: Validate token
  Auth Service → Gateway: Valid / Invalid
  Gateway → Client: AUTH_OK (sessionId, serverTick)

Phase 2: Zone Assignment
  Gateway → World Service: Get zone for player
  World Service → Gateway: ZoneId, ZoneServerAddress
  Gateway → Client: ROUTE_TO_ZONE (zoneServerAddr)

Phase 3: Direct Connection
  Client → Zone Server: DIRECT_CONNECT (sessionId)
  Zone Server → Client: WELCOME (zoneData, initialEntities)

Phase 4: Game Loop
  Client ↔ Zone Server: Tick packets at 20 Hz

5.2 Zone Server (The Core Game Loop)

The Zone Server runs the authoritative game simulation for a specific zone.

Game Loop Architecture

flowchart TD
    subgraph Tick["Game Tick (50ms cycle)"]
        Input["1. Collect Inputs<br/>(buffered client packets)"]
        Predict["2. Client Prediction<br/>(server-side replay)"]
        Simulate["3. Physics Simulation<br/>(movement, collision)"]
        Combat["4. Combat Resolution<br/>(abilities, damage)"]
        AI["5. AI / NPC Updates<br/>(pathfinding, behavior)"]
        Visibility["6. Visibility Culling<br/>(what each player sees)"]
        Serialize["7. State Serialization<br/>(delta encoding)"]
        Broadcast["8. Network Broadcast<br/>(send to clients)"]
        Save["9. Async Persistence<br/>(checkpoint to Redis)"]
    end

    Input --> Predict
    Predict --> Simulate
    Simulate --> Combat
    Combat --> AI
    AI --> Visibility
    Visibility --> Serialize
    Serialize --> Broadcast
    Broadcast --> Save

Detailed Tick Breakdown

Phase Duration Description
1. Input Collection 5ms Gather all client packets received since last tick; discard out-of-order
2. Prediction Reconciliation 5ms Compare client-predicted positions with server-authoritative; correct discrepancies
3. Physics Simulation 10ms Move entities, resolve collisions, apply gravity, terrain checks
4. Combat Resolution 10ms Process ability casts, calculate damage, apply buffs/debuffs, check deaths
5. AI Updates 10ms NPC behavior trees, pathfinding, aggro management, spawn/despawn logic
6. Visibility Culling 3ms Determine which entities each player can see (distance + occlusion)
7. Serialization 4ms Delta-encode state changes, compress packets
8. Broadcast 2ms Send packets to connected clients
9. Async Persistence 1ms Write checkpoint to Redis (non-blocking)
Total 50ms 20 Hz tick rate

Entity Management

Entity Types:
  - Player (controlled by client, authoritative on server)
  - NPC (AI-controlled, static or scripted behavior)
  - Creature (AI-controlled, combat-capable)
  - WorldObject (interactive objects: chests, doors, portals)
  - Projectile (abilities, arrows, spells)
  - Effect (visual effects, AoE zones)

Entity Data Structure:
  {
    entityId:      uint64 (unique, monotonic),
    entityType:    uint16,
    templateId:    uint32 (references asset/template),
    position:      Vector3 { x: float32, y: float32, z: float32 },
    rotation:      float32 (yaw),
    velocity:      Vector3,
    health:        uint16,
    maxHealth:     uint16,
    state:         uint8 (idle, moving, attacking, casting, dead, stunned),
    owner:         uint64 (playerId if player-controlled, 0 otherwise),
    buffs:         Buff[],
    debuffs:       Debuff[],
    lastUpdateTick: uint64,
    interpolationId: uint16 (for client interpolation)
  }

5.3 Real-Time State Synchronization

The Core Challenge

Client A (ping 50ms) ────────┐
                              ├──→ Server (authoritative) ←──┐
Client B (ping 80ms) ────────┘                              └── Client C (ping 60ms)

At 20 Hz with variable latency, the server sees stale client state. The solution combines three techniques:

Technique 1: Client-Side Prediction

Client sends:
  { position: predicted_pos, input: current_input, clientTick: T }

Server receives at tick T+delay:
  1. Replay from last known good state at tick T
  2. Apply client's input from tick T
  3. Compare result with client's predicted position
  4. If within tolerance (±0.5m): accept
  5. If outside tolerance: correct and send correction

Client receives correction:
  1. If current prediction matches server: continue
  2. If mismatch: snap to server position + rewind prediction

Technique 2: Server-Side Reconciliation

flowchart LR
    subgraph Server["Server Reconciliation"]
        A["Receive client packet<br/>(tick T, position P)"]
        B["Find last authoritative<br/>state at tick T"]
        C["Replay inputs from T<br/>to current tick"]
        D["Compare replayed<br/>position with P"]
        E{Within tolerance?}
        F["Accept position"]
        G["Send correction<br/>to client"]
    end

    A --> B --> C --> D --> E
    E -->|Yes| F
    E -->|No| G

Technique 3: Entity Interpolation

Client receives entity positions at 20 Hz but renders at 60+ Hz.

Interpolation:
  For each visible entity:
    prevPos = last received position
    currPos = current received position
    prevTick = tick of prevPos
    currTick = tick of currPos

    renderTick = current render tick
    alpha = (renderTick - prevTick) / (currTick - prevTick)
    alpha = clamp(alpha, 0, 1)

    renderedPos = lerp(prevPos, currPos, alpha)

Extrapolation (for fast-moving entities):
  If no update received for N ticks:
    velocity = (currPos - prevPos) / (currTick - prevTick)
    extrapolatedPos = currPos + velocity × (renderTick - currTick)
    If extrapolation time > threshold: stop extrapolating, freeze

5.4 Lag Compensation

Hit Detection with Lag Compensation

flowchart TD
    A[Attacker fires at target] --> B[Server receives attack packet]
    B --> C["Rewind simulation to<br/>attacker's client tick"]
    C --> D["Check if target was<br/>in line of sight at that tick"]
    D --> E{Valid hit?}
    E -->|Yes| F["Apply damage at<br/>current server tick"]
    E -->|No| G["Reject hit<br/>(target was behind wall)"]
    F --> H["Forward simulation<br/>to current tick"]
    G --> H

Implementation Details

History Buffer:
  - Each entity maintains position history for last 200ms (4 ticks at 20Hz)
  - History entries: { tick, position, rotation, state }
  - Memory: 1000 entities × 4 entries × 32 bytes = 128 KB per zone

Hit Validation:
  1. Attacker sends: { targetId, targetPos_at_fire_time, fireTick }
  2. Server rewinds target to fireTick using history buffer
  3. Server checks line of sight from attacker position to rewound target
  4. Server checks if target was in valid hitbox
  5. If valid: apply damage at current tick
  6. If invalid: reject (potential peeker's advantage abuse)

Anti-Abuse:
  - Maximum rewind window: 200ms (prevents extreme lag exploitation)
  - Clients with high ping (>150ms) get reduced compensation
  - Repeated invalid hits → anti-cheat flag

5.5 World Partitioning and Zone Sharding

Zone Design

flowchart TD
    subgraph Realm["Realm Server"]
        subgraph Continent["Continent"]
            Z1["Zone 1:<br/>Starting Area<br/>(1000 players)"]
            Z2["Zone 2:<br/>Capital City<br/>(1000 players)"]
            Z3["Zone 3:<br/>Wilderness<br/>(1000 players)"]
            Z4["Zone 4:<br/>High-Level Zone<br/>(1000 players)"]
        end

        subgraph Instances["Instances"]
            D1["Dungeon 1<br/>(5-10 players)"]
            D2["Dungeon 2<br/>(5-10 players)"]
            R1["Raid Instance<br/>(20-40 players)"]
            A1["Arena Instance<br/>(5v5)"]
            B1["Battleground<br/>(15v15)"]
        end

        Z1 <--> Z2
        Z2 <--> Z3
        Z3 <--> Z4
        Z1 --> D1
        Z3 --> D2
        Z4 --> R1
    end

Zone Transition Flow

1. Player approaches zone border
2. Client sends ZONE_REQUEST to current zone server
3. Current zone server validates request
4. Zone Manager allocates slot in target zone
5. Character state saved (immediate persist)
6. Target zone server loads player data
7. Player removed from current zone
8. Player added to target zone
9. Zone data sent to client (full state)
10. Client transitions (loading screen if needed)
11. Player appears in new zone

Total transition time: < 2 seconds

Cross-Zone Communication

Party members in different zones:
  - Chat: Routed through Chat Service (Kafka-based)
  - Party invites: Routed through Cross-Zone Router
  - Shared objectives: Tracked in World Service

Implementation:
  Cross-Zone Router maintains:
    - Player → Zone mapping (Redis)
    - Party → Player mapping
    - Message routing table

  When Player A (Zone 1) sends party message:
    1. Zone 1 → Cross-Zone Router: party_msg(partyId, from=A, msg)
    2. Cross-Zone Router looks up party members' zones
    3. Routes to Zone 2, Zone 3, etc.
    4. Each zone server delivers to respective player

5.6 Anti-Cheat System

Multi-Layer Defense

flowchart TD
    subgraph Layers["Anti-Cheat Layers"]
        L1["Layer 1: Client Integrity"]
        L2["Layer 2: Input Validation"]
        L3["Layer 3: Behavior Analysis"]
        L4["Layer 4: Server Authority"]
        L5["Layer 5: Post-Game Review"]
    end

    L1 --> L2 --> L3 --> L4 --> L5

    subgraph L1Detail["Client Integrity"]
        CI1["Binary integrity check<br/>(hash verification)"]
        CI2["Memory scan<br/>(known cheat signatures)"]
        CI3["Anti-debugging<br/>(debugger detection)"]
        CI4["Anti-injection<br/>(DLL/process scan)"]
    end

    subgraph L2Detail["Input Validation"]
        IV1["Speed check<br/>(max movement speed)"]
        IV2["Ability cooldown check"]
        IV3["Line of sight check"]
        IV4["Range validation"]
    end

    subgraph L3Detail["Behavior Analysis"]
        BA1["Aim pattern analysis<br/>(human vs bot)"]
        BA2["Reaction time analysis"]
        BA3["Movement pattern analysis"]
        BA4["Statistical anomaly detection"]
    end

    subgraph L4Detail["Server Authority"]
        SA1["All damage calculated server-side"]
        SA2["All position validation server-side"]
        SA3["All inventory changes server-side"]
        SA4["All combat resolution server-side"]
    end

    subgraph L5Detail["Post-Game Review"]
        PR1["Combat log analysis"]
        PR2["Win rate anomaly detection"]
        PR3["Manual review queue"]
        PR4["Machine learning classification"]
    end

    L1Detail -.-> L1
    L2Detail -.-> L2
    L3Detail -.-> L3
    L4Detail -.-> L4
    L5Detail -.-> L5

Real-Time Detection Pipeline

Per-tick checks (every 50ms):
  1. Speed check: |currentPos - lastPos| <= maxSpeed × deltaTime
  2. Terrain check: position is on valid terrain
  3. Input rate: client sending ≤ 25 packets/sec (20 Hz + 25% tolerance)

Per-action checks:
  1. Ability cooldown: lastCastTime + cooldown <= currentTime
  2. Resource check: mana >= abilityCost
  3. Target validity: target exists, in range, in line of sight
  4. State check: player not dead, not stunned, not in combat lock

Per-session analysis (async):
  1. Movement smoothness: human movement has micro-variations
  2. Aim precision: perfect aim over 100+ shots → flag
  3. Reaction time: sub-100ms consistently → flag
  4. Play pattern: 24/7 play without breaks → flag

5.7 Combat System

Ability Resolution Pipeline

flowchart TD
    A[Client casts ability] --> B["Server receives<br/>ABILITY_CAST"]
    B --> C{Cooldown check}
    C -->|On CD| D[Reject: cooldown]
    C -->|Ready| E{Resource check}
    E -->|Insufficient| F[Reject: not enough mana]
    E -->|Sufficient| G{Target validation}
    G -->|Invalid| H[Reject: invalid target]
    G -->|Valid| I[Apply ability effects]
    I --> J["Calculate damage<br/>(attack vs defense)"]
    J --> K["Apply modifiers<br/>(crit, dodge, block, resist)"]
    K --> L["Apply damage/heal"]
    L --> M["Check death conditions"]
    M --> N["Broadcast combat event<br/>to all affected players"]
    N --> O["Start cooldown"]
    O --> P["Publish to Kafka<br/>(logging, analytics)"]

    D --> Q[Send error to client]
    F --> Q
    H --> Q

Damage Calculation

Base Damage = attacker.attack × ability.multiplier

Modifiers:
  - Critical hit: × 2.0 (if crit roll succeeds)
  - Defense reduction: defender.defense × 0.5
  - Resistance: type-specific (fire resist, etc.)
  - Buffs: +flat or +% from active buffs
  - Debuffs: -flat or -% on target

Final Damage = max(1, Base Damage - Defense Reduction) × Crit × Resistance

Damage Types:
  Physical: Reduced by armor
  Magic (Fire/Frost/Nature/Holy/Shadow): Reduced by spell resistance
  True Damage: Ignored by all defenses

5.8 Matchmaking Service (Dungeons, Raids, PvP)

Queue System

flowchart TD
    A[Player queues for dungeon] --> B["Join queue with<br/>role, level, rating"]
    B --> C["Wait for group<br/>completion"]
    C --> D{Group formed?}
    D -->|Yes| E["Create instance"]
    D -->|No| F{Timeout?}
    F -->|No| C
    F -->|Yes| G["Notify: no match found"]

    E --> H["Assign players to<br/>instance"]
    H --> I["Teleport all players<br/>to instance"]
    I --> J["Instance begins"]

Group Formation Algorithm

Queue entry:
  {
    playerId: uint64,
    role: uint8 (tank, healer, dps),
    level: uint16,
    rating: uint16 (for PvP),
    queueTime: uint64,
    flexibility: uint8 (main role, off role, any)
  }

Matching algorithm:
  1. Sort queue by queueTime (oldest first)
  2. For each player, find compatible group:
     - Role balance: 1 tank, 1-2 healers, rest DPS
     - Level range: ±3 levels
     - Rating range: ±100 (PvP)
     - Latency compatibility: same region preferred
  3. Score each potential group:
     - Time waited (higher = better)
     - Role match quality
     - Rating similarity
     - Latency similarity
  4. Select highest-scoring group
  5. Create instance, assign players
  6. Remove from queue

6. Data Model

6.1 PostgreSQL Schema

-- Accounts
CREATE TABLE accounts (
    account_id    BIGSERIAL PRIMARY KEY,
    username      VARCHAR(64) UNIQUE NOT NULL,
    email         VARCHAR(255) UNIQUE NOT NULL,
    password_hash VARCHAR(255) NOT NULL,
    totp_secret   VARCHAR(64),
    created_at    TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    last_login    TIMESTAMP WITH TIME ZONE,
    is_active     BOOLEAN DEFAULT TRUE,
    ban_reason    TEXT,
    banned_at     TIMESTAMP WITH TIME ZONE
);

-- Characters
CREATE TABLE characters (
    character_id  BIGSERIAL PRIMARY KEY,
    account_id    BIGINT REFERENCES accounts(account_id) ON DELETE CASCADE,
    realm_id      INTEGER NOT NULL,
    name          VARCHAR(32) NOT NULL,
    race          VARCHAR(32) NOT NULL,
    class         VARCHAR(32) NOT NULL,
    gender        CHAR(1) NOT NULL,              -- M/F
    level         INTEGER NOT NULL DEFAULT 1,
    experience    BIGINT NOT NULL DEFAULT 0,
    health        INTEGER NOT NULL,
    max_health    INTEGER NOT NULL,
    mana          INTEGER NOT NULL,
    max_mana      INTEGER NOT NULL,
    strength      INTEGER NOT NULL DEFAULT 10,
    agility       INTEGER NOT NULL DEFAULT 10,
    stamina       INTEGER NOT NULL DEFAULT 10,
    intellect     INTEGER NOT NULL DEFAULT 10,
    zone_id       SMALLINT NOT NULL,
    position_x    REAL NOT NULL,
    position_y    REAL NOT NULL,
    position_z    REAL NOT NULL,
    guild_id      BIGINT,
    is_online     BOOLEAN DEFAULT FALSE,
    last_login    TIMESTAMP WITH TIME ZONE,
    created_at    TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    deleted_at    TIMESTAMP WITH TIME ZONE,
    UNIQUE(account_id, name, realm_id)
);

CREATE INDEX idx_characters_account ON characters(account_id);
CREATE INDEX idx_characters_realm ON characters(realm_id);
CREATE INDEX idx_characters_online ON characters(is_online) WHERE is_online = TRUE;

-- Inventory (sharded by character_id)
CREATE TABLE inventory (
    inventory_id  BIGSERIAL PRIMARY KEY,
    character_id  BIGINT NOT NULL REFERENCES characters(character_id) ON DELETE CASCADE,
    slot          SMALLINT NOT NULL,              -- 0-99
    item_template_id INTEGER NOT NULL,
    quantity      INTEGER NOT NULL DEFAULT 1,
    durability    INTEGER,
    enchantments  JSONB,                          -- [{enchantId, duration}]
    socket_gems   JSONB,                          -- [{socketIndex, gemId}]
    equipped      BOOLEAN DEFAULT FALSE,
    created_at    TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    UNIQUE(character_id, slot)
);

CREATE INDEX idx_inventory_character ON inventory(character_id);

-- Item templates
CREATE TABLE item_templates (
    template_id   INTEGER PRIMARY KEY,
    name          VARCHAR(128) NOT NULL,
    item_type     VARCHAR(32) NOT NULL,           -- weapon, armor, consumable, quest
    item_class    VARCHAR(32),                    -- sword, shield, potion, etc.
    quality       SMALLINT NOT NULL,              -- 0=poor, 1=common, 2=uncommon, 3=rare, 4=epic, 5=legendary
    level_req     INTEGER NOT NULL DEFAULT 1,
    class_req     VARCHAR(32),
    stats         JSONB,                          -- {strength: 10, agility: 5}
    description   TEXT,
    icon_id       INTEGER,
    value         INTEGER NOT NULL DEFAULT 0      -- copper
);

-- Quests
CREATE TABLE quest_templates (
    quest_id      INTEGER PRIMARY KEY,
    name          VARCHAR(128) NOT NULL,
    description   TEXT,
    zone_id       SMALLINT,
    level_req     INTEGER,
    prerequisites JSONB,                          -- [questId1, questId2]
    objectives    JSONB,                          -- [{type: 'kill', target: 'wolf', count: 10}]
    rewards       JSONB,                          -- {xp: 1000, gold: 500, items: [{templateId, qty}]}
    repeatable    BOOLEAN DEFAULT FALSE
);

CREATE TABLE quest_progress (
    character_id  BIGINT REFERENCES characters(character_id) ON DELETE CASCADE,
    quest_id      INTEGER REFERENCES quest_templates(quest_id),
    status        VARCHAR(16) NOT NULL,           -- active, completed, failed
    objectives    JSONB,                          -- Current objective progress
    started_at    TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    completed_at  TIMESTAMP WITH TIME ZONE,
    PRIMARY KEY (character_id, quest_id)
);

-- Skills / Talents
CREATE TABLE skill_templates (
    skill_id      INTEGER PRIMARY KEY,
    name          VARCHAR(64) NOT NULL,
    class         VARCHAR(32),                    -- NULL = all classes
    description   TEXT,
    icon_id       INTEGER,
    cast_time     INTEGER NOT NULL DEFAULT 0,     -- ms
    cooldown      INTEGER NOT NULL DEFAULT 0,     -- ms
    range         REAL NOT NULL,
    mana_cost     INTEGER NOT NULL DEFAULT 0,
    damage_formula TEXT,                          -- "attack * 1.5 + intellect * 0.5"
    effect_type   VARCHAR(32),                    -- damage, heal, buff, debuff, cc
    max_level     INTEGER NOT NULL DEFAULT 1
);

CREATE TABLE character_skills (
    character_id  BIGINT REFERENCES characters(character_id) ON DELETE CASCADE,
    skill_id      INTEGER REFERENCES skill_templates(skill_id),
    level         INTEGER NOT NULL DEFAULT 1,
    points_spent  INTEGER NOT NULL DEFAULT 0,
    PRIMARY KEY (character_id, skill_id)
);

-- Guilds
CREATE TABLE guilds (
    guild_id      BIGSERIAL PRIMARY KEY,
    name          VARCHAR(64) UNIQUE NOT NULL,
    realm_id      INTEGER NOT NULL,
    leader_id     BIGINT REFERENCES characters(character_id),
    motto         VARCHAR(256),
    created_at    TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    level         INTEGER NOT NULL DEFAULT 1,
    experience    BIGINT NOT NULL DEFAULT 0
);

CREATE TABLE guild_members (
    guild_id      BIGINT REFERENCES guilds(guild_id) ON DELETE CASCADE,
    character_id  BIGINT REFERENCES characters(character_id) ON DELETE CASCADE,
    rank          VARCHAR(32) NOT NULL,           -- leader, officer, member
    joined_at     TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    PRIMARY KEY (guild_id, character_id)
);

-- Auction House
CREATE TABLE auctions (
    auction_id    BIGSERIAL PRIMARY KEY,
    seller_id     BIGINT REFERENCES characters(character_id),
    item_template_id INTEGER REFERENCES item_templates(template_id),
    quantity      INTEGER NOT NULL,
    buyout_price  INTEGER,                        -- copper (NULL = no buyout)
    starting_bid  INTEGER NOT NULL,               -- copper
    current_bid   INTEGER NOT NULL,
    bidder_id     BIGINT REFERENCES characters(character_id),
    expires_at    TIMESTAMP WITH TIME ZONE NOT NULL,
    status        VARCHAR(16) DEFAULT 'active',   -- active, sold, expired, cancelled
    created_at    TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

CREATE INDEX idx_auctions_status ON auctions(status, expires_at);
CREATE INDEX idx_auctions_item ON auctions(item_template_id);

-- PvP Ratings
CREATE TABLE pvp_ratings (
    character_id  BIGINT PRIMARY KEY REFERENCES characters(character_id) ON DELETE CASCADE,
    bracket       VARCHAR(32) NOT NULL,           -- arena_2v2, arena_3v3, arena_5v5, rated_bgg
    rating        INTEGER NOT NULL DEFAULT 1500,
    wins          INTEGER NOT NULL DEFAULT 0,
    losses        INTEGER NOT NULL DEFAULT 0,
    draws         INTEGER NOT NULL DEFAULT 0,
    weekly_wins   INTEGER NOT NULL DEFAULT 0,
    season_wins   INTEGER NOT NULL DEFAULT 0,
    updated_at    TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

-- Anti-cheat flags
CREATE TABLE anticheat_flags (
    flag_id       BIGSERIAL PRIMARY KEY,
    account_id    BIGINT REFERENCES accounts(account_id),
    character_id  BIGINT REFERENCES characters(character_id),
    flag_type     VARCHAR(32) NOT NULL,           -- speed_hack, aimbot, memory_edit, etc.
    severity      VARCHAR(16) NOT NULL,           -- low, medium, high, critical
    evidence      JSONB,
    auto_action   VARCHAR(32),                    -- none, warn, kick, temp_ban, perm_ban
    reviewed_by   VARCHAR(64),
    reviewed_at   TIMESTAMP WITH TIME ZONE,
    created_at    TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

6.2 Redis Data Structures

# Hot game state per zone
Key:    zone:{realmId}:{zoneId}:entities
Type:   Hash
Fields: {entityId}: {position, rotation, health, state, owner}
TTL:    None (active zone)

# Player session
Key:    session:{characterId}
Type:   Hash
Fields: realmId, zoneId, gatewayConnId, zoneServerConnId, lastTick, position
TTL:    120s (renewed every tick)

# Player → Zone mapping (for cross-zone routing)
Key:    player:zone:{characterId}
Type:   String
Value:  {realmId}:{zoneId}
TTL:    300s

# Zone player count
Key:    zone:count:{realmId}:{zoneId}
Type:   String
Value:  player_count
TTL:    None

# Entity position history (for lag compensation)
Key:    entity:history:{entityId}
Type:   List
Value:  [{tick, position, rotation}, ...]  (last 4 entries)
TTL:    1s

# Matchmaking queues
Key:    mm:queue:{instanceType}
Type:   Sorted Set
Score:  queue_timestamp
Value:  {characterId}:{role}:{level}:{rating}

# Rate limiting
Key:    ratelimit:{accountId}:{action}
Type:   Sorted Set
Score:  timestamp
Value:  request_id

# Chat channels
Key:    chat:channel:{channelId}
Type:   List
Value:  [{senderId, message, timestamp}, ...]  (last 100 messages)
TTL:    5min

# Leaderboards
Key:    leaderboard:{type}:{realmId}
Type:   Sorted Set
Score:  rating / score
Value:  characterId

6.3 MongoDB Schema (Chat, Logs, Events)

// Chat messages
{
  _id: ObjectId,
  channelId: "party_12345",
  channelType: "party",
  senderId: 98765,
  senderName: "PlayerA",
  message: "Need a healer!",
  timestamp: ISODate("2026-08-28T10:30:00Z"),
  realmId: 1
}

// Combat logs
{
  _id: ObjectId,
  realmId: 1,
  zoneId: 42,
  serverTick: 12345678,
  timestamp: ISODate("2026-08-28T10:30:00Z"),
  events: [
    {
      type: "damage",
      attackerId: 111,
      defenderId: 222,
      abilityId: 101,
      damage: 1500,
      damageType: "physical",
      critical: true
    }
  ]
}

// World events
{
  _id: ObjectId,
  eventId: "world_boss_spawn",
  realmId: 1,
  zoneId: 15,
  position: { x: 1234.5, y: 567.8, z: 12.3 },
  startTime: ISODate("2026-08-28T12:00:00Z"),
  endTime: ISODate("2026-08-28T12:30:00Z"),
  participants: [111, 222, 333],
  outcome: "defeated"
}

7. Key Design Decisions

7.1 TCP vs UDP for Game Transport

Factor TCP UDP (with reliability layer)
Ordering Guaranteed Manual (sequence numbers)
Reliability Guaranteed Selective (only critical packets)
Latency Head-of-line blocking No blocking
Congestion control Built-in Custom
NAT traversal Easier Harder
Decision — UDP with custom reliability

Rationale: Real-time games cannot tolerate head-of-line blocking. A lost packet shouldn’t delay all subsequent packets. We implement:

  • Unreliable channel: Position updates, camera (latest wins, old packets discarded)
  • Reliable channel: Combat events, inventory changes, chat (ordered, retransmitted)
  • Sequenced channel: State updates (ordered but not retransmitted)

7.2 Server Authority Model

Aspect Client Authority Server Authority Hybrid
Anti-cheat Weak Strong Medium
Latency feel Best Good (with prediction) Good
Complexity Low High Medium
Fairness Poor Excellent Good
Decision — — Server Authority + Client Prediction

Rationale: Server authority is non-negotiable for competitive integrity. Client prediction makes the experience feel responsive despite server authority.

7.3 Zone Sharding Strategy

Strategy Pros Cons
Static zones Simple, predictable load Hot zones overflow, cold zones waste resources
Dynamic sharding Optimal resource use Complex migration, state transfer
Decision — Static zones with overflow instances

Implementation:

  • Popular zones (capital cities) get multiple instances
  • Players can choose instance or auto-assign to least populated
  • Cross-realm technology merges low-population realms
  • Dungeon/raid instances are always dynamic

7.4 Save Strategy

Strategy Pros Cons
Save on every change Zero data loss High I/O, performance impact
Periodic save Low I/O Data loss on crash
Decision — Hybrid: periodic + event-triggered

Implementation:

  • Auto-save every 60 seconds (async, non-blocking)
  • Immediate save on: zone transition, death, logout, inventory change
  • Save to Kafka first (WAL), then PostgreSQL
  • Redis holds hot state between saves

7.5 Database Sharding

Sharding key: character_id

Shard 1: character_id % N = 0  (inventory, equipment)
Shard 2: character_id % N = 1  (quests, achievements)
...

Benefits:
  - Even distribution
  - Character data co-located
  - Horizontal scalability

Challenges:
  - Cross-shard transactions (avoided by design)
  - Global queries (use search index / materialized views)

8. Scalability Strategy

8.1 Scaling Dimensions

flowchart TD
    subgraph Scale["Scaling Levers"]
        A["More Zone Servers<br/>(add zone instances)"]
        B["More Realms<br/>(horizontal world scaling)"]
        C["Database Sharding<br/>(more PG shards)"]
        D["Redis Cluster Expansion<br/>(more shards)"]
        E["Kafka Partitions<br/>(more throughput)"]
        F["Gateway Instances<br/>(more connections)"]
    end

    A -->|"Handles more players per zone"| B
    B -->|"Handles more total players"| C
    C -->|"Handles more data"| D

8.2 Capacity Planning

Component Per-Instance Capacity Instances (500K players)
Zone Server 1,000 players 500 zone servers
Realm Server 50,000 players 10 realm servers
Game Gateway 50,000 connections 10 gateways
Redis Node 10M keys, 100K ops/sec 50 nodes (cluster)
Kafka Broker 100K msg/sec 20 brokers, 200 partitions
PostgreSQL 10K writes/sec 1 primary + 10 shards + 5 replicas
MongoDB 50K writes/sec 3-node replica set

8.3 Auto-Scaling

Zone Servers:
  - Player count > 800 → create overflow instance
  - Player count < 200 → merge with adjacent instance
  - CPU > 80% for 1 min → scale up

Gateways:
  - Connection count > 40K → add gateway
  - CPU > 70% → add gateway

Database:
  - Write latency > 10ms → add shard
  - Connection pool > 80% → add replica

8.4 Multi-Region Deployment

Region 1 (US-East) — Primary:
  - 20 realms, 100K players
  - Primary databases, primary Redis
  - Full game services

Region 2 (EU-West):
  - 15 realms, 75K players
  - Independent databases
  - Full game services

Region 3 (APAC):
  - 10 realms, 50K players
  - Independent databases
  - Full game services

Region 4 (US-West):
  - 5 realms, 25K players
  - Shared DB with US-East (low latency)

Cross-region:
  - No cross-region gameplay (latency too high)
  - Cross-region leaderboards (aggregated)
  - Cross-region chat (optional, with latency warning)

8.5 Cross-Realm Technology

When realm population drops below threshold:
  1. Select compatible realms (same patch, same language)
  2. Merge player directories (cross-realm character list)
  3. Enable cross-realm groups
  4. Enable cross-realm PvP queues
  5. Keep inventory/quest data separate per realm

Implementation:
  - Cross-realm directory service (Redis-backed)
  - Player appears with realm tag: "PlayerA [Realm1]"
  - Group formation validates cross-realm compatibility
  - Combat remains within same realm (data isolation)

9. Curveball Scenarios

Curveball 1: “A zone server experiences a sudden spike in entity count (10× normal) due to a world event. The tick rate drops from 20 Hz to 5 Hz. Players experience severe lag. How do you handle this?”

Analysis: World events (world boss spawns, invasions, natural disasters) can dramatically increase entity counts. A zone designed for 1,000 players and 10,000 entities suddenly handles 100,000 entities.

Immediate Response:

Phase 1: Graceful Degradation (0-5 seconds)
  1. Reduce tick rate target: 20 Hz → 10 Hz → 5 Hz
  2. Reduce simulation fidelity:
     - Skip non-critical AI updates
     - Reduce physics precision
     - Disable decorative entity updates
  3. Reduce network bandwidth:
     - Increase entity update interval (every 2 ticks → every 4 ticks)
     - Reduce position precision (float digits)
     - Stop sending non-visible entity updates
  4. Notify clients: "High population event — reduced update rate"

Phase 2: Load Shedding (5-30 seconds)
  1. Freeze non-essential systems:
     - Pause chat processing
     - Pause auction house updates
     - Pause quest progress tracking
  2. Entity culling:
     - Despawn non-essential NPCs temporarily
     - Reduce creature spawn rates
     - Merge nearby entities visually
  3. Client-side compensation:
     - Increase interpolation window
     - Enable client-side entity LOD (level of detail)

Phase 3: Horizontal Scaling (30-120 seconds)
  1. Spawn overflow zone instance
  2. Begin migrating players to overflow:
     - New players → overflow instance
     - Gradual migration of existing players
  3. Split entity processing:
     - Dedicated AI server for NPC processing
     - Dedicated physics server for collision

Prevention:

1. Predictive scaling:
   - World event schedule known in advance
   - Pre-scale zone servers 10 minutes before event
   - Reserve capacity for expected spikes

2. Entity budget system:
   - Each zone has max entity budget
   - World events reserve budget in advance
   - Regular spawns reduced when event active

3. Tiered simulation:
   - Critical entities (players, combat targets): full simulation
   - Important entities (nearby NPCs): reduced simulation
   - Background entities (distant NPCs): minimal simulation

4. Client-side assistance:
   - Clients run simplified AI for distant NPCs
   - Server validates critical actions only
   - Reduces server compute by 30-50%

Curveball 2: “A sophisticated cheat modifies the client’s memory to grant unlimited mana and bypass ability cooldowns. The cheat sends valid-looking packets. How do you detect and prevent this?”

Analysis: Memory modification cheats are among the most dangerous because they don’t modify network packets — they modify the client’s internal state, which then generates “valid” packets from the client’s perspective.

Detection Layers:

Layer 1: Server-Side Validation (Primary Defense)
  ┌─────────────────────────────────────────────┐
  │ Every ability cast is validated server-side: │
  │                                             │
  │ 1. Check actual mana (server-authoritative) │
  │    → Client claims 1000 mana, server says   │
  │      player has 50 mana → REJECT            │
  │                                             │
  │ 2. Check actual cooldown (server-side timer) │
  │    → Client casts fireball, server cooldown │
  │      not expired → REJECT                   │
  │                                             │
  │ 3. Check resource costs server-side          │
  │    → All costs validated against server     │
  │      state, not client claims               │
  └─────────────────────────────────────────────┘

Layer 2: State Consistency Checks
  ┌─────────────────────────────────────────────┐
  │ Periodic state reconciliation:              │
  │                                             │
  │ 1. Server sends periodic state snapshot      │
  │    → Client must acknowledge                │
  │ 2. If client state diverges significantly    │
  │    → Flag for review                        │
  │ 3. Mana regeneration rate check              │
  │    → If player gains mana faster than max   │
  │      regen rate → impossible → flag         │
  └─────────────────────────────────────────────┘

Layer 3: Client Integrity Verification
  ┌─────────────────────────────────────────────┐
  │ Anti-tamper measures:                       │
  │                                             │
  │ 1. Memory integrity checks (periodic)        │
  │    → Hash critical memory regions           │
  │    → Compare with known good values         │
  │ 2. Anti-debugging                            │
  │    → Detect attached debuggers              │
  │    → Detect memory scanning tools           │
  │ 3. Code signature verification               │
  │    → Verify client binary hasn't been       │
  │      modified                               │
  │ 4. Obfuscation                               │
  │    → Critical values stored obfuscated      │
  │    → Mana, cooldowns not in plain memory    │
  └─────────────────────────────────────────────┘

Layer 4: Behavioral Analysis
  ┌─────────────────────────────────────────────┐
  │ Statistical detection:                      │
  │                                             │
  │ 1. Ability usage rate analysis               │
  │    → If player casts abilities faster than  │
  │      possible with cooldowns → flag         │
  │ 2. Damage output analysis                    │
  │    → If DPS exceeds theoretical maximum     │
  │      for class/level/gear → flag            │
  │ 3. Resource consumption analysis             │
  │    → If mana never depletes during combat   │
  │      → impossible → flag                    │
  └─────────────────────────────────────────────┘

Response Protocol:

Detection → Action Matrix:

Confidence Level    Action
─────────────────────────────────────────────
Low (30-50%)        Silent monitoring, collect evidence
Medium (50-70%)     Kick from game, require relaunch
High (70-90%)       Temporary ban (1-7 days), review queue
Critical (90%+)     Permanent ban, hardware ID ban, IP flag

Evidence Collection:
  - Record all packets from suspected client
  - Record server-side state at time of detection
  - Record behavioral metrics (DPS, ability rate, etc.)
  - Store for manual review and ML training

Key Principle: The server is the ultimate authority. The client is an untrusted input device. No client claim about game state is ever trusted — everything is validated against server-authoritative state.


Curveball 3: “You need to perform a zero-downtime game patch that changes the combat formula. Active players are mid-combat. How do you roll out the patch without disrupting gameplay?”

Analysis: Game patches that change core mechanics (combat formulas, ability balances, economy) cannot simply restart servers — active players would be disconnected mid-combat. The patch must be rolled out seamlessly.

Rollout Strategy:

flowchart TD
    A["Prepare patch<br/>(new binary, config)"] --> B["Deploy to canary<br/>realm (1 realm)"]
    B --> C{"Canary stable?<br/>30 min observation"}
    C -->|No| D["Rollback canary"]
    C -->|Yes| E["Batch 1:<br/>10% of realms"]
    E --> F{"Batch 1 stable?<br/>15 min observation"}
    F -->|No| G["Rollback batch 1<br/>Investigate"]
    F -->|Yes| H["Batch 2:<br/>50% of realms"]
    H --> I{"Batch 2 stable?<br/>15 min observation"}
    I -->|No| J["Rollback batch 2"]
    I -->|Yes| K["Batch 3:<br/>Remaining realms"]
    K --> L["Patch complete"]

Technical Implementation:

Step 1: Configuration-Based Combat Formulas
  - Combat formulas are NOT hardcoded
  - Stored in configuration files / database
  - Zone servers load formulas at startup
  - Formulas can be updated without binary patch

  Example config:
  {
    "damage_formula": {
      "physical": "max(1, attacker.attack * ability.multiplier - defender.armor * 0.5)",
      "magic": "max(1, attacker.spellPower * ability.multiplier * (1 - defender.resistance))",
      "crit_chance": "attacker.critRating / 100",
      "crit_multiplier": 2.0
    }
  }

Step 2: Hot-Reload Configuration
  - Zone servers watch for config changes
  - On change: validate new config, apply atomically
  - Active combat uses current formula (no mid-fight change)
  - New combat encounters use new formula

Step 3: Binary Patch (if code changes needed)
  - Blue-green deployment for zone servers:
    a. Deploy new version alongside old version
    b. New version handles new connections only
    c. Old version continues serving existing players
    d. As players log out, they reconnect to new version
    e. When old version has 0 players, shut it down

  - Graceful transition:
    - Players mid-combat: continue on old version
    - Combat ends → player returned to zone on new version
    - No mid-combat disruption

Step 4: Client Patch
  - Client patch required for UI / asset changes
  - Launcher enforces patch before login
  - Grace period: old clients can play for N hours
  - After grace period: old clients rejected

Edge Cases:

Q: What if a player is in a 40-minute raid when the patch deploys?
A: They continue on the old version. After the raid, they're
   transferred to the new version. The raid instance runs to completion.

Q: What if the new formula causes a bug?
A: Canary detection catches it. Rollback reverts config to previous
   version. New config is rejected, old config reloaded.

Q: What about cross-version compatibility?
A: Players on different versions cannot group together.
   Cross-version play is blocked until all players are on the same version.

Curveball 4: “Two zone servers lose connectivity to each other (network partition). Players in both zones are in the same party. What happens and how do you handle it?”

Analysis: Network partitions between zone servers break cross-zone communication. Party members in different zones lose the ability to chat, coordinate, and see each other’s status.

Impact Assessment:

Feature Impact Severity
Party chat Broken Medium
Party status visibility Stale Low
Cross-zone quests Blocked Medium
Group formation Blocked Low
Active combat Unaffected (same zone) None
Zone transition Blocked across partition High

Handling Strategy:

Phase 1: Detection (0-5 seconds)
  - Heartbeat between zone servers fails
  - Cross-Zone Router detects partition
  - Affected parties flagged

Phase 2: Graceful Degradation (5-30 seconds)
  - Party chat: Queue messages, deliver when partition heals
  - Party status: Show "connection unstable" indicator
  - Zone transitions across partition: Block with message
  - Active cross-zone quests: Pause, resume later

Phase 3: Player Communication (30-60 seconds)
  - Notify affected players: "Network issue — some features limited"
  - Provide ETA if known
  - Suggest alternatives (stay in current zone)

Phase 4: Recovery (partition heals)
  - Replay queued messages
  - Resume paused quests
  - Restore full functionality
  - Log incident for analysis

Phase 5: If partition persists > 5 minutes
  - Consider merging affected zones (if geographically close)
  - Offer teleport options to reunite parties
  - Escalate to network team

Prevention:

1. Redundant network paths between zone servers
2. Zone servers co-located in same availability zone
3. Cross-zone communication via Kafka (survives transient failures)
4. Automatic failover to backup network path
5. Regular network partition drills

Curveball 5: “You discover that your entity interpolation causes ‘rubber-banding’ for players with high ping (>150ms). How do you fix this without changing the tick rate?”

Analysis: Rubber-banding occurs when the client’s predicted position diverges significantly from the server’s authoritative position, causing the client to be “snapped back” repeatedly. This is especially problematic for high-ping players.

Root Cause Analysis:

High-ping player scenario:
  - Client predicts movement for 150ms (round-trip time)
  - Server receives stale input, corrects position
  - Client sees correction, snaps back
  - Client predicts again, diverges again
  - Cycle repeats → rubber-banding

Key factors:
  1. Prediction horizon too long (150ms of prediction)
  2. Server correction too aggressive (full snap)
  3. No smoothing on corrections
  4. Client input buffering too short

Solutions:

Solution 1: Extended Client Input Buffering
  - Server accepts inputs with timestamps up to 300ms in the past
  - Server replays from last agreed state, not current state
  - Reduces correction magnitude significantly

Solution 2: Smooth Correction (No Hard Snap)
  - Instead of snapping to server position:
    correctionSpeed = divergence / smoothingTime
    clientPosition += correctionSpeed * deltaTime
  - Smoothing time: 200-500ms (configurable by ping)
  - High-ping players get longer smoothing

Solution 3: Server-Side Client Prediction
  - Server maintains predicted state for each client
  - Server compares prediction with actual
  - Server sends correction deltas, not absolute positions
  - Smaller deltas = less rubber-banding

Solution 4: Adaptive Tick Rate per Client
  - Low ping (< 50ms): 20 Hz updates
  - Medium ping (50-100ms): 15 Hz updates
  - High ping (100-150ms): 10 Hz updates
  - Very high ping (> 150ms): 5 Hz + extrapolation hints
  - Reduces correction frequency for high-ping players

Solution 5: Position Smoothing on Client
  - Client receives server position
  - Instead of using directly: smoothPosition = lerp(smoothPosition, serverPosition, alpha)
  - Alpha decreases with higher ping
  - Creates illusion of smooth movement

Combined Approach:
  1. Extended input buffering (server)
  2. Smooth corrections (client)
  3. Adaptive update rate (server)
  4. Position smoothing (client)
  5. Extrapolation hints for fast movers (server)

Implementation Details:

Server-side changes:
  - Input buffer: 300ms (was 100ms)
  - Correction smoothing: send delta, not absolute
  - Adaptive tick: per-client update rate based on ping

Client-side changes:
  - Position smoothing: lerp with ping-based alpha
  - Extended prediction: use server hints for extrapolation
  - Visual compensation: slight FOV increase for high-ping players

Testing:
  - Simulate various ping levels (50ms, 100ms, 150ms, 200ms)
  - Measure rubber-banding frequency
  - Measure position accuracy
  - Player feedback on movement feel

10. Failure Modes and Mitigations

10.1 Failure Mode Matrix

Component Failure Mode Impact Detection Mitigation RTO
Zone Server Crash Zone goes dark, players disconnected Health check, heartbeat Auto-restart, restore from Redis < 30s
Zone Server Tick rate drop Lag, rubber-banding Tick rate monitoring Graceful degradation, load shed < 10s
Zone Server Memory leak Gradual degradation, OOM crash Memory monitoring Rolling restart, memory limits < 5 min
Game Gateway Connection flood New connections rejected Connection rate alert Rate limiting, auto-scale < 1 min
Redis Node failure Hot state lost for shard Sentinel alert Failover to replica < 10s
Redis Cluster partition Split-brain, data inconsistency Quorum check Pause writes, resolve < 1 min
PostgreSQL Primary failure Character saves blocked Connection errors Patroni failover < 30s
PostgreSQL Replication lag Stale character data Lag monitoring Route to primary < 1 min
Kafka Broker failure Event loss Consumer lag Replica failover < 30s
Network Zone partition Cross-zone features broken Heartbeat miss Queue messages, notify Variable
Network DDoS attack Service degradation Traffic anomaly WAF, rate limiting, scrubbing < 5 min
Client Cheat detected Fairness compromised Anti-cheat flags Kick, ban, review Immediate
Database Disk full Writes fail Disk usage alert Emergency cleanup, expand < 5 min
Config Bad deploy Game-breaking bug Error rate spike Rollback, canary detection < 2 min

10.2 Critical Path: Game Tick Failure Analysis

flowchart TD
    A["Tick starts"] --> B{"Input collection<br/>complete?"}
    B -->|Timeout| C["Use last known inputs<br/>continue tick"]
    B -->|Yes| D["Physics simulation"]
    D --> E{"Physics complete<br/>in time?"}
    E -->|No| F["Reduce simulation<br/>precision, continue"]
    E -->|Yes| G["Combat resolution"]
    G --> H{"Combat complete<br/>in time?"}
    H -->|No| I["Defer non-critical<br/>combat, continue"]
    H -->|Yes| J["Network broadcast"]
    J --> K{"Broadcast complete<br/>in time?"}
    K -->|No| L["Drop non-critical<br/>updates, continue"]
    K -->|Yes| M["Tick complete"]
    C --> D
    F --> G
    I --> G
    L --> M

    M --> N{"Tick took<br/>> 50ms?"}
    N -->|Yes| O["Alert: tick overrun<br/>log details"]
    N -->|No| P["Normal operation"]

10.3 Disaster Recovery Plan

RPO (Recovery Point Objective):
  - Game state: 50ms (last tick in Redis)
  - Character data: 60s (auto-save interval)
  - Inventory: 0 (immediate save on changes)
  - Chat: 5 min (Kafka retention)

RTO (Recovery Time Objective):
  - Zone server: 30 seconds (auto-restart + Redis restore)
  - Game gateway: 1 minute (auto-scale replacement)
  - Redis cluster: 10 seconds (sentinel failover)
  - PostgreSQL: 30 seconds (Patroni failover)
  - Full realm: 2 minutes (cascade restart)
  - Full region: 10 minutes (cross-region failover)

Backup Strategy:
  - PostgreSQL: Continuous WAL archiving + daily full backup
  - Redis: AOF everysec + RDB every 15 min
  - Kafka: 7-day retention (replayable)
  - MongoDB: Daily snapshot + oplog
  - S3: Versioned backups of all critical data

Recovery Procedures:
  1. Zone server crash:
     a. Kubernetes auto-restarts pod
     b. New pod connects to Redis, loads zone state
     c. Players reconnect automatically (client reconnect logic)
     d. Missing players receive "server restart" notification

  2. Redis failure:
     a. Sentinel promotes replica to master
     b. Zone servers reconnect to new master
     c. If no replica: restore from AOF / Kafka replay
     d. Players may experience brief disconnect

  3. PostgreSQL failure:
     a. Patroni promotes replica to primary
     b. Services reconnect to new primary
     c. Character saves resume
     d. If data loss: restore from WAL archive

  4. Full region failure:
     a. DNS failover to secondary region
     b. Players reconnect to secondary region
     c. Data synced from cross-region replication
     d. Estimated downtime: 5-10 minutes

10.4 Monitoring and Alerting

Metric Warning Critical Action
Tick rate < 18 Hz < 10 Hz Scale zone server
Tick duration p99 > 45ms > 60ms Investigate, optimize
Player count per zone > 800 > 950 Create overflow instance
Server CPU > 70% > 90% Auto-scale
Server memory > 80% > 95% Restart, investigate leak
Network latency (internal) > 5ms > 20ms Network investigation
Redis latency p99 > 1ms > 5ms Scale Redis
Redis memory > 75% > 90% Add shard
PG write latency > 5ms > 20ms Scale DB, check queries
PG replication lag > 5s > 30s Route to primary
Kafka consumer lag > 10K > 100K Add consumers
Connection error rate > 0.5% > 2% Check gateway health
Anti-cheat flags/min > 10 > 50 Investigate attack
Packet loss rate > 0.1% > 1% Network investigation
Client disconnect rate > 1%/min > 5%/min Service health check

10.5 Chaos Engineering Plan

Regular chaos experiments:
  1. Kill random zone server → verify auto-restart + player reconnect
  2. Kill Redis node → verify failover + state recovery
  3. Simulate network partition → verify graceful degradation
  4. Inject 200ms latency → verify rubber-banding mitigation
  5. Simulate DDoS → verify rate limiting + auto-scale
  6. Kill PostgreSQL primary → verify Patroni failover
  7. Corrupt Redis data → verify Kafka replay recovery
  8. Deploy bad config → verify canary detection + rollback

Frequency: Weekly automated, monthly full-scale drill

Appendix A: Glossary

Term Definition
Tick One iteration of the game loop (50ms at 20 Hz)
Entity Any object in the game world (player, NPC, projectile, effect)
Interpolation Smoothing entity movement between received positions
Extrapolation Predicting entity position when no update received
Rubber-banding Client position snapped back by server correction
Client Prediction Client predicts outcome before server confirmation
Server Reconciliation Server corrects client prediction discrepancies
Lag Compensation Rewinding simulation to validate actions at client’s perceived time
Zone A partitioned area of the game world managed by one server
Realm A complete game world with multiple zones
Instance A private copy of a zone (dungeons, raids, arenas)
Delta Encoding Sending only changes since last update, not full state
Head-of-Line Blocking TCP behavior where lost packet blocks all subsequent packets
FOW Forward-Only Write — write-once, append-only data model

Appendix B: Technology Stack Summary

Layer Technology Rationale
Game Client Unity / Unreal Engine Mature game engines, cross-platform
Game Server C++ / Rust Performance-critical, low-level control
Network Custom UDP protocol No head-of-line blocking, selective reliability
Gateway Go / Rust High-performance networking
Microservices Go / Java Performance + ecosystem
Game State Redis Cluster Sub-millisecond access, in-memory
Event Bus Apache Kafka Durable, scalable streaming
Primary DB PostgreSQL ACID, mature, JSONB support
Document DB MongoDB Chat, logs, flexible schema
Cache Redis Sessions, leaderboards, hot data
Object Storage S3 / MinIO Backups, assets
Orchestration Kubernetes Auto-scaling, self-healing
Monitoring Prometheus + Grafana Metrics, alerting
Tracing Jaeger Distributed tracing
Log Aggregation ELK Stack Centralized logging
CI/CD Jenkins / GitLab CI Automated deployment

Appendix C: Protocol Wire Format Details

Transport: UDP (custom reliability layer)

Packet Header (10 bytes):
  Byte 0:    Protocol version (1)
  Bytes 1-4: Sequence number (uint32, big-endian)
  Bytes 5-6: Message type (uint16, big-endian)
  Bytes 7-8: Payload length (uint16, big-endian)
  Byte 9:    Flags (reliable, ack, fragment)

Reliability Layer:
  - Reliable packets: Assigned seq number, retransmitted if no ACK
  - Unreliable packets: Fire-and-forget, no retransmission
  - Sequenced packets: Ordered delivery, no retransmission
  - ACK packets: Acknowledge received seq numbers
  - NAK packets: Request retransmission of missing seq numbers

Fragmentation:
  - Max payload: 1400 bytes (MTU safe)
  - Large messages split into fragments
  - Fragment header: {fragmentId, fragmentIndex, totalFragments}
  - Reassembly on receiving end

Encryption:
  - DTLS 1.3 for transport encryption
  - AES-256-GCM for payload encryption
  - Key exchange via SRTP