Analytics & TelemetryQuestion #5ADVANCED

Google Analytics: Real-Time Event Ingestion & Lambda Architecture

High-throughput event tracking, real-time stream ingestion, batch OLAP aggregations, and privacy-preserving analytics at global scale.

Category: Category 4: Analytics & Telemetry•Updated: 2026-08-30
#Analytics#Event Ingestion#Lambda Architecture#Aggregation#Privacy

Design an Analytics System like Google Analytics

Table of Contents

  1. Requirements Clarification
  2. Back-of-Envelope Calculations
  3. System Interface Design
  4. High-Level Architecture
  5. Core Component Design
  6. Data Model
  7. Key Design Decisions
  8. Scalability Strategy
  9. Curveball Scenarios
  10. Failure Modes and Mitigations

1. Requirements Clarification

1.1 Functional Requirements

Event Collection

  • Collect pageview, event, transaction, and custom events from web/mobile apps
  • Support client-side JavaScript snippet and mobile SDKs
  • Handle offline event buffering and batch upload
  • Support custom dimensions and metrics (up to 25 custom dimensions, 25 custom metrics per property)
  • Event parameters: up to 25 per event, key-value pairs

Real-Time Reporting

  • Real-time dashboard showing active users, top pages, traffic sources (last 30 minutes)
  • Updates within 1-2 seconds of event ingestion
  • Real-time user geographic distribution

Historical Reporting

  • Pre-built reports: Audience, Acquisition, Behavior, Conversions
  • Custom report builder with drag-and-drop dimensions/metrics
  • Date range comparison (e.g., this week vs. last week)
  • Cohort analysis, funnel visualization, path analysis
  • Data export to BigQuery-like data warehouse

User Identification

  • Anonymous user tracking via client-generated IDs
  • User ID linking for cross-device tracking (opt-in)
  • Session management (30-minute inactivity timeout)

Privacy & Compliance

  • GDPR compliance: data deletion, consent management
  • CCPA compliance: opt-out mechanisms
  • IP address anonymization
  • Data retention controls (14 months default, configurable)
  • Data processing agreements

Integrations

  • Google Ads, Search Console integration
  • E-commerce platforms (Shopify, WooCommerce)
  • Data export APIs
  • Webhooks for event streaming

1.2 Non-Functional Requirements

Requirement Target Rationale
Event Ingestion Throughput 100M+ events/sec peak Global web traffic scale
Event Collection Latency < 2 seconds (p99) Near-real-time user experience
Real-Time Report Freshness < 5 seconds Dashboard responsiveness
Historical Report Latency < 30 seconds for standard reports Analyst productivity
Data Durability 99.99% Analytics data must not be lost
Availability 99.9% (collection), 99.5% (reporting) Collection is critical path
Privacy GDPR/CCPA compliant Legal requirement
Scalability Support 10M+ websites Market coverage
Cost Efficiency Sub-penny per 1,000 events Free tier viability

2. Back-of-Envelope Calculations

2.1 Assumption Sets

Parameter Conservative Realistic Aggressive
Tracked websites 1M 10M 50M
Daily pageviews/website 10K 50K 100K
Events per pageview 3 5 8
Total daily events 30B 500B 400B
Events/sec average 350K 5.8M 4.6M
Peak events/sec (3×) 1M 17.4M 13.9M
Avg event payload 1 KB 1.5 KB 2 KB
Daily data volume 30 TB 750 TB 800 TB
Monthly data volume 900 TB 22.5 PB 24 PB

2.2 Storage Calculations

Raw Event Storage (before aggregation):

Retention Conservative Realistic Aggressive
7 days (real-time) 210 TB 5.25 PB 5.6 PB
30 days 900 TB 22.5 PB 24 PB
14 months 270 PB 67.5 EB 72 EB

Aggregated Data Storage (1000:1 compression ratio via aggregation):

Granularity Realistic Monthly Notes
Per-second 22.5 TB Real-time dashboard
Per-minute 375 GB Intraday analysis
Per-hour 9.4 GB Daily trend analysis
Per-day 375 MB Long-term trends
Per-month 31 MB Year-over-year

Key Insight: Aggregated data is ~0.01% of raw data. Store raw for 7-30 days, aggregated indefinitely.

2.3 Bandwidth Calculations

Ingestion Pipeline:

  • 5.8M events/sec × 1.5 KB = 8.7 GB/s average ingestion
  • Peak: 26 GB/s
  • Collection endpoints need global CDN + edge processing

Internal Data Movement:

  • Raw → Streaming processing: 8.7 GB/s
  • Streaming → Real-time store: 2 GB/s (filtered/aggregated)
  • Batch → Historical store: 8.7 GB/s (same data, different path)

2.4 Compute Estimates

Collection Endpoints:

  • 17.4M peak events/sec
  • Each server handles ~50K events/sec (simple write + ack)
  • Need ~350 collection servers globally (distributed across edge locations)

Stream Processing:

  • 5.8M events/sec to process
  • Each processing instance handles ~100K events/sec
  • Need ~58 processing instances per pipeline stage

Query Serving:

  • 100K concurrent report queries
  • Each query server handles ~100 QPS
  • Need ~1,000 query servers

3. System Interface Design

3.1 Collection Endpoint (Measurement Protocol)

POST /collect?v=2&tid=GA-XXXXXXXX&cid=abc123def456&t=pageview&dp=/home&dt=Home%20Page&dr=https://example.com&dl=https://example.com/home&uid=user123
Host: collection.analytics.com
Content-Type: application/x-www-form-urlencoded

Batch Collection:
POST /batch
[
  { "v": "2", "tid": "GA-XXXXXXXX", "cid": "abc123", "t": "pageview", "dp": "/home" },
  { "v": "2", "tid": "GA-XXXXXXXX", "cid": "abc123", "t": "event", "ec": "click", "ea": "button", "el": "signup" },
  { "v": "2", "tid": "GA-XXXXXXXX", "cid": "abc123", "t": "screen_view", "sn": "Home" }
]

Response 204 No Content (always succeed, never block the client)

3.2 Real-Time API

GET /v1/properties/{propertyId}/realtime:report
?metrics=activeUsers
&dimensions=country,pageTitle
&filter=country!=Internal
&pageSize=100

Response 200:
{
  "rows": [
    {
      "dimensionValues": [
        { "value": "United States" },
        { "value": "Home Page" }
      ],
      "metricValues": [
        { "value": "15234" }
      ]
    }
  ],
  "rowCount": 1,
  "totalActiveUsers": 45678
}

3.3 Historical Reporting API

POST /v1/properties/{propertyId}:runReport

{
  "dateRanges": [{
    "startDate": "2026-01-01",
    "endDate": "2026-01-31"
  }],
  "dimensions": [
    { "name": "country" },
    { "name": "sessionDefaultChannelGroup" }
  ],
  "metrics": [
    { "name": "activeUsers" },
    { "name": "screenPageViews" },
    { "name": "bounceRate" }
  ],
  "orderBys": [
    { "metric": { "metricName": "activeUsers" }, "desc": true }
  ],
  "limit": 100,
  "segments": [{
    "segmentId": "newVsReturning::new"
  }]
}

Response 200:
{
  "reports": [{
    "dimensionHeaders": [
      { "name": "country", "columnIndex": 0 },
      { "name": "sessionDefaultChannelGroup", "columnIndex": 1 }
    ],
    "metricHeaders": [{
      "metrics": [
        { "name": "activeUsers", "type": "TYPE_INTEGER" },
        { "name": "screenPageViews", "type": "TYPE_INTEGER" },
        { "name": "bounceRate", "type": "TYPE_FLOAT" }
      ]
    }],
    "rows": [
      {
        "dimensionValues": [
          { "value": "United States" },
          { "value": "Organic Search" }
        ],
        "metricValues": [
          { "value": "125000" },
          { "value": "450000" },
          { "value": "0.35" }
        ]
      }
    ]
  }]
}

3.4 Data Export API

POST /v1/properties/{propertyId}/eventStreams:subscribe

{
  "destination": {
    "type": "BIGQUERY",
    "bigqueryDestination": {
      "projectId": "my-project",
      "datasetId": "analytics",
      "tableIdTemplate": "events_{DATE}"
    }
  },
  "filter": {
    "eventFilter": {
      "eventNames": ["page_view", "purchase", "add_to_cart"]
    }
  }
}

3.5 Management API

POST /v1/properties/{propertyId}/customDimensions
{
  "displayName": "User Tier",
  "parameterName": "user_tier",
  "scope": "USER"
}

POST /v1/properties/{propertyId}/dataDeletions
{
  "propertyId": "GA-XXXXXXXX",
  "userId": "user123",
  "deleteEventCondition": {
    "eventDateRange": {
      "startDate": "2026-01-01",
      "endDate": "2026-08-28"
    }
  }
}

4. High-Level Architecture

graph TB
    subgraph "Client Side"
        WEB[Web JS Snippet]
        MOBILE[Mobile SDK]
        SERVER[Server SDK]
        CDP[CDP / Tag Manager]
    end

    subgraph "Edge Collection Layer"
        CDN[Global CDN / Edge Nodes]
        COLLECT1[Collection Endpoint EU]
        COLLECT2[Collection Endpoint US]
        COLLECT3[Collection Endpoint APAC]
    end

    subgraph "Ingestion Layer"
        KAFKA[Kafka Cluster - Raw Events]
        VALIDATE[Event Validation Service]
        ENRICH[Event Enrichment Service]
    end

    subgraph "Real-Time Pipeline"
        FLINK[Stream Processing - Flink]
        RT_AGG[Real-Time Aggregator]
        RT_STORE[Real-Time Store - Redis/Cassandra]
    end

    subgraph "Batch Pipeline"
        BATCH_KAFKA[Kafka - Validated Events]
        SPARK[Batch Processing - Spark]
        AGG_ENGINE[Aggregation Engine]
    end

    subgraph "Storage Layer"
        RAW_STORE[Raw Event Store - Object Storage]
        RT_DB[Real-Time DB]
        HIST_DB[Historical Aggregated DB - Columnar]
        USER_DB[User/Session DB]
        META_DB[Property Metadata DB]
    end

    subgraph "Query & Reporting"
        RT_API[Real-Time API]
        HIST_API[Historical Reporting API]
        QUERY_ENGINE[Query Engine - Presto/Drill]
        CACHE[Query Result Cache]
    end

    subgraph "Management & Admin"
        MGMT_API[Management API]
        CONSENT[Consent Management]
        PRIVACY[Privacy Service]
        EXPORT[Data Export Service]
    end

    subgraph "Dashboard"
        UI[Web Dashboard]
        EXPORT_UI[Export UI]
    end

    WEB --> CDN
    MOBILE --> CDN
    SERVER --> CDN
    CDP --> CDN

    CDN --> COLLECT1
    CDN --> COLLECT2
    CDN --> COLLECT3

    COLLECT1 --> KAFKA
    COLLECT2 --> KAFKA
    COLLECT3 --> KAFKA

    KAFKA --> VALIDATE
    VALIDATE --> ENRICH
    ENRICH --> FLINK
    ENRICH --> BATCH_KAFKA

    FLINK --> RT_AGG
    RT_AGG --> RT_STORE

    BATCH_KAFKA --> SPARK
    SPARK --> AGG_ENGINE
    AGG_ENGINE --> HIST_DB

    ENRICH --> RAW_STORE

    RT_STORE --> RT_DB
    RT_DB --> RT_API

    HIST_DB --> HIST_API
    HIST_DB --> QUERY_ENGINE
    QUERY_ENGINE --> CACHE
    CACHE --> HIST_API

    META_DB --> MGMT_API
    CONSENT --> VALIDATE
    PRIVACY --> RAW_STORE
    PRIVACY --> HIST_DB
    EXPORT --> RAW_STORE

    RT_API --> UI
    HIST_API --> UI
    MGMT_API --> EXPORT_UI

5. Core Component Design

5.1 Event Collection Layer

Design Goals: Maximum throughput, minimum latency, fire-and-forget reliability

graph LR
    subgraph "Browser"
        SNIPPET[Analytics Snippet]
        BEACON[Beacon API / Image Pixel]
        BUFFER[Event Buffer]
    end

    subgraph "Edge"
        CDN_NODE[CDN Edge Node]
        COLLECTOR[Collection Service]
    end

    subgraph "Backend"
        KAFKA[Kafka]
    end

    SNIPPET --> BUFFER
    BUFFER --> BEACON
    BEACON --> CDN_NODE
    CDN_NODE --> COLLECTOR
    COLLECTOR --> KAFKA

Collection Strategy:

  • Beacon API (navigator.sendBeacon): Guaranteed delivery on page unload, async, no blocking
  • Image pixel fallback: For older browsers, 1x1 transparent GIF with event data in URL
  • Batching: Client batches events (up to 25 per batch) to reduce HTTP overhead
  • Offline buffering: Mobile SDKs store events locally, upload when connectivity restored
  • CDN caching: Collection endpoints set Cache-Control: no-store to prevent CDN caching of event data

Collection Endpoint Design:

  • Stateless, horizontally scalable
  • Write events to Kafka and immediately return 204
  • No validation at collection time (validate in pipeline)
  • Rate limiting per property (not per user) to prevent abuse
  • Geographic routing: events routed to nearest regional Kafka cluster

5.2 Event Validation and Enrichment

Validation Pipeline:

graph TD
    RAW[Raw Event from Kafka] --> PARSE[Parse Event]
    PARSE --> VALID{Valid Format?}
    VALID -->|No| DROP[Drop + Count]
    VALID -->|Yes| ENRICH[Enrich Event]

    ENRICH --> GEO[Geo IP Lookup]
    GEO --> DEVICE[Device Detection]
    DEVICE --> CONSENT{Consent Valid?}
    CONSENT -->|No| ANON[Anonymize + Store]
    CONSENT -->|Yes| STORE[Store Enriched Event]

    ANON --> RT[Real-Time Pipeline]
    STORE --> RT
    STORE --> BATCH[Batch Pipeline]
    STORE --> RAW[Raw Archive]

Enrichment Data Sources:

  • GeoIP database: MaxMind or internal IP-to-location mapping (updated daily)
  • Device detection: User-Agent parsing → browser, OS, device type, screen resolution
  • Currency conversion: For e-commerce events, convert to property’s currency
  • Timezone normalization: All timestamps stored in UTC, enriched with user timezone

5.3 Real-Time Processing Pipeline

Architecture:

graph LR
    subgraph "Stream Processing"
        INPUT[Kafka Topic: enriched-events]
        FLINK1[Flink Job: Session Builder]
        FLINK2[Flink Job: Counter Aggregator]
        FLINK3[Flink Job: Top-N Calculator]
    end

    subgraph "Real-Time Storage"
        COUNTER[Counter Store - Redis]
        SESSION[Session Store - Cassandra]
        TOPN[Top-N Store - Redis Sorted Sets]
    end

    INPUT --> FLINK1
    INPUT --> FLINK2
    INPUT --> FLINK3

    FLINK1 --> SESSION
    FLINK2 --> COUNTER
    FLINK3 --> TOPN

Processing Logic:

Metric Computation Window Storage
Active Users Count distinct user IDs in last 30 min Sliding 30-min Redis (TTL 30 min)
Pageviews/sec Rate counter 1-sec tumbling Redis counter
Top Pages Top-N by pageviews 30-min sliding Redis sorted set
Traffic Sources Count by source/medium 30-min sliding Redis hash
Real-Time Events Raw event count 1-min tumbling Redis counter

Session Building:

  • Assign session ID based on client ID + 30-min inactivity gap
  • Maintain session state in Redis with TTL
  • Cross-device session merging via User ID (when provided)

5.4 Batch Processing Pipeline

Lambda Architecture:

graph TB
    subgraph "Speed Layer (Real-Time)"
        STREAM[Stream Processing]
        RT_VIEW[Real-Time View]
    end

    subgraph "Batch Layer"
        BATCH[Batch Processing - Spark]
        BATCH_VIEW[Batch View - Aggregated Tables]
    end

    subgraph "Serving Layer"
        SERVING[Serving Layer]
        MERGE[Merge Batch + Real-Time]
    end

    subgraph "Presentation"
        DASHBOARD[Dashboard / API]
    end

    STREAM --> RT_VIEW
    BATCH --> BATCH_VIEW
    RT_VIEW --> MERGE
    BATCH_VIEW --> MERGE
    MERGE --> SERVING
    SERVING --> DASHBOARD

Batch Processing Stages:

Stage Input Output Schedule
Raw → Cleaned Raw events Validated, enriched events Every 5 min
Session Attribution Cleaned events Session-level data Every 15 min
Daily Aggregation Session data Daily aggregated metrics Daily (next day 02:00 UTC)
Monthly Rollup Daily aggregates Monthly summaries Monthly
Cohort Analysis Session data Cohort retention tables Weekly
Funnel Analysis Event sequences Funnel conversion rates Daily

Aggregation Engine Design:

  • Pre-aggregate at multiple granularities (second, minute, hour, day, month)
  • Use columnar storage (Parquet/ORC) for efficient aggregation queries
  • Materialized views for common report patterns
  • Incremental aggregation: only recompute affected time windows

5.5 Query Engine

Query Optimization:

graph TD
    QUERY[User Query] --> PARSE[Parse Query]
    PARSE --> PLAN[Generate Query Plan]
    PLAN --> ROUTE{Route Decision}

    ROUTE -->|Real-Time| RT[Real-Time Store]
    ROUTE -->|Cached| CACHE[Query Cache]
    ROUTE -->|Aggregated| AGG[Aggregated Tables]
    ROUTE -->|Ad-Hoc| RAW[Raw Event Store]

    RT --> FORMAT[Format Results]
    CACHE --> FORMAT
    AGG --> FORMAT
    RAW --> FORMAT

    FORMAT --> RESULT[Return Results]

    RESULT --> CACHE2{Cache Result?}
    CACHE2 -->|Yes| STORE_CACHE[Store in Cache]
    CACHE2 -->|No| DONE[Done]

Query Routing Strategy:

Query Type Data Source Expected Latency Cache TTL
Real-time dashboard Real-time store < 500ms 5 sec
Standard report (pre-built) Aggregated tables < 2 sec 5 min
Custom report (common) Aggregated tables < 5 sec 1 hour
Custom report (rare) Raw store (Presto) 10-30 sec 24 hours
Export to BigQuery Raw event store Async N/A

5.6 Privacy and Compliance

GDPR Data Deletion Flow:

sequenceDiagram
    participant User
    participant UI as Dashboard
    participant API as Privacy API
    participant RAW as Raw Store
    participant AGG as Aggregated Store
    participant RT as Real-Time Store

    User->>UI: Request data deletion
    UI->>API: Delete events for user_id=X
    API->>RT: Remove real-time data
    RT-->>API: Done
    API->>RAW: Mark events for deletion
    RAW-->>API: Done (lazy deletion)
    API->>AGG: Trigger re-aggregation
    AGG-->>API: Scheduled
    API-->>UI: Deletion initiated
    Note over AGG: Re-aggregation runs<br/>during next batch cycle

Privacy Features:

  • IP anonymization: Last octet(s) stripped before storage
  • Consent mode: Events collected but not processed if consent denied
  • Data retention: Configurable per property (14 months default)
  • User-level deletion: Delete all events for a specific user ID
  • Property-level deletion: Delete all data for a property
  • Audit trail: All deletion requests logged for compliance

6. Data Model

6.1 Raw Event Schema

-- Raw events stored as Parquet files in object storage
-- Partitioned by: property_id, event_date

CREATE EXTERNAL TABLE raw_events (
    event_id            STRING,         -- UUID, generated at collection
    property_id           STRING,         -- GA-XXXXXXXX
    client_id               STRING,         -- Browser-generated unique ID
    user_id                   STRING,         -- Optional, developer-provided
    event_timestamp       BIGINT,         -- Event time in microseconds
    ingestion_timestamp   BIGINT,         -- Server receive time
    event_name            STRING,         -- page_view, click, purchase, etc.
    event_params            MAP<STRING, STRUCT<
        string_value: STRING,
        int_value: BIGINT,
        float_value: DOUBLE,
        double_value: DOUBLE
    >>,
    session_id            STRING,
    user_properties     MAP<STRING, STRUCT<
        value: STRING,
        set_timestamp: BIGINT
    >>,
    geo                 STRUCT<
        country: STRING,
        region: STRING,
        city: STRING,
        sub_continent: STRING,
        continent: STRING
    >,
    device                STRUCT<
        category: STRING,       -- mobile, desktop, tablet
        operating_system: STRING,
        browser: STRING,
        browser_version: STRING,
        mobile_marketing_name: STRING,
        mobile_model_name: STRING,
        mobile_brand_name: STRING,
        is_mobile: BOOLEAN,
        screen_resolution: STRING
    >,
    traffic_source        STRUCT<
        source: STRING,
        medium: STRING,
        campaign: STRING,
        term: STRING,
        content: STRING
    >,
    page                  STRUCT<
        display_name: STRING,
        host: STRING,
        path: STRING,
        referrer: STRING
    >,
    consent               STRUCT<
        ad_personalization: BOOLEAN,
        ad_storage: BOOLEAN,
        analytics_storage: BOOLEAN
    >
)
PARTITIONED BY (property_id STRING, event_date STRING)
STORED AS PARQUET;

6.2 Real-Time Aggregated Schema

-- Real-time counters (Redis-like key-value)
-- Key pattern: {property_id}:{metric}:{dimension}:{window}

-- Active users count
SET ga:12345:active_users:global:30m 45678 EX 1800

-- Pageviews by page (sorted set for top-N)
ZADD ga:12345:pageviews:page:30m 150 "/home"
ZADD ga:12345:pageviews:page:30m 89 "/products"
ZADD ga:12345:pageviews:page:30m 45 "/about"

-- Events by country (hash)
HSET ga:12345:events:country:30m "US" 12345
HSET ga:12345:events:country:30m "UK" 3456
HSET ga:12345:events:country:30m "DE" 2345

6.3 Historical Aggregated Schema

-- Daily aggregated metrics
CREATE TABLE daily_metrics (
    property_id         STRING,
    event_date          DATE,
    dimension_country   STRING,
    dimension_source    STRING,
    dimension_medium    STRING,
    dimension_page_path STRING,
    dimension_device    STRING,
    metric_active_users BIGINT,
    metric_new_users  BIGINT,
    metric_sessions     BIGINT,
    metric_bounces      BIGINT,
    metric_pageviews    BIGINT,
    metric_screens      BIGINT,
    metric_events         BIGINT,
    metric_engaged_sessions BIGINT,
    metric_avg_session_duration_ms BIGINT,
    metric_total_revenue  DECIMAL(18, 2),
    metric_transactions BIGINT,
    PRIMARY KEY (property_id, event_date, dimension_country,
                  dimension_source, dimension_page_path)
)
PARTITIONED BY (property_id, event_date);

-- Session-level data
CREATE TABLE sessions (
    session_id          STRING PRIMARY KEY,
    property_id         STRING,
    client_id           STRING,
    user_id             STRING,
    session_start       TIMESTAMP,
    session_end         TIMESTAMP,
    session_duration_ms BIGINT,
    session_number      INT,
    traffic_source      STRUCT<source: STRING, medium: STRING, campaign: STRING>,
    landing_page    STRING,
    exit_page           STRING,
    pageviews           INT,
    bounces             BOOLEAN,
    revenue             DECIMAL(18, 2),
    country             STRING,
    device_category     STRING,
    browser             STRING,
    os                  STRING
)
PARTITIONED BY (property_id, session_start);

-- Cohort retention
CREATE TABLE cohort_retention (
    property_id         STRING,
    cohort_date         DATE,           -- First session date
    cohort_type         STRING,            -- 'first_date', 'custom'
    retention_day       INT,              -- Days since cohort date
    cohort_size           BIGINT,          -- Users in cohort
    retained_users        BIGINT,            -- Users who returned
    retention_rate        DECIMAL(6, 4),     -- retained / cohort_size
    PRIMARY KEY (property_id, cohort_date, cohort_type, retention_day)
);

-- Funnel analysis
CREATE TABLE funnel_steps (
    property_id         STRING,
    date                DATE,
    funnel_id           STRING,
    step_number         INT,
    step_name           STRING,
    users_entered     BIGINT,
    users_exited        BIGINT,
    conversion_rate   DECIMAL(6, 4),
    PRIMARY KEY (property_id, date, funnel_id, step_number)
);

6.4 Property Metadata Schema

CREATE TABLE properties (
    property_id         STRING PRIMARY KEY,
    account_id            STRING,
    property_name       STRING,
    industry_category   STRING,
    timezone              STRING,
    currency              STRING,
    data_retention_days INT DEFAULT 380,
    created_at            TIMESTAMP,
    updated_at            TIMESTAMP,
    data_stream_ids   ARRAY<STRING>,
    custom_dimensions   ARRAY<STRUCT<
        parameter_name: STRING,
        display_name: STRING,
        scope: STRING
    >>,
    custom_metrics      ARRAY<STRUCT<
        parameter_name: STRING,
        display_name: STRING
    >>
);

CREATE TABLE data_streams (
    stream_id             STRING PRIMARY KEY,
    property_id         STRING REFERENCES properties(property_id),
    platform            STRING,             -- WEB, IOS, ANDROID
    stream_name         STRING,
    measurement_id      STRING,         -- G-XXXXXXXX
    default_uri         STRING,
    app_id              STRING,             -- For mobile apps
    consent_mode        BOOLEAN DEFAULT FALSE,
    enhanced_measurement BOOLEAN DEFAULT TRUE
);

7. Key Design Decisions

7.1 Lambda vs Kappa Architecture

Decision: Lambda Architecture (separate batch and speed layers)

Aspect Lambda Kappa Our Choice
Architecture Batch + Speed layers Stream-only Lambda
Real-time Fast (speed layer) Fast (stream) Lambda
Historical Accurate (batch) Replay from log Lambda
Complexity Higher (two pipelines) Lower (one pipeline) Lambda
Re-processing Re-run batch job Replay Kafka topic Lambda
Data retention Raw data archived Kafka retention needed Lambda

Rationale: Lambda provides the best balance for analytics:

  • Batch layer ensures accuracy for historical reports
  • Speed layer provides real-time dashboards
  • Raw data archived for ad-hoc analysis and compliance

7.2 Event Storage: Row vs Column

Decision: Columnar storage (Parquet) for raw events

Format Read Performance Write Performance Storage Use Case
Row (Avro) Full row scan Fast append Higher Event ingestion
Column (Parquet) Column scan only Slower write 75% less Analytics queries
JSON Flexible Fast Highest API responses

Rationale: Analytics queries typically access 5-10 columns out of 50+. Columnar storage provides 5-10× read performance improvement and 75% storage reduction.

7.3 Real-Time Consistency: Strong vs Eventual

Decision: Eventual consistency for real-time, strong for batch

Layer Consistency Rationale
Real-time dashboard Eventual (seconds) Acceptable for monitoring, enables high throughput
Historical reports Strong Reports must be accurate and reproducible
User counts Approximate (HyperLogLog) Distinct counting at scale requires probabilistic data structures

HyperLogLog for Active Users:

  • Memory: 1.5 KB per HLL instance (vs. billions of bytes for exact set)
  • Error rate: ~0.81% standard error (configurable precision)
  • Mergeable: Can combine HLLs from different partitions

7.4 Client-Side vs Server-Side Tracking

Decision: Support both, default to client-side

Approach Pros Cons
Client-side Easy to deploy, captures user behavior Ad blockers, privacy concerns, unreliable
Server-side Reliable, no ad blockers, privacy-friendly More complex, requires server integration
Hybrid Best of both Most complex

Recommendation: Client-side for standard use, server-side for privacy-sensitive or high-reliability needs.

7.5 Data Retention Strategy

Decision: Tiered retention with automatic lifecycle management

Tier Retention Storage Access
Hot (real-time) 30 minutes In-memory (Redis) Sub-second
Warm (raw) 7-30 days Object storage (Parquet) Seconds
Cool (aggregated) 14 months Columnar DB Sub-second
Archive (rollups) Indefinite Object storage Minutes

8. Scalability Strategy

8.1 Multi-Region Deployment

graph TB
    subgraph "EU Region"
        EU_COLLECT[Collection Endpoints]
        EU_KAFKA[Kafka Cluster]
        EU_FLINK[Flink Cluster]
        EU_SPARK[Spark Cluster]
        EU_STORE[Regional Data Store]
    end

    subgraph "US Region"
        US_COLLECT[Collection Endpoints]
        US_KAFKA[Kafka Cluster]
        US_FLINK[Flink Cluster]
        US_SPARK[Spark Cluster]
        US_STORE[Regional Data Store]
    end

    subgraph "APAC Region"
        AP_COLLECT[Collection Endpoints]
        AP_KAFKA[Kafka Cluster]
        AP_FLINK[Flink Cluster]
        AP_SPARK[Spark Cluster]
        AP_STORE[Regional Data Store]
    end

    subgraph "Global"
        GLOBAL_QUERY[Global Query Service]
        GLOBAL_CACHE[Global Cache]
    end

    EU_STORE --> GLOBAL_QUERY
    US_STORE --> GLOBAL_QUERY
    AP_STORE --> GLOBAL_QUERY
    GLOBAL_QUERY --> GLOBAL_CACHE

Sharding Strategy:

  • By property_id: Each property’s data routed to a consistent region
  • Consistent hashing: Properties distributed evenly across regions
  • Cross-region replication: Aggregated data replicated for redundancy

8.2 Auto-Scaling Triggers

Component Scale-Up Trigger Scale-Down Trigger Min Max
Collection endpoints CPU > 70% for 2 min CPU < 30% for 10 min 50 500
Kafka brokers Disk > 70% Disk < 40% 20 200
Flink tasks Backpressure detected Backpressure cleared 10 100
Spark executors Queue depth > 1000 Queue depth < 100 5 500
Query servers Active queries > 80% capacity Active queries < 20% 20 200

8.3 Data Partitioning Strategy

Raw Events:

  • Partition by property_id (hash) → event_date (range)
  • Each partition: ~100 GB per day for large properties
  • Small properties co-located to avoid small-file problem

Aggregated Data:

  • Partition by property_id → event_date
  • Pre-aggregated at day granularity for fast queries
  • Materialized views for top 100 most-queried dimension combinations

9. Curveball Scenarios

9.1 Curveball 1: A Viral Marketing Campaign Generates 100× Normal Traffic

Scenario: A major brand’s marketing campaign goes viral. Their property receives 50M events/sec (normally 500K/sec) for 2 hours.

Impact Analysis:

  • 100× normal ingestion rate for one property
  • Potential Kafka partition hot-spotting
  • Real-time dashboard may show stale data if pipeline backs up
  • Batch processing delayed for all properties

Response Strategy:

Layer Action Timeline
Collection Rate limiting per property (configurable ceiling). Excess events sampled (1% sampling) rather than dropped. Immediate
Kafka Dynamic partition rebalancing. Hot partitions split automatically. 1-5 min
Stream Processing Flink auto-scales parallelism. Backpressure triggers sampling at processing layer. 1-3 min
Real-Time Dashboard shows “data may be delayed” indicator. Switches to sampled data view. Immediate
Batch Spark auto-scales executors. Priority queue ensures other properties not starved. 5-10 min
Notification Property owner notified of traffic spike and sampling activation. 5 min

Prevention:

  • Gradual sampling: 100% → 50% → 10% → 1% as load increases
  • Per-property rate limits with burst allowance
  • Traffic spike prediction based on historical patterns

9.2 Curveball 2: GDPR Enforcement Requires Immediate Data Deletion

Scenario: A regulatory authority orders deletion of all data for users in a specific country collected over the past 2 years. The property owner has 48 hours to comply.

Impact Analysis:

  • Raw events: Need to scan and delete events matching geo criteria
  • Aggregated data: Must be recomputed after deletion
  • Real-time data: Minimal impact (only last 30 min)
  • Estimated data volume: Petabytes of raw events to scan

Response Strategy:

Phase Action Timeline
Scope Identify affected properties, date ranges, and geo criteria. 1-2 hrs
Raw Deletion Mark events for deletion (lazy deletion via bloom filter). Physical deletion during next compaction cycle. 2-6 hrs
Aggregation Trigger full re-aggregation for affected date ranges. Use incremental approach: recompute daily aggregates, then rollups. 6-24 hrs
Verification Generate deletion certificate: count of deleted events, date ranges covered. 24-36 hrs
Reporting Submit compliance report to regulatory authority. 48 hrs

Technical Approach:

graph TD
    SCAN[Scan Raw Events] --> MATCH{Matches Criteria?}
    MATCH -->|Yes| MARK[Mark for Deletion]
    MATCH -->|No| SKIP[Skip]
    MARK --> COMPACT[Compaction Cycle]
    COMPACT --> PHYSICAL[Physical Deletion]
    PHYSICAL --> REAGG[Re-aggregate Affected Windows]
    REAGG --> VERIFY[Verify Deletion]
    SKIP --> VERIFY

Optimization:

  • Use partition pruning: only scan partitions matching date range
  • Bloom filters for fast geo matching
  • Parallel scanning across thousands of workers

9.3 Curveball 3: A Bug Causes Double-Counting of Events

Scenario: A mobile SDK update sends duplicate events due to a retry bug. A major app with 10M daily active users sends 2× events for 6 hours.

Impact Analysis:

  • 10M extra events for one property
  • All metrics inflated by ~10% for that property
  • Real-time dashboard shows spike
  • Historical reports will be inaccurate if not corrected

Response Strategy:

Phase Action Timeline
Detection Anomaly detection flags 100% increase in event volume for the property. SDK version correlation identified. 5-15 min
Mitigation Enable event deduplication at ingestion: events with same event_id dropped. Notify SDK team. 15-30 min
Correction After SDK fix: re-process affected events with deduplication. Re-aggregate affected time windows. 1-4 hrs
Communication Notify property owner of data quality issue and correction timeline. 1 hr
Prevention Add event_id deduplication as default behavior. SDK version anomaly monitoring. Post-incident

Deduplication Design:

  • Event ID generated by client SDK (UUID)
  • Deduplication window: 24 hours (events with same ID within 24h = duplicate)
  • Implementation: Bloom filter + exact check in Redis
  • Memory: ~100 MB per million events tracked

9.4 Curveball 4: Ad Blockers Block 40% of Analytics Traffic

Scenario: A new generation of ad blockers starts blocking the analytics collection endpoint. Global collection drops 40% overnight.

Impact Analysis:

  • 40% data loss across all web properties
  • Real-time dashboards show dramatic drop
  • Historical trends become unreliable
  • Customer trust at risk

Response Strategy:

Phase Action Timeline
Detection Global anomaly detection flags 40% collection drop. Correlation with known ad blocker lists. 15-30 min
Assessment Identify which collection methods are blocked (JS snippet, image pixel, beacon). 1-2 hrs
Mitigation Push SDK update with alternative collection methods: server-side tracking, CMP integration, first-party domain hosting. 2-24 hrs
Communication Publish guidance for property owners: switch to server-side tracking, use first-party domains. 4 hrs
Long-term Accelerate server-side tracking adoption. Build ad-blocker-resistant collection methods. 1-3 months

Alternative Collection Methods:

  • Server-side tracking: Events sent from user’s server to analytics (not blocked)
  • First-party domain: Host analytics on customer’s domain (not blocked as third-party)
  • CMP integration: Respect consent management platform signals
  • Linker parameters: Pass client ID via URL parameters between domains

9.5 Curveball 5: A Customer’s Property is Used for DDoS Against the Analytics Platform

Scenario: An attacker creates thousands of fake properties and floods the collection endpoint with 100M events/sec of garbage data.

Impact Analysis:

  • Collection endpoints overwhelmed
  • Kafka brokers fill up
  • Legitimate traffic affected
  • Storage costs spike

Response Strategy:

Phase Action Timeline
Detection Rate anomaly detection per IP, per property, and globally. Pattern analysis identifies bot behavior. 1-5 min
Containment Automatic rate limiting per IP. New properties from suspicious IPs blocked. 5-10 min
Filtering Event validation pipeline drops events with invalid patterns (no valid User-Agent, impossible geo). 10-15 min
Cleanup Identify and suspend abusive properties. Recover resources. 1-4 hrs
Prevention CAPTCHA for new property creation. Behavioral analysis for bot detection. Post-incident

Defense Layers:

  1. IP-level rate limiting: 10K events/sec per IP
  2. Property-level rate limiting: 100K events/sec per property (new properties: 1K/sec)
  3. Event validation: Drop events with invalid structure
  4. Behavioral analysis: ML model detects bot patterns
  5. Account verification: Phone/email verification for new accounts

10. Failure Modes and Mitigations

10.1 Failure Mode Matrix

Failure Mode Probability Impact Detection Mitigation RTO Data Loss
Collection endpoint down Medium High (events lost) Health checks, error rate monitoring Multi-region failover, client retry 30 sec Client-side buffer
Kafka broker failure Low High (pipeline stalled) Broker health monitoring Auto-replace, ISR management 1 min 0 (replicated)
Flink job failure Low Medium (real-time delayed) Job manager monitoring Auto-restart, checkpoint recovery 2 min 0 (checkpointed)
Spark job failure Low Low (batch delayed) Job monitoring Auto-retry with exponential backoff 5 min 0 (idempotent)
Real-time store outage Low Medium (dashboard down) Read/write latency monitoring Multi-AZ replication, failover 30 sec 0
Historical DB corruption Very Low High (reports wrong) Checksum validation Backup restoration, re-aggregation 1 hr 0 (raw data exists)
CDN failure Low High (collection down) CDN health monitoring Failover to origin servers 1 min Client-side buffer
GeoIP database stale Medium Low (wrong geo) Data freshness monitoring Daily updates, fallback to previous N/A N/A
Query engine overload Medium Medium (slow reports) Query latency monitoring Query timeout, result caching 5 sec N/A
Data breach Very Low Critical Anomaly detection, audit logs Incident response, notification 1 hr N/A

10.2 Collection Reliability

graph TB
    subgraph "Client-Side Resilience"
        BUFFER[Event Buffer]
        RETRY[Retry Logic]
        BEACON[Beacon API]
    end

    subgraph "Network Resilience"
        CDN1[CDN Primary]
        CDN2[CDN Secondary]
        ORIGIN[Origin Servers]
    end

    subgraph "Backend Resilience"
        KAFKA1[Kafka Primary]
        KAFKA2[Kafka Secondary]
    end

    BUFFER --> RETRY
    RETRY --> BEACON
    BEACON --> CDN1
    CDN1 -->|Failover| CDN2
    CDN2 --> ORIGIN
    ORIGIN --> KAFKA1
    KAFKA1 -->|Failover| KAFKA2

Client-Side Reliability:

  • Buffer: Store up to 50 events in localStorage/IndexedDB
  • Retry: Exponential backoff (1s, 2s, 4s, 8s, 16s), max 5 retries
  • Beacon API: Guaranteed delivery on page unload
  • Offline support: Mobile SDKs persist events, upload on reconnect

Backend Reliability:

  • Kafka replication: 3x replication, min ISR = 2
  • Multi-region collection: Events can be routed to any regional endpoint
  • Dead letter queue: Invalid events sent to DLQ for analysis

10.3 Data Quality Assurance

Automated Quality Checks:

Check Frequency Action on Failure
Event volume anomaly Every 5 min Alert, enable sampling
Schema validation Per event Drop + count
Timestamp validation Per event Drop if > 24h late
Aggregation consistency Daily Re-aggregate if mismatch
Data completeness Daily Alert if < 99% of expected events
GeoIP accuracy Weekly Sample audit, update database

Data Lineage:

Raw Event → Validated → Enriched → Real-Time Aggregation → Real-Time Store
                                        ↓
                                 Batch Aggregation → Historical Store
                                        ↓
                                 Raw Archive (Object Storage)

Each transformation step records metadata for audit and debugging.


Appendix: Glossary

Term Definition
CDP Customer Data Platform
CMP Consent Management Platform
CID Client ID - Browser-generated unique identifier
HLL HyperLogLog - Probabilistic distinct counting algorithm
ISR In-Sync Replicas - Kafka replication mechanism
Parquet Columnar storage format optimized for analytics
Flink Stream processing framework
Spark Distributed data processing engine
Beacon API Browser API for reliable data sending on page unload
Lambda Architecture Batch + Speed layer architecture pattern
Kappa Architecture Stream-only architecture pattern