Design an Analytics System like Google Analytics
Table of Contents
- Requirements Clarification
- Back-of-Envelope Calculations
- System Interface Design
- High-Level Architecture
- Core Component Design
- Data Model
- Key Design Decisions
- Scalability Strategy
- Curveball Scenarios
- Failure Modes and Mitigations
1. Requirements Clarification
1.1 Functional Requirements
Event Collection
- Collect pageview, event, transaction, and custom events from web/mobile apps
- Support client-side JavaScript snippet and mobile SDKs
- Handle offline event buffering and batch upload
- Support custom dimensions and metrics (up to 25 custom dimensions, 25 custom metrics per property)
- Event parameters: up to 25 per event, key-value pairs
Real-Time Reporting
- Real-time dashboard showing active users, top pages, traffic sources (last 30 minutes)
- Updates within 1-2 seconds of event ingestion
- Real-time user geographic distribution
Historical Reporting
- Pre-built reports: Audience, Acquisition, Behavior, Conversions
- Custom report builder with drag-and-drop dimensions/metrics
- Date range comparison (e.g., this week vs. last week)
- Cohort analysis, funnel visualization, path analysis
- Data export to BigQuery-like data warehouse
User Identification
- Anonymous user tracking via client-generated IDs
- User ID linking for cross-device tracking (opt-in)
- Session management (30-minute inactivity timeout)
Privacy & Compliance
- GDPR compliance: data deletion, consent management
- CCPA compliance: opt-out mechanisms
- IP address anonymization
- Data retention controls (14 months default, configurable)
- Data processing agreements
Integrations
- Google Ads, Search Console integration
- E-commerce platforms (Shopify, WooCommerce)
- Data export APIs
- Webhooks for event streaming
1.2 Non-Functional Requirements
| Requirement | Target | Rationale |
|---|---|---|
| Event Ingestion Throughput | 100M+ events/sec peak | Global web traffic scale |
| Event Collection Latency | < 2 seconds (p99) | Near-real-time user experience |
| Real-Time Report Freshness | < 5 seconds | Dashboard responsiveness |
| Historical Report Latency | < 30 seconds for standard reports | Analyst productivity |
| Data Durability | 99.99% | Analytics data must not be lost |
| Availability | 99.9% (collection), 99.5% (reporting) | Collection is critical path |
| Privacy | GDPR/CCPA compliant | Legal requirement |
| Scalability | Support 10M+ websites | Market coverage |
| Cost Efficiency | Sub-penny per 1,000 events | Free tier viability |
2. Back-of-Envelope Calculations
2.1 Assumption Sets
| Parameter | Conservative | Realistic | Aggressive |
|---|---|---|---|
| Tracked websites | 1M | 10M | 50M |
| Daily pageviews/website | 10K | 50K | 100K |
| Events per pageview | 3 | 5 | 8 |
| Total daily events | 30B | 500B | 400B |
| Events/sec average | 350K | 5.8M | 4.6M |
| Peak events/sec (3×) | 1M | 17.4M | 13.9M |
| Avg event payload | 1 KB | 1.5 KB | 2 KB |
| Daily data volume | 30 TB | 750 TB | 800 TB |
| Monthly data volume | 900 TB | 22.5 PB | 24 PB |
2.2 Storage Calculations
Raw Event Storage (before aggregation):
| Retention | Conservative | Realistic | Aggressive |
|---|---|---|---|
| 7 days (real-time) | 210 TB | 5.25 PB | 5.6 PB |
| 30 days | 900 TB | 22.5 PB | 24 PB |
| 14 months | 270 PB | 67.5 EB | 72 EB |
Aggregated Data Storage (1000:1 compression ratio via aggregation):
| Granularity | Realistic Monthly | Notes |
|---|---|---|
| Per-second | 22.5 TB | Real-time dashboard |
| Per-minute | 375 GB | Intraday analysis |
| Per-hour | 9.4 GB | Daily trend analysis |
| Per-day | 375 MB | Long-term trends |
| Per-month | 31 MB | Year-over-year |
Key Insight: Aggregated data is ~0.01% of raw data. Store raw for 7-30 days, aggregated indefinitely.
2.3 Bandwidth Calculations
Ingestion Pipeline:
- 5.8M events/sec × 1.5 KB = 8.7 GB/s average ingestion
- Peak: 26 GB/s
- Collection endpoints need global CDN + edge processing
Internal Data Movement:
- Raw → Streaming processing: 8.7 GB/s
- Streaming → Real-time store: 2 GB/s (filtered/aggregated)
- Batch → Historical store: 8.7 GB/s (same data, different path)
2.4 Compute Estimates
Collection Endpoints:
- 17.4M peak events/sec
- Each server handles ~50K events/sec (simple write + ack)
- Need ~350 collection servers globally (distributed across edge locations)
Stream Processing:
- 5.8M events/sec to process
- Each processing instance handles ~100K events/sec
- Need ~58 processing instances per pipeline stage
Query Serving:
- 100K concurrent report queries
- Each query server handles ~100 QPS
- Need ~1,000 query servers
3. System Interface Design
3.1 Collection Endpoint (Measurement Protocol)
POST /collect?v=2&tid=GA-XXXXXXXX&cid=abc123def456&t=pageview&dp=/home&dt=Home%20Page&dr=https://example.com&dl=https://example.com/home&uid=user123
Host: collection.analytics.com
Content-Type: application/x-www-form-urlencoded
Batch Collection:
POST /batch
[
{ "v": "2", "tid": "GA-XXXXXXXX", "cid": "abc123", "t": "pageview", "dp": "/home" },
{ "v": "2", "tid": "GA-XXXXXXXX", "cid": "abc123", "t": "event", "ec": "click", "ea": "button", "el": "signup" },
{ "v": "2", "tid": "GA-XXXXXXXX", "cid": "abc123", "t": "screen_view", "sn": "Home" }
]
Response 204 No Content (always succeed, never block the client)
3.2 Real-Time API
GET /v1/properties/{propertyId}/realtime:report
?metrics=activeUsers
&dimensions=country,pageTitle
&filter=country!=Internal
&pageSize=100
Response 200:
{
"rows": [
{
"dimensionValues": [
{ "value": "United States" },
{ "value": "Home Page" }
],
"metricValues": [
{ "value": "15234" }
]
}
],
"rowCount": 1,
"totalActiveUsers": 45678
}
3.3 Historical Reporting API
POST /v1/properties/{propertyId}:runReport
{
"dateRanges": [{
"startDate": "2026-01-01",
"endDate": "2026-01-31"
}],
"dimensions": [
{ "name": "country" },
{ "name": "sessionDefaultChannelGroup" }
],
"metrics": [
{ "name": "activeUsers" },
{ "name": "screenPageViews" },
{ "name": "bounceRate" }
],
"orderBys": [
{ "metric": { "metricName": "activeUsers" }, "desc": true }
],
"limit": 100,
"segments": [{
"segmentId": "newVsReturning::new"
}]
}
Response 200:
{
"reports": [{
"dimensionHeaders": [
{ "name": "country", "columnIndex": 0 },
{ "name": "sessionDefaultChannelGroup", "columnIndex": 1 }
],
"metricHeaders": [{
"metrics": [
{ "name": "activeUsers", "type": "TYPE_INTEGER" },
{ "name": "screenPageViews", "type": "TYPE_INTEGER" },
{ "name": "bounceRate", "type": "TYPE_FLOAT" }
]
}],
"rows": [
{
"dimensionValues": [
{ "value": "United States" },
{ "value": "Organic Search" }
],
"metricValues": [
{ "value": "125000" },
{ "value": "450000" },
{ "value": "0.35" }
]
}
]
}]
}
3.4 Data Export API
POST /v1/properties/{propertyId}/eventStreams:subscribe
{
"destination": {
"type": "BIGQUERY",
"bigqueryDestination": {
"projectId": "my-project",
"datasetId": "analytics",
"tableIdTemplate": "events_{DATE}"
}
},
"filter": {
"eventFilter": {
"eventNames": ["page_view", "purchase", "add_to_cart"]
}
}
}
3.5 Management API
POST /v1/properties/{propertyId}/customDimensions
{
"displayName": "User Tier",
"parameterName": "user_tier",
"scope": "USER"
}
POST /v1/properties/{propertyId}/dataDeletions
{
"propertyId": "GA-XXXXXXXX",
"userId": "user123",
"deleteEventCondition": {
"eventDateRange": {
"startDate": "2026-01-01",
"endDate": "2026-08-28"
}
}
}
4. High-Level Architecture
graph TB
subgraph "Client Side"
WEB[Web JS Snippet]
MOBILE[Mobile SDK]
SERVER[Server SDK]
CDP[CDP / Tag Manager]
end
subgraph "Edge Collection Layer"
CDN[Global CDN / Edge Nodes]
COLLECT1[Collection Endpoint EU]
COLLECT2[Collection Endpoint US]
COLLECT3[Collection Endpoint APAC]
end
subgraph "Ingestion Layer"
KAFKA[Kafka Cluster - Raw Events]
VALIDATE[Event Validation Service]
ENRICH[Event Enrichment Service]
end
subgraph "Real-Time Pipeline"
FLINK[Stream Processing - Flink]
RT_AGG[Real-Time Aggregator]
RT_STORE[Real-Time Store - Redis/Cassandra]
end
subgraph "Batch Pipeline"
BATCH_KAFKA[Kafka - Validated Events]
SPARK[Batch Processing - Spark]
AGG_ENGINE[Aggregation Engine]
end
subgraph "Storage Layer"
RAW_STORE[Raw Event Store - Object Storage]
RT_DB[Real-Time DB]
HIST_DB[Historical Aggregated DB - Columnar]
USER_DB[User/Session DB]
META_DB[Property Metadata DB]
end
subgraph "Query & Reporting"
RT_API[Real-Time API]
HIST_API[Historical Reporting API]
QUERY_ENGINE[Query Engine - Presto/Drill]
CACHE[Query Result Cache]
end
subgraph "Management & Admin"
MGMT_API[Management API]
CONSENT[Consent Management]
PRIVACY[Privacy Service]
EXPORT[Data Export Service]
end
subgraph "Dashboard"
UI[Web Dashboard]
EXPORT_UI[Export UI]
end
WEB --> CDN
MOBILE --> CDN
SERVER --> CDN
CDP --> CDN
CDN --> COLLECT1
CDN --> COLLECT2
CDN --> COLLECT3
COLLECT1 --> KAFKA
COLLECT2 --> KAFKA
COLLECT3 --> KAFKA
KAFKA --> VALIDATE
VALIDATE --> ENRICH
ENRICH --> FLINK
ENRICH --> BATCH_KAFKA
FLINK --> RT_AGG
RT_AGG --> RT_STORE
BATCH_KAFKA --> SPARK
SPARK --> AGG_ENGINE
AGG_ENGINE --> HIST_DB
ENRICH --> RAW_STORE
RT_STORE --> RT_DB
RT_DB --> RT_API
HIST_DB --> HIST_API
HIST_DB --> QUERY_ENGINE
QUERY_ENGINE --> CACHE
CACHE --> HIST_API
META_DB --> MGMT_API
CONSENT --> VALIDATE
PRIVACY --> RAW_STORE
PRIVACY --> HIST_DB
EXPORT --> RAW_STORE
RT_API --> UI
HIST_API --> UI
MGMT_API --> EXPORT_UI
5. Core Component Design
5.1 Event Collection Layer
Design Goals: Maximum throughput, minimum latency, fire-and-forget reliability
graph LR
subgraph "Browser"
SNIPPET[Analytics Snippet]
BEACON[Beacon API / Image Pixel]
BUFFER[Event Buffer]
end
subgraph "Edge"
CDN_NODE[CDN Edge Node]
COLLECTOR[Collection Service]
end
subgraph "Backend"
KAFKA[Kafka]
end
SNIPPET --> BUFFER
BUFFER --> BEACON
BEACON --> CDN_NODE
CDN_NODE --> COLLECTOR
COLLECTOR --> KAFKA
Collection Strategy:
- Beacon API (
navigator.sendBeacon): Guaranteed delivery on page unload, async, no blocking - Image pixel fallback: For older browsers, 1x1 transparent GIF with event data in URL
- Batching: Client batches events (up to 25 per batch) to reduce HTTP overhead
- Offline buffering: Mobile SDKs store events locally, upload when connectivity restored
- CDN caching: Collection endpoints set
Cache-Control: no-storeto prevent CDN caching of event data
Collection Endpoint Design:
- Stateless, horizontally scalable
- Write events to Kafka and immediately return 204
- No validation at collection time (validate in pipeline)
- Rate limiting per property (not per user) to prevent abuse
- Geographic routing: events routed to nearest regional Kafka cluster
5.2 Event Validation and Enrichment
Validation Pipeline:
graph TD
RAW[Raw Event from Kafka] --> PARSE[Parse Event]
PARSE --> VALID{Valid Format?}
VALID -->|No| DROP[Drop + Count]
VALID -->|Yes| ENRICH[Enrich Event]
ENRICH --> GEO[Geo IP Lookup]
GEO --> DEVICE[Device Detection]
DEVICE --> CONSENT{Consent Valid?}
CONSENT -->|No| ANON[Anonymize + Store]
CONSENT -->|Yes| STORE[Store Enriched Event]
ANON --> RT[Real-Time Pipeline]
STORE --> RT
STORE --> BATCH[Batch Pipeline]
STORE --> RAW[Raw Archive]
Enrichment Data Sources:
- GeoIP database: MaxMind or internal IP-to-location mapping (updated daily)
- Device detection: User-Agent parsing → browser, OS, device type, screen resolution
- Currency conversion: For e-commerce events, convert to property’s currency
- Timezone normalization: All timestamps stored in UTC, enriched with user timezone
5.3 Real-Time Processing Pipeline
Architecture:
graph LR
subgraph "Stream Processing"
INPUT[Kafka Topic: enriched-events]
FLINK1[Flink Job: Session Builder]
FLINK2[Flink Job: Counter Aggregator]
FLINK3[Flink Job: Top-N Calculator]
end
subgraph "Real-Time Storage"
COUNTER[Counter Store - Redis]
SESSION[Session Store - Cassandra]
TOPN[Top-N Store - Redis Sorted Sets]
end
INPUT --> FLINK1
INPUT --> FLINK2
INPUT --> FLINK3
FLINK1 --> SESSION
FLINK2 --> COUNTER
FLINK3 --> TOPN
Processing Logic:
| Metric | Computation | Window | Storage |
|---|---|---|---|
| Active Users | Count distinct user IDs in last 30 min | Sliding 30-min | Redis (TTL 30 min) |
| Pageviews/sec | Rate counter | 1-sec tumbling | Redis counter |
| Top Pages | Top-N by pageviews | 30-min sliding | Redis sorted set |
| Traffic Sources | Count by source/medium | 30-min sliding | Redis hash |
| Real-Time Events | Raw event count | 1-min tumbling | Redis counter |
Session Building:
- Assign session ID based on client ID + 30-min inactivity gap
- Maintain session state in Redis with TTL
- Cross-device session merging via User ID (when provided)
5.4 Batch Processing Pipeline
Lambda Architecture:
graph TB
subgraph "Speed Layer (Real-Time)"
STREAM[Stream Processing]
RT_VIEW[Real-Time View]
end
subgraph "Batch Layer"
BATCH[Batch Processing - Spark]
BATCH_VIEW[Batch View - Aggregated Tables]
end
subgraph "Serving Layer"
SERVING[Serving Layer]
MERGE[Merge Batch + Real-Time]
end
subgraph "Presentation"
DASHBOARD[Dashboard / API]
end
STREAM --> RT_VIEW
BATCH --> BATCH_VIEW
RT_VIEW --> MERGE
BATCH_VIEW --> MERGE
MERGE --> SERVING
SERVING --> DASHBOARD
Batch Processing Stages:
| Stage | Input | Output | Schedule |
|---|---|---|---|
| Raw → Cleaned | Raw events | Validated, enriched events | Every 5 min |
| Session Attribution | Cleaned events | Session-level data | Every 15 min |
| Daily Aggregation | Session data | Daily aggregated metrics | Daily (next day 02:00 UTC) |
| Monthly Rollup | Daily aggregates | Monthly summaries | Monthly |
| Cohort Analysis | Session data | Cohort retention tables | Weekly |
| Funnel Analysis | Event sequences | Funnel conversion rates | Daily |
Aggregation Engine Design:
- Pre-aggregate at multiple granularities (second, minute, hour, day, month)
- Use columnar storage (Parquet/ORC) for efficient aggregation queries
- Materialized views for common report patterns
- Incremental aggregation: only recompute affected time windows
5.5 Query Engine
Query Optimization:
graph TD
QUERY[User Query] --> PARSE[Parse Query]
PARSE --> PLAN[Generate Query Plan]
PLAN --> ROUTE{Route Decision}
ROUTE -->|Real-Time| RT[Real-Time Store]
ROUTE -->|Cached| CACHE[Query Cache]
ROUTE -->|Aggregated| AGG[Aggregated Tables]
ROUTE -->|Ad-Hoc| RAW[Raw Event Store]
RT --> FORMAT[Format Results]
CACHE --> FORMAT
AGG --> FORMAT
RAW --> FORMAT
FORMAT --> RESULT[Return Results]
RESULT --> CACHE2{Cache Result?}
CACHE2 -->|Yes| STORE_CACHE[Store in Cache]
CACHE2 -->|No| DONE[Done]
Query Routing Strategy:
| Query Type | Data Source | Expected Latency | Cache TTL |
|---|---|---|---|
| Real-time dashboard | Real-time store | < 500ms | 5 sec |
| Standard report (pre-built) | Aggregated tables | < 2 sec | 5 min |
| Custom report (common) | Aggregated tables | < 5 sec | 1 hour |
| Custom report (rare) | Raw store (Presto) | 10-30 sec | 24 hours |
| Export to BigQuery | Raw event store | Async | N/A |
5.6 Privacy and Compliance
GDPR Data Deletion Flow:
sequenceDiagram
participant User
participant UI as Dashboard
participant API as Privacy API
participant RAW as Raw Store
participant AGG as Aggregated Store
participant RT as Real-Time Store
User->>UI: Request data deletion
UI->>API: Delete events for user_id=X
API->>RT: Remove real-time data
RT-->>API: Done
API->>RAW: Mark events for deletion
RAW-->>API: Done (lazy deletion)
API->>AGG: Trigger re-aggregation
AGG-->>API: Scheduled
API-->>UI: Deletion initiated
Note over AGG: Re-aggregation runs<br/>during next batch cycle
Privacy Features:
- IP anonymization: Last octet(s) stripped before storage
- Consent mode: Events collected but not processed if consent denied
- Data retention: Configurable per property (14 months default)
- User-level deletion: Delete all events for a specific user ID
- Property-level deletion: Delete all data for a property
- Audit trail: All deletion requests logged for compliance
6. Data Model
6.1 Raw Event Schema
-- Raw events stored as Parquet files in object storage
-- Partitioned by: property_id, event_date
CREATE EXTERNAL TABLE raw_events (
event_id STRING, -- UUID, generated at collection
property_id STRING, -- GA-XXXXXXXX
client_id STRING, -- Browser-generated unique ID
user_id STRING, -- Optional, developer-provided
event_timestamp BIGINT, -- Event time in microseconds
ingestion_timestamp BIGINT, -- Server receive time
event_name STRING, -- page_view, click, purchase, etc.
event_params MAP<STRING, STRUCT<
string_value: STRING,
int_value: BIGINT,
float_value: DOUBLE,
double_value: DOUBLE
>>,
session_id STRING,
user_properties MAP<STRING, STRUCT<
value: STRING,
set_timestamp: BIGINT
>>,
geo STRUCT<
country: STRING,
region: STRING,
city: STRING,
sub_continent: STRING,
continent: STRING
>,
device STRUCT<
category: STRING, -- mobile, desktop, tablet
operating_system: STRING,
browser: STRING,
browser_version: STRING,
mobile_marketing_name: STRING,
mobile_model_name: STRING,
mobile_brand_name: STRING,
is_mobile: BOOLEAN,
screen_resolution: STRING
>,
traffic_source STRUCT<
source: STRING,
medium: STRING,
campaign: STRING,
term: STRING,
content: STRING
>,
page STRUCT<
display_name: STRING,
host: STRING,
path: STRING,
referrer: STRING
>,
consent STRUCT<
ad_personalization: BOOLEAN,
ad_storage: BOOLEAN,
analytics_storage: BOOLEAN
>
)
PARTITIONED BY (property_id STRING, event_date STRING)
STORED AS PARQUET;
6.2 Real-Time Aggregated Schema
-- Real-time counters (Redis-like key-value)
-- Key pattern: {property_id}:{metric}:{dimension}:{window}
-- Active users count
SET ga:12345:active_users:global:30m 45678 EX 1800
-- Pageviews by page (sorted set for top-N)
ZADD ga:12345:pageviews:page:30m 150 "/home"
ZADD ga:12345:pageviews:page:30m 89 "/products"
ZADD ga:12345:pageviews:page:30m 45 "/about"
-- Events by country (hash)
HSET ga:12345:events:country:30m "US" 12345
HSET ga:12345:events:country:30m "UK" 3456
HSET ga:12345:events:country:30m "DE" 2345
6.3 Historical Aggregated Schema
-- Daily aggregated metrics
CREATE TABLE daily_metrics (
property_id STRING,
event_date DATE,
dimension_country STRING,
dimension_source STRING,
dimension_medium STRING,
dimension_page_path STRING,
dimension_device STRING,
metric_active_users BIGINT,
metric_new_users BIGINT,
metric_sessions BIGINT,
metric_bounces BIGINT,
metric_pageviews BIGINT,
metric_screens BIGINT,
metric_events BIGINT,
metric_engaged_sessions BIGINT,
metric_avg_session_duration_ms BIGINT,
metric_total_revenue DECIMAL(18, 2),
metric_transactions BIGINT,
PRIMARY KEY (property_id, event_date, dimension_country,
dimension_source, dimension_page_path)
)
PARTITIONED BY (property_id, event_date);
-- Session-level data
CREATE TABLE sessions (
session_id STRING PRIMARY KEY,
property_id STRING,
client_id STRING,
user_id STRING,
session_start TIMESTAMP,
session_end TIMESTAMP,
session_duration_ms BIGINT,
session_number INT,
traffic_source STRUCT<source: STRING, medium: STRING, campaign: STRING>,
landing_page STRING,
exit_page STRING,
pageviews INT,
bounces BOOLEAN,
revenue DECIMAL(18, 2),
country STRING,
device_category STRING,
browser STRING,
os STRING
)
PARTITIONED BY (property_id, session_start);
-- Cohort retention
CREATE TABLE cohort_retention (
property_id STRING,
cohort_date DATE, -- First session date
cohort_type STRING, -- 'first_date', 'custom'
retention_day INT, -- Days since cohort date
cohort_size BIGINT, -- Users in cohort
retained_users BIGINT, -- Users who returned
retention_rate DECIMAL(6, 4), -- retained / cohort_size
PRIMARY KEY (property_id, cohort_date, cohort_type, retention_day)
);
-- Funnel analysis
CREATE TABLE funnel_steps (
property_id STRING,
date DATE,
funnel_id STRING,
step_number INT,
step_name STRING,
users_entered BIGINT,
users_exited BIGINT,
conversion_rate DECIMAL(6, 4),
PRIMARY KEY (property_id, date, funnel_id, step_number)
);
6.4 Property Metadata Schema
CREATE TABLE properties (
property_id STRING PRIMARY KEY,
account_id STRING,
property_name STRING,
industry_category STRING,
timezone STRING,
currency STRING,
data_retention_days INT DEFAULT 380,
created_at TIMESTAMP,
updated_at TIMESTAMP,
data_stream_ids ARRAY<STRING>,
custom_dimensions ARRAY<STRUCT<
parameter_name: STRING,
display_name: STRING,
scope: STRING
>>,
custom_metrics ARRAY<STRUCT<
parameter_name: STRING,
display_name: STRING
>>
);
CREATE TABLE data_streams (
stream_id STRING PRIMARY KEY,
property_id STRING REFERENCES properties(property_id),
platform STRING, -- WEB, IOS, ANDROID
stream_name STRING,
measurement_id STRING, -- G-XXXXXXXX
default_uri STRING,
app_id STRING, -- For mobile apps
consent_mode BOOLEAN DEFAULT FALSE,
enhanced_measurement BOOLEAN DEFAULT TRUE
);
7. Key Design Decisions
7.1 Lambda vs Kappa Architecture
Decision: Lambda Architecture (separate batch and speed layers)
| Aspect | Lambda | Kappa | Our Choice |
|---|---|---|---|
| Architecture | Batch + Speed layers | Stream-only | Lambda |
| Real-time | Fast (speed layer) | Fast (stream) | Lambda |
| Historical | Accurate (batch) | Replay from log | Lambda |
| Complexity | Higher (two pipelines) | Lower (one pipeline) | Lambda |
| Re-processing | Re-run batch job | Replay Kafka topic | Lambda |
| Data retention | Raw data archived | Kafka retention needed | Lambda |
Rationale: Lambda provides the best balance for analytics:
- Batch layer ensures accuracy for historical reports
- Speed layer provides real-time dashboards
- Raw data archived for ad-hoc analysis and compliance
7.2 Event Storage: Row vs Column
Decision: Columnar storage (Parquet) for raw events
| Format | Read Performance | Write Performance | Storage | Use Case |
|---|---|---|---|---|
| Row (Avro) | Full row scan | Fast append | Higher | Event ingestion |
| Column (Parquet) | Column scan only | Slower write | 75% less | Analytics queries |
| JSON | Flexible | Fast | Highest | API responses |
Rationale: Analytics queries typically access 5-10 columns out of 50+. Columnar storage provides 5-10× read performance improvement and 75% storage reduction.
7.3 Real-Time Consistency: Strong vs Eventual
Decision: Eventual consistency for real-time, strong for batch
| Layer | Consistency | Rationale |
|---|---|---|
| Real-time dashboard | Eventual (seconds) | Acceptable for monitoring, enables high throughput |
| Historical reports | Strong | Reports must be accurate and reproducible |
| User counts | Approximate (HyperLogLog) | Distinct counting at scale requires probabilistic data structures |
HyperLogLog for Active Users:
- Memory: 1.5 KB per HLL instance (vs. billions of bytes for exact set)
- Error rate: ~0.81% standard error (configurable precision)
- Mergeable: Can combine HLLs from different partitions
7.4 Client-Side vs Server-Side Tracking
Decision: Support both, default to client-side
| Approach | Pros | Cons |
|---|---|---|
| Client-side | Easy to deploy, captures user behavior | Ad blockers, privacy concerns, unreliable |
| Server-side | Reliable, no ad blockers, privacy-friendly | More complex, requires server integration |
| Hybrid | Best of both | Most complex |
Recommendation: Client-side for standard use, server-side for privacy-sensitive or high-reliability needs.
7.5 Data Retention Strategy
Decision: Tiered retention with automatic lifecycle management
| Tier | Retention | Storage | Access |
|---|---|---|---|
| Hot (real-time) | 30 minutes | In-memory (Redis) | Sub-second |
| Warm (raw) | 7-30 days | Object storage (Parquet) | Seconds |
| Cool (aggregated) | 14 months | Columnar DB | Sub-second |
| Archive (rollups) | Indefinite | Object storage | Minutes |
8. Scalability Strategy
8.1 Multi-Region Deployment
graph TB
subgraph "EU Region"
EU_COLLECT[Collection Endpoints]
EU_KAFKA[Kafka Cluster]
EU_FLINK[Flink Cluster]
EU_SPARK[Spark Cluster]
EU_STORE[Regional Data Store]
end
subgraph "US Region"
US_COLLECT[Collection Endpoints]
US_KAFKA[Kafka Cluster]
US_FLINK[Flink Cluster]
US_SPARK[Spark Cluster]
US_STORE[Regional Data Store]
end
subgraph "APAC Region"
AP_COLLECT[Collection Endpoints]
AP_KAFKA[Kafka Cluster]
AP_FLINK[Flink Cluster]
AP_SPARK[Spark Cluster]
AP_STORE[Regional Data Store]
end
subgraph "Global"
GLOBAL_QUERY[Global Query Service]
GLOBAL_CACHE[Global Cache]
end
EU_STORE --> GLOBAL_QUERY
US_STORE --> GLOBAL_QUERY
AP_STORE --> GLOBAL_QUERY
GLOBAL_QUERY --> GLOBAL_CACHE
Sharding Strategy:
- By property_id: Each property’s data routed to a consistent region
- Consistent hashing: Properties distributed evenly across regions
- Cross-region replication: Aggregated data replicated for redundancy
8.2 Auto-Scaling Triggers
| Component | Scale-Up Trigger | Scale-Down Trigger | Min | Max |
|---|---|---|---|---|
| Collection endpoints | CPU > 70% for 2 min | CPU < 30% for 10 min | 50 | 500 |
| Kafka brokers | Disk > 70% | Disk < 40% | 20 | 200 |
| Flink tasks | Backpressure detected | Backpressure cleared | 10 | 100 |
| Spark executors | Queue depth > 1000 | Queue depth < 100 | 5 | 500 |
| Query servers | Active queries > 80% capacity | Active queries < 20% | 20 | 200 |
8.3 Data Partitioning Strategy
Raw Events:
- Partition by
property_id(hash) →event_date(range) - Each partition: ~100 GB per day for large properties
- Small properties co-located to avoid small-file problem
Aggregated Data:
- Partition by
property_id→event_date - Pre-aggregated at day granularity for fast queries
- Materialized views for top 100 most-queried dimension combinations
9. Curveball Scenarios
9.1 Curveball 1: A Viral Marketing Campaign Generates 100× Normal Traffic
Scenario: A major brand’s marketing campaign goes viral. Their property receives 50M events/sec (normally 500K/sec) for 2 hours.
Impact Analysis:
- 100× normal ingestion rate for one property
- Potential Kafka partition hot-spotting
- Real-time dashboard may show stale data if pipeline backs up
- Batch processing delayed for all properties
Response Strategy:
| Layer | Action | Timeline |
|---|---|---|
| Collection | Rate limiting per property (configurable ceiling). Excess events sampled (1% sampling) rather than dropped. | Immediate |
| Kafka | Dynamic partition rebalancing. Hot partitions split automatically. | 1-5 min |
| Stream Processing | Flink auto-scales parallelism. Backpressure triggers sampling at processing layer. | 1-3 min |
| Real-Time | Dashboard shows “data may be delayed” indicator. Switches to sampled data view. | Immediate |
| Batch | Spark auto-scales executors. Priority queue ensures other properties not starved. | 5-10 min |
| Notification | Property owner notified of traffic spike and sampling activation. | 5 min |
Prevention:
- Gradual sampling: 100% → 50% → 10% → 1% as load increases
- Per-property rate limits with burst allowance
- Traffic spike prediction based on historical patterns
9.2 Curveball 2: GDPR Enforcement Requires Immediate Data Deletion
Scenario: A regulatory authority orders deletion of all data for users in a specific country collected over the past 2 years. The property owner has 48 hours to comply.
Impact Analysis:
- Raw events: Need to scan and delete events matching geo criteria
- Aggregated data: Must be recomputed after deletion
- Real-time data: Minimal impact (only last 30 min)
- Estimated data volume: Petabytes of raw events to scan
Response Strategy:
| Phase | Action | Timeline |
|---|---|---|
| Scope | Identify affected properties, date ranges, and geo criteria. | 1-2 hrs |
| Raw Deletion | Mark events for deletion (lazy deletion via bloom filter). Physical deletion during next compaction cycle. | 2-6 hrs |
| Aggregation | Trigger full re-aggregation for affected date ranges. Use incremental approach: recompute daily aggregates, then rollups. | 6-24 hrs |
| Verification | Generate deletion certificate: count of deleted events, date ranges covered. | 24-36 hrs |
| Reporting | Submit compliance report to regulatory authority. | 48 hrs |
Technical Approach:
graph TD
SCAN[Scan Raw Events] --> MATCH{Matches Criteria?}
MATCH -->|Yes| MARK[Mark for Deletion]
MATCH -->|No| SKIP[Skip]
MARK --> COMPACT[Compaction Cycle]
COMPACT --> PHYSICAL[Physical Deletion]
PHYSICAL --> REAGG[Re-aggregate Affected Windows]
REAGG --> VERIFY[Verify Deletion]
SKIP --> VERIFY
Optimization:
- Use partition pruning: only scan partitions matching date range
- Bloom filters for fast geo matching
- Parallel scanning across thousands of workers
9.3 Curveball 3: A Bug Causes Double-Counting of Events
Scenario: A mobile SDK update sends duplicate events due to a retry bug. A major app with 10M daily active users sends 2× events for 6 hours.
Impact Analysis:
- 10M extra events for one property
- All metrics inflated by ~10% for that property
- Real-time dashboard shows spike
- Historical reports will be inaccurate if not corrected
Response Strategy:
| Phase | Action | Timeline |
|---|---|---|
| Detection | Anomaly detection flags 100% increase in event volume for the property. SDK version correlation identified. | 5-15 min |
| Mitigation | Enable event deduplication at ingestion: events with same event_id dropped. Notify SDK team. | 15-30 min |
| Correction | After SDK fix: re-process affected events with deduplication. Re-aggregate affected time windows. | 1-4 hrs |
| Communication | Notify property owner of data quality issue and correction timeline. | 1 hr |
| Prevention | Add event_id deduplication as default behavior. SDK version anomaly monitoring. | Post-incident |
Deduplication Design:
- Event ID generated by client SDK (UUID)
- Deduplication window: 24 hours (events with same ID within 24h = duplicate)
- Implementation: Bloom filter + exact check in Redis
- Memory: ~100 MB per million events tracked
9.4 Curveball 4: Ad Blockers Block 40% of Analytics Traffic
Scenario: A new generation of ad blockers starts blocking the analytics collection endpoint. Global collection drops 40% overnight.
Impact Analysis:
- 40% data loss across all web properties
- Real-time dashboards show dramatic drop
- Historical trends become unreliable
- Customer trust at risk
Response Strategy:
| Phase | Action | Timeline |
|---|---|---|
| Detection | Global anomaly detection flags 40% collection drop. Correlation with known ad blocker lists. | 15-30 min |
| Assessment | Identify which collection methods are blocked (JS snippet, image pixel, beacon). | 1-2 hrs |
| Mitigation | Push SDK update with alternative collection methods: server-side tracking, CMP integration, first-party domain hosting. | 2-24 hrs |
| Communication | Publish guidance for property owners: switch to server-side tracking, use first-party domains. | 4 hrs |
| Long-term | Accelerate server-side tracking adoption. Build ad-blocker-resistant collection methods. | 1-3 months |
Alternative Collection Methods:
- Server-side tracking: Events sent from user’s server to analytics (not blocked)
- First-party domain: Host analytics on customer’s domain (not blocked as third-party)
- CMP integration: Respect consent management platform signals
- Linker parameters: Pass client ID via URL parameters between domains
9.5 Curveball 5: A Customer’s Property is Used for DDoS Against the Analytics Platform
Scenario: An attacker creates thousands of fake properties and floods the collection endpoint with 100M events/sec of garbage data.
Impact Analysis:
- Collection endpoints overwhelmed
- Kafka brokers fill up
- Legitimate traffic affected
- Storage costs spike
Response Strategy:
| Phase | Action | Timeline |
|---|---|---|
| Detection | Rate anomaly detection per IP, per property, and globally. Pattern analysis identifies bot behavior. | 1-5 min |
| Containment | Automatic rate limiting per IP. New properties from suspicious IPs blocked. | 5-10 min |
| Filtering | Event validation pipeline drops events with invalid patterns (no valid User-Agent, impossible geo). | 10-15 min |
| Cleanup | Identify and suspend abusive properties. Recover resources. | 1-4 hrs |
| Prevention | CAPTCHA for new property creation. Behavioral analysis for bot detection. | Post-incident |
Defense Layers:
- IP-level rate limiting: 10K events/sec per IP
- Property-level rate limiting: 100K events/sec per property (new properties: 1K/sec)
- Event validation: Drop events with invalid structure
- Behavioral analysis: ML model detects bot patterns
- Account verification: Phone/email verification for new accounts
10. Failure Modes and Mitigations
10.1 Failure Mode Matrix
| Failure Mode | Probability | Impact | Detection | Mitigation | RTO | Data Loss |
|---|---|---|---|---|---|---|
| Collection endpoint down | Medium | High (events lost) | Health checks, error rate monitoring | Multi-region failover, client retry | 30 sec | Client-side buffer |
| Kafka broker failure | Low | High (pipeline stalled) | Broker health monitoring | Auto-replace, ISR management | 1 min | 0 (replicated) |
| Flink job failure | Low | Medium (real-time delayed) | Job manager monitoring | Auto-restart, checkpoint recovery | 2 min | 0 (checkpointed) |
| Spark job failure | Low | Low (batch delayed) | Job monitoring | Auto-retry with exponential backoff | 5 min | 0 (idempotent) |
| Real-time store outage | Low | Medium (dashboard down) | Read/write latency monitoring | Multi-AZ replication, failover | 30 sec | 0 |
| Historical DB corruption | Very Low | High (reports wrong) | Checksum validation | Backup restoration, re-aggregation | 1 hr | 0 (raw data exists) |
| CDN failure | Low | High (collection down) | CDN health monitoring | Failover to origin servers | 1 min | Client-side buffer |
| GeoIP database stale | Medium | Low (wrong geo) | Data freshness monitoring | Daily updates, fallback to previous | N/A | N/A |
| Query engine overload | Medium | Medium (slow reports) | Query latency monitoring | Query timeout, result caching | 5 sec | N/A |
| Data breach | Very Low | Critical | Anomaly detection, audit logs | Incident response, notification | 1 hr | N/A |
10.2 Collection Reliability
graph TB
subgraph "Client-Side Resilience"
BUFFER[Event Buffer]
RETRY[Retry Logic]
BEACON[Beacon API]
end
subgraph "Network Resilience"
CDN1[CDN Primary]
CDN2[CDN Secondary]
ORIGIN[Origin Servers]
end
subgraph "Backend Resilience"
KAFKA1[Kafka Primary]
KAFKA2[Kafka Secondary]
end
BUFFER --> RETRY
RETRY --> BEACON
BEACON --> CDN1
CDN1 -->|Failover| CDN2
CDN2 --> ORIGIN
ORIGIN --> KAFKA1
KAFKA1 -->|Failover| KAFKA2
Client-Side Reliability:
- Buffer: Store up to 50 events in localStorage/IndexedDB
- Retry: Exponential backoff (1s, 2s, 4s, 8s, 16s), max 5 retries
- Beacon API: Guaranteed delivery on page unload
- Offline support: Mobile SDKs persist events, upload on reconnect
Backend Reliability:
- Kafka replication: 3x replication, min ISR = 2
- Multi-region collection: Events can be routed to any regional endpoint
- Dead letter queue: Invalid events sent to DLQ for analysis
10.3 Data Quality Assurance
Automated Quality Checks:
| Check | Frequency | Action on Failure |
|---|---|---|
| Event volume anomaly | Every 5 min | Alert, enable sampling |
| Schema validation | Per event | Drop + count |
| Timestamp validation | Per event | Drop if > 24h late |
| Aggregation consistency | Daily | Re-aggregate if mismatch |
| Data completeness | Daily | Alert if < 99% of expected events |
| GeoIP accuracy | Weekly | Sample audit, update database |
Data Lineage:
Raw Event → Validated → Enriched → Real-Time Aggregation → Real-Time Store
↓
Batch Aggregation → Historical Store
↓
Raw Archive (Object Storage)
Each transformation step records metadata for audit and debugging.
Appendix: Glossary
| Term | Definition |
|---|---|
| CDP | Customer Data Platform |
| CMP | Consent Management Platform |
| CID | Client ID - Browser-generated unique identifier |
| HLL | HyperLogLog - Probabilistic distinct counting algorithm |
| ISR | In-Sync Replicas - Kafka replication mechanism |
| Parquet | Columnar storage format optimized for analytics |
| Flink | Stream processing framework |
| Spark | Distributed data processing engine |
| Beacon API | Browser API for reliable data sending on page unload |
| Lambda Architecture | Batch + Speed layer architecture pattern |
| Kappa Architecture | Stream-only architecture pattern |