Design a Cloud Solution Provider like AWS
Table of Contents
- Requirements Clarification
- Back-of-Envelope Calculations
- System Interface Design
- High-Level Architecture
- Core Component Design
- Data Model
- Key Design Decisions
- Scalability Strategy
- Curveball Scenarios
- Failure Modes and Mitigations
1. Requirements Clarification
1.1 Functional Requirements
Compute (EC2-like)
- Launch, stop, start, terminate virtual machines of configurable types (CPU, memory, GPU)
- Support multiple OS images (AMI) with user-defined custom images
- Instance metadata service accessible from within instances
- Auto-scaling groups based on metrics (CPU, memory, custom)
- Elastic IP addresses and security groups (stateful firewalls)
Storage (S3-like)
- Object storage with 99.999999999% (11 nines) durability SLA
- Unlimited scale per bucket, billions of objects per account
- Versioning, lifecycle policies, encryption (SSE-S3, SSE-KMS, SSE-C)
- Pre-signed URLs for temporary access
- Event notifications on object creation/deletion
Networking (VPC-like)
- Virtual Private Cloud with configurable CIDR blocks
- Subnets (public/private), route tables, internet gateways, NAT gateways
- VPC peering and Transit Gateway for multi-VPC connectivity
- Elastic Load Balancer (ALB, NLB, GWLB)
- DNS resolution via Route 53-like service
Identity & Access Management (IAM)
- Users, groups, roles with fine-grained policies (JSON-based)
- Multi-factor authentication (MFA)
- Cross-account role assumption (STS)
- Policy evaluation with deny-overrides-allow semantics
Monitoring & Management
- CloudWatch-like metrics, logs, and alarms
- CloudTrail-like audit logging of all API calls
- Cost explorer and budgeting
1.2 Non-Functional Requirements
| Requirement | Target | Rationale |
|---|---|---|
| API Availability | 99.99% (control plane) | Control plane must be highly available |
| Data Durability | 11 nines (S3) | Customer data must never be lost |
| API Latency (p99) | < 500ms for control plane | Developer experience |
| Multi-tenancy Isolation | Strong (hard isolation) | No cross-tenant data leakage |
| Geographic Scale | 6+ regions, 18+ AZs | Global customer base |
| Throughput | Millions of API calls/sec across all services | Enterprise-scale usage |
| Compliance | SOC 2, ISO 27001, HIPAA, PCI-DSS | Enterprise trust |
2. Back-of-Envelope Calculations
2.1 Assumption Sets
We consider three scales:
| Parameter | Startup (10K customers) | Growth (100K customers) | Scale (1M customers) |
|---|---|---|---|
| Active accounts | 10,000 | 100,000 | 1,000,000 |
| Avg instances/account | 50 | 200 | 500 |
| Total instances | 500K | 20M | 500M |
| API calls/day | 10M | 100M | 1B |
| S3 objects total | 10B | 100B | 1T |
| S3 data volume | 10 PB | 100 PB | 1 EB |
| Control plane QPS | ~120 | ~1,200 | ~12,000 |
| Data plane throughput | 100 GB/s | 1 TB/s | 10 TB/s |
2.2 Storage Calculations (S3)
Per-object metadata: ~1 KB (name, size, content-type, timestamps, owner, version ID, tags)
| Scale | Objects | Metadata Storage | Data Storage | Total |
|---|---|---|---|---|
| Startup | 10B | 10 TB | 10 PB | ~10 PB |
| Growth | 100B | 100 TB | 100 PB | ~100 PB |
| Scale | 1T | 1 PB | 1 EB | ~1 EB |
Replication factor: 3x across AZs → multiply all by 3 for physical storage.
Metadata index: Need a distributed key-value store for object → location mapping. At 1T objects × 1KB = 1PB of metadata index, partitioned across thousands of nodes.
2.3 Bandwidth Calculations
Control plane:
- 1B API calls/day = ~11,500 QPS average
- Peak = 3× average = ~35,000 QPS
- Each API response ~2 KB → 70 MB/s average, 210 MB/s peak
Data plane (S3):
- Assume 10% of data accessed daily: 100 PB/day at scale
- 100 PB / 86,400s = ~1.16 TB/s average throughput
- Peak = 3× = ~3.5 TB/s
2.4 Compute Infrastructure
Hypervisor layer:
- 500M instances at scale, assume 10:1 overcommit ratio on average
- Need ~50M physical CPU cores
- At 64 cores/server = ~780K physical servers
Control plane servers:
- 35,000 QPS peak, each server handles ~500 QPS
- Need ~70 servers per service per region (with 3x redundancy = 210 servers)
3. System Interface Design
3.1 EC2 API
POST /v1/instances
Content-Type: application/json
{
"imageId": "ami-0abcdef1234567890",
"instanceType": "m5.xlarge",
"minCount": 1,
"maxCount": 10,
"subnetId": "subnet-abc123",
"securityGroupIds": ["sg-abc123"],
"keyName": "my-key-pair",
"tagSpecifications": [{
"resourceType": "instance",
"tags": [{"key": "Name", "value": "web-server"}]
}]
}
Response 200:
{
"instances": [
{
"instanceId": "i-0abc123def456789",
"state": "pending",
"publicIpAddress": "54.123.45.67",
"privateIpAddress": "10.0.1.50"
}
]
}
POST /v1/instances/{instanceId}/action
{ "action": "stop" | "start" | "terminate" }
GET /v1/instances?filter=tag:Name=web-server
DELETE /v1/instances?instanceIds=i-abc,i-def
3.2 S3 API
PUT /{bucket}/{key}
x-amz-acl: private
x-amz-storage-class: STANDARD
x-amz-server-side-encryption: AES256
Content-Type: application/octet-stream
Content-Length: 10485760
[object body]
Response 200:
x-amz-version-id: "v-abc123"
ETag: "\"d41d8cd98f00b204e9800998ecf8427e\""
GET /{bucket}/{key}?versionId=v-abc123
GET /{bucket}?list-type=2&prefix=logs/&max-keys=1000
DELETE /{bucket}/{key}?versionId=v-abc123
POST /{bucket}?delete (multi-object delete)
3.3 VPC API
POST /v1/vpcs
{
"cidrBlock": "10.0.0.0/16",
"instanceTenancy": "default",
"amazonProvidedIpv6CidrBlock": false
}
POST /v1/subnets
{
"vpcId": "vpc-abc123",
"cidrBlock": "10.0.1.0/24",
"availabilityZone": "us-east-1a"
}
POST /v1/route-tables
{
"vpcId": "vpc-abc123",
"routes": [
{"destinationCidrBlock": "0.0.0.0/0", "gatewayId": "igw-abc123"}
]
}
3.4 IAM API
POST /v1/roles
{
"roleName": "EC2DeployRole",
"assumeRolePolicyDocument": {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Service": "ec2.cloudprovider.com"},
"Action": "sts:AssumeRole"
}]
}
}
POST /v1/policies
{
"policyName": "S3ReadOnly",
"policyDocument": {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:ListBucket"],
"Resource": ["arn:aws:s3:::my-bucket", "arn:aws:s3:::my-bucket/*"]
}]
}
}
POST /v1/roles/{roleName}/attach-policy
{ "policyArn": "arn:aws:iam::policy/S3ReadOnly" }
3.5 STS (Temporary Credentials)
POST /v1/sts/assume-role
{
"roleArn": "arn:aws:iam::123456:role/EC2DeployRole",
"roleSessionName": "deploy-session",
"durationSeconds": 3600
}
Response 200:
{
"credentials": {
"accessKeyId": "ASIA...",
"secretAccessKey": "wJalr...",
"sessionToken": "FwoGZXIv...",
"expiration": "2026-08-28T12:00:00Z"
}
}
4. High-Level Architecture
graph TB
subgraph "Customer"
CLI[CLI / SDK]
Console[Web Console]
end
subgraph "Edge Layer"
DNS[Global DNS / Route 53]
CDN[CDN / CloudFront]
WAF[WAF / Shield]
end
subgraph "API Gateway Layer (per Region)"
GW1[API Gateway - EC2]
GW2[API Gateway - S3]
GW3[API Gateway - VPC]
GW4[API Gateway - IAM]
GW5[API Gateway - ELB]
end
subgraph "Auth Layer"
IAM_SVC[IAM Service]
STS_SVC[STS Service]
KMS_SVC[KMS Service]
end
subgraph "Control Plane Services"
EC2_CP[EC2 Control Plane]
S3_CP[S3 Control Plane]
VPC_CP[VPC Control Plane]
ELB_CP[ELB Control Plane]
AS[Auto Scaling]
end
subgraph "Data Plane"
subgraph "Region A"
subgraph "AZ 1a"
Hyp1[Hypervisor Pool]
S3Node1[S3 Storage Nodes]
VPC1a[VPC Overlay Network]
end
subgraph "AZ 1b"
Hyp2[Hypervisor Pool]
S3Node2[S3 Storage Nodes]
VPC1b[VPC Overlay Network]
end
subgraph "AZ 1c"
Hyp3[Hypervisor Pool]
S3Node3[S3 Storage Nodes]
VPC1c[VPC Overlay Network]
end
end
subgraph "Region B"
S3Node4[S3 Storage Nodes]
Hyp4[Hypervisor Pool]
VPC2[VPC Overlay Network]
end
end
subgraph "Shared Services"
CLOUDWATCH[CloudWatch Metrics]
CLOUDTRAIL[CloudTrail Audit]
CONFIG[Config Service]
EVENTBRIDGE[Event Bridge]
end
subgraph "Metadata & State"
DYNAMO[DynamoDB - Control State]
PARAMETER[Parameter Store]
SECRET[Secrets Manager]
end
CLI --> DNS
Console --> DNS
DNS --> CDN
CDN --> WAF
WAF --> GW1
WAF --> GW2
WAF --> GW3
WAF --> GW4
WAF --> GW5
GW1 --> IAM_SVC
GW2 --> IAM_SVC
GW3 --> IAM_SVC
GW4 --> IAM_SVC
GW5 --> IAM_SVC
IAM_SVC --> STS_SVC
IAM_SVC --> KMS_SVC
GW1 --> EC2_CP
GW2 --> S3_CP
GW3 --> VPC_CP
GW5 --> ELB_CP
EC2_CP --> AS
EC2_CP --> Hyp1
EC2_CP --> Hyp2
EC2_CP --> Hyp3
EC2_CP --> Hyp4
S3_CP --> S3Node1
S3_CP --> S3Node2
S3_CP --> S3Node3
S3_CP --> S3Node4
VPC_CP --> VPC1a
VPC_CP --> VPC1b
VPC_CP --> VPC1c
VPC_CP --> VPC2
EC2_CP --> DYNAMO
S3_CP --> DYNAMO
VPC_CP --> DYNAMO
Hyp1 --> CLOUDWATCH
S3Node1 --> CLOUDWATCH
GW1 --> CLOUDTRAIL
GW2 --> CLOUDTRAIL
EC2_CP --> EVENTBRIDGE
S3_CP --> EVENTBRIDGE
5. Core Component Design
5.1 Multi-Tenancy Architecture
Tenant Isolation Model
graph LR
subgraph "Tenant A"
VPCA[Tenant A VPC]
INSTA[Instances]
S3A[S3 Buckets]
end
subgraph "Tenant B"
VPCB[Tenant B VPC]
INSTB[Instances]
S3B[S3 Buckets]
end
subgraph "Shared Infrastructure"
HW[Physical Hardware]
NET[Network Fabric]
HYPER[Hypervisor]
end
VPCA --> HYPER
VPCB --> HYPER
HYPER --> HW
NET --> VPCA
NET --> VPCB
Isolation Layers:
| Layer | Isolation Mechanism | Guarantee |
|---|---|---|
| Network | VXLAN overlay + security groups | L2/L3 isolation, no cross-tenant traffic |
| Compute | Hardware VMs (Nitro) with dedicated ENI | Memory/CPU isolation via hardware |
| Storage | Encrypted volumes with per-tenant keys | Cryptographic isolation |
| Identity | Separate IAM realms per account | Policy-based access control |
| Metadata | Partitioned by account ID in all DBs | Query-level isolation |
5.2 VPC Design
VXLAN Overlay Network:
graph TB
subgraph "VPC: 10.0.0.0/16"
subgraph "Subnet 10.0.1.0/24 (AZ-a)"
ENI1[ENI: 10.0.1.10]
ENI2[ENI: 10.0.1.11]
end
subgraph "Subnet 10.0.2.0/24 (AZ-b)"
ENI3[ENI: 10.0.2.10]
ENI4[ENI: 10.0.2.11]
end
end
VTEP1[VTEP - AZ-a]
VTEP2[VTEP - AZ-b]
ENI1 --> VTEP1
ENI2 --> VTEP1
ENI3 --> VTEP2
ENI4 --> VTEP2
VTEP1 <-->|VXLAN VNI=1001| VTEP2
IGW[Internet Gateway]
NAT[NAT Gateway]
RT[Route Table]
VTEP1 --> RT
VTEP2 --> RT
RT --> IGW
RT --> NAT
Key Design Points:
- Each VPC assigned a unique VNI (VXLAN Network Identifier), 24-bit space = 16M VPCs
- VTEPs (VXLAN Tunnel Endpoints) run on each hypervisor host
- Security groups implemented as distributed stateful firewalls at the VTEP
- Route tables evaluated per-packet at the VTEP before encapsulation
- Cross-AZ traffic stays within the cloud fabric (no internet transit)
5.3 IAM Design
Policy Evaluation Engine:
graph TD
REQ[API Request] --> EXTRACT[Extract Principal, Action, Resource]
EXTRACT --> CACHE{Policy Cache Hit?}
CACHE -->|Yes| EVAL[Evaluate Cached Policy]
CACHE -->|No| FETCH[Fetch Policies from Store]
FETCH --> COMPILE[Compile Policy Set]
COMPILE --> CACHE2[Cache Policy Set]
CACHE2 --> EVAL
EVAL --> CHECK{Explicit Deny?}
CHECK -->|Yes| DENY[Return DENY]
CHECK -->|No| ALLOW{Explicit Allow?}
ALLOW -->|Yes| COND{Conditions Met?}
ALLOW -->|No| DENY2[Return DENY]
COND -->|Yes| ALLOW2[Return ALLOW]
COND -->|No| DENY3[Return DENY]
DENY --> AUDIT[Audit Log]
DENY2 --> AUDIT
DENY3 --> AUDIT
ALLOW2 --> AUDIT
Policy Store Design:
- Policies stored in a distributed document store (JSON policies)
- Index by: userId, groupId, roleId, policyId, resourceArn
- Policy compilation: merge all attached policies → single decision tree
- Cache compiled policies with TTL (5 min default, invalidated on policy change)
- Explicit deny always wins (short-circuit evaluation)
5.4 EC2 / Compute Service
Instance Lifecycle:
stateDiagram-v2
[*] --> Pending: RunInstances
Pending --> Running: Hypervisor boots VM
Pending --> Failed: Launch failure
Running --> Stopping: StopInstances
Stopping --> Stopped: VM halted
Stopped --> Starting: StartInstances
Starting --> Running: VM resumed
Running --> ShuttingDown: TerminateInstances
ShuttingDown --> Terminated: Resources released
Terminated --> [*]
Failed --> [*]
Placement Strategy:
- Bin-packing algorithm for instance placement across hosts
- Constraints: AZ affinity, host capacity, anti-affinity rules, dedicated hosts
- RACK awareness for fault tolerance
- Capacity reservation for critical workloads
Nitro Hypervisor Architecture:
- Bare-metal servers with Nitro Cards handling networking, storage, and virtualization
- Guest VMs run directly on host CPU (no hypervisor overhead for guest code path)
- ENI (Elastic Network Interface) implemented in FPGA on Nitro Card
- EBS volumes served via NVMe over Fabric from Nitro Card
5.5 S3 Object Storage
Storage Architecture:
graph TB
subgraph "S3 Request Flow"
CLIENT[Client] --> ZONE[Availability Zone Local Endpoint]
ZONE --> INDEX[Metadata Index Service]
INDEX --> ZONE_DATA{Data in this AZ?}
ZONE_DATA -->|Yes| LOCAL[Local Data Node]
ZONE_DATA -->|No| REMOTE[Cross-AZ Fetch]
LOCAL --> CLIENT
REMOTE --> CLIENT
end
subgraph "Data Replication"
DN1[Data Node AZ-1a]
DN2[Data Node AZ-1b]
DN3[Data Node AZ-1c]
DN1 <-->|Async Replication| DN2
DN2 <-->|Async Replication| DN3
DN3 <-->|Async Replication| DN1
end
subgraph "Metadata"
META1[Metadata Shard 1]
META2[Metadata Shard 2]
META3[Metadata Shard N]
end
INDEX --> META1
INDEX --> META2
INDEX --> META3
Key Design Decisions:
- Flat namespace: bucket/key → no directory structure in storage layer
- Metadata separation: Object metadata in distributed KV store, data in erasure-coded storage
- Erasure coding: 12+3 scheme (12 data blocks, 3 parity) for durability with 20% overhead vs 3x replication
- Multi-AZ replication: Write to primary AZ, async replicate to 2 other AZs
- Consistency model: Read-after-write consistent (eventual consistency for cross-region)
- Large object handling: Multi-part upload for objects > 100 MB
Metadata Index:
- Partitioned by hash(bucket + key) across thousands of shards
- Each shard replicated 3x for high availability
- Supports prefix scans via sorted key ordering within shards
5.6 Elastic Load Balancer
ALB (Application Load Balancer):
- Layer 7 load balancer with path/host-based routing
- WebSocket and HTTP/2 support
- Integrated WAF and authentication
NLB (Network Load Balancer):
- Layer 4 load balancer for TCP/UDP
- Static IP assignment, ultra-low latency
- Supports millions of RPS per endpoint
Health Checking:
- Configurable health check endpoints per target group
- Unhealthy targets removed from rotation automatically
- Sticky sessions via cookies when needed
5.7 CloudWatch Monitoring
Metrics Pipeline:
Instances → Agent → Kinesis Firehose → Time-Series DB → Dashboard/Alarms
- Custom metrics via PutMetricData API
- Alarms evaluate metrics at configurable intervals (1 min minimum)
- SNS integration for alarm notifications
- Log ingestion via CloudWatch Logs agent
6. Data Model
6.1 IAM Schema
-- Users
CREATE TABLE iam_users (
user_id UUID PRIMARY KEY,
account_id VARCHAR(12) NOT NULL,
user_name VARCHAR(128) NOT NULL,
created_at TIMESTAMP NOT NULL,
password_hash BYTEA,
mfa_enabled BOOLEAN DEFAULT FALSE,
status VARCHAR(16) DEFAULT 'Active'
);
-- Groups
CREATE TABLE iam_groups (
group_id UUID PRIMARY KEY,
account_id VARCHAR(12) NOT NULL,
group_name VARCHAR(128) NOT NULL
);
-- Roles
CREATE TABLE iam_roles (
role_id UUID PRIMARY KEY,
account_id VARCHAR(12) NOT NULL,
role_name VARCHAR(128) NOT NULL,
assume_role_policy JSONB NOT NULL,
arn VARCHAR(512) GENERATED ALWAYS AS (
'arn:aws:iam::' || account_id || ':role/' || role_name
) STORED
);
-- Policies
CREATE TABLE iam_policies (
policy_id UUID PRIMARY KEY,
account_id VARCHAR(12) NOT NULL,
policy_name VARCHAR(128) NOT NULL,
policy_document JSONB NOT NULL,
is_system BOOLEAN DEFAULT FALSE,
version INTEGER DEFAULT 1
);
-- User-Group Membership
CREATE TABLE iam_user_groups (
user_id UUID REFERENCES iam_users(user_id),
group_id UUID REFERENCES iam_groups(group_id),
PRIMARY KEY (user_id, group_id)
);
-- Policy Attachments (Users, Groups, Roles)
CREATE TABLE iam_policy_attachments (
attachment_id UUID PRIMARY KEY,
entity_type VARCHAR(8) NOT NULL, -- 'USER', 'GROUP', 'ROLE'
entity_id UUID NOT NULL,
policy_id UUID REFERENCES iam_policies(policy_id),
attached_at TIMESTAMP NOT NULL,
UNIQUE (entity_type, entity_id, policy_id)
);
-- Access Keys
CREATE TABLE iam_access_keys (
access_key_id VARCHAR(20) PRIMARY KEY,
user_id UUID REFERENCES iam_users(user_id),
secret_hash BYTEA NOT NULL,
status VARCHAR(16) DEFAULT 'Active',
created_at TIMESTAMP NOT NULL,
last_used_at TIMESTAMP
);
-- Indexes
CREATE INDEX idx_users_account ON iam_users(account_id);
CREATE INDEX idx_roles_account ON iam_roles(account_id);
CREATE INDEX idx_policies_account ON iam_policies(account_id);
CREATE INDEX idx_attachments_entity ON iam_policy_attachments(entity_type, entity_id);
6.2 EC2 Schema
-- Instances
CREATE TABLE ec2_instances (
instance_id VARCHAR(17) PRIMARY KEY, -- i-0abc123def456789
account_id VARCHAR(12) NOT NULL,
region VARCHAR(32) NOT NULL,
availability_zone VARCHAR(32) NOT NULL,
image_id VARCHAR(17) NOT NULL,
instance_type VARCHAR(32) NOT NULL,
state VARCHAR(16) NOT NULL, -- pending/running/stopped/terminated
private_ip INET,
public_ip INET,
host_id UUID, -- physical host
vpc_id VARCHAR(10),
subnet_id VARCHAR(17),
launched_at TIMESTAMP NOT NULL,
terminated_at TIMESTAMP,
user_data BYTEA,
tags JSONB
);
-- AMIs
CREATE TABLE ec2_amis (
image_id VARCHAR(17) PRIMARY KEY,
account_id VARCHAR(12) NOT NULL,
name VARCHAR(256),
description TEXT,
architecture VARCHAR(16), -- x86_64, arm64
root_device_type VARCHAR(16),
block_device_mappings JSONB,
state VARCHAR(16) DEFAULT 'available',
is_public BOOLEAN DEFAULT FALSE,
created_at TIMESTAMP NOT NULL
);
-- Security Groups
CREATE TABLE ec2_security_groups (
group_id VARCHAR(10) PRIMARY KEY, -- sg-abc123
account_id VARCHAR(12) NOT NULL,
vpc_id VARCHAR(10),
group_name VARCHAR(256),
description VARCHAR(256),
created_at TIMESTAMP NOT NULL
);
CREATE TABLE ec2_security_group_rules (
rule_id UUID PRIMARY KEY,
group_id VARCHAR(10) REFERENCES ec2_security_groups(group_id),
direction VARCHAR(8) NOT NULL, -- ingress/egress
ip_protocol VARCHAR(32),
from_port INTEGER,
to_port INTEGER,
cidr_ipv4 INET,
cidr_ipv6 CIDR,
source_group_id VARCHAR(10),
is_egress BOOLEAN
);
-- Indexes
CREATE INDEX idx_instances_account ON ec2_instances(account_id);
CREATE INDEX idx_instances_state ON ec2_instances(account_id, state);
CREATE INDEX idx_instances_host ON ec2_instances(host_id);
CREATE INDEX idx_instances_vpc ON ec2_instances(vpc_id);
6.3 S3 Schema
-- Buckets
CREATE TABLE s3_buckets (
bucket_name VARCHAR(63) PRIMARY KEY,
account_id VARCHAR(12) NOT NULL,
region VARCHAR(32) NOT NULL,
created_at TIMESTAMP NOT NULL,
versioning BOOLEAN DEFAULT FALSE,
lifecycle_rules JSONB,
encryption_default VARCHAR(16),
tags JSONB
);
-- Object Metadata (partitioned by hash of bucket+key)
CREATE TABLE s3_object_metadata (
bucket_name VARCHAR(63) NOT NULL,
object_key TEXT NOT NULL,
version_id VARCHAR(64),
size BIGINT NOT NULL,
content_type VARCHAR(256),
content_md5 VARCHAR(32),
storage_class VARCHAR(16) DEFAULT 'STANDARD',
owner_id VARCHAR(12) NOT NULL,
etag VARCHAR(128),
created_at TIMESTAMP NOT NULL,
last_modified TIMESTAMP NOT NULL,
is_latest BOOLEAN DEFAULT TRUE,
is_delete_marker BOOLEAN DEFAULT FALSE,
user_metadata JSONB,
PRIMARY KEY (bucket_name, object_key, version_id)
);
-- Object Location (maps to physical storage)
CREATE TABLE s3_object_location (
bucket_name VARCHAR(63) NOT NULL,
object_key TEXT NOT NULL,
version_id VARCHAR(64) NOT NULL,
partition_id INTEGER NOT NULL,
replica_az_1 VARCHAR(32) NOT NULL,
replica_az_2 VARCHAR(32) NOT NULL,
replica_az_3 VARCHAR(32) NOT NULL,
PRIMARY KEY (bucket_name, object_key, version_id)
);
-- Indexes
CREATE INDEX idx_objects_bucket_key ON s3_object_metadata(bucket_name, object_key);
CREATE INDEX idx_objects_latest ON s3_object_metadata(bucket_name, object_key)
WHERE is_latest = TRUE;
CREATE INDEX idx_objects_created ON s3_object_metadata(created_at);
6.4 VPC Schema
CREATE TABLE vpc_vpcs (
vpc_id VARCHAR(10) PRIMARY KEY,
account_id VARCHAR(12) NOT NULL,
cidr_block CIDR NOT NULL,
state VARCHAR(16) DEFAULT 'available',
dns_support BOOLEAN DEFAULT TRUE,
dns_hostnames BOOLEAN DEFAULT TRUE,
vxlan_vni INTEGER UNIQUE NOT NULL,
created_at TIMESTAMP NOT NULL
);
CREATE TABLE vpc_subnets (
subnet_id VARCHAR(17) PRIMARY KEY,
vpc_id VARCHAR(10) REFERENCES vpc_vpcs(vpc_id),
cidr_block CIDR NOT NULL,
availability_zone VARCHAR(32) NOT NULL,
map_public_ip BOOLEAN DEFAULT FALSE,
available_ips INTEGER GENERATED ALWAYS AS (
HOSTMASK(cidr_block) - 4
) STORED,
created_at TIMESTAMP NOT NULL
);
CREATE TABLE vpc_route_tables (
route_table_id VARCHAR(17) PRIMARY KEY,
vpc_id VARCHAR(10) REFERENCES vpc_vpcs(vpc_id),
is_main BOOLEAN DEFAULT FALSE,
created_at TIMESTAMP NOT NULL
);
CREATE TABLE vpc_routes (
route_id UUID PRIMARY KEY,
route_table_id VARCHAR(17) REFERENCES vpc_route_tables(route_table_id),
destination_cidr CIDR NOT NULL,
target_type VARCHAR(16) NOT NULL, -- igw, nat, vpc, instance
target_id VARCHAR(20) NOT NULL,
state VARCHAR(16) DEFAULT 'active'
);
CREATE TABLE vpc_internet_gateways (
igw_id VARCHAR(10) PRIMARY KEY,
account_id VARCHAR(12) NOT NULL,
vpc_id VARCHAR(10) REFERENCES vpc_vpcs(vpc_id),
state VARCHAR(16) DEFAULT 'available',
created_at TIMESTAMP NOT NULL
);
7. Key Design Decisions
7.1 Region vs Availability Zone
| Aspect | Region | Availability Zone |
|---|---|---|
| Definition | Geographic area (e.g., us-east-1) | Isolated datacenter within a region |
| Latency between units | 50-200ms+ | < 2ms |
| Failure independence | Independent power, network | Independent power, cooling, network |
| Data replication | Cross-region (async, user-configured) | Cross-AZ (automatic, synchronous for control plane) |
| Use case | Disaster recovery, latency optimization | High availability within a region |
Decision: 3 AZs per region minimum. Each AZ has independent:
- Power feeds from separate substations
- Cooling systems
- Network core switches
- Physical security
7.2 Consistency Model Trade-offs
| Service | Consistency | Rationale |
|---|---|---|
| IAM | Strong | Security-critical, low write volume |
| EC2 Control | Strong | Instance state must be consistent |
| S3 Data | Read-after-write | Developer expectation, achievable with modern distributed systems |
| S3 Cross-Region | Eventual | Latency vs consistency trade-off |
| VPC State | Strong | Network configuration must be consistent |
7.3 Multi-Tenancy Isolation: Hard vs Soft
Decision: Hard isolation (dedicated hardware paths where possible)
| Approach | Pros | Cons | Our Choice |
|---|---|---|---|
| Soft (containers) | High density, fast startup | Weaker isolation, noisy neighbor | Not for customer VMs |
| Hard (VMs with Nitro) | Strong isolation, hardware-backed | Lower density, higher cost | Chosen for EC2 |
| Hybrid | Best of both | Complex | Containers for our own services |
7.4 S3 Durability: Erasure Coding vs Replication
Decision: Erasure coding (12+3) for standard storage, 3x replication for metadata
| Approach | Overhead | Survives | Complexity |
|---|---|---|---|
| 3x Replication | 200% | 2 of 3 failures | Low |
| EC 12+3 | 25% | 3 block failures | Medium |
| EC 8+2 | 25% | 2 block failures | Medium |
Erasure coding chosen for data (87.5% space efficiency vs 33% for replication). Metadata uses 3x replication for faster reads and simpler implementation.
7.5 API Gateway: Per-Service vs Unified
Decision: Per-service API gateways with shared auth middleware
- Each service owns its gateway → independent scaling, deployment
- Shared IAM validation middleware → consistent auth, single source of truth
- API versioning per service → independent evolution
8. Scalability Strategy
8.1 Horizontal Scaling Dimensions
graph LR
subgraph "Scale Axis 1: Regions"
R1[Region US-East]
R2[Region EU-West]
R3[Region AP-South]
end
subgraph "Scale Axis 2: AZs per Region"
R1 --> AZ1a[AZ 1a]
R1 --> AZ1b[AZ 1b]
R1 --> AZ1c[AZ 1c]
end
subgraph "Scale Axis 3: Shards per Service"
AZ1a --> S1[Shard 1]
AZ1a --> S2[Shard 2]
AZ1a --> SN[Shard N]
end
subgraph "Scale Axis 4: Instances per Shard"
S1 --> I1[Instance 1]
S1 --> I2[Instance 2]
end
8.2 Scaling Strategies by Component
| Component | Scaling Strategy | Trigger | Max Scale |
|---|---|---|---|
| API Gateways | Horizontal pod autoscaling | CPU > 70%, queue depth | 1000+ per region |
| Control Plane | Shard by account_id | Shard size > 10K accounts | Unlimited |
| S3 Metadata | Consistent hashing shards | Shard > 100M objects | 10,000+ shards |
| S3 Data | Partition by object hash | Partition > 100 TB | Unlimited |
| Hypervisors | Cluster-based placement | Cluster utilization > 80% | 100K+ hosts |
| VPC VTEPs | Per-host deployment | New host added | 1:1 with hosts |
8.3 Database Sharding Strategy
Control Plane State (DynamoDB-like):
- Primary shard key:
account_id(hash) - Secondary index:
service_type + resource_id - Hot account handling: Sub-sharding by resource type for high-volume accounts
S3 Metadata:
- Shard key:
hash(bucket_name + object_key) - 16-bit partition number → 65,536 partitions
- Each partition fits on a single storage node
- Rebalancing via consistent hashing with virtual nodes
8.4 Capacity Planning
Growth Projections:
| Year | Accounts | Instances | S3 Data | Regions | AZs |
|---|---|---|---|---|---|
| Year 1 | 10K | 500K | 10 PB | 3 | 9 |
| Year 3 | 100K | 20M | 100 PB | 6 | 18 |
| Year 5 | 1M | 500M | 1 EB | 12 | 36 |
Infrastructure Scaling:
- Add AZs before adding regions (deeper before wider)
- Pre-provision capacity 20% above forecast
- Auto-scale data plane independently of control plane
9. Curveball Scenarios
9.1 Curveball 1: A Major Customer Launches a Viral App Overnight
Scenario: A customer’s application goes viral, generating 100× normal traffic in 2 hours. Their S3 bucket serves 50,000 GET requests/sec, and their EC2 instances need to scale from 100 to 10,000.
Impact Analysis:
- S3 GET rate: 50K RPS from one account (potential hot partition)
- EC2 launch: 9,900 new instances in minutes
- Network bandwidth: Potential egress spike to 100 Gbps from one tenant
Response Strategy:
| Layer | Action | Timeline |
|---|---|---|
| S3 | Consistent hashing distributes across partitions; no single hot partition if key distribution is uniform. If hot key detected, enable read replicas automatically. | Immediate |
| EC2 | Auto-scaling group triggers scale-out. Pre-warmed instance pool provides immediate capacity. Spot instances supplement on-demand. | 1-5 min |
| Network | Egress rate limiting per account (configurable). Customer notified if approaching limits. | Immediate |
| Billing | Real-time cost alerting to customer. Budget breach notifications via SNS. | Immediate |
| Infrastructure | Regional capacity monitoring triggers pre-provisioning if regional utilization > 85%. | 5-15 min |
Prevention:
- Customer education on auto-scaling best practices
- Service quotas with easy increase process
- Predictive scaling based on historical patterns
9.2 Curveball 2: An AZ Goes Down Due to Power Failure
Scenario: AZ-1a in us-east-1 loses power due to a substation failure. 50,000 instances and critical S3 data affected.
Impact Analysis:
- All instances in AZ-1a become unreachable
- S3 objects with primary replica in AZ-1a: reads served from AZ-1b/1c replicas
- VPC subnets in AZ-1a: routing disrupted
- Estimated recovery time: 2-4 hours for power restoration
Response Strategy:
| Phase | Action | Timeline |
|---|---|---|
| Detection | Health monitors detect AZ unreachability within 30 seconds. Auto-classify as AZ-down event. | 0-30 sec |
| Failover | S3: Reads automatically served from remaining AZ replicas. Writes queued and redirected to healthy AZs. | 30-60 sec |
| Compute | EC2: Instances marked as impaired. Auto-scaling groups launch replacement instances in healthy AZs. | 1-5 min |
| Network | VPC: Route tables updated to remove AZ-1a routes. Cross-AZ traffic rerouted. | 1-2 min |
| Notification | CloudWatch alarms fire. SNS notifications to affected customers. Status page updated. | 1-3 min |
| Recovery | Power restored → instances in AZ-1a recover. Data consistency verified. Customer instances restarted if needed. | 2-4 hrs |
Data Safety:
- S3: No data loss (3 AZ replication, quorum-based writes)
- EBS: Volumes in AZ-1a may need recovery from cross-AZ snapshots
- RDS: Multi-AZ failover to standby in healthy AZ
9.3 Curveball 3: A Bug in the IAM Policy Engine Causes Mass Denial
Scenario: A deployment of the IAM policy evaluation service introduces a bug that denies all requests for accounts with more than 100 policies attached.
Impact Analysis:
- Affected: Large enterprise accounts with complex IAM setups
- Symptom: All API calls return AccessDenied
- Blast radius: Potentially 10-20% of enterprise customers
Response Strategy:
| Phase | Action | Timeline |
|---|---|---|
| Detection | Error rate monitoring detects spike in AccessDenied responses. Anomaly detection flags the pattern. | 1-3 min |
| Triage | On-call engineer identifies correlation with recent IAM deployment. | 3-5 min |
| Mitigation | Emergency rollback of IAM service to previous version. Circuit breaker activates for affected accounts, bypassing to cached policies. | 5-10 min |
| Verification | Canary validation against test accounts. Gradual traffic shift back. | 10-15 min |
| Root Cause | Post-incident analysis. Bug: integer overflow in policy compilation for >100 policies. | Post-incident |
Prevention:
- Canary deployments with automatic rollback on error rate increase
- Policy evaluation unit tests with edge cases (100+ policies)
- Chaos engineering:定期 inject IAM failures in staging
- Policy compilation result caching as fallback
9.4 Curveball 4: Cross-Region Data Sovereignty Regulation Change
Scenario: A new regulation requires that data for customers in Region EU-West must never leave the EU geographic boundary, including metadata and audit logs.
Impact Analysis:
- Current state: CloudTrail audit logs replicated to us-east-1 for central analysis
- Current state: S3 cross-region replication configured by some EU customers to US regions
- Current state: IAM policy cache may serve from any region
Response Strategy:
| Component | Change | Effort |
|---|---|---|
| CloudTrail | Add data residency tags. EU customers’ logs stay in EU regions. Regional analysis pipelines. | Medium |
| S3 | Enforce cross-region replication restrictions for EU buckets. Add compliance validation. | Low |
| IAM | Regional policy cache isolation. EU accounts evaluate policies within EU. | Medium |
| Billing | Regional billing data isolation. | Low |
| Support | Regional support ticket routing. | Low |
Architecture Change:
graph LR
subgraph "EU Region"
EU_API[EU API Gateway]
EU_IAM[EU IAM Service]
EU_TRAIL[EU CloudTrail]
EU_ANALYTICS[EU Analytics]
end
subgraph "US Region"
US_API[US API Gateway]
US_IAM[US IAM Service]
US_TRAIL[US CloudTrail]
US_ANALYTICS[US Analytics]
end
EU_API --> EU_IAM
EU_API --> EU_TRAIL
EU_TRAIL --> EU_ANALYTICS
US_API --> US_IAM
US_API --> US_TRAIL
US_TRAIL --> US_ANALYTICS
EU_ANALYTICS -.->|No cross-region data| US_ANALYTICS
9.5 Curveball 5: A Customer Discovers They Can Access Another Customer’s Data
Scenario: A security researcher finds that by crafting specific S3 requests with manipulated VNI headers, they can read objects from another tenant’s bucket.
Impact Analysis:
- Severity: CRITICAL - Cross-tenant data leakage
- Root cause: VTEP not validating VNI against tenant’s VPC membership
- Blast radius: All customers sharing VTEP infrastructure
Response Strategy:
| Phase | Action | Timeline |
|---|---|---|
| Detection | Bug bounty report received. Security team validates the exploit. | T+0 |
| Containment | Emergency patch: VTEP validates VNI against tenant authorization before processing. Deploy to all hosts. | T+1 hr |
| Assessment | Scan CloudTrail logs for suspicious cross-tenant access patterns. Identify affected customers. | T+2 hrs |
| Notification | Notify affected customers per breach notification policy. Engage legal and PR. | T+24 hrs |
| Remediation | Full security audit of VTEP code. Additional validation layers. Enhanced monitoring. | T+1 week |
Prevention:
- Defense in depth: VNI validation at multiple layers (VTEP, hypervisor, network fabric)
- Regular penetration testing by third parties
- Bug bounty program with attractive rewards
- Zero-trust network architecture: never trust tenant-supplied network parameters
10. Failure Modes and Mitigations
10.1 Failure Mode Matrix
| Failure Mode | Probability | Impact | Detection | Mitigation | RTO | RPO |
|---|---|---|---|---|---|---|
| Single host failure | High | Low (1-2 instances) | Health monitor | Auto-replace, live migration | 5 min | 0 |
| AZ power loss | Medium | High | Multi-probe monitoring | Cross-AZ replication, failover | 30 min | 0 (S3) |
| Region outage | Low | Critical | Regional health dashboard | Multi-region active-active | 1 hr | 1 hr |
| Control plane DB corruption | Low | Critical | Checksum validation | Multi-AZ sync replication | 15 min | 0 |
| Network partition | Medium | Medium | BFD, ECMP monitoring | Automatic rerouting | 1 min | 0 |
| IAM service outage | Low | Critical | API error rate monitoring | Cached credentials, STS token validity | N/A | N/A |
| S3 metadata corruption | Low | High | CRC validation on reads | 3x replication, automatic repair | 1 hr | 0 |
| Hypervisor compromise | Very Low | Critical | Integrity monitoring | Nitro hardware isolation, attestation | 4 hrs | N/A |
| DDoS attack | Medium | Medium | Traffic anomaly detection | WAF, rate limiting, scrubbing | 5 min | N/A |
| Data center flood/fire | Very Low | High | Physical sensors | Geographic distribution | 4 hrs | 0 |
10.2 Control Plane Resilience
graph TB
subgraph "Control Plane HA"
subgraph "AZ-1a"
CP1[Control Plane Primary]
DB1[State DB Primary]
end
subgraph "AZ-1b"
CP2[Control Plane Standby]
DB2[State DB Replica]
end
subgraph "AZ-1c"
CP3[Control Plane Standby]
DB3[State DB Replica]
end
end
LB[Load Balancer] --> CP1
LB --> CP2
LB --> CP3
CP1 --> DB1
CP2 --> DB2
CP3 --> DB3
DB1 <-->|Sync Replication| DB2
DB2 <-->|Sync Replication| DB3
HEALTH[Health Checker] --> CP1
HEALTH --> CP2
HEALTH --> CP3
HEALTH --> LB
Key Principles:
- Quorum-based writes: Control plane writes require 2-of-3 AZ acknowledgment
- Automatic leader election: If primary fails, standby takes over within 30 seconds
- Read-your-writes: Client sessions pinned to consistent replica during write operations
- Circuit breakers: If a service dependency fails, fallback to cached state or graceful degradation
10.3 Data Plane Resilience
S3 Self-Healing:
- Background scanner runs every hour checking object integrity (checksums)
- Corrupted or missing replicas detected → automatic regeneration from healthy replicas
- If all replicas corrupted → alert operations team (extremely rare with EC + 3 AZ)
- Storage node failure → data redistributed automatically within 15 minutes
EC2 Instance Recovery:
- Host failure detected → instances marked as impaired
- Auto-scaling groups launch replacements in healthy hosts
- EBS volumes remain intact (stored separately from compute)
- Instance recovery: start new instance attached to existing EBS volume
10.4 Disaster Recovery Plan
| Scenario | Strategy | RTO | RPO |
|---|---|---|---|
| AZ failure | Cross-AZ failover (automatic) | 5 min | 0 |
| Region failure | Multi-region active-active (automatic for S3, manual for EC2) | 30 min | 0 (S3), variable (EC2) |
| Global failure | Not designed for (requires geographic distribution at customer level) | N/A | N/A |
| Data corruption | Versioning + cross-region replication | 1 hr | Last replicated version |
| Ransomware | Immutable backups + versioning | 4 hrs | Last clean version |
Appendix: Glossary
| Term | Definition |
|---|---|
| VNI | VXLAN Network Identifier - 24-bit ID for overlay network segmentation |
| VTEP | VXLAN Tunnel Endpoint - encapsulates/decapsulates VXLAN packets |
| ENI | Elastic Network Interface - virtual network card for instances |
| AMI | Amazon Machine Image - template for launching instances |
| STS | Security Token Service - issues temporary credentials |
| Nitro | Hardware-based virtualization architecture using dedicated cards |
| EBS | Elastic Block Store - persistent block storage for instances |
| RTO | Recovery Time Objective - max acceptable downtime |
| RPO | Recovery Point Objective - max acceptable data loss |