Design a File Management System like Amazon S3
Author: System Design Expert | Date: 2026-08-28 Scope: Object storage service with 11 nines durability at planetary scale
Table of Contents
- Requirements Clarification
- Back-of-Envelope Calculations
- System Interface Design
- High-Level Architecture
- Core Component Design
- Data Model
- Key Design Decisions
- Scalability Strategy
- Curveball Scenarios
- Failure Modes and Mitigations
1. Requirements Clarification
1.1 Functional Requirements
| ID | Requirement | Priority |
|---|---|---|
| FR-01 | Object storage — Store and retrieve objects (files) of any type up to 5TB each | Must Have |
| FR-02 | Bucket organization — Namespace objects within buckets; buckets are globally unique | Must Have |
| FR-03 | Multipart upload — Upload large objects in parallel parts; assemble server-side | Must Have |
| FR-04 | Versioning — Maintain multiple versions of an object; enable restore | Must Have |
| FR-05 | Lifecycle policies — Auto-transition objects between storage classes; auto-expire | Must Have |
| FR-06 | Replication — Cross-region and cross-account replication for durability and compliance | Must Have |
| FR-07 | Access control — Fine-grained IAM policies, bucket policies, ACLs, presigned URLs | Must Have |
| FR-08 | Event notifications — Trigger Lambda/SNS/SQS on object events (create, delete, etc.) | Must Have |
| FR-09 | Server-side encryption — SSE-S3, SSE-KMS, SSE-C encryption options | Must Have |
| FR-10 | Storage classes — Standard, Intelligent-Tiering, Standard-IA, Glacier, Glacier Deep Archive | Must Have |
| FR-11 | Object tagging — Key-value metadata tags for organization and lifecycle | Should Have |
| FR-12 | Inventory & metrics — Daily/weekly inventory reports, storage metrics | Should Have |
| FR-13 | Object Lock — WORM (Write Once Read Many) compliance mode | Should Have |
| FR-14 | Select/Query — SQL-like queries on objects in-place (CSV, JSON, Parquet) | Should Have |
| FR-15 | Website hosting — Static website hosting with custom domains | Could Have |
| FR-16 | Transfer acceleration — Optimized transfers via edge locations | Could Have |
1.2 Non-Functional Requirements
| ID | Requirement | Target |
|---|---|---|
| NFR-01 | Durability | 99.999999999% (11 nines) — lose 0.000000001% of objects/year |
| NFR-02 | Availability (Standard) | 99.99% (4 nines) — 52 min downtime/year |
| NFR-03 | Availability (Single AZ) | 99.999999999% (11 nines) for Read-After-Write within AZ |
| NFR-04 | Throughput | Unlimited aggregate; burst to 50,000 PUTs/sec and 5,500 GETs/sec per prefix |
| NFR-05 | Scalability | Trillions of objects, exabytes of data, millions of requests/sec |
| NFR-06 | Consistency | Strong consistency for all operations (as of 2020+) |
| NFR-07 | Latency | P99 < 100ms for GET within same region |
| NFR-08 | Global scale | 30+ regions, 100+ availability zones |
| NFR-09 | Security | AES-256 encryption, TLS 1.3, SOC 2, HIPAA, PCI DSS compliant |
| NFR-10 | Cost efficiency | Pay-per-use pricing; cheaper than on-premises by 30%+ |
2. Back-of-Envelope Calculations
2.1 Assumption Sets
| Parameter | Conservative | Base Case | Aggressive |
|---|---|---|---|
| Total stored data | 5 EB | 20 EB | 100 EB |
| Total objects | 500T | 2T | 10T |
| Avg object size | 10 MB | 10 MB | 10 MB |
| Requests/day (GET) | 100B | 500B | 2T |
| Requests/day (PUT) | 10B | 50B | 200B |
| Requests/day (other) | 5B | 25B | 100B |
| Total requests/day | 115B | 575B | 2.3T |
| Buckets | 10B | 50B | 200B |
| Active users | 5M | 20M | 100M |
2.2 Storage Estimates
Base Case — Raw Data:
├── User data: 20 EB (1 EB = 1,000 PB = 1,000,000 TB)
│
├── Replication factor: 3x (within region, across AZs)
│ └── 20 EB × 3 = 60 EB
│
├── Cross-region replicas: 2x (avg 2 replica regions per bucket)
│ └── 20 EB × 2 = 40 EB
│
├── Metadata per object: 1 KB avg
│ └── 2T objects × 1 KB = 2 PB
│
├── Metadata replication: 3x
│ └── 2 PB × 3 = 6 PB
│
├── Versioning overhead: 2x (avg 2 versions per object)
│ └── 60 EB × 2 = 120 EB
│
├── Erasure coding parity: 0.4x (for Glacier classes)
│ └── 20 EB × 0.4 = 8 EB
│
└── Total physical storage: ~234 EB
2.3 Bandwidth Estimates
Base Case — Request Traffic:
├── Total requests/day: 575B
├── Avg GET size: 500 KB
├── Avg PUT size: 1 MB
├── Avg other size: 1 KB
│
├── GET bandwidth/day: 400B × 500 KB = 200 EB/day
├── PUT bandwidth/day: 50B × 1 MB = 50 EB/day
├── Other bandwidth/day: 25B × 1 KB = 25 PB/day
│
├── Total bandwidth/day: ~250 EB/day
├── Avg bandwidth/sec: 250 EB / 86,400s ≈ 2.9 PB/sec
├── Peak bandwidth (3x): ~8.7 PB/sec
│
└── Request QPS:
├── Avg QPS: 575B / 86,400s ≈ 6.65M QPS
├── Peak QPS: ~20M QPS
├── GET QPS: ~4.6M avg, ~14M peak
└── PUT QPS: ~580K avg, ~1.7M peak
2.4 Infrastructure Estimates
Storage Nodes:
├── Per node capacity: 100 TB usable (10 × 10 TB drives)
├── Nodes for user data: 20 EB / 100 TB = 200,000 nodes
├── Nodes with 3x replica: 600,000 nodes
└── With versioning: ~1.2M nodes
Metadata Servers:
├── Per server capacity: 100M object metadata entries
├── Servers needed: 2T / 100M = 20,000 servers
└── With replication: 60,000 servers
Frontend/Proxy Servers:
├── Per server capacity: 10,000 req/sec
├── Servers needed: 20M QPS / 10K = 2,000 servers
└── With redundancy: 6,000 servers
Total Estimated Infrastructure:
├── Storage nodes: ~1.2M
├── Metadata servers: ~60K
├── Frontend servers: ~6K
├── Management services: ~10K
└── Total: ~1.27M servers
3. System Interface Design
3.1 REST API
# Bucket Operations
PUT / # CreateBucket
DELETE / # DeleteBucket
GET /?location # GetBucketLocation
GET /?versioning # GetBucketVersioning
PUT /?versioning # PutBucketVersioning
GET /?lifecycle # GetBucketLifecycle
PUT /?lifecycle # PutBucketLifecycle
GET /?policy # GetBucketPolicy
PUT /?policy # PutBucketPolicy
DELETE /?policy # DeleteBucketPolicy
GET /?replication # GetBucketReplication
PUT /?replication # PutBucketReplication
GET /?encryption # GetBucketEncryption
PUT /?encryption # PutBucketEncryption
GET /?tagging # GetBucketTagging
PUT /?tagging # PutBucketTagging
# Object Operations
GET /{key} # GetObject
PUT /{key} # PutObject
DELETE /{key} # DeleteObject
HEAD /{key} # HeadObject
GET /{key}?torrent # GetObjectTorrent
GET /{key}?acl # GetObjectAcl
PUT /{key}?acl # PutObjectAcl
GET /{key}?tagging # GetObjectTagging
PUT /{key}?tagging # PutObjectTagging
POST /{key}?restore # RestoreObject
POST /{key}?select # SelectObjectContent
POST /{key}?job # InitiateJob
# Multipart Upload
POST /{key}?uploads # CreateMultipartUpload
PUT /{key}?uploadId={id}&partNumber={n} # UploadPart
POST /{key}?uploadId={id} # CompleteMultipartUpload
POST /{key}?uploadId={id} # AbortMultipartUpload
GET /?uploads # ListMultipartUploads
GET /{key}?uploadId={id} # ListParts
# Listing
GET /?list-type=2 # ListObjectsV2
GET /?list-type=2&prefix={p} # ListObjects with prefix
GET /?list-type=2&delimiter={d} # ListObjects with delimiter
GET /?list-type=2&max-keys={n} # ListObjects with limit
# Versioning
GET /{key}?versionId={id} # Get specific version
DELETE /{key}?versionId={id} # Delete specific version
GET /?versions # ListObjectVersions
# Presigned URL
GET /{key}?X-Amz-Algorithm=AWS4-HMAC-SHA256
&X-Amz-Credential={cred}
&X-Amz-Date={date}
&X-Amz-Expires={seconds}
&X-Amz-SignedHeaders={headers}
&X-Amz-Signature={sig}
3.2 API Request/Response Examples
# CreateBucket
PUT / HTTP/1.1
Host: my-bucket.s3.us-east-1.amazonaws.com
x-amz-bucket-region: us-east-1
x-amz-date: 20260828T120000Z
Authorization: AWS4-HMAC-SHA256 ...
<CreateBucketConfiguration>
<LocationConstraint>us-west-2</LocationConstraint>
</CreateBucketConfiguration>
# Response: 200 OK
---
# PutObject
PUT /photos/vacation.jpg HTTP/1.1
Host: my-bucket.s3.us-east-1.amazonaws.com
Content-Type: image/jpeg
Content-Length: 5242880
x-amz-acl: private
x-amz-server-side-encryption: AES256
x-amz-storage-class: STANDARD
x-amz-metadata-user-id: 12345
x-amz-tagging: project=vacation,year=2026
x-amz-date: 20260828T120000Z
Authorization: AWS4-HMAC-SHA256 ...
<binary data>
# Response: 200 OK
ETag: "d41d8cd98f00b204e9800998ecf8427e"
x-amz-version-id: OTh46nT2EFtbn0VqCiQ6MiHm
x-amz-server-side-encryption: AES256
x-amz-storage-class: STANDARD
---
# GetObject
GET /photos/vacation.jpg HTTP/1.1
Host: my-bucket.s3.us-east-1.amazonaws.com
Range: bytes=0-1048575
x-amz-date: 20260828T120000Z
Authorization: AWS4-HMAC-SHA256 ...
# Response: 206 Partial Content
Content-Range: bytes 0-1048575/5242880
Content-Type: image/jpeg
ETag: "d41d8cd98f00b204e9800998ecf8427e"
x-amz-version-id: OTh46nT2EFtbn0VqCiQ6MiHm
Last-Modified: Fri, 28 Aug 2026 12:00:00 GMT
<binary data>
---
# CreateMultipartUpload
POST /large-video.mp4?uploads HTTP/1.1
Host: my-bucket.s3.us-east-1.amazonaws.com
x-amz-storage-class: STANDARD
x-amz-server-side-encryption: aws:kms
x-amz-server-side-encryption-aws-kms-key-id: arn:aws:kms:...
# Response: 200 OK
<InitiateMultipartUploadResult>
<Bucket>my-bucket</Bucket>
<Key>large-video.mp4</Key>
<UploadId>VXDuQLA8xe2oEdFBzLOAi9DkJ</UploadId>
</InitiateMultipartUploadResult>
---
# UploadPart
PUT /large-video.mp4?uploadId=VXDuQLA8xe2oEdFBzLOAi9DkJ&partNumber=1
Host: my-bucket.s3.us-east-1.amazonaws.com
Content-Length: 10485760 # 10 MB
<binary part data>
# Response: 200 OK
ETag: "a54783c9c3a6e39d9b9a8d3e1f2c4b5a"
---
# CompleteMultipartUpload
POST /large-video.mp4?uploadId=VXDuQLA8xe2oEdFBzLOAi9DkJ
Host: my-bucket.s3.us-east-1.amazonaws.com
<CompleteMultipartUpload>
<Part>
<PartNumber>1</PartNumber>
<ETag>"a54783c9c3a6e39d9b9a8d3e1f2c4b5a"</ETag>
</Part>
<Part>
<PartNumber>2</PartNumber>
<ETag>"b65894d0d4b7f40e0c0b9e4f2g3d5c6b"</ETag>
</Part>
</CompleteMultipartUpload>
# Response: 200 OK
<CompleteMultipartUploadResult>
<Location>https://my-bucket.s3.us-east-1.amazonaws.com/large-video.mp4</Location>
<Bucket>my-bucket</Bucket>
<Key>large-video.mp4</Key>
<ETag>"d41d8cd98f00b204e9800998ecf8427e-2"</ETag>
<VersionId>OTh46nT2EFtbn0VqCiQ6MiHm</VersionId>
</CompleteMultipartUploadResult>
3.3 SDK Interface (Conceptual)
class S3Client:
# Bucket operations
def create_bucket(self, name: str, region: str,
versioning: bool = False,
encryption: EncryptionConfig = None) -> Bucket:
...
def delete_bucket(self, name: str) -> None:
...
def list_buckets(self) -> List[Bucket]:
...
# Object operations
def put_object(self, bucket: str, key: str, body: bytes | IOBase,
content_type: str = None,
storage_class: StorageClass = StorageClass.STANDARD,
tags: Dict[str, str] = None,
metadata: Dict[str, str] = None,
server_side_encryption: str = None,
acl: str = None) -> Object:
...
def get_object(self, bucket: str, key: str,
version_id: str = None,
range: str = None) -> Object:
...
def delete_object(self, bucket: str, key: str,
version_id: str = None) -> None:
...
# Multipart upload
def upload_file(self, filename: str, bucket: str, key: str,
part_size: int = 8*1024*1024, # 8 MB default
concurrency: int = 10,
storage_class: StorageClass = StorageClass.STANDARD) -> None:
...
# Presigned URL
def generate_presigned_url(self, bucket: str, key: str,
expiration: int = 3600,
http_method: str = 'GET') -> str:
...
# Lifecycle
def put_bucket_lifecycle(self, bucket: str,
rules: List[LifecycleRule]) -> None:
...
# Replication
def put_bucket_replication(self, bucket: str,
config: ReplicationConfig) -> None:
...
4. High-Level Architecture
graph TB
subgraph Client["Client Layer"]
APP[Applications / SDKs]
CLI[AWS CLI / Console]
PARTNER[Partner Integrations]
end
subgraph Edge["Edge Layer"]
DNS[Route 53 / DNS]
CDN[CloudFront / Edge Locations]
WAF[WAF / Shield]
LB[Global Load Balancer]
end
subgraph Frontend["Frontend Layer"]
AUTH[Authentication Service<br/>STS / IAM]
API_GW[API Frontend<br/>Request Routing]
QUOTA[Quota & Rate Limiting]
BILLING[Billing Metering]
end
subgraph Control["Control Plane"]
BUCKET_MGR[Bucket Manager<br/>Namespace Service]
POLICY[Policy Engine<br/>ACL / Bucket Policy]
LIFECYCLE[Lifecycle Manager]
REPLICATION[Replication Service]
VERSION_MGR[Version Manager]
ENCRYPTION[Encryption Service<br/>KMS Integration]
EVENT[Event Notification<br/>SNS / SQS / Lambda]
INVENTORY[Inventory & Analytics]
end
subgraph Data["Data Plane"]
subgraph ObjectStorage["Object Storage Engine"]
PARTITION[Partition Router<br/>Consistent Hash]
subgraph AZ1["Availability Zone A"]
STORAGE_A1[Storage Node A1]
STORAGE_A2[Storage Node A2]
STORAGE_A3[Storage Node A3]
end
subgraph AZ2["Availability Zone B"]
STORAGE_B1[Storage Node B1]
STORAGE_B2[Storage Node B2]
STORAGE_B3[Storage Node B3]
end
subgraph AZ3["Availability Zone C"]
STORAGE_C1[Storage Node C1]
STORAGE_C2[Storage Node C2]
STORAGE_C3[Storage Node C3]
end
end
subgraph Metadata["Metadata Service"]
META_DB[(Metadata DB<br/>DynamoDB-like)]
META_CACHE[(Redis Cluster<br/>Hot Metadata)]
end
subgraph Multipart["Multipart Service"]
MP_STATE[Multipart State Store]
MP_ASSEMBLE[Part Assembler]
end
end
subgraph Storage["Replication & Archive"]
REPLICATOR[Async Replicator]
ARCHIVE[Glacier Archive<br/>Cold Storage]
DR_SITE[DR Region]
end
subgraph Infra["Infrastructure"]
MQ[Kafka / Kinesis<br/>Event Stream]
MONITOR[CloudWatch / Monitoring]
LOG[Audit Logging<br/>CloudTrail]
end
APP --> DNS
CLI --> DNS
PARTNER --> DNS
DNS --> CDN
CDN --> WAF
WAF --> LB
LB --> AUTH
LB --> API_GW
API_GW --> QUOTA
QUOTA --> BILLING
QUOTA --> BUCKET_MGR
QUOTA --> POLICY
BUCKET_MGR --> META_DB
POLICY --> META_DB
PARTITION --> STORAGE_A1
PARTITION --> STORAGE_B1
PARTITION --> STORAGE_C1
STORAGE_A1 --> META_DB
STORAGE_B1 --> META_DB
STORAGE_C1 --> META_DB
API_GW --> PARTITION
API_GW --> MP_STATE
MP_STATE --> MP_ASSEMBLE
MP_ASSEMBLE --> PARTITION
LIFECYCLE --> ARCHIVE
REPLICATION --> REPLICATOR
REPLICATOR --> DR_SITE
STORAGE_A1 -->|Events| MQ
STORAGE_B1 -->|Events| MQ
STORAGE_C1 -->|Events| MQ
MQ --> EVENT
MQ --> INVENTORY
MQ --> LOG
MQ --> REPLICATOR
ENCRYPTION --> STORAGE_A1
ENCRYPTION --> STORAGE_B1
ENCRYPTION --> STORAGE_C1
META_CACHE --> META_DB
5. Core Component Design
5.1 Object Storage Engine
The storage engine is the core of S3, responsible for storing, retrieving, and protecting object data across multiple availability zones.
Storage Node Architecture
graph TD
subgraph StorageNode["Storage Node"]
subgraph DiskLayer["Disk Layer"]
D1[Disk 1: 10 TB]
D2[Disk 2: 10 TB]
D3[Disk 3: 10 TB]
D4[Disk 4: 10 TB]
D5[Disk 5: 10 TB]
D6[Disk 6: 10 TB]
D7[Disk 7: 10 TB]
D8[Disk 8: 10 TB]
D9[Disk 9: 10 TB]
D10[Disk 10: 10 TB]
end
subgraph DataLayer["Data Layer"]
EC[Erasure Coding Engine]
CHECKSUM[Checksum Verification<br/>CRC32C + SHA256]
COMPRESSION[Compression<br/>Zstandard / LZ4]
ENCRYPTION[Encryption Engine<br/>AES-256-XTS]
end
subgraph CacheLayer["Cache Layer"]
MEM_CACHE[Memory Cache<br/>Hot Object Blocks]
SSD_CACHE[SSD Cache<br/>Warm Object Blocks]
end
subgraph ProtocolLayer["Protocol Layer"]
READ_HANDLER[Read Handler]
WRITE_HANDLER[Write Handler]
REPAIR_HANDLER[Self-Healing Handler]
end
end
READ_HANDLER --> MEM_CACHE
MEM_CACHE -->|Miss| SSD_CACHE
SSD_CACHE -->|Miss| EC
EC --> D1
EC --> D2
EC --> D3
WRITE_HANDLER --> ENCRYPTION
ENCRYPTION --> COMPRESSION
COMPRESSION --> CHECKSUM
CHECKSUM --> EC
REPAIR_HANDLER -->|Background| EC
REPAIR_HANDLER -->|Verify| CHECKSUM
Data Placement Strategy
Object → Block → Chunk → Disk
1. Object received (e.g., 1 GB file)
2. Split into fixed-size blocks (4 MB each → 256 blocks)
3. Each block erasure-coded into data + parity chunks
- Scheme: 12 data + 3 parity = 15 chunks (25% overhead)
- Each chunk: 4 MB / 12 = 333 KB
4. Chunks distributed across disks and nodes:
- Data chunks: Spread across 12 different nodes
- Parity chunks: Spread across 3 different nodes
- All 15 nodes in different AZs when possible
5. Metadata recorded: which node holds which chunk
Reconstruction:
├── Need any 12 of 15 chunks to reconstruct a block
├── Can tolerate 3 simultaneous node failures
└── Reconstruction triggered automatically on failure
Erasure Coding vs Replication
| Factor | 3x Replication | Erasure Coding (12+3) |
|---|---|---|
| Storage overhead | 200% (3x) | 25% (1.25x) |
| Fault tolerance | Lose 2 of 3 copies | Lose 3 of 15 chunks |
| Read performance | Read any copy (fast) | Read 12 chunks (slower) |
| Write performance | Write 3 copies | Encode + write 15 chunks |
| Rebuild cost | Copy full object | Re-encode from 12 chunks |
| Best for | Standard class | Glacier / IA classes |
| Cost | Higher | ~60% cheaper |
5.2 Metadata Service
Metadata is the most accessed component — every request touches metadata before touching data.
graph TD
subgraph MetadataService["Metadata Service"]
subgraph HotTier["Hot Tier (Redis Cluster)"]
R1[Redis Shard 1]
R2[Redis Shard 2]
R3[Redis Shard N]
end
subgraph WarmTier["Warm Tier (In-Memory DB)"]
W1[Memory Node 1]
W2[Memory Node 2]
end
subgraph ColdTier["Cold Tier (DynamoDB-like)"]
C1[DynamoDB Shard 1]
C2[DynamoDB Shard 2]
end
end
REQ[Request] --> R1
REQ --> R2
REQ --> R3
R1 -->|Miss| W1
R2 -->|Miss| W2
W1 -->|Miss| C1
W2 -->|Miss| C2
C1 -->|Promote| W1
W1 -->|Promote| R1
Metadata Stored Per Object
{
"object_id": "obj-a1b2c3d4e5f6",
"bucket_id": "bucket-xyz789",
"key": "photos/vacation/2026/beach.jpg",
"version_id": "v-20260828-120000-abc123",
"size_bytes": 5242880,
"content_type": "image/jpeg",
"content_md5": "d41d8cd98f00b204e9800998ecf8427e",
"etag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
"storage_class": "STANDARD",
"encryption": {
"algorithm": "AES256",
"key_id": "arn:aws:kms:us-east-1:123456:key/abc-123"
},
"owner": {
"id": "user-12345",
"display_name": "John Doe"
},
"acl": "private",
"tags": {
"project": "vacation",
"year": "2026"
},
"user_metadata": {
"camera": "Canon EOS R5",
"location": "Maldives"
},
"created_at": "2026-08-28T12:00:00Z",
"last_modified": "2026-08-28T12:00:00Z",
"replication_status": "COMPLETED",
"lifecycle_status": "ACTIVE",
"chunk_locations": [
{"chunk_id": "chk-001", "node_id": "node-a1", "az": "us-east-1a"},
{"chunk_id": "chk-002", "node_id": "node-b2", "az": "us-east-1b"},
{"chunk_id": "chk-003", "node_id": "node-c3", "az": "us-east-1c"}
]
}
5.3 Multipart Upload Service
Handles large object uploads by splitting them into parallel parts.
sequenceDiagram
participant C as Client
participant FE as Frontend
participant MP as Multipart Service
participant STORE as Storage Engine
C->>FE: CreateMultipartUpload(key, metadata)
FE->>MP: Create upload session
MP->>STORE: Reserve object ID
MP->>MP: Generate UploadId
MP->>C: Return UploadId
par Parallel Part Uploads
C->>FE: UploadPart(UploadId, partNumber=1, data)
FE->>STORE: Store part 1
STORE->>FE: ETag-1
FE->>C: 200 OK (ETag-1)
and
C->>FE: UploadPart(UploadId, partNumber=2, data)
FE->>STORE: Store part 2
STORE->>FE: ETag-2
FE->>C: 200 OK (ETag-2)
and
C->>FE: UploadPart(UploadId, partNumber=3, data)
FE->>STORE: Store part 3
STORE->>FE: ETag-3
FE->>C: 200 OK (ETag-3)
end
C->>FE: CompleteMultipartUpload(UploadId, [parts])
FE->>MP: Validate all parts present
MP->>MP: Verify ETags match
MP->>STORE: Assemble parts into object
STORE->>STORE: Erasure-code assembled object
STORE->>STORE: Replicate across AZs
STORE->>MP: Object stored successfully
MP->>C: 200 OK (ETag, VersionId)
Note over C,STORE: Abandoned uploads auto-cleaned after 7 days
Multipart Upload Design Details
| Parameter | Value | Rationale |
|---|---|---|
| Minimum part size | 5 MB (except last part) | Balance between overhead and parallelism |
| Maximum part size | 5 GB | Limit memory usage during assembly |
| Maximum parts | 10,000 per upload | Limit metadata overhead |
| Maximum object size | 5 TB | 10,000 parts × 5 GB |
| Part expiration | 7 days (auto-abort) | Clean up abandoned uploads |
| Concurrency | Up to 10,000 parallel parts | SDK-configurable |
| Part storage | Temporary, separate from final object | Avoid wasting space on incomplete uploads |
5.4 Versioning Service
graph TD
subgraph Versioning["Versioning Service"]
PUT[PUT Object] --> CHECK{Versioning Enabled?}
CHECK -->|Yes| VERSION[Create New Version]
CHECK -->|No| OVERWRITE[Overwrite Current]
VERSION --> CURRENT[Mark Previous as Old]
VERSION --> NEW[Store New as Current]
VERSION --> DELETE_MARKER{Same Key?}
DELETE_MARKER -->|DELETE after PUT| DM[Create Delete Marker]
DELETE_MARKER -->|Normal| NORMAL[Normal Version]
OVERWRITE --> REPLACE[Replace Object]
subgraph VersionChain["Version Chain"]
V1[Version 1<br/>v-001<br/>Current]
V2[Version 2<br/>v-002<br/>Current]
V3[Version 3<br/>v-003<br/>Current]
DM1[Delete Marker<br/>dm-001]
end
V1 --> V2 --> V3
V3 --> DM1
end
Version Lifecycle
Object: "report.pdf"
Timeline:
├── T1: PUT report.pdf → Version v-001 (current)
├── T2: PUT report.pdf → Version v-002 (current), v-001 (old)
├── T3: PUT report.pdf → Version v-003 (current), v-002 (old), v-001 (old)
├── T4: DELETE report.pdf → Delete Marker dm-001 (current)
│ └── Object appears deleted, but v-001, v-002, v-003 still exist
├── T5: PUT report.pdf → Version v-004 (current), dm-001 (old)
│ └── Object reappears, delete marker becomes old version
└── T6: DELETE report.pdf version=v-002 → Permanent deletion of v-002
└── Only explicit version deletion permanently removes data
5.5 Lifecycle Management
graph TD
subgraph LifecycleEngine["Lifecycle Engine"]
SCHEDULE[Scheduled Scanner<br/>Every 15 minutes]
EVAL[Evaluator<br/>Match rules]
ACTIONS[Action Executor]
subgraph Rules["Lifecycle Rules"]
R1[Transition Rule<br/>30 days → Standard-IA]
R2[Transition Rule<br/>90 days → Glacier]
R3[Transition Rule<br/>180 days → Glacier Deep Archive]
R4[Expiration Rule<br/>365 days → Delete]
R5[Noncurrent Expiration<br/>7 versions → Delete old]
R6[Abort Incomplete MPU<br/>7 days → Abort]
end
subgraph StorageClasses["Storage Classes"]
STD[Standard<br/>Hot, 3x replica]
STD_IA[Standard-IA<br/>Infrequent, 3x replica]
ONE_ZA[One Zone-IA<br/>Single AZ, 3x replica]
GLACIER[Glacier<br/>Archive, EC]
GLACIER_DEEP[Glacier Deep Archive<br/>Deep archive, EC]
end
R1 --> STD_IA
R2 --> GLACIER
R3 --> GLACIER_DEEP
R4 -->|Delete| X[Deleted]
R5 -->|Delete old versions| X
end
SCHEDULE --> EVAL
EVAL --> R1
EVAL --> R2
EVAL --> R3
EVAL --> R4
EVAL --> R5
EVAL --> R6
R1 --> ACTIONS
R2 --> ACTIONS
R3 --> ACTIONS
R4 --> ACTIONS
R5 --> ACTIONS
R6 --> ACTIONS
Storage Class Comparison
| Class | Availability | Retrieval Time | Min Storage | Min Object Size | Relative Cost |
|---|---|---|---|---|---|
| Standard | 99.99% | Immediate | None | None | 1.0x |
| Intelligent-Tiering | 99.99% | Immediate | $0.0025/object/month | None | 1.0x + monitoring |
| Standard-IA | 99.9% | Immediate | 30 days | 128 KB | 0.6x |
| One Zone-IA | 99.5% | Immediate | 30 days | 128 KB | 0.48x |
| Glacier | 99.999999999% | 1-5 hours | 90 days | 128 KB | 0.004x |
| Glacier Deep Archive | 99.999999999% | 12-48 hours | 180 days | 128 KB | 0.00099x |
5.6 Replication Service
graph TD
subgraph SourceRegion["Source Region (us-east-1)"]
SRC_BUCKET[Source Bucket]
SRC_EVENT[Object Created Event]
SRC_QUEUE[Replication Queue]
end
subgraph ReplicationService["Replication Service"]
DETECT[Event Detector]
FILTER[Rule Filter<br/>Prefix / Tag / Storage Class]
TRANSFORM[Transform<br/>Storage Class / Encryption / Tags]
TRANSFER[Data Transfer<br/>Secure Channel]
VERIFY[Verification<br/>Checksum + Metadata]
STATUS[Status Tracker]
end
subgraph DestRegion["Destination Region (eu-west-1)"]
DST_BUCKET[Destination Bucket]
DST_STORE[Storage Engine]
end
SRC_EVENT --> DETECT
DETECT --> FILTER
FILTER -->|Match| TRANSFORM
FILTER -->|No Match| SKIP[Skip]
TRANSFORM --> TRANSFER
TRANSFER --> DST_STORE
DST_STORE --> VERIFY
VERIFY -->|Success| STATUS
VERIFY -->|Failure| RETRY[Retry with Backoff]
RETRY --> TRANSFER
STATUS -->|Update| SRC_QUEUE
Replication Types
| Type | Description | Use Case |
|---|---|---|
| CRR (Cross-Region) | Replicate to bucket in different region | Disaster recovery, latency reduction |
| SRR (Same-Region) | Replicate to bucket in same region | Compliance, workload isolation |
| Same-account | Source and dest in same AWS account | Simple DR setup |
| Cross-account | Source and dest in different accounts | Multi-account governance |
5.7 Access Control System
graph TD
subgraph AuthFlow["Authentication & Authorization Flow"]
REQ[Incoming Request] --> AUTH{Authenticate}
AUTH -->|IAM User| SIGV4[Signature V4 Verification]
AUTH -->|IAM Role| STS[STS Token Verification]
AUTH -->|Presigned URL| PRESIGN[Presigned URL Verification]
AUTH -->|Anonymous| ANON[Anonymous Check]
SIGV4 --> POLICY_EVAL
STS --> POLICY_EVAL
PRESIGN --> POLICY_EVAL
ANON --> POLICY_EVAL
POLICY_EVAL[Policy Evaluation Engine] --> CHECK1{IAM Policy<br/>Allows?}
CHECK1 -->|Deny| DENY[403 Forbidden]
CHECK1 -->|Not Specified| CHECK2{Bucket Policy<br/>Allows?}
CHECK2 -->|Deny| DENY
CHECK2 -->|Not Specified| CHECK3{ACL<br/>Allows?}
CHECK3 -->|Deny| DENY
CHECK3 -->|Allow| CHECK4{S3 Block Public Access?}
CHECK4 -->|Blocked| DENY
CHECK4 -->|Allowed| ALLOW[200 OK]
end
Permission Evaluation Order
1. Explicit Deny (any policy) → DENY (immediate)
2. S3 Block Public Access settings → DENY if enabled
3. IAM Policy (user/role) → Allow or Not Specified
4. Bucket Policy → Allow, Deny, or Not Specified
5. ACL → Allow or Not Specified
6. Default → DENY (deny by default)
Explicit Deny always wins over Allow.
5.8 Encryption Service
graph TD
subgraph EncryptionOptions["Encryption Options"]
SSE_S3[SSE-S3<br/>Amazon-managed keys]
SSE_KMS[SSE-KMS<br/>Customer-managed keys]
SSE_C[SSE-C<br/>Customer-provided keys]
CSE[CSE-C<br/>Client-side encryption]
end
subgraph SSE_S3_Flow["SSE-S3 Flow"]
DATA1[Plaintext Data] --> ENCRYPT1[AES-256 Encrypt]
KEY1[S3 Master Key] --> ENCRYPT1
ENCRYPT1 --> CIPHER1[Ciphertext]
CIPHER1 --> STORE1[Store Object]
KEY_REF1[Key ID: aws/s3] --> STORE1
end
subgraph SSE_KMS_Flow["SSE-KMS Flow"]
DATA2[Plaintext Data] --> ENCRYPT2[AES-256 Encrypt]
DKE2[Data Key from KMS] --> ENCRYPT2
ENCRYPT2 --> CIPHER2[Ciphertext]
CIPHER2 --> STORE2[Store Object]
WRAPPED2[Encrypted Data Key] --> STORE2
KMS_KEY2[KMS CMK] -->|Encrypt| WRAPPED2
end
subgraph SSE_C_Flow["SSE-C Flow"]
DATA3[Plaintext Data] --> ENCRYPT3[AES-256 Encrypt]
CUST_KEY3[Customer Key in Header] --> ENCRYPT3
ENCRYPT3 --> CIPHER3[Ciphertext]
CIPHER3 --> STORE3[Store Object]
NOTE3[Key NOT stored by S3]
end
6. Data Model
6.1 Metadata Database Schema
-- Bucket metadata
CREATE TABLE buckets (
bucket_id UUID PRIMARY KEY,
bucket_name VARCHAR(63) UNIQUE NOT NULL, -- DNS-compliant
owner_account VARCHAR(12) NOT NULL, -- AWS account ID
region VARCHAR(32) NOT NULL,
creation_time TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
versioning VARCHAR(20) DEFAULT 'Disabled', -- Enabled, Suspended, Disabled
encryption JSONB, -- Default encryption config
lifecycle_rules JSONB, -- Lifecycle configuration
replication_config JSONB, -- Replication configuration
tagging JSONB, -- Bucket tags
policy TEXT, -- Bucket policy (JSON)
object_count BIGINT DEFAULT 0,
total_size_bytes BIGINT DEFAULT 0,
is_public BOOLEAN DEFAULT false,
CONSTRAINT valid_versioning CHECK (versioning IN ('Enabled', 'Suspended', 'Disabled'))
);
-- Object metadata
CREATE TABLE objects (
object_id UUID PRIMARY KEY,
bucket_id UUID NOT NULL REFERENCES buckets(bucket_id),
key VARCHAR(1024) NOT NULL, -- Object key (path)
version_id VARCHAR(64) NOT NULL,
is_latest BOOLEAN DEFAULT true,
is_delete_marker BOOLEAN DEFAULT false,
size_bytes BIGINT NOT NULL,
content_type VARCHAR(255),
content_md5 VARCHAR(32),
etag VARCHAR(128) NOT NULL,
storage_class VARCHAR(32) DEFAULT 'STANDARD',
encryption_alg VARCHAR(32),
encryption_key_id VARCHAR(256),
owner_account VARCHAR(12),
acl VARCHAR(64) DEFAULT 'private',
tagging JSONB,
user_metadata JSONB,
chunk_locations JSONB NOT NULL, -- Array of chunk locations
replication_status VARCHAR(32) DEFAULT 'STORAGE_CLASSAnalysis',
lifecycle_status VARCHAR(32) DEFAULT 'ACTIVE',
creation_time TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
last_modified TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
archive_retrieval_time TIMESTAMP WITH TIME ZONE,
UNIQUE(bucket_id, key, version_id),
CONSTRAINT valid_storage_class CHECK (
storage_class IN ('STANDARD', 'STANDARD_IA', 'ONEZONE_IA',
'GLACIER', 'DEEP_ARCHIVE', 'INTELLIGENT_TIERING')
)
);
-- Multipart upload state
CREATE TABLE multipart_uploads (
upload_id UUID PRIMARY KEY,
bucket_id UUID NOT NULL REFERENCES buckets(bucket_id),
key VARCHAR(1024) NOT NULL,
initiator_account VARCHAR(12),
owner_account VARCHAR(12),
storage_class VARCHAR(32) DEFAULT 'STANDARD',
encryption_alg VARCHAR(32),
encryption_key_id VARCHAR(256),
initiation_time TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
expiration_time TIMESTAMP WITH TIME ZONE, -- Auto-abort time
status VARCHAR(20) DEFAULT 'ACTIVE', -- ACTIVE, COMPLETED, ABORTED
total_parts INT DEFAULT 0,
completed_parts INT DEFAULT 0,
part_metadata JSONB, -- Part ETags and sizes
CONSTRAINT valid_status CHECK (status IN ('ACTIVE', 'COMPLETED', 'ABORTED'))
);
-- Multipart upload parts
CREATE TABLE multipart_parts (
part_id BIGSERIAL PRIMARY KEY,
upload_id UUID NOT NULL REFERENCES multipart_uploads(upload_id),
part_number INT NOT NULL,
size_bytes BIGINT NOT NULL,
etag VARCHAR(128) NOT NULL,
storage_location JSONB NOT NULL, -- Where this part is stored
upload_time TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
UNIQUE(upload_id, part_number)
);
-- Chunk locations (for erasure-coded objects)
CREATE TABLE chunks (
chunk_id UUID PRIMARY KEY,
object_id UUID NOT NULL REFERENCES objects(object_id),
block_index INT NOT NULL, -- Which block this chunk belongs to
chunk_index INT NOT NULL, -- Which chunk within the block (0-14)
node_id VARCHAR(64) NOT NULL, -- Storage node ID
az_id VARCHAR(16) NOT NULL, -- Availability zone
disk_id VARCHAR(32) NOT NULL, -- Physical disk
offset BIGINT NOT NULL, -- Offset on disk
size_bytes INT NOT NULL,
checksum VARCHAR(32) NOT NULL, -- CRC32C
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
last_verified TIMESTAMP WITH TIME ZONE,
status VARCHAR(20) DEFAULT 'HEALTHY', -- HEALTHY, DEGRADED, REPAIRING
UNIQUE(object_id, block_index, chunk_index)
);
-- Replication tasks
CREATE TABLE replication_tasks (
task_id UUID PRIMARY KEY,
source_object_id UUID NOT NULL REFERENCES objects(object_id),
source_bucket_id UUID NOT NULL,
dest_bucket_id UUID NOT NULL,
dest_region VARCHAR(32) NOT NULL,
status VARCHAR(20) DEFAULT 'PENDING', -- PENDING, IN_PROGRESS, COMPLETED, FAILED
error_message TEXT,
retry_count INT DEFAULT 0,
max_retries INT DEFAULT 5,
created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
completed_at TIMESTAMP WITH TIME ZONE,
next_retry_at TIMESTAMP WITH TIME ZONE
);
-- Indexes
CREATE INDEX idx_objects_bucket ON objects(bucket_id, key);
CREATE INDEX idx_objects_bucket_latest ON objects(bucket_id, key) WHERE is_latest = true;
CREATE INDEX idx_objects_version ON objects(bucket_id, key, version_id);
CREATE INDEX idx_objects_storage_class ON objects(storage_class, last_modified);
CREATE INDEX idx_objects_lifecycle ON objects(lifecycle_status, last_modified);
CREATE INDEX idx_objects_creation ON objects(creation_time);
CREATE INDEX idx_mp_uploads_bucket ON multipart_uploads(bucket_id, key);
CREATE INDEX idx_mp_uploads_status ON multipart_uploads(status, expiration_time);
CREATE INDEX idx_chunks_object ON chunks(object_id, block_index);
CREATE INDEX idx_chunks_node ON chunks(node_id, status);
CREATE INDEX idx_replication_status ON replication_tasks(status, next_retry_at);
CREATE INDEX idx_buckets_name ON buckets(bucket_name);
CREATE INDEX idx_buckets_region ON buckets(region);
6.2 Metadata Service (Redis) Schema
# Hot object metadata (TTL: 5 minutes)
HSET meta:hot:{object_id}
key "photos/vacation/beach.jpg"
bucket_id "bucket-xyz789"
size_bytes "5242880"
content_type "image/jpeg"
etag "\"d41d8cd98f00b204e9800998ecf8427e\""
storage_class "STANDARD"
chunk_locations "[{\"node\":\"n1\",\"az\":\"1a\"},{\"node\":\"n2\",\"az\":\"1b\"}]"
EXPIRE meta:hot:{object_id} 300
# Bucket metadata cache
HSET meta:bucket:{bucket_id}
name "my-bucket"
versioning "Enabled"
encryption "{\"algorithm\":\"AES256\"}"
policy "{\"Version\":\"2012-10-17\",\"Statement\":[...]}"
EXPIRE meta:bucket:{bucket_id} 60
# Multipart upload state
HSET mp:{upload_id}
bucket_id "bucket-xyz789"
key "large-video.mp4"
status "ACTIVE"
total_parts "3"
completed_parts "2"
parts "{\"1\":\"etag1\",\"2\":\"etag2\"}"
EXPIRE meta mp:{upload_id} 604800 # 7 days
# Rate limiting
INCR rate:{account_id}:{hour}
EXPIRE rate:{account_id}:{hour} 3600
# Request metering (for billing)
INCR billing:{account_id}:{date}:gets
INCR billing:{account_id}:{date}:puts
INCRBY billing:{account_id}:{date}:bytes_downloaded {bytes}
INCRBY billing:{account_id}:{date}:bytes_uploaded {bytes}
6.3 Object Key Naming and Partitioning
Key Structure:
├── Bucket: globally unique name (3-63 chars, DNS-compliant)
├── Key: hierarchical path (up to 1,024 bytes UTF-8)
│ ├── Prefix: "photos/2026/vacation/"
│ └── Object name: "beach.jpg"
│
Partitioning Strategy:
├── Hash key: SHA-256(bucket_id + key)
├── Partition ID: hash % 1024 (1024 partitions)
├── Each partition handled by a storage group
└── Each storage group spans all AZs in the region
Why 1024 partitions?
├── Balances load evenly across infrastructure
├── Allows independent scaling per partition
├── Limits blast radius of any single failure
└── Can increase to 4096 as needed
7. Key Design Decisions
7.1 Consistency Model Evolution
| Era | Model | Rationale |
|---|---|---|
| 2006-2020 | Eventual consistency | Simpler, better performance, acceptable for most use cases |
| 2020+ | Strong consistency | Customer demand, competitive pressure, infrastructure maturity |
Decision: Strong consistency for all operations
Strong Consistency Implementation:
├── Write path:
│ ├── Accept write to primary AZ
│ ├── Write to quorum of AZs (2 of 3)
│ ├── Return success only after quorum ack
│ └── Async replicate to remaining AZ
│
├── Read path:
│ ├── Read from any AZ
│ ├── If read stale (version check fails):
│ │ ├── Read from all AZs
│ │ ├── Return latest version
│ │ └── Async repair stale copies
│ └── Read repair on every read
│
└── Trade-off:
├── Write latency increases by ~10-20ms
├── Read latency increases by ~5ms (with read repair)
└── Worth it for correctness guarantees
7.2 Erasure Coding vs Replication
| Factor | Decision | Rationale |
|---|---|---|
| Standard class | 3x replication | Performance-critical, can afford 200% overhead |
| IA classes | 3x replication | Still need good read performance |
| Glacier classes | Erasure coding (12+3) | Cost-critical, read performance less important |
| Deep Archive | Erasure coding (8+2) | Maximum cost efficiency |
Decision: Hybrid approach based on storage class
7.3 Object Size Limits
| Limit | Value | Rationale |
|---|---|---|
| Single PUT max | 5 GB | Balance between API simplicity and streaming |
| Multipart max | 5 TB | Covers 99.9% of use cases |
| Minimum part size | 5 MB | Prevents excessive metadata overhead |
| Max parts | 10,000 | Limits metadata complexity |
| Key length | 1,024 bytes | Supports deep hierarchical paths |
7.4 Bucket Naming
Bucket Name Rules:
├── 3-63 characters
├── Lowercase letters, numbers, hyphens, dots
├── Must start with letter or number
├── Cannot be IP address format
├── Globally unique (across all AWS accounts, all regions)
└── Once created, name cannot be changed
Why globally unique?
├── Enables global endpoint: bucket.s3.amazonaws.com
├── Simplifies DNS resolution
├── Prevents naming conflicts
└── Trade-off: Name exhaustion at scale (mitigated by 63-char space)
7.5 Request Routing
Request Routing Strategy:
├── DNS-based routing:
│ ├── bucket.region.s3.amazonaws.com → Region-specific endpoint
│ └── bucket.s3.amazonaws.com → Default region endpoint
│
├── Path-style vs virtual-hosted-style:
│ ├── Virtual-hosted: bucket.s3.region.amazonaws.com/key (preferred)
│ └── Path-style: s3.region.amazonaws.com/bucket/key (legacy)
│
├── Transfer acceleration:
│ ├── bucket.s3-accelerate.amazonaws.com
│ └── Routes through CloudFront edge for optimal path
│
└── Regional endpoints:
├── Reduces latency (stay in-region)
├── Avoids cross-region data transfer charges
└── Required for some operations (e.g., replication)
7.6 Billing Model
Pricing Components:
├── Storage cost: $/GB/month (varies by class and region)
├── Request cost: $/1,000 requests (varies by type)
│ ├── PUT/COPY/POST/LIST: $0.005/1K
│ └── GET/OTHER: $0.0004/1K
├── Data transfer:
│ ├── Inbound: Free
│ └── Outbound: $0.09/GB (first 10 TB/month)
├── Advanced features:
│ ├── Lifecycle transitions: $0.05/1K
│ ├── Replication: Storage + request costs at dest
│ └── Select/Query: $0.01/GB scanned
└── Early deletion (IA classes):
├── Standard-IA: Pro-rated storage + min-charge
└── Glacier: Full minimum storage duration charge
8. Scalability Strategy
8.1 Multi-Level Scaling Architecture
graph TD
subgraph Global["Global Scale"]
R1[Region 1: us-east-1]
R2[Region 2: eu-west-1]
R3[Region 3: ap-southeast-1]
RN[Region N: 30+ regions]
end
subgraph Region["Per-Region Scale"]
AZ1[AZ-A]
AZ2[AZ-B]
AZ3[AZ-C]
AZN[AZ-N: 3-6 AZs per region]
end
subgraph AZ["Per-AZ Scale"]
SG1[Storage Group 1]
SG2[Storage Group 2]
SGN[Storage Group N: 1024 groups]
end
subgraph StorageGroup["Per-Storage-Group Scale"]
SN1[Storage Node 1]
SN2[Storage Node 2]
SN3[Storage Node 3]
SNM[Storage Node M: 100-1000 nodes]
end
Global --> Region
Region --> AZ
AZ --> StorageGroup
StorageGroup --> StorageNode
R1 --> AZ1
R1 --> AZ2
R1 --> AZ3
AZ1 --> SG1
AZ2 --> SG1
AZ3 --> SG1
SG1 --> SN1
SG1 --> SN2
SG1 --> SN3
8.2 Scaling Dimensions
| Dimension | Strategy | Current Scale | Target Scale |
|---|---|---|---|
| Regions | Add new regions | 30+ | 50+ |
| AZs per region | Add AZs | 3-6 | 6-9 |
| Storage groups | Consistent hashing | 1,024 | 4,096 |
| Nodes per group | Auto-scale | 100-1,000 | 1,000-10,000 |
| Frontend servers | Auto-scale | 6,000 | 20,000+ |
| Metadata servers | Sharded | 60,000 | 200,000+ |
8.3 Hot Key / Hot Partition Handling
Problem: A single key or prefix receives disproportionate traffic
Detection:
├── Monitor per-key request rate
├── Alert when > 5,000 GETs/sec or > 500 PUTs/sec per prefix
└── Track top 100 hottest keys
Mitigation:
├── 1. Client-side: Add random suffix to key
│ └── "photo.jpg" → "photo-a1b2c3.jpg"
│
├── 2. CDN caching: Cache hot objects at edge
│ └── CloudFront caches GET responses
│
├── 3. Read replicas: Create additional read replicas
│ └── For extremely hot objects
│
├── 4. Partition splitting: Split hot partition
│ └── Sub-partition by key hash
│
└── 5. Rate limiting: Enforce per-prefix limits
└── 5,500 GETs/sec and 500 PUTs/sec per prefix
8.4 Metadata Scaling
Metadata is the bottleneck — every request touches metadata.
Scaling Strategy:
├── Tier 1: In-memory cache (Redis)
│ ├── Hot objects: Last accessed < 5 min
│ ├── Hit rate: > 95%
│ └── Scale: Add Redis shards
│
├── Tier 2: Warm cache (in-memory DB)
│ ├── Warm objects: Last accessed < 1 hour
│ ├── Hit rate: > 99% (combined with Tier 1)
│ └── Scale: Add memory nodes
│
├── Tier 3: Persistent store (DynamoDB-like)
│ ├── All objects
│ ├── Hit rate: < 1% (cache misses only)
│ └── Scale: Add DynamoDB shards
│
└── Optimization:
├── Metadata compression (protobuf)
├── Batch metadata reads for LIST operations
└── Async metadata updates for non-critical fields
8.5 Data Transfer Optimization
Large Object Transfer:
├── Multipart upload: Parallel parts (10-100 concurrent)
├── Adaptive part size: Based on network bandwidth
├── Retry with exponential backoff: Per-part retry
├── Checksum verification: Per-part and final
└── Progress tracking: Client-side progress bar
Download Optimization:
├── Range requests: Partial downloads
├── Parallel range fetch: Multiple ranges in parallel
├── CDN caching: Edge cache for GET requests
├── Transfer acceleration: CloudFront-optimized path
└── Adaptive bitrate: For streaming use cases
9. Curveball Scenarios
Curveball 1: “How do we achieve 11 nines of durability (99.999999999%)?”
The Math:
11 nines = 99.999999999% durability
Annual failure rate = 0.000000001% = 1 in 1 billion
With 2 trillion objects:
├── Expected annual losses: 2T × 0.000000001% = 2 objects/year
├── That's less than 1 object loss per year per trillion objects stored
└── Over 10 years: ~20 objects lost out of 2T
This means we can lose AT MOST 2 objects per year out of 2 trillion.
Multi-Layer Strategy:
graph TD
subgraph Layer1["Layer 1: Erasure Coding / Replication"]
EC[3x replication across AZs]
note1["Protects against: Disk failure, Node failure, AZ failure"]
end
subgraph Layer2["Layer 2: Continuous Verification"]
VERIFY[Background checksum verification]
note2["Every object verified weekly"]
end
subgraph Layer3["Layer 3: Self-Healing"]
REPAIR[Automatic repair on detection]
note3["Rebuild from surviving copies/chunks"]
end
subgraph Layer4["Layer 4: Cross-Region Replication"]
CRR[Async cross-region replication]
note4["Protects against: Region failure, Natural disaster"]
end
subgraph Layer5["Layer 5: Hardware Quality"]
HW[Enterprise-grade hardware]
note5["Enterprise SSDs/HDDs with low BER"]
end
subgraph Layer6["Layer 6: Scrubbing & Reconciliation"]
SCRUB[Regular data scrubbing]
note6["Detect silent bit rot, repair proactively"]
end
Layer1 --> Layer2 --> Layer3 --> Layer4 --> Layer5 --> Layer6
Detailed Implementation:
| Layer | Mechanism | Failure Protected | Frequency |
|---|---|---|---|
| Erasure Coding | 12+3 parity across 15 nodes | Up to 3 simultaneous node failures | Always |
| 3x Replication | 3 copies in 3 AZs | Up to 2 AZ failures | Always |
| Checksum Verification | CRC32C on every read + weekly scan | Silent bit rot, corruption | Every read + weekly |
| Self-Healing | Auto-rebuild from surviving data | Any detected degradation | Immediate |
| Cross-Region Rep | Async copy to 2+ regions | Region-level disaster | Continuous |
| Hardware Scrubbing | Drive-level surface scan | Impending disk failure | Daily |
| Memory Scrubbing | ECC memory + periodic test | Memory bit flips | Continuous |
| Network Verification | End-to-end checksum on transfer | Network corruption | Every transfer |
Failure Rate Analysis:
Single HDD failure rate: ~2% per year (enterprise drives)
Single node failure rate: ~5% per year (any component)
Single AZ failure rate: ~0.1% per year
Single region failure rate: ~0.01% per year
With 3x replication across 3 AZs:
P(all 3 copies fail) = P(AZ1 fails) × P(AZ2 fails) × P(AZ3 fails)
= 0.001 × 0.001 × 0.001 = 10^-9
= 0.0000001% per year per object
With additional cross-region replication:
P(all replicas fail) = 10^-9 × 10^-9 = 10^-18
= Far exceeds 11 nines requirement
The 11 nines target is achievable through layered defense.
Curveball 2: “A storage node fails — how do we detect, recover, and prevent data loss?”
Detection (Multi-Layer):
graph TD
subgraph Detection["Detection Layers"]
D1[Heartbeat Monitor<br/>30s interval]
D2[Peer Health Check<br/>60s interval]
D3[Read Failure Detection<br/>Immediate]
D4[Checksum Mismatch<br/>On read]
D5[SMART Disk Alerts<br/>Proactive]
end
subgraph Response["Response Pipeline"]
ISOLATE[Isolate Failed Node<br/>Remove from pool]
ASSESS[Assess Data Impact<br/>Which objects affected?]
REBUILD[Rebuild Missing Data<br/>From surviving replicas/chunks]
VERIFY[Verify Rebuilt Data<br/>Checksum validation]
REPLACE[Replace Hardware<br/>New node joins pool]
end
D1 --> ISOLATE
D2 --> ISOLATE
D3 --> ISOLATE
D4 --> ISOLATE
D5 --> ISOLATE
ISOLATE --> ASSESS
ASSESS --> REBUILD
REBUILD --> VERIFY
VERIFY --> REPLACE
Recovery Process:
Step 1: Detection (0-30 seconds)
├── Heartbeat misses (3 consecutive = 90s)
├── Peer nodes report unreachable
├── Read requests to node timeout
└── Alert fired to operations team
Step 2: Isolation (30-60 seconds)
├── Remove node from consistent hash ring
├── Redirect new writes away from node
├── Mark node as "DEGRADED" in metadata
└── Client requests transparently routed to healthy nodes
Step 3: Impact Assessment (1-5 minutes)
├── Query metadata: which objects have chunks on this node?
├── For replicated objects: Check other 2 copies are healthy
├── For erasure-coded objects: Check 12 of 15 chunks survive
└── Calculate rebuild priority (most accessed objects first)
Step 4: Data Rebuild (5-60 minutes)
├── For 3x replication:
│ ├── Copy from surviving replica to new location
│ ├── Verify checksum matches
│ └── Update metadata with new chunk locations
│
├── For erasure coding:
│ ├── Read 12 surviving chunks
│ ├── Reconstruct missing chunks via Reed-Solomon
│ ├── Write reconstructed chunks to new nodes
│ └── Update metadata
│
└── Rebuild bandwidth throttled to 1Gbps
└── Prevent impact on production traffic
Step 5: Verification (continuous)
├── Checksum verify all rebuilt data
├── Read test on rebuilt objects
├── Update node health status
└── Clear alerts when fully recovered
Step 6: Hardware Replacement (1-24 hours)
├── Physical node replaced
├── New node commissioned
├── Joins storage pool
└── Rebalancing distributes data evenly
RPO/RTO Analysis:
| Metric | Target | Actual |
|---|---|---|
| Detection time | < 60s | ~30s (heartbeat) |
| Isolation time | < 60s | ~30s |
| Rebuild time | < 1 hour | ~30 min (typical) |
| RPO | 0 (no data loss) | 0 (redundancy ensures this) |
| RTO | < 5 min (client impact) | ~1 min (transparent failover) |
Curveball 3: “How do we handle a complete region failure (e.g., natural disaster destroys us-east-1)?”
Scenario: Hurricane destroys all infrastructure in us-east-1.
Impact Assessment:
us-east-1 contains:
├── 30% of global S3 data
├── 100M+ objects affected
├── 5M+ active users affected
└── Estimated data: 6 EB
Without preparation: Complete data loss for non-replicated objects
With preparation: Minimal disruption, automatic failover
Disaster Recovery Strategy:
graph TD
subgraph Before["Before Disaster"]
PRIMARY[Primary Region: us-east-1]
REPLICATOR[Async Replicator]
SECONDARY[Secondary Region: us-west-2]
REPLICATOR -->|Continuous sync| SECONDARY
end
subgraph During["During Disaster"]
PRIMARY_DOWN[us-east-1: DOWN]
DNS_FAIL[DNS Failover]
SECONDARY_ACTIVE[us-west-2: ACTIVE]
READERS[Readers routed to us-west-2]
end
subgraph After["After Disaster"]
PROMOTE[Promote us-west-2 to primary]
NEW_REGION[Stand up new region: us-east-1-replacement]
REPLICATE_BACK[Replicate from us-west-2 to new region]
RESTORE[Full service restored]
end
PRIMARY_DOWN --> DNS_FAIL
DNS_FAIL --> SECONDARY_ACTIVE
SECONDARY_ACTIVE --> READERS
READERS --> PROMOTE
PROMOTE --> NEW_REGION
NEW_REGION --> REPLICATE_BACK
REPLICATE_BACK --> RESTORE
Detailed DR Plan:
| Phase | Action | Timeline | Responsible |
|---|---|---|---|
| Detection | Region health check fails | 0-5 min | Automated |
| Alert | Page on-call team, notify customers | 5-10 min | Automated + Human |
| DNS Failover | Route traffic to secondary region | 10-15 min | Automated (Route 53) |
| Promote Secondary | Make secondary the new primary | 15-30 min | Automated |
| Customer Communication | Status page updates, emails | 15-60 min | Human |
| New Region Standup | Provision new infrastructure | 1-24 hours | Automated + Human |
| Data Replication | Replicate from promoted region | 24-72 hours | Automated |
| Full Recovery | All services at pre-disaster level | 3-7 days | Team |
Pre-Disaster Preparation:
Mandatory DR Measures:
├── 1. Cross-region replication enabled for all critical buckets
│ └── CRR configured with automatic failover
│
├── 2. DNS failover configured
│ └── Route 53 health checks + failover routing
│
├── 3. Secondary region always warm
│ └── Infrastructure pre-provisioned, ready to scale
│
├── 4. Regular DR drills
│ └── Quarterly failover tests
│
├── 5. Data consistency verification
│ └── Daily checksum comparison between regions
│
└── 6. Customer communication templates
└── Pre-approved status messages
Data Loss Risk:
With async replication:
├── Replication lag: Typically < 1 minute
├── Worst case lag: Up to 5 minutes (during high load)
├── RPO: Up to 5 minutes of data loss
└── Mitigation: Critical buckets use synchronous replication
Without replication:
├── RPO: Complete data loss
└── Mitigation: Enable CRR for all buckets (default recommendation)
Curveball 4: “How do we prevent and detect data corruption (bit rot)?”
The Problem:
Bit rot (silent data corruption):
├── HDD: ~1 in 10^15 bits flipped per read (unrecovered BER)
├── SSD: ~1 in 10^16 bits flipped per read
├── Memory: ~1 in 10^17 bits flipped per access (with ECC)
├── Network: ~1 in 10^8 bits flipped per packet (TCP catches most)
│
With 20 EB of data:
├── 20 EB = 1.6 × 10^20 bits
├── Expected bit flips per full scan: 1.6 × 10^20 / 10^15 = 160,000
└── Without detection: Silent corruption accumulates
Multi-Layer Protection:
graph TD
subgraph Prevention["Prevention"]
ECC[ECC Memory]
TCP[TCP Checksums]
ENTERPRISE[Enterprise-grade drives]
RAID[RAID / Erasure Coding parity]
end
subgraph Detection["Detection"]
WRITE_CHECK[Write-time checksum<br/>CRC32C + SHA256]
READ_CHECK[Read-time checksum verification]
BG_SCAN[Background scan<br/>Weekly full scan]
ON_DEMAND[On-demand verification]
end
subgraph Repair["Repair"]
AUTO_REPAIR[Auto-repair from replica]
EC_REBUILD[Rebuild from parity]
CLIENT_NOTIFY[Notify client on unrepairable]
end
ECC --> WRITE_CHECK
TCP --> WRITE_CHECK
ENTERPRISE --> WRITE_CHECK
RAID --> WRITE_CHECK
WRITE_CHECK --> READ_CHECK
READ_CHECK --> BG_SCAN
BG_SCAN --> ON_DEMAND
READ_CHECK -->|Mismatch| AUTO_REPAIR
BG_SCAN -->|Mismatch| AUTO_REPAIR
ON_DEMAND -->|Mismatch| AUTO_REPAIR
AUTO_REPAIR -->|No replica| EC_REBUILD
EC_REBUILD -->|No parity| CLIENT_NOTIFY
Checksum Strategy:
| Checksum | Purpose | Algorithm | When Computed |
|---|---|---|---|
| CRC32C | Fast verification | CRC32C (Castagnoli) | Every read/write |
| SHA-256 | Strong integrity | SHA-256 | Write time, stored in metadata |
| MD5 | Backward compatibility | MD5 (ETag) | Write time |
| Per-chunk CRC | Chunk-level verification | CRC32C | Every chunk read/write |
Background Scan Process:
Weekly Full Scan:
├── Scope: All objects in the region
├── Rate: 1% of objects per day (full scan in ~100 days)
├── Priority: Recently accessed objects scanned more frequently
├── Process:
│ 1. Read object data from storage
│ 2. Compute CRC32C and SHA-256
│ 3. Compare with stored checksums
│ 4. If mismatch:
│ a. Try reading from replica
│ b. If replica matches: Replace corrupted copy
│ c. If all copies corrupted: Rebuild from erasure coding
│ d. If unrecoverable: Flag for manual review
│ 5. Update last_verified timestamp
│ 6. Log result for audit
│
└── Metrics Tracked:
├── Objects scanned per day
├── Corruptions detected
├── Auto-repairs completed
└── Unrecoverable corruptions
Curveball 5: “How do we handle the ‘thundering herd’ problem when a popular object goes viral?”
Scenario: A news organization uploads a breaking news video to S3. Within minutes, millions of users try to download it simultaneously.
Problem Analysis:
Normal object: 100 GETs/sec
Viral object: 1,000,000 GETs/sec
Bottlenecks:
├── Single storage node: ~10,000 GETs/sec max
├── Single metadata entry: Lock contention at high concurrency
├── Network bandwidth: 10 Gbps = ~1.25 GB/sec
├── With 500 KB avg response: ~2,500 GETs/sec per 10 Gbps link
└── Without mitigation: 400x overload
Multi-Layer Mitigation:
graph TD
subgraph Layer1["Layer 1: CDN Caching"]
CF[CloudFront Edge Cache]
note1["Cache at 200+ edge locations"]
HIT_RATE["Hit rate: >99% for viral content"]
end
subgraph Layer2["Layer 2: Read Replicas"]
REPL1[Read Replica 1]
REPL2[Read Replica 2]
REPL3[Read Replica N]
note2["Auto-create replicas under load"]
end
subgraph Layer3["Layer 3: Metadata Caching"]
META_CACHE[Metadata in Redis]
note3["Avoid metadata DB hit on every GET"]
end
subgraph Layer4["Layer 4: Rate Limiting"]
RL[Per-key rate limiting]
note4["Fair queuing, not hard cutoff"]
end
subgraph Layer5["Layer 5: Storage Replication"]
SR[Additional storage replicas]
note5["Temporarily increase replication factor"]
end
CF -->|Cache miss| REPL1
CF -->|Cache miss| REPL2
CF -->|Cache miss| REPL3
REPL1 --> META_CACHE
REPL2 --> META_CACHE
REPL3 --> META_CACHE
META_CACHE -->|Miss| RL
RL --> SR
Detailed Strategy:
| Layer | Mechanism | Capacity Increase | Latency Impact |
|---|---|---|---|
| CDN Cache | Edge cache at 200+ locations | 100x+ | Lower (edge closer) |
| Read Replicas | Auto-scale read replicas | 10x | Same |
| Metadata Cache | Redis cache for metadata | 50x | Lower |
| Rate Limiting | Fair queuing per key | N/A | Slight increase |
| Storage Replicas | Temp 3x → 10x replication | 3x | Same |
Auto-Scaling Trigger:
Detection:
├── Monitor per-key GET rate
├── Threshold: > 10,000 GETs/sec for 60 seconds
└── Classification: "Hot object"
Actions (automatic):
├── 1. Enable CDN caching (if not already)
│ └── Cache-Control: public, max-age=300
│
├── 2. Create read replicas
│ └── Scale to 10 replicas across AZs
│
├── 3. Promote to metadata cache
│ └── Pin in Redis with extended TTL
│
├── 4. Increase storage replication
│ └── Temporarily increase from 3x to 10x
│
└── 5. Monitor and auto-degrade
└── After 24 hours of normal traffic, scale back
Cost Consideration:
Viral object costs:
├── CDN egress: $0.085/GB (first 10 TB)
├── S3 GET requests: $0.0004/1,000
├── Additional replicas: Storage cost × 7 extra copies
└── Total for 1 PB viral download: ~$85,000
Customer pays for what they use — this is the cloud model.
10. Failure Modes and Mitigations
10.1 Failure Mode Matrix
| Failure Mode | Probability | Impact | Detection | Mitigation | RTO | RPO |
|---|---|---|---|---|---|---|
| Single disk failure | High (daily) | Low | SMART + heartbeat | Erasure coding / 3x replica, auto-rebuild | 5min | 0 |
| Single node failure | Medium (hourly) | Low | Heartbeat timeout | Redistribute chunks, auto-rebuild | 1min | |
| AZ failure | Low (monthly) | Medium | Health check failure | 3x across AZs, failover | 5min | 0 |
| Region failure | Very Low (yearly) | High | Region health check | Cross-region replication | 15min | 5min |
| Metadata corruption | Very Low | Critical | Checksum mismatch | Redundant metadata, PITR | 15min | 5min |
| Network partition | Low | High | Latency spike | Quorum-based writes, read repair | 30s | 0 |
| Massive DDoS | Medium | High | Traffic anomaly | Rate limiting, WAF, auto-scale | 1min | 0 |
| Encryption key loss | Very Low | Critical | Key access failure | KMS key rotation, backup keys | 1hr | 0 |
| Erasure coding bug | Very Low | Critical | Checksum failure | Fallback to replication, manual rebuild | 1hr | 0 |
| Clock skew | Low | Medium | Timestamp anomaly | NTP sync, logical clocks | N/A | 0 |
| Silent bit rot | Medium (cumulative) | Medium | Background scan | CRC32C + SHA256 verification | N/A | 0 |
| Cascading failure | Low | Critical | Multi-service alerts | Circuit breakers, bulkheads | 5min | 0 |
10.2 Detailed Mitigation Strategies
Single Disk Failure (Most Common)
Frequency: ~1 disk failure per 1,000 disks per month
With 12M disks: ~12,000 disk failures per month
Detection:
├── SMART alerts (predictive, hours before failure)
├── Read error rate spike
├── Checksum mismatch on read
└── Heartbeat from disk controller
Mitigation:
├── Immediate: Mark disk as failed, stop writes
├── Data recovery:
│ ├── For 3x replication: Read from other 2 copies
│ └── For erasure coding: Reconstruct from parity
├── Rebuild: Write recovered data to replacement disk
├── Rebalance: Redistribute data to maintain even load
└── Replace: Physical disk replacement (hot-swap)
Impact on users: None (transparent)
Network Partition (Split-Brain)
graph TD
subgraph Before["Before Partition"]
N1[Node Group A] <-->|Sync| N2[Node Group B]
end
subgraph During["During Partition"]
N1_ISOLATED[Node Group A<br/>Independent]
N2_ISOLATED[Node Group B<br/>Independent]
QUORUM_A{Quorum?}
QUORUM_B{Quorum?}
end
subgraph After["After Healing"]
MERGE[Merge Divergent State]
N1_HEALED[Node Group A]
N2_HEALED[Node Group B]
end
N1_ISOLATED --> QUORUM_A
N2_ISOLATED --> QUORUM_B
QUORUM_A -->|Yes: Continue writes| N1_ISOLATED
QUORUM_A -->|No: Read-only| N1_ISOLATED
QUORUM_B -->|Yes: Continue writes| N2_ISOLATED
QUORUM_B -->|No: Read-only| N2_ISOLATED
N1_HEALED --> MERGE
N2_HEALED --> MERGE
Quorum-Based Write Protocol:
Write Protocol (N=3, W=2, R=2):
├── N = Total replicas (3 AZs)
├── W = Write quorum (2 AZs must ack)
├── R = Read quorum (2 AZs must respond)
└── Guarantee: W + R > N → 2 + 2 > 3 → Always overlap
Write flow:
1. Client sends PUT to any AZ
2. Primary AZ writes locally
3. Primary AZ replicates to 2 other AZs
4. Wait for W=2 acks (including primary)
5. Return success to client
6. Async replicate to remaining AZ
Read flow:
1. Client sends GET to any AZ
2. Read from R=2 AZs
3. Compare versions (using version vector)
4. Return latest version
5. Read repair: Update stale AZs async
During partition:
├── If partition leaves ≥ W nodes: Writes continue
├── If partition leaves < W nodes: Writes blocked (safe)
└── Reads may return stale data if R nodes are partitioned
Cascading Failure Prevention
Circuit Breaker Pattern:
├── Each service has circuit breakers for dependencies
├── States: CLOSED → OPEN → HALF-OPEN → CLOSED
├── Trip threshold: 50% failure rate over 60s window
├── Reset timeout: 30 seconds
└── Half-open: Allow 1 request through, success → CLOSED
Bulkhead Pattern:
├── Isolate failure domains
├── Separate thread pools per service
├── Separate connection pools per dependency
└── Limit blast radius of any single failure
Rate Limiting:
├── Per-account rate limits
├── Per-key rate limits
├── Global rate limits
└── Adaptive limits based on system load
Graceful Degradation:
├── Non-critical features disabled under load
├── Reduced metadata returned in responses
├── Slower storage classes deprioritized
└── Event notifications queued, not dropped
10.3 Monitoring and Alerting
Key Metrics:
├── Storage Health
│ ├── Total capacity used (%)
│ ├── Disk failure rate (per 1,000 disks/month)
│ ├── Rebuild queue depth
│ ├── Erasure coding rebuild time
│ └── Checksum mismatch rate
│
├── Request Performance
│ ├── GET/PUT/DELETE latency (P50/P95/P99)
│ ├── Request success rate (%)
│ ├── Error rate by error code
│ └── Throttled request rate
│
├── Data Integrity
│ ├── Background scan progress
│ ├── Corruptions detected/day
│ ├── Auto-repairs completed/day
│ └── Unrecoverable corruptions
│
├── Replication
│ ├── Replication lag (seconds)
│ ├── Replication success rate
│ └── Failed replication tasks
│
├── Infrastructure
│ ├── Node health score
│ ├── Network bandwidth utilization
│ ├── CPU/Memory utilization
│ └── Disk I/O utilization
│
└── Business
├── Total objects count
├── Total storage used
├── Active buckets
└── Revenue metrics
Alert Thresholds:
├── Critical: P99 latency > 500ms for 5 min
├── Critical: Disk failure rate > 10x baseline
├── Critical: Checksum mismatch rate > 0.01%
├── Critical: Replication lag > 1 hour
├── Warning: P95 latency > 200ms for 10 min
├── Warning: Rebuild queue > 1,000 objects
├── Warning: Storage capacity > 80%
└── Info: New disk failure detected
10.4 Compliance and Audit
Compliance Requirements:
├── SOC 2 Type II: Annual audit
├── HIPAA: Healthcare data protection
├── PCI DSS: Payment card data
├── FedRAMP: Government workloads
├── ISO 27001: Information security
└── GDPR: Data residency and right to erasure
Audit Capabilities:
├── CloudTrail: All API calls logged
├── S3 Server Access Logging: All requests logged
├── S3 Object Lambda: Custom processing with audit
├── VPC Flow Logs: Network-level audit
└── Config: Resource configuration history
Data Residency:
├── Region-specific endpoints
├── Data never leaves chosen region (unless replicated)
├── Cross-region replication is opt-in
└── Government Cloud for classified data
Appendix A: Glossary
| Term | Definition |
|---|---|
| Object | A file and its metadata stored in S3 |
| Bucket | A container for objects; globally unique namespace |
| Key | The full path/name of an object within a bucket |
| ETag | Entity tag; MD5 hash of object content (for non-multipart) |
| Multipart Upload | Uploading large objects in parallel parts |
| Erasure Coding | Data protection using parity chunks instead of full replicas |
| CRC32C | Cyclic Redundancy Check using Castagnoli polynomial |
| BER | Bit Error Rate; probability of bit flip during storage/read |
| RTO | Recovery Time Objective; max acceptable downtime |
| RPO | Recovery Point Objective; max acceptable data loss |
| CRR | Cross-Region Replication |
| SRR | Same-Region Replication |
| SSE | Server-Side Encryption |
| KMS | Key Management Service |
| WORM | Write Once Read Many (Object Lock compliance mode) |
| Quorum | Minimum number of replicas that must agree for read/write |
| Read Repair | Consistency mechanism that repairs stale replicas on read |
| Bit Rot | Silent data corruption over time |
Appendix B: References
- “Designing Data-Intensive Applications” by Martin Kleppmann
- Amazon S3 Documentation: https://docs.aws.amazon.com/AmazonS3/
- “The Amazon S3 Storage System” — AWS re:Invent presentations
- Dynamo paper (Amazon): https://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf
- Reed-Solomon Erasure Coding: https://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction
- CAP Theorem: Brewer’s conjecture on distributed systems
Document Version: 1.0 | Last Updated: 2026-08-28