Collaborative StorageQuestion #16ADVANCED

Amazon S3: Massive-Scale Object Storage & 11 Nines Durability

Exabyte-scale blob storage infrastructure featuring Reed-Solomon erasure coding, LSM-tree metadata partition indexing, bit-rot background scrubbing, and automated lifecycle migration.

Category: Category 10: Collaborative File Storage•Updated: 2026-08-30
#File Sharing#Object Storage#11 Nines#Erasure Coding#Lifecycle

Design a File Management System like Amazon S3

Author: System Design Expert | Date: 2026-08-28 Scope: Object storage service with 11 nines durability at planetary scale


Table of Contents

  1. Requirements Clarification
  2. Back-of-Envelope Calculations
  3. System Interface Design
  4. High-Level Architecture
  5. Core Component Design
  6. Data Model
  7. Key Design Decisions
  8. Scalability Strategy
  9. Curveball Scenarios
  10. Failure Modes and Mitigations

1. Requirements Clarification

1.1 Functional Requirements

ID Requirement Priority
FR-01 Object storage — Store and retrieve objects (files) of any type up to 5TB each Must Have
FR-02 Bucket organization — Namespace objects within buckets; buckets are globally unique Must Have
FR-03 Multipart upload — Upload large objects in parallel parts; assemble server-side Must Have
FR-04 Versioning — Maintain multiple versions of an object; enable restore Must Have
FR-05 Lifecycle policies — Auto-transition objects between storage classes; auto-expire Must Have
FR-06 Replication — Cross-region and cross-account replication for durability and compliance Must Have
FR-07 Access control — Fine-grained IAM policies, bucket policies, ACLs, presigned URLs Must Have
FR-08 Event notifications — Trigger Lambda/SNS/SQS on object events (create, delete, etc.) Must Have
FR-09 Server-side encryption — SSE-S3, SSE-KMS, SSE-C encryption options Must Have
FR-10 Storage classes — Standard, Intelligent-Tiering, Standard-IA, Glacier, Glacier Deep Archive Must Have
FR-11 Object tagging — Key-value metadata tags for organization and lifecycle Should Have
FR-12 Inventory & metrics — Daily/weekly inventory reports, storage metrics Should Have
FR-13 Object Lock — WORM (Write Once Read Many) compliance mode Should Have
FR-14 Select/Query — SQL-like queries on objects in-place (CSV, JSON, Parquet) Should Have
FR-15 Website hosting — Static website hosting with custom domains Could Have
FR-16 Transfer acceleration — Optimized transfers via edge locations Could Have

1.2 Non-Functional Requirements

ID Requirement Target
NFR-01 Durability 99.999999999% (11 nines) — lose 0.000000001% of objects/year
NFR-02 Availability (Standard) 99.99% (4 nines) — 52 min downtime/year
NFR-03 Availability (Single AZ) 99.999999999% (11 nines) for Read-After-Write within AZ
NFR-04 Throughput Unlimited aggregate; burst to 50,000 PUTs/sec and 5,500 GETs/sec per prefix
NFR-05 Scalability Trillions of objects, exabytes of data, millions of requests/sec
NFR-06 Consistency Strong consistency for all operations (as of 2020+)
NFR-07 Latency P99 < 100ms for GET within same region
NFR-08 Global scale 30+ regions, 100+ availability zones
NFR-09 Security AES-256 encryption, TLS 1.3, SOC 2, HIPAA, PCI DSS compliant
NFR-10 Cost efficiency Pay-per-use pricing; cheaper than on-premises by 30%+

2. Back-of-Envelope Calculations

2.1 Assumption Sets

Parameter Conservative Base Case Aggressive
Total stored data 5 EB 20 EB 100 EB
Total objects 500T 2T 10T
Avg object size 10 MB 10 MB 10 MB
Requests/day (GET) 100B 500B 2T
Requests/day (PUT) 10B 50B 200B
Requests/day (other) 5B 25B 100B
Total requests/day 115B 575B 2.3T
Buckets 10B 50B 200B
Active users 5M 20M 100M

2.2 Storage Estimates

Base Case — Raw Data:
├── User data:             20 EB (1 EB = 1,000 PB = 1,000,000 TB)
│
├── Replication factor:    3x (within region, across AZs)
│   └── 20 EB × 3 = 60 EB
│
├── Cross-region replicas: 2x (avg 2 replica regions per bucket)
│   └── 20 EB × 2 = 40 EB
│
├── Metadata per object:   1 KB avg
│   └── 2T objects × 1 KB = 2 PB
│
├── Metadata replication:  3x
│   └── 2 PB × 3 = 6 PB
│
├── Versioning overhead:   2x (avg 2 versions per object)
│   └── 60 EB × 2 = 120 EB
│
├── Erasure coding parity: 0.4x (for Glacier classes)
│   └── 20 EB × 0.4 = 8 EB
│
└── Total physical storage: ~234 EB

2.3 Bandwidth Estimates

Base Case — Request Traffic:
├── Total requests/day:    575B
├── Avg GET size:          500 KB
├── Avg PUT size:          1 MB
├── Avg other size:        1 KB
│
├── GET bandwidth/day:     400B × 500 KB = 200 EB/day
├── PUT bandwidth/day:     50B × 1 MB = 50 EB/day
├── Other bandwidth/day:   25B × 1 KB = 25 PB/day
│
├── Total bandwidth/day:   ~250 EB/day
├── Avg bandwidth/sec:     250 EB / 86,400s ≈ 2.9 PB/sec
├── Peak bandwidth (3x):   ~8.7 PB/sec
│
└── Request QPS:
    ├── Avg QPS:           575B / 86,400s ≈ 6.65M QPS
    ├── Peak QPS:          ~20M QPS
    ├── GET QPS:           ~4.6M avg, ~14M peak
    └── PUT QPS:           ~580K avg, ~1.7M peak

2.4 Infrastructure Estimates

Storage Nodes:
├── Per node capacity:     100 TB usable (10 × 10 TB drives)
├── Nodes for user data:   20 EB / 100 TB = 200,000 nodes
├── Nodes with 3x replica: 600,000 nodes
└── With versioning:       ~1.2M nodes

Metadata Servers:
├── Per server capacity:   100M object metadata entries
├── Servers needed:        2T / 100M = 20,000 servers
└── With replication:      60,000 servers

Frontend/Proxy Servers:
├── Per server capacity:   10,000 req/sec
├── Servers needed:        20M QPS / 10K = 2,000 servers
└── With redundancy:       6,000 servers

Total Estimated Infrastructure:
├── Storage nodes:         ~1.2M
├── Metadata servers:      ~60K
├── Frontend servers:      ~6K
├── Management services:   ~10K
└── Total:                 ~1.27M servers

3. System Interface Design

3.1 REST API

# Bucket Operations
PUT    /                              # CreateBucket
DELETE /                              # DeleteBucket
GET    /?location                     # GetBucketLocation
GET    /?versioning                   # GetBucketVersioning
PUT    /?versioning                   # PutBucketVersioning
GET    /?lifecycle                    # GetBucketLifecycle
PUT    /?lifecycle                    # PutBucketLifecycle
GET    /?policy                       # GetBucketPolicy
PUT    /?policy                       # PutBucketPolicy
DELETE /?policy                       # DeleteBucketPolicy
GET    /?replication                  # GetBucketReplication
PUT    /?replication                  # PutBucketReplication
GET    /?encryption                   # GetBucketEncryption
PUT    /?encryption                   # PutBucketEncryption
GET    /?tagging                      # GetBucketTagging
PUT    /?tagging                      # PutBucketTagging

# Object Operations
GET    /{key}                         # GetObject
PUT    /{key}                         # PutObject
DELETE /{key}                         # DeleteObject
HEAD   /{key}                         # HeadObject
GET    /{key}?torrent                 # GetObjectTorrent
GET    /{key}?acl                     # GetObjectAcl
PUT    /{key}?acl                     # PutObjectAcl
GET    /{key}?tagging                 # GetObjectTagging
PUT    /{key}?tagging                 # PutObjectTagging
POST   /{key}?restore                # RestoreObject
POST   /{key}?select                 # SelectObjectContent
POST   /{key}?job                    # InitiateJob

# Multipart Upload
POST   /{key}?uploads                 # CreateMultipartUpload
PUT    /{key}?uploadId={id}&partNumber={n}  # UploadPart
POST   /{key}?uploadId={id}           # CompleteMultipartUpload
POST   /{key}?uploadId={id}           # AbortMultipartUpload
GET    /?uploads                      # ListMultipartUploads
GET    /{key}?uploadId={id}           # ListParts

# Listing
GET    /?list-type=2                  # ListObjectsV2
GET    /?list-type=2&prefix={p}       # ListObjects with prefix
GET    /?list-type=2&delimiter={d}    # ListObjects with delimiter
GET    /?list-type=2&max-keys={n}     # ListObjects with limit

# Versioning
GET    /{key}?versionId={id}          # Get specific version
DELETE /{key}?versionId={id}          # Delete specific version
GET    /?versions                     # ListObjectVersions

# Presigned URL
GET    /{key}?X-Amz-Algorithm=AWS4-HMAC-SHA256
       &X-Amz-Credential={cred}
       &X-Amz-Date={date}
       &X-Amz-Expires={seconds}
       &X-Amz-SignedHeaders={headers}
       &X-Amz-Signature={sig}

3.2 API Request/Response Examples

# CreateBucket
PUT / HTTP/1.1
Host: my-bucket.s3.us-east-1.amazonaws.com
x-amz-bucket-region: us-east-1
x-amz-date: 20260828T120000Z
Authorization: AWS4-HMAC-SHA256 ...

<CreateBucketConfiguration>
    <LocationConstraint>us-west-2</LocationConstraint>
</CreateBucketConfiguration>

# Response: 200 OK

---

# PutObject
PUT /photos/vacation.jpg HTTP/1.1
Host: my-bucket.s3.us-east-1.amazonaws.com
Content-Type: image/jpeg
Content-Length: 5242880
x-amz-acl: private
x-amz-server-side-encryption: AES256
x-amz-storage-class: STANDARD
x-amz-metadata-user-id: 12345
x-amz-tagging: project=vacation,year=2026
x-amz-date: 20260828T120000Z
Authorization: AWS4-HMAC-SHA256 ...

<binary data>

# Response: 200 OK
ETag: "d41d8cd98f00b204e9800998ecf8427e"
x-amz-version-id: OTh46nT2EFtbn0VqCiQ6MiHm
x-amz-server-side-encryption: AES256
x-amz-storage-class: STANDARD

---

# GetObject
GET /photos/vacation.jpg HTTP/1.1
Host: my-bucket.s3.us-east-1.amazonaws.com
Range: bytes=0-1048575
x-amz-date: 20260828T120000Z
Authorization: AWS4-HMAC-SHA256 ...

# Response: 206 Partial Content
Content-Range: bytes 0-1048575/5242880
Content-Type: image/jpeg
ETag: "d41d8cd98f00b204e9800998ecf8427e"
x-amz-version-id: OTh46nT2EFtbn0VqCiQ6MiHm
Last-Modified: Fri, 28 Aug 2026 12:00:00 GMT

<binary data>

---

# CreateMultipartUpload
POST /large-video.mp4?uploads HTTP/1.1
Host: my-bucket.s3.us-east-1.amazonaws.com
x-amz-storage-class: STANDARD
x-amz-server-side-encryption: aws:kms
x-amz-server-side-encryption-aws-kms-key-id: arn:aws:kms:...

# Response: 200 OK
<InitiateMultipartUploadResult>
    <Bucket>my-bucket</Bucket>
    <Key>large-video.mp4</Key>
    <UploadId>VXDuQLA8xe2oEdFBzLOAi9DkJ</UploadId>
</InitiateMultipartUploadResult>

---

# UploadPart
PUT /large-video.mp4?uploadId=VXDuQLA8xe2oEdFBzLOAi9DkJ&partNumber=1
Host: my-bucket.s3.us-east-1.amazonaws.com
Content-Length: 10485760  # 10 MB

<binary part data>

# Response: 200 OK
ETag: "a54783c9c3a6e39d9b9a8d3e1f2c4b5a"

---

# CompleteMultipartUpload
POST /large-video.mp4?uploadId=VXDuQLA8xe2oEdFBzLOAi9DkJ
Host: my-bucket.s3.us-east-1.amazonaws.com

<CompleteMultipartUpload>
    <Part>
        <PartNumber>1</PartNumber>
        <ETag>"a54783c9c3a6e39d9b9a8d3e1f2c4b5a"</ETag>
    </Part>
    <Part>
        <PartNumber>2</PartNumber>
        <ETag>"b65894d0d4b7f40e0c0b9e4f2g3d5c6b"</ETag>
    </Part>
</CompleteMultipartUpload>

# Response: 200 OK
<CompleteMultipartUploadResult>
    <Location>https://my-bucket.s3.us-east-1.amazonaws.com/large-video.mp4</Location>
    <Bucket>my-bucket</Bucket>
    <Key>large-video.mp4</Key>
    <ETag>"d41d8cd98f00b204e9800998ecf8427e-2"</ETag>
    <VersionId>OTh46nT2EFtbn0VqCiQ6MiHm</VersionId>
</CompleteMultipartUploadResult>

3.3 SDK Interface (Conceptual)

class S3Client:
    # Bucket operations
    def create_bucket(self, name: str, region: str,
                      versioning: bool = False,
                      encryption: EncryptionConfig = None) -> Bucket:
        ...

    def delete_bucket(self, name: str) -> None:
        ...

    def list_buckets(self) -> List[Bucket]:
        ...

    # Object operations
    def put_object(self, bucket: str, key: str, body: bytes | IOBase,
                   content_type: str = None,
                   storage_class: StorageClass = StorageClass.STANDARD,
                   tags: Dict[str, str] = None,
                   metadata: Dict[str, str] = None,
                   server_side_encryption: str = None,
                   acl: str = None) -> Object:
        ...

    def get_object(self, bucket: str, key: str,
                   version_id: str = None,
                   range: str = None) -> Object:
        ...

    def delete_object(self, bucket: str, key: str,
                      version_id: str = None) -> None:
        ...

    # Multipart upload
    def upload_file(self, filename: str, bucket: str, key: str,
                    part_size: int = 8*1024*1024,  # 8 MB default
                    concurrency: int = 10,
                    storage_class: StorageClass = StorageClass.STANDARD) -> None:
        ...

    # Presigned URL
    def generate_presigned_url(self, bucket: str, key: str,
                               expiration: int = 3600,
                               http_method: str = 'GET') -> str:
        ...

    # Lifecycle
    def put_bucket_lifecycle(self, bucket: str,
                             rules: List[LifecycleRule]) -> None:
        ...

    # Replication
    def put_bucket_replication(self, bucket: str,
                               config: ReplicationConfig) -> None:
        ...

4. High-Level Architecture

graph TB
    subgraph Client["Client Layer"]
        APP[Applications / SDKs]
        CLI[AWS CLI / Console]
        PARTNER[Partner Integrations]
    end

    subgraph Edge["Edge Layer"]
        DNS[Route 53 / DNS]
        CDN[CloudFront / Edge Locations]
        WAF[WAF / Shield]
        LB[Global Load Balancer]
    end

    subgraph Frontend["Frontend Layer"]
        AUTH[Authentication Service<br/>STS / IAM]
        API_GW[API Frontend<br/>Request Routing]
        QUOTA[Quota & Rate Limiting]
        BILLING[Billing Metering]
    end

    subgraph Control["Control Plane"]
        BUCKET_MGR[Bucket Manager<br/>Namespace Service]
        POLICY[Policy Engine<br/>ACL / Bucket Policy]
        LIFECYCLE[Lifecycle Manager]
        REPLICATION[Replication Service]
        VERSION_MGR[Version Manager]
        ENCRYPTION[Encryption Service<br/>KMS Integration]
        EVENT[Event Notification<br/>SNS / SQS / Lambda]
        INVENTORY[Inventory & Analytics]
    end

    subgraph Data["Data Plane"]
        subgraph ObjectStorage["Object Storage Engine"]
            PARTITION[Partition Router<br/>Consistent Hash]
            subgraph AZ1["Availability Zone A"]
                STORAGE_A1[Storage Node A1]
                STORAGE_A2[Storage Node A2]
                STORAGE_A3[Storage Node A3]
            end
            subgraph AZ2["Availability Zone B"]
                STORAGE_B1[Storage Node B1]
                STORAGE_B2[Storage Node B2]
                STORAGE_B3[Storage Node B3]
            end
            subgraph AZ3["Availability Zone C"]
                STORAGE_C1[Storage Node C1]
                STORAGE_C2[Storage Node C2]
                STORAGE_C3[Storage Node C3]
            end
        end

        subgraph Metadata["Metadata Service"]
            META_DB[(Metadata DB<br/>DynamoDB-like)]
            META_CACHE[(Redis Cluster<br/>Hot Metadata)]
        end

        subgraph Multipart["Multipart Service"]
            MP_STATE[Multipart State Store]
            MP_ASSEMBLE[Part Assembler]
        end
    end

    subgraph Storage["Replication & Archive"]
        REPLICATOR[Async Replicator]
        ARCHIVE[Glacier Archive<br/>Cold Storage]
        DR_SITE[DR Region]
    end

    subgraph Infra["Infrastructure"]
        MQ[Kafka / Kinesis<br/>Event Stream]
        MONITOR[CloudWatch / Monitoring]
        LOG[Audit Logging<br/>CloudTrail]
    end

    APP --> DNS
    CLI --> DNS
    PARTNER --> DNS

    DNS --> CDN
    CDN --> WAF
    WAF --> LB

    LB --> AUTH
    LB --> API_GW

    API_GW --> QUOTA
    QUOTA --> BILLING
    QUOTA --> BUCKET_MGR
    QUOTA --> POLICY

    BUCKET_MGR --> META_DB
    POLICY --> META_DB

    PARTITION --> STORAGE_A1
    PARTITION --> STORAGE_B1
    PARTITION --> STORAGE_C1

    STORAGE_A1 --> META_DB
    STORAGE_B1 --> META_DB
    STORAGE_C1 --> META_DB

    API_GW --> PARTITION
    API_GW --> MP_STATE

    MP_STATE --> MP_ASSEMBLE
    MP_ASSEMBLE --> PARTITION

    LIFECYCLE --> ARCHIVE
    REPLICATION --> REPLICATOR
    REPLICATOR --> DR_SITE

    STORAGE_A1 -->|Events| MQ
    STORAGE_B1 -->|Events| MQ
    STORAGE_C1 -->|Events| MQ

    MQ --> EVENT
    MQ --> INVENTORY
    MQ --> LOG
    MQ --> REPLICATOR

    ENCRYPTION --> STORAGE_A1
    ENCRYPTION --> STORAGE_B1
    ENCRYPTION --> STORAGE_C1

    META_CACHE --> META_DB

5. Core Component Design

5.1 Object Storage Engine

The storage engine is the core of S3, responsible for storing, retrieving, and protecting object data across multiple availability zones.

Storage Node Architecture

graph TD
    subgraph StorageNode["Storage Node"]
        subgraph DiskLayer["Disk Layer"]
            D1[Disk 1: 10 TB]
            D2[Disk 2: 10 TB]
            D3[Disk 3: 10 TB]
            D4[Disk 4: 10 TB]
            D5[Disk 5: 10 TB]
            D6[Disk 6: 10 TB]
            D7[Disk 7: 10 TB]
            D8[Disk 8: 10 TB]
            D9[Disk 9: 10 TB]
            D10[Disk 10: 10 TB]
        end

        subgraph DataLayer["Data Layer"]
            EC[Erasure Coding Engine]
            CHECKSUM[Checksum Verification<br/>CRC32C + SHA256]
            COMPRESSION[Compression<br/>Zstandard / LZ4]
            ENCRYPTION[Encryption Engine<br/>AES-256-XTS]
        end

        subgraph CacheLayer["Cache Layer"]
            MEM_CACHE[Memory Cache<br/>Hot Object Blocks]
            SSD_CACHE[SSD Cache<br/>Warm Object Blocks]
        end

        subgraph ProtocolLayer["Protocol Layer"]
            READ_HANDLER[Read Handler]
            WRITE_HANDLER[Write Handler]
            REPAIR_HANDLER[Self-Healing Handler]
        end
    end

    READ_HANDLER --> MEM_CACHE
    MEM_CACHE -->|Miss| SSD_CACHE
    SSD_CACHE -->|Miss| EC
    EC --> D1
    EC --> D2
    EC --> D3

    WRITE_HANDLER --> ENCRYPTION
    ENCRYPTION --> COMPRESSION
    COMPRESSION --> CHECKSUM
    CHECKSUM --> EC

    REPAIR_HANDLER -->|Background| EC
    REPAIR_HANDLER -->|Verify| CHECKSUM

Data Placement Strategy

Object → Block → Chunk → Disk

1. Object received (e.g., 1 GB file)
2. Split into fixed-size blocks (4 MB each → 256 blocks)
3. Each block erasure-coded into data + parity chunks
   - Scheme: 12 data + 3 parity = 15 chunks (25% overhead)
   - Each chunk: 4 MB / 12 = 333 KB
4. Chunks distributed across disks and nodes:
   - Data chunks: Spread across 12 different nodes
   - Parity chunks: Spread across 3 different nodes
   - All 15 nodes in different AZs when possible
5. Metadata recorded: which node holds which chunk

Reconstruction:
├── Need any 12 of 15 chunks to reconstruct a block
├── Can tolerate 3 simultaneous node failures
└── Reconstruction triggered automatically on failure

Erasure Coding vs Replication

Factor 3x Replication Erasure Coding (12+3)
Storage overhead 200% (3x) 25% (1.25x)
Fault tolerance Lose 2 of 3 copies Lose 3 of 15 chunks
Read performance Read any copy (fast) Read 12 chunks (slower)
Write performance Write 3 copies Encode + write 15 chunks
Rebuild cost Copy full object Re-encode from 12 chunks
Best for Standard class Glacier / IA classes
Cost Higher ~60% cheaper

5.2 Metadata Service

Metadata is the most accessed component — every request touches metadata before touching data.

graph TD
    subgraph MetadataService["Metadata Service"]
        subgraph HotTier["Hot Tier (Redis Cluster)"]
            R1[Redis Shard 1]
            R2[Redis Shard 2]
            R3[Redis Shard N]
        end

        subgraph WarmTier["Warm Tier (In-Memory DB)"]
            W1[Memory Node 1]
            W2[Memory Node 2]
        end

        subgraph ColdTier["Cold Tier (DynamoDB-like)"]
            C1[DynamoDB Shard 1]
            C2[DynamoDB Shard 2]
        end
    end

    REQ[Request] --> R1
    REQ --> R2
    REQ --> R3

    R1 -->|Miss| W1
    R2 -->|Miss| W2
    W1 -->|Miss| C1
    W2 -->|Miss| C2

    C1 -->|Promote| W1
    W1 -->|Promote| R1

Metadata Stored Per Object

{
  "object_id": "obj-a1b2c3d4e5f6",
  "bucket_id": "bucket-xyz789",
  "key": "photos/vacation/2026/beach.jpg",
  "version_id": "v-20260828-120000-abc123",
  "size_bytes": 5242880,
  "content_type": "image/jpeg",
  "content_md5": "d41d8cd98f00b204e9800998ecf8427e",
  "etag": "\"d41d8cd98f00b204e9800998ecf8427e\"",
  "storage_class": "STANDARD",
  "encryption": {
    "algorithm": "AES256",
    "key_id": "arn:aws:kms:us-east-1:123456:key/abc-123"
  },
  "owner": {
    "id": "user-12345",
    "display_name": "John Doe"
  },
  "acl": "private",
  "tags": {
    "project": "vacation",
    "year": "2026"
  },
  "user_metadata": {
    "camera": "Canon EOS R5",
    "location": "Maldives"
  },
  "created_at": "2026-08-28T12:00:00Z",
  "last_modified": "2026-08-28T12:00:00Z",
  "replication_status": "COMPLETED",
  "lifecycle_status": "ACTIVE",
  "chunk_locations": [
    {"chunk_id": "chk-001", "node_id": "node-a1", "az": "us-east-1a"},
    {"chunk_id": "chk-002", "node_id": "node-b2", "az": "us-east-1b"},
    {"chunk_id": "chk-003", "node_id": "node-c3", "az": "us-east-1c"}
  ]
}

5.3 Multipart Upload Service

Handles large object uploads by splitting them into parallel parts.

sequenceDiagram
    participant C as Client
    participant FE as Frontend
    participant MP as Multipart Service
    participant STORE as Storage Engine

    C->>FE: CreateMultipartUpload(key, metadata)
    FE->>MP: Create upload session
    MP->>STORE: Reserve object ID
    MP->>MP: Generate UploadId
    MP->>C: Return UploadId

    par Parallel Part Uploads
        C->>FE: UploadPart(UploadId, partNumber=1, data)
        FE->>STORE: Store part 1
        STORE->>FE: ETag-1
        FE->>C: 200 OK (ETag-1)
    and
        C->>FE: UploadPart(UploadId, partNumber=2, data)
        FE->>STORE: Store part 2
        STORE->>FE: ETag-2
        FE->>C: 200 OK (ETag-2)
    and
        C->>FE: UploadPart(UploadId, partNumber=3, data)
        FE->>STORE: Store part 3
        STORE->>FE: ETag-3
        FE->>C: 200 OK (ETag-3)
    end

    C->>FE: CompleteMultipartUpload(UploadId, [parts])
    FE->>MP: Validate all parts present
    MP->>MP: Verify ETags match
    MP->>STORE: Assemble parts into object
    STORE->>STORE: Erasure-code assembled object
    STORE->>STORE: Replicate across AZs
    STORE->>MP: Object stored successfully
    MP->>C: 200 OK (ETag, VersionId)

    Note over C,STORE: Abandoned uploads auto-cleaned after 7 days

Multipart Upload Design Details

Parameter Value Rationale
Minimum part size 5 MB (except last part) Balance between overhead and parallelism
Maximum part size 5 GB Limit memory usage during assembly
Maximum parts 10,000 per upload Limit metadata overhead
Maximum object size 5 TB 10,000 parts × 5 GB
Part expiration 7 days (auto-abort) Clean up abandoned uploads
Concurrency Up to 10,000 parallel parts SDK-configurable
Part storage Temporary, separate from final object Avoid wasting space on incomplete uploads

5.4 Versioning Service

graph TD
    subgraph Versioning["Versioning Service"]
        PUT[PUT Object] --> CHECK{Versioning Enabled?}
        CHECK -->|Yes| VERSION[Create New Version]
        CHECK -->|No| OVERWRITE[Overwrite Current]

        VERSION --> CURRENT[Mark Previous as Old]
        VERSION --> NEW[Store New as Current]
        VERSION --> DELETE_MARKER{Same Key?}
        DELETE_MARKER -->|DELETE after PUT| DM[Create Delete Marker]
        DELETE_MARKER -->|Normal| NORMAL[Normal Version]

        OVERWRITE --> REPLACE[Replace Object]

        subgraph VersionChain["Version Chain"]
            V1[Version 1<br/>v-001<br/>Current]
            V2[Version 2<br/>v-002<br/>Current]
            V3[Version 3<br/>v-003<br/>Current]
            DM1[Delete Marker<br/>dm-001]
        end

        V1 --> V2 --> V3
        V3 --> DM1
    end

Version Lifecycle

Object: "report.pdf"

Timeline:
├── T1: PUT report.pdf → Version v-001 (current)
├── T2: PUT report.pdf → Version v-002 (current), v-001 (old)
├── T3: PUT report.pdf → Version v-003 (current), v-002 (old), v-001 (old)
├── T4: DELETE report.pdf → Delete Marker dm-001 (current)
│   └── Object appears deleted, but v-001, v-002, v-003 still exist
├── T5: PUT report.pdf → Version v-004 (current), dm-001 (old)
│   └── Object reappears, delete marker becomes old version
└── T6: DELETE report.pdf version=v-002 → Permanent deletion of v-002
    └── Only explicit version deletion permanently removes data

5.5 Lifecycle Management

graph TD
    subgraph LifecycleEngine["Lifecycle Engine"]
        SCHEDULE[Scheduled Scanner<br/>Every 15 minutes]
        EVAL[Evaluator<br/>Match rules]
        ACTIONS[Action Executor]

        subgraph Rules["Lifecycle Rules"]
            R1[Transition Rule<br/>30 days → Standard-IA]
            R2[Transition Rule<br/>90 days → Glacier]
            R3[Transition Rule<br/>180 days → Glacier Deep Archive]
            R4[Expiration Rule<br/>365 days → Delete]
            R5[Noncurrent Expiration<br/>7 versions → Delete old]
            R6[Abort Incomplete MPU<br/>7 days → Abort]
        end

        subgraph StorageClasses["Storage Classes"]
            STD[Standard<br/>Hot, 3x replica]
            STD_IA[Standard-IA<br/>Infrequent, 3x replica]
            ONE_ZA[One Zone-IA<br/>Single AZ, 3x replica]
            GLACIER[Glacier<br/>Archive, EC]
            GLACIER_DEEP[Glacier Deep Archive<br/>Deep archive, EC]
        end

        R1 --> STD_IA
        R2 --> GLACIER
        R3 --> GLACIER_DEEP
        R4 -->|Delete| X[Deleted]
        R5 -->|Delete old versions| X
    end

    SCHEDULE --> EVAL
    EVAL --> R1
    EVAL --> R2
    EVAL --> R3
    EVAL --> R4
    EVAL --> R5
    EVAL --> R6

    R1 --> ACTIONS
    R2 --> ACTIONS
    R3 --> ACTIONS
    R4 --> ACTIONS
    R5 --> ACTIONS
    R6 --> ACTIONS

Storage Class Comparison

Class Availability Retrieval Time Min Storage Min Object Size Relative Cost
Standard 99.99% Immediate None None 1.0x
Intelligent-Tiering 99.99% Immediate $0.0025/object/month None 1.0x + monitoring
Standard-IA 99.9% Immediate 30 days 128 KB 0.6x
One Zone-IA 99.5% Immediate 30 days 128 KB 0.48x
Glacier 99.999999999% 1-5 hours 90 days 128 KB 0.004x
Glacier Deep Archive 99.999999999% 12-48 hours 180 days 128 KB 0.00099x

5.6 Replication Service

graph TD
    subgraph SourceRegion["Source Region (us-east-1)"]
        SRC_BUCKET[Source Bucket]
        SRC_EVENT[Object Created Event]
        SRC_QUEUE[Replication Queue]
    end

    subgraph ReplicationService["Replication Service"]
        DETECT[Event Detector]
        FILTER[Rule Filter<br/>Prefix / Tag / Storage Class]
        TRANSFORM[Transform<br/>Storage Class / Encryption / Tags]
        TRANSFER[Data Transfer<br/>Secure Channel]
        VERIFY[Verification<br/>Checksum + Metadata]
        STATUS[Status Tracker]
    end

    subgraph DestRegion["Destination Region (eu-west-1)"]
        DST_BUCKET[Destination Bucket]
        DST_STORE[Storage Engine]
    end

    SRC_EVENT --> DETECT
    DETECT --> FILTER
    FILTER -->|Match| TRANSFORM
    FILTER -->|No Match| SKIP[Skip]
    TRANSFORM --> TRANSFER
    TRANSFER --> DST_STORE
    DST_STORE --> VERIFY
    VERIFY -->|Success| STATUS
    VERIFY -->|Failure| RETRY[Retry with Backoff]
    RETRY --> TRANSFER

    STATUS -->|Update| SRC_QUEUE

Replication Types

Type Description Use Case
CRR (Cross-Region) Replicate to bucket in different region Disaster recovery, latency reduction
SRR (Same-Region) Replicate to bucket in same region Compliance, workload isolation
Same-account Source and dest in same AWS account Simple DR setup
Cross-account Source and dest in different accounts Multi-account governance

5.7 Access Control System

graph TD
    subgraph AuthFlow["Authentication & Authorization Flow"]
        REQ[Incoming Request] --> AUTH{Authenticate}
        AUTH -->|IAM User| SIGV4[Signature V4 Verification]
        AUTH -->|IAM Role| STS[STS Token Verification]
        AUTH -->|Presigned URL| PRESIGN[Presigned URL Verification]
        AUTH -->|Anonymous| ANON[Anonymous Check]

        SIGV4 --> POLICY_EVAL
        STS --> POLICY_EVAL
        PRESIGN --> POLICY_EVAL
        ANON --> POLICY_EVAL

        POLICY_EVAL[Policy Evaluation Engine] --> CHECK1{IAM Policy<br/>Allows?}
        CHECK1 -->|Deny| DENY[403 Forbidden]
        CHECK1 -->|Not Specified| CHECK2{Bucket Policy<br/>Allows?}
        CHECK2 -->|Deny| DENY
        CHECK2 -->|Not Specified| CHECK3{ACL<br/>Allows?}
        CHECK3 -->|Deny| DENY
        CHECK3 -->|Allow| CHECK4{S3 Block Public Access?}
        CHECK4 -->|Blocked| DENY
        CHECK4 -->|Allowed| ALLOW[200 OK]
    end

Permission Evaluation Order

1. Explicit Deny (any policy) → DENY (immediate)
2. S3 Block Public Access settings → DENY if enabled
3. IAM Policy (user/role) → Allow or Not Specified
4. Bucket Policy → Allow, Deny, or Not Specified
5. ACL → Allow or Not Specified
6. Default → DENY (deny by default)

Explicit Deny always wins over Allow.

5.8 Encryption Service

graph TD
    subgraph EncryptionOptions["Encryption Options"]
        SSE_S3[SSE-S3<br/>Amazon-managed keys]
        SSE_KMS[SSE-KMS<br/>Customer-managed keys]
        SSE_C[SSE-C<br/>Customer-provided keys]
        CSE[CSE-C<br/>Client-side encryption]
    end

    subgraph SSE_S3_Flow["SSE-S3 Flow"]
        DATA1[Plaintext Data] --> ENCRYPT1[AES-256 Encrypt]
        KEY1[S3 Master Key] --> ENCRYPT1
        ENCRYPT1 --> CIPHER1[Ciphertext]
        CIPHER1 --> STORE1[Store Object]
        KEY_REF1[Key ID: aws/s3] --> STORE1
    end

    subgraph SSE_KMS_Flow["SSE-KMS Flow"]
        DATA2[Plaintext Data] --> ENCRYPT2[AES-256 Encrypt]
        DKE2[Data Key from KMS] --> ENCRYPT2
        ENCRYPT2 --> CIPHER2[Ciphertext]
        CIPHER2 --> STORE2[Store Object]
        WRAPPED2[Encrypted Data Key] --> STORE2
        KMS_KEY2[KMS CMK] -->|Encrypt| WRAPPED2
    end

    subgraph SSE_C_Flow["SSE-C Flow"]
        DATA3[Plaintext Data] --> ENCRYPT3[AES-256 Encrypt]
        CUST_KEY3[Customer Key in Header] --> ENCRYPT3
        ENCRYPT3 --> CIPHER3[Ciphertext]
        CIPHER3 --> STORE3[Store Object]
        NOTE3[Key NOT stored by S3]
    end

6. Data Model

6.1 Metadata Database Schema

-- Bucket metadata
CREATE TABLE buckets (
    bucket_id       UUID PRIMARY KEY,
    bucket_name     VARCHAR(63) UNIQUE NOT NULL,  -- DNS-compliant
    owner_account   VARCHAR(12) NOT NULL,          -- AWS account ID
    region          VARCHAR(32) NOT NULL,
    creation_time   TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    versioning      VARCHAR(20) DEFAULT 'Disabled', -- Enabled, Suspended, Disabled
    encryption      JSONB,                         -- Default encryption config
    lifecycle_rules JSONB,                         -- Lifecycle configuration
    replication_config JSONB,                      -- Replication configuration
    tagging         JSONB,                         -- Bucket tags
    policy          TEXT,                          -- Bucket policy (JSON)
    object_count    BIGINT DEFAULT 0,
    total_size_bytes BIGINT DEFAULT 0,
    is_public       BOOLEAN DEFAULT false,
    CONSTRAINT valid_versioning CHECK (versioning IN ('Enabled', 'Suspended', 'Disabled'))
);

-- Object metadata
CREATE TABLE objects (
    object_id       UUID PRIMARY KEY,
    bucket_id       UUID NOT NULL REFERENCES buckets(bucket_id),
    key             VARCHAR(1024) NOT NULL,        -- Object key (path)
    version_id      VARCHAR(64) NOT NULL,
    is_latest       BOOLEAN DEFAULT true,
    is_delete_marker BOOLEAN DEFAULT false,
    size_bytes      BIGINT NOT NULL,
    content_type    VARCHAR(255),
    content_md5     VARCHAR(32),
    etag            VARCHAR(128) NOT NULL,
    storage_class   VARCHAR(32) DEFAULT 'STANDARD',
    encryption_alg  VARCHAR(32),
    encryption_key_id VARCHAR(256),
    owner_account   VARCHAR(12),
    acl             VARCHAR(64) DEFAULT 'private',
    tagging         JSONB,
    user_metadata   JSONB,
    chunk_locations JSONB NOT NULL,                -- Array of chunk locations
    replication_status VARCHAR(32) DEFAULT 'STORAGE_CLASSAnalysis',
    lifecycle_status VARCHAR(32) DEFAULT 'ACTIVE',
    creation_time   TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    last_modified   TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    archive_retrieval_time TIMESTAMP WITH TIME ZONE,
    UNIQUE(bucket_id, key, version_id),
    CONSTRAINT valid_storage_class CHECK (
        storage_class IN ('STANDARD', 'STANDARD_IA', 'ONEZONE_IA',
                         'GLACIER', 'DEEP_ARCHIVE', 'INTELLIGENT_TIERING')
    )
);

-- Multipart upload state
CREATE TABLE multipart_uploads (
    upload_id       UUID PRIMARY KEY,
    bucket_id       UUID NOT NULL REFERENCES buckets(bucket_id),
    key             VARCHAR(1024) NOT NULL,
    initiator_account VARCHAR(12),
    owner_account   VARCHAR(12),
    storage_class   VARCHAR(32) DEFAULT 'STANDARD',
    encryption_alg  VARCHAR(32),
    encryption_key_id VARCHAR(256),
    initiation_time TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    expiration_time TIMESTAMP WITH TIME ZONE,      -- Auto-abort time
    status          VARCHAR(20) DEFAULT 'ACTIVE',  -- ACTIVE, COMPLETED, ABORTED
    total_parts     INT DEFAULT 0,
    completed_parts INT DEFAULT 0,
    part_metadata   JSONB,                         -- Part ETags and sizes
    CONSTRAINT valid_status CHECK (status IN ('ACTIVE', 'COMPLETED', 'ABORTED'))
);

-- Multipart upload parts
CREATE TABLE multipart_parts (
    part_id         BIGSERIAL PRIMARY KEY,
    upload_id       UUID NOT NULL REFERENCES multipart_uploads(upload_id),
    part_number     INT NOT NULL,
    size_bytes      BIGINT NOT NULL,
    etag            VARCHAR(128) NOT NULL,
    storage_location JSONB NOT NULL,               -- Where this part is stored
    upload_time     TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    UNIQUE(upload_id, part_number)
);

-- Chunk locations (for erasure-coded objects)
CREATE TABLE chunks (
    chunk_id        UUID PRIMARY KEY,
    object_id       UUID NOT NULL REFERENCES objects(object_id),
    block_index     INT NOT NULL,                  -- Which block this chunk belongs to
    chunk_index     INT NOT NULL,                  -- Which chunk within the block (0-14)
    node_id         VARCHAR(64) NOT NULL,          -- Storage node ID
    az_id           VARCHAR(16) NOT NULL,          -- Availability zone
    disk_id         VARCHAR(32) NOT NULL,          -- Physical disk
    offset          BIGINT NOT NULL,               -- Offset on disk
    size_bytes      INT NOT NULL,
    checksum        VARCHAR(32) NOT NULL,           -- CRC32C
    created_at      TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    last_verified   TIMESTAMP WITH TIME ZONE,
    status          VARCHAR(20) DEFAULT 'HEALTHY', -- HEALTHY, DEGRADED, REPAIRING
    UNIQUE(object_id, block_index, chunk_index)
);

-- Replication tasks
CREATE TABLE replication_tasks (
    task_id         UUID PRIMARY KEY,
    source_object_id UUID NOT NULL REFERENCES objects(object_id),
    source_bucket_id UUID NOT NULL,
    dest_bucket_id  UUID NOT NULL,
    dest_region     VARCHAR(32) NOT NULL,
    status          VARCHAR(20) DEFAULT 'PENDING', -- PENDING, IN_PROGRESS, COMPLETED, FAILED
    error_message   TEXT,
    retry_count     INT DEFAULT 0,
    max_retries     INT DEFAULT 5,
    created_at      TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    completed_at    TIMESTAMP WITH TIME ZONE,
    next_retry_at   TIMESTAMP WITH TIME ZONE
);

-- Indexes
CREATE INDEX idx_objects_bucket ON objects(bucket_id, key);
CREATE INDEX idx_objects_bucket_latest ON objects(bucket_id, key) WHERE is_latest = true;
CREATE INDEX idx_objects_version ON objects(bucket_id, key, version_id);
CREATE INDEX idx_objects_storage_class ON objects(storage_class, last_modified);
CREATE INDEX idx_objects_lifecycle ON objects(lifecycle_status, last_modified);
CREATE INDEX idx_objects_creation ON objects(creation_time);
CREATE INDEX idx_mp_uploads_bucket ON multipart_uploads(bucket_id, key);
CREATE INDEX idx_mp_uploads_status ON multipart_uploads(status, expiration_time);
CREATE INDEX idx_chunks_object ON chunks(object_id, block_index);
CREATE INDEX idx_chunks_node ON chunks(node_id, status);
CREATE INDEX idx_replication_status ON replication_tasks(status, next_retry_at);
CREATE INDEX idx_buckets_name ON buckets(bucket_name);
CREATE INDEX idx_buckets_region ON buckets(region);

6.2 Metadata Service (Redis) Schema

# Hot object metadata (TTL: 5 minutes)
HSET meta:hot:{object_id}
    key "photos/vacation/beach.jpg"
    bucket_id "bucket-xyz789"
    size_bytes "5242880"
    content_type "image/jpeg"
    etag "\"d41d8cd98f00b204e9800998ecf8427e\""
    storage_class "STANDARD"
    chunk_locations "[{\"node\":\"n1\",\"az\":\"1a\"},{\"node\":\"n2\",\"az\":\"1b\"}]"
EXPIRE meta:hot:{object_id} 300

# Bucket metadata cache
HSET meta:bucket:{bucket_id}
    name "my-bucket"
    versioning "Enabled"
    encryption "{\"algorithm\":\"AES256\"}"
    policy "{\"Version\":\"2012-10-17\",\"Statement\":[...]}"
EXPIRE meta:bucket:{bucket_id} 60

# Multipart upload state
HSET mp:{upload_id}
    bucket_id "bucket-xyz789"
    key "large-video.mp4"
    status "ACTIVE"
    total_parts "3"
    completed_parts "2"
    parts "{\"1\":\"etag1\",\"2\":\"etag2\"}"
EXPIRE meta mp:{upload_id} 604800  # 7 days

# Rate limiting
INCR rate:{account_id}:{hour}
EXPIRE rate:{account_id}:{hour} 3600

# Request metering (for billing)
INCR billing:{account_id}:{date}:gets
INCR billing:{account_id}:{date}:puts
INCRBY billing:{account_id}:{date}:bytes_downloaded {bytes}
INCRBY billing:{account_id}:{date}:bytes_uploaded {bytes}

6.3 Object Key Naming and Partitioning

Key Structure:
├── Bucket: globally unique name (3-63 chars, DNS-compliant)
├── Key: hierarchical path (up to 1,024 bytes UTF-8)
│   ├── Prefix: "photos/2026/vacation/"
│   └── Object name: "beach.jpg"
│
Partitioning Strategy:
├── Hash key: SHA-256(bucket_id + key)
├── Partition ID: hash % 1024 (1024 partitions)
├── Each partition handled by a storage group
└── Each storage group spans all AZs in the region

Why 1024 partitions?
├── Balances load evenly across infrastructure
├── Allows independent scaling per partition
├── Limits blast radius of any single failure
└── Can increase to 4096 as needed

7. Key Design Decisions

7.1 Consistency Model Evolution

Era Model Rationale
2006-2020 Eventual consistency Simpler, better performance, acceptable for most use cases
2020+ Strong consistency Customer demand, competitive pressure, infrastructure maturity

Decision: Strong consistency for all operations

Strong Consistency Implementation:
├── Write path:
│   ├── Accept write to primary AZ
│   ├── Write to quorum of AZs (2 of 3)
│   ├── Return success only after quorum ack
│   └── Async replicate to remaining AZ
│
├── Read path:
│   ├── Read from any AZ
│   ├── If read stale (version check fails):
│   │   ├── Read from all AZs
│   │   ├── Return latest version
│   │   └── Async repair stale copies
│   └── Read repair on every read
│
└── Trade-off:
    ├── Write latency increases by ~10-20ms
    ├── Read latency increases by ~5ms (with read repair)
    └── Worth it for correctness guarantees

7.2 Erasure Coding vs Replication

Factor Decision Rationale
Standard class 3x replication Performance-critical, can afford 200% overhead
IA classes 3x replication Still need good read performance
Glacier classes Erasure coding (12+3) Cost-critical, read performance less important
Deep Archive Erasure coding (8+2) Maximum cost efficiency

Decision: Hybrid approach based on storage class

7.3 Object Size Limits

Limit Value Rationale
Single PUT max 5 GB Balance between API simplicity and streaming
Multipart max 5 TB Covers 99.9% of use cases
Minimum part size 5 MB Prevents excessive metadata overhead
Max parts 10,000 Limits metadata complexity
Key length 1,024 bytes Supports deep hierarchical paths

7.4 Bucket Naming

Bucket Name Rules:
├── 3-63 characters
├── Lowercase letters, numbers, hyphens, dots
├── Must start with letter or number
├── Cannot be IP address format
├── Globally unique (across all AWS accounts, all regions)
└── Once created, name cannot be changed

Why globally unique?
├── Enables global endpoint: bucket.s3.amazonaws.com
├── Simplifies DNS resolution
├── Prevents naming conflicts
└── Trade-off: Name exhaustion at scale (mitigated by 63-char space)

7.5 Request Routing

Request Routing Strategy:
├── DNS-based routing:
│   ├── bucket.region.s3.amazonaws.com → Region-specific endpoint
│   └── bucket.s3.amazonaws.com → Default region endpoint
│
├── Path-style vs virtual-hosted-style:
│   ├── Virtual-hosted: bucket.s3.region.amazonaws.com/key (preferred)
│   └── Path-style: s3.region.amazonaws.com/bucket/key (legacy)
│
├── Transfer acceleration:
│   ├── bucket.s3-accelerate.amazonaws.com
│   └── Routes through CloudFront edge for optimal path
│
└── Regional endpoints:
    ├── Reduces latency (stay in-region)
    ├── Avoids cross-region data transfer charges
    └── Required for some operations (e.g., replication)

7.6 Billing Model

Pricing Components:
├── Storage cost: $/GB/month (varies by class and region)
├── Request cost: $/1,000 requests (varies by type)
│   ├── PUT/COPY/POST/LIST: $0.005/1K
│   └── GET/OTHER: $0.0004/1K
├── Data transfer:
│   ├── Inbound: Free
│   └── Outbound: $0.09/GB (first 10 TB/month)
├── Advanced features:
│   ├── Lifecycle transitions: $0.05/1K
│   ├── Replication: Storage + request costs at dest
│   └── Select/Query: $0.01/GB scanned
└── Early deletion (IA classes):
    ├── Standard-IA: Pro-rated storage + min-charge
    └── Glacier: Full minimum storage duration charge

8. Scalability Strategy

8.1 Multi-Level Scaling Architecture

graph TD
    subgraph Global["Global Scale"]
        R1[Region 1: us-east-1]
        R2[Region 2: eu-west-1]
        R3[Region 3: ap-southeast-1]
        RN[Region N: 30+ regions]
    end

    subgraph Region["Per-Region Scale"]
        AZ1[AZ-A]
        AZ2[AZ-B]
        AZ3[AZ-C]
        AZN[AZ-N: 3-6 AZs per region]
    end

    subgraph AZ["Per-AZ Scale"]
        SG1[Storage Group 1]
        SG2[Storage Group 2]
        SGN[Storage Group N: 1024 groups]
    end

    subgraph StorageGroup["Per-Storage-Group Scale"]
        SN1[Storage Node 1]
        SN2[Storage Node 2]
        SN3[Storage Node 3]
        SNM[Storage Node M: 100-1000 nodes]
    end

    Global --> Region
    Region --> AZ
    AZ --> StorageGroup
    StorageGroup --> StorageNode

    R1 --> AZ1
    R1 --> AZ2
    R1 --> AZ3
    AZ1 --> SG1
    AZ2 --> SG1
    AZ3 --> SG1
    SG1 --> SN1
    SG1 --> SN2
    SG1 --> SN3

8.2 Scaling Dimensions

Dimension Strategy Current Scale Target Scale
Regions Add new regions 30+ 50+
AZs per region Add AZs 3-6 6-9
Storage groups Consistent hashing 1,024 4,096
Nodes per group Auto-scale 100-1,000 1,000-10,000
Frontend servers Auto-scale 6,000 20,000+
Metadata servers Sharded 60,000 200,000+

8.3 Hot Key / Hot Partition Handling

Problem: A single key or prefix receives disproportionate traffic

Detection:
├── Monitor per-key request rate
├── Alert when > 5,000 GETs/sec or > 500 PUTs/sec per prefix
└── Track top 100 hottest keys

Mitigation:
├── 1. Client-side: Add random suffix to key
│   └── "photo.jpg" → "photo-a1b2c3.jpg"
│
├── 2. CDN caching: Cache hot objects at edge
│   └── CloudFront caches GET responses
│
├── 3. Read replicas: Create additional read replicas
│   └── For extremely hot objects
│
├── 4. Partition splitting: Split hot partition
│   └── Sub-partition by key hash
│
└── 5. Rate limiting: Enforce per-prefix limits
    └── 5,500 GETs/sec and 500 PUTs/sec per prefix

8.4 Metadata Scaling

Metadata is the bottleneck — every request touches metadata.

Scaling Strategy:
├── Tier 1: In-memory cache (Redis)
│   ├── Hot objects: Last accessed < 5 min
│   ├── Hit rate: > 95%
│   └── Scale: Add Redis shards
│
├── Tier 2: Warm cache (in-memory DB)
│   ├── Warm objects: Last accessed < 1 hour
│   ├── Hit rate: > 99% (combined with Tier 1)
│   └── Scale: Add memory nodes
│
├── Tier 3: Persistent store (DynamoDB-like)
│   ├── All objects
│   ├── Hit rate: < 1% (cache misses only)
│   └── Scale: Add DynamoDB shards
│
└── Optimization:
    ├── Metadata compression (protobuf)
    ├── Batch metadata reads for LIST operations
    └── Async metadata updates for non-critical fields

8.5 Data Transfer Optimization

Large Object Transfer:
├── Multipart upload: Parallel parts (10-100 concurrent)
├── Adaptive part size: Based on network bandwidth
├── Retry with exponential backoff: Per-part retry
├── Checksum verification: Per-part and final
└── Progress tracking: Client-side progress bar

Download Optimization:
├── Range requests: Partial downloads
├── Parallel range fetch: Multiple ranges in parallel
├── CDN caching: Edge cache for GET requests
├── Transfer acceleration: CloudFront-optimized path
└── Adaptive bitrate: For streaming use cases

9. Curveball Scenarios

Curveball 1: “How do we achieve 11 nines of durability (99.999999999%)?”

The Math:

11 nines = 99.999999999% durability
Annual failure rate = 0.000000001% = 1 in 1 billion

With 2 trillion objects:
├── Expected annual losses: 2T × 0.000000001% = 2 objects/year
├── That's less than 1 object loss per year per trillion objects stored
└── Over 10 years: ~20 objects lost out of 2T

This means we can lose AT MOST 2 objects per year out of 2 trillion.

Multi-Layer Strategy:

graph TD
    subgraph Layer1["Layer 1: Erasure Coding / Replication"]
        EC[3x replication across AZs]
        note1["Protects against: Disk failure, Node failure, AZ failure"]
    end

    subgraph Layer2["Layer 2: Continuous Verification"]
        VERIFY[Background checksum verification]
        note2["Every object verified weekly"]
    end

    subgraph Layer3["Layer 3: Self-Healing"]
        REPAIR[Automatic repair on detection]
        note3["Rebuild from surviving copies/chunks"]
    end

    subgraph Layer4["Layer 4: Cross-Region Replication"]
        CRR[Async cross-region replication]
        note4["Protects against: Region failure, Natural disaster"]
    end

    subgraph Layer5["Layer 5: Hardware Quality"]
        HW[Enterprise-grade hardware]
        note5["Enterprise SSDs/HDDs with low BER"]
    end

    subgraph Layer6["Layer 6: Scrubbing & Reconciliation"]
        SCRUB[Regular data scrubbing]
        note6["Detect silent bit rot, repair proactively"]
    end

    Layer1 --> Layer2 --> Layer3 --> Layer4 --> Layer5 --> Layer6

Detailed Implementation:

Layer Mechanism Failure Protected Frequency
Erasure Coding 12+3 parity across 15 nodes Up to 3 simultaneous node failures Always
3x Replication 3 copies in 3 AZs Up to 2 AZ failures Always
Checksum Verification CRC32C on every read + weekly scan Silent bit rot, corruption Every read + weekly
Self-Healing Auto-rebuild from surviving data Any detected degradation Immediate
Cross-Region Rep Async copy to 2+ regions Region-level disaster Continuous
Hardware Scrubbing Drive-level surface scan Impending disk failure Daily
Memory Scrubbing ECC memory + periodic test Memory bit flips Continuous
Network Verification End-to-end checksum on transfer Network corruption Every transfer

Failure Rate Analysis:

Single HDD failure rate: ~2% per year (enterprise drives)
Single node failure rate: ~5% per year (any component)
Single AZ failure rate: ~0.1% per year
Single region failure rate: ~0.01% per year

With 3x replication across 3 AZs:
P(all 3 copies fail) = P(AZ1 fails) × P(AZ2 fails) × P(AZ3 fails)
                      = 0.001 × 0.001 × 0.001 = 10^-9
                      = 0.0000001% per year per object

With additional cross-region replication:
P(all replicas fail) = 10^-9 × 10^-9 = 10^-18
                      = Far exceeds 11 nines requirement

The 11 nines target is achievable through layered defense.

Curveball 2: “A storage node fails — how do we detect, recover, and prevent data loss?”

Detection (Multi-Layer):

graph TD
    subgraph Detection["Detection Layers"]
        D1[Heartbeat Monitor<br/>30s interval]
        D2[Peer Health Check<br/>60s interval]
        D3[Read Failure Detection<br/>Immediate]
        D4[Checksum Mismatch<br/>On read]
        D5[SMART Disk Alerts<br/>Proactive]
    end

    subgraph Response["Response Pipeline"]
        ISOLATE[Isolate Failed Node<br/>Remove from pool]
        ASSESS[Assess Data Impact<br/>Which objects affected?]
        REBUILD[Rebuild Missing Data<br/>From surviving replicas/chunks]
        VERIFY[Verify Rebuilt Data<br/>Checksum validation]
        REPLACE[Replace Hardware<br/>New node joins pool]
    end

    D1 --> ISOLATE
    D2 --> ISOLATE
    D3 --> ISOLATE
    D4 --> ISOLATE
    D5 --> ISOLATE

    ISOLATE --> ASSESS
    ASSESS --> REBUILD
    REBUILD --> VERIFY
    VERIFY --> REPLACE

Recovery Process:

Step 1: Detection (0-30 seconds)
├── Heartbeat misses (3 consecutive = 90s)
├── Peer nodes report unreachable
├── Read requests to node timeout
└── Alert fired to operations team

Step 2: Isolation (30-60 seconds)
├── Remove node from consistent hash ring
├── Redirect new writes away from node
├── Mark node as "DEGRADED" in metadata
└── Client requests transparently routed to healthy nodes

Step 3: Impact Assessment (1-5 minutes)
├── Query metadata: which objects have chunks on this node?
├── For replicated objects: Check other 2 copies are healthy
├── For erasure-coded objects: Check 12 of 15 chunks survive
└── Calculate rebuild priority (most accessed objects first)

Step 4: Data Rebuild (5-60 minutes)
├── For 3x replication:
│   ├── Copy from surviving replica to new location
│   ├── Verify checksum matches
│   └── Update metadata with new chunk locations
│
├── For erasure coding:
│   ├── Read 12 surviving chunks
│   ├── Reconstruct missing chunks via Reed-Solomon
│   ├── Write reconstructed chunks to new nodes
│   └── Update metadata
│
└── Rebuild bandwidth throttled to 1Gbps
    └── Prevent impact on production traffic

Step 5: Verification (continuous)
├── Checksum verify all rebuilt data
├── Read test on rebuilt objects
├── Update node health status
└── Clear alerts when fully recovered

Step 6: Hardware Replacement (1-24 hours)
├── Physical node replaced
├── New node commissioned
├── Joins storage pool
└── Rebalancing distributes data evenly

RPO/RTO Analysis:

Metric Target Actual
Detection time < 60s ~30s (heartbeat)
Isolation time < 60s ~30s
Rebuild time < 1 hour ~30 min (typical)
RPO 0 (no data loss) 0 (redundancy ensures this)
RTO < 5 min (client impact) ~1 min (transparent failover)

Curveball 3: “How do we handle a complete region failure (e.g., natural disaster destroys us-east-1)?”

Scenario: Hurricane destroys all infrastructure in us-east-1.

Impact Assessment:

us-east-1 contains:
├── 30% of global S3 data
├── 100M+ objects affected
├── 5M+ active users affected
└── Estimated data: 6 EB

Without preparation: Complete data loss for non-replicated objects
With preparation: Minimal disruption, automatic failover

Disaster Recovery Strategy:

graph TD
    subgraph Before["Before Disaster"]
        PRIMARY[Primary Region: us-east-1]
        REPLICATOR[Async Replicator]
        SECONDARY[Secondary Region: us-west-2]
        REPLICATOR -->|Continuous sync| SECONDARY
    end

    subgraph During["During Disaster"]
        PRIMARY_DOWN[us-east-1: DOWN]
        DNS_FAIL[DNS Failover]
        SECONDARY_ACTIVE[us-west-2: ACTIVE]
        READERS[Readers routed to us-west-2]
    end

    subgraph After["After Disaster"]
        PROMOTE[Promote us-west-2 to primary]
        NEW_REGION[Stand up new region: us-east-1-replacement]
        REPLICATE_BACK[Replicate from us-west-2 to new region]
        RESTORE[Full service restored]
    end

    PRIMARY_DOWN --> DNS_FAIL
    DNS_FAIL --> SECONDARY_ACTIVE
    SECONDARY_ACTIVE --> READERS
    READERS --> PROMOTE
    PROMOTE --> NEW_REGION
    NEW_REGION --> REPLICATE_BACK
    REPLICATE_BACK --> RESTORE

Detailed DR Plan:

Phase Action Timeline Responsible
Detection Region health check fails 0-5 min Automated
Alert Page on-call team, notify customers 5-10 min Automated + Human
DNS Failover Route traffic to secondary region 10-15 min Automated (Route 53)
Promote Secondary Make secondary the new primary 15-30 min Automated
Customer Communication Status page updates, emails 15-60 min Human
New Region Standup Provision new infrastructure 1-24 hours Automated + Human
Data Replication Replicate from promoted region 24-72 hours Automated
Full Recovery All services at pre-disaster level 3-7 days Team

Pre-Disaster Preparation:

Mandatory DR Measures:
├── 1. Cross-region replication enabled for all critical buckets
│   └── CRR configured with automatic failover
│
├── 2. DNS failover configured
│   └── Route 53 health checks + failover routing
│
├── 3. Secondary region always warm
│   └── Infrastructure pre-provisioned, ready to scale
│
├── 4. Regular DR drills
│   └── Quarterly failover tests
│
├── 5. Data consistency verification
│   └── Daily checksum comparison between regions
│
└── 6. Customer communication templates
    └── Pre-approved status messages

Data Loss Risk:

With async replication:
├── Replication lag: Typically < 1 minute
├── Worst case lag: Up to 5 minutes (during high load)
├── RPO: Up to 5 minutes of data loss
└── Mitigation: Critical buckets use synchronous replication

Without replication:
├── RPO: Complete data loss
└── Mitigation: Enable CRR for all buckets (default recommendation)

Curveball 4: “How do we prevent and detect data corruption (bit rot)?”

The Problem:

Bit rot (silent data corruption):
├── HDD: ~1 in 10^15 bits flipped per read (unrecovered BER)
├── SSD: ~1 in 10^16 bits flipped per read
├── Memory: ~1 in 10^17 bits flipped per access (with ECC)
├── Network: ~1 in 10^8 bits flipped per packet (TCP catches most)
│
With 20 EB of data:
├── 20 EB = 1.6 × 10^20 bits
├── Expected bit flips per full scan: 1.6 × 10^20 / 10^15 = 160,000
└── Without detection: Silent corruption accumulates

Multi-Layer Protection:

graph TD
    subgraph Prevention["Prevention"]
        ECC[ECC Memory]
        TCP[TCP Checksums]
        ENTERPRISE[Enterprise-grade drives]
        RAID[RAID / Erasure Coding parity]
    end

    subgraph Detection["Detection"]
        WRITE_CHECK[Write-time checksum<br/>CRC32C + SHA256]
        READ_CHECK[Read-time checksum verification]
        BG_SCAN[Background scan<br/>Weekly full scan]
        ON_DEMAND[On-demand verification]
    end

    subgraph Repair["Repair"]
        AUTO_REPAIR[Auto-repair from replica]
        EC_REBUILD[Rebuild from parity]
        CLIENT_NOTIFY[Notify client on unrepairable]
    end

    ECC --> WRITE_CHECK
    TCP --> WRITE_CHECK
    ENTERPRISE --> WRITE_CHECK
    RAID --> WRITE_CHECK

    WRITE_CHECK --> READ_CHECK
    READ_CHECK --> BG_SCAN
    BG_SCAN --> ON_DEMAND

    READ_CHECK -->|Mismatch| AUTO_REPAIR
    BG_SCAN -->|Mismatch| AUTO_REPAIR
    ON_DEMAND -->|Mismatch| AUTO_REPAIR

    AUTO_REPAIR -->|No replica| EC_REBUILD
    EC_REBUILD -->|No parity| CLIENT_NOTIFY

Checksum Strategy:

Checksum Purpose Algorithm When Computed
CRC32C Fast verification CRC32C (Castagnoli) Every read/write
SHA-256 Strong integrity SHA-256 Write time, stored in metadata
MD5 Backward compatibility MD5 (ETag) Write time
Per-chunk CRC Chunk-level verification CRC32C Every chunk read/write

Background Scan Process:

Weekly Full Scan:
├── Scope: All objects in the region
├── Rate: 1% of objects per day (full scan in ~100 days)
├── Priority: Recently accessed objects scanned more frequently
├── Process:
│   1. Read object data from storage
│   2. Compute CRC32C and SHA-256
│   3. Compare with stored checksums
│   4. If mismatch:
│      a. Try reading from replica
│      b. If replica matches: Replace corrupted copy
│      c. If all copies corrupted: Rebuild from erasure coding
│      d. If unrecoverable: Flag for manual review
│   5. Update last_verified timestamp
│   6. Log result for audit
│
└── Metrics Tracked:
    ├── Objects scanned per day
    ├── Corruptions detected
    ├── Auto-repairs completed
    └── Unrecoverable corruptions

Scenario: A news organization uploads a breaking news video to S3. Within minutes, millions of users try to download it simultaneously.

Problem Analysis:

Normal object: 100 GETs/sec
Viral object: 1,000,000 GETs/sec

Bottlenecks:
├── Single storage node: ~10,000 GETs/sec max
├── Single metadata entry: Lock contention at high concurrency
├── Network bandwidth: 10 Gbps = ~1.25 GB/sec
├── With 500 KB avg response: ~2,500 GETs/sec per 10 Gbps link
└── Without mitigation: 400x overload

Multi-Layer Mitigation:

graph TD
    subgraph Layer1["Layer 1: CDN Caching"]
        CF[CloudFront Edge Cache]
        note1["Cache at 200+ edge locations"]
        HIT_RATE["Hit rate: >99% for viral content"]
    end

    subgraph Layer2["Layer 2: Read Replicas"]
        REPL1[Read Replica 1]
        REPL2[Read Replica 2]
        REPL3[Read Replica N]
        note2["Auto-create replicas under load"]
    end

    subgraph Layer3["Layer 3: Metadata Caching"]
        META_CACHE[Metadata in Redis]
        note3["Avoid metadata DB hit on every GET"]
    end

    subgraph Layer4["Layer 4: Rate Limiting"]
        RL[Per-key rate limiting]
        note4["Fair queuing, not hard cutoff"]
    end

    subgraph Layer5["Layer 5: Storage Replication"]
        SR[Additional storage replicas]
        note5["Temporarily increase replication factor"]
    end

    CF -->|Cache miss| REPL1
    CF -->|Cache miss| REPL2
    CF -->|Cache miss| REPL3
    REPL1 --> META_CACHE
    REPL2 --> META_CACHE
    REPL3 --> META_CACHE
    META_CACHE -->|Miss| RL
    RL --> SR

Detailed Strategy:

Layer Mechanism Capacity Increase Latency Impact
CDN Cache Edge cache at 200+ locations 100x+ Lower (edge closer)
Read Replicas Auto-scale read replicas 10x Same
Metadata Cache Redis cache for metadata 50x Lower
Rate Limiting Fair queuing per key N/A Slight increase
Storage Replicas Temp 3x → 10x replication 3x Same

Auto-Scaling Trigger:

Detection:
├── Monitor per-key GET rate
├── Threshold: > 10,000 GETs/sec for 60 seconds
└── Classification: "Hot object"

Actions (automatic):
├── 1. Enable CDN caching (if not already)
│   └── Cache-Control: public, max-age=300
│
├── 2. Create read replicas
│   └── Scale to 10 replicas across AZs
│
├── 3. Promote to metadata cache
│   └── Pin in Redis with extended TTL
│
├── 4. Increase storage replication
│   └── Temporarily increase from 3x to 10x
│
└── 5. Monitor and auto-degrade
    └── After 24 hours of normal traffic, scale back

Cost Consideration:

Viral object costs:
├── CDN egress: $0.085/GB (first 10 TB)
├── S3 GET requests: $0.0004/1,000
├── Additional replicas: Storage cost × 7 extra copies
└── Total for 1 PB viral download: ~$85,000

Customer pays for what they use — this is the cloud model.

10. Failure Modes and Mitigations

10.1 Failure Mode Matrix

Failure Mode Probability Impact Detection Mitigation RTO RPO
Single disk failure High (daily) Low SMART + heartbeat Erasure coding / 3x replica, auto-rebuild 5min 0
Single node failure Medium (hourly) Low Heartbeat timeout Redistribute chunks, auto-rebuild 1min
AZ failure Low (monthly) Medium Health check failure 3x across AZs, failover 5min 0
Region failure Very Low (yearly) High Region health check Cross-region replication 15min 5min
Metadata corruption Very Low Critical Checksum mismatch Redundant metadata, PITR 15min 5min
Network partition Low High Latency spike Quorum-based writes, read repair 30s 0
Massive DDoS Medium High Traffic anomaly Rate limiting, WAF, auto-scale 1min 0
Encryption key loss Very Low Critical Key access failure KMS key rotation, backup keys 1hr 0
Erasure coding bug Very Low Critical Checksum failure Fallback to replication, manual rebuild 1hr 0
Clock skew Low Medium Timestamp anomaly NTP sync, logical clocks N/A 0
Silent bit rot Medium (cumulative) Medium Background scan CRC32C + SHA256 verification N/A 0
Cascading failure Low Critical Multi-service alerts Circuit breakers, bulkheads 5min 0

10.2 Detailed Mitigation Strategies

Single Disk Failure (Most Common)

Frequency: ~1 disk failure per 1,000 disks per month
With 12M disks: ~12,000 disk failures per month

Detection:
├── SMART alerts (predictive, hours before failure)
├── Read error rate spike
├── Checksum mismatch on read
└── Heartbeat from disk controller

Mitigation:
├── Immediate: Mark disk as failed, stop writes
├── Data recovery:
│   ├── For 3x replication: Read from other 2 copies
│   └── For erasure coding: Reconstruct from parity
├── Rebuild: Write recovered data to replacement disk
├── Rebalance: Redistribute data to maintain even load
└── Replace: Physical disk replacement (hot-swap)

Impact on users: None (transparent)

Network Partition (Split-Brain)

graph TD
    subgraph Before["Before Partition"]
        N1[Node Group A] <-->|Sync| N2[Node Group B]
    end

    subgraph During["During Partition"]
        N1_ISOLATED[Node Group A<br/>Independent]
        N2_ISOLATED[Node Group B<br/>Independent]
        QUORUM_A{Quorum?}
        QUORUM_B{Quorum?}
    end

    subgraph After["After Healing"]
        MERGE[Merge Divergent State]
        N1_HEALED[Node Group A]
        N2_HEALED[Node Group B]
    end

    N1_ISOLATED --> QUORUM_A
    N2_ISOLATED --> QUORUM_B
    QUORUM_A -->|Yes: Continue writes| N1_ISOLATED
    QUORUM_A -->|No: Read-only| N1_ISOLATED
    QUORUM_B -->|Yes: Continue writes| N2_ISOLATED
    QUORUM_B -->|No: Read-only| N2_ISOLATED

    N1_HEALED --> MERGE
    N2_HEALED --> MERGE

Quorum-Based Write Protocol:

Write Protocol (N=3, W=2, R=2):
├── N = Total replicas (3 AZs)
├── W = Write quorum (2 AZs must ack)
├── R = Read quorum (2 AZs must respond)
└── Guarantee: W + R > N → 2 + 2 > 3 → Always overlap

Write flow:
1. Client sends PUT to any AZ
2. Primary AZ writes locally
3. Primary AZ replicates to 2 other AZs
4. Wait for W=2 acks (including primary)
5. Return success to client
6. Async replicate to remaining AZ

Read flow:
1. Client sends GET to any AZ
2. Read from R=2 AZs
3. Compare versions (using version vector)
4. Return latest version
5. Read repair: Update stale AZs async

During partition:
├── If partition leaves ≥ W nodes: Writes continue
├── If partition leaves < W nodes: Writes blocked (safe)
└── Reads may return stale data if R nodes are partitioned

Cascading Failure Prevention

Circuit Breaker Pattern:
├── Each service has circuit breakers for dependencies
├── States: CLOSED → OPEN → HALF-OPEN → CLOSED
├── Trip threshold: 50% failure rate over 60s window
├── Reset timeout: 30 seconds
└── Half-open: Allow 1 request through, success → CLOSED

Bulkhead Pattern:
├── Isolate failure domains
├── Separate thread pools per service
├── Separate connection pools per dependency
└── Limit blast radius of any single failure

Rate Limiting:
├── Per-account rate limits
├── Per-key rate limits
├── Global rate limits
└── Adaptive limits based on system load

Graceful Degradation:
├── Non-critical features disabled under load
├── Reduced metadata returned in responses
├── Slower storage classes deprioritized
└── Event notifications queued, not dropped

10.3 Monitoring and Alerting

Key Metrics:
├── Storage Health
│   ├── Total capacity used (%)
│   ├── Disk failure rate (per 1,000 disks/month)
│   ├── Rebuild queue depth
│   ├── Erasure coding rebuild time
│   └── Checksum mismatch rate
│
├── Request Performance
│   ├── GET/PUT/DELETE latency (P50/P95/P99)
│   ├── Request success rate (%)
│   ├── Error rate by error code
│   └── Throttled request rate
│
├── Data Integrity
│   ├── Background scan progress
│   ├── Corruptions detected/day
│   ├── Auto-repairs completed/day
│   └── Unrecoverable corruptions
│
├── Replication
│   ├── Replication lag (seconds)
│   ├── Replication success rate
│   └── Failed replication tasks
│
├── Infrastructure
│   ├── Node health score
│   ├── Network bandwidth utilization
│   ├── CPU/Memory utilization
│   └── Disk I/O utilization
│
└── Business
    ├── Total objects count
    ├── Total storage used
    ├── Active buckets
    └── Revenue metrics

Alert Thresholds:
├── Critical: P99 latency > 500ms for 5 min
├── Critical: Disk failure rate > 10x baseline
├── Critical: Checksum mismatch rate > 0.01%
├── Critical: Replication lag > 1 hour
├── Warning: P95 latency > 200ms for 10 min
├── Warning: Rebuild queue > 1,000 objects
├── Warning: Storage capacity > 80%
└── Info: New disk failure detected

10.4 Compliance and Audit

Compliance Requirements:
├── SOC 2 Type II: Annual audit
├── HIPAA: Healthcare data protection
├── PCI DSS: Payment card data
├── FedRAMP: Government workloads
├── ISO 27001: Information security
└── GDPR: Data residency and right to erasure

Audit Capabilities:
├── CloudTrail: All API calls logged
├── S3 Server Access Logging: All requests logged
├── S3 Object Lambda: Custom processing with audit
├── VPC Flow Logs: Network-level audit
└── Config: Resource configuration history

Data Residency:
├── Region-specific endpoints
├── Data never leaves chosen region (unless replicated)
├── Cross-region replication is opt-in
└── Government Cloud for classified data

Appendix A: Glossary

Term Definition
Object A file and its metadata stored in S3
Bucket A container for objects; globally unique namespace
Key The full path/name of an object within a bucket
ETag Entity tag; MD5 hash of object content (for non-multipart)
Multipart Upload Uploading large objects in parallel parts
Erasure Coding Data protection using parity chunks instead of full replicas
CRC32C Cyclic Redundancy Check using Castagnoli polynomial
BER Bit Error Rate; probability of bit flip during storage/read
RTO Recovery Time Objective; max acceptable downtime
RPO Recovery Point Objective; max acceptable data loss
CRR Cross-Region Replication
SRR Same-Region Replication
SSE Server-Side Encryption
KMS Key Management Service
WORM Write Once Read Many (Object Lock compliance mode)
Quorum Minimum number of replicas that must agree for read/write
Read Repair Consistency mechanism that repairs stale replicas on read
Bit Rot Silent data corruption over time

Appendix B: References


Document Version: 1.0 | Last Updated: 2026-08-28