i# Design a Payments Gateway like Stripe
Table of Contents
- Requirements Clarification
- Back-of-Envelope Calculations
- System Interface Design
- High-Level Architecture
- Core Component Design
- Data Model
- Key Design Decisions
- Scalability Strategy
- Curveball Scenarios
- Failure Modes and Mitigations
1. Requirements Clarification
1.1 Functional Requirements
- Payment Processing: Accept and process payments from multiple sources (credit cards, debit cards, bank transfers, digital wallets, ACH, SEPA)
- Idempotency: Guarantee exactly-once payment processing even with network retries using idempotency keys
- Multi-Currency Support: Handle transactions in 135+ currencies with real-time FX conversion
- Fraud Detection: Real-time fraud scoring using ML models, rule engines, and velocity checks
- Chargeback Management: Automated chargeback handling, evidence submission, and dispute workflows
- Reconciliation: Daily settlement and reconciliation with acquiring banks and card networks
- Webhooks: Reliable event delivery to merchant systems (payment succeeded, failed, refunded, chargeback)
- Refunds & Captures: Support full/partial refunds, authorization-then-capture flows
- PCI Compliance: Tokenization of card data, PCI DSS Level 1 compliance, no raw PAN in merchant systems
- Dashboard & Reporting: Real-time transaction monitoring, analytics, and exportable reports
- Connect Platform: Support marketplace scenarios with split payments, platform fees, and connected accounts
- Subscription Billing: Recurring billing with proration, trial periods, and dunning management
1.2 Non-Functional Requirements
| Category | Requirement | Rationale |
|---|---|---|
| Availability | 99.99% uptime (52.6 min downtime/year) | Payment downtime = direct revenue loss |
| Latency | p99 < 300ms for authorization | Customer checkout experience |
| Throughput | 10,000+ TPS peak (Black Friday scale) | Handle seasonal spikes |
| Consistency | Strong consistency for financial records | No double-spending, no lost payments |
| Durability | Zero data loss for transactions | Financial audit requirements |
| Security | PCI DSS Level 1, SOC 2 Type II | Regulatory compliance |
| Scalability | Horizontal scaling, multi-region active-active | Global merchant base |
| Auditability | Immutable audit trail, 7-year retention | Regulatory and legal requirements |
2. Back-of-Envelope Calculations
2.1 Assumption Sets
We consider three scales to reason about capacity:
| Parameter | Startup (Year 1) | Growth (Year 3) | Stripe-scale (Today) |
|---|---|---|---|
| Daily Transactions | 100K | 10M | 50M+ |
| Peak TPS | 50 | 5,000 | 50,000+ |
| Monthly Volume (GMV) | $10M | $1B | $50B+ |
| Merchants | 1,000 | 100K | 3M+ |
| Avg Transaction Size | $100 | $100 | $100 |
2.2 Storage Estimates (Stripe-scale)
Per transaction record:
- Core transaction: ~2 KB (JSON metadata)
- Audit trail: ~1 KB/event × 5 events = 5 KB
- Card token reference: ~500 B
- Total per transaction: ~7.5 KB
Daily storage: 50M × 7.5 KB = 375 GB/day
Monthly storage: 375 GB × 30 = 11.25 TB/month
Annual storage: ~135 TB/year
With 3x replication: ~405 TB/year raw storage
7-year retention: ~2.7 PB total
Hot data (90 days): ~100 TB
Warm data (1-2 years): ~500 TB
Cold data (2-7 years): ~2 PB → archive to object storage
2.3 Bandwidth Estimates
Inbound API traffic:
- Request: ~1 KB avg × 50M/day = 50 GB/day
- Response: ~2 KB avg × 50M/day = 100 GB/day
- Total API: ~150 GB/day = ~1.7 GB/min
Outbound webhooks:
- ~3 events/transaction × 50M = 150M events/day
- ~1 KB/event = 150 GB/day
Card network communication:
- ISO 8583 messages: ~500 bytes × 50M = 25 GB/day
- Response messages: ~25 GB/day
Total network I/O: ~350 GB/day ≈ 4 MB/s sustained
Peak (10x burst): ~40 MB/s
2.4 QPS Estimates
Average QPS: 50M / 86,400s = ~580 QPS
Peak QPS (10x burst): ~5,800 QPS
Black Friday (20x burst): ~11,600 QPS
With read amplification (dashboard, webhooks, retries):
- Read QPS: ~5x write = ~2,900 avg, ~29,000 peak
- Total QPS: ~3,500 avg, ~35,000 peak
3. System Interface Design
3.1 Core REST API
Create Payment Intent
POST /v1/payment_intents
Content-Type: application/x-www-form-urlencoded
{
"amount": 2000, // cents
"currency": "usd",
"payment_method": "pm_card_visa_4242",
"capture_method": "automatic", // or "manual"
"confirmation_method": "automatic",
"idempotency_key": "key_abc123",
"metadata": {
"order_id": "ord_12345"
},
"shipping": {
"name": "Jane Doe",
"address": { ... }
}
}
Response 200:
{
"id": "pi_3N2xYz...",
"object": "payment_intent",
"amount": 2000,
"amount_capturable": 0,
"amount_received": 2000,
"status": "succeeded", // requires_action / processing / succeeded / canceled
"currency": "usd",
"created": 1700000000,
"payment_method": "pm_1N2xYz...",
"client_secret": "pi_3N2xYz_...",
"next_action": null,
"last_payment_error": null
}
Create Refund
POST /v1/refunds
{
"payment_intent": "pi_3N2xYz...",
"amount": 1000, // optional, omit for full refund
"reason": "customer_requested", // duplicate / fraudulent / requested_by_customer
"metadata": {
"support_ticket": "ticket_789"
}
}
Create Customer & Payment Method
POST /v1/customers
{
"email": "customer@example.com",
"name": "Jane Doe",
"metadata": { "loyalty_tier": "gold" }
}
POST /v1/payment_methods
{
"type": "card",
"card": {
"number": "4242424242424242",
"exp_month": 12,
"exp_year": 2027,
"cvc": "123"
},
"billing_details": {
"name": "Jane Doe",
"email": "customer@example.com"
}
}
POST /v1/payment_methods/pm_1N2xYz.../attach
{
"customer": "cus_abc123"
}
Webhook Event
POST /merchant-webhook-endpoint
Content-Type: application/json
Stripe-Signature: t=1700000000,v1=abc123...
{
"id": "evt_3N2xYz...",
"object": "event",
"type": "payment_intent.payment_failed",
"data": {
"object": { ... }, // the full PaymentIntent object
"previous_attributes": {
"status": "processing",
"last_payment_error": null
}
},
"created": 1700000000,
"livemode": false
}
3.2 API Design Principles
| Principle | Implementation |
|---|---|
| Idempotency | All POST/PUT accept Idempotency-Key header; stored in idempotency table with TTL |
| Versioning | URL-based (/v1/), with Stripe-Version header for feature gating |
| Pagination | Cursor-based (starting_after / ending_before), never offset-based |
| Filtering | Query params for list endpoints (?status=succeeded&created[gte]=...) |
| Error Handling | Consistent error object with code, type, message, param fields |
| Rate Limiting | Per-merchant, per-endpoint; 429 with Retry-After header |
4. High-Level Architecture
flowchart TB
subgraph CLIENT["Client Layer"]
WEB["Web SDK<br/>(js/stripe.js)"]
MOBILE["Mobile SDKs<br/>(iOS/Android)"]
API["REST API Clients"]
end
subgraph EDGE["Edge Layer"]
CDN["CDN / WAF"]
LB["Global Load Balancer<br/>(GSLB)"]
APIGW["API Gateway<br/>(Rate Limiting / Auth)"]
end
subgraph CORE["Core Services"]
subgraph PAYMENT["Payment Processing"]
PI["PaymentIntent<br/>Service"]
PM["PaymentMethod<br/>Service"]
AUTH["Authorization<br/>Engine"]
CAPTURE["Capture Service"]
REFUND["Refund Service"]
end
subgraph RISK["Risk & Compliance"]
FRAUD["Fraud Detection<br/>(Radar)"]
VEL["Velocity<br/>Checker"]
PCI["PCI Tokenization<br/>Service"]
end
subgraph SETTLE["Settlement"]
FX["FX Conversion<br/>Service"]
BATCH["Batch Settlement<br/>Engine"]
RECON["Reconciliation<br/>Service"]
end
subgraph NOTIF["Notification"]
WEBHOOK["Webhook<br/>Dispatcher"]
EMAIL["Email<br/>Service"]
end
end
subgraph INTEGRATIONS["External Integrations"]
VISA["Visa Network"]
MC["Mastercard Network"]
ACQ["Acquiring Banks"]
ADYEN["Alternative Processors<br/>(Adyen, etc.)"]
end
subgraph DATA["Data Layer"]
TSDB["Time-Series DB<br/>(Metrics)"]
RDB["Primary DB<br/>(PostgreSQL)"]
REDIS["Redis Cluster<br/>(Cache / Sessions)"]
KAFKA["Kafka Cluster<br/>(Event Bus)"]
ES["Elasticsearch<br/>(Search / Logs)"]
S3["Object Storage<br/>(Audit / Archives)"]
end
WEB --> CDN
MOBILE --> CDN
API --> CDN
CDN --> LB
LB --> APIGW
APIGW --> PI
APIGW --> PM
APIGW --> FRAUD
APIGW --> REFUND
PI --> AUTH
PI --> CAPTURE
PI --> FRAUD
PI --> VEL
AUTH --> PCI
AUTH --> VISA
AUTH --> MC
AUTH --> ACQ
CAPTURE --> BATCH
REFUND --> ACQ
BATCH --> FX
BATCH --> RECON
PI --> KAFKA
PM --> KAFKA
REFUND --> KAFKA
KAFKA --> WEBHOOK
KAFKA --> EMAIL
KAFKA --> TSDB
KAFKA --> ES
PI --> RDB
PM --> RDB
AUTH --> RDB
REFUND --> RDB
BATCH --> RDB
RECON --> RDB
PI --> REDIS
FRAUD --> REDIS
VEL --> REDIS
RECON --> S3
ES --> S3
5. Core Component Design
5.1 PaymentIntent Service (Core Transaction Engine)
The PaymentIntent is the central abstraction — it represents the intent to collect payment, decoupling creation from confirmation.
State Machine
stateDiagram-v2
[*] --> RequiresPaymentMethod: create()
RequiresPaymentMethod --> RequiresConfirmation: attach payment method
RequiresConfirmation --> Processing: confirm()
Processing --> RequiresAction: 3DS / SCA challenge
RequiresAction --> Processing: customer completes action
Processing --> Succeeded: auth approved + auto-capture
Processing --> RequiresCapture: auth approved, manual capture
RequiresCapture --> Succeeded: capture()
Processing --> Canceled: cancel() / timeout
Succeeded --> Canceled: full refund
RequiresPaymentMethod --> Canceled: cancel()
RequiresConfirmation --> Canceled: cancel()
RequiresAction --> Canceled: cancel() / timeout
Idempotency Implementation
Idempotency Key Flow:
1. Client sends POST with Idempotency-Key: "key_abc123"
2. API Gateway extracts key, checks Redis:
a. Key exists + IN_PROGRESS → return 409 Conflict
b. Key exists + COMPLETED → return cached 200 response (exact replay)
c. Key not found → create new entry with IN_PROGRESS, pass through
3. After processing:
a. Success → update key status to COMPLETED, cache response body
b. Failure → update key status to FAILED, allow retry (key remains)
4. TTL: 25 hours (allows retries across day boundaries)
Storage: Redis (hot) + PostgreSQL (durability)
Key format: {merchant_id}:{endpoint}:{client_key}
5.2 Fraud Detection (Radar)
Multi-layer fraud prevention pipeline:
flowchart LR
subgraph LAYER1["Layer 1: Real-time Rules (< 50ms)"]
R1["Velocity Checks<br/>(cards/min, IPs/hour)"]
R2["Blocklists<br/>(cards, emails, IPs)"]
R3["BIN Validation"]
R4["AVS / CVV Check"]
end
subgraph LAYER2["Layer 2: ML Scoring (50-200ms)"]
ML1["Device Fingerprint<br/>Score"]
ML2["Behavioral<br/>Analysis"]
ML3["Graph-based<br/>Fraud Rings"]
ML4["Anomaly<br/>Detection"]
end
subgraph LAYER3["Layer 3: Post-Auth (async)"]
P1["Chargeback<br/>Prediction"]
P2["Network<br/>Analysis"]
end
REQ --> LAYER1
LAYER1 -->|Block| REJECT["Reject"]
LAYER1 -->|Pass| LAYER2
LAYER2 -->|High Risk| REVIEW["Manual Review<br/>(Hold)"]
LAYER2 -->|Low Risk| APPROVE["Approve"]
LAYER2 -->|Medium| REVIEW
LAYER2 --> LAYER3
Fraud Signals Catalog
| Signal | Source | Latency | Action |
|---|---|---|---|
| Card velocity > 5/min | Redis counter | < 5ms | Block |
| New card + high amount | Rule engine | < 10ms | Review |
| Device fingerprint mismatch | ML model | ~50ms | Score |
| IP-geo vs billing-geo mismatch | Rule engine | < 10ms | Score |
| Email in blocklist | Redis set | < 5ms | Block |
| Card BIN high-risk | Precomputed | < 5ms | Score |
| Graph: shared device, multiple cards | Graph DB | ~100ms | Score |
| ML ensemble fraud probability | XGBoost model | ~150ms | Score |
5.3 Authorization Engine
Handles the actual card network communication using ISO 8583 / VTS protocols.
sequenceDiagram
participant PI as PaymentIntent
participant AE as Auth Engine
participant PCI as PCI Tokenizer
participant FRAUD as Fraud Service
participant NET as Card Network
participant ISS as Issuing Bank
PI->>AE: authorize(amount, token, currency)
AE->>PCI: detokenize(token) → PAN + exp
PCI-->>AE: PAN (encrypted, in HSM)
AE->>FRAUD: score(transaction_context)
FRAUD-->>AE: risk_score + decision
alt risk_score > threshold
AE-->>PI: REJECT (fraud)
else
AE->>NET: ISO 8583 Auth Request
NET->>ISS: Forward Auth Request
ISS-->>NET: Auth Response (Approve/Decline)
NET-->>AE: Auth Response + Auth Code
AE->>AE: Record auth code, update state
AE-->>PI: APPROVED (auth_code, remaining)
end
Multi-Processor Routing
Routing Strategy:
1. Primary processor: Based on card BIN → best acquiring bank
2. Failover: If primary times out (> 3s), retry with secondary processor
3. Smart routing: A/B test processors for approval rate optimization
4. Cost optimization: Route to lowest-fee processor for same-card types
Processor Pool:
- Chase Paymentech (US cards)
- FIS / First Data (US cards)
- Worldpay (EU cards)
- Adyen (alternative rail)
- Each with independent connection pools
5.4 FX Conversion Service
FX Pipeline:
1. Merchant sets settlement currency (e.g., EUR)
2. Customer pays in transaction currency (e.g., JPY)
3. At auth time:
a. Fetch mid-market rate from 3 providers (Refinitiv, Bloomberg, ECB)
b. Take median rate
c. Apply merchant's FX margin (configured, default 1%)
d. Lock rate for 24 hours (rate lock table)
4. At settlement:
a. Use locked rate for conversion
b. If rate lock expired, re-fetch and re-lock
Rate Storage:
- Redis: Current rates (TTL 60s, updated every 10s)
- PostgreSQL: Historical rates (for reconciliation)
- Rate lock table: locked_rate_id, tx_id, rate, expires_at
5.5 Batch Settlement Engine
Settlement Pipeline (Daily):
Phase 1: Batch Creation (00:00 UTC)
└─ Query all captured, unsettled transactions from previous day
└─ Group by: acquiring bank, currency, settlement date
└─ Create settlement batch per group
Phase 2: File Generation (00:30 UTC)
└─ Generate NACHA (ACH), BACS (UK), SEPA (EU) files
└─ Generate card network settlement files
└─ Apply FX conversion for cross-currency batches
Phase 3: Submission (01:00 UTC)
└─ Submit files via SFTP/API to acquiring banks
└─ Track submission status per batch
Phase 4: Acknowledgment (01:00-06:00 UTC)
└─ Receive acknowledgment from banks
└─ Update batch status: SUBMITTED → ACKNOWLEDGED
Phase 5: Settlement (T+1 to T+2)
└─ Funds arrive in Stripe's operating account
└─ Trigger payout to merchants
Phase 6: Reconciliation (T+2)
└─ Match bank statements to internal records
└─ Flag discrepancies for investigation
5.6 Webhook Dispatcher
Guaranteed delivery with exponential backoff:
Delivery Guarantee: Best-effort with 3-day retry window
Per Event:
1. Publish to Kafka topic: webhook_events.{merchant_id}
2. Consumer picks up, looks up merchant's webhook endpoints
3. For each endpoint:
a. Send HTTP POST with event payload
b. Signature: HMAC-SHA256(timestamp + payload, webhook_secret)
c. On 2xx: Mark DELIVERED
d. On 4xx/5xx/timeout: Schedule retry
Retry Schedule:
Attempt 1: Immediate
Attempt 2: +5 minutes
Attempt 3: +30 minutes
Attempt 4: +2 hours
Attempt 5: +6 hours
Attempt 6: +18 hours
Attempt 7: +36 hours
Attempt 8: +54 hours
Attempt 9: +72 hours (final)
→ After 9 attempts: Mark FAILED, available in dashboard for manual replay
Deduplication:
- Events include unique event ID
- Merchants should check for duplicate event IDs
- Events are NOT guaranteed in-order (use created timestamp)
5.7 Reconciliation Service
Reconciliation Process:
Daily Reconciliation:
1. Pull bank settlement reports (T+2)
2. Pull card network settlement reports
3. Compare against internal transaction ledger:
Internal Ledger Bank Statement
┌─────────────────┐ ┌─────────────────┐
│ tx_001 $100.00 │ ?= │ SETTLE-001 $98.20│
│ tx_002 $200.00 │ ?= │ SETTLE-002 $196.40│
│ tx_003 $ 50.00 │ ?= │ SETTLE-003 $ 49.10│
└─────────────────┘ └─────────────────┘
↓ Mismatch Analysis
Difference = fees + FX spread + chargebacks
4. Discrepancy Handling:
a. Expected differences (fees, FX): Auto-reconcile
b. Unexpected differences: Create investigation ticket
c. Missing transactions: Retry settlement or escalate
5. Output:
a. Reconciliation report (stored in S3)
b. Dashboard summary
c. Alert on unreconciled amount > threshold
6. Data Model
6.1 Core Tables
-- Payment Intents (core transaction record)
CREATE TABLE payment_intents (
id VARCHAR(32) PRIMARY KEY, -- pi_3N2xYz...
merchant_id VARCHAR(32) NOT NULL,
customer_id VARCHAR(32),
amount BIGINT NOT NULL, -- in smallest currency unit
currency CHAR(3) NOT NULL, -- ISO 4217
status VARCHAR(20) NOT NULL, -- state machine status
payment_method VARCHAR(32),
capture_method VARCHAR(20) DEFAULT 'automatic',
client_secret VARCHAR(128),
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
completed_at TIMESTAMPTZ,
canceled_at TIMESTAMPTZ,
last_payment_error JSONB,
next_action JSONB,
metadata JSONB,
idempotency_key VARCHAR(255),
application_fee_amount BIGINT,
receipt_email VARCHAR(255),
shipping JSONB,
description TEXT,
statement_descriptor VARCHAR(22),
CONSTRAINT chk_status CHECK (status IN (
'requires_payment_method', 'requires_confirmation',
'requires_action', 'processing', 'requires_capture',
'succeeded', 'canceled'
))
);
CREATE INDEX pi_merchant_status ON payment_intents(merchant_id, status);
CREATE INDEX pi_customer ON payment_intents(customer_id);
CREATE INDEX pi_created ON payment_intents(created_at DESC);
CREATE INDEX pi_idempotency ON payment_intents(idempotency_key) WHERE idempotency_key IS NOT NULL;
-- Payment Methods (tokenized)
CREATE TABLE payment_methods (
id VARCHAR(32) PRIMARY KEY, -- pm_1N2xYz...
customer_id VARCHAR(32),
merchant_id VARCHAR(32) NOT NULL,
type VARCHAR(20) NOT NULL, -- card / bank_account / alipay
card_token VARCHAR(64), -- token in PCI vault, NOT raw PAN
card_brand VARCHAR(20), -- visa / mastercard
card_last4 CHAR(4),
card_exp_month SMALLINT,
card_exp_year SMALLINT,
card_fingerprint VARCHAR(64), -- dedup across customers
billing_details JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
is_default BOOLEAN DEFAULT false,
CONSTRAINT fk_customer FOREIGN KEY(customer_id) REFERENCES customers(id)
);
CREATE INDEX pm_customer ON payment_methods(customer_id);
CREATE INDEX pm_fingerprint ON payment_methods(card_fingerprint);
-- Customers
CREATE TABLE customers (
id VARCHAR(32) PRIMARY KEY, -- cus_abc123
merchant_id VARCHAR(32) NOT NULL,
email VARCHAR(255),
name VARCHAR(255),
phone VARCHAR(32),
description TEXT,
metadata JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
balance BIGINT DEFAULT 0, -- credit balance for refunds
CONSTRAINT uq_customer_email UNIQUE(merchant_id, email)
);
-- Transactions (immutable ledger)
CREATE TABLE transactions (
id BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
payment_intent_id VARCHAR(32) NOT NULL,
merchant_id VARCHAR(32) NOT NULL,
type VARCHAR(20) NOT NULL, -- auth / capture / refund / chargeback
amount BIGINT NOT NULL,
currency CHAR(3) NOT NULL,
status VARCHAR(20) NOT NULL, -- pending / succeeded / failed
auth_code VARCHAR(32),
processor VARCHAR(50), -- which acquiring bank
processor_ref VARCHAR(128), -- external reference
error_code VARCHAR(50),
error_message TEXT,
risk_score SMALLINT, -- 0-100
risk_decision VARCHAR(20), -- approve / decline / review
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
settled_at TIMESTAMPTZ,
settlement_batch_id VARCHAR(32),
CONSTRAINT fk_pi FOREIGN KEY(payment_intent_id) REFERENCES payment_intents(id)
);
CREATE INDEX tx_pi ON transactions(payment_intent_id);
CREATE INDEX tx_merchant_date ON transactions(merchant_id, created_at DESC);
CREATE INDEX tx_settlement ON transactions(settlement_batch_id) WHERE settlement_batch_id IS NOT NULL;
-- Idempotency Keys
CREATE TABLE idempotency_keys (
key VARCHAR(255) PRIMARY KEY, -- {merchant}:{endpoint}:{client_key}
merchant_id VARCHAR(32) NOT NULL,
endpoint VARCHAR(128) NOT NULL,
request_method VARCHAR(10) NOT NULL,
status VARCHAR(20) NOT NULL, -- in_progress / completed / failed
response_status SMALLINT, -- cached HTTP status
response_body JSONB, -- cached response for replay
request_headers JSONB, -- original request headers
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
expires_at TIMESTAMPTZ NOT NULL, -- 25 hours from creation
CONSTRAINT chk_idem_status CHECK (status IN ('in_progress', 'completed', 'failed'))
);
CREATE INDEX idem_expires ON idempotency_keys(expires_at);
-- Webhook Events
CREATE TABLE webhook_events (
id VARCHAR(32) PRIMARY KEY, -- evt_3N2xYz...
merchant_id VARCHAR(32) NOT NULL,
type VARCHAR(100) NOT NULL, -- payment_intent.succeeded
data JSONB NOT NULL, -- full object snapshot
previous_attrs JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
livemode BOOLEAN NOT NULL DEFAULT false,
pending_webhooks INTEGER NOT NULL DEFAULT 0,
delivered_count INTEGER NOT NULL DEFAULT 0
);
-- Webhook Deliveries
CREATE TABLE webhook_deliveries (
id BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
event_id VARCHAR(32) NOT NULL,
endpoint_id VARCHAR(32) NOT NULL,
status VARCHAR(20) NOT NULL, -- pending / success / failed
http_status SMALLINT,
response_body TEXT,
attempt_count SMALLINT DEFAULT 1,
next_retry_at TIMESTAMPTZ,
delivered_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT fk_event FOREIGN KEY(event_id) REFERENCES webhook_events(id)
);
CREATE INDEX wd_retry ON webhook_deliveries(next_retry_at)
WHERE status = 'pending' AND next_retry_at <= NOW();
-- Settlement Batches
CREATE TABLE settlement_batches (
id VARCHAR(32) PRIMARY KEY,
merchant_id VARCHAR(32) NOT NULL,
processor VARCHAR(50) NOT NULL,
currency CHAR(3) NOT NULL,
status VARCHAR(20) NOT NULL, -- created / submitted / acknowledged / settled / failed
total_amount BIGINT NOT NULL,
total_count INTEGER NOT NULL,
settlement_date DATE NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
submitted_at TIMESTAMPTZ,
settled_at TIMESTAMPTZ,
bank_ref VARCHAR(128),
reconciliation_status VARCHAR(20) DEFAULT 'pending',
CONSTRAINT chk_batch_status CHECK (status IN (
'created', 'submitted', 'acknowledged', 'settled', 'failed'
))
);
-- Refunds
CREATE TABLE refunds (
id VARCHAR(32) PRIMARY KEY, -- re_3N2xYz...
payment_intent_id VARCHAR(32) NOT NULL,
merchant_id VARCHAR(32) NOT NULL,
amount BIGINT NOT NULL,
currency CHAR(3) NOT NULL,
status VARCHAR(20) NOT NULL, -- pending / succeeded / failed / canceled
reason VARCHAR(50),
metadata JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
processed_at TIMESTAMPTZ,
CONSTRAINT fk_pi_refund FOREIGN KEY(payment_intent_id) REFERENCES payment_intents(id)
);
-- Chargebacks
CREATE TABLE chargebacks (
id VARCHAR(32) PRIMARY KEY, -- cb_3N2xYz...
payment_intent_id VARCHAR(32) NOT NULL,
merchant_id VARCHAR(32) NOT NULL,
amount BIGINT NOT NULL,
currency CHAR(3) NOT NULL,
status VARCHAR(20) NOT NULL, -- opened / warning / challenged / won / lost
reason VARCHAR(50), -- fraud / duplicate / product_not_received
network VARCHAR(20), -- visa / mastercard
network_ref VARCHAR(128),
evidence_due TIMESTAMPTZ,
evidence_submitted TIMESTAMPTZ,
decided_at TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT fk_pi_cb FOREIGN KEY(payment_intent_id) REFERENCES payment_intents(id)
);
6.2 Partitioning Strategy
Partitioning:
- payment_intents: Range partition by created_at (monthly partitions)
- transactions: Range partition by created_at (monthly partitions)
- webhook_events: Range partition by created_at (weekly partitions)
- webhook_deliveries: Range partition by created_at (weekly partitions)
Rationale:
- Hot data (last 90 days) on fast NVMe SSDs
- Older partitions can be moved to cheaper storage
- Partition pruning for time-range queries
- Easy DROP for data retention compliance
7. Key Design Decisions
7.1 PaymentIntent vs Direct Charge
| Aspect | PaymentIntent (Chosen) | Direct Charge |
|---|---|---|
| Separation of concerns | Create intent → confirm later (decoupled) | Single atomic call |
| 3DS/SCA support | Natural fit for redirect flows | Complex to handle |
| Retry handling | Confirm can be retried safely | Must re-create entire charge |
| Partial captures | Native support | Requires workarounds |
| Complexity | Higher API surface | Simpler |
| Flexibility | Authorization-only, manual capture | Limited |
Decision: PaymentIntent pattern provides essential flexibility for modern payment flows (3DS, SCA, marketplace splits) at the cost of API complexity. The abstraction pays dividends in extensibility.
7.2 Synchronous vs Asynchronous Processing
| Aspect | Synchronous (Chosen for Auth) | Asynchronous |
|---|---|---|
| Customer experience | Immediate feedback at checkout | “We’ll notify you” |
| Complexity | Simpler error handling | Polling / callback needed |
| Throughput | Limited by slowest path | Higher throughput |
| Timeouts | Must handle network timeouts | No client-side timeout |
Decision: Hybrid approach:
- Authorization: Synchronous (customer waiting at checkout, p99 < 300ms target)
- Settlement: Asynchronous (batch processing, T+1/T+2)
- Webhooks: Asynchronous (fire-and-forget with retry)
- Fraud scoring: Synchronous for Layer 1 (< 50ms), async for Layer 3
7.3 Database Choice: PostgreSQL vs NoSQL
| Aspect | PostgreSQL (Chosen) | MongoDB / Cassandra |
|---|---|---|
| ACID transactions | Native, battle-tested | Limited or eventual |
| Financial integrity | Strong consistency guaranteed | Eventual consistency risky |
| JSON support | JSONB with indexing | Native document model |
| Partitioning | Native declarative partitioning | Manual sharding |
| Complex queries | JOINs, window functions | Limited |
| Scale | Read replicas, Citus for sharding | Horizontal by default |
Decision: PostgreSQL for all financial records. The ACID guarantees are non-negotiable for payment processing. Use Citus extension for horizontal sharding when single-node limits are reached. Redis for caching, velocity checks, and idempotency.
7.4 Tokenization Strategy
Card Data Flow:
1. Client → stripe.js → Stripe Servers (card number NEVER touches merchant server)
2. Stripe → PCI Vault → Token (pm_1N2xYz...)
3. Token returned to merchant
4. Merchant stores token, never sees raw PAN
5. At auth time: Auth Engine → PCI Vault (via HSM) → detokenize → PAN
6. PAN used only within HSM boundary for card network communication
PCI Scope Reduction:
- Tokenization reduces merchant PCI scope to SAQ A (lowest)
- Raw card data only exists in Stripe's PCI Level 1 environment
- HSM (Hardware Security Module) for key management
- PAN never logged, never in plaintext at rest
7.5 Event-Driven Architecture with Kafka
Why Kafka over alternatives:
Kafka vs RabbitMQ:
✓ Higher throughput (millions of msg/sec)
✓ Persistent log (replay events for debugging)
✓ Consumer groups (scale consumers independently)
✓ Exactly-once semantics (with idempotent producers)
Kafka vs Pub/Sub:
✓ Ordered delivery per partition
✓ Replay capability
✓ Better latency control
Topic Design:
- payment.events.{merchant_id} -- per-merchant partitioning
- webhook.dispatch.{region} -- regional dispatch
- fraud.signals -- fraud feature pipeline
- settlement.batch.{processor} -- settlement pipeline
- metrics.{service} -- observability
Retention:
- Financial events: 7 years (compliance)
- Metrics: 90 days
- Fraud signals: 30 days
8. Scalability Strategy
8.1 Horizontal Scaling Layers
flowchart TB
subgraph L1["Layer 1: Stateless Services (Infinite Scale)"]
S1["API Gateway instances"]
S2["PaymentIntent Service instances"]
S3["Fraud Service instances"]
end
subgraph L2["Layer 2: Cache (Linear Scale)"]
C1["Redis Cluster<br/>(12 shards, 3 replicas)"]
end
subgraph L3["Layer 3: Database (Sharded Scale)"]
DB1["PostgreSQL Primary<br/>(merchant_id hash shard)"]
DB2["Read Replica 1"]
DB3["Read Replica 2"]
DB4["Read Replica 3"]
end
subgraph L4["Layer 4: Event Bus (Partition Scale)"]
K1["Kafka Cluster<br/>(24 brokers, 3x replication)"]
end
L1 --> C1
L1 --> L3
L1 --> L4
8.2 Sharding Strategy
Shard Key: merchant_id (consistent hash)
Benefits:
- All data for a merchant co-located
- Cross-merchant queries rare (mostly admin/analytics)
- Natural isolation between tenants
- Easy to add/remove shards
Shard Count:
- Start: 4 shards (4 PostgreSQL nodes)
- Growth: 16 → 64 → 256 shards
- Rebalancing: Virtual shards (vshards) mapped to physical shards
- 10,000 vshards → map to N physical shards
- Rebalance by remapping vshard ranges, not moving data
Read Scaling:
- Primary: All writes
- Replicas: Read-only queries (dashboard, reporting)
- Read/write split at application layer
- Async replication (lag < 1s acceptable for reads)
8.3 Multi-Region Strategy
Active-Active Multi-Region:
Region A (US-East) Region B (EU-West) Region C (APAC)
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ API Gateway │ │ API Gateway │ │ API Gateway │
│ Services │ │ Services │ │ Services │
│ Redis Cluster │ │ Redis Cluster │ │ Redis Cluster │
│ Kafka Cluster │ │ Kafka Cluster │ │ Kafka Cluster │
└────────┬────────┘ └────────┬────────┘ └────────┬────────┘
│ │ │
└──────────┬───────────────┴──────────────────────────┘
│
Global Consensus
(merchant_id → region assignment)
Merchant Region Assignment:
- Merchant assigned to primary region at onboarding
- Cross-region requests routed via GSLB
- Data replicated cross-region for DR (not for active-active reads)
- Idempotency keys replicated cross-region to prevent duplicates
Card Network Connections:
- Each region has independent connections to card networks
- Regional routing reduces latency to issuing banks
8.4 Auto-Scaling Triggers
| Metric | Warning | Scale Up | Scale Down |
|---|---|---|---|
| CPU Utilization | > 60% | > 75% for 2min | < 30% for 5min |
| Request Latency p99 | > 200ms | > 250ms for 1min | < 100ms for 5min |
| Queue Depth (Kafka) | > 10K msgs | > 50K msgs | < 1K msgs |
| Error Rate | > 0.1% | > 0.5% | < 0.01% |
| Connection Pool | > 70% used | > 85% used | < 40% used |
9. Curveball Scenarios
9.1 Curveball 1: “A merchant’s acquiring bank goes down for 6 hours during Black Friday. How do you handle it?”
Impact Assessment:
- Merchant cannot process new authorizations through primary processor
- In-flight authorizations may timeout
- Customer-facing checkout failures spike
Response Strategy:
Phase 1: Detection (0-2 minutes)
- Health checks detect processor unresponsiveness (3 consecutive failures)
- Circuit breaker opens for that processor
- Alert fired to SRE team + affected merchants
Phase 2: Failover (2-5 minutes)
- Smart router switches to secondary processor for affected card BINs
- Secondary processor may have different approval rates / fees
- Merchant dashboard shows "using backup processor" notification
- In-flight requests: retry with secondary processor (idempotency keys prevent duplicates)
Phase 3: Communication (5-15 minutes)
- Automated email to affected merchants
- Dashboard banner with status updates
- API returns processor-specific error codes for client-side handling
Phase 4: Recovery (6+ hours)
- Primary processor health check passes (5 consecutive successes)
- Circuit breaker half-opens, gradual traffic shift back
- Reconciliation catches any transactions lost during failover
- Post-incident report with timeline and metrics
Prevention:
- Multi-processor strategy (never single point of failure)
- Processor health dashboard with SLA tracking
- Regular failover drills (quarterly)
- Processor diversity across different providers
9.2 Curveball 2: “You discover a bug that double-charged 2,000 customers over the past 48 hours. How do you respond?”
Impact Assessment:
- 2,000 customers overcharged (reputation damage)
- Potential chargeback wave incoming
- Regulatory notification may be required
Response Strategy:
Phase 1: Containment (0-30 minutes)
1. IMMEDIATELY: Disable the buggy code path (feature flag kill switch)
2. Verify: Query transactions table for duplicate charges:
SELECT payment_intent_id, COUNT(*) as cnt
FROM transactions
WHERE type = 'auth'
AND created_at >= NOW() - INTERVAL '48 hours'
GROUP BY payment_intent_id
HAVING COUNT(*) > 1;
3. Scope: Identify exact affected transactions, merchants, amounts
Phase 2: Remediation (30 min - 4 hours)
1. Automated refund batch:
- Generate refund for each duplicate charge
- Use internal admin API (bypass normal limits)
- Mark refunds with reason: "system_error_duplicate_charge"
- Total refund amount: sum of duplicate charges
2. Customer notifications:
- Email: "We noticed a billing error and have issued a refund"
- In-app notification for logged-in users
- Proactive communication prevents chargebacks
Phase 3: Verification (4-8 hours)
1. Confirm all refunds processed successfully
2. Verify no remaining duplicates in system
3. Check merchant settlement adjustments
4. Audit trail: Document every action taken
Phase 4: Root Cause & Prevention (1-3 days)
1. Root cause analysis: Why did idempotency fail?
- Likely: Missing idempotency key on internal retry path
- Or: Race condition in state machine transition
2. Fix: Add idempotency to internal paths, add state machine guard
3. Tests: Add regression test for exact scenario
4. Monitoring: Add alert for duplicate charge detection (real-time)
5. Process: Add pre-deployment checklist for payment path changes
Key Principle: Refund BEFORE customers notice. Proactive refunds cost fees.
Reactive chargebacks cost 3x fees + reputation damage.
9.3 Curveball 3: “A sophisticated attacker is using stolen card data with spoofed device fingerprints. Your ML model has 85% accuracy but they’re slipping through. What do you do?”
Impact Assessment:
- Fraud losses increasing
- Legitimate transactions may be impacted by over-tightening
- Attacker adapting to countermeasures
Response Strategy:
Phase 1: Immediate Defense (0-1 hour)
1. Tighten velocity thresholds temporarily:
- Cards: 3/min → 2/min
- IPs: 10/hour → 5/hour
- New cards + amount > $500 → manual review
2. Enable step-up authentication:
- Require 3DS for transactions > $100 from new devices
- Accept temporary friction for security
3. Block high-risk BIN ranges identified in attack pattern
Phase 2: Investigation (1-4 hours)
1. Pattern analysis:
- Common attributes of fraudulent transactions
- Time patterns, amount patterns, merchant patterns
- Device fingerprint analysis (are they truly spoofed or new?)
2. Graph analysis:
- Build device → card → IP → email graph
- Identify fraud rings (connected components)
- Block entire rings, not just individual cards
3. Issuer collaboration:
- Share patterns with card networks
- Request issuer-side blocks on confirmed stolen cards
Phase 3: Model Improvement (4-24 hours)
1. Feature engineering:
- Add new features: mouse movement entropy, typing cadence
- Device fingerprint version detection (spoofed vs genuine)
- Behavioral biometrics from Stripe.js
2. Model retraining:
- Retrain with new fraud examples (online learning)
- Ensemble: combine XGBoost + neural net + rule engine
- Calibrate thresholds per merchant risk tolerance
3. A/B test new model:
- 10% traffic to new model
- Compare: fraud catch rate vs false positive rate
- Full rollout if improvement confirmed
Phase 4: Long-term Defense (1-2 weeks)
1. Machine learning ops:
- Continuous model retraining pipeline
- Automated feature importance monitoring
- Drift detection for model performance
2. Zero-trust architecture:
- Never trust client-side signals alone
- Server-side device attestation
- Behavioral analysis over session lifetime
3. Industry collaboration:
- Share threat intelligence with other processors
- Participate in fraud information sharing groups
9.4 Curveball 4: “Regulatory change: EU requires all payment data to reside in EU data centers. How do you comply without breaking global merchants?”
Response Strategy:
Data Residency Architecture:
Option A: Regional Data Isolation (Chosen)
┌─────────────────────────────────────────────┐
│ EU Region (Frankfurt) │
│ ┌───────────┐ ┌───────────┐ ┌──────────┐ │
│ │ EU Merchants│ │ EU Data │ │ EU Kafka │ │
│ │ DB Shard │ │ Vault │ │ Cluster │ │
│ └───────────┘ └───────────┘ └──────────┘ │
└─────────────────────────────────────────────┘
│ No cross-border payment data │
┌─────────────────────────────────────────────┐
│ US Region (Virginia) │
│ ┌───────────┐ ┌───────────┐ ┌──────────┐ │
│ │ US Merchants│ │ US Data │ │ US Kafka │ │
│ │ DB Shard │ │ Vault │ │ Cluster │ │
│ └───────────┘ └───────────┘ └──────────┘ │
└─────────────────────────────────────────────┘
Implementation:
1. Merchant onboarding: Detect region from registration
2. Data routing: merchant_id hash → regional shard
3. Cross-border transactions:
- EU customer paying US merchant: Process in EU, settle cross-border
- Transaction metadata stays in-region
- Settlement files generated per region
4. Analytics: Regional aggregation, no cross-border raw data
5. Support: Regional support teams with regional data access
Compliance Verification:
- Automated data residency audits (monthly)
- Penetration testing for cross-region data leakage
- Legal review of data flow diagrams
- Customer-facing compliance documentation
9.5 Curveball 5: “Your Kafka cluster loses a majority of brokers in one AZ. Events are piling up. What’s your playbook?”
Response Strategy:
Kafka AZ Failure Playbook:
Detection:
- Kafka broker health checks fail (3 brokers down)
- Producer errors spike (NOT_LEADER_FOR_PARTITION)
- Consumer lag increases rapidly
- Alert: KAFKA_CRITICAL - AZ failure detected
Immediate Actions (0-5 min):
1. Verify: Is it a full AZ failure or partial?
2. Check: Remaining brokers have leader replicas?
- With 3x replication, 1 AZ failure = still operational
- With 2 AZ failure = UNDER_REPLICATED, data at risk
3. If operational: Continue processing, start broker recovery
4. If degraded:
a. Reduce producer rate (throttle non-critical events)
b. Prioritize: payment events > webhook events > metrics
c. Enable producer buffering (increase linger.ms, batch.size)
Service-Level Adaptations:
- Payment processing: Continue (synchronous, doesn't depend on Kafka)
- Webhook delivery: Queue locally in Redis, deliver when Kafka recovers
- Fraud scoring Layer 3: Disable (async, non-blocking)
- Metrics: Buffer in memory, flush when available
Recovery (5-60 min):
1. Start replacement brokers in healthy AZ
2. Kafka auto-rebalances leaders
3. Verify replication catches up
4. Gradually restore throttled services
5. Replay any lost events from service-local buffers
Prevention:
- Rack-aware replication (brokers spread across 3 AZs)
- min.insync.replicas = 2 (ack safety)
- Regular AZ failure drills
- Cross-AZ producer failover configuration
10. Failure Modes and Mitigations
10.1 Failure Mode Matrix
| Failure Mode | Probability | Impact | Detection | Mitigation | RTO | RPO |
|---|---|---|---|---|---|---|
| Single service instance crash | High | Low | Health checks, metrics | Auto-replace, stateless design | < 30s | 0 |
| Database primary failure | Medium | Critical | Connection errors, lag | Automated failover to replica | < 60s | < 1s (WAL) |
| Redis cluster partition | Medium | High | Cache miss spike | Local fallback, re-cache on write | < 30s | Lost cache only |
| Kafka broker failure | Medium | Medium | Producer errors | Replication, producer retry | < 5min | 0 (persisted) |
| Card network outage | Low | Critical | Auth timeout spike | Multi-processor failover | N/A | 0 |
| Region failure | Very Low | Critical | GSLB health checks | Cross-region failover | < 5min | < 1min |
| Data corruption | Very Low | Critical | Checksum mismatch | 3x replication, regular backups | Hours | Last backup |
| DDoS attack | Medium | High | Traffic anomaly | WAF, rate limiting, scrubbing | < 1min | 0 |
| Credential compromise | Low | Critical | Anomalous access patterns | Rotation, MFA, audit alerts | Minutes | 0 |
| FX rate service failure | Low | Medium | Rate fetch errors | Cached rates, fallback provider | < 5min | Last known rate |
10.2 Detailed Mitigation Strategies
Database Primary Failure
Automated Failover Process:
1. Detection (0-5s):
- Patroni / PostgreSQL HA detects primary unresponsive
- 3 consecutive health check failures
2. Promotion (5-30s):
- Select most up-to-date replica (by WAL position)
- Promote to primary (pg_promote)
- Update connection pool DNS / VIP
3. Reconnection (30-60s):
- Application connection pools detect failure
- Reconnect to new primary
- In-flight transactions: rollback, client retries with idempotency keys
4. Recovery (1-5min):
- Old primary (when recovered): demote to replica
- Catch up via WAL replay
- Verify data consistency
Prevention:
- Patroni for automated HA
- Synchronous replication to at least 1 replica
- Regular failover drills (monthly)
- Connection pool with automatic retry
DDoS Attack Mitigation
Defense-in-Depth:
Layer 1: Network (Cloud Provider)
- DDoS protection service (AWS Shield / GCP Armor)
- Traffic scrubbing before reaching our infrastructure
- Volumetric attack absorption
Layer 2: WAF (Edge)
- Rate limiting per IP: 100 req/min
- Rate limiting per merchant: 10,000 req/min
- Geo-blocking for suspicious regions
- Bot detection and challenge
Layer 3: API Gateway
- Per-endpoint rate limiting
- Request size limits
- Connection limits per client
- Circuit breaker for downstream services
Layer 4: Application
- Request queuing with backpressure
- Graceful degradation (disable non-critical features)
- Priority queuing (payment auth > dashboard reads)
Layer 5: Database
- Connection pooling limits
- Query timeout enforcement
- Read replica offloading
Consistency Guarantees
Financial Consistency Model:
Strong Consistency (Required):
- PaymentIntent state transitions (single record, row-level lock)
- Transaction ledger (append-only, never modify/delete)
- Idempotency keys (atomic check-and-set)
- Refund amount <= captured amount (constraint check)
Eventual Consistency (Acceptable):
- Dashboard metrics (count, sum aggregations)
- Search indexes (Elasticsearch)
- Webhook delivery status
- Fraud feature store updates
Consistency Enforcement:
1. Database constraints (CHECK, FOREIGN KEY, UNIQUE)
2. Application-level state machine guards
3. Idempotency keys for all mutating operations
4. Double-entry bookkeeping for financial ledger
5. Daily reconciliation as safety net
6. Audit log for every state change
10.3 Monitoring and Alerting
Critical Alerts (Page On-Call):
- Payment success rate < 95% (5-min window)
- Auth latency p99 > 1s
- Database primary failover triggered
- Kafka consumer lag > 1M messages
- Error rate > 1% on payment endpoints
Warning Alerts (Ticket):
- Payment success rate < 98%
- Auth latency p99 > 500ms
- Redis memory > 80%
- Disk usage > 75%
- Certificate expiry < 30 days
Info Alerts (Dashboard):
- Transaction volume anomalies
- New error codes appearing
- Processor approval rate changes
- FX rate volatility
Appendix A: Glossary
| Term | Definition |
|---|---|
| PaymentIntent | Stripe’s core abstraction representing intent to collect payment |
| Idempotency Key | Client-provided key ensuring exactly-once processing |
| PCI DSS | Payment Card Industry Data Security Standard |
| PAN | Primary Account Number (full card number) |
| Tokenization | Replacing sensitive data with a non-sensitive equivalent |
| 3DS | 3-D Secure, card authentication protocol |
| SCA | Strong Customer Authentication (EU PSD2 requirement) |
| AVS | Address Verification Service |
| BIN | Bank Identification Number (first 6 digits of card) |
| ISO 8583 | Financial transaction card-based message standard |
| Chargeback | Forced refund initiated by cardholder’s bank |
| Dunning | Process of handling failed recurring payments |
| Proration | Partial billing for mid-cycle changes |
| GMV | Gross Merchandise Volume |
| HSM | Hardware Security Module |
Appendix B: Key Metrics Dashboard
| Metric | Target | Measurement |
|---|---|---|
| Payment Success Rate | > 99.5% | Successful auths / Total auth attempts |
| Auth Latency p50 | < 100ms | Server-side timing |
| Auth Latency p99 | < 300ms | Server-side timing |
| Webhook Delivery Rate | > 99.9% | Delivered / Total events (within 3 days) |
| Fraud Catch Rate | > 95% | True positives / (True positives + False negatives) |
| False Positive Rate | < 0.1% | False positives / Total legitimate transactions |
| Settlement Accuracy | 100% | Reconciled amount / Expected amount |
| System Availability | 99.99% | Uptime / Total time |
| Mean Time to Recovery | < 5 min | From detection to full recovery |