Design a Subscription Management System like Adobe
Table of Contents
- Requirements Clarification
- Back-of-Envelope Calculations
- System Interface Design
- High-Level Architecture
- Core Component Design
- Data Model
- Key Design Decisions
- Scalability Strategy
- Curveball Scenarios
- Failure Modes and Mitigations
1. Requirements Clarification
1.1 Functional Requirements
- Subscription Lifecycle Management: Create, activate, pause, resume, cancel, and reactivate subscriptions with proper state transitions
- Flexible Billing Cycles: Support monthly, annual, quarterly, and custom intervals with anchor date management
- Plan Management: Create and manage product plans with features, pricing tiers, and metadata
- Upgrades & Downgrades: Mid-cycle plan changes with proration calculations (credit/debit)
- Proration Engine: Accurate prorated charges/credits for mid-cycle changes, partial refunds
- Dunning Management: Automated retry sequences for failed payments with customer notifications
- Entitlement Management: Real-time access control based on active subscriptions and plan features
- Trial Periods: Configurable free trials with automatic conversion or expiration
- Coupons & Discounts: Percentage/fixed discounts, first-month-free, lifetime deals, stackable promotions
- Invoicing: Generate itemized invoices with tax calculations, PDF delivery, and payment tracking
- Usage-Based Billing: Metered billing for overage charges (API calls, storage, seats)
- Multi-Seat / Team Plans: Per-seat pricing with add/remove seat proration
- Self-Service Portal: Customer-facing dashboard for managing subscriptions, payment methods, and invoices
- Analytics & Reporting: MRR/ARR tracking, churn analysis, cohort reports, LTV calculations
- Tax Compliance: Automated tax calculation (Sales Tax, VAT, GST) with jurisdiction-aware rates
1.2 Non-Functional Requirements
| Category | Requirement | Rationale |
|---|---|---|
| Availability | 99.95% uptime (4.38 hours downtime/year) | Subscription access = product access |
| Consistency | Strong consistency for billing records | No over/under-charging |
| Accuracy | 100% billing accuracy | Financial integrity, customer trust |
| Latency | Entitlement check < 50ms p99 | Product access gate, must be fast |
| Throughput | 1M subscription events/day | Scale for global subscriber base |
| Data Retention | 7+ years for billing records | Tax and audit compliance |
| Multi-Tenancy | Support enterprise customers with custom contracts | B2B2C scenarios |
| Localization | Multi-currency, multi-language invoices | Global customer base |
| Compliance | GDPR, CCPA, SOC 2 | Data privacy and security |
2. Back-of-Envelope Calculations
2.1 Assumption Sets
| Parameter | Startup (Year 1) | Growth (Year 3) | Adobe-scale |
|---|---|---|---|
| Active Subscribers | 50K | 5M | 25M+ |
| Monthly Churn Rate | 5% | 3% | 2% |
| New Subs/Month | 5K | 500K | 2.5M |
| Billing Events/Month | 60K | 6M | 30M |
| Entitlement Checks/Day | 500K | 50M | 250M |
| Avg Revenue/Subscriber | $20/mo | $20/mo | $20/mo |
| MRR | $1M | $100M | $500M |
2.2 Storage Estimates (Adobe-scale)
Per Subscriber:
- Customer profile: ~1 KB
- Active subscription: ~2 KB
- Invoice history (avg 24 invoices): ~5 KB × 24 = 120 KB
- Payment history (avg 24 payments): ~3 KB × 24 = 72 KB
- Event log (avg 50 events): ~500 B × 50 = 25 KB
- Usage records (monthly): ~10 KB
- Total per subscriber: ~230 KB
Active subscribers: 25M × 230 KB = 5.75 TB
Historical (7 years, 50M total ever): 50M × 500 KB = 25 TB
With 3x replication: 75 TB raw storage
Hot data (active + 1 year): ~10 TB
Warm data (1-3 years): ~20 TB
Cold data (3-7 years): ~45 TB → archive to object storage
2.3 QPS Estimates
Entitlement Checks (product access gate):
Average: 250M / 86,400s = ~2,900 QPS
Peak (3x): ~8,700 QPS
Must be < 50ms p99 → heavy caching required
Billing Events:
Average: 30M / 30 days / 86,400s = ~11.6 QPS (spread out)
Billing Day Peak: 30M / 8 hours / 3,600s = ~1,000 QPS
Concentrated billing windows create spikes
API Calls (self-service portal):
Average: ~500 QPS
Peak: ~2,000 QPS
Total System QPS:
Average: ~3,500 QPS
Peak: ~12,000 QPS
2.4 Bandwidth Estimates
Entitlement API:
Request: ~500 B × 250M/day = 125 GB/day
Response: ~1 KB × 250M/day = 250 GB/day
Billing API:
Request: ~2 KB × 30M/day = 60 GB/day
Response: ~5 KB × 30M/day = 150 GB/day
Invoice PDFs:
~200 KB × 30M/month = 6 TB/month (generated on demand, cached)
Webhooks:
~3 events/sub/month × 25M = 75M events/month
~1 KB/event = 75 GB/month
Total: ~400 GB/day API traffic + 6 TB/month PDFs
3. System Interface Design
3.1 Subscription Management API
Create Subscription
POST /v1/subscriptions
Content-Type: application/json
Authorization: Bearer <api_key>
{
"customer_id": "cus_abc123",
"plan_id": "plan_photoshop_annual",
"quantity": 1, // seats
"billing_cycle": "annual", // monthly / annual / quarterly
"trial_period_days": 7, // optional
"coupon_code": "WELCOME20", // optional
"payment_method_id": "pm_1N2xYz...",
"metadata": {
"channel": "web_direct",
"campaign": "summer_2026"
},
"proration_behavior": "create_prorations",
"backdate_start": "2026-07-01" // optional, for enterprise
}
Response 201:
{
"id": "sub_3N2xYz...",
"object": "subscription",
"customer_id": "cus_abc123",
"status": "trialing", // trialing / active / past_due / canceled / paused
"plan_id": "plan_photoshop_annual",
"plan": {
"id": "plan_photoshop_annual",
"name": "Photoshop - Annual",
"amount": 23988, // cents ($239.988 = €22.99 converted)
"currency": "usd",
"interval": "year",
"features": ["photoshop_desktop", "photoshop_web", "cloud_storage_100gb"]
},
"current_period_start": "2026-07-01T00:00:00Z",
"current_period_end": "2027-07-01T00:00:00Z",
"trial_end": "2026-07-08T00:00:00Z",
"cancel_at_period_end": false,
"cancel_at": null,
"canceled_at": null,
"paused_at": null,
"quantity": 1,
"started_at": "2026-07-01T00:00:00Z",
"billing_cycle_anchor": "2026-07-01T00:00:00Z",
"days_until_due": 30,
"collection_method": "charge_automatically", // or send_invoice
"metadata": { ... }
}
Update Subscription (Upgrade/Downgrade)
PATCH /v1/subscriptions/sub_3N2xYz...
{
"plan_id": "plan_creative_cloud_all", // upgrade from single app
"proration_behavior": "create_prorations", // create_invoice_now / always_invoice / none
"proration_date": "2026-08-15T00:00:00Z", // optional, defaults to now
"quantity": 3, // add seats
"metadata": {
"change_reason": "team_expansion"
}
}
Response 200:
{
"id": "sub_3N2xYz...",
"status": "active",
"plan_id": "plan_creative_cloud_all",
"pending_update": null,
"prorations": [
{
"id": "prorate_001",
"type": "credit",
"amount": 5432, // credit for unused Photoshop time
"description": "Proration credit: Photoshop Annual remaining 289 days",
"line_item": "plan_photoshop_annual"
},
{
"id": "prorate_002",
"type": "charge",
"amount": 12500, // charge for new plan remaining time
"description": "Proration charge: Creative Cloud All remaining 289 days",
"line_item": "plan_creative_cloud_all"
},
{
"id": "prorate_003",
"type": "charge",
"amount": 25000, // additional seats
"description": "Quantity change: +2 seats × $83.33/mo × 289 days",
"line_item": "plan_creative_cloud_all_seats"
}
],
"net_proration": 33368, // net charge this billing cycle
"current_period_end": "2027-07-01T00:00:00Z" // anchor preserved
}
Cancel Subscription
POST /v1/subscriptions/sub_3N2xYz.../cancel
{
"cancel_at": "now", // "now" or "period_end"
"reason": "too_expensive", // optional
"feedback": "Found a cheaper alternative",
"refund_proration": true // refund unused time?
}
Response 200:
{
"id": "sub_3N2xYz...",
"status": "canceled",
"canceled_at": "2026-08-15T10:30:00Z",
"cancel_at_period_end": false,
"refund": {
"id": "re_abc123",
"amount": 4500,
"description": "Proration refund for unused 289 days"
}
}
3.2 Entitlement API (Product Access Gate)
GET /v1/entitlements/check
Authorization: Bearer <customer_token>
X-Customer-ID: cus_abc123
Response 200:
{
"customer_id": "cus_abc123",
"entitled": true,
"subscriptions": [
{
"id": "sub_3N2xYz...",
"status": "active",
"plan_id": "plan_creative_cloud_all",
"features": [
"photoshop_desktop",
"photoshop_web",
"illustrator_desktop",
"indesign_desktop",
"cloud_storage_100gb",
"fonts_access",
"behance_pro"
],
"seats_remaining": 2,
"period_end": "2027-07-01T00:00:00Z"
}
],
"blocked_features": [],
"warnings": [
{
"code": "payment_past_due",
"message": "Payment failed. Grace period ends 2026-08-25.",
"grace_period_end": "2026-08-25T00:00:00Z"
}
],
"cached_until": "2026-08-15T10:35:00Z" // 5-minute cache TTL
}
3.3 Usage Reporting API
POST /v1/usage/records
{
"subscription_id": "sub_3N2xYz...",
"metered_feature": "api_calls",
"quantity": 15000,
"timestamp": "2026-08-15T10:30:00Z",
"action": "increment" // increment / set
}
GET /v1/usage/summary?subscription_id=sub_3N2xYz...&period=current
Response 200:
{
"subscription_id": "sub_3N2xYz...",
"period_start": "2026-08-01T00:00:00Z",
"period_end": "2026-08-31T00:00:00Z",
"meters": [
{
"feature": "api_calls",
"current_usage": 850000,
"included_quantity": 1000000,
"overage_quantity": 0,
"overage_charge": 0,
"unit_amount": 0.001 // $0.001 per call over limit
},
{
"feature": "cloud_storage_gb",
"current_usage": 75,
"included_quantity": 100,
"overage_quantity": 0,
"overage_charge": 0,
"unit_amount": 0.10 // $0.10 per GB over limit
}
]
}
3.4 Webhook Events
Event Types:
subscription.created
subscription.activated
subscription.updated
subscription.paused
subscription.resumed
subscription.canceling // cancel_at_period_end = true
subscription.canceled
subscription.reactivated
subscription.trial_ending // 3 days before trial ends
invoice.created
invoice.payment_succeeded
invoice.payment_failed
invoice.payment_overdue
dunning.retry_scheduled
dunning.max_retries_exceeded
entitlement.changed
customer.payment_method_expiring
usage.threshold_reached // 80%, 100%, 120% of included
4. High-Level Architecture
flowchart TB
subgraph CLIENT["Client Layer"]
WEB["Web Portal<br/>(Self-Service)"]
APP["Creative Cloud<br/>Desktop App"]
API["REST API Clients"]
PRODUCT["Product Services<br/>(Photoshop, etc.)"]
end
subgraph EDGE["Edge Layer"]
CDN["CDN / WAF"]
LB["Load Balancer"]
GW["API Gateway"]
end
subgraph CORE["Core Services"]
subgraph SUB["Subscription Management"]
SUB_MGR["Subscription<br/>Lifecycle Service"]
PLAN_MGR["Plan & Pricing<br/>Service"]
PRORATE["Proration<br/>Engine"]
TRIAL["Trial<br/>Manager"]
end
subgraph BILL["Billing & Payments"]
BILLING["Billing Cycle<br/>Engine"]
INVOICE["Invoice<br/>Generator"]
PAYMENT["Payment<br/>Processor"]
DUNNING["Dunning<br/>Manager"]
TAX["Tax Calculation<br/>Service"]
end
subgraph ENT["Entitlement & Access"]
ENTITLE["Entitlement<br/>Service"]
FEATURE["Feature Flag<br/>Service"]
ACCESS["Access Control<br/>(Policy Engine)"]
end
subgraph USAGE["Usage & Metering"]
METER["Usage Metering<br/>Service"]
OVERAGE["Overage<br/>Calculator"]
end
subgraph PROMO["Promotions"]
COUPON["Coupon & Discount<br/>Engine"]
PROMO_VALID["Promo<br/>Validator"]
end
end
subgraph NOTIF["Notification"]
EMAIL["Email Service<br/>(Transactional)"]
PUSH["Push Notifications"]
WEBHOOK["Webhook<br/>Dispatcher"]
end
subgraph DATA["Data Layer"]
RDB["Primary DB<br/>(PostgreSQL)"]
REDIS["Redis Cluster<br/>(Cache / Entitlements)"]
KAFKA["Kafka Cluster<br/>(Event Bus)"]
ES["Elasticsearch<br/>(Search / Analytics)"]
S3["Object Storage<br/>(Invoices / PDFs)"]
TSDB["Time-Series DB<br/>(Usage Metrics)"]
end
subgraph ANALYTICS["Analytics"]
BI["BI Dashboard<br/>(MRR, Churn, LTV)"]
COHORT["Cohort Analysis"]
PREDICT["Churn Prediction<br/>(ML)"]
end
WEB --> CDN
APP --> CDN
API --> CDN
PRODUCT --> CDN
CDN --> LB
LB --> GW
GW --> SUB_MGR
GW --> PLAN_MGR
GW --> ENTITLE
GW --> METER
GW --> BILLING
SUB_MGR --> PRORATE
SUB_MGR --> TRIAL
SUB_MGR --> PLAN_MGR
BILLING --> INVOICE
BILLING --> PAYMENT
BILLING --> DUNNING
BILLING --> TAX
BILLING --> COUPON
ENTITLE --> FEATURE
ENTITLE --> ACCESS
METER --> OVERAGE
OVERAGE --> BILLING
SUB_MGR --> KAFKA
BILLING --> KAFKA
DUNNING --> KAFKA
ENTITLE --> KAFKA
METER --> KAFKA
KAFKA --> EMAIL
KAFKA --> PUSH
KAFKA --> WEBHOOK
KAFKA --> ES
KAFKA --> TSDB
KAFKA --> BI
SUB_MGR --> RDB
PLAN_MGR --> RDB
BILLING --> RDB
INVOICE --> RDB
PAYMENT --> RDB
DUNNING --> RDB
METER --> RDB
ENTITLE --> REDIS
FEATURE --> REDIS
SUB_MGR --> REDIS
INVOICE --> S3
BI --> ES
5. Core Component Design
5.1 Subscription Lifecycle Service
Central state machine managing subscription transitions:
stateDiagram-v2
[*] --> Trialing: create(trial=true)
[*] --> Active: create(trial=false)
Trialing --> Active: trial_end + payment succeeds
Trialing --> Canceled: customer cancels during trial
Trialing --> Past Due: trial_end + payment fails
Active --> Past Due: payment fails
Active --> Canceling: cancel(at_period_end)
Active --> Paused: pause()
Active --> Canceled: cancel(immediate)
Past Due --> Active: dunning retry succeeds
Past Due --> Canceled: max retries exceeded
Past Due --> Past Due: dunning retry scheduled
Canceling --> Canceled: period_end reached
Canceling --> Active: reinstate(cancel_at_period_end=false)
Paused --> Active: resume()
Paused --> Canceled: cancel during pause
Canceled --> Active: reactivate(new payment)
Canceled --> [*]: retention period expired
State Transition Rules
| From | To | Trigger | Side Effects |
|---|---|---|---|
| Trialing → Active | Trial expires + payment succeeds | Cron / event | Charge customer, emit subscription.activated |
| Trialing → Past Due | Trial expires + payment fails | Cron / event | Start dunning sequence, emit invoice.payment_failed |
| Active → Past Due | Payment fails on renewal | Billing cycle | Start dunning, grace period begins |
| Past Due → Active | Retry succeeds | Dunning retry | Resume access, emit invoice.payment_succeeded |
| Past Due → Canceled | Max retries exceeded | Dunning timeout | Revoke access, emit subscription.canceled |
| Active → Canceling | Customer requests cancel | API call | Set cancel_at_period_end=true, no immediate charge |
| Canceling → Canceled | Period end reached | Cron / event | Revoke access, emit subscription.canceled |
| Active → Paused | Customer pauses | API call | Suspend access, stop billing, freeze period end |
| Paused → Active | Customer resumes | API call | Restore access, adjust period end |
| Canceled → Active | Customer reactivates | API call | New charge, new billing cycle |
5.2 Proration Engine
The most complex billing calculation — handles mid-cycle plan changes accurately.
Proration Calculation Algorithm:
Given:
- Current plan: $20/month (Photoshop)
- New plan: $55/month (Creative Cloud All)
- Billing cycle: Monthly
- Current period start: 2026-07-01
- Current period end: 2026-08-01
- Change date: 2026-07-15 (day 15 of 31-day cycle)
Step 1: Calculate remaining days
remaining_days = period_end - change_date = 31 - 15 = 16 days
elapsed_days = change_date - period_start = 15 days
total_days = 31
Step 2: Calculate daily rates
old_daily_rate = old_plan_amount / total_days = $20.00 / 31 = $0.6452/day
new_daily_rate = new_plan_amount / total_days = $55.00 / 31 = $1.7742/day
Step 3: Calculate proration amounts
credit_amount = old_daily_rate × elapsed_days = $0.6452 × 15 = $9.68
charge_amount = new_daily_rate × remaining_days = $1.7742 × 16 = $28.39
Step 4: Net proration
net_charge = charge_amount - credit_amount = $28.39 - $9.68 = $18.71
Step 5: Apply to invoice
- Credit line item: -$9.68 (Photoshop remaining credit)
- Charge line item: +$28.39 (Creative Cloud All remaining)
- Net: +$18.71 charged immediately
Proration Behaviors:
┌─────────────────────┬──────────────────────────────────────────┐
│ Mode │ Behavior │
├─────────────────────┼──────────────────────────────────────────┤
│ create_prorations │ Create credit/charge line items now │
│ create_invoice_now │ Create prorations + immediate invoice │
│ always_invoice │ Always charge (no credits, roll forward) │
│ none │ No proration (change at period end) │
└─────────────────────┴──────────────────────────────────────────┘
Edge Cases:
- Year-to-month: Prorate remaining days of year at monthly rate
- Month-to-year: Credit remaining month days, charge full year
- Quantity changes: Prorate only the delta (not full recalculation)
- Coupon changes: Prorate coupon value proportionally
- Multi-line items: Prorate each line item independently
Proration Precision
Precision Strategy:
- All calculations in smallest currency unit (cents)
- Use decimal arithmetic (NOT floating point)
- Round half-up at the final step only
- Intermediate calculations keep full precision
- Audit trail: Store all intermediate values for reconciliation
Example with cents:
old_daily_rate = 2000¢ / 31 = 64.516129...¢/day
credit = 64.516129... × 15 = 967.741935...¢ → 968¢ (rounded)
new_daily_rate = 5500¢ / 31 = 177.419354...¢/day
charge = 177.419354... × 16 = 2838.709677...¢ → 2839¢ (rounded)
net = 2839 - 968 = 1871¢ = $18.71
5.3 Dunning Management
Automated retry and notification system for failed payments:
flowchart TD
START[Payment Failed] --> R1[Retry 1<br/>+1 hour<br/>Email: Payment Failed]
R1 -->|Success| RESUME[Resume Access<br/>Recover Subscription]
R1 -->|Fail| R2[Retry 2<br/>+3 days<br/>Email: Final Notice]
R2 -->|Success| RESUME
R2 -->|Fail| R3[Retry 3<br/>+7 days<br/>Email + SMS: Last Chance]
R3 -->|Success| RESUME
R3 -->|Fail| R4[Retry 4<br/>+14 days<br/>Email: Account Suspended]
R4 -->|Success| RESUME
R4 -->|Fail| SUSPEND[Suspend Access<br/>Status: Past Due → Canceled]
SUSPEND --> WINBACK[Win-back Campaign<br/>Email: We Miss You<br/>+30 days]
WINBACK -->|Returns| REACTIVATE[Reactivate<br/>New Payment]
WINBACK -->|No Response| ARCHIVE[Archive<br/>+90 days]
style RESUME fill:#90EE90
style SUSPEND fill:#FFB6C1
style REACTIVATE fill:#90EE90
Dunning Configuration
Configurable per merchant / plan:
┌──────────────────┬────────────┬────────────┬──────────────────┐
│ Parameter │ Default │ Range │ Notes │
├──────────────────┼────────────┼────────────┼──────────────────┤
│ Grace Period │ 30 days │ 0-90 days │ Access kept │
│ Retry Attempts │ 4 │ 1-10 │ Total retries │
│ Retry Intervals │ 1h,3d,7d,14d │ Custom │ Exponential │
│ Email Templates │ 3 │ 1-5 │ Per retry stage │
│ SMS Enabled │ false │ bool │ High-value only │
│ Auto-cancel │ true │ bool │ After grace │
│ Win-back │ true │ bool │ Post-cancellation│
│ Card Update Nudge│ true │ bool │ Before expiry │
└──────────────────┴────────────┴────────────┴──────────────────┘
Dunning Metrics:
- Recovery Rate: % of failed payments recovered via retries
- Dunning-Induced Churn: % canceled during grace period
- Average Recovery Time: Mean time from failure to recovery
- Card Update Rate: % updating card before expiry
5.4 Entitlement Service
Real-time access control with caching:
Entitlement Evaluation Pipeline:
1. Request: Product service checks entitlement
GET /entitlements/check?customer_id=cus_abc&feature=photoshop_desktop
2. Cache Check (Redis, < 1ms):
Key: entitlement:{customer_id}:{feature}
TTL: 5 minutes (configurable)
If hit → return cached result
3. If cache miss, evaluate:
a. Look up customer's active subscriptions
b. For each subscription:
- Check status (must be active or in grace period)
- Check plan features (does plan include requested feature?)
- Check quantity (seats remaining?)
- Check billing status (not past_due beyond grace)
c. Combine results:
- entitled = ANY subscription grants feature
- blocked = reasons if not entitled
- warnings = approaching limits
4. Cache Result:
SET entitlement:{customer_id}:{feature} = result TTL 5min
5. Return:
{ entitled: true, features: [...], expires_at: ... }
Invalidation Strategy:
- Subscription state change → invalidate customer's entitlement cache
- Plan update → invalidate all affected customers' caches
- Bulk invalidation via Redis SCAN + DEL pattern
- Max invalidation batch: 10,000 keys/second
Grace Period Handling:
- Past Due + within grace period → entitled = true (with warning)
- Past Due + grace expired → entitled = false
- Canceling (at period end) → entitled = true until period end
- Paused → entitled = false
5.5 Billing Cycle Engine
Deterministic billing schedule management:
Billing Cycle Management:
Anchor Date:
- Set at subscription creation (or backdated)
- All future billing dates derived from anchor
- Survives plan changes (upgrade/downgrade preserve anchor)
- Formula: next_billing = anchor + N × interval
Billing Schedule Generation:
1. Daily job scans subscriptions due for billing:
WHERE current_period_end <= NOW() + INTERVAL '1 day'
AND status IN ('active', 'trialing')
2. For each due subscription:
a. Generate invoice with line items:
- Base plan charge
- Proration adjustments (if any)
- Usage overage charges
- Coupon/discount deductions
- Tax calculations
b. Attempt payment
c. On success: Advance billing period
d. On failure: Start dunning sequence
Billing Window:
- Not all subscribers bill on the same day
- Anchor dates distributed across month
- Reduces peak load on payment processor
- Enterprise: Can align to fiscal month-end
Trial Handling:
- Trial period does NOT create billing cycle
- First charge occurs at trial_end
- Billing anchor = trial_end (not trial_start)
- If trial canceled: No charge, no billing cycle created
5.6 Invoice Generator
Invoice Generation Pipeline:
1. Trigger: Billing cycle engine creates invoice draft
2. Line Items:
┌─────────────────────────────────────────────────────────┐
│ Invoice #INV-2026-001234 │
│ Date: 2026-08-01 │
│ Due: 2026-08-31 │
├──────────────────┬────────────┬──────────┬──────────────┤
│ Description │ Qty │ Unit │ Amount │
├──────────────────┼────────────┼──────────┼──────────────┤
│ Creative Cloud │ 1 │ $54.99 │ $54.99 │
│ All Apps - Aug │ │/mo │ │
├──────────────────┼────────────┼──────────┼──────────────┤
│ Additional Seat │ 2 │ $54.99 │ $109.98 │
│ (Jul 15 change) │ │/mo │ │
├──────────────────┼────────────┼──────────┼──────────────┤
│ Proration Credit │ 1 │ -$9.68 │ -$9.68 │
│ Photoshop Jul │ │ │ │
├──────────────────┼────────────┼──────────┼──────────────┤
│ Summer Discount │ 1 │ -$13.75 │ -$13.75 │
│ (20% off 1 mo) │ │ │ │
├──────────────────┼────────────┼──────────┼──────────────┤
│ Subtotal │ │ │ $141.54 │
│ Tax (VAT 20%) │ │ │ $28.31 │
│ TOTAL │ │ │ $169.85 │
└──────────────────┴────────────┴──────────┴──────────────┘
3. Tax Calculation:
- Determine customer jurisdiction (billing address)
- Look up tax rate (cached, updated monthly)
- Apply tax to taxable line items
- Store tax breakdown for compliance
4. PDF Generation:
- Template-based PDF generation
- Company branding, logo, terms
- Stored in S3, URL provided in invoice record
5. Delivery:
- Email with PDF attachment
- Available in self-service portal
- Webhook: invoice.created
5.7 Usage Metering Service
Usage Ingestion Pipeline:
1. Product Service reports usage:
POST /usage/records
{ subscription_id, feature, quantity, timestamp }
2. Validation:
- Subscription exists and active?
- Feature is metered on this plan?
- Quantity is valid (positive, within reason)?
3. Aggregation:
- Raw events → Kafka topic: usage.raw
- Stream processor aggregates per subscription per day:
SELECT subscription_id, feature, SUM(quantity)
FROM usage_events
WHERE period = current_billing_period
GROUP BY subscription_id, feature
4. Storage:
- Aggregated: PostgreSQL (per-subscription per-day totals)
- Raw events: Time-series DB (for debugging, audit)
- Retention: Raw 30 days, Aggregated 7 years
5. Threshold Alerts:
- Monitor usage vs included quantity
- At 80%: Send warning email
- At 100%: Send overage notice
- At 120%: Send critical notice
6. Overage Calculation (at billing):
- Pull aggregated usage for billing period
- Calculate overage = max(0, usage - included)
- Charge = overage × unit_price
- Add as line item to invoice
6. Data Model
6.1 Core Tables
-- Customers
CREATE TABLE customers (
id VARCHAR(32) PRIMARY KEY, -- cus_abc123
email VARCHAR(255) NOT NULL,
name VARCHAR(255),
phone VARCHAR(32),
default_currency CHAR(3) DEFAULT 'usd',
tax_id VARCHAR(64),
billing_address JSONB,
metadata JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
mrr BIGINT DEFAULT 0, -- current MRR in cents
CONSTRAINT uq_customer_email UNIQUE(email)
);
-- Plans (Product Pricing)
CREATE TABLE plans (
id VARCHAR(64) PRIMARY KEY, -- plan_photoshop_annual
product_id VARCHAR(64) NOT NULL,
name VARCHAR(255) NOT NULL,
description TEXT,
amount BIGINT NOT NULL, -- in smallest currency unit
currency CHAR(3) NOT NULL,
interval VARCHAR(20) NOT NULL, -- day / week / month / year
interval_count SMALLINT DEFAULT 1, // e.g., 3 for quarterly
trial_period_days SMALLINT,
features JSONB NOT NULL, -- ["photoshop_desktop", ...]
metadata JSONB,
active BOOLEAN DEFAULT true,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
sort_order INTEGER DEFAULT 0,
CONSTRAINT chk_interval CHECK (interval IN ('day', 'week', 'month', 'year'))
);
CREATE INDEX idx_plans_product ON plans(product_id);
CREATE INDEX idx_plans_active ON plans(active, sort_order) WHERE active = true;
-- Subscriptions (Core Entity)
CREATE TABLE subscriptions (
id VARCHAR(32) PRIMARY KEY, -- sub_3N2xYz...
customer_id VARCHAR(32) NOT NULL,
plan_id VARCHAR(64) NOT NULL,
status VARCHAR(20) NOT NULL, -- trialing / active / past_due / canceled / paused / canceling
quantity INTEGER DEFAULT 1, -- seats
-- Timing
current_period_start TIMESTAMPTZ NOT NULL,
current_period_end TIMESTAMPTZ NOT NULL,
billing_cycle_anchor TIMESTAMPTZ NOT NULL,
trial_start TIMESTAMPTZ,
trial_end TIMESTAMPTZ,
-- Cancellation
cancel_at_period_end BOOLEAN DEFAULT false,
canceled_at TIMESTAMPTZ,
cancel_reason VARCHAR(100),
cancellation_feedback TEXT,
-- Pause
paused_at TIMESTAMPTZ,
resume_at TIMESTAMPTZ,
-- Billing
collection_method VARCHAR(30) DEFAULT 'charge_automatically',
days_until_due SMALLINT DEFAULT 30,
payment_method_id VARCHAR(32),
default_tax_rates JSONB,
-- Dunning
dunning_status VARCHAR(20) DEFAULT 'none', -- none / in_progress / failed
dunning_attempts SMALLINT DEFAULT 0,
dunning_next_retry TIMESTAMPTZ,
dunning_max_attempts SMALLINT DEFAULT 4,
-- Metadata
metadata JSONB,
started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT fk_customer FOREIGN KEY(customer_id) REFERENCES customers(id),
CONSTRAINT fk_plan FOREIGN KEY(plan_id) REFERENCES plans(id),
CONSTRAINT chk_status CHECK (status IN (
'trialing', 'active', 'past_due', 'canceled', 'paused', 'canceling'
))
);
CREATE INDEX idx_sub_customer ON subscriptions(customer_id, status);
CREATE INDEX idx_sub_period_end ON subscriptions(current_period_end, status)
WHERE status IN ('active', 'trialing');
CREATE INDEX idx_sub_dunning ON subscriptions(dunning_next_retry, dunning_status)
WHERE dunning_status = 'in_progress' AND dunning_next_retry <= NOW();
CREATE INDEX idx_sub_canceling ON subscriptions(current_period_end, status)
WHERE status = 'canceling';
-- Subscription Events (Audit Trail)
CREATE TABLE subscription_events (
id BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
subscription_id VARCHAR(32) NOT NULL,
event_type VARCHAR(50) NOT NULL, -- created / activated / upgraded / downgraded / canceled
previous_status VARCHAR(20),
new_status VARCHAR(20),
previous_plan VARCHAR(64),
new_plan VARCHAR(64),
metadata JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT fk_sub FOREIGN KEY(subscription_id) REFERENCES subscriptions(id)
);
CREATE INDEX idx_sev_sub ON subscription_events(subscription_id, created_at DESC);
-- Invoices
CREATE TABLE invoices (
id VARCHAR(32) PRIMARY KEY, -- inv_3N2xYz...
customer_id VARCHAR(32) NOT NULL,
subscription_id VARCHAR(32),
number VARCHAR(32) NOT NULL, -- INV-2026-001234
status VARCHAR(20) NOT NULL, -- draft / open / paid / void / uncollectible
currency CHAR(3) NOT NULL,
-- Amounts
subtotal BIGINT NOT NULL,
tax_amount BIGINT DEFAULT 0,
discount_amount BIGINT DEFAULT 0,
proration_credit BIGINT DEFAULT 0,
total BIGINT NOT NULL,
amount_paid BIGINT DEFAULT 0,
amount_due BIGINT NOT NULL,
-- Dates
period_start TIMESTAMPTZ NOT NULL,
period_end TIMESTAMPTZ NOT NULL,
due_date TIMESTAMPTZ NOT NULL,
issued_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
paid_at TIMESTAMPTZ,
voided_at TIMESTAMPTZ,
-- Payment
payment_method_id VARCHAR(32),
payment_intent_id VARCHAR(32),
-- Documents
pdf_url VARCHAR(512),
host_url VARCHAR(512),
-- Tax
tax_breakdown JSONB,
billing_address JSONB,
metadata JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT chk_inv_status CHECK (status IN ('draft', 'open', 'paid', 'void', 'uncollectible'))
);
CREATE INDEX idx_inv_customer ON invoices(customer_id, created_at DESC);
CREATE INDEX idx_inv_status ON invoices(status, due_date);
CREATE INDEX idx_inv_number ON invoices(number);
-- Invoice Line Items
CREATE TABLE invoice_line_items (
id BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
invoice_id VARCHAR(32) NOT NULL,
type VARCHAR(20) NOT NULL, -- subscription / proration / usage / coupon / tax
description TEXT NOT NULL,
quantity BIGINT DEFAULT 1,
unit_amount BIGINT NOT NULL, -- can be negative for credits
amount BIGINT NOT NULL, -- quantity × unit_amount
proration_start TIMESTAMPTZ,
proration_end TIMESTAMPTZ,
metadata JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT fk_invoice FOREIGN KEY(invoice_id) REFERENCES invoices(id),
CONSTRAINT chk_li_type CHECK (type IN ('subscription', 'proration', 'usage', 'coupon', 'tax', 'fee'))
);
CREATE INDEX idx_ili_invoice ON invoice_line_items(invoice_id);
-- Coupons
CREATE TABLE coupons (
id VARCHAR(32) PRIMARY KEY, -- coupon_summer20
code VARCHAR(64) NOT NULL,
name VARCHAR(255),
description TEXT,
type VARCHAR(20) NOT NULL, -- percent / amount / duration
percent_off DECIMAL(5,2),
amount_off BIGINT,
currency CHAR(3),
duration VARCHAR(20) NOT NULL, -- once / repeating / forever
duration_in_months INTEGER,
max_redemptions INTEGER,
redemption_count INTEGER DEFAULT 0,
valid_from TIMESTAMPTZ,
valid_until TIMESTAMPTZ,
applicable_plans VARCHAR(64)[], -- specific plans or NULL for all
stackable BOOLEAN DEFAULT false,
active BOOLEAN DEFAULT true,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT uq_coupon_code UNIQUE(code),
CONSTRAINT chk_coupon_type CHECK (type IN ('percent', 'amount', 'duration'))
);
-- Customer-Coupon Associations
CREATE TABLE customer_coupons (
id BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
customer_id VARCHAR(32) NOT NULL,
coupon_id VARCHAR(32) NOT NULL,
subscription_id VARCHAR(32),
deleted BOOLEAN DEFAULT false,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT fk_cust FOREIGN KEY(customer_id) REFERENCES customers(id),
CONSTRAINT fk_coupon FOREIGN KEY(coupon_id) REFERENCES coupons(id),
CONSTRAINT fk_sub FOREIGN KEY(subscription_id) REFERENCES subscriptions(id),
CONSTRAINT uq_cust_coupon_sub UNIQUE(customer_id, coupon_id, subscription_id)
);
-- Usage Records
CREATE TABLE usage_records (
id BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
subscription_id VARCHAR(32) NOT NULL,
feature VARCHAR(100) NOT NULL,
quantity BIGINT NOT NULL,
action VARCHAR(20) DEFAULT 'increment', -- increment / set
timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT fk_sub_usage FOREIGN KEY(subscription_id) REFERENCES subscriptions(id)
);
CREATE INDEX idx_usage_sub_feature ON usage_records(subscription_id, feature, timestamp);
-- Usage Aggregates (materialized, refreshed daily)
CREATE TABLE usage_aggregates (
id BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
subscription_id VARCHAR(32) NOT NULL,
feature VARCHAR(100) NOT NULL,
period_start DATE NOT NULL,
period_end DATE NOT NULL,
total_quantity BIGINT NOT NULL,
overage_quantity BIGINT DEFAULT 0,
overage_charge BIGINT DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT uq_agg UNIQUE(subscription_id, feature, period_start)
);
-- Entitlement Cache (denormalized for fast reads)
CREATE TABLE entitlement_cache (
customer_id VARCHAR(32) PRIMARY KEY,
features JSONB NOT NULL, -- { "photoshop_desktop": true, ... }
subscriptions JSONB NOT NULL, -- summary of active subs
blocked_features JSONB,
warnings JSONB,
version BIGINT NOT NULL, -- increment on change
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
expires_at TIMESTAMPTZ NOT NULL -- TTL for stale detection
);
-- Payment Methods
CREATE TABLE payment_methods (
id VARCHAR(32) PRIMARY KEY,
customer_id VARCHAR(32) NOT NULL,
type VARCHAR(20) NOT NULL, -- card / bank_account
card_token VARCHAR(64),
card_brand VARCHAR(20),
card_last4 CHAR(4),
card_exp_month SMALLINT,
card_exp_year SMALLINT,
is_default BOOLEAN DEFAULT false,
billing_details JSONB,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT fk_cust_pm FOREIGN KEY(customer_id) REFERENCES customers(id)
);
-- Dunning Attempts
CREATE TABLE dunning_attempts (
id BIGINT PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
subscription_id VARCHAR(32) NOT NULL,
invoice_id VARCHAR(32) NOT NULL,
attempt_number SMALLINT NOT NULL,
status VARCHAR(20) NOT NULL, -- scheduled / sent / succeeded / failed
scheduled_at TIMESTAMPTZ NOT NULL,
executed_at TIMESTAMPTZ,
error_code VARCHAR(50),
error_message TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT fk_sub_dunning FOREIGN KEY(subscription_id) REFERENCES subscriptions(id)
);
CREATE INDEX idx_dunning_next ON dunning_attempts(scheduled_at, status)
WHERE status = 'scheduled' AND scheduled_at <= NOW();
6.2 Partitioning and Indexing Strategy
Partitioning:
- invoices: Range partition by created_at (monthly)
- invoice_line_items: Range partition by created_at (monthly)
- subscription_events: Range partition by created_at (monthly)
- usage_records: Range partition by timestamp (monthly)
- dunning_attempts: Range partition by created_at (monthly)
Rationale:
- Billing data is time-accessed (last N months)
- Old partitions can be archived to cold storage
- Partition pruning for date-range queries
- Compliance: Easy DROP for data retention
Critical Indexes:
┌────────────────────────────┬─────────────────────────────────┬──────────────────┐
│ Index │ Columns │ Purpose │
├────────────────────────────┼─────────────────────────────────┼──────────────────┤
│ idx_sub_period_end │ current_period_end, status │ Billing scan │
│ idx_sub_customer │ customer_id, status │ Customer lookup │
│ idx_sub_dunning │ dunning_next_retry, status │ Dunning scan │
│ idx_inv_customer │ customer_id, created_at DESC │ Invoice history │
│ idx_usage_sub_feature │ sub, feature, timestamp │ Usage aggregation│
└────────────────────────────┴─────────────────────────────────┴──────────────────┘
7. Key Design Decisions
7.1 Billing Cycle Anchor vs. Calendar Alignment
| Aspect | Anchor Date (Chosen) | Calendar Month |
|---|---|---|
| Customer experience | Consistent billing date each cycle | Bills on 1st of every month |
| Proration complexity | Simpler (always from anchor) | Complex (partial first/last month) |
| Revenue recognition | Clear period boundaries | Misaligned for mid-month signups |
| Cash flow predictability | Distributed across month | Spike on 1st of month |
| Enterprise alignment | Can backdate to fiscal month | Fixed to calendar |
Decision: Anchor date provides flexibility and simplicity. Enterprise customers can request fiscal-month alignment via backdating.
7.2 Centralized vs. Distributed Entitlements
| Aspect | Centralized Service (Chosen) | Embedded in Product |
|---|---|---|
| Consistency | Single source of truth | Risk of divergence |
| Latency | Added network hop | Zero latency |
| Caching | Centralized cache invalidation | Per-product cache management |
| Complexity | One service to maintain | N services to maintain |
| Cross-product features | Easy (bundle entitlements) | Hard (cross-service calls) |
Decision: Centralized entitlement service with aggressive caching (5-min TTL, Redis). Product services call entitlement API on startup and cache locally. Cache invalidated via event-driven invalidation on subscription changes.
7.3 Synchronous vs. Asynchronous Billing
| Aspect | Async Billing (Chosen) | Synchronous |
|---|---|---|
| Customer experience | No blocking at signup | Immediate charge confirmation |
| Reliability | Retry on failure | Must handle failure inline |
| Batching | Can batch invoices | One-at-a-time |
| Complexity | Event-driven pipeline | Simpler flow |
| Visibility | Invoice available after processing | Immediate |
Decision: Async billing with event-driven pipeline. Trial subscriptions charge at trial_end via scheduled job. Immediate charge option available for enterprise via synchronous API path.
7.4 Proration: Credit vs. Roll-Forward
| Aspect | Credit + Charge (Chosen) | Roll-Forward |
|---|---|---|
| Transparency | Clear credit and charge lines | Hidden in next invoice |
| Cash flow | Immediate net settlement | Deferred to next cycle |
| Customer understanding | “I got $X credit, charged $Y” | “Why is my bill different?” |
| Accounting | Clean period boundaries | Carried-forward complexity |
| Refund handling | Credits can offset charges | Must wait for next invoice |
Decision: Credit + charge provides maximum transparency. Customers see exactly what they’re getting credited and charged for. Configurable to roll-forward for merchants who prefer it.
7.5 Entitlement Cache Strategy
Cache Design:
Tier 1: Product-side LRU Cache (TTL: 60 seconds)
- Each product service caches entitlement results
- Reduces API calls for frequent checks
- Evicted on subscription change event
Tier 2: Entitlement Service Redis Cache (TTL: 5 minutes)
- Key: entitlement:{customer_id}
- Value: Full entitlement snapshot (features, warnings, subs)
- Invalidated on any subscription change for customer
Tier 3: Database (source of truth)
- entitlement_cache table (denormalized)
- Updated asynchronously from subscription events
- Used for cache warming and recovery
Invalidation Flow:
Subscription Change → Kafka Event → Entitlement Service
→ Update DB cache
→ Invalidate Redis key
→ Publish cache_invalidation event
→ Product services receive → evict local cache
8. Scalability Strategy
8.1 Horizontal Scaling Architecture
flowchart TB
subgraph ZONE1["Zone A"]
GW1["API Gateway × 4"]
SUB1["Sub Service × 3"]
BILL1["Billing × 2"]
ENT1["Entitlement × 6"]
REDIS1["Redis Shard 1-4"]
end
subgraph ZONE2["Zone B"]
GW2["API Gateway × 4"]
SUB2["Sub Service × 3"]
BILL2["Billing × 2"]
ENT2["Entitlement × 6"]
REDIS2["Redis Shard 5-8"]
end
subgraph SHARED["Shared Services"]
KAFKA["Kafka Cluster<br/>(6 brokers × 3 AZ)"]
RDB["PostgreSQL<br/>(Primary + 4 Replicas)"]
ES["Elasticsearch<br/>(6 nodes)"]
end
GW1 --> SUB1 & BILL1 & ENT1
GW2 --> SUB2 & BILL12 & ENT2
SUB1 & SUB2 --> KAFKA
BILL1 & BILL12 --> KAFKA
ENT1 --> REDIS1
ENT2 --> REDIS2
SUB1 & SUB2 & BILL1 & BILL12 --> RDB
KAFKA --> ES
8.2 Scaling by Component
| Component | Scaling Strategy | Bottleneck | Max Scale |
|---|---|---|---|
| Entitlement Service | Stateless, auto-scale on QPS | Redis throughput | 50K QPS with caching |
| Subscription Service | Stateless, partition by customer_id | Database write throughput | 10K TPS with sharding |
| Billing Engine | Partitioned by billing window | Payment processor rate limits | 5K TPS (processor-bound) |
| Dunning Manager | Partitioned by retry window | Email/SMS rate limits | 1M retries/day |
| Usage Metering | Stream processing (Kafka Streams) | Aggregation compute | 100K events/sec |
| Invoice Generator | Stateless, queue-based | PDF generation CPU | 10K invoices/min |
8.3 Database Scaling
Phase 1: Single PostgreSQL (0-1M subscribers)
- Primary + 3 read replicas
- Connection pooling (PgBouncer)
- Read/write split at app layer
Phase 2: Citus Sharding (1M-10M subscribers)
- Shard key: customer_id (consistent hash)
- 16 shards across 4 coordinator nodes
- Co-located: customer, subscriptions, invoices
- Distributed queries for analytics
Phase 3: Multi-Region (10M+ subscribers)
- Regional primaries (US, EU, APAC)
- Cross-region replication for DR
- Customer assigned to region at signup
- Analytics: Centralized data warehouse (Snowflake/BigQuery)
Read Scaling:
- Entitlement reads: Redis (99.9% cache hit rate)
- Dashboard reads: Read replicas + materialized views
- Analytics: Elasticsearch + data warehouse
- Invoice history: Read replicas with partition pruning
8.4 Billing Spike Management
Problem: All subscribers with anchor on the 1st bill simultaneously.
Solutions:
1. Anchor Distribution (Prevention):
- New subscribers: Randomize anchor within ±7 days of signup
- Enterprise: Allow specific anchor date
- Result: Billing spread across 15-day window
2. Billing Queue (Absorption):
- Billing requests enter priority queue
- Processed in batches of 1,000
- Rate-limited to payment processor limits
- Queue depth monitored, auto-scale consumers
3. Payment Processor Scaling:
- Multiple processor backends
- Load balance across processors
- Circuit breaker per processor
- Failover on processor degradation
4. Grace Period Buffer:
- Invoices generated ahead of due date
- Payment attempted with 3-day window
- Reduces exact-moment pressure
9. Curveball Scenarios
9.1 Curveball 1: “A major plan price increase causes a 15% churn spike. Customers are upgrading to competitors mid-cycle. How do you handle the billing and entitlement chaos?”
Impact Assessment:
- 15% churn = massive cancellation wave
- Mid-cycle cancellations trigger proration refunds
- Revenue drop impacts cash flow
- Remaining customers may demand price matches
Response Strategy:
Phase 1: Immediate Triage (0-2 hours)
1. Assess scope:
- How many cancellations in the last hour?
- Which plans are most affected?
- Are cancellations immediate or at period end?
2. System protection:
- Monitor proration engine load (cancellations = proration calcs)
- Ensure refund processing queue isn't backing up
- Verify entitlement cache invalidation is keeping up
3. Communication:
- Alert customer success team
- Prepare retention offers for at-risk segments
- Update churn dashboard with real-time metrics
Phase 2: Billing Impact Management (2-8 hours)
1. Proration flood handling:
- Batch proration calculations (not real-time per cancellation)
- Queue cancellations, process proration async
- Customer sees "refund processing" status immediately
- Actual refund within 24 hours
2. Revenue recognition adjustment:
- Recalculate MRR/ARR with new churn rate
- Update finance dashboards
- Flag for board/leadership review
3. Remaining customer impact:
- Ensure no accidental price increases on existing subscriptions
- Price increase should ONLY apply to new signups and renewals
- Verify: No active subscription was charged at new price
Phase 3: Retention & Recovery (8-48 hours)
1. Win-back offers:
- Create limited-time discount for canceling customers
- "Come back within 30 days at old price" coupon
- Automated email sequence for churned customers
2. Price match program:
- For customers citing competitor pricing
- Manual approval workflow for exceptions
3. Product feedback loop:
- Analyze cancellation reasons
- Identify if price was the only factor
- Feed insights to product team
System Resilience:
- Proration engine handled 3x normal load without degradation
- Refund queue processed all backlogged items within 18 hours
- Entitlement cache invalidation kept up (no access leaks)
9.2 Curveball 2: “Your billing engine has a bug that double-billed customers on their annual renewal. 50,000 customers affected. Discovery happens 3 days later. What’s your response?”
Impact Assessment:
- 50,000 customers overcharged (massive reputation risk)
- Chargeback wave incoming
- Regulatory notification may be required
- Customer trust severely damaged
Response Strategy:
Phase 1: Containment (0-1 hour)
1. IMMEDIATELY: Disable annual renewal billing path
- Feature flag: annual_renewal_enabled = false
- Switch to manual review for annual renewals
2. Scope the damage:
SELECT subscription_id, invoice_id, amount
FROM invoices
WHERE period_start >= '2026-07-01'
AND plan_interval = 'year'
AND status = 'paid'
-- Cross-reference with payment records for duplicates
3. Verify: Confirm bug affects ONLY annual renewals
- Monthly renewals: - New signups unaffected
- Proration charges unaffected
Phase 2: Mass Refund (1-6 hours)
1. Automated refund batch:
- Generate refunds for all 50,000 affected customers
- Use internal admin API (bypass normal rate limits)
- Each refund: Full duplicate amount + goodwill credit (1 month free)
- Total refund: ~50,000 × $239.88 = ~$12M
2. Customer notifications (proactive):
- Email: "We made an error and have refunded you"
- Include: Refund amount, expected timeline, apology
- Proactive > reactive (prevents chargebacks)
3. Dashboard update:
- Self-service portal shows refund status
- Updated invoice with credit line item
Phase 3: Verification (6-24 hours)
1. Confirm all refunds initiated
2. Monitor chargeback rate (should be near zero if proactive)
3. Verify no remaining duplicates in system
4. Finance reconciliation: Match refunds to original charges
Phase 4: Root Cause & Prevention (1-5 days)
1. Root cause: Why did double-billing occur?
- Likely: Idempotency missing on renewal path
- Or: Race condition in billing cycle advancement
- Or: Retry logic created duplicate invoices
2. Fix:
- Add idempotency to renewal path
- Add invariant check: No overlapping billing periods
- Add pre-charge validation: No existing paid invoice for period
3. Tests:
- Regression test for exact scenario
- Property-based testing for billing invariants
- Chaos engineering: Inject duplicate billing events
4. Process:
- Pre-deployment checklist for billing changes
- Staged rollout (1% → 10% → 100%) for billing code
- Real-time billing anomaly detection
9.3 Curveball 3: “A customer’s subscription shows as ‘active’ in your system but their payment actually failed 30 days ago. They’ve been using premium features for free. How did this happen and how do you fix it?”
Root Cause Analysis:
Likely Root Causes (ranked by probability):
1. Dunning retry succeeded on a STALE payment method
- Old card expired, but retry used cached token
- Payment processor returned success (test mode / sandbox leak)
- Webhook for payment success never arrived (network issue)
- Subscription status not updated
2. Entitlement cache never invalidated
- Subscription moved to past_due in DB
- Redis cache still shows "active"
- Cache TTL too long or invalidation failed
3. Billing cycle advancement race condition
- Two billing jobs processed same subscription
- First job: Marked past_due
- Second job: Advanced period (overwriting past_due)
4. Webhook delivery failure + no reconciliation
- Payment failed webhook never delivered to subscription service
- No daily reconciliation job to catch discrepancies
Investigation:
1. Check subscription_events table for this customer:
- Last status transition? When?
- Any payment_failed events?
- Any dunning attempts recorded?
2. Check invoices:
- Last paid invoice date?
- Any unpaid invoices in the period?
3. Check payment records:
- Last successful payment?
- Any failed payment attempts?
4. Check entitlement cache:
- Cache version vs DB version
- Last invalidation timestamp
Fix Strategy:
Immediate Fix:
1. Revoke access for affected customer(s)
2. Generate catch-up invoice for the free period
3. Offer payment plan if amount is large
Systemic Fixes:
1. Daily Reconciliation Job:
- Compare subscription status vs. payment status
- Flag: Active subscription + no payment in N days
- Auto-correct or alert for review
2. Entitlement Cache Hard TTL:
- Max cache age: 24 hours (force re-evaluation)
- Even if invalidation fails, cache expires
3. Payment Status Cross-Check:
- On entitlement check: Verify last payment success
- If payment older than billing cycle → flag for review
4. Webhook Reliability:
- Add payment processor API polling as backup
- If webhook not received in 1 hour → poll processor
- Reconcile webhook events with processor records
5. Invariant Monitoring:
- Alert: subscription.status = 'active' AND
last_payment_date < current_period_start
- This should NEVER be true
9.4 Curveball 4: “You need to support a new billing model: ‘Pay as you go’ with no subscription, just metered usage. But your system is built around subscription cycles. How do you adapt?”
Adaptation Strategy:
Architecture Extension:
Option A: Virtual Subscription (Chosen)
- Create a "payg" plan with $0 base price
- Billing cycle: Monthly (for usage aggregation)
- Invoice generated monthly with only usage line items
- No proration, no trial, no dunning for base amount
- Dunning only if usage charges fail
Option B: Standalone Usage Billing
- Separate billing path outside subscription engine
- Usage → Invoice → Payment (no subscription involved)
- More complex, duplicate billing logic
Why Option A:
✓ Reuses existing billing, invoicing, payment, dunning
✓ Customer has one unified billing view
✓ Can combine: Base subscription + usage overage
✓ Minimal new code
Implementation:
1. New plan type: "metered_only"
- amount = 0
- interval = month
- features = metered features only
2. Subscription lifecycle simplified:
- No trial, no proration
- Cancel = stop usage tracking
- Reactivate = resume usage tracking
3. Billing cycle:
- Monthly invoice with usage line items only
- If usage = 0: No invoice generated
- Threshold: Minimum $1 invoice (avoid micro-billing)
4. Entitlement:
- Active payg subscription → access to metered features
- Past due → suspend until payment
- Usage limits can be enforced in real-time
9.5 Curveball 5: “An enterprise customer wants custom billing: Net-60 terms, PO-based invoicing, custom tax rates, and quarterly true-ups. Your self-serve system doesn’t support this.”
Enterprise Billing Extension:
Enterprise Billing Features:
1. Collection Method: send_invoice (vs charge_automatically)
- Invoice generated but NOT auto-charged
- Customer pays via bank transfer / check / PO
- Payment recorded manually or via bank feed
2. Payment Terms: Configurable per customer
- Net-30 (default), Net-60, Net-90
- Due date = invoice_date + terms_days
- Late fee calculation after due date
3. PO Number Support:
- PO number on invoice
- PO validation (expiry, amount limit)
- PO tracking across multiple invoices
4. Quarterly True-Ups:
- Estimate usage monthly, bill monthly
- Quarterly: Compare actual vs estimated
- True-up invoice: Credit or charge the difference
- Formula: true_up = actual_annual - estimated_billed
5. Custom Tax Rates:
- Enterprise tax override (negotiated rate)
- Tax exemption certificates
- Reverse charge for B2B EU transactions
Implementation:
┌─────────────────────────────────────────────────────┐
│ Enterprise Customer Flags: │
│ - collection_method = 'send_invoice' │
│ - payment_terms = 'net_60' │
│ - po_required = true │
│ - true_up_frequency = 'quarterly' │
│ - custom_tax_rate = 0.15 (vs standard 0.20) │
│ - dedicated_account_manager = true │
└─────────────────────────────────────────────────────┘
Billing Flow for Enterprise:
1. Monthly: Generate invoice (no auto-charge)
2. Email invoice to accounts payable
3. Track payment status (unpaid → paid → late)
4. Quarterly: Generate true-up invoice
5. Annual: Generate annual summary for both parties
10. Failure Modes and Mitigations
10.1 Failure Mode Matrix
| Failure Mode | Probability | Impact | Detection | Mitigation | RTO | RPO |
|---|---|---|---|---|---|---|
| Billing job misses a cycle | Medium | High | Reconciliation alert | Daily reconciliation, manual backfill | < 1hr | 0 |
| Entitlement cache stale | Medium | Medium | Cache age monitor | Hard TTL, event invalidation, polling | < 5min | 5min |
| Proration calculation error | Low | Critical | Customer complaint, audit | Decimal arithmetic, audit trail | < 1hr | 0 |
| Dunning emails not sent | Medium | Medium | Email delivery rate | Multi-provider fallback, retry queue | < 30min | 0 |
| Usage data loss | Low | High | Usage reconciliation | Kafka persistence, dual-write | < 1hr | 0 |
| Invoice generation | Low | Critical | Duplicate invoice detection | Idempotency on invoice creation | < 1hr | 0 |
| Payment processor outage | Low | Critical | Payment success rate | Multi-processor failover | N/A | 0 |
| Database primary failure | Low | Critical | Connection errors | Automated failover, replicas | < 60s | < 1s |
| Tax rate change missed | Medium | Medium | Tax compliance audit | Monthly check | Automated tax rate updates | < 1day |
| Webhook delivery failure | Medium | Low | Webhook success rate | Retry with backoff, manual replay | < 3days | 0 |
10.2 Detailed Mitigation Strategies
Billing Job Misses a Cycle
Detection:
- Daily reconciliation job:
SELECT s.id, s.current_period_end, s.status
FROM subscriptions s
WHERE s.status IN ('active', 'trialing')
AND s.current_period_end < NOW() - INTERVAL '1 day'
AND NOT EXISTS (
SELECT 1 FROM invoices i
WHERE i.subscription_id = s.id
AND i.period_end = s.current_period_end
)
- Alert if any subscriptions found
Mitigation:
1. Auto-retry: Billing job retries missed subscriptions
2. Manual backfill: Admin tool to generate backdated invoices
3. Customer notification: "We noticed a billing issue and have corrected it"
4. Goodwill: Consider discount for inconvenience
Prevention:
- Billing job runs twice daily (redundancy)
- Dead man's switch: Alert if billing job doesn't run
- Idempotent billing: Safe to retry without duplicates
- Monitoring: Billing job success rate dashboard
Proration Calculation Error
Prevention Layers:
Layer 1: Code-Level
- Use decimal arithmetic (BigDecimal / Python Decimal)
- NEVER use floating point for money
- Round only at final step
- Store all intermediate values in audit trail
Layer 2: Validation
- Invariant: credit + charge = net_proration
- Invariant: proration_days <= billing_cycle_days
- Invariant: proration_amount <= plan_amount
- Invariant: No negative totals (unless intentional credit)
Layer 3: Testing
- Property-based testing: For all inputs, proration is fair
- Fuzz testing: Random plan changes, verify no negative amounts
- Snapshot testing: Known scenarios produce expected results
Layer 4: Monitoring
- Alert: Proration amount > 2× plan amount
- Alert: Negative proration on upgrade (should be positive)
- Alert: Proration days > billing cycle days
Layer 5: Reconciliation
- Monthly: Sum all prorations, verify against expected
- Customer-facing: Show proration breakdown on invoice
- Support: Tool to recalculate and compare
Entitlement Cache Staleness
Multi-Layer Defense:
Layer 1: Event-Driven Invalidation (Primary)
- Subscription change → Kafka event → Invalidate Redis
- Guaranteed delivery (Kafka persistence)
- Consumer group ensures at-least-once processing
Layer 2: Hard TTL (Safety Net)
- Redis TTL: 5 minutes (configurable)
- Even if invalidation fails, cache expires
- Force re-evaluation from database
Layer 3: Version Checking (Verification)
- Cache includes version number
- On cache hit: Compare version with DB version
- If stale: Evict and re-fetch
Layer 4: Periodic Refresh (Last Resort)
- Every 24 hours: Full cache refresh for active customers
- Ensures no cache is 24-hour age
- Runs during off-peak hours
Layer 5: Product-Side Fallback
- If entitlement service unavailable:
- Use local cache (60-second TTL)
- Log warning, alert on-call
- Fail open (grant access) or fail closed (deny access)
based on product risk assessment
10.3 Monitoring and Alerting
Critical Alerts (Page On-Call):
- Billing job failure (no invoices generated in 24h)
- Payment success rate < 90%
- Entitlement service latency p99 > 200ms
- Database primary failover
- Kafka consumer lag > 1M messages
- Proration calculation errors > 0
Warning Alerts (Ticket):
- Dunning email delivery rate < 95%
- Entitlement cache hit rate < 95%
- Invoice generation queue depth > 10K
- Usage ingestion lag > 1 hour
- Tax rate update failed
Business Alerts (Dashboard):
- MRR change > 5% week-over-week
- Churn rate > threshold
- Trial conversion rate drop
- Dunning recovery rate drop
- Average revenue per user change
10.4 Data Consistency Guarantees
Financial Consistency Model:
Strong Consistency (Required):
- Subscription state transitions (row-level lock)
- Invoice creation (no duplicates for same period)
- Payment recording (exactly once)
- Proration calculations (deterministic, auditable)
Eventual Consistency (Acceptable):
- Entitlement cache (TTL-bounded)
- Analytics dashboards (MRR, ARR aggregates)
- Usage metrics (aggregated daily)
- Search indexes (Elasticsearch)
Consistency Enforcement:
1. Database constraints (CHECK, FOREIGN KEY, UNIQUE)
2. Application-level state machine guards
3. Idempotency keys for all mutating operations
4. Daily reconciliation jobs
5. Audit log for every state change
6. Invariant monitoring (real-time alerts)
Key Invariants:
- A subscription cannot have overlapping billing periods
- An invoice total = sum of line items
- A customer cannot have two active subscriptions for the same plan
- Proration credit + proration charge = net proration
- Entitlement reflects current subscription status (within TTL)
Appendix A: Glossary
| Term | Definition |
|---|---|
| MRR | Monthly Recurring Revenue |
| ARR | Annual Recurring Revenue (MRR × 12) |
| Churn | Percentage of subscribers canceling in a period |
| Proration | Partial billing for mid-cycle changes |
| Dunning | Process of handling failed recurring payments |
| Entitlement | Access rights granted by an active subscription |
| Grace Period | Time after payment failure before access is revoked |
| Anchor Date | Reference date for billing cycle calculation |
| True-Up | Reconciliation of estimated vs actual usage |
| LTV | Lifetime Value of a customer |
| Net-30/60/90 | Payment terms (days until payment due) |
| Metered Billing | Usage-based charging (per API call, GB, etc.) |
| Win-Back | Campaign to re-engage churned customers |
| SCA | Strong Customer Authentication |
| PO | Purchase Order (enterprise billing) |
Appendix B: Key Metrics Dashboard
| Metric | Target | Measurement |
|---|---|---|
| MRR Growth | > 10% MoM | (Current MRR - Previous MRR) / Previous MRR |
| Churn Rate | < 3% monthly | Cancellations / Total subscribers |
| Trial Conversion | > 40% | Activated trials / Total trials |
| Dunning Recovery | > 60% | Recovered failed payments / Total failed |
| Billing Accuracy | 100% | Disputed invoices / Total invoices |
| Entitlement Latency p99 | < 50ms | Server-side timing |
| Invoice Generation Time | < 5 seconds | From trigger to PDF ready |
| System Availability | 99.95% | Uptime / Total time |
| NPS | > 50 | Customer satisfaction survey |
| LTV:CAC Ratio | > 3:1 | Lifetime value / Customer acquisition cost |