Payments & SubscriptionsQuestion #8CORE

Adobe: Subscription Lifecycle, Proration & Entitlements

Enterprise recurring subscription management system handling tier proration, asynchronous dunning state machines, and high-speed cryptographic client entitlements.

Category: Category 5: Payments & Subscriptions•Updated: 2026-08-30
#Payments#Subscriptions#Proration#Dunning#Entitlements

Design a Subscription Management System like Adobe

Table of Contents

  1. Requirements Clarification
  2. Back-of-Envelope Calculations
  3. System Interface Design
  4. High-Level Architecture
  5. Core Component Design
  6. Data Model
  7. Key Design Decisions
  8. Scalability Strategy
  9. Curveball Scenarios
  10. Failure Modes and Mitigations

1. Requirements Clarification

1.1 Functional Requirements

  • Subscription Lifecycle Management: Create, activate, pause, resume, cancel, and reactivate subscriptions with proper state transitions
  • Flexible Billing Cycles: Support monthly, annual, quarterly, and custom intervals with anchor date management
  • Plan Management: Create and manage product plans with features, pricing tiers, and metadata
  • Upgrades & Downgrades: Mid-cycle plan changes with proration calculations (credit/debit)
  • Proration Engine: Accurate prorated charges/credits for mid-cycle changes, partial refunds
  • Dunning Management: Automated retry sequences for failed payments with customer notifications
  • Entitlement Management: Real-time access control based on active subscriptions and plan features
  • Trial Periods: Configurable free trials with automatic conversion or expiration
  • Coupons & Discounts: Percentage/fixed discounts, first-month-free, lifetime deals, stackable promotions
  • Invoicing: Generate itemized invoices with tax calculations, PDF delivery, and payment tracking
  • Usage-Based Billing: Metered billing for overage charges (API calls, storage, seats)
  • Multi-Seat / Team Plans: Per-seat pricing with add/remove seat proration
  • Self-Service Portal: Customer-facing dashboard for managing subscriptions, payment methods, and invoices
  • Analytics & Reporting: MRR/ARR tracking, churn analysis, cohort reports, LTV calculations
  • Tax Compliance: Automated tax calculation (Sales Tax, VAT, GST) with jurisdiction-aware rates

1.2 Non-Functional Requirements

Category Requirement Rationale
Availability 99.95% uptime (4.38 hours downtime/year) Subscription access = product access
Consistency Strong consistency for billing records No over/under-charging
Accuracy 100% billing accuracy Financial integrity, customer trust
Latency Entitlement check < 50ms p99 Product access gate, must be fast
Throughput 1M subscription events/day Scale for global subscriber base
Data Retention 7+ years for billing records Tax and audit compliance
Multi-Tenancy Support enterprise customers with custom contracts B2B2C scenarios
Localization Multi-currency, multi-language invoices Global customer base
Compliance GDPR, CCPA, SOC 2 Data privacy and security

2. Back-of-Envelope Calculations

2.1 Assumption Sets

Parameter Startup (Year 1) Growth (Year 3) Adobe-scale
Active Subscribers 50K 5M 25M+
Monthly Churn Rate 5% 3% 2%
New Subs/Month 5K 500K 2.5M
Billing Events/Month 60K 6M 30M
Entitlement Checks/Day 500K 50M 250M
Avg Revenue/Subscriber $20/mo $20/mo $20/mo
MRR $1M $100M $500M

2.2 Storage Estimates (Adobe-scale)

Per Subscriber:
  - Customer profile: ~1 KB
  - Active subscription: ~2 KB
  - Invoice history (avg 24 invoices): ~5 KB × 24 = 120 KB
  - Payment history (avg 24 payments): ~3 KB × 24 = 72 KB
  - Event log (avg 50 events): ~500 B × 50 = 25 KB
  - Usage records (monthly): ~10 KB
  - Total per subscriber: ~230 KB

Active subscribers: 25M × 230 KB = 5.75 TB
Historical (7 years, 50M total ever): 50M × 500 KB = 25 TB

With 3x replication: 75 TB raw storage

Hot data (active + 1 year): ~10 TB
Warm data (1-3 years): ~20 TB
Cold data (3-7 years): ~45 TB → archive to object storage

2.3 QPS Estimates

Entitlement Checks (product access gate):
  Average: 250M / 86,400s = ~2,900 QPS
  Peak (3x): ~8,700 QPS
  Must be < 50ms p99 → heavy caching required

Billing Events:
  Average: 30M / 30 days / 86,400s = ~11.6 QPS (spread out)
  Billing Day Peak: 30M / 8 hours / 3,600s = ~1,000 QPS
  Concentrated billing windows create spikes

API Calls (self-service portal):
  Average: ~500 QPS
  Peak: ~2,000 QPS

Total System QPS:
  Average: ~3,500 QPS
  Peak: ~12,000 QPS

2.4 Bandwidth Estimates

Entitlement API:
  Request: ~500 B × 250M/day = 125 GB/day
  Response: ~1 KB × 250M/day = 250 GB/day

Billing API:
  Request: ~2 KB × 30M/day = 60 GB/day
  Response: ~5 KB × 30M/day = 150 GB/day

Invoice PDFs:
  ~200 KB × 30M/month = 6 TB/month (generated on demand, cached)

Webhooks:
  ~3 events/sub/month × 25M = 75M events/month
  ~1 KB/event = 75 GB/month

Total: ~400 GB/day API traffic + 6 TB/month PDFs

3. System Interface Design

3.1 Subscription Management API

Create Subscription

POST /v1/subscriptions
Content-Type: application/json
Authorization: Bearer <api_key>

{
  "customer_id": "cus_abc123",
  "plan_id": "plan_photoshop_annual",
  "quantity": 1,                      // seats
  "billing_cycle": "annual",          // monthly / annual / quarterly
  "trial_period_days": 7,             // optional
  "coupon_code": "WELCOME20",         // optional
  "payment_method_id": "pm_1N2xYz...",
  "metadata": {
    "channel": "web_direct",
    "campaign": "summer_2026"
  },
  "proration_behavior": "create_prorations",
  "backdate_start": "2026-07-01"      // optional, for enterprise
}

Response 201:
{
  "id": "sub_3N2xYz...",
  "object": "subscription",
  "customer_id": "cus_abc123",
  "status": "trialing",               // trialing / active / past_due / canceled / paused
  "plan_id": "plan_photoshop_annual",
  "plan": {
    "id": "plan_photoshop_annual",
    "name": "Photoshop - Annual",
    "amount": 23988,                  // cents ($239.988 = €22.99 converted)
    "currency": "usd",
    "interval": "year",
    "features": ["photoshop_desktop", "photoshop_web", "cloud_storage_100gb"]
  },
  "current_period_start": "2026-07-01T00:00:00Z",
  "current_period_end": "2027-07-01T00:00:00Z",
  "trial_end": "2026-07-08T00:00:00Z",
  "cancel_at_period_end": false,
  "cancel_at": null,
  "canceled_at": null,
  "paused_at": null,
  "quantity": 1,
  "started_at": "2026-07-01T00:00:00Z",
  "billing_cycle_anchor": "2026-07-01T00:00:00Z",
  "days_until_due": 30,
  "collection_method": "charge_automatically",  // or send_invoice
  "metadata": { ... }
}

Update Subscription (Upgrade/Downgrade)

PATCH /v1/subscriptions/sub_3N2xYz...
{
  "plan_id": "plan_creative_cloud_all",  // upgrade from single app
  "proration_behavior": "create_prorations",  // create_invoice_now / always_invoice / none
  "proration_date": "2026-08-15T00:00:00Z",  // optional, defaults to now
  "quantity": 3,                            // add seats
  "metadata": {
    "change_reason": "team_expansion"
  }
}

Response 200:
{
  "id": "sub_3N2xYz...",
  "status": "active",
  "plan_id": "plan_creative_cloud_all",
  "pending_update": null,
  "prorations": [
    {
      "id": "prorate_001",
      "type": "credit",
      "amount": 5432,           // credit for unused Photoshop time
      "description": "Proration credit: Photoshop Annual remaining 289 days",
      "line_item": "plan_photoshop_annual"
    },
    {
      "id": "prorate_002",
      "type": "charge",
      "amount": 12500,          // charge for new plan remaining time
      "description": "Proration charge: Creative Cloud All remaining 289 days",
      "line_item": "plan_creative_cloud_all"
    },
    {
      "id": "prorate_003",
      "type": "charge",
      "amount": 25000,          // additional seats
      "description": "Quantity change: +2 seats × $83.33/mo × 289 days",
      "line_item": "plan_creative_cloud_all_seats"
    }
  ],
  "net_proration": 33368,       // net charge this billing cycle
  "current_period_end": "2027-07-01T00:00:00Z"  // anchor preserved
}

Cancel Subscription

POST /v1/subscriptions/sub_3N2xYz.../cancel
{
  "cancel_at": "now",              // "now" or "period_end"
  "reason": "too_expensive",       // optional
  "feedback": "Found a cheaper alternative",
  "refund_proration": true         // refund unused time?
}

Response 200:
{
  "id": "sub_3N2xYz...",
  "status": "canceled",
  "canceled_at": "2026-08-15T10:30:00Z",
  "cancel_at_period_end": false,
  "refund": {
    "id": "re_abc123",
    "amount": 4500,
    "description": "Proration refund for unused 289 days"
  }
}

3.2 Entitlement API (Product Access Gate)

GET /v1/entitlements/check
Authorization: Bearer <customer_token>
X-Customer-ID: cus_abc123

Response 200:
{
  "customer_id": "cus_abc123",
  "entitled": true,
  "subscriptions": [
    {
      "id": "sub_3N2xYz...",
      "status": "active",
      "plan_id": "plan_creative_cloud_all",
      "features": [
        "photoshop_desktop",
        "photoshop_web",
        "illustrator_desktop",
        "indesign_desktop",
        "cloud_storage_100gb",
        "fonts_access",
        "behance_pro"
      ],
      "seats_remaining": 2,
      "period_end": "2027-07-01T00:00:00Z"
    }
  ],
  "blocked_features": [],
  "warnings": [
    {
      "code": "payment_past_due",
      "message": "Payment failed. Grace period ends 2026-08-25.",
      "grace_period_end": "2026-08-25T00:00:00Z"
    }
  ],
  "cached_until": "2026-08-15T10:35:00Z"  // 5-minute cache TTL
}

3.3 Usage Reporting API

POST /v1/usage/records
{
  "subscription_id": "sub_3N2xYz...",
  "metered_feature": "api_calls",
  "quantity": 15000,
  "timestamp": "2026-08-15T10:30:00Z",
  "action": "increment"    // increment / set
}

GET /v1/usage/summary?subscription_id=sub_3N2xYz...&period=current
Response 200:
{
  "subscription_id": "sub_3N2xYz...",
  "period_start": "2026-08-01T00:00:00Z",
  "period_end": "2026-08-31T00:00:00Z",
  "meters": [
    {
      "feature": "api_calls",
      "current_usage": 850000,
      "included_quantity": 1000000,
      "overage_quantity": 0,
      "overage_charge": 0,
      "unit_amount": 0.001  // $0.001 per call over limit
    },
    {
      "feature": "cloud_storage_gb",
      "current_usage": 75,
      "included_quantity": 100,
      "overage_quantity": 0,
      "overage_charge": 0,
      "unit_amount": 0.10   // $0.10 per GB over limit
    }
  ]
}

3.4 Webhook Events

Event Types:
  subscription.created
  subscription.activated
  subscription.updated
  subscription.paused
  subscription.resumed
  subscription.canceling          // cancel_at_period_end = true
  subscription.canceled
  subscription.reactivated
  subscription.trial_ending       // 3 days before trial ends
  invoice.created
  invoice.payment_succeeded
  invoice.payment_failed
  invoice.payment_overdue
  dunning.retry_scheduled
  dunning.max_retries_exceeded
  entitlement.changed
  customer.payment_method_expiring
  usage.threshold_reached         // 80%, 100%, 120% of included

4. High-Level Architecture

flowchart TB
    subgraph CLIENT["Client Layer"]
        WEB["Web Portal<br/>(Self-Service)"]
        APP["Creative Cloud<br/>Desktop App"]
        API["REST API Clients"]
        PRODUCT["Product Services<br/>(Photoshop, etc.)"]
    end

    subgraph EDGE["Edge Layer"]
        CDN["CDN / WAF"]
        LB["Load Balancer"]
        GW["API Gateway"]
    end

    subgraph CORE["Core Services"]
        subgraph SUB["Subscription Management"]
            SUB_MGR["Subscription<br/>Lifecycle Service"]
            PLAN_MGR["Plan & Pricing<br/>Service"]
            PRORATE["Proration<br/>Engine"]
            TRIAL["Trial<br/>Manager"]
        end

        subgraph BILL["Billing & Payments"]
            BILLING["Billing Cycle<br/>Engine"]
            INVOICE["Invoice<br/>Generator"]
            PAYMENT["Payment<br/>Processor"]
            DUNNING["Dunning<br/>Manager"]
            TAX["Tax Calculation<br/>Service"]
        end

        subgraph ENT["Entitlement & Access"]
            ENTITLE["Entitlement<br/>Service"]
            FEATURE["Feature Flag<br/>Service"]
            ACCESS["Access Control<br/>(Policy Engine)"]
        end

        subgraph USAGE["Usage & Metering"]
            METER["Usage Metering<br/>Service"]
            OVERAGE["Overage<br/>Calculator"]
        end

        subgraph PROMO["Promotions"]
            COUPON["Coupon & Discount<br/>Engine"]
            PROMO_VALID["Promo<br/>Validator"]
        end
    end

    subgraph NOTIF["Notification"]
        EMAIL["Email Service<br/>(Transactional)"]
        PUSH["Push Notifications"]
        WEBHOOK["Webhook<br/>Dispatcher"]
    end

    subgraph DATA["Data Layer"]
        RDB["Primary DB<br/>(PostgreSQL)"]
        REDIS["Redis Cluster<br/>(Cache / Entitlements)"]
        KAFKA["Kafka Cluster<br/>(Event Bus)"]
        ES["Elasticsearch<br/>(Search / Analytics)"]
        S3["Object Storage<br/>(Invoices / PDFs)"]
        TSDB["Time-Series DB<br/>(Usage Metrics)"]
    end

    subgraph ANALYTICS["Analytics"]
        BI["BI Dashboard<br/>(MRR, Churn, LTV)"]
        COHORT["Cohort Analysis"]
        PREDICT["Churn Prediction<br/>(ML)"]
    end

    WEB --> CDN
    APP --> CDN
    API --> CDN
    PRODUCT --> CDN
    CDN --> LB
    LB --> GW

    GW --> SUB_MGR
    GW --> PLAN_MGR
    GW --> ENTITLE
    GW --> METER
    GW --> BILLING

    SUB_MGR --> PRORATE
    SUB_MGR --> TRIAL
    SUB_MGR --> PLAN_MGR

    BILLING --> INVOICE
    BILLING --> PAYMENT
    BILLING --> DUNNING
    BILLING --> TAX
    BILLING --> COUPON

    ENTITLE --> FEATURE
    ENTITLE --> ACCESS

    METER --> OVERAGE
    OVERAGE --> BILLING

    SUB_MGR --> KAFKA
    BILLING --> KAFKA
    DUNNING --> KAFKA
    ENTITLE --> KAFKA
    METER --> KAFKA

    KAFKA --> EMAIL
    KAFKA --> PUSH
    KAFKA --> WEBHOOK
    KAFKA --> ES
    KAFKA --> TSDB
    KAFKA --> BI

    SUB_MGR --> RDB
    PLAN_MGR --> RDB
    BILLING --> RDB
    INVOICE --> RDB
    PAYMENT --> RDB
    DUNNING --> RDB
    METER --> RDB

    ENTITLE --> REDIS
    FEATURE --> REDIS
    SUB_MGR --> REDIS

    INVOICE --> S3
    BI --> ES

5. Core Component Design

5.1 Subscription Lifecycle Service

Central state machine managing subscription transitions:

stateDiagram-v2
    [*] --> Trialing: create(trial=true)
    [*] --> Active: create(trial=false)

    Trialing --> Active: trial_end + payment succeeds
    Trialing --> Canceled: customer cancels during trial
    Trialing --> Past Due: trial_end + payment fails

    Active --> Past Due: payment fails
    Active --> Canceling: cancel(at_period_end)
    Active --> Paused: pause()
    Active --> Canceled: cancel(immediate)

    Past Due --> Active: dunning retry succeeds
    Past Due --> Canceled: max retries exceeded
    Past Due --> Past Due: dunning retry scheduled

    Canceling --> Canceled: period_end reached
    Canceling --> Active: reinstate(cancel_at_period_end=false)

    Paused --> Active: resume()
    Paused --> Canceled: cancel during pause

    Canceled --> Active: reactivate(new payment)
    Canceled --> [*]: retention period expired

State Transition Rules

From To Trigger Side Effects
Trialing → Active Trial expires + payment succeeds Cron / event Charge customer, emit subscription.activated
Trialing → Past Due Trial expires + payment fails Cron / event Start dunning sequence, emit invoice.payment_failed
Active → Past Due Payment fails on renewal Billing cycle Start dunning, grace period begins
Past Due → Active Retry succeeds Dunning retry Resume access, emit invoice.payment_succeeded
Past Due → Canceled Max retries exceeded Dunning timeout Revoke access, emit subscription.canceled
Active → Canceling Customer requests cancel API call Set cancel_at_period_end=true, no immediate charge
Canceling → Canceled Period end reached Cron / event Revoke access, emit subscription.canceled
Active → Paused Customer pauses API call Suspend access, stop billing, freeze period end
Paused → Active Customer resumes API call Restore access, adjust period end
Canceled → Active Customer reactivates API call New charge, new billing cycle

5.2 Proration Engine

The most complex billing calculation — handles mid-cycle plan changes accurately.

Proration Calculation Algorithm:

Given:
  - Current plan: $20/month (Photoshop)
  - New plan: $55/month (Creative Cloud All)
  - Billing cycle: Monthly
  - Current period start: 2026-07-01
  - Current period end: 2026-08-01
  - Change date: 2026-07-15 (day 15 of 31-day cycle)

Step 1: Calculate remaining days
  remaining_days = period_end - change_date = 31 - 15 = 16 days
  elapsed_days = change_date - period_start = 15 days
  total_days = 31

Step 2: Calculate daily rates
  old_daily_rate = old_plan_amount / total_days = $20.00 / 31 = $0.6452/day
  new_daily_rate = new_plan_amount / total_days = $55.00 / 31 = $1.7742/day

Step 3: Calculate proration amounts
  credit_amount = old_daily_rate × elapsed_days = $0.6452 × 15 = $9.68
  charge_amount = new_daily_rate × remaining_days = $1.7742 × 16 = $28.39

Step 4: Net proration
  net_charge = charge_amount - credit_amount = $28.39 - $9.68 = $18.71

Step 5: Apply to invoice
  - Credit line item: -$9.68 (Photoshop remaining credit)
  - Charge line item: +$28.39 (Creative Cloud All remaining)
  - Net: +$18.71 charged immediately

Proration Behaviors:
  ┌─────────────────────┬──────────────────────────────────────────┐
  │ Mode                │ Behavior                                 │
  ├─────────────────────┼──────────────────────────────────────────┤
  │ create_prorations   │ Create credit/charge line items now      │
  │ create_invoice_now  │ Create prorations + immediate invoice    │
  │ always_invoice      │ Always charge (no credits, roll forward) │
  │ none                │ No proration (change at period end)      │
  └─────────────────────┴──────────────────────────────────────────┘

Edge Cases:
  - Year-to-month: Prorate remaining days of year at monthly rate
  - Month-to-year: Credit remaining month days, charge full year
  - Quantity changes: Prorate only the delta (not full recalculation)
  - Coupon changes: Prorate coupon value proportionally
  - Multi-line items: Prorate each line item independently

Proration Precision

Precision Strategy:
  - All calculations in smallest currency unit (cents)
  - Use decimal arithmetic (NOT floating point)
  - Round half-up at the final step only
  - Intermediate calculations keep full precision
  - Audit trail: Store all intermediate values for reconciliation

Example with cents:
  old_daily_rate = 2000¢ / 31 = 64.516129...¢/day
  credit = 64.516129... × 15 = 967.741935...¢ → 968¢ (rounded)
  new_daily_rate = 5500¢ / 31 = 177.419354...¢/day
  charge = 177.419354... × 16 = 2838.709677...¢ → 2839¢ (rounded)
  net = 2839 - 968 = 1871¢ = $18.71

5.3 Dunning Management

Automated retry and notification system for failed payments:

flowchart TD
    START[Payment Failed] --> R1[Retry 1<br/>+1 hour<br/>Email: Payment Failed]
    R1 -->|Success| RESUME[Resume Access<br/>Recover Subscription]
    R1 -->|Fail| R2[Retry 2<br/>+3 days<br/>Email: Final Notice]
    R2 -->|Success| RESUME
    R2 -->|Fail| R3[Retry 3<br/>+7 days<br/>Email + SMS: Last Chance]
    R3 -->|Success| RESUME
    R3 -->|Fail| R4[Retry 4<br/>+14 days<br/>Email: Account Suspended]
    R4 -->|Success| RESUME
    R4 -->|Fail| SUSPEND[Suspend Access<br/>Status: Past Due → Canceled]
    SUSPEND --> WINBACK[Win-back Campaign<br/>Email: We Miss You<br/>+30 days]
    WINBACK -->|Returns| REACTIVATE[Reactivate<br/>New Payment]
    WINBACK -->|No Response| ARCHIVE[Archive<br/>+90 days]

    style RESUME fill:#90EE90
    style SUSPEND fill:#FFB6C1
    style REACTIVATE fill:#90EE90

Dunning Configuration

Configurable per merchant / plan:

┌──────────────────┬────────────┬────────────┬──────────────────┐
│ Parameter        │ Default    │ Range      │ Notes            │
├──────────────────┼────────────┼────────────┼──────────────────┤
│ Grace Period     │ 30 days    │ 0-90 days  │ Access kept      │
│ Retry Attempts   │ 4          │ 1-10       │ Total retries    │
│ Retry Intervals  │ 1h,3d,7d,14d │ Custom  │ Exponential      │
│ Email Templates  │ 3          │ 1-5        │ Per retry stage  │
│ SMS Enabled      │ false      │ bool       │ High-value only  │
│ Auto-cancel      │ true       │ bool       │ After grace      │
│ Win-back         │ true       │ bool       │ Post-cancellation│
│ Card Update Nudge│ true       │ bool       │ Before expiry    │
└──────────────────┴────────────┴────────────┴──────────────────┘

Dunning Metrics:
  - Recovery Rate: % of failed payments recovered via retries
  - Dunning-Induced Churn: % canceled during grace period
  - Average Recovery Time: Mean time from failure to recovery
  - Card Update Rate: % updating card before expiry

5.4 Entitlement Service

Real-time access control with caching:

Entitlement Evaluation Pipeline:

1. Request: Product service checks entitlement
   GET /entitlements/check?customer_id=cus_abc&feature=photoshop_desktop

2. Cache Check (Redis, < 1ms):
   Key: entitlement:{customer_id}:{feature}
   TTL: 5 minutes (configurable)
   If hit → return cached result

3. If cache miss, evaluate:
   a. Look up customer's active subscriptions
   b. For each subscription:
      - Check status (must be active or in grace period)
      - Check plan features (does plan include requested feature?)
      - Check quantity (seats remaining?)
      - Check billing status (not past_due beyond grace)
   c. Combine results:
      - entitled = ANY subscription grants feature
      - blocked = reasons if not entitled
      - warnings = approaching limits

4. Cache Result:
   SET entitlement:{customer_id}:{feature} = result TTL 5min

5. Return:
   { entitled: true, features: [...], expires_at: ... }

Invalidation Strategy:
  - Subscription state change → invalidate customer's entitlement cache
  - Plan update → invalidate all affected customers' caches
  - Bulk invalidation via Redis SCAN + DEL pattern
  - Max invalidation batch: 10,000 keys/second

Grace Period Handling:
  - Past Due + within grace period → entitled = true (with warning)
  - Past Due + grace expired → entitled = false
  - Canceling (at period end) → entitled = true until period end
  - Paused → entitled = false

5.5 Billing Cycle Engine

Deterministic billing schedule management:

Billing Cycle Management:

Anchor Date:
  - Set at subscription creation (or backdated)
  - All future billing dates derived from anchor
  - Survives plan changes (upgrade/downgrade preserve anchor)
  - Formula: next_billing = anchor + N × interval

Billing Schedule Generation:
  1. Daily job scans subscriptions due for billing:
     WHERE current_period_end <= NOW() + INTERVAL '1 day'
       AND status IN ('active', 'trialing')
  2. For each due subscription:
     a. Generate invoice with line items:
        - Base plan charge
        - Proration adjustments (if any)
        - Usage overage charges
        - Coupon/discount deductions
        - Tax calculations
     b. Attempt payment
     c. On success: Advance billing period
     d. On failure: Start dunning sequence

Billing Window:
  - Not all subscribers bill on the same day
  - Anchor dates distributed across month
  - Reduces peak load on payment processor
  - Enterprise: Can align to fiscal month-end

Trial Handling:
  - Trial period does NOT create billing cycle
  - First charge occurs at trial_end
  - Billing anchor = trial_end (not trial_start)
  - If trial canceled: No charge, no billing cycle created

5.6 Invoice Generator

Invoice Generation Pipeline:

1. Trigger: Billing cycle engine creates invoice draft
2. Line Items:
   ┌─────────────────────────────────────────────────────────┐
   │ Invoice #INV-2026-001234                                │
   │ Date: 2026-08-01                                        │
   │ Due: 2026-08-31                                         │
   ├──────────────────┬────────────┬──────────┬──────────────┤
   │ Description      │ Qty        │ Unit     │ Amount       │
   ├──────────────────┼────────────┼──────────┼──────────────┤
   │ Creative Cloud   │ 1          │ $54.99   │ $54.99       │
   │ All Apps - Aug   │            │/mo       │              │
   ├──────────────────┼────────────┼──────────┼──────────────┤
   │ Additional Seat  │ 2          │ $54.99   │ $109.98      │
   │ (Jul 15 change)  │            │/mo       │              │
   ├──────────────────┼────────────┼──────────┼──────────────┤
   │ Proration Credit │ 1          │ -$9.68   │ -$9.68       │
   │ Photoshop Jul    │            │          │              │
   ├──────────────────┼────────────┼──────────┼──────────────┤
   │ Summer Discount  │ 1          │ -$13.75  │ -$13.75      │
   │ (20% off 1 mo)   │            │          │              │
   ├──────────────────┼────────────┼──────────┼──────────────┤
   │ Subtotal         │            │          │ $141.54      │
   │ Tax (VAT 20%)    │            │          │ $28.31       │
   │ TOTAL            │            │          │ $169.85      │
   └──────────────────┴────────────┴──────────┴──────────────┘

3. Tax Calculation:
   - Determine customer jurisdiction (billing address)
   - Look up tax rate (cached, updated monthly)
   - Apply tax to taxable line items
   - Store tax breakdown for compliance

4. PDF Generation:
   - Template-based PDF generation
   - Company branding, logo, terms
   - Stored in S3, URL provided in invoice record

5. Delivery:
   - Email with PDF attachment
   - Available in self-service portal
   - Webhook: invoice.created

5.7 Usage Metering Service

Usage Ingestion Pipeline:

1. Product Service reports usage:
   POST /usage/records
   { subscription_id, feature, quantity, timestamp }

2. Validation:
   - Subscription exists and active?
   - Feature is metered on this plan?
   - Quantity is valid (positive, within reason)?

3. Aggregation:
   - Raw events → Kafka topic: usage.raw
   - Stream processor aggregates per subscription per day:
     SELECT subscription_id, feature, SUM(quantity)
     FROM usage_events
     WHERE period = current_billing_period
     GROUP BY subscription_id, feature

4. Storage:
   - Aggregated: PostgreSQL (per-subscription per-day totals)
   - Raw events: Time-series DB (for debugging, audit)
   - Retention: Raw 30 days, Aggregated 7 years

5. Threshold Alerts:
   - Monitor usage vs included quantity
   - At 80%: Send warning email
   - At 100%: Send overage notice
   - At 120%: Send critical notice

6. Overage Calculation (at billing):
   - Pull aggregated usage for billing period
   - Calculate overage = max(0, usage - included)
   - Charge = overage × unit_price
   - Add as line item to invoice

6. Data Model

6.1 Core Tables

-- Customers
CREATE TABLE customers (
    id              VARCHAR(32)  PRIMARY KEY,  -- cus_abc123
    email           VARCHAR(255) NOT NULL,
    name            VARCHAR(255),
    phone           VARCHAR(32),
    default_currency CHAR(3)    DEFAULT 'usd',
    tax_id          VARCHAR(64),
    billing_address JSONB,
    metadata        JSONB,
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    updated_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    mrr             BIGINT       DEFAULT 0,    -- current MRR in cents

    CONSTRAINT uq_customer_email UNIQUE(email)
);

-- Plans (Product Pricing)
CREATE TABLE plans (
    id              VARCHAR(64)  PRIMARY KEY,  -- plan_photoshop_annual
    product_id      VARCHAR(64)  NOT NULL,
    name            VARCHAR(255) NOT NULL,
    description     TEXT,
    amount          BIGINT       NOT NULL,     -- in smallest currency unit
    currency        CHAR(3)      NOT NULL,
    interval        VARCHAR(20)  NOT NULL,     -- day / week / month / year
    interval_count  SMALLINT     DEFAULT 1,    // e.g., 3 for quarterly
    trial_period_days SMALLINT,
    features        JSONB      NOT NULL,       -- ["photoshop_desktop", ...]
    metadata        JSONB,
    active          BOOLEAN    DEFAULT true,
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    sort_order      INTEGER    DEFAULT 0,

    CONSTRAINT chk_interval CHECK (interval IN ('day', 'week', 'month', 'year'))
);

CREATE INDEX idx_plans_product ON plans(product_id);
CREATE INDEX idx_plans_active ON plans(active, sort_order) WHERE active = true;

-- Subscriptions (Core Entity)
CREATE TABLE subscriptions (
    id              VARCHAR(32)  PRIMARY KEY,  -- sub_3N2xYz...
    customer_id     VARCHAR(32)  NOT NULL,
    plan_id         VARCHAR(64)  NOT NULL,
    status          VARCHAR(20)  NOT NULL,     -- trialing / active / past_due / canceled / paused / canceling
    quantity        INTEGER    DEFAULT 1,       -- seats

    -- Timing
    current_period_start TIMESTAMPTZ NOT NULL,
    current_period_end   TIMESTAMPTZ NOT NULL,
    billing_cycle_anchor TIMESTAMPTZ NOT NULL,
    trial_start          TIMESTAMPTZ,
    trial_end            TIMESTAMPTZ,

    -- Cancellation
    cancel_at_period_end BOOLEAN    DEFAULT false,
    canceled_at          TIMESTAMPTZ,
    cancel_reason        VARCHAR(100),
    cancellation_feedback TEXT,

    -- Pause
    paused_at            TIMESTAMPTZ,
    resume_at            TIMESTAMPTZ,

    -- Billing
    collection_method    VARCHAR(30) DEFAULT 'charge_automatically',
    days_until_due       SMALLINT   DEFAULT 30,
    payment_method_id    VARCHAR(32),
    default_tax_rates    JSONB,

    -- Dunning
    dunning_status       VARCHAR(20) DEFAULT 'none',  -- none / in_progress / failed
    dunning_attempts     SMALLINT   DEFAULT 0,
    dunning_next_retry   TIMESTAMPTZ,
    dunning_max_attempts SMALLINT   DEFAULT 4,

    -- Metadata
    metadata             JSONB,
    started_at           TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    created_at           TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    updated_at           TIMESTAMPTZ  NOT NULL DEFAULT NOW(),

    CONSTRAINT fk_customer FOREIGN KEY(customer_id) REFERENCES customers(id),
    CONSTRAINT fk_plan FOREIGN KEY(plan_id) REFERENCES plans(id),
    CONSTRAINT chk_status CHECK (status IN (
        'trialing', 'active', 'past_due', 'canceled', 'paused', 'canceling'
    ))
);

CREATE INDEX idx_sub_customer ON subscriptions(customer_id, status);
CREATE INDEX idx_sub_period_end ON subscriptions(current_period_end, status)
    WHERE status IN ('active', 'trialing');
CREATE INDEX idx_sub_dunning ON subscriptions(dunning_next_retry, dunning_status)
    WHERE dunning_status = 'in_progress' AND dunning_next_retry <= NOW();
CREATE INDEX idx_sub_canceling ON subscriptions(current_period_end, status)
    WHERE status = 'canceling';

-- Subscription Events (Audit Trail)
CREATE TABLE subscription_events (
    id              BIGINT       PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
    subscription_id VARCHAR(32)  NOT NULL,
    event_type      VARCHAR(50)  NOT NULL,     -- created / activated / upgraded / downgraded / canceled
    previous_status VARCHAR(20),
    new_status      VARCHAR(20),
    previous_plan   VARCHAR(64),
    new_plan        VARCHAR(64),
    metadata        JSONB,
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW(),

    CONSTRAINT fk_sub FOREIGN KEY(subscription_id) REFERENCES subscriptions(id)
);

CREATE INDEX idx_sev_sub ON subscription_events(subscription_id, created_at DESC);

-- Invoices
CREATE TABLE invoices (
    id              VARCHAR(32)  PRIMARY KEY,  -- inv_3N2xYz...
    customer_id     VARCHAR(32)  NOT NULL,
    subscription_id VARCHAR(32),
    number          VARCHAR(32)  NOT NULL,     -- INV-2026-001234
    status          VARCHAR(20)  NOT NULL,     -- draft / open / paid / void / uncollectible
    currency        CHAR(3)      NOT NULL,

    -- Amounts
    subtotal        BIGINT       NOT NULL,
    tax_amount      BIGINT       DEFAULT 0,
    discount_amount BIGINT       DEFAULT 0,
    proration_credit BIGINT      DEFAULT 0,
    total           BIGINT       NOT NULL,
    amount_paid     BIGINT       DEFAULT 0,
    amount_due      BIGINT       NOT NULL,

    -- Dates
    period_start    TIMESTAMPTZ  NOT NULL,
    period_end      TIMESTAMPTZ  NOT NULL,
    due_date        TIMESTAMPTZ  NOT NULL,
    issued_at       TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    paid_at         TIMESTAMPTZ,
    voided_at       TIMESTAMPTZ,

    -- Payment
    payment_method_id VARCHAR(32),
    payment_intent_id VARCHAR(32),

    -- Documents
    pdf_url         VARCHAR(512),
    host_url        VARCHAR(512),

    -- Tax
    tax_breakdown   JSONB,
    billing_address JSONB,

    metadata        JSONB,
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW(),

    CONSTRAINT chk_inv_status CHECK (status IN ('draft', 'open', 'paid', 'void', 'uncollectible'))
);

CREATE INDEX idx_inv_customer ON invoices(customer_id, created_at DESC);
CREATE INDEX idx_inv_status ON invoices(status, due_date);
CREATE INDEX idx_inv_number ON invoices(number);

-- Invoice Line Items
CREATE TABLE invoice_line_items (
    id              BIGINT       PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
    invoice_id      VARCHAR(32)  NOT NULL,
    type            VARCHAR(20)  NOT NULL,     -- subscription / proration / usage / coupon / tax
    description     TEXT       NOT NULL,
    quantity        BIGINT     DEFAULT 1,
    unit_amount     BIGINT     NOT NULL,       -- can be negative for credits
    amount          BIGINT     NOT NULL,       -- quantity × unit_amount
    proration_start TIMESTAMPTZ,
    proration_end   TIMESTAMPTZ,
    metadata        JSONB,
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW(),

    CONSTRAINT fk_invoice FOREIGN KEY(invoice_id) REFERENCES invoices(id),
    CONSTRAINT chk_li_type CHECK (type IN ('subscription', 'proration', 'usage', 'coupon', 'tax', 'fee'))
);

CREATE INDEX idx_ili_invoice ON invoice_line_items(invoice_id);

-- Coupons
CREATE TABLE coupons (
    id              VARCHAR(32)  PRIMARY KEY,  -- coupon_summer20
    code            VARCHAR(64)  NOT NULL,
    name            VARCHAR(255),
    description     TEXT,
    type            VARCHAR(20)  NOT NULL,     -- percent / amount / duration
    percent_off     DECIMAL(5,2),
    amount_off      BIGINT,
    currency        CHAR(3),
    duration        VARCHAR(20)  NOT NULL,     -- once / repeating / forever
    duration_in_months INTEGER,
    max_redemptions INTEGER,
    redemption_count INTEGER DEFAULT 0,
    valid_from      TIMESTAMPTZ,
    valid_until     TIMESTAMPTZ,
    applicable_plans VARCHAR(64)[],            -- specific plans or NULL for all
    stackable       BOOLEAN    DEFAULT false,
    active          BOOLEAN    DEFAULT true,
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW(),

    CONSTRAINT uq_coupon_code UNIQUE(code),
    CONSTRAINT chk_coupon_type CHECK (type IN ('percent', 'amount', 'duration'))
);

-- Customer-Coupon Associations
CREATE TABLE customer_coupons (
    id              BIGINT       PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
    customer_id     VARCHAR(32)  NOT NULL,
    coupon_id       VARCHAR(32)  NOT NULL,
    subscription_id VARCHAR(32),
    deleted         BOOLEAN    DEFAULT false,
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW(),

    CONSTRAINT fk_cust FOREIGN KEY(customer_id) REFERENCES customers(id),
    CONSTRAINT fk_coupon FOREIGN KEY(coupon_id) REFERENCES coupons(id),
    CONSTRAINT fk_sub FOREIGN KEY(subscription_id) REFERENCES subscriptions(id),
    CONSTRAINT uq_cust_coupon_sub UNIQUE(customer_id, coupon_id, subscription_id)
);

-- Usage Records
CREATE TABLE usage_records (
    id              BIGINT       PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
    subscription_id VARCHAR(32)  NOT NULL,
    feature         VARCHAR(100) NOT NULL,
    quantity        BIGINT       NOT NULL,
    action          VARCHAR(20)  DEFAULT 'increment',  -- increment / set
    timestamp       TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW(),

    CONSTRAINT fk_sub_usage FOREIGN KEY(subscription_id) REFERENCES subscriptions(id)
);

CREATE INDEX idx_usage_sub_feature ON usage_records(subscription_id, feature, timestamp);

-- Usage Aggregates (materialized, refreshed daily)
CREATE TABLE usage_aggregates (
    id              BIGINT       PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
    subscription_id VARCHAR(32)  NOT NULL,
    feature         VARCHAR(100) NOT NULL,
    period_start    DATE       NOT NULL,
    period_end      DATE       NOT NULL,
    total_quantity  BIGINT     NOT NULL,
    overage_quantity BIGINT    DEFAULT 0,
    overage_charge  BIGINT     DEFAULT 0,
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    updated_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW(),

    CONSTRAINT uq_agg UNIQUE(subscription_id, feature, period_start)
);

-- Entitlement Cache (denormalized for fast reads)
CREATE TABLE entitlement_cache (
    customer_id     VARCHAR(32)  PRIMARY KEY,
    features        JSONB      NOT NULL,       -- { "photoshop_desktop": true, ... }
    subscriptions   JSONB      NOT NULL,       -- summary of active subs
    blocked_features JSONB,
    warnings        JSONB,
    version         BIGINT     NOT NULL,       -- increment on change
    updated_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW(),
    expires_at      TIMESTAMPTZ  NOT NULL      -- TTL for stale detection
);

-- Payment Methods
CREATE TABLE payment_methods (
    id              VARCHAR(32)  PRIMARY KEY,
    customer_id     VARCHAR(32)  NOT NULL,
    type            VARCHAR(20)  NOT NULL,     -- card / bank_account
    card_token      VARCHAR(64),
    card_brand      VARCHAR(20),
    card_last4      CHAR(4),
    card_exp_month  SMALLINT,
    card_exp_year   SMALLINT,
    is_default      BOOLEAN    DEFAULT false,
    billing_details JSONB,
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW(),

    CONSTRAINT fk_cust_pm FOREIGN KEY(customer_id) REFERENCES customers(id)
);

-- Dunning Attempts
CREATE TABLE dunning_attempts (
    id              BIGINT       PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
    subscription_id VARCHAR(32)  NOT NULL,
    invoice_id      VARCHAR(32)  NOT NULL,
    attempt_number  SMALLINT   NOT NULL,
    status          VARCHAR(20)  NOT NULL,     -- scheduled / sent / succeeded / failed
    scheduled_at    TIMESTAMPTZ  NOT NULL,
    executed_at     TIMESTAMPTZ,
    error_code      VARCHAR(50),
    error_message   TEXT,
    created_at      TIMESTAMPTZ  NOT NULL DEFAULT NOW(),

    CONSTRAINT fk_sub_dunning FOREIGN KEY(subscription_id) REFERENCES subscriptions(id)
);

CREATE INDEX idx_dunning_next ON dunning_attempts(scheduled_at, status)
    WHERE status = 'scheduled' AND scheduled_at <= NOW();

6.2 Partitioning and Indexing Strategy

Partitioning:
  - invoices: Range partition by created_at (monthly)
  - invoice_line_items: Range partition by created_at (monthly)
  - subscription_events: Range partition by created_at (monthly)
  - usage_records: Range partition by timestamp (monthly)
  - dunning_attempts: Range partition by created_at (monthly)

Rationale:
  - Billing data is time-accessed (last N months)
  - Old partitions can be archived to cold storage
  - Partition pruning for date-range queries
  - Compliance: Easy DROP for data retention

Critical Indexes:
  ┌────────────────────────────┬─────────────────────────────────┬──────────────────┐
  │ Index                      │ Columns                         │ Purpose          │
  ├────────────────────────────┼─────────────────────────────────┼──────────────────┤
  │ idx_sub_period_end         │ current_period_end, status      │ Billing scan     │
  │ idx_sub_customer           │ customer_id, status             │ Customer lookup  │
  │ idx_sub_dunning            │ dunning_next_retry, status      │ Dunning scan     │
  │ idx_inv_customer           │ customer_id, created_at DESC    │ Invoice history  │
  │ idx_usage_sub_feature      │ sub, feature, timestamp         │ Usage aggregation│
  └────────────────────────────┴─────────────────────────────────┴──────────────────┘

7. Key Design Decisions

7.1 Billing Cycle Anchor vs. Calendar Alignment

Aspect Anchor Date (Chosen) Calendar Month
Customer experience Consistent billing date each cycle Bills on 1st of every month
Proration complexity Simpler (always from anchor) Complex (partial first/last month)
Revenue recognition Clear period boundaries Misaligned for mid-month signups
Cash flow predictability Distributed across month Spike on 1st of month
Enterprise alignment Can backdate to fiscal month Fixed to calendar

Decision: Anchor date provides flexibility and simplicity. Enterprise customers can request fiscal-month alignment via backdating.

7.2 Centralized vs. Distributed Entitlements

Aspect Centralized Service (Chosen) Embedded in Product
Consistency Single source of truth Risk of divergence
Latency Added network hop Zero latency
Caching Centralized cache invalidation Per-product cache management
Complexity One service to maintain N services to maintain
Cross-product features Easy (bundle entitlements) Hard (cross-service calls)

Decision: Centralized entitlement service with aggressive caching (5-min TTL, Redis). Product services call entitlement API on startup and cache locally. Cache invalidated via event-driven invalidation on subscription changes.

7.3 Synchronous vs. Asynchronous Billing

Aspect Async Billing (Chosen) Synchronous
Customer experience No blocking at signup Immediate charge confirmation
Reliability Retry on failure Must handle failure inline
Batching Can batch invoices One-at-a-time
Complexity Event-driven pipeline Simpler flow
Visibility Invoice available after processing Immediate

Decision: Async billing with event-driven pipeline. Trial subscriptions charge at trial_end via scheduled job. Immediate charge option available for enterprise via synchronous API path.

7.4 Proration: Credit vs. Roll-Forward

Aspect Credit + Charge (Chosen) Roll-Forward
Transparency Clear credit and charge lines Hidden in next invoice
Cash flow Immediate net settlement Deferred to next cycle
Customer understanding “I got $X credit, charged $Y” “Why is my bill different?”
Accounting Clean period boundaries Carried-forward complexity
Refund handling Credits can offset charges Must wait for next invoice

Decision: Credit + charge provides maximum transparency. Customers see exactly what they’re getting credited and charged for. Configurable to roll-forward for merchants who prefer it.

7.5 Entitlement Cache Strategy

Cache Design:

Tier 1: Product-side LRU Cache (TTL: 60 seconds)
  - Each product service caches entitlement results
  - Reduces API calls for frequent checks
  - Evicted on subscription change event

Tier 2: Entitlement Service Redis Cache (TTL: 5 minutes)
  - Key: entitlement:{customer_id}
  - Value: Full entitlement snapshot (features, warnings, subs)
  - Invalidated on any subscription change for customer

Tier 3: Database (source of truth)
  - entitlement_cache table (denormalized)
  - Updated asynchronously from subscription events
  - Used for cache warming and recovery

Invalidation Flow:
  Subscription Change → Kafka Event → Entitlement Service
    → Update DB cache
    → Invalidate Redis key
    → Publish cache_invalidation event
    → Product services receive → evict local cache

8. Scalability Strategy

8.1 Horizontal Scaling Architecture

flowchart TB
    subgraph ZONE1["Zone A"]
        GW1["API Gateway × 4"]
        SUB1["Sub Service × 3"]
        BILL1["Billing × 2"]
        ENT1["Entitlement × 6"]
        REDIS1["Redis Shard 1-4"]
    end

    subgraph ZONE2["Zone B"]
        GW2["API Gateway × 4"]
        SUB2["Sub Service × 3"]
        BILL2["Billing × 2"]
        ENT2["Entitlement × 6"]
        REDIS2["Redis Shard 5-8"]
    end

    subgraph SHARED["Shared Services"]
        KAFKA["Kafka Cluster<br/>(6 brokers × 3 AZ)"]
        RDB["PostgreSQL<br/>(Primary + 4 Replicas)"]
        ES["Elasticsearch<br/>(6 nodes)"]
    end

    GW1 --> SUB1 & BILL1 & ENT1
    GW2 --> SUB2 & BILL12 & ENT2
    SUB1 & SUB2 --> KAFKA
    BILL1 & BILL12 --> KAFKA
    ENT1 --> REDIS1
    ENT2 --> REDIS2
    SUB1 & SUB2 & BILL1 & BILL12 --> RDB
    KAFKA --> ES

8.2 Scaling by Component

Component Scaling Strategy Bottleneck Max Scale
Entitlement Service Stateless, auto-scale on QPS Redis throughput 50K QPS with caching
Subscription Service Stateless, partition by customer_id Database write throughput 10K TPS with sharding
Billing Engine Partitioned by billing window Payment processor rate limits 5K TPS (processor-bound)
Dunning Manager Partitioned by retry window Email/SMS rate limits 1M retries/day
Usage Metering Stream processing (Kafka Streams) Aggregation compute 100K events/sec
Invoice Generator Stateless, queue-based PDF generation CPU 10K invoices/min

8.3 Database Scaling

Phase 1: Single PostgreSQL (0-1M subscribers)
  - Primary + 3 read replicas
  - Connection pooling (PgBouncer)
  - Read/write split at app layer

Phase 2: Citus Sharding (1M-10M subscribers)
  - Shard key: customer_id (consistent hash)
  - 16 shards across 4 coordinator nodes
  - Co-located: customer, subscriptions, invoices
  - Distributed queries for analytics

Phase 3: Multi-Region (10M+ subscribers)
  - Regional primaries (US, EU, APAC)
  - Cross-region replication for DR
  - Customer assigned to region at signup
  - Analytics: Centralized data warehouse (Snowflake/BigQuery)

Read Scaling:
  - Entitlement reads: Redis (99.9% cache hit rate)
  - Dashboard reads: Read replicas + materialized views
  - Analytics: Elasticsearch + data warehouse
  - Invoice history: Read replicas with partition pruning

8.4 Billing Spike Management

Problem: All subscribers with anchor on the 1st bill simultaneously.

Solutions:
1. Anchor Distribution (Prevention):
   - New subscribers: Randomize anchor within ±7 days of signup
   - Enterprise: Allow specific anchor date
   - Result: Billing spread across 15-day window

2. Billing Queue (Absorption):
   - Billing requests enter priority queue
   - Processed in batches of 1,000
   - Rate-limited to payment processor limits
   - Queue depth monitored, auto-scale consumers

3. Payment Processor Scaling:
   - Multiple processor backends
   - Load balance across processors
   - Circuit breaker per processor
   - Failover on processor degradation

4. Grace Period Buffer:
   - Invoices generated ahead of due date
   - Payment attempted with 3-day window
   - Reduces exact-moment pressure

9. Curveball Scenarios

9.1 Curveball 1: “A major plan price increase causes a 15% churn spike. Customers are upgrading to competitors mid-cycle. How do you handle the billing and entitlement chaos?”

Impact Assessment:

  • 15% churn = massive cancellation wave
  • Mid-cycle cancellations trigger proration refunds
  • Revenue drop impacts cash flow
  • Remaining customers may demand price matches

Response Strategy:

Phase 1: Immediate Triage (0-2 hours)
  1. Assess scope:
     - How many cancellations in the last hour?
     - Which plans are most affected?
     - Are cancellations immediate or at period end?

  2. System protection:
     - Monitor proration engine load (cancellations = proration calcs)
     - Ensure refund processing queue isn't backing up
     - Verify entitlement cache invalidation is keeping up

  3. Communication:
     - Alert customer success team
     - Prepare retention offers for at-risk segments
     - Update churn dashboard with real-time metrics

Phase 2: Billing Impact Management (2-8 hours)
  1. Proration flood handling:
     - Batch proration calculations (not real-time per cancellation)
     - Queue cancellations, process proration async
     - Customer sees "refund processing" status immediately
     - Actual refund within 24 hours

  2. Revenue recognition adjustment:
     - Recalculate MRR/ARR with new churn rate
     - Update finance dashboards
     - Flag for board/leadership review

  3. Remaining customer impact:
     - Ensure no accidental price increases on existing subscriptions
     - Price increase should ONLY apply to new signups and renewals
     - Verify: No active subscription was charged at new price

Phase 3: Retention & Recovery (8-48 hours)
  1. Win-back offers:
     - Create limited-time discount for canceling customers
     - "Come back within 30 days at old price" coupon
     - Automated email sequence for churned customers

  2. Price match program:
     - For customers citing competitor pricing
     - Manual approval workflow for exceptions

  3. Product feedback loop:
     - Analyze cancellation reasons
     - Identify if price was the only factor
     - Feed insights to product team

System Resilience:
  - Proration engine handled 3x normal load without degradation
  - Refund queue processed all backlogged items within 18 hours
  - Entitlement cache invalidation kept up (no access leaks)

9.2 Curveball 2: “Your billing engine has a bug that double-billed customers on their annual renewal. 50,000 customers affected. Discovery happens 3 days later. What’s your response?”

Impact Assessment:

  • 50,000 customers overcharged (massive reputation risk)
  • Chargeback wave incoming
  • Regulatory notification may be required
  • Customer trust severely damaged

Response Strategy:

Phase 1: Containment (0-1 hour)
  1. IMMEDIATELY: Disable annual renewal billing path
     - Feature flag: annual_renewal_enabled = false
     - Switch to manual review for annual renewals

  2. Scope the damage:
     SELECT subscription_id, invoice_id, amount
     FROM invoices
     WHERE period_start >= '2026-07-01'
       AND plan_interval = 'year'
       AND status = 'paid'
     -- Cross-reference with payment records for duplicates

  3. Verify: Confirm bug affects ONLY annual renewals
     - Monthly renewals:     - New signups unaffected
     - Proration charges unaffected

Phase 2: Mass Refund (1-6 hours)
  1. Automated refund batch:
     - Generate refunds for all 50,000 affected customers
     - Use internal admin API (bypass normal rate limits)
     - Each refund: Full duplicate amount + goodwill credit (1 month free)
     - Total refund: ~50,000 × $239.88 = ~$12M

  2. Customer notifications (proactive):
     - Email: "We made an error and have refunded you"
     - Include: Refund amount, expected timeline, apology
     - Proactive > reactive (prevents chargebacks)

  3. Dashboard update:
     - Self-service portal shows refund status
     - Updated invoice with credit line item

Phase 3: Verification (6-24 hours)
  1. Confirm all refunds initiated
  2. Monitor chargeback rate (should be near zero if proactive)
  3. Verify no remaining duplicates in system
  4. Finance reconciliation: Match refunds to original charges

Phase 4: Root Cause & Prevention (1-5 days)
  1. Root cause: Why did double-billing occur?
     - Likely: Idempotency missing on renewal path
     - Or: Race condition in billing cycle advancement
     - Or: Retry logic created duplicate invoices

  2. Fix:
     - Add idempotency to renewal path
     - Add invariant check: No overlapping billing periods
     - Add pre-charge validation: No existing paid invoice for period

  3. Tests:
     - Regression test for exact scenario
     - Property-based testing for billing invariants
     - Chaos engineering: Inject duplicate billing events

  4. Process:
     - Pre-deployment checklist for billing changes
     - Staged rollout (1% → 10% → 100%) for billing code
     - Real-time billing anomaly detection

9.3 Curveball 3: “A customer’s subscription shows as ‘active’ in your system but their payment actually failed 30 days ago. They’ve been using premium features for free. How did this happen and how do you fix it?”

Root Cause Analysis:

Likely Root Causes (ranked by probability):

1. Dunning retry succeeded on a STALE payment method
   - Old card expired, but retry used cached token
   - Payment processor returned success (test mode / sandbox leak)
   - Webhook for payment success never arrived (network issue)
   - Subscription status not updated

2. Entitlement cache never invalidated
   - Subscription moved to past_due in DB
   - Redis cache still shows "active"
   - Cache TTL too long or invalidation failed

3. Billing cycle advancement race condition
   - Two billing jobs processed same subscription
   - First job: Marked past_due
   - Second job: Advanced period (overwriting past_due)

4. Webhook delivery failure + no reconciliation
   - Payment failed webhook never delivered to subscription service
   - No daily reconciliation job to catch discrepancies

Investigation:
  1. Check subscription_events table for this customer:
     - Last status transition? When?
     - Any payment_failed events?
     - Any dunning attempts recorded?

  2. Check invoices:
     - Last paid invoice date?
     - Any unpaid invoices in the period?

  3. Check payment records:
     - Last successful payment?
     - Any failed payment attempts?

  4. Check entitlement cache:
     - Cache version vs DB version
     - Last invalidation timestamp

Fix Strategy:

Immediate Fix:
  1. Revoke access for affected customer(s)
  2. Generate catch-up invoice for the free period
  3. Offer payment plan if amount is large

Systemic Fixes:
  1. Daily Reconciliation Job:
     - Compare subscription status vs. payment status
     - Flag: Active subscription + no payment in N days
     - Auto-correct or alert for review

  2. Entitlement Cache Hard TTL:
     - Max cache age: 24 hours (force re-evaluation)
     - Even if invalidation fails, cache expires

  3. Payment Status Cross-Check:
     - On entitlement check: Verify last payment success
     - If payment older than billing cycle → flag for review

  4. Webhook Reliability:
     - Add payment processor API polling as backup
     - If webhook not received in 1 hour → poll processor
     - Reconcile webhook events with processor records

  5. Invariant Monitoring:
     - Alert: subscription.status = 'active' AND
               last_payment_date < current_period_start
     - This should NEVER be true

9.4 Curveball 4: “You need to support a new billing model: ‘Pay as you go’ with no subscription, just metered usage. But your system is built around subscription cycles. How do you adapt?”

Adaptation Strategy:

Architecture Extension:

Option A: Virtual Subscription (Chosen)
  - Create a "payg" plan with $0 base price
  - Billing cycle: Monthly (for usage aggregation)
  - Invoice generated monthly with only usage line items
  - No proration, no trial, no dunning for base amount
  - Dunning only if usage charges fail

Option B: Standalone Usage Billing
  - Separate billing path outside subscription engine
  - Usage → Invoice → Payment (no subscription involved)
  - More complex, duplicate billing logic

Why Option A:
  ✓ Reuses existing billing, invoicing, payment, dunning
  ✓ Customer has one unified billing view
  ✓ Can combine: Base subscription + usage overage
  ✓ Minimal new code

Implementation:
  1. New plan type: "metered_only"
     - amount = 0
     - interval = month
     - features = metered features only

  2. Subscription lifecycle simplified:
     - No trial, no proration
     - Cancel = stop usage tracking
     - Reactivate = resume usage tracking

  3. Billing cycle:
     - Monthly invoice with usage line items only
     - If usage = 0: No invoice generated
     - Threshold: Minimum $1 invoice (avoid micro-billing)

  4. Entitlement:
     - Active payg subscription → access to metered features
     - Past due → suspend until payment
     - Usage limits can be enforced in real-time

9.5 Curveball 5: “An enterprise customer wants custom billing: Net-60 terms, PO-based invoicing, custom tax rates, and quarterly true-ups. Your self-serve system doesn’t support this.”

Enterprise Billing Extension:

Enterprise Billing Features:

1. Collection Method: send_invoice (vs charge_automatically)
   - Invoice generated but NOT auto-charged
   - Customer pays via bank transfer / check / PO
   - Payment recorded manually or via bank feed

2. Payment Terms: Configurable per customer
   - Net-30 (default), Net-60, Net-90
   - Due date = invoice_date + terms_days
   - Late fee calculation after due date

3. PO Number Support:
   - PO number on invoice
   - PO validation (expiry, amount limit)
   - PO tracking across multiple invoices

4. Quarterly True-Ups:
   - Estimate usage monthly, bill monthly
   - Quarterly: Compare actual vs estimated
   - True-up invoice: Credit or charge the difference
   - Formula: true_up = actual_annual - estimated_billed

5. Custom Tax Rates:
   - Enterprise tax override (negotiated rate)
   - Tax exemption certificates
   - Reverse charge for B2B EU transactions

Implementation:
  ┌─────────────────────────────────────────────────────┐
  │ Enterprise Customer Flags:                          │
  │ - collection_method = 'send_invoice'                │
  │ - payment_terms = 'net_60'                          │
  │ - po_required = true                                │
  │ - true_up_frequency = 'quarterly'                   │
  │ - custom_tax_rate = 0.15 (vs standard 0.20)         │
  │ - dedicated_account_manager = true                  │
  └─────────────────────────────────────────────────────┘

  Billing Flow for Enterprise:
  1. Monthly: Generate invoice (no auto-charge)
  2. Email invoice to accounts payable
  3. Track payment status (unpaid → paid → late)
  4. Quarterly: Generate true-up invoice
  5. Annual: Generate annual summary for both parties

10. Failure Modes and Mitigations

10.1 Failure Mode Matrix

Failure Mode Probability Impact Detection Mitigation RTO RPO
Billing job misses a cycle Medium High Reconciliation alert Daily reconciliation, manual backfill < 1hr 0
Entitlement cache stale Medium Medium Cache age monitor Hard TTL, event invalidation, polling < 5min 5min
Proration calculation error Low Critical Customer complaint, audit Decimal arithmetic, audit trail < 1hr 0
Dunning emails not sent Medium Medium Email delivery rate Multi-provider fallback, retry queue < 30min 0
Usage data loss Low High Usage reconciliation Kafka persistence, dual-write < 1hr 0
Invoice generation Low Critical Duplicate invoice detection Idempotency on invoice creation < 1hr 0
Payment processor outage Low Critical Payment success rate Multi-processor failover N/A 0
Database primary failure Low Critical Connection errors Automated failover, replicas < 60s < 1s
Tax rate change missed Medium Medium Tax compliance audit Monthly check Automated tax rate updates < 1day
Webhook delivery failure Medium Low Webhook success rate Retry with backoff, manual replay < 3days 0

10.2 Detailed Mitigation Strategies

Billing Job Misses a Cycle

Detection:
  - Daily reconciliation job:
    SELECT s.id, s.current_period_end, s.status
    FROM subscriptions s
    WHERE s.status IN ('active', 'trialing')
      AND s.current_period_end < NOW() - INTERVAL '1 day'
      AND NOT EXISTS (
        SELECT 1 FROM invoices i
        WHERE i.subscription_id = s.id
          AND i.period_end = s.current_period_end
      )
  - Alert if any subscriptions found

Mitigation:
  1. Auto-retry: Billing job retries missed subscriptions
  2. Manual backfill: Admin tool to generate backdated invoices
  3. Customer notification: "We noticed a billing issue and have corrected it"
  4. Goodwill: Consider discount for inconvenience

Prevention:
  - Billing job runs twice daily (redundancy)
  - Dead man's switch: Alert if billing job doesn't run
  - Idempotent billing: Safe to retry without duplicates
  - Monitoring: Billing job success rate dashboard

Proration Calculation Error

Prevention Layers:

Layer 1: Code-Level
  - Use decimal arithmetic (BigDecimal / Python Decimal)
  - NEVER use floating point for money
  - Round only at final step
  - Store all intermediate values in audit trail

Layer 2: Validation
  - Invariant: credit + charge = net_proration
  - Invariant: proration_days <= billing_cycle_days
  - Invariant: proration_amount <= plan_amount
  - Invariant: No negative totals (unless intentional credit)

Layer 3: Testing
  - Property-based testing: For all inputs, proration is fair
  - Fuzz testing: Random plan changes, verify no negative amounts
  - Snapshot testing: Known scenarios produce expected results

Layer 4: Monitoring
  - Alert: Proration amount > 2× plan amount
  - Alert: Negative proration on upgrade (should be positive)
  - Alert: Proration days > billing cycle days

Layer 5: Reconciliation
  - Monthly: Sum all prorations, verify against expected
  - Customer-facing: Show proration breakdown on invoice
  - Support: Tool to recalculate and compare

Entitlement Cache Staleness

Multi-Layer Defense:

Layer 1: Event-Driven Invalidation (Primary)
  - Subscription change → Kafka event → Invalidate Redis
  - Guaranteed delivery (Kafka persistence)
  - Consumer group ensures at-least-once processing

Layer 2: Hard TTL (Safety Net)
  - Redis TTL: 5 minutes (configurable)
  - Even if invalidation fails, cache expires
  - Force re-evaluation from database

Layer 3: Version Checking (Verification)
  - Cache includes version number
  - On cache hit: Compare version with DB version
  - If stale: Evict and re-fetch

Layer 4: Periodic Refresh (Last Resort)
  - Every 24 hours: Full cache refresh for active customers
  - Ensures no cache is 24-hour age
  - Runs during off-peak hours

Layer 5: Product-Side Fallback
  - If entitlement service unavailable:
    - Use local cache (60-second TTL)
    - Log warning, alert on-call
    - Fail open (grant access) or fail closed (deny access)
      based on product risk assessment

10.3 Monitoring and Alerting

Critical Alerts (Page On-Call):
  - Billing job failure (no invoices generated in 24h)
  - Payment success rate < 90%
  - Entitlement service latency p99 > 200ms
  - Database primary failover
  - Kafka consumer lag > 1M messages
  - Proration calculation errors > 0

Warning Alerts (Ticket):
  - Dunning email delivery rate < 95%
  - Entitlement cache hit rate < 95%
  - Invoice generation queue depth > 10K
  - Usage ingestion lag > 1 hour
  - Tax rate update failed

Business Alerts (Dashboard):
  - MRR change > 5% week-over-week
  - Churn rate > threshold
  - Trial conversion rate drop
  - Dunning recovery rate drop
  - Average revenue per user change

10.4 Data Consistency Guarantees

Financial Consistency Model:

Strong Consistency (Required):
  - Subscription state transitions (row-level lock)
  - Invoice creation (no duplicates for same period)
  - Payment recording (exactly once)
  - Proration calculations (deterministic, auditable)

Eventual Consistency (Acceptable):
  - Entitlement cache (TTL-bounded)
  - Analytics dashboards (MRR, ARR aggregates)
  - Usage metrics (aggregated daily)
  - Search indexes (Elasticsearch)

Consistency Enforcement:
  1. Database constraints (CHECK, FOREIGN KEY, UNIQUE)
  2. Application-level state machine guards
  3. Idempotency keys for all mutating operations
  4. Daily reconciliation jobs
  5. Audit log for every state change
  6. Invariant monitoring (real-time alerts)

Key Invariants:
  - A subscription cannot have overlapping billing periods
  - An invoice total = sum of line items
  - A customer cannot have two active subscriptions for the same plan
  - Proration credit + proration charge = net proration
  - Entitlement reflects current subscription status (within TTL)

Appendix A: Glossary

Term Definition
MRR Monthly Recurring Revenue
ARR Annual Recurring Revenue (MRR × 12)
Churn Percentage of subscribers canceling in a period
Proration Partial billing for mid-cycle changes
Dunning Process of handling failed recurring payments
Entitlement Access rights granted by an active subscription
Grace Period Time after payment failure before access is revoked
Anchor Date Reference date for billing cycle calculation
True-Up Reconciliation of estimated vs actual usage
LTV Lifetime Value of a customer
Net-30/60/90 Payment terms (days until payment due)
Metered Billing Usage-based charging (per API call, GB, etc.)
Win-Back Campaign to re-engage churned customers
SCA Strong Customer Authentication
PO Purchase Order (enterprise billing)

Appendix B: Key Metrics Dashboard

Metric Target Measurement
MRR Growth > 10% MoM (Current MRR - Previous MRR) / Previous MRR
Churn Rate < 3% monthly Cancellations / Total subscribers
Trial Conversion > 40% Activated trials / Total trials
Dunning Recovery > 60% Recovered failed payments / Total failed
Billing Accuracy 100% Disputed invoices / Total invoices
Entitlement Latency p99 < 50ms Server-side timing
Invoice Generation Time < 5 seconds From trigger to PDF ready
System Availability 99.95% Uptime / Total time
NPS > 50 Customer satisfaction survey
LTV:CAC Ratio > 3:1 Lifetime value / Customer acquisition cost