Payment systems are among the most critical and complex components in software engineering. A bug in a payment system can charge a customer twice, lose money, or violate financial regulations. This guide covers the architecture of a production payment system — from charge initiation to settlement — with focus on idempotency, consistency, and the patterns that prevent financial errors. Essential knowledge for system design interviews at fintech companies and any role involving payment processing.
Payment Flow Architecture
A typical payment flow: (1) Checkout — the user selects items and confirms the order. The frontend collects payment method details and tokenizes them (Stripe.js, Stripe Elements). The raw card number never touches your server — it goes directly to Stripe, which returns a token. (2) Payment intent creation — your backend creates a PaymentIntent with Stripe: amount, currency, payment method token, and an idempotency key. Stripe returns a payment_intent_id with status “requires_confirmation.” (3) Confirmation — your backend confirms the PaymentIntent. Stripe attempts the charge with the card network (Visa, Mastercard). (4) Authorization — the card network checks with the issuing bank: does the cardholder have sufficient funds? Is the card valid? Is this transaction suspicious? The bank approves or declines. (5) Capture — if authorized, Stripe captures the funds (moves them from the cardholder to the merchant). Some flows separate authorization and capture (hotel bookings authorize on check-in, capture on check-out). (6) Settlement — Stripe transfers the captured funds (minus fees) to your bank account, typically in 2 business days. Your backend receives webhook events at each stage: payment_intent.created, payment_intent.succeeded, payment_intent.payment_failed.
Idempotency and Double-Charge Prevention
The most critical requirement: never charge a customer twice for the same order. Network failures cause retries — if the initial charge request succeeded but the response was lost, the client retries and the customer is charged twice. Idempotency key pattern: the client generates a unique key per payment attempt (typically the order_id or a UUID) and includes it in every request header. Server behavior: on first request, process the payment and store the result keyed by the idempotency key. On duplicate request (same key), return the stored result without re-processing. Stripe requires an Idempotency-Key header on all POST requests and handles deduplication server-side. Your backend must also implement idempotency: before calling Stripe, check if this order_id has already been charged. Use a database unique constraint on (order_id, charge_status) to prevent concurrent duplicate charges. The order state machine: PENDING -> CHARGING -> CHARGED -> FULFILLED. Use optimistic locking (version column) or SELECT FOR UPDATE to prevent two concurrent requests from both transitioning from PENDING to CHARGING.
The Payment Ledger
A ledger is an append-only record of every financial transaction. Every payment, refund, fee, and payout is recorded as a ledger entry. Double-entry bookkeeping: every transaction creates two entries that sum to zero. A $100 charge creates: debit customer_receivable +$100, credit revenue +$100. A $100 refund creates: debit revenue -$100, credit customer_receivable -$100. The ledger is the source of truth for all financial state. Account balances are derived by summing ledger entries — never stored as a mutable field (which would be vulnerable to race conditions and data loss). Ledger entries are immutable — mistakes are corrected by adding a new reversal entry, not by modifying the original. This provides a complete audit trail. Schema: ledger_entry_id, transaction_id, account_id, amount (positive for debit, negative for credit), currency, created_at, description, metadata. Index on account_id + created_at for balance queries. The ledger must be strongly consistent — use a single database (PostgreSQL) with SERIALIZABLE isolation for critical financial operations. Do not shard the ledger unless absolutely necessary (sharding complicates cross-account consistency).
Reconciliation
Reconciliation verifies that your internal records match the payment processor records. Discrepancies indicate bugs, fraud, or processing errors. Daily reconciliation process: (1) Export — download the daily settlement report from Stripe (all charges, refunds, disputes, and fees for the day). (2) Match — for each Stripe transaction, find the corresponding record in your ledger by payment_intent_id. (3) Compare — verify amounts match, statuses match, and no transactions are missing from either side. (4) Flag discrepancies — transactions in Stripe but not in your ledger (your system lost track of a charge), transactions in your ledger but not in Stripe (a charge you think succeeded actually failed), and amount mismatches (partial capture, currency conversion). (5) Investigate and resolve — each discrepancy is investigated. Common causes: webhook delivery failure (your system did not receive the payment_succeeded event), race conditions (charge succeeded but your database update failed), and timezone boundary issues (transactions near midnight may appear on different days). Automate reconciliation to run daily. Alert on any discrepancy. In fintech, unresolved discrepancies older than 24 hours should escalate to engineering leadership.
PCI Compliance
PCI DSS (Payment Card Industry Data Security Standard) governs how card data is handled. The simplest path to compliance: never touch card data. Use Stripe.js or Stripe Elements — the card number goes directly from the user browser to Stripe servers. Your backend receives only a token representing the card. This puts you in PCI SAQ-A (the simplest compliance level — a self-assessment questionnaire). If you handle card data directly (type it into your backend, store it, or log it), you are subject to PCI SAQ-D (the most stringent level — requires annual on-site audit, network segmentation, encryption at rest and in transit, and extensive documentation). Rules: (1) Never log card numbers, CVVs, or full card data. Not in application logs, not in error tracking, not in database audit logs. (2) Never store CVV/CVC — even encrypted. Card networks prohibit storing the security code after authorization. (3) Use tokenization — replace the card number with a token from Stripe/Braintree. The token is safe to store and log. (4) Use HTTPS for all payment pages. (5) Regularly scan your infrastructure for vulnerabilities.
Webhook Reliability
Stripe sends webhook events to notify your backend of payment state changes. Your webhook handler must be reliable — a missed event can leave an order unpaid or a refund unprocessed. Reliability patterns: (1) Idempotent handler — process each event exactly once. Store the event_id in a processed_events table. Before processing, check if the event_id exists. If it does, return 200 (already processed). (2) Verify the signature — Stripe signs every webhook with a secret. Verify the signature before processing to prevent forged events. (3) Return 200 quickly — acknowledge receipt within 5 seconds. Do not perform slow operations (database writes, API calls) synchronously in the handler. Enqueue the event for async processing and return 200 immediately. If you return a non-2xx or timeout, Stripe retries with exponential backoff for up to 3 days. (4) Reconciliation as backup — do not rely solely on webhooks. Run a periodic job (every 5 minutes) that queries Stripe for recent PaymentIntents and reconciles with your database. This catches any events missed due to webhook delivery failure. (5) Event ordering — Stripe does not guarantee event delivery order. Your handler must tolerate receiving payment_intent.succeeded before payment_intent.created. Design handlers to be order-independent by checking the current state before applying transitions.