Payment systems are among the most critical and complex components in software engineering. A bug in a payment system can charge a customer twice, lose money, or violate financial regulations. This guide covers the architecture of a production payment system — from charge initiation to settlement — with focus on idempotency, consistency, and the patterns that prevent financial errors. Essential knowledge for system design interviews at fintech companies and any role involving payment processing.
Payment Flow Architecture
A typical payment flow: (1) Checkout — the user selects items and confirms the order. The frontend collects payment method details and tokenizes them (Stripe.js, Stripe Elements). The raw card number never touches your server — it goes directly to Stripe, which returns a token. (2) Payment intent creation — your backend creates a PaymentIntent with Stripe: amount, currency, payment method token, and an idempotency key. Stripe returns a payment_intent_id with status “requires_confirmation.” (3) Confirmation — your backend confirms the PaymentIntent. Stripe attempts the charge with the card network (Visa, Mastercard). (4) Authorization — the card network checks with the issuing bank: does the cardholder have sufficient funds? Is the card valid? Is this transaction suspicious? The bank approves or declines. (5) Capture — if authorized, Stripe captures the funds (moves them from the cardholder to the merchant). Some flows separate authorization and capture (hotel bookings authorize on check-in, capture on check-out). (6) Settlement — Stripe transfers the captured funds (minus fees) to your bank account, typically in 2 business days. Your backend receives webhook events at each stage: payment_intent.created, payment_intent.succeeded, payment_intent.payment_failed.
Idempotency and Double-Charge Prevention
The most critical requirement: never charge a customer twice for the same order. Network failures cause retries — if the initial charge request succeeded but the response was lost, the client retries and the customer is charged twice. Idempotency key pattern: the client generates a unique key per payment attempt (typically the order_id or a UUID) and includes it in every request header. Server behavior: on first request, process the payment and store the result keyed by the idempotency key. On duplicate request (same key), return the stored result without re-processing. Stripe requires an Idempotency-Key header on all POST requests and handles deduplication server-side. Your backend must also implement idempotency: before calling Stripe, check if this order_id has already been charged. Use a database unique constraint on (order_id, charge_status) to prevent concurrent duplicate charges. The order state machine: PENDING -> CHARGING -> CHARGED -> FULFILLED. Use optimistic locking (version column) or SELECT FOR UPDATE to prevent two concurrent requests from both transitioning from PENDING to CHARGING.
The Payment Ledger
A ledger is an append-only record of every financial transaction. Every payment, refund, fee, and payout is recorded as a ledger entry. Double-entry bookkeeping: every transaction creates two entries that sum to zero. A $100 charge creates: debit customer_receivable +$100, credit revenue +$100. A $100 refund creates: debit revenue -$100, credit customer_receivable -$100. The ledger is the source of truth for all financial state. Account balances are derived by summing ledger entries — never stored as a mutable field (which would be vulnerable to race conditions and data loss). Ledger entries are immutable — mistakes are corrected by adding a new reversal entry, not by modifying the original. This provides a complete audit trail. Schema: ledger_entry_id, transaction_id, account_id, amount (positive for debit, negative for credit), currency, created_at, description, metadata. Index on account_id + created_at for balance queries. The ledger must be strongly consistent — use a single database (PostgreSQL) with SERIALIZABLE isolation for critical financial operations. Do not shard the ledger unless absolutely necessary (sharding complicates cross-account consistency).
Reconciliation
Reconciliation verifies that your internal records match the payment processor records. Discrepancies indicate bugs, fraud, or processing errors. Daily reconciliation process: (1) Export — download the daily settlement report from Stripe (all charges, refunds, disputes, and fees for the day). (2) Match — for each Stripe transaction, find the corresponding record in your ledger by payment_intent_id. (3) Compare — verify amounts match, statuses match, and no transactions are missing from either side. (4) Flag discrepancies — transactions in Stripe but not in your ledger (your system lost track of a charge), transactions in your ledger but not in Stripe (a charge you think succeeded actually failed), and amount mismatches (partial capture, currency conversion). (5) Investigate and resolve — each discrepancy is investigated. Common causes: webhook delivery failure (your system did not receive the payment_succeeded event), race conditions (charge succeeded but your database update failed), and timezone boundary issues (transactions near midnight may appear on different days). Automate reconciliation to run daily. Alert on any discrepancy. In fintech, unresolved discrepancies older than 24 hours should escalate to engineering leadership.
PCI Compliance
PCI DSS (Payment Card Industry Data Security Standard) governs how card data is handled. The simplest path to compliance: never touch card data. Use Stripe.js or Stripe Elements — the card number goes directly from the user browser to Stripe servers. Your backend receives only a token representing the card. This puts you in PCI SAQ-A (the simplest compliance level — a self-assessment questionnaire). If you handle card data directly (type it into your backend, store it, or log it), you are subject to PCI SAQ-D (the most stringent level — requires annual on-site audit, network segmentation, encryption at rest and in transit, and extensive documentation). Rules: (1) Never log card numbers, CVVs, or full card data. Not in application logs, not in error tracking, not in database audit logs. (2) Never store CVV/CVC — even encrypted. Card networks prohibit storing the security code after authorization. (3) Use tokenization — replace the card number with a token from Stripe/Braintree. The token is safe to store and log. (4) Use HTTPS for all payment pages. (5) Regularly scan your infrastructure for vulnerabilities.
Webhook Reliability
Stripe sends webhook events to notify your backend of payment state changes. Your webhook handler must be reliable — a missed event can leave an order unpaid or a refund unprocessed. Reliability patterns: (1) Idempotent handler — process each event exactly once. Store the event_id in a processed_events table. Before processing, check if the event_id exists. If it does, return 200 (already processed). (2) Verify the signature — Stripe signs every webhook with a secret. Verify the signature before processing to prevent forged events. (3) Return 200 quickly — acknowledge receipt within 5 seconds. Do not perform slow operations (database writes, API calls) synchronously in the handler. Enqueue the event for async processing and return 200 immediately. If you return a non-2xx or timeout, Stripe retries with exponential backoff for up to 3 days. (4) Reconciliation as backup — do not rely solely on webhooks. Run a periodic job (every 5 minutes) that queries Stripe for recent PaymentIntents and reconciles with your database. This catches any events missed due to webhook delivery failure. (5) Event ordering — Stripe does not guarantee event delivery order. Your handler must tolerate receiving payment_intent.succeeded before payment_intent.created. Design handlers to be order-independent by checking the current state before applying transitions.
{“@context”:”https://schema.org”,”@type”:”FAQPage”,”mainEntity”:[{“@type”:”Question”,”name”:”How do you prevent double-charging customers in a payment system?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”Double charges occur when a network failure causes a retry of a successful payment. Prevention: (1) Idempotency keys — the client generates a unique key per payment (order_id or UUID) and includes it in every request. The server stores the result keyed by this value. On duplicate request, return the stored result without re-processing. Stripe requires an Idempotency-Key header. (2) Database constraints — add a unique constraint on (order_id, charge_status) in your orders table. Two concurrent charge attempts for the same order will conflict at the database level. (3) State machine — the order transitions through states: PENDING -> CHARGING -> CHARGED. Use optimistic locking (version column) or SELECT FOR UPDATE to prevent two concurrent requests from both seeing PENDING and proceeding. (4) Reconciliation — run daily reconciliation matching your records against Stripe settlement reports to catch any discrepancies.”}},{“@type”:”Question”,”name”:”What is a payment ledger and why use double-entry bookkeeping?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”A ledger is an append-only record of every financial transaction. Double-entry bookkeeping creates two entries per transaction that sum to zero. A $100 charge creates: debit customer_receivable +$100, credit revenue +$100. A refund reverses it. Benefits: (1) Account balances are derived from summing entries, not stored as mutable fields. No race conditions on balance updates. (2) Every entry is immutable — mistakes are corrected with reversal entries, preserving a complete audit trail. (3) The books always balance (total debits = total credits). An imbalance indicates a bug. (4) Regulatory compliance — financial auditors require transaction-level detail. The ledger should use SERIALIZABLE isolation in PostgreSQL for critical operations. Do not shard unless absolutely necessary — cross-shard consistency is extremely difficult for financial data.”}},{“@type”:”Question”,”name”:”How do you ensure payment webhook reliability?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”Stripe webhooks notify your backend of payment events (succeeded, failed, refunded). Reliability patterns: (1) Idempotent handler — store processed event_ids. Before processing, check if the event was already handled. Return 200 for duplicates. (2) Verify signatures — Stripe signs every webhook. Verify before processing to prevent forged events. (3) Return 200 quickly — acknowledge within 5 seconds. Enqueue the event for async processing rather than doing heavy work synchronously. If you timeout, Stripe retries. (4) Reconciliation backup — do not rely solely on webhooks. Run a periodic job (every 5 minutes) querying Stripe for recent PaymentIntents and reconciling with your database. This catches missed events. (5) Handle out-of-order delivery — Stripe does not guarantee order. Your handler might receive payment_intent.succeeded before payment_intent.created. Check current state before applying transitions.”}},{“@type”:”Question”,”name”:”What is PCI compliance and how do you achieve it for a payment system?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”PCI DSS governs how card data is handled. The simplest compliance path: never touch card data. Use Stripe.js or Stripe Elements — the card number goes directly from the browser to Stripe. Your backend only receives a token. This qualifies you for PCI SAQ-A, the simplest compliance level (self-assessment questionnaire). If you handle raw card data, you need SAQ-D (annual on-site audit, network segmentation, encryption everywhere, extensive documentation). Rules: never log card numbers or CVVs (not in app logs, error tracking, or database audit logs). Never store CVV after authorization (card networks prohibit it). Use tokenization — replace card numbers with tokens from Stripe. Use HTTPS for all payment pages. Scan infrastructure for vulnerabilities regularly. The cost difference between SAQ-A and SAQ-D is enormous — always use tokenization through a certified payment processor.”}}]}