system design

Designing a payment system that never charges a card twice

The question that decides this round usually sounds harmless: “The card network approved the charge, but your service crashed before it wrote anything to the database. The client retries. What happens now?” Answer that the customer gets billed twice and the rest of your design stops mattering. Payment interviews reward the candidate who treats money as something that can be lost, duplicated, or left in an unknown state, and who builds for that from the first whiteboard box.

Scope it before you draw anything

Start by asking who you are in the money flow. Are you a merchant charging your own customers at checkout, or a platform moving money on behalf of other merchants the way Stripe or Adyen does? The second is harder, because you now owe money to sellers and have to pay them out, and interviewers usually mean that version. Assume you sit in front of one or more payment service providers (PSPs) and never touch card networks or raw PCI card data directly. That single assumption cuts a lot of scope and shows you know where the real difficulty lives.

Nail down the operations: authorize and capture a card payment, refund it partially or fully, and pay out net funds to merchants. Get rough numbers on the board. A thousand payments per second at peak, average ticket around $40, gives you roughly 86 million transactions a day and a few billion a year that you keep for compliance and dispute windows. None of that is large by read-QPS standards. The pressure here is correctness, not throughput, and saying so early tells the interviewer you understand the domain.

The rules you commit to before any boxes

Money is stored as integer minor units, so $40.00 is the integer 4000, never a float. Every amount carries a currency code, because a bare number is a bug waiting to settle in the wrong denomination. Every state-changing call is idempotent. And the ledger is append-only: you never update or delete a row to fix a mistake, you append a correcting entry. State these four things out loud early and a good interviewer relaxes, because most weak answers violate all four by the thirty-minute mark.

Model the payment as a state machine

A payment object moves through a small set of states: created, processing, succeeded, failed, and later refunded. A refund is its own object that points back at the payment, not a mutation of it. The state that matters most is processing with an unknown PSP result. That is the crash-after-charge case, and your whole design is really about how a payment leaves that state safely instead of guessing.

Idempotency, and why persist-before-call is the whole game

The client generates an idempotency key, a UUID, and sends it with the payment request. On the server: open a transaction, insert a payments row with a UNIQUE constraint on that key, set status to processing, commit, and only then call the PSP. If the insert hits the unique constraint, this is a retry, so you read the existing row and replay its stored result instead of charging again. The database constraint is what actually prevents the double charge. Application-level “check if it exists, then insert” does not, because two retries race between the check and the write.

Here is the part that separates a senior answer from a staff one. You persisted intent, called the PSP, the PSP approved, and then your process died before recording success. The retry finds the row sitting in processing. You must not fire a fresh charge. Instead you forward the same idempotency key to the PSP, which exposes its own idempotency mechanism for exactly this reason, or you query the PSP for that key’s status. Idempotency has to run end to end, your key mapped to the PSP’s key, or the guarantee stops at your own front door and the money moves anyway.

The double-entry ledger

Every movement of money writes two entries, a debit and a credit, that sum to zero. A card payment credits what you owe the merchant and debits the cash you received from the PSP, with the processing fee booked as its own line. The invariant is worth stating as your correctness test: across any closed time window, the sum of debits equals the sum of credits. If that ever fails, money has been invented or destroyed, and you want a system that pages someone.

Ledger account Debit (cents) Credit (cents)
PSP clearing (funds received from the processor) 4000 0
Merchant payable (amount owed to the seller) 0 3882
Processing fee income 0 118
Total for one $40.00 payment (debits must equal credits) 4000 4000

A refund does not edit the original entries. It appends new ones that reverse the portion being returned, so the full history stays auditable. That is why finance teams and regulators want double-entry, and why reaching for an UPDATE balances SET amount = ... is an instant red flag in this interview.

Talking to the PSP without lying to yourself

Wrap the PSP behind an adapter with real timeouts, since a charge call can take anywhere from two to thirty seconds. When that call times out, the result is unknown, not failed, and treating unknown as failed is how you strand or duplicate money. Retries reuse the same idempotency key with backoff. The synchronous response is a hint. The authoritative outcome arrives later as a webhook, which you verify by signature and process idempotently, because PSPs redeliver webhooks freely. For emitting your own events reliably, a confirmation email or a downstream payout trigger, write them to a transactional outbox in the same database transaction as the ledger write, then ship them with a separate poller so an event never fires for a payment that rolled back.

Reconciliation is the safety net, not an afterthought

Every night the PSP hands you a settlement file listing what it actually processed and will pay out. A batch job walks that file against your ledger and flags anything that disagrees: a payment stuck in processing, an amount that differs by a cent, a charge the PSP has that you never recorded, or one you recorded that never settled. Those exceptions route to an automated rule or a human. This is what resolves the crash-after-charge payment from earlier, hours later, deterministically. Reconciliation usually runs on three fronts at once.

  • your ledger against the PSP’s settlement report
  • your ledger against the bank statement once funds actually land
  • expected merchant payouts against what actually went out

Bring reconciliation up before the interviewer does. It signals you have run a real payment system, where the code is never the last line of defense.

Storage and the consistency you refuse to give up

Put the ledger in a strongly consistent relational database, Postgres or MySQL, sharded by merchant or account id so one seller’s entries live on one shard. Balance reads have to see every prior write, so you do not serve them from an async replica that lags behind. This is the one place you trade scalability for correctness without apology. A merchant checking their balance mid-second and seeing stale numbers is a support ticket. A ledger that double-counts under replica lag is a lawsuit. Sharding by account also keeps a single payment’s entries and its idempotency record together, so the whole write stays in one local transaction.

How far you take each piece tracks the level you are interviewing for. At L5 the probe is idempotency keys and retry semantics, and a clean persist-before-call answer usually clears it. At L6 and staff the interviewer pushes on the ledger under concurrency, what happens when the bank approves and then reverses days later, multi-currency entries, and exactly-once delivery across the outbox and webhooks. A few phrasings you should be ready for:

  • “Walk me through a charge that times out at the PSP. How do you avoid a double charge?”
  • “A refund partially succeeds. What do the ledger entries look like?”
  • “How do you prove your books are correct after a full day of transactions?”

The candidates who clear the staff bar say the word “unknown” out loud the moment a network call times out, and they design the reconciliation path before anyone asks for it. Everything else on the board is table stakes. That instinct is the actual signal.

newsletter

What's actually being asked right now

Interview patterns & comp trends, straight to your inbox.

No spam. Unsubscribe anytime.

newsletter

What's actually being asked right now

Interview patterns & comp trends, straight to your inbox.

No spam. Unsubscribe anytime.

1972 Soviet postage stamp commemorating the Mars 2 probe

worth a read

Mars For The Rest of Us — a weekly-or-more deep dive on the technical side of Mars exploration: rocket propulsion, microbiology, mission architecture, and everything in between. Written by Maciej Ceglowski.

Read it on Substack →
Scroll to Top