Design a Mobile Receipt-Scan and Expense App

Updated July 26, 2026 · techinterview.org

Receipt-scanning expense apps (Expensify, Zoho Expense, modern fintech wallets) have become routine for business travelers and freelancers. The interview tests whether you understand the OCR pipeline, the categorization layer, and integration with accounting systems.

Functional requirements

• Capture a receipt with the camera. Clarify whether the app also accepts emailed PDFs, photo-library imports, and multi-receipt pages, since each adds a different intake path.
• Extract amount, vendor, date, line items via OCR. Interviewers want you to separate raw text recognition from structured field parsing, because they fail in different ways.
• Categorize for expense reports. Decide up front whether categories are company-defined policy buckets or generic ones, since that changes the ML and lookup design.
• Sync to accounting system (QuickBooks, NetSuite). Treat this as an async, retryable job rather than an inline call, because third-party APIs are slow and rate-limited.
• Submit for reimbursement. Scope whether the app owns the approval workflow or just hands expenses off to an existing corporate system.
• Mileage tracking via GPS. Flag early that this is a battery and privacy problem, not just a distance calculation.

Architecture

Three pipelines: capture, extract, sync.

Camera capture

Quality matters for OCR accuracy:

• Auto-detect receipt edges and crop. This removes the table, hand, and background clutter that otherwise confuses OCR into reading noise as text.
• De-skew and de-warp. Receipts photographed at an angle or curled up produce slanted, curved text lines; correcting geometry before OCR meaningfully raises accuracy.
• Enhance contrast for faded receipts. Thermal-paper receipts fade to near-gray, so adaptive thresholding or binarization recovers characters the camera barely captured.
• Capture as high-res JPEG (1080p or 4K). Enough resolution to read small line items, but compress before upload so slow mobile connections don’t stall the user.

Platform APIs: VisionKit on iOS (DataScannerViewController), ML Kit on Android. Both have receipt-detection models.

OCR

Two strategies:

• On-device: faster, private, works offline. Modern phones have on-device text recognition. Good for an instant preview the moment the shutter fires, and it keeps sensitive receipt images off your servers.
• Cloud: better accuracy for tricky receipts. Send the image to a service like Veryfi, AWS Textract, or Google Document AI. These are tuned for tables and faded thermal paper that generic on-device models miss.

Most production apps use cloud for accuracy with on-device for instant preview.

Field extraction

OCR returns text blocks. Field extraction parses semantic fields:

• Total amount (look for “Total”, “Amount Due”, currency symbols). The trap is a receipt with subtotal, tax, tip, and total all present; pick the largest bottom-most value and validate that subtotal plus tax equals it.
• Vendor name (typically the largest text at the top). Normalize it against a merchant database, because the printed name (“SQ *BLUE BOTTLE”) rarely matches the clean brand name a user expects.
• Date (multiple formats; locale-dependent). Watch for ambiguous formats like 03/04/2026, and reject dates in the future or implausibly far in the past as OCR errors.
• Line items (table-like structure). This is the hardest field; you need column alignment from bounding-box coordinates, not just the text stream, and interviewers often probe here.
• Tax (separate line, often labeled). Capturing tax separately matters for VAT reclaim and accounting, so parse it as its own field rather than folding it into the total.

Modern systems use LLMs (cloud-hosted) for robust extraction across receipt formats.

Categorization

Map vendor → category:

• Lookup table for common vendors (Starbucks → “Meals”). Fast and deterministic for the long tail of frequent merchants, and it costs nothing to run per receipt.
• ML categorization for unknown vendors based on keywords and patterns. Features like merchant category code, line-item text, and amount let you guess a bucket when the vendor isn’t in the table.
• User correction trains the model over time. Log every override as a labeled example and weight per-user corrections, since one traveler’s “Meals” is another company’s “Client Entertainment”.

Mileage tracking

For driving expense claims:

• Background location tracking when user starts a trip. Request the right OS permission tier and be explicit about it, because always-on location is what triggers app-store scrutiny and user distrust.
• Auto-detect start/stop based on movement patterns. Use the low-power motion/activity APIs to notice driving begins, so you only spin up GPS when the user is actually moving.
• Calculate distance from GPS path. Smooth the raw points and snap to roads, since summing noisy GPS fixes overstates distance and inflates reimbursement.
• Apply IRS or company mileage rate. Keep rates as configurable, dated values so a rate change doesn’t silently rewrite past trips.

Battery: the most expensive part of these apps. Be conservative.

Accounting integration

Push expenses to:

• QuickBooks, Xero, NetSuite, FreshBooks (most common). Each has its own object model and quirks, so build an adapter layer rather than coupling your core to one vendor’s API.
• Sync via OAuth + REST API. Store and refresh per-user tokens, and handle the token-expiry and re-consent flows that break silently months after setup.
• Map your category to their chart of accounts. This mapping is per-organization and set by the finance team, so make it configurable rather than hardcoded.
• Handle approval workflows. Model expenses as a state machine (draft, submitted, approved, exported) so a failed sync can retry without double-posting.

Submitting for reimbursement

For corporate reimbursement:

• Group expenses into a report. Let users bundle by trip or date range, and validate against policy (missing receipts, over-limit amounts) before submission.
• Submit for manager approval. Route based on org hierarchy and approval limits, with escalation when the direct manager is out.
• Approval triggers reimbursement (ACH, payroll, expense card). Make the payout step idempotent so a retry after a timeout never pays the same report twice.

Frequently Asked Questions

How accurate is mobile OCR for receipts?

For typical English-language receipts: 95%+ on amount and date. Lower on line items, especially handwritten or thermal-faded receipts.

How is fraud prevented?

Image fingerprinting (same receipt submitted twice triggers flag). Velocity rules. Manager approval as a gate. None of these are perfect.

Why is mileage tracking battery-intensive?

Continuous GPS during the drive. Modern apps use motion-detection to auto-start, then high-accuracy GPS until the user stops moving.