Low Level Design: File Upload Service – Tech Interview Dot Org

What Is a File Upload Service?

A file upload service provides a reliable, scalable mechanism for clients to transfer files to a backend system. It abstracts storage details, enforces access control, handles metadata, and integrates with downstream processing pipelines such as virus scanning, image resizing, or indexing. At the low level, the service must manage byte streams, track upload state, and coordinate between the client, application servers, and object storage (e.g., S3, GCS, or Azure Blob).

Data Model

The core entities are Upload and FileMetadata. A minimal SQL-style schema looks like this:

CREATE TABLE uploads (
    upload_id     UUID PRIMARY KEY,
    user_id       BIGINT NOT NULL,
    filename      VARCHAR(512) NOT NULL,
    mime_type     VARCHAR(128),
    size_bytes    BIGINT,
    storage_key   VARCHAR(1024),
    status        ENUM('pending', 'in_progress', 'complete', 'failed'),
    created_at    TIMESTAMP DEFAULT NOW(),
    updated_at    TIMESTAMP DEFAULT NOW()
);

CREATE TABLE file_metadata (
    file_id       UUID PRIMARY KEY REFERENCES uploads(upload_id),
    checksum_md5  CHAR(32),
    checksum_sha256 CHAR(64),
    tags          JSONB,
    visibility    ENUM('private', 'public'),
    owner_id      BIGINT NOT NULL
);

The storage_key column stores the path within the object store. Status transitions are: pending → in_progress → complete (or failed on error).

Core Workflow: Presigned URL Pattern

The most scalable pattern avoids routing large file bytes through application servers entirely. Instead, the server issues a short-lived presigned URL that allows the client to write directly to object storage:

Client requests upload: POST /api/uploads with filename, size, and MIME type.
Server creates an upload record (status = pending), then calls the storage SDK to generate a presigned PUT URL with a TTL of 15–60 minutes.
Server returns the upload_id and the presigned URL to the client.
Client PUTs the file directly to the presigned URL. No application server traffic for the file bytes.
Storage fires a completion event (S3 event notification, GCS Pub/Sub) to a backend consumer, which updates the record to complete and triggers post-processing.

Alternatively, the server can poll or the client can call a confirm endpoint after upload completes.

Failure Handling

Failures can occur at multiple stages. Key mitigations:

Network interruption during PUT: The client must retry from scratch (single PUT is not resumable). For large files, chunked upload is preferred (see companion post).
Presigned URL expiry: The client should request a fresh URL if the TTL expires before upload starts. Implement a /api/uploads/{id}/refresh-url endpoint.
Storage event loss: Use an idempotent reconciliation job that scans in_progress rows older than N minutes and checks their storage existence directly via a HEAD request to the storage API.
Duplicate uploads: Hash the file client-side (SHA-256) and include it in the initial request; deduplicate at the metadata layer before issuing a presigned URL.

Scalability Considerations

Offloading byte transfer to object storage is the single biggest scalability win—application servers become lightweight coordinators. Additional considerations:

Rate limiting: Enforce per-user quotas at the upload initiation endpoint to prevent abuse.
CDN integration: Serve completed files through a CDN by mapping storage keys to a distribution domain. Never serve directly from the bucket in production.
Async post-processing: Publish a message (Kafka, SQS) on upload completion. Downstream workers handle scanning, thumbnailing, and indexing without blocking the upload path.
Database sharding: Partition the uploads table by user_id range or hash once write volume grows beyond a single primary.
Storage lifecycle policies: Use bucket lifecycle rules to move infrequently accessed files to cold storage (S3 Glacier, GCS Nearline) automatically.

Summary

A well-designed file upload service keeps application servers out of the data path by using presigned URLs for direct client-to-storage transfers. The database tracks upload lifecycle state, and event-driven or reconciliation-based mechanisms ensure consistency even under failure. This architecture scales horizontally with minimal backend load and integrates cleanly with async processing pipelines.

{
“@context”: “https://schema.org”,
“@type”: “FAQPage”,
“mainEntity”: [
{
“@type”: “Question”,
“name”: “What are the core components of a file upload service design?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “A file upload service typically consists of an API gateway to receive upload requests, a metadata store (e.g., a relational database) to track file records, an object storage backend (such as Amazon S3 or Google Cloud Storage) for storing file bytes, and a CDN for fast retrieval. Additional components include authentication/authorization, virus scanning, and event queues for async processing.”
}
},
{
“@type”: “Question”,
“name”: “How do you handle large file uploads reliably in a system design interview?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Large file uploads are best handled by splitting the file into smaller chunks on the client side, uploading each chunk independently, and reassembling them on the server or in object storage. This approach supports retrying individual failed chunks, enables parallel uploads for better throughput, and avoids timeouts associated with single long-running HTTP requests.”
}
},
{
“@type”: “Question”,
“name”: “How would you scale a file upload service to handle millions of concurrent users?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “Scaling a file upload service involves using horizontally scalable stateless upload servers behind a load balancer, pre-signed URLs to offload direct client-to-storage uploads (bypassing your servers entirely), distributed metadata storage with sharding or a managed database, and a message queue (e.g., Kafka or SQS) to decouple post-upload processing jobs such as thumbnail generation or virus scanning.”
}
},
{
“@type”: “Question”,
“name”: “What consistency and durability guarantees should a file upload service provide?”,
“acceptedAnswer”: {
“@type”: “Answer”,
“text”: “A file upload service should guarantee at-least-once delivery of file data to durable storage, with idempotent upload operations so retries do not create duplicates. Files should be replicated across multiple availability zones or regions for durability (e.g., 11 nines durability like S3). Metadata and file state should be kept consistent using transactions or optimistic locking to avoid scenarios where metadata says a file exists but the bytes are missing.”
}
}
]
}