Media Upload Service Low-Level Design

By /

What is a Media Upload Service?

A media upload service handles ingest, validation, processing, and storage of user-generated files: profile pictures, product photos, video clips, documents. The service must handle large files without blocking web servers, process uploads asynchronously (resize, transcode, virus scan), and serve processed media efficiently from a CDN. Instagram, YouTube, and Dropbox are built on similar patterns.

Requirements

  • Upload images (up to 20MB) and videos (up to 2GB)
  • Validate file type (MIME check, not just extension) and scan for malware
  • Process images: resize to multiple resolutions (thumbnail, medium, large)
  • Process videos: transcode to H.264/AAC, generate thumbnail at 1s mark
  • Serve processed media via CDN with cache headers
  • 50K image uploads/day, 5K video uploads/day

Upload Flow: Presigned URLs

Never stream large files through your application server — it wastes resources and adds latency. Use S3 presigned URLs to upload directly from the client to object storage:

# 1. Client requests an upload URL from your API
POST /api/media/upload-url
{ "filename": "photo.jpg", "content_type": "image/jpeg", "size_bytes": 4200000 }

# 2. API server generates presigned URL (no file goes through your server)
def get_upload_url(filename, content_type, size_bytes):
    media_id = uuid4()
    object_key = f'uploads/raw/{media_id}/{filename}'
    presigned_url = s3.generate_presigned_url(
        'put_object',
        Params={'Bucket': RAW_BUCKET, 'Key': object_key,
                'ContentType': content_type},
        ExpiresIn=900  # 15 minutes
    )
    # Record pending upload
    db.insert(Media(id=media_id, status='PENDING', raw_key=object_key, ...))
    return {'media_id': media_id, 'upload_url': presigned_url}

# 3. Client PUTs file directly to S3
PUT {presigned_url}
Content-Type: image/jpeg
[file bytes]

# 4. Client notifies API that upload is complete
POST /api/media/{media_id}/complete

Async Processing Pipeline

After upload is complete, trigger async processing via a job queue:

Media(media_id UUID, user_id UUID, original_filename VARCHAR,
      content_type VARCHAR, size_bytes BIGINT,
      status ENUM(PENDING, PROCESSING, READY, FAILED),
      raw_key VARCHAR,         -- S3 key of original file
      processed_keys JSONB,    -- {'thumbnail': 'media/thumb/...', 'medium': '...'}
      error_message TEXT,
      created_at, processed_at)

Processing worker (triggered by SQS/RabbitMQ message):

def process_media(media_id):
    media = db.get(media_id)
    raw_file = s3.get_object(RAW_BUCKET, media.raw_key)

    # 1. Validate MIME type (read magic bytes, not file extension)
    actual_type = magic.from_buffer(raw_file[:2048], mime=True)
    if actual_type != media.content_type:
        mark_failed(media_id, 'MIME mismatch'); return

    # 2. Malware scan
    result = clamav.scan(raw_file)
    if result.found:
        mark_failed(media_id, 'Malware detected'); return

    if media.content_type.startswith('image/'):
        process_image(media, raw_file)
    elif media.content_type.startswith('video/'):
        process_video(media, raw_file)

def process_image(media, data):
    img = PIL.Image.open(io.BytesIO(data))
    processed = {}
    for name, size in [('thumbnail', (150,150)), ('medium', (800,800)), ('large', (2000,2000))]:
        resized = img.copy()
        resized.thumbnail(size, PIL.Image.LANCZOS)
        key = f'media/{name}/{media.id}.jpg'
        s3.put_object(PROCESSED_BUCKET, key, resized.tobytes(), 'image/jpeg')
        processed[name] = key
    db.update(media.id, status='READY', processed_keys=processed)

CDN Serving

Processed media in S3 is served via CloudFront. S3 bucket is private; CloudFront uses an Origin Access Identity. Public URLs:

https://media.example.com/thumbnail/{media_id}.jpg
  → CloudFront → S3: media/thumbnail/{media_id}.jpg

Cache-Control: public, max-age=31536000, immutable
# Content never changes (media_id is UUID); safe to cache forever

On READY: the API returns the CDN URL. The CDN URL is stored in the processed_keys JSONB as the full URL for the client to use directly. Raw uploads bucket is never exposed publicly.

Key Design Decisions

  • Presigned S3 URLs — client uploads directly to S3, zero load on application servers
  • Magic byte MIME validation — file extension lies; magic bytes don’t
  • Async processing queue — upload completes immediately; resizing/transcoding happens in background
  • Separate raw and processed buckets — raw bucket is private (contains unvalidated content); processed is CDN-accessible
  • Immutable CDN cache headers — media_id is a UUID, content never changes; safe for long-lived caching

{“@context”:”https://schema.org”,”@type”:”FAQPage”,”mainEntity”:[{“@type”:”Question”,”name”:”How do you handle large file uploads without overloading your servers?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”Use S3 presigned URLs for direct client-to-S3 uploads. Your API server generates a presigned PUT URL (valid for 15 minutes) and returns it to the client. The client uploads the file directly to S3 — the bytes never touch your application server. This eliminates server memory pressure, bandwidth costs, and connection timeouts for large files (GBs).”}},{“@type”:”Question”,”name”:”How do you validate that an uploaded file is actually the claimed type?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”Check magic bytes, not file extensions. Read the first 2048 bytes of the uploaded file and pass them to a library like python-magic (which wraps libmagic). Magic bytes are embedded in the file itself and cannot be easily faked by renaming .exe to .jpg. For images, also use PIL/Pillow to verify the file can actually be decoded — a valid JPEG header with corrupted content would still fail this check.”}},{“@type”:”Question”,”name”:”How does async media processing work after upload?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”After the client calls your upload-complete API, publish a job message to a queue (SQS, RabbitMQ). A pool of processing workers picks up the job, downloads the raw file from S3, runs validation (MIME check, malware scan), and processes it (resize images to thumbnail/medium/large, transcode video). On completion, update the Media record status to READY and store processed S3 keys. The client polls or receives a webhook when READY.”}},{“@type”:”Question”,”name”:”How do you serve processed media efficiently?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”Store processed files in a private S3 bucket served by CloudFront CDN. Use immutable cache headers (Cache-Control: public, max-age=31536000, immutable) because the media_id is a UUID — the content never changes, so CDN caching is permanent. Keep the raw uploads bucket private (never exposed publicly, contains unvalidated content). The processed bucket is accessible only via CloudFront using Origin Access Identity.”}},{“@type”:”Question”,”name”:”How do you handle video transcoding at scale?”,”acceptedAnswer”:{“@type”:”Answer”,”text”:”Use a dedicated transcoding queue separate from image processing. Videos require significantly more CPU/time (a 1GB video may take minutes). Use AWS Elastic Transcoder, MediaConvert, or FFmpeg workers on GPU instances. Store the original video in raw S3, push a transcode job to a separate high-priority queue, and track status in the Media record. Generate a thumbnail at the 1-second mark using FFmpeg: `ffmpeg -i input.mp4 -ss 1 -vframes 1 thumbnail.jpg`.”}}]}

Media upload and processing systems are core to Meta system design interview guide.

Large-scale media upload architecture is discussed in Snap system design interview questions.

Media storage and CDN delivery design is covered in Google system design interview preparation.

Scroll to Top