# Integration Platform Low-Level Design: Connector Framework, Data Mapping, and Error Handling

Source: https://www.techinterview.org/post/3233470082/lld-integration-platform/
Updated: 2026-07-13 · techinterview.org

## Integration Platform: Low-Level Design

Integration platforms let non-engineers wire together SaaS products and internal services without writing custom glue code for each pair of systems. The building blocks are pluggable connectors, a declarative data mapping engine, a transformation pipeline, and robust error handling including dead-letter queuing.

## Requirements

### Functional

- Connect source and destination systems via pluggable connectors (REST, GraphQL, database, file, message queue)

- Define data mappings declaratively: field rename, type cast, computed fields, conditional logic

- Execute transformation pipelines with ordered steps: filter, enrich, split, aggregate

- Retry failed records with exponential backoff; route permanently failed records to a dead-letter store

- Provide a UI and API for managing integrations, viewing run history, and replaying dead-letter records

### Non-Functional

- Throughput: 50,000 records per second across all active integrations

- Pipeline execution latency under 500 ms p95 for single-record synchronous mode

- At-least-once delivery semantics per integration

## Data Model

- **connectors**: connector_id (UUID), type (ENUM: rest, graphql, db, s3, kafka), config (JSONB, encrypted credentials), owner_id, created_at

- **integrations**: integration_id (UUID), name, source_connector_id, destination_connector_id, trigger (ENUM: schedule, webhook, event), mapping_spec (JSONB), pipeline_spec (JSONB), enabled (BOOL)

- **pipeline_runs**: run_id (UUID), integration_id, started_at, finished_at, status (ENUM: running, success, partial, failed), records_in (INT), records_out (INT), records_failed (INT)

- **dead_letter_records**: dlr_id (UUID), run_id, integration_id, raw_payload (JSONB), error_message (TEXT), attempt_count (INT), last_attempted_at, resolved (BOOL)

## Core Algorithms

### Connector Framework

Each connector implements a standard interface with three methods: `read(cursor, batch_size)` returns a record batch and a next cursor; `write(records)` returns a write result with success and failure counts; `test_connection()` validates credentials. Connectors are packaged as isolated Docker images and loaded dynamically by the pipeline executor. Configuration schemas are defined as JSON Schema, validated at integration creation time.

### Declarative Data Mapping

The mapping spec is a JSON array of field-level rules. Each rule specifies source path (JSONPath), destination path, and an optional transform expression (powered by a sandboxed JavaScript engine, e.g. GraalVM). The mapping engine processes each record in a single pass, applying rules in order. Computed fields (e.g., concatenate first and last name) use template expressions. Conditional mappings use a when predicate evaluated against the source record.

### Pipeline Execution

The pipeline spec is a directed acyclic graph (DAG) of steps. The executor topologically sorts steps and executes independent branches in parallel using a worker thread pool. Step types include: Filter (drop records matching a predicate), Enrich (call an external lookup service and merge results), Split (fan one record into N based on an array field), Aggregate (buffer records and emit a summary), and Map (apply the field mapping spec). Each step is wrapped in a try-catch; step-level failures emit the record to the dead-letter store rather than halting the pipeline.

## Scalability and Architecture

Integrations triggered by schedule are managed by a distributed scheduler (backed by Postgres advisory locks to prevent double-execution). Webhook-triggered integrations receive events via a gateway that publishes to a Kafka topic per integration. Pipeline executors are stateless workers that consume from Kafka, execute the DAG, and write outputs via the destination connector.

- Horizontal scaling: add executor workers to increase throughput; partition Kafka topic by integration_id for ordered processing

- Connector credential secrets are stored encrypted (AES-256-GCM) with key management via AWS KMS or HashiCorp Vault

- Dead-letter records are stored in Postgres with full payload; a replay API re-enqueues them to the head of the pipeline

- Observability: per-integration metrics (records/sec, error rate, latency histogram) exported to Prometheus; run history queryable via API for 90 days

## API Design

### Integration Lifecycle

- `POST /v1/integrations` — create integration with source, destination, mapping, and pipeline spec

- `PUT /v1/integrations/{integration_id}/enable` / `/disable` — toggle integration

- `GET /v1/integrations/{integration_id}/runs?status=STRING&limit=INT` — paginated run history

### Dead-Letter Management

- `GET /v1/integrations/{integration_id}/dead-letter?resolved=false` — list unresolved DLQ records

- `POST /v1/dead-letter/{dlr_id}/replay` — re-enqueue a single record for reprocessing

- `POST /v1/integrations/{integration_id}/dead-letter/replay-all` — bulk replay, returns a batch job ID

### Connector Testing

`POST /v1/connectors/{connector_id}/test` — validates credentials and connectivity, returns status and a sample record from the source.

## Interview Tips

Discuss the schema evolution problem: when a source system changes its payload structure, existing mapping specs may break silently. Mitigate with schema versioning on the source connector and a compatibility check step that validates sample records against the expected schema before enabling an integration. Also address the split-step fan-out risk: a single input record expanding to 10,000 output records can overwhelm the destination connector; implement a configurable per-step rate limiter and back-pressure mechanism that pauses the source read when the destination write queue exceeds a depth threshold.

See also: [Atlassian Interview Guide](https://www.techinterview.org/post/3233459919/atlassian-interview/)

See also: [Scale AI Interview Guide 2026: Data Infrastructure, RLHF Pipelines, and ML Engineering](https://www.techinterview.org/post/3233460275/scale-ai-interview-guide-2026-data-infrastructure-rlhf-pipelines-and-ml-engineering/)

See also: [Stripe Interview Guide 2026: Process, Bug Bash Round, and Payment Systems](https://www.techinterview.org/post/3233460268/stripe-interview-guide-2026-process-bug-bash-round-and-payment-systems/)