How do you integrate webhooks from multiple shipping carriers?

Each carrier exposes its own webhook format, so the integration layer deploys a per-carrier adapter that authenticates the inbound request (HMAC signature or IP allowlist), parses the payload, and publishes a normalized event to an internal Kafka topic. A registry maps carrier codes to adapter classes, enabling new carriers to be onboarded without changing downstream consumers. Retries and deduplication are handled at the ingestion gateway using idempotency keys derived from carrier tracking number plus event timestamp.

How does an event normalization pipeline work for shipment tracking?

The normalization pipeline reads raw carrier events from a staging topic, applies a carrier-specific transformer to produce a canonical TrackingEvent schema (tracking_number, carrier, status_code, location, occurred_at, raw_payload), and writes the result to a normalized topic. Status codes are mapped to a unified taxonomy (e.g., USPS '010' and FedEx 'PU' both become PICKED_UP). Schema validation with Avro rejects malformed events to a dead-letter queue for alerting and manual review.

What is an efficient storage schema for shipment checkpoints?

Checkpoints are stored in a wide-column store (e.g., Cassandra) with a partition key of tracking_number and a clustering key of (occurred_at DESC, event_id). This layout collocates all events for a shipment and returns the latest checkpoint in O(1) reads. A separate materialized view keyed by (carrier, status_code, date) supports carrier-level SLA reporting. Older checkpoints are tiered to object storage via TTL-based compaction to control storage growth.

How is ML-based ETA prediction integrated into a shipping tracker?

A feature store assembles real-time features per shipment: current checkpoint, historical transit times for that carrier x route x service-tier, weather signals, and carrier capacity indices. A gradient-boosted model (XGBoost or LightGBM) trained on historical deliveries predicts remaining transit hours. The prediction service is called asynchronously on each new checkpoint event and stores the updated ETA alongside the checkpoint. Confidence intervals are surfaced to users, and model freshness is monitored via a drift detector that triggers retraining when p-value thresholds are breached.

Shipping Tracker Low-Level Design: Carrier Integration, Event Normalization, and ETA Prediction

⏱ 5 min read

Requirements and Constraints

A shipping tracker aggregates parcel status from multiple carriers (FedEx, UPS, USPS, DHL, regional last-mile), normalizes heterogeneous event formats into a unified model, stores checkpoint history, and predicts estimated delivery time. Functional requirements: ingest carrier webhooks and polling responses, normalize to canonical event types, store full checkpoint history per shipment, expose tracking APIs to customers and internal systems, and provide ETA prediction. Non-functional: ingest 50,000 events per minute, deliver status updates to end users within 5 seconds of carrier event, maintain 5 years of tracking history, and achieve ETA accuracy within 2 hours for 80% of shipments.

Core Data Model

shipments(shipment_id PK, tracking_number, carrier_id FK, service_level, origin_zip, dest_zip, weight_oz, label_created_at, estimated_delivery_date, actual_delivery_at, status ENUM('label_created','in_transit','out_for_delivery','delivered','exception'))
carriers(carrier_id PK, name, webhook_secret, polling_enabled, poll_interval_seconds, api_base_url)
checkpoints(checkpoint_id PK, shipment_id FK, carrier_event_code, canonical_event_type, location_city, location_state, location_zip, occurred_at, received_at, raw_payload JSONB)
eta_predictions(prediction_id PK, shipment_id, predicted_delivery_at, confidence_score, model_version, features_snapshot JSONB, created_at)
carrier_polling_cursors(carrier_id, last_polled_at, next_poll_at)

Carrier Integration Architecture

Each carrier integration is a plugin implementing a common interface with two methods: verify_webhook(headers, body) bool and parse_events(payload) []RawEvent. Webhook endpoints are carrier-specific paths (/webhooks/fedex, /webhooks/ups) that route to the appropriate plugin after HMAC signature verification using the stored webhook_secret. For carriers that do not support push webhooks, a polling adapter reads carrier_polling_cursors, calls the carrier tracking API for shipments due for a poll, and emits synthetic RawEvent objects that flow through the same normalization pipeline.

Raw events are published to a partitioned Kafka topic (carrier-events-raw) keyed by tracking_number. This decouples ingestion from processing, provides a durable replay buffer, and allows horizontal scaling of normalization workers.

Event Normalization

Each carrier uses different event codes and schemas. A normalization worker consumes from carrier-events-raw and applies carrier-specific mapping tables stored in the database (event_code_mappings: carrier_id, raw_code, canonical_event_type, description). Canonical types are a fixed enum: LABEL_CREATED, PICKED_UP, IN_TRANSIT, OUT_FOR_DELIVERY, DELIVERED, DELIVERY_ATTEMPTED, EXCEPTION, RETURNED_TO_SENDER.

After mapping, the worker writes a checkpoint row and updates the shipment status. Status transitions are validated against an allowed-transitions matrix to prevent out-of-order events from regressing status (e.g., an IN_TRANSIT event cannot overwrite a DELIVERED status). Duplicate detection uses a composite unique index on (shipment_id, carrier_event_code, occurred_at).

ETA Prediction

ETA prediction is a regression model trained on historical shipment data. Features include: carrier, service level, origin-dest zip pair distance bucket, current checkpoint type, days since label created, day of week, and holiday proximity flag. The model outputs a predicted delivery timestamp and a confidence score.

Prediction runs asynchronously after each new checkpoint is written. The prediction service fetches recent checkpoints for the shipment, assembles the feature vector, calls the model serving endpoint (ONNX Runtime or a lightweight HTTP model server), and upserts the latest eta_predictions row. The features_snapshot column stores the exact input vector for auditability and offline model evaluation.

For interview purposes: discuss how you would retrain the model (nightly batch job on delivered shipments with ground truth), how to handle cold start for new carriers (fall back to carrier-published SLA windows), and how to A/B test model versions using the model_version column.

Scalability Considerations

Hot tracking numbers: High-volume shippers generate bursts of events for the same tracking numbers. Kafka partitioning by tracking_number ensures ordered processing per shipment while parallelizing across partitions.
Checkpoint storage: checkpoints is an append-only table; partition by received_at month. Archive old partitions to object storage with a metadata index for rare historical lookups.
Fanout notifications: After status update, publish to a shipment-status-changed topic consumed by a notification service (SMS/email), the fulfillment service, and customer-facing WebSocket push servers.
Polling efficiency: Batch carrier polling calls; use bulk tracking APIs where available (UPS, FedEx support up to 30 tracking numbers per request) to reduce API quota consumption.

API Design

POST /shipments — register a new shipment with tracking number and carrier; triggers initial status poll
GET /shipments/{tracking_number} — returns current status, latest ETA, and full checkpoint history
GET /shipments/{tracking_number}/checkpoints — paginated checkpoint list with canonical and raw event data
POST /webhooks/{carrier} — carrier-facing webhook receiver; returns 200 immediately, processes async
GET /shipments/{tracking_number}/eta — latest ETA prediction with confidence score and model version
GET /carriers/{id}/exception-rate — operational dashboard: exception rate over rolling 24h window by carrier