IoT platforms ingest telemetry from millions of devices with high write throughput, store time-series data efficiently, and enable real-time alerting and analytics.
Device Registry
Device: device_id, serial_number, device_type, firmware_version, owner_id, location, status ENUM (active/inactive/decommissioned), last_seen_at, metadata_json. DeviceCredential: device_id, client_id, certificate_pem, created_at, expires_at. Devices authenticate with X.509 certificates or pre-shared keys. The registry is the source of truth for device provisioning.
MQTT Broker
Devices publish telemetry to a topic hierarchy: devices/{device_id}/telemetry. The broker (e.g., EMQX or AWS IoT Core) handles millions of persistent connections. QoS levels: QoS 0 (fire and forget), QoS 1 (at least once with ACK), QoS 2 (exactly once). Retained messages store the last-known state. The broker bridges to a Kafka topic for downstream processing.
Ingestion Pipeline
Kafka consumers read from the telemetry topic. Each message contains device_id, timestamp, and payload (JSON or Protobuf). The consumer validates the schema, enriches with device metadata from a registry cache, and normalizes units. Valid records are written to a time-series database (TimescaleDB, InfluxDB, or Apache Cassandra with time-partitioned tables). Bad messages are routed to a dead letter queue.
Time-Series Storage
Schema: (device_id, metric_name, timestamp, value) with a hypertable partitioned by time (weekly chunks). Indexes on (device_id, metric_name, timestamp DESC) for fast range queries. A retention policy compresses or drops data older than a configured threshold (e.g., raw data 90 days, hourly aggregates 2 years). A downsampling job computes hourly and daily aggregates for long-term storage.
Stream Processing and Alerting
Flink or Spark Streaming consumes telemetry. Sliding window aggregations compute metrics such as average temperature over the last 5 minutes. A rule engine evaluates alert conditions — metric crosses threshold, device goes offline (no heartbeat for more than 5 minutes). Alert events are published to a notification service. Alert rules are stored in a database and loaded into the stream processor as broadcast state.
OTA Firmware Updates
An operator uploads new firmware to S3 and creates an UpdateCampaign (campaign_id, device_type, firmware_version, target_device_filter, rollout_pct, created_at). The campaign scheduler assigns devices to update. Each device receives an update command via its MQTT command topic, downloads the firmware from a pre-signed S3 URL, and reports status (downloading/applying/success/failure). Failed devices are automatically excluded from future rollout cohorts.
Frequently Asked Questions: IoT Data Ingestion Platform
What are the differences between MQTT QoS levels 0, 1, and 2?
MQTT defines three Quality of Service levels. QoS 0 (at most once) fires the message and forgets it—no acknowledgment, no retry; best for high-frequency sensor telemetry where occasional loss is acceptable. QoS 1 (at least once) requires a PUBACK from the receiver; the sender retains and retransmits until acknowledged, which means duplicates are possible. QoS 2 (exactly once) uses a four-way handshake (PUBLISH → PUBREC → PUBREL → PUBCOMP) to guarantee delivery without duplicates; it has the highest overhead and suits critical commands or billing events.
How does hypertable partitioning work in a time-series database?
A hypertable (as in TimescaleDB) is a logical table that is automatically partitioned into chunks along the time dimension—for example, one chunk per day or per week. Each chunk is a regular PostgreSQL table stored separately on disk, enabling chunk-level operations: pruning queries to only scan relevant time ranges, dropping old data in O(1) by deleting a chunk, and tiering cold chunks to cheaper storage. Space partitioning (by device ID hash) can be added as a second dimension to parallelize ingest and prevent hot spots on a single node.
How do you detect when an IoT device has gone offline using heartbeat TTL?
Each device publishes a heartbeat message on a fixed interval (e.g., every 30 seconds). The platform stores the last-seen timestamp for each device ID in a fast store such as Redis. A background monitor compares the current time against last-seen; if the gap exceeds a configurable TTL (e.g., 90 seconds—three missed heartbeats), the device is marked offline and an alert is emitted. Redis TTL keys can encode this natively: the device sets or refreshes a key with a TTL equal to the offline threshold, and key expiration triggers a Keyspace Notification that the monitor consumes.
How should an OTA firmware update be rolled out to a large IoT fleet with automated gates?
A staged OTA rollout starts by deploying the firmware to a canary cohort (e.g., 1% of devices) and monitoring error rates, crash reports, and connectivity metrics for a bake period (e.g., 24 hours). Automated gates compare these metrics against baseline thresholds; if they pass, the rollout advances to the next stage (e.g., 5%, 20%, 50%, 100%). If a gate fails, the pipeline halts and optionally initiates an automatic rollback by pushing the previous firmware version. Each device reports its installed version and installation status, feeding the deployment dashboard in near real time.
How do Flink sliding window aggregations enable real-time alerting on IoT streams?
Apache Flink’s sliding windows compute continuous aggregates (sum, average, max) over a moving time range. For example, a window of size 5 minutes sliding every 1 minute emits an aggregate every minute covering the last 5 minutes of data. For IoT alerting, a Flink job consumes sensor readings from Kafka, keys the stream by device ID, and applies a sliding window to compute rolling averages of temperature, vibration, or pressure. When an aggregate breaches a threshold, the job emits an alert event downstream to a notification service. Flink’s event-time processing with watermarks handles out-of-order messages from intermittently connected devices.
See also: Scale AI Interview Guide 2026: Data Infrastructure, RLHF Pipelines, and ML Engineering
See also: Databricks Interview Guide 2026: Spark Internals, Delta Lake, and Lakehouse Architecture