5.7 KiB
#Highlights
Stream Processing with IoT Data
Date: 2021-01-05 Time: 19:32 URL: Stream Processing with IoT Data: Best Practices & Techniques Obsidian-Highlights Tags: #dataprocessing #IOT #streamprocessing
Intro
The rise of IoT devices means that we have to collect, process, and analyze orders of magnitude more data than ever before
...healthcare can now track patient health in real time and even provide on-demand care; manufacturing is able to understand the details of production lines and predict issues before they happen;
At Tesla, we unlock insights into our fleet by processing trillions of events per day from every part of the business, with just a handful of people.
This has helped Tesla become an industry leader by opening up new abilities, like being able to run complex predictions against our entire fleet to optimize efficiency or inform the next generation of manufacturing.
...is enabled by a system capable of ingesting, processing, storing, and serving these trillions of data points.
IoT data streams look a lot like common web server log events. You have events being generated, sometimes at high volumes, and they need to be processed and either made available to downstream consumers or stored in databases.
...it turns out you have all of the usual “server log” challenges and then a slew of new ones to contend with as well. Instead of web servers that are probably within your network and under your control, you now have a large number of devices with variable connectivity, leading to bursty data and a long tail of firmware versions (you can’t just stop supporting some versions) with old data formats.
...some of these devices can go “insane” and start dumping piles of data on your infrastructure, which can feel like a Denial-of-Service (DoS) attack.
Unfortunately, it gets even worse. Some of your data streams—especially if you have devices that are at all related to medical or health & safety—could be high priority and require very low latency.
These data streams can be mixed in with streams that are just high-volume “normal operations” streams used by analysts for evaluating and understanding the health of your device fleet. Now you have mixed service levels in a shared environment that you need to worry about.
Designing from first principles
A common organizational split is to have a firmware team that deals with making the device actually work and a server-side data team that handles the rest of the data pipeline, from collecting the events to processing, storing, and serving that data to the rest of the organization.
Before even deciding on technologies, the first question we should ask is: What kind of capabilities do I need from my system? There are a number of things you might want to include, but for IoT, here is a nice set of core requirements:
-
Durable storage
-
Easy horizontal scalability
-
High throughput
-
Low latency
...our challenge is to receive that data and quickly process it, making it available to downstream users so that they can produce the insights and improvements that really move a company forward.
The core piece of technology that enables us to meet all of these goals is Apache Kafka®.
Having settled on Kafka as the core of our infrastructure, we can start to sketch out some of the rest of the pieces of our processing system.
Let’s look at what we need to build around this core. We need to land the data from the devices into Apache Kafka, implement stream processing to make that raw data usable, and make the data accessible to others.
Raw data ingest
We need a small intermediary to abstract the Kafka client complexities from our devices, which often have computational and bandwidth constraints.
Great! Now that we have an API in place, we can do some simple routing and management of events.
...he API server also has one more job beyond routing: making sure that the data is durable before it confirms success to the client, so that the client knows if it should resend the data. However, for devices that have short-lived, on-device storage, or that are sending their “last gasp” data before they die, we might only have one chance to catch that data, so it is often vitally important that the data lands the first time, whenever possible
Handling large messages
One of the unique challenges with device data streams, as opposed to web server log data streams, is that it is normal for devices to become disconnected for a long while—easily months—and to then send a huge amount of data as they come back online.
I’ve seen Kafka happy with messages up to 1 MB, with very little tuning of configuration parameters needed. At the same time, I have heard of people regularly handling messages of 20 MB (e.g., Vorstella) and upwards, but that requires significant tuning.
One approach is to split Kafka clusters by workload type: one for small messages, one for larger ones.
Here are two common, alternative approaches that we can take to handle these large messages within a single cluster:
-
Chunking up the message into pieces
-
Storing a message reference in Kafka to an external store
Using an external store, our API will write messages into Kafka with a small wrapper that includes the data (or the reference, for large messages), as well as some helpful details about the message:
Message { string device_id; optional string reference; optional bytes body; }