2.1 KiB
2.1 KiB
#Highlights
IoT Cloud and Machine Learning The Building Blocks
Date: 2021-01-05 Time: 22:03 URL: (https://www.opensourceforu.com/2020/10/iot-cloud-and-machine-learning-the-building-blocks/) Obsidian-Highlights Tags: #machinelearning #IOT #dataprocessing
Main building blocks of an ML infrastructure
Given below are the tools and frameworks needed for an Apache Spark based ML infrastructure.
- Spark 2: Comes up with all the necessary tools from the Spark ecosystem — Hadoop, Mlib, etc. You can get more details from https://spark.apache.org/docs/latest/.
- Hadoop: You can either install Spark2 with Hadoop or Hadoop as standalone.
- Python3/Scala/Java: This depends on what language you prefer to write ML programs
- PostgreSQL/MongoDB: Install this if you have to store data into traditional databases other than Hadoop HDFS for future use and reference.
- MLib/Tensorflow/Keras/Scikit learn: Choose from these ML libraries based on your needs.
- Data analytics tools: Choose these based on your needs.
Common ML use cases in IoT
Given below are a few common use cases based on the data received from devices (strictly based on my experience and may differ in your business case).
- Analysis of data patterns for a specific period: As an example, if data comes from a temperature sensor, then the pattern of temperature data for a location where the device is installed for a day can be analysed for that specific period.
- Data missing/changes in duration/changes in pattern, etc: It is important to understand the missing data or changes in frequency because immediate action is required, or it can lead to potential errors in our analytics/forecasts.
- Inactivities or other ambiguities in data flow: These errors in data processing must be avoided.
- Difference between the forecasted and real data: This may lead to a correction in data models and trainings.
- User and location behaviour from device to device: Data for each device may differ if user and location behaviour contribute some points to the data.
- Frequency of maintenance and root causes for that: This may be specific to location, usage, transaction volume, etc.