29 lines
2.1 KiB
Markdown
29 lines
2.1 KiB
Markdown
#Highlights
|
|
|
|
---
|
|
|
|
# **IoT Cloud and Machine Learning The Building Blocks**
|
|
Date: 2021-01-05
|
|
Time: 22:03
|
|
URL: (https://www.opensourceforu.com/2020/10/iot-cloud-and-machine-learning-the-building-blocks/) [[Obsidian-Highlights]]
|
|
Tags: #machinelearning #IOT #dataprocessing
|
|
|
|
---
|
|
# Main building blocks of an ML infrastructure
|
|
|
|
Given below are the tools and frameworks needed for an Apache Spark based ML infrastructure.
|
|
- _Spark 2:_ Comes up with all the necessary tools from the Spark ecosystem — Hadoop, Mlib, etc. You can get more details from https://spark.apache.org/docs/latest/.
|
|
- _Hadoop:_ You can either install Spark2 with Hadoop or Hadoop as standalone.
|
|
- _Python3/Scala/Java:_ This depends on what language you prefer to write ML programs
|
|
- PostgreSQL/MongoDB: Install this if you have to store data into traditional databases other than Hadoop HDFS for future use and reference.
|
|
- _MLib/Tensorflow/Keras/Scikit learn_: Choose from these ML libraries based on your needs.
|
|
- _Data analytics tools:_ Choose these based on your needs.
|
|
# Common ML use cases in IoT
|
|
|
|
Given below are a few common use cases based on the data received from devices (strictly based on my experience and may differ in your business case).
|
|
- _Analysis of data patterns for a specific period_: As an example, if data comes from a temperature sensor, then the pattern of temperature data for a location where the device is installed for a day can be analysed for that specific period.
|
|
- _Data missing/changes in duration/changes in pattern, etc:_ It is important to understand the missing data or changes in frequency because immediate action is required, or it can lead to potential errors in our analytics/forecasts.
|
|
- _Inactivities or other ambiguities in data flow:_ These errors in data processing must be avoided.
|
|
- _Difference between the forecasted and real data:_ This may lead to a correction in data models and trainings.
|
|
- _User and location behaviour from device to device:_ Data for each device may differ if user and location behaviour contribute some points to the data.
|
|
- _Frequency of maintenance and root causes for that:_ This may be specific to location, usage, transaction volume, etc. |