214 lines
11 KiB
Markdown
214 lines
11 KiB
Markdown
file:: [Roman et al. - 2021 - Big Data Pipelines on the Computing Continuum Eco.pdf](file://C:\Users\david\Zotero\/storage/TVIRF2SK/Roman et al. - 2021 - Big Data Pipelines on the Computing Continuum Eco.pdf)
|
||
file-path:: file://C:\Users\david\Zotero\/storage/TVIRF2SK/Roman et al. - 2021 - Big Data Pipelines on the Computing Continuum Eco.pdf
|
||
|
||
- nsure low latency pre-processing and filtering close to the data sources
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 63c66a35-1f15-4f9d-9565-1220cd22a3d1
|
||
- management of various phases of Big Data processing on the Computing Continuum
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 63c66a40-f770-49b9-a952-e098276d5ddb
|
||
- Big Data pipelines in the Computing Continuum
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 63c66a4e-d008-43a1-b863-8911599fd4e0
|
||
- Cloud Computing cannot fully meet the requirements of Big Data processing applications and their data transfer overheads.
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: green
|
||
id:: 63c66a8a-a5e9-4708-8c81-92f719c34dc4
|
||
- Computing Continuum creates a fluid ecosystem by extending the Cloud Computing with the emerging Fog and Edge Computing paradigms by pushing the services that are traditionally bounded within data centres towards remote network node
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: purple
|
||
id:: 63c66b8a-10a6-4945-be4d-ad7620a933d8
|
||
hl-stamp:: 1673948045506
|
||
- discovery, modelling, simulation, and deployment of Big Data pipelines over heterogeneous resources from a diverse set of providers (i.e., using trustworthy resources).
|
||
ls-type:: annotation
|
||
hl-page:: 1
|
||
hl-color:: purple
|
||
id:: 63c66bac-57cd-4521-88a9-0e19f39704fa
|
||
- methods to support the complete Big Data pipeline processing, enabling their discovery, definition, model-based analysis and optimisation, simulation, deployment, adaptive run-time provisioning, and monitoring on top of decentralised heterogeneous infrastructures on the Computing Continuum.
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: purple
|
||
id:: 63c66be1-9c35-4c36-afcb-e8cde7371892
|
||
- ig Data pipelines traceable, trustable, manageable, analysable, and optimisable.
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: purple
|
||
id:: 63c676b9-3c80-453e-8e82-888a24d62550
|
||
- Pipeline discovery
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: purple
|
||
id:: 63c67761-93e1-441f-b01b-93c67128effa
|
||
- Pipeline definition
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: purple
|
||
id:: 63c67767-f374-4416-abd9-e4cdc643e482
|
||
- Pipeline simulation
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: purple
|
||
id:: 63c6776e-b855-485d-9178-85227241a96c
|
||
- Resource provisioning
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: purple
|
||
id:: 63c67775-9798-489b-8d2e-e3819c980642
|
||
- Pipeline deployment
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: purple
|
||
id:: 63c6777c-51fa-4067-ad61-ec776a011663
|
||
- Pipeline adaptation
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: purple
|
||
id:: 63c67781-dcc5-4b63-8759-8e5e988f0e61
|
||
- P. Castro, V. Ishakian, V. Muthusamy, and A. Slominski, “The Rise of Serverless Computing,” Communications of the ACM, vol. 62, no. 12, p.44–54, Nov. 2019
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: purple
|
||
id:: 63c677a1-f70a-45e8-a2bd-7915bf68a9e5
|
||
- The ecosystem separates the design-time from the runtime deployment of pipelines and complements modern serverless approaches [4]
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: purple
|
||
id:: 63c67815-1393-4785-ae2d-cac4996d6e6f
|
||
- design-time, pipelines are learned/discovered from the data sources, designed, customised, and simulated based on the provisioned resources, and deployed as-a-service
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: purple
|
||
id:: 63c6a293-9c12-4bb2-a9f1-8a74cbfd15c2
|
||
hl-stamp:: 1673962252115
|
||
- At run-time, new data from the data providers are served as input to the deployed pipelines, which execute and deliver data as output in the form of smart data and insights that represent actionable knowledge for data consumers.
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: purple
|
||
id:: 63c6a2c1-766b-4738-8569-cdf9613db539
|
||
hl-stamp:: 1673962254151
|
||
- utilising Big Data pipelines on the Computing Continuum for their businesses.
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 63c6a3d1-68f4-4813-9395-e393e57cd8d2
|
||
- A. Smart Mobile Marketing Campaigns
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 63c6a4ce-d2b0-4db5-a758-2aba8e5342c4
|
||
- data-driven management of marketing campaigns
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 63c6a4f2-e605-47da-bc55-7a9d28babbfe
|
||
- big historical data set containing the main performance indicator
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 63c6a50d-69e1-451f-a3b0-5fa7c4392420
|
||
- However, due to the high amount and diversity of data, and the enormous amount of potential configuration and AI libraries, the company is not capable of implementing a trusted, robust, and reliable data pipeline to generate the required insights guiding the marketing investment strategy.
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: purple
|
||
id:: 63c6a527-ff3b-4b40-801c-8987d9f17400
|
||
- The company has experienced many problems related to node configuration or memory capacity distribution when implementing novel data pipelines for new insights generation.
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: purple
|
||
id:: 63c6a57c-4b14-4a25-9bc9-3e49467e37c1
|
||
- In the proposed ecosystem, pipelines are defined and simulated by domain experts and data scientists to decide the “best” configuration in terms of extract-transform-load and analytical tools (e.g., algorithms, models, training data sets). Once the model is selected, the “best” technological architecture can be deployed to generate the expected information
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: purple
|
||
id:: 63c6a5be-1031-4248-8a92-5c52da5182f4
|
||
- B. Automatic Live Sports Content Annotation
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 63c6a69f-3f14-4554-881a-d2eff3874c4d
|
||
- Dedicated trusted connections between the cameras and the Cloud could lower the data vulnerability. If available, computational resources near the event facility could lower the latency, reduce costs, and improve the general experience.
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: purple
|
||
id:: 63c6a81d-8ab4-4044-8313-8213b8248625
|
||
- data scientists and video experts discover, define, and simulate multiple events pipelines to obtain the “best” algorithms for detecting events, and the“best” processes to enrich the videos with the obtained metadata. Pipelines can then be deployed to generate the expected information and annotate the videos. This process must be supported through intelligent resource management.
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: purple
|
||
id:: 63c6a855-718c-47b7-b5ed-92c8f647615d
|
||
- Digital Health System
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 63c6a87e-7e1f-4ade-bab7-3c22f745ec99
|
||
- Continuous progress in the remote healthcare domain has led to the invention of new wireless devices and wearables, which can obtain a comprehensive view of the peoples’ overall health based on their vitals. They collectively trace diseases of individuals and the overall community to understand the evolution of a viral infection, for example.
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 63c6a943-f0b7-4a23-8c99-bf00e6792433
|
||
- The system needs to gather data from medical devices and cameras and perform local processing, such as filtering, encryption, anonymisation and local storage for security and privacy reasons. The system also needs to orchestrate a set of Software-as-a-Service applications for patients and health personnel and to integrate and exchange data to third party services such as national Electronic Health Record systems. Overall, the system has to manage and orchestrate data pipelines in a trustworthy way across the Edge, Fog, and Cloud resources.
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: purple
|
||
id:: 63c6a95e-4e60-47b3-a529-23ca0f4ee525
|
||
- data processing pipelines on Edge devices are learned and specified based on the various IoT patient monitoring sensors. Such pipelines are then tested through simulation and deployed on personal health gateways and eHealth platforms, which make smart use of resources through adaptation.
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: purple
|
||
id:: 63c6a98a-46f9-4642-9c5d-ff1dc1ccf7a8
|
||
- Analytics of Manufacturing Assets
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 63c6a9f1-a61f-4a0b-a185-f7a1c270b872
|
||
- Predicting Deformations in Ceramics Manufacturing
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 63c6a9f7-1e9a-4ea2-8122-cc591e7290ce
|
||
- Industry 4.0 production processes require a stack of analytical applications for constant monitoring, diagnostics, and optimisation of production assets, including assembly lines and manufacturing robots
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: green
|
||
id:: 63c6aa80-c3f9-4ea2-a8f1-dea3d35d5d55
|
||
- This requires equipping production assets with sensors reporting characteristics, such as temperature, state, and operation errors.
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: purple
|
||
id:: 63c6ab66-d0e0-49f2-9f06-b5abc77b8d27
|
||
hl-stamp:: 1673964394037
|
||
- The analytics of manufacturing assets solutions offered by a producer of sensors and related chips include failure root-cause analyses, process quality monitoring, production quality monitoring, and anomaly detection. In the proposed ecosystem, data collected from different sources, such as SMD and SJI, are used to discover, define, and simulate data pipelines. Data scientists specify appropriate AI methods for the simulation and deployment of the pipelines on a pool of resources, which are actively adapted over the provisioned trusted resources.
|
||
ls-type:: annotation
|
||
hl-page:: 3
|
||
hl-color:: purple
|
||
id:: 63c6abbb-f2be-4c69-aa2c-9bae5f2281e9
|
||
- PIPELINE LIFECYCLE
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: purple
|
||
id:: 63c6ac0e-e9cf-464f-9828-40c38c524c26
|
||
- STAKEHOLDERS
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: purple
|
||
id:: 63c6ac16-1e5c-4334-ba49-d395dc85c89c
|
||
- We proposed an ecosystem for managing Big Data pipelines on the Computing Continuum including several phases, ranging from discovery to adaptation, and a set of stakeholders ranging from data providers to consumers. Our ecosystem separated the design-time and run-time aspects of pipeline execution. Finally, we provided a set of real-life use cases from different application domains to exemplify the possible use of the proposed ecosystem. We plan to realise the ecosystem by developing novel techniques and tools to support each phase in the pipeline lifecycle, and by further specifying and implementing the proposed use cases.
|
||
ls-type:: annotation
|
||
hl-page:: 4
|
||
hl-color:: purple
|
||
id:: 63c6ad3a-659f-4997-94b0-10a0bb252067
|
||
- [:span]
|
||
ls-type:: annotation
|
||
hl-page:: 2
|
||
hl-color:: green
|
||
id:: 63e602e2-f405-4b5a-98f5-15e20e365860
|
||
hl-type:: area
|
||
hl-stamp:: 1676018401277 |