logseq/pages/hls__Roman et al. - 2021 - Big Data Pipelines on the Computing Continuum Eco.md

file:: [Roman et al. - 2021 - Big Data Pipelines on the Computing Continuum Eco.pdf](file://C:\Users\david\Zotero/storage/TVIRF2SK/Roman et al. - 2021 - Big Data Pipelines on the Computing Continuum Eco.pdf) file-path:: file://C:\Users\david\Zotero/storage/TVIRF2SK/Roman et al. - 2021 - Big Data Pipelines on the Computing Continuum Eco.pdf

• nsure low latency pre-processing and filtering close to the data sources ls-type:: annotation hl-page:: 1 hl-color:: green id:: 63c66a35-1f15-4f9d-9565-1220cd22a3d1
• management of various phases of Big Data processing on the Computing Continuum ls-type:: annotation hl-page:: 1 hl-color:: green id:: 63c66a40-f770-49b9-a952-e098276d5ddb
• Big Data pipelines in the Computing Continuum ls-type:: annotation hl-page:: 1 hl-color:: green id:: 63c66a4e-d008-43a1-b863-8911599fd4e0
• Cloud Computing cannot fully meet the requirements of Big Data processing applications and their data transfer overheads. ls-type:: annotation hl-page:: 1 hl-color:: green id:: 63c66a8a-a5e9-4708-8c81-92f719c34dc4
• Computing Continuum creates a fluid ecosystem by extending the Cloud Computing with the emerging Fog and Edge Computing paradigms by pushing the services that are traditionally bounded within data centres towards remote network node ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 63c66b8a-10a6-4945-be4d-ad7620a933d8 hl-stamp:: 1673948045506
• discovery, modelling, simulation, and deployment of Big Data pipelines over heterogeneous resources from a diverse set of providers (i.e., using trustworthy resources). ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 63c66bac-57cd-4521-88a9-0e19f39704fa
• methods to support the complete Big Data pipeline processing, enabling their discovery, definition, model-based analysis and optimisation, simulation, deployment, adaptive run-time provisioning, and monitoring on top of decentralised heterogeneous infrastructures on the Computing Continuum. ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c66be1-9c35-4c36-afcb-e8cde7371892
• ig Data pipelines traceable, trustable, manageable, analysable, and optimisable. ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c676b9-3c80-453e-8e82-888a24d62550
• Pipeline discovery ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c67761-93e1-441f-b01b-93c67128effa
• Pipeline definition ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c67767-f374-4416-abd9-e4cdc643e482
• Pipeline simulation ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c6776e-b855-485d-9178-85227241a96c
• Resource provisioning ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c67775-9798-489b-8d2e-e3819c980642
• Pipeline deployment ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c6777c-51fa-4067-ad61-ec776a011663
• Pipeline adaptation ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c67781-dcc5-4b63-8759-8e5e988f0e61
• P. Castro, V. Ishakian, V. Muthusamy, and A. Slominski, “The Rise of Serverless Computing,” Communications of the ACM, vol. 62, no. 12, p.44–54, Nov. 2019 ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 63c677a1-f70a-45e8-a2bd-7915bf68a9e5
• The ecosystem separates the design-time from the runtime deployment of pipelines and complements modern serverless approaches [4] ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c67815-1393-4785-ae2d-cac4996d6e6f
• design-time, pipelines are learned/discovered from the data sources, designed, customised, and simulated based on the provisioned resources, and deployed as-a-service ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c6a293-9c12-4bb2-a9f1-8a74cbfd15c2 hl-stamp:: 1673962252115
• At run-time, new data from the data providers are served as input to the deployed pipelines, which execute and deliver data as output in the form of smart data and insights that represent actionable knowledge for data consumers. ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c6a2c1-766b-4738-8569-cdf9613db539 hl-stamp:: 1673962254151
• utilising Big Data pipelines on the Computing Continuum for their businesses. ls-type:: annotation hl-page:: 2 hl-color:: green id:: 63c6a3d1-68f4-4813-9395-e393e57cd8d2
• A. Smart Mobile Marketing Campaigns ls-type:: annotation hl-page:: 2 hl-color:: green id:: 63c6a4ce-d2b0-4db5-a758-2aba8e5342c4
• data-driven management of marketing campaigns ls-type:: annotation hl-page:: 2 hl-color:: green id:: 63c6a4f2-e605-47da-bc55-7a9d28babbfe
• big historical data set containing the main performance indicator ls-type:: annotation hl-page:: 2 hl-color:: green id:: 63c6a50d-69e1-451f-a3b0-5fa7c4392420
• However, due to the high amount and diversity of data, and the enormous amount of potential configuration and AI libraries, the company is not capable of implementing a trusted, robust, and reliable data pipeline to generate the required insights guiding the marketing investment strategy. ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c6a527-ff3b-4b40-801c-8987d9f17400
• The company has experienced many problems related to node configuration or memory capacity distribution when implementing novel data pipelines for new insights generation. ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c6a57c-4b14-4a25-9bc9-3e49467e37c1
• In the proposed ecosystem, pipelines are defined and simulated by domain experts and data scientists to decide the “best” configuration in terms of extract-transform-load and analytical tools (e.g., algorithms, models, training data sets). Once the model is selected, the “best” technological architecture can be deployed to generate the expected information ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c6a5be-1031-4248-8a92-5c52da5182f4
• B. Automatic Live Sports Content Annotation ls-type:: annotation hl-page:: 2 hl-color:: green id:: 63c6a69f-3f14-4554-881a-d2eff3874c4d
• Dedicated trusted connections between the cameras and the Cloud could lower the data vulnerability. If available, computational resources near the event facility could lower the latency, reduce costs, and improve the general experience. ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 63c6a81d-8ab4-4044-8313-8213b8248625
• data scientists and video experts discover, define, and simulate multiple events pipelines to obtain the “best” algorithms for detecting events, and the“best” processes to enrich the videos with the obtained metadata. Pipelines can then be deployed to generate the expected information and annotate the videos. This process must be supported through intelligent resource management. ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 63c6a855-718c-47b7-b5ed-92c8f647615d
• Digital Health System ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63c6a87e-7e1f-4ade-bab7-3c22f745ec99
• Continuous progress in the remote healthcare domain has led to the invention of new wireless devices and wearables, which can obtain a comprehensive view of the peoples’ overall health based on their vitals. They collectively trace diseases of individuals and the overall community to understand the evolution of a viral infection, for example. ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63c6a943-f0b7-4a23-8c99-bf00e6792433
• The system needs to gather data from medical devices and cameras and perform local processing, such as filtering, encryption, anonymisation and local storage for security and privacy reasons. The system also needs to orchestrate a set of Software-as-a-Service applications for patients and health personnel and to integrate and exchange data to third party services such as national Electronic Health Record systems. Overall, the system has to manage and orchestrate data pipelines in a trustworthy way across the Edge, Fog, and Cloud resources. ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 63c6a95e-4e60-47b3-a529-23ca0f4ee525
• data processing pipelines on Edge devices are learned and specified based on the various IoT patient monitoring sensors. Such pipelines are then tested through simulation and deployed on personal health gateways and eHealth platforms, which make smart use of resources through adaptation. ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 63c6a98a-46f9-4642-9c5d-ff1dc1ccf7a8
• Analytics of Manufacturing Assets ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63c6a9f1-a61f-4a0b-a185-f7a1c270b872
• Predicting Deformations in Ceramics Manufacturing ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63c6a9f7-1e9a-4ea2-8122-cc591e7290ce
• Industry 4.0 production processes require a stack of analytical applications for constant monitoring, diagnostics, and optimisation of production assets, including assembly lines and manufacturing robots ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63c6aa80-c3f9-4ea2-a8f1-dea3d35d5d55
• This requires equipping production assets with sensors reporting characteristics, such as temperature, state, and operation errors. ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 63c6ab66-d0e0-49f2-9f06-b5abc77b8d27 hl-stamp:: 1673964394037
• The analytics of manufacturing assets solutions offered by a producer of sensors and related chips include failure root-cause analyses, process quality monitoring, production quality monitoring, and anomaly detection. In the proposed ecosystem, data collected from different sources, such as SMD and SJI, are used to discover, define, and simulate data pipelines. Data scientists specify appropriate AI methods for the simulation and deployment of the pipelines on a pool of resources, which are actively adapted over the provisioned trusted resources. ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 63c6abbb-f2be-4c69-aa2c-9bae5f2281e9
• PIPELINE LIFECYCLE ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 63c6ac0e-e9cf-464f-9828-40c38c524c26
• STAKEHOLDERS ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 63c6ac16-1e5c-4334-ba49-d395dc85c89c
• We proposed an ecosystem for managing Big Data pipelines on the Computing Continuum including several phases, ranging from discovery to adaptation, and a set of stakeholders ranging from data providers to consumers. Our ecosystem separated the design-time and run-time aspects of pipeline execution. Finally, we provided a set of real-life use cases from different application domains to exemplify the possible use of the proposed ecosystem. We plan to realise the ecosystem by developing novel techniques and tools to support each phase in the pipeline lifecycle, and by further specifying and implementing the proposed use cases. ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 63c6ad3a-659f-4997-94b0-10a0bb252067
• [:span] ls-type:: annotation hl-page:: 2 hl-color:: green id:: 63e602e2-f405-4b5a-98f5-15e20e365860 hl-type:: area hl-stamp:: 1676018401277