logseq/pages/hls__Nikolov et al. - 2021 - Conceptualization and scalable execution of big da.md

file:: [Nikolov et al. - 2021 - Conceptualization and scalable execution of big da.pdf](file://C:\Users\david\Zotero/storage/QYUBVQAD/Nikolov et al. - 2021 - Conceptualization and scalable execution of big da.pdf) file-path:: file://C:\Users\david\Zotero/storage/QYUBVQAD/Nikolov et al. - 2021 - Conceptualization and scalable execution of big da.pdf

• Cloud infrastructures’ elasticity for scalability ls-type:: annotation hl-page:: 1 hl-color:: green id:: 63c90619-1b5b-4b7e-a916-b31e78362af8
• highly scalable workflow execution and abstract workflow definition. ls-type:: annotation hl-page:: 1 hl-color:: green id:: 63c90629-cf05-43e6-825b-9eea51752930
• specification and scalable execution of Big Data workflows. ls-type:: annotation hl-page:: 1 hl-color:: green id:: 63c90635-dddc-4b0a-a752-15137dfcf884
• Big Data4 workflows are composed of multiple orchestrated steps, such as workflow activities that perform various data analytical5 tasks ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 63c90654-38ab-4c17-b1c6-f4e345bf2a8f
• ynamic, process heterogeneous data,6 and are executed in parallel instead of a sequential set of scientific operators ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 63c9065f-e8a7-47be-b8d4-21e63552fb9b
• econdly, given the fact that IoT, Edge and Cloud technologies converge towards10 a computing continuum, workflow steps need to be mapped dynamically to heterogeneous computing and storage11 resources to ensure scalability ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 63c9067a-740c-4655-87e3-5b3bd74dac41
• a scalable, general-purpose solution for Big Data workflows that a12 broad audience can use is an open research issue ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 63c906c7-b466-43a5-b10f-60888e50c43b
• scaling up individual steps introduces race conditions between step instances18 that attempt to process the same piece of data simultaneously. ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c906ee-8b20-4a86-977d-ae6227effaa5
• Another major challenge is achieving usability by19 multiple stakeholders as most Big Data processing solutions are focused on ad-hoc processing models that only trained20 professionals can use ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c9073e-b5ea-489d-a7c7-073fc7f42ade
• organizations typically operate on specific software stacks, and getting experts in Big21 Data technology can introduce costs that are not affordable or practical. ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c90746-161a-4dff-a651-0e37cd1bdd6d
• data workflow steps pertain to specific domain-dependent knowledge, which is possessed by the23 domain experts rather than the data scientists who set up the data workflows ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c913c2-8cb1-4b36-8a39-9d871c440104
• conceptualization of Big Data workflows using a domain-specific language (DSL) ls-type:: annotation hl-page:: 2 hl-color:: blue id:: 63c913e1-53f6-4423-a3a1-2c73281aeed6
• calable execution of Big Data workflows using software container technologies and message-oriented middle-28 ware solutions. ls-type:: annotation hl-page:: 2 hl-color:: blue id:: 63c913e6-9a90-4d8b-a6a6-6094b67725ca
• . D. Dessalk, et al., Scalable Execution of Big Data Workflows Using Software Containers, in: Proc. of the MEDES 2020, 2020, p. 76–83. ls-type:: annotation hl-page:: 22 hl-color:: green id:: 63c91415-08b4-4568-8fa6-64e037c3faa4
• Dynamic Deployment of Scientific Workflows in the Cloud Using Container Virtualization ls-type:: annotation hl-page:: 22 hl-color:: green id:: 63c9142a-05c3-473f-a5c3-d5fb9b604d13
• rchestrating Big Data Analysis Workflows in the Cloud: Research Challenges, Survey, and Future Directions ls-type:: annotation hl-page:: 22 hl-color:: green id:: 63c91432-9f9d-4299-82d0-52e53ff68ca8
• Docker container-based big data processing system in multiple clouds for everyone ls-type:: annotation hl-page:: 22 hl-color:: green id:: 63c91458-593e-455e-bc6c-3b36640bcbea
• WorkflowDSL: scalable workflow execution with provenance for data analysis application ls-type:: annotation hl-page:: 22 hl-color:: green id:: 63c9147a-a5aa-4140-9185-e9136a20b65a
• ntegrating Containers into Workflows: A Case Study Using Makeflow, Work Queue, and Docker, ls-type:: annotation hl-page:: 22 hl-color:: green id:: 63c91483-d338-464c-84e4-2e1bf5f05189
• ClowdFlows: Online workflows for distributed big data mining, F ls-type:: annotation hl-page:: 22 hl-color:: green id:: 63c91493-0548-4c09-b19c-c150b881c5b0
• software container technologies, message-oriented middleware (MOM), and a DSL to enable highly scalable workflow31 execution and abstract workflow definition ls-type:: annotation hl-page:: 2 hl-color:: blue id:: 63c914ad-7e32-44d7-b7ca-d41009884e75
• scaling up on the level of individual workflow32 steps on top of heterogeneous infrastructures while avoiding race conditions through a system of inter- and intra-33 step coordination. ls-type:: annotation hl-page:: 2 hl-color:: green id:: 63c914eb-f0d1-4df9-be58-6be410a276a7 hl-stamp:: 1674122478299
• analysis of the requirements for enabling43 Big Data workflows on the Computing Continuum ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 63c9152d-cd81-4fb1-868c-accf76a80a14
• rious processing models can be applied for parallelizing data process-56 ing known as workflow data pattern ls-type:: annotation hl-page:: 2 hl-color:: green id:: 63c9154c-7a80-445f-ab17-bdfc98260e84
• Pipe and Filter (P&F) i ls-type:: annotation hl-page:: 2 hl-color:: green id:: 63c91554-5412-41b1-8cf3-1e14d651be36
• MOM provides an infrastructure for loosely-66 coupled and asynchronous inter-process communication using messaging capabilities ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63c9159c-f327-4667-8831-6cf5af6d3050
• middleware can act as a medium for communication, whereby step instances coordinate passing70 intermediate results through sending/receiving messages in MOM queues ls-type:: annotation hl-page:: 3 hl-color:: purple id:: 63c915c0-0c8f-407a-8cc1-02e4f2660fed
• Docker is one of74 the most well-known platforms for organizing solutions based on container technologies. ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63c917c3-4cd4-4418-bcb7-fa96e0829460
• deploying distributed scalable applications ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63c917cf-4fd9-43d7-a3bf-343fd002140b
• Orchestration tools use a configuration file to define container images, network, and related deployment schemes of79 an application ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63c917e1-c190-4b3f-85b2-08b31fe43945
• Big Data workflow solutions that share similar design principles while fulfilling the94 needs of various groups of users and use cas ls-type:: annotation hl-page:: 3 hl-color:: green id:: 63c91803-7913-4947-9c5a-6b6bc31adfb8
• [:span] ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 63c91848-d3d0-47cb-93e2-61070479dc67 hl-type:: area hl-stamp:: 1674123334656
• usability for non-technical experts ls-type:: annotation hl-page:: 4 hl-color:: purple id:: 63c918ac-ac89-46d3-b0eb-b8dbf2353276
• However, Argo Workflows117 does not have a middleware solution to handle inter-step communication, which may result in step instances running118 into race conditions when scaled horizontally. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 63c918ee-63f9-4679-a65c-74102020ec38
• Nextflow ls-type:: annotation hl-page:: 4 hl-color:: green id:: 63c9190a-b4e2-4158-aa13-c37939483635
• Nextflow does not provide a clear separation of concerns between workflow definition and125 implementation and relies on a specific software stack (through the DSL) for workflow step implementation ls-type:: annotation hl-page:: 4 hl-color:: green id:: 63c91915-137b-4d57-b65f-a48df4c7b0e9
• achyderm is based on Docker and Kubernetes, and provides advanced features such as pluralisation132 and incremental processing. Users need technical knowledge to define workflows. ls-type:: annotation hl-page:: 4 hl-color:: green id:: 63c9194a-1856-4489-8b8c-aa793fe36ee6
• challenges of scalability, resource provisioning,148 scheduling, orchestration, and data management of data workflows ls-type:: annotation hl-page:: 5 hl-color:: purple id:: 63c91996-30c1-4a25-8ff0-c3d72a66f54a
• Commonly used data processing frameworks (such as Spark, Flink, Beam) are designed with ad-hoc process-177 ing models that technical experts on specific technology stacks can only use. ls-type:: annotation hl-page:: 5 hl-color:: green id:: 63c919e0-a840-483e-828e-949b004b0361
• lthough most approaches and tools use185 some form of DSL or UI, the abstraction level is not sufficient to allow for separation of concerns between definition186 and implementation, which is necessary to effectively involve domain experts in workflow definition ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 63c91a5f-d65f-4093-bd68-857ebf016a2d
• ynamic mapping workflow steps to heterogeneous com-188 puting and storage resources to ensure scalability. ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 63c91a89-f40a-4b4a-8ecb-404b4bf28319
• scalability needs to be organized and orchestrated over heterogeneous computing192 resources. ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 63c91a96-6ade-41d0-9ce3-d5d1ca3c8906
• no approach is able to unlock the full197 potential for achieving step-level scalability of workflows, which relies on workflow and step encapsulation. ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 63c91ab8-3355-4a4b-bb07-0833b5c5f506
• A communication solution must decouple the communication between the steps to be200 scaled up while maintaining race-condition-free data access ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 63c91aca-f8fb-4cad-843e-f3349b9281d6
• a workflow definition mechanism with a clear separation between design- and run-time aspects and not limited206 to a specific technology stack, application domain or ad-hoc processing models; ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 63c91ae7-3ed2-479a-8f55-94648bce938d
• workflow run-time support that considers workflows as separate units, rather than as a single unit, for individ-208 ual workflow steps; and ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 63c91af0-388e-45d1-af53-5575d5f78215
• a workflow enactment approach with event driven execution and support for race-condition-free parallel execu-210 tion. ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 63c91af6-5d06-4797-be43-bc584eef01fe
• Workflow Modelling Manager ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 63c91b10-0fd9-4f83-b66a-49288efbe420
• Deployment Service Runtime ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 63c91b13-aca5-487e-a353-1db9fd550fc9
• Data218 Storage/Sharing Ecosystem ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 63c91b18-e977-4bf8-b17b-55c18c73ecd9
• The Workflow Modelling Manager ls-type:: annotation hl-page:: 6 hl-color:: purple id:: 63c91b84-b61b-478f-8fbf-c6296331b759
• com-222 ponent handles storage configurations, data preparation, and step-level data processing and transformation operations ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 63c91c2f-ddf2-4e61-a95f-1fdc4b4b9967
• deployable data workflow ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 63c91c36-5290-49f4-8ed7-49578b2fd266
• Deployment Service Runtime ls-type:: annotation hl-page:: 6 hl-color:: blue id:: 63c91c3c-ebce-41ab-9609-3141e8701524
• we assume that the DSL should be used by non-technical experts271 and, therefore, should avoid complex constructions as much as possible. ls-type:: annotation hl-page:: 8 hl-color:: purple id:: 63c91cc6-aa26-4a0f-9f09-ee485f231f39
• Components of workflow step template ls-type:: annotation hl-page:: 9 hl-color:: purple id:: 63c91d0d-18cd-4ec5-a534-232b2d185c50
• he template is modified by adding the installation script for any necessary software libraries in the Docker373 image building configuration and injecting relevant code scripts (step processing code) in a specific place in the374 template logic ls-type:: annotation hl-page:: 11 hl-color:: purple id:: 63c91d3e-1e40-4941-81ce-3576c7dc3c18
• The DSL description of the prototype workflow ls-type:: annotation hl-page:: 12 hl-color:: purple id:: 63c91d67-84dc-437f-8a18-2362ba07520c
• eight ls-type:: annotation hl-page:: 14 hl-color:: purple id:: 63c91d90-732e-4f71-a669-169408b63a4f
• share a distributed file system, and run the Docker engine440 connected to Rancher. ls-type:: annotation hl-page:: 14 hl-color:: purple id:: 63c91d9b-43ae-47f6-b3e1-97df59157b03
• quality monitoring for resistance spot welding ls-type:: annotation hl-page:: 17 hl-color:: purple id:: 63c91f67-46d0-44e4-94a5-6e3c8cbc6352
• n addition to the sensor data from the welding machines, the use case involves reference data535 that provide information about target parameters of the equipment as well as the settings of the individual machines536 and welding programs. ls-type:: annotation hl-page:: 17 hl-color:: purple id:: 63c91fb6-4463-4747-af54-259b2f796f8f
• workflow for offline batch preparation of data for537 machine learning ls-type:: annotation hl-page:: 17 hl-color:: purple id:: 63c91ffa-f2da-414b-907d-8d79c268fd8d
• online scenario for real-time monitoring using trained ML model ls-type:: annotation hl-page:: 17 hl-color:: purple id:: 63c92001-e70a-4c74-a2bf-5665989ecff7
• distributed) file system ls-type:: annotation hl-page:: 17 hl-color:: purple id:: 63c9201b-ebfa-4072-82db-fdd5a3f4536b
• [:span] ls-type:: annotation hl-page:: 8 hl-color:: purple id:: 63ca5656-a823-48d9-a94a-b150c12863fe hl-type:: area hl-stamp:: 1674204757282