[logseq-plugin-git:commit] 2025-06-05T08:36:10.944Z

2025-06-05 10:36:12 +02:00
parent b2116c22df
commit e3725ac9e7
6304 changed files with 708459 additions and 464 deletions
@@ -0,0 +1,338 @@
+file:: [Nikolov et al. - 2021 - Conceptualization and scalable execution of big da.pdf](file://C:\Users\david\Zotero\/storage/QYUBVQAD/Nikolov et al. - 2021 - Conceptualization and scalable execution of big da.pdf)
+file-path:: file://C:\Users\david\Zotero\/storage/QYUBVQAD/Nikolov et al. - 2021 - Conceptualization and scalable execution of big da.pdf
+
+- Cloud infrastructures’ elasticity for scalability
+  ls-type:: annotation
+  hl-page:: 1
+  hl-color:: green
+  id:: 63c90619-1b5b-4b7e-a916-b31e78362af8
+- highly scalable workflow execution and abstract workflow definition. 
+  ls-type:: annotation
+  hl-page:: 1
+  hl-color:: green
+  id:: 63c90629-cf05-43e6-825b-9eea51752930
+- specification and scalable execution of Big Data workflows. 
+  ls-type:: annotation
+  hl-page:: 1
+  hl-color:: green
+  id:: 63c90635-dddc-4b0a-a752-15137dfcf884
+- Big Data4 workflows are composed of multiple orchestrated steps, such as workflow activities that perform various data analytical5 tasks
+  ls-type:: annotation
+  hl-page:: 1
+  hl-color:: purple
+  id:: 63c90654-38ab-4c17-b1c6-f4e345bf2a8f
+- ynamic, process heterogeneous data,6 and are executed in parallel instead of a sequential set of scientific operators 
+  ls-type:: annotation
+  hl-page:: 1
+  hl-color:: purple
+  id:: 63c9065f-e8a7-47be-b8d4-21e63552fb9b
+- econdly, given the fact that IoT, Edge and Cloud technologies converge towards10 a computing continuum, workflow steps need to be mapped dynamically to heterogeneous computing and storage11 resources to ensure scalability
+  ls-type:: annotation
+  hl-page:: 1
+  hl-color:: purple
+  id:: 63c9067a-740c-4655-87e3-5b3bd74dac41
+- a scalable, general-purpose solution for Big Data workflows that a12 broad audience can use is an open research issue 
+  ls-type:: annotation
+  hl-page:: 1
+  hl-color:: purple
+  id:: 63c906c7-b466-43a5-b10f-60888e50c43b
+- scaling up individual steps introduces race conditions between step instances18 that attempt to process the same piece of data simultaneously.
+  ls-type:: annotation
+  hl-page:: 2
+  hl-color:: purple
+  id:: 63c906ee-8b20-4a86-977d-ae6227effaa5
+- Another major challenge is achieving usability by19 multiple stakeholders as most Big Data processing solutions are focused on ad-hoc processing models that only trained20 professionals can use
+  ls-type:: annotation
+  hl-page:: 2
+  hl-color:: purple
+  id:: 63c9073e-b5ea-489d-a7c7-073fc7f42ade
+- organizations typically operate on specific software stacks, and getting experts in Big21 Data technology can introduce costs that are not affordable or practical.
+  ls-type:: annotation
+  hl-page:: 2
+  hl-color:: purple
+  id:: 63c90746-161a-4dff-a651-0e37cd1bdd6d
+- data workflow steps pertain to specific domain-dependent knowledge, which is possessed by the23 domain experts rather than the data scientists who set up the data workflows
+  ls-type:: annotation
+  hl-page:: 2
+  hl-color:: purple
+  id:: 63c913c2-8cb1-4b36-8a39-9d871c440104
+- conceptualization of Big Data workflows using a domain-specific language (DSL) 
+  ls-type:: annotation
+  hl-page:: 2
+  hl-color:: blue
+  id:: 63c913e1-53f6-4423-a3a1-2c73281aeed6
+- calable execution of Big Data workflows using software container technologies and message-oriented middle-28 ware solutions.
+  ls-type:: annotation
+  hl-page:: 2
+  hl-color:: blue
+  id:: 63c913e6-9a90-4d8b-a6a6-6094b67725ca
+- . D. Dessalk, et al., Scalable Execution of Big Data Workflows Using Software Containers, in: Proc. of the MEDES 2020, 2020, p. 76–83.
+  ls-type:: annotation
+  hl-page:: 22
+  hl-color:: green
+  id:: 63c91415-08b4-4568-8fa6-64e037c3faa4
+- Dynamic Deployment of Scientific Workflows in the Cloud Using Container Virtualization
+  ls-type:: annotation
+  hl-page:: 22
+  hl-color:: green
+  id:: 63c9142a-05c3-473f-a5c3-d5fb9b604d13
+- rchestrating Big Data Analysis Workflows in the Cloud: Research Challenges, Survey, and Future Directions
+  ls-type:: annotation
+  hl-page:: 22
+  hl-color:: green
+  id:: 63c91432-9f9d-4299-82d0-52e53ff68ca8
+- Docker container-based big data processing system in multiple clouds for everyone
+  ls-type:: annotation
+  hl-page:: 22
+  hl-color:: green
+  id:: 63c91458-593e-455e-bc6c-3b36640bcbea
+- WorkflowDSL: scalable workflow execution with provenance for data analysis application
+  ls-type:: annotation
+  hl-page:: 22
+  hl-color:: green
+  id:: 63c9147a-a5aa-4140-9185-e9136a20b65a
+- ntegrating Containers into Workflows: A Case Study Using Makeflow, Work Queue, and Docker,
+  ls-type:: annotation
+  hl-page:: 22
+  hl-color:: green
+  id:: 63c91483-d338-464c-84e4-2e1bf5f05189
+- ClowdFlows: Online workflows for distributed big data mining, F
+  ls-type:: annotation
+  hl-page:: 22
+  hl-color:: green
+  id:: 63c91493-0548-4c09-b19c-c150b881c5b0
+- software container technologies, message-oriented middleware (MOM), and a DSL to enable highly scalable workflow31 execution and abstract workflow definition
+  ls-type:: annotation
+  hl-page:: 2
+  hl-color:: blue
+  id:: 63c914ad-7e32-44d7-b7ca-d41009884e75
+- scaling up on the level of individual workflow32 steps on top of heterogeneous infrastructures while avoiding race conditions through a system of inter- and intra-33 step coordination.
+  ls-type:: annotation
+  hl-page:: 2
+  hl-color:: green
+  id:: 63c914eb-f0d1-4df9-be58-6be410a276a7
+  hl-stamp:: 1674122478299
+- analysis of the requirements for enabling43 Big Data workflows on the Computing Continuum
+  ls-type:: annotation
+  hl-page:: 2
+  hl-color:: purple
+  id:: 63c9152d-cd81-4fb1-868c-accf76a80a14
+- rious processing models can be applied for parallelizing data process-56 ing known as workflow data pattern
+  ls-type:: annotation
+  hl-page:: 2
+  hl-color:: green
+  id:: 63c9154c-7a80-445f-ab17-bdfc98260e84
+- Pipe and Filter (P&F) i
+  ls-type:: annotation
+  hl-page:: 2
+  hl-color:: green
+  id:: 63c91554-5412-41b1-8cf3-1e14d651be36
+- MOM provides an infrastructure for loosely-66 coupled and asynchronous inter-process communication using messaging capabilities
+  ls-type:: annotation
+  hl-page:: 3
+  hl-color:: green
+  id:: 63c9159c-f327-4667-8831-6cf5af6d3050
+- middleware can act as a medium for communication, whereby step instances coordinate passing70 intermediate results through sending/receiving messages in MOM queues
+  ls-type:: annotation
+  hl-page:: 3
+  hl-color:: purple
+  id:: 63c915c0-0c8f-407a-8cc1-02e4f2660fed
+- Docker is one of74 the most well-known platforms for organizing solutions based on container technologies. 
+  ls-type:: annotation
+  hl-page:: 3
+  hl-color:: green
+  id:: 63c917c3-4cd4-4418-bcb7-fa96e0829460
+- deploying distributed scalable applications
+  ls-type:: annotation
+  hl-page:: 3
+  hl-color:: green
+  id:: 63c917cf-4fd9-43d7-a3bf-343fd002140b
+- Orchestration tools use a configuration file to define container images, network, and related deployment schemes of79 an application
+  ls-type:: annotation
+  hl-page:: 3
+  hl-color:: green
+  id:: 63c917e1-c190-4b3f-85b2-08b31fe43945
+- Big Data workflow solutions that share similar design principles while fulfilling the94 needs of various groups of users and use cas
+  ls-type:: annotation
+  hl-page:: 3
+  hl-color:: green
+  id:: 63c91803-7913-4947-9c5a-6b6bc31adfb8
+- [:span]
+  ls-type:: annotation
+  hl-page:: 4
+  hl-color:: purple
+  id:: 63c91848-d3d0-47cb-93e2-61070479dc67
+  hl-type:: area
+  hl-stamp:: 1674123334656
+- usability for non-technical experts
+  ls-type:: annotation
+  hl-page:: 4
+  hl-color:: purple
+  id:: 63c918ac-ac89-46d3-b0eb-b8dbf2353276
+- However, Argo Workflows117 does not have a middleware solution to handle inter-step communication, which may result in step instances running118 into race conditions when scaled horizontally.
+  ls-type:: annotation
+  hl-page:: 4
+  hl-color:: green
+  id:: 63c918ee-63f9-4679-a65c-74102020ec38
+- Nextflow 
+  ls-type:: annotation
+  hl-page:: 4
+  hl-color:: green
+  id:: 63c9190a-b4e2-4158-aa13-c37939483635
+- Nextflow does not provide a clear separation of concerns between workflow definition and125 implementation and relies on a specific software stack (through the DSL) for workflow step implementation
+  ls-type:: annotation
+  hl-page:: 4
+  hl-color:: green
+  id:: 63c91915-137b-4d57-b65f-a48df4c7b0e9
+- achyderm is based on Docker and Kubernetes, and provides advanced features such as pluralisation132 and incremental processing. Users need technical knowledge to define workflows.
+  ls-type:: annotation
+  hl-page:: 4
+  hl-color:: green
+  id:: 63c9194a-1856-4489-8b8c-aa793fe36ee6
+- challenges of scalability, resource provisioning,148 scheduling, orchestration, and data management of data workflows
+  ls-type:: annotation
+  hl-page:: 5
+  hl-color:: purple
+  id:: 63c91996-30c1-4a25-8ff0-c3d72a66f54a
+- Commonly used data processing frameworks (such as Spark, Flink, Beam) are designed with ad-hoc process-177 ing models that technical experts on specific technology stacks can only use. 
+  ls-type:: annotation
+  hl-page:: 5
+  hl-color:: green
+  id:: 63c919e0-a840-483e-828e-949b004b0361
+- lthough most approaches and tools use185 some form of DSL or UI, the abstraction level is not sufficient to allow for separation of concerns between definition186 and implementation, which is necessary to effectively involve domain experts in workflow definition
+  ls-type:: annotation
+  hl-page:: 6
+  hl-color:: purple
+  id:: 63c91a5f-d65f-4093-bd68-857ebf016a2d
+- ynamic mapping workflow steps to heterogeneous com-188 puting and storage resources to ensure scalability.
+  ls-type:: annotation
+  hl-page:: 6
+  hl-color:: purple
+  id:: 63c91a89-f40a-4b4a-8ecb-404b4bf28319
+- scalability needs to be organized and orchestrated over heterogeneous computing192 resources.
+  ls-type:: annotation
+  hl-page:: 6
+  hl-color:: purple
+  id:: 63c91a96-6ade-41d0-9ce3-d5d1ca3c8906
+- no approach is able to unlock the full197 potential for achieving step-level scalability of workflows, which relies on workflow and step encapsulation.
+  ls-type:: annotation
+  hl-page:: 6
+  hl-color:: purple
+  id:: 63c91ab8-3355-4a4b-bb07-0833b5c5f506
+- A communication solution must decouple the communication between the steps to be200 scaled up while maintaining race-condition-free data access
+  ls-type:: annotation
+  hl-page:: 6
+  hl-color:: purple
+  id:: 63c91aca-f8fb-4cad-843e-f3349b9281d6
+- a workflow definition mechanism with a clear separation between design- and run-time aspects and not limited206 to a specific technology stack, application domain or ad-hoc processing models;
+  ls-type:: annotation
+  hl-page:: 6
+  hl-color:: purple
+  id:: 63c91ae7-3ed2-479a-8f55-94648bce938d
+- workflow run-time support that considers workflows as separate units, rather than as a single unit, for individ-208 ual workflow steps; and
+  ls-type:: annotation
+  hl-page:: 6
+  hl-color:: purple
+  id:: 63c91af0-388e-45d1-af53-5575d5f78215
+- a workflow enactment approach with event driven execution and support for race-condition-free parallel execu-210 tion.
+  ls-type:: annotation
+  hl-page:: 6
+  hl-color:: purple
+  id:: 63c91af6-5d06-4797-be43-bc584eef01fe
+- Workflow Modelling Manager
+  ls-type:: annotation
+  hl-page:: 6
+  hl-color:: purple
+  id:: 63c91b10-0fd9-4f83-b66a-49288efbe420
+- Deployment Service Runtime
+  ls-type:: annotation
+  hl-page:: 6
+  hl-color:: purple
+  id:: 63c91b13-aca5-487e-a353-1db9fd550fc9
+- Data218 Storage/Sharing Ecosystem
+  ls-type:: annotation
+  hl-page:: 6
+  hl-color:: purple
+  id:: 63c91b18-e977-4bf8-b17b-55c18c73ecd9
+- The Workflow Modelling Manager
+  ls-type:: annotation
+  hl-page:: 6
+  hl-color:: purple
+  id:: 63c91b84-b61b-478f-8fbf-c6296331b759
+- com-222 ponent handles storage configurations, data preparation, and step-level data processing and transformation operations
+  ls-type:: annotation
+  hl-page:: 6
+  hl-color:: blue
+  id:: 63c91c2f-ddf2-4e61-a95f-1fdc4b4b9967
+- deployable data workflow
+  ls-type:: annotation
+  hl-page:: 6
+  hl-color:: blue
+  id:: 63c91c36-5290-49f4-8ed7-49578b2fd266
+- Deployment Service Runtime
+  ls-type:: annotation
+  hl-page:: 6
+  hl-color:: blue
+  id:: 63c91c3c-ebce-41ab-9609-3141e8701524
+- we assume that the DSL should be used by non-technical experts271 and, therefore, should avoid complex constructions as much as possible.
+  ls-type:: annotation
+  hl-page:: 8
+  hl-color:: purple
+  id:: 63c91cc6-aa26-4a0f-9f09-ee485f231f39
+- Components of workflow step template
+  ls-type:: annotation
+  hl-page:: 9
+  hl-color:: purple
+  id:: 63c91d0d-18cd-4ec5-a534-232b2d185c50
+- he template is modified by adding the installation script for any necessary software libraries in the Docker373 image building configuration and injecting relevant code scripts (step processing code) in a specific place in the374 template logic
+  ls-type:: annotation
+  hl-page:: 11
+  hl-color:: purple
+  id:: 63c91d3e-1e40-4941-81ce-3576c7dc3c18
+- The DSL description of the prototype workflow
+  ls-type:: annotation
+  hl-page:: 12
+  hl-color:: purple
+  id:: 63c91d67-84dc-437f-8a18-2362ba07520c
+- eight 
+  ls-type:: annotation
+  hl-page:: 14
+  hl-color:: purple
+  id:: 63c91d90-732e-4f71-a669-169408b63a4f
+- share a distributed file system, and run the Docker engine440 connected to Rancher.
+  ls-type:: annotation
+  hl-page:: 14
+  hl-color:: purple
+  id:: 63c91d9b-43ae-47f6-b3e1-97df59157b03
+- quality monitoring for resistance spot welding
+  ls-type:: annotation
+  hl-page:: 17
+  hl-color:: purple
+  id:: 63c91f67-46d0-44e4-94a5-6e3c8cbc6352
+- n addition to the sensor data from the welding machines, the use case involves reference data535 that provide information about target parameters of the equipment as well as the settings of the individual machines536 and welding programs. 
+  ls-type:: annotation
+  hl-page:: 17
+  hl-color:: purple
+  id:: 63c91fb6-4463-4747-af54-259b2f796f8f
+- workflow for offline batch preparation of data for537 machine learning
+  ls-type:: annotation
+  hl-page:: 17
+  hl-color:: purple
+  id:: 63c91ffa-f2da-414b-907d-8d79c268fd8d
+- online scenario for real-time monitoring using trained ML model
+  ls-type:: annotation
+  hl-page:: 17
+  hl-color:: purple
+  id:: 63c92001-e70a-4c74-a2bf-5665989ecff7
+- distributed) file system
+  ls-type:: annotation
+  hl-page:: 17
+  hl-color:: purple
+  id:: 63c9201b-ebfa-4072-82db-fdd5a3f4536b
+- [:span]
+  ls-type:: annotation
+  hl-page:: 8
+  hl-color:: purple
+  id:: 63ca5656-a823-48d9-a94a-b150c12863fe
+  hl-type:: area
+  hl-stamp:: 1674204757282