251 lines
18 KiB
Markdown
251 lines
18 KiB
Markdown
-
|
||
status:: [[REJECTED]]
|
||
deadline-submission:: [[29-03-2023]]
|
||
todoist:: https://todoist.com/showTask?id=6420720229
|
||
project-type:: [[EU-PROJECT]]
|
||
external-links:: [CONTIMINER-extended-abstract - Google Docs](https://docs.google.com/document/d/1GnF-F_jHgkJgPAkXUjbo7RV4EEaKvi95Cdm6IdU_yuk/edit)
|
||
meeting:: [Search Funding & Tenders (europa.eu)](https://ec.europa.eu/info/funding-tenders/opportunities/portal/screen/opportunities/topic-search;callCode=HORIZON-CL4-2023-DATA-01;freeTextSearchKeyword=;matchWholeText=true;typeCodes=1;statusCodes=31094501,31094502,31094503;programmePeriod=null;programCcm2Id=null;programDivisionCode=null;focusAreaCode=null;destinationGroup=null;missionGroup=null;geographicalZonesCode=null;programmeDivisionProspect=null;startDateLte=null;startDateGte=null;crossCuttingPriorityCode=null;cpvCode=null;performanceOfDelivery=null;sortQuery=sortStatus;orderBy=asc;onlyTenders=false;topicListKey=callTopicSearchTableState)
|
||
|
||
- {{renderer :tocgen2}}
|
||
-
|
||
- # NOTES FROM Google doc
|
||
- **
|
||
- ADDITIONAL NOTES / COMMENTS
|
||
- 
|
||
- LUCIANO BARESI
|
||
- I am not sure I am using this document the right way, but I want to leave these notes here for further discussion tomorrow:
|
||
- What is the role of NiFi? How do we model the computing infrastructure within NiFi?
|
||
- Do we want to consider data placement, replication, etc?
|
||
- The way we slice pipes must consider the data each peace needs
|
||
- We should decide whether the underlying infrastructure can be seen as given or it can change dynamically
|
||
- If it can change, what can we do with nodes that disappear?
|
||
- Which kind of resources do we want to consider: CPUs, GPUs, memory, others?
|
||
- Which kind of deployment architecture do we have in mind? Is everything hidden behind NiFi?
|
||
- PoliMi could contribute:
|
||
- Work on dynamic management of computing resources: CPU and GPU cores allocated to microservices/containers
|
||
- Luciano Baresi, Alberto Leva, Giovanni Quattrocchi: Fine-Grained Dynamic Resource Allocation for Big-Data Applications. IEEE Trans. Software Eng. 47(8): 1668-1682 (2021)
|
||
- Luciano Baresi, Giovanni Quattrocchi, Nicholas Rasi: Resource Management for TensorFlow Inference. ICSOC 2021: 238-253
|
||
- Work on service placement on edge nodes given actual workload and available resources
|
||
- Luciano Baresi, Davide Yi Xian Hu, Giovanni Quattrocchi, Luca Terracciano:
|
||
- NEPTUNE: Network- and GPU-aware Management of Serverless Functions at the Edge. SEAMS 2022: 144-155
|
||
- Work on data management (with the involvement of Alessandro Margara)
|
||
- DAVIDE
|
||
- UDA can support the development of workflows by means of model recommenders. The idea is to rely on reusable components (see for instance [Enterprise NiFi: Implementing Reusable Components ... - Cloudera Community - 247775](https://community.cloudera.com/t5/Community-Articles/Enterprise-NiFi-Implementing-Reusable-Components-and-an-SDLC/ta-p/247775)) that can be retrieved and used according to some criteria (the way such criteria have to be specified is part of the research). The envisioned technologies underpinning the development of model recommenders, can be based on simply query languages or on more advanced ML algorithms depending on the availability of training data.
|
||
- OLD APACHE NIFI EXAMPLE
|
||
- Figure 2 illustrates a straightforward data pipeline that has been implemented utilizing Apache NiFi. The pipeline enables the collection of sensor data that is published to an MQTT broker located at the edge, and its subsequent transfer to a central server for comprehensive analysis and graphical representation through advanced dashboard tools.
|
||
- 
|
||
- YANNIS KORKONTZELOS - aLTERNATIVE SCENARIO/EXAMPLE
|
||
- Disaster Scenario
|
||
- The first step in managing the effect of a natural disaster is to assess the extent of the disaster and the issues involved. This operation is of key importance as the first 48 hours after a disaster has struck are critical. Therefore, the needs assessment phase has to be completed quickly and efficiently in order to produce an informed action plan.
|
||
- N drones:
|
||
- fly over the affected area
|
||
- record images, video and audio
|
||
- run simple processing tasks to summarise / reduce the size of data to be transmitted.
|
||
- wirelessly transmit data to a local base/server.
|
||
- The drones can be managed from a distance making this solution suitable to any type of disaster, either natural, such as earthquake, hurricane, tsunami, flood, landslide and wildfire or man-made, such as an industrial accident.
|
||
- Data can be transferred to the cloud or be processed in the local base/server before being transmitted.
|
||
- Connection between the drones and the local base/server or the local base/server and the cloud can interrupt at any point in time.
|
||
- **
|
||
- # Work Programme
|
||
- 
|
||
-
|
||
- ((63a2eb22-a909-4fca-8212-843062fb482f))
|
||
- https://ec.europa.eu/info/funding-tenders/opportunities/portal/screen/opportunities/topic-details/horizon-cl4-2023-data-01-04
|
||
- ((63a2eb41-a558-425d-97c2-1676e4ae0401))
|
||
- [[Highlights]]
|
||
- ((63a33208-aa3b-46c5-9e02-41c86de40026))
|
||
- ## Expected outcomes
|
||
- ((63b932e4-8ba3-4671-9274-91802e90e46f))
|
||
- ((63b932f2-a4d8-4f64-87ea-d65aaa04dfc7))
|
||
id:: 63b29701-94a3-412c-8b15-87871d3351a1
|
||
- ((63b93321-d49f-4f27-b542-0a388e4ec775))
|
||
- ((63b93336-2520-478c-bd87-e5c6f11cadb3))
|
||
- ((63b9335f-c708-4438-be3d-0723f5fe940d))
|
||
- ((63b9337a-0066-4725-b5f5-d9b3fc6d4651))
|
||
- ((63b933d6-15af-4b5e-a043-768ebb8d6c65))
|
||
- ((63b933e3-8412-4237-967d-6172d399ee8b))
|
||
- ((63b93408-f7b5-49e3-8882-3280e59a0e98))
|
||
- {{embed ((63b934ef-c941-42ae-a61c-d0694db9243f))}}
|
||
-
|
||
- # Previous submission
|
||
- _1671621273857_0.pdf)
|
||
- _1671621321758_0.pdf)
|
||
- ((63a2edf8-5c97-4578-aa31-adb6576d9a52))
|
||
- ((63a2ee78-765a-456a-b751-d99874c5a7a9))
|
||
- ((63a2ee90-ecfb-464f-b279-e4d1fca6aa3c))
|
||
- ((63a2ef17-a310-4818-860b-c7b7bb55b80e))
|
||
- ((63a2ef25-a170-4a3d-aaa4-9f6dacf65ed2))
|
||
- ((63bbd62f-9a8f-41ce-9b77-f1ef933bafce))
|
||
- 
|
||
- ((63baecf8-171f-4c23-a378-35abc1a6a500))
|
||
- ((63baeb5f-d703-498f-816b-112fab152a7c))
|
||
- ((63ca557d-5436-42b8-ab84-90754bf4b2dc))
|
||
- # Vision
|
||
- ((63e602e2-f405-4b5a-98f5-15e20e365860))
|
||
- # Consortium
|
||
- ## R&D partners
|
||
- **York** - WP2 leader
|
||
- **UDA** - WP3 leader
|
||
- **ATB** - WP4 leader
|
||
- **POLIMI** - WP5 leader
|
||
- **CLMS** - Platform integration (WP6?)
|
||
- **CEA** - Contribute with security tasks
|
||
- **EHU** - Contribute with ML / NLP / Federated learning tasks
|
||
- **TOG** - The Open Group
|
||
- ## Industrial partners
|
||
- **EDP** and **City of Maia** from Portugal have Edge processing power and are interested if it's possible to reduce data lake processing costs by pushing processing out to existing Edge resources, and also reduce the needs for human resources for data mining tasks through AI and automation of pipelines.
|
||
- As replacement for Kachelman, Scott has invited the government funded **Brno Communications** company (Czech Republic) that handles all the city of Brno automation and infrastructures. Waiting to hear if they are interested.
|
||
- Scott had an exchange with **[Triply](https://triply.net/)** from Salzburg (Austria) who he knows from a previous smart city proposal who build data driven mobility applications. They said they are interested in pushing their Cloud based mobility data services towards more localized processing on smart devices.
|
||
- **Infotripla**: Finnish company working as a data integrator and service provider in the Smart Traffic, Smart City, and Smart Infrastructure Treating business areas
|
||
-
|
||
- # Challenges <-> Objectives <-> Workpackage
|
||
- | **Challenges** | **Objectives** | **Workpackages**|
|
||
| Enabling deployment-independent data-mining specifications| Design and implementation of a polyglot data pipeline development environment| WP2|
|
||
|Providing intelligent assistance during data-mining processes|Development of intelligent recommenders for data mining assistance|WP3|
|
||
| Optimizing the deployment and execution of data-mining processes| Development of an optimised data mining execution environment | WP4|
|
||
|Autonomous self-organising of data-mining processes in the cloud-edge computing continuum| Development of run-time monitors for intelligent resource allocations in the continuum | WP5 |
|
||
- # Proposed approach
|
||
- {:height 382, :width 630}
|
||
-
|
||
-
|
||
- # Tasks for the next call
|
||
- DONE Update STOs and WPs according to what has been discussed on [[10-02-2023]] (**DAVIDE**)
|
||
- DONE A new iteration on the motivating example (**Davide**)
|
||
- DONE Ambition sections (baseline and innovation) (**R&D partners** according to the leaders identified above)
|
||
- DONE Initial draft of WP descriptions (**R&D partners**)
|
||
- An always updated PDF of the proposal is available at [http://vps.diruscio.org/nc/index.php/s/XmB5LFMsY8qokaB](http://vps.diruscio.org/nc/index.php/s/XmB5LFMsY8qokaB)
|
||
- GitHub repository: [davidediruscio/computing-continuum (github.com)](https://github.com/davidediruscio/computing-continuum)
|
||
- Look at the paper *A Lightweight and Efficient Approach to Java Confidential Computing on SGX*
|
||
-
|
||
- # Brainstorming
|
||
collapsed:: true
|
||
- ## MAIN CONCEPTS OF THE CALL
|
||
collapsed:: true
|
||
- #CON1 - AUTOMATED MANAGEMENT TOOLS
|
||
- #CON2 - PROGRAMMING MODELS
|
||
- #CON3 - LEARNING AND DECISION-MAKING METHODS
|
||
- *CONTINUOUS FEDERATED LEARNING FROM DATA DISTRIBUTED OVER THE EDGE AND IN THE NETWORK*
|
||
- PROVIDING THE RIGHT BALANCE BETWEEN CENTRALIZED AND DECENTRALIZED SOLUTIONS TO MAXIMIZE THE ENERGY EFFICIENCY, RESILIENCE AND EFFECTIVENESS OF THE SYSTEM
|
||
- INCREASING PRIVACY AND INTARACTION BETWEEN *DIFFERENT ORGANIZATIONS* WITHOUT EXPLICIT SHARING OF DATA
|
||
- #CON4 - END-TO-END SECURITY AND IDENTITY MANAGEMENT
|
||
- *STATE-OF-THE-ART TECHNOLOGIES, SYNERGIES WITH CLUSTER 3*
|
||
- #CON5 - RESOURCES HETEROGENEITY
|
||
- DIVERSITY OF DEVICES EQUIPPED WITH STORAGE AND PROCESSING CAPACITIES AT THE EDGE
|
||
- #CON6 - EXTREME SCALE AND FAULT-TOLERANCE
|
||
- ELASTICITY TO FLEXIBILITY ALLOCATE RESOURCES AN TASKS
|
||
- #CON7 - OPTIMIZATION OF ENERGY EFFICIENCY AND ECOLOGICAL SUSTAINABILITY
|
||
- END-TO-END DATA PROCESSING ACROSS THE CONTINUUM
|
||
- OPEN STANDARDS
|
||
- OPEN PLATFORMS
|
||
- INTEROPERABILTY MODELS
|
||
- ### MAIN SCOPE ELEMENTS OF THE CALL
|
||
- [[SERVICES AND DATA TO BE PROCESSED ACROSS VARIOUS PROVIDERS]]
|
||
- [[ADAPTIVE HYBRID COMPUTING COGNITIVE CLOUDS]]
|
||
- [[COMPUTE CONTINUUM]]
|
||
- Strong capacities at the edge and fog/IoT edge in an energy efficient and trustworthy manner
|
||
- Intelligent compute, data and *code orchestration* mechanisms
|
||
- Efficient value extraction from the huge volumes of generated data at the edge of the network
|
||
- [[HYPER-DISTRIBUTED COMPUTING APPROACHES]]
|
||
- Resources fom IoT and far-edge contrained devices, to federated fog/edge computing nodes to central cloud computing centres and hybrid cloud models
|
||
- Artificial Intelligence techniques to advance automation and dynamic adaptation of resource managemnt in Cloud and Edge systems.
|
||
- Intelligently balance computing tasks across decentral and central computing environments to optimize resources and quality of services
|
||
- Swarm computing and decentralized intelligence
|
||
- [[INNOVATIVE MANAGEMENT TECHNIQUES AND COMPUTATIONAL METHODS]]
|
||
- | ORIGINAL CHALLENGES | ORIGINAL STOs | NOTES FOR IMPROVEMENTS / LINKS WITH THE NEW CALL |
|
||
| ((63badd17-ef03-4ae5-ab1e-7937b5f03599)) |((63badd3b-6650-43e1-a431-d5b652eeb808)) | |
|
||
| ((63badd4b-f211-4497-925a-1dcce4ba4809)) | ((63badd93-b6a9-42ad-b007-763e49e78143)) | Development of an AI-assisted data pipeline development environment (#CON2 )|
|
||
| ((63badd4e-126a-4634-a1c0-29b69e5c4a1a)) | ((63badda1-cb3e-4602-b4cd-4713c2c0a745)) | Development of AI-based techniques for deep data mining (#CON3 ) |
|
||
| ((63badd52-8265-48d6-ace0-f718647e0ff8)) | ((63baddab-1b7d-4fe8-864c-c9d179cdea64))| AI-based techniques for intelligent specification and execution of data pipelines on hyper-distributed environments (#CON1, #CON3 )|
|
||
| ((63badd57-20db-43cd-9425-7a0b9154ba73))| ((63baddb9-7bcb-4d2b-9895-82f0889269b1)) | Development of a cognitive data pipeline execution environment (#CON1, #CON3, #CON5, #CON6 ) |
|
||
- #CON4 and #CON7 are not covered (i.e., security / privacy, and energy efficiency)
|
||
- STO1 can be incorporated by STO2 so the have space for covering at least one of them?
|
||
- ## Tentative new title
|
||
- Hyper-distributed and Cognitive Polyglot Data Pipelines
|
||
- ### Partners
|
||
- ((63dcd25d-186d-4e13-a01c-33d35b594ff7)) -> Security?
|
||
- ((63dcd260-172c-48b8-b8fc-06d709f0c1da)) -> for AI/NLP tasks ?
|
||
- ((63dcd259-db41-4f13-9245-f4272cd270bf)) -> Replaced by PoliMI???
|
||
- Some of the partners should take care of *simulation*
|
||
- Possibly CEA List?
|
||
- ((63dcd4be-51b4-4579-8437-ecf84db48cd2)) -> To check if they have enough edge processing power
|
||
-
|
||
-
|
||
- # [[New Motivations]]
|
||
collapsed:: true
|
||
- ((63b989cc-6342-45de-a273-65483500905c))
|
||
- ((63b98a16-d8ba-4415-87b3-88e9802c7957))
|
||
- ((63b9b89a-c3ed-4acc-b755-4d65fc85c0e1))
|
||
- ((63b9b8de-0d31-4b3a-80da-5e4697b690df))
|
||
- ((63b9b8f3-4553-453e-a9f8-c07dea30e0ad))
|
||
- ((63b9b906-62f2-464d-b9d1-2806380bd4e8)) [[COMPUTE CONTINUUM]]
|
||
- # [[New Examples]]
|
||
- # Meetings
|
||
- type:: [[meeting]]
|
||
external-links::
|
||
tags::
|
||
people::
|
||
date:: [[21-12-2022]] - 17:00
|
||
- {:height 403, :width 526}
|
||
- type:: [[meeting]]
|
||
date:: [[09-01-2023]] - 10:41
|
||
- 
|
||
- {:height 486, :width 499}
|
||
- DONE Lavorare su un nuovo motivating example per la proposta
|
||
- Devices for different domains
|
||
- Pipelines capabilities
|
||
- Standard way to specify edge capabilities
|
||
- Energy Efficiency aspects
|
||
- Intelligencs vs Context (different possible pipelines to be executed depending on the context)
|
||
- # Related projects
|
||
collapsed:: true
|
||
- ## [[Kalix]]
|
||
- [Partners | AI-Sprint (ai-sprint-project.eu)](https://www.ai-sprint-project.eu/partners)
|
||
type:: EU RIA
|
||
date-start:: [[01-01-2021]]
|
||
duration:: 36 months
|
||
collapsed:: true
|
||
- ((63b98437-a9d8-41db-8f36-6289c6d95f43))
|
||
- **AI-SPRINT Objectives**
|
||
- ((63b95eaa-4593-4cb4-a951-1b80bd3b22c0))
|
||
- ((63b95ebb-fd4e-40ef-b2e5-14bd9d480d29))
|
||
- ((63b95ec6-2c7a-4a9f-ae3c-5d8f82244acd))
|
||
- ((63b95edb-ec85-4385-a90b-08520ffd3b41))
|
||
- [[References]]
|
||
- 
|
||
- 
|
||
- 
|
||
- [Artificial Intelligence in Secure PRIvacy-preserving computing coNTinuum - Dimensions](https://app.dimensions.ai/details/grant/grant.9523373)
|
||
collapsed:: true
|
||
- ((63b959ca-ec84-4be7-95bc-93118171f531))
|
||
- ((63b959d0-4056-41fd-88c8-def2ebc882be))
|
||
- ((63b959e4-3f9d-4d68-a254-3e9b753f18f0))
|
||
-
|
||
- [Distributed Serverless Computing in Cloud-Edge Environments - Dimensions](https://app.dimensions.ai/details/grant/grant.13022663)
|
||
- [Cognitive edge-cloud with serverless computing - Dimensions](https://app.dimensions.ai/details/grant/grant.13029988)
|
||
- [Big data pRocessing and Artificial Intelligence at the Network Edge - Dimensions](https://app.dimensions.ai/details/grant/grant.9244474)
|
||
collapsed:: true
|
||
- [Big data pRocessing and Artificial Intelligence at the Network Edge | BRAINE Project | Fact Sheet | H2020 | CORDIS | European Commission (europa.eu)](https://cordis.europa.eu/project/id/876967)
|
||
- [DataCloud | Enabling the big data pipeline lifecycle on the computing continuum (datacloudproject.eu)](https://datacloudproject.eu/)
|
||
- [[@Big Data Pipelines on the Computing Continuum: Ecosystem and Use Cases Overview]]
|
||
- ((63c6ad3a-659f-4997-94b0-10a0bb252067))
|
||
- [[@Conceptualization and scalable execution of big data workflows using domain-specific languages and software containers]]
|
||
- # Related work
|
||
- [The European Cloud, Edge and IoT Continuum – EUCloudEdgeIOT](https://eucloudedgeiot.eu/the-eu-vision-on-the-cei-continuum/)
|
||
- Edge computing techniques allow these data to be processed locally, reduce security risks by data transmission to the clouds, enable real-time analysis, and offer greater privacy – such as helping organisations to take safe and reliable decisions at the point of interest.
|
||
- The Edge Computing paradigm will spur the next wave of Innovation of decentralised intelligence by optimising operations in a broad section of industries like mobility, farming, home/buildings, energy, logistics and manufacturing, and by mastering increasing volumes of data in their green and digital transition.
|
||
- [Understanding Cloud-Edge-IoT: Challenges and Opportunities - Webinar Highlights | Zenodo](https://zenodo.org/record/7185383#.Y7lX9tXMK4Q)
|
||
- 
|
||
- [Advancing Design and Runtime Management of AI Applications with AI-SPRINT - PositionPaper .pdf (polimi.it)](https://re.public.polimi.it/bitstream/11311/1172291/2/PositionPaper%20.pdf)
|
||
- [[@Efficient Data Streaming Analytic Designs for Parallel and Distributed Processing]]
|
||
- [[@Towards a computing continuum: Enabling edge-to-cloud integration for data-driven workflows]]
|
||
- [[@Computing in the Continuum: Combining Pervasive Devices and Services to Support Data-Driven Applications]]
|
||
- [[@Enterprise Restaurant Compute.pdf]]
|
||
- [[@On distributed computing continuum systems]]
|
||
- [[@Stream Processing on Clustered Edge Devices]]
|
||
- # People
|
||
- [Maria Giuffrida (0000-0001-8655-0190) (orcid.org)](https://orcid.org/0000-0001-8655-0190)
|
||
- [Lorenzo Calamai (0000-0002-5961-6227) (orcid.org)](https://orcid.org/0000-0002-5961-6227)
|
||
- [Prof. Danilo Ardagna (polimi.it)](https://ardagna.faculty.polimi.it/)
|
||
-
|
||
-
|
||
- |