From 6ac1a5003c2de6b314f99af9ca62b854439e1f24 Mon Sep 17 00:00:00 2001 From: Davide Di Ruscio Date: Wed, 4 Feb 2026 19:24:04 +0100 Subject: [PATCH] [logseq-plugin-git:commit] 2026-02-04T18:24:03.854Z --- pages/ECMFA_2026_15.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 pages/ECMFA_2026_15.md diff --git a/pages/ECMFA_2026_15.md b/pages/ECMFA_2026_15.md new file mode 100644 index 00000000..343065d5 --- /dev/null +++ b/pages/ECMFA_2026_15.md @@ -0,0 +1,24 @@ +collapsed:: true +type:: [[REVIEWS]] +tags:: +year:: 2026 +venue:: [[ECMFA]] +full-title:: A Model-Driven Approach To Support The Understanding Of Machine Learning Pipelines +date-start:: [[04-02-2026]] - 19:20 +date-submitted:: +external-links:: +status:: [[DONE]] +deadline-submission:: +file:: +parent:: +todoist:: https://app.todoist.com/app/task/14-nicolas-lacroix-mireille-blay-fornarino-philippe-collet-frederic-precioso-and-6frhCMhFhcFm3hPc + + - ### [[Comments]] + - Summary: This paper proposes a model-driven approach to improve the understanding of ML pipelines. The core idea is to reverse-engineer pipeline code into a lightweight, UML-based representation, and then incrementally enrich it with semantic annotations produced by profiling functions. The authors introduce a dedicated pattern language (Canopus) to query and compare pipelines, and they validate the approach on a corpus of 105 Kaggle notebooks, illustrating use cases such as structural filtering, compliance checking, and iterative refinement for (anti-)pattern investigation. + - Comments: Overall, I find the paper interesting and well motivated. My main minor concern is about the positioning of the related work. Section 3 does a good job at covering model querying approaches (OCL, IncQuery) and graph query languages, and it argues that many of these approaches either assume a stable/rich metamodel or introduce accidental complexity when expressing ML-specific motifs over pipelines. However, I do not see an explicit discussion of existing, widely used "pipeline modeling languages" and environments from the data mining/data science tooling ecosystem (not necessarily from the MDE community) that already let users design and execute pipelines with reusable constructs (e.g., Knime, Orange, RapidMiner). Given the topic and the paper's claims about supporting pipeline understanding and analysis, I think the reader expects at least a short paragraph explaining why these systems are out of scope (e.g., prescriptive design/execution vs descriptive reverse engineering of arbitrary code) and what the conceptual relationship is (complementary, alternative, potential integration). Without this, the impression is that the language and tooling were devised in isolation from an important body of relevant practice. + - Related to this, I would like the paper to clarify more directly how Canopus relates to tools like Orange. Orange gives users a "language" (even if mostly graphical) to assemble pipelines out of reusable blocks, while Canopus is presented as a language to query reverse-engineered pipelines. That difference is likely the key novelty, but it is not stated clearly enough: are you trying to eventually model pipelines for execution, or are you strictly targeting understanding/quality control of existing pipelines? If the latter, it would help to say so explicitly and explain why importing/describing pipelines from Orange/Knime/RapidMiner is not the focus (and whether it could become a source of descriptive models in future work). + - On the modeling side, I understand the rationale for a lightweight structural core plus an extensible description mechanism, and I agree that this separation can be a good way to avoid overfitting to a fixed taxonomy. That said, the generalizability of the approach still looks fragile in practice, because the utility of the patterns depends heavily on the availability and correctness of the attached semantic descriptions (step categories, functions, libraries, etc.). The paper mentions that the metamodel can be extended with new Information subclasses (benchmarks, model properties such as number of layers or tree depth), but it is not clear "how" from an engineering point of view: does this extension require only (i) adding new Information types in the metamodel and (ii) writing new profiling functions, or do users also need to adjust extraction rules and migrations of the persisted knowledge base? A brief concrete example would make the extensibility claim much more convincing. + - The evaluation section also exposes a major threat to validity: classification quality. The whole pipeline querying story works only if the profiling functions and the mapping from code instructions to step categories are reliable. You reuse the parser and classification mapping from Biswas et al. and extend it with function-name extraction, but the paper does not provide enough detail on how the step classification is performed, and how robust the conclusions are to misclassification. Interestingly, later in the evaluation, you explicitly detect a classification issue (DMatrix being labeled as Modeling instead of Data Preparation), and you acknowledge that such issues can affect results and interpretation. This is good and honest, but I think it should be treated more explicitly as a limitation/threat earlier, because it underpins all compliance checking patterns such as "Data Acquisition" or "Data Preparation". + - I also have a minor presentation concern about the "findings" and "observations" and the numbering. The evaluation references "Finding 3/4/5" and then lists "Observation 1/2/3"; to me, they read very similar in nature, and the numbering starting at 3 is confusing. Please, also clarify what you mean by "finding" vs "observation", because as it stands, it is not obvious that the distinction is meaningful for the reader. + - Finally, the compliance-checking results raise a wording issue about "representative pipeline". You report that the strict pattern has no matches and that even a relaxed variant matches only 15 pipelines, which you correctly note is small for something presented as "representative" in the original article, and you explain (at least partially) via the infrequency of Evaluation steps. This is a valuable critique, but it would be good to be more explicit about the interpretation: is the point that the "representative pipeline" is more of an idealized template than an empirically common structure, or that the operationalization via your patterns does not align with the original study's notion of representativeness? +- \ No newline at end of file