9.9 KiB
9.9 KiB
type:: REVIEWS tags:: year:: 2024 venue:: MODELS-WS full-title:: White-box LLM-supported Low-code Engineering: A Vision and First Insights date-start:: 29-07-2024 - 09:54 date-submitted:: 29-07-2024 external-links:: status:: done deadline-submission:: file:: @White-box LLM-supported Low-code Engineering: A Vision and First Insights parent:: todoist:: https://app.todoist.com/app/task/5-thomas-buchmann-rene-peinl-and-felix-schwaegerl-white-box-llm-supported-low-co-6Vm6GwWrHXCJHhhc
- ### [[Highlights]]
- # Annotazioni
- (27/7/2024, 12:18:08)
- - “grow complex” ([Buchmann et al., 2018, p. 1](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=1&annotation=WW4HZZ8P)) #ff6666
- - “offers source code generation from natural language but suffers from its blackbox nature.” ([Buchmann et al., 2018, p. 1](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=1&annotation=IMK9ZAB8)) #5fb236
- - “white-box approach” ([Buchmann et al., 2018, p. 1](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=1&annotation=KFZSLPAJ)) #ffd400
- *What do you mean?*
- - “We report on a case study in manufacturing execution systems, where the implementation of the approach provides first insights.” ([Buchmann et al., 2018, p. 1](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=1&annotation=7BA354BH)) #5fb236
- - “prompt” ([Buchmann et al., 2018, p. 1](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=1&annotation=7KEYQQH8)) #ffd400
- *THere are too many papers in CORR. I would replace them with papers appeared in peer-reviewed venues!*
- - “All further constraints and operations are phrased as natural-language annotations and remain attached to the domain model until the application stub has been generated.” ([Buchmann et al., 2018, p. 1](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=1&annotation=PTQE6UJU)) #ffd400
- *It's a chain-of-though approach. What's the novelty?*
- - “well-known weakness in doing maths can be overcome by teaching them how to use a calculator” ([Buchmann et al., 2018, p. 2](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=2&annotation=PWDNX24W)) #5fb236
- - “tool-use” of LLMs” ([Buchmann et al., 2018, p. 2](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=2&annotation=CISH3Y4A)) #5fb236
- - “,” ([Buchmann et al., 2018, p. 2](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=2&annotation=HRZ3RDXY)) #ff6666
- - “do few-shot learning by presenting them some examples to be able to generate the desired output.” ([Buchmann et al., 2018, p. 2](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=2&annotation=JIUX283Q)) #5fb236
- - “One of the first to evaluate the use of LLMs in model driven development is [3].” ([Buchmann et al., 2018, p. 2](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=2&annotation=PTJVSFE8)) #a28ae5
- - “applicability of LLMs to Business Process Management” ([Buchmann et al., 2018, p. 2](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=2&annotation=NIPV4FGM)) #a28ae5
- - “identification of processes based on documents” ([Buchmann et al., 2018, p. 2](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=2&annotation=D55I4LBU)) #5fb236
- - “Vidgof et al.” ([Buchmann et al., 2018, p. 2](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=2&annotation=PE22Y6P8)) #ff6666
- *Missing ref.*
- - ““which tasks can be achieved with already existing models” and that data sets, [[benchmarks]] and new LLMs specialized on process management are required.” ([Buchmann et al., 2018, p. 2](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=2&annotation=WSHW9BDT)) #5fb236
- - “[4] use” ([Buchmann et al., 2018, p. 2](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=2&annotation=89P468CV)) #ffd400
- *Avoid using references as subject of sentences. Write instead Forell et. al [4] use...*
- - “mining declarative process models from textual descriptions,” ([Buchmann et al., 2018, p. 2](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=2&annotation=4UAJ6QDV)) #ff6666
- *Repeatead !!!*
- - “Design of an initial domain model” ([Buchmann et al., 2018, p. 2](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=2&annotation=QYAX57L9)) #5fb236
- - “Partial code generation” ([Buchmann et al., 2018, p. 2](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=2&annotation=3DELMZ6M)) #5fb236
- - “sage of a LLM contextualized with the existing source code in order to fill in the generated stubs one after another” ([Buchmann et al., 2018, p. 2](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=2&annotation=7N8HLAZW)) #5fb236
- - “LLM GitHub Copilot 3.” ([Buchmann et al., 2018, p. 4](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=4&annotation=9N68VPV4)) #5fb236
- - “Creating the resulting source code requires manual copyand-paste in our experimentation, but will be automated in our envisioned approach.” ([Buchmann et al., 2018, p. 4](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=4&annotation=R8CLAN5M)) #5fb236
- - “seven of the nine operation bodies were provided correctly by the LLM, while we had to re-phrase two of them.” ([Buchmann et al., 2018, p. 4](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=4&annotation=A9A79UVN)) #5fb236
- - “conversational nature of LLMs.” ([Buchmann et al., 2018, p. 4](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=4&annotation=MG6ZTP5E)) #5fb236
- - “We expect, however, that the here envisioned approach is applicable, also to input field validation and the like.” ([Buchmann et al., 2018, p. 4](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=4&annotation=DLN75NRJ)) #ffd400
- *too high level, too preliminary. I understand this is a workshop submission, but I think the propometed ideas are very high level. Aspects like generalizability,ways to mitigate hallucinations, etc are not mentioned at all. I see the proposed approach as a kind of chain of thoughs, and as a such I don't see clearly explained the research contribution of the envisioned work. Moreover, I'm missing to see the envisioned approach contextualized in some modeling/software engineering process.
- The white-box of the proposed approach is limited, because it relies on LLMs that by nature are black box and as users we can "touch" the inputs and analyze the outputs without any control of the inner behaviour of the model.*
- - “deterministic generation of applications” ([Buchmann et al., 2018, p. 5](zotero://select/library/items/YA3WBXNL)) ([pdf](zotero://open-pdf/library/items/IA6G7ECR?page=5&annotation=XDEDXXA3)) #ffd400
- *I'm not convinced about the deterministic characteristcs of the generation phase!*
- ### [[Comments]]
- The paper presents an approach combining large language models (LLMs) with low-code development (LCD) platforms to empower users to create domain models and business applications.
- Even though the paper is about an interesting topic, I have some concerns related to the following issues:
- The "white-box approach" concept is mentioned but not clearly defined. The paper should elaborate on what constitutes this white-box nature, especially given that LLMs are inherently black-box systems.
- The case study and examples provided are too high-level and preliminary. The paper reports initial insights from a case study in manufacturing execution systems but needs more detailed results and a thorough analysis. Additionally, the novelty of the approach, which resembles a chain-of-thought method, is not well-articulated. More concrete examples and a clearer explanation of the research contributions and practical applications are necessary.
- The references and citations need improvement. There are instances where references are used as the subject of sentences, which should be avoided. Moreover, there are too many papers in CORR. I would replace them with papers that appeared in peer-reviewed venues.
- While the paper outlines future automation plans and potential applications, these sections are too abstract and high-level. The envisioned approach's generalizability, ways to mitigate issues like hallucinations, and integration into existing modeling and software engineering processes are not adequately discussed.
- Finally, concerning the following sentence in the conclusion section: "This fusion of LCD and LLM technology promises precise, explainable, and deterministic generation of applications, bridging the gap between technical complexity and non-technical user proficiency." I'm not convinced about the deterministic characteristics of the generation phase.
- ### [[REVIEWS/Notes]]
- ### YELLOW CONCERNS
background-color:: yellow
- {{query (and [[ffd400]] [[MDE_Intelligence_2024_paper_5]] )}}
collapsed:: true
- ### ❓️Questions
- {{query (and [[question]] [[MDE_Intelligence_2024_paper_5]] )[[question]]}}
query-table:: true
query-properties:: [:block]