38 KiB
38 KiB
type:: REVIEWS tags:: year:: 2025 venue:: ICSE full-title:: On Automating Configuration Dependency Validation via Retrieval-Augmented Generation date-start:: 16-05-2025 - 17:19 date-submitted:: external-links:: status:: DONE deadline-submission:: file:: @On automating configuration dependency validation via retrieval-augmented generation parent:: todoist:: https://app.todoist.com/app/task/1465-on-automating-configuration-dependency-validation-via-retrieval-augmented-g-6XXr6PrWMWjc9wW6
- ### [[Highlights]]
- # Annotazioni
(16/5/2025, 17:16:43)
- “Configuration dependencies arise when multiple technologies within a software system require coordinated settings for correct interplay.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=1&annotation=2Z2M3C8H)) #5fb236
- “high false-positive rates,” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=1&annotation=SSYXPGX9)) #a28ae5
- “large language models (LLMs) for dependency validation still suffers from inaccuracies due to projectand technology-specific variations, as well as from missing contextual information.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=1&annotation=G9KX4D8N)) #a28ae5
- “it is unclear which architectural decisions, which information resources, and what kind of information help best to automate the validation of configuration dependencies” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=1&annotation=USN3RIQW)) #a28ae5
- “we conducted a large empirical study on validating configuration dependencies.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=1&annotation=6P49UFXJ)) #5fb236
- “LLMs already demonstrate solid validation abilities,” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=1&annotation=9NDSYVN7)) #5fb236
- “while RAG has only marginal or even negative effects on the validation performance of the models.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=1&annotation=K3USACMP)) #5fb236
- “Simplistic RAG systems may not benefit from additional information if it is not tailored to the task at hand” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=1&annotation=XWNXM4G5)) #a28ae5
- “and it is often unclear upfront what kind of information yields improved performance.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=1&annotation=DT44ZJBD)) #a28ae5
*This is interesting!!!*
- “Consider a typical Spring Boot application: the database credentials specified in the application.yml in Listing 1 must match those defined in the docker-compose.yml in Listing 2 for the application to successfully connect to the database. This constraint constitutes a configuration dependency. Such dependencies can occur within a single technology (intra-technology) and across multiple technologies (cross-technology).” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=1&annotation=TVSCS6EB)) #e56eee
- “configuration dependencies are rarely fully documented [6], leading to incomplete and often outdated documentation [14, 16, 26].” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=1&annotation=H5379W6C)) #5fb236
- “For example, a recent configuration update at CrowdStrike caused a massive outage, crashing millions Windows systems [17]. In 2021, a minor configuration change at Facebook resulted in a six-hour global service outage [8]. Similarly, a misconfiguration at Microsoft left the Azure Compute service unavailable for more than two hours [24].” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=2&annotation=PSX6LWF8)) #5fb236
- “Several dependency detection approaches have been proposed, using heuristics [6, 7, 35, 42], domain-specific languages [2, 12], and machine learning techniques [29, 30, 45]. Unfortunately, these approaches often suffer from a high rate of false positives, where, for example, independent configuration options are incorrectly linked due to coincidental similarities.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=2&annotation=IRE7JQ25)) #5fb236
- “outdated training data of LLMs (e.g., due to changes in configuration formats and frameworks), the unavailability of project-specific information (e.g., module organization or available resources), and ambiguous naming schemes for configuration options.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=2&annotation=CHNBK82Z)) #a28ae5
- “In this work, we set out to evaluate the usefulness of RAG for configuration dependency validation. To this end, we analyze the validation effectiveness of vanilla LLMs compared to RAG, the effect of different RAG architectures on the validation performance, and what types of information LLMs actually require for effective dependency validation.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=2&annotation=HRAUJI3E)) #e56eee
- “we develop a novel RAG system for dependency validation, which integrates four widely-used context sources (i.e., technology docs, Stack Overflow posts, GitHub repositories, and Web search results) to provide additional context information to LLMs.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=2&annotation=4CCMRWGK)) #e56eee
- “(1) reducing false positives by providing accurate classification capabilities in a automated and technology-agnostic way” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=2&annotation=EDJWK4NR)) #2ea8e5
- “(2) assessing the usefulness of RAG for dependency validation” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=2&annotation=A6NMTNFF)) #2ea8e5
- “(3) identifying what information LLMs actually require for effective dependency validation” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=2&annotation=ZFQHPVA8)) #2ea8e5
- “350 configuration dependencies” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=2&annotation=LXC3M9IM)) #5fb236
- “six state-of-the-art LLMs” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=2&annotation=6559MI3W)) #5fb236
- “eight RAG variants” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=2&annotation=WCI7KTH7)) #5fb236
- “Our results show that vanilla LLMs already exhibit solid validation abilities, while RAG, contrary to its promise, only has marginal or even negative effects on their validation effectiveness.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=2&annotation=BCQQQ3HZ)) #5fb236
- “it often lacks dependency-related information and thus increases noise for LLMs rather than providing meaningful information, negatively affecting their validation performance.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=2&annotation=3UCJQ7NB)) #e56eee
- “the key takeaway is that merely adding contextual information does not guarantee better validation effectiveness of LLMs. Instead, a systematic refinement of RAG components and particularly context sources is essential to unleash the full potential of RAG systems for dependency validation, and thus needs to be done in any RAG development activity.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=2&annotation=SEHTWSGV)) #a28ae5
- “We provide a supplementary website with all code scripts, dependency datasets, prompt templates, and validation results at https://anonymous.4open.science/r/rag-config-val-D115.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=3&annotation=LJC9RFQF)) #5fb236
- “Configuration Dependency Detection” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=3&annotation=RJGQGQZ9)) #2ea8e5
- “These approaches offer valuable solutions for detecting configuration dependencies in software systems, but often require extensive additional validation efforts. In contrast to the aforementioned approaches, our goal is to enable an automated and technologyagnostic validation of dependencies using RAG.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=3&annotation=VPL55ZUS)) #a28ae5
- “Configuration Validation” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=3&annotation=ZTKJHCDX)) #2ea8e5
- “Both approaches require manual engineering to define specification rules.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=3&annotation=YQI7IVKY)) #5fb236
- “The work most closely related to ours is that of Lian et. al [20], which investigates the effectiveness of using LLMs for configuration validation” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=3&annotation=42GWMZJR)) #5fb236
- “They introduce Ciri, an LLM-based framework, to detect constraint and dependency violations in single configuration files.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=3&annotation=UQ7J85BQ)) #5fb236
- “While constraint violations (e.g., syntax and value range) are more common, dependency violations are often project- and technologyspecific and thus harder for LLMs to capture without fine-tuning on dependency-related data or additional relevant contextual information.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=3&annotation=J2E582BG)) #5fb236
- “The key difference to our work is that Lian et al. [20] focus on single-file constraint and dependency violations, while our approach explicitly focuses on validating cross-technology configuration dependencies using RAG.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=3&annotation=4TAGIQ4Q)) #a28ae5
- “3 CONFIGURATION DEPENDENCY VALIDATION WITH RAG” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=3&annotation=4EXGSH46)) #ffd400
*Overall the section is a bit high level. It is supposed to describe and explain the main technical aspects of the system shown in Fig. 1. Unfortunately, the subsections stay too high level without conveying the technical details that reader is supposed to see to grasp how the system works.*
- “To index content derived from the context sources, we first generate documents with metadata, such as the origin of a context source.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=3&annotation=8PDKNPVH)) #5fb236
- “we generate a retrieval query for each validation request by adding all relevant information specific to the configuration dependency of interest.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=4&annotation=932U3IAN)) #ffd400
*How? See my previous comment.*
- “retrieved context information” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=4&annotation=PSG6THTL)) #ffd400
*Can you give some example?*
- “configuration options” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=4&annotation=T3GF2X96)) #ffd400
*How are they distilled?*
- “Inconsistencies in these configuration values can lead to configuration errors. Importantly, configuration options may have equal values by accident, meaning that there is no actual dependency, but it just happens that they have equal values.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=4&annotation=HET9GPA9)) #ffd400
*It would be interesting to read why you had to add such a clarification. What happened during the iterative defintion of the prompt? without such a clarification, what did the author experience?*
- “{context}” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=4&annotation=K8Z3Y2T7)) #ffd400
*We need a concrete example of context here?*
- “(1) plan: a string that describes the plan how to validate the dependency based on the contextual information step by step; (2) rationale: a string that explains whether and why the configuration options depend on each other or not; and (3) isDependency: a boolean indicating the final decision of the LLM whether the given dependency is a true dependency or a false positive.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=4&annotation=9W9G8F6R)) #ffd400
*Also in this case it is necessary to give some examples. Authors correctly added in the introduction section an explanatory example to present the problem of dependency violation. Why not relying on the same case to give explanatory examples of the different components of the process and related artifacts?*
- “RAG for Dependency Validation” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=4&annotation=44RI3PUE)) #2ea8e5
- “We evaluate whether and to what extent RAG systems improve over vanilla LLMs for dependency validation” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=4&annotation=365XFRAU)) #5fb236
- “How effective are vanilla LLMs compared to RAG in validating configuration dependencies?” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=5&annotation=3PLMSYT7)) #5fb236
- “To what extent do RAG architecture variants affect the validation performance?” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=5&annotation=2HGUSJ5D)) #5fb236
- “What information does RAG retrieve for dependency validation?” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=5&annotation=RXF3AMQ5)) #5fb236
- “Types of Information Needed for Dependency Validation” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=5&annotation=FST86GA5)) #2ea8e5
- “With a RAG system at our hand, we can qualitatively analyze the causes of failures as we have explicit contextual information available.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=5&annotation=3DFIGFJL)) #5fb236
- “What kind of failures occur in LLM-based dependency validation?” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=5&annotation=NMD96YYM)) #5fb236
- “What types of information do LLMs require for accurate dependency validation?” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=5&annotation=67266UL3)) #5fb236
- “RAG variants.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=5&annotation=RGQTVGNB)) #ffd400
*Authors mention may times "RAG variants" without clarifying what are the peculiar characteristics of each of them. What are the variability components that have been analyzed to produce the mentioned variants?*
- “This procedure ensures that all LLMs receive exactly the same context for a given dependency, such that we do not have effects of order or time and, thus, enabling a sound comparison of the validation performance across different LLMs and RAG variants.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=5&annotation=TWGS89Y5)) #ffd400
*At this stage reviewers have not received enough details to appreciate sentences like this one. No concrete elements have been previously given. Reviewers is supposed to guess what are the different ingredients.*
- “We examine names and values of configuration options, as well as the technologies involved and assess whether the LLM’s reasoning aligns with its incorrect final assessmen” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=5&annotation=BY2VUXJT)) #ffd400
*In sentences like this one you can refer to the example given at the beginning of the paper and that unfortunately you have no longer considered in the paper to explain the details of the proposed approach. The readability of the paper is really affected by the lack of details.*
- “From these failure categories, we derive what type of information an LLM needs to improve dependency validation.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=5&annotation=8BG4DTUE)) #ffd400
*Again, see my previous comment. Can you give an intuition about a possible improvement?*
- “Datasets” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=5&annotation=Q5A4E6UY)) #2ea8e5
- “pring Boot microservice architecture as it represents a technology stack of significant practical relevance” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=5&annotation=9HSG4VLF)) #5fb236
- “To obtain a dataset of configuration dependencies, we ran CfgNet on the subject systems. For each software project, we sampled 50 potential dependencies, leading to a final set of 500 dependency candidates — the largest cross-technology dependency dataset we are aware of. We manually check each candidate in detail and classify it into one of three categories: (1) true if it represents a valid dependency, (2) false if it is a false positive, and (3) borderline if no decision can be made based on domain knowledge.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=5&annotation=SZNUGKA6)) #5fb236
- “350 configuration dependencies, is used to investigate whether RAG enables effective dependency validation.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=6&annotation=CZ9G2EVX)) #a28ae5
- “150 configuration dependencies, serves as an unseen test set to evaluate the refined RAG variant’s ability to reliably validate configuration dependencies.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=6&annotation=3E42BL6Z)) #a28ae5
- “RAG Variants and studied LLMs.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=6&annotation=D7T4U32S)) #2ea8e5
- “Hence, we adopt eight different RAG variants that differ in several core components, such as the embedding model (ada2 vs. Qwen2), the re-ranking algorithm (Colbert vs. Sentence Transformer), and the final number of context chunks provided to the LLM (3 vs. 5)” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=6&annotation=XZMVSLPC)) #ffd400
*That's good, it answer one of my previous concern. I suggest anticipating this paragraph before the first reference of "RAG Variants" or adding references to this paragraph when "RAG Variants" are mentioned in the previous sections of the paper.*
- “a 4” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=6&annotation=H3293547)) #ff6666
- “Context Sources” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=6&annotation=BFEDIQIX)) #2ea8e5
- “All these sources were scraped parior to generation queries (i.e., representing static context), whereas the Web (i.e., dynamic context) was scraped dynamically for each generation query.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=6&annotation=SDWA4JTN)) #5fb236
- “Evaluation Metrics.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=6&annotation=YGIPPF5K)) #2ea8e5
- “hese metrics highlight the trade-off between identifying as many true dependencies as possible (recall) and ensuring the correctness of positively labeled dependencies (precision).” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=6&annotation=U7K6TWRE)) #5fb236
- “The F1-score, the harmonic mean of precision and recall, provides a single scalar for better comparability.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=6&annotation=L5J6FLNU)) #5fb236
- “By contrast, the additional context of a RAG system leads to a higher recall, which comes with a reduced precision” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 7](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=7&annotation=HCQFIIX9)) #5fb236
- “Good news is that a single architecture is constantly better than others, indicating that not model performance is affecting the outcome, but actual architectural decisions.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 7](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=7&annotation=QCBXV9GA)) #5fb236
- “We refined RAG variant R1: (1) We incorporated project-specific details from each subject project by extracting its project structure, module organization, implementation details, and available resources. (2) We refined the validation prompt to make the definition of configuration dependencies more precise and avoiding any ambiguity. (3) We collected exemplary dependencies from each failure category along with their correct classification and added the two most similar examples based on cosine similarity. We apply refinement (2) also to the vanilla LLMs as this is clearly a measure, which can improve the baseline and is not RAG specific. To not jeopardize the validity of the results, we evaluate the final validation effectiveness of both the refined vanilla LLMs and the refined RAG variant R1 on an unseen dataset of 150 real-world configuration dependencies.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 9](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=9&annotation=NC2S4S4C)) #ffd400
**It would be interesting to see how this is actually done in practice and to what extent this process can be automated.**
- “vanilla LLMs miss many dependencies, but are less prone to misclassifications than RAG systems” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 10](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=10&annotation=DCM6UWYZ)) #5fb236
- “when enriched with special information derived from a qualitative analysis of validation failures, such as projectspecific details, precise and unambiguous prompts, as well as examples of similar dependencies, our proposed RAG system achieves substantially more accurate validation results for all models, with improvements of precision by 35 % and recall by 133 % over vanilla LLMs.” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 10](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=10&annotation=GFEGCDE5)) #ffd400
**The qualitative analysis need to be added by humans. What's the price/effort of adding such information? **#**question**
- “The key takeaway is that merely adding context does not automatically enhance the validation effectiveness of LLMs. Instead, a systemic refinement of RAG components and particularly context sources is necessary to unleash the full potential of RAG for configuration dependency validation” ([“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 10](zotero://select/library/items/PLHRFXWP)) ([pdf](zotero://open-pdf/library/items/A8VRMAR4?page=10&annotation=HPUBR2IU)) #ffd400
**Related to the previous comment. Can you elaborate more on the efforts required to do so? **#**question**
- ### [[Comments]]
- #.tabular
- ### Paper summary
- This paper is on automating the validation of configuration dependencies using large language models (LLMs). In particular, it focuses on the adoption of retrieval-augmented generation (RAG) and analyze to what extent RAG can improve the detection of configuration dependencies compared to vanilla LLMs. The authors perform a set of experiments involving multiple LLMs and RAG variants, analyze performance across 500 real-world configuration dependencies, and assess the impact of architectural and contextual design choices on validation outcomes.
- ### Strengths
- + Interesting and relevant topic
- + Extensive empirical study including a comparison involving several LLMs and RAG variants
- ### Weaknesses
- - Key technical aspects of the proposed approach, especially the implementation of the RAG system, are presented at a high level, limiting reader comprehension.
- - The role of manual effort in refining prompts and integrating context is underexplored.
- ### Detailed comments for authors
- Novelty: The paper provides an interesting comparison between vanilla LLMs and multiple RAG configurations for configuration dependency validation. The work distinguishes itself by focusing on cross-technology configuration dependencies, unlike previous approaches limited to single-file checks.
Rigor: The experimental design includes various models and metrics, but the underlying implementation decisions for core components are not detailed (see the detailed section below).
Relevance: The tackled problem is indeed of high relevance for practitioners working with modern software stacks.
Verifiability & transparency: The dataset and scripts are made available, supporting replication.
Presentation: This is the main issue of the paper. In particular, the paper's readability suffers due to abstract and high-level descriptions of core mechanisms. Important clarifications and architectural details are deferred to later sections; some should be anticipated earlier (see the detailed comments below). Reuse of the motivating example throughout the paper would significantly improve understanding.
Detailed comments:
- Page 3 - "3 CONFIGURATION DEPENDENCY VALIDATION WITH RAG": The section is intended to describe the technical core of the system (Fig. 1), but it remains too high-level. It lacks the technical details that would allow readers to understand how the system operates.
- Page 4 - "we generate a retrieval query...": It is unclear how the retrieval query is constructed. Please explain the logic or heuristics used, possibly supported by an example.
- Page 4 - "retrieved context information": Provide one or more examples of what the retrieved context looks like. This would clarify how context is integrated into the validation process.
- Page 4 - "configuration options": Explain how relevant configuration options are identified and distilled. Is there a manual selection, or are heuristics or automated tools used?
- Page 4 - "configuration values may be equal by accident...": This is an important clarification. It would be valuable to elaborate on whether this was based on observed model behavior or errors during earlier iterations of prompt design.
- Page 4 - "{context}": A concrete instance of what is meant by `{context}` would help to better understand how this element contributes to the LLM's input.
- Page 4 - "plan, rationale, isDependency": These output components should be exemplified using the motivating example provided earlier in the paper. Doing so would greatly improve reader comprehension.
- Page 5 - "RAG variants.": The phrase "RAG variants" appears multiple times before their specific characteristics are described (on Page 6). Either move that explanation earlier or insert forward references when the variants are first mentioned.
- Page 5 - "ensures that all LLMs receive exactly the same context...": This presumes prior knowledge of the context construction mechanism, which has not yet been detailed. Consider adding that explanation earlier to make this sentence meaningful.
- Page 5 - "LLM’s reasoning aligns...": Use the example from the introduction to clarify how the reasoning process is judged and aligned with dependency correctness.
- Page 5 - "type of information an LLM needs...": Offer a brief concrete example of information that improved or degraded LLM performance, based on your analysis.
- Page 6 - "RAG Variants and studied LLMs.": Since this paragraph clarifies the meaning of the RAG variants, consider moving it earlier or citing it explicitly when the term is first introduced.
- Page 9 - "We refined RAG variant R1...": This paragraph is insightful, but more details are needed. For each refinement step, clarify how it was performed and whether the process could be automated or scaled.
- Page 10 - "when enriched with special information derived from a qualitative analysis...": The added information seems to be manually curated. What is the effort required to perform such qualitative analysis? Please provide an estimate or qualitative description of this cost.
- Page 10 - "systemic refinement... is necessary": Further elaborate on what is involved in this refinement. Is it manual? What is the typical workload or expertise required to achieve effective results?
- Questions
- 1. What is the effort required to perform the qualitative analysis that improves the prompt and adds domain-specific context? Could it be automated?
2. Can you elaborate on how retrieval queries and configuration options are constructed/distilled in your RAG pipeline?
- ### [[REVIEWS/Notes]]
- ### YELLOW CONCERNS
background-color:: yellow
- {{query (and [[ffd400]] [[icse2026-paper1465]] )}}
collapsed:: true
- ### ❓️Questions
- {{query (and [[question]] [[icse2026-paper1465]] )[[question]]}}
query-table:: true
query-properties:: [:block]