logseq/pages/@On automating configuration dependency validation via retrieval-augmented generation.md

tags:: #zotero date:: 2026 title:: @On automating configuration dependency validation via retrieval-augmented generation item-type:: journalArticle original-title:: On automating configuration dependency validation via retrieval-augmented generation language:: en library-catalog:: Zotero links:: Local library, Web library

• Abstract
  • Configuration dependencies arise when multiple technologies within a software system require coordinated settings for correct interplay. Existing approaches for detecting such dependencies often suffer from high false-positive rates, require additional validation mechanisms, and are typically limited to specific projects or technologies. Recent work that incorporates large language models (LLMs) for dependency validation still suffers from inaccuracies due to projectand technology-specific variations, as well as from missing contextual information. A promising solution to missing contextual information represents retrieval-augmented generation (RAG) systems, which can dynamically retrieve project- and technology-specific knowledge for validating configuration dependencies. However, it is unclear which architectural decisions, which information resources, and what kind of information help best to automate the validation of configuration dependencies. In this work, we set out to evaluate whether RAG can improve LLM-based validation of configuration dependencies and which architectural decisions, as well as contextual information are needed to overcome the static knowledge base of LLMs. To this end, we conducted a large empirical study on validating configuration dependencies. Our evaluation of six state-of-the-art LLMs and eight RAG variants shows that vanilla LLMs already demonstrate solid validation abilities, while RAG has only marginal or even negative effects on the validation performance of the models. By incorporating tailored contextual information into the RAG system–derived from a qualitative analysis of validation failures–we achieve significantly more accurate validation results across all models, with an average precision of 0.84 and recall of 0.70, representing improvements of 35 % and 133 % over vanilla LLMs, respectively. In addition, these results offer two important insights: Simplistic RAG systems may not benefit from additional information if it is not tailored to the task at hand, and it is often unclear upfront what kind of information yields improved performance.

• Attachments

  • PDF {{zotero-imported-file A8VRMAR4, "2026 - On automating configuration dependency validation via retrieval-augmented generation.pdf"}}

• Notes

  • I'm reviewing a research paper and I took the following notes:

    Annotazioni

    (16/5/2025, 17:16:43)

    • “Configuration dependencies arise when multiple technologies within a software system require coordinated settings for correct interplay.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1) #5fb236

    • “high false-positive rates,” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1) #a28ae5

    • “large language models (LLMs) for dependency validation still suffers from inaccuracies due to projectand technology-specific variations, as well as from missing contextual information.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1) #a28ae5

    • “it is unclear which architectural decisions, which information resources, and what kind of information help best to automate the validation of configuration dependencies” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1) #a28ae5

    • “we conducted a large empirical study on validating configuration dependencies.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1) #5fb236

    • “LLMs already demonstrate solid validation abilities,” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1) #5fb236

    • “while RAG has only marginal or even negative effects on the validation performance of the models.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1) #5fb236

    • “Simplistic RAG systems may not benefit from additional information if it is not tailored to the task at hand” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1) #a28ae5

    • “and it is often unclear upfront what kind of information yields improved performance.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1) #a28ae5 This is interesting!!!

    • “Consider a typical Spring Boot application: the database credentials specified in the application.yml in Listing 1 must match those defined in the docker-compose.yml in Listing 2 for the application to successfully connect to the database. This constraint constitutes a configuration dependency. Such dependencies can occur within a single technology (intra-technology) and across multiple technologies (cross-technology).” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1) #e56eee

    • “configuration dependencies are rarely fully documented [6], leading to incomplete and often outdated documentation [14, 16, 26].” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 1) #5fb236

    • “For example, a recent configuration update at CrowdStrike caused a massive outage, crashing millions Windows systems [17]. In 2021, a minor configuration change at Facebook resulted in a six-hour global service outage [8]. Similarly, a misconfiguration at Microsoft left the Azure Compute service unavailable for more than two hours [24].” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2) #5fb236

    • “Several dependency detection approaches have been proposed, using heuristics [6, 7, 35, 42], domain-specific languages [2, 12], and machine learning techniques [29, 30, 45]. Unfortunately, these approaches often suffer from a high rate of false positives, where, for example, independent configuration options are incorrectly linked due to coincidental similarities.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2) #5fb236

    • “outdated training data of LLMs (e.g., due to changes in configuration formats and frameworks), the unavailability of project-specific information (e.g., module organization or available resources), and ambiguous naming schemes for configuration options.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2) #a28ae5

    • “In this work, we set out to evaluate the usefulness of RAG for configuration dependency validation. To this end, we analyze the validation effectiveness of vanilla LLMs compared to RAG, the effect of different RAG architectures on the validation performance, and what types of information LLMs actually require for effective dependency validation.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2) #e56eee

    • “we develop a novel RAG system for dependency validation, which integrates four widely-used context sources (i.e., technology docs, Stack Overflow posts, GitHub repositories, and Web search results) to provide additional context information to LLMs.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2) #e56eee

    • “(1) reducing false positives by providing accurate classification capabilities in a automated and technology-agnostic way” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2) #2ea8e5

    • “(2) assessing the usefulness of RAG for dependency validation” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2) #2ea8e5

    • “(3) identifying what information LLMs actually require for effective dependency validation” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2) #2ea8e5

    • “350 configuration dependencies” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2) #5fb236

    • “six state-of-the-art LLMs” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2) #5fb236

    • “eight RAG variants” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2) #5fb236

    • “Our results show that vanilla LLMs already exhibit solid validation abilities, while RAG, contrary to its promise, only has marginal or even negative effects on their validation effectiveness.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2) #5fb236

    • “it often lacks dependency-related information and thus increases noise for LLMs rather than providing meaningful information, negatively affecting their validation performance.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2) #e56eee

    • “the key takeaway is that merely adding contextual information does not guarantee better validation effectiveness of LLMs. Instead, a systematic refinement of RAG components and particularly context sources is essential to unleash the full potential of RAG systems for dependency validation, and thus needs to be done in any RAG development activity.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 2) #a28ae5

    • “We provide a supplementary website with all code scripts, dependency datasets, prompt templates, and validation results at https://anonymous.4open.science/r/rag-config-val-D115.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3) #5fb236

    • “Configuration Dependency Detection” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3) #2ea8e5

    • “These approaches offer valuable solutions for detecting configuration dependencies in software systems, but often require extensive additional validation efforts. In contrast to the aforementioned approaches, our goal is to enable an automated and technologyagnostic validation of dependencies using RAG.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3) #a28ae5

    • “Configuration Validation” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3) #2ea8e5

    • “Both approaches require manual engineering to define specification rules.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3) #5fb236

    • “The work most closely related to ours is that of Lian et. al [20], which investigates the effectiveness of using LLMs for configuration validation” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3) #5fb236

    • “They introduce Ciri, an LLM-based framework, to detect constraint and dependency violations in single configuration files.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3) #5fb236

    • “While constraint violations (e.g., syntax and value range) are more common, dependency violations are often project- and technologyspecific and thus harder for LLMs to capture without fine-tuning on dependency-related data or additional relevant contextual information.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3) #5fb236

    • “The key difference to our work is that Lian et al. [20] focus on single-file constraint and dependency violations, while our approach explicitly focuses on validating cross-technology configuration dependencies using RAG.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3) #a28ae5

    • “3 CONFIGURATION DEPENDENCY VALIDATION WITH RAG” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3) #ffd400 Overall the section is a bit high level. It is supposed to describe and explain the main technical aspects of the system shown in Fig. 1. Unfortunately, the subsections stay too high level without conveying the technical details that reader is supposed to see to grasp how the system works.

    • “To index content derived from the context sources, we first generate documents with metadata, such as the origin of a context source.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 3) #5fb236

    • “we generate a retrieval query for each validation request by adding all relevant information specific to the configuration dependency of interest.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4) #ffd400 How? See my previous comment.

    • “retrieved context information” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4) #ffd400 Can you give some example?

    • “configuration options” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4) #ffd400 How are they distilled?

    • “Inconsistencies in these configuration values can lead to configuration errors. Importantly, configuration options may have equal values by accident, meaning that there is no actual dependency, but it just happens that they have equal values.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4) #ffd400 It would be interesting to read why you had to add such a clarification. What happened during the iterative defintion of the prompt? without such a clarification, what did the author experience?

    • “{context}” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4) #ffd400 We need a concrete example of context here?

    • “(1) plan: a string that describes the plan how to validate the dependency based on the contextual information step by step; (2) rationale: a string that explains whether and why the configuration options depend on each other or not; and (3) isDependency: a boolean indicating the final decision of the LLM whether the given dependency is a true dependency or a false positive.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4) #ffd400 Also in this case it is necessary to give some examples. Authors correctly added in the introduction section an explanatory example to present the problem of dependency violation. Why not relying on the same case to give explanatory examples of the different components of the process and related artifacts?

    • “RAG for Dependency Validation” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4) #2ea8e5

    • “We evaluate whether and to what extent RAG systems improve over vanilla LLMs for dependency validation” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 4) #5fb236

    • “How effective are vanilla LLMs compared to RAG in validating configuration dependencies?” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5) #5fb236

    • “To what extent do RAG architecture variants affect the validation performance?” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5) #5fb236

    • “What information does RAG retrieve for dependency validation?” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5) #5fb236

    • “Types of Information Needed for Dependency Validation” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5) #2ea8e5

    • “With a RAG system at our hand, we can qualitatively analyze the causes of failures as we have explicit contextual information available.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5) #5fb236

    • “What kind of failures occur in LLM-based dependency validation?” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5) #5fb236

    • “What types of information do LLMs require for accurate dependency validation?” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5) #5fb236

    • “RAG variants.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5) #ffd400 Authors mention may times "RAG variants" without clarifying what are the peculiar characteristics of each of them. What are the variability components that have been analyzed to produce the mentioned variants?

    • “This procedure ensures that all LLMs receive exactly the same context for a given dependency, such that we do not have effects of order or time and, thus, enabling a sound comparison of the validation performance across different LLMs and RAG variants.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5) #ffd400 At this stage reviewers have not received enough details to appreciate sentences like this one. No concrete elements have been previously given. Reviewers is supposed to guess what are the different ingredients.

    • “We examine names and values of configuration options, as well as the technologies involved and assess whether the LLM’s reasoning aligns with its incorrect final assessmen” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5) #ffd400 In sentences like this one you can refer to the example given at the beginning of the paper and that unfortunately you have no longer considered in the paper to explain the details of the proposed approach. The readability of the paper is really affected by the lack of details.

    • “From these failure categories, we derive what type of information an LLM needs to improve dependency validation.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5) #ffd400 Again, see my previous comment. Can you give an intuition about a possible improvement?

    • “Datasets” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5) #2ea8e5

    • “pring Boot microservice architecture as it represents a technology stack of significant practical relevance” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5) #5fb236

    • “To obtain a dataset of configuration dependencies, we ran CfgNet on the subject systems. For each software project, we sampled 50 potential dependencies, leading to a final set of 500 dependency candidates — the largest cross-technology dependency dataset we are aware of. We manually check each candidate in detail and classify it into one of three categories: (1) true if it represents a valid dependency, (2) false if it is a false positive, and (3) borderline if no decision can be made based on domain knowledge.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 5) #5fb236

    • “350 configuration dependencies, is used to investigate whether RAG enables effective dependency validation.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6) #a28ae5

    • “150 configuration dependencies, serves as an unseen test set to evaluate the refined RAG variant’s ability to reliably validate configuration dependencies.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6) #a28ae5

    • “RAG Variants and studied LLMs.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6) #2ea8e5

    • “Hence, we adopt eight different RAG variants that differ in several core components, such as the embedding model (ada2 vs. Qwen2), the re-ranking algorithm (Colbert vs. Sentence Transformer), and the final number of context chunks provided to the LLM (3 vs. 5)” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6) #ffd400 That's good, it answer one of my previous concern. I suggest anticipating this paragraph before the first reference of "RAG Variants" or adding references to this paragraph when "RAG Variants" are mentioned in the previous sections of the paper.

    • “a 4” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6) #ff6666

    • “Context Sources” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6) #2ea8e5

    • “All these sources were scraped prior to generation queries (i.e., representing static context), whereas the Web (i.e., dynamic context) was scraped dynamically for each generation query.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6) #5fb236

    • “Evaluation Metrics.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6) #2ea8e5

    • “hese metrics highlight the trade-off between identifying as many true dependencies as possible (recall) and ensuring the correctness of positively labeled dependencies (precision).” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6) #5fb236

    • “The F1-score, the harmonic mean of precision and recall, provides a single scalar for better comparability.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 6) #5fb236

    • “By contrast, the additional context of a RAG system leads to a higher recall, which comes with a reduced precision” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 7) #5fb236

    • “Good news is that a single architecture is constantly better than others, indicating that not model performance is affecting the outcome, but actual architectural decisions.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 7) #5fb236

    • “We refined RAG variant R1: (1) We incorporated project-specific details from each subject project by extracting its project structure, module organization, implementation details, and available resources. (2) We refined the validation prompt to make the definition of configuration dependencies more precise and avoiding any ambiguity. (3) We collected exemplary dependencies from each failure category along with their correct classification and added the two most similar examples based on cosine similarity. We apply refinement (2) also to the vanilla LLMs as this is clearly a measure, which can improve the baseline and is not RAG specific. To not jeopardize the validity of the results, we evaluate the final validation effectiveness of both the refined vanilla LLMs and the refined RAG variant R1 on an unseen dataset of 150 real-world configuration dependencies.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 9) #ffd400 It would be interesting to see how this is actually done in practice and to what extent this process can be automated.

    • “vanilla LLMs miss many dependencies, but are less prone to misclassifications than RAG systems” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 10) #5fb236

    • “when enriched with special information derived from a qualitative analysis of validation failures, such as projectspecific details, precise and unambiguous prompts, as well as examples of similar dependencies, our proposed RAG system achieves substantially more accurate validation results for all models, with improvements of precision by 35 % and recall by 133 % over vanilla LLMs.” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 10) #ffd400 The qualitative analysis need to be added by humans.  What's the price/effort of adding such information? #question

    • “The key takeaway is that merely adding context does not automatically enhance the validation effectiveness of LLMs. Instead, a systemic refinement of RAG components and particularly context sources is necessary to unleash the full potential of RAG for configuration dependency validation” (“On automating configuration dependency validation via retrieval-augmented generation”, 2026, p. 10) #ffd400 Related to the previous comment. Can you elaborate more on the efforts required to do so? #question

    COnsider that those that are tagget with #5fb236 are just highlights, those that are tagged with #e56eee and #a28ae5 are imporant sentences. Please pay attention instead to the notes that are tagged with #ffd400. Those that are tagged with #ff6666 are typos or errors. Could you please draft a review by organizing it as follows:

    SUMMARY: Just a few sentence to summarize the work

    STRENGHTS:

    WEAKNESSES:

    COMMENTS: Organize the notes with respect to the following criteria:

    Novelty

    Rigor

    Relevance (of the contribution)

    Verifiability and Transparency

    Presentation

    And then add a Detailed Comments section to report the notets that contain issues or typos.

    Can you also formulate explicit questions by considering the comments that are tagged with #question? Please merge those that are about the same topics and lead to 4 questions.