logseq/pages/@Domain-specific semantic-rich software knowledge graph construction through human-LLM team working.md

tags:: #zotero date:: 2026 title:: @Domain-specific semantic-rich software knowledge graph construction through human-LLM team working item-type:: journalArticle original-title:: Domain-specific semantic-rich software knowledge graph construction through human-LLM team working language:: en library-catalog:: Zotero links:: Local library, Web library

• Abstract
  • In software engineering (SE), while graph-based tools such as user models and control flow graphs effectively capture behavioral and structural aspects of systems, they inadequately represent the semantic information essential for discerning complex relationships within software artifacts. Recognizing that Knowledge Graphs (KGs) excel in modeling semantic data, we introduce the Software Knowledge Graph (SKG), which adapts KGs to encapsulate entities, relationships, and categories pertinent to SE. Conventional KG construction approaches fall into two primary paradigms: the first one is the predefined-schema-guided method, which achieves high precision through strict adherence to established schema, but suffers from limited completeness, as such predefined schema often fail to capture all relevant entities and relationships; the second one is schema-free method, which leverages Large Language Models (LLMs) to enhance completeness but consequently introduces noise and inconsistency in the absence of schematic constraints. To reconcile these inherent trade-offs, this paper introduces the Do-While Human-LLM Team Working (DHTW) method, an evolutionary schema exploration paradigm that integrates LLM-driven autonomy with expert validation to balance precision and completeness in KG construction. In the “Do” phase, LLMs autonomously extract candidate schema elements, such as entity classes, attributes, and relationships, from domain-specific corpora based on learned user preferences rather than rigid predefined schema, thereby fostering expansive knowledge discovery. Subsequently, in the “While” phase, domain experts iteratively refine and validate these elements to ensure semantic consistency and precise alignment with domain requirements; this process continues until the corpus is exhaustively explored. By synergistically combining LLM-driven exploration with human-guided validation, the DHTW method effectively overcomes the rigidity and incompleteness of predefinedschema-guided methods while mitigating the noise and irrelevance of schema-free approaches. The comprehensive experimental evaluations underscore the effectiveness of the DHTW method. Furthermore, the generality of this method provides a blueprint for Human-LLM interaction in complex domains.

• Attachments

  • PDF {{zotero-imported-file GH84YKWC, "2026 - Domain-Specific Semantic-Rich Software Knowledge Graph Construction through Human-LLM Team Working.pdf"}}

• Notes

  • I'm reviewing a research paper and I took the following notes:

    Annotazioni

    (17/5/2025, 15:28:22)

    • “Software Knowledge Graph Constructio” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 1) #ffd400 Software Knowledge Graphs should be constructed with a goal in mind, isn't it? How the goal is given and taken into account during the graph construction process?

    • “Human-LLM Team Working” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 1) #5fb236

    • “Recognizing that Knowledge Graphs (KGs) excel in modeling semantic data, we introduce the Software Knowledge Graph (SKG), which adapts KGs to encapsulate entities, relationships, and categories pertinent to SE.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 1) #5fb236

    • “predefined-schema-guided method, which achieves high precision through strict adherence to established schema, but suffers from limited completeness, as such predefined schema often fail to capture all relevant entities and relationships” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 1) #5fb236

    • “schema-free method, which leverages Large Language Models (LLMs) to enhance completeness but consequently introduces noise and inconsistency in the absence of schematic constraints” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 1) #5fb236

    • “Do-While Human-LLM Team Working (DHTW) method, an evolutionary schema exploration paradigm that integrates LLM-driven autonomy with expert validation to balance precision and completeness in KG construction” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 1) #5fb236

    • “These traditional graph representations primarily focus on capturing the behavioral and structural properties of systems” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 1) #5fb236

    • “represent entities, their relationships, and categorical information pertinent to SE.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 1) #ffd400 How do you embed or take into account the goal of the encoding of software systems in terms of KG? Knowledge Graphs can be done to capture different levels of abstraction!

    • “Do-While Human-LLM” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 1) #5fb236

    • “Team Working (DHTW)” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #5fb236

    • “In the "Do" phase, LLMs autonomously explore candidate schema elements, such as entity classes, attributes, and relationships, from specific corpora, guided by learned user preferences rather than rigid, predefined schema, thus facilitating expansive domain knowledge acquisition with LLM’s inherent divergence.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #5fb236

    • “"While" phase, users, typically domain experts, iteratively refine, validate, and adjust these elements, ensuring semantic consistency, domain alignment, and fidelity to user-specified requirements through targeted corrections and adjustments.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #5fb236

    • “To evaluate the effectiveness of the proposed DHTW method, we conducted experiments in two domains, focusing first on the API” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #ffd400 It's not clear if the API domain is just a domain that has been used to evaluate the approach or if it is the main target of the work. This is not clarified in the paper. It seems that the approach is API domain specific.

    • “API domain, where extensive schema can be derived from the widely adopted Unified Modeling Language (UML)” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #5fb236

    • “but also precisely aligned with the specific requirements of the domain.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #ffd400 How is this given to the elicitation process? This is not clear to me.

    • “balancing completeness, precision, and contextual relevance.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #ffd400 Exaclty, these are all relevant characteristics even though it is not clear at this stage how they are managed and given as input to the elicitation process.

    • “the absence of guidance frequently generates noise and leads to deviations from domain-specific standards, undermining the precision and alignment of the resulting KGs with user requirements” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #5fb236

    • “DHTW method presents a disciplined yet flexible alternative, leveraging an iterative process to address these limitations.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #5fb236

    • “In the method, the "Do" phase employs LLMs to identify and extract diverse schema elements from heterogeneous data sources, while the "While" phase incorporates expert validation to refine and organize these elements, ensuring iterative improvement.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #ffd400 This is a repetition. This concept has been already said earlier in the paper.

    • “This work introduces the DHTW method for implementing "Human-in-the-Loop" by decomposing tasks into iterative cycles.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #a28ae5

    • “In each iteration, LLMs execute subtasks (Do phase) using human-derived feedback (e.g., operational optimizations, clarified requirements), while domain experts refine outputs (While phase) to ensure precision and inject actionable guidance for subsequent rounds.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #a28ae5

    • “This closedloop process harmonizes human expertise with LLM ability, fostering continuous improvement in both precision and completeness.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #ffd400 This is a kind of chain of thoughts process, isn't it?

    • “DHTW method take the advantages of both predefinedschema-guided method and schema-free method, through Do-While schema exploration and schema-guided KG construction.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #a28ae5

    • “The former iteratively discovers schema that adheres to domain standards while transcending predefined schema” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #5fb236

    • “the latter leverages refined schema to construct KG from large-scale data, resolving the tension between rigidity and over-generalization in traditional approaches” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #5fb236

    • “KGs for APIs (Application Programming Interface) can capture complex relationships and dependencies among large number of APIs, which can help developers better manage, optimize, and automate within API ecosystems” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #5fb236

    • “reflecting domainspecific nuance” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #a28ae5

    • “Figure 1 illustrates their differences, highlighting limitations and the need for a new approach.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #5fb236

    • “KW&H, by Huang et al. [13], is a rule-based method with a predefined schema. It achieves high precision by analyzing software documentation but struggles with evolving needs, missing relationships outside predefined rules” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #5fb236

    • “Reflecting on these methods, KW&H ensures precision but sacrifices completeness, while GraphRAG and EDC excel in completeness but compromise precision and semantic richness.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 2) #ffd400 It's still vague.

    • “o API functional dependencies or interactions,” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 3) #ffd400 If this is the interest of the study, it has to be clearly stated upfront in the paper.

    • “Figure 1: Comparison of Knowledge Graph Construction Methods in the API Domain” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 3) #ffd400 Why not reusing existing approaches based on static analysis and just filter out those relationship triples that are not relevant for the goal at hand?

    • “Collections.sort(), relies on, Arrays.asList” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 3) #ffd400 This example gives some hints on the wanted granularity of the poposed approach even though it is not explicitely mentioned. It is important to specify upfront what is the granularity of the proposed approach. Is it at method level?Class level? Component level? What are the entities of interest? What are they relationships of interest?

    • “avoiding pitfalls like (Files.readAllLines(), throws, IOException),” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 3) #ffd400 Why this case are not of interest at all? It depens of the usage of the KGs, isn't it?

    • “3.1.1 Seed Text Preparation and Data Chunking.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 3) #2ea8e5

    • “In the initial phase of schema exploration, we first extract a small but representative set of seed texts (e.g., API documentation, code snippets, or Stack Overflow posts) from the target domain [21]. These seed texts serve as input data, providing domain-specific context for schema exploration” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 3) #ffd400 Ok this answers my comments about the domain previously given. It's important to add clarifications about this aspect earlier in the paper. By the way it is necessary to clarify what are the characteristics that need to be satisfied by the text and data that need to be prepared at this stage. Is data chunking automated? Depending on it, the subsequent phases are affected.

    • “[specific criteria TBD by user].” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 3) #ff6666 I guess this is a sentence to be completed. isn't it?

    • “The chunking strategy is based on size, ensuring each chunk is suitable for processing” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 3) #ffd400 Details are needed here.

    • “3.1.2 Do Phase: LLM-Based Schema Extraction and Definition.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 3) #2ea8e5

    • “In the Do phase, the LLM extracts schema from the current data chunk, aiming to ensure the completeness of the generated schema.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 3) #ffd400 Completeness with respect to what?

    • “The LLM first performs entity extraction to identify domain-related entities in the data chunk, such as a function name or variable from a code snippet.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 3) #ffd400 Again, it is necessary to clarify what's the target domain of the approach. Is it for generating KGs for APIs? Can it be applied to other kinds of artifacts? The envisioned target use of the approach is not clear to me.

    • “Subsequently, the LLM inputs the extracted entities into the entity type labeling unit, which aggregates entities into preliminary labels, for example, grouping function-related terms under a “function” category. These labels are then input into the entity type fusion unit, which further abstracts low-dimensional entity types into high-dimensional types, such as combining “function” and “procedure” into a broader “callable” type. Relation types are abstracted through the relation type fusion unit, such as merging “calls” and “invokes” into a “call” category. Through this fusion mechanism, the method unifies and standardizes diverse entity and relation types, ensuring higher abstraction and consistency in the generated schema, unlike purely automated methods that may lack such refinement.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 3) #ffd400 This is related to my previous points. Authors seem to have specific categories of interest that should be enumerated and justified in the paper.

    • “3.1.3 While Phase: Human Review and Feedback.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 4) #2ea8e5

    • “LLM generates two types of suggestions to optimize the next iteration. First, the LLM generates operational suggestions, such as suggesting not to extract temporary variables in the next iteration if the user does not need them,” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 4) #5fb236

    • “Second, the LLM generates clarification suggestions, such as suggesting to clarify if the user needs dependency relations, ensuring the schema meets specific needs.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 4) #5fb236

    • “3.1.4 Iterative Loop (Do-While Loop).” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 4) #2ea8e5

    • “until all chunks are processed” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 4) #5fb236

    • “which involves extracting API KGs from large-scale data based on validated schema.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 4) #5fb236

    • “the LLM identifies domainrelated entities in the data based on the entity types defined in the schema, such as functions or classes.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 4) #5fb236

    • “LLM identifies relationships between entities based on the predefined relation types in the schema, such as a “call” relation.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 4) #5fb236

    • “Through schema constraints, strict quality control is implemented, filtering out noise and correcting inconsistencies.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 4) #5fb236

    • “The final output of high-quality API KGs can be directly applied to practical domain tasks, such as code generation, code search, and vulnerability detection [25].” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 4) #ffd400 This statement needs to be substantiated.

    • “code generation, code search, and vulnerability detection” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 4) #5fb236

    • “balance precision, adaptability, and completeness,” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 4) #ffd400 I expect to see an experiment section that clearly show how the approach outperforms existing techniques with respect to three peculiar characteristics.

    • “we designed the method of DHTW which combines human expertise with the scalability of LLMs to deliver a KG that captures accurate relationships while avoiding noise and errors.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 4) #5fb236

    • “alignment with user needs and domainspecific nuances.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 5) #5fb236

    • “For instance, they assess whether “NullPointerException” is relevant to the user’s goals, and if deemed redundant, revise the schema by marking triples like “(Arrays.asList, throws, NullPointerException)” for exclusion.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 5) #ffd400 I?m wondering how scalable is this manual process with respect to the size of the input data. Indeed this can be an error-prone and strenuous task.

    • “Predefined-schema-guided methods ensure precision but often lack completeness, as predefined schemas may miss relevant entities and relationships, leading to incomplete domain representations.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 5) #ffd400 This is not properly supported. At the end, with the proposed approach, human implicitely defines the schema of interest. Why not giving this upfront to an algortihmic approach?

    • “Our research questions aim to investigate the indispensable role of Do-While schema exploration and definition coupled with the Human-LLM team working mechanism, in enhancing the applicability of constructed KGs.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 5) #ffd400 Repeated manh times.

    • “thereby establishing a robust framework for effectively capturing the intricate semantics inherent in SE artifacts.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 6) #ffd400 Te paper is missing an explicit definition of the SE artifact of interest or in other words that are managed by the proposed approach.

    • “RQ1: How does DHTW method outperform predefinedschema-guided and schema-free methods in constructing domain-specific KGs?” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 6) #2ea8e5

    • “RQ2: How does Human-LLM team working mechanism enhance the precision and completeness of the constructed domain-specific KGs?” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 6) #2ea8e5

    • “RQ3: How does KG schema exploration and definition strategy enhance the precision and completeness of the constructed domain-specific KGs?” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 6) #ffd400 This seems to be covered by the previous two RQs, isn't it?

    • “There are two datasets used in the experiments” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 6) #5fb236

    • “The first dataset focuses on the API domain. It includes two parts, the first part includes 206 Stack Overflow texts about Java obtained from work [13] (via GitHub [1]), which were used to summarize API entities and relationship types. The second part comprises 32,505 Java tutorial documents [14] provided by the same work [13]. However, not all Java tutorial documents contain relevant API entities and relationships. For example, some documents discuss general programming concepts without specific API references, such as: "This article focuses on two common operations: adding/removing elements...".” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 6) #ffd400 The usage of the term domain is not clear. Both datasets are for creating KGs related to APIs. What do you mean with domain?  #question

    • “Applying these criteria, we extracted API entities and relationships from 5,047 texts to construct the API KG” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 6) #ffd400 Right, it seems your goal is generating API KGs. Do you support other kinds of software artifacts? #question

    • “The second dataset is the WebNLG+2020 (v3.0) [5] semantic parsing task test set, which is also used in the EDC method [31]. WebNLG+2020 includes 1,165 pairs of texts and triplets, focusing on urban information, with reference triplet patterns covering 159 unique relationship types. We sampled 333 texts from the WebNLG+2020 training set to construct seed texts. However, we observed that the reference triplets in WebNLG+2020 are often non-exhaustive and may include information external to the text. This issue may lead to inaccuracies in evaluation results. To address this problem, we arranged for annotators to construct reference answers for 367 sampled data points in the urban information domain, following the same method as the first dataset, as an additional test set. The Cohen’s Kappa coefficients for this annotation process were 0.83 and 0.81, demonstrating high inter-annotator agreement.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 6) #ffd400 What is the wanted KG supposed to represent with this dataset?

    • “4.4 Experimental Evaluation Metrics” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 7) #ffd400 The confusion about the scope of the paper is also testified in this section. Even though in the previous section mentions that two different "domains" are considered, the evaluation metrics that are presented in section 4.4. all refer ti the API knowledge extraction problem.

    • “5.1 Evaluation of RQ1” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 7) #ffd400 Also in this section, only the API KG extraction problem is considered. No trace about the second application domain mentioned in section 4.3. Actually the same problem occurs for the whole experimental evaluation section.

    • “While they excel in breadth, they fall short in delivering a semantic-rich schema.” (“Domain-specific semantic-rich software knowledge graph construction through human-LLM team working”, 2026, p. 7) #ffd400 Can you elaborate more on this? It is not clear the problem that  you want to address. The challenges that are caused  by existing approaches and that this paper aims to address are not clear #question

    COnsider that those that are tagget with #5fb236 are just highlights, those that are tagged with #e56eee and #a28ae5 are imporant sentences. Please pay attention instead to the notes that are tagged with #ffd400. Those that are tagged with #ff6666 are typos or errors. Could you please draft a review by organizing it as follows:

    SUMMARY: Just a few sentence to summarize the work

    STRENGHTS:

    WEAKNESSES:

    COMMENTS: Organize the notes with respect to the following criteria:

    Novelty

    Rigor

    Relevance (of the contribution)

    Verifiability and Transparency

    Presentation

    And then add a Detailed Comments section to report the notes that contain issues or typos. Can you also formulate three explicit questions by considering the comments that are tagged with #question ?