type:: [[REVIEWS]]
tags:: 
year:: 2025
venue:: [[ICSE]] 
full-title::  Domain-Specific Semantic-Rich Software Knowledge Graph Construction through Human-LLM Team Working
date-start:: [[17-05-2025]] - 15:30
date-submitted::
external-links::
status:: [[DONE]] 
deadline-submission:: 
file:: [[@Domain-specific semantic-rich software knowledge graph construction through human-LLM team working]] 
parent::
todoist:: https://app.todoist.com/app/task/2338-domain-specific-semantic-rich-software-knowledge-graph-construction-through-6XXr6Q485pffwfC6

	- ### [[Highlights]]
	- ### [[Comments]]
		- #.tabular
			- ### Paper summary
				- The paper presents an approach called Do-While Human-LLM Team Working (DHTW) to support the construction of Software Engineering Knowledge Graphs (SKGs). The proposed method defines an iterative, human-in-the-loop process in which Large Language Models (LLMs) autonomously generate partial knowledge graphs, which are then validated and refined by human experts before proceeding to the next iteration. The primary objective is to balance precision and completeness in the resulting knowledge graphs. The approach is evaluated using two datasets and compared against several baseline methods.
			- ### Strengths
				- + Interesting problem in Software Engineering
				- + Human-LLM collaborative methodology for the construction of KGs
			- ### Weaknesses
				- - The scope of the approach is not clearly defined. Despite mentions of multiple domains, the work seems focused on API KGs
				- - The scalability of the human validation phase is questionable and unaddressed.
			- ### Detailed comments for authors
				- **Novelty**: The integration of human validation with LLM-based schema exploration is an interesting contribution. However, the approach seems to repackage ideas already present in schema learning and interactive KG construction with limited conceptual advancement. The focus on API KGs, if intended to be the central contribution, should be clearly stated upfront. Even though the authors claim that the approach is for representing Knowledge Graphs of Software Engineering artifacts, most of the work focuses on API. This is confusing and reduces the overall quality of the paper.
				- **Rigor**: The paper lacks rigorous justification for critical methodological decisions. In particular, I have several concerns that are related to the following issues:
					- What characteristics make seed texts representative?
					- How are completeness, precision, and contextual relevance operationalized and measured?
					- The fusion process for entity and relation types appears ad hoc and lacks reproducibility.
					- Repetition weakens the methodological exposition (e.g., Do/While phases are described multiple times).
				- **Relevance**: The application of SKG construction to the API domain is valuable but the paper suggests broader applicability without providing evidence or discussion about transferability. Terms such as "domain-specific" and "software artifact" are used inconsistently, creating confusion about what the contribution targets (see additional comments below).
				- **Verifiability & transparency**: A GitHub repository is made available even though the README file should be improved to make clear the links among the content of the repository with the process shown in Fig. 2
				- **Presentation**: The writing includes redundant contents (e.g., the Do/While phase descriptions), vague assertions (e.g., "semantic-rich"), and some confusing terminology (e.g., the usage of "domain", which is not clear to). Certain design choices (e.g., level of granularity of KGs, types of entities/relations extracted) are not discussed early enough and remain unclear. There is one incomplete sentence ([specific criteria TBD by user]), which should be fixed.
				- Additional comments:
					- Page 1 – *“Software Knowledge Graphs should be constructed with a goal in mind…”*: Clarify how user goals are explicitely captured and integrated into the KG construction.
					- Page 1 – *“represent entities, their relationships…”*: Clarify how abstraction level is selected; KGs can represent many levels of granularity. It can be software components, software dependencies, compilation units, etc.
					- Page 2 – *“the API domain…”*: Clarify whether the API domain is the main target or just a test case.
					- Page 2 – *“aligned with the specific requirements…”*: How are requirements elicited and fed into the process?
					- Page 3 – *“chunking strategy…”*: Add details on how sizes are chosen. What are the criteria that should be followed? Can chunking be automated? what are the consequences?
					- Page 3 – *“Completeness…”*: With respect to what? The user’s goal? Schema? Ground truth?
					- Page 3 – *“LLM inputs the extracted entities…”*: Categories seem pre-defined. Please enumerate and justify them.
					- Page 4 – *“code generation, code search, and vulnerability detection”*: These are important claims and should be backed by use case scenarios or evidence.
					- Page 4 – *“high-quality API KGs…”*: This needs to be substantiated. What makes them “high quality”?
					- Page 5 – *“marking triples for exclusion…”*: The scalability of this process is not discussed. How feasible is manual filtering at scale?
					- Page 5 – *“human implicitly defines the schema…”*: Why not make this explicit and use algorithmic schema guidance?
					- Page 6 – *“domain usage in datasets…”*: Terminology is confusing. Are both datasets API-specific? If so, clarify what “domain” variation is meant.
					- Page 6 – *“What is the wanted KG…”*: The role of WebNLG+2020 dataset is underexplained. What kind of software artifact is modeled?
					- Page 7 – *“Evaluation metrics all refer to API…”*: There’s no evidence of a second application domain especially in the evaluation metrics section.
					- Page 7 – *“breadth vs semantic-rich schema…”*: Elaborate on what constitutes a “semantic-rich schema.”
			- Questions
				- How do you define and represent the goal of the knowledge graph construction process, and how does it influence the schema learning and entity-relation extraction phases?
				- How do you ensure that the constructed SKG reflects the right level of abstraction and captures entities/relations relevant to the user’s intended use cases (e.g., code generation, search, etc.)?
				- The scalability of the proposed process is not discussed. How feasible is manual filtering at scale?
	- ### [[REVIEWS/Notes]]
	- ### YELLOW CONCERNS
	  background-color:: yellow
		- {{query (and [[ffd400]] [[icse2026-paper2338]] )}}
		  collapsed:: true
	- ### ❓️Questions
		- {{query (and [[question]] [[icse2026-paper2338]] )[[question]]}}
		  query-table:: true
		  query-properties:: [:block]