13 KiB
13 KiB
type:: REVIEWS tags:: year:: 2024 venue:: SOSYM full-title:: date-start:: 10-10-2024 - 15:00 date-submitted:: external-links:: status:: DONE deadline-submission:: file:: @SOSYM-24-00004734_Proof_hi parent:: todoist:: https://app.todoist.com/app/task/manuscript-id-sosym-24-00004734-now-in-your-reviewer-center-software-and-systems-6W6r44cfwPc7XqXg
- ### [[Highlights]]
-
- # Annotazioni
collapsed:: true
- (10/10/2024, 14:54:36)
- ### SOSYM\-24\-00004734_Proof_hi
- \- “chemaless graph datasets” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=3&annotation=LM9TW69W)) #5fb236
- \- “Design Science effort” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=3&annotation=YHIHTJ6G)) #5fb236
- \- “KG development and integration method” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=3&annotation=Y6BTEASQ)) #5fb236
- \- “students who tend to perceive symbolic AI as a niche of questionable relevance” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=4&annotation=FMIAUBIC)) #5fb236
- \- “"citizen knowledge acquisition".” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=4&annotation=EB3MV7CC)) #a28ae5
- \- ““low\-code knowledge graph building”” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=4&annotation=VDKJGZGH)) #a28ae5
- \- “reate KG structures” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=4&annotation=CDJ4GBPR)) #a28ae5
- \- “RDF/OWL jargon or the ontology\-first experience of a tool like Protégé” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=4&annotation=WUM4DNMV)) #5fb236
- \- “This paper's proposal is a DSML (and associated modeling method deployed as an ADOxx modeling tool) that obscures as much as possible the formal details of knowledge representation standards such as RDF/S and OWL” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=5&annotation=8PHJKSMV)) #5fb236
- \- “but the focus is still on the schema layer and not on the creation of arbitrary graph structure” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=5&annotation=E9GGAW54)) #5fb236
- \- “Section 2 provides extensive information about the problem identification objectives definition. Section 3 discusses design decisions and implementation details. Section 4 presents two application use\-cases, Section 5 discusses a series of evaluations. Section 6 comments on related works, and Section 7 concludes our work with a SWOT analysis and outlook” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=6&annotation=6RLH4HUF)) #5fb236
- \- “Moreover, the current visual tools are limited to saving the created schemas in some basic format and do not care about specialized usage scenarios such as those mentioned above, that may require interoperability.” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=6&annotation=8LQB2B6K)) #5fb236
- \- “Exposing such exemplars to immediate consumption (the SEO and the GraphRAG use cases described above, but also the cognitive acquisition for novice students) is therefore prioritized.” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=7&annotation=GRP4AW26)) #5fb236
- \- “roblem Statement and Objectives Definition” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=7&annotation=FY2PE2AV)) #ffd400
- *The section needs improvements. It does not clearly present the problem, it is verbose and should be more focused by clearly presenting the problem, e.g., by discussing explanatory cases and examples.*
- \- “We aim for a Pareto\-driven approach where 80% of the user needs are served by 20% of the RDF\-based specifications captured in a DSML\-based modeling method” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=7&annotation=VZVXXNJJ)) #ffd400
- *What do you mean with "served"?*
- \- “eveloped here as a Design Science artifact” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=7&annotation=SETZSN9M)) #ffd400
- \- “weakly structured interviews were used to collect pain points about the tooling they can offer to students and the learning objective affordances of such tooling.” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=8&annotation=5DKGM6VL)) #5fb236
- \- “GraphRAG scenario was adopted from recent technical propositions available in the literature” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=8&annotation=SGLKC9XC)) #5fb236
- \- “nd also from KG platform vendors” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=8&annotation=XQ4TV2ST)) #5fb236
- \- “graphical note\-taking tools have established this as a popular practice of structuring ideas and was the originating motivation for repurposing here the practice of DSML engineering;” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=9&annotation=9J7ZKG5U)) #ffd400
- *Why are they not enough to support the specification of concepts and relationships among them (thus box and lines?)*
- \- “KG building is done by semantic technology experts in specialized tools that first require familiarity with standards such as OWL and RDF, if not with underlying formal logic” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=9&annotation=RPFEINLY)) #a28ae5
- \- “TBOX” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=9&annotation=4KDG3G7C)) #ffd400
- *Still on the readability of the paper. The work mentions acronyms or concepts, assuming that the reader is aware of them. The authors should revise the paper by ensuring that all the mentioned concepts / technologies are properly introduced especially for the readers that are not experts of the considered domain. A related minor issue is that references re not always given the first time when a given technology is mentioned (e.g. see Langchain, it is mentioned at page 10 the first time, whereas the reference is given at page 17).*
- \- “usability requirements” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=9&annotation=GQFHCGSL)) #ffd400
- *How have you elicited such requirements? It is important to discuss how you have elicited them, e.g., if they come from the interview you have done with stakeholders, or if they come only by the experience of the authors.*
- \- “schema can be postpone” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=9&annotation=IKADSH2F)) #5fb236
- \- “different visual loads:” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=9&annotation=Y6XUAL6R)) #5fb236
- \- “RDF/OWL jargon should be avoided wherever possible,” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=10&annotation=L2QL2M8G)) #5fb236
- \- “LLM services” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=10&annotation=GM6BEGI5)) #ffd400
- *What do you mean?*
- \- “model\-driven editing layer over a standards\-compliant KG platform” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=10&annotation=VWMD7UQD)) #5fb236
- \- “Langchain” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=11&annotation=MSABGQCX)) #ffd400
- \- “We decided to add less abstract classes because ADOxx as a metamodeling environment allows us to enforce a series of relation constraints” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=12&annotation=ZBL4CCLZ)) #5fb236
- \- “Prefix changing” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=17&annotation=D3NMLK6F)) #ffd400
- *It's not explicit the connection between "Prefix changing" and "URI generation" with the requirements given in Section 2.*
- \- “coupling RAG with RDF” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=19&annotation=Q398QJQZ)) #5fb236
- \- “AdoScripts in the modeling tool take care of collecting the natural language query and passing it to a Python proxy script that performs the actual KG and LLM access” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=20&annotation=WM3GE6VC)) #5fb236
- \- “This serves to obtain a subgraph of entities and relations connected to the subject of the user query.” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=20&annotation=RSV6N5MC)) #5fb236
- \- “Figure 6 presents the flow of the LLM integration through LangChain, emphasizing on the interactions with the KG and the extraction of a subgraph by navigating radial relationships starting from the subject node;” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=21&annotation=FJKRPPXN)) #5fb236
- \- “the” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=21&annotation=7U3H5D9E)) #ff6666
- \- “Step 5.” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=21&annotation=EJ4VG3B7)) #ffd400
- *A concrete example with the variable prompt and results actualized is necessary.*
- \- “The modeling tool can serve to both building ontologies and graph data as shown in Figure 8.” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=23&annotation=9JJWCGN9)) #5fb236
- \- “GRAPHxx capabilities were employed in this demo case for designing and querying a KG that can be rolled out on a real world scenario of analyzing interactions between business actors, employees and other constituting elements.” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=24&annotation=UPTPFGC4)) #5fb236
- \- “We used ADOxx's scripting language to automatically populate diagrams in GRAPHxx.” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=25&annotation=M4KXCDW5)) #5fb236
- \- “4500 concepts per diagram, also included only prefixes, URIs and labels for ThingIdentifier nodes and RelatedTo connectors” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=26&annotation=K4J4V48S)) #5fb236
- \- “conceptual schemas expressed as UML diagrams to graph equivalents.” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=27&annotation=QKZ3TKRW)) #5fb236
- \- “RDF schema generations directly from an intuitive diagramming environment that empowers users to think in free networks of associations close to the RDF data model” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=27&annotation=UX8PVY5A)) #5fb236
- \- “BPMN2KG acts in a similar way, but does not rely on the BPMN standard, since it provides a visual layer bridging the RDF formalisms with ADOxx meta\-metamodeling capabilities.” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=28&annotation=BKTNUUHF)) #ffd400
- *WHat are the limitations of tools like BPMN2KG with respect to the requirements presented in Sec. 2? It is necessary to discuss existing works with respect to a set of elicited requirements and then show what are the challenges that still need to be addressed.*
- \- “SWOT” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=30&annotation=V5D75UXT)) #5fb236
- \- “Strengths” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=30&annotation=6XTCI4MD)) #2ea8e5
- \- “Opportunities” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=31&annotation=Q54VENPI)) #2ea8e5
- \- “Weaknesses” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=31&annotation=WGKGHJJQ)) #2ea8e5
- \- “Threats” ([pdf](zotero://open\-pdf/library/items/RP5CFRYG?page=31&annotation=TRGLGZY2)) #2ea8e5
- ### [[Comments]]
- Summary: The paper presents a tool-supported approach to specifying knowledge graphs. The approach has been implemented on top of the AIDOxx platform, and it permits to query specified graphs by means of LLMs. In particular, the tool allows the export of the nodes of interest, specifies queries, and composes them in a prompt to be sent to Open API via LangChain.
- Comments: The paper is about an interesting topic and it has been applied on two different use cases to show its adoption in different contexts. However, I have some concerns that are mainly related to the following main issues:
- Presentation: The paper is not always well written, the reader is supposed to be aware of many concepts that are mentioned in the paper without a proper introduction and presentation.
- Strengths/Novelty: The presented tool for KG modeling offers interesting features, particularly the integration with LLMs. However, it requires a more rigorous comparison with existing tools. Section 2 outlines the requirements but lacks a clear elicitation process. Additionally, these requirements should be systematically used to contrast GRAPHxx with similar tools. This would highlight the unique contributions and limitations of GRAPHxx in comparison to existing solutions, providing a clearer understanding of its innovation and applicability.
- Evaluation: The evaluation needs a major revision. While the paper references existing KG tools, it must thoroughly assess them against the requirements outlined in Section 2. By directly comparing tools like BPMN2KG, Protégé, and other DSML-based KG editors to GRAPHxx, the paper could strengthen its argument for the tool’s novelty and address gaps in existing technologies. This section should also clarify the limitations of current visual tools concerning schema creation, usability, and interoperability, which GRAPHxx aims to address. Moreover, the functionality of generating prompts to query specified KBs should be properly evaluated by defining research questions, a ground truth to compare with, and evaluation metrics.
-
- ### [[REVIEWS/Notes]]
- ### YELLOW CONCERNS
background-color:: yellow
- {{query (and [[ffd400]] [[SOSYM-24-00004734_Proof_hi]] )}}
collapsed:: true
- ### ❓️Questions
- {{query (and [[question]] [[SOSYM-24-00004734_Proof_hi]] )[[question]]}}
query-table:: true
query-properties:: [:block]