4.6 KiB
4.6 KiB
collapsed:: true type:: REVIEWS tags:: year:: 2026 venue:: ICSE full-title:: Determining Application Test Results Using Adaptive JSON date-start:: 14-09-2025 - 18:01 date-submitted:: external-links:: status:: DONE deadline-submission:: file:: @Determining Application Test Results Using Adaptive JSON parent:: todoist:: https://app.todoist.com/app/task/3924-determining-application-test-results-using-adaptive-json-6cVHmvg3mM8ghg48
- ### [[Highlights]]
- ### [[Comments]]
- #.tabular
- - ### Paper summary
- The paper presents an approach to test JSON-based APIs. The main building blocks of the proposed approach are "Skeleton JSON", "Adaptive JSON", and "Adaptation Sets". By means of these components, engineers can precisely define which parts of an API response have to be validated and how. The application of the approach on two different case studies has been shown.
- - ### Strengths
- + Relevant problem
- + Promising approach based on the declarative "Adaptation Set" mechanism for tuning the testing process
- - ### Weaknesses
- - The paper is not always well written
- - Several parts ar repetitive, vague, or poorly written.
- - The writing includes multiple formatting and typographic errors (e.g., Markdown synatx, missing spaces)
- - Gray literature is used even critical and motivational parts of the paper that would require peer-reviewed references
- - Comparisons with existing techniques (especially other schema-inference-based approaches) are missing or superficial
- - The validation of the claimed advantages (e.g., flexibility, maintainability, reduction of brittleness) is not adequately supported by empirical evidence.
- - ### Detailed comments for authors
- Novelty: The paper does not clearly compare the proposed approach with existing methods that extract or synthesize schemas from JSON payloads. A comparison of the proposed testing mechanism with existing baselines is also missing.
- Rigor: The methodology is described conceptually, but formal definitions, measurable properties, and an empirical evaluation are missing. Moreover, claims such as “resilient to insignificant changes” or “lightweight” need quantification or validation to be taken seriously.
- Relevance: The paper targets a relevant and timely problem in automated software testing, especially in DevOps and CI/CD pipelines that depend on stable, low-maintenance test suites. However, without a comparative evaluation, the practical significance is not clear.
- Verifiability & transparency: There is no mention of tool availability, datasets, or replication packages. The paper should provide a GitHub repository or similar so that readers can downlod tools and related artifacts to play with the proposed approach. Two case studies have been discussed, even though the given details are not enough to replicate them.
- Presentation: This is the main issue of the paper. There are redundant sentences, the writing presents the approach only at conceptual level and several formatting errors are present. Many references from non-peer-reviewed sources (e.g., Medium, Postman blogs) are given.
- Detailed comments:
- p.1: The limitation of JSON-DDT is vaguely mentioned; its impact or relation to the contribution is not evident.
- p.2: “resilient to insignificant changes” - This is too vague. Define what constitutes an “insignificant” change.
p.2: JSON Schema is mentioned, but no comparative discussion is provided - how does the proposed approach differ from or improve upon it?
- p.2: The supposed “distinct advantages” of the technique need to be clarified and situated in the literature (e.g., compare with 10.1016/j.knosys.2016.03.020 and 10.1109/MODELS50736.2021.00033).
- p.2: Claim that Skeleton JSON is “lightweight” and suitable for test automation is not supported by any empirical or architectural argument.
- p.2: What were the requirements that led to the definition of Adaptive JSON? These are not made explicit.
- p.7: The claim about brittleness and spurious failures in Snapshot Testing should be supported by examples or quantitative evidence.
- p.8: The authors state that the technique was “demonstrated in case studies” - but there is no real comparison with other approaches, no quantitative results, and no ablation study.
- ### [[REVIEWS/Notes]]
- ### YELLOW CONCERNS
background-color:: yellow
- {{query (and [[ffd400]] [[ICSE2026-paper3924]] )}}
collapsed:: true
- ### ❓️Questions
- {{query (and [[question]] [[ICSE2026-paper3924]] )[[question]]}}
query-table:: true
query-properties:: [:block]