logseq/pages/TOSEM-2023-0169_Proof_hi.md

type:: REVIEWS tags:: #empirical year:: 2023 venue:: TOSEM full-title:: An Empirical Analysis of Issue Templates Usage in Large-Scale Projects on GitHub date-start:: 17-08-2023 - 09:13 date-submitted:: 22-08-2023 external-links:: https://mc.manuscriptcentral.com/tosem?URL_MASK=58c5e1714ad44b3b8f3731d3a4d7645d status:: DONE deadline-submission:: file::

- #Highlights
	- #+BEGIN_IMPORTANT
	  ((64de89fe-6607-46f8-95bf-921f86f8f931))
	  ((64e46c1d-0e12-4b8e-ad42-6b214b750044))
	  ((64e46e15-13d9-4d43-b227-3c6a1ec8cdda))
	  ((64e46e45-6c3d-426d-9529-0972789280eb))
	  ((64e49d6a-b4a2-4ed9-8fe9-d595f8ff920d))
	  #+END_IMPORTANT
		- The paper aims at answering the following research questions:
			- ((64e46e75-f0cf-4d3e-86b2-0628d56aeffd))
			- ((64e46e7b-41f9-4bd6-8c98-bbaf713e1fb7))
			- ((64e46e85-7b9d-406a-8222-cca9cf1254f0))
			- ((64e46f06-7778-406c-810d-c4dd95fae92f))
			- ((64e46f0c-d60a-4e3b-ab26-40a0a5b4fa5b))
	- ((64e46bfb-ea9a-4e13-9690-dfd2350889b3))
		- This seems a negative comment compared with the positive results of the performed investigation.
	- Issues templates have been analyzed with respect to ((64e47adb-72df-4339-8683-d2b8984c7d9e)):
		- *categorization of templates with respect to their use cases*
			- ((64e47f49-fb2c-4189-8b76-35b2c30c9d80))
				- It is not evident where these 350 templates have been elicited.
			- ((64e47f7e-65bf-4517-8365-4a92832b2cd4))
		- *Identification of the template components*
			- ((64e47f09-153a-419a-886a-baad5d577fc9))
		- *Analysis of the template chooser menu*
		- *Classify the use cases of the template chooser menu*
	- ((64e4a066-0252-487a-b9ff-fbc0db0ea3a7))
		- Check if there is an explicit threat to validity for this point. Making this step wrongly can compromise the whole work.
		- #+BEGIN_IMPORTANT
		  The part about issue parsing has to be elaborated and detailed in order to convince that templates are correctly detected. Explanatory examples, showing concrete issues and how they are parsed can improve the quality of the paper and make it stronger.
		  #+END_IMPORTANT
		-
	- ((64e49c59-ee48-496b-adbb-593e10a3febd))
		- The way these expressions are evaluated is a bit obscure
			- es. "valid body"???
		- ((64e49c92-a10e-498a-ac93-787e3afc37bd))
			- I guess the comparison should consider the semantics of the titles, isn't it?
		- ((64e49cbc-f2a1-41b9-a153-6a30cc4dbf1d))
			- This is not enough!
	- ((64e49d3a-b50d-450c-ae0f-004af4143eaf))
		- Minutes? I guess most of the issues can take days to be closed, am I wrong?
		- ((64e4a535-08f0-4702-98db-b7294a90dfe4))
			- **This is also a bit obscure!!!**
			- ((64e4a768-c90e-4675-8465-93a2e36b92e8))
				- Not clear how this threat has been mitigated!
	- ((64e49f5c-ec04-4c11-a287-a52cc91fefdd))
		- how is this done?
		- how to detect is a given template is used, or on the other way round, given an issue in input, how to identify the templates that are used to create it?
	- ((64e4a418-2c50-4f2c-a39a-79ea7819abeb))
		- What can you say about the other issues?
	- ((64e4a7a6-d866-4e41-a257-84c993087e1f))
		- We don't know to what extent the 100 projects are representative of large-scale GitHub projects. I suggest to reduce the expectation by refining the title/abstract/intro by making the boundaries of the study more clear, e.g., by clearly defining the subjects of the analysis instead of waiting the details given in the methodology and results sections.
- #Comments
	- The paper explores the utilization of templates for defining GitHub issues. Based on the conducted analysis, the authors demonstrate that adopting templates substantially reduces issue resolution times, minimizes the occurrence of issue reopenings, and leads to more concise related discussions.
	- The paper is well-written and well-organized. The underlying hypothesis is logical, and the research questions pursued by the authors are well-defined. Overall, I found the paper interesting; however, I do hold particular concerns related to the following aspects:
		- **Analysis of Issue Templates**: The authors are encouraged to provide additional details regarding the methodology used for parsing issues. It is critical to thoroughly discuss the accuracy of the developed mechanism used to detect the presence of specific templates. Errors in this phase could undermine the entire paper. The section focusing on issue parsing should be expanded and elucidated, incorporating concrete examples of issues and their parsing process. This inclusion could substantiate the validity of the template detection process.
		- **Conformance Measurement Algorithm**: Further elaboration is needed in explaining the second conditional statement. What constitutes a "valid body"? The authors mention the identification of the strings "No response" or "N/A," but this approach might require more sophistication. It is advisable to justify the representativeness of such strings or explore alternatives that offer a more comprehensive analysis. Additionally, comparing anticipated title headings and actual titles within issues warrants detailed clarification, especially concerning semantic considerations during this phase.
		- **Number of Templates**: The authors introduce the figure of 350 templates without clear contextualization. It would be beneficial to elucidate the source or rationale behind this number.
		- **Effects of Issue Templates**: The authors mention that out of "1,661,788 issues, 209,654 issues have zero conformance to templates, while 394,285 issues have non-zero conformance." The disposition of the remaining issues needs to be addressed. Providing insights into how these unmentioned issues fit into the framework of the study would enhance comprehensiveness.
		- **Sampled Projects**: The extent to which the sampled 100 projects accurately represent large-scale GitHub projects requires clarification. To enhance precision, I recommend refining the title, abstract, and introduction to establish the scope of the study upfront. Defining the specific parameters of analysis early on, rather than waiting until the methodology and results sections, would provide a clearer delineation of the research.
	-
- #REVIEWS/Notes
  collapsed:: true
	- This is the link to Perceval tool
	- ((64e473c8-b750-448b-8f29-0ca477f9bd15))
		- It is a mining tool that has been used in this paper to collect issues of GitHub projects.