logseq/pages/PhD_Thesis_Mastropaolo_Antonio.md

type:: REVIEWS/PhDThesis tags:: REVIEWS/PhDThesis year:: 2024 venue::
full-title:: EXPLORING THE USAGE OF PRE-TRAINED MODELS FOR CODE-RELATED TASKS date-start::
date-submitted:: 08-05-2024 external-links:: status:: DONE deadline:: 08-05-2024 file:: @antonio-mastropaolo-phd-thesis.pdf parent:: Antonio Mastropaolo - Student of Gabriele Bavota todoist:: https://app.todoist.com/app/task/ph-d-antonio-mastropaolo-7863812588

- ### [[Highlights]]
	- The problem is that the models that are the subjects of the evaluation are not only complex, but they also change all the time. So, I wonder, how many of the experimental results would hold after next year. What about five years from now?
	- Overall, I found this to be a very convincing and very well-written dissertation. The work is relevant, timely, and original, and it represents a precious addition to the body of knowledge on software engineering especially in the context of using deep learning for code-related tasks. The thesis was an excellent read. Given the significance, novelty, and depth of contributions, and the quality of the thesis manuscript and the work reported therein, I have no hesitation to recommend that Antonio Mastropaolo be granted the PhD title.
	- Main presentation points
		- Pre-training for code-related tasks
			- Use of text-to-text transformer (proposed by Google) - T5
			- Semantic equivalence (slide 55)
		- automatically comment source code
			- method-level code summarization vs code snippet code summarization
			- Dataset to be built manually
		- New metric (SIDE)
	- ---
	- **QUESTIONS**
		- What are the strong points of your work and thus the take-away messages of your thesis despite the wide adoption of LLMs. (Slide 96)
		  background-color:: green
			- What about sustainability? Why are we supposed to use always LLMs?
			  collapsed:: true
				- GEMINI has 100 Trillion parameters
					- Better data -> less data -> shorter training -> $$ -> less Co2 emission (Green AI for software engineering)
		- How would you extend your work to deal with the problem of continual learning? All the experiments and the evaluation have been done on datasets created intentionally. As you know, repositories evolve, and a typical problem for this kind of work is that they might become obsolete or, in other words, conclusions might change. How would you refine your work to mitigate this problem?
		  background-color:: green
		- In section 4, you analyzed PEGASUS and TP approaches for generating paraphrasis for each original code description. A minor question is: what are the criteria that these tools employ to generate different variants of the original test?
		- Still in section 4
			- You used Levenshtein to measure the distance between two descriptions. You know this is a syntactical metric, I'm wondering why you have not employed different metrics, like the one you introduced in your recent ICSE paper
				- Evaluating Code Summarization Techniques: A New Metric and an Empirical Characterization A. Mastropaolo, M. Ciniselli, G. Bavota, M. Di Penta. In 46th International Conference on Software Engineering (ICSE 2024),
			- Have you considered variating also the size of the code preceding and following the emptied method to see if something change in the results?
		- Have you ever considered sharing your work with the NLP community events?
		  background-color:: green
	- ---
	- # Annotazioni  
	  
	  (8/5/2024, 16:51:39)
	- “Prof. Davide Di Ruscio” ([“antonio-mastropaolo-phd-thesis.pdf”, p. i](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=3&annotation=JHBN52QB)) #f0ff00
	- “log statements” ([“antonio-mastropaolo-phd-thesis.pdf”, p. iii](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=5&annotation=YUMUQY4M)) #f0ff00  
	  
	  *What are the goals?*
	- “William & Mary, Virginia (USA).” ([“antonio-mastropaolo-phd-thesis.pdf”, p. vi](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=8&annotation=HJILZU47)) #f0ff00
	- “Pre-trained deep learning models can effectively support the automation of tasks characterized by both code and natural language. Novel metrics are needed to properly assess their performance in the context of generative tasks.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 4](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=18&annotation=G65SJQKB)) #a28ae5  
	  
	  *NOVEL METRICS*
	- “investigate the effectiveness of pre-trained models for several code related tasks, including those characterized by bi-modal data” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 5](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=19&annotation=STLP4H9Y)) #5fb236
	- “snippet-level code summarization” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 5](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=19&annotation=B6T95I9J)) #5fb236
	- “log injection” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 5](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=19&annotation=CQTPDZKE)) #5fb236
	- “novel metric” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 5](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=19&annotation=5JSXPGQX)) #5fb236
	- “natural language summary is appropriate for a given code” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 5](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=19&annotation=Y2U95PPH)) #5fb236
	- “its sensitivity to the prompt provided as input” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 5](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=19&annotation=MLJLSYM8)) #5fb236
	- “three high-level categories” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 5](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=19&annotation=INVBHP3I)) #a28ae5
	- “definition and experimentation of DL-based techniques exploiting pre-trained models to automate code-related tasks” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 5](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=19&annotation=7LXA5WXX)) #a28ae5
	- “novel metric aimed at assessing the effectiveness of code summarization methods” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 5](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=19&annotation=FQ4L4X7N)) #a28ae5
	- “empirical assessment of the robustness of GitHub Copilot [cop] for the task of code generation, covered in Chapter 4.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 5](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=19&annotation=9B6L7UAQ)) #a28ae5
	- “The higher the textual similarity between the generated summary and a reference summary written by developers, the higher its quality.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 6](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=20&annotation=QZ4V3F53)) #a28ae5
	- “capturing the extent to which the generated summary aligns with the semantics of the documented code snippet, independently from the reference summary.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 6](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=20&annotation=YW2MCLLK)) #a28ae5
	- “ontrastive learning” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 7](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=21&annotation=LHFKMYX4)) #a28ae5
	- “it is still unclear the extent to which semantic-preserving changes in the natural language description provided to the model have an effect on the generated code” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 7](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=21&annotation=ZU6JUN7V)) #a28ae5
	- “different but semantically equivalent natural language descriptions result in the same recommended function” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 7](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=21&annotation=I6EV3DP4)) #a28ae5
	- “DL-based techniques for code-related tasks and software engineering automation” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 7](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=21&annotation=P2LAVSZQ)) #a28ae5
	- “advantages and limitations of AI-driven solutions for software development” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 7](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=21&annotation=5J2498XQ)) #5fb236
	- “We compared our technique against the state-of-the-art techniques proposed at that time for solving the same tasks.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 7](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=21&annotation=SR9J4YGV)) #5fb236
	- “logging” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 8](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=22&annotation=3VHLPAUD)) #a28ae5
	- “code summarization” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 8](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=22&annotation=6MVZNXEC)) #a28ae5
	- “Chapter 3” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 11](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=25&annotation=QTBCHV5G)) #a28ae5
	- “499,618 English sentences and 1,569,889 source code components (i.e., Java methods).” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 11](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=25&annotation=6DVMSV47)) #a28ae5
	- “develop four different models” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 11](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=25&annotation=JNKBAHUV)) #5fb236
	- “One open question related to the adoption of this tool is what is the impact of the wording used when defining the prompt on the generated source code” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 11](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=25&annotation=3GN9J8MA)) #a28ae5
	- “The basic idea is to provide the model with the buggy version of a code (e.g., a buggy Java method) asking it to generate its fixed version.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 13](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=27&annotation=SGB9SJ5P)) #a28ae5
	- “s[” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 14](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=28&annotation=W2C4KYME)) #ff6666
	- “Transformer architecture” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 14](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=28&annotation=2IMAIBYL)) #5fb236
	- “The T5 model we adopted, has then been also exploited in different fash- ions in several of the subsequent works” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 14](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=28&annotation=AS8FL5BU)) #5fb236
	- “These models undergo extensive pre-training across diverse datasets, encompassing billions of instances that include not just code files, but also a vast array of textual content sourced from the internet.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 14](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=28&annotation=ZMD9X3U6)) #5fb236
	- “Figure 2.1. Example of bug fix using OpenAI Codex LLM [CTJ+21]” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 15](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=29&annotation=JW5L5XKV)) #ff6666  
	  
	  *The code in (3) has nothing to do with the code in the given example (I meas related to the fault used for the training by means of the shot).*
	- “Contribution in the area” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 15](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=29&annotation=R88LNZS8)) #ffd400  
	  
	  *I understand this was produced before ChatGPT. What can you say about the novelty of your contribution despite the ChatGPT/GenAI era. (Automated Bug Fixing)*
	- “predefined catalogues of mutation operators” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 16](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=30&annotation=93JASIK4)) #5fb236
	- “we were the first employing pre-trained transformers to mutate source code” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 16](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=30&annotation=4WP6AMU2)) #5fb236
	- “generation of assert statements.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 17](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=31&annotation=YEG3TZP6)) #5fb236
	- “being trained to predict how to generate an appropriate assert statement for the input test method” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 17](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=31&annotation=JECSBY5N)) #5fb236
	- ““pretrain-then-finetune” training strategy helps in substantially boost performance when generating assert statements as compared to the state-of-the-art” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 17](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=31&annotation=IB2EJRV4)) #a28ae5
	- “Method-level Code Summarization” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 17](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=31&annotation=EWNTBJS7)) #a28ae5
	- “extractive” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 17](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=31&annotation=MSVYMWXG)) #a28ae5
	- “abstractive” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 17](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=31&annotation=DLTFZVMQ)) #a28ae5
	- “In our exploration of the role pre-trained models play in facilitating code-related tasks (Chapter 3), we demonstrated that our bi-modal pre-trained technique significantly surpassed the then-current state of the art [HLWM20].” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 19](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=33&annotation=5P34TS9F)) #ffd400  
	  
	  *The same question that I have dove for the other three chapters presenting the contribution in the area.*
	- “y computing their Cyclomatic Complexity and Cognitive Complexity” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 19](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=33&annotation=KNFKMWJW)) #5fb236
	- “this advancement calls for a critical evaluation of these AI-driven recommenders for software engineering tasks. In this context, we explored the aspect of Robustness, particularly how different but semantically equivalent prompts impact the performance of Copilot in code generation (Chapter 4).” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 20](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=34&annotation=9EPYZ7LN)) #ffd400  
	  
	  *What else do you envision?*
	- “Automatic bug-fixing” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 21](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=35&annotation=8DTHI7ZV)) #a28ae5
	- “Injection of code mutants” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 21](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=35&annotation=ZBP8RQZR)) #a28ae5
	- “Generation of assert statements in test methods” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 21](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=35&annotation=KJT8RZNJ)) #a28ae5
	- “Code Summarization” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 21](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=35&annotation=ZJBF5RAD)) #a28ae5
	- “pre-training phase,” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 22](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=36&annotation=YA3SNGUI)) #5fb236
	- “multi-task fine-tuning” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 22](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=36&annotation=ESHMXGGE)) #5fb236
	- “Considering the available computational resources, we decided to use the simplest T5small model.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 23](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=37&annotation=4P52NH59)) #5fb236
	- “masking tokens in natural language sentences and asking the model to guess the masked tokens.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 23](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=37&annotation=TBS4R43Y)) #5fb236
	- “can handle any sort of training instance as long as it can be formulated as a text-to-text transformation” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 25](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=39&annotation=AYRWHDIC)) #5fb236
	- “automate code-related tasks” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 29](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=43&annotation=ML2Q2UY5)) #5fb236
	- “role of pre-training” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 29](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=43&annotation=9XLX6YX7)) #5fb236
	- “role of multi-task learning” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 29](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=43&annotation=AU86R5UP)) #5fb236
	- “able 3.11. Training time (hours) for the trained T5 models” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 41](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=55&annotation=4ZCV5SH2)) #a28ae5
	- “that” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 45](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=59&annotation=NKBP6IL3)) #ff6666  
	  
	  *than*
	- “(ii) did not need the additional ∼22 hours of computation required by the pre-training.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 45](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=59&annotation=XCFEKETD)) #ffd400  
	  
	  *How would you organize a tradeoff analysis?*
	- “Copilot is able to automatically synthesize entire functions just starting from their signature and natural language descriptions.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 47](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=61&annotation=NILLKD2U)) #5fb236
	- “This means that two developers providing different natural language descriptions for the same function they would like to automatically generate could receive two different recommendations.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 47](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=61&annotation=MLFUGU4L)) #5fb236
	- “semantically equivalent natural language descriptions would pose questions on the robustness and usability of DL-based code recommenders.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 47](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=61&annotation=VNAGABN7)) #a28ae5
	- “we generated different paraphrases” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 48](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=62&annotation=6KAEUDJI)) #ffd400  
	  
	  *What are the criteria that have been used to this end?*
	- “four of the authors,” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 48](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=62&annotation=ZZR53PH2)) #ff6666
	- “automated paraphrasing techniques” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 48](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=62&annotation=JE3SSJSK)) #5fb236
	- “1,401 repositories” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 49](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=63&annotation=GQTJNXGF)) #5fb236
	- “projects to use a testing framework and to be compilable” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 49](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=63&annotation=IAPUVINW)) #5fb236
	- “This gives us confidence that the related test cases exercise an acceptable number of behaviors and, therefore, could allow to spot cases in which different generated functions for semantically-equivalent descriptions actually behave differently.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 49](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=63&annotation=ADGN4KA6)) #a28ae5
	- “PEGASUS” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 51](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=65&annotation=BSDBIB7N)) #5fb236
	- “Trans- lation Pivoting (TP)” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 51](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=65&annotation=LVTE2L8T)) #5fb236
	- “TP failed to generate a valid paraphrase (i.e., a sentence different from the original one) in 100 cases (out of 892), while this only happened once with PEGASUS.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 51](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=65&annotation=872ZX7HJ)) #5fb236
	- “For this reason, we developed a toolchain able to automatically invoke Copilot on the subject instances: We exploit the AppleScript language to automate this task on a MacBook Pro, simulating the developer’s interaction with Visual Studio Code (vscode).” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 52](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=66&annotation=7KH6CZRQ)) #a28ae5
	- “preceding and following” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 52](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=66&annotation=AWFDB9MK)) #ffd400  
	  
	  *Have you considered to variate also the size of the code preceding and following the emptied method?*
	- “TLev representing the token-level Levenshtein distance between the two descriptions” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 53](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=67&annotation=MG9EMAN4)) #ffd400  
	  
	  *You wanted to do an analysis based on semantically different or similar sentences.... Why have you selected Levenshtein, which is a "syntactical" metric*
	- “CodeBLEU measures how similar two methods are” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 53](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=67&annotation=JJB9H77V)) #5fb236
	- “State-of-the-art paraphrasing techniques” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 55](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=69&annotation=KWH2A4R7)) #5fb236
	- “high percentage of Java methods (∼73% for the original and ∼72% for the paraphrased description) for which Copilot was not able to synthesize a method passing the related unit tests.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 55](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=69&annotation=JWUM3HM3)) #a28ae5
	- “higher similarity between the compared methods” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 55](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=69&annotation=STRDEPFU)) #a28ae5
	- “Different (but semantically equivalent) natural language descriptions of the same method are likely to result in different code recommendations generated by DL-based code generation models.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 61](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=75&annotation=9DZFDDF3)) #a28ae5
	- “semantically equivalent natural language descriptions” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 62](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=76&annotation=RSQ79WEY)) #ffd400
	- “∼46% of cases semantically equivalent but different method descrip- tions result in different code recommendations” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 62](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=76&annotation=JAAGPLUE)) #a28ae5
	- “Our results highlight the importance of providing a proper code description when asking DL-based recommenders to synthesize code.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 62](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=76&annotation=2I88F2IV)) #ffd400  
	  
	  *A dsl for Prompt Engineering???*
	- “LANCE (Log stAtemeNt reCommEnder)” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 65](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=79&annotation=YQYW49EA)) #a28ae5
	- “LANCE must generate a complete log statement and inject it in the proper location.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 65](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=79&annotation=2RZKC8HE)) #5fb236
	- “assumes that only one log statement is needed in a Java method provided as input.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 65](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=79&annotation=PDG4M7P5)) #5fb236
	- “LANCE cannot assess whether log statements are needed at all.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 65](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=79&annotation=AC2ILZ8H)) #5fb236
	- “1 to n log statements in a given method” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 70](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=84&annotation=LIGRYWP4)) #a28ae5
	- “We used srcML [src] to extracted all Java methods in the selected projects.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 71](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=85&annotation=NDGABEZD)) #5fb236
	- “randomly masking 15% of the tokens composing a training instance (i.e., a Java method) asking the model to predict them” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 72](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=86&annotation=CAHMPNZE)) #ffd400  
	  
	  *This partially answer the comment below. However, the percentage of the masking can be a dimension to investigate, isn't it?*
	- “Figure 6.1. Example of Pre-training instance” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 72](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=86&annotation=65BH7FBX)) #ffd400  
	  
	  *This comment applies to different works, where you have used such a pre-training technique. How have you selected the number of MASKS, and where to locate them in the training data?*
	- “Jaccard similarity” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 74](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=88&annotation=PPSAX9QB)) #ffd400  
	  
	  *This is a syntactic similarity. How to cover semantically similar log messages that are syntactically different?*
	- “For this reason, we also compute the following four metrics used in Natural Language Processing (NLP) for the assessment of automatically generated t” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 80](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=94&annotation=YWRQKBR6)) #ffd400
	- “A New Metric and an Empirical Characterization” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 94](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=108&annotation=S9STPE7B)) #5fb236
	- “ct as a proxy for the quality of au- tomatically generated text (e.g., a translation) by comparing it with a reference (expected) text: The higher the words’ overlap between the generated and the reference text, the higher the assessed quality.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 135](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=149&annotation=4IHY9EQU)) #5fb236
	- “no guarantee that the reference text is of high quality” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 135](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=149&annotation=2E9IJR74)) #a28ae5
	- “Metrics based on word overlap penalize generated summaries for being different but semantically equivalent to the reference one, thus again not being good proxies for the summary quality.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 135](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=149&annotation=6BZFX647)) #a28ae5
	- “a high similarity to a low-quality reference summary may provide a misleading good assessment of a generated summary” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 136](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=150&annotation=KZ68SKNJ)) #5fb236
	- “SIDE” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 136](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=150&annotation=6WMMN8ER)) #a28ae5
	- “viability of utilizing pre-trained deep learning (DL) models for code-related tasks, with a specific focus on tasks requiring the manipulation of bi-modal data, such as code summarization and the generation of log statements” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 157](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=171&annotation=2TUZPE7M)) #a28ae5
	- “Java as the primary programming language” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 157](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=171&annotation=MW7JW52W)) #a28ae5
	- “driving 112 gasoline-powered vehicles a year” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 159](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=173&annotation=VM5A7J5E)) #a28ae5
	- “Despite the substantial environmental impact associated with training these LLMs, the inference process (i.e., prompting) has been projected to become the primary contributors to carbon emissions, as indicated by recent research” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 159](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=173&annotation=PQYHJ5YY)) #a28ae5
	- “This leaves us with the question of how software engineering will change in the near future, and how developers will need to adapt in this context. These are questions that will inform our future research.” ([“antonio-mastropaolo-phd-thesis.pdf”, p. 159](zotero://select/library/items/BD6CUQXU)) ([pdf](zotero://open-pdf/library/items/62B7GTXP?page=173&annotation=L696S3XL)) #e56eee
- ### [[Comments]]
- ### [[REVIEWS/Notes]]
  collapsed:: true
	- #### Abstract Summary:
	  collapsed:: true
		- Antonio Mastropaolo's dissertation investigates the application of deep learning (DL) models, specifically focusing on the "pretrain-then-finetune" paradigm, to automate various code-related tasks. These tasks, often demanding due to their technical nature involving both code and natural language, include bug-fixing, code summarization, and code review. Mastropaolo's research aims to expand the capability of these models to assist developers more effectively by leveraging pre-trained models to understand and manipulate code in conjunction with technical English.
	- #### Research Goals and Methodology:
	  collapsed:: true
		- The primary objective of the thesis is to explore how pre-trained DL models can be effectively utilized to support developers in tasks that involve both programming code and natural language descriptions. The research methodologically embraces the "pretrain-then-finetune" strategy where models are initially pre-trained on vast datasets to learn generic patterns and subsequently fine-tuned for specific tasks such as code summarization and bug-fixing.
	- #### Key Findings:
	  collapsed:: true
		- **Code Summarization:** Mastropaolo introduces novel metrics for evaluating the quality of automatically generated code summaries, addressing the challenge of assessing such summaries objectively.
		- **Log Statement Generation:** The thesis presents a novel approach for the automatic generation and injection of log statements into code, demonstrating the practical utility of DL models in routine programming tasks.
		- **Empirical Study on GitHub Copilot:** An empirical analysis of GitHub Copilot illustrates its capabilities and limitations, particularly its sensitivity to variations in natural language descriptions used to generate code.
	- #### Contributions:
	  collapsed:: true
		- **Theoretical Contribution:** The research contributes to the theoretical understanding of applying the "pretrain-then-finetune" paradigm in the context of code-related tasks, extending its applicability and efficacy.
		- **Practical Tools:** Development of practical tools and metrics that can be directly utilized by software developers and incorporated into existing software development workflows.
		- **Empirical Insights:** Provides empirical insights into the behavior of state-of-the-art tools like GitHub Copilot, enhancing understanding of AI-driven code recommendation systems.
	- #### Significance:
	  collapsed:: true
		- Mastropaolo's work significantly advances the field of software engineering by integrating cutting-edge AI techniques with traditional code development processes. His findings offer robust evidence supporting the efficiency of DL models in automating complex code-related tasks, potentially reducing the cognitive load on developers and improving productivity and code quality.
	- #### Future Work:
	  collapsed:: true
		- The thesis outlines several areas for future research, including the refinement of model training processes, exploration of additional tasks that could benefit from AI support, and further development of tools that can adapt to varying contexts within software development.
	- Chapter summaries
	  collapsed:: true
		- **Chapter 1: Introduction**
		  This chapter introduces the context and motivation behind the research, framing the challenges developers face in code-related tasks due to tight deadlines and complexity. It outlines the potential of pre-trained deep learning models to address these challenges by automating tasks like bug fixing, code summarization, and code review. The thesis statement is presented, asserting that pre-trained models can significantly enhance productivity and accuracy in handling code combined with natural language.
		- **Chapter 2: Background and Related Work**
		  Mastropaolo reviews existing literature and methodologies related to the automation of code-related tasks using deep learning. He discusses the evolution from traditional methods to the current state involving sophisticated DL models, highlighting key technologies like the Transformer architecture and their impact on the field.
		- **Chapter 3: Towards Automating Code-Related Tasks via Pre-trained Models of Code**
		  This chapter details the empirical investigations into the use of DL models for automating code-related tasks. It focuses on the "pretrain-then-finetune" approach, demonstrating its application across several code-related tasks and comparing its effectiveness with traditional methods.
		- **Chapter 4: Evaluating the Robustness of DL-based techniques for Generating Code**
		  Focusing on GitHub Copilot, this chapter presents an empirical study to assess the robustness of DL-based code generation techniques. Mastropaolo examines how minor variations in the wording of prompts affect the generated code, providing insights into the model's sensitivity and potential areas for enhancement.
		- **Chapter 5: Background and Related Work on Log Statement Generation**
		  This chapter sets the stage for discussing automated log statement generation, reviewing prior work and technological foundations that support the automated integration of log statements into code.
		- **Chapter 6: Log Statement Generation via Deep Learning**
		  Mastropaolo introduces a novel DL-based approach for generating and injecting log statements into software, which helps in debugging and maintaining software systems. He evaluates this approach in practical settings to demonstrate its effectiveness and efficiency.
		- **Chapter 7: Background and Related Work on Code Summarization**
		  The chapter reviews existing approaches and challenges in code summarization, setting the groundwork for introducing new techniques that leverage DL to improve summarization accuracy and relevance.
		- **Chapter 8: Towards Summarizing Code Snippets**
		  This chapter presents innovative methods developed to automate the summarization of code snippets using pre-trained models. Mastropaolo introduces new evaluation metrics designed to objectively assess the quality of code summaries.
		- **Chapter 9: Supporting Code Summarization via Comment Completion Techniques**
		  Expanding on code summarization, this chapter explores the use of DL models to complete partial code comments automatically. It addresses the practical aspects of deploying such technologies in real-world development environments.
		- **Chapter 10: A New Metric for Evaluating Code Summarization Techniques**
		  Mastropaolo proposes a novel metric that better aligns with human judgment for evaluating code summarization techniques, a significant advancement over traditional metrics which often fail to capture the semantic accuracy of summaries.
		- **Epilogue: Conclusions and Future Work**
		  The final chapter synthesizes the research findings, discussing the implications for both theory and practice. Mastropaolo outlines potential future research directions, emphasizing the need for further innovation in the integration of AI in software development processes. He also reflects on the broader impacts of his work on the software engineering community and technology development.
-
-
- QUESTIONS
  collapsed:: true
	- **Integration with Existing Development Tools:**
	  collapsed:: true
		- How do the developed approaches integrate with existing software development tools and workflows?
	- **Effectiveness of Pre-trained Models:**
	  collapsed:: true
		- What were the most significant challenges encountered when adapting pre-trained models for specific code-related tasks, and how were they overcome?
	- **Evaluation of Novel Metrics:**
	  collapsed:: true
		- How were the novel metrics evaluated in terms of their effectiveness and reliability in real-world coding scenarios?
	- **Impact of Research on Future Practices:**
	  collapsed:: true
		- Based on the findings of the thesis, what changes do you foresee in software development practices, particularly with regard to the use of AI and machine learning?
	- **Comparative Analysis:**
	  collapsed:: true
		- Could you detail the comparative analysis conducted between the proposed methods and the state-of-the-art techniques? What were the key differentiators?
	- **Robustness of Code Generation Techniques:**
	  collapsed:: true
		- How does the thesis address the robustness of code generation techniques like GitHub Copilot when faced with variable input prompts?
	- **Future Directions in AI-Driven Development:**
	  collapsed:: true
		- What future research directions does Mastropaolo suggest based on the outcomes of his PhD work?
	- **Implications for Software Maintenance:**
	  collapsed:: true
		- How might the advancements in automated log statement generation and code summarization impact long-term software maintenance and debugging?
	- **Training Data and Model Bias:**
	  collapsed:: true
		- How did Mastropaolo address potential biases in training data, and what measures were taken to ensure the generalizability of the models?
	- **Performance Metrics and Evaluation:**
	  collapsed:: true
		- What specific performance metrics were prioritized in evaluating the models, and how do these align with industry needs for software development tools?
-