logseq/pages/COLA-D-24-00156.md

type:: REVIEWS tags:: year:: 2025 venue:: COLA full-title:: Impact of Context-based Word Embedding Techniques on Software Defect Prediction date-start:: 06-03-2025 - 14:45 date-submitted:: external-links:: status:: DONE deadline-submission:: file:: @COLA-D-24-00156.PDF parent:: todoist:: https://app.todoist.com/app/task/cola-d-24-00156-6Wxx76rWx3QQ6r88

- ### [[Highlights]]
	- *# Annotazioni  *
	  
	  (6/3/2025, 14:47:58)
	- “Context-based Word Embedding” ([“COLA-D-24-00156.PDF”, p. 3](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=3&annotation=UK26CKVH)) *#5fb236  *
	- “Software Defect Prediction” ([“COLA-D-24-00156.PDF”, p. 3](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=3&annotation=5DWKERNP)) *#5fb236  *
	- “software defect prediction” ([“COLA-D-24-00156.PDF”, p. 3](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=3&annotation=QR9QVQJJ)) *#5fb236  *
	- “PROMISE” ([“COLA-D-24-00156.PDF”, p. 3](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=3&annotation=ZPESDAWN)) *#2ea8e5  *
	- “5 deep learning models” ([“COLA-D-24-00156.PDF”, p. 3](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=3&annotation=H9AN3DFR)) *#5fb236  *
	- “eighteen out of twenty-five scenarios” ([“COLA-D-24-00156.PDF”, p. 3](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=3&annotation=VAUG27EP)) *#5fb236  *
	- “seven out of twenty-five cases” ([“COLA-D-24-00156.PDF”, p. 3](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=3&annotation=JWK3XRNM)) *#5fb236  *
	- “The supremacy of RoBERTa is accredited to its large training data, advanced pre-training strategies, and distinctive architecture.” ([“COLA-D-24-00156.PDF”, p. 3](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=3&annotation=HPGS82AM)) *#a28ae5  *
	- “SDP has emerged as a valuable technique, alleviating the burden on developers by early detection and prevention of defects.” ([“COLA-D-24-00156.PDF”, p. 3](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=3&annotation=IL5GIN9S)) *#5fb236  *
	- “A software defect or a software bug denotes a mistake or defect in a software program that affects it such that it acts in an unforeseen or undesirable manner” ([“COLA-D-24-00156.PDF”, p. 4](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=4&annotation=F9VWJWZB)) *#5fb236  *
	- “SDP is the progression which utilizes data mining and machine learning methods to spot possible software defects before they occur.” ([“COLA-D-24-00156.PDF”, p. 4](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=4&annotation=MCKSCPEE)) *#a28ae5  *
	- “Predictive models are constructed by analyzing historical data on software deficiencies (Khalid et al., 2023) like bug reports, code changes, and testing results to detect distinct behaviors and patterns that would suggest future defects.” ([“COLA-D-24-00156.PDF”, p. 4](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=4&annotation=4NVTF2NN)) *#5fb236  *
	- “SDP is a technique that aims to identify and fix defects early in the development process so that they don’t lead to delays.” ([“COLA-D-24-00156.PDF”, p. 4](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=4&annotation=TB4THYSB)) *#5fb236  *
	- “SDP attempts to use data mining and machine learning methods to predict possible defects well before they happen.” ([“COLA-D-24-00156.PDF”, p. 4](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=4&annotation=CIB6RQY4)) *#ffd400  *
	  
	  *Quite repetitive*
	- “It could be used to prioritize testing efforts and allocate resources toward improving the quality of the entire software” ([“COLA-D-24-00156.PDF”, p. 4](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=4&annotation=Y7MWVXIV)) *#a28ae5  *
	- “extracting semantic information using ASTs” ([“COLA-D-24-00156.PDF”, p. 4](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=4&annotation=249R78R2)) *#2ea8e5  *
	- “word embedding technique” ([“COLA-D-24-00156.PDF”, p. 4](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=4&annotation=8EEABZIU)) *#2ea8e5  *
	- “deep-learning classifier that predicts possible defects within the software” ([“COLA-D-24-00156.PDF”, p. 4](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=4&annotation=HDAEQK39)) *#2ea8e5  *
	- “as for the semantic features inherent in the coding, it became apparent that static code metrics had a drawback.” ([“COLA-D-24-00156.PDF”, p. 4](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=4&annotation=JXLX52US)) *#5fb236  *
	- “Javalang is used to parse Java source code and generate ASTs in this paper.” ([“COLA-D-24-00156.PDF”, p. 4](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=4&annotation=B6UB3EZR)) *#ffd400  *
	  
	  *Is it necessary to give this details at this stage when it is not clear yet what the contribution of this paper is?*
	- “SDP were based on static code metrics to evaluate code quality. These metrics cannot capture semantic aspects of the code to predict defects accurately.” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=HFVIDV8U)) *#5fb236  *
	- “limitations have been further increased by recent improvements in software development process.” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=9VMFCK73)) *#5fb236  *
	- “(” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=MT5NTJML)) *#ff6666  *
	- “understand code semantics” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=QVT3PUVQ)) *#a28ae5  *
	- “Bidirectional Encoder Representations from Transformers (BERT)” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=6P7UV25C)) *#2ea8e5  *
	- “Code Bidirectional Encoder Representations from Transformers (CodeBERT)” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=GCDM33VH)) *#2ea8e5  *
	- “Robustly optimized Bidirectional Encoder Representations from Transformers approach (RoBERTa)” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=X9SLVVY7)) *#2ea8e5  *
	- “XLNet” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=UWMN76U5)) *#2ea8e5  *
	- “identifying the optimal word embedding technique for SDP” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=FTDD7K4G)) *#a28ae5  *
	- “The lack of comparative analysis of these techniques for SDP presents a challenge for researchers in determining the most effective method.” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=57Y67G4X)) *#a28ae5  *
	- “This research study aims to bridge this gap through comparative analysis of BERT, CodeBERT, RoBERTa and XLNet to find the best word embedding technique for SDP based on their relative performance. The study will help to identify more accurate and efficient defect prediction models by evaluating relative performance and applicability of these techniques.” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=AH9AZDV4)) *#e56eee  *
	  
	  *That's important and represents the goal of this paper.*
	- “fine-tune particular tasks or datasets” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=CT7HHRIP)) #5fb236
	- “The motivation behind this study is to investigate which advanced word embedding techniques can be considered optimal for SDP.” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=GWETGYLN)) #a28ae5
	- “Perform a comparative analysis” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=LFIFWW5Q)) #2ea8e5
	- “Identify the best-performing word embedding technique for SDP,” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=KQRSHI3G)) #2ea8e5
	- “Provide statistical support for the determination of the superior context-based word embedding technique among those evaluated.” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=88LNUJ8V)) #2ea8e5
	- “selected word embedding techniques” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=BGLRM3KE)) #2ea8e5
	- “overview of related work” ([“COLA-D-24-00156.PDF”, p. 5](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=5&annotation=WMBIFDXJ)) #2ea8e5
	- “three methodological phases” ([“COLA-D-24-00156.PDF”, p. 6](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=6&annotation=8WBV3PZB)) #2ea8e5
	- “Word embeddings techniques are used to represent words as numerical vectors in NL” ([“COLA-D-24-00156.PDF”, p. 6](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=6&annotation=ULDTNTZB)) #5fb236
	- “two categories of word embeddings” ([“COLA-D-24-00156.PDF”, p. 6](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=6&annotation=74TZD8AF)) #2ea8e5
	- “non-context-based” ([“COLA-D-24-00156.PDF”, p. 6](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=6&annotation=C4KTNEXR)) #5fb236
	- “context-based.” ([“COLA-D-24-00156.PDF”, p. 6](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=6&annotation=DBC67QIB)) #5fb236
	- “context-based techniques dynamically represent words by incorporating the surrounding context in which they are used” ([“COLA-D-24-00156.PDF”, p. 6](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=6&annotation=L65F9C4K)) #5fb236
	- “BERT (Devlin et al., 2019) is trained on a massive dataset of 3.3 billion words, including Wikipedia and BooksCorpus from Google.” ([“COLA-D-24-00156.PDF”, p. 6](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=6&annotation=7VWDY9MS)) #5fb236
	- “This specialized model undergoes training on large-scale code corpora, aiming to capture contextual information and semantic relationships within programming languages. CodeBERT exhibits a similar model architecture to BERT, making it adept at understanding both programming languages (PL) and natural language (NL).” ([“COLA-D-24-00156.PDF”, p. 7](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=7&annotation=E4EBN28D)) #5fb236
	- “The optimization strategy adopted for RoBERTa involves Adam, employing specific parameters such as β1 = 0.9, β2 = 0.999, ɛ = 0.000001, and using an L2 weight decay of 0.01. Noteworthy adjustments in the training of RoBERTa include the learning rate warming up over the initial 10,000 steps to a maximum value of 0.0001, followed by a linear decay. Key modifications during the training of RoBERTa involve gradually increasing the learning rate over the initial 10,000 steps to reach a maximum value of 0.0001, followed by linear decay. Additionally, RoBERTa incorporates a dropout rate of 0.1 on all layers and attention weights, along with a GELU activation function. The pretraining phase consists of 1,000,000 updates, with minibatches containing 256 sequences, each having a” ([“COLA-D-24-00156.PDF”, p. 7](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=7&annotation=K6QSMG4C)) #ffd400  
	  
	  *Are these details relevant for the sake of this work here?*
	- “maximizing the expected value of the sum of the log probabilities of predicting each token in the sequence, given the tokens that precede it, over all possible permutations of the token sequence” ([“COLA-D-24-00156.PDF”, p. 8](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=8&annotation=5JIQ8UD9)) #5fb236
	- “The exploration of effective techniques in this domain has been a persistent area of interest within the research community. Some common ways explored by the researchers are mentioned in this section.” ([“COLA-D-24-00156.PDF”, p. 8](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=8&annotation=RYB9NZX5)) #5fb236
	- “Tested on ten open-source projects, this model achieved superior F1-scores compared to state-of-the-art models.” ([“COLA-D-24-00156.PDF”, p. 8](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=8&annotation=SMDPJA3C)) #5fb236
	- “Softmax neural network” ([“COLA-D-24-00156.PDF”, p. 8](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=8&annotation=K2RXYXPU)) #5fb236
	- “Defect Prediction via Self-Attention mechanism (DPSAM)” ([“COLA-D-24-00156.PDF”, p. 8](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=8&annotation=PEGJ2AG9)) #5fb236
	- “Cross-Project Defect Prediction (CPDP)” ([“COLA-D-24-00156.PDF”, p. 8](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=8&annotation=Q3STA9XZ)) #e56eee
	- “Within-Project Defect Prediction (WPDP)” ([“COLA-D-24-00156.PDF”, p. 8](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=8&annotation=2WSJ5UMM)) #a28ae5
	- “comprehensive comparative analysis of context-based word embeddings.” ([“COLA-D-24-00156.PDF”, p. 9](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=9&annotation=9L7UNZWR)) #a28ae5
	- “Addressing this gap in the research landscape holds significant importance as it can potentially guide practitioners and researchers towards selecting the most suitable context-based word embeddings for improved accuracy and generalizability in SDP.” ([“COLA-D-24-00156.PDF”, p. 10](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=10&annotation=Z4SPESRE)) #5fb236
	- “extracting word tokens from AST” ([“COLA-D-24-00156.PDF”, p. 10](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=10&annotation=C5I5NUNT)) #5fb236
	- “The methodology begins with extracting semantic features from designated open-source Java projects using javalang (https://pypi.org/project/javalang), an open-source Python tool for parsing Java source code. This tool enables the conversion of code into AST format, focusing on three categories of AST nodes: controlflow type, declaration type, and method invocation type as determined by existing studies” ([“COLA-D-24-00156.PDF”, p. 10](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=10&annotation=EXXA823Z)) #5fb236
	- “This selective approach was intended to mitigate potential noise introduced by specific AST node types, such as intrinsic type declarations.” ([“COLA-D-24-00156.PDF”, p. 10](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=10&annotation=UQFZ85E2)) #5fb236
	- “The application of multiple models (Xu et al., 2021) enables a comprehensive evaluation of the embedding techniques' effectiveness within the realm of defect prediction.” ([“COLA-D-24-00156.PDF”, p. 12](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=12&annotation=TVUBGFFD)) *#5fb236  *
	- “Attention Mechanism (ANN-AM)” ([“COLA-D-24-00156.PDF”, p. 12](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=12&annotation=MJEA6VVB)) *#5fb236  *
	- “Self-attention was chosen because it allows the model to dynamically weigh the importance of different parts of the input sequence, capturing long-range dependencies effectively” ([“COLA-D-24-00156.PDF”, p. 12](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=12&annotation=V3M739QQ)) *#5fb236  *
	- “softmax activation” ([“COLA-D-24-00156.PDF”, p. 12](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=12&annotation=QLRRSEJ2)) *#5fb236  *
	- “WPDP” ([“COLA-D-24-00156.PDF”, p. 12](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=12&annotation=E3ICV2F4)) *#5fb236  *
	- “This section provides a thorough examination of the experimental setting used to assess the performance of various word embedding algorithms for defect prediction.” ([“COLA-D-24-00156.PDF”, p. 12](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=12&annotation=8AAEJYGC)) *#5fb236  *
	- “It covers the dataset description, outlines the proposed methodology, and details the evaluation metrics.” ([“COLA-D-24-00156.PDF”, p. 12](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=12&annotation=2QL8QJ2X)) *#a28ae5  *
	- “y Sp” ([“COLA-D-24-00156.PDF”, p. 12](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=12&annotation=RHKKB3WX)) *#ff6666  *
	- “The second and third models utilize an LSTM layer and a GRU layer, respectively. Operating on input sequences with a length of 768, both models process the data through their respective recurrent layers and generate predictions via a fully connected layer.” ([“COLA-D-24-00156.PDF”, p. 14](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=14&annotation=3A7ICBVE)) *#5fb236  *
	- “The output size is set to 2, corresponding to the binary classification task. The BiLSTM model (Uddin et al., 2022) follows a similar structure, with the LSTM layer replaced by a Bidirectional LSTM layer. The size of the fully connected layer is doubled (200) to accommodate bidirectionality” ([“COLA-D-24-00156.PDF”, p. 14](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=14&annotation=A2JK4IJ7)) #5fb236
	- “The paper concludes that RoBERTa is the most effective word embedding technique in predicting software defects, outperforming others in accuracy” ([“COLA-D-24-00156.PDF”, p. 19](zotero://select/library/items/VLJ6RUW2)) ([pdf](zotero://open-pdf/library/items/BT72DCNF?page=19&annotation=L4A7IC7Y)) #5fb236
- ### [[Comments]]
	- SUMMARY: The paper analyzes the effects of different word embedding techniques on the prediction of software defects. In particular, four word embedding techniques, namely BERT, CodeBERT, RoBERTa, XLNet have been used while employing 5 different deep-learning models. The dataset used for the experiments includes eight well-known projects. According to the performed experiments, RoBERTa is the most effective word embedding technique to predict software defects with respect to different metrics, including accuracy.
	- COMMENTS: The paper is about an interesting topic. Overall, it is well-written and structured. I liked the paper. There are some typos to be fixed, and I would like to see a dedicated section on the implications of this study. The authors could discuss how both practitioners and researchers might benefit from the outcomes of the paper. Another issue that must be fixed is the lack of a proper replication package. This is necessary to make the presented experiments reproducible and also to foster further research on this topic.
	-