21 KiB
21 KiB
type:: REVIEWS tags:: year:: 2024 venue:: ICSE full-title:: date-start:: 01-10-2024 - 23:42 date-submitted:: 01-10-2024 external-links:: status:: DONE deadline-submission:: file:: @icse2025-paper797 parent:: todoist:: https://app.todoist.com/app/task/797-curating-reviews-on-vulnerable-code-by-active-learning-6WFCX7PH66w9PjRc
- ### [[Highlights]]
-
- # Annotazioni
- (1/10/2024, 23:07:48)
- - “Detecting vulnerabilities and curating corresponding datasets is important for cybersecurity” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=N3QQMC5V)) #5fb236
- - “vulnerabilities are very rare.” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=2SH5VJF6)) #5fb236
- - “classifier iteratively searches which observations are labeled next, and then repeatedly learns from such labeling feedback” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=5LEEZW2M)) #a28ae5
- - “We contribute novel simulations that evaluate the properties of active learning under well-defined and repeatable assumptions, concluding on performance, threats, and limitations. We contribute a novel empirical study where we evaluate active learning in the setting of curating pull request reviews on vulnerable code. The method behaves as expected from the simulation. We present a corresponding dataset of 234 vulnerability related reviews, while we only need to label 1247 reviews. This is a percentage of 18.7% vulnerabilities during manual labeling.” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=PWSL28VP)) #ffd400
- *The abstract is not effective in presenting the contribution of the paper.*
- - “simulations” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=8YIHR6F5)) #2ea8e5
- - “fails in producing datasets of high quality” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=XVBEZ3IB)) #a28ae5
- - “This affects the classifiers for detection and causes ineffective cybersecurity whenever classifiers are used” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=28VLWSG8)) #a28ae5
- - “A dataset is created (e.g., [5]) on which then a classifier is trained (e.g., [6])” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=I8WTT78A)) #5fb236
- - “one million commits Liu et al. find that 1462 are vulnerable” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=N95NGTQB)) #a28ae5
- - “0.1% chance is frustrating” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=TYR6LQNF)) #a28ae5
- - “selectively label instances for a dataset,” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=J6QQVEK3)) #5fb236
- - “active learning” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=8KDRWG9A)) #a28ae5
- - “We study this active learning loop in the context of curating pull request reviews on vulnerable code” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=NAJCXD2R)) #a28ae5
- - “GitHub and collecting 4,191,892 unlabeled pull request reviews.” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=XAPCWL9N)) #5fb236
- - “We reach a fraction of 18.7% vulnerabilities labels, which is far beyond 0.1%, and acceptable for manual labeling” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=EA48ZZNS)) #ffd400
- *It depends if 18.7% is correctly labeled.*
- - “a fine-tuned language model for sequence classification.” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=R7JKYHDB)) #a28ae5
- - “transfers active learning to vulnerability detection.” ([“icse2025-paper797”, p. 1](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=1&annotation=DHV6GWXB)) #5fb236
- - “evaluating the method in a simulation and empirical study” ([“icse2025-paper797”, p. 2](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=2&annotation=FRKYWMMM)) #5fb236
- - “Sec. III starts with the active learning method transferred to curate a dataset of pull request reviews on vulnerable code.” ([“icse2025-paper797”, p. 2](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=2&annotation=R2EXA385)) #ffd400
- *Why you start section II by referring Section III?*
- - “We then evaluate the method in a simulation study (in Sec. IV) and in an empirical study (in Sec. V). For a methodological underpinning of such hybrid design, we refer to [17] and [18]. We expect both studies to strongly correspond, but they are orthogonal in their conclusions:” ([“icse2025-paper797”, p. 2](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=2&annotation=BVD43N2Z)) #ffd400
- *I don't understand why you describe the structure of the next sections at the beginning of the study design section.*
- - “The simulation study stresses active learning under transparent and repeatable assumptions on a simulated scenario in which the method is applied. We simulate different scenarios, varying the predictability of the labels, or the correlation in the data. We repeat the simulation until confidence intervals sufficiently converged to rule out randomness in our conclusions entirely. The simulation is highly parametric. We do not examine all parameters, only those we consider most important. We stress the method to its limit, to detect where it does not work, and what might be a potential threat. We also use it to test our code. We avoid an additional threats and limitation sections in this paper because the simulation study is basically this. Every piece of the simulation is operational and deployed online. It is fully reproducible” ([“icse2025-paper797”, p. 2](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=2&annotation=F242EKWB)) #ffd400
- *This is vaguous and generic. Any context is given to help the reader understand.*
- - “We expect the method to behave as in the simulation. This is the case. We interpret minor differences in this section and show that the method can curate a real dataset successfully. We also show a more realistic classifier by fine-tuning an existing language model. The empirical study is only reproducible to a certain degree because GitHub is a moving target. For readers that search for confidence intervals or p-values in the empirical study: We don’t have independent measurements of properties of the method because the method is not repeatable in the same way as the simulation. Reporting on this would be fraud.” ([“icse2025-paper797”, p. 2](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=2&annotation=IF2XHSWL)) #ffd400
- *See my previous point.*
- - “Active learning is an umbrella term for methods that learn a model (or classifier) iteratively by a (human) feedback mechanism” ([“icse2025-paper797”, p. 2](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=2&annotation=GM5U69BY)) #a28ae5
- - “pull request reviews discuss vulnerable code” ([“icse2025-paper797”, p. 2](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=2&annotation=LJQIKHG3)) #a28ae5
- - “The label if a review is about vulnerable code is the expensive feature that we miss” ([“icse2025-paper797”, p. 2](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=2&annotation=L8VEAZKQ)) #a28ae5
- - “used to train the classifier.” ([“icse2025-paper797”, p. 2](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=2&annotation=UGZNHH6J)) #a28ae5
- - “The classifier selects the candidates RC,i as a subset of those with unknown labels from the previous iteration RU,i−1.” ([“icse2025-paper797”, p. 2](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=2&annotation=8JEUR5L4)) #a28ae5
- - “The particular model type used depends on the problem. The upcoming simulations will use a basic feed-forward neural network for binary classification.” ([“icse2025-paper797”, p. 2](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=2&annotation=PTZ9L73K)) #5fb236
- - “We assume an imbalance in label types” ([“icse2025-paper797”, p. 2](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=2&annotation=WZJVYJEA)) #5fb236
- - “This will later be vulnerability related (rare-label) vs. non-vulnerability related (majority-label).” ([“icse2025-paper797”, p. 2](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=2&annotation=CHYISSTP)) #2ea8e5
- - “elect those with the highest entropy” ([“icse2025-paper797”, p. 3](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=3&annotation=PPRTQXMP)) #ffd400
- *I would remove section 2 or revise it with the aim of gain some space and for instance add explanatory examples to discuss the different steps of the process and give some evidence about relevant concepts, like entropy in the context of this work.*
- - “classifier is most certain” ([“icse2025-paper797”, p. 3](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=3&annotation=Y28Q6Q8L)) #ffd400
- *How do you assess that? Do you assign a probability to the outputs of the classifier? This is not clear.*
- - “We borrow a classifier to set the candidates of the first iteration. We use this in our empirical study.” ([“icse2025-paper797”, p. 3](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=3&annotation=MUN43E7N)) #ffd400
- *This is not clear. What are the characteristics of the considered classifier? Can you be more precise on this?*
- - “The simulation study examines properties of our method under well-defined and repeatable assumptions. The section answers RQ3.” ([“icse2025-paper797”, p. 3](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=3&annotation=FAGJSY8L)) #ffd400
- *This is a presentation issue. Do you start with answering RQ3?*
- - “We now have synthetic data for our input xs and the corresponding labels ys. Since we simulate, we know both. We will now stress active learning methods that try to recover many rare lables in ys without having exhaustive access to ys. We repeat running the simulation to rule out randomness entirely.” ([“icse2025-paper797”, p. 4](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=4&annotation=IZQJ88TZ)) #ffd400
- *The section is not properly connected with the problem under investigation. It is not clear how the presented code and discussion is linked to the problem of detecting pull request reviews that discuss vulnerable code.*
- - “Additional Simulation Assumptions There are more assumptions where we omit a discussion based on code. The code can be found in the online material. We present the big picture of such assumptions, itemized here.” ([“icse2025-paper797”, p. 4](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=4&annotation=7VP625YQ)) #ffd400
- *Same problem, see my previous comment. It is not easy to understand the context of the given esentences. COncepts are introduced without any context. For instance "To produce imbalance, we modify the last bias of the network.", which network are the authors referring to here? Many sentences are like this one.*
- - “However, the simulation just makes a statement that overfitting matters. Finding a good configuration may differ from case to case. Cross-validation may help. In our empirical study, we used cross-validation. We assume that also other methods to prevent overfitting will work, like regularization, dropout, or reduction of the model size.” ([“icse2025-paper797”, p. 6](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=6&annotation=FTDJJ92D)) #ffd400
- *Again this just another example of paragraphs that do not convey any specific content.*
- - “4,191,892 pull request reviews, from 4,937 distinct repositories, and 492,266 distinct pull requests.” ([“icse2025-paper797”, p. 6](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=6&annotation=FZMV5ZCC)) #5fb236
- - “We also label if bots are writing a review by Bot.” ([“icse2025-paper797”, p. 7](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=7&annotation=HQRHAQK9)) #ffd400
- *WHat does it mean?*
- - “while we consider the Vuldev classification as an interesting but premature artifact. We plan to follow up on the Vuldev dataset in future work” ([“icse2025-paper797”, p. 7](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=7&annotation=MCHUFFDV)) #ffd400
- *THis is a limitation of the paper, whereas it is supposed to be one of the novel aspects of the proposed approach.*
- - “We are not interested in an exhaustive enumeration of keywords here, since this is the task for the following active learning loop.” ([“icse2025-paper797”, p. 7](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=7&annotation=2JRKJPKT)) #5fb236
- - “2) Active Learning Classifier: We rely on transfer learning by fine-tuning an existing language model for sequence classification (Roberta-base [24]). The input of the model is the tokenized text of the pull request (up to 64 tokens). The output of the model is the category that we manually assigned during labeling.” ([“icse2025-paper797”, p. 7](zotero://select/library/items/C69B6BRI)) ([pdf](zotero://open-pdf/library/items/5ECWC5VA?page=7&annotation=QBUNIAQ7)) #ffd400
- *By looking at this paragraph it is not clear if Roberta is used as it is for the classification or how it is used for the active learning process.*
- ### [[Comments]]
- #.tabular
- - ### Paper summary
- The paper presents an approach to address the problem of detecting vulnerabilities in code repositories by using active learning techniques to curate a vulnerability dataset from pull request reviews. The paper introduces a study to evaluate the properties of active learning under controlled assumptions and then transitions to an empirical study where the method is applied to a GitHub dataset of pull request reviews.
- - ### Strengths
- The use of active learning to address the problem of vulnerability detection is interesting.
- The dataset produced (Vuldev) can be a valuable contribution to future research in vulnerability detection.
- - ### Weaknesses
-
- The abstract does not effectively communicate the core contributions of the paper and the potential impact of the proposed method on the field.
- Several sections of the paper are difficult to follow, particularly the description of the simulation study and its connection to the real-world problem.
- - ### Detailed comments for authors
-
- Novelty: The application of active learning to vulnerability detection in pull request reviews is a novel idea. However, the novelty could be better emphasized in the early sections of the paper to help readers quickly grasp what sets this work apart from previous studies.
- Rigor: The paper presents its dual study design (simulation and empirical). However, more context is needed in some parts of the methodology, particularly the simulation. For instance, the assumptions behind the choice of parameters and the structure of the neural networks are not always clearly justified. Furthermore, the paper lacks a thorough discussion of potential biases or weaknesses in the labeled data and the classifier used.
- Relevance: The work is highly relevant to the field of vulnerability detection, especially given the increasing importance of automation and machine learning in cybersecurity. The use of real-world data from GitHub makes this work potentially applicable to real-world scenarios. However, improvements in the clarity of the method's application need to be operated.
- Verifiability & Transparency: The authors have provided online resources for the simulation code and results.
- Presentation: It is the main problem of the paper. There are several structural issues (e.g., references to later sections early in the paper) that disrupt the flow. Additionally, some terminology needs to be adequately explained, particularly in the simulation study. More effort could be made to guide the reader through the technical details and to link the sections more coherently. Visual aids or explanatory examples are needed.
- #### Detailed Comments
- The abstract should more clearly convey the contributions of the paper.
- The section introducing simulations does not explain the connection between the simulated and empirical scenarios. A deeper discussion of how the simulation results translate to the real-world problem of vulnerability detection would be beneficial.
- The discussion on entropy and rare-label sampling is potentially interesting, but the explanation of why certain sampling methods work better than others could be expanded.
- There is a presentation issue in describing the study structure too early in the study design section. This could be confusing for readers who are not yet familiar with the method.
- The sentence "To produce imbalance, we modify the last bias of the network" lacks sufficient context. It is unclear what network the authors are referring to; more explanation is needed here.
- The discussion on overfitting and model tuning is vague. More specifics on how overfitting was mitigated in the empirical study would enhance the clarity.
- The introduction of Vuldev as a "premature" dataset could be reframed to highlight its potential and limitations while still emphasizing its current utility.
- I recommend either removing Section 2 or revising it to make better use of space. For example, the section could be shortened and improved by adding explanatory examples that clarify the different steps of the process. Additionally, providing concrete evidence and detailed explanations of key concepts, such as entropy, in the context of this work would help readers better understand the methodology.
- Page 3 - "...We select those instances where the classifier is most certain to be of the majority-label...": How do you assess this? It's like you are assigning probabilities to classifier outputs, but indeed this is not the case.
- End of Section IVB: The last paragraph is not properly connected with the problem under investigation. It is not clear how the presented code and discussion are linked to the problem of detecting pull request reviews that discuss vulnerable code. There are many paragraphs in the paper affected by the same problem.
- Page 7 - "Active learning classifier": By looking at this paragraph, it is not clear if Roberta is used as it is for the classification or how it is used for the active learning process.
- To summarize, while the paper is about a relevant problem in software engineering, clarity and presentation improvements are needed to enhance its quality.
- ### [[REVIEWS/Notes]]
- ### YELLOW CONCERNS
background-color:: yellow
- {{query (and [[ffd400]] [[icse2025-paper797]] )}}
collapsed:: true
- ### ❓️Questions
- {{query (and [[question]] [[icse2025-paper797]] )[[question]]}}
query-table:: true
query-properties:: [:block]