logseq/pages/@icse2025-paper797.md

tags:: #zotero title:: @icse2025-paper797 item-type:: document original-title:: icse2025-paper797 links:: Local library, Web library

• Attachments

  • PDF {{zotero-imported-file 5ECWC5VA, "icse2025-paper797.pdf"}}

• Notes

  • Annotazioni

    (1/10/2024, 23:07:48)

    • “Detecting vulnerabilities and curating corresponding datasets is important for cybersecurity” (“icse2025-paper797”, p. 1) #5fb236

    • “vulnerabilities are very rare.” (“icse2025-paper797”, p. 1) #5fb236

    • “classifier iteratively searches which observations are labeled next, and then repeatedly learns from such labeling feedback” (“icse2025-paper797”, p. 1) #a28ae5

    • “We contribute novel simulations that evaluate the properties of active learning under well-defined and repeatable assumptions, concluding on performance, threats, and limitations. We contribute a novel empirical study where we evaluate active learning in the setting of curating pull request reviews on vulnerable code. The method behaves as expected from the simulation. We present a corresponding dataset of 234 vulnerability related reviews, while we only need to label 1247 reviews. This is a percentage of 18.7% vulnerabilities during manual labeling.” (“icse2025-paper797”, p. 1) #ffd400 The abstract is not effective in presenting the contribution of the paper.

    • “simulations” (“icse2025-paper797”, p. 1) #2ea8e5

    • “fails in producing datasets of high quality” (“icse2025-paper797”, p. 1) #a28ae5

    • “This affects the classifiers for detection and causes ineffective cybersecurity whenever classifiers are used” (“icse2025-paper797”, p. 1) #a28ae5

    • “A dataset is created (e.g., [5]) on which then a classifier is trained (e.g., [6])” (“icse2025-paper797”, p. 1) #5fb236

    • “one million commits Liu et al. find that 1462 are vulnerable” (“icse2025-paper797”, p. 1) #a28ae5

    • “0.1% chance is frustrating” (“icse2025-paper797”, p. 1) #a28ae5

    • “selectively label instances for a dataset,” (“icse2025-paper797”, p. 1) #5fb236

    • “active learning” (“icse2025-paper797”, p. 1) #a28ae5

    • “We study this active learning loop in the context of curating pull request reviews on vulnerable code” (“icse2025-paper797”, p. 1) #a28ae5

    • “GitHub and collecting 4,191,892 unlabeled pull request reviews.” (“icse2025-paper797”, p. 1) #5fb236

    • “We reach a fraction of 18.7% vulnerabilities labels, which is far beyond 0.1%, and acceptable for manual labeling” (“icse2025-paper797”, p. 1) #ffd400 It depends if 18.7% is correctly labeled.

    • “a fine-tuned language model for sequence classification.” (“icse2025-paper797”, p. 1) #a28ae5

    • “transfers active learning to vulnerability detection.” (“icse2025-paper797”, p. 1) #5fb236

    • “evaluating the method in a simulation and empirical study” (“icse2025-paper797”, p. 2) #5fb236

    • “Sec. III starts with the active learning method transferred to curate a dataset of pull request reviews on vulnerable code.” (“icse2025-paper797”, p. 2) #ffd400 Why you start section II by referring Section III?

    • “We then evaluate the method in a simulation study (in Sec. IV) and in an empirical study (in Sec. V). For a methodological underpinning of such hybrid design, we refer to [17] and [18]. We expect both studies to strongly correspond, but they are orthogonal in their conclusions:” (“icse2025-paper797”, p. 2) #ffd400 I don't understand why you describe the structure of the next sections at the beginning of the study design section.

    • “The simulation study stresses active learning under transparent and repeatable assumptions on a simulated scenario in which the method is applied. We simulate different scenarios, varying the predictability of the labels, or the correlation in the data. We repeat the simulation until confidence intervals sufficiently converged to rule out randomness in our conclusions entirely. The simulation is highly parametric. We do not examine all parameters, only those we consider most important. We stress the method to its limit, to detect where it does not work, and what might be a potential threat. We also use it to test our code. We avoid an additional threats and limitation sections in this paper because the simulation study is basically this. Every piece of the simulation is operational and deployed online. It is fully reproducible” (“icse2025-paper797”, p. 2) #ffd400 This is vaguous and generic. Any context is given to help the reader understand.

    • “We expect the method to behave as in the simulation. This is the case. We interpret minor differences in this section and show that the method can curate a real dataset successfully. We also show a more realistic classifier by fine-tuning an existing language model. The empirical study is only reproducible to a certain degree because GitHub is a moving target. For readers that search for confidence intervals or p-values in the empirical study: We don’t have independent measurements of properties of the method because the method is not repeatable in the same way as the simulation. Reporting on this would be fraud.” (“icse2025-paper797”, p. 2) #ffd400 See my previous point.

    • “Active learning is an umbrella term for methods that learn a model (or classifier) iteratively by a (human) feedback mechanism” (“icse2025-paper797”, p. 2) #a28ae5

    • “pull request reviews discuss vulnerable code” (“icse2025-paper797”, p. 2) #a28ae5

    • “The label if a review is about vulnerable code is the expensive feature that we miss” (“icse2025-paper797”, p. 2) #a28ae5

    • “used to train the classifier.” (“icse2025-paper797”, p. 2) #a28ae5

    • “The classifier selects the candidates RC,i as a subset of those with unknown labels from the previous iteration RU,i−1.” (“icse2025-paper797”, p. 2) #a28ae5

    • “The particular model type used depends on the problem. The upcoming simulations will use a basic feed-forward neural network for binary classification.” (“icse2025-paper797”, p. 2) #5fb236

    • “We assume an imbalance in label types” (“icse2025-paper797”, p. 2) #5fb236

    • “This will later be vulnerability related (rare-label) vs. non-vulnerability related (majority-label).” (“icse2025-paper797”, p. 2) #2ea8e5

    • “elect those with the highest entropy” (“icse2025-paper797”, p. 3) #ffd400 I would remove section 2 or revise it with the aim of gain some space and for instance add explanatory examples to discuss the different steps of the process and give some evidence about relevant concepts, like entropy in the context of this work.

    • “classifier is most certain” (“icse2025-paper797”, p. 3) #ffd400 How do you assess that? Do you assign a probability to the outputs of the classifier? This is not clear.

    • “We borrow a classifier to set the candidates of the first iteration. We use this in our empirical study.” (“icse2025-paper797”, p. 3) #ffd400 This is not clear. What are the characteristics of the considered classifier? Can you be more precise on this?

    • “The simulation study examines properties of our method under well-defined and repeatable assumptions. The section answers RQ3.” (“icse2025-paper797”, p. 3) #ffd400 This is a presentation issue. Do you start with answering RQ3?

    • “We now have synthetic data for our input xs and the corresponding labels ys. Since we simulate, we know both. We will now stress active learning methods that try to recover many rare lables in ys without having exhaustive access to ys. We repeat running the simulation to rule out randomness entirely.” (“icse2025-paper797”, p. 4) #ffd400 The section is not properly connected with the problem under investigation. It is not clear how the presented code and discussion is linked to the problem of detecting pull request reviews that discuss vulnerable code.

    • “Additional Simulation Assumptions There are more assumptions where we omit a discussion based on code. The code can be found in the online material. We present the big picture of such assumptions, itemized here.” (“icse2025-paper797”, p. 4) #ffd400 Same problem, see my previous comment. It is not easy to understand the context of the given esentences. COncepts are introduced without any context. For instance "To produce imbalance, we modify the last bias of the network.", which network are the authors referring to here? Many sentences are like this one.

    • “However, the simulation just makes a statement that overfitting matters. Finding a good configuration may differ from case to case. Cross-validation may help. In our empirical study, we used cross-validation. We assume that also other methods to prevent overfitting will work, like regularization, dropout, or reduction of the model size.” (“icse2025-paper797”, p. 6) #ffd400 Again this just another example of paragraphs that do not convey any specific content.

    • “4,191,892 pull request reviews, from 4,937 distinct repositories, and 492,266 distinct pull requests.” (“icse2025-paper797”, p. 6) #5fb236

    • “We also label if bots are writing a review by Bot.” (“icse2025-paper797”, p. 7) #ffd400 WHat does it mean?

    • “while we consider the Vuldev classification as an interesting but premature artifact. We plan to follow up on the Vuldev dataset in future work” (“icse2025-paper797”, p. 7) #ffd400 THis is a limitation of the paper, whereas it is supposed to be one of the novel aspects of the proposed approach.

    • “We are not interested in an exhaustive enumeration of keywords here, since this is the task for the following active learning loop.” (“icse2025-paper797”, p. 7) #5fb236

    • “2) Active Learning Classifier: We rely on transfer learning by fine-tuning an existing language model for sequence classification (Roberta-base [24]). The input of the model is the tokenized text of the pull request (up to 64 tokens). The output of the model is the category that we manually assigned during labeling.” (“icse2025-paper797”, p. 7) #ffd400 By looking at this paragraph it is not clear if Roberta is used as it is for the classification or how it is used for the active learning process.