logseq/pages/SANER2024_paper_11.md

type:: REVIEWS tags:: year:: 2024 venue:: SANER full-title:: ALANCA: Active Learning Guided Adversarial Attacks for Code Comprehension on Diverse Pre-trained and Large Language Models date-start:: 15-11-2023 - 22:37 date-submitted:: 10-12-2023 external-links:: status:: DONE deadline-submission:: 09-12-2023 file::

- [[Highlights]]
	- #+BEGIN_IMPORTANT
	  -- ((6575dab7-69bf-420b-a3f9-9536e2371a88))
	  -- ((6575d887-11b4-42cf-bbb0-35f2edb9a2c9))
	  -- ((6575db74-e580-4e8f-9fe1-4a1f38cfa462))
	  -- ** ((6575dce2-7118-4127-a285-a98b23de4a12)) **
	  #+END_IMPORTANT
	- ((6575daf9-cbc0-40bf-b84a-d2907592593c))
	- ((6575db4a-5e63-40fe-82b0-50de4c47d428))
	- ![image.png](../assets/image_1702223603638_0.png){:height 317, :width 503}
		- ((6575dfaf-7d00-4762-a5f9-28690486f3ca))
			- What are the various attack strategies mentioned for the Guided Code Transformers steps? [[question]]
		- The description of Fig. 1 requires some concrete and explanatory example, otherwise it is not easy to grasp the details of the proposed approach and of the 3 main components.
	- ((6575e2bc-e630-4510-9694-4164a29b379c))
		- An illustrative example is needed here.
		- The two code needs the be similar from a functional point of view and inspectable for humans.
			- If code are similar from a functional point of view, in what respect are we talking about adversarial attack? Can you clearly define what do you mean with adversarial attack in the particular considered domain? [[question]]
			- ((6575e352-ebdd-424c-b1d0-60207026e293))
				- How to ensure that? [[question]]
					- The answer is given in the following table
					- ![image.png](../assets/image_1702224782317_0.png)
	- ((6575e468-5282-4bf9-8854-35c8d0b68c45))
		- Can you define this? When an attack can be considered successful? [[question]]
	- ((6575e52d-42cc-42f7-84ad-7ab967568727))
		- Example [[question]]
		  id:: 6575e533-8197-48b0-aa23-1f677b5fe30a
	- ((6575e568-fb57-49df-9f69-31f5e2c1464d))
		- The problem on the limited number of interactions with the target models is addressed by employing a mechanism that can predict the outpout of the considered model.
			- **This is a possible threat to validity or bias to be discussed.**
	- ((6575e67a-6067-4c0f-962b-7dc424247900))
		- Why Code Classification is considered as a code-to-code scenario [[question]]
	- ((6575e81b-3f49-469c-beb7-c81bda6adf41))
		- Prediction capabilities of what?
	- ((6575e84d-d1b5-48ab-84c2-33f8f2519cd7))
		- Why predicted labels when you are considering different tasks including summarization. In such a case, how do you assess if an attack has success or not?
	- ((6575e970-ec66-47ad-ab2f-0049c6d0c1ca))
		- This seems to be similar to ASR. Why not defining Robustness as follows?
			- $Ro = \frac{\sum_{(x_i,x'_{ij}) \in D_T} I (P(x_i) = P(x'_{ij}))}{|D_T|}$
	- Why according to Table III, Code classification is a task, which appears to be not supported by LLMs? [[question]]
	- ((6575ee7e-212a-42a3-8a26-9a142754745c))
		- The active-learning component is not evident.
		-
- [[Comments]]
	- The paper introduces ALANCA, an approach for conducting adversarial attacks on code comprehension using various pre-trained and large language models. ALANCA relies on semantic-preserving translations on code and employs adversarial discriminator and token selector mechanisms . The evaluations demonstrate ALANCA's ability to confuse different neural models used in software engineering tasks, including code summarization, method name prediction, code classification, and code detection. A replication package has been made available.
	-
	- Overall, I appreciate the paper's timeliness, relevance, and well-structured presentation. However, I have some concerns:
		- The paper lacks explanatory examples supporting the building blocks of ALANCA as presented in Fig. 1. Reducing the presentation of Algorithm 1 could potentially free up space for more clarity.
		- The "active-learning" component of ALANCA is not adequately explained. Sentences such as "we have incorporated a learnable discriminator module within ALANCA..." need clearer explanations with illustrative examples.
		- It appears that the limited number of interactions with the target models is addressed by a mechanism predicting the output of the model. This introduces a potential threat to validity or bias that should be discussed.
		- In Section IV.A, Code Classification is treated as a code-to-code scenario, which requires clarification.
		- The definition of Attack Success Rate, as the rate at which predicted labels are changed after the transformation process, needs revision. The inclusion of tasks like summarization, which does not produce labels, raises questions about how success is measured in such cases.
		- The Robustness metrics seem similar to Attack Success Rate, and its definition is unclear. A suggestion is to consider defining Robustness as something like $Ro = \frac{\sum_{(x_i,x'_{ij}) \in D_T} I (P(x_i) = P(x'_{ij}))}{|D_T|}$
		- According to Table III, Code Classification is labeled as a task not supported by Large Language Models (LLMs). This requires clarification and justification.
- [[REVIEWS/Notes]]
	- ((6575e6ff-54da-4b73-acd8-8e4662d73efd))
		- AST-based models, e.g. Code2Vec and Code2Seq
		- Pre-trained transformed models, e.g., CodeBert, GraphCodeBert, PLBART, and CodeT5
		- LLMs, including ChatGPT and ChatGLM2

• {{embed SANER2024-REVIEW-INSTRUCTIONS}}