14 KiB
14 KiB
type:: REVIEWS tags:: year:: 2024 venue:: ICSE full-title:: Magika: AI-Powered Content-Type Detection date-start:: 16-04-2024 - 14:02 date-submitted:: external-links:: status:: DONE deadline-submission:: file:: @icse2025-paper239.pdf parent:: todoist:: https://app.todoist.com/app/task/239-magika-ai-powered-content-type-detection-7858696715
- ### [[Highlights]]
-
- # Annotazioni
- (5/5/2024, 18:38:10)
- - “programming language when a user creates a new file without specifying a file extension [7]. While the original guesslang is currently unmaintained (last commit in September 2021) and relies on deprecated TensorFlow abstractions, VS Code developers maintain a Node.js client that facilitates the testing of the underlying model [26].” ([“icse2025-paper239.pdf”, p. 2](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=2&annotation=698UBFAE)) #f0ff00
- *Interesting. I have to check it*
- - “a stratified” ([“icse2025-paper239.pdf”, p. 3](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=3&annotation=4RPFTFZR)) #f0ff00
- *What do you mean?*
- - “100 most prominent file extensions” ([“icse2025-paper239.pdf”, p. 3](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=3&annotation=W6U5AKKK)) #f0ff00
- *What's the coverage of this with respect those in references [29]... Below?*
- - “detection errors (in the case of trusting one tool above others). We use four heuristics for validation: file size, magic bytes (for binary files), character encoding (for text files), and file trustworthiness. For file size, we require any sample in our” ([“icse2025-paper239.pdf”, p. 3](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=3&annotation=CQ5WM896)) #f0ff00
- *Hope the four heuristics are described later. Here they are only mentioned without explanation*
- - “set of necessary but not sufficient” ([“icse2025-paper239.pdf”, p. 3](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=3&annotation=N6EWHFA9)) #f0ff00
- *It can be source of bias right?*
- - “obtain 10K samples per content type for” ([“icse2025-paper239.pdf”, p. 3](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=3&annotation=RT5NA5ID)) #f0ff00
- *Check the math*
- - “validation dataset consists” ([“icse2025-paper239.pdf”, p. 3](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=3&annotation=HMCTCE2N)) #f0ff00
- *Check the usage of this.*
- - “multi-class classification problem, we estimate per-type precision as:” ([“icse2025-paper239.pdf”, p. 4](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=4&annotation=CP6S3GFR)) #f0ff00
- *multi-class because binary, text, exe?.....*
- - “[11].” ([“icse2025-paper239.pdf”, p. 5](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=5&annotation=359QS6S7)) #f0ff00
- *Da vedere.*
- - “content types, with an average F1 score of 96% when using the MIME flag,” ([“icse2025-paper239.pdf”, p. 5](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=5&annotation=HHNVE4S3)) #f0ff00
- *This mime management needs to be better understood.*
- - “signatures created by experts.” ([“icse2025-paper239.pdf”, p. 6](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=6&annotation=U6E4AIDU)) #f0ff00
- *Check why signatures need to be created by users.*
- - “Speed.” ([“icse2025-paper239.pdf”, p. 6](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=6&annotation=MGCADFSS)) #f0ff00
- *How can you affect this aspect? What are the intervention dimensions?*
- - “Reshape size, number and size of Dense layers, normalization type, and amount of dropout applied before converging on this specific design.” ([“icse2025-paper239.pdf”, p. 6](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=6&annotation=KIMQ42SJ)) #ff4400
- *Does not parse.*
- - ([“icse2025-paper239.pdf”, p. 7](zotero://select/library/items/N2DCQ73C)) #f0ff00
- *Dropout?*
- - “not enough bytes” ([“icse2025-paper239.pdf”, p. 7](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=7&annotation=MUUA2XUC)) #f0ff00
- *Hat does it mean? When you get not enough types?*
- - “a Dense layer.” ([“icse2025-paper239.pdf”, p. 7](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=7&annotation=AQWW4J6X)) #f0ff00
- *To be checked.*
- - “gelu” ([“icse2025-paper239.pdf”, p. 7](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=7&annotation=CVMY8JEM)) #f0ff00
- *?*
- - “softmax activation” ([“icse2025-paper239.pdf”, p. 7](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=7&annotation=T63VRH3Z)) #f0ff00
- *?*
- - “an argmax” ([“icse2025-paper239.pdf”, p. 7](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=7&annotation=3K545WHE)) #f0ff00
- *?*
- - “C. Training” ([“icse2025-paper239.pdf”, p. 7](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=7&annotation=8VQKK7DH)) #f0ff00
- *Interesting because it is a typical setting.*
- - “for data augmentation,” ([“icse2025-paper239.pdf”, p. 7](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=7&annotation=RY3Z4AXV)) #f0ff00
- *Why data augmentation? When is it performed in the process?*
- - “30 epochs” ([“icse2025-paper239.pdf”, p. 7](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=7&annotation=3XJ6VDJK)) #f0ff00
- *Check*
- - “Figure 3 shows how the validation loss and validation accuracy progress with the number of epochs. Note how the” ([“icse2025-paper239.pdf”, p. 7](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=7&annotation=9D9B7W98)) #f0ff00
- *Validation loss and accuracy*
- - “deep learning model either through a command line or API. As part of this, we use OnnxRuntime [39] (instead of Tensorflow and Keras, which we use for training), because it is roughly 15x faster in loading the model (while having a similar model” ([“icse2025-paper239.pdf”, p. 8](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=8&annotation=A3D9RDV5)) #f0ff00
- *How is this possible?*
- - “[11].” ([“icse2025-paper239.pdf”, p. 8](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=8&annotation=X76CJQUC)) #f0ff00
- *To check*
- - “-1%” ([“icse2025-paper239.pdf”, p. 9](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=9&annotation=49VH8XK8)) #f0ff00
- *Negative?*
- - “MIME flag), followed by MAGIKA, exiftool, trid, and guesslang.” ([“icse2025-paper239.pdf”, p. 9](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=9&annotation=4THPWB8L)) #f0ff00
- - “OnnxRuntime.” ([“icse2025-paper239.pdf”, p. 9](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=9&annotation=6K9CNC8F)) #f0ff00
- *But this is for execution time.*
- - “a command line wrapper” ([“icse2025-paper239.pdf”, p. 9](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=9&annotation=TRRJPWBX)) #f0ff00
- *?*
- - “, on a daily basis, MAGIKA routes to MS Officespecific malware scanners” ([“icse2025-paper239.pdf”, p. 10](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=10&annotation=TE3DMZVJ)) #f0ff00
- *This is a different problem, isn't it?*
- - “alternatives. We subsequently engaged in a dialogue regarding current feature gaps, primarily focusing on the necessity for more fine-grained detection of specific content types (e.g., C vs. C++, JAVASCRIPT vs. TYPESCRIPT, INI vs. TOML), which we are currently exploring.VIII. DISCUSSION” ([“icse2025-paper239.pdf”, p. 10](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=10&annotation=UDHKPD66)) #f0ff00
- *How difficult is this process?*
- - “Trained on 24M samples” ([“icse2025-paper239.pdf”, p. 10](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=10&annotation=XH5HHB95)) #f0ff00
- - “for malware analysis,” ([“icse2025-paper239.pdf”, p. 10](zotero://select/library/items/N2DCQ73C)) ([pdf](zotero://open-pdf/library/items/XYE4ESPB?page=10&annotation=IVJR7LR6)) #f0ff00
- *How content type detection is linked to this?*
- ### [[Comments]]
- #.tabular
- ### Paper summary
- The paper introduces MAGIKA, an AI-powered content-type detection tool. The tool's architecture consists of several layers and is based on an introduced encoding of the input files. MAGIKA outperforms existing content-type detection tools (including file and guesslang) with an average F1 score of 99% across over a hundred content types and a dataset of 26.5 million items. The paper emphasizes the adoption of MAGIKA by an email provider for attachment scanning, by VirusTotal for malware analysis, and mentions a plan to integrate the tool with Visual Studio Code.
- ### Strengths
- + Well structured and well presented paper
- + Relevant approach
- + High performance AI-based tool
- ### Weaknesses
- - Minor presentation issues that need to be fixed
- ### Detailed comments for authors
- Novelty: The paper introduces an innovative approach to content-type detection using AI, which significantly overcomes existing methods.
- Rigor: Overall, the paper is well presented, and the evaluation is properly done. However, some details need clarification, as discussed in the detailed comments below.
- Relevance: Content-type detection is a critical task with widespread applications, making MAGIKA highly relevant to various domains. However, while the authors suggest applicability in malware analysis, how content-type detection contributes to this task remains unclear.
- Verifiability & Transparency: The authors have made the tool open-source, providing an anonymized link to the source code. Additionally, a comprehensive table comparing MAGIKA with other tools is also provided.
- Presentation: Certain technical terms and methodologies require clearer explanations to improve the paper's accessibility and readability (see detailed comments below).
- Detailed comments
- Page 3 - "Rather than simulate any single environment, we instead use GitHub and VirusTotal to source a stratified sample of content types including source code, executables, documents, media, archives, and more.": The concept "stratified" is not clear at this point of the paper. Please clarify.
- Page 3 - Regarding the discussion on "content types," could you provide an analysis of the coverage achieved by the 128 considered file extensions compared to those curated in references [29]-[31]?
- Page 3 - While the paper mentions using four heuristics for validation, their description should be improved to increase clarity and understanding for readers. In particular, concerning size, the authors require a lower bound without discussing later in the paper the extent to which the size of the input affects the performance of MAGIKA. Concerning the magic bytes, the authors apply a set of "necessary but not sufficient" rules to validate a file extension. Authors should discuss if such rules can be sources of bias.
- Page 3 - Final dataset: There is no explicit reference to the size of the considered samples for experiments. Moreover, the mentioned sizes of the samples used for training, testing, and validation need to be motivated and described.
- Page 6 - the sentence "Reshape size, number and size of Dense layers, normalization type, and amount of dropout applied before converging on this specific design." does not parse, please rewrite.
- Section V.B - the presented model architecture should also be discussed with respect to the requirements shown in Sec. V.A. It is necessary to describe the components and the technical decisions contributing to the initial requirements. I know there is a discussion later in the paper. However, some preliminary discussion on the decisions taken can be given in Section V.B.
- Page 6 - "If there are not enough bytes in a file, we pad the encoding with a special character (represented by the integer 256)." What does it mean "not enough"? How can you decide if a file contains enough bytes? Is there any threshold that you identified?
- Page 7 - To make the paper self-contained, I suggest giving at least links to the used argmax function and of the gelu and softmax activations.
- Section V.C (Training) - the data augmentation process is quickly mentioned and needs clarification. As far as I understood by reading Section III, synthetic data are generated for UNKNOWN and TXT content types. The motivation behind such a decision needs to be clarified.
- Questions
- Could you elaborate on how MAGIKA's content-type detection capabilities can contribute to malware analysis?
- Why was data augmentation necessary in your approach?
-
- ### [[REVIEWS/Notes]]