3.8 KiB
3.8 KiB
type:: REVIEWS tags:: year:: 2024 venue:: ICSE full-title:: date-start:: 02-10-2024 - 21:30 date-submitted:: external-links:: status:: DONE deadline-submission:: file:: @icse2025-paper818 parent:: todoist:: https://app.todoist.com/app/task/818-auto-dgr-dnn-model-recommendation-based-on-dual-graph-representation-learnin-6WFCX7JPX4FCjP3c
- ### [[Highlights]]
- ### [[Comments]]
- #.tabular
- - ### Paper summary
- The paper presents AutoDGR, an approach to recommend pre-trained models from a given model hub, such as HuggingFace, that best fit the tasks that users want to perform (e.g., image classification). AutoDGR exploits the usage of a dual-graph representation system to encode datasets and models separately. A graph convolutional network is used to embed them into a high-dimensional space. The approach has been experimented with existing work by focusing on the problem of image classification, and it outperforms them.
- - ### Strengths
- +
- + Interesting and relevant problem
- + The dual-graph representation for both tasks and models is interesting
- - ### Weaknesses
- -
- The evaluation is primarily focused on image classification tasks, which may limit the applicability of the approach to other domains.
- The model's success relies on assumptions regarding the similarity of tasks and models, which might not hold in more diverse scenarios.
- The approach has not been validated on tasks outside of image classification.
- - ### Detailed comments for authors
- Novelty: AutoDGR presents an interesting approach by combining dual-graph representation learning with GCNs to recommend models for deep learning tasks.
- Rigor: The approach is properly evaluated using a carefully constructed benchmark dataset, and results are compared against relevant baselines. However, the strong assumptions underpinning the method, such as models with similar architectures or tasks exhibiting similar performance, need to be substantiated with more empirical evidence. For instance, assuming that models with similar parameters will perform similarly on the same dataset seems too simplistic and could limit the method's applicability in real-world settings.
- Relevance: The problem of recommending suitable pre-trained models for new tasks is critical for the widespread adoption of DNNs, especially in non-expert settings. Extending the evaluation to non-image domains would enhance its relevance further.
- Verifiability & Transparency: The authors need to provide more details on how the ground truth is created. It is important to ensure that the process used to determine a task's "optimal" model is clearly described and reproducible.
- Presentation: The paper is well-structured and easy to follow, with clear explanations of the method and results. The figures are helpful in illustrating the key concepts, such as the dual-graph structure and performance comparisons.
- #### Additional Comments:
- The process for generating the ground-truth optimal model is not described in detail, raising questions about the reliability of the recommendations. Clarifying this process would improve transparency.
- As recognized by the authors, the experiments that were performed were affected by some threats to external validity. While the approach performs well on image classification tasks, its generality across different types of tasks, especially in the software engineering domain (e.g., code generation, code completion, library recommendation), remains untested.
- ### [[REVIEWS/Notes]]
- ### YELLOW CONCERNS
background-color:: yellow
- {{query (and [[ffd400]] [[icse2025-paper818]] )}}
collapsed:: true
- ### ❓️Questions
- {{query (and [[question]] [[icse2025-paper818]] )[[question]]}}
query-table:: true
query-properties:: [:block]