logseq/pages/@An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry.md

links:: Local library, Web library authors:: Wenxin Jiang, Nicholas Synovic, Matt Hyatt, Taylor R. Schorlemmer, Rohan Sethi, Yung-Hsiang Lu, George K. Thiruvathukal, James C. Davis tags:: Computer Science - Artificial Intelligence, Computer Science - Machine Learning, Computer Science - Software Engineering, #zotero date:: 04-03-2023 item-type:: preprint title:: @An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry

• Abstract
  • Deep Neural Networks (DNNs) are being adopted as components in software systems. Creating and specializing DNNs from scratch has grown increasingly difficult as state-of-the-art architectures grow more complex. Following the path of traditional software engineering, machine learning engineers have begun to reuse large-scale pre-trained models (PTMs) and fine-tune these models for downstream tasks. Prior works have studied reuse practices for traditional software packages to guide software engineers towards better package maintenance and dependency management. We lack a similar foundation of knowledge to guide behaviors in pre-trained model ecosystems. In this work, we present the first empirical investigation of PTM reuse. We interviewed 12 practitioners from the most popular PTM ecosystem, Hugging Face, to learn the practices and challenges of PTM reuse. From this data, we model the decision-making process for PTM reuse. Based on the identified practices, we describe useful attributes for model reuse, including provenance, reproducibility, and portability. Three challenges for PTM reuse are missing attributes, discrepancies between claimed and actual performance, and model risks. We substantiate these identified challenges with systematic measurements in the Hugging Face ecosystem. Our work informs future directions on optimizing deep learning ecosystems by automated measuring useful attributes and potential attacks, and envision future research on infrastructure and standardization for model registries.

• Attachments

  • arXiv.org Snapshot {{zotero-imported-file JBL2ELEQ, "2303.html"}}
  • Jiang et al_2023_An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning.pdf {{zotero-imported-file FC5QI4KB, "Jiang et al_2023_An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning.pdf"}}

• Notes

  • Comment: Proceedings of the ACM/IEEE 45th International Conference on Software Engineering (ICSE) 2023
• Highlights
  • ((65a99501-0072-404d-b1f0-d862178a8a9e))
  • DNN
    • pre-trained on large datasets
    • fine-tuned to solve specialized tasks
  • ((65baa9b4-3525-42c9-867a-0fbf563056b1)): a collaborative model hub where teams can share DL moels. Examples are:
    • Hugging Face (It offers the largest and most diverse set of pre trained models \approx 60,000 PTMs)
    • TensorFlow Hub
    • PyTorch Hub
    • ONNX Model Zoo
  • DL Traceability is limited because authors often omit training logs and documentation.
  • Interestingly, the work discusses:
    • how engineers select PTM
    • the PTM attributes that facilitate PTM reuse
    • the challenges of PTM reuse
    • risks related to the reuse of PTMs
  • Two main reuse techniques are employed:
    • Transfer learning
    • Quantization techniques
  • Expertes find easier to reuse PTMs from DL model registries than adopting PTMs from GitHub projects. In particular, typically users take PTMs from model registries and apply tranfer larning techniques to the model. They reuse is performed by fine-tuning an existing PTM by optionally extending the architecture and training on a task-specific dataset, or by building a new model on top of the pre-trained one.
  • The decision-making process to select PTMs that are more appropriate for the task to be performed typically considers at least the following aspects:
    • Reusability assessment by considering requirements related to model input and output, latency, size and licensing. The availability of enough computational resources is also an important aspect, which is considering when evaluating PTMs
    • Downstream evaluation: after selecting candidate PTMs, engineers conduct a downstream evaluation for their rspecific task, thus they finetune them, test them, and compare them. This is a strenuos activity because models might not work tool well directly, and it can happen to miss adequate documantion or observing discrepancies within existing documantions. The effort required to deploy PTMs is also a relevant characteristics, which is considered when selecting them.
  • The typical attributes, which are typically considered when evaluating and comparing PTMs are:
    • Popularity, assessed e.g., by considering the number of download
    • Provenance, e.g., information about original paper, dataset, and architecture.
    • Reproducibility, it comes from two aspects: (1) the configuration of training (e.g., hardware types, required memory, training scripts, hyperparameters), (2) the understanding of the model (e.g., availability of notebook demo, and documentation)
    • Portability, it includes hardware specification and environment.
  • The challenges of PTM reuse
    • Missing attributes in the model registries including datasets, licensing, and model details. For instance, Hugging Face does not enforce any form of documentation. In any case, model registries, do not provide automated approaches to measure model attributes.
    • Descrepancies, e.g., models are over-promising, models are not named correctly, or provided scripts are broken. A reasing for this kind of problem is that training configuration details (i.e., hyper-parameters) are hard to find.
    • Model risks including privacy and ethics aspects. In particular, users can be reluctant to send their sensitive dataset to Hugging Face. Moreover, if a model is trained with malicious intents, it could have a lot of consequences. This indicates the potential risks of a malicious model being uploaded to model registries.
• • Summary for the SOTA of the PROJECTS/MOSAICO project proposal:
    • DNNs are often pre-trained on large datasets and fine-tuned for specialized tasks. Model registries like Hugging Face, TensorFlow Hub, and PyTorch Hub serve as collaborative hubs for sharing DNN models. In [X] authors explore how engineers select Pre-Trained Models (PTMs), emphasizing PTM attributes that facilitate reuse, challenges, and associated risks. In the performed investigation, the preference for reusing PTMs from model registries over GitHub projects is noted. The decision-making process for selecting PTMs considers factors like reusability, downstream evaluation, and deployment effort. Attributes like popularity, provenance, reproducibility, and portability are crucial in evaluating and comparing PTMs. The mains challenges in PTM reuse include missing attributes in registries, discrepancies in model promises, and potential risks related to privacy and ethics, emphasizing the need for robust documentation and governance in collaborative model sharing platforms.