200 lines
7.4 KiB
Markdown
200 lines
7.4 KiB
Markdown
file:: [Jiang et al_2023_An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning.pdf](file://C:/Users/david/Zotero/storage/FC5QI4KB/Jiang et al_2023_An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning.pdf)
|
|
file-path:: file://C:/Users/david/Zotero/storage/FC5QI4KB/Jiang et al_2023_An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning.pdf
|
|
|
|
- An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: green
|
|
id:: 65a99501-0072-404d-b1f0-d862178a8a9e
|
|
- Deep Neural Networks (DNNs) are being adopted as components in software systems
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: green
|
|
id:: 65ba6360-f1e9-437a-8c8c-f83664a34972
|
|
- reuse large-scale pre-trained models (PTMs) andfine-tune these models for downstream tasks
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: purple
|
|
id:: 65ba6377-6b5d-467e-b2b8-a426272be340
|
|
- Prior works have studied reuse practices for traditional software packages to guide software engineers towards better package maintenance and dependency managemen
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: blue
|
|
id:: 65ba6381-5211-475c-a49a-470cab1f72fc
|
|
hl-stamp:: 1706714652863
|
|
- . We lack a similar foundation of knowledge to guide behaviors in pre-trained model ecosystems
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: purple
|
|
id:: 65ba638d-569e-4c2d-b7a3-8d997633322a
|
|
hl-stamp:: 1706714650511
|
|
- decision-making process for PTM reuse
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: purple
|
|
id:: 65ba663e-1304-4dfb-9a14-149c8c36c840
|
|
- provenance
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: purple
|
|
id:: 65ba664d-8354-4c39-b8c3-abac806d0fc0
|
|
- reproducibility
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: purple
|
|
id:: 65ba6650-e186-4ad6-90bf-d6cb4e3ec03d
|
|
- portability
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: purple
|
|
id:: 65ba6652-d3b8-4fb8-9cb2-88c5159d585a
|
|
- hree challenges for PTM reuse are missing attributes, discrepancies between claimed and actual performance, and model risks
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: blue
|
|
id:: 65ba665d-5d51-4878-98c7-ed30863879dd
|
|
- automated measuring useful attributes and potential attacks, and envision future research on infrastructure and standardization for model registries
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: purple
|
|
id:: 65ba6670-d3d6-4437-8a4b-6b7f20c85900
|
|
- hese problems can be addressed by reusing pre-trained DNN models (PTMs) to amortize DNN development costs across multiple projects and organizations
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: green
|
|
id:: 65ba6696-2844-41eb-aafc-32c68b23571f
|
|
- we present the first empirical study of pretrained model reuse
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: purple
|
|
id:: 65ba6aca-3d58-49c6-8821-b44dace2b352
|
|
- Hugging Face DL model registry
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: purple
|
|
id:: 65baa376-bf86-4d5d-b32e-68273597908e
|
|
- which is the largest PTM registry at presen
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: purple
|
|
id:: 65baa37f-c0bf-4f7e-8074-9523ba8cafe4
|
|
- Our findings indicate that PTM reuse workflows are similar to those for traditional software package reuse, but that engineers follow practices and experience challenges specific to deep learning.
|
|
ls-type:: annotation
|
|
hl-page:: 1
|
|
hl-color:: purple
|
|
id:: 65baa3b3-f08c-4390-9184-85f98742c59a
|
|
- PTM reuse
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: blue
|
|
id:: 65baa46b-438b-47ee-b668-be3279bce724
|
|
- determine attributes of PTMs
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: blue
|
|
id:: 65baa481-7113-4534-a2da-821511ece855
|
|
- improve the quality of their offerings
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: purple
|
|
id:: 65baa48f-8448-4707-8cb9-a50f1eeb2e80
|
|
- S. Oladele, “Ml model registry: What it is, why it matters, how to implement it,” 2022
|
|
ls-type:: annotation
|
|
hl-page:: 13
|
|
hl-color:: green
|
|
id:: 65baa4d4-e9dd-472d-baf8-9296343f0a5c
|
|
- . Prior work shows that engineers may improve their software selection with insights into the decision-making process and an understanding of relevant factors
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: green
|
|
id:: 65baa826-a176-479b-b14a-788175a6f73b
|
|
- The extent to which reuse practices for traditional software will transfer to the reuse of PTM packages is unclear.
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: blue
|
|
id:: 65baa832-e8c8-4924-837b-20603ccfb9cb
|
|
- Through transfer learning,
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: green
|
|
id:: 65baa8a6-da7c-44f6-b522-606378c8ab2a
|
|
- DNNs can be pre-trained on large datasets and fine-tuned to solve specialized tasks
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: green
|
|
id:: 65baa8c4-9bbf-46ea-9207-43100e20affa
|
|
- To better reuse the PTMs, it is important to monitor the performance of deployed models, track changes in data characteristics, and to retrain and revalidate them frequently
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: purple
|
|
id:: 65baa958-8736-4d89-bb63-af0a66b4529b
|
|
- DL model registry
|
|
ls-type:: annotation
|
|
hl-page:: 2
|
|
hl-color:: green
|
|
id:: 65baa9b4-3525-42c9-867a-0fbf563056b1
|
|
- DL traceability is hampered because authors often omit training logs and documentation
|
|
ls-type:: annotation
|
|
hl-page:: 3
|
|
hl-color:: green
|
|
id:: 65baadfe-1e3d-4921-8c13-ec764d6bbaf9
|
|
- qualitative and quantitative aspects
|
|
ls-type:: annotation
|
|
hl-page:: 3
|
|
hl-color:: green
|
|
id:: 65bab0ec-f0a4-4d47-8e1e-0a798aec2d26
|
|
- RQ1 How do engineers select PTMs?
|
|
ls-type:: annotation
|
|
hl-page:: 3
|
|
hl-color:: green
|
|
id:: 65bab10d-f9d8-495a-b456-41352cea8f60
|
|
- PTM attributes
|
|
ls-type:: annotation
|
|
hl-page:: 3
|
|
hl-color:: green
|
|
id:: 65bab115-d5f0-4dbf-ba4e-86cb951f8e86
|
|
- challenges
|
|
ls-type:: annotation
|
|
hl-page:: 3
|
|
hl-color:: green
|
|
id:: 65bab118-cb7f-4748-baaf-04db90e07feb
|
|
- risks
|
|
ls-type:: annotation
|
|
hl-page:: 3
|
|
hl-color:: green
|
|
id:: 65bab11f-29ec-4040-8f12-dd0d8183c19a
|
|
- our participants reported using only two: transfer learning and quantization techniques. When reusing, participants find PTMs from DL model registries easier to adopt than PTMs from GitHub projects
|
|
ls-type:: annotation
|
|
hl-page:: 4
|
|
hl-color:: green
|
|
id:: 65bb5edf-9f47-450a-9bd3-5ad8b052cf22
|
|
- This indicates the potential risks of a malicious model being uploaded to model registrie
|
|
ls-type:: annotation
|
|
hl-page:: 7
|
|
hl-color:: purple
|
|
id:: 65bb6be6-0a4c-4260-a52c-cd31cc9f3526
|
|
- Figure 5, to represent the how models are created and shared
|
|
ls-type:: annotation
|
|
hl-page:: 7
|
|
hl-color:: blue
|
|
id:: 65bb6c2d-844f-48c0-a9fa-612334eec393
|
|
- The common unit of reuse on Hugging Face is the repository, classified into datasets (input/output data for supervised or unsupervised learning) and models (PTM architecture, weights, and configuration, cf. Figure 2)
|
|
ls-type:: annotation
|
|
hl-page:: 7
|
|
hl-color:: purple
|
|
id:: 65bb6c45-05c4-40ac-bd69-340b08c2d7c2
|
|
- Organization Verification
|
|
ls-type:: annotation
|
|
hl-page:: 7
|
|
hl-color:: green
|
|
id:: 65bb6c87-ec84-4bb6-a253-183748ec306d
|
|
- Out of 6,243 organizations, only199 (3.19%) were verified
|
|
ls-type:: annotation
|
|
hl-page:: 7
|
|
hl-color:: blue
|
|
id:: 65bb6ca2-b465-42a0-9db1-a743326e3ec8
|
|
- Universal Dependencies dataset [79] is the most popular dataset on Hugging Face, with6,834 models depending upon it
|
|
ls-type:: annotation
|
|
hl-page:: 8
|
|
hl-color:: green
|
|
id:: 65bb6cf4-d491-4054-9909-ecb3ee21df30 |