logseq/pages/hls__Jiang et al_2023_An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning.md

file:: [Jiang et al_2023_An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning.pdf](file://C:/Users/david/Zotero/storage/FC5QI4KB/Jiang et al_2023_An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning.pdf) file-path:: file://C:/Users/david/Zotero/storage/FC5QI4KB/Jiang et al_2023_An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning.pdf

• An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry ls-type:: annotation hl-page:: 1 hl-color:: green id:: 65a99501-0072-404d-b1f0-d862178a8a9e
• Deep Neural Networks (DNNs) are being adopted as components in software systems ls-type:: annotation hl-page:: 1 hl-color:: green id:: 65ba6360-f1e9-437a-8c8c-f83664a34972
• reuse large-scale pre-trained models (PTMs) andfine-tune these models for downstream tasks ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 65ba6377-6b5d-467e-b2b8-a426272be340
• Prior works have studied reuse practices for traditional software packages to guide software engineers towards better package maintenance and dependency managemen ls-type:: annotation hl-page:: 1 hl-color:: blue id:: 65ba6381-5211-475c-a49a-470cab1f72fc hl-stamp:: 1706714652863
• . We lack a similar foundation of knowledge to guide behaviors in pre-trained model ecosystems ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 65ba638d-569e-4c2d-b7a3-8d997633322a hl-stamp:: 1706714650511
• decision-making process for PTM reuse ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 65ba663e-1304-4dfb-9a14-149c8c36c840
• provenance ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 65ba664d-8354-4c39-b8c3-abac806d0fc0
• reproducibility ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 65ba6650-e186-4ad6-90bf-d6cb4e3ec03d
• portability ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 65ba6652-d3b8-4fb8-9cb2-88c5159d585a
• hree challenges for PTM reuse are missing attributes, discrepancies between claimed and actual performance, and model risks ls-type:: annotation hl-page:: 1 hl-color:: blue id:: 65ba665d-5d51-4878-98c7-ed30863879dd
• automated measuring useful attributes and potential attacks, and envision future research on infrastructure and standardization for model registries ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 65ba6670-d3d6-4437-8a4b-6b7f20c85900
• hese problems can be addressed by reusing pre-trained DNN models (PTMs) to amortize DNN development costs across multiple projects and organizations ls-type:: annotation hl-page:: 1 hl-color:: green id:: 65ba6696-2844-41eb-aafc-32c68b23571f
• we present the first empirical study of pretrained model reuse ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 65ba6aca-3d58-49c6-8821-b44dace2b352
• Hugging Face DL model registry ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 65baa376-bf86-4d5d-b32e-68273597908e
• which is the largest PTM registry at presen ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 65baa37f-c0bf-4f7e-8074-9523ba8cafe4
• Our findings indicate that PTM reuse workflows are similar to those for traditional software package reuse, but that engineers follow practices and experience challenges specific to deep learning. ls-type:: annotation hl-page:: 1 hl-color:: purple id:: 65baa3b3-f08c-4390-9184-85f98742c59a
• PTM reuse ls-type:: annotation hl-page:: 2 hl-color:: blue id:: 65baa46b-438b-47ee-b668-be3279bce724
• determine attributes of PTMs ls-type:: annotation hl-page:: 2 hl-color:: blue id:: 65baa481-7113-4534-a2da-821511ece855
• improve the quality of their offerings ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 65baa48f-8448-4707-8cb9-a50f1eeb2e80
• S. Oladele, “Ml model registry: What it is, why it matters, how to implement it,” 2022 ls-type:: annotation hl-page:: 13 hl-color:: green id:: 65baa4d4-e9dd-472d-baf8-9296343f0a5c
• . Prior work shows that engineers may improve their software selection with insights into the decision-making process and an understanding of relevant factors ls-type:: annotation hl-page:: 2 hl-color:: green id:: 65baa826-a176-479b-b14a-788175a6f73b
• The extent to which reuse practices for traditional software will transfer to the reuse of PTM packages is unclear. ls-type:: annotation hl-page:: 2 hl-color:: blue id:: 65baa832-e8c8-4924-837b-20603ccfb9cb
• Through transfer learning, ls-type:: annotation hl-page:: 2 hl-color:: green id:: 65baa8a6-da7c-44f6-b522-606378c8ab2a
• DNNs can be pre-trained on large datasets and fine-tuned to solve specialized tasks ls-type:: annotation hl-page:: 2 hl-color:: green id:: 65baa8c4-9bbf-46ea-9207-43100e20affa
• To better reuse the PTMs, it is important to monitor the performance of deployed models, track changes in data characteristics, and to retrain and revalidate them frequently ls-type:: annotation hl-page:: 2 hl-color:: purple id:: 65baa958-8736-4d89-bb63-af0a66b4529b
• DL model registry ls-type:: annotation hl-page:: 2 hl-color:: green id:: 65baa9b4-3525-42c9-867a-0fbf563056b1
• DL traceability is hampered because authors often omit training logs and documentation ls-type:: annotation hl-page:: 3 hl-color:: green id:: 65baadfe-1e3d-4921-8c13-ec764d6bbaf9
• qualitative and quantitative aspects ls-type:: annotation hl-page:: 3 hl-color:: green id:: 65bab0ec-f0a4-4d47-8e1e-0a798aec2d26
• RQ1 How do engineers select PTMs? ls-type:: annotation hl-page:: 3 hl-color:: green id:: 65bab10d-f9d8-495a-b456-41352cea8f60
• PTM attributes ls-type:: annotation hl-page:: 3 hl-color:: green id:: 65bab115-d5f0-4dbf-ba4e-86cb951f8e86
• challenges ls-type:: annotation hl-page:: 3 hl-color:: green id:: 65bab118-cb7f-4748-baaf-04db90e07feb
• risks ls-type:: annotation hl-page:: 3 hl-color:: green id:: 65bab11f-29ec-4040-8f12-dd0d8183c19a
• our participants reported using only two: transfer learning and quantization techniques. When reusing, participants find PTMs from DL model registries easier to adopt than PTMs from GitHub projects ls-type:: annotation hl-page:: 4 hl-color:: green id:: 65bb5edf-9f47-450a-9bd3-5ad8b052cf22
• This indicates the potential risks of a malicious model being uploaded to model registrie ls-type:: annotation hl-page:: 7 hl-color:: purple id:: 65bb6be6-0a4c-4260-a52c-cd31cc9f3526
• Figure 5, to represent the how models are created and shared ls-type:: annotation hl-page:: 7 hl-color:: blue id:: 65bb6c2d-844f-48c0-a9fa-612334eec393
• The common unit of reuse on Hugging Face is the repository, classified into datasets (input/output data for supervised or unsupervised learning) and models (PTM architecture, weights, and configuration, cf. Figure 2) ls-type:: annotation hl-page:: 7 hl-color:: purple id:: 65bb6c45-05c4-40ac-bd69-340b08c2d7c2
• Organization Verification ls-type:: annotation hl-page:: 7 hl-color:: green id:: 65bb6c87-ec84-4bb6-a253-183748ec306d
• Out of 6,243 organizations, only199 (3.19%) were verified ls-type:: annotation hl-page:: 7 hl-color:: blue id:: 65bb6ca2-b465-42a0-9db1-a743326e3ec8
• Universal Dependencies dataset [79] is the most popular dataset on Hugging Face, with6,834 models depending upon it ls-type:: annotation hl-page:: 8 hl-color:: green id:: 65bb6cf4-d491-4054-9909-ecb3ee21df30