9.0 KiB
9.0 KiB
type:: JournalPaper external-links:: Sosym-Systematic benchmarking of ML tools for MDE Rebuttal Letter - Google Docs todoist:: Todoist year:: 2023 deadline-submission:: 12-05-2023 status:: REVISED venue:: SOSYM date-submitted:: 12-05-2023 full-title:: Systematic benchmarking of ML tools in MDE priority:: P4
- ## TODOs
- DONE Vedere sezione 3
date-submitted:: [[09-05-2023]]
- DONE Finalizzare descizione architettura
date-submitted:: [[11-05-2023]]
- DONE GUardare lo spreadsheet di Jesus
date-submitted:: [[11-05-2023]]
- Finalizzare fig 3
- DONE Rivedere parte iniziale sez 4
date-submitted:: [[09-05-2023]]
- DONE Scrivere Conclusioni
date-submitted:: [[09-05-2023]]
- [[10-05-2023]]
- DONE Rivedere Intro
date-submitted::[[10-05-2023]]
- DONE Scrivere [[Abstract]]
date-submitted:: [[10-05-2023]]
- [[12-05-2023]]
- DONE Rivedere sezione 5
date-submitted:: [[12-05-2023]]
- DONE [[people/riccardo]] rivede la sezione related [[WORK]]
date-submitted:: [[12-05-2023]]
- DONE Rivedere sezione 3
date-submitted:: [[12-05-2023]]
- DONE Lettura completa
date-submitted:: [[12-05-2023]]
- DONE Fare figura componenti da inserire in sezione 3
date-submitted:: [[02-05-2023]]
- DONE Rivedere parte scritta da Riccardo
date-submitted:: [[02-05-2023]]
- DONE Listare challenges a fine sezione [[2]]
date-submitted:: [[02-05-2023]]
-
- Ideas
- We could propose a tool demo [[paper]] at MODELS
- [MODELS 2023 - Tools and Demonstrations - MODELS 2023 (researchr.org)](https://conf.researchr.org/track/models-2023/models-2023-tools-and-demonstrations)
- We could even organize an informal hackathon to extend the catalogue
-
- Bisogna descrivere un processo tipico di benchmarking.
- A tale scopo bisogna guardare [microsoft/CodeXGLUE: CodeXGLUE (github.com)](https://github.com/microsoft/CodeXGLUE)
- microsoft/CodeXGLUE: CodeXGLUEmicrosoft/CodeXGLUE: CodeXGLUE - https://github.com/microsoft/CodeXGLUE -
- 23.9 million professional developers in 2019, and the population is expected to reach 28.7 million in 2024
- Microsoft offers the most complete toolchain for developers, bringing together the best of GitHub, Visual Studio, and Microsoft Azure to help developers to go from idea to code and code to cloud.
- **Recent years have seen a surge of applying of statistical models, including neural nets, to code intelligence tasks**
- However, the area of code intelligence lacks a benchmark suite that covers a wide range of tasks.
- **CodeXGLUE, a benchmark dataset and open challenge for code intelligence.**
- collection of code intelligence tasks and a platform for model evaluation and comparison.
-
- The questions / points to elaborate are as follows:
- the need of benchmarking when applying ML tools when automating some process
- how typical benchmarking processes look like
- what are the issues when benchmarking ML tools
-
- Initial draft on notes:
% Benchmarking ML Tools: Why and How
- Machine learning (ML) is a rapidly evolving field that offers many benefits for various domains and applications. However, choosing the right ML tool for a specific task can be challenging, as there are many factors to consider, such as data capacity, training speed, inference speed, and model precision. Therefore, benchmarking ML tools is a crucial step to evaluate their performance and suitability for different scenarios.
- Benchmarking is the practice of comparing tools based on some key performance indicators (KPIs) that reflect their strengths and weaknesses. Benchmarking can help users make informed decisions about which tool to use, as well as identify areas for improvement and optimization. However, benchmarking ML tools is not a straightforward process, as it involves many steps and challenges.
In this blog post, we will elaborate on:
- The need of benchmarking when applying ML tools when automating some process
- How typical benchmarking processes look like
- What are the issues when benchmarking ML tools
The need of benchmarking when applying ML tools when automating some process
ML tools are often used to automate some process that involves data analysis, prediction, or classification. For example, ML tools can be used to automate image recognition, sentiment analysis, fraud detection, or recommendation systems. However, not all ML tools are equally effective or efficient for these tasks. Some tools may have higher accuracy, but lower speed; some tools may handle large datasets better than others; some tools may have more features or functionalities than others.
Therefore, benchmarking ML tools is necessary to measure their performance and compare them against each other or against some baseline. Benchmarking can help users select the best tool for their specific needs and goals, as well as optimize their workflow and resource allocation. Benchmarking can also help developers improve their tools by identifying bottlenecks and gaps in performance.
How typical benchmarking processes look like
A typical benchmarking process consists of the following steps:
- Define the ==objective and scope of the benchmarking==. What is the purpose of the benchmarking? What are the criteria and metrics to evaluate the performance? What are the tools to compare?
- Choose a ==reference dataset and a reference model==. The dataset should be relevant and representative of the task at hand. The model should be appropriate and consistent for the chosen tools.
- Choose a ==reference computer and a reference training strategy==. The computer should have sufficient hardware and software specifications to run the tools. The training strategy should include parameters such as loss function, optimization algorithm, and stopping criterion.
- Run the tools on the dataset using the model and the training strategy. ==Record the results== and metrics for each tool.
- ==Analyze and compare the results==. Use visualizations and statistics to summarize and interpret the results. Identify the strengths and weaknesses of each tool.
-
- What are the issues when benchmarking ML tools
- Benchmarking ML tools is not without challenges. Some of the common issues are:
- The ==lack of standardization and reproducibility==. Different tools may have different implementations, configurations, or versions that affect their performance. Moreover, different datasets, models, or training strategies may also introduce variability and inconsistency in the results. Therefore, it is important to ensure that the benchmarking is fair and reliable by using standardized and reproducible methods.
- The ==trade-off between complexity and simplicity==. Some tools may have more features or functionalities than others, which can make them more powerful or flexible, but also more complex or difficult to use. On the other hand, some tools may be simpler or easier to use, but also more limited or constrained in their capabilities. Therefore, it is important to balance between complexity and simplicity when choosing or evaluating a tool.
- The ==trade-off between accuracy and efficiency==. Some tools may have higher accuracy or precision than others, but also lower speed or scalability. On the other hand, some tools may be faster or more scalable than others, but also less accurate or precise. Therefore, it is important to balance between accuracy and efficiency when choosing or evaluating a tool.
- Conclusion
- Benchmarking ML tools is an essential step to assess their performance and suitability for different tasks and applications. However, benchmarking ML tools is not a simple process, as it involves many steps and challenges. In this blog post, we have elaborated on why benchmarking is needed when applying ML tools when automating some process; how typical benchmarking processes look like; and what are the issues when benchmarking ML tools.
- Machine learning (ML) is a fast-growing field with many applications in various domains and areas including MDE. Selecting the appropriate ML tool for a particular task can be difficult, as different ML tools may have different strengths and weaknesses. In this [[paper]], we presented the \modelxglue framework, which we have designed to facilitate benchmarking ML [[MODELS]] specifically created to address MDE tasks. The framework has been designed to be able to manage different datasets, metrics, and execution environments. The aim is to automate benchmarking processess [[by]] simplifying comparisong processess [[by]] limiting the burden related to the installation and management of the different artifacts that are typically involved. A catalogue of already available [[benchmarks]] has been presented and its execution has been presented.
- [[RelatedWork]]
- [[@Benchmarking Machine Learning Solutions in Production]]
- [[@PMLB: a large benchmark suite for machine learning evaluation and comparison]]