logseq/pages/Benchmarks.md

tags:: PROJECTS/MOSAICO, P1

- In one of the ICSE papers that I reviewed [[ICSE2026-paper1656]] there is this reference to look at:
	- >[21] René Just, Darioush Jalali, and Michael D. Ernst. 2014.  **Defects4j: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs**. In International Symposium on Software Testing and Analysis, ISSTA ’14, San Jose, CA, USA - July 21 - 26, 2014. ACM, 437–440.
	- It made me think that when benhmarking AI agents, we need to follow something similar with what we did with Jesus ([ModelXGlue: a benchmarking framework for ML tools in MDE | Software and Systems Modeling](https://link-springer-com.univaq.idm.oclc.org/article/10.1007/s10270-024-01183-z))
	- In particular, this concept of database of existing subjects to be investigated should be made explicit in the language for specifying benchmarks.