logseq/pages/ReadingNotes/Reproducibility crisis of ML experiments.md

tags

tags
#readingnotes #projects/proposals #machinelearning

The factors that compromise the reproducibility of ML experiments are manifold including the following ones:

• Unavailability or outdated source code
• Unavailability of the datasets used for training and evaluation
• Unavailability of the reference implementation
• Unclear or missing description of the parameters that need to be set for obtaining the presented results
• Missing guidelines on the selection of the training, test and evaluation data
• Missing information of the required libraries/packages and their corresponding version
• Possible tweaks performed in the source code and that are not mentioned in the paper
• Missing information about the underpinning techniques, which have been used
• Lack of documentation about the preprocessing phases, like data preparation and cleaning
• Hardware requirements that could be not satisfied to train the large neural network