A summary of possible enhancements #1
Description
Summarized based on the suggestions from Tomas Kalibera.
Should be divided into several small issues.
- A more general harness
- e.g. running also benchmarks that need preprocessed inputs such as fasta based shootouts,
- preprocessing those inputs, repetitions of VM invocations as well as benchmark iterations,
- unification of seed setting for random number generation,
- automated sizing of benchmarks based on a given VM,
- collection of raw measurements and some initial summarization in R,
- some common framework/approach to running on VMs with restricted features like FastR or specific needs like ORBIT - the .internal calls).
- The benchmarks should each support validation of results (which is hard indeed, but at least they should provide an output that could be verified by hand)