4 Benchmarks

4.1 Running Benchmarks

To run a benchmark, use the run.sh script in ~/Desktop. For example, to run the morsecode benchmark, run the command:

./run.sh benchmarks/morsecode

This will produce two relevant files: benchmarks/morsecode.rktd which contains the runtimes for all runs performed and benchmarks/morsecode.png, which contains the L-step N/M usable results as in Figure 4 of the paper.

The run-all.sh script simply calls the run.sh script on every benchmark in the benchmarks/ directory.

Many of the benchmarks take a very long time to run due to the high overheads of contract checking and the large number of runs (exponential in the number of modules). The gregor and quad benchmarks take an especially long time to run all of the lattice configurations so we do not recommend trying to run them to completion.

The rest of the benchmarks are in the following subdirectories:

• gregor/

• kcfa/

• lnm/

• quad/

• sieve/

• snake/

• suffixtree/

• synth/

• tetris/

• zordoz/

Walkthrough: Creating and Analyzing a Benchmark Program contains a short-running benchmark as well.

4.2 Benchmark Structure

A benchmark directory contains at least 2 subdirectories: typed/ and untyped/. The two directories contain the typed and untyped versions of each module in the benchmark program.

The directory may contain 2 other subdirectories: both/ and base/. Both contain files that should not change based on the typed/untyped configuration. These include typed adaptor modules (as explained in section 3.1.1 in the paper), external libraries not included in the configuration space, and non-racket files used in the benchmark. The only difference is where they are accessed they are placed relative to the configurations. Files in the both/ directory will be copied into the directory of each configuration, whereas files in the base/ directory will be copied into a single neighboring base/ directory, i.e., they are accessible by the relative path ../base/.

Typed adaptors need to be in the both/ directory since they use require/typed/check and therefore have different behavior in different configurations. In principle, everything in the base/ directory could be placed in the both/ directory instead but it would be a large waste in storage space and runtime for large benchmarks.