expfile.md

Experiment files

In dimebox, experiments are specified with YAML files called expfiles. This document will explain the possible fields used to create an expfile.

Name and description

These two fields together exist to make it easy to identify the kind of experiment that you will be running. Example:

name: my_experiment
desc: Testing various parameters of k on quality of result

The name field is used when computing each job name, as well as the names of the job files. The name should not contain any characters that are unwieldy to deal with in filenames, such as spaces and tabs.

The description field allows the user to give a more detailed description about the kind of experiment that will be run. This is an opportunity to provide detail at high level about what parameters are used, what changes to the code were made, and other information that will be helpful when looking at how this experiment differs from the others that have run. This field's sole purpose is to make it easy for the user to understand the context of this experiment when returning to it later.

Processor counts

The p field specifies which processor counts this experiment will use. Example:

p: [1, 2, 4, 8, 16, 32]

There will be one job created for each processor count.

One can specify the number of threads for each processor count with the depth field (depthvar is the environment variable used to specify threads). For example:

depth: [1, 2, 4]
depthvar: OMP_NUM_THREADS

Commands

The commands object specifies which executables will be run and with which parameters. Example:

cmds:
  foo: ./myexe 1000
   bar: ./yourexe 1000

As the commands have multiple key-value pairs, this allows for a job to run multiple executables. Each executable's output will be stored in a different file, where the filename is derived from the keys in this object. In our example, the output of myexe will be stored in a file starting with 'foo' and the output of yourexe will be in a file starting with 'bar.'

In each command, arguments that are specified in optargs or weakargs will be available for use, with the name of the argument preceded by a dollar sign (e.g., $arg).

Arguments

There are three kinds of arguments in dimebox, optargs, pairargs and weakargs

optargs

This field allows the user to specify a matrix of arguments that will be used to create jobs. Example:

optargs: 
  tau: [1, 16, 64, 256, 4096]
  k: [0, 1, 2, 4, 8]

In this example, there are two different arguments, each with 5 possibilities, allowing for 25 different combinations of arguments. These arguments can be specified in a command like so:

cmds:
  foo: ./myexe $tau $k

pairargs

This field allows the user to specify lists of arguments. Unlike optargs, the cross product isn't used to combine these arguments. Example:

pairargs:
  n: [1,2,3]
  m: [4,5,6]
  s: [7,8,9]

This will create only 3 jobs, with n = 1 and m = 4 and s = 7, n = 2 and m = 5 and s = 8, etc.

Combining this with optargs takes the cartesian product of both. Example:

optargs:
  tau: [1, 16, 64, 256, 4096]
  k: [0, 1, 2, 4, 8]
pairargs:
  n: [1,2,3]
  m: [4,5,6]
  s: [7,8,9]

As stated above, there are 25 different combinations of the optargs and 3 for the pairargs so there will be 75 jobs created.

weakargs

Weak arguments are arguments that depend on the processor count p. Example:

weakargs:
  scale: 11 + log2(p)

This will create a $scale argument whose value depends on the value of p. Note that each weakarg is essentially a JavaScript expression that will be evaluated during generation. It is preprocessed by appending Math or Number in front of every identifier that isn't p. If there are any errors in this expression, or if it does not generate a number or string, then generation will fail.

Walltime

To change the walltime for the jobs in an experiment, use the wall keyword:

wall: 03:00:00

Queues

The q field exists to specify queues for your jobs. Example:

q:
  small: 16
  med: 128
  big: 512

The q field should be key-value pairs of queue names and processor counts. For each job with a given processor count, a queue will be computed for that job based on the smallest queue that is less than or equal to the processor counts given in q.

In our example, a job with 16 processors will be assigned to the small queue, whereas a job with 32 processors will be assigned to the med queue. If a processor count higher than 512 is specified, generation will abort.

Environment variables

Environment variables can be specified with the env field. For example, if you wanted to specify export BG_MAPCOMMONHEAP=1, you would add the following to your expfile:

env:
  BG_MAPCOMMONHEAP: 1

Trials

You can specify that the commands in your experiment be run multiple times using the trials field:

trials: 32

This will run the commands 32 times and the result of each command will be appended to your output file.

Raw headers and flags

If dimebox doesn't include support for a flag that should be in your job file, it's possible to explicitly specify a flag that should be in a job's header, or called with the MPIRUN command:

raw:
  headers: ['--myheader ${p*5}']
  runFlags: ["-S ${Math.log2(p)}"]

This will result in a job, where p=8, that may include the following:

#PBS --myheader 40
mpirun -np 8 -S 3 ...

You can evaluate expressions inside of ${}, where the values of p and depth for a given job are available.

Prologue and epilogue

Commands that happen at the beginning and end of your job can be specified with the prologue and epilogue fields. Example:

prologue: 'echo "Subject: starting job" | sendmail -v my@email.com'
epilogue: rm core.*

Workspaces

Specifying a workspace in your expfile will allow jobs to be run in their own isolated directory. Example:

workspace:
  links: [myexe]

The nested links field should provide an array of symbolic links that will be avaiable in the isolated directory. Links are relative to the current working directory.