Campaigns

A campaign is a set of simulation runs of one project, planned from one file: the product of named cases, a parameter grid, random samples and replicates, each run one scenario with its own seed. taktwerk campaign prints the plan; with --write it runs it, each run a taktwerk run process of its own, and picks up where it stopped when run again.

taktwerk campaign sweep.toml            # the plan, nothing run
taktwerk campaign sweep.toml --write    # run what is not done

The campaign file

project = "project.toml"         # relative to this file
out = "out/sweep"                # default: the file's stem, beside it
jobs = 2                         # runs at a time; --jobs overrides
speed = "max"                    # --speed of every run: max (default) or a factor
seed = 1                         # the campaign's seed
scenario = "disturbance.toml"    # a scenario every run starts from

[set]                            # overrides of every run
"ctrl.sp" = [1.0]

[stop]                           # stops of every run; every run needs cycles or time_s
time_s = 1200.0

[[event]]                        # events of every run, as in a scenario file
at_s = 300.0
add = "plant.q_dist"
value = 0.2

[[case]]                         # the list axis
name = "nominal"
[[case]]
name = "small-outlet"
scenario = "outlet.toml"         # merged over the common scenario
set = { "plant.k_out" = 0.4 }

[grid]                           # the grid axis: every combination
"ctrl.kp" = [0.5, 1.0, 2.0]
"ctrl.ki" = [0.01, 0.05]

[sample]                         # the random axis: n draws of every variable
n = 50
method = "lhs"                   # Latin hypercube (default) or "uniform"
"plant.gain" = { uniform = [0.8, 1.2] }
"plant.tau" = { log_uniform = [0.5, 2.0] }
"plant.bias" = { normal = [0.0, 0.01] }
"plant.mode" = { choice = [1, 2, 3] }

replicates = 1                   # repeats of every run with another seed

• Runs. The product of the axes present, in the order case, grid point, sample, replicate. A run is named by its axes: small-outlet-g3-s07, or g03 for a grid alone, run for a campaign without axes.
• What a run runs. The common scenario file, then the campaign’s [set], [stop] and events, then the case’s scenario file and set, then the grid point’s and the sample’s values: later values win, stop conditions and events add up. The result is the run’s run.toml, a scenario file that taktwerk run --scenario reruns on its own.
• Seeds. Run seeds derive from the campaign’s seed and the run’s sample and replicate, 63 bits of splitmix64: runs that differ only in case or grid point share their seed, so a comparison of cases or gains sees the same noise (common random numbers). A grid or sample key seed sets the run seed instead.
• Samples. A Latin hypercube puts one draw of each variable in each of n equal-probability strata, shuffled per variable; each variable draws from its own stream of the campaign’s seed, so adding a variable leaves the others’ draws as they were. normal maps through the inverse normal distribution, choice picks one of its values.
• Every run is checked when the plan is made: a bad case name, an empty grid, a bad distribution or a run without a stop limit fails the plan before anything runs.

Running and resuming

Each run is taktwerk run <project> --speed <speed> --no-connectors --scenario run.toml --record-dir <run directory>, jobs at a time; a library with process-wide state is no problem, since every run is its own process.

out/sweep/
  campaign.json                   the plan: file and project hashes, every run's values, seed, spec hash
  runs.csv                        one row per run: axes, seed, status, stop, cycles, wall time, values
  runs/small-outlet-g3-s07/
    run.toml                      the run's scenario
    status.json                   status, exit code, stop, cycles, wall time, sha256 of every record file
    stderr.log                    the run's log
    project/main/<run-id>/        the run's records: run.json, parts, events.csv

• Status. A run is ok when it exits cleanly on a limit or a condition, failed otherwise; a run stopped by SIGINT or SIGTERM is not ok.
• Resume. --write again runs only what is not done: a run is done when its status.json says ok and its spec hash is the planned one. The spec hash covers the taktwerk version, the project file, the project’s model packages and the run’s scenario, so a changed project or a rebuilt package runs everything again. An interrupted or failed run’s directory is cleared and run again.
• Another plan. When the output already holds a plan with other runs, --write refuses; --force clears the runs and starts over.

runs.csv and campaign.json are made to be read back: the Python reader loads a campaign into a table of runs and each run’s signals into arrays.