Run record format

A run record is what one recorder writes for one run of an engine: a directory of plain files that any tool can read with gzip, CSV and JSON. The engine’s own file sink and taktwerk record write the same files; taktwerk serve, the Python package and taktwerk-record read them. This page specifies format 1, the format in run.json. How a recorder chooses what to record is in Recording.

Layout

<dir>/<project>/<recorder>/<run-id>/
  run.json                   what was recorded and how the run was started
  part-0001.csv.gz           closed parts: never change again
  part-0002.csv.gz.open      the part being written; renamed without .open once closed
  events.csv                 the run's events
  SHA256SUMS                 one line per closed file
  CLOSED                     empty; written last, once the run is closed

• <dir> is the recorder’s dir (or --record-dir), or --dir of taktwerk record.
• <project> is the project file’s name without extension; <recorder> the recorder’s id (ASCII letters, digits, _, -).
• <run-id> is YYYYMMDDTHHMMSSZ-<hash>: the recorder host’s UTC when the run opened and the first 8 hex digits of the project file’s SHA-256. A second run of the same second gets -2, then -3, and so on.
• Parts are numbered from 0001, four digits at least; the order of the numbers is the order of the rows.

Life of a run

1. The run opens: the directory, run.json (without end), events.csv with its header and part-0001.csv.gz.open.
2. Rows go into the open part. When it closes (rotation or the end of the run) it is finished, synced to disk and renamed to part-NNNN.csv.gz, and <sha256> part-NNNN.csv.gz is appended to SHA256SUMS; the next part opens.
3. The run closes: the last part closes, run.json is replaced with end, the lines for events.csv and run.json are appended to SHA256SUMS, and CLOSED is created.

• run.json is written as run.json.tmp and renamed, never rewritten in place.
• SHA256SUMS lines are <64 lowercase hex> <file name>, the format sha256sum -c checks.
• A run without CLOSED is still recording, or was cut off (SIGKILL, a crash, power). Its last part keeps the .open name and ends at its last flush.
• Retention deletes closed parts of a recorder’s runs, oldest first; their lines stay in SHA256SUMS (sha256sum -c --ignore-missing). run.json, events.csv and SHA256SUMS are never deleted.
• Files ending in .tmp are temporary: readers and sync tools ignore them.

run.json

One JSON object, pretty-printed.

A model’s hash is sha256:<hex> over the package: of the file for an .fmu; for a directory, over the files that make the package in path order, each as its relative path, a NUL byte and its bytes. A raw package’s files are taktwerk-model.toml and everything below lib/; an extracted FMU’s are modelDescription.xml and everything below binaries/ and resources/. Other files in the directory are not part of the package.

invocation:

Absent fields were not used. Device clocks are not trusted: rows carry cycles and engine time, and every UTC timestamp comes from the host that wrote the record.

Parts

A part is a gzip file of CSV text (RFC 4180 quoting, \n line ends): a header line, then one line per row. The gzip header carries no file name and modification time 0.

cycle,t_engine,filt.y[0],filt.y[1],filt.y[2],filt.gain,taktwerk.status
0,0.0,0.9090909090909091,1.9607843137254901,2.9702970297029703,1.0,1
1,0.01,0.8264462809917356,1.9223375624759707,2.9408881482207625,1.0,1

• Header: cycle, t_engine, then each column’s name from run.json, in that order. Every part repeats it. A scalar’s name is its signal; an array element is <signal>[i], a matrix element <signal>[i,j] with logical indices, quoted since it holds a comma; a text signal is one column named like its signal.
• cycle: the tick, an unsigned integer. Consecutive rows differ by every; a larger step is a loss (see Gaps and drops). Skipped ticks have rows.
• t_engine: start_time + cycle · tick_s, seconds.
• Floats (f64, f32): the shortest text that reads back to the same value: 0.1, 1e-7, 1.0, NaN, inf, -inf.
• Integers: decimal. Booleans: 0 or 1.
• Text: the bytes up to the first NUL, as UTF-8 (invalid sequences replaced), quoted when they hold a comma, a quote, CR or LF.
• The stream is flushed every 10 s of engine time and at close, never on wall time, so the same rows give the same bytes.
• A part still being written, or cut off, ends at its last flush without the gzip trailer: a reader decodes until the stream ends and keeps the complete lines.

events.csv

CSV with the header cycle,t_engine,kind,source,detail; one line per event, in the order the engine logged them. A puller’s gap line is written when the gap is found, so it may follow an event of a later cycle. source and detail are quoted when they need it. Flushed about once a second on wall time, so a run still recording may end in half a line.

A value in a detail is written as in the parts; an array is [a, b, c] in storage order, text is in double quotes. A reader should accept kinds it does not know.

Tunables

events.csv holds the value of every tunable at every cycle:

• At init, before the first tick, one start <value> per tunable, in signal order.
• Every change: <old> -> <new> by <writer>, the writer the id of the connector that wrote it. A write of the value in force logs nothing.
• Every refused write: refused <value> by <writer>: <reason> (wrong type, length or text capacity, outside the tunable’s range, or rejected by a client’s mapping). The value in force stays.
• A scenario’s set, add or ramp of a tunable is a scenario event; the scenario is in run.json.

The cycle of a start or a change is the first tick that fetches the value: an instance stepping at that cycle or later steps with it. The value of a tunable at cycle c is that of its last start or change line with a cycle of at most c; the lines of one tunable are in the order of its values.

Reading a record

1. Read run.json; check format.
2. List part-*.csv.gz and part-*.csv.gz.open, sort by name, and decode each as far as its gzip stream goes, keeping complete lines. Check each header against columns.
3. Read events.csv up to its last line break.
4. A run is complete when CLOSED exists; its closed files check with sha256sum -c --ignore-missing SHA256SUMS.

examples/recording/read_run.py does steps 1 and 2 with the Python standard library.

Pull interface

taktwerk record reads a running engine’s recorders through its OPC UA server; another puller can do the same. Per recorder, next to its counters, in the signal namespace (s= NodeIds):

• <prefix>.recorder.<id>.manifest, a String variable: run.json without the run’s own fields (run_id, host, source, start, end), as JSON.
• <prefix>.recorder.<id>.read_rows(from_cycle: UInt64, max_rows: UInt32) returns engine_run (String), oldest_cycle, next_cycle (UInt64) and rows (ByteString). Rows start at the first row at or after from_cycle still held, 0 meaning the oldest. Each row is its cycle and then one word per column word (words in the manifest), all little-endian u64: a float’s bits (f32 in the low 32), an integer sign- or zero-extended, a bool 0 or 1, a text byte per word. A reply holds at most 8 MiB. Ask again from next_cycle; a jump in the cycles is a gap.
• <prefix>.recorder.<id>.read_events(from_seq: UInt64, max: UInt32) returns engine_run, oldest_seq, next_seq, then cycles (UInt64[]), kinds, sources and details (String[]), one entry per event, at most 4096 per call.

<prefix> is the engine’s system_prefix. The methods are called on the recorder’s folder object, NodeId s=<prefix>.recorder.<id>/. A changed engine_run means the engine restarted and its cycles count from 0 again.