The old subprocess.run(capture_output=True) buffered every score result in
memory until the engine exited, then parsed+scored them all at once. If the
engine crashed (or was killed) at request 39/40 you lost everything -- a
multi-hour run with zero information, which is exactly what happened on the
g64 eval against a disk-I/O-bound 400GB model.
Now streams engine stdout line-by-line via Popen:
- each score result lands in the CSV immediately (flushed), with full
provenance (task/qi/oi/gold/logprob/greedy + a header with snap/tasks/
limit/seed/timestamp)
- a [progress] line every 5 requests: N/total, elapsed, req/s, ETA, last
request scored
- the engine's own [score N req] stderr heartbeat streams live too
- on partial completion (crash/kill at request N): keeps 1..N-1, fills the
rest with -inf, and scores the partial results -- never a wasted run
New --out <path> writes the incremental CSV; without it, behaves as before
(results to stdout only). Backward compatible: all existing args unchanged.
woolcoxm
86987154f5