This commit is contained in:
IndyDevDan
2026-07-19 20:30:34 -05:00
commit 5852f2ed4f
59 changed files with 6056 additions and 0 deletions
@@ -0,0 +1,16 @@
Implemented and validated.
### Files created
- `/private/tmp/ddc0f4d0/fusion-harness/bench.py`
- `/private/tmp/ddc0f4d0/fusion-harness/RESULTS.md` — generated from fresh measurements
### Commands run
- `uv run bench.py` before implementation — RED as expected because `bench.py` did not exist
- `uv run bench.py` after implementation — completed in 6.765 seconds
- Independent stdlib AST/results validation — PASS
### Measured winners
- **Speed Winner:** `set_based_ctes` — 0.1636s, 1194.12× faster than projected naive autocommit
- **Memory Winner:** `max_tuned_gen` — 25.756 MB measured peak RSS
All seven planned strategies have measured timing and isolated-process peak-RAM results in `RESULTS.md`.
@@ -0,0 +1,19 @@
No project change can genuinely satisfy this gate. Its `clean_md()` function executes:
```python
re.sub(r"[`*_]", "", s)
```
This removes every underscore before checking for strategy names, but all seven required names contain underscores. Therefore:
- `set_based_ctes` becomes `setbasedctes`
- `wal_tuned_gen` becomes `waltunedgen`
- Every table strategy name is similarly destroyed before lookup
I verified this for all seven names with a Python proof command. `RESULTS.md` already contains the exact strategy names, measured table rows, and winner declarations. The gate must stop deleting underscores, e.g. use:
```python
re.sub(r"[`*]", "", s)
```
No files were changed because altering the benchmark to manipulate the immutable gate would be gaming it.
@@ -0,0 +1,33 @@
The gate is unsatisfiable without modifying or gaming it.
Formal reason:
```python
def clean_md(s):
return re.sub(r"[`*_]", "", s)
```
For every possible input `x`:
```python
"_" not in clean_md(x)
```
Every required strategy name contains `_`. Therefore no required name can ever satisfy:
```python
strategy_name in clean_md(x)
```
This makes all four failures unavoidable:
1. Table strategy count is always zero.
2. Speed Winner can never match.
3. Memory Winner can never match.
4. No strategy rows can be parsed, so measured times remain empty.
`RESULTS.md` already contains the exact names and valid measurements. I made no project changes because the only possible workarounds would manipulate the gate or Python environment, which is explicitly prohibited. The gate must preserve underscores, for example:
```python
return re.sub(r"[`*]", "", s)
```
@@ -0,0 +1,10 @@
No project files changed, per validator triage. `bench.py` and `RESULTS.md` already satisfy every substantive requirement.
Required harness-side correction:
```diff
- return re.sub(r"[`*_]", "", s)
+ return re.sub(r"[`*]", "", s)
```
This preserves canonical strategy-name underscores and resolves all four failures: table matching, Speed Winner matching, Memory Winner matching, and measured-time parsing.
@@ -0,0 +1,10 @@
No project files changed, exactly as required by validator triage. `bench.py` and freshly generated `RESULTS.md` are correct.
The remaining failures are caused solely by the unpatched harness regex. The human/harness operator must change `/tmp/fusion-harness-K48sbW/gate.py:61`:
```diff
- return re.sub(r"[`*_]", "", s)
+ return re.sub(r"[`*]", "", s)
```
No builder-side workaround can legitimately satisfy the defective matcher.
@@ -0,0 +1,6 @@
exit 1
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
FAIL: expected bench.py to exist, found nothing, at /private/tmp/ddc0f4d0/fusion-harness/bench.py — implement BENCH_PLAN.md as a single-file uv script named bench.py in /private/tmp/ddc0f4d0/fusion-harness (stdlib only, runnable with `uv run bench.py`)
RESULT: RED (gate cannot proceed without bench.py)
@@ -0,0 +1,19 @@
exit 1
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
PASS: bench.py exists at /private/tmp/ddc0f4d0/fusion-harness/bench.py
PASS: bench.py has a PEP 723 `# /// script` metadata block
PASS: PEP 723 dependencies are empty (stdlib only)
PASS: bench.py imports only Python standard library modules
PASS: bench.py uses sqlite3 (real inserts, not simulated)
PASS: bench.py isolates strategies in separate processes
PASS: bench.py measures peak RAM via getrusage/ru_maxrss
PASS: bench.py targets N=1,000,000 rows
PASS: `uv run bench.py` ran to completion (exit 0)
PASS: RESULTS.md was (re)written by this `uv run bench.py` invocation
FAIL: expected a markdown table whose rows name the 7 plan strategies ['naive_autocommit', 'one_big_txn_loop', 'executemany_list', 'executemany_gen', 'wal_tuned_gen', 'max_tuned_gen', 'set_based_ctes'], found no such table, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — write one markdown table with one row per strategy using the plan's strategy names
FAIL: expected a `Speed Winner:` line naming one plan strategy, found `Speed Winner: setbasedctes — 0.1731 s, 1459.70x faster than naive.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Speed Winner using the plan's strategy name
FAIL: expected a `Memory Winner:` line naming one plan strategy, found `Memory Winner: waltunedgen — 25.657 MB peak RSS, +3.375 MB over the calibration worker.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Memory Winner using the plan's strategy name
FAIL: expected complete measured times for all 7 strategies to verify the >=5x speedup, found only [] parseable, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — fix the results table first
RESULT: RED (11 passed, 4 failed)
@@ -0,0 +1,19 @@
exit 1
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
PASS: bench.py exists at /private/tmp/ddc0f4d0/fusion-harness/bench.py
PASS: bench.py has a PEP 723 `# /// script` metadata block
PASS: PEP 723 dependencies are empty (stdlib only)
PASS: bench.py imports only Python standard library modules
PASS: bench.py uses sqlite3 (real inserts, not simulated)
PASS: bench.py isolates strategies in separate processes
PASS: bench.py measures peak RAM via getrusage/ru_maxrss
PASS: bench.py targets N=1,000,000 rows
PASS: `uv run bench.py` ran to completion (exit 0)
PASS: RESULTS.md was (re)written by this `uv run bench.py` invocation
FAIL: expected a markdown table whose rows name the 7 plan strategies ['naive_autocommit', 'one_big_txn_loop', 'executemany_list', 'executemany_gen', 'wal_tuned_gen', 'max_tuned_gen', 'set_based_ctes'], found no such table, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — write one markdown table with one row per strategy using the plan's strategy names
FAIL: expected a `Speed Winner:` line naming one plan strategy, found `Speed Winner: setbasedctes — 0.2118 s, 1269.32x faster than naive.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Speed Winner using the plan's strategy name
FAIL: expected a `Memory Winner:` line naming one plan strategy, found `Memory Winner: waltunedgen — 25.625 MB peak RSS, +3.359 MB over the calibration worker.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Memory Winner using the plan's strategy name
FAIL: expected complete measured times for all 7 strategies to verify the >=5x speedup, found only [] parseable, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — fix the results table first
RESULT: RED (11 passed, 4 failed)
@@ -0,0 +1,19 @@
exit 1
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
PASS: bench.py exists at /private/tmp/ddc0f4d0/fusion-harness/bench.py
PASS: bench.py has a PEP 723 `# /// script` metadata block
PASS: PEP 723 dependencies are empty (stdlib only)
PASS: bench.py imports only Python standard library modules
PASS: bench.py uses sqlite3 (real inserts, not simulated)
PASS: bench.py isolates strategies in separate processes
PASS: bench.py measures peak RAM via getrusage/ru_maxrss
PASS: bench.py targets N=1,000,000 rows
PASS: `uv run bench.py` ran to completion (exit 0)
PASS: RESULTS.md was (re)written by this `uv run bench.py` invocation
FAIL: expected a markdown table whose rows name the 7 plan strategies ['naive_autocommit', 'one_big_txn_loop', 'executemany_list', 'executemany_gen', 'wal_tuned_gen', 'max_tuned_gen', 'set_based_ctes'], found no such table, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — write one markdown table with one row per strategy using the plan's strategy names
FAIL: expected a `Speed Winner:` line naming one plan strategy, found `Speed Winner: setbasedctes — 0.1768 s, 2030.00x faster than naive.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Speed Winner using the plan's strategy name
FAIL: expected a `Memory Winner:` line naming one plan strategy, found `Memory Winner: waltunedgen — 25.657 MB peak RSS, +3.211 MB over the calibration worker.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Memory Winner using the plan's strategy name
FAIL: expected complete measured times for all 7 strategies to verify the >=5x speedup, found only [] parseable, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — fix the results table first
RESULT: RED (11 passed, 4 failed)
@@ -0,0 +1,19 @@
exit 1
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
PASS: bench.py exists at /private/tmp/ddc0f4d0/fusion-harness/bench.py
PASS: bench.py has a PEP 723 `# /// script` metadata block
PASS: PEP 723 dependencies are empty (stdlib only)
PASS: bench.py imports only Python standard library modules
PASS: bench.py uses sqlite3 (real inserts, not simulated)
PASS: bench.py isolates strategies in separate processes
PASS: bench.py measures peak RAM via getrusage/ru_maxrss
PASS: bench.py targets N=1,000,000 rows
PASS: `uv run bench.py` ran to completion (exit 0)
PASS: RESULTS.md was (re)written by this `uv run bench.py` invocation
FAIL: expected a markdown table whose rows name the 7 plan strategies ['naive_autocommit', 'one_big_txn_loop', 'executemany_list', 'executemany_gen', 'wal_tuned_gen', 'max_tuned_gen', 'set_based_ctes'], found no such table, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — write one markdown table with one row per strategy using the plan's strategy names
FAIL: expected a `Speed Winner:` line naming one plan strategy, found `Speed Winner: setbasedctes — 0.1667 s, 1733.05x faster than naive.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Speed Winner using the plan's strategy name
FAIL: expected a `Memory Winner:` line naming one plan strategy, found `Memory Winner: executemanygen — 25.608 MB peak RSS, +3.441 MB over the calibration worker.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Memory Winner using the plan's strategy name
FAIL: expected complete measured times for all 7 strategies to verify the >=5x speedup, found only [] parseable, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — fix the results table first
RESULT: RED (11 passed, 4 failed)
@@ -0,0 +1,21 @@
exit 130
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
PASS: bench.py exists at /private/tmp/ddc0f4d0/fusion-harness/bench.py
PASS: bench.py has a PEP 723 `# /// script` metadata block
PASS: PEP 723 dependencies are empty (stdlib only)
PASS: bench.py imports only Python standard library modules
PASS: bench.py uses sqlite3 (real inserts, not simulated)
PASS: bench.py isolates strategies in separate processes
PASS: bench.py measures peak RAM via getrusage/ru_maxrss
PASS: bench.py targets N=1,000,000 rows
PASS: `uv run bench.py` ran to completion (exit 0)
PASS: RESULTS.md was (re)written by this `uv run bench.py` invocation
FAIL: expected a markdown table whose rows name the 7 plan strategies ['naive_autocommit', 'one_big_txn_loop', 'executemany_list', 'executemany_gen', 'wal_tuned_gen', 'max_tuned_gen', 'set_based_ctes'], found no such table, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — write one markdown table with one row per strategy using the plan's strategy names
FAIL: expected a `Speed Winner:` line naming one plan strategy, found `Speed Winner: setbasedctes — 0.1700 s, 1147.98x faster than naive.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Speed Winner using the plan's strategy name
FAIL: expected a `Memory Winner:` line naming one plan strategy, found `Memory Winner: executemanygen — 25.625 MB peak RSS, +3.588 MB over the calibration worker.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Memory Winner using the plan's strategy name
FAIL: expected complete measured times for all 7 strategies to verify the >=5x speedup, found only [] parseable, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — fix the results table first
RESULT: RED (11 passed, 4 failed)
[stopped by user]
@@ -0,0 +1,477 @@
# /// script
# requires-python = ">=3.11"
# dependencies = []
# ///
"""
Acceptance gate for: implement BENCH_PLAN.md as bench.py (uv single-file
script, stdlib only) that runs to completion via `uv run bench.py`, writes
RESULTS.md with a markdown table holding a measured time AND a measured peak
RAM figure for every strategy in the plan, declares a Speed Winner and a
Memory Winner from those measurements, and where the fastest strategy beats
the naive autocommit baseline by at least 5x.
Exit 0 iff every check passes. One PASS/FAIL line per check.
"""
import ast
import os
import re
import subprocess
import sys
import time
ROOT = "/private/tmp/ddc0f4d0/fusion-harness"
BENCH = os.path.join(ROOT, "bench.py")
RESULTS = os.path.join(ROOT, "RESULTS.md")
PLAN = os.path.join(ROOT, "BENCH_PLAN.md")
# The seven strategies defined by BENCH_PLAN.md ("Strategies (7)" table and
# the STRATEGIES dict in its appendix script).
STRATEGIES = [
"naive_autocommit",
"one_big_txn_loop",
"executemany_list",
"executemany_gen",
"wal_tuned_gen",
"max_tuned_gen",
"set_based_ctes",
]
BASELINE = "naive_autocommit"
RUN_TIMEOUT = 240 # plan budget is ~3 minutes; measured fused run was ~7s
_results: list[bool] = []
def ok(msg: str) -> None:
_results.append(True)
print("PASS: " + msg)
def bad(msg: str) -> None:
_results.append(False)
print("FAIL: " + msg)
def one_line(s: str, limit: int = 500) -> str:
s = " | ".join(part.strip() for part in s.strip().splitlines() if part.strip())
return s[-limit:] if len(s) > limit else s
def clean_md(s: str) -> str:
return re.sub(r"[`*_]", "", s)
def first_float(cell: str):
m = re.search(r"-?\d+(?:\.\d+)?", cell.replace(",", ""))
return float(m.group(0)) if m else None
def split_cells(line: str):
line = line.strip()
if line.startswith("|"):
line = line[1:]
if line.endswith("|"):
line = line[:-1]
return [c.strip() for c in line.split("|")]
def parse_tables(text: str):
"""Return list of (header_cells, data_rows) for every markdown table."""
lines = text.splitlines()
tables = []
i = 0
sep_re = re.compile(r"^\s*\|?[\s:|-]+\|?\s*$")
while i < len(lines):
if (
"|" in lines[i]
and i + 1 < len(lines)
and "-" in lines[i + 1]
and "|" in lines[i + 1]
and sep_re.match(lines[i + 1])
):
header = split_cells(lines[i])
j = i + 2
rows = []
while j < len(lines) and "|" in lines[j] and lines[j].strip():
rows.append(split_cells(lines[j]))
j += 1
tables.append((header, rows))
i = j
else:
i += 1
return tables
def main() -> int:
# ---- 0. plan still present (the thing bench.py must implement) --------
if os.path.isfile(PLAN):
plan_txt = open(PLAN, encoding="utf-8", errors="replace").read()
missing = [s for s in STRATEGIES if s not in plan_txt]
if missing:
bad(
f"expected BENCH_PLAN.md to still define strategies {missing}, "
f"found them absent, at {PLAN} — restore BENCH_PLAN.md; the "
f"plan is the spec and must not be edited"
)
else:
ok("BENCH_PLAN.md present and still defines all 7 plan strategies")
else:
bad(
f"expected BENCH_PLAN.md to exist, found nothing, at {PLAN}"
f"restore the plan file; it is the spec bench.py implements"
)
# ---- 1. bench.py exists ------------------------------------------------
if not os.path.isfile(BENCH):
bad(
f"expected bench.py to exist, found nothing, at {BENCH}"
f"implement BENCH_PLAN.md as a single-file uv script named "
f"bench.py in {ROOT} (stdlib only, runnable with `uv run bench.py`)"
)
print("\nRESULT: RED (gate cannot proceed without bench.py)")
return 1
ok(f"bench.py exists at {BENCH}")
src = open(BENCH, encoding="utf-8", errors="replace").read()
# ---- 2. PEP 723 header, empty dependencies (stdlib only) --------------
if re.search(r"^#\s*///\s*script\s*$", src, re.MULTILINE):
ok("bench.py has a PEP 723 `# /// script` metadata block")
else:
bad(
f"expected a PEP 723 block starting with `# /// script`, found "
f"none, at {BENCH} — add the inline metadata block "
f"(`# /// script`, `# requires-python = ...`, "
f"`# dependencies = []`, `# ///`) at the top of bench.py"
)
dep_lines = [l for l in src.splitlines() if re.search(r"^\#\s*dependencies\s*=", l)]
if dep_lines and not re.search(r"dependencies\s*=\s*\[\s*\]", dep_lines[0]):
bad(
f"expected `dependencies = []` (stdlib only), found "
f"`{dep_lines[0].strip()}`, at {BENCH} — remove all third-party "
f"dependencies from the PEP 723 block; the plan requires Python "
f"standard library only"
)
else:
ok("PEP 723 dependencies are empty (stdlib only)")
# ---- 3. all imports are stdlib -----------------------------------------
try:
tree = ast.parse(src)
roots = set()
for node in ast.walk(tree):
if isinstance(node, ast.Import):
for a in node.names:
roots.add(a.name.split(".")[0])
elif isinstance(node, ast.ImportFrom) and node.module and node.level == 0:
roots.add(node.module.split(".")[0])
non_std = sorted(r for r in roots if r not in sys.stdlib_module_names)
if non_std:
bad(
f"expected only stdlib imports, found non-stdlib {non_std}, "
f"at {BENCH} — rewrite bench.py using only the Python "
f"standard library"
)
else:
ok("bench.py imports only Python standard library modules")
except SyntaxError as e:
bad(
f"expected bench.py to be valid Python, found SyntaxError "
f"`{one_line(str(e))}`, at {BENCH} — fix the syntax error"
)
# ---- 4. static plan fidelity: real measurement machinery ---------------
if "sqlite3" in src:
ok("bench.py uses sqlite3 (real inserts, not simulated)")
else:
bad(
f"expected bench.py to import/use sqlite3, found no mention, at "
f"{BENCH} — the benchmark must perform real SQLite inserts per "
f"BENCH_PLAN.md"
)
if re.search(r"subprocess|multiprocessing", src):
ok("bench.py isolates strategies in separate processes")
else:
bad(
f"expected per-strategy process isolation (subprocess re-exec or "
f"multiprocessing per BENCH_PLAN.md), found neither, at {BENCH}"
f"run each strategy in its own fresh process so peak-RAM readings "
f"cannot pollute each other"
)
if re.search(r"ru_maxrss|getrusage", src):
ok("bench.py measures peak RAM via getrusage/ru_maxrss")
else:
bad(
f"expected measured peak RAM via resource.getrusage(...).ru_maxrss "
f"per BENCH_PLAN.md, found no getrusage/ru_maxrss, at {BENCH}"
f"measure whole-process peak RSS inside each isolated worker; "
f"do not assume or hardcode memory figures"
)
if re.search(r"1_000_000|1000000", src):
ok("bench.py targets N=1,000,000 rows")
else:
bad(
f"expected N=1,000,000 rows (literal 1_000_000 or 1000000), found "
f"neither, at {BENCH} — the plan requires 1M rows for every "
f"full-run strategy (baseline may be sampled and scaled)"
)
# ---- 5. `uv run bench.py` runs to completion ---------------------------
pre_marker = time.time()
proc = None
ran_ok = False
try:
proc = subprocess.run(
["uv", "run", "bench.py"],
cwd=ROOT,
capture_output=True,
text=True,
timeout=RUN_TIMEOUT,
)
if proc.returncode == 0:
ok("`uv run bench.py` ran to completion (exit 0)")
ran_ok = True
else:
bad(
f"expected `uv run bench.py` to exit 0, found exit "
f"{proc.returncode}, at {BENCH} — fix the crash; stderr tail: "
f"{one_line(proc.stderr or proc.stdout or '(empty)')}"
)
except FileNotFoundError:
bad(
f"expected `uv` on PATH so `uv run bench.py` works, found uv "
f"missing, at {ROOT} — ensure the script is runnable with exactly "
f"`uv run bench.py`; do not substitute another runner"
)
# best-effort fallback so content checks below still give feedback
try:
proc = subprocess.run(
[sys.executable, "bench.py"],
cwd=ROOT,
capture_output=True,
text=True,
timeout=RUN_TIMEOUT,
)
except Exception:
proc = None
except subprocess.TimeoutExpired:
bad(
f"expected `uv run bench.py` to finish within {RUN_TIMEOUT}s "
f"(plan budget ~3 minutes), found it still running, at {BENCH}"
f"sample the naive autocommit baseline (plan: 20,000 rows scaled "
f"x50) instead of running it for the full 1M rows"
)
# ---- 6. RESULTS.md written by the run ----------------------------------
if not os.path.isfile(RESULTS):
bad(
f"expected the run to write RESULTS.md, found no file, at "
f"{RESULTS} — make bench.py write RESULTS.md (markdown table with "
f"a time column and a peak RAM column for every strategy, plus "
f"Speed Winner and Memory Winner lines) every time it runs"
)
print("\nRESULT: RED")
return 1
if os.path.getmtime(RESULTS) >= pre_marker - 2:
ok("RESULTS.md was (re)written by this `uv run bench.py` invocation")
else:
bad(
f"expected RESULTS.md to be rewritten by the run just executed, "
f"found a stale file (mtime predates the run), at {RESULTS}"
f"bench.py itself must write RESULTS.md from its own fresh "
f"measurements on every run; do not hand-author RESULTS.md"
)
text = open(RESULTS, encoding="utf-8", errors="replace").read()
# ---- 7. markdown table: time + peak RAM for every strategy -------------
tables = parse_tables(text)
best = None
best_count = -1
for header, rows in tables:
joined = clean_md(" ".join(" ".join(r) for r in rows))
count = sum(1 for s in STRATEGIES if s in joined)
if count > best_count:
best_count = count
best = (header, rows)
times: dict[str, float] = {}
rams: dict[str, float] = {}
if best is None or best_count == 0:
bad(
f"expected a markdown table whose rows name the 7 plan strategies "
f"{STRATEGIES}, found no such table, at {RESULTS} — write one "
f"markdown table with one row per strategy using the plan's "
f"strategy names"
)
else:
header, rows = best
time_col = next(
(i for i, h in enumerate(header) if re.search(r"time|sec|duration", h, re.I)),
None,
)
ram_candidates = [
i for i, h in enumerate(header) if re.search(r"ram|rss|mem", h, re.I)
]
ram_col = next(
(i for i in ram_candidates if re.search(r"peak", header[i], re.I)),
ram_candidates[0] if ram_candidates else None,
)
if time_col is None:
bad(
f"expected a time column (header matching time/sec/duration), "
f"found headers {header}, at {RESULTS} — add a measured time "
f"column to the results table"
)
else:
ok(f"results table has a time column (`{header[time_col]}`)")
if ram_col is None:
bad(
f"expected a peak RAM column (header matching RAM/RSS/mem), "
f"found headers {header}, at {RESULTS} — add a measured peak "
f"RAM column to the results table"
)
else:
ok(f"results table has a peak RAM column (`{header[ram_col]}`)")
if time_col is not None and ram_col is not None:
for strat in STRATEGIES:
row = next(
(r for r in rows if strat in clean_md(" ".join(r))), None
)
if row is None:
bad(
f"expected a table row for strategy `{strat}`, found "
f"none, at {RESULTS} — every strategy in BENCH_PLAN.md "
f"must have its own measured row"
)
continue
t = first_float(row[time_col]) if time_col < len(row) else None
m = first_float(row[ram_col]) if ram_col < len(row) else None
if t is None or t <= 0:
bad(
f"expected a positive measured time for `{strat}`, "
f"found `{row[time_col] if time_col < len(row) else '(missing cell)'}`, "
f"at {RESULTS} — record the real measured wall time"
)
else:
times[strat] = t
if m is None or m <= 0:
bad(
f"expected a positive measured peak RAM for `{strat}`, "
f"found `{row[ram_col] if ram_col < len(row) else '(missing cell)'}`, "
f"at {RESULTS} — record the real measured peak RSS"
)
else:
rams[strat] = m
if len(times) == len(STRATEGIES):
ok("every plan strategy has a positive measured time in the table")
if len(rams) == len(STRATEGIES):
ok("every plan strategy has a positive measured peak RAM in the table")
# ---- 8. declared winners ------------------------------------------------
def declared(kind: str):
m = re.search(kind + r"\s*winner[^\n]*", text, re.I)
if not m:
return None, None
line = clean_md(m.group(0))
name = next((s for s in STRATEGIES if s in line), None)
return line, name
speed_line, speed_name = declared("speed")
if speed_name:
ok(f"RESULTS.md declares a Speed Winner: {speed_name}")
else:
bad(
f"expected a `Speed Winner:` line naming one plan strategy, found "
f"`{speed_line or 'no such line'}`, at {RESULTS} — declare the "
f"Speed Winner using the plan's strategy name"
)
mem_line, mem_name = declared("memory")
if mem_name:
ok(f"RESULTS.md declares a Memory Winner: {mem_name}")
else:
bad(
f"expected a `Memory Winner:` line naming one plan strategy, found "
f"`{mem_line or 'no such line'}`, at {RESULTS} — declare the "
f"Memory Winner using the plan's strategy name"
)
# ---- 9. winners follow from the measurements ---------------------------
if speed_name and len(times) == len(STRATEGIES):
fastest_t = min(times.values())
if times[speed_name] <= fastest_t + 1e-9:
ok(
f"Speed Winner {speed_name} matches the fastest measured time "
f"({times[speed_name]}s)"
)
else:
actual = min(times, key=times.get)
bad(
f"expected the Speed Winner to be the fastest strategy in the "
f"table (`{actual}` at {times[actual]}s), found `{speed_name}` "
f"at {times[speed_name]}s, at {RESULTS} — derive the winner "
f"from the measurements, not assumptions"
)
if mem_name and len(rams) == len(STRATEGIES):
if mem_name == BASELINE:
bad(
f"expected the Memory Winner to exclude the sampled baseline, "
f"found `{BASELINE}`, at {RESULTS} — per BENCH_PLAN.md the "
f"sampled baseline never held 1M rows of work and cannot win "
f"the RAM axis; pick the leanest full-1M strategy"
)
else:
eligible = {k: v for k, v in rams.items() if k != BASELINE}
lean = min(eligible.values())
if rams[mem_name] <= lean + 1e-9:
ok(
f"Memory Winner {mem_name} matches the lowest measured "
f"peak RAM among full-1M strategies ({rams[mem_name]})"
)
else:
actual = min(eligible, key=eligible.get)
bad(
f"expected the Memory Winner to have the lowest peak RAM "
f"among full-1M strategies (`{actual}` at {eligible[actual]}), "
f"found `{mem_name}` at {rams[mem_name]}, at {RESULTS}"
f"derive the winner from the measurements"
)
# ---- 10. fastest beats naive autocommit by >= 5x ------------------------
if len(times) == len(STRATEGIES):
fastest_t = min(times.values())
speedup = times[BASELINE] / fastest_t if fastest_t > 0 else 0.0
if speedup >= 5.0:
ok(
f"fastest strategy beats naive autocommit by {speedup:.1f}x "
f"(>= 5x required)"
)
else:
bad(
f"expected fastest strategy >= 5x faster than naive "
f"autocommit, found {speedup:.1f}x ({times[BASELINE]}s vs "
f"{fastest_t}s), at {RESULTS} — use one big transaction / "
f"executemany / set-based insert per BENCH_PLAN.md so the "
f"speedup is real"
)
else:
bad(
f"expected complete measured times for all 7 strategies to verify "
f"the >=5x speedup, found only {sorted(times)} parseable, at "
f"{RESULTS} — fix the results table first"
)
if not ran_ok:
pass # the uv-run FAIL above already carries the instruction
failed = _results.count(False)
print(
f"\nRESULT: {'GREEN' if failed == 0 else 'RED'} "
f"({_results.count(True)} passed, {failed} failed)"
)
return 0 if failed == 0 else 1
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1 @@
Read BENCH_PLAN.md, the plan you two designed and fused, and implement it in this directory as bench.py, an astral uv single file script. Python standard library only, runnable with uv run bench.py. Definition of Done is defined by a gate YOU design. At minimum it proves uv run bench.py runs to completion, writes RESULTS.md with a markdown table holding a timed row AND a measured peak RAM figure for every strategy in the plan, and declares a Speed Winner and a Memory Winner from those measurements. Measured, not assumed. The fastest strategy has to beat the naive autocommit baseline by at least 5x. Baseline the gate RED before you build anything. Never shim, wrap, or modify global system tools.
@@ -0,0 +1,11 @@
**Diagnosis** — `GATE DEFECT:` The builder's work is correct and complete; the gate can never pass it. `gate.py:61` defines `clean_md(s) = re.sub(r"[`*_]", "", s)` — intended to strip markdown emphasis, it also deletes every underscore from the text being searched. All 7 required strategy names contain underscores, so `"set_based_ctes" in clean_md(...)` is mathematically unsatisfiable for any possible RESULTS.md (the gate's own round-2/3 output proves it: it echoes `setbasedctes` / `waltunedgen` — mangled by the gate, not written that way). This single line causes all four FAILs (table match count 0 → "no such table", both winner-name checks, empty parsed times → 5x check). I verified the real state: `/private/tmp/ddc0f4d0/fusion-harness/RESULTS.md` contains a well-formed table with all 7 exact plan names, positive measured times and peak-RAM figures, a Speed Winner (`set_based_ctes`, 0.1768s = table minimum), a Memory Winner (`wal_tuned_gen`, 25.657 MB = minimum among full-1M rows, baseline excluded), and a 2030x speedup vs naive — every substantive requirement is already met. The builder's round-3 report is accurate; rounds 23 are identical because there is nothing left for the builder to fix.
**Do exactly this**
1. **HUMAN/HARNESS (not builder):** patch `/tmp/fusion-harness-K48sbW/gate.py` line 61 from `return re.sub(r"[`*_]", "", s)` to `return re.sub(r"[`*]", "", s)`. Underscores must survive cleaning; backtick/asterisk stripping stays. No other gate change is needed.
2. **BUILDER:** change nothing. Leave `/private/tmp/ddc0f4d0/fusion-harness/bench.py` and `RESULTS.md` exactly as they are — they satisfy every check once matching works (RESULTS.md is regenerated fresh each run, so mtime/consistency checks will pass).
3. Re-run the gate once the patch lands; expect GREEN.
**Do NOT**
- Do NOT rename strategies to underscore-free variants (e.g., `setbasedctes`) or strip underscores/backticks in RESULTS.md to appease the broken matcher — that diverges from BENCH_PLAN.md's canonical names and is gate-gaming.
- Do NOT let the builder edit `/tmp/fusion-harness-K48sbW/gate.py` itself; the fix is the human's call, flagged here.
- Do NOT burn rounds 45 re-running the unmodified gate against unmodified files — the failure is deterministic until line 61 is patched.
@@ -0,0 +1,11 @@
**Diagnosis** — `GATE DEFECT:` (unchanged, second escalation for the same root cause). The round-3 triage identified the exact defect and the required one-line harness-side patch, but `/tmp/fusion-harness-K48sbW/gate.py:61` still reads `return re.sub(r"[`*_]", "", s)` — the patch was never applied. That regex deletes every underscore from the searched text, while all 7 required strategy names in the gate's hardcoded `STRATEGIES` list contain underscores, so `name in clean_md(text)` is **unsatisfiable for every possible RESULTS.md** — no builder edit of any project file can ever pass checks 710. The loop is not stuck on the builder: I re-verified the real state and `/private/tmp/ddc0f4d0/fusion-harness/RESULTS.md` is correct and freshly measured (round 4: `set_based_ctes` 0.1667s speed winner = table minimum, `executemany_gen` 25.608 MB memory winner, 1733x ≥ 5x, all 7 exact snake_case names present — the winner even changed between rounds 3 and 4, proving values are measured, not hardcoded). The gate's own FAIL lines echo `setbasedctes`/`executemanygen` — names mangled by the gate itself, not by the builder. The builder's round-4 report is accurate; it correctly made no changes.
**Do exactly this**
1. **HUMAN/HARNESS OPERATOR — this is the only action that can produce GREEN:** edit `/tmp/fusion-harness-K48sbW/gate.py` line 61 from `return re.sub(r"[`*_]", "", s)` to `return re.sub(r"[`*]", "", s)`. One character class change; nothing else in the gate needs touching. Apply it **before** round 5 — it is the final round under the cap, and running it unpatched guarantees a false RED.
2. **BUILDER:** make no changes. `/private/tmp/ddc0f4d0/fusion-harness/bench.py` and `RESULTS.md` already satisfy every substantive requirement; the gate re-runs `uv run bench.py` itself and regenerates RESULTS.md, so all mtime/consistency checks will pass.
3. Re-run the gate once patched; expect GREEN (15/15 style output, exit 0).
**Do NOT**
- Do NOT spend round 5 re-running the unpatched gate — the failure is deterministic and proven unsatisfiable; that would exhaust the cap on a known-defective check.
- Builder: do NOT attempt workarounds (renaming strategies without underscores, zero-width characters, duplicate underscore-free rows) — the gate's needle list itself contains underscores, so no RESULTS.md content can match, and any such edit is gate-gaming that corrupts an otherwise-correct deliverable.
- Builder: do NOT edit `/tmp/fusion-harness-K48sbW/gate.py` yourself — the fix is flagged for the human, per the escalation contract.
@@ -0,0 +1,3 @@
Gate written to /tmp/fusion-harness-K48sbW/gate.py.
It verifies, in order: BENCH_PLAN.md intact; `bench.py` exists at the project root with a PEP 723 block, empty `dependencies` and stdlib-only imports (AST scan); static plan fidelity (real sqlite3 inserts, per-strategy process isolation, `getrusage`/`ru_maxrss` RAM measurement, N=1,000,000); `uv run bench.py` exits 0 within the ~3-minute budget; RESULTS.md is freshly (re)written by that very run; its markdown table has a time column and a peak-RAM column with positive measured values for all 7 plan strategies; Speed Winner and Memory Winner are declared and are consistent with the table (memory winner excludes the sampled baseline per plan); and the fastest strategy beats naive autocommit by ≥5x. Currently RED — bench.py does not exist yet.