🚀
This commit is contained in:
@@ -0,0 +1,16 @@
|
||||
Implemented and validated.
|
||||
|
||||
### Files created
|
||||
- `/private/tmp/ddc0f4d0/fusion-harness/bench.py`
|
||||
- `/private/tmp/ddc0f4d0/fusion-harness/RESULTS.md` — generated from fresh measurements
|
||||
|
||||
### Commands run
|
||||
- `uv run bench.py` before implementation — RED as expected because `bench.py` did not exist
|
||||
- `uv run bench.py` after implementation — completed in 6.765 seconds
|
||||
- Independent stdlib AST/results validation — PASS
|
||||
|
||||
### Measured winners
|
||||
- **Speed Winner:** `set_based_ctes` — 0.1636s, 1194.12× faster than projected naive autocommit
|
||||
- **Memory Winner:** `max_tuned_gen` — 25.756 MB measured peak RSS
|
||||
|
||||
All seven planned strategies have measured timing and isolated-process peak-RAM results in `RESULTS.md`.
|
||||
@@ -0,0 +1,19 @@
|
||||
No project change can genuinely satisfy this gate. Its `clean_md()` function executes:
|
||||
|
||||
```python
|
||||
re.sub(r"[`*_]", "", s)
|
||||
```
|
||||
|
||||
This removes every underscore before checking for strategy names, but all seven required names contain underscores. Therefore:
|
||||
|
||||
- `set_based_ctes` becomes `setbasedctes`
|
||||
- `wal_tuned_gen` becomes `waltunedgen`
|
||||
- Every table strategy name is similarly destroyed before lookup
|
||||
|
||||
I verified this for all seven names with a Python proof command. `RESULTS.md` already contains the exact strategy names, measured table rows, and winner declarations. The gate must stop deleting underscores, e.g. use:
|
||||
|
||||
```python
|
||||
re.sub(r"[`*]", "", s)
|
||||
```
|
||||
|
||||
No files were changed because altering the benchmark to manipulate the immutable gate would be gaming it.
|
||||
@@ -0,0 +1,33 @@
|
||||
The gate is unsatisfiable without modifying or gaming it.
|
||||
|
||||
Formal reason:
|
||||
|
||||
```python
|
||||
def clean_md(s):
|
||||
return re.sub(r"[`*_]", "", s)
|
||||
```
|
||||
|
||||
For every possible input `x`:
|
||||
|
||||
```python
|
||||
"_" not in clean_md(x)
|
||||
```
|
||||
|
||||
Every required strategy name contains `_`. Therefore no required name can ever satisfy:
|
||||
|
||||
```python
|
||||
strategy_name in clean_md(x)
|
||||
```
|
||||
|
||||
This makes all four failures unavoidable:
|
||||
|
||||
1. Table strategy count is always zero.
|
||||
2. Speed Winner can never match.
|
||||
3. Memory Winner can never match.
|
||||
4. No strategy rows can be parsed, so measured times remain empty.
|
||||
|
||||
`RESULTS.md` already contains the exact names and valid measurements. I made no project changes because the only possible workarounds would manipulate the gate or Python environment, which is explicitly prohibited. The gate must preserve underscores, for example:
|
||||
|
||||
```python
|
||||
return re.sub(r"[`*]", "", s)
|
||||
```
|
||||
@@ -0,0 +1,10 @@
|
||||
No project files changed, per validator triage. `bench.py` and `RESULTS.md` already satisfy every substantive requirement.
|
||||
|
||||
Required harness-side correction:
|
||||
|
||||
```diff
|
||||
- return re.sub(r"[`*_]", "", s)
|
||||
+ return re.sub(r"[`*]", "", s)
|
||||
```
|
||||
|
||||
This preserves canonical strategy-name underscores and resolves all four failures: table matching, Speed Winner matching, Memory Winner matching, and measured-time parsing.
|
||||
@@ -0,0 +1,10 @@
|
||||
No project files changed, exactly as required by validator triage. `bench.py` and freshly generated `RESULTS.md` are correct.
|
||||
|
||||
The remaining failures are caused solely by the unpatched harness regex. The human/harness operator must change `/tmp/fusion-harness-K48sbW/gate.py:61`:
|
||||
|
||||
```diff
|
||||
- return re.sub(r"[`*_]", "", s)
|
||||
+ return re.sub(r"[`*]", "", s)
|
||||
```
|
||||
|
||||
No builder-side workaround can legitimately satisfy the defective matcher.
|
||||
@@ -0,0 +1,6 @@
|
||||
exit 1
|
||||
|
||||
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
|
||||
FAIL: expected bench.py to exist, found nothing, at /private/tmp/ddc0f4d0/fusion-harness/bench.py — implement BENCH_PLAN.md as a single-file uv script named bench.py in /private/tmp/ddc0f4d0/fusion-harness (stdlib only, runnable with `uv run bench.py`)
|
||||
|
||||
RESULT: RED (gate cannot proceed without bench.py)
|
||||
@@ -0,0 +1,19 @@
|
||||
exit 1
|
||||
|
||||
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
|
||||
PASS: bench.py exists at /private/tmp/ddc0f4d0/fusion-harness/bench.py
|
||||
PASS: bench.py has a PEP 723 `# /// script` metadata block
|
||||
PASS: PEP 723 dependencies are empty (stdlib only)
|
||||
PASS: bench.py imports only Python standard library modules
|
||||
PASS: bench.py uses sqlite3 (real inserts, not simulated)
|
||||
PASS: bench.py isolates strategies in separate processes
|
||||
PASS: bench.py measures peak RAM via getrusage/ru_maxrss
|
||||
PASS: bench.py targets N=1,000,000 rows
|
||||
PASS: `uv run bench.py` ran to completion (exit 0)
|
||||
PASS: RESULTS.md was (re)written by this `uv run bench.py` invocation
|
||||
FAIL: expected a markdown table whose rows name the 7 plan strategies ['naive_autocommit', 'one_big_txn_loop', 'executemany_list', 'executemany_gen', 'wal_tuned_gen', 'max_tuned_gen', 'set_based_ctes'], found no such table, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — write one markdown table with one row per strategy using the plan's strategy names
|
||||
FAIL: expected a `Speed Winner:` line naming one plan strategy, found `Speed Winner: setbasedctes — 0.1731 s, 1459.70x faster than naive.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Speed Winner using the plan's strategy name
|
||||
FAIL: expected a `Memory Winner:` line naming one plan strategy, found `Memory Winner: waltunedgen — 25.657 MB peak RSS, +3.375 MB over the calibration worker.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Memory Winner using the plan's strategy name
|
||||
FAIL: expected complete measured times for all 7 strategies to verify the >=5x speedup, found only [] parseable, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — fix the results table first
|
||||
|
||||
RESULT: RED (11 passed, 4 failed)
|
||||
@@ -0,0 +1,19 @@
|
||||
exit 1
|
||||
|
||||
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
|
||||
PASS: bench.py exists at /private/tmp/ddc0f4d0/fusion-harness/bench.py
|
||||
PASS: bench.py has a PEP 723 `# /// script` metadata block
|
||||
PASS: PEP 723 dependencies are empty (stdlib only)
|
||||
PASS: bench.py imports only Python standard library modules
|
||||
PASS: bench.py uses sqlite3 (real inserts, not simulated)
|
||||
PASS: bench.py isolates strategies in separate processes
|
||||
PASS: bench.py measures peak RAM via getrusage/ru_maxrss
|
||||
PASS: bench.py targets N=1,000,000 rows
|
||||
PASS: `uv run bench.py` ran to completion (exit 0)
|
||||
PASS: RESULTS.md was (re)written by this `uv run bench.py` invocation
|
||||
FAIL: expected a markdown table whose rows name the 7 plan strategies ['naive_autocommit', 'one_big_txn_loop', 'executemany_list', 'executemany_gen', 'wal_tuned_gen', 'max_tuned_gen', 'set_based_ctes'], found no such table, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — write one markdown table with one row per strategy using the plan's strategy names
|
||||
FAIL: expected a `Speed Winner:` line naming one plan strategy, found `Speed Winner: setbasedctes — 0.2118 s, 1269.32x faster than naive.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Speed Winner using the plan's strategy name
|
||||
FAIL: expected a `Memory Winner:` line naming one plan strategy, found `Memory Winner: waltunedgen — 25.625 MB peak RSS, +3.359 MB over the calibration worker.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Memory Winner using the plan's strategy name
|
||||
FAIL: expected complete measured times for all 7 strategies to verify the >=5x speedup, found only [] parseable, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — fix the results table first
|
||||
|
||||
RESULT: RED (11 passed, 4 failed)
|
||||
@@ -0,0 +1,19 @@
|
||||
exit 1
|
||||
|
||||
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
|
||||
PASS: bench.py exists at /private/tmp/ddc0f4d0/fusion-harness/bench.py
|
||||
PASS: bench.py has a PEP 723 `# /// script` metadata block
|
||||
PASS: PEP 723 dependencies are empty (stdlib only)
|
||||
PASS: bench.py imports only Python standard library modules
|
||||
PASS: bench.py uses sqlite3 (real inserts, not simulated)
|
||||
PASS: bench.py isolates strategies in separate processes
|
||||
PASS: bench.py measures peak RAM via getrusage/ru_maxrss
|
||||
PASS: bench.py targets N=1,000,000 rows
|
||||
PASS: `uv run bench.py` ran to completion (exit 0)
|
||||
PASS: RESULTS.md was (re)written by this `uv run bench.py` invocation
|
||||
FAIL: expected a markdown table whose rows name the 7 plan strategies ['naive_autocommit', 'one_big_txn_loop', 'executemany_list', 'executemany_gen', 'wal_tuned_gen', 'max_tuned_gen', 'set_based_ctes'], found no such table, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — write one markdown table with one row per strategy using the plan's strategy names
|
||||
FAIL: expected a `Speed Winner:` line naming one plan strategy, found `Speed Winner: setbasedctes — 0.1768 s, 2030.00x faster than naive.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Speed Winner using the plan's strategy name
|
||||
FAIL: expected a `Memory Winner:` line naming one plan strategy, found `Memory Winner: waltunedgen — 25.657 MB peak RSS, +3.211 MB over the calibration worker.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Memory Winner using the plan's strategy name
|
||||
FAIL: expected complete measured times for all 7 strategies to verify the >=5x speedup, found only [] parseable, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — fix the results table first
|
||||
|
||||
RESULT: RED (11 passed, 4 failed)
|
||||
@@ -0,0 +1,19 @@
|
||||
exit 1
|
||||
|
||||
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
|
||||
PASS: bench.py exists at /private/tmp/ddc0f4d0/fusion-harness/bench.py
|
||||
PASS: bench.py has a PEP 723 `# /// script` metadata block
|
||||
PASS: PEP 723 dependencies are empty (stdlib only)
|
||||
PASS: bench.py imports only Python standard library modules
|
||||
PASS: bench.py uses sqlite3 (real inserts, not simulated)
|
||||
PASS: bench.py isolates strategies in separate processes
|
||||
PASS: bench.py measures peak RAM via getrusage/ru_maxrss
|
||||
PASS: bench.py targets N=1,000,000 rows
|
||||
PASS: `uv run bench.py` ran to completion (exit 0)
|
||||
PASS: RESULTS.md was (re)written by this `uv run bench.py` invocation
|
||||
FAIL: expected a markdown table whose rows name the 7 plan strategies ['naive_autocommit', 'one_big_txn_loop', 'executemany_list', 'executemany_gen', 'wal_tuned_gen', 'max_tuned_gen', 'set_based_ctes'], found no such table, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — write one markdown table with one row per strategy using the plan's strategy names
|
||||
FAIL: expected a `Speed Winner:` line naming one plan strategy, found `Speed Winner: setbasedctes — 0.1667 s, 1733.05x faster than naive.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Speed Winner using the plan's strategy name
|
||||
FAIL: expected a `Memory Winner:` line naming one plan strategy, found `Memory Winner: executemanygen — 25.608 MB peak RSS, +3.441 MB over the calibration worker.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Memory Winner using the plan's strategy name
|
||||
FAIL: expected complete measured times for all 7 strategies to verify the >=5x speedup, found only [] parseable, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — fix the results table first
|
||||
|
||||
RESULT: RED (11 passed, 4 failed)
|
||||
@@ -0,0 +1,21 @@
|
||||
exit 130
|
||||
|
||||
PASS: BENCH_PLAN.md present and still defines all 7 plan strategies
|
||||
PASS: bench.py exists at /private/tmp/ddc0f4d0/fusion-harness/bench.py
|
||||
PASS: bench.py has a PEP 723 `# /// script` metadata block
|
||||
PASS: PEP 723 dependencies are empty (stdlib only)
|
||||
PASS: bench.py imports only Python standard library modules
|
||||
PASS: bench.py uses sqlite3 (real inserts, not simulated)
|
||||
PASS: bench.py isolates strategies in separate processes
|
||||
PASS: bench.py measures peak RAM via getrusage/ru_maxrss
|
||||
PASS: bench.py targets N=1,000,000 rows
|
||||
PASS: `uv run bench.py` ran to completion (exit 0)
|
||||
PASS: RESULTS.md was (re)written by this `uv run bench.py` invocation
|
||||
FAIL: expected a markdown table whose rows name the 7 plan strategies ['naive_autocommit', 'one_big_txn_loop', 'executemany_list', 'executemany_gen', 'wal_tuned_gen', 'max_tuned_gen', 'set_based_ctes'], found no such table, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — write one markdown table with one row per strategy using the plan's strategy names
|
||||
FAIL: expected a `Speed Winner:` line naming one plan strategy, found `Speed Winner: setbasedctes — 0.1700 s, 1147.98x faster than naive.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Speed Winner using the plan's strategy name
|
||||
FAIL: expected a `Memory Winner:` line naming one plan strategy, found `Memory Winner: executemanygen — 25.625 MB peak RSS, +3.588 MB over the calibration worker.`, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — declare the Memory Winner using the plan's strategy name
|
||||
FAIL: expected complete measured times for all 7 strategies to verify the >=5x speedup, found only [] parseable, at /private/tmp/ddc0f4d0/fusion-harness/RESULTS.md — fix the results table first
|
||||
|
||||
RESULT: RED (11 passed, 4 failed)
|
||||
|
||||
[stopped by user]
|
||||
@@ -0,0 +1,477 @@
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = []
|
||||
# ///
|
||||
"""
|
||||
Acceptance gate for: implement BENCH_PLAN.md as bench.py (uv single-file
|
||||
script, stdlib only) that runs to completion via `uv run bench.py`, writes
|
||||
RESULTS.md with a markdown table holding a measured time AND a measured peak
|
||||
RAM figure for every strategy in the plan, declares a Speed Winner and a
|
||||
Memory Winner from those measurements, and where the fastest strategy beats
|
||||
the naive autocommit baseline by at least 5x.
|
||||
|
||||
Exit 0 iff every check passes. One PASS/FAIL line per check.
|
||||
"""
|
||||
|
||||
import ast
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
ROOT = "/private/tmp/ddc0f4d0/fusion-harness"
|
||||
BENCH = os.path.join(ROOT, "bench.py")
|
||||
RESULTS = os.path.join(ROOT, "RESULTS.md")
|
||||
PLAN = os.path.join(ROOT, "BENCH_PLAN.md")
|
||||
|
||||
# The seven strategies defined by BENCH_PLAN.md ("Strategies (7)" table and
|
||||
# the STRATEGIES dict in its appendix script).
|
||||
STRATEGIES = [
|
||||
"naive_autocommit",
|
||||
"one_big_txn_loop",
|
||||
"executemany_list",
|
||||
"executemany_gen",
|
||||
"wal_tuned_gen",
|
||||
"max_tuned_gen",
|
||||
"set_based_ctes",
|
||||
]
|
||||
BASELINE = "naive_autocommit"
|
||||
RUN_TIMEOUT = 240 # plan budget is ~3 minutes; measured fused run was ~7s
|
||||
|
||||
_results: list[bool] = []
|
||||
|
||||
|
||||
def ok(msg: str) -> None:
|
||||
_results.append(True)
|
||||
print("PASS: " + msg)
|
||||
|
||||
|
||||
def bad(msg: str) -> None:
|
||||
_results.append(False)
|
||||
print("FAIL: " + msg)
|
||||
|
||||
|
||||
def one_line(s: str, limit: int = 500) -> str:
|
||||
s = " | ".join(part.strip() for part in s.strip().splitlines() if part.strip())
|
||||
return s[-limit:] if len(s) > limit else s
|
||||
|
||||
|
||||
def clean_md(s: str) -> str:
|
||||
return re.sub(r"[`*_]", "", s)
|
||||
|
||||
|
||||
def first_float(cell: str):
|
||||
m = re.search(r"-?\d+(?:\.\d+)?", cell.replace(",", ""))
|
||||
return float(m.group(0)) if m else None
|
||||
|
||||
|
||||
def split_cells(line: str):
|
||||
line = line.strip()
|
||||
if line.startswith("|"):
|
||||
line = line[1:]
|
||||
if line.endswith("|"):
|
||||
line = line[:-1]
|
||||
return [c.strip() for c in line.split("|")]
|
||||
|
||||
|
||||
def parse_tables(text: str):
|
||||
"""Return list of (header_cells, data_rows) for every markdown table."""
|
||||
lines = text.splitlines()
|
||||
tables = []
|
||||
i = 0
|
||||
sep_re = re.compile(r"^\s*\|?[\s:|-]+\|?\s*$")
|
||||
while i < len(lines):
|
||||
if (
|
||||
"|" in lines[i]
|
||||
and i + 1 < len(lines)
|
||||
and "-" in lines[i + 1]
|
||||
and "|" in lines[i + 1]
|
||||
and sep_re.match(lines[i + 1])
|
||||
):
|
||||
header = split_cells(lines[i])
|
||||
j = i + 2
|
||||
rows = []
|
||||
while j < len(lines) and "|" in lines[j] and lines[j].strip():
|
||||
rows.append(split_cells(lines[j]))
|
||||
j += 1
|
||||
tables.append((header, rows))
|
||||
i = j
|
||||
else:
|
||||
i += 1
|
||||
return tables
|
||||
|
||||
|
||||
def main() -> int:
|
||||
# ---- 0. plan still present (the thing bench.py must implement) --------
|
||||
if os.path.isfile(PLAN):
|
||||
plan_txt = open(PLAN, encoding="utf-8", errors="replace").read()
|
||||
missing = [s for s in STRATEGIES if s not in plan_txt]
|
||||
if missing:
|
||||
bad(
|
||||
f"expected BENCH_PLAN.md to still define strategies {missing}, "
|
||||
f"found them absent, at {PLAN} — restore BENCH_PLAN.md; the "
|
||||
f"plan is the spec and must not be edited"
|
||||
)
|
||||
else:
|
||||
ok("BENCH_PLAN.md present and still defines all 7 plan strategies")
|
||||
else:
|
||||
bad(
|
||||
f"expected BENCH_PLAN.md to exist, found nothing, at {PLAN} — "
|
||||
f"restore the plan file; it is the spec bench.py implements"
|
||||
)
|
||||
|
||||
# ---- 1. bench.py exists ------------------------------------------------
|
||||
if not os.path.isfile(BENCH):
|
||||
bad(
|
||||
f"expected bench.py to exist, found nothing, at {BENCH} — "
|
||||
f"implement BENCH_PLAN.md as a single-file uv script named "
|
||||
f"bench.py in {ROOT} (stdlib only, runnable with `uv run bench.py`)"
|
||||
)
|
||||
print("\nRESULT: RED (gate cannot proceed without bench.py)")
|
||||
return 1
|
||||
ok(f"bench.py exists at {BENCH}")
|
||||
|
||||
src = open(BENCH, encoding="utf-8", errors="replace").read()
|
||||
|
||||
# ---- 2. PEP 723 header, empty dependencies (stdlib only) --------------
|
||||
if re.search(r"^#\s*///\s*script\s*$", src, re.MULTILINE):
|
||||
ok("bench.py has a PEP 723 `# /// script` metadata block")
|
||||
else:
|
||||
bad(
|
||||
f"expected a PEP 723 block starting with `# /// script`, found "
|
||||
f"none, at {BENCH} — add the inline metadata block "
|
||||
f"(`# /// script`, `# requires-python = ...`, "
|
||||
f"`# dependencies = []`, `# ///`) at the top of bench.py"
|
||||
)
|
||||
dep_lines = [l for l in src.splitlines() if re.search(r"^\#\s*dependencies\s*=", l)]
|
||||
if dep_lines and not re.search(r"dependencies\s*=\s*\[\s*\]", dep_lines[0]):
|
||||
bad(
|
||||
f"expected `dependencies = []` (stdlib only), found "
|
||||
f"`{dep_lines[0].strip()}`, at {BENCH} — remove all third-party "
|
||||
f"dependencies from the PEP 723 block; the plan requires Python "
|
||||
f"standard library only"
|
||||
)
|
||||
else:
|
||||
ok("PEP 723 dependencies are empty (stdlib only)")
|
||||
|
||||
# ---- 3. all imports are stdlib -----------------------------------------
|
||||
try:
|
||||
tree = ast.parse(src)
|
||||
roots = set()
|
||||
for node in ast.walk(tree):
|
||||
if isinstance(node, ast.Import):
|
||||
for a in node.names:
|
||||
roots.add(a.name.split(".")[0])
|
||||
elif isinstance(node, ast.ImportFrom) and node.module and node.level == 0:
|
||||
roots.add(node.module.split(".")[0])
|
||||
non_std = sorted(r for r in roots if r not in sys.stdlib_module_names)
|
||||
if non_std:
|
||||
bad(
|
||||
f"expected only stdlib imports, found non-stdlib {non_std}, "
|
||||
f"at {BENCH} — rewrite bench.py using only the Python "
|
||||
f"standard library"
|
||||
)
|
||||
else:
|
||||
ok("bench.py imports only Python standard library modules")
|
||||
except SyntaxError as e:
|
||||
bad(
|
||||
f"expected bench.py to be valid Python, found SyntaxError "
|
||||
f"`{one_line(str(e))}`, at {BENCH} — fix the syntax error"
|
||||
)
|
||||
|
||||
# ---- 4. static plan fidelity: real measurement machinery ---------------
|
||||
if "sqlite3" in src:
|
||||
ok("bench.py uses sqlite3 (real inserts, not simulated)")
|
||||
else:
|
||||
bad(
|
||||
f"expected bench.py to import/use sqlite3, found no mention, at "
|
||||
f"{BENCH} — the benchmark must perform real SQLite inserts per "
|
||||
f"BENCH_PLAN.md"
|
||||
)
|
||||
if re.search(r"subprocess|multiprocessing", src):
|
||||
ok("bench.py isolates strategies in separate processes")
|
||||
else:
|
||||
bad(
|
||||
f"expected per-strategy process isolation (subprocess re-exec or "
|
||||
f"multiprocessing per BENCH_PLAN.md), found neither, at {BENCH} — "
|
||||
f"run each strategy in its own fresh process so peak-RAM readings "
|
||||
f"cannot pollute each other"
|
||||
)
|
||||
if re.search(r"ru_maxrss|getrusage", src):
|
||||
ok("bench.py measures peak RAM via getrusage/ru_maxrss")
|
||||
else:
|
||||
bad(
|
||||
f"expected measured peak RAM via resource.getrusage(...).ru_maxrss "
|
||||
f"per BENCH_PLAN.md, found no getrusage/ru_maxrss, at {BENCH} — "
|
||||
f"measure whole-process peak RSS inside each isolated worker; "
|
||||
f"do not assume or hardcode memory figures"
|
||||
)
|
||||
if re.search(r"1_000_000|1000000", src):
|
||||
ok("bench.py targets N=1,000,000 rows")
|
||||
else:
|
||||
bad(
|
||||
f"expected N=1,000,000 rows (literal 1_000_000 or 1000000), found "
|
||||
f"neither, at {BENCH} — the plan requires 1M rows for every "
|
||||
f"full-run strategy (baseline may be sampled and scaled)"
|
||||
)
|
||||
|
||||
# ---- 5. `uv run bench.py` runs to completion ---------------------------
|
||||
pre_marker = time.time()
|
||||
proc = None
|
||||
ran_ok = False
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
["uv", "run", "bench.py"],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=RUN_TIMEOUT,
|
||||
)
|
||||
if proc.returncode == 0:
|
||||
ok("`uv run bench.py` ran to completion (exit 0)")
|
||||
ran_ok = True
|
||||
else:
|
||||
bad(
|
||||
f"expected `uv run bench.py` to exit 0, found exit "
|
||||
f"{proc.returncode}, at {BENCH} — fix the crash; stderr tail: "
|
||||
f"{one_line(proc.stderr or proc.stdout or '(empty)')}"
|
||||
)
|
||||
except FileNotFoundError:
|
||||
bad(
|
||||
f"expected `uv` on PATH so `uv run bench.py` works, found uv "
|
||||
f"missing, at {ROOT} — ensure the script is runnable with exactly "
|
||||
f"`uv run bench.py`; do not substitute another runner"
|
||||
)
|
||||
# best-effort fallback so content checks below still give feedback
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[sys.executable, "bench.py"],
|
||||
cwd=ROOT,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=RUN_TIMEOUT,
|
||||
)
|
||||
except Exception:
|
||||
proc = None
|
||||
except subprocess.TimeoutExpired:
|
||||
bad(
|
||||
f"expected `uv run bench.py` to finish within {RUN_TIMEOUT}s "
|
||||
f"(plan budget ~3 minutes), found it still running, at {BENCH} — "
|
||||
f"sample the naive autocommit baseline (plan: 20,000 rows scaled "
|
||||
f"x50) instead of running it for the full 1M rows"
|
||||
)
|
||||
|
||||
# ---- 6. RESULTS.md written by the run ----------------------------------
|
||||
if not os.path.isfile(RESULTS):
|
||||
bad(
|
||||
f"expected the run to write RESULTS.md, found no file, at "
|
||||
f"{RESULTS} — make bench.py write RESULTS.md (markdown table with "
|
||||
f"a time column and a peak RAM column for every strategy, plus "
|
||||
f"Speed Winner and Memory Winner lines) every time it runs"
|
||||
)
|
||||
print("\nRESULT: RED")
|
||||
return 1
|
||||
if os.path.getmtime(RESULTS) >= pre_marker - 2:
|
||||
ok("RESULTS.md was (re)written by this `uv run bench.py` invocation")
|
||||
else:
|
||||
bad(
|
||||
f"expected RESULTS.md to be rewritten by the run just executed, "
|
||||
f"found a stale file (mtime predates the run), at {RESULTS} — "
|
||||
f"bench.py itself must write RESULTS.md from its own fresh "
|
||||
f"measurements on every run; do not hand-author RESULTS.md"
|
||||
)
|
||||
|
||||
text = open(RESULTS, encoding="utf-8", errors="replace").read()
|
||||
|
||||
# ---- 7. markdown table: time + peak RAM for every strategy -------------
|
||||
tables = parse_tables(text)
|
||||
best = None
|
||||
best_count = -1
|
||||
for header, rows in tables:
|
||||
joined = clean_md(" ".join(" ".join(r) for r in rows))
|
||||
count = sum(1 for s in STRATEGIES if s in joined)
|
||||
if count > best_count:
|
||||
best_count = count
|
||||
best = (header, rows)
|
||||
|
||||
times: dict[str, float] = {}
|
||||
rams: dict[str, float] = {}
|
||||
if best is None or best_count == 0:
|
||||
bad(
|
||||
f"expected a markdown table whose rows name the 7 plan strategies "
|
||||
f"{STRATEGIES}, found no such table, at {RESULTS} — write one "
|
||||
f"markdown table with one row per strategy using the plan's "
|
||||
f"strategy names"
|
||||
)
|
||||
else:
|
||||
header, rows = best
|
||||
time_col = next(
|
||||
(i for i, h in enumerate(header) if re.search(r"time|sec|duration", h, re.I)),
|
||||
None,
|
||||
)
|
||||
ram_candidates = [
|
||||
i for i, h in enumerate(header) if re.search(r"ram|rss|mem", h, re.I)
|
||||
]
|
||||
ram_col = next(
|
||||
(i for i in ram_candidates if re.search(r"peak", header[i], re.I)),
|
||||
ram_candidates[0] if ram_candidates else None,
|
||||
)
|
||||
if time_col is None:
|
||||
bad(
|
||||
f"expected a time column (header matching time/sec/duration), "
|
||||
f"found headers {header}, at {RESULTS} — add a measured time "
|
||||
f"column to the results table"
|
||||
)
|
||||
else:
|
||||
ok(f"results table has a time column (`{header[time_col]}`)")
|
||||
if ram_col is None:
|
||||
bad(
|
||||
f"expected a peak RAM column (header matching RAM/RSS/mem), "
|
||||
f"found headers {header}, at {RESULTS} — add a measured peak "
|
||||
f"RAM column to the results table"
|
||||
)
|
||||
else:
|
||||
ok(f"results table has a peak RAM column (`{header[ram_col]}`)")
|
||||
|
||||
if time_col is not None and ram_col is not None:
|
||||
for strat in STRATEGIES:
|
||||
row = next(
|
||||
(r for r in rows if strat in clean_md(" ".join(r))), None
|
||||
)
|
||||
if row is None:
|
||||
bad(
|
||||
f"expected a table row for strategy `{strat}`, found "
|
||||
f"none, at {RESULTS} — every strategy in BENCH_PLAN.md "
|
||||
f"must have its own measured row"
|
||||
)
|
||||
continue
|
||||
t = first_float(row[time_col]) if time_col < len(row) else None
|
||||
m = first_float(row[ram_col]) if ram_col < len(row) else None
|
||||
if t is None or t <= 0:
|
||||
bad(
|
||||
f"expected a positive measured time for `{strat}`, "
|
||||
f"found `{row[time_col] if time_col < len(row) else '(missing cell)'}`, "
|
||||
f"at {RESULTS} — record the real measured wall time"
|
||||
)
|
||||
else:
|
||||
times[strat] = t
|
||||
if m is None or m <= 0:
|
||||
bad(
|
||||
f"expected a positive measured peak RAM for `{strat}`, "
|
||||
f"found `{row[ram_col] if ram_col < len(row) else '(missing cell)'}`, "
|
||||
f"at {RESULTS} — record the real measured peak RSS"
|
||||
)
|
||||
else:
|
||||
rams[strat] = m
|
||||
if len(times) == len(STRATEGIES):
|
||||
ok("every plan strategy has a positive measured time in the table")
|
||||
if len(rams) == len(STRATEGIES):
|
||||
ok("every plan strategy has a positive measured peak RAM in the table")
|
||||
|
||||
# ---- 8. declared winners ------------------------------------------------
|
||||
def declared(kind: str):
|
||||
m = re.search(kind + r"\s*winner[^\n]*", text, re.I)
|
||||
if not m:
|
||||
return None, None
|
||||
line = clean_md(m.group(0))
|
||||
name = next((s for s in STRATEGIES if s in line), None)
|
||||
return line, name
|
||||
|
||||
speed_line, speed_name = declared("speed")
|
||||
if speed_name:
|
||||
ok(f"RESULTS.md declares a Speed Winner: {speed_name}")
|
||||
else:
|
||||
bad(
|
||||
f"expected a `Speed Winner:` line naming one plan strategy, found "
|
||||
f"`{speed_line or 'no such line'}`, at {RESULTS} — declare the "
|
||||
f"Speed Winner using the plan's strategy name"
|
||||
)
|
||||
mem_line, mem_name = declared("memory")
|
||||
if mem_name:
|
||||
ok(f"RESULTS.md declares a Memory Winner: {mem_name}")
|
||||
else:
|
||||
bad(
|
||||
f"expected a `Memory Winner:` line naming one plan strategy, found "
|
||||
f"`{mem_line or 'no such line'}`, at {RESULTS} — declare the "
|
||||
f"Memory Winner using the plan's strategy name"
|
||||
)
|
||||
|
||||
# ---- 9. winners follow from the measurements ---------------------------
|
||||
if speed_name and len(times) == len(STRATEGIES):
|
||||
fastest_t = min(times.values())
|
||||
if times[speed_name] <= fastest_t + 1e-9:
|
||||
ok(
|
||||
f"Speed Winner {speed_name} matches the fastest measured time "
|
||||
f"({times[speed_name]}s)"
|
||||
)
|
||||
else:
|
||||
actual = min(times, key=times.get)
|
||||
bad(
|
||||
f"expected the Speed Winner to be the fastest strategy in the "
|
||||
f"table (`{actual}` at {times[actual]}s), found `{speed_name}` "
|
||||
f"at {times[speed_name]}s, at {RESULTS} — derive the winner "
|
||||
f"from the measurements, not assumptions"
|
||||
)
|
||||
if mem_name and len(rams) == len(STRATEGIES):
|
||||
if mem_name == BASELINE:
|
||||
bad(
|
||||
f"expected the Memory Winner to exclude the sampled baseline, "
|
||||
f"found `{BASELINE}`, at {RESULTS} — per BENCH_PLAN.md the "
|
||||
f"sampled baseline never held 1M rows of work and cannot win "
|
||||
f"the RAM axis; pick the leanest full-1M strategy"
|
||||
)
|
||||
else:
|
||||
eligible = {k: v for k, v in rams.items() if k != BASELINE}
|
||||
lean = min(eligible.values())
|
||||
if rams[mem_name] <= lean + 1e-9:
|
||||
ok(
|
||||
f"Memory Winner {mem_name} matches the lowest measured "
|
||||
f"peak RAM among full-1M strategies ({rams[mem_name]})"
|
||||
)
|
||||
else:
|
||||
actual = min(eligible, key=eligible.get)
|
||||
bad(
|
||||
f"expected the Memory Winner to have the lowest peak RAM "
|
||||
f"among full-1M strategies (`{actual}` at {eligible[actual]}), "
|
||||
f"found `{mem_name}` at {rams[mem_name]}, at {RESULTS} — "
|
||||
f"derive the winner from the measurements"
|
||||
)
|
||||
|
||||
# ---- 10. fastest beats naive autocommit by >= 5x ------------------------
|
||||
if len(times) == len(STRATEGIES):
|
||||
fastest_t = min(times.values())
|
||||
speedup = times[BASELINE] / fastest_t if fastest_t > 0 else 0.0
|
||||
if speedup >= 5.0:
|
||||
ok(
|
||||
f"fastest strategy beats naive autocommit by {speedup:.1f}x "
|
||||
f"(>= 5x required)"
|
||||
)
|
||||
else:
|
||||
bad(
|
||||
f"expected fastest strategy >= 5x faster than naive "
|
||||
f"autocommit, found {speedup:.1f}x ({times[BASELINE]}s vs "
|
||||
f"{fastest_t}s), at {RESULTS} — use one big transaction / "
|
||||
f"executemany / set-based insert per BENCH_PLAN.md so the "
|
||||
f"speedup is real"
|
||||
)
|
||||
else:
|
||||
bad(
|
||||
f"expected complete measured times for all 7 strategies to verify "
|
||||
f"the >=5x speedup, found only {sorted(times)} parseable, at "
|
||||
f"{RESULTS} — fix the results table first"
|
||||
)
|
||||
|
||||
if not ran_ok:
|
||||
pass # the uv-run FAIL above already carries the instruction
|
||||
|
||||
failed = _results.count(False)
|
||||
print(
|
||||
f"\nRESULT: {'GREEN' if failed == 0 else 'RED'} "
|
||||
f"({_results.count(True)} passed, {failed} failed)"
|
||||
)
|
||||
return 0 if failed == 0 else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1 @@
|
||||
Read BENCH_PLAN.md, the plan you two designed and fused, and implement it in this directory as bench.py, an astral uv single file script. Python standard library only, runnable with uv run bench.py. Definition of Done is defined by a gate YOU design. At minimum it proves uv run bench.py runs to completion, writes RESULTS.md with a markdown table holding a timed row AND a measured peak RAM figure for every strategy in the plan, and declares a Speed Winner and a Memory Winner from those measurements. Measured, not assumed. The fastest strategy has to beat the naive autocommit baseline by at least 5x. Baseline the gate RED before you build anything. Never shim, wrap, or modify global system tools.
|
||||
@@ -0,0 +1,11 @@
|
||||
**Diagnosis** — `GATE DEFECT:` The builder's work is correct and complete; the gate can never pass it. `gate.py:61` defines `clean_md(s) = re.sub(r"[`*_]", "", s)` — intended to strip markdown emphasis, it also deletes every underscore from the text being searched. All 7 required strategy names contain underscores, so `"set_based_ctes" in clean_md(...)` is mathematically unsatisfiable for any possible RESULTS.md (the gate's own round-2/3 output proves it: it echoes `setbasedctes` / `waltunedgen` — mangled by the gate, not written that way). This single line causes all four FAILs (table match count 0 → "no such table", both winner-name checks, empty parsed times → 5x check). I verified the real state: `/private/tmp/ddc0f4d0/fusion-harness/RESULTS.md` contains a well-formed table with all 7 exact plan names, positive measured times and peak-RAM figures, a Speed Winner (`set_based_ctes`, 0.1768s = table minimum), a Memory Winner (`wal_tuned_gen`, 25.657 MB = minimum among full-1M rows, baseline excluded), and a 2030x speedup vs naive — every substantive requirement is already met. The builder's round-3 report is accurate; rounds 2–3 are identical because there is nothing left for the builder to fix.
|
||||
|
||||
**Do exactly this**
|
||||
1. **HUMAN/HARNESS (not builder):** patch `/tmp/fusion-harness-K48sbW/gate.py` line 61 from `return re.sub(r"[`*_]", "", s)` to `return re.sub(r"[`*]", "", s)`. Underscores must survive cleaning; backtick/asterisk stripping stays. No other gate change is needed.
|
||||
2. **BUILDER:** change nothing. Leave `/private/tmp/ddc0f4d0/fusion-harness/bench.py` and `RESULTS.md` exactly as they are — they satisfy every check once matching works (RESULTS.md is regenerated fresh each run, so mtime/consistency checks will pass).
|
||||
3. Re-run the gate once the patch lands; expect GREEN.
|
||||
|
||||
**Do NOT**
|
||||
- Do NOT rename strategies to underscore-free variants (e.g., `setbasedctes`) or strip underscores/backticks in RESULTS.md to appease the broken matcher — that diverges from BENCH_PLAN.md's canonical names and is gate-gaming.
|
||||
- Do NOT let the builder edit `/tmp/fusion-harness-K48sbW/gate.py` itself; the fix is the human's call, flagged here.
|
||||
- Do NOT burn rounds 4–5 re-running the unmodified gate against unmodified files — the failure is deterministic until line 61 is patched.
|
||||
@@ -0,0 +1,11 @@
|
||||
**Diagnosis** — `GATE DEFECT:` (unchanged, second escalation for the same root cause). The round-3 triage identified the exact defect and the required one-line harness-side patch, but `/tmp/fusion-harness-K48sbW/gate.py:61` still reads `return re.sub(r"[`*_]", "", s)` — the patch was never applied. That regex deletes every underscore from the searched text, while all 7 required strategy names in the gate's hardcoded `STRATEGIES` list contain underscores, so `name in clean_md(text)` is **unsatisfiable for every possible RESULTS.md** — no builder edit of any project file can ever pass checks 7–10. The loop is not stuck on the builder: I re-verified the real state and `/private/tmp/ddc0f4d0/fusion-harness/RESULTS.md` is correct and freshly measured (round 4: `set_based_ctes` 0.1667s speed winner = table minimum, `executemany_gen` 25.608 MB memory winner, 1733x ≥ 5x, all 7 exact snake_case names present — the winner even changed between rounds 3 and 4, proving values are measured, not hardcoded). The gate's own FAIL lines echo `setbasedctes`/`executemanygen` — names mangled by the gate itself, not by the builder. The builder's round-4 report is accurate; it correctly made no changes.
|
||||
|
||||
**Do exactly this**
|
||||
1. **HUMAN/HARNESS OPERATOR — this is the only action that can produce GREEN:** edit `/tmp/fusion-harness-K48sbW/gate.py` line 61 from `return re.sub(r"[`*_]", "", s)` to `return re.sub(r"[`*]", "", s)`. One character class change; nothing else in the gate needs touching. Apply it **before** round 5 — it is the final round under the cap, and running it unpatched guarantees a false RED.
|
||||
2. **BUILDER:** make no changes. `/private/tmp/ddc0f4d0/fusion-harness/bench.py` and `RESULTS.md` already satisfy every substantive requirement; the gate re-runs `uv run bench.py` itself and regenerates RESULTS.md, so all mtime/consistency checks will pass.
|
||||
3. Re-run the gate once patched; expect GREEN (15/15 style output, exit 0).
|
||||
|
||||
**Do NOT**
|
||||
- Do NOT spend round 5 re-running the unpatched gate — the failure is deterministic and proven unsatisfiable; that would exhaust the cap on a known-defective check.
|
||||
- Builder: do NOT attempt workarounds (renaming strategies without underscores, zero-width characters, duplicate underscore-free rows) — the gate's needle list itself contains underscores, so no RESULTS.md content can match, and any such edit is gate-gaming that corrupts an otherwise-correct deliverable.
|
||||
- Builder: do NOT edit `/tmp/fusion-harness-K48sbW/gate.py` yourself — the fix is flagged for the human, per the escalation contract.
|
||||
@@ -0,0 +1,3 @@
|
||||
Gate written to /tmp/fusion-harness-K48sbW/gate.py.
|
||||
|
||||
It verifies, in order: BENCH_PLAN.md intact; `bench.py` exists at the project root with a PEP 723 block, empty `dependencies` and stdlib-only imports (AST scan); static plan fidelity (real sqlite3 inserts, per-strategy process isolation, `getrusage`/`ru_maxrss` RAM measurement, N=1,000,000); `uv run bench.py` exits 0 within the ~3-minute budget; RESULTS.md is freshly (re)written by that very run; its markdown table has a time column and a peak-RAM column with positive measured values for all 7 plan strategies; Speed Winner and Memory Winner are declared and are consistent with the table (memory winner excludes the sampled baseline per plan); and the fastest strategy beats naive autocommit by ≥5x. Currently RED — bench.py does not exist yet.
|
||||
Reference in New Issue
Block a user