serve: honor --ngen when the client omits max_tokens; convert: print the resolved plan (#382, #383)

Two operator-surprise fixes:

openai_server.py (#382, LordMZTE): a request without max_tokens defaulted to
min(256, limit) — so `coli serve --ngen 32768` still cut every answer at 256
tokens for clients that omit the field. The operator's configured budget IS the
default now; generation still ends at EOS, so it's a cap, not a target.

convert_fp8_to_int4.py (#383, bokiko): the resolved conversion plan (mode,
source, ebits/io/x bits, grouped-vs-per-row) prints as a [PLAN] line before any
work. The two traps in #383 — --mtp defaulting ebits to 8 with the grouped
branch silently gated off, and --mtp appearing to ignore --indir — are already
defused by the #355 fix (the --mtp pass emits ONLY head tensors on the local
path), but a 3.5-hour job must show its plan at second 1, not in a post-hoc
size sanity check. bokiko's exact invocation now prints:
  [PLAN] mode: MTP head only | source: local ./fp8 | experts 8-bit, ... |
         PER-ROW (grouped branch needs bits<=4; ebits=8 disables it)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
JustVugg
2026-07-18 12:28:27 +02:00
parent 8a1dccae3b
commit f8e4dc95b5
2 changed files with 14 additions and 1 deletions
+4 -1
View File
@@ -406,7 +406,10 @@ def generation_options(body, limit):
maximum = body.get("max_tokens")
maximum_param = "max_tokens"
if maximum is None:
maximum = min(256, limit)
# Client omitted max_tokens: honor the operator's configured budget (--max-tokens /
# --ngen), not an arbitrary 256 — `coli serve --ngen 32768` must mean 32768 (#382).
# Generation still ends at EOS, so this is a cap, not a target.
maximum = limit
temperature = body.get("temperature")
top_p = body.get("top_p")
temperature = 0.7 if temperature is None else temperature