fix(sleep): address review — CLI choice, auth guard, provider-neutral kwargs, tests
Addresses maintainer review on the OpenAI-compatible endpoints PR: 1. CLI: accept --backend azure_openai in skillopt_sleep/__main__.py (the documented command was rejected by the argparse choices). 2. Example runner: exit with the child's return code so watchdog/supervisors see a failed sleep run as a failure. 3. Error state: clear last_call_error when a retry recovers; set an explicit "empty response on all N attempts" diagnostic when every attempt returns empty text. 4. Security guard: the managed-identity path now refuses to send an Azure AD bearer token to any endpoint outside *.openai.azure.com / *.cognitiveservices.azure.com — a custom endpoint requires explicit AZURE_OPENAI_AUTH_MODE=openai_compatible + API key. 5. Provider-neutral requests: compat mode sends only the standard contract (max_tokens, default 8192 via SKILLOPT_SLEEP_COMPAT_MAX_TOKENS); provider-specific body fields are opt-in via SKILLOPT_SLEEP_CHAT_EXTRA_BODY (JSON) — the deepseek model-name inference is removed. 6. Docs: removed the unimplemented OPTIMIZER_*/TARGET_* env-var claim; added a configuration reference matching the implementation exactly. 7. Tests: tests/test_azure_openai_compat.py — 17 deterministic no-network unittest cases covering CLI acceptance, compat-vs-Azure client selection, endpoint resolution, the credential guard, request kwargs (opt-in extra body / token cap), retry-success error clearing, empty-response diagnostics, and runner exit-code propagation. Re-verified live against DeepSeek (deepseek-v4-pro, openai_compatible mode) after the rework: client type OpenAI, completion returned, no error state.
This commit is contained in:
Alphaxalchemy
@@ -72,12 +72,14 @@ def main() -> None:
|
||||
# OpenAI-compatible path — see docs/sleep/openai-compatible-endpoints.md
|
||||
backend = "azure_openai"
|
||||
env["PYTHONIOENCODING"] = "utf-8"
|
||||
for prefix in ("", "OPTIMIZER_", "TARGET_"):
|
||||
env[f"{prefix}AZURE_OPENAI_AUTH_MODE"] = "openai_compatible"
|
||||
env[f"{prefix}AZURE_OPENAI_ENDPOINT"] = PROVIDER_ENDPOINT
|
||||
env[f"{prefix}AZURE_OPENAI_API_KEY"] = env["DEEPSEEK_API_KEY"]
|
||||
env["TARGET_DEPLOYMENT"] = PROVIDER_MODEL
|
||||
env["OPTIMIZER_DEPLOYMENT"] = PROVIDER_MODEL
|
||||
env["AZURE_OPENAI_AUTH_MODE"] = "openai_compatible"
|
||||
env["AZURE_OPENAI_ENDPOINT"] = PROVIDER_ENDPOINT
|
||||
env["AZURE_OPENAI_API_KEY"] = env["DEEPSEEK_API_KEY"]
|
||||
# Provider-specific request fields are opt-in, never inferred from the
|
||||
# model name. For DeepSeek reasoning models, enable the thinking channel:
|
||||
env.setdefault("SKILLOPT_SLEEP_CHAT_EXTRA_BODY",
|
||||
json.dumps({"thinking": {"type": "enabled"}}))
|
||||
env.setdefault("SKILLOPT_SLEEP_COMPAT_MAX_TOKENS", "8192")
|
||||
else:
|
||||
# OPTIONAL, UNVERIFIED fallback: route the `claude` CLI backend through a
|
||||
# local Anthropic-compatible proxy (e.g. LiteLLM) to reach Gemini. There
|
||||
@@ -93,7 +95,10 @@ def main() -> None:
|
||||
"--model", PROVIDER_MODEL, "--project", PROJECT_DIR]
|
||||
|
||||
print(f"Running: {' '.join(args)}")
|
||||
subprocess.run(args, env=env, check=False)
|
||||
# Propagate the child's exit code so supervisors (watchdog.py, systemd,
|
||||
# Task Scheduler) see a failed sleep run as a failure, not a success.
|
||||
proc = subprocess.run(args, env=env, check=False)
|
||||
sys.exit(proc.returncode)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# OpenAI-compatible endpoints for SkillOpt-Sleep (DeepSeek, local vLLM, …)
|
||||
|
||||
This document describes a small enhancement to the `azure_openai` backend in
|
||||
This document describes an enhancement to the `azure_openai` backend in
|
||||
`skillopt_sleep/backend.py` that lets SkillOpt-Sleep drive **any
|
||||
OpenAI-compatible chat-completions endpoint** — for example DeepSeek's hosted
|
||||
API or a self-hosted vLLM/Ollama server — in addition to native Azure OpenAI
|
||||
@@ -9,48 +9,69 @@ nightly sleep cycle inside the Antigravity IDE against DeepSeek.
|
||||
|
||||
## What changed
|
||||
|
||||
Three focused changes in `AzureOpenAIBackend` (`skillopt_sleep/backend.py`),
|
||||
all backward-compatible — the default managed-identity path is unchanged:
|
||||
All changes are backward-compatible — the default managed-identity Azure path
|
||||
is unchanged:
|
||||
|
||||
1. **Endpoint resolution honors `AZURE_OPENAI_ENDPOINT`.**
|
||||
`__init__` now resolves the endpoint as `explicit arg` → `AZURE_OPENAI_ENDPOINT`
|
||||
env → the built-in `_AZURE_ENDPOINTS` table. Previously a non-Azure endpoint
|
||||
could not be supplied at all, so calls always went to a hardcoded
|
||||
`*.openai.azure.com` host.
|
||||
1. **CLI acceptance.** `skillopt-sleep run --backend azure_openai` is now an
|
||||
accepted choice in `skillopt_sleep/__main__.py` (it was previously rejected
|
||||
by argparse even though `get_backend()` understood the name).
|
||||
|
||||
2. **`openai_compatible` auth mode.** When
|
||||
2. **Endpoint resolution honors `AZURE_OPENAI_ENDPOINT`.**
|
||||
`AzureOpenAIBackend.__init__` resolves the endpoint as `explicit arg` →
|
||||
`AZURE_OPENAI_ENDPOINT` env → the built-in `_AZURE_ENDPOINTS` table.
|
||||
Previously a non-Azure endpoint could not be supplied at all.
|
||||
|
||||
3. **`openai_compatible` auth mode.** When
|
||||
`AZURE_OPENAI_AUTH_MODE=openai_compatible` (also accepts `compat`/`openai`),
|
||||
`_get_client()` builds a plain `openai.OpenAI(base_url=…)` client instead of
|
||||
an `AzureOpenAI` client. This mirrors the auth mode already supported by the
|
||||
sibling `skillopt/model/azure_openai.py` module, so the two are consistent.
|
||||
`_get_client()` builds a plain `openai.OpenAI(base_url=…)` client with
|
||||
`AZURE_OPENAI_API_KEY` instead of an `AzureOpenAI` client. This mirrors the
|
||||
auth mode already supported by the sibling `skillopt/model/azure_openai.py`
|
||||
module. (The `AzureOpenAI` client rewrites request URLs with Azure-only
|
||||
`?api-version=…` query params and deployment path segments, which non-Azure
|
||||
servers reject with `404 Resource not found` — the sleep cycle then scores
|
||||
every rollout `0.0` with no diagnostic.)
|
||||
|
||||
3. **DeepSeek request shape + error surfacing.** `_call()` sends `max_tokens`
|
||||
plus `extra_body={"thinking": {"type": "enabled"}}` for `deepseek*`
|
||||
deployments (the DeepSeek reasoning models expect this), and records the last
|
||||
exception in `self.last_call_error` so a failed night is diagnosable instead
|
||||
of collapsing to a silent empty response scored `0.0`.
|
||||
4. **Managed-identity credential guard.** The managed-identity path attaches an
|
||||
Azure AD bearer token to every request. If a custom endpoint outside
|
||||
`*.openai.azure.com` / `*.cognitiveservices.azure.com` is configured without
|
||||
explicit compat auth, the backend now raises a clear `ValueError` instead of
|
||||
sending Azure credentials to an arbitrary host.
|
||||
|
||||
### Why the previous behavior failed on non-Azure servers
|
||||
5. **Provider-neutral request shape.** In compat mode the backend sends only the
|
||||
standard OpenAI-compatible contract (`model`, `messages`, `max_tokens`).
|
||||
Provider-specific request fields are **opt-in** via environment variables
|
||||
(below) — nothing is inferred from model-name substrings.
|
||||
|
||||
The `AzureOpenAI` SDK client rewrites request URLs with Azure-only structure —
|
||||
a `?api-version=…` query string and deployment path segments. A non-Azure
|
||||
OpenAI-compatible server (DeepSeek, vLLM, …) does not recognize those routes and
|
||||
responds `404 Resource not found`. SkillOpt-Sleep then receives an empty
|
||||
response, the judge scores it `0.0`, and every night reports
|
||||
`baseline 0.0 -> candidate 0.0` with no usable diagnostic. Selecting a plain
|
||||
`OpenAI` client via `openai_compatible` mode avoids the Azure URL rewriting and
|
||||
talks to the endpoint directly.
|
||||
6. **Reliable error state.** `_call()` records the last exception in
|
||||
`self.last_call_error` (surfaced in `diagnostics.json`), clears it when a
|
||||
retry recovers, and sets an explicit `"empty response on all N attempts"`
|
||||
diagnostic when every attempt returns empty text.
|
||||
|
||||
## Configuration reference
|
||||
|
||||
SkillOpt-Sleep's `azure_openai` backend reads these environment variables
|
||||
(unprefixed only — the `OPTIMIZER_*`/`TARGET_*` dual-role variables belong to
|
||||
the separate `skillopt.model.azure_openai` module and are **not** used by the
|
||||
sleep cycle):
|
||||
|
||||
| Variable | Meaning |
|
||||
|---|---|
|
||||
| `AZURE_OPENAI_AUTH_MODE` | `openai_compatible` (or `compat`/`openai`) selects the plain OpenAI client. Unset/other = Azure managed identity (default). |
|
||||
| `AZURE_OPENAI_ENDPOINT` | Base URL of the server, e.g. `https://api.deepseek.com`. |
|
||||
| `AZURE_OPENAI_API_KEY` | API key sent by the compat client. |
|
||||
| `SKILLOPT_SLEEP_COMPAT_MAX_TOKENS` | Optional int (default `8192`): `max_tokens` sent in compat mode. |
|
||||
| `SKILLOPT_SLEEP_CHAT_EXTRA_BODY` | Optional JSON object passed as `extra_body` for provider-specific fields. |
|
||||
|
||||
## How to use it
|
||||
|
||||
Set four environment variables and run the cycle with `--backend azure_openai`:
|
||||
|
||||
```bash
|
||||
export AZURE_OPENAI_AUTH_MODE=openai_compatible
|
||||
export AZURE_OPENAI_ENDPOINT=https://api.deepseek.com # no /v1, no trailing path
|
||||
export AZURE_OPENAI_API_KEY=sk-... # your provider key
|
||||
# optimizer/target dual-role overrides are also honored, e.g.
|
||||
# OPTIMIZER_AZURE_OPENAI_AUTH_MODE / TARGET_AZURE_OPENAI_AUTH_MODE, etc.
|
||||
|
||||
# DeepSeek reasoning models: enable the thinking channel (opt-in, not inferred)
|
||||
export SKILLOPT_SLEEP_CHAT_EXTRA_BODY='{"thinking": {"type": "enabled"}}'
|
||||
export SKILLOPT_SLEEP_COMPAT_MAX_TOKENS=8192
|
||||
|
||||
skillopt-sleep run \
|
||||
--backend azure_openai \
|
||||
@@ -59,7 +80,9 @@ skillopt-sleep run \
|
||||
```
|
||||
|
||||
The same pattern works for any OpenAI-compatible server — point
|
||||
`AZURE_OPENAI_ENDPOINT` at it and set a matching `--model`.
|
||||
`AZURE_OPENAI_ENDPOINT` at it, set a matching `--model`, and omit
|
||||
`SKILLOPT_SLEEP_CHAT_EXTRA_BODY` unless your provider needs extra request
|
||||
fields.
|
||||
|
||||
## End-to-end integration: Antigravity + DeepSeek
|
||||
|
||||
@@ -68,14 +91,15 @@ was wired into the [Antigravity](https://antigravity.google/) agent IDE so the
|
||||
sleep cycle runs unattended:
|
||||
|
||||
- **`examples/runner.py`** — a thin launcher that loads a provider key from an
|
||||
`.env` file, exports the four variables above, and invokes `skillopt-sleep run`
|
||||
with the DeepSeek backend. It also implements a `session-end` hook that appends
|
||||
task-outcome metadata to a rollout-evidence log (wired to Antigravity's `Stop`
|
||||
hook) so future nights have richer sessions to mine.
|
||||
`.env` file, exports the variables above, invokes `skillopt-sleep run` with
|
||||
the DeepSeek backend, and **exits with the child's return code** so
|
||||
supervisors see failures as failures. It also implements a `session-end` hook
|
||||
that appends task-outcome metadata to a rollout-evidence log (wired to
|
||||
Antigravity's `Stop` hook) so future nights have richer sessions to mine.
|
||||
- **`examples/watchdog.py`** — a minimal supervisor loop that invokes the runner
|
||||
on a fixed interval (e.g. every 4 hours). On Windows this is registered as a
|
||||
Scheduled Task so it survives logout; on Linux/macOS a `systemd` timer or cron
|
||||
entry serves the same role.
|
||||
on a fixed interval (e.g. every 4 hours) and logs non-zero exits as failures.
|
||||
On Windows this is registered as a Scheduled Task so it survives logout; on
|
||||
Linux/macOS a `systemd` timer or cron entry serves the same role.
|
||||
|
||||
### Verified result
|
||||
|
||||
@@ -95,6 +119,11 @@ On a Windows 11 host, driving the cycle against `deepseek-v4-pro` in
|
||||
**rejected** a non-improving proposal (`0.3 → 0.3`), confirming the validation
|
||||
gate behaves normally on the new backend.
|
||||
|
||||
Deterministic no-network coverage for the new behavior lives in
|
||||
`tests/test_azure_openai_compat.py` (CLI acceptance, client selection,
|
||||
endpoint/auth guard, request kwargs, retry error-state, empty-response
|
||||
diagnostics, and runner exit-code propagation).
|
||||
|
||||
## A note on Gemini (optional, unverified fallback)
|
||||
|
||||
`examples/runner.py` also contains a fallback branch that, when only a Gemini key
|
||||
|
||||
Reference in New Issue
Block a user