fix(sleep): address review — CLI choice, auth guard, provider-neutral kwargs, tests

Addresses maintainer review on the OpenAI-compatible endpoints PR:

1. CLI: accept --backend azure_openai in skillopt_sleep/__main__.py (the
   documented command was rejected by the argparse choices).
2. Example runner: exit with the child's return code so watchdog/supervisors
   see a failed sleep run as a failure.
3. Error state: clear last_call_error when a retry recovers; set an explicit
   "empty response on all N attempts" diagnostic when every attempt returns
   empty text.
4. Security guard: the managed-identity path now refuses to send an Azure AD
   bearer token to any endpoint outside *.openai.azure.com /
   *.cognitiveservices.azure.com — a custom endpoint requires explicit
   AZURE_OPENAI_AUTH_MODE=openai_compatible + API key.
5. Provider-neutral requests: compat mode sends only the standard contract
   (max_tokens, default 8192 via SKILLOPT_SLEEP_COMPAT_MAX_TOKENS);
   provider-specific body fields are opt-in via SKILLOPT_SLEEP_CHAT_EXTRA_BODY
   (JSON) — the deepseek model-name inference is removed.
6. Docs: removed the unimplemented OPTIMIZER_*/TARGET_* env-var claim; added a
   configuration reference matching the implementation exactly.
7. Tests: tests/test_azure_openai_compat.py — 17 deterministic no-network
   unittest cases covering CLI acceptance, compat-vs-Azure client selection,
   endpoint resolution, the credential guard, request kwargs (opt-in extra
   body / token cap), retry-success error clearing, empty-response
   diagnostics, and runner exit-code propagation.

Re-verified live against DeepSeek (deepseek-v4-pro, openai_compatible mode)
after the rework: client type OpenAI, completion returned, no error state.
This commit is contained in:
Alphaxalchemy
2026-07-14 01:22:19 -06:00
parent af33f6c338
commit 4ff77b71ff
5 changed files with 423 additions and 57 deletions
+12 -7
View File
@@ -72,12 +72,14 @@ def main() -> None:
# OpenAI-compatible path — see docs/sleep/openai-compatible-endpoints.md # OpenAI-compatible path — see docs/sleep/openai-compatible-endpoints.md
backend = "azure_openai" backend = "azure_openai"
env["PYTHONIOENCODING"] = "utf-8" env["PYTHONIOENCODING"] = "utf-8"
for prefix in ("", "OPTIMIZER_", "TARGET_"): env["AZURE_OPENAI_AUTH_MODE"] = "openai_compatible"
env[f"{prefix}AZURE_OPENAI_AUTH_MODE"] = "openai_compatible" env["AZURE_OPENAI_ENDPOINT"] = PROVIDER_ENDPOINT
env[f"{prefix}AZURE_OPENAI_ENDPOINT"] = PROVIDER_ENDPOINT env["AZURE_OPENAI_API_KEY"] = env["DEEPSEEK_API_KEY"]
env[f"{prefix}AZURE_OPENAI_API_KEY"] = env["DEEPSEEK_API_KEY"] # Provider-specific request fields are opt-in, never inferred from the
env["TARGET_DEPLOYMENT"] = PROVIDER_MODEL # model name. For DeepSeek reasoning models, enable the thinking channel:
env["OPTIMIZER_DEPLOYMENT"] = PROVIDER_MODEL env.setdefault("SKILLOPT_SLEEP_CHAT_EXTRA_BODY",
json.dumps({"thinking": {"type": "enabled"}}))
env.setdefault("SKILLOPT_SLEEP_COMPAT_MAX_TOKENS", "8192")
else: else:
# OPTIONAL, UNVERIFIED fallback: route the `claude` CLI backend through a # OPTIONAL, UNVERIFIED fallback: route the `claude` CLI backend through a
# local Anthropic-compatible proxy (e.g. LiteLLM) to reach Gemini. There # local Anthropic-compatible proxy (e.g. LiteLLM) to reach Gemini. There
@@ -93,7 +95,10 @@ def main() -> None:
"--model", PROVIDER_MODEL, "--project", PROJECT_DIR] "--model", PROVIDER_MODEL, "--project", PROJECT_DIR]
print(f"Running: {' '.join(args)}") print(f"Running: {' '.join(args)}")
subprocess.run(args, env=env, check=False) # Propagate the child's exit code so supervisors (watchdog.py, systemd,
# Task Scheduler) see a failed sleep run as a failure, not a success.
proc = subprocess.run(args, env=env, check=False)
sys.exit(proc.returncode)
if __name__ == "__main__": if __name__ == "__main__":
+67 -38
View File
@@ -1,6 +1,6 @@
# OpenAI-compatible endpoints for SkillOpt-Sleep (DeepSeek, local vLLM, …) # OpenAI-compatible endpoints for SkillOpt-Sleep (DeepSeek, local vLLM, …)
This document describes a small enhancement to the `azure_openai` backend in This document describes an enhancement to the `azure_openai` backend in
`skillopt_sleep/backend.py` that lets SkillOpt-Sleep drive **any `skillopt_sleep/backend.py` that lets SkillOpt-Sleep drive **any
OpenAI-compatible chat-completions endpoint** — for example DeepSeek's hosted OpenAI-compatible chat-completions endpoint** — for example DeepSeek's hosted
API or a self-hosted vLLM/Ollama server — in addition to native Azure OpenAI API or a self-hosted vLLM/Ollama server — in addition to native Azure OpenAI
@@ -9,48 +9,69 @@ nightly sleep cycle inside the Antigravity IDE against DeepSeek.
## What changed ## What changed
Three focused changes in `AzureOpenAIBackend` (`skillopt_sleep/backend.py`), All changes are backward-compatible — the default managed-identity Azure path
all backward-compatible — the default managed-identity path is unchanged: is unchanged:
1. **Endpoint resolution honors `AZURE_OPENAI_ENDPOINT`.** 1. **CLI acceptance.** `skillopt-sleep run --backend azure_openai` is now an
`__init__` now resolves the endpoint as `explicit arg``AZURE_OPENAI_ENDPOINT` accepted choice in `skillopt_sleep/__main__.py` (it was previously rejected
env → the built-in `_AZURE_ENDPOINTS` table. Previously a non-Azure endpoint by argparse even though `get_backend()` understood the name).
could not be supplied at all, so calls always went to a hardcoded
`*.openai.azure.com` host.
2. **`openai_compatible` auth mode.** When 2. **Endpoint resolution honors `AZURE_OPENAI_ENDPOINT`.**
`AzureOpenAIBackend.__init__` resolves the endpoint as `explicit arg`
`AZURE_OPENAI_ENDPOINT` env → the built-in `_AZURE_ENDPOINTS` table.
Previously a non-Azure endpoint could not be supplied at all.
3. **`openai_compatible` auth mode.** When
`AZURE_OPENAI_AUTH_MODE=openai_compatible` (also accepts `compat`/`openai`), `AZURE_OPENAI_AUTH_MODE=openai_compatible` (also accepts `compat`/`openai`),
`_get_client()` builds a plain `openai.OpenAI(base_url=…)` client instead of `_get_client()` builds a plain `openai.OpenAI(base_url=…)` client with
an `AzureOpenAI` client. This mirrors the auth mode already supported by the `AZURE_OPENAI_API_KEY` instead of an `AzureOpenAI` client. This mirrors the
sibling `skillopt/model/azure_openai.py` module, so the two are consistent. auth mode already supported by the sibling `skillopt/model/azure_openai.py`
module. (The `AzureOpenAI` client rewrites request URLs with Azure-only
`?api-version=…` query params and deployment path segments, which non-Azure
servers reject with `404 Resource not found` — the sleep cycle then scores
every rollout `0.0` with no diagnostic.)
3. **DeepSeek request shape + error surfacing.** `_call()` sends `max_tokens` 4. **Managed-identity credential guard.** The managed-identity path attaches an
plus `extra_body={"thinking": {"type": "enabled"}}` for `deepseek*` Azure AD bearer token to every request. If a custom endpoint outside
deployments (the DeepSeek reasoning models expect this), and records the last `*.openai.azure.com` / `*.cognitiveservices.azure.com` is configured without
exception in `self.last_call_error` so a failed night is diagnosable instead explicit compat auth, the backend now raises a clear `ValueError` instead of
of collapsing to a silent empty response scored `0.0`. sending Azure credentials to an arbitrary host.
### Why the previous behavior failed on non-Azure servers 5. **Provider-neutral request shape.** In compat mode the backend sends only the
standard OpenAI-compatible contract (`model`, `messages`, `max_tokens`).
Provider-specific request fields are **opt-in** via environment variables
(below) — nothing is inferred from model-name substrings.
The `AzureOpenAI` SDK client rewrites request URLs with Azure-only structure — 6. **Reliable error state.** `_call()` records the last exception in
a `?api-version=…` query string and deployment path segments. A non-Azure `self.last_call_error` (surfaced in `diagnostics.json`), clears it when a
OpenAI-compatible server (DeepSeek, vLLM, …) does not recognize those routes and retry recovers, and sets an explicit `"empty response on all N attempts"`
responds `404 Resource not found`. SkillOpt-Sleep then receives an empty diagnostic when every attempt returns empty text.
response, the judge scores it `0.0`, and every night reports
`baseline 0.0 -> candidate 0.0` with no usable diagnostic. Selecting a plain ## Configuration reference
`OpenAI` client via `openai_compatible` mode avoids the Azure URL rewriting and
talks to the endpoint directly. SkillOpt-Sleep's `azure_openai` backend reads these environment variables
(unprefixed only — the `OPTIMIZER_*`/`TARGET_*` dual-role variables belong to
the separate `skillopt.model.azure_openai` module and are **not** used by the
sleep cycle):
| Variable | Meaning |
|---|---|
| `AZURE_OPENAI_AUTH_MODE` | `openai_compatible` (or `compat`/`openai`) selects the plain OpenAI client. Unset/other = Azure managed identity (default). |
| `AZURE_OPENAI_ENDPOINT` | Base URL of the server, e.g. `https://api.deepseek.com`. |
| `AZURE_OPENAI_API_KEY` | API key sent by the compat client. |
| `SKILLOPT_SLEEP_COMPAT_MAX_TOKENS` | Optional int (default `8192`): `max_tokens` sent in compat mode. |
| `SKILLOPT_SLEEP_CHAT_EXTRA_BODY` | Optional JSON object passed as `extra_body` for provider-specific fields. |
## How to use it ## How to use it
Set four environment variables and run the cycle with `--backend azure_openai`:
```bash ```bash
export AZURE_OPENAI_AUTH_MODE=openai_compatible export AZURE_OPENAI_AUTH_MODE=openai_compatible
export AZURE_OPENAI_ENDPOINT=https://api.deepseek.com # no /v1, no trailing path export AZURE_OPENAI_ENDPOINT=https://api.deepseek.com # no /v1, no trailing path
export AZURE_OPENAI_API_KEY=sk-... # your provider key export AZURE_OPENAI_API_KEY=sk-... # your provider key
# optimizer/target dual-role overrides are also honored, e.g.
# OPTIMIZER_AZURE_OPENAI_AUTH_MODE / TARGET_AZURE_OPENAI_AUTH_MODE, etc. # DeepSeek reasoning models: enable the thinking channel (opt-in, not inferred)
export SKILLOPT_SLEEP_CHAT_EXTRA_BODY='{"thinking": {"type": "enabled"}}'
export SKILLOPT_SLEEP_COMPAT_MAX_TOKENS=8192
skillopt-sleep run \ skillopt-sleep run \
--backend azure_openai \ --backend azure_openai \
@@ -59,7 +80,9 @@ skillopt-sleep run \
``` ```
The same pattern works for any OpenAI-compatible server — point The same pattern works for any OpenAI-compatible server — point
`AZURE_OPENAI_ENDPOINT` at it and set a matching `--model`. `AZURE_OPENAI_ENDPOINT` at it, set a matching `--model`, and omit
`SKILLOPT_SLEEP_CHAT_EXTRA_BODY` unless your provider needs extra request
fields.
## End-to-end integration: Antigravity + DeepSeek ## End-to-end integration: Antigravity + DeepSeek
@@ -68,14 +91,15 @@ was wired into the [Antigravity](https://antigravity.google/) agent IDE so the
sleep cycle runs unattended: sleep cycle runs unattended:
- **`examples/runner.py`** — a thin launcher that loads a provider key from an - **`examples/runner.py`** — a thin launcher that loads a provider key from an
`.env` file, exports the four variables above, and invokes `skillopt-sleep run` `.env` file, exports the variables above, invokes `skillopt-sleep run` with
with the DeepSeek backend. It also implements a `session-end` hook that appends the DeepSeek backend, and **exits with the child's return code** so
task-outcome metadata to a rollout-evidence log (wired to Antigravity's `Stop` supervisors see failures as failures. It also implements a `session-end` hook
hook) so future nights have richer sessions to mine. that appends task-outcome metadata to a rollout-evidence log (wired to
Antigravity's `Stop` hook) so future nights have richer sessions to mine.
- **`examples/watchdog.py`** — a minimal supervisor loop that invokes the runner - **`examples/watchdog.py`** — a minimal supervisor loop that invokes the runner
on a fixed interval (e.g. every 4 hours). On Windows this is registered as a on a fixed interval (e.g. every 4 hours) and logs non-zero exits as failures.
Scheduled Task so it survives logout; on Linux/macOS a `systemd` timer or cron On Windows this is registered as a Scheduled Task so it survives logout; on
entry serves the same role. Linux/macOS a `systemd` timer or cron entry serves the same role.
### Verified result ### Verified result
@@ -95,6 +119,11 @@ On a Windows 11 host, driving the cycle against `deepseek-v4-pro` in
**rejected** a non-improving proposal (`0.3 → 0.3`), confirming the validation **rejected** a non-improving proposal (`0.3 → 0.3`), confirming the validation
gate behaves normally on the new backend. gate behaves normally on the new backend.
Deterministic no-network coverage for the new behavior lives in
`tests/test_azure_openai_compat.py` (CLI acceptance, client selection,
endpoint/auth guard, request kwargs, retry error-state, empty-response
diagnostics, and runner exit-code propagation).
## A note on Gemini (optional, unverified fallback) ## A note on Gemini (optional, unverified fallback)
`examples/runner.py` also contains a fallback branch that, when only a Gemini key `examples/runner.py` also contains a fallback branch that, when only a Gemini key
+2 -1
View File
@@ -70,7 +70,8 @@ def _add_common(p: argparse.ArgumentParser) -> None:
p.add_argument("--project", default="") p.add_argument("--project", default="")
p.add_argument("--scope", default="", choices=["", "all", "invoked"]) p.add_argument("--scope", default="", choices=["", "all", "invoked"])
p.add_argument("--backend", default="", p.add_argument("--backend", default="",
choices=["", "mock", "claude", "codex", "copilot", "handoff"]) choices=["", "mock", "claude", "codex", "copilot", "handoff",
"azure_openai"])
p.add_argument("--model", default="") p.add_argument("--model", default="")
p.add_argument("--codex-path", default="", help="path to the real @openai/codex binary") p.add_argument("--codex-path", default="", help="path to the real @openai/codex binary")
p.add_argument("--claude-home", default="", help="override ~/.claude (also isolates state)") p.add_argument("--claude-home", default="", help="override ~/.claude (also isolates state)")
+94 -11
View File
@@ -1222,16 +1222,33 @@ _AZURE_MI_CLIENT_ID = "8cafa2b1-a2a7-4ad9-814a-ffe4aed7e800"
class AzureOpenAIBackend(CliBackend): class AzureOpenAIBackend(CliBackend):
"""Drives Azure OpenAI gpt-5.x deployments via managed identity. """Drives Azure OpenAI gpt-5.x deployments via managed identity, or any
OpenAI-compatible chat-completions server via explicit compat auth.
Mirrors the intern's blog_1 setup (avail_api.md): managed-identity auth, the Mirrors the intern's blog_1 setup (avail_api.md): managed-identity auth, the
same endpoints/deployments. Reuses CliBackend's attempt/judge/reflect prompts same endpoints/deployments. Reuses CliBackend's attempt/judge/reflect prompts
and JSON parsing; only _call() differs. openai + azure-identity are lazy and JSON parsing; only _call() differs. openai + azure-identity are lazy
imported so the mock/CLI paths stay dependency-free. imported so the mock/CLI paths stay dependency-free.
OpenAI-compatible mode (opt-in, matching skillopt.model.azure_openai):
* AZURE_OPENAI_AUTH_MODE=openai_compatible selects a plain ``openai.OpenAI``
client with AZURE_OPENAI_API_KEY against AZURE_OPENAI_ENDPOINT.
* SKILLOPT_SLEEP_COMPAT_MAX_TOKENS (int, default 8192) caps completion
length via the standard ``max_tokens`` parameter in compat mode.
* SKILLOPT_SLEEP_CHAT_EXTRA_BODY (JSON object) is passed as ``extra_body``
for provider-specific request fields (e.g. DeepSeek's
``{"thinking": {"type": "enabled"}}``). Nothing provider-specific is
inferred from model names.
""" """
name = "azure" name = "azure"
_COMPAT_MODES = {"openai_compatible", "compat", "openai"}
# Hosts the managed-identity (AAD bearer token) path may talk to. A custom
# endpoint outside these domains must use explicit compat auth — we never
# send Azure credentials to an arbitrary host.
_AZURE_HOST_SUFFIXES = (".openai.azure.com", ".cognitiveservices.azure.com")
def __init__(self, deployment: str = "", endpoint: str = "", timeout: int = 180, def __init__(self, deployment: str = "", endpoint: str = "", timeout: int = 180,
api_version: str = "2024-12-01-preview") -> None: api_version: str = "2024-12-01-preview") -> None:
super().__init__(model=deployment or "gpt-5.5", timeout=timeout) super().__init__(model=deployment or "gpt-5.5", timeout=timeout)
@@ -1248,6 +1265,9 @@ class AzureOpenAIBackend(CliBackend):
self.api_version = api_version self.api_version = api_version
self.name = f"azure:{self.deployment}" self.name = f"azure:{self.deployment}"
self._client = None self._client = None
# Opt-in request knobs (read once; see class docstring).
self.compat_max_tokens = self._read_compat_max_tokens()
self.chat_extra_body = self._read_chat_extra_body()
@staticmethod @staticmethod
def _endpoint_for(deployment: str) -> str: def _endpoint_for(deployment: str) -> str:
@@ -1256,6 +1276,44 @@ class AzureOpenAIBackend(CliBackend):
return ep return ep
return "https://oaidr9.openai.azure.com/" return "https://oaidr9.openai.azure.com/"
@classmethod
def _compat_mode(cls) -> bool:
return os.environ.get("AZURE_OPENAI_AUTH_MODE", "").strip().lower() in cls._COMPAT_MODES
@staticmethod
def _read_compat_max_tokens() -> int:
raw = os.environ.get("SKILLOPT_SLEEP_COMPAT_MAX_TOKENS", "").strip()
try:
val = int(raw) if raw else 8192
return val if val > 0 else 8192
except ValueError:
import logging
logging.getLogger("skillopt_sleep").warning(
"ignoring non-integer SKILLOPT_SLEEP_COMPAT_MAX_TOKENS=%r", raw)
return 8192
@staticmethod
def _read_chat_extra_body() -> Optional[Dict[str, Any]]:
raw = os.environ.get("SKILLOPT_SLEEP_CHAT_EXTRA_BODY", "").strip()
if not raw:
return None
try:
parsed = json.loads(raw)
except ValueError:
parsed = None
if isinstance(parsed, dict) and parsed:
return parsed
import logging
logging.getLogger("skillopt_sleep").warning(
"ignoring SKILLOPT_SLEEP_CHAT_EXTRA_BODY: not a non-empty JSON object: %r",
raw[:100])
return None
def _is_azure_host(self) -> bool:
from urllib.parse import urlparse
host = (urlparse(self.endpoint).hostname or "").lower()
return host.endswith(self._AZURE_HOST_SUFFIXES)
def _get_client(self): def _get_client(self):
if self._client is None: if self._client is None:
# AZURE_OPENAI_AUTH_MODE=openai_compatible (matching the sibling # AZURE_OPENAI_AUTH_MODE=openai_compatible (matching the sibling
@@ -1264,8 +1322,7 @@ class AzureOpenAIBackend(CliBackend):
# endpoint work: the AzureOpenAI client would otherwise rewrite the # endpoint work: the AzureOpenAI client would otherwise rewrite the
# URL with Azure-only query params (?api-version=...) and deployment # URL with Azure-only query params (?api-version=...) and deployment
# path segments, which non-Azure servers reject with a 404. # path segments, which non-Azure servers reject with a 404.
auth_mode = os.environ.get("AZURE_OPENAI_AUTH_MODE", "").strip().lower() if self._compat_mode():
if auth_mode in {"openai_compatible", "compat", "openai"}:
from openai import OpenAI from openai import OpenAI
self._client = OpenAI( self._client = OpenAI(
base_url=self.endpoint.rstrip("/"), base_url=self.endpoint.rstrip("/"),
@@ -1273,6 +1330,17 @@ class AzureOpenAIBackend(CliBackend):
max_retries=4, max_retries=4,
) )
else: else:
# Security guard: the managed-identity path attaches an Azure AD
# bearer token to every request. Refuse to do that for a custom
# non-Azure endpoint — leaking a live AAD token to an arbitrary
# host must be impossible by (mis)configuration.
if not self._is_azure_host():
raise ValueError(
"azure_openai backend: refusing to send Azure managed-identity "
f"credentials to non-Azure endpoint {self.endpoint!r}. For "
"OpenAI-compatible servers set AZURE_OPENAI_AUTH_MODE="
"openai_compatible and AZURE_OPENAI_API_KEY."
)
from azure.identity import ManagedIdentityCredential, get_bearer_token_provider from azure.identity import ManagedIdentityCredential, get_bearer_token_provider
from openai import AzureOpenAI from openai import AzureOpenAI
cred = ManagedIdentityCredential(client_id=_AZURE_MI_CLIENT_ID) cred = ManagedIdentityCredential(client_id=_AZURE_MI_CLIENT_ID)
@@ -1298,20 +1366,25 @@ class AzureOpenAIBackend(CliBackend):
client = self._get_client() client = self._get_client()
last_exc = None last_exc = None
for attempt in range(max(1, retries)): n_attempts = max(1, retries)
for attempt in range(n_attempts):
try: try:
kwargs: Dict[str, Any] = { kwargs: Dict[str, Any] = {
"model": self.deployment, "model": self.deployment,
"messages": [{"role": "user", "content": prompt}], "messages": [{"role": "user", "content": prompt}],
} }
# DeepSeek's reasoning models expect `max_tokens` and an # Provider-neutral request shape: compat mode speaks the
# extra_body flag to enable the thinking channel; the Azure # standard OpenAI-compatible contract (`max_tokens`); the Azure
# gpt-5.x deployments use `max_completion_tokens`. # gpt-5.x deployments require `max_completion_tokens`. Any
if "deepseek" in self.deployment.lower(): # provider-specific body fields are opt-in via
kwargs["max_tokens"] = 8192 # SKILLOPT_SLEEP_CHAT_EXTRA_BODY (see class docstring) — never
kwargs["extra_body"] = {"thinking": {"type": "enabled"}} # inferred from the model name.
if self._compat_mode():
kwargs["max_tokens"] = self.compat_max_tokens
else: else:
kwargs["max_completion_tokens"] = 16384 kwargs["max_completion_tokens"] = 16384
if self.chat_extra_body:
kwargs["extra_body"] = self.chat_extra_body
resp = client.chat.completions.create(**kwargs) resp = client.chat.completions.create(**kwargs)
text = (resp.choices[0].message.content or "").strip() text = (resp.choices[0].message.content or "").strip()
try: try:
@@ -1320,6 +1393,9 @@ class AzureOpenAIBackend(CliBackend):
except Exception: except Exception:
pass pass
if text: if text:
# A recovered retry must not leave a stale error behind:
# last_call_error always reflects the LATEST outcome.
self.last_call_error = ""
return text return text
# empty but no exception: model genuinely returned nothing — one # empty but no exception: model genuinely returned nothing — one
# quick retry can help (reasoning models occasionally yield empty) # quick retry can help (reasoning models occasionally yield empty)
@@ -1330,8 +1406,15 @@ class AzureOpenAIBackend(CliBackend):
# from a mis-pointed endpoint) instead of a silent empty->0. # from a mis-pointed endpoint) instead of a silent empty->0.
self.last_call_error = str(e)[:500] self.last_call_error = str(e)[:500]
# backoff before next try (skip after the final attempt) # backoff before next try (skip after the final attempt)
if attempt < retries - 1: if attempt < n_attempts - 1:
_t.sleep(min(8.0, (2 ** attempt) * 0.5) + _r.random() * 0.4) _t.sleep(min(8.0, (2 ** attempt) * 0.5) + _r.random() * 0.4)
if last_exc == "empty-response":
# All attempts "succeeded" HTTP-wise but carried no text — say so,
# otherwise a run of empty completions is indistinguishable from a
# never-called backend in diagnostics.
self.last_call_error = (
f"{self.deployment}: empty response on all {n_attempts} attempts"
)
return "" return ""
+248
View File
@@ -0,0 +1,248 @@
"""Tests for the azure_openai backend's OpenAI-compatible mode.
Pure-stdlib (unittest), deterministic, NO network, no API key. Covers the
integration points requested in review of the openai-compatible feature:
* CLI acceptance of --backend azure_openai
* compat-vs-Azure client selection
* endpoint resolution and the managed-identity credential guard
* provider-neutral request kwargs (opt-in extra body / token cap)
* retry-success error clearing + empty-response diagnostics
* example runner exit-code propagation
Run: python -m unittest tests.test_azure_openai_compat
"""
from __future__ import annotations
import argparse
import contextlib
import importlib.util
import io
import json
import os
import sys
import unittest
from pathlib import Path
from types import SimpleNamespace
from unittest import mock
from skillopt_sleep.__main__ import _add_common
from skillopt_sleep.backend import AzureOpenAIBackend
try:
import openai # noqa: F401
HAVE_OPENAI = True
except ImportError:
HAVE_OPENAI = False
COMPAT_ENV = {
"AZURE_OPENAI_AUTH_MODE": "openai_compatible",
"AZURE_OPENAI_ENDPOINT": "https://compat.example.test",
"AZURE_OPENAI_API_KEY": "sk-test-not-a-real-key",
}
def _resp(text):
return SimpleNamespace(
choices=[SimpleNamespace(message=SimpleNamespace(content=text))],
usage=None,
)
class _FakeCompletions:
"""Scripted chat.completions.create: replies are strings or Exceptions."""
def __init__(self, replies):
self.replies = list(replies)
self.calls = []
def create(self, **kwargs):
self.calls.append(kwargs)
item = self.replies.pop(0)
if isinstance(item, Exception):
raise item
return _resp(item)
def _fake_client(replies):
return SimpleNamespace(chat=SimpleNamespace(completions=_FakeCompletions(replies)))
def _backend_with(replies, env):
"""Build a backend under `env` with a scripted fake client injected."""
with mock.patch.dict(os.environ, env, clear=True):
be = AzureOpenAIBackend(deployment="any-model")
be._client = _fake_client(replies)
return be
class TestCliAcceptance(unittest.TestCase):
def test_backend_azure_openai_is_accepted(self):
p = argparse.ArgumentParser()
_add_common(p)
ns = p.parse_args(["--backend", "azure_openai"])
self.assertEqual(ns.backend, "azure_openai")
class TestClientSelection(unittest.TestCase):
@unittest.skipUnless(HAVE_OPENAI, "openai package not installed")
def test_compat_mode_builds_plain_openai_client(self):
from openai import AzureOpenAI, OpenAI
# Overlay (clear=False): constructing a real OpenAI client builds an
# SSL context, which needs the ambient cert-related env vars intact.
with mock.patch.dict(os.environ, COMPAT_ENV, clear=False):
be = AzureOpenAIBackend(deployment="some-model", endpoint=COMPAT_ENV["AZURE_OPENAI_ENDPOINT"])
client = be._get_client()
self.assertIsInstance(client, OpenAI)
self.assertNotIsInstance(client, AzureOpenAI)
self.assertIn("compat.example.test", str(client.base_url))
def test_managed_identity_refuses_non_azure_endpoint(self):
# No compat auth mode + a custom non-Azure endpoint: the backend must
# raise instead of sending an Azure AD bearer token to that host. The
# guard fires before azure.identity is imported, so this test needs
# neither azure-identity nor any network.
env = {"AZURE_OPENAI_ENDPOINT": "https://api.deepseek.com"}
with mock.patch.dict(os.environ, env, clear=True):
be = AzureOpenAIBackend(deployment="some-model")
with self.assertRaises(ValueError) as ctx:
be._get_client()
self.assertIn("openai_compatible", str(ctx.exception))
def test_azure_host_detection(self):
with mock.patch.dict(os.environ, {}, clear=True):
be = AzureOpenAIBackend(deployment="gpt-5.5") # table endpoint
self.assertTrue(be._is_azure_host())
be2 = AzureOpenAIBackend(
deployment="gpt-5.5", endpoint="https://api.deepseek.com")
self.assertFalse(be2._is_azure_host())
class TestEndpointResolution(unittest.TestCase):
def test_env_endpoint_is_honored(self):
env = {"AZURE_OPENAI_ENDPOINT": "https://compat.example.test"}
with mock.patch.dict(os.environ, env, clear=True):
be = AzureOpenAIBackend(deployment="some-model")
self.assertEqual(be.endpoint, "https://compat.example.test")
def test_explicit_arg_beats_env(self):
env = {"AZURE_OPENAI_ENDPOINT": "https://env.example.test"}
with mock.patch.dict(os.environ, env, clear=True):
be = AzureOpenAIBackend(deployment="m", endpoint="https://arg.example.test")
self.assertEqual(be.endpoint, "https://arg.example.test")
def test_table_fallback_without_env(self):
with mock.patch.dict(os.environ, {}, clear=True):
be = AzureOpenAIBackend(deployment="gpt-5.5")
self.assertTrue(be.endpoint.endswith(".openai.azure.com/"))
class TestRequestKwargs(unittest.TestCase):
def test_compat_mode_sends_standard_max_tokens(self):
be = _backend_with(["hi"], COMPAT_ENV)
with mock.patch.dict(os.environ, COMPAT_ENV, clear=True):
out = be._call("p", retries=1)
self.assertEqual(out, "hi")
(call,) = be._client.chat.completions.calls
self.assertEqual(call["max_tokens"], 8192)
self.assertNotIn("max_completion_tokens", call)
self.assertNotIn("extra_body", call)
def test_compat_max_tokens_env_override(self):
env = dict(COMPAT_ENV, SKILLOPT_SLEEP_COMPAT_MAX_TOKENS="4096")
be = _backend_with(["hi"], env)
with mock.patch.dict(os.environ, env, clear=True):
be._call("p", retries=1)
(call,) = be._client.chat.completions.calls
self.assertEqual(call["max_tokens"], 4096)
def test_extra_body_is_opt_in_via_env_not_model_name(self):
# A deepseek-named model WITHOUT the env knob gets a pure standard
# request (nothing inferred from the name)...
with mock.patch.dict(os.environ, COMPAT_ENV, clear=True):
be = AzureOpenAIBackend(deployment="deepseek-v4-pro")
be._client = _fake_client(["hi"])
be._call("p", retries=1)
(call,) = be._client.chat.completions.calls
self.assertNotIn("extra_body", call)
# ...and any model WITH the env knob gets exactly the configured body.
body = {"thinking": {"type": "enabled"}}
env = dict(COMPAT_ENV, SKILLOPT_SLEEP_CHAT_EXTRA_BODY=json.dumps(body))
be2 = _backend_with(["hi"], env)
with mock.patch.dict(os.environ, env, clear=True):
be2._call("p", retries=1)
(call2,) = be2._client.chat.completions.calls
self.assertEqual(call2["extra_body"], body)
def test_malformed_extra_body_is_ignored(self):
env = dict(COMPAT_ENV, SKILLOPT_SLEEP_CHAT_EXTRA_BODY="{not json")
be = _backend_with(["hi"], env)
with mock.patch.dict(os.environ, env, clear=True):
be._call("p", retries=1)
(call,) = be._client.chat.completions.calls
self.assertNotIn("extra_body", call)
def test_azure_mode_sends_max_completion_tokens(self):
be = _backend_with(["hi"], {}) # no compat auth mode
with mock.patch.dict(os.environ, {}, clear=True):
be._call("p", retries=1)
(call,) = be._client.chat.completions.calls
self.assertEqual(call["max_completion_tokens"], 16384)
self.assertNotIn("max_tokens", call)
class TestErrorState(unittest.TestCase):
def test_recovered_retry_clears_last_call_error(self):
be = _backend_with([RuntimeError("transient boom"), "recovered"], COMPAT_ENV)
with mock.patch.dict(os.environ, COMPAT_ENV, clear=True), \
mock.patch("time.sleep"):
out = be._call("p", retries=2)
self.assertEqual(out, "recovered")
self.assertEqual(be.last_call_error, "")
def test_all_empty_responses_set_diagnostic(self):
be = _backend_with(["", ""], COMPAT_ENV)
with mock.patch.dict(os.environ, COMPAT_ENV, clear=True), \
mock.patch("time.sleep"):
out = be._call("p", retries=2)
self.assertEqual(out, "")
self.assertIn("empty response on all 2 attempts", be.last_call_error)
def test_persistent_exception_is_surfaced(self):
be = _backend_with([RuntimeError("boom-1"), RuntimeError("boom-2")], COMPAT_ENV)
with mock.patch.dict(os.environ, COMPAT_ENV, clear=True), \
mock.patch("time.sleep"):
out = be._call("p", retries=2)
self.assertEqual(out, "")
self.assertIn("boom-2", be.last_call_error)
class TestRunnerExitCode(unittest.TestCase):
RUNNER = Path(__file__).resolve().parent.parent / "docs" / "sleep" / "examples" / "runner.py"
def _load_runner(self):
spec = importlib.util.spec_from_file_location("example_runner", self.RUNNER)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
return mod
def _run_with_child_rc(self, rc):
env = {"DEEPSEEK_API_KEY": "sk-test-not-a-real-key"}
with mock.patch.dict(os.environ, env, clear=True):
mod = self._load_runner()
fake = SimpleNamespace(run=lambda *a, **k: SimpleNamespace(returncode=rc))
with mock.patch.object(mod, "subprocess", fake), \
mock.patch.object(sys, "argv", ["runner.py", "run"]), \
contextlib.redirect_stdout(io.StringIO()):
with self.assertRaises(SystemExit) as ctx:
mod.main()
return ctx.exception.code
def test_child_failure_propagates(self):
self.assertEqual(self._run_with_child_rc(7), 7)
def test_child_success_exits_zero(self):
self.assertEqual(self._run_with_child_rc(0), 0)
if __name__ == "__main__":
unittest.main()