Add a New Model Backend

SkillOpt's model layer is function-based: each chat backend is a Python module that exposes the call, token-tracking, and deployment-setting functions used by skillopt.model. There is no backend base class or registry object to subclass.

Built-in: the generic OpenAI-compatible backend

!!! note "Version requirement" This backend landed after v0.2.0. Install from the latest main until it is included in the next release.

Before writing a new backend, check whether your provider already speaks the OpenAI Chat Completions protocol. Most do, in which case you can use the built-in openai_compatible backend (skillopt/model/openai_compatible_backend.py) with no code changes.

A single base_url + api_key pair lets you point SkillOpt at, for example:

Provider	`base_url`	Example model
DeepSeek	`https://api.deepseek.com/v1`	`deepseek-chat`
Groq	`https://api.groq.com/openai/v1`	`llama-3.3-70b-versatile`
Together AI	`https://api.together.xyz/v1`	`meta-llama/Llama-3.3-70B-Instruct-Turbo`
Ollama (local)	`http://localhost:11434/v1`	`qwen2.5:7b`
vLLM / SGLang / TGI	`http://localhost:8000/v1`	your served model
LiteLLM proxy	`http://localhost:4000`	any proxied model
OpenRouter / Fireworks / xAI / …	provider base URL	provider model id

Python API

Select and configure the backend directly when embedding SkillOpt as a Python library:

import skillopt.model as model

# Use the generic backend for both optimizer and target calls.
model.set_backend("openai_compatible")
model.configure_openai_compatible(
    base_url="https://api.deepseek.com/v1",
    api_key="sk-...",
    model="deepseek-chat",
)

configure_openai_compatible() also accepts optimizer_* and target_* arguments when the two roles use different endpoints or models.

Environment variables

The shared variables below configure both roles. Role-specific OPTIMIZER_OPENAI_COMPATIBLE_* and TARGET_OPENAI_COMPATIBLE_* variables take precedence:

export OPENAI_COMPATIBLE_BASE_URL="https://api.groq.com/openai/v1"
export OPENAI_COMPATIBLE_API_KEY="gsk_..."
export OPENAI_COMPATIBLE_MODEL="llama-3.3-70b-versatile"
# Optional: OPENAI_COMPATIBLE_TEMPERATURE, _MAX_TOKENS, _TIMEOUT_SECONDS

For direct library use, OPTIMIZER_BACKEND=openai_compatible and/or TARGET_BACKEND=openai_compatible select the role. The training and evaluation scripts resolve backend selection from their config, so set the split fields explicitly there:

model:
  optimizer_backend: openai_compatible
  target_backend: openai_compatible
  optimizer: llama-3.3-70b-versatile
  target: llama-3.3-70b-versatile

Equivalently, override those fields on the command line:

python scripts/train.py --config configs/searchqa/default.yaml \
  --cfg-options \
  model.optimizer_backend=openai_compatible \
  model.target_backend=openai_compatible \
  model.optimizer=llama-3.3-70b-versatile \
  model.target=llama-3.3-70b-versatile

Do not rely on the legacy high-level model.backend label to replace the two role-specific fields in a structured config.

The generic backend uses the official openai SDK and the Chat Completions API. It records token usage through the shared tracker, supports provider tool calling through chat_*_messages(..., tools=...), and exposes count_tokens() (tiktoken when available, with a character-based fallback). Provider-specific Responses API features are outside this backend's contract.

Only write a new backend when the provider is not compatible with this surface or requires behavior that cannot be expressed by its configuration.

Backend architecture

The active split optimizer/target dispatcher is the public skillopt/model/__init__.py module:

skillopt/model/
├── common.py                       # aliases, default models, token/response helpers
├── backend_config.py               # optimizer/target whitelists and runtime selection
├── __init__.py                     # public API and split-role dispatch
├── openai_compatible_backend.py    # generic Chat Completions example
├── qwen_backend.py                 # raw-HTTP chat example with per-role config
├── minimax_backend.py              # compact raw-HTTP chat example
├── codex_harness.py                # target-only exec harnesses
└── router.py                       # legacy single-backend compatibility surface

router.py is not the dispatcher used by the current training loop. Update it only if the new backend must also be exposed through that legacy single-backend API.

Step 1: implement the module contract

Create a module such as skillopt/model/your_backend.py. Copy the signatures from openai_compatible_backend.py or qwen_backend.py; model calls in the current framework are synchronous.

For a chat backend that supports both roles, the public module surface is:

Function	Purpose
`chat_optimizer(...)`	Optimizer system/user call; returns `(text, usage)`
`chat_target(...)`	Target system/user call; returns `(text, usage)`
`chat_optimizer_messages(...)`	Optimizer message-list call, including optional tools
`chat_target_messages(...)`	Target message-list call, including optional tools
`get_token_summary()`	Return per-stage counters plus `_total`
`reset_token_tracker()`	Clear this backend's counters
`set_optimizer_deployment(name)`	Change the optimizer model at runtime
`set_target_deployment(name)`	Change the target model at runtime
`set_reasoning_effort(effort)`	Apply or safely ignore the shared reasoning setting

Every call returns a usage dict with prompt_tokens, completion_tokens, and total_tokens. Use TokenTracker from skillopt.model.common and record each call exactly once. Message-list calls that accept tools should return the compatibility message objects from common.py when return_message=True.

Provider-specific configuration helpers and count_tokens() are optional, but their state must be safe to update while calls may run concurrently. Keep credentials out of logs and persisted artifacts.

Exec-style targets do not implement this chat contract. They are target-only and are integrated through codex_harness.py plus environment-specific rollout code.

Step 2: register and route the backend

A new backend normally requires all of the following:

Add its canonical name, aliases, and default model to skillopt/model/common.py.
Add the canonical name to the appropriate optimizer and/or target whitelist in skillopt/model/backend_config.py. Do not advertise a role the module cannot execute.
Import the module in skillopt/model/__init__.py and add dispatch branches for every supported call surface.
Include its counters in get_token_summary() / reset_token_tracker() and forward the shared deployment/reasoning setters where applicable.
If it has YAML settings, add structured-to-flat mappings in skillopt/config.py, wire them through scripts/train.py and scripts/eval_only.py, and document their precedence over environment variables.
Update router.py only when legacy single-backend compatibility is part of the intended feature.

Backend selection in scripts/train.py must use model.optimizer_backend and model.target_backend. A high-level model.backend alias alone is not a substitute for this explicit split.

Step 3: test the integration

Add focused tests under tests/ that do not call a live provider. At minimum, cover:

optimizer and target whitelist validation;
routing for text and message-list calls;
role-specific configuration precedence;
tool-call compatibility, if supported;
deployment/reasoning setters;
token accounting, including a single correct _total;
actionable errors for missing credentials or invalid responses.

Then run the focused test, the full suite, and the documentation build:

python -m pytest tests/test_your_backend.py -q
python -m pytest tests/ -q
mkdocs build --strict

Also update .env.example, the configuration reference, and the backend table in the API reference. Add an optional dependency extra only when the backend requires a package that is not already a core dependency.

8.2 KiB Raw Blame History