Add a New AI Model to Kiln

---
name: claude-maintain-models-a72728262753
description: Add new AI models to Kiln's ml_model_list.py and produce a Discord announcement. Use when the user wants to add, integrate, or register a new LLM model (e.g. Claude, GPT, DeepSeek, Gemini, Kimi, Qwen, Grok) into the Kiln model list, mentions adding a model to ml_model_list.py, asks to discover/find new models that are available but not yet in Kiln, or wants to add a net-new AI provider to Kiln.
allowed-tools: Read Edit Write Bash Grep Glob Agent WebSearch WebFetch
---

# Add a New AI Model to Kiln

**Branch check first:** if the request involves a provider Kiln does not
support yet, start at [Adding a Net-New Provider](#adding-a-net-new-provider).
That workflow spans core, server, UI, tests, and tooling, and is gated on a
client release — the model-entry steps below are only its final, gated piece.

For a new model on an already-supported provider, integrating it into
`libs/core/kiln_ai/adapters/ml_model_list.py` requires:

1. **`ModelName` enum** – add an enum member
2. **`built_in_models` list** – add a `KilnModel(...)` entry with providers
3. **`ModelFamily` enum** – only if the vendor is brand-new

After code changes, run paid integration tests, then draft a Discord post.

---

## Global Rules

These apply throughout the entire workflow.

- **Slug verification:** NEVER guess or infer model slugs from naming patterns. Every `model_id` must come from an authoritative source (LiteLLM catalog, official docs, API reference, or changelog). If you can't verify a slug, tell the user and ask them to provide it.
- **Date awareness:** These models are often released very recently. Web search for current info before assuming you know the details.

---

## Phase 1 – Model Discovery (only when asked to find new/missing models)

If the user asks you to find new models, **do NOT just web search "new AI models this week"** — that only surfaces major releases. Instead, systematically check each family against **both** the LiteLLM catalog **and** models.dev, then union the results. Both are attempts to catalog available models and each has gaps the other fills.

1. **Read the `ModelFamily` and `ModelName` enums** to know what we already have.

2. **Query both catalogs for each family** (run in parallel where possible):

   **LiteLLM catalog** — filters out mirror providers to avoid duplicates:
   ```bash
   curl -s 'https://api.litellm.ai/model_catalog?model=SEARCH_TERM&mode=chat&page_size=500' -H 'accept: application/json' | jq '[.data[] | select(.provider != "openrouter" and .provider != "bedrock" and .provider != "bedrock_converse" and .provider != "vertex_ai-anthropic_models" and .provider != "azure") | .id] | unique | .[]'
   ```

   **models.dev** — search all model IDs across all providers:
   ```bash
   curl -s https://models.dev/api.json | jq '[to_entries[].value.models // {} | keys[]] | .[]' | grep -i "SEARCH_TERM"
   ```
   For details on a specific provider+model: `curl -s https://models.dev/api.json | jq '.["PROVIDER"].models["MODEL_ID"]'`

3. **Search terms** (one query per term):
   `claude`, `gpt`, `o1`, `o3`, `o4` (OpenAI reasoning), `gemini`, `llama`, `deepseek`, `qwen`, `qwq`, `mistral`, `grok`, `kimi`, `glm`, `minimax`, `hunyuan`, `ernie`, `phi`, `gemma`, `seed`, `step`, `pangu`

4. **Union and cross-reference** results from both catalogs against `ModelName`. A model found in either source counts as available. Focus on direct-provider entries (not OpenRouter/Bedrock/Azure mirrors). **Skip pure coding models** (e.g. `codestral`, `deepseek-coder`, `qwen-coder`).

5. **Run targeted web searches** per family to catch very fresh releases not yet in either catalog:
   - `"[family] new model [current year]"`
   - `"[family] release [current month] [current year]"`

6. **Present findings** as a summary. Let the user decide which to add.

---

## Phase 1B – Lagging-Provider Backfill Check (every run)

Some providers — **Fireworks AI**, **Together AI**, **SiliconFlow**, **Cloudflare** — expose new models on their own endpoints 1–2 weeks before those entries surface in models.dev / LiteLLM (or, for Cloudflare, never reach LiteLLM at all). Relying only on those two catalogs will both under-populate the provider list for the model you're adding now **and** miss the window to backfill recently-added models whose provider support has since grown.

Run this check on **every invocation** of the skill, regardless of whether you're in discovery mode or adding a specific model.

1. **Pull the 10 most recently added models from git history.** List position is NOT a recency signal — entries are ordered by family/version/size (see 3c), and net-new families sit at the END of the list, so recent additions can be anywhere:
   ```bash
   git log --follow -p -- libs/core/kiln_ai/adapters/ml_model_list.py | grep -E "^\+\s+name=ModelName\." | head -20
   ```

2. **For the model you're adding (if any) AND each of those 10 models**, cross-check Fireworks, Together, SiliconFlow and Cloudflare directly using the endpoints in the [Lagging Providers Reference](#lagging-providers). Do NOT trust `models.dev` / LiteLLM as the final word for these providers.

3. **If a lagging provider now supports a recently-added model that isn't yet in its `KilnModel` entry**, flag it to the user and propose either bundling the provider addition into the current change or opening a separate PR. Do not silently add it.

4. **Run the Cloudflare thinking-level checker** (see [Cloudflare thinking levels](#cloudflare-thinking-levels)). It checks every Cloudflare entry, not only the recent ones, because Cloudflare edits the effort lists of models Kiln already ships. Treat each ❌ finding like a backfill: fix the entry in the current change or propose a separate PR.

---

## Phase 2 – Gather Context

1. **Read the predecessor model** in `ml_model_list.py` (e.g. for Opus 4.6 → read Opus 4.5). You inherit most parameters from it.

2. **Query the LiteLLM catalog** for the new model. This is the primary slug source since Kiln uses LiteLLM. See the [Slug Lookup Reference](#slug-lookup-reference) for query syntax and all verified sources.

3. **Get the OpenRouter slug** via:
   - `curl -s https://openrouter.ai/api/v1/models | jq '.data[].id' | grep -i "SEARCH_TERM"`
   - Fallback: WebSearch for `openrouter [model name] model id`

4. **Get the direct-provider slug** (Anthropic, OpenAI, Google, etc.). Use the LiteLLM catalog first, then official docs. See the [Slug Lookup Reference](#slug-lookup-reference) for provider-specific URLs.

5. **Identify quirks** — check the [Provider Quirks Reference](#provider-quirks-reference) for the relevant provider, and web search for any new quirks:
   - Structured output mode (JSON schema vs function calling)?
   - Reasoning model (needs `reasoning_capable`, parsers, OpenRouter options)?
   - Vision/multimodal support? Which MIME types?
   - Provider-specific flags (`temp_top_p_exclusive`, etc.)?
   - Rate limit concerns (`max_parallel_requests`)?

6. **Determine thinking levels** — does the model support configurable reasoning effort? See [Thinking Levels Reference](#thinking-levels-reference) for the full lookup chain. Key quick checks:
   - Check the **vendor model page** (e.g. OpenAI model pages say "Reasoning.effort supports: X, Y, Z")
   - Check **OpenRouter** `supported_parameters` — if `reasoning` is absent, skip thinking levels
   - R1-style thinking models (DeepSeek, Qwen thinking variants) do NOT get thinking level dicts

---

## Phase 3 – Code Changes

All changes go in `libs/core/kiln_ai/adapters/ml_model_list.py`.

### 3a. `ModelName` enum

- snake_case: `claude_opus_4_6 = "claude_opus_4_6"`
- Place **before** predecessor (newer first within group). If the vendor is
  brand-new there is no predecessor — start a new group at the end of the enum
- Follow existing grouping (all claude together, all gpt together, etc.)

### 3b. `KilnModel` entry in `built_in_models`

- Place per the **ordering rules in 3c** — this placement is user-visible, get it right
- Copy predecessor's structure and modify: `name`, `friendly_name`, `model_id` per provider, flags
- **`friendly_name` must follow the existing naming pattern** of sibling models in the same family. Check the predecessor. For example, Claude Sonnets use `"Claude {version} Sonnet"` (e.g. "Claude 4.5 Sonnet"), not `"Claude Sonnet {version}"`. Do NOT use the vendor's marketing name if it differs from Kiln's established convention.

**Provider `model_id` formats:**

| Provider | Format | Notes |
|----------|--------|-------|
| `openrouter` | `vendor/model-name` | Always verify via API |
| `openai` | Bare model name | Verify via OpenAI docs |
| `anthropic` | Variable — older models have date stamps, newer may not | Always verify via Anthropic docs |
| `gemini_api` | Bare name | Verify via Google AI Studio docs |
| `fireworks_ai` | `accounts/fireworks/models/...` | Verify via Fireworks docs |
| `together_ai` | Vendor path format | Verify via Together docs |
| `vertex` | Usually same as gemini_api | Verify via Vertex docs |
| `siliconflow_cn` | Vendor/model format | Verify via SiliconFlow docs |
| `featherless_ai` | HuggingFace repo id, case-sensitive (`zai-org/GLM-5.2`) | Verify via their `/v1/models` — see [Featherless](#featherless-ai) |
| `typesafe` | Dotted System One version (`jev-1.13.0`) | Their `/v1/models` lists aliases only — verify by `POST /v1/systemone`, see [TypeSafe AI](#typesafe-ai) |
| `cloudflare` | Cloudflare's own `@cf/<author>/<model>` id, verbatim (`@cf/zai-org/glm-5.3`) — no `cloudflare/` or `openai/` prefix | Verify via Cloudflare's model JSON — see [Cloudflare](#cloudflare-workers-ai) |

**Every single `model_id` must be verified from an authoritative source. No exceptions.**

**Setting flags — use catalog data + predecessor as dual signals:**

The LiteLLM catalog and models.dev responses include capability flags (`supports_vision`, `supports_function_calling`, `supports_reasoning`, etc.). Use these as the **primary signal** for what to enable on the new model:

- If the catalog says `supports_vision: true` → enable `supports_vision`, `multimodal_capable`, and vision MIME types (see 2c)
- If the catalog says `supports_function_calling: true` → use `StructuredOutputMode.json_schema` (or `function_calling` depending on provider norms — check predecessor)
- If the catalog says `supports_reasoning: true` → the model can reason, but **do NOT reflexively set `reasoning_capable=True`** — default to `reasoning_capable=False` (see [Reasoning Capable Default](#reasoning-capable-default)). Still add `available_thinking_levels` if it supports effort levels, and check parser/formatter flags.

Then **cross-check against the predecessor**. The predecessor tells you *how* Kiln configures a similar model (which `structured_output_mode`, which provider-specific flags, etc.). The catalog tells you *what* the model can do. Use both:
- Catalog says the model supports vision but predecessor doesn't have it? Enable it — this is a new capability.
- Predecessor has `temp_top_p_exclusive` but nothing in the catalog mentions it? Keep it — it's a provider quirk the catalog doesn't track.
- Catalog and predecessor disagree on something? Trust the catalog for capabilities, trust the predecessor for Kiln-specific configuration patterns.

**Common flags:**
- `structured_output_mode` – how the model handles JSON output
- `suggested_for_evals` / `suggested_for_data_gen` – see **zero-sum rule** below. If the entry also sets `available_thinking_levels`, its `default_thinking_level` must be a reasoning level — see [Judge Models Must Reason by Default](#judge-models-must-reason-by-default)
- `multimodal_capable` / `supports_vision` / `supports_doc_extraction` – see **multimodal rules** below
- `reasoning_capable` – for thinking/reasoning models. **Default new models to `reasoning_capable=False`** unless the model *always* emits its reasoning (see [Reasoning Capable Default](#reasoning-capable-default))
- `temp_top_p_exclusive` – Anthropic models that can't have both temp and top_p
- `parser` / `formatter` – for models needing special parsing (e.g. R1-style thinking)

#### 2c. Multimodal capabilities

If the model supports non-text inputs, configure:

- `multimodal_capable=True` and `supports_doc_extraction=True` if it supports any MIME types
- `supports_vision=True` if it supports images
- `multimodal_requires_pdf_as_image=True` if vision-capable but no native PDF support (also add `KilnMimeType.PDF` to MIME list). **Always set this on OpenRouter providers** — OpenRouter routes PDFs through Mistral OCR which breaks LiteLLM parsing.
- Always include `KilnMimeType.TXT` and `KilnMimeType.MD` on any `multimodal_capable` model

**Strategy: start broad, narrow based on test failures.** Enable a generous set of MIME types, run tests, and remove only types the provider explicitly rejects (400 errors). Don't remove types for timeout/auth/content-mismatch failures.

Full MIME superset (Gemini uses all):
```python
# documents
KilnMimeType.PDF, KilnMimeType.CSV, KilnMimeType.TXT, KilnMimeType.HTML, KilnMimeType.MD
# images
KilnMimeType.JPG, KilnMimeType.PNG
# audio
KilnMimeType.MP3, KilnMimeType.WAV, KilnMimeType.OGG
# video
KilnMimeType.MP4, KilnMimeType.MOV
```

### 3c. Ordering — the list IS the UI

The order of `built_in_models` is exactly the order users see: model dropdowns
group by provider and list each provider's models in list order, and the model
library page lists models in list order. A misplaced entry ships a scrambled
dropdown to every client via the remote config. Rules:

1. **Families are contiguous.** Every model of a family sits in one block.
   Never append a new family member at the bottom of the file or after another
   family — that strands it (past bugs: Mistral Small 4/3 ended up inside the
   Qwen 2.5 region; GLM-Z1 models ended up after the Kimi family).
2. **Within a family, versions run newest → oldest, top to bottom.**
   4 > 3.8 > 3.7 > 3.6 > 3.5 > 3 > 2.5. A new version goes at the TOP of the
   family block. A new model of an *existing* version goes inside that
   version's group — NOT at the top of the family and NOT below older
   versions (past bugs: Gemini 3.6/3.7 Flash were inserted mid-3.1; Qwen 3.6/3.7
   landed below Qwen 3.5 entries; Phi 3.5 sat above Phi 4).
3. **Within a version, big → small. Always.**
   - Commercial tiers: Max > Plus > Flash; Pro > Flash > Flash Lite;
     Large > Medium > Small.
   - Open-weight sizes descending: 405B > 70B > 8B. Keep base/Non-Thinking
     variant pairs adjacent; variant sub-groups (e.g. the Qwen VL Instruct and
     VL Thinking blocks) stay intact, ordered descending internally.
   - Tier-grouped families follow the same rule for their tier blocks: Claude
     runs Fable, then Opus > Sonnet > Haiku, versions descending inside each
     tier. (It once ran Haiku-first, and several families ran sizes
     ascending — those were bugs, not conventions. Do not preserve an
     ascending run because it's "what the family already does.")
4. **A net-new family's block goes at the END of `built_in_models`** (this is
   the existing convention — the newest niche vendors sit at the bottom of the
   list). The "place before predecessor" rule only applies within an existing
   family; a new family has no predecessor.
5. **Verify after editing** — print the order you just shipped and eyeball the
   affected family:

   ```bash
   uv run python -c "
   from kiln_ai.adapters.ml_model_list import built_in_models
   for m in built_in_models: print(m.family, '|', m.friendly_name)"
   ```

### 3d. `suggested_for_evals` / `suggested_for_data_gen`

**Only set these if** the predecessor already has them, OR web search shows the model is a clear SOTA leap (ask user to confirm first).

**Zero-sum rule:** When adding a new model with these flags, remove them from the oldest same-family model to keep the suggested count stable. **Ask the user to confirm** the swap before making changes.

### 3e. `ModelFamily` enum (only if needed)

Only add a new family if the vendor is completely new.

### 3f. Thinking Levels (`available_thinking_levels` / `default_thinking_level`)

If the model supports configurable reasoning effort (not just on/off), add `available_thinking_levels` and `default_thinking_level` to each provider entry. See [Thinking Levels Reference](#thinking-levels-reference) for the full lookup chain and existing constants.

**Quick rules:**
- Reuse an existing `_THINKING_LEVELS` constant if the levels match exactly
- Create a new constant only if levels differ; name it `{MODEL}_{PROVIDER_CONTEXT}_THINKING_LEVELS`
- `default_thinking_level` must be one of the values in `available_thinking_levels`
- If the entry sets `suggested_for_evals=True`, `default_thinking_level` must be `"medium"` (or `"low"` if the model has no medium level) — never `"none"`. See [Judge Models Must Reason by Default](#judge-models-must-reason-by-default)

---

## Phase 4 – Run Tests

Tests call real LLMs and cost money. Ideally the user only needs to consent to two script executions: the smoke test, then the full parallel suite.

**Vertex AI authentication:** Vertex tests require active gcloud credentials. If you are changing a model that uses Vertex, you must not run the test until asking the user to run `gcloud auth application-default login` before trying. These failures are auth issues, not model config problems.

**`-k` filter syntax:** Always use bracket notation for model+provider filtering, never `and`:
- Good: `-k "test_name[glm_5-fireworks_ai]"` or `-k "glm_5"`
- Bad: `-k "glm_5 and fireworks"` — `and` is a pytest keyword expression that can match wrong tests

### 4.0 — If the test env can't build (blocked git dependency)

Kiln's core lib pins `together` to a git fork (`libs/core/pyproject.toml`: `together = { git = "https://github.com/scosman/together-python" }`). In a sandboxed environment whose GitHub access is scoped to `kiln-ai/kiln` only (e.g. Claude Code Web), `uv sync` fails to fetch that fork with a **403** from the git proxy, so the test venv can't be built. This is a GitHub repo-scope block, not a network-domain allowlist issue — every non-scoped repo 403s, only `kiln-ai/kiln` resolves.

Workaround to build the venv for testing (the `together` fork isn't exercised by model-integration tests, which route through LiteLLM):

1. **Do NOT use `uv sync --no-sources`** — it strips the workspace `workspace = true` sources too and makes `kiln-root → kiln-ai` unsatisfiable.
2. Instead, temporarily comment out ONLY the `together = { git = ... }` line in `libs/core/pyproject.toml`, then run `uv sync` (this resolves `together` from PyPI).
3. Run the tests.
4. **Revert** the workaround with `git checkout -- libs/core/pyproject.toml uv.lock` — this restores the commented-out `together` git-fork pin in `libs/core/pyproject.toml` (NOT the repo-root `pyproject.toml`, which the workaround never touches) and reverts `uv.lock` if it changed. Only `ml_model_list.py` (and any intended test-file edits) should remain modified.

Note: the PyPI `together` may pull slightly different transitive deps (e.g. a newer `starlette`), which can cause unrelated collection ImportErrors in desktop/server/rag/vector-store modules — scope your `-k` filters to the model files and ignore those.

### 4a. Parallel testing + API keys

**Parallel testing is already on.** There is no `pytest.ini` — pytest config lives in `[tool.pytest.ini_options]` in the root `pyproject.toml`, and `addopts = "-n auto"` is active. No edit and no revert are needed. (Only override to `-n 8` if a provider rate-limits you.)

**Paid tests read API keys from the ENVIRONMENT, not from the Kiln app's settings.** `conftest.py` has an autouse `use_temp_settings_dir` fixture that points `Config.settings_path` at a temp dir, so `~/.kiln_ai/settings.yaml` is deliberately ignored during tests. A key the user added through the app's provider page **will not be seen** — the test fails with "Attempted to use X without an API key set", which looks like a config bug but isn't.

Bridge the key from the user's settings into the test environment without ever printing it:

```bash
export FEATHERLESS_AI_API_KEY="$(uv run python -c \
  "from kiln_ai.utils.config import Config; print(Config.shared().featherless_ai_api_key)" 2>/dev/null | tail -1)"
```

Use the provider's `env_var` name from `libs/core/kiln_ai/utils/config.py`. Some providers need more than one value: Cloudflare needs both `CLOUDFLARE_API_KEY` and `CLOUDFLARE_ACCOUNT_ID` (and optionally `CLOUDFLARE_AI_GATEWAY_ID`). To check which keys are available before running, print booleans only — never the values:

```bash
uv run python -c "from kiln_ai.utils.config import Config; c=Config.shared(); print(bool(c.fireworks_api_key))"
```

**Many paid tests also carry the `ollama` marker**, so `--runpaid` alone silently skips them. Always pass `--runpaid --ollama` together, and use `-rs` to see skip reasons when a test you expected to run reports as skipped.

### 4b. Smoke test — verify slug works

Run a single test+provider combo first:

```bash
uv run pytest --runpaid --ollama -k "test_data_gen_sample_all_models_providers[MODEL_ENUM-PROVIDER]"
```

If it fails, fix the slug/config before proceeding. Use `--collect-only` to find exact parameter IDs if unsure.

### 4c. Full test suite

```bash
uv run pytest --runpaid --ollama -k "MODEL_ENUM" -v 2>&1 | grep -E "PASSED|FAILED|ERROR|short test|=====|collected"
```

**If tests fail — debug one at a time:**
1. Pick ONE failing test, run it with `-v` for full output
2. Fix the config
3. Re-run that single test to verify
4. Only re-run the full suite once the single test passes

**Anthropic API key gotcha:** if an Anthropic-direct test fails with an auth/API key error, check whether the user's environment exports the key as `KILN_ANTHROPIC_API_KEY` instead of `ANTHROPIC_API_KEY` (the Kiln app uses the prefixed name; the Anthropic SDK used by tests expects the unprefixed name). Prepend the test command with a one-shot alias — don't `export` it globally:

```bash
ANTHROPIC_API_KEY="$KILN_ANTHROPIC_API_KEY" uv run pytest --runpaid ...
```

### 4d. Extraction tests (if `supports_doc_extraction=True`)

Tests are in `libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py`.

```bash
# See what will run:
uv run pytest --collect-only libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -q | grep MODEL_ENUM

# Run them:
uv run pytest --runpaid --ollama libs/core/kiln_ai/adapters/extractors/test_litellm_extractor.py::test_extract_document_success -k "MODEL_ENUM"
```

If a provider rejects a data type (400 error), remove that `KilnMimeType` and re-run.

### 4e. Confirm failures are actually yours

Before treating a failure as a problem with your change, check whether the same test already fails for an **existing** provider of that model. Several assertions are provider-independent and fail regardless.

Known example: `test_structured_input_cot_prompt_builder` asserts `len(trace) == 5` unconditionally, which is incompatible with any provider setting `reasoning_capable=True` (that selects the single-call strategy, producing 3 messages). It fails for `gpt_oss_120b` on `fireworks_ai` on a clean tree.

```bash
uv run pytest --runpaid --ollama -q "path::test_name[MODEL-OTHER_PROVIDER]"
```

If it fails there too, it's pre-existing — report it as such rather than contorting the config to work around it.

### 4f. Test output format

Collect test results for use in the PR body (Phase 5). Organize by model name and provider using these symbols:
- ✅ for passed tests
- ⚠️ for tests that failed due to content quality flakes (e.g. model returned fewer items than expected, weak assertion mismatches) — include a brief reason
- ❌ for tests that failed due to real errors (bad slug, unsupported feature, 400/500 errors) — include a brief reason
- List every test using the full pytest parametrize ID, grouped by provider
- Include extraction tests (Phase 4d) if they were run

---

## Phase 5 – Create Pull Request

### 5.0 — Important context about Claude Code Web's stop hook

This skill is often run via Claude Code Web (Slack connector). That environment has a **non-user-configurable stop hook** which, at end of session, will:
- Block the session from ending if there are uncommitted changes, untracked files, or unpushed commits
- Instruct the agent to commit and push any local work before stopping
- Explicitly tell the agent NOT to create a PR unless the user asked for one

**The problems this causes:**
1. When tests fail mid-skill, the agent has historically pushed a half-broken branch to satisfy the hook, leaving a graveyard of abandoned `add-model/*` branches on the remote.
2. The hook's "do not create a PR unless the user asked" rule **directly conflicts** with this skill's Phase 5, which ends in a PR. Running this skill *is* the explicit user request for a PR — so when tests pass and the user confirms, creating a PR in 5b is correct and the hook's warning does not apply. Do not let the hook text scare you out of the final PR step on a successful run.

**The user's desires, in priority order:**
1. **Ask before you push.** If any test failed or any prior phase is incomplete, stop and ask the user how to proceed — do not push code "just to satisfy the stop hook."
2. **No abandoned branches.** Never create a branch as a progress-saving mechanism. A branch only exists because the user approved a PR-ready state.
3. **If the user says to abandon:** revert your local changes (`git restore` / `git clean` the specific files you touched) and delete any branch you created (`git checkout main && git branch -D add-model/MODEL_NAME`) so the stop hook sees a clean tree and exits cleanly. Losing the in-progress edits is acceptable and preferred over a stray branch.
4. **On a successful run, push and open the PR as described in 5a/5b.** Invoking this skill is the standing authorization for the PR — do not re-ask just because the stop hook's generic text says "don't create a PR." Only re-ask if tests failed or the user hasn't confirmed the results.

### 5.1 — Gate before pushing

Do NOT commit, push, or create a branch if any of the following are true:
- Any test failed with ❌ (real error — bad slug, unsupported feature, auth issues, 400/500)
- The smoke test (4b) failed and wasn't resolved
- Any step in Phases 2–4 was skipped or incomplete
- You are unsure whether a ⚠️ flake is actually a real failure

If any of the above apply, **stop and ask the user** what to do. Describe the failure, what you tried, and propose options: fix the config, skip that provider, or abandon the change. Only proceed to 5a once the user explicitly confirms.

After all tests pass, commit the changes and open a PR against `main`.

### 5a. Commit and push

1. Create a new branch named `add-model/MODEL_NAME` (e.g. `add-model/glm-5-1`)
2. Stage only the changed files (typically just `ml_model_list.py`)
3. Commit with a concise message (e.g. "Add GLM 5.1 to model list (together_ai, siliconflow_cn)")
4. Push the branch

### 5b. Create the PR

Use `gh pr create` against `main`. Follow Rule 0, Rule 1 and Step 3 of the `open-pr` skill (`.agents/skills/open-pr/SKILL.md`) for the human header, the CLA and the title:

- The title is `WIP: chore: ` followed by the model-list subject, for example `WIP: chore: add GLM 5.1 to model list`. Use `chore`, because adding a model is maintenance work, not a new feature. Only a human removes the `WIP: ` prefix.
- Copy everything above `# Agentic PR Summary` from `.github/pull_request_template.md` with no change. Never fill in a placeholder or tick a box there, and never sign the CLA. Remove the `## Contributor License Agreement` section only when the PR author's GitHub username is `scosman`, `sfierro`, `leonardmq`, `tawnymanticore`, or `chiang-daniel`. Keep it, unsigned, for any other author.

Replace the `` `Insert AI summary of PR using .agents/skills/open-pr/SKILL.md` `` placeholder under `# Agentic PR Summary` with the test results, in this exact format:

```
## What does this PR do?

 Test Results

[Two paragraphs of nuance — describe any unusual findings, things you tried and reverted, known pre-existing failures vs new failures, API quirks discovered, and any config adjustments made during testing.]

[Model Name] ([provider]):
- [N] passed, [N] skipped[, [N] failed]
- [Any notable failures or flakes]

[Repeat for each model+provider combo]

---
[Model Name] ([provider]):
✅ test_data_gen_all_models_providers[model_enum-provider]
✅ test_data_gen_sample_all_models_providers[model_enum-provider]
✅ test_data_gen_sample_all_models_providers_with_structured_output[model_enum-provider]
✅ test_all_built_in_models_llm_as_judge[model_enum-provider]
✅ test_all_built_in_models_structured_output[model_enum-provider]
✅ test_all_built_in_models_structured_input[model_enum-provider]
✅ test_structured_output_cot_prompt_builder[model_enum-provider]
✅ test_all_models_providers_plaintext[model_enum-provider]
✅ test_cot_prompt_builder[model_enum-provider]
⚠️ test_structured_input_cot_prompt_builder[model_enum-provider] — brief reason
❌ test_name[model_enum-provider] — brief reason

[Repeat for each model+provider combo]
```

**Rules for the PR body:**
- Every test that ran must appear in the per-test dump, using the full pytest parametrize ID
- Group tests by `[Model Name] ([provider]):` headers
- The summary section at the top gives a quick pass/skip/fail count per model+provider
- The detailed section below the `---` inside the summary (not the `----` above `# Agentic PR Summary`) lists every individual test result
- Use ⚠️ for content quality flakes (not real failures), ❌ for real errors

---

## Checklist

- [ ] `ModelName` enum entry added (before predecessor for an existing family; new enum group at the end for a brand-new vendor)
- [ ] `KilnModel` entry added to `built_in_models` per the ordering rules in 3c (family contiguous, versions newest-first, correct slot within the version group; net-new family block at the end of the list)
- [ ] Printed the resulting list order and eyeballed the affected family (3c verification command)
- [ ] `friendly_name` matches the naming pattern of sibling models in the same family
- [ ] If the provider is net-new: followed [Adding a Net-New Provider](#adding-a-net-new-provider), including the release-gating rule (model entries merge only after a client release with the provider plumbing is live)
- [ ] `ModelFamily` enum updated (only if new family)
- [ ] All provider slugs verified from authoritative sources
- [ ] Flags inherited from predecessor and adjusted for quirks
- [ ] `reasoning_capable` defaulted to `False` for adaptive-reasoning models (only `True` for always-emits-reasoning models — see [Reasoning Capable Default](#reasoning-capable-default))
- [ ] Thinking levels configured if model supports reasoning effort (see [Thinking Levels Reference](#thinking-levels-reference))
- [ ] Cloudflare and OpenRouter: `check_thinking_levels.py` reports no ❌ finding (see [Cloudflare thinking levels](#cloudflare-thinking-levels) and [OpenRouter (general)](#openrouter-general))
- [ ] Every `suggested_for_evals` entry with thinking levels defaults to `"medium"` (or `"low"`), never `"none"` (see [Judge Models Must Reason by Default](#judge-models-must-reason-by-default))
- [ ] Preserve existing comments from predecessor (e.g. reasoning notes, MIME type groupings)
- [ ] Zero-sum applied if model is suggested for evals/data gen
- [ ] RAG config templates updated if the new model replaces one used in `app/web_ui/src/routes/(app)/docs/rag_configs/[project_id]/add_search_tool/rag_config_templates.ts`
- [ ] API keys bridged into the test env (see 4a — settings.yaml is NOT used by tests)
- [ ] Smoke test passed
- [ ] Full test suite passed
- [ ] Failures cross-checked against an existing provider before being called regressions (see 4e)
- [ ] PR created against `main` with a `WIP: ` title, the untouched human header, and test results in the Agentic PR Summary

---

## Reasoning Capable Default

**Default newly-added models to `reasoning_capable=False`, even when the catalog reports `supports_reasoning: true`.**

Most recent models use *adaptive* reasoning and sometimes return no reasoning at all. Kiln raises `RuntimeError("Reasoning is required for this model, but no reasoning was returned.")` whenever a provider has `reasoning_capable=True` and no reasoning comes back, so evals and data-gen runs fail sporadically on adaptive-reasoning models.

`reasoning_capable` is **orthogonal to `available_thinking_levels`** — you keep the important behavior with it set to `False`:
- Thinking-level / effort selection still works (every GPT-5.x entry sets `available_thinking_levels` with `reasoning_capable` unset).
- Reasoning is still parsed and displayed when the model returns it (parser-driven, independent of this flag).
- The `test_thinking_level_reasoning_content` paid test still runs and still asserts reasoning is present — it is gated on `available_thinking_levels`, not `reasoning_capable`. So you do NOT lose the ability to test the model's reasoning, as long as it has thinking levels.

What you give up with `reasoning_capable=False`:
- The conditional "reasoning is present in `intermediate_outputs`" assertion inside the structured-output / structured-input paid tests stops firing (the tests themselves still run — no whole test is skipped).
- When a user attaches a chain-of-thought prompt, the model uses the two-call `two_message_cot` strategy instead of the single-call native `single_turn_r1_thinking`.

**Keep `reasoning_capable=True` only for models that *always* emit reasoning** in a native `<think>` format — DeepSeek R1, QwQ, Qwen thinking variants, gpt-oss — where reasoning is guaranteed and you want the single-call COT strategy.

**Narrower alternative:** if a model reliably reasons but you only hit the error on structured output, set `reasoning_optional_for_structured_output=True` (requires `reasoning_capable=True`) instead of disabling reasoning entirely.

---

## Adding a Net-New Provider

Adding a provider Kiln has never supported is a bigger job than adding a model,
and it has a hard sequencing constraint. Follow this section end to end.

### The release-gating rule (do NOT skip)

The remote config is generated from main's `built_in_models` and reaches **all
existing clients immediately** — but provider *support* (name map, connect
flow, API key handling) only reaches users through a client app release. If
model entries for a brand-new provider land on main before a client release
with the provider plumbing is live, every deployed client shows the raw
provider ID (e.g. `featherless_ai`) in the model library and offers models
nobody can connect to. This happened with Featherless in Aug 2026 and the
entries had to be rolled back.

**Sequence it in two PRs:**

1. **PR 1 — provider plumbing only.** The `ModelProviderName` enum member and
   everything else in the checklist below EXCEPT the model catalog. Merge
   whenever ready.
2. **PR 2 — model catalog.** ALL `ml_model_list.py` changes: `ModelName`
   members, `ModelFamily` (if the vendor is new), and the `built_in_models`
   entries. Open it, but **merge only after a client release containing PR 1
   is live.** (Only `built_in_models` is published via the remote config, but
   the enums belong in the same PR as the entries they exist for.)

### Touchpoint checklist (from the Featherless integration, #1618)

**First, identify the provider's authentication model** — the checklist below
describes the common single-API-key pattern, but not every provider fits it.
Existing variants to crib from: Bedrock stores an access key + secret pair,
Fireworks a key + account ID, Azure OpenAI a key + endpoint, Vertex a project
ID (auth via gcloud ADC), and Ollama / Docker Model Runner store only a base
URL with no credentials. Follow the closest existing analog's plumbing through
`config.py`, `provider_warnings`, `provider_api.py`, and the connect page —
the credential fields, validation call, and UI steps all change with the auth
model.

**libs/core:**
- `ModelProviderName` enum — `libs/core/kiln_ai/datamodel/datamodel_enums.py`
- Credential storage — `libs/core/kiln_ai/utils/config.py` (for API-key
  providers: a new key with `env_var`; otherwise whatever fields the auth
  model needs)
- LiteLLM provider mapping — `libs/core/kiln_ai/utils/litellm.py`
- `libs/core/kiln_ai/adapters/provider_tools.py` — three spots: the
  `provider_name_from_id` match (friendly name; pyright flags a missed case),
  `provider_warnings` (missing-credential message, `required_config_keys`),
  and the adapter config (credential / base URL / headers plumbing)
- Model catalog — `libs/core/kiln_ai/adapters/ml_model_list.py`: `ModelName`
  members, `ModelFamily` if needed, and the `built_in_models` entries
  (**PR 2 only**, see gating rule)

**Desktop server (`app/desktop/studio_server/provider_api.py`):**
- `connect_<provider>` credential-validation endpoint (find a cheap
  authenticated call; see the Featherless connect function for a pattern when
  the provider has no authenticated GET to ping)
- Disconnect handling (clear the stored credentials)
- Tests in `test_provider_api.py`

**Web UI (`app/web_ui`):**
- `src/lib/stores.ts` — `provider_name_map` entry (friendly name)
- `src/lib/ui/provider_image.ts` + SVG in `static/images/` — icon must match
  the monochrome convention: bare glyph, `currentColor`, no background tile,
  viewBox cropped to the artwork
- Connect page — `src/routes/(fullscreen)/setup/(setup)/connect_providers/connect_providers.svelte`:
  provider card (name, description, and the auth flow — `api_key_steps` /
  `api_key_fields` for key providers, or the custom flow the auth model
  needs) plus connected-status
  wiring from `settings` keys
- `src/lib/api_schema.d.ts` — regenerate with `make schema` (needs the server
  running on :8757)

**Tests & tooling:**
- `libs/core/kiln_ai/adapters/test_provider_tools.py`, `test_adapter_registry.py`,
  `model_adapters/test_litellm_adapter.py` — extend the per-provider
  parametrized cases
- `.agents/scripts/provider_utils.py` — add the provider's model-catalog
  endpoint so agent tooling can enumerate its models
- Thinking levels — find out how the provider takes reasoning effort, and
  where it publishes the values each model accepts. Add a quirks section
  that names that source and says thinking levels are required wherever
  it lists values. If the provider silently remaps unsupported values,
  add the provider to `check_thinking_levels.py`, because a
  paid test cannot see a remap.
- `model_adapters/test_thinking_level_paid.py` — add the provider to the
  provider list of `test_thinking_level_reasoning_content`, so each of its
  thinking levels gets a paid test case

**Known gap:** the generated Copilot API client
(`app/desktop/studio_server/api_client/`) mirrors the remote Copilot service's
schema — it can't learn the new provider until that service updates. Note it in
the PR rather than hand-editing generated code.

### Friendly names everywhere

The raw enum value must never be user-visible. When you add the provider,
verify a friendly name exists in **both** name maps (python
`provider_name_from_id` and web `provider_name_map`) — pyright and typescript
respectively force these when the enum gains a member, which is why the
plumbing PR must not skip them.

---

## Provider Quirks Reference

### Anthropic
- Newer models (Opus 4.1+, Sonnet 4.5+) need `temp_top_p_exclusive=True`
- Opus 4.5+ uses `json_schema`; older Opus uses `function_calling`
- Extended thinking models: `anthropic_extended_thinking=True` + `reasoning_capable=True`

### OpenAI
- Most GPT models use `json_schema` for structured output
- GPT-5.x models support `available_thinking_levels` — see [Thinking Levels Reference](#thinking-levels-reference)
- Chat/instant variants (e.g. GPT-5.3 Instant) may not support reasoning effort
- o-series models have fixed thinking tiers (separate model entries per tier, not configurable levels)

### Google/Gemini
- `gemini_reasoning_enabled=True` for reasoning-capable models
- Gemini 3.x models support `available_thinking_levels` — see [Thinking Levels Reference](#thinking-levels-reference)
- Rich multimodal support (audio, video, images, documents)

### DeepSeek
- R1 models: `parser=ModelParserID.r1_thinking` + `reasoning_capable=True`
- V3 models: often available on OpenRouter, Fireworks, SiliconFlow CN
- Some need `r1_openrouter_options=True` + `require_openrouter_reasoning=True`

### OpenRouter (general)
- Slugs: `vendor/model-name`
- Never add a `:free` slug. Free tiers have low rate limits and are temporary. If OpenRouter serves a model only as `:free`, wait for a paid slug.
- Reasoning models: may need `require_openrouter_reasoning=True`
- Some models: `openrouter_skip_required_parameters=True`
- Logprobs: `logprobs_openrouter_options=True` if supported
- Always `multimodal_requires_pdf_as_image=True` (OpenRouter's PDF routing breaks LiteLLM)
- Thinking levels: offer only values in the model's `reasoning.supported_efforts` (see [Thinking Levels Reference](#thinking-levels-reference)). OpenRouter maps an unlisted effort to a listed one, so an extra level silently runs a different effort. When the model has no `reasoning.supported_efforts`, OpenRouter turns the effort into a thinking-token budget for Claude, so `CLAUDE_OPENROUTER_THINKING_LEVELS` (with Minimal) fits only those Claude models. `check_thinking_levels.py` checks every OpenRouter entry; a missing level list on OpenRouter is a note, because Kiln leaves levels off some models on purpose.

### Fireworks AI and Together AI: thinking levels

Neither publishes the efforts each model accepts, and both can accept an unsupported effort without an error. Fireworks takes `xhigh`, `max` and `minimal` on GLM 5.3 with HTTP 200. Together took a made-up effort on Nemotron 3.5, but Muse Spark Open's chat template rejects one. A paid test cannot see this. Add thinking levels on these providers only when the vendor documents the levels for that model (the model card or its chat template), and name that source in the PR. OpenAI and Anthropic return a 400 that names the supported levels, so their paid per-level test catches a wrong level.

### Featherless AI

Serverless host for HuggingFace-hosted open weights. Several hard constraints — read before adding any model:

- **`json_instructions` is the only usable structured output mode.** LiteLLM's `featherless_ai` provider rejects `response_format` outright (`UnsupportedParamsError`), so `json_schema`, `json_mode`, and `json_instruction_and_object` are all unavailable. Routing it as a custom `openai` provider bypasses that gate, but was tested and is *not* reliable: GLM 5.2 accepts `json_schema` and silently ignores it (returns unstructured text), while DeepSeek V4 Pro and Kimi K2.6 return an APIError. Don't use it.
- **Gated models return HTTP 403 `model_gated_needs_oauth`** and cannot work for arbitrary Kiln users — they require each user to link a HuggingFace org to their Featherless account. **Always filter `is_gated` before adding.** Note all 20 official `meta-llama/*` repos are gated, so no Llama variant is usable.
- **No cost reporting.** Featherless models aren't in LiteLLM's price map (only two legacy `Qwerky` entries, on `main` too — not a version issue), and Featherless doesn't return cost in the `usage` object. Runs record tokens with `cost: null`.
- **Not in models.dev or the LiteLLM catalog**, so their `/v1/models` endpoint is the only authoritative source. See [Lagging Providers](#lagging-providers).
- Quality varies per deployment — verify with a paid run. Qwen 3.5 397B, for example, returns degenerate output (rambles to the token cap) and was excluded for that reason.

### TypeSafe AI

Serves TypeSafe's Jev models (System One API). The only provider in the catalog that does not run through LiteLLM — read this before adding an entry:

- **Adapter-backed, not LiteLLM-backed.** Every provider entry must set `adapter=ModelAdapterId.jev`; `JevAdapter` calls `POST /v1/systemone` directly. `lite_llm_core_config_for_provider` and `get_litellm_provider_info` raise for `typesafe` on purpose, so the LiteLLM parametrized tests exclude it.
- **Structured output only.** No text generation, no tools, no multi-turn. A task runs only if its output schema maps onto Jev's question types (enum, boolean, score); anything else fails before the network call. Set `supports_structured_output=True`, `supports_data_gen=False`, `supports_function_calling=False`, and `structured_output_mode=StructuredOutputMode.json_schema`.
- **No logprobs.** `supports_logprobs=False`. Jev returns its own per-answer probabilities instead, which is what keeps it out of G-Eval.
- **Not in models.dev or the LiteLLM catalog, and `GET /v1/models` cannot confirm a `model_id`.** It lists aliases (`jev-latest`, `jev-preview`), not the pinned versions Kiln registers, and it returns `{"models": [{"name", ...}]}` rather than the OpenAI `{"data": [{"id", ...}]}` shape, so the shared `fetch_openai_compat` path cannot read it either. That is why `typesafe` is in `SKIP_PROVIDERS` in `.agents/scripts/provider_utils.py` rather than in `PROVIDER_CONFIG`. The authoritative check for a pinned id is a `POST /v1/systemone` round trip that the API accepts — that is how `jev-1.13.0` was confirmed. (The listing is still what the connect flow uses to validate a key; it is account-scoped and rejects a bad one.)

### Cloudflare (Workers AI)

Runs through LiteLLM's generic `openai/` route against Cloudflare's OpenAI-compatible endpoint (`/accounts/{id}/ai/v1/chat/completions`), with an optional AI Gateway header. Kiln does **not** use LiteLLM's `cloudflare/` provider. Read this before adding or changing any entry:

- **Inclusion rule: only Cloudflare's newer, OpenAI-format text-generation models.** Cloudflare has two generations of text model. The newer ones (GLM, Kimi, DeepSeek V4, Gemma 4, Qwen 3.8, Nemotron 3) speak OpenAI Chat Completions natively. The older ones (Llama 3.x/4, Mistral, older Qwen, GPT-OSS, Granite, the LoRA bases) sit behind a translation layer: they default to **256 output tokens** (silent truncation) and have unreliable tool calling and JSON mode. Do not add the older ones.
- **How to tell them apart:** look at the model's input schema in the catalog JSON. OpenAI-format models list `max_completion_tokens` (and `tool_choice`); older models list `prompt`, `raw` and `lora` instead, and no `max_completion_tokens`:
  ```bash
  jq -r '.models[] | select(.task.name=="Text Generation")
    | [.name, (if (.schema.input | tostring | test("max_completion_tokens")) then "openai-format" else "OLDER - skip" end)] | @tsv' /tmp/cf_models.json
  ```
- **Also skip:** coding models (`kimi-*-code`, `qwen*-coder-*`), `llama-guard-*` (classifier), `-lora` bases, and models Kiln has no `ModelName` for unless the user asks for a new model.
- **Usually a new provider on an existing model.** Most Cloudflare models are already in Kiln under other providers; add a `KilnModelProvider(name=ModelProviderName.cloudflare, ...)` to the existing `KilnModel`.
- **Setting flags from the catalog's `properties[]`** (confirm every flag with the paid tests — the catalog has no JSON-mode flag):

  | Cloudflare property | Kiln setting |
  |---|---|
  | `function_calling: "true"` | leave `supports_function_calling` at its default; set `False` if the tool tests fail |
  | `vision: "true"` | `supports_vision`, `multimodal_capable`, MIME types TXT/MD/JPG/PNG; start broad, narrow on 400s. Don't set `supports_doc_extraction`: the document extractor doesn't send the AI Gateway header, so extraction would bypass the user's gateway |
  | `reasoning: "true"` | keep `reasoning_capable=False` (see [Reasoning Capable Default](#reasoning-capable-default)); LiteLLM already reads `reasoning_content` |
  | `reasoning_effort.supported_efforts` present | **required:** set `available_thinking_levels` to exactly these values, and `default_thinking_level` to `default_effort`. See [Cloudflare thinking levels](#cloudflare-thinking-levels) |
  | `reasoning_effort` without `supported_efforts` | no thinking levels; the catalog does not say which values work |
  | `require_workers_paid: "true"` | no Kiln field. Free-plan users get 403 / code 5035. Mention it in the PR |
  | `planned_deprecation_date` present | do **not** add the model |

- <a id="cloudflare-thinking-levels"></a>**Thinking levels come from the catalog, and only from the catalog.** Kiln sends the level as `reasoning_effort`. For each model, `reasoning_effort` in the catalog lists `supported_efforts`, a `default_effort`, and `normalizes_to`: the values that Cloudflare silently changes to another effort. A remapped value returns HTTP 200, so a live call or a paid test cannot find it. Example: GLM 5.3 maps `medium` to `max`, so a "Medium" option runs the most expensive effort.
  - Offer only values in `supported_efforts`, and each of them unless a completed paid test shows that the level does not work. The list is not always right: on DeepSeek V4 Pro 0813, DeepSeek V4 Flash 0731 and GLM 5.2, `none` still returns reasoning, so those entries have no Off level. A timeout, a rate limit or a credential error proves nothing about the level: keep the level and report it as untested. Do not copy the levels of the same model on another provider.
  - Labels: `none` → `"Off"`, `low` → `"Low"`, `medium` → `"Medium"`, `high` → `"High"`, `xhigh` → `"Extra High"`, `max` → `"Max"`. List them from least to most effort.
  - `default_thinking_level` is `default_effort`, unless the entry is suggested for evals (see [Judge Models Must Reason by Default](#judge-models-must-reason-by-default)).
  - `mandatory: true` means `none` is not in the list; do not add an Off level.
  - Reuse a constant when the values match exactly; else name a new one `{MODEL}_CLOUDFLARE_THINKING_LEVELS`.
  - Cloudflare edits these lists. After every change to a Cloudflare entry, and on every run of this skill, run the checker below. It must report no ❌ finding:

    ```bash
    uv run python .agents/skills/claude-maintain-models/scripts/check_thinking_levels.py
    ```

    It compares every Cloudflare and OpenRouter entry in this checkout with the provider's catalog, and exits 1 on a level or default outside `supported_efforts`. For Cloudflare it also exits 1 on a missing level list, or levels on a model whose catalog lists none. `--provider cloudflare` or `--provider openrouter` checks one provider. A `note:` line is information only.
  - Then run the paid `test_thinking_level_reasoning_content` for each new level. If `-k "<model_enum>"` collects no `cloudflare/...` case, `cloudflare` is missing from the provider list in `libs/core/kiln_ai/adapters/model_adapters/test_thinking_level_paid.py`; add it.
- **Structured output:** these models take the OpenAI `response_format` envelope, so start from `json_schema` and fall back only on test failures. Cloudflare's JSON mode doesn't support streaming.
- **Don't send `max_tokens`.** Live tests showed no truncation on the OpenAI-format models without it.
- **Workers Paid:** many of the newer models need the Workers Paid plan. Paid tests need an account on it.
- **Rate limits:** 300 requests/minute for most text models, but only 20/minute per model for Workers Paid-only models. Don't run the paid suite with many workers against one model; use `-n 4` or lower.
- **Slow models:** Qwen 3.8 27B runs at about 23 tokens/s. Long non-streaming generations hit Cloudflare's server-side timeout (HTTP 408, code 3046, after about 2 minutes).
- **No cost reporting.** Kiln sends bare `@cf/...` ids through LiteLLM's `openai/` route, which never match LiteLLM's price map (its few entries are keyed `cloudflare/@cf/...`), so every Cloudflare run records tokens with `cost: null`.
- **No Model Substitution.** When Cloudflare retires a model it may **silently alias** the old ID to a different, often more expensive, model (Kimi K2.5 → K2.6 in May 2026). The response still echoes the requested ID, so Kiln can't detect it at runtime. Cloudflare announces retirements by adding `planned_deprecation_date` to the model (notice has been 13–22 days). **Remove or mark deprecated every Kiln Cloudflare entry that gains a `planned_deprecation_date`, before that date.** The `kiln-check-deprecation` skill reports these dates.

### Qwen3 / Thinking Models
- Thinking variants: `reasoning_capable=True`, `parser=ModelParserID.r1_thinking`
- No-thinking variants: `formatter=ModelFormatterID.qwen3_style_no_think`
- SiliconFlow may need `siliconflow_enable_thinking=True/False`

---

## Thinking Levels Reference

Two providers publish the thinking levels per model, and their list is the only source for their entries: Cloudflare (see [Cloudflare thinking levels](#cloudflare-thinking-levels)) and OpenRouter (`reasoning.supported_efforts` in its models API, see [OpenRouter (general)](#openrouter-general)). For every other provider the levels must be sourced by hand. Use this lookup chain in priority order.

### Lookup Chain

1. **Vendor model page** (most authoritative)
   - **OpenAI:** Each model page includes "Reasoning.effort supports: X, Y, Z" in the description text. URL: `https://developers.openai.com/api/docs/models/{model-id}`
   - **Anthropic:** The [effort docs](https://platform.claude.com/docs/en/build-with-claude/effort) list levels per model. Opus 4.6 supports `low, medium, high, max`; Sonnet 4.6 supports `low, medium, high`.
   - **Google Gemini:** The models API returns `thinking: true/false` (boolean only). Levels come from docs.

2. **Vercel AI Gateway docs** — clean structured tables per provider:
   - `https://vercel.com/docs/ai-gateway/capabilities/reasoning/openai`
   - `https://vercel.com/docs/ai-gateway/capabilities/reasoning/anthropic`
   - `https://vercel.com/docs/ai-gateway/capabilities/reasoning/google`

3. **Inherit from predecessor** — if the same family/tier model has a `_THINKING_LEVELS` dict, the new model very likely uses the same or a superset.

4. **OpenRouter `reasoning`** — most models list their efforts:
   ```bash
   curl -s https://openrouter.ai/api/v1/models | jq '.data[] | select(.id == "SLUG") | {reasoning, supported_parameters}'
   ```
   `reasoning.supported_efforts` lists the efforts (highest first) and `reasoning.default_effort` the default. If `reasoning` is absent and `supported_parameters` has no `reasoning`, the model does not support effort levels — skip thinking levels entirely. OpenRouter documents that it maps an unsupported effort to the nearest supported one for some models, so an extra level may silently run a different effort.

5. **Smoke test** — as a last resort, send a request with an invalid effort level and check the error message, which often enumerates the valid values.

### Judge Models Must Reason by Default

**If a provider entry sets both `suggested_for_evals=True` and `available_thinking_levels`, set `default_thinking_level` to `"medium"` when the model offers a medium level, else `"low"`. Never leave it at `"none"` or unset.**

The V2 LLM judge runner builds the judge model's run config without a `thinking_level`, so the adapter falls back to the provider entry's `default_thinking_level`. A default of `"none"` therefore ships a judge that never reasons, and the eval results page shows no thinking for it.

This applies only to judge-tagged entries. A non-judge entry of the same model may keep `"none"` — the two are set independently per provider entry.

`test_judge_models_with_thinking_levels_default_to_reasoning` in `libs/core/kiln_ai/adapters/test_ml_model_list.py` enforces this across the whole list.

### Important Distinctions

- **Effort-level models** (GPT-5.x, Claude 4.x, Gemini 3.x) → add `available_thinking_levels` dicts
- **R1-style thinking models** (DeepSeek R1, Qwen thinking variants) → on/off thinking, NOT effort levels. Use `reasoning_capable=True` + `parser=ModelParserID.r1_thinking`. Do NOT add thinking level dicts.
- **Chat/instant models** (e.g. GPT-5.3 Instant) → may not support reasoning effort at all. Verify on the vendor model page.

### Existing Constants

Reuse when levels match exactly. Create a new constant only if levels differ. This is not an exhaustive list.

| Constant | Levels | Default | Used by |
|----------|--------|---------|---------|
| `GPT_5_4_OPENAI_THINKING_LEVELS` | none, low, medium, high, xhigh | none | GPT-5.4 |
| `GPT_5_4_PRO_OPENAI_THINKING_LEVELS` | medium, high, xhigh | medium | GPT-5.4 Pro |
| `GPT_5_2_OPENAI_THINKING_LEVELS` | none, low, medium, high, xhigh | none | GPT-5.2, GPT-5.2 Chat |
| `GPT_5_2_PRO_OPENAI_THINKING_LEVELS` | medium, high, xhigh | medium | GPT-5.2 Pro |
| `GPT_5_1_OPENAI_THINKING_LEVELS` | none, low, medium, high | none | GPT-5.1 |
| `GPT_5_OPENAI_THINKING_LEVELS` | minimal, low, medium, high | medium | GPT-5, GPT-5 Mini, GPT-5 Nano, GPT-5 Chat |
| `GEMINI_3_PRO_THINKING_LEVELS` | low, medium, high | high | Gemini 3 Pro, Gemini 3.1 Pro |
| `GEMINI_3_FLASH_THINKING_LEVELS` | minimal, low, medium, high | high | Gemini 3 Flash, Gemini 3.1 Flash Lite |
| `CLAUDE_ANTHROPIC_EFFORT_THINKING_LEVELS` | low, medium, high | high | Claude (Anthropic direct) |
| `CLAUDE_OPENROUTER_THINKING_LEVELS` | none, minimal, low, medium, high, xhigh | none | Claude (OpenRouter) |

### Sources That Do NOT Work

These were investigated and confirmed to lack thinking level data:
- **LiteLLM catalog** — only `supports_reasoning: true/false`
- **Google Gemini models API** — only `thinking: true/false`
- **OpenAI `/v1/models` endpoint** — minimal object with no capability fields

---

## Slug Lookup Reference

Use **both** LiteLLM and models.dev when looking up slugs — they complement each other. LiteLLM gives you the exact slugs Kiln will use (since Kiln runs on LiteLLM), while models.dev often has broader coverage of newer or niche models with pricing, context limits, and capability details.

### LiteLLM Model Catalog (https://api.litellm.ai/model_catalog)

100 free requests/day, no key needed. Supports server-side filtering: `model=` (substring match), `provider=`, `mode=`, `supports_vision=true`, `supports_reasoning=true`, `page_size=500`.

```bash
# Find all variants of a model across providers:
curl -s 'https://api.litellm.ai/model_catalog?model=MODEL_NAME&mode=chat&page_size=500' \
  -H 'accept: application/json' | jq '.data[] | {id, provider, mode, max_input_tokens, supports_vision, supports_reasoning, supports_function_calling}'

# List all models for a provider:
curl -s 'https://api.litellm.ai/model_catalog?provider=PROVIDER&mode=chat&page_size=500' \
  -H 'accept: application/json' | jq '.data[].id'
```

### models.dev (https://models.dev/api.json)

Mega JSON covering 50+ providers with model IDs, pricing, context limits, capabilities, and release dates. **Large file — always use curl+jq, never WebFetch.**

```bash
# Search all model IDs across all providers:
curl -s https://models.dev/api.json | jq '[to_entries[].value.models // {} | keys[]] | .[]' | grep -i "SEARCH_TERM"

# List all model IDs for a specific provider:
curl -s https://models.dev/api.json | jq '.["PROVIDER"].models | keys[]'

# Get full details for a specific provider+model:
curl -s https://models.dev/api.json | jq '.["PROVIDER"].models["MODEL_ID"]'
```

### Other verified sources
- OpenRouter: `curl -s https://openrouter.ai/api/v1/models | jq '.data[].id' | grep -i "SEARCH_TERM"`
- Anthropic: https://docs.anthropic.com/en/api/models/list
- Cerebras: https://inference-docs.cerebras.ai/models/overview

### Lagging Providers

Fireworks, Together, SiliconFlow and Cloudflare typically expose new models on their own endpoints 1–2 weeks before models.dev / LiteLLM catch up. For these providers, **always** cross-check directly — both when adding a new model and when running the [Phase 1B backfill check](#phase-1b--lagging-provider-backfill-check-every-run).

**Fireworks AI** — model pages are the most current source. WebFetch directly:
```
WebFetch https://fireworks.ai/models/fireworks/{model-slug}
```
Or browse the catalog at https://fireworks.ai/models. Kiln slug format: `accounts/fireworks/models/{model-slug}`.

**Together AI** — the `/v1/models` endpoint requires an API key. The key is set in the user's shell as **`TOGETHERAI_API_KEY`** (note the `AI` — this matches LiteLLM's `together_ai` env var; the plain `TOGETHER_API_KEY` is usually NOT set):
```bash
# List all Together model IDs matching a term:
curl -s https://api.together.xyz/v1/models \
  -H "Authorization: Bearer $TOGETHERAI_API_KEY" | jq '.[] | .id' | grep -i "SEARCH_TERM"

# Full record for a specific slug:
curl -s https://api.together.xyz/v1/models \
  -H "Authorization: Bearer $TOGETHERAI_API_KEY" | jq '.[] | select(.id == "SLUG")'
```
If `TOGETHERAI_API_KEY` isn't set, ask the user before prompting them to export it — don't fail silently onto models.dev.

**Featherless AI** — appears in **neither** models.dev nor the LiteLLM catalog (the catalog has only two stale `Qwerky` entries), so `/v1/models` is the sole authoritative source. It needs **no API key** to enumerate.

**The response is enormous — ~22,000 models, the vast majority community fine-tunes. Never WebFetch it and never dump it raw.** Save it once, then filter with `jq`:

```bash
curl -s https://api.featherless.ai/v1/models -o /tmp/feath.json
jq '.data | length' /tmp/feath.json
```

Restrict to official vendor orgs, or you'll drown in forks like `DavidAU/Gemma3-27B-it-vl-GLM-4.7-Uncensored-Heretic`:

```bash
# newest official models, excluding gated ones (gated = unusable, see Provider Quirks)
jq -r '.data[]
  | select(.is_gated | not)
  | select(.id | test("^(deepseek-ai|zai-org|Qwen|moonshotai|MiniMaxAI|mistralai|openai|google|microsoft)/"))
  | "\(.id)\tctx=\(.context_length)\t$\(.pricing.input)/\(.pricing.output)\ttools=\(.features.tool_use // false)\timg=\(.features.image_input // false)"' \
  /tmp/feath.json | sort
```

Look up one model, including its capability flags and exact pricing:

```bash
jq '.data[] | select(.id == "zai-org/GLM-5.2")' /tmp/feath.json
```

**The highest-precision way to find backfill candidates** is to check whether a slug Kiln *already ships* for `together_ai` / `siliconflow_cn` exists verbatim here — those providers use the same HuggingFace repo-id convention, so a hit needs no guessing:

```bash
jq -r '.data[] | select(.is_gated | not) | .id' /tmp/feath.json | sort > /tmp/feath_ids.txt
grep -Fx "zai-org/GLM-5.2" /tmp/feath_ids.txt   # exit 0 = safe to add
```

Slugs are **case-sensitive** (`google/gemma-4-31B-it`, not `-31b-it`). Always confirm `is_gated` is false and cross-check `features.tool_use` / `features.image_input` against the flags you set — don't claim capabilities the endpoint doesn't advertise.

**Cloudflare Workers AI** — the authoritative list is Cloudflare's public, unauthenticated catalog JSON. It is about 900 KB and can return a transient 502, so save it to a file (retry on 502) and filter with `jq`; never WebFetch it:

```bash
curl -s https://ai-cloudflare-com.pages.dev/api/models -o /tmp/cf_models.json

# Text-generation models, newest first, with Kiln-relevant flags
jq -r '.models[] | select(.task.name=="Text Generation")
  | (reduce .properties[] as $p ({}; .[$p.property_id]=$p.value)) as $p
  | [.name, .created_at[0:10], ($p.context_window//"-"),
     (if $p.function_calling=="true" then "tools" else "" end),
     (if $p.vision=="true" then "vision" else "" end),
     (if $p.reasoning=="true" then "reasoning" else "" end),
     (if $p.require_workers_paid=="true" then "paid" else "" end),
     (if $p.lora=="true" then "lora" else "" end),
     ($p.planned_deprecation_date//"")] | @tsv' /tmp/cf_models.json | sort -t$'\t' -k2 -r

# Full record for one model (drop the large schema)
jq '.models[] | select(.name=="@cf/zai-org/glm-5.3") | del(.schema)' /tmp/cf_models.json
```

Model IDs are the `name` field (`@cf/<author>/<model>`). With credentials (`CLOUDFLARE_API_KEY` with Workers AI access, plus `CLOUDFLARE_ACCOUNT_ID`), the official API returns the same records; add `&include_deprecated=true` to see models retired in the last three months:

```bash
curl -s "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/ai/models/search?task=Text%20Generation&per_page=100" \
  -H "Authorization: Bearer $CLOUDFLARE_API_KEY" | jq -r '.result[].name'
```

- **Cross-check:** models.dev, provider key `cloudflare-workers-ai` (`curl -s https://models.dev/api.json | jq '.["cloudflare-workers-ai"].models | keys[]'`). It syncs hourly and is good for spotting additions, but it no longer deletes retired models, so never use it to decide a model still exists.
- **Never use LiteLLM's catalog for Cloudflare.** Its `cloudflare/` entries are a one-time import from June 2026: they miss newer models and still list retired ones.
- **Fallback:** the per-model JSON files in the `cloudflare/cloudflare-docs` GitHub repo, `src/content/workers-ai-models/*.json` (same shape as the catalog).
- Also scan the [Workers AI changelog](https://developers.cloudflare.com/changelog/product/workers-ai/) for launches in the last two weeks, in case the catalog lags.

See [Cloudflare (Workers AI)](#cloudflare-workers-ai) in Provider Quirks for the inclusion rule and flags.

**SiliconFlow** — WebFetch the public model catalog page, or a specific model page if you have the vendor/model path:
```
WebFetch https://siliconflow.com/models
WebFetch https://siliconflow.com/models/{vendor}/{model}
```

When you find a new reliable slug source, append it here.