--- name: kiln-check-finetune-deprecation-597fae4d54cb description: Check Kiln's fine-tunable model list for deprecated or unsupported base models. Use when the user wants to audit fine-tuning support, check if fine-tune base models are still valid, or mentions fine-tune model deprecation. --- # Check Fine-Tune Model Deprecation in Kiln Audit the fine-tunable models listed in Kiln to find base models that are no longer supported for fine-tuning by their providers. This is separate from general model deprecation — a model can still be available for inference but no longer supported as a fine-tuning base. --- ## Global Rules - **Sandbox:** All `curl`, `uv run`, and `python3` commands that hit the network MUST use `required_permissions: ["all"]`. The sandbox blocks network access. - **Env vars:** Source `.env` before running check scripts: `export $(grep -v '^#' .env | xargs)` - **Vertex AI auth:** The Vertex check requires `gcloud` CLI authentication. Before running, ask the user to run `gcloud auth login` if they haven't recently. If `gcloud auth print-access-token` fails, prompt the user to authenticate. - **Non-destructive:** This skill only reports findings. It does not automatically remove or deprecate models. --- ## Supported Providers | Provider | Env Var | Auth | |----------|---------|------| | OpenAI | `OPENAI_API_KEY` | Bearer | | Together AI | `TOGETHER_API_KEY` | Bearer | | Vertex AI | `VERTEX_PROJECT_ID` + `gcloud` CLI auth | OAuth (gcloud) | | Fireworks AI | `FIREWORKS_API_KEY` | Bearer | --- ## Background Kiln has two types of fine-tunable model entries: 1. **Static entries** — Models in `libs/core/kiln_ai/adapters/ml_model_list.py` with `provider_finetune_id` set. Currently used by Together AI and Vertex AI. 2. **Dynamic entries (Fireworks)** — Fetched at runtime from `api.fireworks.ai/v1/accounts/fireworks/models` filtering by `tunable=True`. This list is managed by Fireworks and may include stale/unsupported models. The API endpoint that serves the fine-tune dropdown is `GET /api/finetune_providers` in `app/desktop/studio_server/finetune_api.py`. --- ## Phase 1 – Check Static Fine-Tune IDs Run the check script from the repo root: ```bash uv run python3 .agents/skills/kiln-check-finetune-deprecation/scripts/check_finetune.py static > /tmp/kiln_finetune_static.json ``` This extracts all `provider_finetune_id` entries from `ml_model_list.py` and checks each provider's docs/API to see if the model is still available for fine-tuning: - **Together AI:** Scrapes the Together AI docs page for the list of supported fine-tune models. Together uses full HuggingFace-style IDs (e.g. `meta-llama/Meta-Llama-3.1-8B-Instruct-Reference`). - **Vertex AI:** Checks the Vertex AI publisher models API. Vertex fine-tuning uses specific versioned model IDs (e.g. `gemini-2.0-flash-001`). **Output:** JSON to stdout with per-provider results, human summary to stderr. --- ## Phase 2 – Check Fireworks Dynamic Models Fireworks fine-tunable models are fetched dynamically from their API, not stored in `ml_model_list.py`. The script checks the API's `supervisedLoraTunable` and `supervisedFullParameterTunable` fields (not the old `tunable` field, which is stale) and cross-references against our allowlist. ```bash uv run python3 .agents/skills/kiln-check-finetune-deprecation/scripts/check_finetune.py fireworks > /tmp/kiln_finetune_fireworks.json ``` This script: 1. Fetches all models with `supervisedLoraTunable=True` or `supervisedFullParameterTunable=True` from `api.fireworks.ai/v1/accounts/fireworks/models` 2. Cross-references against `FIREWORKS_SUPPORTED_FINETUNE_MODELS` from `libs/core/kiln_ai/adapters/fine_tune/fireworks_finetune.py` — the same allowlist that filters the runtime fine-tune dropdown 3. Checks both directions: - **Stale allowlist entries**: models in our allowlist that the API no longer marks as supervised-tunable — these may need to be removed - **Missing from allowlist**: models the API marks as supervised-tunable but aren't in our allowlist — these are candidates to add The canonical allowlist lives in `fireworks_finetune.py` and is shared between the runtime dropdown filter and this audit skill. There is a single place to update when Fireworks changes their supported models. **Interpreting results:** The supervised API fields return ~43 models, but our allowlist intentionally filters to ~20 that match Fireworks' documented supported models. Seeing ~23 "models in API but NOT in allowlist" is normal — these are models Fireworks marks as technically tunable but doesn't officially document or support. Only flag these if a model appears in the Fireworks docs but is missing from our allowlist. The important direction is **stale allowlist entries** — models we ship that the API no longer marks as supervised-tunable. **Output:** JSON to stdout, human summary to stderr. --- ## Phase 3 – Report Findings Read the JSON outputs and present findings in a clear table: ```text Fine-Tune Deprecation Audit Results ==================================== STATIC PROVIDERS (provider_finetune_id in ml_model_list.py): ✅ OpenAI: 5/5 found gpt-4.1-2025-04-14, gpt-4.1-mini-2025-04-14, gpt-4.1-nano-2025-04-14, gpt-4o-2024-08-06, gpt-4o-mini-2024-07-18 ✅ Vertex AI: 2/2 found gemini-2.0-flash-001, gemini-2.0-flash-lite-001 ❌ Together AI: 2/4 missing ✅ Qwen/Qwen2.5-72B-Instruct — found ✅ Qwen/Qwen2.5-14B-Instruct — found ❌ meta-llama/Meta-Llama-3.1-8B-Instruct-Reference — NOT FOUND ❌ meta-llama/Meta-Llama-3.1-70B-Instruct-Reference — NOT FOUND ⏭️ SKIPPED (no credentials): - vertex (VERTEX_PROJECT_ID not set) DYNAMIC PROVIDER (Fireworks): ⚠️ 167 models marked tunable in API ⚠️ 15 models in known-good docs list ✅ 11 models in both API and docs ❌ 156 models in API but NOT in docs (likely stale) - accounts/fireworks/models/qwen2-72b-instruct - accounts/fireworks/models/code-llama-7b - ... ⚠️ 4 models in docs but NOT in API - accounts/fireworks/models/gemma-2-9b-it - ... ``` Note: The example above is illustrative. Actual model counts and entries will vary. --- ## Phase 4 – Remediation Based on findings, recommend specific actions: ### For static entries (`provider_finetune_id`): - If a model's fine-tune ID is no longer valid, either: - **Update** the `provider_finetune_id` to a newer version if one exists - **Remove** the `provider_finetune_id` field to stop offering fine-tuning for that model on that provider - **Set `deprecated=True`** on the provider entry if the model is fully gone ### For Fireworks dynamic entries: - The stale models come from Fireworks' API, not our code. Options: - **Filter in our code** — update `fetch_fireworks_finetune_models()` in `finetune_api.py` to filter against a known-good list - **Report to Fireworks** — flag the stale models to Fireworks support - **Both** — filter now, report later **Always ask the user to confirm** before making any code changes. --- ## Phase 5 – Verify After any changes, run the relevant tests: ```bash uv run python3 -m pytest app/desktop/studio_server/test_finetune_api.py -q ``` --- ## Checklist - [ ] Env vars sourced from `.env` - [ ] Vertex auth confirmed (`gcloud auth print-access-token` works, or skip Vertex) - [ ] Static fine-tune IDs checked against provider APIs (Together, Vertex) - [ ] Fireworks dynamic models cross-referenced against known-good list - [ ] Findings reported to user with clear table - [ ] User confirmed remediation approach - [ ] Code changes made (if any) - [ ] Tests pass after changes - [ ] FIREWORKS_SUPPORTED_FINETUNE_MODELS in fireworks_finetune.py updated if docs have changed --- Source attribution: Kiln-AI Original project: https://github.com/Kiln-AI/Kiln License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License Copyright (c) 2026 Gabriel Moreira Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: Kiln-AI Original project: https://github.com/Kiln-AI/Kiln Attribution basis: Upstream repository explicitly recorded in package mirror path Fields of work: Data & Analytics, Development & Engineering Roles: Data Analyst, Data Scientist, ML / AI Engineer Specialties: API Tools: Curl, Gcloud Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: e3264014a5adda95fa6f8c6f637ff34bf4e363394a5b178e56b6988e307ee7d2