--- name: workflow-ai-enhancement description: Restore, upscale, and enhance existing footage using 2026 open-source AI models — Real-ESRGAN/SwinIR/HAT super-resolution, RIFE/FILM interpolation, DeepFilterNet/RNNoise audio denoise, rembg/BiRefNet/RVM matting, Depth-Anything v2 depth — with strict OSI-open commercial-safe license filter. Use when the user says "upscale old footage", "remaster", "enhance quality", "30 to 60fps", "AI denoise", "restore VHS", "remove background from video", or anything about AI-driven footage restoration. argument-hint: [source] --- # Workflow — AI Enhancement **What:** Take existing footage and make it visually/sonically better using open-source AI. Strict license discipline: Apache-2 / MIT / BSD / GPL only. NC / research-only models are documented-and-dropped. ## Skills used `media-upscale`, `media-interpolate`, `media-matte`, `media-depth`, `media-denoise-ai`, `media-demucs`, `ffmpeg-denoise-restore`, `ffmpeg-stabilize`, `ffmpeg-ivtc`, `ffmpeg-hdr-color`, `ffmpeg-lut-grade`, `ffmpeg-ocio-colorpro`, `ffmpeg-transcode`, `ffmpeg-hwaccel`, `ffmpeg-quality`. ## Pipeline ### Step 1 — Probe source `ffmpeg-probe`. Capture resolution, frame rate, codec artifacts, audio noise floor, color space (BT.601 / BT.709 / SDR), bit depth, chroma. ### Step 2 — Undo legacy artifacts BEFORE AI - **Telecined 29.97i → 23.976p** — `ffmpeg-ivtc` (`fieldmatch → decimate`). - **True interlaced** — `yadif` via `ffmpeg-video-filter`. - **Compression noise** — `ffmpeg-denoise-restore` (`nlmeans`, `hqdn3d`). - **Unstable handheld** — `ffmpeg-stabilize` two-pass (`vidstabdetect` → `vidstabtransform`). ### Step 3 — AI super-resolution Use `media-upscale`: - **Real-ESRGAN x4plus** — live-action default. - **Real-ESRGAN anime6b** — animation. - **SwinIR** — graphics / text. - **HAT** — best-quality slowest. - **Face-enhance** — for close-ups; never use CodeFormer (NC). ### Step 4 — Frame interpolation `media-interpolate`: - **RIFE v4.6** (MIT) — handles scene cuts. - **FILM** (Apache-2) — slightly smoother, drops on scene cuts. Target integer multiples for clean math (23.976 → 47.952 exact 2×). 23.976 → 60 is 2.504× — quality suffers on fractional positions. ### Step 5 — AI audio denoise `media-denoise-ai`: - **DeepFilterNet** — general, 16 kHz or 48 kHz MONO only. - **RNNoise** — lightweight, hardcoded 48 kHz mono 16-bit. - **Resemble Enhance** — speech super-res. Isolate stems first with `media-demucs` if source is mixed. ### Step 6 — Background removal (optional) `media-matte`: - **rembg** (MIT) — fastest. - **BiRefNet / RMBG-2.0** — stills only; per-frame for video produces temporal flicker. - **RVM (RobustVideoMatting)** — temporally coherent, GPL-3 (propagates if shipped embedded; commercial OK via dynamic linking). ### Step 7 — Depth estimation (optional) `media-depth`: - **Depth-Anything v2** (Apache-2) — fastest, relative depth. - **MiDaS** — for relighting / 3D reprojection. ### Step 8 — Color + HDR finish LUT (`ffmpeg-lut-grade`), OCIO ACES (`ffmpeg-ocio-colorpro`), or SDR→HLG tone-map (`ffmpeg-hdr-color`). ### Step 9 — QC + final encode `ffmpeg-quality` VMAF vs source (80–95 expected). `ffmpeg-transcode` to delivery: H.264 10-bit `yuv420p10le` for broad compat, AV1 for bandwidth-constrained. ## Variants - **Animation** — Real-ESRGAN anime variant + RIFE with smooth motion. - **Face-focused** — GFPGAN (MIT) for talking heads. NEVER CodeFormer (NC). - **VHS → 4K archive** — detect cadence → `yadif` → heavy `hqdn3d` → upscale 480p → 1440p → color correct. - **Game capture 30 → 120 fps** — RIFE with higher factors. - **Handheld + upscale** — stabilize FIRST (crop introduced), THEN upscale. ## Gotchas - **License landmines — always-drop list:** CodeFormer (NC), DAIN (research), XTTS-v2 / F5-TTS (CPML NC), Stable Video Diffusion (NC), Wav2Lip / SadTalker (research), MusicGen Meta (CC-BY-NC), Surya (commercial restriction). Each AI skill's `references/LICENSES.md` pins the allow-list. - **Upscaling amplifies noise.** ALWAYS denoise first — upscaler + noisy source = HD noise. - **RIFE handles scene cuts; FILM doesn't.** Watch for ghost frames on FILM output. - **Real-ESRGAN default tile size 400** — drop to 200 if VRAM-bound. Model doesn't "see" the whole frame; artifacts at tile borders on wild tile sizes. - **After AI upscale, force `-pix_fmt yuv420p`** (or `yuv420p10le` for 10-bit deliver). AI models often output RGB or YUV 4:4:4; consumer players need 4:2:0. - **10-bit upscale into 8-bit codec = banding.** If upscale to 10-bit, deliver in 10-bit: `libx264 -pix_fmt yuv420p10le` or equivalent HEVC. - **DeepFilterNet expects 16 kHz or 48 kHz MONO.** Feed right format or it downsamples badly. - **RNNoise hardcoded 48 kHz mono 16-bit.** Resample first or it fails silently. - **Demucs expects 44.1 kHz or 48 kHz STEREO.** Mono input → degraded stems. - **RVM is GPL-3.** If shipped embedded in proprietary code, GPL-3 propagates. Commercial OK via dynamic linking. - **rembg default `u2net`** is fine; `isnet-general-use` is better but heavier. - **BiRefNet is image-only.** Per-frame video = temporal flicker. Use RVM for temporal coherence. - **Depth-Anything v2 outputs RELATIVE depth.** Good for VFX; NOT for measurement. - **Hugging Face model licenses can change without notice.** Pin exact commit hash in `references/LICENSES.md`. - **AI inference benefits 10× from GPU.** On CPU, Real-ESRGAN 1080p30s = hours. - **`ffmpeg -hwaccel` is decode-only.** AI models run on their own CUDA / Metal / ROCm paths. - **Apple Silicon (M1–M4) MPS:** set `PYTORCH_ENABLE_MPS_FALLBACK=1` for unported ops. ## Example — Upscale + interpolate old 720p30 → 4K60 Probe source → denoise with `hqdn3d` → Real-ESRGAN x4plus to 2880×? tile=400 → RIFE v4.6 ×2 → LUT grade → ffmpeg-transcode to HEVC 10-bit HDR-ready `yuv420p10le`. VMAF check against source at 720p (upscale reference). Deliver. ## Related - `workflow-ai-generation` — pure AI-generated media (not enhancement of existing). - `workflow-vod-post-production` — traditional color/stabilize/denoise path without AI. - `workflow-hdr` — if AI output needs HDR mastering. --- Source attribution: damionrashford Original project: https://github.com/damionrashford/media-os/blob/5014c1e8b23fd3e18d49926d9aa147d15a3aa08e/skills/workflow-ai-enhancement/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License Copyright (c) 2026 Media OS Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: damionrashford Original project: https://github.com/damionrashford/media-os/blob/5014c1e8b23fd3e18d49926d9aa147d15a3aa08e/skills/workflow-ai-enhancement/SKILL.md Attribution basis: Source repository record Fields of work: Content & Media, Design, Marketing Roles: Content Marketer, Graphic Designer, Video Producer Specialties: API Tools: FFmpeg, Birefnet, Deepfilternet, Depth Anything, Film, Hat, Real Esrgan, Rembg, Rife, Rnnoise, Rvm, Swinir Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: d93c318829d4a0842e7d4ab494a0b37c3809a1bb99e989a318a8812919262094