--- name: media-whisper description: > Speech-to-text transcription and subtitle generation with whisper.cpp and faster-whisper: extract spoken text from audio/video, generate SRT/VTT subtitles, translate speech to English, word-level timestamps, multilingual support, speaker diarization optional. Use when the user asks to transcribe a video, auto-generate subtitles from speech, create captions, translate a foreign-language clip to English, extract dialogue as text, build an ASR pipeline, or process podcast audio into a transcript. argument-hint: "[input]" --- # Media Whisper **Context:** $ARGUMENTS ## Quick start - **Transcribe video → SRT:** extract 16 kHz mono WAV (Step 2) → run ASR (Step 3) → SRT is written next to the input. - **Translate foreign speech → English SRT:** Step 2, then `whisper-cpp -tr ...` or faster-whisper `task="translate"` (Step 3). - **Burn-in captions:** generate SRT here, then hand off to the `ffmpeg-subtitles` skill. - **Soft-mux SRT into MP4:** `scripts/whisper.py srt-mux --video in.mp4 --srt out.srt --output tagged.mp4`. ## When to use - Auto-generate captions (SRT/VTT) from audio or video. - Translate non-English audio to an English transcript. - Produce a searchable text transcript or word-level JSON for a podcast/interview. - Build an ASR stage in a larger pipeline (precedes diarization, search indexing, summarization). - Do NOT use for speaker identification — whisper doesn't do diarization reliably. Hand off to `pyannote-audio` (see `references/whisper.md`). ## Step 1 — Pick a backend Two mature local backends. Pick one: | Backend | Install | Strength | Use when | | --------------- | ---------------------------------------------------------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------ | | whisper.cpp | `brew install whisper-cpp` (macOS) or build from source on Linux | Zero Python deps, Metal on Apple Silicon, SRT/VTT built in | Shell-only pipelines, Apple Silicon, want a static binary | | faster-whisper | `pip install faster-whisper` (or `uv pip install faster-whisper`) | ~4× faster on CPU (CTranslate2), word timestamps, VAD | Python pipeline, need word-level timing, GPU CUDA, long files with VAD | | OpenAI API | n/a (HTTP) | Best-quality large-v3, no local compute | Network available, ok paying per minute, confidentiality is not an issue | Check what's present: ```bash uv run .claude/skills/media-whisper/scripts/whisper.py check ``` Download a GGML model for whisper.cpp (one-time): ```bash # Installed with brew: models land in /opt/homebrew/share/whisper.cpp/models bash /opt/homebrew/share/whisper.cpp/models/download-ggml-model.sh base.en # Options: tiny / tiny.en / base / base.en / small / small.en / medium / medium.en / large-v3 / large-v3-q5_0 ``` faster-whisper downloads models automatically on first use into `~/.cache/huggingface/`. ## Step 2 — Extract 16 kHz mono WAV whisper.cpp **requires** 16 kHz mono PCM-s16le WAV. faster-whisper accepts almost any format but benefits from the same normalization. Always do this first: ```bash ffmpeg -y -i in.mp4 -vn -ar 16000 -ac 1 -c:a pcm_s16le audio.wav ``` `scripts/whisper.py transcribe` runs this step automatically if the input is not already a WAV. **Pre-processing for better accuracy** (optional, for podcast / phone audio): loudnorm + denoise before ASR. See `ffmpeg-audio-filter` skill — `loudnorm` two-pass to hit -16 LUFS, then `afftdn=nr=12` for broadband noise. Whisper hallucinates less on normalized input. ## Step 3 — Run ASR ### Option A: whisper.cpp ```bash # Basic: English, write SRT + VTT + plain text whisper-cpp \ -m /opt/homebrew/share/whisper.cpp/models/ggml-base.en.bin \ -f audio.wav \ -l en \ -osrt -ovtt -otxt \ -of out # output basename → out.srt / out.vtt / out.txt # Auto language detection (first 30s used for detection) whisper-cpp -m ggml-large-v3.bin -f audio.wav -l auto -osrt -of out # Translate any language → English SRT whisper-cpp -m ggml-large-v3.bin -f audio.wav -tr -osrt -of out # Word-level timestamps (TSV) whisper-cpp -m ggml-large-v3.bin -f audio.wav -owts -of out # Diarized-ish timestamps (tinydiarize — only works with ggml-small.en-tdrz.bin model) whisper-cpp -m ggml-small.en-tdrz.bin -f audio.wav -tdrz -osrt -of out ``` Useful flags: `-t N` threads, `-p N` processors, `-ml N` max segment length (chars), `-sow` split on word, `-pp` print progress, `-nt` no timestamps (plain text mode), `-et` entropy threshold for hallucination guard. ### Option B: faster-whisper (Python) ```python from faster_whisper import WhisperModel # compute_type: int8 (fast CPU), int8_float16 (GPU balance), float16 (GPU best) model = WhisperModel("large-v3", device="cpu", compute_type="int8") segments, info = model.transcribe( "audio.wav", language="en", # or None for auto-detect task="transcribe", # or "translate" → English word_timestamps=True, vad_filter=True, # removes silence, reduces hallucinations vad_parameters={"min_silence_duration_ms": 500}, ) print(f"Detected language: {info.language} (prob {info.language_probability:.2f})") for seg in segments: print(f"[{seg.start:.2f} -> {seg.end:.2f}] {seg.text}") ``` `segments` is a generator — iterate once, write SRT as you go. `scripts/whisper.py transcribe --backend faster` handles this. ### Option C: Driver script (recommended) ```bash # Auto picks whisper.cpp if binary is on PATH, else faster-whisper uv run .claude/skills/media-whisper/scripts/whisper.py transcribe \ --input podcast.mp3 --output-srt podcast.srt --model base.en --lang en # Force backend uv run .../whisper.py transcribe --backend cpp --input in.mp4 --output-srt out.srt uv run .../whisper.py transcribe --backend faster --input in.mp4 --output-srt out.srt \ --word-timestamps # Translate uv run .../whisper.py translate --input spanish.mp4 --output english.srt --model large-v3 ``` ## Step 4 — Post-process and mux **Soft-mux SRT into MP4** (no re-encode): ```bash ffmpeg -i in.mp4 -i out.srt -c copy -c:s mov_text out.mp4 # or through the script uv run .../whisper.py srt-mux --video in.mp4 --srt out.srt --output tagged.mp4 ``` **Burn-in (hard-coded) subtitles:** hand off to `ffmpeg-subtitles` skill. Or quick form: ```bash ffmpeg -i in.mp4 -vf "subtitles=out.srt" -c:a copy burned.mp4 # or uv run .../whisper.py srt-burn --video in.mp4 --srt out.srt --output burned.mp4 ``` **Trim end-of-file hallucinations:** whisper commonly emits a phantom "Thank you for watching" / "Subtitles by..." at EOF. Check the last cue; drop it if it appears after the audio ended. ## Gotchas - whisper.cpp **requires** 16 kHz mono PCM-s16le WAV. Other formats sometimes work via internal conversion but are unreliable — always normalize with ffmpeg first. - Model sizes (FP16 GGML): **tiny 39M · base 74M · small 244M · medium 769M · large-v3 1.5GB**. RAM need ~= model size + 1GB working set. - **English-only models (`.en` suffix) are faster AND more accurate for English** than the multilingual equivalents. Never use multilingual if you know the content is English. - Multilingual models can **translate to English** (`-tr` / `task="translate"`) but cannot translate between two non-English languages. - **Auto language detection uses only the first 30 s** of audio. Unreliable for code-switching / mixed-language content. Pass `-l ` explicitly when you know the language. - Use **quantized models** (`large-v3-q5_0` ~1.1GB vs 1.5GB FP16) when RAM-constrained. Accuracy loss is minimal; inference ~same speed or faster on CPU. - **faster-whisper is ~4× faster than whisper.cpp on CPU** (CTranslate2 int8). Use it for long files. whisper.cpp wins on Apple Silicon with Metal for short files due to faster warm-up. - **compute_type** tradeoff: `int8` fast CPU, `int8_float16` good on GPU, `float16` best quality but GPU-only, `float32` reference. - **Smaller models drop punctuation** on complex sentences. Use `large-v3` or a post-processing punctuation restorer (e.g., `deepmultilingualpunctuation`). - **End-of-file hallucinations** are common — whisper trained on YouTube often appends "Thanks for watching", "Subtitles by ..." after real speech ends. Check final cue and drop if it extends past audio duration. - **Hallucinations on silence** are the #1 quality issue. Mitigate: enable VAD (`vad_filter=True` in faster-whisper), pre-trim silence with `ffmpeg-audio-filter` `silenceremove`, or bump `-et` entropy threshold in whisper.cpp. - `chunk_length` default is 30 s — audio longer than that is processed in overlapping windows. Usually fine; long silences between chunks can cause boundary glitches. - **Diarization (who spoke when) is NOT whisper.** `-tdrz` (tinydiarize) is a token hint, not real diarization. For speaker labels, run `pyannote-audio` separately and merge on timestamps. See `references/whisper.md`. - **GPU support:** whisper.cpp has Metal (Apple) + CUDA. faster-whisper has CUDA only (no Metal). On Apple Silicon, whisper.cpp is usually faster. - `-owts` in whisper.cpp writes a **.wts** karaoke-style script, not JSON. For word-level JSON, use faster-whisper `word_timestamps=True`. - SRT timing granularity is ms — don't expect sample-accurate boundaries. Word timestamps can drift ±200 ms from true onsets. ## Examples ### Example 1: English podcast → SRT ```bash uv run .../whisper.py transcribe \ --input episode42.mp3 \ --output-srt episode42.srt \ --model small.en \ --lang en \ --verbose ``` Uses whisper.cpp if available (via `small.en` GGML); falls back to faster-whisper. `small.en` is the sweet-spot for English podcasts on CPU. ### Example 2: Spanish film clip → English captions burned into video ```bash # 1. Translate uv run .../whisper.py translate --input scene.mov --output scene.srt --model large-v3 # 2. Burn into video (720p output) uv run .../whisper.py srt-burn --video scene.mov --srt scene.srt --output scene.dubbed.mp4 ``` ### Example 3: Interview → word-level JSON for search index ```bash uv run .../whisper.py transcribe \ --input interview.wav \ --output-srt interview.srt \ --backend faster \ --model large-v3 \ --word-timestamps # Script also writes interview.words.json when --word-timestamps is set. ``` ### Example 4: Long lecture with lots of silence ```bash # Enable VAD to skip silent stretches; dramatic speedup + fewer hallucinations. uv run .../whisper.py transcribe \ --input lecture.m4a --output-srt lecture.srt \ --backend faster --model medium.en # Script enables vad_filter=True by default for faster-whisper. ``` ## Troubleshooting ### Error: `whisper-cpp: command not found` Cause: whisper.cpp binary not installed. Solution: `brew install whisper-cpp` (macOS) or build from https://github.com/ggerganov/whisper.cpp. Or force faster-whisper: `--backend faster`. ### Error: `ModuleNotFoundError: No module named 'faster_whisper'` Cause: package not installed in active Python. Solution: `pip install faster-whisper` (or `uv pip install faster-whisper`). Alternatively use `--backend cpp`. ### Error: `failed to load model: ggml-base.en.bin` Cause: model file missing. Solution: `bash /opt/homebrew/share/whisper.cpp/models/download-ggml-model.sh base.en`. Pass the full path via `-m` if it's stored elsewhere. ### Transcript contains "Thanks for watching, don't forget to subscribe" Cause: EOF hallucination — model trained on YouTube. Solution: drop the last cue if its start time is past the audio duration. Or pre-trim the trailing silence with `ffmpeg -af silenceremove=stop_periods=1:stop_duration=2:stop_threshold=-40dB`. ### Subtitles drift vs. audio on long file Cause: boundary glitches between 30 s chunks, or variable frame rate source. Solution: remux to CFR first (`ffmpeg -i in.mp4 -vsync cfr out.mp4`), or run the `media-subtitle-sync` skill (alass / ffsubsync) on the generated SRT to re-align. ### Punctuation missing from transcript Cause: small / tiny models produce unpunctuated output. Solution: use `medium` or `large-v3`, or post-process with `deepmultilingualpunctuation` / `punctuators`. faster-whisper's `large-v3` is generally best. ### faster-whisper is slow on my Mac Cause: faster-whisper uses CTranslate2 CPU path on Apple Silicon (no Metal). Solution: switch to whisper.cpp (`--backend cpp`) which has Metal acceleration. Or run faster-whisper on a CUDA box. ### Auto-detected language is wrong Cause: first 30 s of file is music / silence / wrong language. Solution: force `--lang `. See language codes in `references/whisper.md`. --- Source attribution: damionrashford Original project: https://github.com/damionrashford/media-os/blob/5014c1e8b23fd3e18d49926d9aa147d15a3aa08e/skills/media-whisper/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Mega AI Hub. ## Upstream license notice MIT License Copyright (c) 2026 Media OS Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Mega AI Hub resource metadata Creator / publisher: damionrashford Original project: https://github.com/damionrashford/media-os/blob/5014c1e8b23fd3e18d49926d9aa147d15a3aa08e/skills/media-whisper/SKILL.md Attribution basis: Source repository record Fields of work: Content & Media, Education & Research, Marketing Roles: Content Marketer, Copywriter, Podcast Producer, Researcher, Video Producer Specialties: API Tools: FFmpeg, Whisper Country applicability: Country not specified Classification basis: Source catalog labels Instruction SHA-256: 3b5e9cc2248bb1b9d786b3a24261e3126ea296b9be0f1b09c193480cc9012b57