Video Editing

---
name: video-editing-944e2cdf14a3
description: >-
  Non-destructive video editing in Clips — the editsJson model, trim / split /
  cut / speed / blur, transcript-based editing, and ffmpeg.wasm export. Use
  when building the editor UI, adding a new edit operation, or wiring the
  export pipeline.
---

# Video Editing

## When to use

Reach for this skill any time you modify how recordings are edited: new edit operations, the timeline UI, transcript-driven cuts ("remove the ums"), the preview overlay, or the export flow. Editing **must** be non-destructive — the recording blob is immutable after upload.

## Data model touched

- **`recordings.edits_json`** — a JSON column with the edit document. Shape:

  ```json
  {
    "version": 1,
    "trims": [
      { "startMs": 0, "endMs": 3200, "excluded": true },
      { "startMs": 12000, "endMs": 12000, "excluded": false }
    ],
    "blurs": [],
    "thumbnail": null
  }
  ```

- **`recordings.chapters_json`** — JSON array of `{ startMs, title }`.
- **`application_state.editor-draft`** — in-progress editor state the user is previewing (recording id, playhead, preview playback speed, zoom, and edits JSON). Persisted edit operations still write directly to `edits_json` through actions.

## Rules

1. **Non-destructive.** Never re-encode on edit. The original webm/mp4 stays intact at `recordings.video_url`. Edits only change the JSON.
2. **Single source of truth.** The player renders edits at playback time — read `edits_json`, compute the virtual timeline, and skip excluded ranges via `HTMLVideoElement.currentTime` seeks. Do not fork the edit model for the editor vs the player.
3. **Export is explicit.** The user must click Export to render a new file. MP4 export runs in the browser with ffmpeg.wasm; long recordings get a warning and an option to download the original or continue exporting.
4. **Use edit actions.** Agents must use the recording actions below instead of writing `edits_json` directly.

## Timestamp-based editing actions

Timestamp arguments are integer milliseconds on the original video timeline,
even when earlier edits have shortened playback. Convert clock timestamps to
milliseconds before calling an action; for example, `1:23` is `83000` ms.

| Action | Required arguments | Effect |
| --- | --- | --- |
| `trim-recording` | `recordingId`, `startMs`, `endMs` | Excludes the range from playback. `endMs` must be greater than `startMs`; overlapping or adjacent excluded ranges are merged. |
| `split-recording` | `recordingId`, `atMs` | Adds a zero-width split marker for the editor UI. It does not physically split or rewrite the video and does not change playback. |

## Transcript-based editing

Users love editing by text — click a filler word in the transcript, remove it. The editor uses `recording_transcripts.segments_json` to map words to `{ startMs, endMs }` ranges; clicking a segment creates a `cut` edit for that range.

"Remove the filler words" is **not** solved inline — it delegates to the agent. See the `ai-video-tools` skill. The agent analyzes the transcript, proposes a list of cuts, writes them to `editor-draft` for review, and the user one-click approves.

## Export

The editor toolbar calls `exportMp4()` from `app/lib/ffmpeg-export.ts` in the
browser. It loads ffmpeg.wasm on the first export, processes the source video
and kept ranges, then downloads the resulting MP4. There is no server-side
export queue or automatic fallback for long recordings.

When the edited duration exceeds `LONG_EXPORT_THRESHOLD_MS` (10 minutes), the
toolbar warns that browser export may run out of memory and offers **Download
original** or **Export anyway**. Choosing **Export anyway** runs the same
browser export path. Export errors are surfaced to the user; do not report a
failed export as queued or continuing elsewhere.

## Player integration

The player is the canonical renderer for edits. Given `edits_json`, compute a `VirtualTimeline`:

```ts
interface VirtualTimeline {
  // Maps virtual ms -> source ms, skipping cuts/trims.
  toSource(virtualMs: number): number;
  // Total virtual duration after edits applied.
  durationMs: number;
  // Ranges to skip on playback: the player listens for timeupdate and seeks past them.
  excludedRanges: { startMs: number; endMs: number }[];
}
```

Never call `video.currentTime` with a raw segment index — always go through `toSource`.

## Rules

- The edit UI writes to `editor-draft` on every change, and only writes to `edits_json` on Save (so Cmd+Z is cheap and the DB stays clean).
- Never mutate `edits_json` from `db-exec`; use the recording actions above. Use `clear-edits` to reset edits.
- Speed edits compound (`1.5 × 2 = 3x`) — validate the result is in `[0.25, 4]`.
- Blur coordinates are in **source resolution**, not display pixels. Always normalize against `recordings.width` / `recordings.height`.

## Related skills

- `ai-video-tools` — filler-word removal and chapter generation propose edits via the agent.
- `real-time-sync` — when the agent writes `edits_json`, the player must reflect it; `edits_json` is part of the `recordings` row which is already on the sync list.
- `video-sharing` — exports honor the share's `enableDownloads` flag.
- `storing-data` — why edits live in a JSON column rather than a separate table.

---

## Editor implementation reference

### Concrete `editsJson` shape (editor team)

The editor writes and reads this exact shape. Any new edit operation MUST
preserve this shape so the player, editor, and export pipeline stay in sync.

```ts
interface EditsJson {
  version: 1;
  // Ripple-style trim ranges. `excluded:true` ranges are skipped during
  // playback and collapsed during export. `excluded:false` entries where
  // `startMs === endMs` are SPLIT MARKERS used by the editor UI — they
  // never affect playback.
  trims: Array<{ startMs: number; endMs: number; excluded: boolean }>;
  blurs: Array<{
    id: string;
    startMs: number;
    endMs: number;
    x: number; y: number; w: number; h: number; // normalized 0..1
    intensity: number;
  }>;
  thumbnail?:
    | { kind: "url"; value: string /* absolute URL */ }
    | { kind: "frame"; value: string /* timeMs as string */ }
    | { kind: "gif"; value: string /* JSON: { url, startMs, durationMs } */ }
    | null;
  // Provenance for stitched recordings — set by `stitch-recordings`.
  stitchedFrom?: string[];
}
```

Comment `videoTimestampMs`, reaction `videoTimestampMs`, and
`recording_transcripts.segmentsJson` timestamps all refer to **original**
video time. The player converts to edited time; never assume the two are
equal.

### Timestamp helpers

`app/lib/timestamp-mapping.ts` is the single source of truth:

| Helper                                 | Returns                                                                 |
| -------------------------------------- | ----------------------------------------------------------------------- |
| `parseEdits(raw)`                      | A fully-populated `EditsJson` from the DB column (empty on bad input)   |
| `serializeEdits(edits)`                | Stringified JSON ready to write back                                    |
| `getExcludedRanges(edits)`             | Sorted, non-overlapping excluded ranges                                 |
| `getKeptRanges(durationMs, edits)`     | Ordered "kept" ranges in original time — the export pipeline iterates   |
| `originalToEdited(ms, edits)`          | Maps original → edited (playback) timeline                              |
| `editedToOriginal(ms, edits)`          | Maps edited → original (used when seeking the underlying `<video>`)     |
| `effectiveDuration(durationMs, edits)` | Edited duration after excluded ranges are removed                       |
| `isExcluded(ms, edits)`                | True if an original ms falls inside an excluded range                   |
| `mergeExcluded(edits, startMs, endMs)` | Append an excluded range; merges overlapping/adjacent entries           |
| `popLastExcluded(edits)`               | Remove the most-recently-added excluded range (used by `undo-edit`)     |
| `appendSplit(edits, atMs)`             | Append a zero-width split marker                                        |
| `formatMs(ms)`                         | `0:42`, `1:23:04` formatting for timestamps                             |

The player team may have its own copy of some of these. If so, consolidate on
`app/lib/timestamp-mapping.ts` — never let the two drift.

### Editor actions

| Action               | Writes                                   | Purpose                                           |
| -------------------- | ---------------------------------------- | ------------------------------------------------- |
| `set-thumbnail`      | `thumbnailUrl` / `animatedThumbnailUrl` / `editsJson.thumbnail` | Three modes: upload / frame / gif |
| `set-chapters`       | `chaptersJson`                           | Overwrites the chapter array                      |
| `stitch-recordings`  | new `recordings` row                     | Client-side ffmpeg concat + upload + insert       |
| `undo-edit`          | `editsJson.trims`                        | Pop the last excluded range (no redo)             |
| `clear-edits`        | `editsJson`                              | Reset to defaults (chapters/thumbnailUrl kept)    |

Every mutation ends with `writeAppState("refresh-signal", { ts: Date.now() })`.

Chapter writes go through `actions/lib/save-chapters.ts`, which only writes
over the list it was checked against: `set-chapters` needs `expectedChapters`
(as `get-recording-player-data` returns them) or `expectedVersion`, and always
needs `expectedMediaUpdatedAt` from that same response. Pass `expectedCuts`
when times were mapped through cuts. `update-recording --chaptersJson` needs
`expectedChapters` and `expectedMediaUpdatedAt`. A new writer of
`chaptersJson` must check both values, or match the row it read in its update's
WHERE, as `apply-rewind-extension` does. Stored chapters that can't be read
refuse every save (`chapters_unreadable`); `set-chapters --discardUnreadable`
replaces them, and only when the user asked to.

### ffmpeg.wasm usage

`app/lib/ffmpeg-export.ts` lazy-loads `@ffmpeg/ffmpeg` (only fetched on first
Export click; core wasm is ~30MB). Three entry points:

```ts
exportMp4(recording, edits, onProgress)  // kept-range concat → H.264+AAC MP4
exportGif(recording, startMs, durationMs, onProgress)  // animated thumbnail
exportConcat(sources, onProgress)  // stitching N recordings into one MP4
```

**Assumed limits (tested in practice — update if you hit new ceilings):**

- Single-threaded WASM, ~2GB memory ceiling per tab.
- Roughly 10 minutes of 1080p WebM→MP4 is the practical upper bound before
  tabs run out of memory. The editor surfaces a confirm dialog when
  `effectiveDuration(...) > LONG_EXPORT_THRESHOLD_MS` (10 min) and offers
  "Download original" as an escape hatch.
- Both the trim/concat paths and the `exportConcat` (for stitching) re-encode
  to H.264+AAC so the output plays everywhere.

### Stitching

Decision: **client-side ffmpeg concat** rather than a virtual playlist that
plays N sources sequentially. Rationale:

- A real combined MP4 plays in share links, embeds, iframes, and any mobile
  browser without special-case player logic.
- The virtual-playlist approach couples the player to the stitched structure
  and breaks sharing / thumbnails / comments (which are keyed off
  `recording_id`).

Flow from the UI:

1. `stitch-manager.tsx` collects the ordered list of source recordings.
2. `exportConcat()` fetches each `videoUrl` and concatenates with ffmpeg.wasm.
3. We upload the resulting blob via `uploadFile()` (or fall back to a data
   URL when no provider is configured — dev mode).
4. We call `stitch-recordings` with the uploaded URL + total duration; the
   action inserts a new `recordings` row with `status: "ready"` and
   `editsJson.stitchedFrom` set to the source IDs for provenance.

### Waveform peak caching

`computePeaks()` decodes the video's audio track via the Web Audio API and
downsamples to a 2000-point peaks array. Peaks are cached in
`application_state` under `waveform-<recordingId>` so remounts don't
recompute. The editor reads the cache first and falls back to computing only
when the key is missing or corrupted.

### Timeline filmstrip

The trim track shows video frames behind the waveform. Two paths, and the order
matters:

1. **Sprite (preferred).** `generate-filmstrip` runs one ffmpeg pass
   (`fps` → `scale` → `pad` → `tile`) into a single JPEG grid, uploads it, and
   stores the URL plus grid geometry on the recording. The editor renders cells
   with CSS `background-position` — one cached image request, no video decoding
   in the browser.
2. **Browser extraction (fallback).** `extractFilmstripThumbnails()` seeks a
   detached `<video>` once per frame. Only used when there is no sprite, which
   means hosts without ffmpeg and local/dev media the server cannot fetch.

Three traps this design exists to avoid:

- **Pass the proxied URL.** Frame extraction must use `getWaveformMediaUrl()`,
  never `recording.videoUrl` — reading pixels back out of a cross-origin video
  taints the canvas and `toDataURL` throws, so provider media silently yields no
  filmstrip at all.
- **Both paths sample cell midpoints,** not `0 … duration` endpoints. The strip
  renders N equal cells, so cell `i` must show the middle of the slot it
  occupies or every thumbnail sits up to half a cell from the time beneath it.
- **Cell count comes from the geometry,** `trackWidth / (height × aspect)`, not
  from the sprite's frame count. Rendering all 40 frames across an unzoomed
  track makes each cell portrait, and `object-cover` then shows a narrow centre
  slice of each frame instead of a recognisable thumbnail.

The filmstrip never replaces the waveform — peaks still draw on top of a scrim,
because frames say nothing about the audio.

### Keyboard shortcuts (editor scope)

- **Space** — play / pause (even while focused in the editor area)
- **Cmd/Ctrl+Z** — undo the last trim (no redo stack)
- **I / O** — mark in / out of the current trim selection
- **Hold R and drag on the timeline** — select an exact range; use Cut
  Selection or Delete to remove it. A normal click still selects and seeks to a
  section, and dragging a section edge still changes that edge.
- **S** — split at playhead (via `split-button.tsx`)
- **Delete / Backspace** — while the transcript editor has a selection: trim
  the selected text's timestamp range

### File map

```
actions/
  trim-recording.ts   split-recording.ts   set-thumbnail.ts
  set-chapters.ts     stitch-recordings.ts
  undo-edit.ts        clear-edits.ts
app/lib/
  timestamp-mapping.ts   waveform-peaks.ts   ffmpeg-export.ts
app/components/editor/
  editor-layout.tsx       editor-toolbar.tsx
  waveform.tsx            trim-handles.tsx      timeline.tsx
  transcript-editor.tsx   chapters-editor.tsx
  thumbnail-picker.tsx    stitch-manager.tsx    split-button.tsx
```

When the recording route enters edit mode, render `<EditorLayout recordingId={id} />` — it wires the toolbar, video preview, transcript editor, waveform, trim handles, timeline, and chapters sidebar. The toolbar includes a Loom-style preview-speed dropdown next to the playhead time; it changes `video.playbackRate` for trimming/review only and writes `playbackSpeed` into `editor-draft` so the agent can see how the user is previewing. The dialogs for thumbnail picking and stitching are mounted inside the layout and toggled by the toolbar.