# Provisional segments & cost-aware generation

Generating real media (especially cloud video/TTS) costs money and time, so the editor works with
**provisional segments first** and generates the real thing **late** — at final render, or on request per
segment. Lengths start as estimates and snap to the real duration once generated.

## The lifecycle of a segment

1. **Provisional.** A clip/cue exists before its media does:
   - An **un-voiced line** (cue with no `clip`) gets its length **estimated from the text**
     (`estimateSpeechFrames`, ~150 wpm) — `add_cue` fills `to` and sets `estimated: true`.
   - A **shot compiled from a storyboard** becomes a **pending video** asset (`status: "pending"`) whose
     still is a **draft `thumb`**; the clip is `estimated: true` with the shot's intended duration.
   The editor shows these as **drafts** — the thumb at reduced opacity, a "draft" badge, a dashed border,
   and a "~" before the length.
2. **Generate — when, not just what.** Match the model to the moment:
   - **Local models** (Wan/LTX images, local TTS/SFX) are cheap/offline → fine to generate **live / eagerly**.
     Images are fast; **local video is slow**, so even locally prefer a still `thumb` draft until asked.
   - **Cloud models** (fal, xAI, ElevenLabs, …) cost money → **defer**: generate at **final render**, or when
     the operator clicks **generate** on a segment (the ⚡ button → asks the agent to produce just that one).
3. **Snap to real.** After generating a segment's media, call **`clip_fit { projectPath, clipId }`** — it
   measures the file's real duration (ffmpeg) and applies `resize_clip { durationFrames, ripple: true }`
   for you, so the clip takes its true length, `estimated` clears, and later clips on the track **ripple**
   to stay gap-free. (You can still call `resize_clip` directly if you already know the length.) Linked cues
   follow automatically (reconcileCues). For a voiced line, set the cue's `clip` so it snaps to the audio.

## Rules of thumb for the agent

- Don't generate cloud media just to preview — leave it provisional (thumb + estimate) until render or a
  generate request.
- Local video is the one exception to "local = eager": still-thumb it, generate the clip on request/render.
- Always `clip_fit` (which measures + `resize_clip`s with ripple) after generating, so the provisional
  length becomes the real one and the sequence stays tight — no need to compute the duration yourself.
- A **final render** = generate every still-pending segment (respecting the operator's model choice), then
  snap all lengths with a single `clip_fit` sweep (call it with **no `clipId`** — it measures every clip
  whose media is on disk and ripples them to their real length), then run `render_project`.

## Model duration limits & expansion

Video models have a **max clip length** and often a fixed/quantized native length. Run `00 media caps`
for each model's `maxSeconds`, whether it's `variable`, and `firstFrame`/`lastFrame` support, plus
`installed` + an `installHint`. Rules:

- **Never expand a segment past the model's `maxSeconds`** — clamp (helper `clampExpand`). One generation
  can't exceed it; going longer means chaining extensions (see below).
- **Too-short request → generate the next-larger size, then trim in playback.** `videoGenPlan(requested,
  model)` returns `generateSeconds` (≥ requested, ≤ max) and `trim: true`. Generate that clip and set the
  clip's `trimIn`/`trimOut` (or `from`/`to`) to the requested window — don't re-encode; it's an offset, so
  you can later move the in/out points to pick a different part of the generated footage.
- **Extend an existing clip** — before or after its ends:
  - **After** needs image-to-video (`firstFrame`): extract the clip's **last** frame, generate a
    continuation from it (i2v), place it after, ripple. (All our video models support this.)
  - **Before** needs end-frame conditioning (`lastFrame`): generate a clip that **ends** on the clip's
    **first** frame. Several start/end-frame models in the store do this — **Kling O1/2.1/2.5/2.6/3.0,
    Seedance 1/1.5 Pro/2.0/2.0 Mini, PixVerse C1/V6/5, Wan 2.7, Google Veo 3.1, LTX 2.3, Hailuo 02** —
    reachable via `FAL_KEY` (Veo via `GEMINI_API_KEY`). When none is installed the editor shows the
    "expand before" option **dimmed with a ⚙ install hint**. Each model advertises its `maxSeconds`
    (e.g. LTX 2.3 → 20s, Kling 3.0 / Seedance / PixVerse → 15s), `audio`, `resolution`, and `reference`/
    `multiShot` support in the capability snapshot.

## Show the option even when the model isn't installed

For every modality (video, image, music, sfx, speech, voice), surface the generation/expand affordance
**even if no model is ready** — rendered at reduced opacity with a **guided install** (⚙): the capability
snapshot's `installHint` says whether to install a local model (Settings → media models) or add a provider
API key. Clicking it asks the agent to set one up, then perform the action. This way the operator always
sees what's *possible*, not just what's currently installed.

## Cost: warn before you spend (every modality)

Cloud generation costs money. **Before triggering any cloud generative action — video, image, audio/
music/SFX, or speech — state the estimated cost and get a go-ahead.** Local models are free (compute
only); say so and proceed. Never silently run a paid generation.

- **Where prices live.** Each model carries its price in the capability metadata (centralized in R2):
  `pricePerSecond` (video / music / sfx), `pricePerImage` (image), `pricePerKChar` (speech), with a
  `priceNote` when it's a ballpark. Prices are **approximate** — present them as estimates, not quotes.
- **Per action.** Video ≈ `seconds × pricePerSecond`; image ≈ `pricePerImage` per image; speech ≈
  `chars/1000 × pricePerKChar`. The `image_generate` / `video_generate` / `generate_speech` tools also
  append the estimated cost to their result, so the spend is on the record after the fact too.
- **Per render (the whole video).** Before a full render, sum the cost of everything still pending and
  show the total. `00 media cost --project <project> [--speech <ttsModelId>]` returns exactly this — a
  per-item breakdown + `total`, with `unknownCount` for models that have no price on record.
  The editor shows the same as a **"render ≈ $X"** chip. Report the total (and flag any unknowns) and
  confirm before generating.
- **Cheaper first.** When cost matters, prefer a local model (free) or a cheaper cloud model, and only
  generate cloud media on request / at render — not just to preview (see the provisional-segment flow
  above).

## Where the model metadata lives (centralized)

Capabilities are **centralized in the remote R2 catalog** (`dl.0-0.chat/catalogs/media-models`), under a
`caps` map keyed by model id — durations, `firstFrame`/`lastFrame`, `audio`, `resolution`, `reference`,
`multiShot`, `pricePerSecond`, and (for cloud/key models declared only there) `modality` + `install`. It's
validated field-by-field and can be **updated without an app release**. The engine overlays it on the
built-in specs (`mergedModelSpecs`), so R2 is the source of truth: fixing a max duration or adding a new
model is a bucket edit. The merged metadata surfaces in **two** places — the editor's expand/duration logic
(`/api/agents/:id/media-capabilities`, with per-agent installed status) and the **media-settings model
store** (`/api/media-capabilities`, intrinsic), where each model shows its capability chips.

## Editing images: inpaint & outpaint

`image_generate` edits an existing image, not just generates one. Two spatial edits sit on top of the
plain "edit with a prompt" path:

- **Inpaint** — regenerate just a rectangle, keep the rest. Pass `maskRect: {x, y, w, h}` (source
  pixels, origin top-left) alongside `imagePaths: ["<one image>"]` and a prompt describing what goes
  there. The engine synthesizes a mask (transparent = the region to redraw) and calls the model's edit
  endpoint. Example: `image_generate({ imagePaths: ["files/hero.png"], maskRect: {x:120,y:60,w:400,h:300}, prompt: "a calm sunset sky" })`.
- **Outpaint / expand ("zoom out")** — grow the canvas and paint new content into the new border. Pass
  `outpaint: {left, right, top, bottom}` (pixels to add per side) with one `imagePaths` entry. The engine
  pads the canvas transparently (ffmpeg) and masks the border for the model to fill.

Rules:
- Exactly **one** input image, and a model that advertises the capability. The masked path routes to the
  OpenAI **GPT Image** family (needs `OPENAI_API_KEY`) — **gpt-image-2** (current, preferred), gpt-image-1.5,
  gpt-image-1-mini, or the legacy gpt-image-1 (deprecates Oct 2026); pass `model` to pick one. The
  capability snapshot marks each image model's `edit` / `inpaint` / `outpaint` support. Pass either
  `maskRect` **or** `outpaint`, never both.
- The result is saved as a new PNG in `files/`. To replace an in-use asset, generate then point the
  clip/asset `src` at the new file (or overwrite and let the passive reload pick it up).

### From the UI (no coordinates by hand)
The operator picks the region for you, so you receive an explicit call:
- **Standalone image preview** — an **Edit** toggle turns on a canvas. **Drag** a rectangle → an inpaint
  request; **scroll/zoom out** past fit → an outpaint request sized to the zoom-out. The picked region
  rides the composer with the exact `maskRect`/`outpaint`, and the operator types what to paint.
- **Video editor** — selecting an **image clip** shows **✎ edit image**, which opens the same surface on
  that clip's still; the picked edit arrives as an `edit-image` instruction naming the asset's path.

In both cases the instruction already contains the precise `image_generate` arguments — just run it (add
the operator's prompt), then snap the asset/clip to the new file.

## Schema touchpoints
- `asset.status: "pending" | "ready"`, `asset.thumb` (draft image), `asset.durationFrames` (measured real length).
- `clip.estimated`, `cue.estimated` — length is a guess (provisional).
- `project.speech.wpm` — override the estimate rate.