The useful unit of control is not a prompt. It is a versioned chain from premise and ending to beats, scenes, shots, source assets, and the final timeline. Video models generate material inside that chain; editing, compositing, color, sound, and story judgment make it a film.
Start by fixing the dramatic contract
Long-form coherence fails before generation if the story has no destination. I would lock four things first: a premise, a protagonist who wants something concrete, a dramatic question, and an ending that answers it. The ending may change during revision, but every scene must still earn or complicate that answer. This turns “interesting shots” into a system with a direction.
Next comes a beat sheet with a rough tension target. The curve is not a screenplay formula or a score for artistic quality. It is an editorial hypothesis: pressure should rise through complications, change meaning near the midpoint, collapse into a crisis, peak at a decisive choice, then release enough energy for the ending to feel resolved.
Compile story state into shots
A scene graph stores causal structure: which beat a scene advances, what each character knows before and after it, which object states change, and what setup must pay off later. A shot graph then turns a scene into camera-sized work units. Every shot points back to its narrative purpose, so a beautiful render with no function is easy to cut.
The asset bible is the continuity memory shared by every shot: approved character turnarounds, costume and prop states, location geometry, lighting rules, palette, lens language, voice references, and negative constraints. Shot specifications reference immutable asset versions instead of repeatedly describing the same character in prose.
| Artifact | What it controls | Gate before moving on |
|---|---|---|
| Story brief | Premise, dramatic question, ending, theme, audience promise | The ending answers the opening question |
| Beat sheet | Reversals, reveals, decisions, and intended tension | Every beat changes story state |
| Scene graph | Causality, knowledge, object state, setup and payoff | No scene exists only for exposition |
| Shot manifest | Framing, duration, action, continuity inputs, audio intent | Each shot has one editorial job |
| Asset bible | Identity, wardrobe, locations, props, palette, lens rules | References are approved and versioned |
| Timeline + QC log | Picture, sound, color, provenance, defects, approvals | Only failed shots return to generation |
The machine-readable manifest is the handoff between writing, generation, and finishing. A shot record can be small:
{
"shot_id": "s07",
"beat": "the apparent victory becomes a trap",
"duration_s": 4.5,
"continuity_in": ["hero_v03", "red_key_v02", "rainy_hall_v04"],
"camera": {"size": "medium", "move": "slow_push", "lens_mm": 50},
"action": "hero notices the key is already wet",
"generation": {"workflow": "video_i2v_v06.json", "seed": 184221},
"acceptance": ["hand keeps key", "screen direction L→R", "no dialogue"]
}The production graph and its feedback loop
Use ComfyUI as a reproducible renderer
ComfyUI fits the generation layer because its workflows are explicit node graphs and can be stored as JSON. I would version the graph, model hashes, adapter versions, seed, scheduler, conditioning inputs, and output checksums with every candidate. A controller can compile a shot manifest into a workflow job, submit variants, and attach the returned assets to the same shot ID. The model remains replaceable; the production contract does not.
Control inputs should be chosen per failure mode: approved reference frames for identity and wardrobe, masks for local edits, depth or pose when composition matters, first/last frames for transitions, and deterministic plates for text, screens, logos, or props that must be exact. Custom nodes are executable dependencies, so I would pin revisions, review their code, isolate the environment, and keep an allowlist; ComfyUI’s own documentation warns that unverified nodes can compromise a system.
Finish with traditional tools
The first assembly happens before expensive polish. If a beat does not land in a rough cut, a sharper grade will not fix it. DaVinci Resolve owns the editorial timeline, dialogue, sound design, music, and color. Fusion’s node-based 2D/3D tools handle tracking, cleanup, compositing, set extension, titles, and fixes that are more controllable than regenerating a whole shot. FFmpeg provides scriptable conform, frame-rate and color-space checks, filter graphs, proxies, review renders, loudness measurements, and deterministic delivery encodes.
Sound should be authored against the locked cut rather than generated as an afterthought. Dialogue intelligibility, room tone, perspective, transient placement, and moments of silence carry continuity even when images change. The final master keeps stems and a cue sheet so picture and audio revisions remain separable.
Continuity QC is an executable review
Each shot passes four layers. Narrative QC asks whether the intended beat is readable. Continuity QC compares identity, costume, props, lighting, eyelines, screen direction, and character knowledge with neighboring shots. Temporal QC flags warping, object births, unstable hands, and impossible motion. Technical QC checks cadence, gamut, clipping, loudness, captions, and delivery format.
A failure record names the exact frame range, defect class, severity, and upstream variable most likely responsible. The next candidate changes one factor at a time. That makes iteration debuggable and preserves approved work instead of relying on prompt archaeology.
The first build I would ship
The initial target is deliberately small: a 60–90 second film, one protagonist, two locations, one consequential prop, and roughly 12–18 shots. Those are scope targets, not completed results. The deliverable includes the film, story and shot graphs, asset bible, versioned ComfyUI workflows, edit timeline, generation manifest, and QC log.
- Story gate: a viewer can state what the protagonist chose and why the ending follows.
- Continuity gate: every prop and character-state change is either motivated or logged as a defect.
- Control gate: a single failed shot can be regenerated without changing approved neighbors.
- Reproducibility gate: every frame traces to a shot specification, workflow revision, inputs, and seed.
- Production gate: picture, stems, captions, color-managed master, and web encode pass automated checks.
The larger research question is not whether a model can make a striking clip. It is how much story-level intent survives a long sequence of tool handoffs, and how cheaply the system can locate and repair the point where that intent was lost.
Primary tool references
- ComfyUI, Workflow concepts and JSON persistence.
- ComfyUI, programmatic workflow jobs and API format.
- ComfyUI, custom-node installation and security guidance.
- FFmpeg, official filter-graph documentation.
- Blackmagic Design, Fusion’s node-based compositing and 2D/3D toolset.
- OpenAI, Sora technical report and documented temporal/physics limitations (2024).