Storyboard Studio · Backend AI layer

AI Orchestration Layer

Three agents on two orchestrators: one plan-and-execute agent with replan for Script and Image (the LLM plans once, code runs and checks, replan only on problems), a single planner loop for Quick Video. Plus the stack, the eval behind the switch, and the pain points found in real use.

1 · Script agent 2 · Image agent 3 · Quick Video agent 4 · Tech stack 5 · Graph vs agent eval & tuning 6 · Pain points
October 2026
Agent 1 · Script — plan-and-execute with replan

Plan once, run in code, replan only when a step is blocked or a draft fails

One chat turn = one background job. Drafts film script, episode outline, episode script, scene prompts and keyframes. The old fixed graph stays as a fallback.

LLM call Code Checks in code Input / output
LLM calls per turn: Q&A 1 · ask back 1 · k tools k + 2 · each fix round +2 · caps: 2 fix rounds, 3 replans, 15 calls, $0.05, 120 s
Agent 2 · Image — same plan-and-execute agent

Same agent, different tools: this agent drafts the visual cards

Separate chat thread per project. Drafts the style, character cards and location cards; images are generated only when the user clicks.

LLM call Code Checks in code Input / output User action Background job
The AI never writes to the board: the user fills the form and saves
Agent 3 · Quick Video — planner orchestrator

No graph: one planner call per turn, then paid jobs only after approval

Chat prompt + a few images → a video under 60 seconds. The LLM picks the step; code checks the rules and holds the money gate.

LLM call Code Decision Input / output Background job
Turn output: reply + one card (ideas, brief or plan)
Tech stack

One Python codebase, two processes: API and worker

Modular monolith split by layer. Every AI call runs in the worker, never inside an HTTP request.

Runtime & web

  • Python 3.12
  • FastAPI + uvicorn
  • Pydantic v2, pydantic-settings

Data

  • PostgreSQL, JSONB for semi-structured data
  • SQLAlchemy 2 async + psycopg 3
  • Alembic migrations

Jobs & events

  • ARQ on Redis: 4 jobs at once, 20 min timeout, no auto-retry
  • redis-py pub/sub for SSE, rate limit

Agent / orchestration

  • langgraph: plan-and-execute agent with replan (fixed graph as fallback)
  • langgraph-checkpoint-postgres chat memory
  • langchain-core interfaces · Langfuse tracing (dev)

Model access

  • httpx REST to OpenAI and Vidu, no SDK
  • Strict structured output from Pydantic schemas
  • Fake or real provider per type

Models (defaults)

  • LLM: OpenAI gpt-6-luna
  • Image: OpenAI gpt-image-2
  • Video: Vidu viduq3-turbo

Security

  • cryptography Fernet: provider keys, decrypted only in worker
  • pyjwt (HS256), argon2-cffi (Argon2id)

Media

  • FFmpeg / ffprobe: frames, joining clips
  • boto3 for S3 (signed URLs), Pillow

Build & quality

  • uv + uv.lock · Docker image python:3.12-slim with dev / prod targets · pytest + pytest-asyncio (fake providers only) · ruff · eval suite: 10 cases, LLM judge, real model on demand
Layer rule: core, utils ← db, schemas ← ai, media ← services ← jobs/handlers, middlewares ← routes (checked by a test)
Fixed graph vs plan-and-execute agent · eval

Plan + replan beats the fixed graph on quality with fewer LLM calls

3 runs per case, gpt-6-luna runs and judges (2026-10-13). #1 right tool call (code) · #2 meets the request · #3 usable by a director (LLM judge).

TotalFixed graphReAct agentPlan + replan+ light reasoning (D)
Runs10 × 310 × 311 × 311 × 3
#1 Right tool call80%73%88%91%
#2 Meets the request60%58%71%67%
#3 Usable by a director58%51%66%70%
LLM calls per run3.22.42.52.6
Tool calls (drafts made)15152325
Cost (assistant only)$0.0443$0.0336$0.0519$0.0512
Planner time p50——10.0 s6.6 s
Hit the cap · errors— · 00 · 00 · 00 · 0

Tuning reasoning per node · 11 cases × 3 runs, gpt-6-luna

  • A no reasoning setting (baseline)
  • B planner, replanner, respond, clarify at low
  • C B + drafting tools at low
  • D B + compact responder (default)
  • E1 only the planner at low (round 2)
A · baselineB · light coordinationC · B + light toolsD · B + compact respondE1 · only planner at low
#1 / #2 / #3 quality91 / 69 / 7088 / 66 / 6491 / 64 / 6991 / 67 / 7097 / 74 / 67
Assistant cost$0.0607$0.0508 (−16%)$0.0474 (−22%)$0.0512 (−16%)$0.0590
Time per turn (median)22.9 s17.1 s (−25%)20.3 s (−11%)20.4 s (−11%)25.3 s
Planner (median per call)10.0 s7.4 s (−27%)8.1 s6.6 s (−34%)—
Source: apps/api/evals/assistant/reports/compare-graph-vs-agent-plan.md (jobs 463, 462, 465, 469) · tuning: reports/tuning-2026-10-13-005.md · D rerun 91 / 63 / 66 vs D 91 / 67 / 70: noise · D1 (shared prompt prefix): no cache gain · E1 is round 2: vs D rerun +9% cost, +29% time
Video model · where it plugs into the agent · design under exploration, not built yet

The video model is chosen between the scene description and the prompt, by the user or by code

USER Configures the model

  • Where: on the board, per scene, or one default per episode (today).
  • What: the API type (T2V, I2V, R2V, V2V) and the model.
  • Checked by: the capability table, so the user cannot pick a type or an image count the model does not support.
  • Wins over auto: a forced choice is never overwritten by the router.
  • V2V is only here: it needs an existing video, so it runs when the user asks to edit a scene (edit_of: scene N) and code makes a new take.

AUTO Code picks the fitting model

  1. No character: composition must match and a background exists → I2V, else T2V.
  2. Character + exact composition: keyframe exists → I2V start-end, else R2V.
  3. Character, continuous with the previous scene: R2V + last frame of the previous scene.
  4. Everything else: R2V.
  5. Model: filter by the capability table, then by the episode quality level; none left → downgrade I2V start-end → I2V → R2V.
Deterministic: the same facts always give the same choice, and the reason is shown on the board so the user can override.
Source: docs/10-model-routing.md §3, §5 (exploring, no proposal yet) · related: PP-012 · most platforms (Vidu, Flow) and every API make the user / caller pick the mode explicitly
Video API types · T2V, I2V, R2V, V2V

Four ways to call a video model: the output is always a video, the inputs differ

Pros and cons are still hypotheses, to be tested by a dev on the web UI before any of it becomes a routing rule.

TypeInputProsConsIn the app today
T2Vtext to videoPrompt onlyNeeds only a prompt: no image to prepare firstNothing anchors the face or the compositionNot used
I2Vimage to video · start / start-endPrompt + start image (+ end image)Composition and setting match the start (and end) image; Keyframe mode already has these framesNeeds the image to exist first; the prompt must not contradict itOnly sheet_cut with first frame (img2video)
R2Vreference to videoPrompt + subject images (characters, background)Keeps a character's face across many scenes, from the character card imagesDoes not lock the composition; needs @Name and @background in the promptDefault for almost every scene (reference2video)
V2Vvideo to videoPrompt + source videoEdits or restyles an existing video without redoing the whole sceneNeeds a source video, so it cannot be chosen at planning timeNot used (planned for the post-production tab)
Two shared currencies: image and videoEach carries a role: start_frame, end_frame, subject:<name>, background, source_video. A last frame of T2V can feed I2V; an I2V output can be an R2V subject.
Prompt is written after the type is chosenEach type needs a different prompt style. Changing type or model after a prompt exists marks it stale and the agent rewrites it.
Open questionsDoes one Vidu call accept a start image and subjects together? How do T2V / I2V compare with R2V on the same scene? Who picks the model: user, a fixed quality level, or both?
Source: docs/10-model-routing.md §1, §2, §4, §9 · code: services/video.py::to_vidu_request, vidu_endpoint
Pain points

Found while making real films: the flow works, the output does not always

Pain pointSeverityFix
ConsistencyPP-005 · 006 · 013Context is lost between scenes
HighTry I2V (start frame), start/end frame, V2V / extend
ConsistencyPP-004Real people cannot be used as references
HighWarn about real faces at the Cast step, before any money is spent
LanguagePP-001Dialogue comes out in Chinese
HighAmbient audio from Vidu, voice from our own subtitles / TTS
LanguagePP-011Vietnamese prompts may make Vidu skip details
LowCompare VI vs EN prompts, keep the better one as default
Also logged: cost and preview before paying (PP-003, 007, 012) · speed and editing (PP-008, 002)
13 logged · 11 open · 2 resolved
1 / 10