# Unified asset references — `{image:N}` for Assets-handle connections

**Date:** 2026-06-29
**Status:** ✅ SHIPPED to dev + prod (2026-06-30) — **Phase 1 (video) PR #3692**, **Phase 2 (image) PR #3693**. Reached production via the dev→main deploy PR #3691. See **As-built notes** at the bottom for where the implementation diverged from this design.

## Goal / user story

> When an image is connected to a generate-image / generate-video node via the **Image Refs** handle, the user can reference it in the prompt as `{image:N:label}`, which resolves to `the {label} from @image_N` and points at the N-th reference image. We want the **same behavior for ASSETS** — character / location / object / animal(creature) — connected via the **Assets** handle.

A connected asset must (a) contribute its reference image to `referenceImageUrls`, (b) be referenceable by a positional token that **resolves** (today it silently drops), and (c) be numbered in **one** sequence shared with the plain image refs.

## Decisions (locked)

| # | Decision |
|---|----------|
| D1 | **Unified `{image:N}` numbering.** Assets share ONE reference-image numbering with the Image Refs handle. An asset at position N → `@image_N`. No typed `{character:N}` tokens. |
| D2 | **All four asset types** (character, location, object, animal/creature) become first-class positional refs with guaranteed URL contribution. Characters & locations **keep** their richer `@name` mentions on top — those are just one input to the same unified numbering. |
| D3 | **Order = Image-refs first, then assets.** `referenceImageUrls = [Image-Refs-handle conns (existing order)] + [Assets-handle conns (edge order)]`; **frames stay tail-appended, unnumbered** (`resolveSeedance2Inputs`). |
| D4 | **Spec + phased build.** Phase 1 = generate-video; Phase 2 = generate-image. Each ships with the numbering-invariant + preview↔run parity tests. |
| D5 | **One ordering on ALL surfaces** = image-refs-first (D3), including **flipping #3683's `assembleVideoConnectedReferences`** (today assets-first) to number after the leading flat refs. Converges canvas / orchestrator / API / MCP / SDK / CLI onto one `@image_N`, fixes the orchestrator's latent `@image_1` collision AND #3683's image-route-vs-video-route mismatch. |

## Builds on #3683 (merged to dev 2026-06-29, commit 9365cb24b)

PR #3683 — *"catalog-driven reference coverage + connectedReferences across API/MCP/SDK"* — landed right before this work and is **orthogonal, not conflicting**. It built the **external-caller** half; this feature builds the **editor/canvas** half on top of it. Reuse, don't duplicate:

- **`connectedReferences` is first-class across API / MCP / SDK** (NOT CLI — see Coverage). Routes (`generate-video`/`generate-image`/`text-to-video`) accept `connectedReferences` + `referenceOrder`, validated by `connectedReferenceSchema` (`backend/src/lib/connected-reference-schema.ts`), and **assemble server-side** via the SAME shared core (`assembleVideoConnectedReferences` in `routes/generate-video.ts:166`, `assembleImageInput` for image) → `resolveVideoReferenceCore`. So a direct API/SDK/MCP run already binds inline refs exactly like the canvas.
- **Catalog-driven per-provider reference CAP**: `VIDEO_REF_LIMITS_BY_PROVIDER` + `imageReferenceLimit(provider)` (`model-constants.ts:465,1191`), drift-guarded (`video-ref-limits.test.ts`). Caps are applied **before** numbering so no `@image_(cap+1)` directive points at a dropped slot. **Our unified numbering MUST respect this cap** (flat refs consume budget first, then assets).
- **What #3683 did NOT do:** the editor Assets-handle entities still assemble via the local canvas path (`assembleVideoPrompt`/`execute-node`) and orchestrator `payload-builder` — characters only (no object/creature expander; location is image-only). **That gap is exactly this feature.**

Net effect on the plan: the assembly core, the caps, and the API/MCP/SDK surface already exist. Phase 1 shrinks to **feeding the editor's Assets-handle entities (all 4 types) into that same structured pipeline** on canvas + orchestrator, plus the object/creature/location gaps, plus **CLI parity** (the one surface #3683 skipped).

## Current state (grounded in code)

### The editor↔resolver split
- **Editor** (`connected-references.ts::buildImageConnectedReferences`): `getConnectedSources` returns *all* edges; only `startFrame`/`endFrame` are excluded. So character/location/object/creature wired to **Assets** already flow into the prompt-editor ref list — characters/locations as rich `@slug:N` mentions, object/creature/face fall through to the generic **`{image:N}`** chip bucket (`connected-references.ts:233-244`). Indices are assigned in **edge order**, mixing handles.
- **Resolver** (`@nodaro/shared/node-refs.ts`): `referenceModalityForHandle("assets")` → **`null`** (not in `REFERENCE_HANDLE_MAP`), so `countRefModalityEdges` does **not** count the Assets handle. → A `{image:N}` chip the editor offers for an object/animal **drops** at resolve time (`N > imageRefCount`).

### Numbering: video vs image
- **Video** (`video-reference-resolver.ts::resolveVideoReferenceCore`): numbers asset URLs from **1**, unaware of plain refs. Backend then assembles `referenceImageUrls = [plain refs…, resolver.additionalUrls…]` (`payload-builder.ts:2432-2443` t2v, `:2305-2380` i2v). → directive `@image_1` and body-token `@image_1` can mean **different** images when both plain refs and assets are present (latent bug, rare today). Only `expandWiredCharacterRefs` runs in the video path; **`expandWiredLocationRefs` is image-only**; there is **no** object/creature expansion at all.
- **Image** (`prompt-builder.ts::buildImagePrompt`): already numbers `@image_N` over `[pre-existing referenceImageUrls] + [mentions] + [canonical fallback]` (`:596, :483, :879`) — i.e. **plain refs first, then assets**, with directive ordinals and worker indices designed to agree. **This is the proven pattern Phase 1 mirrors.**

### Editor preview
Preview uses the same resolver core (`assembleVideoPrompt` → `resolveVideoPromptMentions` → `resolveVideoReferenceCore`). Fixing the core fixes preview **and** run together (the preview↔run parity test from #3676 enforces this).

## Design

### Canonical reference order (single source of truth)
Introduce one ordering used by **both** the editor numbering and the backend assembly:

```
referenceImageUrls (excluding frames) =
  [ Image-Refs handle connections, in existing order
    (manual data.referenceImageUrls + connectedRefImageOrder) ]
  ++ [ Assets handle connections, in edge order, each expanded:
       character → @mentioned variant URLs then canonical;
       location  → canonical (+ bucketed variants if @mentioned);
       object    → canonical (auto-attach, no mention machinery);
       animal    → canonical (auto-attach, like object) ]
frames (startFrame/endFrame) appended at the TAIL, unnumbered.
```

`@image_N` (body tokens **and** directive ordinals) numbers this list 1..K. `K` = its length. Editor chip N must equal backend position N.

### Numbering authority = the resolver (mirror the image side)
Extend `resolveVideoReferenceCore` so it is the **single** place numbering happens:
1. Accept the **leading plain image refs** (`leadingRefUrls: string[]`) — the Image-Refs-handle URLs already resolved by the input resolver. The resolver seeds `position` and `merged` with these so every directive ordinal and body `{image:N}` is offset past them.
2. Expand & include **all four** entity types (not just characters): accept pre-expanded `wiredAssetRefs` covering character + location + object + animal (callers build these via `to-connected-references.ts` + new `expandWired{Location,Object,Creature}Refs` reused from the image path).
3. Body-token count = `merged.length` (the assembled total), so tokens up to K resolve and beyond-K still drop to bare label (unchanged contract).
4. `REF_BINDING` swap-point unchanged — still emits `@image_N`.

Backend `payload-builder` video cases (i2v/t2v) stop doing the post-hoc `[plain, ...additionalUrls]` merge; they pass `leadingRefUrls` **in** and use the resolver's single merged output for `referenceImageUrls`.

### Count consistency
`countRefModalityEdges` stays the **edge** counter for plain refs. The **token-resolution** count becomes the resolver's `merged.length` (already how `resolveReferenceTokens` is called — `tokenCounts(merged.length)`), so assets are counted by construction. We do **not** need to hack `referenceModalityForHandle("assets")` → image for counting; instead the assets are real entries in `merged`. (Revisit only if a non-resolver call site needs the asset-inclusive count.)

### Editor numbering parity
`buildImageConnectedReferences` / `connectedReferencesToRefImages` must emit indices in the **canonical order** (D3): Image-Refs-handle entries first, then Assets-handle entries. Extract a shared ordering helper (or sort key) so the editor and backend can't drift. Object/animal entries already become `{image:N}` chips; ensure their N matches the canonical order and they carry a sensible default label (entity name).

## Phase 1 — generate-video (ship first)

Converge canvas + orchestrator on the SAME `resolveVideoReferenceCore` the #3683 routes already use, fed with ALL four asset types + leading refs, **capped by `imageReferenceLimit(provider)`**.

Touchpoints:
1. `packages/shared/src/video-reference-resolver.ts` — add `leadingRefUrls` + multi-entity `wiredAssetRefs`; seed `position`/`merged`; number directives + tokens over the unified list. Respect the per-provider cap (flat/leading refs consume budget first — mirror `prompt-builder.ts:1063-1080` and `assembleVideoConnectedReferences`).
2. `packages/shared/src/to-connected-references.ts` — ensure object/creature/location map to the resolver's expected shape (canonical auto-attach for object/animal).
3. `backend/.../payload-builder.ts` — video i2v/t2v: build leading refs + expand all 4 entity types (add object/creature expanders; wire location into video), pass into resolver, drop the post-hoc merge. Where practical, reuse `assembleVideoConnectedReferences` (already exported from `routes/generate-video.ts:166`) so orchestrator ≡ route.
4. `frontend/.../execute-node.ts` — mirror the same on the single-node run path (it calls `resolveVideoPromptMentions`).
5. `frontend/.../video-prompt-assembly.ts` — preview path inherits the core change; `imageRefCount` becomes/aligns with the unified, capped count.
6. `frontend/.../connected-references.ts` (+ `video-audio-ref-items.ts`) — canonical-order indexing for the editor chips; object/animal default labels; cap the offered chips at `imageReferenceLimit`.
7. Autocomplete (`suggestion-list.tsx`): object/animal show as resolving `{image:N}` chips (already offered; verify label + number).
8. **CLI** (`packages/cli/`) — #3683 skipped it. Add `--connected-references <json|@file>` (+ `--reference-order`) passthrough on the `run`/node commands so CLI reaches parity with the SDK it wraps. (Small; can ship in Phase 1 or as a fast-follow.)

## Coverage (API / MCP / SDK / CLI)

| Surface | Status | Notes |
|---------|--------|-------|
| **API** | ✅ done (#3683) | routes accept `connectedReferences` + `referenceOrder`, assemble via the shared core; verify Assets-entity sources resolve once Phase 1 expands them |
| **MCP** | ✅ done (#3683) | `verbs-image` / `verbs-video` / `video-director` expose `connected_references` + `reference_order` |
| **SDK** | ✅ done (#3683) | `@nodaro/sdk` `StructuredReferenceParams`, typed `run`/`runAndWait` overloads, `ConnectedReference` export |
| **CLI** | ❌ **gap** | not touched by #3683 — add passthrough (touchpoint 8) |
| **Editor / canvas / orchestrator** | ❌ **this feature** | Assets-handle entities (all 4) into the unified, capped `{image:N}` numbering |

## Phase 2 — generate-image (after Phase 1 ships)

The image-side `buildImagePrompt` already numbers plain-refs-first. Phase 2 mostly **verifies** parity and **fills gaps**:
- Confirm object/animal(creature) get expanded + numbered on the image path (characters/locations already do).
- Editor chip numbering parity for image nodes (same shared ordering helper).
- Extend the image preview↔run parity + numbering-invariant tests to assets.

## Non-agreement / conflict rules

How the system behaves when references, surfaces, or numbering *don't* agree. Two kinds:

### A. Design-time non-agreement (two surfaces could compute different results)
The governing rule is **make disagreement structurally impossible, not fixed once**:
1. **Single source of truth.** ONE shared function owns the canonical reference order + `@image_N` numbering; every surface (canvas preview, canvas run, orchestrator, API route, MCP, SDK, CLI) *calls* it — none re-derives. Authorities: `resolveVideoReferenceCore` (order + numbering), `VIDEO_REF_LIMITS_BY_PROVIDER` / `imageReferenceLimit` (caps), `referenceModalityForHandle` (modality). Adding a parallel path is the bug.
2. **Invariant + guard test, not "remember to update."** The agreement is pinned by a test that fails on drift: **editor chip N ≡ orchestrator payload position N ≡ route payload position N ≡ directive ordinal**. The red test is the signal (mirrors the #3676 preview↔run parity + #3683 cap drift-guard). CLAUDE.md: prefer an invariant + guard over a hand-maintained list.
3. **Data/capability-driven, not hardcoded.** Provider ref-support + caps come from the catalog, never a hand-kept enum.
4. **When surfaces ALREADY disagree** (image-route plain-first vs video-route assets-first, today): **converge to the proven convention and delete the divergent path** — never add a third. Here → image-refs-first (D5).

### B. Runtime non-agreement (a reference can't be satisfied) — defined fallbacks
1. **Out-of-range token** (`{image:N}`, N > available refs): drop to the **bare label**; never ship a raw `{image:N}` to the model. (`resolveReferenceTokens` contract.)
2. **Over-cap references** (more than the provider accepts): **leading flat refs consume the budget first**, excess assets dropped **before** numbering, so no `@image_(cap+1)` directive points at a slot the worker never gets. (#3683 cap rule, now applied in the shared core.)
3. **Non-ref-capable provider**: strip ALL `{image:N}` tokens to bare labels, attach nothing (`{video:N}`/`{audio:N}` still resolve against their flat counts).
4. **Determinism / tie-break**: canonical order is deterministic — image-refs in handle order, then assets in edge order — so the same graph always numbers identically (no run-to-run drift).

These rules are the acceptance contract for the numbering-invariant test below.

## Numbering invariant + tests

- **Invariant test (new):** for a graph with P plain refs + assets on the Assets handle, assert: editor chip index for each ref == its 1-based position in the backend-assembled `referenceImageUrls` == the ordinal in its directive/binding. One shared fixture drives editor-side and backend-side.
- **Preview↔run parity (extend `preview-run-parity.test.ts`):** add cases with assets on the Assets handle (character, location, object, animal) — assert `assembleVideoPrompt(...)===` the run's prompt, tokens resolved to `@image_N` at the right N.
- **Resolver unit tests (`video-reference-resolver.test.ts`):** leading refs + each entity type; ordering; dedup; frames at tail unaffected; `{image:N}` beyond K still drops.
- Full backend + frontend vitest before each PR (provider/ref change → run the whole suite, not targeted).

## Risks / edge cases
- **Mention coexistence:** a character both `@mentioned` and auto-canonical must not double-count URLs (existing dedup in `merged` covers this; assert in tests).
- **`referenceOrder` / `connectedRefImageOrder`:** user reorder must apply over the unified list, then renumber (the image side's `(@image_|Image )` renumber regex is the reference).
- **Frames:** must remain tail-appended and unnumbered (Seedance) — leading-ref offset must not shift the frame-suffix ordinals.
- **Providers without `reference-image`:** still strip tokens to bare label (gate unchanged).
- **Drift guard:** one shared ordering helper for editor + backend, covered by the invariant test (CLAUDE.md: invariant + guard over "remember to update").

## Out of scope
- Typed `@character_N` / `@location_N` bindings (rejected — D1).
- New editor token syntax (reuse `{image:N}` + existing `@mention`).
- Re-architecting the image-side builder (it already numbers correctly; Phase 2 only fills gaps).

---

## As-built notes (what actually shipped vs this design)

The design above is accurate in intent; two things evolved during implementation. Recorded here so this doc matches the code.

### 1. `leadingRefUrls` (URL-based) vs `ordinalOffset` (edge-based) — the offset is supplied two ways
The design said the orchestrator/canvas would "pass plain refs as `leadingRefUrls`." In practice the **count must be the EDGE count, not the resolved-URL count**, for FE↔BE parity — the canvas preview has no URL-resolution layer, so it can only edge-count, and the orchestrator must match it (the existing `{image:N}` edge-count contract, `payload-builder-video-mentions.test.ts`). So `resolveVideoReferenceCore` gained **two** ways to specify the leading count:

- **`leadingRefUrls: string[]`** — the core OWNS the URLs: prepends them to `additionalUrls` + offset = their count. Used by the **route** (`assembleVideoConnectedReferences`), where the external caller passes real URLs (no edge layer).
- **`ordinalOffset: number`** — the CALLER owns + merges the leading URLs; the core only offsets the numbering + `{image:N}` count, no prepend. `ordinalOffset = countRefModalityEdges(node, "image")` (EDGE count). Used by the **orchestrator** (i2v/t2v/generate-video) and the **canvas** (execute-node run + video-prompt-assembly preview), which keep their own frame-promotion + merge.

`applyReferenceOrder` also gained an `ordinalOffset` so leading refs keep `@image_1..offset` and only the asset tail renumbers.

### 2. Entity contribution split by path (parity-safe)
- **Video** path: `expandWiredEntityExtraRefs` (BE) + `expandWiredEntityExtrasForVideo` (FE) return `{url, description}` auto-attach extras for location/object/creature, gated by `includeWiredEntities` on ref-capable providers. **RAW canonical image, no location smart-variant**, so preview === run.
- **Image** path: `expandWiredObjectCreatureRefs` (BE) returns `wired-object`/`wired-creature` `ConnectedReference[]`, threaded into `buildConnectedRefsForGenerate`/`FromUrls`. Locations + characters already had image-path expanders; only object/creature were missing on the orchestrator (the FE + `buildImagePrompt` already handled them).

### 3. Editor chip numbering (follow-up #1 — mostly closed)
`connectedReferencesToRefImages` numbers the `{image:N}` chip over **auto-attaching refs only** (plain image + canonical character/location + object/creature/face), **skipping character/location VARIANT entries**, image-refs-first. So an Object/Animal wired alongside a Character (which expands to canonical + K variant rows for the `@`-mention drill) now gets its **true** `@image_N` — the variant rows no longer inflate it. Safe because a variant's `index` is never read for insertion (mentions derive their own N by scanning the prompt); it's display/filter/key only. Guarded by `connected-references-image-n-numbering.test.ts`.

**Remaining (genuinely out of scope — the existing `@mention` mechanism, not this feature):**
- `@kira:N` / `@loc:N` insertion (`computeNextMentionIndex`) is **offset-agnostic** — `maxExistingMentionN + 1`, ignoring preceding image-refs/entities. So a character's *own* mention N doesn't reflect its `@image_N` position when image-refs precede it. Reconciling that means making the `@mention` counter run-order-aware — a broader change to the long-standing mention system (touches all existing workflows), with its own risk, so it's deliberately not bundled here.
- When a SPECIFIC variant is `@`-mentioned, the run attaches that variant instead of the canonical; the editor can't know that without parsing mentions, so a later object's chip-N can still drift by one in that case.

### Coverage delivered
Canvas preview · single-node Run · workflow run (orchestrator) · API · MCP · SDK (all via #3683's `connectedReferences`) · CLI (`--params-file`, documented). Image + video both done.

### Tests (guarding the invariant)
`video-reference-leading-refs.test.ts` (resolver leading/offset) · `generate-video.test.ts` (route D5 ordinal) · `payload-builder-video-mentions.test.ts` (orchestrator object→@image_2) · `preview-run-parity.test.ts` case (j) (object→@image_1, preview≡run) · `payload-builder.test.ts` (`expandWiredObjectCreatureRefs`).