Discord
此页面暂无中文版本 — 以下显示英文内容。 查看英文版

Live video SDKs

JavaScript SDK

An ES-module client for Node 18+ and browsers: zero dependencies, uses the platform fetch, and ships hand-written TypeScript types. Read the SDK overview first for the concepts (broker model, tokens, connect, heartbeat).

Published. Both SDKs are on PyPI and npm as videorouter-live. A live-video model can only be booked once its price is verified (see Live video & world models), and the SDKs surface that as PricingUnavailableError.
npm install videorouter-live

The package is ESM-only ("type": "module"). From CommonJS, load it with await import("videorouter-live").

Quickstart: server → browser

import { LiveVideo } from "videorouter-live";

const lv = new LiveVideo({ apiKey: process.env.VIDEOROUTER_API_KEY });

// Your server, per page load / per user. The token is safe to hand to a browser.
const { token } = await lv.createToken({
  models: ["reactor/happy-oyster-adventure"],
  expiresIn: 300,                                  // seconds
  maxSessions: 1,
  maxSessionDurationSeconds: 60,
});
res.json({ token });                               // e.g. from your /api/live-token route

The browser never sees your API key: it gets a short-lived token. tokenProvider mirrors the apiKeyProvider pattern of realtime SDKs — it is called for the first request, and once more if the server answers 401 (an expired token).

withSession

import { LiveVideo } from "videorouter-live";

const lv = new LiveVideo({ apiKey: process.env.VIDEOROUTER_API_KEY });

// open -> wait until ready -> heartbeat -> your code -> settle (in a finally)
const result = await lv.withSession(
  { model: "decart/lucy-2.5", maxDurationSeconds: 120, options: { speed: "standard" } },
  async (session) => {
    await sendToBrowser(session.connect);      // host credentials - returned once
    return await waitForUserToFinish();
  },
);

Options (third argument): waitUntilReady = true, autoHeartbeat = true, readyTimeout = 300000, pollInterval = 4000, heartbeatInterval (ms). It returns whatever your function returns; the session is always settled in a finally, and a failure to settle never masks your own error.

Manual lifecycle & events

const s = await lv.createSession({
  model: "reactor/happy-oyster-adventure",
  maxDurationSeconds: 60,
});
console.log(s.status);                         // "ready" - connect is available now

s.on("statusChanged", (st) => console.log("->", st));
s.once("ended", (sess) => console.log("settled:", sess.costUsd));
s.on("error", (e) => console.warn("heartbeat problem:", e.code));

await s.waitUntilReady({ timeout: 180_000, pollInterval: 4_000 });   // the poll that sees it ready returns s.connect
s.startHeartbeat();                            // setInterval, unref()'d in Node
use(s.connect);

await s.end({ hostTravelId: "..." });          // optional (Happy Oyster): bill on the host's own duration + fetch the replay

Use the manual flow when the session outlives one function. getSession(id) rebuilds a handle from an id (it never carries connect). waitUntilReady also accepts an AbortSignal.

Model wrappers

One class per live model, so a browser app reads like the model it drives. Each wrapper does the broker half with LiveVideo (open, wait until ready, heartbeat, settle) and the host half with the host’s own browser SDK, which it loads lazily. Install that SDK next to this one — @happy-oyster/js-sdk, @decartai/sdk, @odysseyml/odyssey are all optional peer dependencies — or pass it in as hostSdk.

import { HappyOysterModel } from "videorouter-live";

const happyOyster = new HappyOysterModel({ mode: "adventure", videoElement });
await happyOyster.connect(token);          // a browser token from lv.createToken() on your server

// Describe the world
await happyOyster.createWorld({ prompt: "A sunlit meadow with an old oak." });

// Start streaming
await happyOyster.startTravel();
happyOyster.sendCommand({ translation: "Front" });

await happyOyster.end();                   // stops the stream, then settles the session
import { LucyModel } from "videorouter-live";

const lucy = new LucyModel({ model: "decart/lucy-2.5" });
await lucy.connect(token);

const stream = await navigator.mediaDevices.getUserMedia({ video: true });
await lucy.start(stream, { prompt: "Anime style", onRemoteStream: (s) => (video.srcObject = s) });
lucy.setPrompt("Cyberpunk");               // change the edit on the fly
await lucy.end();
import { OdysseyModel } from "videorouter-live";

const odyssey = new OdysseyModel();
await odyssey.connect(token);
await odyssey.start({ prompt: "A cat walking through a neon city" });
odyssey.interact("make it rain");
await odyssey.end();
import { ViduAvatarModel } from "videorouter-live";

// Needs Alibaba Cloud RTC in the page: `npm i aliyun-rtc-sdk`, or load its CDN script, or pass { hostSdk: { AliRtcEngine } }.
const tina = new ViduAvatarModel({ callMode: "video" });       // or "audio"
await tina.connect(token);
await tina.start({
  persona: "You are Tina, a friendly product guide.",
  imageUri: "https://example.com/tina.png",
  voice: "Tina",
  videoElement: document.getElementById("character"),   // her video
  previewElement: document.getElementById("me"),
});
tina.say("Hi! What can you show me?");                  // she answers in audio and video
tina.interrupt();                                       // cut her off mid-sentence
await tina.end();                                       // hangs up, then settles
import { ViduEditingModel } from "videorouter-live";

const edit = new ViduEditingModel({ editingType: "virtual_tryon" });
await edit.connect(token);
await edit.start({ imageUrl: "https://example.com/jacket.png", videoElement: document.getElementById("result") });
edit.switchPrompt({ imageUrl: "https://example.com/coat.png" });   // no ack on success; each call is a real switch
edit.switchPrompt({ editingType: "background_replacement" });
await edit.end();
import { ViduComponentAvatarModel } from "videorouter-live";
import WebSocket from "ws";                       // server: a WebSocket that accepts headers (an API key never goes in a URL)

const tina = new ViduComponentAvatarModel({ webSocket: WebSocket });
await tina.connect({ apiKey: process.env.VIDEOROUTER_API_KEY });   // in a browser: connect(token) and omit webSocket
await tina.start({
  imageUri: "https://example.com/tina.png",
  rtcInfo: { provider: "agora", app_id, channel_id, user_id, token },   // omit it and Vidu hosts a channel: tina.managedRtc
});
tina.sendAudio(pcmChunk);                // raw PCM, 16-bit signed little-endian, 24 kHz, mono
tina.sendUserText("What time is it?");   // what the user said (your ASR)
tina.sendReplyText("It is noon.");       // what your model answered
tina.interrupt();                        // the user started talking over her
await tina.end();                        // hangs up and settles; closing the stream does the same
Preview. The wrappers are unit-tested against fake host SDKs; their call shapes follow each host’s public docs but have not been run end to end against a live session yet. The Vidu wrappers are the exception: they were run end to end against Vidu’s live API in headless Chrome with a fake camera and microphone — control connection, Alibaba Cloud RTC join, decoded remote video, hangup and billing all worked. Happy Oyster’s directing mode assumes the browser SDK model id happyoyster-1.0-directing — override it with hostModel if the host says otherwise. Vidu wrappers also take webSocket (Node), ackTimeout and retryDelay. Options: hostSdk, maxDurationSeconds, pollInterval, baseUrl.

Nuance Live

Nuance Live (nuance/nuance-live) is a real-time AI persona you video-call over LiveKit. NuanceLiveModel does the broker half (open, heartbeat, settle) and the Nuance half: it joins the room with livekit-client (an optional peer dependency, or pass hostSdk: { Room, RoomEvent }), publishes the microphone, answers the clock-sync pings, joins message.part packets, waits for session.started before it sends anything, and runs the tool round trip. Your page never holds a Nuance credential: the gateway creates and follows the conversation and hands you only the room.

import { NuanceLiveModel } from "videorouter-live";

const persona = new NuanceLiveModel();
await persona.connect(token);                       // a browser token from createToken, as for every model
await persona.start({
  personaId: "sonia",                               // "sam" (default) | "sonia" | "mira"
  systemPrompt: "You are Sonia, the front-desk assistant at Riverside Library. Hand any booking request to your backend.",
  expressiveness: 1, timezone: "Europe/London", endUserId: "user_42",
  delegation: { type: "responses", responses: { model: "default", instructions: "Use find_book, then answer in one sentence.",
    tools: [{ type: "function", name: "find_book", description: "Find a book by title.", parameters: { type: "object", properties: { title: { type: "string" } }, required: ["title"] } }] } },
  runtimeTools: { end_call: true },
  videoElement: document.querySelector("video"), audioElement: document.querySelector("audio"),
});
console.log(persona.warnings);                      // tools the session will run without, and why

persona.onEvent((e) => { if (e.type === "session.input_transcript.done") console.log("user:", e.transcript); });
persona.onToolCall(async (call) => {                // each call arrives once, as soon as the backend makes it
  const out = await findBook(call.arguments.title);
  await persona.sendToolResult(call.callId, out);   // sent with the step's other results once the step is complete
});

await persona.append("thinking", "The user just opened the catalog.", { whileSpeaking: "revise" });
await persona.createResponse("Greet the user.");    // she speaks first once you ask her to
await persona.interrupt();   await persona.holdFloor({ seconds: 30, reason: "The user is reading." });   await persona.releaseFloor();
await persona.end();                                // session.close, leave the room, settle the broker session

Every operation in Nuance's docs is covered:

start({ personaId, systemPrompt, … })Persona, prompt (English only), expressiveness 0–2, timezone, endUserId, greenScreen, delegation (client / responses / none), runtimeTools, camera, raw interfaceSetup.
append(kind, content, opts)thinking, instructions, commentary; whileSpeaking finish / revise; delegationId for client-delegated tasks.
createResponse(instructions?) · sendTyped(text)response.create and typed words (needs a responses delegation).
interrupt() · holdFloor() · releaseFloor()Stop the reply; keep the persona from taking a turn (seconds, endsOnSpeech, reason).
muteInput() · unmuteInput()The persona hears the microphone as silence. setMicrophoneEnabled / setCameraEnabled control your own tracks.
onToolCall() · sendToolResult() · setTools() · update()The tool round trip, tool availability, and session.update.
onEvent(handler) · sendEvent(event)Every raw event in and out; events over 64,000 bytes go as a text stream.
closed · sessionInfo · conversationId · warningssession.closed reason, the session as started, the host conversation id, and the mint warnings.

The session ending on Nuance's side (end_call, expired, a policy or age-check end) or the room dropping settles the broker session for you. Nuance speaks English only and caps each conversation, so maxDurationSeconds is held to what the model allows. The model is bookable once its price is verified; until then it is refused like any unpriced model (PricingUnavailableError).

Decart Lucy and Reactor

These two hosts are metered from the host’s own record of the session, so each needs one extra step on your side. Everything else (opening, heartbeats, settling) is the same as for every model. Details and the reasons: How Decart and Reactor are metered.

Decart Lucy. session.connect is { api_key, base_url, model, max_session_duration_seconds }. api_key is a token for this session only (not a Decart credential) and base_url is VideoRouter’s signaling URL. Decart’s SDK must connect there: pass realtimeBaseUrl. LucyModel does this for you. You pay the seconds Decart reports, and a session that never starts streaming is free. Connect within about 40 s of opening; there is no reconnect (a dropped stream means a new session).

import { createDecartClient, models } from "@decartai/sdk";

const client = createDecartClient({ apiKey: session.connect.api_key, realtimeBaseUrl: session.connect.base_url });
const rt = await client.realtime.connect(camera, { model: models.realtime(session.connect.model), onRemoteStream });
// ... await session.end() stops the Decart stream.

Reactor. session.connect is { jwt, base_url, model, expires_at, max_session_duration_seconds, host_session_path, connect_within_seconds }. Your browser creates the Reactor session itself with the jwt (valid 60 s) and then reports it with reportHostSession. VideoRouter then ends exactly that session when you end the booking, and you pay Reactor’s own record of how long it ran; a booking whose browser never connected is free.

import { LiveVideo } from "videorouter-live";
import { Reactor } from "@reactor-team/js-sdk";

const lv = new LiveVideo({ token });                       // a browser token from createToken on your server
const session = await lv.createSession({ model: "reactor/helios", maxDurationSeconds: 120 });
session.startHeartbeat();                                  // for Reactor this is the only sign that you are alive

const reactor = new Reactor({ modelName: session.connect.model, jwt: session.connect.jwt });
await reactor.connect();                                   // within 60 s of opening
await session.reportHostSession(reactor.getSessionId());   // tell VideoRouter which session this is
// ... steer it with Reactor's commands
await reactor.disconnect();
await session.end();                                       // settles: you pay Reactor's own record
  • reportHostSession(hostSessionId) rejects with a LiveVideoError whose code is invalid_host_session (status 400) if the id is not a session of this booking, or host_session_already_set (409) if another session is attached. Repeating the same id is fine.
  • Without the report we pick the one session of that model created since you opened. If you hold several sessions of one model at once and do not report them, none can be told from another: none is ended early and each is billed its full duration.

Generating videos

The same client wraps the /v1/videos generation API. Params are the API’s own fields, passed through unchanged (see Video generation). It needs an API key — a browser token from createToken is live-video only. Creating a job is never retried, so a flaky network cannot bill you twice.

import { LiveVideo } from "videorouter-live";

const lv = new LiveVideo();                // reads VIDEOROUTER_API_KEY

// Create and wait
const job = await lv.videos.generate({
  model: "heygen-video-1",
  prompt: "A presenter says: 'Welcome to VideoRouter.'",
  duration_secs: 5,
});
console.log(job.url);

// Or step by step
const queued = await lv.videos.create({ model: "wan-3.0-prime", prompt: "A robotic arm in a lab" });
await queued.wait({ timeout: 600_000, pollInterval: 5_000 });   // throws VideoGenerationError / TimeoutError
const again = await lv.videos.get(queued.id);

job.status, job.done, job.failed, job.url / job.urls, job.error and job.raw are available on every job.

Handling errors

import { InsufficientCreditsError, ConcurrencyLimitError, HostCapacityError, PricingUnavailableError, LiveVideoError } from "videorouter-live";

try {
  await lv.createSession({ model: "reactor/happy-oyster-adventure", maxDurationSeconds: 60 });
} catch (e) {
  if (e instanceof InsufficientCreditsError) { /* top up, or book fewer seconds */ }
  else if (e instanceof ConcurrencyLimitError) { /* end a session first; e.retryAfter is set */ }
  else if (e instanceof HostCapacityError) { await sleep((e.retryAfter ?? 5) * 1000); }
  else if (e instanceof PricingUnavailableError) { /* model.available === false */ }
  else if (e instanceof LiveVideoError) { console.log(e.status, e.code, e.message); }
  else throw e;
}

The full class table is in the overview. Every class extends LiveVideoError (itself an Error) with status, code, type, retryAfter (seconds) and body.

Reference

LiveVideo

Constructor optionDescription
apiKeyYour server key. Falls back to process.env.VIDEOROUTER_API_KEY in Node. Never in a browser.
tokenA short-lived token from createToken — browser-safe.
tokenProvider() => string | Promise<string>. Called for the first request and once more on a 401.
baseUrlhttps://videorouter.sh/api/v1 by default.
fetchA custom fetch (tests, proxies, older runtimes).
maxRetriesDefault 2, for reads / heartbeats / end only. Creates are never retried.
timeoutPer-request timeout in ms, default 30000.
MethodReturns / behavior
models()Promise<Model[]>. available is false until the model has a verified price.
createToken({ models, expiresIn, maxSessions, maxSessionDurationSeconds })Promise<Token>. Call from your server. expiresIn is seconds, 1–21,600.
createSession({ model, maxDurationSeconds, options })Promise<LiveSession>. Never retried.
getSession(id)Promise<LiveSession> (no connect).
withSession(params, fn, opts)See above.

LiveSession

MemberDescription
id · getSessionId()The session id (lvs_...).
modelThe model slug.
status · getStatus()"waiting" | "ready" | "ended".
connectHost credentials, or null (waiting, or a re-fetched session).
worldId, billedSeconds, costUsd, endReason, recordingUrl, expiresAtServer fields; the billing fields are set once ended.
rawA copy of the last server response.
refresh()Re-read. For a waiting session this advances it; the call that sees the world built returns connect.
waitUntilReady({ timeout, pollInterval, signal })Throws TimeoutError or SessionStateError (ended while waiting).
heartbeat()One beat. Throws SessionStateError on an ended session.
startHeartbeat(ms?) · stopHeartbeat()Background beat via setInterval (unref()’d in Node so it never keeps a process alive). Default: the server’s 5 s. Stops itself on 404 / 401.
reportHostSession(hostSessionId)Reactor: report the Reactor session your browser created (reactor.getSessionId()). We then end exactly that session with this one and bill Reactor’s own record of it. Throws SessionStateError on an ended session. See Decart Lucy and Reactor.
end({ hostTravelId })Settle. Idempotent. hostTravelId (Happy Oyster) bills on the host’s own duration and fetches the replay.
on / once / offEvents statusChanged, ended, heartbeat, error. Unknown names throw.
getLastError()The last background-heartbeat error.

TypeScript

import { LiveVideo, type LiveSession, type Model, type SessionStatus } from "videorouter-live";

const lv = new LiveVideo({ token });
const models: Model[] = await lv.models();
const bookable = models.filter((m) => m.available && m.kind === "world");

const session: LiveSession = await lv.createSession({ model: bookable[0].id });
session.on("statusChanged", (status: SessionStatus) => {});
// session.on("frame", ...)   // type error: not an event

The exported types are LiveVideoOptions, Model, Token, CreateTokenParams, CreateSessionParams, WithSessionOptions, SessionStatus, EndReason, Connect, SessionEvents and every error class. Event names are checked at compile time.

Testing your code

import { LiveVideo } from "videorouter-live";

const lv = new LiveVideo({
  apiKey: "test",
  baseUrl: "http://test/v1",
  fetch: async () => new Response(JSON.stringify({ object: "list", data: [] }), { status: 200 }),
});
console.assert((await lv.models()).length === 0);