Discord

Real-time

Bring your own room or voice

Two live-avatar models only render the face and lip-sync the speech you feed them, so they need infrastructure of your own: Higgs Avatar needs your own LiveKit room and voice agent, the Vidu component avatar needs your own real-time channel. The managed avatars (HeyGen LiveAvatar, Tavus, Anam) need nothing: see Live avatars, which is also where to choose between all of them. The session lifecycle, billing and errors are the same as for every live-video model: Live video guide.

Higgs Avatar in your LiveKit room

boson/higgs-avatar-live puts a talking-head avatar into a LiveKit Cloud room you own. The avatar renders the face and lip-sync; you supply the room, the conversation and the voice. No video or audio passes through VideoRouter — we open and meter the avatar session, nothing more.

You need:

  • A LiveKit Cloud project (its wss://<project>.livekit.cloud URL, API key and API secret). Self-hosted LiveKit is not supported. LiveKit bills you for it directly.
  • A voice agent that joins the room and produces the avatar’s speech as 24 kHz, 16-bit, mono PCM — any TTS or speech-to-speech model works. We do not provide one.
  • A front end that joins the room as a viewer and shows the avatar’s video and audio tracks.

Your LiveKit API key and secret never come to VideoRouter. You sign a short-lived token for the avatar with them, on your own server, and send us only that token.

1. Mint the tokens (your server)

Three participants join the room: the avatar, your agent, and your viewer. Each gets its own token. Give the avatar token a lifetime longer than the session (max_duration_seconds plus a few minutes) — if it expires first, the avatar drops out of the room.

# pip install livekit-api
from datetime import timedelta
from livekit import api

LK_KEY, LK_SECRET = "APIxxxx", "xxxx"          # your LiveKit project's credentials - they stay on your server
ROOM, MAX_SECONDS = "avatar-demo", 600

def token(identity, *, publish, subscribe, ttl, sources=None):
    grants = api.VideoGrants(room_join=True, room=ROOM, can_publish=publish, can_subscribe=subscribe,
                             can_publish_data=publish, can_publish_sources=sources)
    return (api.AccessToken(LK_KEY, LK_SECRET).with_identity(identity)
            .with_grants(grants).with_ttl(timedelta(seconds=ttl)).to_jwt())

avatar_token = token("boson-avatar", publish=True, subscribe=False, sources=["camera", "microphone"],
                     ttl=MAX_SECONDS + 300)                       # goes to VideoRouter in step 2
agent_token  = token("agent",  publish=True,  subscribe=True,  ttl=3600)   # stays with your voice agent
viewer_token = token("viewer", publish=False, subscribe=True,  ttl=3600)   # goes to your front end

The avatar token must allow joining the room and publishing camera and microphone tracks; it does not need to subscribe. The avatar and agent identities must differ.

2. Open the session

Send the avatar token and room details in options.session_params.transport. We forward them to Boson once; the avatar is in your room when the call returns. publisher_identity must be the identity your agent joins with — the avatar accepts audio from that participant only.

POST /v1/live-video/sessions
{
  "model": "boson/higgs-avatar-live",
  "max_duration_seconds": 600,
  "options": { "session_params": {
    "avatar_id": "builtin_claire_v1",          // or builtin_emma_v1 / builtin_james_v1
    "transport": {
      "url": "wss://my-project.livekit.cloud",
      "room_name": "avatar-demo",
      "participant_token": "<LiveKit JWT for the avatar identity>",
      "participant_identity": "boson-avatar",   // the avatar
      "publisher_identity": "agent"             // your voice agent (must differ)
    }
  } }
}

The response is the usual session object; connect carries avatar_session_id, avatar_identity, room_name, publisher_identity and expires_at — never your token. Keep calling heartbeat about every 5 s.

3. Send the avatar its speech (your voice agent)

Join the room as agent and stream your speech to the avatar over a LiveKit data stream, using the LiveKit Agents avatar audio output. The avatar publishes the lip-synced video and the audio; your agent must not publish its own audio track.

# pip install livekit-agents
from livekit import rtc
from livekit.agents.voice.avatar import DataStreamAudioOutput

to_avatar = DataStreamAudioOutput(room, destination_identity="boson-avatar", sample_rate=24000)

await to_avatar.capture_frame(rtc.AudioFrame(pcm, 24000, 1, len(pcm) // 2))   # 24 kHz, 16-bit, mono PCM
to_avatar.flush()          # end of this utterance
to_avatar.clear_buffer()   # the user interrupted: drop whatever the avatar has not played yet

Listen for the playback_finished event to know when the avatar has finished speaking. Check LiveKit Agents’ avatar documentation for the exact constructor and event API of the version you use. Boson’s own livekit-plugins-boson-avatar creates the session at Boson with a Boson key, so it does not go through VideoRouter — use the data stream directly as above.

4. Show it (your front end)

Join the room with viewer_token using any LiveKit client SDK and attach the video and audio tracks published by the participant whose identity is boson-avatar.

5. End it

Call POST /v1/live-video/sessions/{id}/end when the conversation is over; we release the avatar and settle the bill. The session also ends by itself at max_duration_seconds or when every other participant has left the room, so a closed browser tab does not keep billing.

Prices are live data, not a doc table: GET /v1/live-video/models returns price_per_second_usd and price_per_hour_usd per row. Session limits are the host’s own and differ by mode.

Billing. The general rule above: whole seconds, the larger of Boson’s own created_at→ended_at and our measured ready→end time, never more than your cap. Boson does not publish its metering rule, so we bill in the safer direction. Rate: $0.07/min at 480x640 (Boson's default; other sizes, including 640x640, are not available in real time). That is the whole VideoRouter charge: LiveKit Cloud is billed to you by LiveKit, under your own project, for the avatar and your voice agent as participants in your room. Capacity. Avatar capacity at Boson is shared and not documented; a full pool returns 429 host_capacity. Only Boson’s builtin avatars (builtin_claire_v1, builtin_emma_v1, builtin_james_v1) can be used.

Offline version. To turn a face image and an audio file (up to 60 s) into a finished video instead, use boson/higgs-avatar on POST /v1/lipsync. It is the same Boson model, billed per second of audio rather than per session second.

Preview. The real-time Higgs Avatar needs your own LiveKit Cloud project and has been exercised against test doubles only; the offline version is verified end to end. Tell us if a session misbehaves.

Component avatar (bring your own RTC)

vidu/vidu-s2-avatar-component is for teams that already run their own real-time channel (Alibaba, Tencent, Agora or Volcano Engine RTC) and their own speech pipeline. You stream the character’s speech and text to us; Vidu renders the character and pushes the video into your channel. Nothing else about the session differs: same hold, same settle, same limits.

  1. Open it with options.session_params: image_uri (required, http(s) or data:image/) and optionally rtc_info — provider (artc, trtc, agora, volcengine) and token are required, app_id, channel_id, user_id optional strings. Without rtc_info Vidu hosts a channel and connect.managed_rtc carries the credentials to join it and watch. Your join token is forwarded to Vidu and never stored in our logs.
  2. Stream to wss://<host>/api/v1/live-video/sessions/{id}/stream?conn_id=<random> (the path is in connect.stream_path). Servers authenticate with Authorization: Bearer; a browser, which cannot set headers, passes a short-lived live-video token as ?token= — a long-lived API key in a URL is refused.
  3. Frames. Text frames are JSON signals, binary frames are raw audio. First send {"type":1,"seq_id":1,"payload":{"conn_init":{"version":1}}} and wait for conn_init_ack.success; on NOT_READY just send it again a couple of seconds later (do not reconnect). Then stream PCM 16-bit signed little-endian, 24 kHz, mono (48 bytes per millisecond; 20 ms or 100 ms chunks suit it), plus type 9 (what the user said) and type 10 (what your model replied) as {"payload":{"text_msg":{"msg_id","content","timestamp"}}}, type 7 to interrupt, type 5 to hang up. We forward every frame verbatim in both directions.
Close codeMeaning
1000Normal: you hung up, the session was ended over HTTP, or the host ended it. The session is settled.
4410Your max_duration_seconds was reached. We hung the host up and settled the session.
4409The session is not open, already has a stream attached, or the host refused the stream. Nothing was settled; you may retry.
4401 / 4403 / 4404 / 4400Bad credentials / missing live_video scope / no such session (or another organization’s) / the model has no stream.
4502We could not reach the host. Nothing was settled; you may retry.
  • Why it is relayed. Vidu’s own data socket admits one connection, so whoever held it could never be cut at the session cap by anyone else. The relay holds that connection for you, which is what makes the cap, the hold and the settle real. Only audio and text cross it; the video goes straight to your RTC channel.
  • One stream per session, and detaching ends it. Closing the stream hangs the host up immediately and settles the session; to continue, open a new session. The stream itself counts as the heartbeat, so you do not call POST .../heartbeat.
  • Billing is the host’s own meter, as for the other Vidu models: 1 credit per second from the host’s acknowledgement of conn_init to the hang-up, with a short warm-up that is not billed.