> ## Documentation Index
> Fetch the complete documentation index at: https://spatialwalk-6179c111.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# RTC Mode & Framework Plugin

> Use LiveKit as the transport layer for SpatialReal avatars

## What is RTC Mode?

RTC Mode enables **real-time voice communication** with avatars through LiveKit. It builds on top of [Host Mode](https://spatialwalk-6179c111.mintlify.site/guide/host-mode.md) — the `@spatialwalk/avatarkit-rtc` adapter package handles the RTC connection and feeds audio/animation data to the SDK automatically.

```mermaid theme={null}
flowchart LR
    A["🎤 Microphone"] --> B["AvatarPlayer<br/>(RTC Adapter)"]
    B <-->|WebRTC| C["RTC Server<br/>(LiveKit)"]
    B --> D["AvatarKit SDK"]
    D --> E["🖥️ Avatar Rendering"]
```

> [!NOTE]
>
> **Web Only:** RTC Mode is currently available for Web applications only. iOS and Android support is planned for a future release. In the meantime, native mobile platforms can use [Host Mode](https://spatialwalk-6179c111.mintlify.site/guide/host-mode.md) with your own RTC implementation.

> **[AvatarKit Voice Agent Demo](https://github.com/spatialwalk/avatarkit-voice-agent-demo)**
>
> A full reference repository with implementation details, including different frontend UI options and multiple backend agent patterns.

## When to Use

* **Real-time voice conversation** — users talk to an avatar via microphone
* **Low-latency interaction** — WebRTC provides sub-second latency
* **Server-side AI** — your RTC server processes audio and generates responses

## Packages Required

| Package | Purpose | Required |
| - | - | - |
| `@spatialwalk/avatarkit` | Avatar rendering SDK | Yes |
| `@spatialwalk/avatarkit-rtc` | RTC adapter | Yes |
| `livekit-client@2.16.1` | LiveKit RTC SDK | Yes |

> [!WARNING]
>
> **Critical:** We are only compatable with `livekit-client` under version `2.17`, since `2.17` introduced single PC by default. We are working on adding support for `2.17`+ in a future release, but for now please ensure you install version `2.16.1` to avoid compatibility issues.

## LiveKit Browser Compatibility

| Browser | Minimum Version |
| - | - |
| **Chrome** | 94+ |
| **Firefox** | 117+ |
| **Safari** | 15.4+ |
| **Edge** | 94+ |

## How It Works

RTC Mode uses the SDK in **Host Mode** internally. The `AvatarPlayer` acts as a bridge:

1. Initializes the avatar SDK with `DrivingServiceMode.host`
2. Connects to the RTC server via the chosen provider
3. Publishes your microphone audio to the RTC server
4. Receives animation and audio data from the RTC server
5. Feeds **animation data** into the avatar SDK for rendering; **audio is played through the native WebRTC audio track** by the RTC provider

You don't need to call `yieldAudioData()` or `yieldFramesData()` manually — the adapter handles this.

### What RTC Mode Does NOT Use

Although RTC Mode builds on Host Mode internally, the following SDK/Host Mode features are **not used** in RTC Mode:

| Feature | Used in SDK/Host Mode | Used in RTC Mode |
| - | - | - |
| `initializeAudioContext()` | Yes — required for Web Audio API playback | **No** — audio is played via WebRTC tracks |
| Internal audio player | Yes — SDK decodes and plays audio internally | **No** — RTC provider handles audio playback |
| `yieldAudioData()` / `yieldFramesData()` | Yes — you call these manually | **No** — the adapter calls them internally |
| `start()` / `send()` / `close()` | Yes (SDK Mode only) | **No** |

> [!WARNING]
>
> **Audio path difference:** In SDK Mode and Host Mode, the SDK plays avatar audio through its internal audio player (Web Audio API). In RTC Mode, avatar audio arrives as a **native WebRTC audio track** and is played by the browser's WebRTC stack directly — the SDK's internal audio player is not involved. This distinction matters for audio processing (e.g., echo cancellation, noise suppression).

## Server-Side Setup

Your backend is responsible for sending audio to the avatar service and having the resulting avatar stream published to your RTC room. Two approaches are available:

| Platform | Framework plugin | Server SDK + egress |
| - | - | - |
| **LiveKit** | [LiveKit Agents plugin](https://github.com/spatialwalk/livekit-plugins-spatialreal) — hooks into your agent pipeline and publishes to the room | [LiveKit Server](https://spatialwalk-6179c111.mintlify.site/guide/rtc-livekit-server.md) (section 2) — use AvatarKit Server SDK with LiveKit egress config |

* **Framework plugin**: Best if you already use LiveKit Agents. The plugin handles audio → SpatialReal → RTC publish for you.
* **Server SDK + egress**: Use the [Golang](https://spatialwalk-6179c111.mintlify.site/server/go-sdk/go-sdk) or [Python](https://spatialwalk-6179c111.mintlify.site/server/python-sdk/python-sdk) Server SDK, create a session with LiveKit egress config, and send audio. The avatar service publishes audio and animation directly to the RTC room; your server does not relay that data.

Client-side setup is the same either way: the client joins the room and uses `@spatialwalk/avatarkit-rtc` to render the avatar. See the client guides below.

## Get Started

**Client (browser)**

> **[LiveKit Client](https://spatialwalk-6179c111.mintlify.site/guide/rtc-livekit-client.md)**
>
> Connect with LiveKit RTC

**Server (backend)**

> **[LiveKit Server](https://spatialwalk-6179c111.mintlify.site/guide/rtc-livekit-server.md)**
>
> Plugin or Server SDK with LiveKit egress


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.