> ## Documentation Index
> Fetch the complete documentation index at: https://spatialwalk-6179c111.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# LiveKit Client

> Real-time voice communication with avatars using LiveKit

```mermaid theme={null}
flowchart LR
    A["🎤 Microphone"] --> B["AvatarPlayer"]
    B <-->|WebRTC| C["LiveKit Server"]
    B --> D["AvatarKit SDK"]
    D --> E["🖥️ Avatar Rendering"]
```

> [!NOTE]
>
> **Web Only:** RTC Mode is currently available for Web applications only.

> [!WARNING]
>
> **Do not call `initializeAudioContext()`** — it is not needed in RTC Mode. Avatar audio is delivered as a native WebRTC audio track by the LiveKit client SDK, not through the AvatarKit SDK's internal audio player. The `AvatarPlayer` adapter only feeds **animation data** to the SDK for rendering; audio playback is handled entirely by LiveKit's WebRTC stack.

## Installation

> [!WARNING]
>
> **Critical:** We are only compatable with `livekit-client` under version `2.17`, since `2.17` introduced single PC by default. We are working on adding support for `2.17`+ in a future release, but for now please ensure you install version `2.16.1` to avoid compatibility issues.

**pnpm**

```bash theme={null}
pnpm add @spatialwalk/avatarkit @spatialwalk/avatarkit-rtc livekit-client@2.16.1
```

**npm**

```bash theme={null}
npm install @spatialwalk/avatarkit @spatialwalk/avatarkit-rtc livekit-client@2.16.1
```

**yarn**

```bash theme={null}
yarn add @spatialwalk/avatarkit @spatialwalk/avatarkit-rtc livekit-client@2.16.1
```

You also need to configure your build tool for WASM files — see [Build Tool Configuration](https://spatialwalk-6179c111.mintlify.site/guide/sdk-mode-web.md#build-tool-configuration).

## Authentication

| Credential | How to Obtain | Notes |
| - | - | - |
| **App ID** | [SpatialReal Studio](https://app.spatialreal.ai) → Create App | For SDK initialization |
| **Session Token** | Your backend → AvatarKit Server | For avatar loading (max 24 hours) |
| **LiveKit Token** | Your backend → LiveKit Server | For RTC room connection |

## Quick Start

For a complete client + server implementation, see [AvatarKit Voice Agent Demo](https://github.com/spatialwalk/avatarkit-voice-agent-demo/tree/main/livekit-cascade-voice-agent).

**1. Initialize SDK in Host Mode**

```typescript theme={null}
import { AvatarSDK, AvatarManager, AvatarView, DrivingServiceMode, Environment } from '@spatialwalk/avatarkit'

await AvatarSDK.initialize('your-app-id', {
  environment: Environment.intl,
  drivingServiceMode: DrivingServiceMode.host,  // MUST be host for RTC
})

AvatarSDK.setSessionToken('your-session-token')
```

**2. Load Avatar & Create View**

```typescript theme={null}
const avatar = await AvatarManager.shared.load('avatar-id')
const container = document.getElementById('avatar-container')!
const avatarView = new AvatarView(avatar, container)
```

**3. Create Player with LiveKit Provider**

```typescript theme={null}
import { AvatarPlayer, LiveKitProvider } from '@spatialwalk/avatarkit-rtc'

const provider = new LiveKitProvider()
const player = new AvatarPlayer(provider, avatarView, {
  logLevel: 'warning',
})
```

**4. Connect to LiveKit Server**

```typescript theme={null}
await player.connect({
  url: 'wss://your-livekit-server.com',
  token: 'your-livekit-token',
  roomName: 'room-name',
})
```

**5. Start Voice Interaction**

```typescript theme={null}
// Start microphone publishing
await player.startPublishing()

// Stop microphone
await player.stopPublishing()

// Disconnect when done
await player.disconnect()
```

## AvatarPlayer API

### Constructor

```typescript theme={null}
new AvatarPlayer(provider: LiveKitProvider, avatarView: AvatarView, options?: AvatarPlayerOptions)
```

### AvatarPlayerOptions

```typescript theme={null}
interface AvatarPlayerOptions {
  /** Start speaking transition frames, default 5 (~200ms at 25fps) */
  transitionStartFrameCount?: number

  /** End speaking transition frames, default 40 (~1600ms at 25fps) */
  transitionEndFrameCount?: number

  /** Log level: 'info' | 'warning' | 'error' | 'none', default 'warning' */
  logLevel?: LogLevel
}
```

### Connection

```typescript theme={null}
// Connect to LiveKit server
await player.connect(config: LiveKitConnectionConfig)

// Disconnect and clean up
await player.disconnect()

// Reconnect using last config (useful after stalls)
await player.reconnect()

// Check connection status
player.isConnected  // boolean
player.getConnectionState()  // ConnectionState
```

#### LiveKitConnectionConfig

```typescript theme={null}
interface LiveKitConnectionConfig {
  url: string       // LiveKit server URL (wss://...)
  token: string     // Auth token from your backend
  roomName: string  // Room name
}
```

### Microphone Control

```typescript theme={null}
// Start microphone (requests permission automatically)
await player.startPublishing()

// Stop microphone
await player.stopPublishing()
```

### Custom Audio Publishing

For non-microphone audio sources like audio elements or Web Audio API.

```typescript theme={null}
// Publish a custom audio track
await player.publishAudio(track: MediaStreamTrack)

// Stop custom audio
await player.unpublishAudio()
```

| Audio Source | How to Obtain Track |
| - | - |
| `<audio>` element | `audioElement.captureStream().getAudioTracks()[0]` |
| Screen share audio | `getDisplayMedia({ audio: true })` |
| Web Audio API | `audioContext.createMediaStreamDestination().stream.getAudioTracks()[0]` |

> [!NOTE]
>
> `unpublishAudio()` does **not** stop the track. You are responsible for calling `track.stop()` to release the resource.

### Native Client Access

Access the underlying LiveKit `Room` instance for advanced features.

```typescript theme={null}
import { LiveKitRoom } from '@spatialwalk/avatarkit-rtc'

// Via provider (recommended — full type safety)
const room = provider.getNativeClient()  // LiveKitRoom | null

// Via player (requires type assertion)
const room = player.getNativeClient() as LiveKitRoom | null

console.log('Remote participants:', room?.remoteParticipants.size)
```

### ConnectionState

```typescript theme={null}
type ConnectionState = 'disconnected' | 'connecting' | 'connected' | 'reconnecting' | 'failed'
```

## Events

```typescript theme={null}
player.on(event: string, handler: Function)
player.off(event: string, handler: Function)
```

| Event | Callback | Description |
| - | - | - |
| `'connected'` | `()` | Connected to RTC server |
| `'disconnected'` | `()` | Disconnected from RTC server |
| `'error'` | `(error: Error)` | An error occurred |
| `'connection-state-changed'` | `(state: ConnectionState)` | Connection state changed |
| `'stalled'` | `()` | Data stream stalled (no frames for 3s) |

> [!NOTE]
>
> When a `stalled` event fires, the avatar automatically transitions to idle animation. Consider calling `player.reconnect()` to recover.

**Stall recovery example:**

```typescript theme={null}
player.on('stalled', async () => {
  console.log('Stream stalled, reconnecting...')
  try {
    await player.reconnect()
  } catch (error) {
    console.error('Reconnection failed:', error)
  }
})
```

## Complete Example

```typescript theme={null}
import { AvatarPlayer, LiveKitProvider } from '@spatialwalk/avatarkit-rtc'
import { AvatarSDK, AvatarView, AvatarManager, DrivingServiceMode, Environment } from '@spatialwalk/avatarkit'

async function init() {
  // Initialize SDK in host mode
  await AvatarSDK.initialize('your-app-id', {
    environment: Environment.intl,
    drivingServiceMode: DrivingServiceMode.host,
  })
  AvatarSDK.setSessionToken('your-session-token')

  // Load avatar and create view
  const avatar = await AvatarManager.shared.load('character-id')
  const container = document.getElementById('avatar-container')!
  const avatarView = new AvatarView(avatar, container)

  // Create player
  const provider = new LiveKitProvider()
  const player = new AvatarPlayer(provider, avatarView, {
    logLevel: 'info',
    transitionStartFrameCount: 5,
    transitionEndFrameCount: 40,
  })

  // Listen to events
  player.on('connected', () => console.log('Connected!'))
  player.on('disconnected', () => console.log('Disconnected!'))
  player.on('error', (err) => console.error('Error:', err))
  player.on('stalled', async () => {
    console.log('Stream stalled, reconnecting...')
    await player.reconnect()
  })

  // Connect to LiveKit server
  await player.connect({
    url: 'wss://your-livekit-server.com',
    token: 'your-livekit-token',
    roomName: 'my-room',
  })

  // Start microphone
  await player.startPublishing()
}
```

## Browser Compatibility

| Browser | Minimum Version | Notes |
| - | - | - |
| Chrome | 94+ | VP8 + RTCRtpScriptTransform |
| Firefox | 117+ | RTCRtpScriptTransform required |
| Safari | 15.4+ | RTCRtpScriptTransform supported |
| Edge | 94+ | Same as Chrome |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.