diff --git a/.github/meet_sample_screenshot.png b/.github/meet_sample_screenshot.png new file mode 100644 index 00000000..b5c32b1f --- /dev/null +++ b/.github/meet_sample_screenshot.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:816d49d248767098776a2f2834c4ddeb110247fec7c0b23878f0782c908b6379 +size 826744 diff --git a/.github/youtube_tutorial_screenshot.png b/.github/youtube_tutorial_screenshot.png new file mode 100644 index 00000000..a5fe808a --- /dev/null +++ b/.github/youtube_tutorial_screenshot.png @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:0b6dffa6b3e952327e386cc1213bc7a1ca517b502d7c2c128da5a40fb8943b01 +size 3138615 diff --git a/README.md b/README.md index 585848a7..5e9ca9f1 100644 --- a/README.md +++ b/README.md @@ -1,84 +1,50 @@ - - - - - The LiveKit icon, the name of the repository and some sample code in the background. - - - - - # LiveKit Unity SDK - - -⚠️ Warning - - - -> This SDK is currently in Developer Preview mode and not ready for production use. There will be bugs and APIs may change during this period. - -> - -> We welcome and appreciate any feedback or contributions. You can create issues here or chat live with us and other users in our community at https://community.livekit.io/. - - - -Use this SDK to add realtime video, audio and data features to your Unity app. By connecting to LiveKit Cloud or a self-hosted server, you can quickly build applications such as multi-modal AI, live streaming, or video calls with just a few lines of code. +Use this SDK to add realtime video, audio and data features to your Unity app. By connecting to LiveKit Cloud or a self-hosted server, you can quickly build applications such as multi-modal AI like voice AI agents for NPCs, live streaming, or video calls with just a few lines of code. - - [SDK Reference »](https://livekit.github.io/client-sdk-unity) - +## Tutorial + +Screenshot of the LiveKit Unity SDK Youtube tutorial + +[Youtube Agents NPC Tutorial](https://www.youtube.com/watch?v=cLreGfOKzN8) ## Platform Support - +We officially support Unity 2022 onwards, we test on Unity 2022.3.62 and Unity 6000.3.10. - [x] Windows - - [x] MacOS - - [x] Linux - - [x] iOS - - [x] Android - - [ ] WebGL - - We plan to support all Unity platforms with this SDK. WebGL is currently supported with [client-sdk-unity-web](https://github.com/livekit/client-sdk-unity-web). - - ## Installation - ### Using Git Before cloning the repo, make sure that Git LFS is installed and setup. - You can either clone the repo and import from the local Unity package files: - ```sh git clone https://github.com/livekit/client-sdk-unity.git @@ -87,7 +53,6 @@ cd client-sdk-unity ``` - Or you can import the Git url `https://github.com/livekit/client-sdk-unity.git` from the package manager. If you want to use tagged release versions, use `https://github.com/livekit/client-sdk-unity.git#vX.X.X`, for example with `#v1.3.5`. @@ -95,125 +60,73 @@ If you want to use tagged release versions, use `https://github.com/livekit/clie The package is also hosted in the OpenUPM package registry. Here is the guide on how to use the OpenUPM registry to import the package: https://openupm.com/packages/io.livekit.livekit-sdk/#modal-manualinstallation - ## Local Development - ### Building LiveKit plugins locally - - For local development, initialize the Git submodule containing the Rust code for the LiveKit plugin libraries. - - There is a [helper script](https://github.com/livekit/client-sdk-unity/blob/main/Scripts~/build_ffi_locally.sh) to build the libraries locally and exchange the downloaded libraries with the local build artifacts in the correct `Runtime/Plugins` folder. - - Currently, the build script supports the following arguments: - macos - - android - - ios - - In the following options: - debug (default) - - release - - So a build command is for example: -`./Scripts~/build_ffi_locally.sh macos release` - - +`./Scripts~/build_ffi_locally.sh macos release` ### VSCode setup - - Look at the Unity-SDK.code-workspace setup for VSCode. This will use the Meet Sample as the Unity project and the Unity SDK package as two roots in a multi-root workspace and the Meet.sln as the `dotnet.defaultSolution`, enabling Rust and C# IDE support. - - ### Debugging - - For C# debugging, there is a simple attach launch option called `C# Unity`, for example in the `Meet/.vscode/launch.json`. - - For Rust / C++ debugging on MacOS, you need to install the [CodeLLDB](https://marketplace.visualstudio.com/items?itemName=vadimcn.vscode-lldb) extension. The debug attach is defined in `.vscode/launch.json`. - - 1. Build the livekit-ffi lib locally in debug mode with `./Scripts~/build_ffi_locally.sh macos debug` - 2. Start the Unity Editor - 3. Attach to the Unity Editor process (either auto or manual process picker) - 4. Start the Scene in Editor - - ### iOS - - Add dependent frameworks to your Unity project - - select `Unity-iPhone` -> `TARGETS` -> `UnityFramework` -> `General` -> `Frameworks and Libraries` -> `+` - - add the following frameworks: - - `OpenGLES.framework` `MetalKit.framework` `GLKit.framework` `MetalKit.framework` `VideoToolBox.framework` `Network.framework` - - add other linker flags to `UnityFramework`: - - `-ObjC` - - - -Since `libiPhone-lib.a` has built-in old versions of `celt` and `libvpx` (This will cause the opus and vp8/vp9 codecs to not be called correctly and cause a crash.), you need to ensure that `liblivekit_ffi.a` is linked before `libiPhone-lib.a`. - +Since `libiPhone-lib.a` has built-in old versions of `celt` and `libvpx` (This will cause the opus and vp8/vp9 codecs to not be called correctly and cause a crash.), you need to ensure that `liblivekit_ffi.a` is linked before `libiPhone-lib.a`. The package now applies an iOS post-build fix that rewrites the exported Xcode project so `libiPhone-lib.a` is moved after `liblivekit_ffi.a` in `UnityFramework -> Frameworks and Libraries`. - - It also strips the old CELT object cluster from the exported `Libraries/libiPhone-lib.a` so Xcode cannot resolve those codec symbols from Unity's archive. - - If your project disables package editor scripts or uses a custom Xcode export pipeline that overwrites `project.pbxproj` after LiveKit runs, you may still need to adjust the order manually by removing and re-adding `libiPhone-lib.a`. - - ## Examples - +The repo contains these sample projects: +- [Meet](https://github.com/livekit/client-sdk-unity/tree/main/Samples~/Meet) +- [Agents](https://github.com/livekit/client-sdk-unity/tree/main/Samples~/Agents) -Use the samples of the package to see how to use the SDK. +Most of the following functionalities and code snippets can be found in the samples in a similar form to try out. ### Tokens @@ -234,17 +147,19 @@ For production. Point to your own token endpoint URL and add any required authen Add a `TokenSourceComponent` to a GameObject, assign your `TokenSourceComponentConfig` asset, then fetch connection details before connecting: +`FetchConnectionDetails` returns a `TaskYieldInstruction`, so you can `yield return` it from a coroutine, `await` it, or bridge it with `.AsUniTask()`. Inspect the result on the instruction itself — `IsError` / `Exception` / `Result` — rather than on a `Task`: + ```cs -var connectionDetailsTask = _tokenSourceComponent.FetchConnectionDetails(); -yield return new WaitUntil(() => connectionDetailsTask.IsCompleted); +var fetch = _tokenSourceComponent.FetchConnectionDetails(); +yield return fetch; -if (connectionDetailsTask.IsFaulted) +if (fetch.IsError) { - Debug.LogError($"Failed to fetch connection details: {connectionDetailsTask.Exception?.InnerException?.Message}"); + Debug.LogError($"Failed to fetch connection details: {fetch.Exception?.Message}"); yield break; } -var details = connectionDetailsTask.Result; +var details = fetch.Result; _room = new Room(); var connect = _room.Connect(details.ServerUrl, details.ParticipantToken, new RoomOptions()); ``` @@ -252,808 +167,543 @@ var connect = _room.Connect(details.ServerUrl, details.ParticipantToken, new Roo Per-call overrides (e.g. dynamic room or participant names) can be passed via `TokenSourceFetchOptions`; any field set there wins over the asset, and unset fields fall back to the config: ```cs -var task = _tokenSourceComponent.FetchConnectionDetails(new TokenSourceFetchOptions +var fetch = _tokenSourceComponent.FetchConnectionDetails(new TokenSourceFetchOptions { RoomName = "lobby-" + System.Guid.NewGuid(), ParticipantName = playerName, }); ``` -To skip the ScriptableObject entirely, instantiate a token source directly: +To skip the ScriptableObject entirely, instantiate a token source directly. Each returns the same `TaskYieldInstruction` from `FetchConnectionDetails`, so it can be yielded, awaited, or `.AsUniTask()`-bridged just like the component: ```cs +// Fixed sources take no per-call options: ITokenSourceFixed source = new TokenSourceLiteral("wss://your.livekit.host", ""); -// or: new TokenSourceSandbox(""); -// or: new TokenSourceEndpoint("https://your.token-server/api/token", headers); // or: new TokenSourceCustom(async () => await MyAuthFlow()); + +var fetch = source.FetchConnectionDetails(); +yield return fetch; +var details = fetch.Result; + +// Configurable sources accept TokenSourceFetchOptions per call: +ITokenSourceConfigurable configurable = new TokenSourceSandbox(""); +// or: new TokenSourceEndpoint("https://your.token-server/api/token", headers); + +var configurableFetch = configurable.FetchConnectionDetails(new TokenSourceFetchOptions { RoomName = "lobby" }); +yield return configurableFetch; ``` - +### Connecting to a room +```cs +IEnumerator ConnectToRoom() +{ + var serverUrl = "< your server url >"; + var token = "< your token >"; -### Connect to a room + var room = new Room(); + var connect = room.Connect(serverUrl, token, new RoomOptions()); - + yield return connect; +} +``` -```cs +### Video -using LiveKit; +#### Publishing a texture (e.g Unity Camera) - +```cs +IEnumerator PublishCamera(Room room) +{ + // Option 1: publish a WebCamera + // var source = new TextureVideoSource(webCamTexture); + + // Option 2: publish a screen share + // var source = new ScreenVideoSource(); + + // Option 3: publishing a Unity Camera + var source = new CameraVideoSource(Camera.main); + + var track = LocalVideoTrack.CreateVideoTrack("my-video-track", source, room); + + var videoCoding = new VideoEncoding + { + MaxBitrate = 512000, + MaxFramerate = frameRate + }; + + var options = new TrackPublishOptions + { + VideoCodec = VideoCodec.Vp8, + VideoEncoding = videoCoding, + Simulcast = true, + Source = TrackSource.SourceCamera + }; + + var publish = room.LocalParticipant.PublishTrack(track, options); + yield return publish; + + if (!publish.IsError) + { + Debug.Log("Track published!"); + } + + source.Start(); + StartCoroutine(source.Update()); +} +``` -IEnumerator Start() +#### Receiving video +```cs +void OnTrackSubscribed(IRemoteTrack track, RemoteTrackPublication publication, RemoteParticipant participant) { + if (track is RemoteVideoTrack videoTrack) + { + var rawImage = GetComponent(); + var stream = new VideoStream(videoTrack); + stream.TextureReceived += (tex) => + { + rawImage.texture = tex; + }; + + StartCoroutine(stream.Update()); + } +} +``` -var room = new Room(); +### Audio -room.TrackSubscribed += TrackSubscribed; +There are two options to handle audio input and output, via "Unity Audio" or via "Platform Audio". - +#### Unity Audio -var connect = room.Connect("ws://localhost:7880", ""); +Using Unity Audio, the Unity Microphone and AudioSource APIs are used to pipe the audio frames. Use Unity Audio if your game needs to: +- read the audio frames, e.g. for lip syncing +- manipulate the audio frames -yield return connect; +On mobile platforms, the WebRTC audio is handled within the audio session owned by Unity, which reduces complexity. -if (!connect.IsError) +#### Unity Audio Input +```cs +IEnumerator PublishLocalMicrophoneUnity(Room room) { + Debug.Log("Publishing microphone using Unity Audio"); + GameObject microphoneObject = new GameObject("my-audio-source"); + var rtcSource = new MicrophoneSource(Microphone.devices[0], microphoneObject); + var track = LocalAudioTrack.CreateAudioTrack("my-audio-track", rtcSource, room); -Debug.Log("Connected to " + room.Name); + var options = new TrackPublishOptions(); + options.AudioEncoding = new AudioEncoding(); + options.AudioEncoding.MaxBitrate = 64000; + options.Source = TrackSource.SourceMicrophone; -} + var publish = room.LocalParticipant.PublishTrack(track, options); + yield return publish; -} + if (!publish.IsError) + { + Debug.Log("Track published!"); + } + rtcSource.Start(); +} ``` - - -## Asynchronous programming: coroutines, async/await, and UniTask - -The SDK exposes three interchangeable styles for awaiting asynchronous operations. Coroutines, async/await and UniTask. - -**1. Coroutines (default, no dependency)** — shown throughout this README. - -**2. async/await (no dependency)** — every operation returns an awaitable instruction (`ConnectInstruction`, `PublishTrackInstruction`, `PerformRpcInstruction`, the stream read instructions, …), so you can `await` it directly. As with coroutines, you inspect success/failure on the instruction (`IsError`) — `await` does not throw. Continuations resume on Unity's main thread. +#### Unity Audio Output ```cs -async void Start() +void TrackSubscribed(IRemoteTrack track, RemoteTrackPublication publication, RemoteParticipant participant) { - var room = new Room(); - var connect = room.Connect("ws://localhost:7880", "", new RoomOptions()); - await connect; - if (!connect.IsError) - Debug.Log("Connected to " + room.Name); + if (track is RemoteAudioTrack audioTrack) + { + GameObject audioOutputObject = new GameObject(audioTrack.Sid); + var source = audioOutputObject.AddComponent(); + var stream = new AudioStream(audioTrack, source); + } } ``` -> Use `async void` only for top-level event handlers (e.g. button callbacks); its exceptions surface to Unity's log rather than to a caller. Prefer `async Task`/`async UniTaskVoid` elsewhere. +#### Platform Audio -**3. UniTask (optional)** — install [UniTask](https://github.com/Cysharp/UniTask) (`com.cysharp.unitask`). The SDK auto-detects it via the `LIVEKIT_UNITASK` scripting define and enables the `LiveKit.UniTask` assembly, which adds `CancellationToken` support, composition, and async streams. +With Platform Audio, the audio input and output are managed by the native ADM of WebRTC. This unlocks echo cancellation, noise suppression, auto gain control and hardware processing if available. -Cancellation (abandon-awaiter semantics — the underlying request is not cancelled on the wire): +There are some known issues with Platform Audio, that we are working on resolving: +- On iOS, disposing of Platform Audio object stops Unity audio output +- On iOS and Unity 6, backgrounding the app breaks Platform Audio +- On MacOS with bluetooth headset, unmuting can break audio output -```cs -await room.Connect("ws://localhost:7880", "", new RoomOptions()) - .AsUniTask(cancellationToken); -``` +#### Initialize Platform Audio -Run operations in parallel. `AsUniTask` does not throw on failure (matching the -coroutine path), so keep the instructions and check `IsError` on each after the -`await` — otherwise a failed operation passes silently: +Make sure to initialize Platform Audio before connecting to a call. ```cs -var publishCamera = room.LocalParticipant.PublishTrack(cameraTrack, cameraOptions); -var publishMicrophone = room.LocalParticipant.PublishTrack(microphoneTrack, microphoneOptions); - -await UniTask.WhenAll(publishCamera.AsUniTask(ct), publishMicrophone.AsUniTask(ct)); - -if (publishCamera.IsError || publishMicrophone.IsError) - Debug.LogError("Failed to publish one or more tracks"); +void InitializePlatformAudio() +{ + try + { + var platformAudio = new PlatformAudio(); + Debug.Log($"PlatformAudio initialized: {platformAudio.RecordingDeviceCount} mics, " + + $"{platformAudio.PlayoutDeviceCount} speakers"); + + var (recording, playout) = platformAudio.GetDevices(); + Debug.Log("Recording devices:"); + foreach (var device in recording) + Debug.Log($" [{device.Index}] {device.Name}"); + + Debug.Log("Playout devices:"); + foreach (var device in playout) + Debug.Log($" [{device.Index}] {device.Name}"); + + if (platformAudio.RecordingDeviceCount > 0) + platformAudio.SetRecordingDevice(0); + if (platformAudio.PlayoutDeviceCount > 0) + platformAudio.SetPlayoutDevice(0); + + Debug.Log($"PlatformAudio ready. AEC={echoCancellation}, NS={noiseSuppression}, AGC={autoGainControl}, HW={preferHardwareProcessing}"); + } + catch (System.Exception e) + { + Debug.LogError($"Failed to initialize PlatformAudio, falling back to Unity audio: {e.Message}"); + usePlatformAudio = false; + platformAudio = null; + } +} ``` -Consume an incremental stream with `await foreach`. The sequence ends at end-of-stream; if the stream ends with an error it throws a `StreamError`: +#### Platform Audio Input ```cs -try -{ - await foreach (var chunk in reader.ReadIncremental().AsAsyncEnumerable(ct)) - Process(chunk); -} -catch (StreamError e) +IEnumerator PublishLocalMicrophonePlatform(PlatformAudio platformAudio, Room room) { - Debug.LogError(e.Message); + if (platformAudio != null) + { + yield return platformAudio.StartRecording(); + } + + var audioOptions = new AudioProcessingOptions + { + EchoCancellation = echoCancellation, + NoiseSuppression = noiseSuppression, + AutoGainControl = autoGainControl, + PreferHardware = preferHardwareProcessing + }; + + var platformAudioSource = new PlatformAudioSource(platformAudio, audioOptions); + var localAudioTrack = LocalAudioTrack.CreateAudioTrack(LocalAudioTrackName, platformAudioSource, room); + + var options = new TrackPublishOptions + { + AudioEncoding = new AudioEncoding { MaxBitrate = 64000 }, + Source = TrackSource.SourceMicrophone + }; + + var publish = room.LocalParticipant.PublishTrack(localAudioTrack, options); + yield return publish; + + if (publish.IsError) + { + Debug.LogError("Failed to publish microphone track"); + platformAudioSource?.Dispose(); + yield break; + } + + Debug.Log("Microphone published via PlatformAudio (AEC enabled)"); } ``` -> Error-handling differs by API: awaiting an instruction (and `AsUniTask`) never throws on a -> failed operation — you inspect `IsError` after the `await`. The stream enumerable is the -> exception: `await foreach` has no post-loop point to check `IsError`, so a mid-stream failure -> surfaces by throwing `StreamError`. - +#### Platform Audio Output -### Publishing microphone +Using Platform Audio, for audio output of subscribed remote audio tracks you don't need any Unity handling. +### RPC +Perform your own predefined method calls from one participant to another. -```cs +This feature is especially powerful when used with [Agents](https://docs.livekit.io/agents), for instance to forward LLM function calls to your client application. -var localSid = "my-audio-source"; +The following is a brief overview but [more detail is available in the documentation](https://docs.livekit.io/home/client/data/rpc). -GameObject audObject = new GameObject(localSid); +#### Registering an RPC method + +The participant who implements the method and will receive its calls must first register support. Your method handler will be an async callback that receives an `RpcInvocationData` object: -_audioObjects[localSid] = audObject; +```cs +void OnRoomConnected(Room room) +{ + room.LocalParticipant.RegisterRpcMethod("greet", HandleGreeting); +} -var rtcSource = new MicrophoneSource(Microphone.devices[0], _audioObjects[localSid]); +async Task HandleGreeting(RpcInvocationData data) +{ + Debug.Log($"Received greeting from {data.CallerIdentity}: {data.Payload}"); + return $"Hello, {data.CallerIdentity}!"; +} +``` -var track = LocalAudioTrack.CreateAudioTrack("my-audio-track", rtcSource, room); +In addition to the payload, `RpcInvocationData` also contains `responseTimeout`, which informs you the maximum time available to return a response. If you are unable to respond in time, the call will result in an error on the caller's side. - +#### Performing an RPC request -var options = new TrackPublishOptions(); +The caller may initiate an RPC call using coroutines: -options.AudioEncoding = new AudioEncoding(); +```cs +IEnumerator PerformRpcCoroutine(Room room) +{ + var rpcCall = room.LocalParticipant.PerformRpc(new PerformRpcParams + { + DestinationIdentity = "recipient-identity", + Method = "greet", + Payload = "Hello from RPC!" + }); + + yield return rpcCall; + + if (rpcCall.IsError) + { + Debug.Log($"RPC call failed: {rpcCall.Error}"); + } + else + { + Debug.Log($"RPC response: {rpcCall.Payload}"); + } +} +``` -options.AudioEncoding.MaxBitrate = 64000; +You may find it useful to adjust the `ResponseTimeout` parameter, which indicates the amount of time you will wait for a response. We recommend keeping this value as low as possible while still satisfying the constraints of your application. -options.Source = TrackSource.SourceMicrophone; +#### Errors - +LiveKit is a dynamic realtime environment and RPC calls can fail for various reasons. -var publish = room.LocalParticipant.PublishTrack(track, options); +You may throw errors of the type `RpcError` with a string `message` in an RPC method handler and they will be received on the caller's side with the message intact. Other errors will not be transmitted and will instead arrive to the caller as `1500` ("Application Error"). Other built-in errors are detailed in the [docs](https://docs.livekit.io/home/client/data/rpc/#errors). -yield return publish; +### Sending text - +Use text streams to send any amount of text between participants. -if (!publish.IsError) +#### Sending text all at once +```cs +IEnumerator SendText() { + var text = "Lorem ipsum dolor sit amet..."; + var sendTextInstruction = room.LocalParticipant.SendText("Hello from Unity", "Chat"); + yield return sendTextInstruction; +} +``` -Debug.Log("Track published!"); +#### Streaming text incrementally +```cs +IEnumerator StreamText(Room room) +{ + var streamWriter = room.LocalParticipant.StreamText("Chat"); + yield return streamWriter; + string[] textChunks = {"Lorem ", "ipsum ", "dolor ", "sit ", "amet..."}; + foreach (var textChunk in textChunks) + { + Debug.Log($"Sending {textChunk}"); + var instruction = streamWriter.Writer.Write(textChunk); + yield return instruction; + } + + yield return streamWriter.Writer.Close(); } +``` - +#### Handling incoming streams -rtcSource.Start(); +```cs +void OnRoomConnected(Room room) +{ + room.RegisterTextStreamHandler("Chat", (reader, identity) => StartCoroutine(OnTextStream(reader, identity))); +} +IEnumerator OnTextStream(TextStreamReader reader, string identity) +{ + // Option 1: Process the stream incrementally + var readIncremental = reader.ReadIncremental(); + while (true) + { + readIncremental.Reset(); + yield return readIncremental; + if (readIncremental.IsEos) + break; + Debug.Log(readIncremental.Text); + } + + // Option 2: Get the entire text after the stream completes + var readAllCall = reader.ReadAll(); + yield return readAllCall; + Debug.Log($"Received text: {readAllCall.Text}"); +} ``` - +### Sending files & bytes -### Publishing a texture (e.g Unity Camera) +Use byte streams to send files, images, or any other kind of data between participants. - +#### Sending files ```cs +IEnumerator SendFile() +{ + var filePath = "path/to/file.jpg"; + Debug.Log($"Sending file {filePath}"); + var sendFileCall = room.LocalParticipant.SendFile(filePath, "my-topic"); + yield return sendFileCall; + + if (sendFileCall.IsError) + { + Debug.LogError("File not found"); + } +} +``` -// publish a WebCamera +#### Streaming bytes -//var source = new TextureVideoSource(webCamTexture); +```cs +IEnumerator StreamBytes() +{ + var streamBytesCall = room.LocalParticipant.StreamBytes("my-topic"); + yield return streamBytesCall; - + var writer = streamBytesCall.Writer; + Debug.Log($"Opened byte stream with ID: {writer.Info.Id}"); -// Publish the entire screen + var dataChunks = new[] + { + new byte[] { 0x00, 0x01 }, + new byte[] { 0x02, 0x03 } + }; -//var source = new ScreenVideoSource(); + foreach (var chunk in dataChunks) + { + yield return writer.Write(chunk); + } - + yield return writer.Close(); +} +``` -// Publishing a Unity Camera +#### Handling incoming streams -//Camera.main.enabled = true; +```cs +void OnRoomConnected(Room room) +{ + room.RegisterByteStreamHandler("my-topic", (reader, identity) => StartCoroutine(HandleByteStream(reader, identity))); +} -//var source = new CameraVideoSource(Camera.main); +IEnumerator HandleByteStream(ByteStreamReader reader, string participantIdentity) +{ + var info = reader.Info; + + // Option 1: Process the stream incrementally + var readIncremental = reader.ReadIncremental(); + while (true) + { + readIncremental.Reset(); + yield return readIncremental; + if (readIncremental.IsEos) break; + foreach (var dataByte in readIncremental.Bytes) + Debug.Log($"Received {dataByte}"); + } + + // Option 2: Get the entire file after the stream completes + var readAllCall = reader.ReadAll(); + yield return readAllCall; + var data = readAllCall.Bytes; + foreach (var dataByte in data) + Debug.Log($"Received {dataByte}"); + + // Option 3: Write the stream to a local file on disk as it arrives + var writeToFileCall = reader.WriteToFile(); + yield return writeToFileCall; + var path = writeToFileCall.FilePath; + Debug.Log($"Wrote to file: {path}"); + + Debug.Log($@" + Byte stream received from {participantIdentity} + Topic: {info.Topic} + Timestamp: {info.Timestamp} + ID: {info.Id} + Size: {info.TotalLength} (only available if the stream was sent with `SendFile`) + "); +} +``` - +## Asynchronous programming: coroutines, async/await, and UniTask -var rt = new UnityEngine.RenderTexture(1920, 1080, 24, RenderTextureFormat.ARGB32); +The SDK exposes three interchangeable styles for awaiting asynchronous operations. Coroutines, async/await and UniTask. -rt.Create(); +**1. Coroutines (default, no dependency)** — shown throughout this README. -Camera.main.targetTexture = rt; +**2. async/await (no dependency)** — every operation returns an awaitable instruction (`ConnectInstruction`, `PublishTrackInstruction`, `PerformRpcInstruction`, the stream read instructions, …), so you can `await` it directly. As with coroutines, you inspect success/failure on the instruction (`IsError`) — `await` does not throw. Continuations resume on Unity's main thread. -var source = new TextureVideoSource(rt); +```cs +async void Start() +{ + var room = new Room(); + var connect = room.Connect("ws://localhost:7880", "", new RoomOptions()); + await connect; + if (!connect.IsError) + Debug.Log("Connected to " + room.Name); +} +``` -var track = LocalVideoTrack.CreateVideoTrack("my-video-track", source, room); +> Use `async void` only for top-level event handlers (e.g. button callbacks); its exceptions surface to Unity's log rather than to a caller. Prefer `async Task`/`async UniTaskVoid` elsewhere. - +**3. UniTask (optional)** — install [UniTask](https://github.com/Cysharp/UniTask) (`com.cysharp.unitask`). The SDK auto-detects it via the `LIVEKIT_UNITASK` scripting define and enables the `LiveKit.UniTask` assembly, which adds `CancellationToken` support, composition, and async streams. -var options = new TrackPublishOptions(); +Cancellation (abandon-awaiter semantics — the underlying request is not cancelled on the wire): -options.VideoCodec = VideoCodec.Vp8; +```cs +await room.Connect("ws://localhost:7880", "", new RoomOptions()) + .AsUniTask(cancellationToken); +``` -var videoCoding = new VideoEncoding(); - -videoCoding.MaxBitrate = 512000; - -videoCoding.MaxFramerate = frameRate; - -options.VideoEncoding = videoCoding; - -options.Simulcast = true; - -options.Source = TrackSource.SourceCamera; - - - -var publish = room.LocalParticipant.PublishTrack(track, options); - -yield return publish; - - - -if (!publish.IsError) - -{ - -Debug.Log("Track published!"); - -} - - - -source.Start(); - -StartCoroutine(source.Update()); - -``` - - - -### Receiving tracks - - - -```cs - -IEnumerator Start() - -{ - -var room = new Room(); - -room.TrackSubscribed += TrackSubscribed; - - - -var connect = room.Connect("ws://localhost:7880", ""); - -yield return connect; - - - -// .... - -} - - - -void TrackSubscribed(IRemoteTrack track, RemoteTrackPublication publication, RemoteParticipant participant) - -{ - -if (track is RemoteVideoTrack videoTrack) - -{ - -var rawImage = GetComponent(); - -var stream = new VideoStream(videoTrack); - -stream.TextureReceived += (tex) => - -{ - -rawImage.texture = tex; - -}; - -StartCoroutine(stream.Update()); - -// The video data is displayed on the rawImage - -} - -else if (track is RemoteAudioTrack audioTrack) - -{ - -GameObject audObject = new GameObject(audioTrack.Sid); - -var source = audObject.AddComponent(); - -var stream = new AudioStream(audioTrack, source); - -// Audio is being played on the source .. - -} - -} - -``` - - - -### RPC - - - -Perform your own predefined method calls from one participant to another. - - - -This feature is especially powerful when used with [Agents](https://docs.livekit.io/agents), for instance to forward LLM function calls to your client application. - - - -The following is a brief overview but [more detail is available in the documentation](https://docs.livekit.io/home/client/data/rpc). - - - -#### Registering an RPC method - - - -The participant who implements the method and will receive its calls must first register support. Your method handler will be an async callback that receives an `RpcInvocationData` object: - - - -```cs - -// Define your method handler - -async Task HandleGreeting(RpcInvocationData data) - -{ - -Debug.Log($"Received greeting from {data.CallerIdentity}: {data.Payload}"); - -return $"Hello, {data.CallerIdentity}!"; - -} - - - -// Register the method after connection to the room - -room.LocalParticipant.RegisterRpcMethod("greet", HandleGreeting); - -``` - - - -In addition to the payload, `RpcInvocationData` also contains `responseTimeout`, which informs you the maximum time available to return a response. If you are unable to respond in time, the call will result in an error on the caller's side. - - - -#### Performing an RPC request - - - -The caller may initiate an RPC call using coroutines: - - - -```cs - -IEnumerator PerformRpcCoroutine() - -{ - -var rpcCall = room.LocalParticipant.PerformRpc(new PerformRpcParams - -{ - -DestinationIdentity = "recipient-identity", - -Method = "greet", - -Payload = "Hello from RPC!" - -}); - - - -yield return rpcCall; - - - -if (rpcCall.IsError) - -{ - -Debug.Log($"RPC call failed: {rpcCall.Error}"); - -} - -else - -{ - -Debug.Log($"RPC response: {rpcCall.Payload}"); - -} - -} - - - -// Start the coroutine from another MonoBehaviour method - -StartCoroutine(PerformRpcCoroutine()); - -``` - - - -You may find it useful to adjust the `ResponseTimeout` parameter, which indicates the amount of time you will wait for a response. We recommend keeping this value as low as possible while still satisfying the constraints of your application. - - - -#### Errors - - - -LiveKit is a dynamic realtime environment and RPC calls can fail for various reasons. - - - -You may throw errors of the type `RpcError` with a string `message` in an RPC method handler and they will be received on the caller's side with the message intact. Other errors will not be transmitted and will instead arrive to the caller as `1500` ("Application Error"). Other built-in errors are detailed in the [docs](https://docs.livekit.io/home/client/data/rpc/#errors). - - - -### Sending text - - - -Use text streams to send any amount of text between participants. - - - -#### Sending text all at once - - - -```cs - -IEnumerator PerformSendText() - -{ - -var text = "Lorem ipsum dolor sit amet..."; - -var sendTextCall = room.LocalParticipant.SendText(text, "some-topic"); - -yield return sendTextCall; - - - -Debug.Log($"Sent text with stream ID {sendTextCall.Info.Id}"); - -} - -``` - - - -#### Streaming text incrementally - - - -```cs - -IEnumerator PerformStreamText() - -{ - -var streamTextCall = room.LocalParticipant.StreamText("my-topic"); - -yield return streamTextCall; - - - -var writer = streamTextCall.Writer; - -Debug.Log($"Opened text stream with ID: {writer.Info.Id}"); - - - -// In a real app, you would generate this text asynchronously / incrementally as well - -var textChunks = new[] { "Lorem ", "ipsum ", "dolor ", "sit ", "amet..." }; - -foreach (var chunk in textChunks) - -{ - -yield return writer.Write(chunk); - -} - - - -// The stream must be explicitly closed when done - -yield return writer.Close(); - - - -Debug.Log($"Closed text stream with ID: {writer.Info.Id}"); - -} - -``` - - - -#### Handling incoming streams - - - -```cs - -IEnumerator HandleTextStream(TextStreamReader reader, string participantIdentity) - -{ - -var info = reader.Info; - -Debug.Log($@" - -Text stream received from {participantIdentity} - -Topic: {info.Topic} - -Timestamp: {info.Timestamp} - -ID: {info.Id} - -Size: {info.TotalLength} (only available if the stream was sent with `SendText`) - -"); - - - -// Option 1: Process the stream incrementally - -var readIncremental = reader.ReadIncremental(); - -while (true) - -{ - -readIncremental.Reset(); - -yield return readIncremental; - -if (readIncremental.IsEos) break; - -Debug.Log($"Next chunk: {readIncremental.Text}"); - -} - - - -// Option 2: Get the entire text after the stream completes - -var readAllCall = reader.ReadAll(); - -yield return readAllCall; - -Debug.Log($"Received text: {readAllCall.Text}") - -} - - - -// Register the topic before connecting to the room - -room.RegisterTextStreamHandler("my-topic", (reader, identity) => - -StartCoroutine(HandleTextStream(reader, identity)) - -); - -``` - - - -### Sending files & bytes - - - -Use byte streams to send files, images, or any other kind of data between participants. - - - -#### Sending files - - +Run operations in parallel. `AsUniTask` does not throw on failure (matching the +coroutine path), so keep the instructions and check `IsError` on each after the +`await` — otherwise a failed operation passes silently: ```cs +var publishCamera = room.LocalParticipant.PublishTrack(cameraTrack, cameraOptions); +var publishMicrophone = room.LocalParticipant.PublishTrack(microphoneTrack, microphoneOptions); -IEnumerator PerformSendFile() - -{ - -var filePath = "path/to/file.jpg"; - -var sendFileCall = room.LocalParticipant.SendFile(filePath, "some-topic"); - -yield return sendFileCall; - - - -Debug.Log($"Sent file with stream ID: {sendFileCall.Info.Id}"); - -} +await UniTask.WhenAll(publishCamera.AsUniTask(ct), publishMicrophone.AsUniTask(ct)); +if (publishCamera.IsError || publishMicrophone.IsError) + Debug.LogError("Failed to publish one or more tracks"); ``` - - -#### Streaming bytes - - +Consume an incremental stream with `await foreach`. The sequence ends at end-of-stream; if the stream ends with an error it throws a `StreamError`: ```cs - -IEnumerator PerformStreamBytes() - -{ - -var streamBytesCall = room.LocalParticipant.StreamBytes("my-topic"); - -yield return streamBytesCall; - - - -var writer = streamBytesCall.Writer; - -Debug.Log($"Opened byte stream with ID: {writer.Info.Id}"); - - - -// Example sending arbitrary binary data - -// For sending files, use `SendFile` instead - -var dataChunks = new[] { - -new byte[] { 0x00, 0x01 }, - -new byte[] { 0x03, 0x04 } - -}; - -foreach (var chunk in dataChunks) - +try { - -yield return writer.Write(chunk); - -} - - - -// The stream must be explicitly closed when done - -yield return writer.Close(); - - - -Debug.Log($"Closed byte stream with ID: {writer.Info.Id}"); - + await foreach (var chunk in reader.ReadIncremental().AsAsyncEnumerable(ct)) + Process(chunk); } - -``` - - - -#### Handling incoming streams - - - -```cs - -IEnumerator HandleByteStream(ByteStreamReader reader, string participantIdentity) - -{ - -var info = reader.Info; - - - -// Option 1: Process the stream incrementally - -var readIncremental = reader.ReadIncremental(); - -while (true) - +catch (StreamError e) { - -readIncremental.Reset(); - -yield return readIncremental; - -if (readIncremental.IsEos) break; - -Debug.Log($"Next chunk: {readIncremental.Bytes}"); - -} - - - -// Option 2: Get the entire file after the stream completes - -var readAllCall = reader.ReadAll(); - -yield return readAllCall; - -var data = readAllCall.Bytes; - - - -// Option 3: Write the stream to a local file on disk as it arrives - -var writeToFileCall = reader.WriteToFile(); - -yield return writeToFileCall; - -var path = writeToFileCall.FilePath; - -Debug.Log($"Wrote to file: {path}"); - - - -Debug.Log($@" - -Byte stream received from {participantIdentity} - -Topic: {info.Topic} - -Timestamp: {info.Timestamp} - -ID: {info.Id} - -Size: {info.TotalLength} (only available if the stream was sent with `SendFile`) - -"); - + Debug.LogError(e.Message); } - - - -// Register the topic after connection to the room - -room.RegisterByteStreamHandler("my-topic", (reader, identity) => - -StartCoroutine(HandleByteStream(reader, identity)) - -); - ``` +> Error-handling differs by API: awaiting an instruction (and `AsUniTask`) never throws on a +> failed operation — you inspect `IsError` after the `await`. The stream enumerable is the +> exception: `await foreach` has no post-loop point to check `IsError`, so a mid-stream failure +> surfaces by throwing `StreamError`. - ## Verbose Logging - - To enable verbose logging, define the `LK_VERBOSE` symbol: - - 1. Navigate to Project Settings → Player - 2. Select your platform tab (e.g., Mac, iOS, Android). - -3. Under Other Settings → Scripting Define Symbols, add `LK_VERBOSE`. - - +3. Under Other Settings → Scripting Define Symbols, add `LK_VERBOSE`. diff --git a/Samples~/Agents/ProjectSettings/ProjectSettings.asset b/Samples~/Agents/ProjectSettings/ProjectSettings.asset index f28f8e67..91ba1f5f 100644 --- a/Samples~/Agents/ProjectSettings/ProjectSettings.asset +++ b/Samples~/Agents/ProjectSettings/ProjectSettings.asset @@ -265,7 +265,7 @@ PlayerSettings: useCustomGradlePropertiesTemplate: 0 useCustomGradleSettingsTemplate: 0 useCustomProguardFile: 0 - AndroidTargetArchitectures: 1 + AndroidTargetArchitectures: 2 AndroidTargetDevices: 0 AndroidSplashScreenScale: 0 androidSplashScreen: {fileID: 0} @@ -763,7 +763,8 @@ PlayerSettings: scriptingDefineSymbols: {} additionalCompilerArguments: {} platformArchitecture: {} - scriptingBackend: {} + scriptingBackend: + Android: 1 il2cppCompilerConfiguration: {} il2cppCodeGeneration: {} managedStrippingLevel: diff --git a/Samples~/Agents/README.md b/Samples~/Agents/README.md new file mode 100644 index 00000000..e021e48a --- /dev/null +++ b/Samples~/Agents/README.md @@ -0,0 +1,25 @@ +## Agents sample + +Connect to a voice AI agent. + +## Tutorial + +Screenshot of the LiveKit Unity SDK Youtube tutorial + +[Youtube Agents NPC Tutorial](https://www.youtube.com/watch?v=cLreGfOKzN8) + +## Getting started + +### Project setup + +The sample can either be imported via the package manager to get access to the assets or opened as a full Unity project. The project is set up to build and run on all supported platforms. + +### Agent dispatch + +To talk to an agent, the app needs a token to connect to a LiveKit room and dispatch an agent. The project is already configured to automatically connect to the LiveKit homepage agent you can try at www.livekit.com. + +To connect to your own LiveKit server, edit the token source component config with your projects token source. + +### Common sample package + +In order to get access to common sample functions like the on device scrolling log, make sure to import the [Common](https://github.com/livekit/client-sdk-unity/tree/main/Samples~/Common) sample from the LiveKit Unity Package in the package manager. \ No newline at end of file diff --git a/Samples~/Meet/README.md b/Samples~/Meet/README.md new file mode 100644 index 00000000..4ffae2cc --- /dev/null +++ b/Samples~/Meet/README.md @@ -0,0 +1,23 @@ +## Meet sample + +Connect to a video call with other remote participants. + +Screenshot of the Meet sample + +## Getting started + +### Project setup + +The sample can either be imported via the package manager to get access to the assets or opened as a full Unity project. The project is set up to build and run on all supported platforms. + +### Token source + +In order to connect to your LiveKit server, configure the token source component config used in the scene. To learn more about tokens, see https://docs.livekit.io/frontends/build/authentication. + +### Audio system + +The LiveKit Unity SDK offers two audio systems. The Unity audio path uses the Unity APIs for audio input and output. Platform Audio is the alternative, where the native LiveKit plugin manages audio input and output. You can select which path to use on the MeetManager component. + +### Common sample package + +In order to get access to common sample functions like the on device scrolling log, make sure to import the [Common](https://github.com/livekit/client-sdk-unity/tree/main/Samples~/Common) sample from the LiveKit Unity Package in the package manager. \ No newline at end of file