[Article] OpenUI for Voice Agents: Pairing LiveKit with Generative UI for Real-Time Visual Feedback

[manja316]

Closes #6

Summary

A technical article + conceptual guide breaking down how to combine LiveKit's real-time voice infrastructure with OpenUI for multimodal voice agents. Based on the actual thesysdev/voice-agent-generativeui reference implementation. Covers:

• The dual-output architecture: speech path (TTS) + visual path (C1 streaming) running in parallel
• The show_ui tool implementation: fire-and-forget async streaming via LiveKit text channels
• Browser-side progressive rendering with C1Component and registerTextStreamHandler
• Bidirectional interaction: UI button clicks flowing back to the voice agent via chat messages
• Prompt strategy for aggressive visual output ("show, don't read")
• Realtime vs pipeline agent modes for tool narration
• Use cases: customer support, financial advising, medical triage, travel planning

~2,500 words. Code examples pulled directly from the reference implementation source code (agent.ts, show-ui.ts, GenUIPanel.tsx, VoiceUI.tsx, prompt.ts).

[entelligence-ai-pr-reviews]

EntelligenceAI PR Summary

Introduces a new technical article (articles/openui-voice-agents-livekit.md) detailing the design and implementation of a multimodal voice agent system.

• Documents LiveKit real-time voice pipeline integration (STT → LLM → TTS) paired with Thesys C1 generative UI model
• Describes show_ui tool design using fire-and-forget async streaming via LiveKit text streams with abort-on-new-request pattern
• Covers Next.js frontend integration using C1Component for progressive/streaming React UI rendering
• Details bidirectional action handling enabling UI components to send events back to the voice agent
• Explains prompt engineering strategy for aggressive visual delegation to UI
• Outlines dual agent operation modes: realtime mode and pipeline narration mode

---

Confidence Score: 5/5 - Safe to Merge

Safe to merge — this PR introduces a single new documentation article (articles/openui-voice-agents-livekit.md) with no runtime code changes, meaning there is zero risk of introducing bugs, regressions, or security vulnerabilities. The article documents a well-structured multimodal voice agent architecture covering the LiveKit STT→LLM→TTS pipeline, the show_ui fire-and-forget async streaming tool with abort-on-new-request pattern, and Next.js frontend integration via C1Component. No review comments were generated and no pre-existing unresolved concerns are associated with this PR.

Key Findings:

• The entire changeset is a single new Markdown article file (articles/openui-voice-agents-livekit.md), which carries no runtime execution risk whatsoever — documentation-only changes cannot introduce crashes, logic bugs, or security issues.
• The described architecture patterns (fire-and-forget async streaming, abort-on-new-request via LiveKit text streams) are clearly explained in the article and do not represent code being merged into a production codebase, so their correctness is an editorial concern rather than a safety concern for this PR.
• Zero review comments were generated by automated analysis and zero pre-existing unresolved issues are flagged, giving high confidence that even the documentation content is coherent and well-formed.

Files requiring special attention

• articles/openui-voice-agents-livekit.md

[entelligence-ai-pr-reviews]

Walkthrough

Adds a new technical article documenting the architecture and implementation of a multimodal voice agent system combining LiveKit's real-time voice pipeline (STT → LLM → TTS) with Thesys's C1 generative UI model. Covers the show_ui tool design, Next.js frontend integration, bidirectional UI-to-agent action handling, prompt strategy for visual delegation, and dual agent operation modes.

Changes

File(s)

Summary

articles/openui-voice-agents-livekit.md

New technical article documenting multimodal voice agent architecture pairing LiveKit's real-time voice pipeline with Thesys C1 generative UI, covering show_ui tool design (fire-and-forget async streaming, abort-on-new-request pattern), Next.js frontend integration via C1Component, bidirectional UI-to-agent action handling, aggressive visual delegation prompt strategy, and dual agent modes (realtime vs. pipeline narration).

Sequence Diagram

This diagram shows the interactions between components:

sequenceDiagram
    actor User
    participant Browser as "Browser (Next.js)"
    participant LiveKit as "LiveKit Room"
    participant Agent as "VoiceAgent"
    participant ShowUI as "ShowUITool"
    participant LLM as "Voice LLM"
    participant C1 as "Thesys C1 Model"

    User->>LiveKit: Speaks (audio)
    LiveKit->>Agent: STT transcript
    Agent->>LLM: Send transcript + tools

    Note over LLM: Decides what to show visually
    LLM-->>Agent: Call show_ui(content)
    Agent->>ShowUI: execute({ content })

    Note over ShowUI: Fire-and-forget background task
    ShowUI-->>Agent: return "UI is loading on screen"
    Agent->>LLM: Tool result (immediate)
    LLM-->>Agent: Generate spoken response
    Agent->>LiveKit: TTS audio stream
    LiveKit->>User: Agent speaks

    par Background UI Generation
        ShowUI->>LiveKit: streamText({ topic: "genui" }) open writer
        ShowUI->>C1: chat.completions.create(content, stream:true)
        loop Streaming chunks
            C1-->>ShowUI: openui-lang code chunk
            ShowUI->>LiveKit: writer.write(chunk)
            LiveKit->>Browser: genui text stream chunk
            Browser->>Browser: Accumulate & re-render C1Component
        end
        ShowUI->>LiveKit: writer.close()
    end

    Note over Browser: User sees components appear while agent talks
    opt User clicks UI element
        Browser->>LiveKit: sendChatMessage(llmFriendlyMessage)
        LiveKit->>Agent: Chat message
        Agent->>LLM: Send UI action as user input
        LLM-->>Agent: Spoken response
        Agent->>LiveKit: TTS audio stream
        LiveKit->>User: Agent acknowledges action
    end

Loading

---

🔗 Cross-Repository Impact Analysis

Enable automatic detection of breaking changes across your dependent repositories. → Set up now

Learn more about Cross-Repository Analysis

What It Does

• Automatically identifies repositories that depend on this code
• Analyzes potential breaking changes across your entire codebase
• Provides risk assessment before merging to prevent cross-repo issues

How to Enable

1. Visit Settings → Code Management
2. Configure repository dependencies
3. Future PRs will automatically include cross-repo impact analysis!

Benefits

• 🛡️ Prevent breaking changes across repositories
• 🔍 Catch integration issues before they reach production
• 📊 Better visibility into your multi-repo architecture

---

[entelligence-ai-pr-reviews]

LGTM 👍 No issues found.