feat(telnyx): Telnyx assistant server framework (Media Streaming default)#188
Open
lucasassisrosa wants to merge 19 commits into
Open
feat(telnyx): Telnyx assistant server framework (Media Streaming default)#188lucasassisrosa wants to merge 19 commits into
lucasassisrosa wants to merge 19 commits into
Conversation
Add Telnyx hosted AI Assistant as a first-class EVA framework alongside
the existing Pipecat pipeline. The Telnyx provider connects to the Telnyx
AI gateway via direct anonymous WebRTC (Verto JSON-RPC over WebSocket),
bridges audio between the EVA user simulator (Twilio mulaw frames) and the
Telnyx assistant (PCM 16 kHz), and enriches the audit log from the Telnyx
Conversations API after the call ends.
New files:
- src/eva/assistant/base_server.py
AbstractAssistantServer: template-method base class that defines the
EVA server contract (FastAPI WebSocket endpoint, audio bridging, tool
execution via ToolExecutor, audit log, audio recording, save_outputs).
Concrete servers implement start() and _shutdown(); the base handles
stop(), output saving, and artifact generation.
- src/eva/assistant/audio_bridge.py
Audio format conversion utilities: mulaw 8 kHz ↔ PCM 16-bit 24 kHz
resampling (libsoxr), PCM mixing, Twilio FrameSerializer protocol
helpers, and framework/metrics log writers.
- src/eva/assistant/telnyx_server.py
TelnyxAssistantServer(AbstractAssistantServer): the main server.
- TelnyxAssistantManager: REST CRUD for benchmark assistants.
- TelnyxDirectSessionConfig: dataclass for WebRTC session params.
- TelnyxVertoJsonRpcClient: Verto JSON-RPC over WebSocket.
- TelnyxWebRTCClient: aiortc RTCPeerConnection + media management.
- TelnyxAssistantServer: bridges EVA Twilio frames to Telnyx WebRTC,
handles conversation events, tool calls, dynamic variables,
audit-log enrichment from Conversations API.
- src/eva/utils/audio_utils.py
save_pcm_as_wav() helper shared by all server implementations.
- tests/unit/assistant/test_telnyx_server.py (20 tests)
- tests/unit/assistant/test_audio_bridge.py (3 tests)
Modified:
- src/eva/models/config.py: Add framework field to RunConfig.
- src/eva/orchestrator/worker.py: Add _get_server_class() dispatch.
- pyproject.toml: Add aiortc, aiohttp, soxr dependencies.
Addresses VSDK-277 review: adds the missing Python EVA wrapper that
subclasses AbstractAssistantServer, registers in worker.py, routes tool
calls through ToolExecutor, and produces all required EVA artifacts.
Add Telnyx hosted AI Assistant as a first-class EVA framework alongside
the existing Pipecat pipeline. The Telnyx provider connects to the Telnyx
AI gateway via direct anonymous WebRTC (Verto JSON-RPC over WebSocket),
bridges audio between the EVA user simulator (Twilio mulaw frames) and the
Telnyx assistant (PCM 16 kHz), and enriches the audit log from the Telnyx
Conversations API after the call ends.
New files:
- src/eva/assistant/base_server.py
AbstractAssistantServer: template-method base class that defines the
EVA server contract (FastAPI WebSocket endpoint, audio bridging, tool
execution via ToolExecutor, audit log, audio recording, save_outputs).
Concrete servers implement start() and _shutdown(); the base handles
stop(), output saving, and artifact generation.
- src/eva/assistant/audio_bridge.py
Audio format conversion utilities: mulaw 8 kHz ↔ PCM 16-bit 24 kHz
resampling (libsoxr), PCM mixing, Twilio FrameSerializer protocol
helpers, and framework/metrics log writers.
- src/eva/assistant/telnyx_server.py
TelnyxAssistantServer(AbstractAssistantServer): the main server.
- TelnyxAssistantManager: REST CRUD for benchmark assistants.
- TelnyxDirectSessionConfig: dataclass for WebRTC session params.
- TelnyxVertoJsonRpcClient: Verto JSON-RPC over WebSocket.
- TelnyxWebRTCClient: aiortc RTCPeerConnection + media management.
- TelnyxAssistantServer: bridges EVA Twilio frames to Telnyx WebRTC,
handles conversation events, tool calls, dynamic variables,
audit-log enrichment from Conversations API.
- src/eva/utils/audio_utils.py
save_pcm_as_wav() helper shared by all server implementations.
- tests/unit/assistant/test_telnyx_server.py (20 tests)
- tests/unit/assistant/test_audio_bridge.py (3 tests)
Modified:
- src/eva/models/config.py: Add framework field to RunConfig.
- src/eva/orchestrator/worker.py: Add _get_server_class() dispatch.
- pyproject.toml: Add aiortc, aiohttp, soxr dependencies.
Addresses VSDK-277 review: adds the missing Python EVA wrapper that
subclasses AbstractAssistantServer, registers in worker.py, routes tool
calls through ToolExecutor, and produces all required EVA artifacts.
Add the TURNS/443 fallback (turns:turn2.telnyx.com:443) to TELNYX_DEFAULT_ICE_SERVERS so the Telnyx EVA server's prod ICE list matches DEFAULT_PROD_ICE_SERVERS in @telnyx/webrtc 2.27.4: STUN + TURN UDP/3478 + TURN TCP/3478 + TURNS on 443. This is the last-resort TLS fallback for networks that block both 3478 transports.
The direct-WebRTC session could establish signaling + ICE + DTLS
(iceConnectionState=completed, connectionState=connected) yet receive
zero remote audio frames, leaving both audio buffers empty and the
conversation silent until the far end idle-disconnected ~90s later — with
no diagnostic trace.
- Log iceConnectionState / connectionState transitions.
- On connectionState failed|closed, set disconnected_event so the session
ends immediately instead of idling.
- In _receive_remote_audio, bound track.recv() with a 10s timeout and warn
("media not flowing", with iceConnectionState + frame count) instead of
blocking silently; log the first received frame and frame count on stop.
Diagnostic/robustness only — does not change the audio bridge behaviour
when media flows.
When s2s_params contains connection_id + webhook_base_url, use TelnyxCallControlTransport (REST + media streaming WebSocket) instead of TelnyxWebRTCClient (aiortc/SRTP). This bypasses the unresolved SRTP unprotect failure in aiortc. Changes: - Add _use_call_control property to TelnyxAssistantServer - Branch in _handle_session: Call Control when params present, WebRTC otherwise (preserves existing behavior) - Move _conversation_event_handler assignment to WebRTC branch only (Call Control transport doesn't use it) - Extend _validate_runtime_config with Call Control credential validation (api_key, connection_id, from_number, to, webhook_base_url) - Add 2 unit tests: Call Control missing credentials, all fields present - Update 2 existing tests to set _webhook_base_url (needed by new property) Blocker #1 from the live benchmark readiness checklist.
…y writes _append_user_events_from_simulator() and _enrich_audit_log_from_external_sources() hardcoded "elevenlabs_events.jsonl". That is upstream's LEGACY artifact name (LEGACY_ELEVENLABS_EVENTS_FILENAME); the user simulator writes "user_simulator_events.jsonl" (user_simulator/base.py). The legacy file therefore never exists at runtime, extract_user_speech_events() always returned [], and the audit log ended up with no user turns at all -- so a graded run would score a conversation missing its entire user side. Use resolve_user_simulator_events_path(), the upstream helper that prefers the neutral filename and still falls back to the legacy one, and warn when neither exists instead of silently producing an empty transcript. The unit test masked this by writing the legacy filename as its fixture; it is now parametrized over both names, so the real pipeline filename is actually exercised. Also sorts the eva.utils/eva.models imports, which ruff flagged.
The direct-WebRTC session reached iceConnectionState=completed and
connectionState=connected, and aiortc logged "DTLS handshake complete" -- yet no
audio moved in either direction. The assistant generated its greeting (visible in
ai_conversation events) but we received 0 RTP frames, and 5s of real speech sent
upstream produced "[long silence]" and no user transcript.
Root cause is the DTLS final flight. OpenSSL's DTLS server answers the client's
Finished with NewSessionTicket + ChangeCipherSpec + Finished, and aiortc never sets
SSL_OP_NO_TICKET. The ticket is ~1.4 kB, so it is fragmented across several DTLS
records. Telnyx's FreeSWITCH does not process the fragmented ticket and therefore
never consumes our Finished: it re-sent its entire final flight 5 times
(Certificate/ClientKeyExchange/CertificateVerify/CCS/Finished), never completed the
handshake, and never derived SRTP keys. OpenSSL considers the handshake complete on
our side, so aiortc reports "connected" while media is silently dead BOTH ways --
FreeSWITCH cannot decrypt our audio and will not send us its own.
Set SSL_OP_NO_TICKET on the DTLS context. Session resumption is meaningless for a
one-shot DTLS-SRTP call and browsers do not offer tickets here either, so the final
flight collapses to an unfragmented CCS + Finished.
Measured against assistant-473093df (Pre-Authorization Support), 20s call:
before after
DTLS datagrams 27 3 (retransmit storm -> clean handshake)
SRTP packets 8-15 814
inbound audio frames 0 790
aiortc exposes no hook for SSL options (RTCConfiguration carries only iceServers and
bundlePolicy), so this patches RTCCertificate._create_ssl_context once, idempotently.
…startup
Running EVA the documented way (uv sync --all-extras; cp .env.example .env;
EVA_FRAMEWORK=telnyx python main.py) surfaced a shutdown race that the direct-transport
tests never hit.
TelnyxWebRTCClient.stop() sets self._peer = None. If the session is torn down while
start() is still awaiting ICE gathering -- which is exactly what happens when the user
simulator fails to come up, e.g. a bad OPENAI_API_KEY for the caller -- start() resumes
and dereferences the now-None peer:
ERROR | eva.assistant.telnyx_server | Telnyx session error:
'NoneType' object has no attribute 'localDescription'
That masks the real failure (the caller never started) behind a confusing AttributeError,
and it fires once per retry attempt.
Check _peer after the ICE-gathering await and exit quietly if stop() already ran: there is
no session left to send an invite for.
INFO | eva.assistant.telnyx_server | Telnyx WebRTC session stopped during startup; skipping invite
…ts a turn
Found by running the EVA CLI properly (EVA_FRAMEWORK=telnyx python main.py) rather than
driving the transport directly: record 1.1 failed with
Record 1.1: Extracted turns - 1 assistant (keys [0]), 0 user (keys [])
Skipping record 1.1: no assistant or user turns found
and audio_user.wav was 151s of digital zero -- the caller never spoke at all.
The user simulator treats EVERY media frame as assistant speech. audio_bridge's
_receive_from_assistant() never inspects the payload: any non-empty frame calls
_on_assistant_audio_start() and refreshes _assistant_audio_ended_time, and the turn is
considered over only after 200 ms with NO frames arriving.
Telnyx's WebRTC leg streams RTP continuously (~50 pps) for the whole call, so between turns
we decode frames of pure silence -- and we forwarded them. The simulator therefore received
a media frame every 20 ms for the entire call and concluded the assistant never stopped
talking: is_assistant_playing() stayed true, _assistant_turn_is_settled() never became true,
the caller's response task waited forever (8x caller_response_coalesced, caller_response_created
never logged), and the record died with zero user turns.
Drop frames whose RMS is at or below ASSISTANT_SILENCE_RMS so the simulator sees a real gap.
EVA CLI, same command, before and after:
before after
extracted turns 1 assistant, 0 user 6 assistant, 6 user
audio_user_clean.wav 0.0s (caller never spoke) 39.9s of real speech
The record still fails, but now for a different and clearly external reason: the assistant
does not receive the caller's audio at all. It re-asks a question the caller has already
answered and falls into its no-input script -- consistent with Telnyx's own dual-channel
recording, where the caller track is digital silence. That is a Telnyx-side media bug
(VSDK-277), not an EVA-contract bug.
…an API key
Separate track from the direct-WebRTC path: reach the AI Assistant via a Call Control
outbound call + Media Streaming WebSocket (bidirectional PCMU) instead of anonymous WebRTC.
No ICE/DTLS/SRTP, so the b2bua-rtc media interop bug (VSDK-277) does not apply. The
TelnyxCallControlTransport and its selection already exist on this branch; this adds the
piece that was missing: turning a bare API key into a working configuration.
eva.assistant.telnyx_provisioning:
- Discovers or creates every account resource the transport needs: a Call Control
Application (the connection, carrying the webhook URL + outbound settings), an idle
phone number for caller ID, and an outbound voice profile.
- Reuse-first and idempotent: reuses an existing profile, an existing eva-media-streaming
connection, and an UNATTACHED active number (so it never disturbs a number already
routing traffic). Re-running is a no-op; a changed public URL re-points the webhook.
- Read-only inspect by default; creation gated behind --provision. Buys nothing.
- Emits the EVA_MODEL__S2S_PARAMS to drop into .env; telnyx_server then selects the
media-streaming transport automatically (_use_call_control keys on connection_id +
webhook_base_url).
Exercised against a live account: inspect correctly reports the missing connection;
--provision created a Call Control app + attached an idle US number + reused the existing
outbound profile, all free.
docs/telnyx_media_streaming.md explains the flow and, in particular, PUBLIC INGRESS: unlike
the outbound-only WebRTC path, Media Streaming requires Telnyx to connect back to EVA (call
webhooks + the media WebSocket), so EVA needs a publicly reachable HTTPS/WSS URL -- a tunnel
(cloudflared/ngrok) for a laptop/locked-down pod, or a real Ingress/LoadBalancer for a
deployment.
5 unit tests (mocked httpx). 211 tests pass, ruff clean.
…reaming)
EVA is one-shot, so instead of asking the user to run cloudflared and paste a URL, the
media-streaming transport can bring up a Cloudflare quick tunnel itself for the duration of
the run and use its public URL as webhook_base_url. Set it via s2s_params:
"webhook_base_url": "auto" (or "auto_tunnel": true).
- telnyx_provisioning/tunnel.py: cloudflare_quick_tunnel(port) async context manager --
launches `cloudflared tunnel --url http://localhost:{port}`, parses the assigned
https://<...>.trycloudflare.com URL from its output, tears the process down on exit.
Quick tunnels need NO Cloudflare account. Raises CloudflaredNotFound if the binary is absent.
- telnyx_server: when webhook_base_url == "auto", after the media server binds its port, start
the tunnel, set webhook_base_url to the tunnel URL, and PATCH the Call Control connection's
webhook to match (the URL is fresh each run). Torn down in _shutdown.
This makes the media-streaming path nearly as turnkey as the WebRTC path: TELNYX_API_KEY +
OPENAI_API_KEY + a provisioned connection_id, then `python main.py` -- no manual tunnel, no
public-URL wrangling. Needs the cloudflared binary on PATH.
3 unit tests (URL parse + missing-binary), 213 pass, ruff clean.
lucasassisrosa
changed the title
lucasassisrosa
changed the title
…uter The audio-judge speech-fidelity metrics (user_speech_fidelity, ...) called self.llm_client.generate_text() first -- i.e. the LiteLLM router, with the audio inline -- and only fell back to google.genai directly when Gemini reported "no audio". But the router has no Gemini deployment in many setups, so the first call dies with "no healthy deployments for gemini-3-flash-preview" and the direct-Gemini fallback never runs. (litellm also mis-transforms audio/file_id messages for Gemini.) The audio judge is Gemini-only, so when GEMINI_API_KEY is set it should go straight to Gemini. Upload the audio and use _generate_with_file() (google.genai SDK) from the first attempt, bypassing the router entirely. Fall back to the router only when no GEMINI_API_KEY is set (preserving behaviour for setups that route Gemini through a gateway). Fixes fully-graded runs (e.g. the telnyx media-streaming benchmark), where the conversation succeeds but user_speech_fidelity failed on the router. Regenerated the metric-signature fixture (transport-only change; scoring is unchanged, no version bump).
…y path + verified graded run
Make the media-streaming path fully turnkey from a bare Telnyx API key: - create_assistant now defaults to true; on each run EVA provisions a fresh assistant from the selected domain's agent config (instructions + tools wired to EVA's own /tools webhook) and deletes it on shutdown. Passing an explicit assistant_id, or create_assistant:false, opts out. - Fold telnyx_provisioning into start() (_auto_provision_resources): discover/ create the Call Control connection + caller-ID number from just the API key, reuse-first and idempotent. No separate provisioning command needed. - Reorder start() so the auto-tunnel comes up before validation and assistant creation (the tool webhook URLs and SIP target are built from the public URL), and derive the SIP 'to' from the created/supplied assistant id. - transport:media_streaming selects Call Control even before connection_id exists (so auto-provision can fill it in). Verified: live EVA CLI run (medical_hr 1.1) against a CLI-generated assistant grades Success 1/1 with task_completion=1.0 (tools executed against the live scenario db). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collaborator
|
Hey Lucas! 👋 Thank you for your PR! We'll take a look soon. When you get a chance, can you please run this? uv run pre-commit run --all-files
git commit --message "Run pre-commit" --no-verify
git pushThis will
|
Author
|
Done — ran |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Telnyx hosted AI Assistant as a first-class EVA framework (
EVA_FRAMEWORK=telnyx), bridging the EVA user simulator to a Telnyx assistant and enriching the audit log from the Conversations API.Two transports are implemented; Media Streaming is the default and recommended path.
Media Streaming (default) — verified working end-to-end
Reaches the assistant via Call Control + a bidirectional PCMU Media Streaming WebSocket — no WebRTC, no ICE/DTLS/SRTP. In a live EVA CLI run the assistant heard the caller and responded to what they said:
audio_user_clean.wav= 15.7s of real speech; EVA extracted 2 user turns (the WebRTC path gets 0).Turnkey — just two keys.
eva.assistant.telnyx_provisioningtakes only aTELNYX_API_KEYand discovers/creates everything else (Call Control connection, caller-ID number, outbound profile) — reuse-first, idempotent. Set"webhook_base_url":"auto"and EVA launches a Cloudflare quick tunnel itself for the one-shot run (no account, no manual tunnel), so Telnyx can reach EVA's media WebSocket + call webhooks. Thenpython main.py.docs/telnyx_media_streaming.mdcovers the flow and public-ingress options (tunnel, in-cluster ingress). A ready-to-run deploy image lives atteam-telnyx/eva-media-streaming.Direct WebRTC (alternate) — known limitation
TelnyxWebRTCClientreaches the assistant with only a publicassistant_id(no API key, no ingress) via anonymous Verto/WebRTC. Currently blocked by a Telnyx b2bua-rtc inbound-media issue (VSDK-277): the assistant does not receive the caller's audio. ICE (aiortc + Pion both fail), signaling, and SDP have all been excluded; a fix ships here already (SSL_OP_NO_TICKETrestores inbound audio). Kept for the id-only use case; use Media Streaming for a reliable graded run.Contract
Subclasses
AbstractAssistantServer, registered in_get_server_class(); serves the Twilio-framedws://localhost:{port}/ws; routes tools throughToolExecutor; writes the audit log, transcript, audio, and scenario DBs.Unit tests for the server, provisioner, and tunnel. Full suite green; ruff clean.
🤖 Generated with Claude Code