Frame metadata: no publish-side API to attach user_data from the JS SDK

[mcuelenaere]

Describe the feature

livekit-client can publish frame-metadata trailers with user_timestamp and frame_id, and (since #1983, shipped in livekit-client@2.20.1) it can now parse a user_data trailer (tag 0x03) on the receive side. However, there is still no way to write user_data on the publish side from the JS SDK.

• FrameMetadataPublishOptions only exposes timestamp and frameId.
• The encode transform (appendPacketTrailerToEncodedFrame in src/frameMetadata/frameMetadata.ts) only ever writes the timestamp + frameId TLVs; there is no plumbing to attach an arbitrary per-frame user_data payload.
• By contrast, the Rust SDK supports this today via VideoFrame.frame_metadata.user_data passed to capture_frame (Add support for frame level user timestamp rust-sdks#890), and the blog post (https://livekit.com/blog/frame-metadata) advertises user_data as a first-class capability across SDKs.

So JS clients can read user_data published by Rust/other SDKs, but a JS publisher cannot emit it.

Use case

Frame-accurate correlation of application events with the published video: attach a small user_data payload (e.g. an event id / tag) to the frame that was live when the event fired, so a downstream recording/analysis pipeline can line up video with application state with no separate sync channel — exactly the "frame-accurate correlation" use case from the frame-metadata blog post.

Today this requires reimplementing the SDK's encoded-frame transform in a custom worker just to add the user_data TLV (reusing the exact LKTS XOR-TLV wire format), which duplicates SDK internals and risks drifting from the wire format across releases.

Proposal

Expose a publish-side user_data mechanism, e.g.:

• a userData flag in FrameMetadataPublishOptions (to advertise the feature on the track), and
• an API to set the user_data for upcoming frame(s), e.g. LocalVideoTrack.setFrameUserData(bytes: Uint8Array) (attach to the next encoded frame, or valid until replaced), mirroring the Rust capture_frame model.

Questions

• Was the publish-side user_data writer intentionally deferred, or is it planned as a follow-up to Add support for user data frame metadata trailer type #1983?
• Is there an issue/PR already tracking this that I missed?

SDK

livekit-client@2.20.1

[mcuelenaere]

#2015 is an attempt at implementing this

[chenosaurus]

@mcuelenaere can you describe the use case? We intentionally did not add metadata publishing from the JS sdk because there was not a way to publish frames directly.

[mcuelenaere]

@chenosaurus we'd like to correlate certain events (eg an user pressing a button or the agent taking a certain action) with the videostream that comes out of the egress. Right now we have to do this by trying to align timestamps of multiple sources, which is brittle and tricky to get right. Attaching in-stream metadata to the video seems like a perfect solution for that problem

[chenosaurus]

@mcuelenaere I see, so precise alignment isn't a requirement, just need it to be able to attach metadata to the latest frame?

[mcuelenaere]

@chenosaurus exactly, it doesn't need frame accuracy. Just within a couple of ms is good enough

[mcuelenaere]

Claude made a fair point: we could enable timestamp: true and link our events to the timestamps of the client machine and store these references out-of-band, that should work for our usecase as well. If you feel like this API is too awkward/too much of a wrong fit for this SDK (given that you can't accurately attach metadata per frame), then we'll just go with that approach instead

[mcuelenaere]

After more investigation, it looks we require changes in livekit/egress and not really in livekit/client-sdk-js, so I've opened an issue there.

If you want, I can close the draft PR here

[1egoman]

@mcuelenaere I see - I will close it for now and we can revisit once there's some more demand for this. Thanks for digging in!