RFC: optional structure-derived timing layer for agent-authored documents

[varo-yang]

I wanted to open this as a discussion rather than a bug report. No renderer changes proposed here.

After wiring HyperFrames into a server-side HTML-to-video pipeline, the failure pattern has been pretty consistent: the renderer is usually fine, but LLM-authored documents keep breaking around timeline bookkeeping.

The common failures are things like same-track overlaps, drift after a local duration edit, captions/audio being slightly out of sync, or adjacent clips whose data-start values are off by a small arithmetic mistake. I suspect anyone running agent-authored HF docs at scale will see a similar distribution.

The track model is great as a renderer/debug target, but as a source format for agents it asks the model to keep a lot of timing state in its head. In practice the model has to:

• keep doing chained arithmetic for data-start
• remember global invariants like "no overlap on this track"
• align clips that may be far apart in the HTML, separated by styling and animation code

That feels like the wrong side of the human/agent split. Humans get a visual timeline; the LLM just gets a long text file full of absolute numbers.

One direction I want to explore is an optional source layer that compiles down to the current track model. The renderer would still consume ordinary HyperFrames output; this would only change the authoring surface.

Rough shape:

<seq>
  <scene dur="3s">
    ...plain HTML / CSS / GSAP, with local t=0 at scene start...
  </scene>

  <scene dur="auto">
    <vo src="vo_02_9c81.mp3">Narration text as inscription.</vo>
    ...scene duration is derived from the probed audio length...
  </scene>

  <scene until="beat:drop">
    ...
  </scene>
</seq>

<layer>watermark.svg</layer>

The goal is not to replace tracks. Tracks would still be the executable artifact and the debugging view. The extra layer would just let authors express the common case structurally, then have a deterministic compiler derive the absolute timing.

So instead of asking the LLM to write:

<div data-start="0" data-duration="3">...</div>
<div data-start="3" data-duration="4.7">...</div>
<div data-start="7.7" data-duration="2.2">...</div>

it can write the sequence and let the compiler do the sums. Same for media-driven durations: dur="auto" can come from local probing with ffprobe. No TTS/network/API behavior needs to live in this layer; media provenance stays out of scope, same as HF already treats assets.

The property I care about is pretty simple: a whole class of timeline errors becomes hard or impossible to write in the source format, while the renderer still gets the exact same kind of track HTML it gets today.

I'm not sure yet whether the richer version (scene / vo / beat anchors / global layers) is worth it, or whether the useful cut is just a minimal seq / par tree. I can test that instead of guessing.

What I'm planning to run is a three-way benchmark on real failing docs from my pipeline:

1. current track HTML
2. a minimal seq / par tree that compiles to tracks
3. the richer screenplay-ish layer above, with media-derived durations

Same prompts, same validation loop, then measure structural error rate, edit success rate, iterations-to-valid, and renderer parity after compilation.

If this direction seems useful, I'll share the harness/results here before proposing any implementation.

cc @miguel-heygen @jrusso1020

[miguel-heygen]

@varo-yang this shouldn't be the right direction, even if the LLMs need to write the data-attributes, I think there is a deeper problem. Feel free to share the harness/results, we need to dig deeper here, my main concern is that this hides something bigger than we expected and we need to fix.