QVAC-21981 feat: ABot-World interactive walk sessions + native scene creation - #22
Open
DmitryMalishev wants to merge 17 commits into
Open
QVAC-21981 feat: ABot-World interactive walk sessions + native scene creation#22DmitryMalishev wants to merge 17 commits into
DmitryMalishev wants to merge 17 commits into
Conversation
In Resample::forward (upsample3d), the first latent-frame chunk skipped time_conv entirely and never seeded the temporal feat cache, so the second chunk ran time_conv against zero padding instead of chunk-0 context. The reference implementation (Wan vae2_2.py, "Rep" cache semantics) runs time_conv with causal zero padding on chunk 0, doubles the frames, trims the first one, and stores the input tail in the cache for the next chunk. Effect on Wan2.2 VAE decode vs the PyTorch reference (5-frame 480x832, CPU backend, f16 weights): cosine 0.9959 / 27.2 dB PSNR before, cosine 1.000000 / 79.0 dB after. Output was visually near-identical, so this only shows up in numeric parity testing. Encode and TAEHV paths are unaffected.
…ction adapter) ABot-World-0-5B-LF (acvlab, Apache-2.0) is a causal/interactive derivative of Wan2.2-TI2V-5B with a keyboard-action conditioning adapter. This adds: - VERSION_ABOT_WORLD: detected via the act_control_adapter.conv.weight tensor; helpers sd_version_is_abot_world() / sd_version_is_wan_ti2v_family() (the latter now gates the shared TI2V 48ch/16x latent-space branches in vae.hpp, tae.hpp and stable-diffusion.cpp instead of == VERSION_WAN2_2_TI2V) - WAN::ActControlAdapter block (SimpleAdapter in the reference impl: conv 2x2/s2 over pixel-unshuffled action planes + one ReLU residual block) registered by WAN::Wan when WanParams.abot_world is set, so all 831 checkpoint tensors (F16/Q8_0 GGUF or safetensors) map and load - WanRunner: ABot-World-5B config (Wan2.2-TI2V-5B dims + abot_world flag) - generate_video(): explicit, documented rejection for ABot-World models -- the causal interactive session (KV cache, per-block actions, 4-step distilled schedule) lands in a follow-up; running these weights through the batch path executes the wrong recipe by design No behavior change for any existing model version: every new branch is gated on VERSION_ABOT_WORLD, and sd_version_is_wan_ti2v_family(v) == (v == VERSION_WAN2_2_TI2V) for all previously existing versions. Verified: ABot Q8_0 GGUF (5.86 GB, 831 tensors incl. adapter) detects as "ABot-World", instantiates ABot-World-5B, and completes tensor loading; generation attempt fails fast with the explanatory error.
sd-abot-decode: a native, Python-free example that loads the taehv (taew2_2)
decoder GGUF and a fixed-scene "walk" of precomputed latents (raw f32), decodes
them on the ggml engine, and writes a video. This proves the native taehv
decode path and the S3 -> GPU-runner -> build -> video -> validate CI pipeline
end to end; the DiT causal denoise replaces the "load latents" input in the
follow-up (docs/abot_world_mvp_spec.md).
- examples/abot-decode/{main.cpp,CMakeLists.txt}: standalone decoder (mirrors
the upscaler.cpp standalone-runner pattern); loads TinyVideoAutoEncoder for
VERSION_ABOT_WORLD, decodes video latents, writes MJPG-AVI.
- examples/abot-decode/data/walk_latents.{bin,json}: 12-frame golden latent
walk (from the reference pipeline) as the CI decode input.
- .github/workflows/build.yml: temporary abot-walk-mvp-gpu job on
qvac-ubuntu2204-x64-gpu — Vulkan build, pull models from S3 (tether-ai-dev via
OIDC), run the decode, validate the video, upload it as an artifact. Appended
to the existing workflow so a feature-branch PR can trigger it; to be split
into abot-walk-mvp.yml at merge.
Builds locally (MinGW, CPU). GPU/S3 path validated on CI.
GPU testing of ABot moves to the diffusion addon (monorepo) via a vcpkg overlay port, so the engine PR no longer needs the fork-side GPU workflow or the standalone taehv-decode example: - revert build.yml to the pristine matrix (drop the abot-walk-mvp-gpu job that had no available self-hosted runner in this fork) - delete .github/workflows/abot-walk-mvp.yml scaffold - delete examples/abot-decode/ (native taehv decode + committed latent fixture) This PR is now scoped to engine support only: recognize + load ABot-World (VERSION_ABOT_WORLD, action-adapter block, TI2V loading) with batch generate_video() guarded. Docs (docs/abot_world*.md) and the load-validation script (script/validate_abot_world.sh) are kept.
DmitryMalishev
added a commit
to tetherto/qvac
that referenced
this pull request
Jul 20, 2026
…test Builds the diffusion addon against the unmerged sd.cpp PR tetherto/qvac-ext-stable-diffusion.cpp#22 (ABot-World support) using a vcpkg overlay port, so ABot can be GPU-tested through the addon's existing integration CI without merging the engine PR or bumping the shared registry. - vcpkg/overlay-ports/stable-diffusion-cpp: verbatim copy of the registry port (2026-07-03, port-version 5) with REF + SHA512 repinned to PR #22's head (da3f6b26). vcpkg-configuration.json gains overlay-ports so it wins over the registry port. Temporary — removed when the engine PR merges + the registry port is bumped. - scripts/download-model-abot.sh: fetch the ABot GGUF set (DiT q8_0/f16, wan2.2 VAE, taew2_2) from corp S3 (tether-ai-dev, OIDC) with SHA256 check. - test/integration/abot-world.test.js: skip-guarded. Asserts the ABot set loads via the addon and that batch generate is rejected with the documented "interactive session not implemented" error (ABot is causal/interactive, not one-shot). Auto-discovered by test:integration, so it runs on the existing qvac-*-gpu integration workflow. Replaced by a real walk assertion when the causal core lands.
Drop the internal implementation-planning spec (belongs with the consuming project, not this public repo) and remove a reference to non-public tooling from abot_world.md. No code changes.
olyasir
reviewed
Jul 21, 2026
| *num_frames_out = 0; | ||
| } | ||
| int64_t t0 = ggml_time_ms(); | ||
| if (sd_version_is_abot_world(sd_ctx->sd->version)) { |
There was a problem hiding this comment.
This fail-fast guard only covers one of the two doors into generation, and the capability layer disagrees with it.
1. generate_image() is unguarded. sd_version_is_wan() was (correctly) extended to include VERSION_ABOT_WORLD, so ABot travels through all the shared Wan latent/VAE/sampling paths used by both generate_video() and generate_image()/generate_image_internal(). But the rejection was added only here. A caller that loads an ABot GGUF and calls generate_image() runs the bidirectional/one-shot Wan recipe on a causal model with no check at all — silently corrupted output, or a raw GGML_ASSERT abort if it hits a shape assumption that was never true for ABot. That's exactly the hazard this guard exists to prevent.
2. The capability query now lies. sd_version_supports_video_generation() (line 3104) is ... || sd_version_is_wan(version) || ..., so it now reports ABot as video-capable. The example server's /vid_gen route (examples/server/routes_sdcpp.cpp) uses that query to reject unsupported models cleanly before reaching the engine — it will instead wave ABot through to this return false, bypassing the intended clean-rejection path.
Suggested fix: reject in one shared choke point instead of per-entrypoint, and keep the capability query honest:
- exclude ABot from
sd_version_supports_video_generation()(and any image-capability query), e.g.... && !sd_version_is_abot_world(version); - move the actual rejection to a point both entrypoints pass through (
GenerationRequest/SamplePlan::resolve()), or reject atnew_sd_ctx()load time when the version isVERSION_ABOT_WORLDand no causal-session API was requested.
Loading still works; generation is then refused everywhere with the same message, and the server pre-screen agrees with the engine.
Author
There was a problem hiding this comment.
Both points confirmed and fixed in 23d8fd5 — thanks, this was a real gap.
- Shared choke point: the rejection moved into
GenerationRequest::resolve()(same pattern asvalidate_ideogram4_uncond_model), sogenerate_image()andgenerate_video()are both covered by one check; the per-entrypoint guard here was removed. Message now names both entrypoints. - Honest capability queries:
sd_version_supports_video_generation()andsd_version_supports_image_generation()both returnfalsefor ABot (image could not simply stay!video, or it would have flipped totrue). script/validate_abot_world.shgrew animg_genlane asserting the image path rejects too; verified locally against the ABot Q8_0 GGUF (detection + full load unchanged, both batch paths reject, stock-Wan lane unaffected).
On load-time rejection: kept loading permitted intentionally — downstream (tetherto/qvac#3352) exercises load on GPU CI ahead of the causal session API, and the capability queries now give front-ends the clean pre-screen.
…apability queries Review feedback (#22): the fail-fast guard covered only generate_video(), while generate_image() ran the batch recipe on the causal model unchecked, and sd_version_supports_video_generation() still reported ABot as video-capable, so front-ends pre-screening on the capability query would wave it through. - Move the rejection into GenerationRequest::resolve() (the stage both generate_image() and generate_video() pass through, same pattern as validate_ideogram4_uncond_model): one check now covers every batch door. Message updated to name both entrypoints. - sd_version_supports_video_generation() and sd_version_supports_image_generation() both return false for ABot-World (image could not simply mirror !video, or it would flip to true). - Drop the now-redundant guard in generate_video(). - script/validate_abot_world.sh: assert the image path is rejected too; docs/abot_world.md wording updated. Verified locally against the ABot Q8_0 GGUF: detection + full tensor load unchanged; vid_gen and img_gen both reject with the shared message; stock-Wan regression lane unchanged.
aegioscy
approved these changes
Jul 21, 2026
Implements the actual interactive walk for ABot-World models (the batch
generate_image()/generate_video() APIs keep rejecting them):
- src/abot_world.hpp: causal walk core using the recompute formulation --
each denoise step re-derives attention over [ref tokens | clean history
| current block] with a per-row trailing-window mask, reproducing the
reference KV cache exactly (including eviction). Absolute-counter RoPE
with negative-time reference ids; per-frame timestep modulation via a
row-indexed e0 table; PixelUnshuffle'd keyboard-action planes through
the act_control_adapter. Includes a minimal scene-pack reader
(safetensors F32), a seed-stable RNG, and AbotWalkSession combining the
4-step distilled flow-matching block loop with taehv pixel decoding
(stateless per-block decode with a 3-latent-frame overlap standing in
for the reference's streaming decoder cache).
- include/stable-diffusion.h, src/stable-diffusion.cpp: public C API --
sd_abot_session_{params_init,new,step,frames_free,free}. A session is
standalone (own DiT + taehv + scene pack, no sd_ctx_t); step() takes an
8-key action mask and returns the next block's decoded RGB frames.
- src/wan.hpp: WanAttentionBlock::forward takes an optional additive
self-attention mask (defaulted, no behavior change for existing paths).
- src/tae.hpp: patchify/unpatchify marked inline (they now live in a
header included from more than one translation unit).
Validation vs the PyTorch reference (F16 GGUF, coastal scene goldens):
- Single-step parity across every regime -- first block, all timesteps,
history recompute, window onset, deep eviction: 7/7 gates at cosine
0.9996-0.9999 (gate 0.999).
- Native taehv decode vs reference streaming decode on identical latents:
PSNR mean 46.4 dB, min 33.5 dB.
Three small executables used to prove the causal walk core against the PyTorch reference and to exercise the public session C API end to end: - examples/abot-parity: runs one golden denoise step (dumped by the reference implementation) through the causal forward and writes the block x0 prediction for offline cosine comparison. - examples/abot-walk: full block-loop walk over a fixed scene; replays the reference's recorded noise for deterministic full-chain validation against the golden per-block final latents, or free-walks with a seeded RNG. Emits raw latents. - examples/abot-session: drives the public sd_abot_session_* C API for a fixed-scene walk, writing decoded frames as PNGs; --mode decode is a taehv-only lane for PSNR cross-checks against the reference decoder on identical latents.
The abot-parity/walk/session examples link against engine internals (ModelLoader, SDBackendManager, the runner classes) that are not exported from the shared library, so -DSD_BUILD_SHARED_LIBS=ON lanes failed to link them. Gate the three harnesses to static builds; the public sd_abot_session_* C API itself is exported like every other sd_* symbol and is unaffected.
RTX 5090 (32 GiB, sm_120) validation of the causal walk exposed unbounded
per-block growth: every denoise graph carried the full walk history (the
attention window was enforced only by masking), so VRAM grew ~1.3 GiB and
per-block time ~1.6 s per block until CUDA OOM'd at walk block 5
(20.1 -> 29.1 GiB measured at 832x480 F16).
Bounded history (default): each block's graph now carries the refs, walk
block 0 (pinned - ref rows re-derive their K/V against it and its own window
is refs + itself, so it stays exact), the trailing local_attn_size frames
(the current block's exact window), and the current block. RoPE ids and the
mask switch to absolute frame ids so every attended pairwise relation is
unchanged; for walks <= window + fpb frames the kept set equals the full
history, so the golden/parity regime is bit-identical. history_keep < 0
restores the previous full-recompute behavior.
Per-layer KV cache (opt-in, ABOT_KV_CACHE=1): roped K / per-token V of
refs+block0 ("base") and a 5-frame ring (window - fpb) are captured once per
finalized block via the runner's named-cache buffers; denoise graphs then
carry only the current block's rows against the cached context, and a
context pass at t=0 appends the finalized block (reference cache-append
semantics). The cache-append pass overlaps with the taehv decode on a
worker thread; with taehv on a second GPU (backend spec
"diffusion=cuda0,vae=cuda1") measured steady state is 1.9 s/block, flat,
16/16 blocks (previously 8.5 s/block growing, OOM at 5).
Validation on RTX 5090 CUDA (both paths):
- parity 7/7 cases >= 0.9995 (gate 0.999), recompute path bit-identical to
the unpatched engine
- golden-replay walk chain: recompute 0.99989/0.99872/0.99708/0.99201;
KV cache 0.99989/0.99866/0.99689/0.99226 (gate 0.99)
- sd-abot-session gains --mode walkval (--golden replay via noise_override,
--latents-out for compare_walk.py) to gate any walk path end to end
Also adds an opt-in ABOT_FLASH_ATTN=1 toggle: the masked flash-attention
path is ~23% faster but currently produces wrong results for any block with
history (block 0 passes, history blocks collapse to cosine ~0.2 vs goldens
on CUDA) - exposed for debugging the flash mask handling only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two independent defects in the opt-in ABOT_KV_CACHE=1 path, found while validating PR #22 on Apple Silicon / Metal (platform 2): 1. Captured K/V tensors were graph intermediates that the attention itself also consumes, so the graph allocator could reuse their memory before the post-compute cache persist read them. CUDA's allocation layout happened to keep the bytes intact; on Metal and CPU every block with history collapsed to cosine ~0.25 vs goldens. Materialize view captures and pin every capture as a graph output. No computed value changes on any backend; the plain path and parity outputs are bit-identical pre/post fix. 2. The taehv-decode / cache-append overlap ran two graphs concurrently on ONE ggml backend instance when DiT and taehv share a device (single-GPU Metal), wedging Metal's command queue (command buffer stuck in Scheduled, backend poisoned). ggml backends are not thread-safe; serialize the overlap unless the two modules run on distinct backend instances. The dual-GPU overlap config (diffusion=cuda0,vae=cuda1) is unchanged. Validated on Apple M3 Ultra 96 GB (Metal): KV walkval golden gate PASS (0.99995 / 0.99897 / 0.99736 / 0.99294), 16-block walk 10.7 s/block steady vs ~35 s plain, memory flat at ~30 GiB, frames coherent. The pre-fix failure also reproduced on the CPU backend, confirming the capture bug was latent on all backends. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… lookup) GGMLBlock::init appends "." to a non-empty prefix; the ABot runner and the parity/walk harnesses passed "model.diffusion_model." WITH a trailing dot, so every init_params get_type lookup used a "model.diffusion_model.." key, missed the tensor storage map, and fell back to GGML_TYPE_F32. The loader (which composes names itself) still found and converted every tensor, so loading "worked" - but the whole 5B DiT ran with F32 weights: - 19.7 GiB of weights in VRAM instead of 9.8 GiB (session peak 30.2 -> 20.7) - F32 GEMM paths instead of F16 tensor-core paths (KV-cache walk pass 320 ms -> 244 ms; steady block 1.9 s -> 1.5 s on RTX 5090) - quantized GGUFs silently dequantized to F32, so Q8_0 changed neither VRAM nor speed Measured after the fix (RTX 5090, 832x480, window 8): parity 7/7 >= 0.9995, golden-replay chain PASS on both walk paths with block-3 cosine improving 0.992 -> 0.9968 (true-F16 weights match the goldens' F16 provenance more closely than the F32 conversions did). Also adds opt-in ABOT_PROF=1 per-phase timing (host prep / graph compute / taehv decode) used to find this. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…first frame) Scene packs no longer need offline PyTorch extraction: sd_abot_scene_create builds one on-device from a prompt and a first-frame image. - prompt -> umT5-XXL embeddings via T5CLIPEmbedder (umt5 tokenizer, seq 512, attention-masked, padding rows zeroed) matching the reference WanTextEncoder semantics (u[v:] = 0) - image -> PIL ImageOps.fit-equivalent aspect crop + antialiased Catmull-Rom resize (stb) -> Wan2.2 VAE encode -> (z - mean) / std diffusion latents - zero-filled reference slots + safetensors writer (AbotSceneWriter) - sd-abot-session gains --mode create-scene Validated against the reference coastal pack (F16 models, CPU backend): prompt_embeds cosine 0.9973, first_frame_latents cosine 0.9987 (gate 0.995). Encoders load standalone and are freed before returning; the walk session path is untouched. Co-Authored-By: Claude <noreply@anthropic.com>
…int-limited sd_abot_scene_create now accepts a null init_image: the pack is written with a zero first-frame latent and a new first_frame_mask tensor ([1,1], 0 = no first frame; missing = 1 for full back-compat), and the walk skips the three block-0 pinning sites for such packs so frame 0 is generated from noise under the prompt alone (TI2V-style). sd-abot-session --mode create-scene makes --image optional (--vae required only alongside an image). Validated on RTX 5090 (2026-07): the plumbing works end to end, but the distilled ci2v checkpoint cannot bootstrap a coherent first frame in its 4 distilled steps - text-only scenes render as degenerate, non-navigable texture that does not recover over blocks. The capability is therefore kept gated (front-ends should require an image) and becomes useful only with a T2V-capable checkpoint; the scene format change is forward-compatible. Also documents validated image-input behavior at the create API: stb decode is magic-byte based (extension irrelevant; JPEG/PNG reliable, no WebP/AVIF/HEIC), EXIF orientation is not honored, alpha is dropped, and arbitrary input resolutions are cover-scaled + center-cropped (small inputs upscale softly; extreme aspect ratios lose the periphery). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review-pass cleanup, no functional changes: - docs/abot_world.md: rewritten to describe what actually shipped - the interactive session C API, the opt-in KV cache, native scene creation, and the gated text-only pack support - instead of the pre-session "generation not yet supported" status. Adds the model-files table. - stable-diffusion.cpp: the batch-rejection error and capability-query comment no longer claim the session API "is not implemented yet"; they point at sd_abot_session_* (the validate script's grep contract is unchanged). The text-only branch of sd_abot_scene_create is restructured into a properly indented if/else (was a non-reindented wrap ending in a confusing double brace), and the completion log now says "first frame zeroed - text-only scene" instead of the misleading "encoded". - stable-diffusion.h: sd_abot_scene_params_t documents the optional init_image/vae_path (text-only pack semantics and its gating). - validate_abot_world.sh: comment updated for the landed session API. Validated after the restructure (CPU, F16 encoders): scene pack vs the golden extraction - prompt_embeds cosine 0.9973, first_frame_latents 0.9987, SCENE PASS (identical to the pre-restructure numbers); text-only pack writes prompt embeds + zeroed latents + first_frame_mask=0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DmitryMalishev
changed the title
DmitryMalishev
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds full ABot-World support to the engine: model loading, the interactive walk session, and native scene creation. ABot-World-0-5B-LF (Apache-2.0) is a causal, interactive derivative of Wan2.2-TI2V-5B driven block-by-block by keyboard actions. See
docs/abot_world.mdfor the feature overview.Increments, in review order:
VERSION_ABOT_WORLDdetection (via theact_control_adapter.conv.weighttensor), theActControlAdapterblock (keyboard-action conditioning), Wan2.2-TI2V-5B-family loading, and a guard that rejects ABot models in the batchgenerate_image()/generate_video()APIs at the sharedGenerationRequest::resolve()choke point. Both capability queries reportfalsefor ABot.src/abot_world.hpp+ session C API): the causal walk core — recompute formulation with a per-row trailing-window mask that reproduces the reference KV-cache semantics exactly (action adapter, zero-masked reference tokens, negative-time RoPE ids, per-frame timestep modulation, 4-step distilled flow matching, taehv per-block decode with 3-latent-frame overlap). Public API:sd_abot_session_{params_init,new,step,frames_free,free};step(action_mask)(bits 0..7 = W,A,S,D,I,J,K,L) returns the next block's decoded RGB frames.ABOT_KV_CACHE=1): history K/V captured once per finalized block into per-layer base/ring tensors instead of recomputed every denoise step (~3.7x fewer frame-passes per block in steady state); walk-graph memory bounded by the attention window. Correct on shared/single-GPU backends (decode/append overlap serialized when DiT and taehv share a backend).sd_abot_scene_create+sd-abot-session --mode create-scene): umT5-XXL prompt encode (masked, padding rows zeroed) + Wan2.2 VAE first-frame encode (PIL-fit-equivalent aspect crop + antialiased Catmull-Rom resize) + zero-filled reference slots, written as the scene-pack safetensors the session loads. Replaces the reference implementation's offline PyTorch extraction.first_frame_mask): the format and walk support packs without a first-frame image (block 0 from noise), but the distilled checkpoint cannot bootstrap a coherent first frame, so the capability is gated and documented; front-ends should require an image. Back-compatible: packs without the mask tensor behave exactly as before.Validation (vs the PyTorch reference)
Regression safety
Effective changes to existing pipelines at merge time are additive only (verified with a two-dot diff against the base — the Wan VAE first-chunk fix visible in the three-dot view is already on the base branch via #21):
VERSION_ABOT_WORLDmodels or through the newsd_abot_*entry points.sd_version_is_wan_ti2v_family()refactorsVERSION_WAN2_2_TI2Vchecks; behavior for existing versions is bit-identical (pure extension to ABot).attn_mask/context_img_lenparameter onWanAttentionBlock::forward;inlineon two free functions intae.hpp(ODR fix).script/validate_abot_world.sh, which also (optionally) verifies a stock Wan model still detects and generates unchanged.abot-parity/abot-walk/abot-sessionharnesses build only for static libs (they link engine internals the shared library does not export).