StreamArena Evaluation Toolkit

Repository layout

StreamArena_code/
├── streamarena/           # shared library
│   ├── data.py            # dataset loader + EvalRecord/EvalOutput helpers
│   ├── tools.py           # Config, Frame, FrameBuffer, Serper ToolBox
│   ├── media.py           # frame extraction + subtitles + audio slicing + clip helpers
│   ├── protocols.py       # tool-call protocol parsers, thinking splitter
│   ├── backends.py        # OpenAIBackend + GeminiBackend
│   ├── runner.py          # unified multi-turn runner (offline backends)
│   └── judge.py           # LLM-as-Judge scorer
├── method/                # one folder per evaluated method
│   ├── offline/           # Qwen / MiMo / Kimi / Qwen-Omni / Gemini    (available)
│   ├── streammind/        # Streaming eval driver + StreamingAgent base class (driver available; paper's agent coming soon)
│   ├── aura/              # AURA fixed-window streamer                 (client only; server = official vLLM)
│   ├── minicpm/           # MiniCPM-o-4.5 unified                      (available; HF weights)
│   ├── vst/               # VST text-summary streamer                  (available; HF weights)
│   ├── streamforest/      # StreamForest native streamer               (available; llava/ vendored)
│   └── thinkstream/       # ThinkStream native streamer                (available; thinkstream/ vendored)
├── judge/                 # universal LLM-as-Judge scorer              (available)
├── requirements.txt
├── LICENSE
└── README.md

Every method writes the same records JSONL schema (see method/offline/README.md for field details), so the scorer in judge/ treats them all identically.

Methods status

Folder	Description	Status
method/offline/	Turn-based MLLMs (Qwen / MiMo / Kimi / Qwen-Omni / Gemini) with the Serper agentic loop	Available
method/streammind/	Streaming eval driver: wall-clock 2 fps frame push + non-blocking ask / e-watch events + JSONL writer. Subclass StreamingAgent to plug in your own agent. Paper's StreamMind agent coming soon.	Driver available; agent coming soon
method/aura/	AURA fixed-window streamer; client only, vLLM server per aurateam/AURA	Available
method/minicpm/	MiniCPM-o-4.5 unified; weights openbmb/MiniCPM-o-4_5	Available
method/vst/	VST text-summary streamer; weights Catalan258/VST-7B	Available
method/streamforest/	StreamForest native streamer; llava/ vendored from MCG-NJU/StreamForest (Apache-2.0)	Available
method/thinkstream/	ThinkStream native streamer; thinkstream/ vendored from CASIA-IVA-Lab/ThinkStream (MIT)	Available
judge/	LLM-as-Judge, shared across every method above	Available

Paper highlights

• 243 full-length videos, average duration 88.8 min, total ~300 GB.
• 3,646 timestamped open-ended questions across four capabilities: Real-Time Perception (RTP), Historical Retrospection (HR), External Tool Use (Tool), Proactive Interaction (Pro).
• HR / Pro are stratified into horizon buckets up to >30 min for HR and >4 min for Pro.
• Two independent annotators + one blind auditor removed ~27% of drafts, leaving 3,646 validated tasks.
• StreamMind is the only streaming method evaluated on all four capabilities and reduces pooled query-to-answer latency by 66.2% on the same Qwen3.5-397B-A17B backbone.

Installation

Python 3.10+, FFmpeg, and enough disk space for the videos you want.

cd StreamArena_code
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

1. Question annotations (always required)

The question set is a plain JSONL at the root of the dataset repo. Grab whichever language you need (English by default):

hf download hkuzxc/StreamArena \
  --repo-type dataset \
  --include "question.en.jsonl" \
  --local-dir ./data/StreamArena
# for Chinese: --include "question.jsonl"

Skip this step if you plan to pass the Hugging Face id directly (DATASET=hkuzxc/StreamArena) -- the loader will fetch the JSONL for you on first use.

2. Videos

One video (smoke test, ~1--2 GB):

hf download hkuzxc/StreamArena \
  --repo-type dataset \
  --include "videos/-J3qSQ2z4Nc.tar" \
  --local-dir ./data/StreamArena

Full ~300 GB bundle:

hf download hkuzxc/StreamArena \
  --repo-type dataset \
  --include "videos/*.tar" \
  --local-dir ./data/StreamArena

3. Extract video tars

Each videos/<video_id>.tar bundles that video's .mp4 and any subtitles. You have two options:

Option A -- do nothing. The runner auto-extracts a tar the first time it needs the video (see _extract_tar_if_needed in streamarena/data.py). Grabbing one tar is enough for a smoke test.

Option B -- pre-extract everything. Handy when you want the full benchmark ready on a shared disk before starting a batch run:

cd ./data/StreamArena/videos
for f in *.tar; do
  tar -xf "$f" && rm "$f"   # drop `&& rm "$f"` if you want to keep the tars
done
cd -

After extraction each video sits at ./data/StreamArena/videos/<video_id>/<video_id>.mp4 alongside its .vtt subtitles, which is the layout the runner expects.

Environment variables

Variable	Purpose
GEMINI_API_URL / GEMINI_API_KEY	Gemini endpoint + key (used by offline Gemini backend and the judge)
SERPER_API_KEY	Serper API key for text_search / image_search
JUDGE_URL / JUDGE_API_KEY	Judge endpoint + key
STREAMARENA_DATASET	Default dataset id / local path
STREAMARENA_VIDEO_DIR	Default video directory or tar archive folder
STREAMARENA_LANGUAGE	en or zh (default en)

End-to-end example (Qwen + auto-judge)

Two ways to test the Qwen backbones. The paper numbers come from option A; option B is provided for convenience when local GPUs are not available.

A) Local vLLM (the default; what the paper reports). Stand up Qwen3.5-397B-A17B behind an OpenAI-compatible endpoint following https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3.5-397B-A17B.html, then point the driver at it:

export SERPER_API_KEY=...
export JUDGE_URL="https://generativelanguage.googleapis.com/v1beta/models/{model}:generateContent"
export JUDGE_API_KEY=...

AUTO_JUDGE=1 \
BASE_URL=http://localhost:12347/v1 MODEL=qwen3.5 \
VIDEO_DIR=./data/StreamArena/videos \
bash method/offline/scripts/run_qwen.sh

All open-source backbones we report on (Qwen / MiMo / Kimi / MiniCPM / VST / StreamForest / ThinkStream / AURA) are served locally through vLLM or SGLang in the paper.

B) Qwen Official API (Alibaba Cloud, OpenAI-compatible mode). If you don't have GPUs, subscribe to a Qwen model at https://qwen.ai/apiplatform, then:

BASE_URL=https://dashscope.aliyuncs.com/compatible-mode/v1 \
API_KEY="sk-..." \
MODEL=qwen3-max-preview \
VIDEO_DIR=./data/StreamArena/videos \
bash method/offline/scripts/run_qwen.sh

The hosted API has its own model ids, rate limits, and request-schema quirks. Consult the official docs at https://qwen.ai/apiplatform and adjust the driver accordingly if something 4xx's -- we do not track hosted-API compatibility here.

Run + score separately:

bash method/offline/scripts/run_qwen.sh
bash judge/scripts/run_judge.sh method/offline/results/offline_qwen_qwen3.5.jsonl

Citation

@misc{zhang2026streamarenacontinuousinteractivelonghorizon,
  title={StreamArena: Toward Continuous, Interactive, and Long-Horizon Agentic Streaming Video Understanding},
  author={Xichen Zhang and Guankai Li and Yinghao Zhu and Shijian Wang and Sitong Wu and Shaozuo Yu and Meng Chu and Yuan Lu and Jiaya Jia},
  year={2026},
  eprint={2608.05703},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2608.05703}
}

Licenses

Evaluation code is Apache-2.0. StreamArena annotations are CC-BY-NC-4.0. Video copyrights remain with the original YouTube uploaders; consult the dataset card before redistribution or commercial use.

Third-party notices

Portions of two upstream baselines are vendored inside this repository under method/*/_vendor/, together with their original LICENSE:

• method/streamforest/_vendor/llava/ is the llava/ package from MCG-NJU/StreamForest (Apache-2.0). See method/streamforest/_vendor/LICENSE.
• method/thinkstream/_vendor/thinkstream/ is the thinkstream/ package from CASIA-IVA-Lab/ThinkStream (MIT). See method/thinkstream/_vendor/LICENSE.

AURA, MiniCPM-o-4.5, and VST are used through their published model checkpoints only; no code is vendored. See each folder's README for setup instructions and links to the upstream repos / weights.