feat(images): paste a screenshot, ask about it

[ndemianc]

Implements docs/IMAGES.md — local, session-attached, no upload. Slices I1–I5 of seven.

The cap is measured, not chosen

The plan argued 1568px from first principles. Claude Code's transcripts are on disk, so I read them instead of guessing: 24 images, every re-encoded one exactly 2000px on the long edge. That's the threshold the vision docs name for staying clear of the stricter per-image limit above 20 images per request — the largest size that is never unsafe, and above both tier caps, so the server does the final downscale and we never discard fidelity it would have kept.

The same transcripts confirmed a rule I'd derived rather than observed: under the cap, pass through untouched in the original format; only oversize images are resized and re-encoded to WebP. Re-encoding what didn't need resizing only stacks artifacts, worst on screenshots of code.

Verified in a real browser against real images:

case	source	sent
4K screenshot	3840×2160 png 764KB	2000×1125 webp 115KB	resized
retina window	3024×1964 png 654KB	2000×1299 webp 137KB	resized
small PNG	1160×480 png 121KB	1160×480 png 121KB	pass-through
under-cap JPEG	1600×900 jpg 208KB	1600×900 jpg 208KB	pass-through
wide panorama	2400×600 png 178KB	2000×500 webp 51KB	resized
TIFF		refused, with what to do instead

Aspect ratios come out exact.

Slices

I1 — the wire seam, and a bug that predates images. toOpenAIMessages fell through on any block it didn't recognise, so an unknown type vanished between composer and wire with no error and no log line — the model would answer confidently about content it was never sent. Now a throw naming the type. Images map to image_url; a Files-API source is refused rather than sent as something.

I2 — the vision gate, deliberately stricter than the tools gate: unknown model means no vision, because attaching to a blind model costs a composed message while a disabled button costs a click.

I3 — imageCost.js. The formula I had in mind (w*h/750) was stale. It's ⌈w/28⌉ × ⌈h/28⌉ over 28px patches with per-tier caps. Doc-parity test reproduces every worked example in the vision docs on both tiers.

I4 — local store + the meter. Content-addressed under the session's media/, tmp+rename, 5MB cap, and isRef pins the shape so ../../etc/passwd is not a ref. The meter now counts images by real visual cost — a test pins that the storage shape doesn't move the number.

I5 — paste, drop, chips, refusal. The refusal runs at attach time, not send time, because the composer clears on send. canSeeImages ships from both config paths — a test pins that, since one silently allowing is the kind of half-gate that passes review.

Why a media/ dir rather than Claude Code's inline base64

Claude Code inlines bytes in its JSONL and that's fine there. It isn't here, for a reason specific to this codebase: sessionStore.scanProject does readFileSync + JSON.parse on every session file whenever index.json is missing, malformed, or on an older schema — first run, and after any schema bump. Inlined bytes would make drawing a list of session titles parse every screenshot in every session.

Not in this PR

I6 — the Add Files picker doesn't yet treat an image as an image. I7 — the remaining refusals (too many images, oversize at the host boundary) are partly in place via the store's caps but not surfaced as first-class messages.

37 suites green; 29 bypasses verified across the five slices.

[Copilot]

Pull request overview

Adds local screenshot paste/drop support for LevelCode AI, including image normalization, storage, vision gating, provider translation, and token accounting.

Changes:

• Adds paste/drop attachment UI with resizing and validation.
• Stores images content-addressably under session media.
• Adds image-aware provider integration, capability checks, accounting, and tests.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Final review findings
extensions/levelcode-ai/test/translate.test.js	No final comments.
extensions/levelcode-ai/test/imageStore.test.js	No final comments.
extensions/levelcode-ai/test/imageCost.test.js	No final comments.
extensions/levelcode-ai/test/imageAttach.test.js	No final comments.
extensions/levelcode-ai/sessions.js	Moderate (2 votes): Add media lifecycle/GC or accurately document retention and storage scope.
extensions/levelcode-ai/providers/translate.js	Moderate (2 votes): Unsupported assistant content blocks can still be silently dropped.
extensions/levelcode-ai/providers/catalog.js	Moderate (2 votes): Vision gating ignores provider-level capabilities for custom endpoints.
extensions/levelcode-ai/media/chat.html	Moderate (2 votes each): Image removal is overridden by the generic chip handler; reset does not clear images; asynchronous attachment normalization can race with sending.
extensions/levelcode-ai/imageStore.js	No final comments.
extensions/levelcode-ai/imageCost.js	No final comments.
extensions/levelcode-ai/extension.js	Critical (2 votes): OpenAI-compatible chat receives unsupported Anthropic image blocks. Moderate (4 votes): Agent mode omits image blocks from requests and persisted turns. Moderate (3 votes): Materialized image messages lose generated agent history. Moderate (2 votes each): Normal-chat media refs are not persisted; vision capability is not revalidated after model changes.
extensions/levelcode-ai/agentMemory.js	Moderate (3 votes): High-resolution image costs are undercounted when compacting memory without the active model ID.

Suppressed comments (13)

extensions/levelcode-ai/agentMemory.js:60

• Once an agent goal carries [image, text] blocks, findCompactionCut still cannot recognize it as a goal boundary because isGoalBoundary only accepts string content (lines 14–15). Image-bearing turns would then prevent compaction from finding a safe cut, so long agent sessions stop compacting after this feature is wired through. Treat non-tool_result user block arrays as boundaries, consistently with sessionResume.isTurnStart.

		if (!Array.isArray(m.content)) { chars += JSON.stringify(m).length; continue; }
		chars += 24;   // role + envelope, roughly what the object costs around its blocks
		for (const b of m.content) {
			if (b && b.type === 'image') { imageTokens += imageBlockTokens(b, modelId); }

extensions/levelcode-ai/extension.js:1831

• The vision flag only gates attachment time; it can change after an image is attached or after an earlier image turn is in conversation. withImages(conversation) materializes every stored image on every request, so switching to a non-vision model still sends those images (and can make the next text-only turn fail). Revalidate the current provider/model at this wire boundary, or otherwise prevent image history from being sent to a blind model.

	post({ type: 'assistantStart' });

extensions/levelcode-ai/extension.js:1749

• A host-side store failure is caught and the image is skipped, but the webview has already rendered every submitted image as a sent user message and cleared the tray. For an image-only oversize/invalid payload this returns from handleSend with no request or error, while the transcript claims it was sent; with text, the image is shown even though only text was transmitted. Do not silently continue here—acknowledge accepted images back to the webview or reject the send before the optimistic render.

}

/**
 * Store what the webview normalized, and return the blocks that will ride the conversation.

extensions/levelcode-ai/extension.js:1647

• withImages deliberately returns a new array when it materializes a stored ref, but runAgent mutates ctx.messages by appending assistant, tool-result, and continuation messages. That leaves those messages on the temporary copy; agentMessages is then persisted and reused without the completed turn. Any agent run containing an image consequently loses its generated history. Materialize at the provider boundary or merge the mutations back into the canonical ref-backed transcript.

	}
	const diagBaseline = verifyCfg.enabled ? snapshotDiagnostics() : new Map();

extensions/levelcode-ai/extension.js:1832

• withImages produces Anthropic {type: 'image', source: ...} blocks here, but providers.streamChat dispatches to streamOpenAI, which passes its messages directly to buildChatBody; toOpenAIMessages is only used by the agent adapter. OpenAI-compatible chat therefore sends an unsupported image shape instead of image_url, so pasted images fail for normal BYOK/OpenRouter chat.

	if (auto.names.length) { post({ type: 'autoContext', names: auto.names }); }
	pendingContext = null;
	post({ type: 'clearContext' });
	post({ type: 'assistantStart' });

extensions/levelcode-ai/imageCost.js:60

• A valid extreme aspect ratio such as 1×10,000 rounds the narrow axis to 0 at the chosen scale. fitToTier then reports a zero-width result, and the equivalent rounding in clientTarget/normalizeImage can create a zero-sized canvas and reject the attachment. Clamp every resized dimension to at least one pixel in both the pure target calculations and the webview normalizer.

	let hi = Math.min(1, t.edge / Math.max(w, h));
	const at = (s) => [Math.round(w * s), Math.round(h * s)];

extensions/levelcode-ai/imageCost.js:40

• The implementation documents high-res as Claude 4.7+, but only claude-opus-4-(7|8|9) is recognized here; a Claude Sonnet or Haiku 4.7+ id falls back to the standard tier. imageBlockTokens then under-counts the visual cost (for example, 1560 instead of the high-res 4784 cap), making the context meter inaccurate. Match all supported Claude families at 4.7+ and add a regression case.

	if (/claude-(opus|sonnet|fable|mythos)-5/.test(id)) { return 'high'; }
	if (/claude-opus-4-(7|8|9)/.test(id)) { return 'high'; }

extensions/levelcode-ai/imageStore.js:102

• Keeping refs in the persisted transcript means sessions.resume will plan from these blocks before materialize runs, but sessionResume.estimateTokens charges JSON.stringify(content).length / 4. A stored 4K screenshot is therefore counted as roughly the ref length instead of thousands of visual tokens, so a resume can load far beyond its configured budget and only fail at the provider request. Feed the same model-aware image-cost estimator into resume planning, not only the compact-memory meter.

function materialize(root, slug, block) {
	if (!block || block.type !== 'image') { return block; }
	if (block.source) { return block; }   // already materialized (or an inline block from elsewhere)
	const data = read(root, slug, block.ref);
	if (!data) { throw new Error('imageStore: attached image is missing from disk: ' + String(block.ref)); }
	return { type: 'image', source: { type: 'base64', media_type: mediaTypeOf(block.ref), data } };

extensions/levelcode-ai/media/chat.html:1585

• This element is exposed as a focusable role="button", but it only handles onclick; Enter or Space on a focused image remove control does nothing. Add keyboard activation, consistent with the existing keyboard handling for file chips, so keyboard users can remove an attachment.

    chipsEl.querySelectorAll('.imgx').forEach(function(x){
      x.onclick = function(e){ e.stopPropagation(); removeImage(x.getAttribute('data-img')); };

extensions/levelcode-ai/media/chat.html:3230

• The dragover handler accepts every file drag, but this handler returns before preventDefault() when the drop contains no images. Dropping a non-image file onto the panel can therefore trigger the browser's default open/navigate behavior and replace the chat. Cancel file drops before filtering, then ignore non-image files.

    if (!ev.dataTransfer || !ev.dataTransfer.files || !ev.dataTransfer.files.length) { return; }
    const imgs = Array.from(ev.dataTransfer.files).filter(function(f){ return /^image\//.test(f.type); });
    if (!imgs.length) { return; }

extensions/levelcode-ai/media/chat.html:3106

• The vision check only runs in attachImageFiles. If a user queues an image under a vision model and then switches models, doSend still emits images: imgs without revalidating canSeeImages, so the strict gate is bypassed and the blind provider receives the composed request. Reject or clear pending images when the selected model changes, or check again before building this payload.

    const imgs = pendingImages.map(function(i){
      return { media_type: i.media_type, base64: i.base64, w: i.w, h: i.h, bytes: i.bytes };
    });

extensions/levelcode-ai/media/chat.html:3195

• attachImageFiles is async and both paste/drop handlers invoke it without awaiting, so two rapid attachment events can enter this loop while pendingImages.length is still below the cap. Each call then passes the check before its normalizeImage await resolves, allowing more than IMG_MAX_PER_TURN images to be added. Serialize attachment jobs or reserve the remaining slots before the first await.

    for (const f of list) {
      if (pendingImages.length >= IMG_MAX_PER_TURN) {
        note('Up to ' + IMG_MAX_PER_TURN + ' images per message — the rest were not attached.');
        break;
      }
      try {
        const im = await normalizeImage(f);

extensions/levelcode-ai/providers/catalog.js:133

• Dynamic model discovery preserves vision on each choice, but after selection supportsVisionForModel recomputes only from static/heuristic modelCaps. A dynamically discovered OpenRouter model that advertises image input under an unrecognized id is consequently treated as blind and cannot use the new attachment flow. Retain the dynamic capability metadata per provider/model or pass it into the gate.

function supportsVisionForModel(providerId, modelId) {
	const p = getProvider(providerId);
	if (!p) { return false; }
	return modelCaps(modelId).vision === true;

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

extensions/levelcode-ai/media/chat.html:3299

• Sending while an image is still "preparing" will post an image object with missing base64/media_type (because placeholders have loading: true). The host then refuses it as an empty image, so the user's intended attachment is silently lost on send.

Block send until all pending images are ready (or filter out placeholders and warn).

    const imgs = pendingImages.map(function(i){
      return { media_type: i.media_type, base64: i.base64, w: i.w, h: i.h, bytes: i.bytes };
    });
    add('user', (imgs.length ? imgs.map(function(i, k){
      return '<img class="msgimg" src="' + escAttr(pendingImages[k].url) + '" alt="attached image">';
    }).join('') : '') + render(t));