Skip to content

Commit 17d4e6e

Browse files
committed
feat(images) I5: paste a screenshot, ask about it
Paste, drop, a thumbnail chip you can take back off, and the refusal that has to happen before anything is lost. THE CAP IS 2000px ON THE LONG EDGE, AND IT IS MEASURED, NOT CHOSEN. The plan argued 1568 from first principles. Claude Code's own transcripts are on this machine, so I read them instead: 24 images, and every single one it re-encodes is exactly 2000 on the long edge. That is the threshold the vision docs name for staying clear of the stricter per-image dimension limit above 20 images per request — the largest size that is never unsafe. It also sits above both model tiers' caps, so the server does the final downscale and we never throw away fidelity it would have kept. Changed 1568 -> 2000. The same transcripts confirmed the rule I had derived rather than observed: images UNDER the cap are passed through untouched IN THEIR ORIGINAL FORMAT (their PNGs stay PNG, their JPEGs stay JPEG), and only oversize ones are resized and re-encoded to WebP. Re-encoding something that did not need resizing only stacks artifacts, and that is worst on screenshots of code, which is most of what gets pasted. Verified in a real browser against real images, not just in unit tests: 4K screenshot 3840x2160 png 764KB -> 2000x1125 webp 115KB resized retina window 3024x1964 png 654KB -> 2000x1299 webp 137KB resized small PNG 1160x480 png 121KB -> 1160x480 png 121KB PASS-THROUGH under-cap JPEG 1600x900 jpg 208KB -> 1600x900 jpg 208KB PASS-THROUGH wide panorama 2400x600 png 178KB -> 2000x500 webp 51KB resized TIFF -> refused, with what to do instead Aspect ratios come out exact (16:9 stays 16:9, 4:1 stays 4:1). THE REFUSAL RUNS AT ATTACH TIME, NOT SEND TIME. The composer clears on send, so refusing host-side would throw away what someone had written. canSeeImages travels with the model config instead — from BOTH config paths, gateway and BYOK, which a test pins, because one of them silently allowing is exactly the kind of half-gate that looks fine in review. Other decisions worth naming: - An image with no words is a valid message. "Look at this" is implied by attaching it. - The chip carries its own thumbnail. An attachment you cannot see is one you cannot check, and a screenshot is the case where the wrong one looks exactly like the right one in a filename. - Drop is bound to the whole panel, not the composer: someone dragging a screenshot aims at the conversation, which is the far bigger target. - A paste with no image falls through untouched, and the bail comes BEFORE preventDefault — reversing those two silently breaks ordinary text pasting. - withImages() copies. agentMessages persists across runs and is what recordTurn writes to the session log; materializing in place would put megabytes of base64 in both. 9 tests in imageAttach, 37 suites green.
1 parent e66d524 commit 17d4e6e

4 files changed

Lines changed: 371 additions & 13 deletions

File tree

extensions/levelcode-ai/extension.js

Lines changed: 69 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,8 @@ const { registerLmProvider } = require('./lmProvider');
2222
const { registerInlineComplete } = require('./inlineComplete');
2323
const { runAgent } = require('./agent');
2424
const { findCompactionCut, estimateMsgTokens } = require('./agentMemory');
25+
const imageStore = require('./imageStore');
26+
const { supportsVisionForModel } = require('./providers/catalog');
2527
const sessionStore = require('./sessionStore');
2628
const sessionEvents = require('./sessionEvents');
2729
const sessionMemory = require('./sessionMemory');
@@ -1642,7 +1644,7 @@ async function agentFlow(text) {
16421644
dbg('verify.config', { enabled: verifyCfg.enabled, hasCommand: !!verifyCfg.command, maxRounds: verifyCfg.maxRounds, includeWarnings: verifyCfg.includeWarnings });
16431645
try {
16441646
await runAgent({
1645-
messages: agentMessages, // persists across runs → the agent remembers the session
1647+
messages: withImages(agentMessages), // persists across runs → the agent remembers the session
16461648
providerId: req.providerId, // Anthropic native, or an OpenAI-shaped provider via translation (P2)
16471649
baseURL: req.baseURL, // for the custom / Ollama endpoints
16481650
label: req.label, // route name for error attribution — "LevelCode Cloud" on the gateway,
@@ -1724,8 +1726,57 @@ async function agentFlow(text) {
17241726
}
17251727
}
17261728

1727-
async function handleSend(text) {
1728-
if (!text || !text.trim()) { return; }
1729+
/**
1730+
* Store what the webview normalized, and return the blocks that will ride the conversation.
1731+
*
1732+
* Bytes land in the session's own media/ directory and the message keeps only a ref. Refused
1733+
* images are reported and skipped rather than failing the whole send — someone who pasted three
1734+
* screenshots and one unreadable file should still get their question answered.
1735+
*/
1736+
function storeImages(images) {
1737+
const out = [];
1738+
if (!Array.isArray(images) || !images.length) { return out; }
1739+
const m = sessionsManager();
1740+
const paths = m && m.mediaRoot ? m.mediaRoot() : null;
1741+
if (!paths) { vscode.window.showWarningMessage('Images need a session to attach to.'); return out; }
1742+
for (const im of images) {
1743+
try {
1744+
const { ref, bytes } = imageStore.put(paths.root, paths.slug, im.base64, im.media_type);
1745+
out.push({ type: 'image', ref, w: Number(im.w) || 0, h: Number(im.h) || 0, bytes });
1746+
} catch (e) {
1747+
const msg = String((e && e.message) || e).replace(/^imageStore: /, '');
1748+
vscode.window.showWarningMessage('Could not attach an image: ' + msg);
1749+
dbg('image.store.failed', { msg });
1750+
}
1751+
}
1752+
return out;
1753+
}
1754+
1755+
/**
1756+
* A copy of `msgs` with every stored image turned into a real wire block.
1757+
*
1758+
* A COPY, deliberately. `agentMessages` persists across runs and is what recordTurn writes to the
1759+
* session log — materializing in place would put megabytes of base64 into both.
1760+
*/
1761+
function withImages(msgs) {
1762+
if (!Array.isArray(msgs)) { return msgs; }
1763+
const m = sessionsManager();
1764+
const paths = m && m.mediaRoot ? m.mediaRoot() : null;
1765+
if (!paths) { return msgs; }
1766+
let touched = false;
1767+
const out = msgs.map((msg) => {
1768+
if (!msg || !Array.isArray(msg.content)) { return msg; }
1769+
if (!msg.content.some((b) => b && b.type === 'image' && b.ref)) { return msg; }
1770+
touched = true;
1771+
return { ...msg, content: msg.content.map((b) => imageStore.materialize(paths.root, paths.slug, b)) };
1772+
});
1773+
return touched ? out : msgs;
1774+
}
1775+
1776+
async function handleSend(text, images) {
1777+
const imageBlocks = storeImages(images);
1778+
if ((!text || !text.trim()) && !imageBlocks.length) { return; }
1779+
text = text || '';
17291780
if (ctx) { ctx.globalState.update('levelcode.ai.hasSentMessage', true); } // user engaged → stop auto-revealing the panel on launch
17301781
if (agentMode) { await agentFlow(text); return; }
17311782
const cfg = aiConfig();
@@ -1750,7 +1801,12 @@ async function handleSend(text) {
17501801

17511802
if (pendingContext) { blocks.push(pendingContext); }
17521803
const userContent = blocks.length ? (blocks.join('\n\n') + '\n\n' + text) : text;
1753-
conversation.push({ role: 'user', content: userContent });
1804+
// Blocks only when there is an image; a text-only turn stays a plain string so every cached
1805+
// prefix keeps the bytes it already had. Images lead — the model reads them best before the
1806+
// text that asks about them.
1807+
conversation.push(imageBlocks.length
1808+
? { role: 'user', content: [...imageBlocks, { type: 'text', text: userContent }] }
1809+
: { role: 'user', content: userContent });
17541810
post({ type: 'userMessage', text });
17551811
if (auto.names.length) { post({ type: 'autoContext', names: auto.names }); }
17561812
pendingContext = null;
@@ -1772,7 +1828,7 @@ async function handleSend(text) {
17721828
const doStream = (r) => providers.streamChat({
17731829
providerId: r.providerId, apiKey: r.apiKey, baseURL: r.baseURL, label: r.label,
17741830
model: r.model, maxTokens: r.maxTokens, system: SYSTEM_PROMPT,
1775-
messages: conversation, signal: abort.signal, onDelta
1831+
messages: withImages(conversation), signal: abort.signal, onDelta
17761832
});
17771833
try {
17781834
await doStream(req);
@@ -2240,14 +2296,16 @@ function sendConfigToWebview() {
22402296
type: 'config', provider: 'gateway', proseSize, proseWidth, model: gatewayModelLabel(model), modelId: model,
22412297
providerLabel: 'LevelCode Cloud', contextLimit: contextLimitFor('openai', capsModel(model)),
22422298
gateway: true, plan: cloudPlanName() || 'Free', paid: isPaidCloudPlan(cloudPlanName()),
2243-
groupActivity: groupActivity
2299+
groupActivity: groupActivity, canSeeImages: supportsVisionForModel('openai', capsModel(model))
22442300
});
22452301
return;
22462302
}
22472303
const providerId = currentProviderId();
22482304
const p = providers.getProvider(providerId) || providers.getProvider('claude');
22492305
// Carry the model's context window so the footer meter updates the moment the model changes.
2250-
post({ type: 'config', provider: providerId, proseSize, proseWidth, model: activeModel(cfg, providerId), providerLabel: p.label, contextLimit: currentContextLimit(), groupActivity: groupActivity });
2306+
// canSeeImages travels with the model so the composer can refuse an attachment BEFORE anything is
2307+
// typed and lost, rather than after a send that the provider would reject.
2308+
post({ type: 'config', provider: providerId, proseSize, proseWidth, model: activeModel(cfg, providerId), providerLabel: p.label, contextLimit: currentContextLimit(), groupActivity: groupActivity, canSeeImages: supportsVisionForModel(providerId, activeModel(cfg, providerId)) });
22512309
}
22522310

22532311
/**
@@ -2280,7 +2338,10 @@ class ChatViewProvider {
22802338
case 'ready': cloudSignedIn = !!(ctx && await ctx.secrets.get(ACCOUNT_TOKEN_KEY)); autopilot = aiConfig().get('agent.autopilot', false); sendConfigToWebview(); postActiveFile(); postContextFiles(); post({ type: 'mode', agent: agentMode }); post({ type: 'autopilot', on: autopilot }); postAccount(); buildFileIndex(); post({ type: 'contextUsage', input: 0, limit: currentContextLimit() }); if (review) { review.resync(); } postMemoryDigest(); if (pendingTranscriptReplay) { const t = pendingTranscriptReplay; pendingTranscriptReplay = ''; replayLiveTranscript(t); } break;
22812339
case 'setMode': agentMode = !!msg.agent; post({ type: 'mode', agent: agentMode }); break;
22822340
case 'setAutopilot': autopilot = !!msg.on; aiConfig().update('agent.autopilot', autopilot, vscode.ConfigurationTarget.Global); dbg('autopilot.set', { on: autopilot }); post({ type: 'autopilot', on: autopilot }); break;
2283-
case 'send': await handleSend(msg.text); break;
2341+
case 'send': await handleSend(msg.text, msg.images); break;
2342+
// One surface for "that could not be attached" — VS Code's own, not a second one
2343+
// invented inside the transcript.
2344+
case 'notice': if (msg.text) { vscode.window.showWarningMessage(String(msg.text)); } break;
22842345
case 'stop': dbg('stop.clicked', { running: commandStops.size }); for (const [, stop] of commandStops) { try { stop(); } catch (e) { /* gone */ } } if (abort) { abort.abort(); } clearApprovals(); clearQuestions(); break;
22852346
case 'stopCommand': { dbg('stopCommand', { id: msg.id }); const s = commandStops.get(msg.id); if (s) { try { s(); } catch (e) { /* gone */ } } break; }
22862347
case 'approvalResponse': {

0 commit comments

Comments
 (0)