Allow agent to control light/dark color theme, and display screenshots in the chat by shaunandrews · Pull Request #3982 · Automattic/studio

shaunandrews · 2026-06-28T04:05:25Z

Proposed Changes

Makes screenshots captured by Studio Code appear inline in the conversation as media artifacts, so users can see the captured image instead of only reading a text summary — in both the agentic UI and the legacy Studio Code session view.
Adds light/dark color scheme arguments to screenshot and visual-inspection tools, making it easier for agents to verify theme-aware pages in the intended color scheme.
Screenshots are emitted as structural chat artifacts only — the raw mediaWidgetPayload= text lines are gone from tool output (a shared stripper keeps old transcripts clean), and the studio_present screenshot rule was replaced so the agent no longer re-presents the same capture, which caused the duplicate image reported in review.
Captures are stored in a per-session <session>.screenshots/ directory next to the session file instead of the OS temp dir, so inline images keep working in older transcripts; deleting a session removes its screenshots.
Remote-site sessions now emit chat artifacts for all Studio tools (previously only take_screenshot was wrapped, so e.g. site_create never emitted its site-preview artifact).
Browser builds (hosted / studio ui) can't read local files, so local-only screenshot artifacts are hidden there via a new readLocalMedia connector capability instead of rendering guaranteed "Image unavailable" boxes.

Screenshots

Testing Instructions

Run npm test -- apps/cli/ai/tests packages/common/ai apps/ui/src/ui-classic/components/session-view/conversation apps/studio/src/components/studio-code-session/conversation
Run npm run typecheck
In Studio Code, ask the agent to take a screenshot and confirm exactly one captured image appears inline in the conversation (the duplicate is gone).
Confirm the capture is saved under ~/Library/Application Support/Studio/sessions/<date>/<session>.screenshots/ and still renders after restarting the app and reopening the session.
Ask for light and dark captures with colorScheme: "all" and confirm both artifacts are produced, labeled (desktop light) / (desktop dark).
Open the same session in the legacy Studio Code view and confirm the screenshot renders inline there too (check light + dark mode).
Delete the session and confirm its .screenshots/ directory is removed.
Open a pre-existing session whose transcript contains mediaWidgetPayload= lines and confirm they stay hidden in the expanded tool details.

Copilot

Pull request overview

Adds first-class screenshot artifacts to Studio’s agent conversations and expands screenshot/inspection tools to support explicit light/dark capture, improving theme-aware visual verification in the Agentic UI and CLI runtime.

Changes:

Render studio.chat_artifact media widgets (local screenshots) inline in the classic conversation transcript UI, with a loading/unavailable state.
Extend take_screenshot, share_screenshot, and inspect_design tools with colorScheme support (light/dark, plus all for take_screenshot).
Suppress raw mediaWidgetPayload*= marker lines from the conversation’s expanded tool-details UI while still emitting artifacts via chat events.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
apps/ui/src/ui-classic/components/session-view/conversation/style.module.css	Adds layout + skeleton/unavailable styles for inline media artifacts.
apps/ui/src/ui-classic/components/session-view/conversation/index.tsx	Renders chat artifact images inline; hides screenshot payload markers in tool output.
apps/ui/src/ui-classic/components/session-view/conversation/index.test.ts	Adds coverage for payload-line hiding and inline local screenshot rendering.
apps/cli/ai/tools/take-screenshot.ts	Adds `colorScheme` + multi-capture labeling and returns `studioArtifacts` for artifact emission.
apps/cli/ai/tools/share-screenshot.ts	Adds `colorScheme` option for prefers-color-scheme sharing captures.
apps/cli/ai/tools/screenshot-helpers.ts	Threads `colorScheme` into Playwright `emulateMedia` and temp filename suffixing.
apps/cli/ai/tools/inspect-design.ts	Adds `colorScheme` emulation for rendered DOM inspection.
apps/cli/ai/tools/index.ts	Exports `withChatArtifactEmission` for reuse and ensures tool execution can emit artifacts.
apps/cli/ai/tests/tools.test.ts	Adds/extends tests for `colorScheme`, multi-scheme capture, and artifact emission behavior.
apps/cli/ai/runtimes/pi/index.ts	Wraps remote-site screenshot tool to emit chat artifacts when enabled.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

wpmobilebot · 2026-06-29T18:32:32Z

📊 Performance Test Results

Comparing a6c3f8f vs trunk

app-size

Metric	trunk	`a6c3f8f`	Diff	Change
App Size (Mac)	1345.48 MB	1345.51 MB	+0.03 MB	⚪ 0.0%

site-editor

Metric	trunk	`a6c3f8f`	Diff	Change
load	1104 ms	754 ms	350 ms	🟢 -31.7%

site-startup

Metric	trunk	`a6c3f8f`	Diff	Change
siteCreation	6523 ms	6504 ms	19 ms	⚪ 0.0%
siteStartup	1860 ms	1863 ms	+3 ms	⚪ 0.0%

Results are median values from multiple test runs.

Legend: 🟢 Improvement (faster) | 🔴 Regression (slower) | ⚪ No change (<50ms diff)

bcotrim

While testing I noticed the agent appears to be taking 2 screenshots, or printing the screenshot twice.

Attempt 1	Attempt2

bcotrim · 2026-07-01T11:45:20Z

+	} );
+}
+
+function stripScreenshotMediaPayloadLines( text: string ): string {


stripScreenshotMediaPayloadLines is duplicated in apps/studio/src/components/studio-code-session/conversation/index.tsx:68, and the mediaWidgetPayload=/mediaWidgetPayloads= prefixes are hardcoded here, there, and in the producer (apps/cli/ai/tools/take-screenshot.ts:146). If the CLI renames the prefix, both strippers silently break. Lift the prefixes + helper into tools/common? Non-blocking.

bcotrim · 2026-07-01T11:45:49Z

 	return optionLabels.has( answer ) ? answer : undefined;
 }

+function isRecord( value: unknown ): value is Record< string, unknown > {


Dupes the unexported isRecord in tools/common/ai/chat-artifacts.ts:38 — export that one and import it instead?

shaunandrews · 2026-07-03T15:37:42Z

While testing I noticed the agent appears to be taking 2 screenshots, or printing the screenshot twice.

Its taking the screenshot in light and dark mode, but since your site doesn't change anything, it looks weird.

shaunandrews · 2026-07-03T15:55:38Z

Its taking the screenshot in light and dark mode, but since your site doesn't change anything, it looks weird.

Hmmmm, maybe not. Looks like it might be showing them twice for some reason. Looking at a fix now.

…io tools Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…o/common Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…urally Removes the mediaWidgetPayload= lines from take_screenshot output and replaces the screenshot-local-media presentation rule, which told the model to re-present the auto-emitted screenshot via studio_present and produced duplicate images in the conversation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Screenshot artifacts persisted into session transcripts pointed at mkdtemp directories that macOS purges within days, leaving permanent 'Image unavailable' boxes. Captures now live in <session>.screenshots/ next to the session JSONL (with unique filenames) and are deleted with the session; standalone CLI/MCP runs still fall back to a temp directory. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Validates persisted chat-artifact entries before rendering, caches local screenshot reads through react-query and object URLs instead of base64 data URLs, hides local-only artifacts on connectors that cannot read local files, and moves legacy payload-line stripping into the memoized render-item pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…tion The legacy view hid the payload line but had no way to see the capture at all. Chat artifact entries now render as inline images (read over IPC with a per-path cache), using frame color tokens for light/dark support. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

blob: URLs are not allowed by the desktop renderer CSP (img-src permits data:), and revoke-on-cleanup broke the image when the artifact row remounted after a run finished. Data URLs need no lifecycle; conversion happens once inside the cache layer so files are still read over IPC only once per path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

shaunandrews · 2026-07-03T19:07:52Z

👋 Update on the review feedback — a batch of fixes just landed on this branch (rebased onto trunk first). Here's the tour:

The double screenshot @bcotrim spotted turned out not to be two captures at all — it was one capture presented twice. The tool now auto-emits its artifact, but the old screenshot-local-media prompt rule was also telling the model to re-present the same image via studio_present. Fixed at the source (4bbec3c): the mediaWidgetPayload= lines are gone from tool output entirely, and the rule now tells the model screenshots are already shown inline. One screenshot, one image.

Which also resolves the two duplication comments: with the producer no longer writing marker lines, the strippers only exist for old transcripts — there's now a single stripMediaWidgetPayloadLines (plus the marker constants) in @studio/common/ai/chat-artifacts, used by both renderers (cf66b73). isRecord is exported from there too, and the third private copy in studio-widgets.ts now imports it as well.

On Copilot's data-URL performance note: the conversion now happens once inside a cache layer — react-query in the classic UI, a per-path map in the legacy view — so each file crosses IPC and gets encoded a single time per app lifetime. (Fun detour: we tried blob: object URLs first, but the desktop CSP only allows data: in img-src, and revoke-on-cleanup broke under StrictMode remounts. Data URLs it is — a6c3f8f.)

A few more things surfaced while testing that seemed worth fixing in the same pass:

Screenshots now live next to the session file in a <session>.screenshots/ folder instead of os.tmpdir() (686cb25). Before, macOS would purge the temp files within a few days and every screenshot in an older transcript became a permanent "Image unavailable" box. Deleting a session cleans up its screenshots too.
The legacy Studio Code view renders screenshots inline now as well (85b2b81) — previously it hid the payload line but showed nothing in its place, so the capture was unreachable there.
Remote sessions were only wrapping take_screenshot for artifact emission — site_create returns a site-preview artifact too, and unwrapped it never fired (so session placement never happened for remote-created sites). All the remote studio tools are wrapped uniformly now, and the wrapper forwards signal/onUpdate and keeps details intact (df716ff).
Guarded the artifact render path against malformed persisted entries (one bad JSONL line used to be able to take down the whole conversation view), hid local-file artifacts in the browser builds where the connector can't read disk, and a handful of small dedups (shared colorScheme schema, shared emulateMedia helper, test mock factory).

Typecheck is clean and the affected suites pass (~235 tests, including new coverage for the storage, delete sweep, and malformed-entry cases). Still worth a human eyeball on the inline images in light + dark mode.

Written by Claude (working with @shaunandrews), who also did most of the implementation in this batch.

github-actions Bot assigned shaunandrews Jun 28, 2026

shaunandrews marked this pull request as ready for review June 29, 2026 16:51

shaunandrews requested a review from Copilot June 29, 2026 16:51

Copilot started reviewing on behalf of shaunandrews June 29, 2026 16:52 View session

Copilot AI reviewed Jun 29, 2026

View reviewed changes

Comment thread apps/ui/src/ui-classic/components/session-view/conversation/index.tsx Outdated

shaunandrews changed the title ~~Add screenshot artifacts and color scheme capture~~ Agentic UI: Add screenshot artifacts and color scheme capture Jun 29, 2026

shaunandrews changed the title ~~Agentic UI: Add screenshot artifacts and color scheme capture~~ Add screenshot artifacts and color scheme capture Jun 29, 2026

shaunandrews requested a review from a team June 30, 2026 00:29

shaunandrews changed the title ~~Add screenshot artifacts and color scheme capture~~ Allow agent to control light/dark color theme, and display screenshots in the chat Jun 30, 2026

bcotrim reviewed Jul 1, 2026

View reviewed changes

shaunandrews force-pushed the extract-screenshot-artifacts branch 2 times, most recently from 9757d73 to f54ed57 Compare July 3, 2026 16:40

shaunandrews and others added 4 commits July 3, 2026 12:41

Add AI screenshot artifacts and color scheme capture

c6172aa

Improve local screenshot data URL conversion

e160e95

Hide screenshot payloads in Studio Code transcript

63ce0e8

Forward tool execute args and emit chat artifacts for all remote stud…

df716ff

…io tools Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

shaunandrews force-pushed the extract-screenshot-artifacts branch from f54ed57 to df716ff Compare July 3, 2026 16:42

shaunandrews and others added 7 commits July 3, 2026 12:44

Add shared media artifact helpers and payload-line stripper to @studi…

cf66b73

…o/common Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Share the colorScheme schema and media emulation across screenshot tools

e007735

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Allow agent to control light/dark color theme, and display screenshots in the chat#3982

Allow agent to control light/dark color theme, and display screenshots in the chat#3982
shaunandrews wants to merge 11 commits into
trunkfrom
extract-screenshot-artifacts

shaunandrews commented Jun 28, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

wpmobilebot commented Jun 29, 2026 •

edited

Loading

Uh oh!

bcotrim left a comment •

edited

Loading

Uh oh!

bcotrim Jul 1, 2026

Uh oh!

bcotrim Jul 1, 2026

Uh oh!

shaunandrews commented Jul 3, 2026

Uh oh!

shaunandrews commented Jul 3, 2026

Uh oh!

shaunandrews commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

shaunandrews commented Jun 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Changes

Screenshots

Testing Instructions

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

wpmobilebot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Performance Test Results

app-size

site-editor

site-startup

Uh oh!

bcotrim left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bcotrim Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

bcotrim Jul 1, 2026

Choose a reason for hiding this comment

Uh oh!

shaunandrews commented Jul 3, 2026

Uh oh!

shaunandrews commented Jul 3, 2026

Uh oh!

shaunandrews commented Jul 3, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shaunandrews commented Jun 28, 2026 •

edited

Loading

wpmobilebot commented Jun 29, 2026 •

edited

Loading

bcotrim left a comment •

edited

Loading