fix(sanitization): secret exposure in function and agent trace spans (#6000)

* fix trace span secret sanitization

* sanitize workflow output logs

* preserve streaming usage estimates

* Fix logging session test after staging merge

* secrets sanitization correctness

* fix(execution): address review regressions

* fix(execution): harden secret trace provenance

* fix(execution): preserve functional state during trace projection

---------

Co-authored-by: Bill Leoutsakos <billleoutsakos@Mac.localdomain>
Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>

Author: 3 people

apps/docs/content/docs/en/agents/custom-tools.mdx

@@ -156,7 +156,7 @@ From **Settings → Custom Tools** you can:
 
 <FAQ items={[
   { question: "Can I use custom tools in standalone blocks (not agents)?", answer: "No. Custom tools are designed for use within Agent blocks, where the AI model decides when to call them. For deterministic tool execution, use the Function block instead." },
-  { question: "How do I pass API keys to my custom tool code?", answer: "Use environment variables with double curly brace syntax: {{MY_API_KEY}}. Create the environment variable in Settings → Secrets, and it will be injected at execution time without appearing in logs." },
+  { question: "How do I pass API keys to my custom tool code?", answer: "Use double curly brace syntax such as {{MY_API_KEY}} for a value saved under Settings → Secrets. The real value is injected at execution time, while exact occurrences are masked in the trace copy. If the tool returns the secret, the raw result still reaches the Agent. See Execution log protection under Secrets for details." },
   { question: "Can I use external npm packages?", answer: "No. Custom tool code runs in a sandboxed environment with access to built-in Node.js modules and fetch(), but not external packages. For complex dependencies, consider calling an external API that wraps the functionality you need." },
   { question: "What's the difference between custom tools and the Function block?", answer: "Custom tools are called by AI agents when they decide the tool is relevant — the agent chooses when to use it. Function blocks run deterministically at a fixed point in the workflow. Use custom tools for agent-driven actions and Function blocks for predictable data transformations." },
   { question: "Are custom tools shared across the workspace?", answer: "Yes. Custom tools are workspace-scoped, so all workspace members can use them in their workflows." },

apps/docs/content/docs/en/agents/mcp.mdx

@@ -72,6 +72,8 @@ Authorization: Bearer {{MCP_API_TOKEN}}
 
 When you type `{{` in the URL or header fields, a dropdown appears showing available workspace environment variables.
 
+When a saved secret is successfully substituted this way, exact occurrences of its value are masked in stored MCP tool-call traces. The real URL or header value still reaches the MCP server unchanged. See [Execution log protection](/platform/credentials#execution-log-protection) for the exact scope and limitations.
+
 ### Testing and Validation
 
 Click **Test Connection** before saving to verify the server is reachable and discover available tools. The test response shows the number of tools found and the protocol version.
@@ -169,4 +171,4 @@ import { FAQ } from '@/components/ui/faq'
   { question: "How do I update MCP tool schemas after a server changes its available tools?", answer: "Click the Refresh button on the MCP server in your workspace settings. This fetches the latest tool schemas from the server and automatically updates any agent blocks that use those tools with the new parameter definitions." },
   { question: "Can permission groups restrict access to MCP tools?", answer: "Yes. On Enterprise-entitled workspaces, any workspace admin can create a permission group that disables MCP tools for its members using the disableMcpTools option. When this is enabled, affected users will not be able to add or use MCP tools in workflows that belong to that workspace." },
   { question: "What happens if an MCP server goes offline during workflow execution?", answer: "If the MCP server is unreachable during execution, the tool call will fail and return an error. In an Agent block, the AI may attempt to handle the failure gracefully. In a standalone MCP Tool block, the workflow step will fail. Check MCP server logs and verify the server is running and accessible to troubleshoot connectivity issues." },
-]} />
+]} />

apps/docs/content/docs/en/logs-debugging/index.mdx

@@ -43,7 +43,7 @@ This is the level you debug at, because a run fails when one of its blocks fails
 
 ### Input and output
 
-Each block in the sidebar has two tabs. The **Input** tab shows the resolved values the block actually ran with: the literal values you typed, and the earlier outputs it read by reference (with API keys redacted). The **Output** tab shows what the block produced, formatted as an object, with markdown rendered for agent-generated text.
+Each block in the sidebar has two tabs. The **Input** tab shows the resolved values the block actually ran with: the literal values you typed, and the earlier outputs it read by reference. Exact values successfully substituted from Secrets through `{{KEY}}` are masked in this trace copy; see [Execution log protection](/platform/credentials#execution-log-protection). The **Output** tab shows what the block produced, formatted as an object, with markdown rendered for agent-generated text.
 
 The input tab is the important one. A block reads earlier outputs by name, written `<blockName.field>`, and the input tab shows what those references resolved to at run time. If a reference pointed at a value that was not there, you see it here as missing or wrong, not as the tag you wrote. See [how blocks pass data](/workflows/data-flow) for how those references resolve.
 

apps/docs/content/docs/en/logs-debugging/logging.mdx

@@ -57,7 +57,7 @@ Click any entry to open its sidebar: the run's timeline (start/end, total durati
     The block's result — JSON-formatted structured data, markdown rendering for AI content, and a copy button.
   </Tab>
   <Tab>
-    What the block received — resolved variable values, referenced outputs, and environment variables. API keys are automatically redacted.
+    What the block received — resolved variable values, referenced outputs, and environment variables. Exact secret values activated by a successful `{{KEY}}` substitution are masked in this trace view. See [Execution log protection](/platform/credentials#execution-log-protection).
   </Tab>
 </Tabs>
 
@@ -95,7 +95,7 @@ import { FAQ } from '@/components/ui/faq'
 <FAQ items={[
   { question: "How long are run logs retained?", answer: "Free plans retain logs for 7 days — after that, logs are archived to cloud storage and deleted from the database. Pro, Team, and Enterprise plans retain logs indefinitely with no automatic cleanup." },
   { question: "What data is captured in each run log?", answer: "Each log entry includes the run ID, workflow ID, trigger type, start and end timestamps, total duration in milliseconds, cost breakdown (total cost, token counts, and per-model breakdowns), run data with trace spans, final output, and any associated files. The log details sidebar lets you inspect block-level inputs and outputs." },
-  { question: "Are API keys visible in the logs?", answer: "No. API keys and credentials are automatically redacted in the log input tab for security. You can safely inspect block inputs without exposing sensitive values." },
+  { question: "Are saved secrets visible in logs?", answer: "When a value saved under Secrets is successfully substituted through {{KEY}}, exact, case-sensitive occurrences are masked throughout the log-facing copy, including the live block-log display, Logs Overview input and output, Trace, log-read APIs, and the Logs block's Get Run Details output. This is not a general redactor: hardcoded or directly read values do not activate masking by themselves, and encoded, hashed, or transformed values are not matched. Functional execution responses, streams, and callbacks remain unchanged. See Execution log protection under Secrets for details." },
   { question: "What is a workflow snapshot?", answer: "A frozen copy of the workflow's structure (blocks, connections, and configuration) captured at run time, so you can see the exact state behind a particular run — useful for debugging workflows that have been modified since." },
   { question: "Can I access logs programmatically?", answer: "Yes. The External API provides endpoints to query logs with filtering by workflow, time range, trigger type, duration, cost, and model. You can also set up webhook, email, or Slack notifications for real-time alerts when runs complete." },
   { question: "What does Live mode do on the Logs page?", answer: "It refreshes the Logs page in real time so new entries appear as they are recorded — useful during deployments or when monitoring active workflows." },

apps/docs/content/docs/en/platform/credentials.mdx

@@ -65,8 +65,14 @@ Select the secret you want to use. The reference appears highlighted in blue and
   height={200}
 />
 
+### Execution log protection
+
+When a saved secret is successfully substituted through a `{{KEY}}` reference, Sim masks exact, case-sensitive occurrences of its resolved value in log-facing content. This includes the editor's live block-log display, Logs Overview input and output, stored execution traces, log-read API responses, and the Logs block's **Get Run Details** output. Function and Agent span inputs, outputs, and errors, Agent thinking, and Agent tool-call arguments, results, and errors are protected. The replacement is normally shown as `{{KEY}}`.
+
+This is an observability projection only. Secret resolution and workflow behavior are unchanged: blocks, tools, models, and downstream steps receive the real runtime value. Stored functional execution data, workflow execution responses, streams, callbacks, block state, and snapshots are not rewritten. Log-facing views and read APIs receive a separate protected copy, so the Logs Overview **Workflow Input** and **Workflow Output** are masked without changing the underlying workflow result.
+
 <Callout type="warn">
-Secret values are never exposed in the workflow editor or execution logs — they are only resolved during execution.
+Masking is activated only when Sim successfully resolves a value from **Settings → Secrets** through `{{KEY}}`. A hardcoded literal, direct `environmentVariables['KEY']` read, or shell `$KEY` read does not activate it by itself. Once activated, every exact occurrence of that value in the run's log-facing content is masked. Encoded, hashed, or otherwise transformed versions are not matched. Do not deliberately return or print secrets.
 </Callout>
 
 ## Secret Details
@@ -115,7 +121,8 @@ When a workflow runs, secrets resolve in this order:
 - **Never hardcode secrets** in workflow input fields — always use `{{KEY}}` references
 
 <FAQ items={[
-  { question: "Are my secrets encrypted at rest?", answer: "Yes. Secret values are encrypted before being stored in the database using server-side encryption, so raw values are never persisted in plaintext. They are also never exposed in the workflow editor, logs, or API responses." },
+  { question: "Are my secrets encrypted at rest?", answer: "Yes. Values saved under Secrets are encrypted before being stored in the database." },
+  { question: "Can a saved secret still appear in a workflow result?", answer: "Yes. Functional data is not rewritten, so the raw value can still reach downstream blocks, tools, and models and can appear in workflow execution responses, streams, or callbacks if your workflow deliberately returns or prints it. Log-facing views and read APIs, including Workflow Output on the Logs Overview and the Logs block's Get Run Details output, receive a protected copy after a successful {{KEY}} substitution." },
   { question: "What happens if both a workspace secret and a personal secret have the same key name?", answer: "The workspace secret takes precedence. During execution, the resolver checks workspace secrets first and uses personal secrets only as a fallback. This ensures production workflows use the shared, team-managed value." },
   { question: "Who determines which personal secret is used for automated runs?", answer: "For manual runs, the personal secrets of the user who clicked Run are used as fallback. For automated runs triggered by API, webhook, or schedule, the personal secrets of the workflow owner are used instead." },
   { question: "Can I import secrets from a .env file?", answer: "Yes. Paste .env-style content (KEY=VALUE format) into any key or value field and the secrets will be auto-populated. The parser supports export KEY=VALUE, quoted values, and inline comments." },

apps/docs/content/docs/en/workflows/blocks/function.mdx

@@ -138,14 +138,19 @@ number reported. A sandbox may declare up to 50 packages.
 
 When a Function block is used as an Agent tool, its code can read every workspace
 secret by default — both `{{MY_SECRET}}` and `environmentVariables['MY_SECRET']`.
+Use `{{MY_SECRET}}` when the value may appear in execution logs: a successful
+double-brace substitution activates [execution-trace masking](/platform/credentials#execution-log-protection),
+while direct `environmentVariables['MY_SECRET']` access alone does not activate
+it by itself.
 
 To narrow that, set **Secret access** to *Selected secrets* in the block's
 tool configuration and pick the names the code may read. Two things change:
 
 - Only those secrets are injected. `{{OTHER_SECRET}}` no longer resolves either.
 - The selected **names** are added to the tool's description, so the model knows
-  what it can reference. Values are never sent to the model — they are injected
-  server-side at execution.
+  what it can reference. Values are injected server-side when the Function runs.
+  If the Function returns a secret, the raw result is still sent to the Agent;
+  masking affects only the trace and display copy.
 
 Leaving the default (*All secrets*) resolves the list at run time, so a secret
 added next month is included automatically.

apps/docs/content/docs/en/workflows/blocks/logs.mdx

@@ -46,7 +46,10 @@ This pairs naturally with the [Sim trigger](/workflows/triggers/sim): the trigge
 
 <Callout type="info">
   The block always operates on the current workspace. Costs are denominated in credits, both for
-  the cost filter and the cost output.
+  the cost filter and the cost output. **Get Run Details** is a log-read path, so both `traceSpans`
+  and `finalOutput` come from the protected log-facing projection described under
+  [Execution log protection](/platform/credentials#execution-log-protection). The underlying runtime
+  result remains unchanged.
 </Callout>
 
 <FAQ items={[

apps/docs/content/docs/en/workflows/variables.mdx

@@ -74,9 +74,9 @@ Each run starts fresh from the values defined in the panel. A change made during
 
 ## Environment variables
 
-Environment variables store sensitive values like API keys, tokens, and configuration that should never appear in logs or on the canvas. Create them under **Settings → Secrets** by adding a key-value pair.
+Environment variables let you keep sensitive values like API keys and tokens out of your saved workflow configuration. Create them under **Settings → Secrets** by adding a key-value pair.
 
-Reference them with double curly braces in any block field, including Agent system prompts and Function block code. The value is substituted before the block runs, and API keys are redacted in logs.
+Reference them with double curly braces in any block field, including Agent system prompts and Function block code. The value is substituted before the block runs. When a saved secret is successfully substituted through `{{KEY}}`, exact occurrences of that value are masked in the live block-log display and stored execution trace without changing the runtime value. See [Execution log protection](/platform/credentials#execution-log-protection) for the exact scope and limitations.
 
 ```
 {{API_KEY}}

apps/sim/app/api/function/execute/route.test.ts

@@ -796,16 +796,115 @@ describe('Function Execute API Route', () => {
 
   describe('Template Variable Resolution', () => {
     it.concurrent('should resolve environment variables with {{var_name}} syntax', async () => {
-      const req = createMockRequest('POST', {
-        code: 'return {{API_KEY}}',
-        envVars: {
-          API_KEY: 'secret-key-123',
+      const req = createMockRequest(
+        'POST',
+        {
+          code: 'return {{API_KEY}}',
+          envVars: {
+            API_KEY: 'secret-key-123',
+          },
         },
-      })
+        {
+          'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
+        }
+      )
 
       const response = await POST(req)
+      const data = await response.json()
 
       expect(response.status).toBe(200)
+      expect(data.__resolvedSecretNames).toEqual(['API_KEY'])
+    })
+
+    it('reports only successful references sourced from scoped environment variables', async () => {
+      const envResponse = await POST(
+        createMockRequest(
+          'POST',
+          {
+            code: 'return {{SHARED}} + {{ENV_ONLY}} + {{MISSING}}',
+            params: { SHARED: 'param-value', MISSING: 'ordinary-param' },
+            envVars: { SHARED: 'secret-value', ENV_ONLY: 'other-secret' },
+          },
+          {
+            'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
+          }
+        )
+      )
+      const envData = await envResponse.json()
+
+      const directResponse = await POST(
+        createMockRequest(
+          'POST',
+          {
+            code: 'return environmentVariables.API_KEY + params.API_KEY',
+            params: { API_KEY: 'ordinary-param' },
+            envVars: { API_KEY: 'secret-value' },
+          },
+          {
+            'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
+          }
+        )
+      )
+      const directData = await directResponse.json()
+
+      expect(envData.__resolvedSecretNames).toEqual(['ENV_ONLY', 'SHARED'])
+      expect(directData.__resolvedSecretNames).toEqual([])
+    })
+
+    it('reports shell {{NAME}} substitutions but not direct shell environment access', async () => {
+      envFlagsMock.isRemoteSandboxEnabled = true
+
+      const referencedResponse = await POST(
+        createMockRequest(
+          'POST',
+          {
+            code: 'printf "%s" "{{API_KEY}}"',
+            language: 'shell',
+            envVars: { API_KEY: 'secret-value' },
+          },
+          {
+            'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
+          }
+        )
+      )
+      const referencedData = await referencedResponse.json()
+
+      const directResponse = await POST(
+        createMockRequest(
+          'POST',
+          {
+            code: 'printf "%s" "$API_KEY"',
+            language: 'shell',
+            envVars: { API_KEY: 'secret-value' },
+          },
+          {
+            'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
+          }
+        )
+      )
+      const directData = await directResponse.json()
+
+      expect(referencedData.__resolvedSecretNames).toEqual(['API_KEY'])
+      expect(directData.__resolvedSecretNames).toEqual([])
+    })
+
+    it('reports only substitutions allowed by the Function secret scope', async () => {
+      const response = await POST(
+        createMockRequest(
+          'POST',
+          {
+            code: 'return {{ALLOWED}} + {{BLOCKED}}',
+            envVars: { ALLOWED: 'allowed-secret', BLOCKED: 'blocked-secret' },
+            secretScope: 'selected',
+            mountedSecrets: ['ALLOWED'],
+          },
+          {
+            'x-sim-request-private-tool-metadata': 'resolved-secret-names-v1',
+          }
+        )
+      )
+
+      expect((await response.json()).__resolvedSecretNames).toEqual(['ALLOWED'])
     })
 
     it.concurrent('should resolve tag variables with <tag_name> syntax', async () => {