Skip to content

Commit d3555ef

Browse files
committed
feat(api): add proxyUrl for residential/custom proxy egress on the API block
The HTTP/API block egresses from the app runtime's fixed datacenter IPs via secureFetchWithPinnedIP, so targets behind Cloudflare/WAF that block datacenter IPs (e.g. state .gov license portals) return 403/429 even when the identical request works from a browser. There was no way to route a request through a residential/custom proxy. Add an optional `proxyUrl` field (Advanced) to the API block. When set, the request routes through the given http:// proxy so it egresses from that proxy's IP. Security: - validateAndPinProxyUrl resolves the proxy host's DNS and blocks private/reserved/loopback IPs (same SSRF guard as target URLs), then pins the connection by rewriting the host to the resolved IP (creds/port preserved), closing the DNS-rebinding window. - Restricted to the http: proxy scheme (https/socks rejected) so host pinning is safe without breaking TLS-to-proxy SNI. - Target-IP pinning is intentionally bypassed when a proxy is active (the proxy resolves the target); target URL validation still runs. Threaded block field -> http tool param -> formatRequestParams -> executeToolRequest (validate + pin) -> secureFetchWithPinnedIP, which swaps its pinned Node agent for HttpsProxyAgent/HttpProxyAgent (keyed off target protocol) when proxyUrl is set.
1 parent 52659d4 commit d3555ef

12 files changed

Lines changed: 250 additions & 6 deletions

File tree

apps/sim/blocks/blocks/api.ts

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,15 @@ Example:
123123
description: 'Allow retries for POST/PATCH requests (may create duplicate requests)',
124124
mode: 'advanced',
125125
},
126+
{
127+
id: 'proxyUrl',
128+
title: 'Proxy URL',
129+
type: 'short-input',
130+
placeholder: 'http://user:pass@proxy.host:port',
131+
description:
132+
'Optional. Route this request through an http:// proxy (e.g. a residential proxy). Must be http://; the proxy host must be publicly reachable.',
133+
mode: 'advanced',
134+
},
126135
],
127136
tools: {
128137
access: ['http_request'],
@@ -144,6 +153,10 @@ Example:
144153
type: 'boolean',
145154
description: 'Allow retries for non-idempotent methods like POST/PATCH',
146155
},
156+
proxyUrl: {
157+
type: 'string',
158+
description: 'Optional http:// proxy URL to route the request through',
159+
},
147160
},
148161
outputs: {
149162
data: { type: 'json', description: 'API response data (JSON, text, or other formats)' },

apps/sim/lib/core/security/input-validation.server.ts

Lines changed: 76 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,8 @@ import type { LookupFunction } from 'net'
55
import { createLogger } from '@sim/logger'
66
import { toError } from '@sim/utils/errors'
77
import { omit } from '@sim/utils/object'
8+
import { HttpProxyAgent } from 'http-proxy-agent'
9+
import { HttpsProxyAgent } from 'https-proxy-agent'
810
import * as ipaddr from 'ipaddr.js'
911
import { Agent, type RequestInit as UndiciRequestInit, fetch as undiciFetch } from 'undici'
1012
import { isHosted, isPrivateDatabaseHostsAllowed } from '@/lib/core/config/env-flags'
@@ -148,6 +150,61 @@ export async function validateUrlWithDNS(
148150
}
149151
}
150152

153+
/**
154+
* Result of validating a user-supplied HTTP proxy URL.
155+
*/
156+
export interface ProxyValidationResult {
157+
isValid: boolean
158+
/** Proxy URL with hostname rewritten to the resolved IP (creds/port preserved) to pin the proxy connection. */
159+
pinnedProxyUrl?: string
160+
error?: string
161+
}
162+
163+
/**
164+
* Validates a user-supplied HTTP proxy URL and returns an IP-pinned form.
165+
*
166+
* When a request routes through a proxy, the TCP connection targets the proxy
167+
* host (the proxy resolves the destination), so target-IP pinning no longer
168+
* governs egress and the proxy URL becomes the SSRF surface. This function:
169+
* 1. Enforces the `http:` scheme (raw TCP to the proxy, no TLS-to-proxy SNI to
170+
* reconcile, so the host can be safely rewritten to an IP).
171+
* 2. Resolves the proxy host's DNS and blocks private/reserved/loopback IPs via
172+
* {@link validateUrlWithDNS}.
173+
* 3. Pins the connection by rewriting the hostname to the resolved IP while
174+
* preserving credentials/port, closing the DNS-rebinding (TOCTOU) window.
175+
*
176+
* @param proxyUrl - The proxy URL (e.g. `http://user:pass@host:port`)
177+
*/
178+
export async function validateAndPinProxyUrl(
179+
proxyUrl: string | null | undefined
180+
): Promise<ProxyValidationResult> {
181+
if (!proxyUrl || typeof proxyUrl !== 'string') {
182+
return { isValid: false, error: 'proxyUrl must be a string' }
183+
}
184+
185+
let parsed: URL
186+
try {
187+
parsed = new URL(proxyUrl)
188+
} catch {
189+
return { isValid: false, error: 'proxyUrl must be a valid URL' }
190+
}
191+
192+
if (parsed.protocol !== 'http:') {
193+
return {
194+
isValid: false,
195+
error: 'proxyUrl must use http:// (https/socks proxies are not supported)',
196+
}
197+
}
198+
199+
const validation = await validateUrlWithDNS(proxyUrl, 'proxyUrl', { allowHttp: true })
200+
if (!validation.isValid) {
201+
return { isValid: false, error: validation.error }
202+
}
203+
204+
parsed.hostname = validation.resolvedIP!
205+
return { isValid: true, pinnedProxyUrl: parsed.toString() }
206+
}
207+
151208
/**
152209
* Validates a database hostname by resolving DNS and checking the resolved IP
153210
* against private/reserved ranges to prevent SSRF via database connections.
@@ -343,6 +400,12 @@ export interface SecureFetchOptions {
343400
signal?: AbortSignal
344401
/** Drop the Authorization header when following a redirect, so it is not sent to the redirect target's origin. */
345402
stripAuthOnRedirect?: boolean
403+
/**
404+
* Pre-validated, IP-pinned `http://` proxy URL (see {@link validateAndPinProxyUrl}).
405+
* When set, the connection routes through this proxy and target-IP pinning is
406+
* bypassed (the proxy resolves the target).
407+
*/
408+
proxyUrl?: string
346409
}
347410

348411
export class SecureFetchHeaders {
@@ -678,11 +741,19 @@ export async function secureFetchWithPinnedIP(
678741
const defaultPort = isHttps ? 443 : 80
679742
const port = parsed.port ? Number.parseInt(parsed.port, 10) : defaultPort
680743

681-
const lookup = createPinnedLookup(resolvedIP)
682-
683-
const agentOptions: http.AgentOptions = { lookup }
684-
685-
const agent = isHttps ? new https.Agent(agentOptions) : new http.Agent(agentOptions)
744+
let agent: http.Agent
745+
if (options.proxyUrl) {
746+
// The proxy connection is already IP-pinned by validateAndPinProxyUrl; routing
747+
// through the proxy intentionally bypasses target-IP pinning (the proxy
748+
// resolves the target). Agent choice keys off the target protocol: an
749+
// https target tunnels via CONNECT, an http target uses absolute-URI
750+
// forwarding - both over the plain-http proxy connection.
751+
agent = isHttps ? new HttpsProxyAgent(options.proxyUrl) : new HttpProxyAgent(options.proxyUrl)
752+
} else {
753+
const lookup = createPinnedLookup(resolvedIP)
754+
const agentOptions: http.AgentOptions = { lookup }
755+
agent = isHttps ? new https.Agent(agentOptions) : new http.Agent(agentOptions)
756+
}
686757

687758
const { 'accept-encoding': _, ...sanitizedHeaders } = options.headers ?? {}
688759

apps/sim/lib/core/security/input-validation.test.ts

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ import {
2828
} from '@/lib/core/security/input-validation'
2929
import {
3030
isPrivateOrReservedIP,
31+
validateAndPinProxyUrl,
3132
validateDatabaseHost,
3233
validateUrlWithDNS,
3334
} from '@/lib/core/security/input-validation.server'
@@ -843,6 +844,54 @@ describe('validateDatabaseHost', () => {
843844
})
844845
})
845846

847+
describe('validateAndPinProxyUrl', () => {
848+
it('should reject a null/empty proxy URL', async () => {
849+
expect((await validateAndPinProxyUrl(null)).isValid).toBe(false)
850+
expect((await validateAndPinProxyUrl('')).isValid).toBe(false)
851+
})
852+
853+
it('should reject a malformed URL', async () => {
854+
const result = await validateAndPinProxyUrl('not a url')
855+
expect(result.isValid).toBe(false)
856+
expect(result.error).toContain('valid URL')
857+
})
858+
859+
it('should reject an https:// proxy scheme', async () => {
860+
const result = await validateAndPinProxyUrl('https://proxy.example.com:8080')
861+
expect(result.isValid).toBe(false)
862+
expect(result.error).toContain('http://')
863+
})
864+
865+
it('should reject a socks5:// proxy scheme', async () => {
866+
const result = await validateAndPinProxyUrl('socks5://proxy.example.com:1080')
867+
expect(result.isValid).toBe(false)
868+
expect(result.error).toContain('http://')
869+
})
870+
871+
it('should reject a proxy host that is a private IP', async () => {
872+
const result = await validateAndPinProxyUrl('http://user:pass@192.168.1.1:8080')
873+
expect(result.isValid).toBe(false)
874+
expect(result.error).toMatch(/private IP|blocked IP/)
875+
})
876+
877+
it('should reject a proxy host that is the metadata IP', async () => {
878+
const result = await validateAndPinProxyUrl('http://169.254.169.254:80')
879+
expect(result.isValid).toBe(false)
880+
expect(result.error).toMatch(/private IP|blocked IP/)
881+
})
882+
883+
it('should accept a public proxy host and pin the hostname to the resolved IP, preserving creds/port', async () => {
884+
const result = await validateAndPinProxyUrl('http://user:pass@8.8.8.8:8080')
885+
expect(result.isValid).toBe(true)
886+
const pinned = new URL(result.pinnedProxyUrl!)
887+
expect(pinned.protocol).toBe('http:')
888+
expect(pinned.hostname).toBe('8.8.8.8')
889+
expect(pinned.username).toBe('user')
890+
expect(pinned.password).toBe('pass')
891+
expect(pinned.port).toBe('8080')
892+
})
893+
})
894+
846895
describe('validateInteger', () => {
847896
describe('valid integers', () => {
848897
it.concurrent('should accept positive integers', () => {

apps/sim/package.json

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -155,6 +155,8 @@
155155
"gray-matter": "^4.0.3",
156156
"groq-sdk": "^0.15.0",
157157
"html-to-text": "^9.0.5",
158+
"http-proxy-agent": "7.0.2",
159+
"https-proxy-agent": "7.0.6",
158160
"idb-keyval": "6.2.2",
159161
"image-size": "1.2.1",
160162
"imapflow": "1.2.4",

apps/sim/tools/http/request.ts

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,11 @@ export const requestTool: ToolConfig<RequestParams, RequestResponse> = {
5151
visibility: 'user-or-llm',
5252
description: 'Form data to send (will set appropriate Content-Type)',
5353
},
54+
proxyUrl: {
55+
type: 'string',
56+
visibility: 'user-only',
57+
description: 'Route the request through an http:// proxy (e.g. http://user:pass@host:port).',
58+
},
5459
timeout: {
5560
type: 'number',
5661
visibility: 'user-only',

apps/sim/tools/http/types.ts

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ export interface RequestParams {
88
params?: TableRow[] | string
99
pathParams?: Record<string, string>
1010
formData?: Record<string, string | Blob>
11+
proxyUrl?: string
1112
timeout?: number
1213
retries?: number
1314
retryDelayMs?: number

apps/sim/tools/index.test.ts

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -958,6 +958,77 @@ describe('Automatic Internal Route Detection', () => {
958958
Object.assign(tools, originalTools)
959959
})
960960

961+
it('should validate + pin a proxyUrl param and pass it to secureFetchWithPinnedIP', async () => {
962+
inputValidationMockFns.mockValidateAndPinProxyUrl.mockResolvedValue({
963+
isValid: true,
964+
pinnedProxyUrl: 'http://user:pass@1.2.3.4:8080/',
965+
})
966+
967+
const mockTool = {
968+
id: 'test_external_proxy',
969+
name: 'Test External Proxy Tool',
970+
description: 'A test tool that routes through a proxy',
971+
version: '1.0.0',
972+
params: {},
973+
request: {
974+
url: 'https://api.example.com/endpoint',
975+
method: 'GET',
976+
headers: () => ({ 'Content-Type': 'application/json' }),
977+
},
978+
transformResponse: vi.fn().mockResolvedValue({ success: true, output: {} }),
979+
}
980+
981+
const originalTools = { ...tools }
982+
;(tools as any).test_external_proxy = mockTool
983+
984+
await executeTool('test_external_proxy', { proxyUrl: 'http://user:pass@proxy.host:8080' })
985+
986+
expect(inputValidationMockFns.mockValidateAndPinProxyUrl).toHaveBeenCalledWith(
987+
'http://user:pass@proxy.host:8080'
988+
)
989+
expect(mockSecureFetchWithPinnedIP).toHaveBeenCalledWith(
990+
'https://api.example.com/endpoint',
991+
'93.184.216.34',
992+
expect.objectContaining({ proxyUrl: 'http://user:pass@1.2.3.4:8080/' })
993+
)
994+
995+
Object.assign(tools, originalTools)
996+
})
997+
998+
it('should throw when the proxyUrl param fails validation', async () => {
999+
inputValidationMockFns.mockValidateAndPinProxyUrl.mockResolvedValue({
1000+
isValid: false,
1001+
error: 'proxyUrl must use http:// (https/socks proxies are not supported)',
1002+
})
1003+
1004+
const mockTool = {
1005+
id: 'test_external_bad_proxy',
1006+
name: 'Test External Bad Proxy Tool',
1007+
description: 'A test tool with an invalid proxy',
1008+
version: '1.0.0',
1009+
params: {},
1010+
request: {
1011+
url: 'https://api.example.com/endpoint',
1012+
method: 'GET',
1013+
headers: () => ({ 'Content-Type': 'application/json' }),
1014+
},
1015+
transformResponse: vi.fn().mockResolvedValue({ success: true, output: {} }),
1016+
}
1017+
1018+
const originalTools = { ...tools }
1019+
;(tools as any).test_external_bad_proxy = mockTool
1020+
1021+
const result = await executeTool('test_external_bad_proxy', {
1022+
proxyUrl: 'https://proxy.host:8080',
1023+
})
1024+
1025+
expect(result.success).toBe(false)
1026+
expect(result.error).toContain('Invalid proxy URL')
1027+
expect(mockSecureFetchWithPinnedIP).not.toHaveBeenCalled()
1028+
1029+
Object.assign(tools, originalTools)
1030+
})
1031+
9611032
it('should handle dynamic URLs that resolve to internal routes', async () => {
9621033
const mockTool = {
9631034
id: 'test_dynamic_internal',

apps/sim/tools/index.ts

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ import { DEFAULT_EXECUTION_TIMEOUT_MS, getMaxExecutionTimeout } from '@/lib/core
1414
import { getHostedKeyRateLimiter } from '@/lib/core/rate-limiter'
1515
import {
1616
secureFetchWithPinnedIP,
17+
validateAndPinProxyUrl,
1718
validateUrlWithDNS,
1819
} from '@/lib/core/security/input-validation.server'
1920
import { PlatformEvents } from '@/lib/core/telemetry'
@@ -1809,13 +1810,23 @@ async function executeToolRequest(
18091810
throw new Error(`Invalid tool URL: ${urlValidation.error}`)
18101811
}
18111812

1813+
let proxyOption: string | undefined
1814+
if (requestParams.proxyUrl) {
1815+
const proxyValidation = await validateAndPinProxyUrl(requestParams.proxyUrl)
1816+
if (!proxyValidation.isValid) {
1817+
throw new Error(`Invalid proxy URL: ${proxyValidation.error}`)
1818+
}
1819+
proxyOption = proxyValidation.pinnedProxyUrl
1820+
}
1821+
18121822
const secureResponse = await secureFetchWithPinnedIP(fullUrl, urlValidation.resolvedIP!, {
18131823
method: requestParams.method,
18141824
headers: headersRecord,
18151825
body: requestParams.body ?? undefined,
18161826
timeout: requestParams.timeout,
18171827
maxResponseBytes: MAX_TOOL_RESPONSE_BODY_BYTES,
18181828
signal,
1829+
proxyUrl: proxyOption,
18191830
})
18201831

18211832
const responseHeaders = new Headers(secureResponse.headers.toRecord())

apps/sim/tools/utils.test.ts

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -209,6 +209,17 @@ describe('formatRequestParams', () => {
209209

210210
expect(result.body).toBe('{"prompt": "Hello"}\n{"prompt": "World"}')
211211
})
212+
213+
it.concurrent('should pass through a non-empty proxyUrl (trimmed)', () => {
214+
const result = formatRequestParams(mockTool, { proxyUrl: ' http://user:pass@host:8080 ' })
215+
expect(result.proxyUrl).toBe('http://user:pass@host:8080')
216+
})
217+
218+
it.concurrent('should omit proxyUrl when blank, whitespace, or absent', () => {
219+
expect(formatRequestParams(mockTool, {}).proxyUrl).toBeUndefined()
220+
expect(formatRequestParams(mockTool, { proxyUrl: '' }).proxyUrl).toBeUndefined()
221+
expect(formatRequestParams(mockTool, { proxyUrl: ' ' }).proxyUrl).toBeUndefined()
222+
})
212223
})
213224

214225
describe('validateRequiredParametersAfterMerge', () => {

apps/sim/tools/utils.ts

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,7 @@ export interface RequestParams {
8282
headers: Record<string, string>
8383
body?: string
8484
timeout?: number
85+
proxyUrl?: string
8586
}
8687

8788
/**
@@ -137,7 +138,12 @@ export function formatRequestParams(tool: ToolConfig, params: Record<string, any
137138
? Math.min(timeout, MAX_TIMEOUT_MS)
138139
: undefined
139140

140-
return { url, method, headers, body, timeout: validTimeout }
141+
const proxyUrl =
142+
typeof params.proxyUrl === 'string' && params.proxyUrl.trim()
143+
? params.proxyUrl.trim()
144+
: undefined
145+
146+
return { url, method, headers, body, timeout: validTimeout, proxyUrl }
141147
}
142148

143149
/**

0 commit comments

Comments
 (0)