diff --git a/concepts/checks.mdx b/concepts/checks.mdx index 8d59e15c..b92e87bd 100644 --- a/concepts/checks.mdx +++ b/concepts/checks.mdx @@ -80,7 +80,7 @@ Prompt-driven synthetic monitoring where an agent discovers a user journey, crea - AI-powered support, search, or assistant flows where the exact response is nondeterministic - Early monitoring coverage before writing deterministic scripts - Flows that benefit from AI-assisted discovery and maintenance -- Checks that need explicit access to selected skills and environment variables +- Checks that need selected skills or account variables referenced in the prompt diff --git a/constructs/agentic-check.mdx b/constructs/agentic-check.mdx index b869c67a..0a1c88b0 100644 --- a/constructs/agentic-check.mdx +++ b/constructs/agentic-check.mdx @@ -16,7 +16,7 @@ Before creating Agentic Checks, ensure you have: - An initialized Checkly CLI project - Access to Agentic Checks on your Checkly account - Available Agentic Check active capacity if the check is active -- Any required test credentials saved as Checkly environment variables +- Any required test credentials saved as Checkly variables or secrets For additional setup information, see [CLI overview](/cli/overview). @@ -34,21 +34,17 @@ new AgenticCheck('homepage-health', { }) ``` -```typescript Runtime Access +```typescript Skills import { AgenticCheck, Frequency } from 'checkly/constructs' new AgenticCheck('login-flow', { name: 'Login Flow', - prompt: 'Sign in to the test account and verify that the dashboard loads.', + prompt: 'Sign in with {{TEST_USER_EMAIL}} and {{TEST_USER_PASSWORD}}, then verify that the dashboard loads.', activated: true, frequency: Frequency.EVERY_1H, tags: ['agentic', 'critical'], agentRuntime: { skills: ['addyosmani/web-quality-skills'], - exposeEnvironmentVariables: [ - 'TEST_USER_EMAIL', - { name: 'TEST_USER_PASSWORD', description: 'Password for the test account' }, - ], }, }) ``` @@ -66,7 +62,7 @@ The Agentic Check configuration consists of Agentic-specific options and a subse |-----------|------|----------|---------|-------------| | `prompt` | `string` | Yes | - | Monitoring goal the agent should verify. Maximum 10,000 characters. | | `frequency` | `AgenticCheckFrequency` | No | `Frequency.EVERY_30M` | How often the check should run. Agentic Checks support the same frequency values as other checks, down to `5` minutes. | -| `agentRuntime` | `object` | No | `{ skills: [], exposeEnvironmentVariables: [] }` | Runtime access the agent is allowed to use during execution. | +| `agentRuntime` | `object` | No | `{ skills: [] }` | Extra runtime skills the agent is allowed to use during execution. | @@ -100,7 +96,7 @@ See [Prompt best practices](/detect/synthetic-monitoring/agentic-checks/overview A reliable Agentic Check prompt usually states: - **Goal**: The user journey or system behavior to monitor. -- **What you are given**: The start URL, test account context, and any required environment variables. +- **What you are given**: The start URL, test account context, and any required variables or secrets using `{{VARIABLE_NAME}}` notation. - **Success looks like**: The exact outcome that should pass. - **Failure looks like**: The states that should fail the check. - **Strictness**: Whether the agent should assert exact values or evaluate the outcome more flexibly. @@ -134,9 +130,6 @@ new AgenticCheck('login-flow', { ## Failure looks like The login form rejects valid credentials, the page stays on the login screen, an MFA prompt blocks the flow, or the dashboard shows a 4xx/5xx error. `, - agentRuntime: { - exposeEnvironmentVariables: ['TEST_USER_EMAIL', 'TEST_USER_PASSWORD'], - }, }) ``` @@ -193,31 +186,26 @@ new AgenticCheck('regional-login-flow', { ### `agentRuntime` -Controls what the agent can access while executing the check. +Controls extra skills the agent can access while executing the check. | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| | `skills` | `string[]` | No | `[]` | Extra skill packages to load into the agent runtime. | -| `exposeEnvironmentVariables` | `(string \| object)[]` | No | `[]` | Environment variables the agent is allowed to read. | -`exposeEnvironmentVariables` accepts either a string variable name or an object with `name` and optional `description`. +Reference account variables and secrets in the `prompt` with double curly brackets, such as `{{API_TOKEN}}`. This is the only supported way to expose account variables and secrets to an Agentic Check. The `AgenticCheck` construct does not expose a separate environment-variable option. ```typescript -new AgenticCheck('authenticated-api-check', { - name: 'Authenticated API Check', - prompt: 'Verify that the authenticated API health endpoint returns a healthy response.', +new AgenticCheck('web-quality-check', { + name: 'Web Quality Check', + prompt: 'Review the homepage and verify that the main navigation and primary call to action are usable.', agentRuntime: { skills: ['addyosmani/web-quality-skills'], - exposeEnvironmentVariables: [ - 'API_KEY', - { name: 'TEST_USER_PASSWORD', description: 'Login password for the test account' }, - ], }, }) ``` -Variables not listed in `agentRuntime.exposeEnvironmentVariables` are not exposed to the agent runtime, even if they exist in your Checkly account. +Do not paste secret values into the prompt. Store secrets in Checkly and reference them by name, for example `{{TEST_USER_PASSWORD}}`. ## Billing behavior diff --git a/detect/synthetic-monitoring/agentic-checks/configuration.mdx b/detect/synthetic-monitoring/agentic-checks/configuration.mdx index e51137eb..26e7aa1a 100644 --- a/detect/synthetic-monitoring/agentic-checks/configuration.mdx +++ b/detect/synthetic-monitoring/agentic-checks/configuration.mdx @@ -1,6 +1,6 @@ --- title: 'Agentic Checks Configuration' -description: 'Configure Agentic Checks with prompts, runtime access, scheduling, locations, and active capacity.' +description: 'Configure Agentic Checks with prompts, variables, skills, scheduling, locations, and active capacity.' sidebarTitle: 'Configuration' --- @@ -10,7 +10,7 @@ Before creating Agentic Checks, ensure you have: - Access to Agentic Checks on your Checkly account - Available Agentic Check active capacity if the check will be active - An initialized Checkly CLI project if you want to manage the check as code -- Any required test credentials saved as Checkly environment variables +- Any required test credentials saved as Checkly variables or secrets For CLI setup, see the [CLI overview](/cli/overview). @@ -37,32 +37,26 @@ Good prompts include: Avoid prompts that ask the agent to test too many unrelated paths in one check. Create separate Agentic Checks for separate critical flows. -## Configure runtime access +## Reference variables and secrets -Agentic Checks do **not** automatically expose account environment variables to the agent. Add only the variables the agent needs with `agentRuntime.exposeEnvironmentVariables`. +Agentic Checks do **not** automatically expose all account variables or secrets to the agent. To expose one to an Agentic Check, reference it directly in the prompt with double curly brackets, such as `{{TEST_USER_EMAIL}}`. ```typescript checkout.agentic.check.ts import { AgenticCheck, Frequency } from 'checkly/constructs' new AgenticCheck('checkout-flow', { name: 'Checkout Flow', - prompt: 'Sign in to the test account and verify that a user can reach the checkout review step.', + prompt: 'Sign in with {{TEST_USER_EMAIL}} and {{TEST_USER_PASSWORD}}, then verify that a user can reach the checkout review step.', frequency: Frequency.EVERY_1H, activated: true, - agentRuntime: { - exposeEnvironmentVariables: [ - { name: 'TEST_USER_EMAIL', description: 'Email address for the checkout test account' }, - { name: 'TEST_USER_PASSWORD', description: 'Password for the checkout test account' }, - ], - }, }) ``` -Do not paste secrets directly into the prompt. Store secrets as Checkly environment variables and expose only the variables the agent needs for this check. +Do not paste secret values directly into the prompt. Store secrets in Checkly and reference them by name with `{{SECRET_NAME}}`. -Some applications block automated monitoring unless requests include required headers, test bypass tokens, or feature flags. If your flow depends on a header value, expose that value as an environment variable and say in the prompt which header the agent should send. Do not paste the header value directly into the prompt. +Some applications block automated monitoring unless requests include required headers, test bypass tokens, or feature flags. If your flow depends on a header value, store that value as a Checkly variable or secret and say in the prompt which header the agent should send. Do not paste the header value directly into the prompt. ## Add skills diff --git a/detect/synthetic-monitoring/agentic-checks/overview.mdx b/detect/synthetic-monitoring/agentic-checks/overview.mdx index ea0b370a..13d48eda 100644 --- a/detect/synthetic-monitoring/agentic-checks/overview.mdx +++ b/detect/synthetic-monitoring/agentic-checks/overview.mdx @@ -29,7 +29,7 @@ The prompt is the monitoring contract for an Agentic Check. Write it as a focuse A reliable prompt usually includes: - **Goal**: The user journey or system behavior to monitor. -- **What you are given**: The start URL, test account context, and any required environment variables. +- **What you are given**: The start URL, test account context, and any required variables or secrets using `{{VARIABLE_NAME}}` notation. - **Success looks like**: The exact outcome that should pass. - **Failure looks like**: The states that should fail the check. - **Strictness**: Whether the agent should assert exact values or evaluate the outcome more flexibly. @@ -59,18 +59,19 @@ Each run can produce: Agentic Checks are not a replacement for every deterministic check. Use them for important flows where the agent's discovery and self-healing behavior is valuable. Use [API Checks](/detect/synthetic-monitoring/api-checks/overview), [Browser Checks](/detect/synthetic-monitoring/browser-checks/overview), or [Playwright Check Suites](/detect/synthetic-monitoring/playwright-checks/overview) when you need fully deterministic scripted monitoring. -## Runtime access +## Variables, secrets, and skills -Agentic Checks run with a small default runtime for browser automation. You can extend that runtime with skills and explicitly expose environment variables that the agent is allowed to read. +Agentic Checks run with a small default runtime for browser automation. You can extend that runtime with skills. -Treat runtime access as a security boundary: +To use account variables or secrets, reference them directly in the prompt with double curly brackets, such as `{{TEST_USER_EMAIL}}` or `{{API_TOKEN}}`. This is the only way to expose Checkly account variables and secrets to an Agentic Check. -- Only expose the variables the agent needs. -- Add descriptions for credentials so the agent knows when to use them. +Treat this access as a security boundary: + +- Only reference the variables and secrets the agent needs. - Do not put secrets directly in prompts. - Review generated checks before deploying them to production monitoring. -Learn how to configure runtime access in the [Agentic Checks configuration guide](/detect/synthetic-monitoring/agentic-checks/configuration). +Learn how to configure variables, secrets, and skills in the [Agentic Checks configuration guide](/detect/synthetic-monitoring/agentic-checks/configuration). ## Billing and entitlements @@ -91,7 +92,7 @@ Your billing page is the source of truth for the exact Agentic Check capacity in -Define prompts, runtime access, and scheduling behavior. +Define prompts, skills, variables, and scheduling behavior. diff --git a/package.json b/package.json index 1b43cac2..69a89d5a 100644 --- a/package.json +++ b/package.json @@ -23,5 +23,5 @@ "testing" ], "author": "Checkly Team", - "license": "MIT", + "license": "MIT" }