Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion concepts/checks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ Prompt-driven synthetic monitoring where an agent discovers a user journey, crea
- AI-powered support, search, or assistant flows where the exact response is nondeterministic
- Early monitoring coverage before writing deterministic scripts
- Flows that benefit from AI-assisted discovery and maintenance
- Checks that need explicit access to selected skills and environment variables
- Checks that need selected skills or account variables referenced in the prompt
</Accordion>

<Accordion title="Multistep Checks">
Expand Down
34 changes: 11 additions & 23 deletions constructs/agentic-check.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Before creating Agentic Checks, ensure you have:
- An initialized Checkly CLI project
- Access to Agentic Checks on your Checkly account
- Available Agentic Check active capacity if the check is active
- Any required test credentials saved as Checkly environment variables
- Any required test credentials saved as Checkly variables or secrets

For additional setup information, see [CLI overview](/cli/overview).
</Accordion>
Expand All @@ -34,21 +34,17 @@ new AgenticCheck('homepage-health', {
})
```

```typescript Runtime Access
```typescript Skills
import { AgenticCheck, Frequency } from 'checkly/constructs'

new AgenticCheck('login-flow', {
name: 'Login Flow',
prompt: 'Sign in to the test account and verify that the dashboard loads.',
prompt: 'Sign in with {{TEST_USER_EMAIL}} and {{TEST_USER_PASSWORD}}, then verify that the dashboard loads.',
activated: true,
frequency: Frequency.EVERY_1H,
tags: ['agentic', 'critical'],
agentRuntime: {
skills: ['addyosmani/web-quality-skills'],
exposeEnvironmentVariables: [
'TEST_USER_EMAIL',
{ name: 'TEST_USER_PASSWORD', description: 'Password for the test account' },
],
},
})
```
Expand All @@ -66,7 +62,7 @@ The Agentic Check configuration consists of Agentic-specific options and a subse
|-----------|------|----------|---------|-------------|
| `prompt` | `string` | Yes | - | Monitoring goal the agent should verify. Maximum 10,000 characters. |
| `frequency` | `AgenticCheckFrequency` | No | `Frequency.EVERY_30M` | How often the check should run. Agentic Checks support the same frequency values as other checks, down to `5` minutes. |
| `agentRuntime` | `object` | No | `{ skills: [], exposeEnvironmentVariables: [] }` | Runtime access the agent is allowed to use during execution. |
| `agentRuntime` | `object` | No | `{ skills: [] }` | Extra runtime skills the agent is allowed to use during execution. |

</Tab>
<Tab title="General Check">
Expand Down Expand Up @@ -100,7 +96,7 @@ See [Prompt best practices](/detect/synthetic-monitoring/agentic-checks/overview
A reliable Agentic Check prompt usually states:

- **Goal**: The user journey or system behavior to monitor.
- **What you are given**: The start URL, test account context, and any required environment variables.
- **What you are given**: The start URL, test account context, and any required variables or secrets using `{{VARIABLE_NAME}}` notation.
- **Success looks like**: The exact outcome that should pass.
- **Failure looks like**: The states that should fail the check.
- **Strictness**: Whether the agent should assert exact values or evaluate the outcome more flexibly.
Expand Down Expand Up @@ -134,9 +130,6 @@ new AgenticCheck('login-flow', {
## Failure looks like
The login form rejects valid credentials, the page stays on the login screen, an MFA prompt blocks the flow, or the dashboard shows a 4xx/5xx error.
`,
agentRuntime: {
exposeEnvironmentVariables: ['TEST_USER_EMAIL', 'TEST_USER_PASSWORD'],
},
})
```

Expand Down Expand Up @@ -193,31 +186,26 @@ new AgenticCheck('regional-login-flow', {

### `agentRuntime`

Controls what the agent can access while executing the check.
Controls extra skills the agent can access while executing the check.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `skills` | `string[]` | No | `[]` | Extra skill packages to load into the agent runtime. |
| `exposeEnvironmentVariables` | `(string \| object)[]` | No | `[]` | Environment variables the agent is allowed to read. |

`exposeEnvironmentVariables` accepts either a string variable name or an object with `name` and optional `description`.
Reference account variables and secrets in the `prompt` with double curly brackets, such as `{{API_TOKEN}}`. This is the only supported way to expose account variables and secrets to an Agentic Check. The `AgenticCheck` construct does not expose a separate environment-variable option.

```typescript
new AgenticCheck('authenticated-api-check', {
name: 'Authenticated API Check',
prompt: 'Verify that the authenticated API health endpoint returns a healthy response.',
new AgenticCheck('web-quality-check', {
name: 'Web Quality Check',
prompt: 'Review the homepage and verify that the main navigation and primary call to action are usable.',
agentRuntime: {
skills: ['addyosmani/web-quality-skills'],
exposeEnvironmentVariables: [
'API_KEY',
{ name: 'TEST_USER_PASSWORD', description: 'Login password for the test account' },
],
},
})
```

<Warning>
Variables not listed in `agentRuntime.exposeEnvironmentVariables` are not exposed to the agent runtime, even if they exist in your Checkly account.
Do not paste secret values into the prompt. Store secrets in Checkly and reference them by name, for example `{{TEST_USER_PASSWORD}}`.
</Warning>

## Billing behavior
Expand Down
20 changes: 7 additions & 13 deletions detect/synthetic-monitoring/agentic-checks/configuration.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: 'Agentic Checks Configuration'
description: 'Configure Agentic Checks with prompts, runtime access, scheduling, locations, and active capacity.'
description: 'Configure Agentic Checks with prompts, variables, skills, scheduling, locations, and active capacity.'
sidebarTitle: 'Configuration'
---

Expand All @@ -10,7 +10,7 @@ Before creating Agentic Checks, ensure you have:
- Access to Agentic Checks on your Checkly account
- Available Agentic Check active capacity if the check will be active
- An initialized Checkly CLI project if you want to manage the check as code
- Any required test credentials saved as Checkly environment variables
- Any required test credentials saved as Checkly variables or secrets

For CLI setup, see the [CLI overview](/cli/overview).
</Accordion>
Expand All @@ -37,32 +37,26 @@ Good prompts include:

Avoid prompts that ask the agent to test too many unrelated paths in one check. Create separate Agentic Checks for separate critical flows.

## Configure runtime access
## Reference variables and secrets

Agentic Checks do **not** automatically expose account environment variables to the agent. Add only the variables the agent needs with `agentRuntime.exposeEnvironmentVariables`.
Agentic Checks do **not** automatically expose all account variables or secrets to the agent. To expose one to an Agentic Check, reference it directly in the prompt with double curly brackets, such as `{{TEST_USER_EMAIL}}`.

```typescript checkout.agentic.check.ts
import { AgenticCheck, Frequency } from 'checkly/constructs'

new AgenticCheck('checkout-flow', {
name: 'Checkout Flow',
prompt: 'Sign in to the test account and verify that a user can reach the checkout review step.',
prompt: 'Sign in with {{TEST_USER_EMAIL}} and {{TEST_USER_PASSWORD}}, then verify that a user can reach the checkout review step.',
frequency: Frequency.EVERY_1H,
activated: true,
agentRuntime: {
exposeEnvironmentVariables: [
{ name: 'TEST_USER_EMAIL', description: 'Email address for the checkout test account' },
{ name: 'TEST_USER_PASSWORD', description: 'Password for the checkout test account' },
],
},
})
```

<Warning>
Do not paste secrets directly into the prompt. Store secrets as Checkly environment variables and expose only the variables the agent needs for this check.
Do not paste secret values directly into the prompt. Store secrets in Checkly and reference them by name with `{{SECRET_NAME}}`.
</Warning>

Some applications block automated monitoring unless requests include required headers, test bypass tokens, or feature flags. If your flow depends on a header value, expose that value as an environment variable and say in the prompt which header the agent should send. Do not paste the header value directly into the prompt.
Some applications block automated monitoring unless requests include required headers, test bypass tokens, or feature flags. If your flow depends on a header value, store that value as a Checkly variable or secret and say in the prompt which header the agent should send. Do not paste the header value directly into the prompt.

## Add skills

Expand Down
17 changes: 9 additions & 8 deletions detect/synthetic-monitoring/agentic-checks/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ The prompt is the monitoring contract for an Agentic Check. Write it as a focuse
A reliable prompt usually includes:

- **Goal**: The user journey or system behavior to monitor.
- **What you are given**: The start URL, test account context, and any required environment variables.
- **What you are given**: The start URL, test account context, and any required variables or secrets using `{{VARIABLE_NAME}}` notation.
- **Success looks like**: The exact outcome that should pass.
- **Failure looks like**: The states that should fail the check.
- **Strictness**: Whether the agent should assert exact values or evaluate the outcome more flexibly.
Expand Down Expand Up @@ -59,18 +59,19 @@ Each run can produce:
Agentic Checks are not a replacement for every deterministic check. Use them for important flows where the agent's discovery and self-healing behavior is valuable. Use [API Checks](/detect/synthetic-monitoring/api-checks/overview), [Browser Checks](/detect/synthetic-monitoring/browser-checks/overview), or [Playwright Check Suites](/detect/synthetic-monitoring/playwright-checks/overview) when you need fully deterministic scripted monitoring.
</Note>

## Runtime access
## Variables, secrets, and skills

Agentic Checks run with a small default runtime for browser automation. You can extend that runtime with skills and explicitly expose environment variables that the agent is allowed to read.
Agentic Checks run with a small default runtime for browser automation. You can extend that runtime with skills.

Treat runtime access as a security boundary:
To use account variables or secrets, reference them directly in the prompt with double curly brackets, such as `{{TEST_USER_EMAIL}}` or `{{API_TOKEN}}`. This is the only way to expose Checkly account variables and secrets to an Agentic Check.

- Only expose the variables the agent needs.
- Add descriptions for credentials so the agent knows when to use them.
Treat this access as a security boundary:

- Only reference the variables and secrets the agent needs.
- Do not put secrets directly in prompts.
- Review generated checks before deploying them to production monitoring.

Learn how to configure runtime access in the [Agentic Checks configuration guide](/detect/synthetic-monitoring/agentic-checks/configuration).
Learn how to configure variables, secrets, and skills in the [Agentic Checks configuration guide](/detect/synthetic-monitoring/agentic-checks/configuration).

## Billing and entitlements

Expand All @@ -91,7 +92,7 @@ Your billing page is the source of truth for the exact Agentic Check capacity in

<Columns cols={2}>
<Card title="Configure Agentic Checks" href="/detect/synthetic-monitoring/agentic-checks/configuration">
Define prompts, runtime access, and scheduling behavior.
Define prompts, skills, variables, and scheduling behavior.
</Card>

<Card title="Agentic Check Construct" href="/constructs/agentic-check">
Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -23,5 +23,5 @@
"testing"
],
"author": "Checkly Team",
"license": "MIT",
"license": "MIT"
}