From 637a1a6ca11709b35a06a37e461120ed05626b61 Mon Sep 17 00:00:00 2001 From: Herve Labas Date: Wed, 6 May 2026 15:50:03 +0200 Subject: [PATCH 1/4] Document Agentic Checks for self-service billing --- .agents/skills/checkly/SKILL.md | 97 ++++++++++ AGENTS.md | 102 ++++++++++ concepts/checks.mdx | 13 +- constructs/agentic-check.mdx | 174 ++++++++++++++++++ detect/overview.mdx | 7 +- .../agentic-checks/configuration.mdx | 134 ++++++++++++++ .../agentic-checks/overview.mdx | 80 ++++++++ detect/synthetic-monitoring/overview.mdx | 10 +- docs.json | 8 + 9 files changed, 619 insertions(+), 6 deletions(-) create mode 100644 .agents/skills/checkly/SKILL.md create mode 100644 AGENTS.md create mode 100644 constructs/agentic-check.mdx create mode 100644 detect/synthetic-monitoring/agentic-checks/configuration.mdx create mode 100644 detect/synthetic-monitoring/agentic-checks/overview.mdx diff --git a/.agents/skills/checkly/SKILL.md b/.agents/skills/checkly/SKILL.md new file mode 100644 index 00000000..896a4fd0 --- /dev/null +++ b/.agents/skills/checkly/SKILL.md @@ -0,0 +1,97 @@ +--- +name: checkly +description: Set up, create, test and manage monitoring checks using the Checkly CLI. Use when working with API Checks, Browser Checks, URL Monitors, ICMP Monitors, Playwright Check Suites, Heartbeat Monitors, Alert Channels, Dashboards, or Status Pages. +allowed-tools: Bash(npx:checkly:*) Bash(npm:install:*) +metadata: + author: checkly +--- + +# Checkly + +The Checkly CLI provides all the required information via the `npx checkly skills` command. + +Use `npx checkly skills install` to install this skill into your project (supports Codex, Cursor, Codex and more). + +Use `npx checkly skills` to list all available actions, and `npx checkly skills ` to access up-to-date information on how to use the Checkly CLI for each action. + +## Progressive Disclosure via `npx checkly skills` + +The skill is structured for efficient context usage: + +1. **Metadata** (~80 tokens): Name and description in frontmatter +2. **Core Instructions** (~1K tokens): Main SKILL.md content with links to reference commands +3. **Reference Commands** (loaded on demand): Detailed construct documentation with examples + +Agents load what they need for each task. + +## Plan Awareness + +Before configuring checks, run `npx checkly account plan --output json` to see what features, locations, and limits are available on the current plan. Disabled features include an `upgradeUrl` pointing to the self-service checkout page or the enterprise contact sales page — share these with the user when they need a feature that's not on their plan. + +Run `npx checkly skills manage` for the full reference. + +## Confirmation Protocol + +Write commands (e.g. `incidents create`, `deploy`, `destroy`) return exit code 2 with a `confirmation_required` JSON envelope instead of executing. **Always present the `changes` to the user and wait for approval before running the `confirmCommand`.** Never auto-append `--force`. This applies to every write command individually — updates and resolutions need confirmation too, not just the initial create. + +Run `npx checkly skills communicate` for the full protocol details. + +### `npx checkly skills initialize` +Learn how to initialize and set up a new Checkly CLI project from scratch. + +### `npx checkly skills configure` +Learn how to create and manage monitoring checks using Checkly constructs and the CLI. + +#### `npx checkly skills configure api-checks` +Api Check construct (`ApiCheck`), assertions, and authentication setup scripts + +#### `npx checkly skills configure browser-checks` +Browser Check construct (`BrowserCheck`) with Playwright test files + +#### `npx checkly skills configure playwright-checks` +Playwright Check Suite construct (`PlaywrightCheck`) for multi-browser test suites + +#### `npx checkly skills configure multistep-checks` +Multistep Check construct (`MultiStepCheck`) for complex user flows + +#### `npx checkly skills configure tcp-monitors` +TCP Monitor construct (`TcpMonitor`) with assertions + +#### `npx checkly skills configure url-monitors` +URL Monitor construct (`UrlMonitor`) with assertions + +#### `npx checkly skills configure dns-monitors` +DNS Monitor construct (`DnsMonitor`) with assertions + +#### `npx checkly skills configure icmp-monitors` +ICMP Monitor construct (`IcmpMonitor`) with latency and packet loss assertions + +#### `npx checkly skills configure heartbeat-monitors` +Heartbeat Monitor construct (`HeartbeatMonitor`) + +#### `npx checkly skills configure check-groups` +CheckGroupV2 construct (`CheckGroupV2`) for organizing checks + +#### `npx checkly skills configure alert-channels` +Email (`EmailAlertChannel`), Phone (`PhoneCallAlertChannel`), and Slack (`SlackAlertChannel`) alert channels + +#### `npx checkly skills configure supporting-constructs` +Status pages (`StatusPage`), dashboards (`Dashboard`), maintenance windows (`MaintenanceWindow`), and private locations (`PrivateLocation`) + +### `npx checkly skills investigate` +Access check status, analyze failures, and investigate errors. + +#### `npx checkly skills investigate checks` +Inspecting checks (`checks list`, `checks get`) and triggering on-demand runs + +### `npx checkly skills communicate` +Open incidents and lead customer communications via status pages. + +#### `npx checkly skills communicate incidents` +Incident lifecycle (`incidents create`, `update`, `resolve`, `list`) and status pages + +### `npx checkly skills manage` +Understand your account plan, entitlements, and feature limits. + +#### `npx checkly skills manage plan` +Check account plan, entitlements, feature limits, and available locations (`account plan`) diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..41e10d5f --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,102 @@ +# Mintlify documentation + +## Working relationship +- You can push back on ideas-this can lead to better documentation. Cite sources and explain your reasoning when you do so +- ALWAYS ask for clarification rather than making assumptions +- NEVER lie, guess, or make up information + +## Project context +- Framework: Mintlify. https://mintlify.com/docs/llms.txt +- Format: MDX files with YAML frontmatter +- Config: docs.json for navigation, theme, settings. Tabs set top level navigation, Groups set sidebar navigation within Tabs. +- Components: We use Mintlify components and documentation. + -- Headers and text + -- Images and embeds + -- Lists and tables + -- Code + -- Reusable snippets + -- Accordions + -- Callouts + -- Cards + -- Columns + -- Code groups + -- Examples + -- Expandables + -- Fields + -- Frames + -- Icons + -- Mermaid + -- Panel + -- Steps + -- Tabs + -- Tooltips + -- Update + +## Content strategy +- Search for existing information before adding new content. Avoid duplication unless it is done for a strategic reason +- Check existing patterns for consistency +- Start by making the smallest reasonable changes +- Document just enough for user success - not too much. Don't be verbose. +- Prioritize accuracy and usability of information +- Make content evergreen + +## docs.json + +- Refer to the [docs.json schema](https://mintlify.com/docs.json) when building the docs.json file and site navigation +- Tabs +-- Groups +--- Pages + +## Frontmatter requirements for pages +- title: Clear, descriptive page title +- sidebarTitle: Concise title for sidebar +- description: Concise summary for SEO/navigation + +## Writing standards +- Second-person voice ("you") +- Prerequisites at start of procedural content - ALWAYS wrap prerequisites in an Accordion component +- Test all code examples before publishing +- Match style and formatting of existing pages +- Include both basic and advanced use cases +- Language tags on all code blocks +- Alt text on all images +- Relative paths for internal links + +## Component usage rules +- Prerequisites Should Always be wrapped in an Accordion component with title "Prerequisites" or "Before you begin" +- Use Steps component for sequential procedures sparingly. +- Don't use Cards to display information. Only for navigation on Overview type pages +- Use CodeGroups and Dropdowns for code examples that show the same example in different languages +- Use Note/Tip/Warning callouts appropriately + +## CodeGroup Dropdown +- Use `` for code examples that show the same functionality in different programming languages +- The dropdown automatically creates a language selector based on the code block language identifiers +- Example usage: + + +```typescript example.ts +// TypeScript code here +``` + +```javascript example.js +// JavaScript code here +``` + + +- This replaces the older `` pattern for language-specific code examples + +## Git workflow +- NEVER use --no-verify when committing +- Ask how to handle uncommitted changes before starting +- Create a new branch when no clear branch exists for changes +- Commit frequently throughout development +- NEVER skip or disable pre-commit hooks + +## Do not +- Skip frontmatter on any MDX file +- Use absolute URLs for internal links +- Include untested code examples +- Make assumptions - always ask for clarification +- overuse components like Cards or Accordion +​ diff --git a/concepts/checks.mdx b/concepts/checks.mdx index 23f1692f..8d59e15c 100644 --- a/concepts/checks.mdx +++ b/concepts/checks.mdx @@ -72,6 +72,17 @@ Advanced API testing that goes beyond uptime monitoring. Use setup and teardown - Integration testing between services + +Prompt-driven synthetic monitoring where an agent discovers a user journey, creates a reusable check script, and can self-heal when that saved script starts failing. Use Agentic Checks when the expected intent is stable, but the exact implementation or response can vary. + +**Perfect for:** +- Critical user journeys that change often +- AI-powered support, search, or assistant flows where the exact response is nondeterministic +- Early monitoring coverage before writing deterministic scripts +- Flows that benefit from AI-assisted discovery and maintenance +- Checks that need explicit access to selected skills and environment variables + + Write Node.js scripts that run multiple API requests in sequence with arbitrary code between requests. Perfect for testing complex workflows involving authentication, data manipulation, and multi-step processes that span multiple endpoints. @@ -126,5 +137,3 @@ Teardown scripts are run *before* any assertions are validated All check types at Checkly have a Construct or API endpoint - meaning you can use Checkly to bring your monitoring process right into your repository. We call this [Monitoring as Code](/concepts/monitoring-as-code). This approach transforms monitoring from a manual, UI-driven process into a programmable, scalable system that grows with your application. You can version control your monitoring logic, collaborate on it through code reviews, and deploy monitoring changes through the same CI/CD pipelines you use for application code. - - diff --git a/constructs/agentic-check.mdx b/constructs/agentic-check.mdx new file mode 100644 index 00000000..787c2d57 --- /dev/null +++ b/constructs/agentic-check.mdx @@ -0,0 +1,174 @@ +--- +title: 'Agentic Check Construct' +description: 'Learn how to configure Agentic Checks with the Checkly CLI.' +sidebarTitle: 'Agentic Check' +--- + + +Learn more about Agentic Checks in [the Agentic Checks overview](/detect/synthetic-monitoring/agentic-checks/overview). + + +Use Agentic Checks to monitor user journeys from a natural-language prompt. The agent discovers the target, creates a reusable check script, and can self-heal when the saved script starts failing. + + +Before creating Agentic Checks, ensure you have: + +- An initialized Checkly CLI project +- Access to Agentic Checks on your Checkly account +- Available Agentic Check active capacity if the check is active +- Any required test credentials saved as Checkly environment variables + +For additional setup information, see [CLI overview](/cli/overview). + + + + +```typescript Basic Example +import { AgenticCheck, Frequency } from 'checkly/constructs' + +new AgenticCheck('homepage-health', { + name: 'Homepage Health Check', + prompt: 'Navigate to https://example.com and verify the page loads correctly.', + activated: true, + frequency: Frequency.EVERY_1H, +}) +``` + +```typescript Runtime Access +import { AgenticCheck, Frequency } from 'checkly/constructs' + +new AgenticCheck('login-flow', { + name: 'Login Flow', + prompt: 'Sign in to the test account and verify that the dashboard loads.', + activated: true, + frequency: Frequency.EVERY_1H, + tags: ['agentic', 'critical'], + agentRuntime: { + skills: ['addyosmani/web-quality-skills'], + exposeEnvironmentVariables: [ + 'TEST_USER_EMAIL', + { name: 'TEST_USER_PASSWORD', description: 'Password for the test account' }, + ], + }, +}) +``` + + + +## Configuration + +The Agentic Check configuration consists of Agentic-specific options and a subset of inherited general check options. + + + + +| Parameter | Type | Required | Default | Description | +|-----------|------|----------|---------|-------------| +| `prompt` | `string` | Yes | - | Monitoring goal the agent should verify. Maximum 10,000 characters. | +| `frequency` | `AgenticCheckFrequency` | No | `Frequency.EVERY_30M` | How often the check should run. Agentic Checks support the same frequency values as other checks, down to `5` minutes. | +| `agentRuntime` | `object` | No | `{ skills: [], exposeEnvironmentVariables: [] }` | Runtime access the agent is allowed to use during execution. | + + + + +| Property | Type | Required | Default | Description | +|----------|------|----------|---------|-------------| +| `name` | `string` | Yes | - | Friendly name for your check | +| `description` | `string` | No | `null` | Description of the check. Supports markdown. Max 500 characters | +| `activated` | `boolean` | No | `true` | Whether the check is enabled | +| `alertChannels` | `AlertChannel[]` | No | `[]` | Alert channels for notifications | +| `alertEscalationPolicy` | `AlertEscalationPolicy` | No | - | Advanced alert settings | +| `group` | `CheckGroup` | No | - | The CheckGroup this check belongs to | +| `locations` | `(keyof Region)[]` | No | Project default or `['us-east-1']` | Public Checkly locations where the check runs | +| `muted` | `boolean` | No | `false` | Whether alert notifications are muted | +| `tags` | `string[]` | No | `[]` | Tags to organize checks | +| `testOnly` | `boolean` | No | `false` | Only run with `test`, not during deploy | + + + + + +The Agentic Check construct intentionally does not expose `privateLocations`, `runParallel`, `retryStrategy`, `shouldFail`, `doubleCheck`, or `triggerIncident`. These options are not currently honored for Agentic Checks. + + +### `prompt` + +The monitoring goal the agent should verify. Write the prompt in terms of user-visible behavior and expected outcomes. + +```typescript +new AgenticCheck('checkout-flow', { + name: 'Checkout Flow', + prompt: ` + Go to https://shop.example.com. + Sign in with the test account. + Add one item to the cart and verify the checkout review page loads. + `, +}) +``` + +### `frequency` + +How often the Agentic Check should run. Use one of the supported `Frequency` constants or the equivalent number of minutes. + +```typescript +import { AgenticCheck, Frequency } from 'checkly/constructs' + +new AgenticCheck('hourly-agentic-check', { + name: 'Hourly Agentic Check', + prompt: 'Verify the homepage and pricing page load correctly.', + frequency: Frequency.EVERY_1H, +}) +``` + +Supported values: `5`, `10`, `15`, `30`, `60`, `120`, `180`, `360`, `720`, and `1440`. + +### `locations` + +Public Checkly locations where the Agentic Check should run. Accounts can select up to three locations by default. Enterprise accounts can contact sales for unlimited locations. + +```typescript +new AgenticCheck('regional-login-flow', { + name: 'Regional Login Flow', + prompt: 'Verify that a user can sign in and reach the dashboard.', + locations: ['us-east-1', 'eu-west-1', 'ap-southeast-1'], +}) +``` + +### `agentRuntime` + +Controls what the agent can access while executing the check. + +| Parameter | Type | Required | Default | Description | +|-----------|------|----------|---------|-------------| +| `skills` | `string[]` | No | `[]` | Extra skill packages to load into the agent runtime. | +| `exposeEnvironmentVariables` | `(string \| object)[]` | No | `[]` | Environment variables the agent is allowed to read. | + +`exposeEnvironmentVariables` accepts either a string variable name or an object with `name` and optional `description`. + +```typescript +agentRuntime: { + skills: ['addyosmani/web-quality-skills'], + exposeEnvironmentVariables: [ + 'API_KEY', + { name: 'TEST_USER_PASSWORD', description: 'Login password for the test account' }, + ], +} +``` + + +Variables not listed in `agentRuntime.exposeEnvironmentVariables` are not exposed to the agent runtime, even if they exist in your Checkly account. + + +## Billing behavior + +Agentic Checks use active-capacity billing. Active checks consume capacity; inactive checks do not. Use `activated: false` to keep an Agentic Check as a deployed draft without consuming active capacity. + +```typescript +new AgenticCheck('draft-agentic-check', { + name: 'Draft Agentic Check', + prompt: 'Verify the account settings page loads for a signed-in user.', + activated: false, +}) +``` + +If an active Agentic Check would exceed your available capacity, deployment fails with a billing entitlement error. Deactivate an existing Agentic Check or purchase additional Agentic Check capacity before deploying. diff --git a/detect/overview.mdx b/detect/overview.mdx index 9a54fed6..e640d1fc 100644 --- a/detect/overview.mdx +++ b/detect/overview.mdx @@ -12,7 +12,7 @@ Monitor availability and response times across your website and infrastructure -Proactively validate real user interactions with API, browser, and multistep checks +Proactively validate real user interactions with API, browser, multistep, and Agentic Checks @@ -62,6 +62,7 @@ Implement testing and monitoring across multiple layers for a comprehensive view - **Uptime Monitoring** — Monitor servers, databases, and network connectivity to ensure availability and responsiveness - **API Synthetic Monitoring** — Validate API endpoints and microservice communication - **Browser Synthetic Monitoring** — Validate user journeys and interactions in a real browser environment +- **Agentic Synthetic Monitoring** — Prompt an agent to discover, verify, and maintain checks for critical user journeys ## Getting Started @@ -73,11 +74,11 @@ Ready to start monitoring your applications? Choose the approach that best fits Begin with automated testing for development workflows - + Set up basic availability monitoring for critical services - + Create comprehensive user experience validation diff --git a/detect/synthetic-monitoring/agentic-checks/configuration.mdx b/detect/synthetic-monitoring/agentic-checks/configuration.mdx new file mode 100644 index 00000000..efefe5ff --- /dev/null +++ b/detect/synthetic-monitoring/agentic-checks/configuration.mdx @@ -0,0 +1,134 @@ +--- +title: 'Agentic Checks Configuration' +description: 'Configure Agentic Checks with prompts, runtime access, scheduling, locations, and active capacity.' +sidebarTitle: 'Configuration' +--- + + +Before creating Agentic Checks, ensure you have: + +- Access to Agentic Checks on your Checkly account +- Available Agentic Check active capacity if the check will be active +- An initialized Checkly CLI project if you want to manage the check as code +- Any required test credentials saved as Checkly environment variables + +For CLI setup, see the [CLI overview](/cli/overview). + + +## Define the check goal + +Write the prompt as a monitoring objective, not as implementation instructions. Include the target URL, the user journey, and the observable success criteria. + +```text Prompt +Go to https://app.example.com/login. +Sign in with the test account. +Verify that the dashboard loads, the account name is visible, and there are no blocking console errors. +``` + +Good prompts include: + +- The exact page or flow to start from +- The expected end state +- Important assertions +- Known constraints, such as test account behavior +- What should count as a failure + +Avoid prompts that ask the agent to test too many unrelated paths in one check. Create separate Agentic Checks for separate critical flows. + +## Configure runtime access + +Agentic Checks do **not** automatically expose account environment variables to the agent. Add only the variables the agent needs with `agentRuntime.exposeEnvironmentVariables`. + +```typescript checkout.agentic.check.ts +import { AgenticCheck, Frequency } from 'checkly/constructs' + +new AgenticCheck('checkout-flow', { + name: 'Checkout Flow', + prompt: 'Sign in to the test account and verify that a user can reach the checkout review step.', + frequency: Frequency.EVERY_1H, + activated: true, + agentRuntime: { + exposeEnvironmentVariables: [ + { name: 'TEST_USER_EMAIL', description: 'Email address for the checkout test account' }, + { name: 'TEST_USER_PASSWORD', description: 'Password for the checkout test account' }, + ], + }, +}) +``` + + +Do not paste secrets directly into the prompt. Store secrets as Checkly environment variables and expose only the variables the agent needs for this check. + + +Some applications block automated monitoring unless requests include required headers, test bypass tokens, or feature flags. If your flow depends on a header value, expose that value as an environment variable and say in the prompt which header the agent should send. Do not paste the header value directly into the prompt. + +## Add skills + +Agentic Checks include a default browser automation skill. You can add extra skills when the agent needs domain-specific capabilities. + +```typescript agentic-runtime.check.ts +import { AgenticCheck } from 'checkly/constructs' + +new AgenticCheck('web-quality-check', { + name: 'Web Quality Agentic Check', + prompt: 'Review the homepage and verify that the main navigation and primary call to action are usable.', + agentRuntime: { + skills: ['addyosmani/web-quality-skills'], + }, +}) +``` + +Keep the skill list short. Every added skill expands what the agent can do during execution. + +## Scheduling and locations + +Agentic Checks can run from up to three public locations by default. Enterprise accounts can contact sales to enable unlimited locations. + +The CLI construct supports these frequencies: + +| Frequency in minutes | Frequency constant | +|----------------------|--------------------| +| `5` | `Frequency.EVERY_5M` | +| `10` | `Frequency.EVERY_10M` | +| `15` | `Frequency.EVERY_15M` | +| `30` | `Frequency.EVERY_30M` | +| `60` | `Frequency.EVERY_1H` | +| `120` | `Frequency.EVERY_2H` | +| `180` | `Frequency.EVERY_3H` | +| `360` | `Frequency.EVERY_6H` | +| `720` | `Frequency.EVERY_12H` | +| `1440` | `Frequency.EVERY_24H` | + +## Active capacity + +Only active Agentic Checks consume Agentic Check capacity. + +Use `activated: false` when you want to deploy an Agentic Check draft without consuming active capacity: + +```typescript draft.agentic.check.ts +import { AgenticCheck } from 'checkly/constructs' + +new AgenticCheck('draft-agentic-check', { + name: 'Draft Agentic Check', + prompt: 'Verify the account settings page loads for a signed-in user.', + activated: false, +}) +``` + +You can keep inactive checks in code and activate them later when your account has enough capacity. + +## Test and deploy + +Test your project before deploying: + +```bash Terminal +npx checkly test +``` + +Deploy the check when you are ready: + +```bash Terminal +npx checkly deploy +``` + +If deployment fails because of active capacity, deactivate an existing Agentic Check or purchase enough Agentic Check capacity before deploying the new active check. diff --git a/detect/synthetic-monitoring/agentic-checks/overview.mdx b/detect/synthetic-monitoring/agentic-checks/overview.mdx new file mode 100644 index 00000000..fc92d60c --- /dev/null +++ b/detect/synthetic-monitoring/agentic-checks/overview.mdx @@ -0,0 +1,80 @@ +--- +title: 'Agentic Checks Overview' +description: 'Monitor critical user journeys and AI-powered experiences with prompt-driven Agentic Checks that discover, verify, and maintain synthetic monitoring scripts.' +sidebarTitle: 'Overview' +--- + + +**Monitoring as Code**: Learn more about the [Agentic Check Construct](/constructs/agentic-check). + + +## What are Agentic Checks? + +Agentic Checks are AI-powered synthetic checks that turn a monitoring goal into an executable check. You describe what the agent should verify, and Checkly runs the agent in a browser-capable runtime to explore the target, evaluate assertions, and persist what it learns for future runs. + +Agentic Checks are useful when the behavior you need to monitor is clear, but the exact script is expensive to write or maintain by hand. They also work well for nondeterministic application behavior, such as AI-powered support chat, search, or assistant flows where the exact response can vary while the expected outcome stays stable. + +**Agentic Checks are a good fit for:** + +- Critical user journeys that change often +- Authentication or onboarding flows that need contextual checks +- AI-powered product flows where the monitor should evaluate whether the user goal was completed, not whether the output matched exact text +- Pages where you want the monitor to adapt when implementation details change +- Early monitoring coverage before you invest in a hand-written Browser Check or Playwright Check Suite + +## How Agentic Checks work + +The first run uses your prompt to discover the target and create a reusable check script. Later runs execute the saved script first for faster, lower-cost monitoring. If the saved script fails, the agent can wake up in self-heal mode, diagnose the failure, update the script, and verify the result before the run finishes. + +Each run can produce: + +- A human-readable summary +- Assertions the agent evaluated +- Errors and failure details +- Agent steps and tool activity +- Suggestions for improving the check +- User-facing artifacts collected during the run + + +Agentic Checks are not a replacement for every deterministic check. Use them for important flows where the agent's discovery and self-healing behavior is valuable. Use [API Checks](/detect/synthetic-monitoring/api-checks/overview), [Browser Checks](/detect/synthetic-monitoring/browser-checks/overview), or [Playwright Check Suites](/detect/synthetic-monitoring/playwright-checks/overview) when you need fully deterministic scripted monitoring. + + +## Runtime access + +Agentic Checks run with a small default runtime for browser automation. You can extend that runtime with skills and explicitly expose environment variables that the agent is allowed to read. + +Treat runtime access as a security boundary: + +- Only expose the variables the agent needs. +- Add descriptions for credentials so the agent knows when to use them. +- Do not put secrets directly in prompts. +- Review generated checks before deploying them to production monitoring. + +Learn how to configure runtime access in the [Agentic Checks configuration guide](/detect/synthetic-monitoring/agentic-checks/configuration). + +## Billing and entitlements + +Agentic Checks use active-capacity billing: + +- Active Agentic Checks consume purchased active capacity. +- Inactive Agentic Checks keep their configuration and history without consuming active capacity. +- Starter and Team plans include Agentic Check capacity; you can purchase additional active capacity from the billing page. +- Agentic Check runs still count toward usage analytics, but they are billed by active capacity, not per-run runtime usage. +- Plan changes that would leave you with more active Agentic Checks than the target subscription allows are blocked until you deactivate checks or purchase enough capacity. +- Accounts can run Agentic Checks from up to three locations by default. Enterprise accounts can contact sales for unlimited locations. + + +Your billing page is the source of truth for the exact Agentic Check capacity included in your plan and any additional capacity you have purchased. + + +## Next steps + + + +Define prompts, runtime access, and scheduling behavior. + + + +Configure Agentic Checks as code with the Checkly CLI. + + diff --git a/detect/synthetic-monitoring/overview.mdx b/detect/synthetic-monitoring/overview.mdx index ec37495b..397c67cf 100644 --- a/detect/synthetic-monitoring/overview.mdx +++ b/detect/synthetic-monitoring/overview.mdx @@ -11,9 +11,13 @@ Synthetic monitoring goes beyond simple uptime checks by proactively validating ## Types of Synthetic Monitoring Checks -Checkly provides four powerful check types to comprehensively monitor your applications: +Checkly provides five powerful check types to comprehensively monitor your applications: + +Monitor critical user journeys with prompt-driven checks that can discover, verify, and self-heal. + + Monitor individual API endpoints with detailed request/response validation. @@ -48,6 +52,10 @@ Ready to implement comprehensive synthetic monitoring? Choose your starting poin Start with backend API monitoring and validation + +Prompt an agent to monitor a critical user journey + + Monitor user experiences with automated browser testing diff --git a/docs.json b/docs.json index fe0430fe..ccb5b2b5 100644 --- a/docs.json +++ b/docs.json @@ -185,6 +185,13 @@ "group": "Synthetic Monitoring", "pages": [ "detect/synthetic-monitoring/overview", + { + "group": "Agentic Checks", + "pages": [ + "detect/synthetic-monitoring/agentic-checks/overview", + "detect/synthetic-monitoring/agentic-checks/configuration" + ] + }, { "group": "API Checks", "pages": [ @@ -458,6 +465,7 @@ "constructs/tcp-monitor", "constructs/icmp-monitor", "constructs/heartbeat-monitor", + "constructs/agentic-check", "constructs/api-check", "constructs/multistep-check", "constructs/browser-check", From a136497dea627a52ae42db3dd82baca6a7bc4926 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Herv=C3=A9=20Labas?= <431257+thebiglabasky@users.noreply.github.com> Date: Wed, 6 May 2026 17:11:17 +0200 Subject: [PATCH 2/4] Delete .agents/skills/checkly/SKILL.md --- .agents/skills/checkly/SKILL.md | 97 --------------------------------- 1 file changed, 97 deletions(-) delete mode 100644 .agents/skills/checkly/SKILL.md diff --git a/.agents/skills/checkly/SKILL.md b/.agents/skills/checkly/SKILL.md deleted file mode 100644 index 896a4fd0..00000000 --- a/.agents/skills/checkly/SKILL.md +++ /dev/null @@ -1,97 +0,0 @@ ---- -name: checkly -description: Set up, create, test and manage monitoring checks using the Checkly CLI. Use when working with API Checks, Browser Checks, URL Monitors, ICMP Monitors, Playwright Check Suites, Heartbeat Monitors, Alert Channels, Dashboards, or Status Pages. -allowed-tools: Bash(npx:checkly:*) Bash(npm:install:*) -metadata: - author: checkly ---- - -# Checkly - -The Checkly CLI provides all the required information via the `npx checkly skills` command. - -Use `npx checkly skills install` to install this skill into your project (supports Codex, Cursor, Codex and more). - -Use `npx checkly skills` to list all available actions, and `npx checkly skills ` to access up-to-date information on how to use the Checkly CLI for each action. - -## Progressive Disclosure via `npx checkly skills` - -The skill is structured for efficient context usage: - -1. **Metadata** (~80 tokens): Name and description in frontmatter -2. **Core Instructions** (~1K tokens): Main SKILL.md content with links to reference commands -3. **Reference Commands** (loaded on demand): Detailed construct documentation with examples - -Agents load what they need for each task. - -## Plan Awareness - -Before configuring checks, run `npx checkly account plan --output json` to see what features, locations, and limits are available on the current plan. Disabled features include an `upgradeUrl` pointing to the self-service checkout page or the enterprise contact sales page — share these with the user when they need a feature that's not on their plan. - -Run `npx checkly skills manage` for the full reference. - -## Confirmation Protocol - -Write commands (e.g. `incidents create`, `deploy`, `destroy`) return exit code 2 with a `confirmation_required` JSON envelope instead of executing. **Always present the `changes` to the user and wait for approval before running the `confirmCommand`.** Never auto-append `--force`. This applies to every write command individually — updates and resolutions need confirmation too, not just the initial create. - -Run `npx checkly skills communicate` for the full protocol details. - -### `npx checkly skills initialize` -Learn how to initialize and set up a new Checkly CLI project from scratch. - -### `npx checkly skills configure` -Learn how to create and manage monitoring checks using Checkly constructs and the CLI. - -#### `npx checkly skills configure api-checks` -Api Check construct (`ApiCheck`), assertions, and authentication setup scripts - -#### `npx checkly skills configure browser-checks` -Browser Check construct (`BrowserCheck`) with Playwright test files - -#### `npx checkly skills configure playwright-checks` -Playwright Check Suite construct (`PlaywrightCheck`) for multi-browser test suites - -#### `npx checkly skills configure multistep-checks` -Multistep Check construct (`MultiStepCheck`) for complex user flows - -#### `npx checkly skills configure tcp-monitors` -TCP Monitor construct (`TcpMonitor`) with assertions - -#### `npx checkly skills configure url-monitors` -URL Monitor construct (`UrlMonitor`) with assertions - -#### `npx checkly skills configure dns-monitors` -DNS Monitor construct (`DnsMonitor`) with assertions - -#### `npx checkly skills configure icmp-monitors` -ICMP Monitor construct (`IcmpMonitor`) with latency and packet loss assertions - -#### `npx checkly skills configure heartbeat-monitors` -Heartbeat Monitor construct (`HeartbeatMonitor`) - -#### `npx checkly skills configure check-groups` -CheckGroupV2 construct (`CheckGroupV2`) for organizing checks - -#### `npx checkly skills configure alert-channels` -Email (`EmailAlertChannel`), Phone (`PhoneCallAlertChannel`), and Slack (`SlackAlertChannel`) alert channels - -#### `npx checkly skills configure supporting-constructs` -Status pages (`StatusPage`), dashboards (`Dashboard`), maintenance windows (`MaintenanceWindow`), and private locations (`PrivateLocation`) - -### `npx checkly skills investigate` -Access check status, analyze failures, and investigate errors. - -#### `npx checkly skills investigate checks` -Inspecting checks (`checks list`, `checks get`) and triggering on-demand runs - -### `npx checkly skills communicate` -Open incidents and lead customer communications via status pages. - -#### `npx checkly skills communicate incidents` -Incident lifecycle (`incidents create`, `update`, `resolve`, `list`) and status pages - -### `npx checkly skills manage` -Understand your account plan, entitlements, and feature limits. - -#### `npx checkly skills manage plan` -Check account plan, entitlements, feature limits, and available locations (`account plan`) From 2c4d9af0139ab93bc97ba62dd095ee9064829e6c Mon Sep 17 00:00:00 2001 From: Herve Labas Date: Wed, 6 May 2026 17:21:29 +0200 Subject: [PATCH 3/4] Expand Agentic Check prompt best practices --- constructs/agentic-check.mdx | 75 +++++++++++++++++-- .../agentic-checks/configuration.mdx | 2 + .../agentic-checks/overview.mdx | 20 +++++ 3 files changed, 90 insertions(+), 7 deletions(-) diff --git a/constructs/agentic-check.mdx b/constructs/agentic-check.mdx index 787c2d57..4bda5e2f 100644 --- a/constructs/agentic-check.mdx +++ b/constructs/agentic-check.mdx @@ -95,6 +95,16 @@ The Agentic Check construct intentionally does not expose `privateLocations`, `r The monitoring goal the agent should verify. Write the prompt in terms of user-visible behavior and expected outcomes. +See [Prompt best practices](/detect/synthetic-monitoring/agentic-checks/overview#prompt-best-practices) for guidance on scope, strictness, and nondeterministic flows. + +A reliable Agentic Check prompt usually states: + +- **Goal**: The user journey or system behavior to monitor. +- **What you are given**: The start URL, test account context, and any required environment variables. +- **Success looks like**: The exact outcome that should pass. +- **Failure looks like**: The states that should fail the check. +- **Strictness**: Whether the agent should assert exact values or evaluate the outcome more flexibly. + ```typescript new AgenticCheck('checkout-flow', { name: 'Checkout Flow', @@ -106,6 +116,53 @@ new AgenticCheck('checkout-flow', { }) ``` +Use stricter prompts when the output should be predictable: + +```typescript +new AgenticCheck('login-flow', { + name: 'Login Flow', + prompt: ` + ## Goal + Sign in to https://app.example.com/login and confirm the dashboard loads. + + ## What you are given + Use TEST_USER_EMAIL and TEST_USER_PASSWORD from the environment. + + ## Success looks like + The authenticated dashboard is visible, the account name appears, and the final URL is inside app.example.com. + + ## Failure looks like + The login form rejects valid credentials, the page stays on the login screen, an MFA prompt blocks the flow, or the dashboard shows a 4xx/5xx error. + `, + agentRuntime: { + exposeEnvironmentVariables: ['TEST_USER_EMAIL', 'TEST_USER_PASSWORD'], + }, +}) +``` + +Use outcome-based prompts when the application is intentionally nondeterministic: + +```typescript +new AgenticCheck('support-assistant-flow', { + name: 'Support Assistant Flow', + prompt: ` + ## Goal + Verify that the support assistant can help a user understand how to update billing details. + + ## What you are given + Start at https://app.example.com/support and ask: "How do I update my billing details?" + + ## Success looks like + The assistant gives a relevant answer, asks for missing context if needed, and guides the user toward a billing-settings or support handoff path. + + ## Failure looks like + The assistant does not respond, answers an unrelated question, loops without progress, or gives instructions that cannot lead to completion. + `, +}) +``` + +Keep prompts focused on one flow. If you want to monitor navigation, login, and record creation, create separate Agentic Checks for each flow. + ### `frequency` How often the Agentic Check should run. Use one of the supported `Frequency` constants or the equivalent number of minutes. @@ -146,13 +203,17 @@ Controls what the agent can access while executing the check. `exposeEnvironmentVariables` accepts either a string variable name or an object with `name` and optional `description`. ```typescript -agentRuntime: { - skills: ['addyosmani/web-quality-skills'], - exposeEnvironmentVariables: [ - 'API_KEY', - { name: 'TEST_USER_PASSWORD', description: 'Login password for the test account' }, - ], -} +new AgenticCheck('authenticated-api-check', { + name: 'Authenticated API Check', + prompt: 'Verify that the authenticated API health endpoint returns a healthy response.', + agentRuntime: { + skills: ['addyosmani/web-quality-skills'], + exposeEnvironmentVariables: [ + 'API_KEY', + { name: 'TEST_USER_PASSWORD', description: 'Login password for the test account' }, + ], + }, +}) ``` diff --git a/detect/synthetic-monitoring/agentic-checks/configuration.mdx b/detect/synthetic-monitoring/agentic-checks/configuration.mdx index efefe5ff..e51137eb 100644 --- a/detect/synthetic-monitoring/agentic-checks/configuration.mdx +++ b/detect/synthetic-monitoring/agentic-checks/configuration.mdx @@ -19,6 +19,8 @@ For CLI setup, see the [CLI overview](/cli/overview). Write the prompt as a monitoring objective, not as implementation instructions. Include the target URL, the user journey, and the observable success criteria. +For broader guidance, see [Prompt best practices](/detect/synthetic-monitoring/agentic-checks/overview#prompt-best-practices). + ```text Prompt Go to https://app.example.com/login. Sign in with the test account. diff --git a/detect/synthetic-monitoring/agentic-checks/overview.mdx b/detect/synthetic-monitoring/agentic-checks/overview.mdx index fc92d60c..ea0b370a 100644 --- a/detect/synthetic-monitoring/agentic-checks/overview.mdx +++ b/detect/synthetic-monitoring/agentic-checks/overview.mdx @@ -22,6 +22,26 @@ Agentic Checks are useful when the behavior you need to monitor is clear, but th - Pages where you want the monitor to adapt when implementation details change - Early monitoring coverage before you invest in a hand-written Browser Check or Playwright Check Suite +## Prompt best practices + +The prompt is the monitoring contract for an Agentic Check. Write it as a focused objective with clear pass and fail criteria. + +A reliable prompt usually includes: + +- **Goal**: The user journey or system behavior to monitor. +- **What you are given**: The start URL, test account context, and any required environment variables. +- **Success looks like**: The exact outcome that should pass. +- **Failure looks like**: The states that should fail the check. +- **Strictness**: Whether the agent should assert exact values or evaluate the outcome more flexibly. + +Choose strictness based on the application behavior: + +- Use strict assertions for deterministic flows, such as a login page loading, an API response containing a required field, or a dashboard showing an account name. +- Use outcome-based assertions for nondeterministic flows, such as an AI support assistant answering a billing question or a search experience returning relevant results. In these cases, describe the intent and acceptance criteria instead of requiring exact copy. +- Avoid vague prompts like "monitor my app". If the prompt is broad, tell the agent what surface to cover and what failures should matter. + +Keep prompts focused on one flow. If you want to monitor navigation, login, and record creation, create separate Agentic Checks for each flow. + ## How Agentic Checks work The first run uses your prompt to discover the target and create a reusable check script. Later runs execute the saved script first for faster, lower-cost monitoring. If the saved script fails, the agent can wake up in self-heal mode, diagnose the failure, update the script, and verify the result before the run finishes. From 374afe9abb26b071dc5562d1fa72d4b7c30c21ac Mon Sep 17 00:00:00 2001 From: Herve Labas Date: Wed, 6 May 2026 17:22:11 +0200 Subject: [PATCH 4/4] docs: use env var notation in agentic prompt --- constructs/agentic-check.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/constructs/agentic-check.mdx b/constructs/agentic-check.mdx index 4bda5e2f..b869c67a 100644 --- a/constructs/agentic-check.mdx +++ b/constructs/agentic-check.mdx @@ -126,7 +126,7 @@ new AgenticCheck('login-flow', { Sign in to https://app.example.com/login and confirm the dashboard loads. ## What you are given - Use TEST_USER_EMAIL and TEST_USER_PASSWORD from the environment. + Use {{TEST_USER_EMAIL}} and {{TEST_USER_PASSWORD}} from the environment. ## Success looks like The authenticated dashboard is visible, the account name appears, and the final URL is inside app.example.com.