-
Notifications
You must be signed in to change notification settings - Fork 9
Add Agentic Checks documentation #296
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
637a1a6
a136497
2c4d9af
374afe9
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| # Mintlify documentation | ||
|
|
||
| ## Working relationship | ||
| - You can push back on ideas-this can lead to better documentation. Cite sources and explain your reasoning when you do so | ||
| - ALWAYS ask for clarification rather than making assumptions | ||
| - NEVER lie, guess, or make up information | ||
|
|
||
| ## Project context | ||
| - Framework: Mintlify. https://mintlify.com/docs/llms.txt | ||
| - Format: MDX files with YAML frontmatter | ||
|
Check warning on line 10 in AGENTS.md
|
||
| - Config: docs.json for navigation, theme, settings. Tabs set top level navigation, Groups set sidebar navigation within Tabs. | ||
| - Components: We use Mintlify components and documentation. | ||
|
Check warning on line 12 in AGENTS.md
|
||
| -- Headers and text | ||
| -- Images and embeds | ||
| -- Lists and tables | ||
| -- Code | ||
| -- Reusable snippets | ||
| -- Accordions | ||
| -- Callouts | ||
| -- Cards | ||
| -- Columns | ||
| -- Code groups | ||
| -- Examples | ||
| -- Expandables | ||
| -- Fields | ||
| -- Frames | ||
| -- Icons | ||
| -- Mermaid | ||
| -- Panel | ||
| -- Steps | ||
| -- Tabs | ||
| -- Tooltips | ||
| -- Update | ||
|
|
||
| ## Content strategy | ||
| - Search for existing information before adding new content. Avoid duplication unless it is done for a strategic reason | ||
|
Check warning on line 36 in AGENTS.md
|
||
| - Check existing patterns for consistency | ||
| - Start by making the smallest reasonable changes | ||
| - Document just enough for user success - not too much. Don't be verbose. | ||
| - Prioritize accuracy and usability of information | ||
| - Make content evergreen | ||
|
|
||
| ## docs.json | ||
|
|
||
| - Refer to the [docs.json schema](https://mintlify.com/docs.json) when building the docs.json file and site navigation | ||
| - Tabs | ||
| -- Groups | ||
| --- Pages | ||
|
|
||
| ## Frontmatter requirements for pages | ||
| - title: Clear, descriptive page title | ||
| - sidebarTitle: Concise title for sidebar | ||
| - description: Concise summary for SEO/navigation | ||
|
|
||
| ## Writing standards | ||
| - Second-person voice ("you") | ||
| - Prerequisites at start of procedural content - ALWAYS wrap prerequisites in an Accordion component | ||
| - Test all code examples before publishing | ||
| - Match style and formatting of existing pages | ||
| - Include both basic and advanced use cases | ||
| - Language tags on all code blocks | ||
| - Alt text on all images | ||
| - Relative paths for internal links | ||
|
|
||
| ## Component usage rules | ||
| - Prerequisites Should Always be wrapped in an Accordion component with title "Prerequisites" or "Before you begin" | ||
|
Check warning on line 66 in AGENTS.md
|
||
| - Use Steps component for sequential procedures sparingly. | ||
| - Don't use Cards to display information. Only for navigation on Overview type pages | ||
| - Use CodeGroups and Dropdowns for code examples that show the same example in different languages | ||
| - Use Note/Tip/Warning callouts appropriately | ||
|
|
||
| ## CodeGroup Dropdown | ||
| - Use `<CodeGroup dropdown={true}>` for code examples that show the same functionality in different programming languages | ||
| - The dropdown automatically creates a language selector based on the code block language identifiers | ||
| - Example usage: | ||
|
|
||
| <CodeGroup dropdown={true}> | ||
| ```typescript example.ts | ||
| // TypeScript code here | ||
| ``` | ||
|
|
||
| ```javascript example.js | ||
| // JavaScript code here | ||
| ``` | ||
| </CodeGroup> | ||
|
|
||
| - This replaces the older `<Tabs>` pattern for language-specific code examples | ||
|
|
||
| ## Git workflow | ||
| - NEVER use --no-verify when committing | ||
| - Ask how to handle uncommitted changes before starting | ||
| - Create a new branch when no clear branch exists for changes | ||
| - Commit frequently throughout development | ||
| - NEVER skip or disable pre-commit hooks | ||
|
Check warning on line 94 in AGENTS.md
|
||
|
|
||
| ## Do not | ||
| - Skip frontmatter on any MDX file | ||
|
Check warning on line 97 in AGENTS.md
|
||
| - Use absolute URLs for internal links | ||
| - Include untested code examples | ||
| - Make assumptions - always ask for clarification | ||
| - overuse components like Cards or Accordion | ||
| | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,235 @@ | ||
| --- | ||
| title: 'Agentic Check Construct' | ||
| description: 'Learn how to configure Agentic Checks with the Checkly CLI.' | ||
| sidebarTitle: 'Agentic Check' | ||
| --- | ||
|
|
||
| <Tip> | ||
| Learn more about Agentic Checks in [the Agentic Checks overview](/detect/synthetic-monitoring/agentic-checks/overview). | ||
| </Tip> | ||
|
|
||
| Use Agentic Checks to monitor user journeys from a natural-language prompt. The agent discovers the target, creates a reusable check script, and can self-heal when the saved script starts failing. | ||
|
|
||
| <Accordion title="Prerequisites"> | ||
| Before creating Agentic Checks, ensure you have: | ||
|
|
||
| - An initialized Checkly CLI project | ||
| - Access to Agentic Checks on your Checkly account | ||
| - Available Agentic Check active capacity if the check is active | ||
| - Any required test credentials saved as Checkly environment variables | ||
|
|
||
| For additional setup information, see [CLI overview](/cli/overview). | ||
| </Accordion> | ||
|
|
||
| <CodeGroup> | ||
|
|
||
| ```typescript Basic Example | ||
| import { AgenticCheck, Frequency } from 'checkly/constructs' | ||
|
|
||
| new AgenticCheck('homepage-health', { | ||
| name: 'Homepage Health Check', | ||
| prompt: 'Navigate to https://example.com and verify the page loads correctly.', | ||
| activated: true, | ||
| frequency: Frequency.EVERY_1H, | ||
| }) | ||
| ``` | ||
|
|
||
| ```typescript Runtime Access | ||
| import { AgenticCheck, Frequency } from 'checkly/constructs' | ||
|
|
||
| new AgenticCheck('login-flow', { | ||
| name: 'Login Flow', | ||
| prompt: 'Sign in to the test account and verify that the dashboard loads.', | ||
| activated: true, | ||
| frequency: Frequency.EVERY_1H, | ||
| tags: ['agentic', 'critical'], | ||
| agentRuntime: { | ||
| skills: ['addyosmani/web-quality-skills'], | ||
| exposeEnvironmentVariables: [ | ||
| 'TEST_USER_EMAIL', | ||
| { name: 'TEST_USER_PASSWORD', description: 'Password for the test account' }, | ||
| ], | ||
| }, | ||
| }) | ||
| ``` | ||
|
|
||
| </CodeGroup> | ||
|
|
||
| ## Configuration | ||
|
|
||
| The Agentic Check configuration consists of Agentic-specific options and a subset of inherited general check options. | ||
|
|
||
| <Tabs> | ||
| <Tab title="Agentic Check"> | ||
|
|
||
| | Parameter | Type | Required | Default | Description | | ||
| |-----------|------|----------|---------|-------------| | ||
| | `prompt` | `string` | Yes | - | Monitoring goal the agent should verify. Maximum 10,000 characters. | | ||
| | `frequency` | `AgenticCheckFrequency` | No | `Frequency.EVERY_30M` | How often the check should run. Agentic Checks support the same frequency values as other checks, down to `5` minutes. | | ||
| | `agentRuntime` | `object` | No | `{ skills: [], exposeEnvironmentVariables: [] }` | Runtime access the agent is allowed to use during execution. | | ||
|
|
||
| </Tab> | ||
| <Tab title="General Check"> | ||
|
|
||
| | Property | Type | Required | Default | Description | | ||
| |----------|------|----------|---------|-------------| | ||
| | `name` | `string` | Yes | - | Friendly name for your check | | ||
| | `description` | `string` | No | `null` | Description of the check. Supports markdown. Max 500 characters | | ||
| | `activated` | `boolean` | No | `true` | Whether the check is enabled | | ||
| | `alertChannels` | `AlertChannel[]` | No | `[]` | Alert channels for notifications | | ||
| | `alertEscalationPolicy` | `AlertEscalationPolicy` | No | - | Advanced alert settings | | ||
| | `group` | `CheckGroup` | No | - | The CheckGroup this check belongs to | | ||
| | `locations` | `(keyof Region)[]` | No | Project default or `['us-east-1']` | Public Checkly locations where the check runs | | ||
| | `muted` | `boolean` | No | `false` | Whether alert notifications are muted | | ||
| | `tags` | `string[]` | No | `[]` | Tags to organize checks | | ||
| | `testOnly` | `boolean` | No | `false` | Only run with `test`, not during deploy | | ||
|
|
||
| </Tab> | ||
| </Tabs> | ||
|
|
||
| <Info> | ||
| The Agentic Check construct intentionally does not expose `privateLocations`, `runParallel`, `retryStrategy`, `shouldFail`, `doubleCheck`, or `triggerIncident`. These options are not currently honored for Agentic Checks. | ||
| </Info> | ||
|
|
||
| ### `prompt` | ||
|
|
||
| The monitoring goal the agent should verify. Write the prompt in terms of user-visible behavior and expected outcomes. | ||
|
|
||
| See [Prompt best practices](/detect/synthetic-monitoring/agentic-checks/overview#prompt-best-practices) for guidance on scope, strictness, and nondeterministic flows. | ||
|
|
||
| A reliable Agentic Check prompt usually states: | ||
|
|
||
| - **Goal**: The user journey or system behavior to monitor. | ||
| - **What you are given**: The start URL, test account context, and any required environment variables. | ||
| - **Success looks like**: The exact outcome that should pass. | ||
| - **Failure looks like**: The states that should fail the check. | ||
| - **Strictness**: Whether the agent should assert exact values or evaluate the outcome more flexibly. | ||
|
|
||
| ```typescript | ||
| new AgenticCheck('checkout-flow', { | ||
| name: 'Checkout Flow', | ||
| prompt: ` | ||
| Go to https://shop.example.com. | ||
| Sign in with the test account. | ||
| Add one item to the cart and verify the checkout review page loads. | ||
| `, | ||
| }) | ||
| ``` | ||
|
|
||
| Use stricter prompts when the output should be predictable: | ||
|
|
||
| ```typescript | ||
| new AgenticCheck('login-flow', { | ||
| name: 'Login Flow', | ||
| prompt: ` | ||
| ## Goal | ||
| Sign in to https://app.example.com/login and confirm the dashboard loads. | ||
|
|
||
| ## What you are given | ||
| Use {{TEST_USER_EMAIL}} and {{TEST_USER_PASSWORD}} from the environment. | ||
|
|
||
| ## Success looks like | ||
| The authenticated dashboard is visible, the account name appears, and the final URL is inside app.example.com. | ||
|
|
||
| ## Failure looks like | ||
| The login form rejects valid credentials, the page stays on the login screen, an MFA prompt blocks the flow, or the dashboard shows a 4xx/5xx error. | ||
| `, | ||
| agentRuntime: { | ||
| exposeEnvironmentVariables: ['TEST_USER_EMAIL', 'TEST_USER_PASSWORD'], | ||
| }, | ||
| }) | ||
| ``` | ||
|
|
||
| Use outcome-based prompts when the application is intentionally nondeterministic: | ||
|
|
||
| ```typescript | ||
| new AgenticCheck('support-assistant-flow', { | ||
| name: 'Support Assistant Flow', | ||
| prompt: ` | ||
| ## Goal | ||
| Verify that the support assistant can help a user understand how to update billing details. | ||
|
|
||
| ## What you are given | ||
| Start at https://app.example.com/support and ask: "How do I update my billing details?" | ||
|
|
||
| ## Success looks like | ||
| The assistant gives a relevant answer, asks for missing context if needed, and guides the user toward a billing-settings or support handoff path. | ||
|
|
||
| ## Failure looks like | ||
| The assistant does not respond, answers an unrelated question, loops without progress, or gives instructions that cannot lead to completion. | ||
| `, | ||
| }) | ||
| ``` | ||
|
|
||
| Keep prompts focused on one flow. If you want to monitor navigation, login, and record creation, create separate Agentic Checks for each flow. | ||
|
|
||
| ### `frequency` | ||
|
|
||
| How often the Agentic Check should run. Use one of the supported `Frequency` constants or the equivalent number of minutes. | ||
|
|
||
| ```typescript | ||
| import { AgenticCheck, Frequency } from 'checkly/constructs' | ||
|
|
||
| new AgenticCheck('hourly-agentic-check', { | ||
| name: 'Hourly Agentic Check', | ||
| prompt: 'Verify the homepage and pricing page load correctly.', | ||
| frequency: Frequency.EVERY_1H, | ||
| }) | ||
| ``` | ||
|
|
||
| Supported values: `5`, `10`, `15`, `30`, `60`, `120`, `180`, `360`, `720`, and `1440`. | ||
|
|
||
| ### `locations` | ||
|
|
||
| Public Checkly locations where the Agentic Check should run. Accounts can select up to three locations by default. Enterprise accounts can contact sales for unlimited locations. | ||
|
|
||
| ```typescript | ||
| new AgenticCheck('regional-login-flow', { | ||
| name: 'Regional Login Flow', | ||
| prompt: 'Verify that a user can sign in and reach the dashboard.', | ||
| locations: ['us-east-1', 'eu-west-1', 'ap-southeast-1'], | ||
| }) | ||
| ``` | ||
|
|
||
| ### `agentRuntime` | ||
|
|
||
| Controls what the agent can access while executing the check. | ||
|
|
||
| | Parameter | Type | Required | Default | Description | | ||
| |-----------|------|----------|---------|-------------| | ||
| | `skills` | `string[]` | No | `[]` | Extra skill packages to load into the agent runtime. | | ||
| | `exposeEnvironmentVariables` | `(string \| object)[]` | No | `[]` | Environment variables the agent is allowed to read. | | ||
|
|
||
| `exposeEnvironmentVariables` accepts either a string variable name or an object with `name` and optional `description`. | ||
|
|
||
| ```typescript | ||
| new AgenticCheck('authenticated-api-check', { | ||
| name: 'Authenticated API Check', | ||
| prompt: 'Verify that the authenticated API health endpoint returns a healthy response.', | ||
| agentRuntime: { | ||
| skills: ['addyosmani/web-quality-skills'], | ||
| exposeEnvironmentVariables: [ | ||
| 'API_KEY', | ||
| { name: 'TEST_USER_PASSWORD', description: 'Login password for the test account' }, | ||
| ], | ||
| }, | ||
| }) | ||
| ``` | ||
|
|
||
| <Warning> | ||
| Variables not listed in `agentRuntime.exposeEnvironmentVariables` are not exposed to the agent runtime, even if they exist in your Checkly account. | ||
| </Warning> | ||
|
|
||
| ## Billing behavior | ||
|
|
||
| Agentic Checks use active-capacity billing. Active checks consume capacity; inactive checks do not. Use `activated: false` to keep an Agentic Check as a deployed draft without consuming active capacity. | ||
|
|
||
| ```typescript | ||
| new AgenticCheck('draft-agentic-check', { | ||
| name: 'Draft Agentic Check', | ||
| prompt: 'Verify the account settings page loads for a signed-in user.', | ||
| activated: false, | ||
| }) | ||
| ``` | ||
|
|
||
| If an active Agentic Check would exceed your available capacity, deployment fails with a billing entitlement error. Deactivate an existing Agentic Check or purchase additional Agentic Check capacity before deploying. | ||
Uh oh!
There was an error while loading. Please reload this page.