Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 102 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# Mintlify documentation

Check warning on line 1 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L1

Did you really mean 'Mintlify'?

## Working relationship
- You can push back on ideas-this can lead to better documentation. Cite sources and explain your reasoning when you do so
- ALWAYS ask for clarification rather than making assumptions
- NEVER lie, guess, or make up information

Check warning on line 6 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L6

Spell out 'NEVER', if it's unfamiliar to the audience.

## Project context
- Framework: Mintlify. https://mintlify.com/docs/llms.txt

Check warning on line 9 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L9

Did you really mean 'Mintlify'?
- Format: MDX files with YAML frontmatter

Check warning on line 10 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L10

Spell out 'MDX', if it's unfamiliar to the audience.

Check warning on line 10 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L10

Did you really mean 'frontmatter'?
- Config: docs.json for navigation, theme, settings. Tabs set top level navigation, Groups set sidebar navigation within Tabs.

Check warning on line 11 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L11

Did you really mean 'Config'?
- Components: We use Mintlify components and documentation.

Check warning on line 12 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L12

Try to avoid using first-person plural like 'We'.

Check warning on line 12 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L12

Did you really mean 'Mintlify'?
-- Headers and text
-- Images and embeds
-- Lists and tables
-- Code
-- Reusable snippets
-- Accordions
-- Callouts

Check warning on line 19 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L19

Did you really mean 'Callouts'?
-- Cards
-- Columns
-- Code groups
-- Examples
-- Expandables

Check warning on line 24 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L24

Did you really mean 'Expandables'?
-- Fields
-- Frames
-- Icons
-- Mermaid
-- Panel
-- Steps
-- Tabs
-- Tooltips

Check warning on line 32 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L32

Did you really mean 'Tooltips'?
-- Update

## Content strategy
- Search for existing information before adding new content. Avoid duplication unless it is done for a strategic reason

Check warning on line 36 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L36

'it is' is too wordy.

Check warning on line 36 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L36

Use 'it's' instead of 'it is'.

Check warning on line 36 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L36

'is done' may be passive voice. Use active voice if you can.

Check warning on line 36 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L36

Try to avoid using 'is'.

Check warning on line 36 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L36

In general, use active voice instead of passive voice ('is done').
- Check existing patterns for consistency
- Start by making the smallest reasonable changes
- Document just enough for user success - not too much. Don't be verbose.

Check warning on line 39 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L39

Try to avoid using 'be'.
- Prioritize accuracy and usability of information

Check warning on line 40 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L40

'Prioritize' is too wordy.
- Make content evergreen

## docs.json

Check warning on line 43 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L43

'docs.json' should use sentence-style capitalization.

- Refer to the [docs.json schema](https://mintlify.com/docs.json) when building the docs.json file and site navigation
- Tabs
-- Groups
--- Pages

## Frontmatter requirements for pages

Check warning on line 50 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L50

Did you really mean 'Frontmatter'?
- title: Clear, descriptive page title
- sidebarTitle: Concise title for sidebar

Check warning on line 52 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L52

Did you really mean 'sidebarTitle'?
- description: Concise summary for SEO/navigation

Check warning on line 53 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L53

Spell out 'SEO', if it's unfamiliar to the audience.

## Writing standards
- Second-person voice ("you")

Check warning on line 56 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L56

Use parentheses judiciously.
- Prerequisites at start of procedural content - ALWAYS wrap prerequisites in an Accordion component
- Test all code examples before publishing
- Match style and formatting of existing pages
- Include both basic and advanced use cases
- Language tags on all code blocks
- Alt text on all images
- Relative paths for internal links

## Component usage rules
- Prerequisites Should Always be wrapped in an Accordion component with title "Prerequisites" or "Before you begin"

Check warning on line 66 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L66

In general, use active voice instead of passive voice ('be wrapped').

Check warning on line 66 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L66

Try to avoid using 'be'.

Check warning on line 66 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L66

'be wrapped' may be passive voice. Use active voice if you can.
- Use Steps component for sequential procedures sparingly.
- Don't use Cards to display information. Only for navigation on Overview type pages
- Use CodeGroups and Dropdowns for code examples that show the same example in different languages
- Use Note/Tip/Warning callouts appropriately

Check warning on line 70 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L70

Did you really mean 'callouts'?

## CodeGroup Dropdown

Check warning on line 72 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L72

'CodeGroup Dropdown' should use sentence-style capitalization.
- Use `<CodeGroup dropdown={true}>` for code examples that show the same functionality in different programming languages

Check warning on line 73 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L73

Use 'capability' or 'feature' instead of 'functionality'.
- The dropdown automatically creates a language selector based on the code block language identifiers
- Example usage:

<CodeGroup dropdown={true}>
```typescript example.ts
// TypeScript code here
```

```javascript example.js
// JavaScript code here
```
</CodeGroup>

- This replaces the older `<Tabs>` pattern for language-specific code examples

## Git workflow
- NEVER use --no-verify when committing

Check warning on line 90 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L90

Spell out 'NEVER', if it's unfamiliar to the audience.
- Ask how to handle uncommitted changes before starting
- Create a new branch when no clear branch exists for changes
- Commit frequently throughout development
- NEVER skip or disable pre-commit hooks

Check warning on line 94 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L94

Spell out 'NEVER', if it's unfamiliar to the audience.

Check warning on line 94 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L94

Use 'turn off' or 'off' instead of 'disable'.

## Do not

Check warning on line 96 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L96

Use 'don't' instead of 'Do not'.
- Skip frontmatter on any MDX file

Check warning on line 97 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L97

Did you really mean 'frontmatter'?

Check warning on line 97 in AGENTS.md

View check run for this annotation

Mintlify / Mintlify Validation (checkly-422f444a) - vale-spellcheck

AGENTS.md#L97

Spell out 'MDX', if it's unfamiliar to the audience.
- Use absolute URLs for internal links
- Include untested code examples
- Make assumptions - always ask for clarification
- overuse components like Cards or Accordion
13 changes: 11 additions & 2 deletions concepts/checks.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,17 @@ Advanced API testing that goes beyond uptime monitoring. Use setup and teardown
- Integration testing between services
</Accordion>

<Accordion title="Agentic Checks">
Prompt-driven synthetic monitoring where an agent discovers a user journey, creates a reusable check script, and can self-heal when that saved script starts failing. Use Agentic Checks when the expected intent is stable, but the exact implementation or response can vary.

**Perfect for:**
- Critical user journeys that change often
- AI-powered support, search, or assistant flows where the exact response is nondeterministic
- Early monitoring coverage before writing deterministic scripts
- Flows that benefit from AI-assisted discovery and maintenance
- Checks that need explicit access to selected skills and environment variables
</Accordion>

<Accordion title="Multistep Checks">
Write Node.js scripts that run multiple API requests in sequence with arbitrary code between requests. Perfect for testing complex workflows involving authentication, data manipulation, and multi-step processes that span multiple endpoints.

Expand Down Expand Up @@ -126,5 +137,3 @@ Teardown scripts are run *before* any assertions are validated
All check types at Checkly have a Construct or API endpoint - meaning you can use Checkly to bring your monitoring process right into your repository. We call this [Monitoring as Code](/concepts/monitoring-as-code).

This approach transforms monitoring from a manual, UI-driven process into a programmable, scalable system that grows with your application. You can version control your monitoring logic, collaborate on it through code reviews, and deploy monitoring changes through the same CI/CD pipelines you use for application code.


235 changes: 235 additions & 0 deletions constructs/agentic-check.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,235 @@
---
title: 'Agentic Check Construct'
description: 'Learn how to configure Agentic Checks with the Checkly CLI.'
sidebarTitle: 'Agentic Check'
---

<Tip>
Learn more about Agentic Checks in [the Agentic Checks overview](/detect/synthetic-monitoring/agentic-checks/overview).
</Tip>

Use Agentic Checks to monitor user journeys from a natural-language prompt. The agent discovers the target, creates a reusable check script, and can self-heal when the saved script starts failing.

<Accordion title="Prerequisites">
Before creating Agentic Checks, ensure you have:

- An initialized Checkly CLI project
- Access to Agentic Checks on your Checkly account
- Available Agentic Check active capacity if the check is active
- Any required test credentials saved as Checkly environment variables

For additional setup information, see [CLI overview](/cli/overview).
</Accordion>

<CodeGroup>

```typescript Basic Example
import { AgenticCheck, Frequency } from 'checkly/constructs'

new AgenticCheck('homepage-health', {
name: 'Homepage Health Check',
prompt: 'Navigate to https://example.com and verify the page loads correctly.',
activated: true,
frequency: Frequency.EVERY_1H,
})
```

```typescript Runtime Access
import { AgenticCheck, Frequency } from 'checkly/constructs'

new AgenticCheck('login-flow', {
name: 'Login Flow',
prompt: 'Sign in to the test account and verify that the dashboard loads.',
activated: true,
frequency: Frequency.EVERY_1H,
tags: ['agentic', 'critical'],
agentRuntime: {
skills: ['addyosmani/web-quality-skills'],
exposeEnvironmentVariables: [
'TEST_USER_EMAIL',
{ name: 'TEST_USER_PASSWORD', description: 'Password for the test account' },
],
},
})
```

</CodeGroup>

## Configuration

The Agentic Check configuration consists of Agentic-specific options and a subset of inherited general check options.

<Tabs>
<Tab title="Agentic Check">

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `prompt` | `string` | Yes | - | Monitoring goal the agent should verify. Maximum 10,000 characters. |
| `frequency` | `AgenticCheckFrequency` | No | `Frequency.EVERY_30M` | How often the check should run. Agentic Checks support the same frequency values as other checks, down to `5` minutes. |
| `agentRuntime` | `object` | No | `{ skills: [], exposeEnvironmentVariables: [] }` | Runtime access the agent is allowed to use during execution. |

</Tab>
<Tab title="General Check">

| Property | Type | Required | Default | Description |
|----------|------|----------|---------|-------------|
| `name` | `string` | Yes | - | Friendly name for your check |
| `description` | `string` | No | `null` | Description of the check. Supports markdown. Max 500 characters |
| `activated` | `boolean` | No | `true` | Whether the check is enabled |
| `alertChannels` | `AlertChannel[]` | No | `[]` | Alert channels for notifications |
| `alertEscalationPolicy` | `AlertEscalationPolicy` | No | - | Advanced alert settings |
| `group` | `CheckGroup` | No | - | The CheckGroup this check belongs to |
| `locations` | `(keyof Region)[]` | No | Project default or `['us-east-1']` | Public Checkly locations where the check runs |
| `muted` | `boolean` | No | `false` | Whether alert notifications are muted |
| `tags` | `string[]` | No | `[]` | Tags to organize checks |
| `testOnly` | `boolean` | No | `false` | Only run with `test`, not during deploy |

</Tab>
</Tabs>

<Info>
The Agentic Check construct intentionally does not expose `privateLocations`, `runParallel`, `retryStrategy`, `shouldFail`, `doubleCheck`, or `triggerIncident`. These options are not currently honored for Agentic Checks.
</Info>

### `prompt`

The monitoring goal the agent should verify. Write the prompt in terms of user-visible behavior and expected outcomes.

See [Prompt best practices](/detect/synthetic-monitoring/agentic-checks/overview#prompt-best-practices) for guidance on scope, strictness, and nondeterministic flows.

A reliable Agentic Check prompt usually states:

- **Goal**: The user journey or system behavior to monitor.
- **What you are given**: The start URL, test account context, and any required environment variables.
- **Success looks like**: The exact outcome that should pass.
- **Failure looks like**: The states that should fail the check.
- **Strictness**: Whether the agent should assert exact values or evaluate the outcome more flexibly.

```typescript
new AgenticCheck('checkout-flow', {
name: 'Checkout Flow',
prompt: `
Go to https://shop.example.com.
Sign in with the test account.
Add one item to the cart and verify the checkout review page loads.
`,
})
```
Comment thread
thebiglabasky marked this conversation as resolved.

Use stricter prompts when the output should be predictable:

```typescript
new AgenticCheck('login-flow', {
name: 'Login Flow',
prompt: `
## Goal
Sign in to https://app.example.com/login and confirm the dashboard loads.

## What you are given
Use {{TEST_USER_EMAIL}} and {{TEST_USER_PASSWORD}} from the environment.

## Success looks like
The authenticated dashboard is visible, the account name appears, and the final URL is inside app.example.com.

## Failure looks like
The login form rejects valid credentials, the page stays on the login screen, an MFA prompt blocks the flow, or the dashboard shows a 4xx/5xx error.
`,
agentRuntime: {
exposeEnvironmentVariables: ['TEST_USER_EMAIL', 'TEST_USER_PASSWORD'],
},
})
```

Use outcome-based prompts when the application is intentionally nondeterministic:

```typescript
new AgenticCheck('support-assistant-flow', {
name: 'Support Assistant Flow',
prompt: `
## Goal
Verify that the support assistant can help a user understand how to update billing details.

## What you are given
Start at https://app.example.com/support and ask: "How do I update my billing details?"

## Success looks like
The assistant gives a relevant answer, asks for missing context if needed, and guides the user toward a billing-settings or support handoff path.

## Failure looks like
The assistant does not respond, answers an unrelated question, loops without progress, or gives instructions that cannot lead to completion.
`,
})
```

Keep prompts focused on one flow. If you want to monitor navigation, login, and record creation, create separate Agentic Checks for each flow.

### `frequency`

How often the Agentic Check should run. Use one of the supported `Frequency` constants or the equivalent number of minutes.

```typescript
import { AgenticCheck, Frequency } from 'checkly/constructs'

new AgenticCheck('hourly-agentic-check', {
name: 'Hourly Agentic Check',
prompt: 'Verify the homepage and pricing page load correctly.',
frequency: Frequency.EVERY_1H,
})
```

Supported values: `5`, `10`, `15`, `30`, `60`, `120`, `180`, `360`, `720`, and `1440`.

### `locations`

Public Checkly locations where the Agentic Check should run. Accounts can select up to three locations by default. Enterprise accounts can contact sales for unlimited locations.

```typescript
new AgenticCheck('regional-login-flow', {
name: 'Regional Login Flow',
prompt: 'Verify that a user can sign in and reach the dashboard.',
locations: ['us-east-1', 'eu-west-1', 'ap-southeast-1'],
})
```

### `agentRuntime`

Controls what the agent can access while executing the check.

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `skills` | `string[]` | No | `[]` | Extra skill packages to load into the agent runtime. |
| `exposeEnvironmentVariables` | `(string \| object)[]` | No | `[]` | Environment variables the agent is allowed to read. |

`exposeEnvironmentVariables` accepts either a string variable name or an object with `name` and optional `description`.

```typescript
new AgenticCheck('authenticated-api-check', {
name: 'Authenticated API Check',
prompt: 'Verify that the authenticated API health endpoint returns a healthy response.',
agentRuntime: {
skills: ['addyosmani/web-quality-skills'],
exposeEnvironmentVariables: [
'API_KEY',
{ name: 'TEST_USER_PASSWORD', description: 'Login password for the test account' },
],
},
})
```

<Warning>
Variables not listed in `agentRuntime.exposeEnvironmentVariables` are not exposed to the agent runtime, even if they exist in your Checkly account.
</Warning>

## Billing behavior

Agentic Checks use active-capacity billing. Active checks consume capacity; inactive checks do not. Use `activated: false` to keep an Agentic Check as a deployed draft without consuming active capacity.

```typescript
new AgenticCheck('draft-agentic-check', {
name: 'Draft Agentic Check',
prompt: 'Verify the account settings page loads for a signed-in user.',
activated: false,
})
```

If an active Agentic Check would exceed your available capacity, deployment fails with a billing entitlement error. Deactivate an existing Agentic Check or purchase additional Agentic Check capacity before deploying.
7 changes: 4 additions & 3 deletions detect/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Monitor availability and response times across your website and infrastructure
</Card>

<Card title="Synthetic Monitoring" href="/detect/synthetic-monitoring/overview" icon="robot" horizontal>
Proactively validate real user interactions with API, browser, and multistep checks
Proactively validate real user interactions with API, browser, multistep, and Agentic Checks
</Card>

<Card title="Testing" href="/detect/testing/overview" icon="flask" horizontal>
Expand Down Expand Up @@ -62,6 +62,7 @@ Implement testing and monitoring across multiple layers for a comprehensive view
- **Uptime Monitoring** — Monitor servers, databases, and network connectivity to ensure availability and responsiveness
- **API Synthetic Monitoring** — Validate API endpoints and microservice communication
- **Browser Synthetic Monitoring** — Validate user journeys and interactions in a real browser environment
- **Agentic Synthetic Monitoring** — Prompt an agent to discover, verify, and maintain checks for critical user journeys


## Getting Started
Expand All @@ -73,11 +74,11 @@ Ready to start monitoring your applications? Choose the approach that best fits
Begin with automated testing for development workflows
</Card>

<Card title="Create Your First Uptime Monitor" href="/detect/uptime-monitoring/url-monitors/creating-your-first-url-monitor">
<Card title="Create Your First Uptime Monitor" href="/quickstarts/url-monitor">
Set up basic availability monitoring for critical services
</Card>

<Card title="Create Your First Synthetic Check" href="/detect/synthetic-monitoring/api-checks/creating-your-first-api-check">
<Card title="Create Your First Synthetic Check" href="/quickstarts/api-check">
Create comprehensive user experience validation
</Card>
</CardGroup>
Expand Down
Loading