Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion packages/cli/src/ai-context/references/communicate.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,13 @@ Write commands (`incidents create`, `incidents update`, `incidents resolve`, `de
**Rules for agents:**

1. When exit code is 2 and output contains `"status": "confirmation_required"`, **always present the `changes` array to the user** and ask for confirmation.
2. **Never auto-append `--force`** — only run the `confirmCommand` after the user explicitly approves.
2. **Run the `confirmCommand` verbatim, and only after the user explicitly approves.** It is normally the command you just ran, plus `--force` to skip the second prompt. Don't append `--force` to anything yourself, and don't edit the `confirmCommand` — changing its flags changes what you were authorized to do.
3. This applies to **every** write command, not just the first one. Incident updates and resolutions also require confirmation.
4. Use `--dry-run` to preview what a command will do without executing or prompting.
5. Read-only commands (`incidents list`, `status-pages list`) execute immediately without confirmation.

The `confirmCommand` omits flags left at their default, so a bare `npx checkly deploy` confirms as `checkly deploy --force` rather than echoing back every boolean the parser filled in. Treat every flag you see there as deliberate.

## Available Commands

Parse and read further reference documentation when tasked with any of the following:
Expand Down
23 changes: 23 additions & 0 deletions packages/cli/src/ai-context/references/configure.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,3 +75,26 @@ Run `npx checkly skills manage plan` for the full reference.
## Deploying

- Deploy checks using the `npx checkly deploy` command. Use `--output` to see the created, updated, and deleted resources. Use `--verbose` to also include each resource's name and physical ID (UUID), which is useful for programmatically referencing deployed resources (e.g. `npx checkly checks get <id>`).
- Use `--preview` to see what a deploy would change without applying it.

### Deleted resources

A deploy makes the account match the code. **Any resource that was deployed before and is no longer in the code gets deleted, along with its run history.** Pass `--preserve-resources` to keep those resources and their history in the Checkly account instead, where the user can manage them from the web app.

This matters when the local project isn't the whole picture — a partial checkout, or a project whose checks were also edited elsewhere. If you're not sure the code is the complete source of truth, say so before deploying.

### Confirmation

`deploy` is a write command: without `--force` it returns exit code 2 and a `confirmation_required` envelope. Present its `changes` to the user and run the `confirmCommand` verbatim only after they approve.

That confirmation happens **before the project is parsed**, so it cannot tell you which resources would be deleted — its `changes` only warn that deletion is possible. There is a second, itemised guard that lists each doomed resource by name, but it is skipped by `--force`, and `--force` is exactly what the `confirmCommand` carries. **An agent following the confirmation protocol never sees that list.** It is a prompt for humans deploying by hand.

So when resources may have been removed from the code, don't rely on the confirmation to surface it — preview first:

```bash
npx checkly deploy --preview
```

This is a dry run: nothing is applied and nothing is confirmed. It prints the resources that would be created, updated, and deleted (add `--verbose` for names and IDs). Show the user what would be deleted, and only then deploy.

Run `npx checkly skills communicate` for the full protocol.
6 changes: 4 additions & 2 deletions packages/cli/src/ai-context/skill.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,11 @@ Run `npx checkly skills manage` for the full reference.

## Confirmation Protocol

Write commands (e.g. `incidents create`, `deploy`, `destroy`) return exit code 2 with a `confirmation_required` JSON envelope instead of executing. **Always present the `changes` to the user and wait for approval before running the `confirmCommand`.** Never auto-append `--force`. This applies to every write command individually — updates and resolutions need confirmation too, not just the initial create.
Write commands (e.g. `incidents create`, `deploy`, `destroy`) return exit code 2 with a `confirmation_required` JSON envelope instead of executing. **Always present the `changes` to the user and wait for approval before running the `confirmCommand`.** This applies to every write command individually — updates and resolutions need confirmation too, not just the initial create.

Run `npx checkly skills communicate` for the full protocol details.
The `confirmCommand` is the approved command, ready to run verbatim: it repeats the flags you passed and already ends in `--force`. Run it as-is once the user approves — don't add `--force` to a command yourself, and don't add flags the user didn't ask for.

Run `npx checkly skills communicate` for the full protocol details, or `npx checkly skills configure` for what `deploy` confirms.

## API Pass-Through (fallback for any endpoint)

Expand Down
47 changes: 47 additions & 0 deletions packages/cli/src/commands/__tests__/confirm-flow-deploy.spec.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { Parser } from '@oclif/core'
import { beforeEach, describe, expect, it, vi } from 'vitest'

vi.mock('../../helpers/cli-mode', () => ({
Expand Down Expand Up @@ -39,11 +40,23 @@ vi.mock('prompts', () => ({
}))

import { detectCliMode } from '../../helpers/cli-mode.js'
import { buildConfirmCommand } from '../../helpers/command-preview.js'
import * as api from '../../rest/api.js'
import { parseProject } from '../../services/project-parser.js'
import { AuthCommand } from '../authCommand.js'
import Deploy from '../deploy.js'

/** Turn a generated confirmCommand back into argv, so oclif can re-parse it. */
function flagArgv (confirmCommand: string): string[] {
return [...confirmCommand.matchAll(/--[a-zA-Z0-9-]+(?:="[^"]*")?/g)]
.map(([token]) => token.replace(/="(.*)"$/, '=$1'))
}

async function confirmCommandFor (argv: string[]): Promise<string> {
const { flags, metadata } = await Parser.parse(argv, { flags: Deploy.flags, strict: true })
return buildConfirmCommand('deploy', flags, undefined, metadata.flags)
}

function createConfirmContext () {
const logged: string[] = []
let exitCodeValue: number | undefined
Expand Down Expand Up @@ -156,6 +169,17 @@ describe('deploy confirmation flow', () => {
'debug-bundle': false,
'debug-bundle-output-file': './debug-bundle.json',
},
metadata: {
flags: {
'preview': { setFromDefault: true },
'output': { setFromDefault: true },
'verbose': { setFromDefault: true },
'schedule-on-deploy': { setFromDefault: true },
'verify-runtime-dependencies': { setFromDefault: true },
'debug-bundle': { setFromDefault: true },
'debug-bundle-output-file': { setFromDefault: true },
},
},
})

await expect(
Expand All @@ -172,5 +196,28 @@ describe('deploy confirmation flow', () => {
expect(output.command).toBe('deploy')
expect(output.classification.idempotent).toBe(true)
expect(output.confirmCommand).toContain('--force')
expect(output.confirmCommand).not.toContain('--no-preview')
})
})

describe('deploy confirmCommand', () => {
it('echoes only the flags the user typed', async () => {
expect(await confirmCommandFor([])).toBe('checkly deploy --force')
expect(await confirmCommandFor(['--preserve-resources'])).toBe('checkly deploy --preserve-resources --force')
})

it('keeps an explicit --no-<flag> for flags that allow it', async () => {
expect(await confirmCommandFor(['--no-schedule-on-deploy']))
.toBe('checkly deploy --no-schedule-on-deploy --force')
})

it('generates a command oclif can parse back', async () => {
for (const argv of [[], ['--preserve-resources'], ['--no-schedule-on-deploy'], ['--verbose']]) {
const confirmCommand = await confirmCommandFor(argv)
await expect(
Parser.parse(flagArgv(confirmCommand), { flags: Deploy.flags, strict: true }),
`confirmCommand for "checkly deploy ${argv.join(' ')}" must be runnable: ${confirmCommand}`,
).resolves.toBeDefined()
}
})
})
34 changes: 32 additions & 2 deletions packages/cli/src/commands/__tests__/confirm-flow-destroy.spec.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { Parser } from '@oclif/core'
import { beforeEach, describe, expect, it, vi } from 'vitest'

vi.mock('../../helpers/cli-mode', () => ({
Expand Down Expand Up @@ -29,14 +30,26 @@ vi.mock('prompts', () => ({
import { detectCliMode } from '../../helpers/cli-mode.js'
import prompts from 'prompts'
import * as api from '../../rest/api.js'
import { buildConfirmCommand } from '../../helpers/command-preview.js'
import { AuthCommand } from '../authCommand.js'
import Destroy from '../destroy.js'

function createCommandContext (parsed: unknown) {
/** Turn a generated confirmCommand back into argv, so oclif can re-parse it. */
function flagArgv (confirmCommand: string): string[] {
return [...confirmCommand.matchAll(/--[a-zA-Z0-9-]+(?:="[^"]*")?/g)]
.map(([token]) => token.replace(/="(.*)"$/, '=$1'))
}

async function confirmCommandFor (argv: string[]): Promise<string> {
const { flags, metadata } = await Parser.parse(argv, { flags: Destroy.flags, strict: true })
return buildConfirmCommand('destroy', flags, undefined, metadata.flags)
}

function createCommandContext (parsed: { flags: Record<string, unknown>, metadata?: unknown }) {
const logged: string[] = []
let exitCodeValue: number | undefined
return {
parse: vi.fn().mockResolvedValue(parsed),
parse: vi.fn().mockResolvedValue({ metadata: { flags: {} }, ...parsed }),
log: vi.fn((msg?: string) => {
if (msg) logged.push(msg)
}),
Expand Down Expand Up @@ -172,3 +185,20 @@ describe('destroy confirmation flow', () => {
expect(Destroy.idempotent).toBe(false)
})
})

describe('destroy confirmCommand', () => {
it('echoes only the flags the user typed', async () => {
expect(await confirmCommandFor([])).toBe('checkly destroy --force')
expect(await confirmCommandFor(['--preserve-resources'])).toBe('checkly destroy --preserve-resources --force')
})

it('generates a command oclif can parse back', async () => {
for (const argv of [[], ['--preserve-resources'], ['--cancel-in-progress-deployment']]) {
const confirmCommand = await confirmCommandFor(argv)
await expect(
Parser.parse(flagArgv(confirmCommand), { flags: Destroy.flags, strict: true }),
`confirmCommand for "checkly destroy ${argv.join(' ')}" must be runnable: ${confirmCommand}`,
).resolves.toBeDefined()
}
})
})
4 changes: 3 additions & 1 deletion packages/cli/src/commands/deploy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -110,7 +110,7 @@ export default class Deploy extends AuthCommand {
}

async run (): Promise<void> {
const { flags } = await this.parse(Deploy)
const { flags, metadata } = await this.parse(Deploy)
const {
force,
preview,
Expand Down Expand Up @@ -146,6 +146,7 @@ export default class Deploy extends AuthCommand {
: 'Delete any resources removed from code, losing their run history. Pass --preserve-resources to keep them in your Checkly account instead',
],
flags,
flagMetadata: metadata.flags,
classification: {
readOnly: Deploy.readOnly,
destructive: Deploy.destructive,
Expand Down Expand Up @@ -316,6 +317,7 @@ export default class Deploy extends AuthCommand {
`Delete ${PRETTY_RESOURCE_TYPES[resourceType] ?? resourceType}: ${logicalId}`),
],
flags,
flagMetadata: metadata.flags,
classification: {
readOnly: Deploy.readOnly,
destructive: Deploy.destructive,
Expand Down
3 changes: 2 additions & 1 deletion packages/cli/src/commands/destroy.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ export default class Destroy extends AuthCommand {
}

async run (): Promise<void> {
const { flags } = await this.parse(Destroy)
const { flags, metadata } = await this.parse(Destroy)
const {
config: configFilename,
'preserve-resources': preserveResources,
Expand All @@ -52,6 +52,7 @@ export default class Destroy extends AuthCommand {
: `PERMANENTLY delete ALL resources associated with the project "${checklyConfig.projectName}" in account "${account.name}"`,
],
flags,
flagMetadata: metadata.flags,
classification: {
readOnly: Destroy.readOnly,
destructive: Destroy.destructive,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,12 +50,12 @@ const incidentFixture = {
services: [{ id: 'svc-1', name: 'Checkout API', accountId: 'acc-1' }],
}

function createCommandContext (parsed: unknown) {
function createCommandContext (parsed: { flags: Record<string, unknown>, args?: unknown, metadata?: unknown }) {
const logged: string[] = []
let didExit = false
let exitCodeValue: number | undefined
return {
parse: vi.fn().mockResolvedValue(parsed),
parse: vi.fn().mockResolvedValue({ metadata: { flags: {} }, ...parsed }),
log: vi.fn((msg?: string) => {
if (msg) logged.push(msg)
}),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,13 @@ const statusPageFixture: StatusPage = {
updated_at: '2026-02-25T10:00:00.000Z',
}

function createCommandContext (parsed: unknown, CommandClass: any = IncidentsCreate) {
function createCommandContext (
parsed: { flags: Record<string, unknown>, args?: unknown, metadata?: unknown },
CommandClass: any = IncidentsCreate,
) {
const logged: string[] = []
return {
parse: vi.fn().mockResolvedValue(parsed),
parse: vi.fn().mockResolvedValue({ metadata: { flags: {} }, ...parsed }),
log: vi.fn((msg?: string) => {
if (msg) logged.push(msg)
}),
Expand Down
3 changes: 2 additions & 1 deletion packages/cli/src/commands/incidents/create.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ export default class IncidentsCreate extends AuthCommand {
}

async run (): Promise<void> {
const { flags } = await this.parse(IncidentsCreate)
const { flags, metadata } = await this.parse(IncidentsCreate)
this.style.outputFormat = flags.output

const statusPage = await api.statusPages.get(flags['status-page-id'])
Expand All @@ -73,6 +73,7 @@ export default class IncidentsCreate extends AuthCommand {
: 'Subscribers will NOT be notified',
],
flags,
flagMetadata: metadata.flags,
classification: {
readOnly: IncidentsCreate.readOnly,
destructive: IncidentsCreate.destructive,
Expand Down
3 changes: 2 additions & 1 deletion packages/cli/src/commands/incidents/resolve.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ export default class IncidentsResolve extends AuthCommand {
}

async run (): Promise<void> {
const { args, flags } = await this.parse(IncidentsResolve)
const { args, flags, metadata } = await this.parse(IncidentsResolve)
this.style.outputFormat = flags.output

const incident = await api.incidents.get(args.id)
Expand All @@ -48,6 +48,7 @@ export default class IncidentsResolve extends AuthCommand {
: 'Subscribers will NOT be notified',
],
flags,
flagMetadata: metadata.flags,
args: { id: args.id },
classification: {
readOnly: IncidentsResolve.readOnly,
Expand Down
3 changes: 2 additions & 1 deletion packages/cli/src/commands/incidents/update.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ export default class IncidentsUpdate extends AuthCommand {
}

async run (): Promise<void> {
const { args, flags } = await this.parse(IncidentsUpdate)
const { args, flags, metadata } = await this.parse(IncidentsUpdate)
this.style.outputFormat = flags.output

const incident = await api.incidents.get(args.id)
Expand All @@ -69,6 +69,7 @@ export default class IncidentsUpdate extends AuthCommand {
description: 'Post progress update to incident',
changes,
flags,
flagMetadata: metadata.flags,
args: { id: args.id },
classification: {
readOnly: IncidentsUpdate.readOnly,
Expand Down
18 changes: 18 additions & 0 deletions packages/cli/src/helpers/__tests__/command-preview.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -124,4 +124,22 @@ describe('buildConfirmCommand', () => {
const result = buildConfirmCommand('incidents update', { message: 'Fix deployed' }, { id: 'inc-123' })
expect(result).toBe('checkly incidents update inc-123 --message="Fix deployed" --force')
})

it('omits flags oclif filled in from their default', () => {
const result = buildConfirmCommand('deploy', {
'preview': false,
'schedule-on-deploy': true,
'preserve-resources': true,
}, undefined, {
'preview': { setFromDefault: true },
'schedule-on-deploy': { setFromDefault: true },
'preserve-resources': { setFromDefault: false },
})
expect(result).toBe('checkly deploy --preserve-resources --force')
})

it('keeps default-valued flags when no metadata is given', () => {
const result = buildConfirmCommand('deploy', { 'schedule-on-deploy': true })
expect(result).toBe('checkly deploy --schedule-on-deploy --force')
})
})
13 changes: 12 additions & 1 deletion packages/cli/src/helpers/command-preview.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,18 @@ export type CommandClassification = {
idempotent: boolean
}

/**
* The subset of oclif's parse metadata we rely on: which flags were filled in from
* `default:` rather than typed by the user. Pass `metadata.flags` from `this.parse()`.
*/
export type FlagMetadata = Record<string, { setFromDefault?: boolean } | undefined>

export type CommandPreview = {
command: string
description: string
changes: string[]
flags: Record<string, unknown>
flagMetadata?: FlagMetadata
args?: Record<string, unknown>
classification: CommandClassification
}
Expand All @@ -28,6 +35,7 @@ export function buildConfirmCommand (
command: string,
flags: Record<string, unknown>,
args?: Record<string, unknown>,
flagMetadata?: FlagMetadata,
): string {
const parts = ['checkly', command]

Expand All @@ -40,6 +48,9 @@ export function buildConfirmCommand (
for (const [key, value] of Object.entries(flags)) {
if (OMITTED_FLAGS.has(key)) continue
if (value === undefined || value === null) continue
// Defaults are not part of what the user asked for, and a default `false` renders as
// `--no-x`, which only parses when the flag sets `allowNo: true`.
if (flagMetadata?.[key]?.setFromDefault) continue

if (Array.isArray(value)) {
for (const item of value) {
Expand All @@ -66,7 +77,7 @@ export function formatPreviewForAgent (
description: preview.description,
classification: preview.classification,
changes: preview.changes,
confirmCommand: buildConfirmCommand(preview.command, preview.flags, preview.args),
confirmCommand: buildConfirmCommand(preview.command, preview.flags, preview.args, preview.flagMetadata),
}
}

Expand Down
6 changes: 4 additions & 2 deletions skills/checkly/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,11 @@ Run `npx checkly skills manage` for the full reference.

## Confirmation Protocol

Write commands (e.g. `incidents create`, `deploy`, `destroy`) return exit code 2 with a `confirmation_required` JSON envelope instead of executing. **Always present the `changes` to the user and wait for approval before running the `confirmCommand`.** Never auto-append `--force`. This applies to every write command individually — updates and resolutions need confirmation too, not just the initial create.
Write commands (e.g. `incidents create`, `deploy`, `destroy`) return exit code 2 with a `confirmation_required` JSON envelope instead of executing. **Always present the `changes` to the user and wait for approval before running the `confirmCommand`.** This applies to every write command individually — updates and resolutions need confirmation too, not just the initial create.

Run `npx checkly skills communicate` for the full protocol details.
The `confirmCommand` is the approved command, ready to run verbatim: it repeats the flags you passed and already ends in `--force`. Run it as-is once the user approves — don't add `--force` to a command yourself, and don't add flags the user didn't ask for.

Run `npx checkly skills communicate` for the full protocol details, or `npx checkly skills configure` for what `deploy` confirms.

## API Pass-Through (fallback for any endpoint)

Expand Down
Loading