diff --git a/packages/cli/src/ai-context/references/communicate.md b/packages/cli/src/ai-context/references/communicate.md index 745cd505b..506b42448 100644 --- a/packages/cli/src/ai-context/references/communicate.md +++ b/packages/cli/src/ai-context/references/communicate.md @@ -20,11 +20,13 @@ Write commands (`incidents create`, `incidents update`, `incidents resolve`, `de **Rules for agents:** 1. When exit code is 2 and output contains `"status": "confirmation_required"`, **always present the `changes` array to the user** and ask for confirmation. -2. **Never auto-append `--force`** — only run the `confirmCommand` after the user explicitly approves. +2. **Run the `confirmCommand` verbatim, and only after the user explicitly approves.** It is normally the command you just ran, plus `--force` to skip the second prompt. Don't append `--force` to anything yourself, and don't edit the `confirmCommand` — changing its flags changes what you were authorized to do. 3. This applies to **every** write command, not just the first one. Incident updates and resolutions also require confirmation. 4. Use `--dry-run` to preview what a command will do without executing or prompting. 5. Read-only commands (`incidents list`, `status-pages list`) execute immediately without confirmation. +The `confirmCommand` omits flags left at their default, so a bare `npx checkly deploy` confirms as `checkly deploy --force` rather than echoing back every boolean the parser filled in. Treat every flag you see there as deliberate. + ## Available Commands Parse and read further reference documentation when tasked with any of the following: diff --git a/packages/cli/src/ai-context/references/configure.md b/packages/cli/src/ai-context/references/configure.md index a45c2e1b4..c32c53b84 100644 --- a/packages/cli/src/ai-context/references/configure.md +++ b/packages/cli/src/ai-context/references/configure.md @@ -75,3 +75,26 @@ Run `npx checkly skills manage plan` for the full reference. ## Deploying - Deploy checks using the `npx checkly deploy` command. Use `--output` to see the created, updated, and deleted resources. Use `--verbose` to also include each resource's name and physical ID (UUID), which is useful for programmatically referencing deployed resources (e.g. `npx checkly checks get `). +- Use `--preview` to see what a deploy would change without applying it. + +### Deleted resources + +A deploy makes the account match the code. **Any resource that was deployed before and is no longer in the code gets deleted, along with its run history.** Pass `--preserve-resources` to keep those resources and their history in the Checkly account instead, where the user can manage them from the web app. + +This matters when the local project isn't the whole picture — a partial checkout, or a project whose checks were also edited elsewhere. If you're not sure the code is the complete source of truth, say so before deploying. + +### Confirmation + +`deploy` is a write command: without `--force` it returns exit code 2 and a `confirmation_required` envelope. Present its `changes` to the user and run the `confirmCommand` verbatim only after they approve. + +That confirmation happens **before the project is parsed**, so it cannot tell you which resources would be deleted — its `changes` only warn that deletion is possible. There is a second, itemised guard that lists each doomed resource by name, but it is skipped by `--force`, and `--force` is exactly what the `confirmCommand` carries. **An agent following the confirmation protocol never sees that list.** It is a prompt for humans deploying by hand. + +So when resources may have been removed from the code, don't rely on the confirmation to surface it — preview first: + +```bash +npx checkly deploy --preview +``` + +This is a dry run: nothing is applied and nothing is confirmed. It prints the resources that would be created, updated, and deleted (add `--verbose` for names and IDs). Show the user what would be deleted, and only then deploy. + +Run `npx checkly skills communicate` for the full protocol. diff --git a/packages/cli/src/ai-context/skill.md b/packages/cli/src/ai-context/skill.md index 9f28eac82..c24f11267 100644 --- a/packages/cli/src/ai-context/skill.md +++ b/packages/cli/src/ai-context/skill.md @@ -34,9 +34,11 @@ Run `npx checkly skills manage` for the full reference. ## Confirmation Protocol -Write commands (e.g. `incidents create`, `deploy`, `destroy`) return exit code 2 with a `confirmation_required` JSON envelope instead of executing. **Always present the `changes` to the user and wait for approval before running the `confirmCommand`.** Never auto-append `--force`. This applies to every write command individually — updates and resolutions need confirmation too, not just the initial create. +Write commands (e.g. `incidents create`, `deploy`, `destroy`) return exit code 2 with a `confirmation_required` JSON envelope instead of executing. **Always present the `changes` to the user and wait for approval before running the `confirmCommand`.** This applies to every write command individually — updates and resolutions need confirmation too, not just the initial create. -Run `npx checkly skills communicate` for the full protocol details. +The `confirmCommand` is the approved command, ready to run verbatim: it repeats the flags you passed and already ends in `--force`. Run it as-is once the user approves — don't add `--force` to a command yourself, and don't add flags the user didn't ask for. + +Run `npx checkly skills communicate` for the full protocol details, or `npx checkly skills configure` for what `deploy` confirms. ## API Pass-Through (fallback for any endpoint) diff --git a/packages/cli/src/commands/__tests__/confirm-flow-deploy.spec.ts b/packages/cli/src/commands/__tests__/confirm-flow-deploy.spec.ts index 9e85e5e1b..ce37f1c1f 100644 --- a/packages/cli/src/commands/__tests__/confirm-flow-deploy.spec.ts +++ b/packages/cli/src/commands/__tests__/confirm-flow-deploy.spec.ts @@ -1,3 +1,4 @@ +import { Parser } from '@oclif/core' import { beforeEach, describe, expect, it, vi } from 'vitest' vi.mock('../../helpers/cli-mode', () => ({ @@ -39,11 +40,23 @@ vi.mock('prompts', () => ({ })) import { detectCliMode } from '../../helpers/cli-mode.js' +import { buildConfirmCommand } from '../../helpers/command-preview.js' import * as api from '../../rest/api.js' import { parseProject } from '../../services/project-parser.js' import { AuthCommand } from '../authCommand.js' import Deploy from '../deploy.js' +/** Turn a generated confirmCommand back into argv, so oclif can re-parse it. */ +function flagArgv (confirmCommand: string): string[] { + return [...confirmCommand.matchAll(/--[a-zA-Z0-9-]+(?:="[^"]*")?/g)] + .map(([token]) => token.replace(/="(.*)"$/, '=$1')) +} + +async function confirmCommandFor (argv: string[]): Promise { + const { flags, metadata } = await Parser.parse(argv, { flags: Deploy.flags, strict: true }) + return buildConfirmCommand('deploy', flags, undefined, metadata.flags) +} + function createConfirmContext () { const logged: string[] = [] let exitCodeValue: number | undefined @@ -156,6 +169,17 @@ describe('deploy confirmation flow', () => { 'debug-bundle': false, 'debug-bundle-output-file': './debug-bundle.json', }, + metadata: { + flags: { + 'preview': { setFromDefault: true }, + 'output': { setFromDefault: true }, + 'verbose': { setFromDefault: true }, + 'schedule-on-deploy': { setFromDefault: true }, + 'verify-runtime-dependencies': { setFromDefault: true }, + 'debug-bundle': { setFromDefault: true }, + 'debug-bundle-output-file': { setFromDefault: true }, + }, + }, }) await expect( @@ -172,5 +196,28 @@ describe('deploy confirmation flow', () => { expect(output.command).toBe('deploy') expect(output.classification.idempotent).toBe(true) expect(output.confirmCommand).toContain('--force') + expect(output.confirmCommand).not.toContain('--no-preview') + }) +}) + +describe('deploy confirmCommand', () => { + it('echoes only the flags the user typed', async () => { + expect(await confirmCommandFor([])).toBe('checkly deploy --force') + expect(await confirmCommandFor(['--preserve-resources'])).toBe('checkly deploy --preserve-resources --force') + }) + + it('keeps an explicit --no- for flags that allow it', async () => { + expect(await confirmCommandFor(['--no-schedule-on-deploy'])) + .toBe('checkly deploy --no-schedule-on-deploy --force') + }) + + it('generates a command oclif can parse back', async () => { + for (const argv of [[], ['--preserve-resources'], ['--no-schedule-on-deploy'], ['--verbose']]) { + const confirmCommand = await confirmCommandFor(argv) + await expect( + Parser.parse(flagArgv(confirmCommand), { flags: Deploy.flags, strict: true }), + `confirmCommand for "checkly deploy ${argv.join(' ')}" must be runnable: ${confirmCommand}`, + ).resolves.toBeDefined() + } }) }) diff --git a/packages/cli/src/commands/__tests__/confirm-flow-destroy.spec.ts b/packages/cli/src/commands/__tests__/confirm-flow-destroy.spec.ts index c4f65c381..bb41fce53 100644 --- a/packages/cli/src/commands/__tests__/confirm-flow-destroy.spec.ts +++ b/packages/cli/src/commands/__tests__/confirm-flow-destroy.spec.ts @@ -1,3 +1,4 @@ +import { Parser } from '@oclif/core' import { beforeEach, describe, expect, it, vi } from 'vitest' vi.mock('../../helpers/cli-mode', () => ({ @@ -29,14 +30,26 @@ vi.mock('prompts', () => ({ import { detectCliMode } from '../../helpers/cli-mode.js' import prompts from 'prompts' import * as api from '../../rest/api.js' +import { buildConfirmCommand } from '../../helpers/command-preview.js' import { AuthCommand } from '../authCommand.js' import Destroy from '../destroy.js' -function createCommandContext (parsed: unknown) { +/** Turn a generated confirmCommand back into argv, so oclif can re-parse it. */ +function flagArgv (confirmCommand: string): string[] { + return [...confirmCommand.matchAll(/--[a-zA-Z0-9-]+(?:="[^"]*")?/g)] + .map(([token]) => token.replace(/="(.*)"$/, '=$1')) +} + +async function confirmCommandFor (argv: string[]): Promise { + const { flags, metadata } = await Parser.parse(argv, { flags: Destroy.flags, strict: true }) + return buildConfirmCommand('destroy', flags, undefined, metadata.flags) +} + +function createCommandContext (parsed: { flags: Record, metadata?: unknown }) { const logged: string[] = [] let exitCodeValue: number | undefined return { - parse: vi.fn().mockResolvedValue(parsed), + parse: vi.fn().mockResolvedValue({ metadata: { flags: {} }, ...parsed }), log: vi.fn((msg?: string) => { if (msg) logged.push(msg) }), @@ -172,3 +185,20 @@ describe('destroy confirmation flow', () => { expect(Destroy.idempotent).toBe(false) }) }) + +describe('destroy confirmCommand', () => { + it('echoes only the flags the user typed', async () => { + expect(await confirmCommandFor([])).toBe('checkly destroy --force') + expect(await confirmCommandFor(['--preserve-resources'])).toBe('checkly destroy --preserve-resources --force') + }) + + it('generates a command oclif can parse back', async () => { + for (const argv of [[], ['--preserve-resources'], ['--cancel-in-progress-deployment']]) { + const confirmCommand = await confirmCommandFor(argv) + await expect( + Parser.parse(flagArgv(confirmCommand), { flags: Destroy.flags, strict: true }), + `confirmCommand for "checkly destroy ${argv.join(' ')}" must be runnable: ${confirmCommand}`, + ).resolves.toBeDefined() + } + }) +}) diff --git a/packages/cli/src/commands/deploy.ts b/packages/cli/src/commands/deploy.ts index c09554cbe..8bd258395 100644 --- a/packages/cli/src/commands/deploy.ts +++ b/packages/cli/src/commands/deploy.ts @@ -110,7 +110,7 @@ export default class Deploy extends AuthCommand { } async run (): Promise { - const { flags } = await this.parse(Deploy) + const { flags, metadata } = await this.parse(Deploy) const { force, preview, @@ -146,6 +146,7 @@ export default class Deploy extends AuthCommand { : 'Delete any resources removed from code, losing their run history. Pass --preserve-resources to keep them in your Checkly account instead', ], flags, + flagMetadata: metadata.flags, classification: { readOnly: Deploy.readOnly, destructive: Deploy.destructive, @@ -316,6 +317,7 @@ export default class Deploy extends AuthCommand { `Delete ${PRETTY_RESOURCE_TYPES[resourceType] ?? resourceType}: ${logicalId}`), ], flags, + flagMetadata: metadata.flags, classification: { readOnly: Deploy.readOnly, destructive: Deploy.destructive, diff --git a/packages/cli/src/commands/destroy.ts b/packages/cli/src/commands/destroy.ts index 3054c8581..4ec85eaaf 100644 --- a/packages/cli/src/commands/destroy.ts +++ b/packages/cli/src/commands/destroy.ts @@ -31,7 +31,7 @@ export default class Destroy extends AuthCommand { } async run (): Promise { - const { flags } = await this.parse(Destroy) + const { flags, metadata } = await this.parse(Destroy) const { config: configFilename, 'preserve-resources': preserveResources, @@ -52,6 +52,7 @@ export default class Destroy extends AuthCommand { : `PERMANENTLY delete ALL resources associated with the project "${checklyConfig.projectName}" in account "${account.name}"`, ], flags, + flagMetadata: metadata.flags, classification: { readOnly: Destroy.readOnly, destructive: Destroy.destructive, diff --git a/packages/cli/src/commands/incidents/__tests__/confirm-flow.spec.ts b/packages/cli/src/commands/incidents/__tests__/confirm-flow.spec.ts index 0dad4ee1c..193e3f9c7 100644 --- a/packages/cli/src/commands/incidents/__tests__/confirm-flow.spec.ts +++ b/packages/cli/src/commands/incidents/__tests__/confirm-flow.spec.ts @@ -50,12 +50,12 @@ const incidentFixture = { services: [{ id: 'svc-1', name: 'Checkout API', accountId: 'acc-1' }], } -function createCommandContext (parsed: unknown) { +function createCommandContext (parsed: { flags: Record, args?: unknown, metadata?: unknown }) { const logged: string[] = [] let didExit = false let exitCodeValue: number | undefined return { - parse: vi.fn().mockResolvedValue(parsed), + parse: vi.fn().mockResolvedValue({ metadata: { flags: {} }, ...parsed }), log: vi.fn((msg?: string) => { if (msg) logged.push(msg) }), diff --git a/packages/cli/src/commands/incidents/__tests__/notify-subscribers.spec.ts b/packages/cli/src/commands/incidents/__tests__/notify-subscribers.spec.ts index c00d3fe67..7dcabf76a 100644 --- a/packages/cli/src/commands/incidents/__tests__/notify-subscribers.spec.ts +++ b/packages/cli/src/commands/incidents/__tests__/notify-subscribers.spec.ts @@ -43,10 +43,13 @@ const statusPageFixture: StatusPage = { updated_at: '2026-02-25T10:00:00.000Z', } -function createCommandContext (parsed: unknown, CommandClass: any = IncidentsCreate) { +function createCommandContext ( + parsed: { flags: Record, args?: unknown, metadata?: unknown }, + CommandClass: any = IncidentsCreate, +) { const logged: string[] = [] return { - parse: vi.fn().mockResolvedValue(parsed), + parse: vi.fn().mockResolvedValue({ metadata: { flags: {} }, ...parsed }), log: vi.fn((msg?: string) => { if (msg) logged.push(msg) }), diff --git a/packages/cli/src/commands/incidents/create.ts b/packages/cli/src/commands/incidents/create.ts index 88dcebf1a..490c76348 100644 --- a/packages/cli/src/commands/incidents/create.ts +++ b/packages/cli/src/commands/incidents/create.ts @@ -51,7 +51,7 @@ export default class IncidentsCreate extends AuthCommand { } async run (): Promise { - const { flags } = await this.parse(IncidentsCreate) + const { flags, metadata } = await this.parse(IncidentsCreate) this.style.outputFormat = flags.output const statusPage = await api.statusPages.get(flags['status-page-id']) @@ -73,6 +73,7 @@ export default class IncidentsCreate extends AuthCommand { : 'Subscribers will NOT be notified', ], flags, + flagMetadata: metadata.flags, classification: { readOnly: IncidentsCreate.readOnly, destructive: IncidentsCreate.destructive, diff --git a/packages/cli/src/commands/incidents/resolve.ts b/packages/cli/src/commands/incidents/resolve.ts index d76b8997c..16039842c 100644 --- a/packages/cli/src/commands/incidents/resolve.ts +++ b/packages/cli/src/commands/incidents/resolve.ts @@ -33,7 +33,7 @@ export default class IncidentsResolve extends AuthCommand { } async run (): Promise { - const { args, flags } = await this.parse(IncidentsResolve) + const { args, flags, metadata } = await this.parse(IncidentsResolve) this.style.outputFormat = flags.output const incident = await api.incidents.get(args.id) @@ -48,6 +48,7 @@ export default class IncidentsResolve extends AuthCommand { : 'Subscribers will NOT be notified', ], flags, + flagMetadata: metadata.flags, args: { id: args.id }, classification: { readOnly: IncidentsResolve.readOnly, diff --git a/packages/cli/src/commands/incidents/update.ts b/packages/cli/src/commands/incidents/update.ts index c798c7771..1c87f8c2f 100644 --- a/packages/cli/src/commands/incidents/update.ts +++ b/packages/cli/src/commands/incidents/update.ts @@ -50,7 +50,7 @@ export default class IncidentsUpdate extends AuthCommand { } async run (): Promise { - const { args, flags } = await this.parse(IncidentsUpdate) + const { args, flags, metadata } = await this.parse(IncidentsUpdate) this.style.outputFormat = flags.output const incident = await api.incidents.get(args.id) @@ -69,6 +69,7 @@ export default class IncidentsUpdate extends AuthCommand { description: 'Post progress update to incident', changes, flags, + flagMetadata: metadata.flags, args: { id: args.id }, classification: { readOnly: IncidentsUpdate.readOnly, diff --git a/packages/cli/src/helpers/__tests__/command-preview.spec.ts b/packages/cli/src/helpers/__tests__/command-preview.spec.ts index 93ea1b707..32a1dffb4 100644 --- a/packages/cli/src/helpers/__tests__/command-preview.spec.ts +++ b/packages/cli/src/helpers/__tests__/command-preview.spec.ts @@ -124,4 +124,22 @@ describe('buildConfirmCommand', () => { const result = buildConfirmCommand('incidents update', { message: 'Fix deployed' }, { id: 'inc-123' }) expect(result).toBe('checkly incidents update inc-123 --message="Fix deployed" --force') }) + + it('omits flags oclif filled in from their default', () => { + const result = buildConfirmCommand('deploy', { + 'preview': false, + 'schedule-on-deploy': true, + 'preserve-resources': true, + }, undefined, { + 'preview': { setFromDefault: true }, + 'schedule-on-deploy': { setFromDefault: true }, + 'preserve-resources': { setFromDefault: false }, + }) + expect(result).toBe('checkly deploy --preserve-resources --force') + }) + + it('keeps default-valued flags when no metadata is given', () => { + const result = buildConfirmCommand('deploy', { 'schedule-on-deploy': true }) + expect(result).toBe('checkly deploy --schedule-on-deploy --force') + }) }) diff --git a/packages/cli/src/helpers/command-preview.ts b/packages/cli/src/helpers/command-preview.ts index 7d59c7060..64f8368d1 100644 --- a/packages/cli/src/helpers/command-preview.ts +++ b/packages/cli/src/helpers/command-preview.ts @@ -4,11 +4,18 @@ export type CommandClassification = { idempotent: boolean } +/** + * The subset of oclif's parse metadata we rely on: which flags were filled in from + * `default:` rather than typed by the user. Pass `metadata.flags` from `this.parse()`. + */ +export type FlagMetadata = Record + export type CommandPreview = { command: string description: string changes: string[] flags: Record + flagMetadata?: FlagMetadata args?: Record classification: CommandClassification } @@ -28,6 +35,7 @@ export function buildConfirmCommand ( command: string, flags: Record, args?: Record, + flagMetadata?: FlagMetadata, ): string { const parts = ['checkly', command] @@ -40,6 +48,9 @@ export function buildConfirmCommand ( for (const [key, value] of Object.entries(flags)) { if (OMITTED_FLAGS.has(key)) continue if (value === undefined || value === null) continue + // Defaults are not part of what the user asked for, and a default `false` renders as + // `--no-x`, which only parses when the flag sets `allowNo: true`. + if (flagMetadata?.[key]?.setFromDefault) continue if (Array.isArray(value)) { for (const item of value) { @@ -66,7 +77,7 @@ export function formatPreviewForAgent ( description: preview.description, classification: preview.classification, changes: preview.changes, - confirmCommand: buildConfirmCommand(preview.command, preview.flags, preview.args), + confirmCommand: buildConfirmCommand(preview.command, preview.flags, preview.args, preview.flagMetadata), } } diff --git a/skills/checkly/SKILL.md b/skills/checkly/SKILL.md index 0c1e17744..c44432618 100644 --- a/skills/checkly/SKILL.md +++ b/skills/checkly/SKILL.md @@ -34,9 +34,11 @@ Run `npx checkly skills manage` for the full reference. ## Confirmation Protocol -Write commands (e.g. `incidents create`, `deploy`, `destroy`) return exit code 2 with a `confirmation_required` JSON envelope instead of executing. **Always present the `changes` to the user and wait for approval before running the `confirmCommand`.** Never auto-append `--force`. This applies to every write command individually — updates and resolutions need confirmation too, not just the initial create. +Write commands (e.g. `incidents create`, `deploy`, `destroy`) return exit code 2 with a `confirmation_required` JSON envelope instead of executing. **Always present the `changes` to the user and wait for approval before running the `confirmCommand`.** This applies to every write command individually — updates and resolutions need confirmation too, not just the initial create. -Run `npx checkly skills communicate` for the full protocol details. +The `confirmCommand` is the approved command, ready to run verbatim: it repeats the flags you passed and already ends in `--force`. Run it as-is once the user approves — don't add `--force` to a command yourself, and don't add flags the user didn't ask for. + +Run `npx checkly skills communicate` for the full protocol details, or `npx checkly skills configure` for what `deploy` confirms. ## API Pass-Through (fallback for any endpoint)