diff --git a/images/docs/images/cicd/github_setup_1.png b/images/docs/images/cicd/github_setup_1.png deleted file mode 100644 index da9aff12..00000000 Binary files a/images/docs/images/cicd/github_setup_1.png and /dev/null differ diff --git a/images/docs/images/cicd/github_setup_2.png b/images/docs/images/cicd/github_setup_2.png deleted file mode 100644 index f88a355f..00000000 Binary files a/images/docs/images/cicd/github_setup_2.png and /dev/null differ diff --git a/images/docs/images/cicd/github_setup_3.png b/images/docs/images/cicd/github_setup_3.png deleted file mode 100644 index 44569c10..00000000 Binary files a/images/docs/images/cicd/github_setup_3.png and /dev/null differ diff --git a/images/docs/images/cicd/github_setup_4.png b/images/docs/images/cicd/github_setup_4.png deleted file mode 100644 index 8fe07d0a..00000000 Binary files a/images/docs/images/cicd/github_setup_4.png and /dev/null differ diff --git a/images/docs/images/cicd/github_setup_5.png b/images/docs/images/cicd/github_setup_5.png deleted file mode 100644 index 6cc2e0bf..00000000 Binary files a/images/docs/images/cicd/github_setup_5.png and /dev/null differ diff --git a/images/docs/images/cicd/github_setup_6.png b/images/docs/images/cicd/github_setup_6.png deleted file mode 100644 index eddec4ba..00000000 Binary files a/images/docs/images/cicd/github_setup_6.png and /dev/null differ diff --git a/integrations/ci-cd/github/actions.mdx b/integrations/ci-cd/github/actions.mdx index 5469bba4..52054362 100644 --- a/integrations/ci-cd/github/actions.mdx +++ b/integrations/ci-cd/github/actions.mdx @@ -1,157 +1,235 @@ --- title: Integrating Checkly in GitHub Actions -description: "Run Checkly checks from GitHub Actions, triggering on deployment_status or push events to validate deployments before promoting your checks to monitors." +description: "Run Checkly checks from GitHub Actions and report test session results in pull requests." sidebarTitle: GitHub Actions canonical: 'https://www.checklyhq.com/docs/integrations/ci-cd/github/actions/' --- -## Workflow scenarios +Use the Checkly GitHub Action to run `checkly test` or `checkly trigger` from GitHub Actions. This lets you validate Monitoring as Code changes in pull requests, run checks against preview deployments, and report the test session result back to GitHub. -GitHub Actions workflows can run on different events. In most cases, you will want to trigger the Checkly CLI after a -**deployment** is done based on a recent `git push` to a branch, when a branch is merged or when a pull request is created. + + - A Checkly project with checks managed in code or deployed checks you want to trigger. + - A GitHub repository with GitHub Actions enabled. + - `CHECKLY_API_KEY` stored as a GitHub Actions secret. + - `CHECKLY_ACCOUNT_ID` stored as a GitHub Actions variable. + - Checkly CLI `8.15.0` or newer. The action uses the latest compatible CLI version by default. + - Optional: install the Checkly GitHub integration from [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations) to receive a separate GitHub Check for detached Checkly runs. + -GitHub created the [deployments API](https://docs.github.com/en/rest/deployments/deployments) for this and many 3rd party -integrators like Vercel and Heroku use this to relay deployment status to GitHub. However, you can also call this API -yourself and trigger a `deployment_status` event. + + Install the Checkly GitHub App from inside the Checkly web app: go to [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations) and click **Integrate with GitHub**. Installing the app directly from GitHub does not create the matching Checkly integration, so GitHub Check reporting will not work correctly. + -> Make sure to set your `CHECKLY_API_KEY` as a [secret](https://docs.github.com/en/actions/security-guides/encrypted-secrets) and `CHECKLY_ACCOUNT_ID` as a [variable](https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/store-information-in-variables) in your GitHub Actions settings before you get started. +## Choose a GitHub workflow -## Running on `deployment_status` events +There are two GitHub CI/CD paths in Checkly: -The `deployment_status` event is the preferred event to trigger a GH Actions workflow that executes your checks. However, -this comes with some peculiarities that are native to GH Actions and a bit different from using the `push` event. +| Use case | Guide | +| --- | --- | +| Run a Checkly test session from a GitHub Actions workflow and report the result on a pull request. | Continue with this guide. | +| Listen for GitHub deployment events and trigger checks linked to a repository, using the deployment environment URL as the test target. | Use [GitHub deployment hooks](/integrations/ci-cd/github/deployments). | -- The deployment event holds core information about your deployment, i.e. the environment name and an optional `ENVIRONMENT_URL`. -- The full git repo with full history is not available. We have to jump through some hoops to properly set the branch name -for instance. -- We have no access to the original pull request that triggered the deployment event. +## Run checks on pull requests + +Use `command: test` when your checks are defined in the repository. The action checks out the pull request code, installs dependencies, and runs `checkly test` on Checkly's infrastructure. ```yaml .github/workflows/checkly.yml -name: 'checkly' -on: [deployment_status] - -# Set the necessary credentials and export variables we can use to instrument our test run. Use the ENVIRONMENT_URL -# to run your checks against staging, preview or production. -env: - CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} - CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} - ENVIRONMENT_URL: ${{ github.event.deployment_status.environment_url }} - CHECKLY_TEST_ENVIRONMENT: ${{ github.event.deployment_status.environment }} +name: Checkly + +on: + pull_request: + jobs: - test-e2e: - if: github.event.deployment_status.state == 'success' # Only run when the deployment was successful. - name: Test E2E on Checkly + checkly: runs-on: ubuntu-latest - timeout-minutes: 10 steps: - - uses: actions/checkout@v3 - with: - ref: "${{ github.event.deployment_status.deployment.ref }}" - fetch-depth: 0 - - name: Set branch name # workaround to detect branch name in "deployment_status" actions - run: echo "CHECKLY_TEST_REPO_BRANCH=$(git show -s --pretty=%D HEAD | tr -s ',' '\n' | sed 's/^ //' | grep -e 'origin/' | head -1 | sed 's/\origin\///g')" >> $GITHUB_ENV - - uses: actions/setup-node@v3 + - uses: actions/checkout@v4 + + - uses: checkly/checkly-action@v1 with: - node-version-file: '.nvmrc' - - name: Restore or cache node_modules - id: cache-node-modules - uses: actions/cache@v3 + command: test + cli-version: latest + install-command: npm ci + reporting: auto + github-check-name: Checkly PR checks + env: + CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} + CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} +``` + +With `reporting: auto`, the action uses GitHub Check reporting when the Checkly GitHub App can report on the repository. If GitHub Check reporting is unavailable, the action waits for the test session and reports through the GitHub Actions job summary instead. + +## Run selected checks + +Filter the checks that run in a pull request with `tags`, `grep`, or file patterns. + +```yaml .github/workflows/checkly-filtered.yml +name: Checkly + +on: + pull_request: + +jobs: + checkly: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: checkly/checkly-action@v1 with: - path: node_modules - key: node-modules-${{ hashFiles('package-lock.json') }} - - name: Install dependencies - if: steps.cache-node-modules.outputs.cache-hit != 'true' - run: npm ci - - name: Run checks # run the checks passing in the ENVIRONMENT_URL and recording a test session. - id: run-checks - run: npx checkly test -e ENVIRONMENT_URL=${{ env.ENVIRONMENT_URL }} --reporter=github --record - - name: Create summary # export the markdown report to the job summary. - id: create-summary - run: cat checkly-github-report.md > $GITHUB_STEP_SUMMARY - - name: Deploy checks # if the test run was successful and we are on Production, deploy the checks - id: deploy-checks - if: steps.run-checks.outcome == 'success' && github.event.deployment_status.environment == 'Production' - run: npx checkly deploy --force + command: test + cli-version: latest + install-command: npm ci + reporting: auto + tags: | + production,webapp + production,api + grep: checkout + files: | + __checks__/**/*.check.ts + env: + CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} + CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} ``` -## Usage with mono repos and Vercel deployments +Multiple `tags` lines are treated as OR filters. Comma-separated tags on one line are treated as AND filters. -If you are running a mono repo, you might bump into issues that multiple `deployment_status` events will randomly trigger -your workflow. GitHub Actions does not have a great way to filter for this, but there are two strategies you can follow. +## Report results -### Filtering with `if` statements. +The `reporting` input controls where the Checkly result is reported. -If part of the URL for your deployments deterministically maps to one of the apps in your repo, you can just extend -the `if` statement in your workflow, as shown below. +| Value | Behavior | +| --- | --- | +| `auto` | Use a detached Checkly run with GitHub Check reporting when available. Otherwise, wait in the GitHub Actions job and report there. | +| `github-check` | Require detached GitHub Check reporting. The action fails before running checks if the Checkly GitHub App cannot report on the repository. | +| `github-actions` | Always wait in the GitHub Actions job and write the Checkly summary to the GitHub Actions job summary. | + +Use `github-check` when the Checkly GitHub Check should be a required status check on pull requests. + +```yaml .github/workflows/checkly-required.yml +name: Checkly required + +on: + pull_request: -```yaml .github/workflows/checkly-filtered.yml jobs: - test-e2e: - if: github.event.deployment_status.state == 'success' && contains(github.event.deployment_status.environment_url, 'webapp') - name: Test E2E on Checkly + checkly: runs-on: ubuntu-latest - timeout-minutes: 10 + steps: + - uses: actions/checkout@v4 + + - uses: checkly/checkly-action@v1 + with: + command: test + cli-version: latest + reporting: github-check + github-check-name: Checkly required checks + env: + CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} + CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} ``` -### Using branch protection rules +Use `github-check-name` to set the GitHub Check name that appears on pull requests. It defaults to `Checkly`. + + + GitHub Check reporting requires the Checkly GitHub App to be installed through [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations) in Checkly. Without the app, use `reporting: auto` or `reporting: github-actions`. + + +## Test preview deployments + +For pull request preview deployments, pass the preview URL to your checks as `ENVIRONMENT_URL`. In Browser checks and Playwright Check Suites, read `process.env.ENVIRONMENT_URL` in your test code. -In many cases, the above solution with `if` statements is not enough. When using Vercel and mono repos for instance, GitHub -Actions will recognize any "skipped" statuses as "passed". This can cause havoc. A way to sidestep this issue is by setting -a branch protection rule on your repo that is generated by your GH Action workflow. +```yaml .github/workflows/checkly-preview.yml +name: Checkly preview -You do this by adding the [LouisBrunner/checks-action@v1.6.0](https://github.com/marketplace/actions/github-checks) action -to your workflow and assigning some name. This name can be fixed or dynamic. See the example below. +on: + pull_request: -```yaml .github/workflows/checkly-branch-protection.yml -name: 'Run Checkly MaC Checks' -on: [deployment_status] -env: - ENVIRONMENT_URL: ${{ github.event.deployment_status.environment_url }} - CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} - CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} -run-name: Run MaC checks against ${{ github.event.deployment_status.environment_url }} jobs: - test-e2e: - if: github.event.deployment_status.state == 'success' && contains(github.event.deployment_status.environment_url, 'webapp') - name: Test E2E on Checkly + checkly: runs-on: ubuntu-latest - timeout-minutes: 10 steps: - # All steps as normal... - - uses: LouisBrunner/checks-action@v1.6.0 - if: always() + - uses: actions/checkout@v4 + + - id: preview + run: echo "url=https://preview.example.com" >> "$GITHUB_OUTPUT" + + - uses: checkly/checkly-action@v1 with: - token: ${{ secrets.GITHUB_TOKEN }} - name: Test E2E Passed Notification - conclusion: ${{ job.status }} + command: test + cli-version: latest + reporting: auto + env: | + ENVIRONMENT_URL=${{ steps.preview.outputs.url }} + env: + CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} + CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} + CHECKLY_TEST_ENVIRONMENT: preview + ENVIRONMENT_URL: ${{ steps.preview.outputs.url }} ``` -The will generate a custom status for your called **Test E2E Passed Notification**. +If your deployment provider already emits GitHub deployment events with an environment URL, you can also use [GitHub deployment hooks](/integrations/ci-cd/github/deployments) instead of a GitHub Actions workflow. -Then, add a [branch protection rule](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/managing-a-branch-protection-rule) -for your repo. +## Trigger deployed checks -![GitHub Branch Protection Ruke](/images/docs/images/cicd/gh_branch_protection_rule.png) +Use `command: trigger` when you want to run checks that are already deployed in your Checkly account. This is useful for smoke testing existing production or staging checks from a release workflow. -Together with any optional `if` statements, your GH Actions will now run only against the apps you want in your mono repo. +```yaml .github/workflows/checkly-trigger.yml +name: Checkly trigger -## Using the GitHub Markdown Reporter +on: + workflow_dispatch: -You can print a user friendly summary report to your GitHub Actions Job Summary by adding the `--reporter=github` flag -to the `test` command and export the resulting `checkly-github-report.md` file to the `$GITHUB_STEP_SUMMARY` variable. +jobs: + checkly: + runs-on: ubuntu-latest + steps: + - uses: checkly/checkly-action@v1 + with: + command: trigger + cli-version: latest + reporting: auto + tags: production + check-id: abc123,def456 + env: + CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} + CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} +``` + +## Use with monorepos + +In monorepos, combine GitHub Actions path filters with the action's `working-directory`, `tags`, `grep`, or `files` inputs. + +```yaml .github/workflows/checkly-monorepo.yml +name: Checkly webapp -In the example, you can see this in the following step: +on: + pull_request: + paths: + - apps/webapp/** + - packages/checks/** -```yaml .github/workflows/checkly-reporter.yml -- name: Run checks - id: run-checks - run: npx checkly test - -e ENVIRONMENT_URL=${{ env.ENVIRONMENT_URL }} - --reporter=github - --record -- name: Create summary - id: create-summary - run: cat checkly-github-report.md > $GITHUB_STEP_SUMMARY +jobs: + checkly: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: checkly/checkly-action@v1 + with: + command: test + cli-version: latest + working-directory: apps/webapp + install-command: npm ci + reporting: auto + tags: webapp + env: + CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }} + CHECKLY_ACCOUNT_ID: ${{ vars.CHECKLY_ACCOUNT_ID }} ``` +## Next steps +- Learn more about [`checkly test`](/cli/checkly-test). +- Learn more about [`checkly trigger`](/cli/checkly-trigger). +- Use [GitHub deployment hooks](/integrations/ci-cd/github/deployments) when you want Checkly to react to GitHub deployment webhook events without a GitHub Actions workflow. diff --git a/integrations/ci-cd/github/deployments.mdx b/integrations/ci-cd/github/deployments.mdx index 122fcbcf..4a0d3add 100644 --- a/integrations/ci-cd/github/deployments.mdx +++ b/integrations/ci-cd/github/deployments.mdx @@ -1,48 +1,46 @@ --- -title: Integrating Checkly in GitHub Deployments -description: "Trigger Checkly checks from GitHub deployment events fired by services like Vercel and Heroku, running against both preview and production deployments." -sidebarTitle: GitHub Deployments +title: GitHub deployment hooks +description: "Trigger linked Checkly checks from GitHub deployment events and use deployment environment URLs as test targets." +sidebarTitle: Deployment hooks canonical: 'https://www.checklyhq.com/docs/integrations/ci-cd/github/deployments/' --- +Use GitHub deployment hooks when you want Checkly to listen for GitHub `deployment_status` events and trigger checks linked to a repository. This is separate from the [Checkly GitHub Action](/integrations/ci-cd/github/actions), which starts test sessions from a GitHub Actions workflow. -GitHub deployments are webhook events triggered by GitHub whenever a deployment service reports a successful deployment -event. This works out-of-the-box with the following services. - -- [Vercel for GitHub](https://vercel.com/docs/v2/git-integrations/vercel-for-github) -- [Heroku Pipelines](https://devcenter.heroku.com/articles/pipelines) - -Any (SaaS) service that interacts with the [GitHub deployments API](https://developer.github.com/v3/repos/deployments/) and -reports the correct "success" message will work, both with production and preview deployments. + + - The Checkly GitHub integration installed from [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations). + - A deployment provider that sends successful GitHub `deployment_status` events, such as Vercel for GitHub or Heroku Pipelines. + - One or more Checkly checks linked to the GitHub repository from the check's **CI/CD** settings. + -This integration has some [known limitations](/integrations/ci-cd/github/deployments#limitations). We recommend using the [Checkly CLI](/cli/overview) in your CI/CD pipeline instead. + Install the Checkly GitHub App from inside the Checkly web app: go to [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations) and click **Integrate with GitHub**. Installing the app directly from GitHub does not create the matching Checkly integration, so deployment hooks and GitHub Check reporting will not work correctly. -## Setting up your GitHub integration +GitHub `deployment_status` webhook events are triggered whenever a deployment service reports a successful deployment. This works with services that report successful deployment statuses through the GitHub Deployments API, including: -1. Go to the **integrations tab** in your account settings and click the "Integrate with GitHub" button. -![set up GitHub integration step 1](/images/docs/images/cicd/github_setup_1.png) +- [Vercel for GitHub](https://vercel.com/docs/v2/git-integrations/vercel-for-github) +- [Heroku Pipelines](https://devcenter.heroku.com/articles/pipelines) +- Other deployment services that use the [GitHub deployments API](https://docs.github.com/en/rest/deployments/deployments) and report successful deployment statuses. -2. You will be redirected to GitHub and prompted to select an account and install the Checkly GitHub App. -![set up GitHub integration step 2](/images/docs/images/cicd/github_setup_2.png) + + This integration has some [known limitations](/integrations/ci-cd/github/deployments#limitations). Use the [Checkly GitHub Action](/integrations/ci-cd/github/actions) when you want GitHub Actions to start a recorded test session and report the result on a pull request. + -3. When accepting the installation, you are redirected back to Checkly and the integration is installed. -Now, go to the **CI/CD tab** of the check you want to link to a GitHub repository. In Browser checks, this will be under "Change settings". -![set up GitHub integration step 3](/images/docs/images/cicd/github_setup_3.png) +## Set up GitHub deployment hooks -4. Click the **"Link GitHub repo"** button and select the repository you want to link. -![set up GitHub integration step 4](/images/docs/images/cicd/github_setup_4.png) +1. Go to [Account settings > Integrations](https://app.checklyhq.com/settings/account/integrations) and click **Integrate with GitHub**. +2. In GitHub, choose the account where you want to install the Checkly GitHub App and grant it access to the repositories you want to connect. After installation, GitHub redirects you to Checkly. +3. Open the check or check group you want to trigger and go to its **CI/CD** settings. +4. Click **Link GitHub repo** and select the repository that reports your deployment events. -5. On each deployment event, we will run your check and report the results directly in **your GitHub timeline and PR overview**. -![set up GitHub integration step 5](/images/docs/images/cicd/github_setup_5.png) +After you link the repository, Checkly listens for successful deployment events and runs the linked check or check group. You can also choose a run location, filter deployments by environment or environment URL, and include screenshots in the pull request comment. -6. You will also get an overview of the check result in the *details* section -![set up GitHub integration step 6](/images/docs/images/cicd/github_setup_6.png) - -> You can hook up multiple checks to the same repo. We will just run all of them as a test suite. + + You can link multiple checks to the same repository. Checkly runs the linked checks as a test suite when GitHub reports a successful deployment. + -## Using environment URLs +## Use deployment environment URLs GitHub reports an **environment URL** on each deployment event. Depending on what deployment service you use, this environment URL can be used to run your check on different target environments than configured in your check. @@ -54,7 +52,7 @@ So, when you enable the **"Use environment URL from deployment trigger"** checkb ### API checks & environment URLs -With API checks, we replace the hostname part of your request url with the host in the environment URL. +With API checks, Checkly replaces the hostname part of your request URL with the host in the environment URL. For example: - Your configured URL: `https://api.acme.com/v1/customers?page=1` @@ -68,7 +66,7 @@ Notice we only replace the **host** part, not the URL path or any query paramete For browser checks, the environment URL is exposed as the `ENVIRONMENT_URL` environment variable. This means you can use that variable in your code to replace any hardcoded URL you might have, i.e.: - + ```typescript test.spec.ts import { expect, test } from '@playwright/test' @@ -92,18 +90,18 @@ test('assert response status of page', async ({ page }) => { ``` -This way we are setting the `targetUrl` variable to either the `ENVIRONMENT_URL` or just our main production URL. +This sets the `targetUrl` variable to either the `ENVIRONMENT_URL` or your main production URL. -Whenever a **Preview** deploy happens on GitHub, this check gets called and runs the script against the preview environment. This check also runs on a set interval, and checks our production environment. +When a **Preview** deployment happens on GitHub, this check runs the script against the preview environment. This check also runs on a set interval and checks your production environment. -This way, we kill two birds with one stone and don't need separate checks for separate environments. +With one check, you can validate preview deployments from GitHub events and keep running the same check on your production schedule. ## Limitations -GitHub deployments lacks support for several features: +GitHub deployment hooks lack support for several features: * Client certificates are not applied. * OpenTelemetry integration headers are not applied. * Private locations are not available. -As an alternative, we recommend using the [Checkly CLI](/cli/overview) in your CI/CD pipeline. The CLI is much more powerful and fully supported. +Use the [Checkly GitHub Action](/integrations/ci-cd/github/actions) or the [Checkly CLI](/cli/overview) in your CI/CD pipeline when you need these features. diff --git a/integrations/ci-cd/overview.mdx b/integrations/ci-cd/overview.mdx index 1579d764..a3760bb1 100644 --- a/integrations/ci-cd/overview.mdx +++ b/integrations/ci-cd/overview.mdx @@ -26,7 +26,7 @@ The preferred and most flexible way to integrate Checkly with your CI/CD is thro ## CI/CD Integration using vendor webhooks - [Vercel](/integrations/ci-cd/vercel/overview) -- [GitHub deployments](/integrations/ci-cd/github/deployments) +- [GitHub deployment hooks](/integrations/ci-cd/github/deployments) ## CI/CD Basics We've optimized the [Checkly CLI](/cli/overview) to work in any CI/CD workflow. Here are the basics you need to know that @@ -41,4 +41,4 @@ in your CI/CD platform. The CLI will attempt to parse `git` specific information from the environment to display in the test session as metadata. You can also set these data items specifically by using environment variables. - \ No newline at end of file + diff --git a/snippets/main-cicd-cards.mdx b/snippets/main-cicd-cards.mdx index 75607214..679d54dd 100644 --- a/snippets/main-cicd-cards.mdx +++ b/snippets/main-cicd-cards.mdx @@ -4,7 +4,7 @@ icon="/images/github-icon-dark.svg" href="/integrations/ci-cd/github/actions" > - Run the Checkly CLI from GitHub Actions, export summary reports and integrate with mono repos + Run Checkly test sessions from GitHub Actions and report results in pull requests.