diff --git a/docs/admin/executors/index.mdx b/docs/admin/executors/index.mdx index 346bb53f1..0f5d0e5b4 100644 --- a/docs/admin/executors/index.mdx +++ b/docs/admin/executors/index.mdx @@ -5,26 +5,105 @@ Available via the Web app. -Executors are Sourcegraph's solution for isolating and running workloads in a secure and controllable way. Executors provide a sandbox that can run resource-intensive or untrusted tasks on behalf of the Sourcegraph instance, such as: +Executors run resource-intensive or untrusted work on behalf of Sourcegraph. They are used to: -- [Automatically indexing a repository for precise code navigation](/code-navigation/auto-indexing) -- [Running batch changes](/batch-changes/server-side) +- [Automatically index repositories for precise code navigation](/code-navigation/auto-indexing) +- [Run batch changes server-side](/batch-changes/server-side) + +This page covers the Sourcegraph configuration and administrative tools that apply to every executor. If you operate a self-hosted Sourcegraph instance and need to deploy executor infrastructure, see [Deploy executors](/self-hosted/executors). ## Why use executors? -Running untrusted code is a core requirement of features such as precise code navigation [auto-indexing](/code-navigation/auto-indexing), and [running batch changes server-side](/batch-changes/server-side). +Running untrusted code is a core requirement of features such as precise code navigation [auto-indexing](/code-navigation/auto-indexing) and [running batch changes server-side](/batch-changes/server-side). + +Auto-indexing jobs, in particular, invoke project build tools to resolve dependencies. Post-install hooks, insecure [package management tools](https://github.com/golang/go/issues/29230), and package manager proxy attacks can allow malicious code to consume compute resources or exfiltrate data. This is especially dangerous when the work runs inside a Sourcegraph instance that has repository data on disk and unprotected services on its internal network. + +Executors move this work outside the Sourcegraph instance. A job receives an isolated workspace containing a clone of only the repository it needs, limiting its access to other repository data and Sourcegraph services. + +## How executors work + +Executors run jobs outside the Sourcegraph instance. Each executor connects to Sourcegraph, requests work from a queue, prepares an isolated workspace containing one repository, runs the job, and sends its progress and result back to Sourcegraph. + +Keeping this work outside the Sourcegraph instance limits the access that untrusted build tools, package managers, and user-defined commands have to Sourcegraph services and repository data. The exact isolation boundary depends on how the executor is deployed. Self-hosted administrators should review the [deployment and isolation options](/self-hosted/executors#choose-a-deployment-method) before deploying executors. + +## Sandboxing model + +Executors can run each job inside a [Firecracker](https://sourcegraph.com/github.com/firecracker-microvm/firecracker) MicroVM to isolate the job from other jobs and the executor host. Firecracker provides the strongest supported isolation boundary and requires Linux KVM extensions. On cloud infrastructure, this generally requires a bare-metal instance on AWS or nested virtualization on Google Cloud. + +Executors can also run jobs in Docker containers or as Kubernetes Jobs without KVM-based isolation. These methods are easier to deploy on common infrastructure, but they provide a weaker isolation boundary and require additional host, node, and network controls. + +See [Firecracker isolation](/self-hosted/executors/firecracker) for architecture and caveats, or [choose a deployment method](/self-hosted/executors#choose-a-deployment-method) based on your infrastructure and security requirements. + +## Setting up executors + +The following steps configure a self-hosted Sourcegraph instance to accept connections from executors. They are separate from deploying the executor service onto infrastructure such as a Linux host, cloud virtual machine, or Kubernetes cluster. + + + Sourcegraph Cloud includes Sourcegraph-managed executor infrastructure by + default. Cloud customers using these managed executors do not need to + configure executor authentication or deploy their own executor + infrastructure. + + +If you operate a self-hosted Sourcegraph instance, complete the steps below and then follow the [self-hosted deployment guide](/self-hosted/executors) to choose a deployment method and install executors on your infrastructure. + +Self-hosted executors authenticate to Sourcegraph with a shared secret: + +1. Generate a random secret containing at least 20 characters. For example, use OpenSSL to generate a 64-character hexadecimal secret: + + ```bash + openssl rand -hex 32 + ``` + +1. Open **Site admin > Configuration > Advanced configuration**. +1. Set `executors.accessToken` to the generated secret and save the site configuration. + + ``` + { "executors.accessToken": "" } + ``` + +1. Configure the same secret on each executor as `EXECUTOR_FRONTEND_PASSWORD`, or through the equivalent setting for its deployment method. + +The executor authentication secret is not a Sourcegraph user access token and is not managed on the **Access tokens** page. See [Site configuration](/admin/config/site-config#view-and-edit-site-configuration) for more information about editing site configuration. + +## Monitor executor instances + + + The **Executor instances** page becomes available only after + `executors.accessToken` is configured in site configuration. + + +Open **Site admin > Maintenance > Executor instances** to see executors that have connected to Sourcegraph. The page shows whether each executor is active, its Sourcegraph compatibility, and details including its hostname and installed runtime versions. -Auto-indexing jobs, in particular, require the invocation of arbitrary and untrusted code to support the resolution of project dependencies. Invocation of post-install hooks, use of insecure [package management tools](https://github.com/golang/go/issues/29230), and package manager proxy attacks can create opportunities in which an adversary can gain unlimited use of compute or exfiltrate data. The latter outcome is particularly dangerous for on-premise installations of Sourcegraph, which is the chosen option for companies wanting to maintain strict privacy of their code property. +![Executor instances page showing active and inactive executors](https://storage.googleapis.com/sourcegraph-assets/docs/images/executors/executor-instances.png) -Instead of performing this work within the Sourcegraph instance, where code is available on disk and unprotected internal services are available over the local network, we move untrusted compute into a sandboxed environment, the _executor_, that has access only to the clone of a single repository on disk (its _workspace_) and to the public internet. +Use this page to confirm that a new or upgraded executor is connected and reporting the expected version. For host, service, or runtime failures, see [Troubleshoot executors](/self-hosted/executors/executors-troubleshooting). -## Sandboxing Model +## Manage executor secrets -Executors can be deployed with [Firecracker](https://sourcegraph.com/github.com/firecracker-microvm/firecracker) isolation in accordance with our [sandboxing model](/admin/executors/#how-it-works) to isolate jobs from each other and the host. -This requires executors to be run on machines capable of running Linux KVM extensions. On the most popular cloud providers, this either means running executors on bare-metal machines (AWS) or machines capable of nested virtualization (GCP). +Executor secrets provide environment variables to jobs without placing secret values in a batch spec. Global secrets are available to every eligible job, while user and organization secrets apply only to jobs in their respective namespaces. Namespaced secrets override global secrets with the same name. -Optionally, executors can be run without using KVM-based isolation, which is less secure but might be easier to run on common machines. +Site administrators manage global secrets under **Site admin > Batch Changes > Secrets**. Users and organization administrators manage their scoped secrets from the corresponding user or organization settings. -## How it works +Learn how to create, rotate, and remove [executor secrets](/admin/executors/executor-secrets). -Executor instances are capable of being deployed in a variety of ways. Each runtime varies in how jobs are executed. + + + + + diff --git a/docs/agentic-batch-changes/index.mdx b/docs/agentic-batch-changes/index.mdx index f47cedf4b..da6d718bf 100644 --- a/docs/agentic-batch-changes/index.mdx +++ b/docs/agentic-batch-changes/index.mdx @@ -34,7 +34,7 @@ Before creating an Agentic Batch Change, make sure your Sourcegraph instance can - [Configure code host connections](/admin/code-hosts) for the repositories you want the agent to search and modify. - [Configure Batch Changes credentials](/batch-changes/configuring-credentials) for each code host where the agent should publish pull requests or merge requests. -- [Deploy Sourcegraph executors](/self-hosted/executors/deploy-executors) so Agentic Batch Changes can run code-modification jobs in isolated workspaces. Executors are available with no additional setup on [Sourcegraph Cloud](/cloud). +- If you use self-hosted Sourcegraph, [deploy Sourcegraph executors](/self-hosted/executors) so Agentic Batch Changes can run code-modification jobs in isolated workspaces. [Sourcegraph Cloud](/cloud) includes Sourcegraph-managed executors by default, so Cloud customers using them do not need to deploy executor infrastructure. ### Prompting diff --git a/docs/batch-changes/faq.mdx b/docs/batch-changes/faq.mdx index 452759d61..934be401f 100644 --- a/docs/batch-changes/faq.mdx +++ b/docs/batch-changes/faq.mdx @@ -226,7 +226,7 @@ They can! Each changeset that is computed can be assigned to a separate executor ### What additional resources do I need to provision to run batch changes server-side? -See [deploying executors](/self-hosted/executors/deploy-executors) page. You'll require little as a single compute instance and a docker registry mirror if you just want to process batch changes at a small scale; an autoscaling group of instances if you want to process large batch changes very fast. +For self-hosted Sourcegraph, see [deploying executors](/self-hosted/executors). You may require as little as a single compute instance and a Docker registry mirror to process Batch Changes at a small scale, or an autoscaling group of instances to process large Batch Changes quickly. Sourcegraph Cloud includes Sourcegraph-managed executors by default, so Cloud customers using them do not need to deploy executor infrastructure. ### Can someone accidentally take down the Sourcegraph instance if they run too big a batch change? diff --git a/docs/batch-changes/how-src-executes-a-batch-spec.mdx b/docs/batch-changes/how-src-executes-a-batch-spec.mdx index fe9d2add0..84f0bde31 100644 --- a/docs/batch-changes/how-src-executes-a-batch-spec.mdx +++ b/docs/batch-changes/how-src-executes-a-batch-spec.mdx @@ -7,6 +7,10 @@ Here, you will learn what happens when a user applies or previews a Batch Spec by running `src batch apply` or `src batch preview` commands. +![Local Batch Changes execution with src-cli](https://storage.googleapis.com/sourcegraph-assets/executor_src_local_arch.png) + +When Batch Changes runs locally, `src` resolves and downloads the target repositories, then runs each batch-spec step sequentially in a Docker container with the repository mounted as its workspace. `src` collects the container logs and resulting diffs, then sends the changeset specifications and batch-spec result to Sourcegraph. + ## Overview `src batch apply` and `src batch preview` execute a batch spec the same way by following these steps: diff --git a/docs/batch-changes/server-side.mdx b/docs/batch-changes/server-side.mdx index e4efadc14..c598f2304 100644 --- a/docs/batch-changes/server-side.mdx +++ b/docs/batch-changes/server-side.mdx @@ -11,7 +11,7 @@ By default, Batch Changes uses a command line interface in your local environment to [compute diffs](/batch-changes/how-src-executes-a-batch-spec) and create changesets. This can be impractical for creating batch changes affecting hundreds or thousands of repositories, with large numbers of workspaces, or if the batch change steps require CPU, memory, or disk resources that are unavailable locally. -Instead of computing Batch Changes locally using `src-cli`, you can offload this task to one or many remote server called an [executor](/self-hosted/executors/deploy-executors). Executors are also required to enable code navigation [auto-indexing](/code-navigation/auto-indexing). +Instead of computing Batch Changes locally using `src-cli`, you can offload this task to one or many remote servers called [executors](/admin/executors). Executors are also required to enable code navigation [auto-indexing](/code-navigation/auto-indexing). This allows to: @@ -21,16 +21,15 @@ This allows to: ## Setup -This is a one-time process. Once a site-admin of the Sourcegraph instance sets up executors and enables running batch changes server-side, all users of the Sourcegraph instance can get started with no additional setup required. +Sourcegraph Cloud includes Sourcegraph-managed executors by default, so Cloud customers using them do not need to deploy executor infrastructure. -Make sure that [executors are deployed and are online](/self-hosted/executors/deploy-executors). +If you operate a self-hosted Sourcegraph instance, [deploy executors](/self-hosted/executors) and confirm that they are online. This is a one-time process. Once a site administrator sets up executors and enables running Batch Changes server-side, users can get started without additional executor setup. ## Limitations -- Running batch changes server-side requires setting up executors. Executors are configured ready-to-use on Sourcegraph Cloud +- Self-hosted Sourcegraph instances require executor setup. Executors are ready to use on Sourcegraph Cloud. - Running batch changes server-side is limited to user namespaces - The newly introduced APIs for server-side are still experimental and will likely change -- Executors can only be deployed using Terraform (AWS or GCP) or using pre-built binaries (see [deploying executors](/self-hosted/executors/deploy-executors)). Running batch changes server-side has been tested to run a simple **45K changeset batch change**. Actual performance and setup requirements depend on the complexity of the batch change. diff --git a/docs/code-navigation/auto-indexing.mdx b/docs/code-navigation/auto-indexing.mdx index ce1bd23a7..25647f9c8 100644 --- a/docs/code-navigation/auto-indexing.mdx +++ b/docs/code-navigation/auto-indexing.mdx @@ -21,7 +21,7 @@ The following docs explains how to turn on [auto-indexing](/code-navigation/auto This step is only required if you are on self-hosted Sourcegraph. -First, [deploy the executor service](/self-hosted/executors/deploy-executors) targeting your Sourcegraph instance. This will provide the necessary compute resources that clone the target Git repository, securely analyze the code to produce a code graph data index, then upload that index to your Sourcegraph instance for processing. +First, [deploy the executor service](/self-hosted/executors) targeting your Sourcegraph instance. This will provide the necessary compute resources that clone the target Git repository, securely analyze the code to produce a code graph data index, then upload that index to your Sourcegraph instance for processing. ### Enable index job scheduling diff --git a/docs/self-hosted/executors/custom-certificates.mdx b/docs/self-hosted/executors/custom-certificates.mdx new file mode 100644 index 000000000..552f7fcec --- /dev/null +++ b/docs/self-hosted/executors/custom-certificates.mdx @@ -0,0 +1,108 @@ +# Use custom certificates with executors + +By default, executors will search for certificates in the following files and directories: + +| Directory or file | Distribution | +| --------------------------------------------------- | ------------------------- | +| `/etc/ssl/certs/ca-certificates.crt` | Debian/Ubuntu/Gentoo etc. | +| `/etc/pki/tls/certs/ca-bundle.crt` | Fedora/RHEL 6 | +| `/etc/ssl/ca-bundle.pem` | OpenSUSE | +| `/etc/pki/tls/cacert.pem` | OpenELEC | +| `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` | CentOS/RHEL 7 | +| `/etc/ssl/cert.pem` | Alpine Linux | +| `/etc/ssl/certs` | SLES10/SLES11 | +| `/etc/pki/tls/certs` | Fedora/RHEL | +| `/system/etc/security/cacerts` | Android | + +If your environment makes use of custom certificates, you can add them to one of these locations in order for executors to pick them up. + +## Add certificates to a binary deployment + +> NOTE: see the [troubleshooting guide](/self-hosted/executors/executors-troubleshooting#connecting-to-cloud-provider-executor-instances) for instructions on how to connect to cloud provider VMs. + +After successfully [deploying binaries](/self-hosted/executors/deploy-executors-binary), follow these steps: + +1. Copy your certificates to `/etc/ssl/certs`. +1. If you are using systemd, run `systemctl restart executor`. If not, proceed to the next step. +1. Run `executor run` on the VM in order to restart the executor service. + +### Add certificates with Firecracker + +When running executors with the [firecracker runtime](/self-hosted/executors/firecracker), custom certificates need to be added in +the container that is running within the Firecracker VM. To add custom certificates, you must create a new Docker image +that contains the certificates. For example, + +```dockerfile +FROM upstream:tag + +# Copy the certificates into the container +COPY customcert.crt /usr/local/share/ca-certificates/customcert.crt +# Update the certificate store +RUN chmod 644 /usr/local/share/ca-certificates/customcert.crt && update-ca-certificates +# ... +``` + +#### Code navigation + +Once the custom image is built, you can configure the executor to use it by setting +the `codeIntelAutoIndexing.indexerMap` to use the custom image. For example, + +```json +"codeIntelAutoIndexing.indexerMap": { + "go": "myregistry.company.com/scip-go:custom" +} +``` + +## Add certificates to a Kubernetes deployment using manifests + +First, add the certificate data as a secret in your preferred namespace: + +```shell +SECRET_NAME=custom-certs +CERT_PATH=/path/to/cert.pem +kubectl create secret generic $SECRET_NAME --from-file=customcert.crt=$CERT_PATH +``` + +Or as a declarative manifest: + +```yaml +apiVersion: v1 +kind: Secret +metadata: + name: custom-certs +data: + customcert.crt: $(base64 -i /path/to/cert.pem) +type: Opaque +``` + +Next, mount the secret in the executor deployment. Add the following snippet to `spec.template.spec.volumes` of each executor deployment: + +```yaml +- name: custom-certs + secret: + secretName: custom-certs +``` + +Also add this snippet to `spec.template.spec.containers.volumeMounts` of each executor deployment (specifically, the executor container, in case you inject any sidecars): + +```yaml +- mountPath: /etc/ssl/certs + name: custom-certs + readOnly: true +``` + +Next, apply the updated YAML manifests. Once the executors have rolled out, they should be picking up your custom certificates. + +## Add certificates to a Kubernetes deployment using Helm + +You may follow the same instructions for the manifest deployment to set custom certificates. + +## Add certificates to a Docker Compose deployment + +First, ensure that the certificate file is present on the host machine. Next, add the volume to the [executor compose file](https://sourcegraph.com/github.com/sourcegraph/deploy-sourcegraph-docker/-/blob/docker-compose/executors/executor.docker-compose.yaml?L26-30): + +```yaml +- '/path/to/certs:/etc/ssl/certs' +``` + +Next, restart the deployment with `docker-compose down` and `docker-compose up -d`. diff --git a/docs/self-hosted/executors/deploy-executors-binary-offline.mdx b/docs/self-hosted/executors/deploy-executors-binary-offline.mdx index bc8bfd4cf..03b9c06d7 100644 --- a/docs/self-hosted/executors/deploy-executors-binary-offline.mdx +++ b/docs/self-hosted/executors/deploy-executors-binary-offline.mdx @@ -2,6 +2,22 @@ When running in an air-gap environment, the executor binary can be deployed with this guide. +## How it works + +An offline deployment uses the same executor binary and job runtime as a standard [Linux binary deployment](/self-hosted/executors/deploy-executors-binary#how-it-works). Download the executor and its dependencies from a connected machine, transfer them into the offline environment, and configure internal container registries where required. + +**On this page** + +- [Initial dependencies](#initial-dependencies) +- [Install the binary](#install-binary) +- [Configure Docker](#configure-docker) +- [Environment variables](#environment-variables) +- [Batch Changes](#batch-changes) +- [Auto-indexing](#auto-indexing) +- [Firecracker setup](#firecracker-setup) +- [Validation](#validation) +- [Related configuration](#related-configuration) + ## Initial Dependencies Executors @@ -195,3 +211,8 @@ $ executor test-vm [--repo=github.com/sourcegraph/sourcegraph --revision=main] ``` This should succeed and print a command to use to attach to the guest VM. If it is able to spin up properly, that is a good indication that everything is set up correctly. + +## Related configuration + +- [Add custom certificates to a binary deployment](/self-hosted/executors/custom-certificates#add-certificates-to-a-binary-deployment) +- [Use private container registries](/self-hosted/executors/private-registries) diff --git a/docs/self-hosted/executors/deploy-executors-binary.mdx b/docs/self-hosted/executors/deploy-executors-binary.mdx index 934bdb49b..e754752ee 100644 --- a/docs/self-hosted/executors/deploy-executors-binary.mdx +++ b/docs/self-hosted/executors/deploy-executors-binary.mdx @@ -1,10 +1,22 @@ # Deploying Sourcegraph executors on linux machines +![Executor architecture when installed as a Linux binary](https://storage.googleapis.com/sourcegraph-assets/executor_binary_arch.png) + +## How it works + +The executor runs as a service on a Linux host and polls Sourcegraph for work. After dequeuing a job, it clones the target repository into a job workspace and runs each step sequentially in a Docker container. Batch Changes jobs that do not use native execution invoke `src`; code navigation and native Batch Changes jobs invoke Docker directly. The executor streams step logs and the final job status back to Sourcegraph. + +**On this page** + +- [Installation](#installation) +- [Upgrading executors](#upgrading-executors) +- [Related configuration](#related-configuration) + ## Installation > Note: See [offline installation guide](/self-hosted/executors/deploy-executors-binary-offline) for instructions on how to install executors in an air-gapped environment. -The following steps will guide you through the process of installing executors on a linux machine. +The following steps will guide you through the process of installing executors on a Linux machine. ### Dependencies @@ -30,7 +42,7 @@ If [Firecracker isolation will be used](/self-hosted/executors/firecracker): _(r ### **Step 0:** Confirm that virtualization is enabled (if using Firecracker) -KVM (virtualization) support is required for our [sandboxing model](/self-hosted/executors#sandboxing-model) with Firecracker. The following command checks whether virtualization is enabled on the machine (it should print something): +KVM (virtualization) support is required for [Firecracker isolation](/self-hosted/executors/firecracker). The following command checks whether virtualization is enabled on the machine (it should print something): ```bash $ lscpu | grep Virtualization @@ -109,7 +121,7 @@ executor validate If any issues are found, correct them before proceeding. -If you use our [sandboxing model](/self-hosted/executors#sandboxing-model) with Firecracker _(recommended)_, you can also verify that the executor is able to spin up the isolation VMs properly. For that, use the following command: +If you use [Firecracker isolation](/self-hosted/executors/firecracker) _(recommended)_, you can also verify that the executor is able to spin up the isolation VMs properly. For that, use the following command: ```bash # Optionally provide a repo to clone into the VMs workspace, to verify that cloning works properly as well. @@ -226,3 +238,8 @@ systemctl status executor ``` Otherwise, run `executor run` directly. Your executor should start listening for jobs and be visible under **Site admin > Executors > Instances**. + +## Related configuration + +- [Add custom certificates to a binary deployment](/self-hosted/executors/custom-certificates#add-certificates-to-a-binary-deployment) +- [Use private container registries](/self-hosted/executors/private-registries) diff --git a/docs/self-hosted/executors/deploy-executors-dind.mdx b/docs/self-hosted/executors/deploy-executors-dind.mdx index 5e6b3dab8..c386eea77 100644 --- a/docs/self-hosted/executors/deploy-executors-dind.mdx +++ b/docs/self-hosted/executors/deploy-executors-dind.mdx @@ -13,11 +13,26 @@ Executors on Kubernetes require privileged access to a container runtime daemon in order to operate correctly. To ensure maximum capability across Kubernetes versions and container runtimes, a [Docker in Docker](https://www.docker.com/blog/docker-can-now-run-within-docker/) sidecar is deployed with each executor pod so that the executor does not have to access the host container runtime directly. +![Docker-in-Docker Kubernetes executor architecture](https://storage.googleapis.com/sourcegraph-assets/executor_kubernetes_dind_arch.png) + +### How it works + +The executor and a Docker-in-Docker daemon run as containers in the same Kubernetes pod. The executor polls Sourcegraph, dequeues a job, and clones its repository into a shared workspace. It then uses the sidecar Docker daemon to run each job step sequentially in a container. Logs and the final job status are streamed back to Sourcegraph. + +**On this page** + +- [Step-by-step guide](#step-by-step-guide) +- [Upgrading executors](#upgrading-executors) +- [Security considerations](#security-considerations) +- [Related configuration](#related-configuration) + ### Step-by-step Guide - Before deploying, ensure your Sourcegraph instance is configured to accept executor connections. - See [Executor authentication](/self-hosted/executors/deploy-executors#executor-authentication) for instructions on setting the required `executors.accessToken` in your site configuration. + Before deploying, ensure your Sourcegraph instance is configured to accept + executor connections. See [Setting up + executors](/admin/executors#setting-up-executors) for instructions on + setting the required `executors.accessToken` in your site configuration. Ensure you have the following tools installed: @@ -126,3 +141,9 @@ We strongly recommend applying the following controls when running docker-in-doc 3. **Restrict network access.** Apply Kubernetes [NetworkPolicies](https://kubernetes.io/docs/concepts/services-networking/network-policies/) and/or cloud firewall rules to the executor node pool to: - Block access to internal and private resources and services. - Block access to the cloud instance metadata service at `169.254.169.254` (AWS / GCP). + +## Related configuration + +- [Add custom certificates using Kubernetes manifests](/self-hosted/executors/custom-certificates#add-certificates-to-a-kubernetes-deployment-using-manifests) +- [Add custom certificates using Helm](/self-hosted/executors/custom-certificates#add-certificates-to-a-kubernetes-deployment-using-helm) +- [Use private container registries](/self-hosted/executors/private-registries) diff --git a/docs/self-hosted/executors/deploy-executors-docker.mdx b/docs/self-hosted/executors/deploy-executors-docker.mdx index 2d8526822..1e3bac888 100644 --- a/docs/self-hosted/executors/deploy-executors-docker.mdx +++ b/docs/self-hosted/executors/deploy-executors-docker.mdx @@ -6,6 +6,20 @@ A [docker-compose file](https://github.com/sourcegraph/deploy-sourcegraph-docker/blob/master/docker-compose/executors/executor.docker-compose.yaml) is provided to deploy executors standlone, or alongside your existing Sourcegraph deployment. +![Executor architecture when deployed with Docker](https://storage.googleapis.com/sourcegraph-assets/executor_docker_arch.png) + +## How it works + +The executor service itself runs in a Docker container and polls Sourcegraph for work. After dequeuing a job, it clones the target repository into a workspace and uses the Docker daemon to run each job step sequentially in another container. Batch Changes jobs that do not use native execution invoke `src`; code navigation and native Batch Changes jobs invoke Docker directly. Logs and the final job status are streamed back to Sourcegraph. + +**On this page** + +- [Requirements](#requirements) +- [Deployment](#deployment) +- [Upgrading executors](#upgrading-executors) +- [Related configuration](#related-configuration) +- [Security note](#note) + ## Requirements Privileged containers are required to run executors in docker-compose. This is because executors require access to the docker daemon running on the host. @@ -64,8 +78,13 @@ Upgrading Docker Compose executors requires updating the executor image tag in t 4. Confirm the executor is online and reporting the expected version under **Site admin > Executors > Instances**. +## Related configuration + +- [Add custom certificates to a Docker Compose deployment](/self-hosted/executors/custom-certificates#add-certificates-to-a-docker-compose-deployment) +- [Use private container registries](/self-hosted/executors/private-registries) + ## Note -Executors deployed via docker-compose do not use [Firecracker](/admin/executors/#how-it-works), meaning they require [privileged access](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) to the docker daemon running on the host. +Executors deployed via Docker Compose do not use [Firecracker](/self-hosted/executors/firecracker), meaning they require [privileged access](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) to the Docker daemon running on the host. If you have security concerns, consider deploying via [terraform](/self-hosted/executors/deploy-executors-terraform) or [installing the binary](/self-hosted/executors/deploy-executors-binary) directly. diff --git a/docs/self-hosted/executors/deploy-executors-kubernetes.mdx b/docs/self-hosted/executors/deploy-executors-kubernetes.mdx index 1ac0720b4..07d42edb1 100644 --- a/docs/self-hosted/executors/deploy-executors-kubernetes.mdx +++ b/docs/self-hosted/executors/deploy-executors-kubernetes.mdx @@ -6,7 +6,21 @@ [Linux binary](/self-hosted/executors/deploy-executors-binary) for better long-term support. -The native Kubernetes Executors have a master Executor pod that schedules worker pods via the Kubernetes API. The master Executor pod manages the lifecycle of jobs, while the worker pods process the actual [batch change](/batch-changes/server-side) or [precise auto indexing](/code-navigation/auto-indexing) job specs. For more details, see [how it works](/admin/executors#native-kubernetes). +The native Kubernetes Executors have a master Executor pod that schedules worker pods via the Kubernetes API. The master Executor pod manages the lifecycle of jobs, while the worker pods process the actual [batch change](/batch-changes/server-side) or [precise auto indexing](/code-navigation/auto-indexing) job specs. Review the [deployment method comparison](/self-hosted/executors#choose-a-deployment-method) for its isolation and support status. + +![Native Kubernetes executor architecture](https://storage.googleapis.com/sourcegraph-assets/executor_kubernetes_native_arch.png) + +## How it works + +The executor runs as a pod and polls Sourcegraph for work. After dequeuing a job, it clones the target repository into a shared workspace and asks the Kubernetes API to create a job pod. Each step runs sequentially as an init container in that pod and shares the workspace through a volume. The executor streams the step logs and final job status back to Sourcegraph. + +**On this page** + +- [Requirements](#requirements) +- [Deployment](#deployment) +- [Upgrading executors](#upgrading-executors) +- [Additional notes](#additional-notes) +- [Related configuration](#related-configuration) ## Requirements @@ -129,7 +143,7 @@ Upgrading native Kubernetes executors follows the same pattern as the initial de ### Firecracker -Executors deployed on Kubernetes do not use [Firecracker](/admin/executors/#how-it-works). +Executors deployed on Kubernetes do not use [Firecracker](/self-hosted/executors/firecracker). If you have security concerns, consider deploying via [terraform](/self-hosted/executors/deploy-executors-terraform) or [installing the binary](/self-hosted/executors/deploy-executors-binary) directly. @@ -179,3 +193,9 @@ The Executor Docker image is available on Docker Hub at [`sourcegraph/executor-k ### Example Configuration YAML For an example of how to configure the Executor in Kubernetes, see the [`sourcegraph-executor-k8s` Helm chart values reference](https://github.com/sourcegraph/deploy-sourcegraph-helm/blob/main/charts/sourcegraph-executor/k8s/values.yaml). + +## Related configuration + +- [Add custom certificates using Kubernetes manifests](/self-hosted/executors/custom-certificates#add-certificates-to-a-kubernetes-deployment-using-manifests) +- [Add custom certificates using Helm](/self-hosted/executors/custom-certificates#add-certificates-to-a-kubernetes-deployment-using-helm) +- [Use private container registries](/self-hosted/executors/private-registries) diff --git a/docs/self-hosted/executors/deploy-executors-terraform-aws.mdx b/docs/self-hosted/executors/deploy-executors-terraform-aws.mdx index eeb50d724..5c070cbe8 100644 --- a/docs/self-hosted/executors/deploy-executors-terraform-aws.mdx +++ b/docs/self-hosted/executors/deploy-executors-terraform-aws.mdx @@ -5,6 +5,21 @@ provision machines running executors on AWS. See also: [Deploying on Google Cloud](/self-hosted/executors/deploy-executors-terraform-gcp) +## How it works + +The Terraform module provisions the AWS networking, Docker registry mirror, launch template, and optional autoscaling resources needed by executor instances. Each instance runs the executor binary, polls Sourcegraph for jobs, and processes them using [Firecracker isolation](/self-hosted/executors/firecracker) by default. The underlying job flow is the same as a [Linux binary deployment](/self-hosted/executors/deploy-executors-binary). + +**On this page** + +- [Basic definition](#basic-definition) +- [Terraform version](#terraform-version) +- [Permissions](#permissions) +- [Supported regions](#supported-regions) +- [Examples](#examples) +- [Step-by-step guide](#step-by-step-guide) +- [Auto-scaling](#auto-scaling) +- [Upgrading executors](#upgrading-executors) + ## Basic Definition The following is the minimum required definition to deploy an executor on AWS. @@ -46,7 +61,7 @@ module "executors" { | `private_networking` | If `true`, the executors and Docker registry mirror will live in a private subnet and communicate with the internet through a NAT Gateway. Defaults to `false`. See the [Private Single Executor](#private-single-executor) example. | | `randomize_resource_names` | Use randomized names for resources. Defaults to `false`. Existing resources are updated in-place when enabled. | | `permissions_boundary_arn` | The ARN of an IAM policy to use as the [permissions boundary](https://docs.aws.amazon.com/IAM/latest/UserGuide/access_policies_boundaries.html) for IAM roles and users created by the module. Optional. | -| `private_ca_cert_path` | Path to a private CA certificate file. Use this when executors need to communicate with a Sourcegraph instance that uses a certificate signed by a private/internal CA. Optional. | +| `private_ca_cert_path` | Path to a private CA certificate file. Use this when executors need to communicate with a Sourcegraph instance that uses a certificate signed by a private/internal CA. See [Use custom certificates with executors](/self-hosted/executors/custom-certificates). Optional. | See the [AWS Terraform module variables](https://github.com/sourcegraph/terraform-aws-executors/blob/main/modules/executors/variables.tf) for additional configurations. @@ -222,7 +237,7 @@ ubuntu@ip-10-0-1-42:~$ curl > NOTE: Auto scaling is currently not supported > when [downloading and running executor binaries yourself](/self-hosted/executors/deploy-executors-binary), -> and on managed instances when using self-hosted executors, since it requires deployment adjustments. +> and on Sourcegraph Cloud when using customer-managed executors, since it requires deployment adjustments. Auto-scaling of executor instances can help to increase concurrency of jobs, without paying for unused resources. With auto-scaling, you can scale down to 0 instances when no workload exist and scale up as far as you like and your cloud provider can support. Auto-scaling needs to be configured separately. @@ -306,7 +321,7 @@ Next, you can test whether the number of executors rises and shrinks as load spi ## Upgrading executors Upgrading executors is relatively uninvolved. Simply follow the instructions below. -Also, check the [changelog](https://sourcegraph.com/changelog) for any Executors related breaking changes or new features or flags that you might want to configure. See [Executors maintenance](/self-hosted/executors/deploy-executors#Maintaining-and-upgrading-executors) for version compatibility. +Also, check the [changelog](https://sourcegraph.com/changelog) for any executor-related breaking changes, features, or configuration flags before [upgrading executors](#upgrading-executors). ### **Step 1:** Update the source version of the terraform modules diff --git a/docs/self-hosted/executors/deploy-executors-terraform-gcp.mdx b/docs/self-hosted/executors/deploy-executors-terraform-gcp.mdx index bd2dc26e2..65d68669c 100644 --- a/docs/self-hosted/executors/deploy-executors-terraform-gcp.mdx +++ b/docs/self-hosted/executors/deploy-executors-terraform-gcp.mdx @@ -5,6 +5,21 @@ provision machines running executors on Google Cloud. See also: [Deploying on AWS](/self-hosted/executors/deploy-executors-terraform-aws) +## How it works + +The Terraform module provisions the Google Cloud networking, Docker registry mirror, instance template, and optional autoscaling resources needed by executor instances. Each instance runs the executor binary, polls Sourcegraph for jobs, and processes them using [Firecracker isolation](/self-hosted/executors/firecracker) by default. The underlying job flow is the same as a [Linux binary deployment](/self-hosted/executors/deploy-executors-binary). + +**On this page** + +- [Basic definition](#basic-definition) +- [Terraform version](#terraform-version) +- [Permissions](#permissions) +- [Supported regions](#supported-regions) +- [Examples](#examples) +- [Step-by-step guide](#step-by-step-guide) +- [Auto-scaling](#auto-scaling) +- [Upgrading executors](#upgrading-executors) + ## Basic Definition The following is the minimum required definition to deploy an executor on Google Cloud. @@ -47,7 +62,7 @@ module "executors" { | `executor_use_firecracker` | Whether to use [Firecracker](/self-hosted/executors/firecracker) sandboxing for job execution. Requires nested virtualization support. Defaults to `true`. | | `private_networking` | If `true`, the executors and Docker registry mirror will live in a private subnet and communicate with the internet through Cloud NAT. Defaults to `false`. See the [Private Single Executor](#private-single-executor) example. | | `randomize_resource_names` | Use randomized names for resources. Defaults to `false`. Enabling this on existing deployments will **recreate** executor resources. | -| `private_ca_cert_path` | Path to a private CA certificate file. Use this when executors need to communicate with a Sourcegraph instance that uses a certificate signed by a private/internal CA. Optional. | +| `private_ca_cert_path` | Path to a private CA certificate file. Use this when executors need to communicate with a Sourcegraph instance that uses a certificate signed by a private/internal CA. See [Use custom certificates with executors](/self-hosted/executors/custom-certificates). Optional. | See the [Google Cloud Terraform module variables](https://github.com/sourcegraph/terraform-google-executors/blob/main/modules/executors/variables.tf) for additional configurations. @@ -243,7 +258,7 @@ you@sourcegraph-executor-h0rv:~$ curl > NOTE: Auto scaling is currently not supported > when [downloading and running executor binaries yourself](/self-hosted/executors/deploy-executors-binary), -> and on managed instances when using self-hosted executors, since it requires deployment adjustments. +> and on Sourcegraph Cloud when using customer-managed executors, since it requires deployment adjustments. Auto-scaling of executor instances can help to increase concurrency of jobs, without paying for unused resources. With auto-scaling, you can scale down to 0 instances when no workload exist and scale up as far as you like and your cloud provider can support. Auto-scaling needs to be configured separately. @@ -325,7 +340,7 @@ Next, you can test whether the number of executors rises and shrinks as load spi ## Upgrading executors Upgrading executors is relatively uninvolved. Simply follow the instructions below. -Also, check the [changelog](https://sourcegraph.com/changelog) for any Executors related breaking changes or new features or flags that you might want to configure. See [Executors maintenance](/self-hosted/executors/deploy-executors#Maintaining-and-upgrading-executors) for version compatibility. +Also, check the [changelog](https://sourcegraph.com/changelog) for any executor-related breaking changes, features, or configuration flags before [upgrading executors](#upgrading-executors). ### **Step 1:** Update the source version of the terraform modules diff --git a/docs/self-hosted/executors/deploy-executors.mdx b/docs/self-hosted/executors/deploy-executors.mdx deleted file mode 100644 index 352e00a6f..000000000 --- a/docs/self-hosted/executors/deploy-executors.mdx +++ /dev/null @@ -1,322 +0,0 @@ -# Deploying Sourcegraph executors - -Executors can be deployed in a variety of manners. The supported deployment options are: - -- [Linux Binary Service](/self-hosted/executors/deploy-executors-binary) ([Firecracker](./firecracker) compatible) -- Terraform ([Firecracker](./firecracker) compatible): [AWS](/self-hosted/executors/deploy-executors-terraform-aws) | [Google Cloud](/self-hosted/executors/deploy-executors-terraform-gcp) -- [Native Kubernetes](/self-hosted/executors/deploy-executors-kubernetes) *(Beta — not recommended for production)* -- [Docker-in-Docker on Kubernetes](/self-hosted/executors/deploy-executors-dind) *(Beta — not recommended for production)* -- [Docker-Compose](/self-hosted/executors/deploy-executors-docker) - -See [deciding which executor deployment method to use ](../executors#deciding-which-executor-deployment-method-to-use) for more information on these different deployment options. - - - Executors are available with no additional setup required on Sourcegraph - Cloud. - - -### Resource recommendations - -It is recommended to set the resources based on the number of jobs an instance will process in parallel. - -A single job should have the following resources available. - -- **CPU:** 4 -- **Memory:** 12GB -- **Disk:** 20GB - -So, if you expect an Executor instance to process up to 4 jobs in parallel, the recommended resources for the machine are - -- **CPU:** 16 -- **Memory:** 48GB -- **Disk:** 80GB - -The above recommended resources can be changed to fit your constraints. See below for configuring resources for a Job. - - - The smallest machine type on AWS that can support Executors with - Firecracker is `c5n.metal` (72 vCPU and 192GB of Memory), but concurrency - can be turned up for the additional cost. - - -#### Job configuration - -The maximum number of Jobs an Executor instance can run in parallel is configured by the Environment Variable `EXECUTOR_MAXIMUM_NUM_JOBS`. - -The CPU and Memory usage of an individual Job is configured by the Environment Variables `EXECUTOR_JOB_NUM_CPUS` -and `EXECUTOR_JOB_MEMORY`. - -See [executor configuration](/self-hosted/executors/executors-config) for a full list of configuration options. - - - Changing CPU and Memory for jobs will affect the overall requirements - for an Executor instance. - - -#### AWS - -It is recommended to add the following **Disk** configuration in AWS. - -- **IOPS:** Equal to the Disk Size (so if **Disk** is 100GB, then IOPS is 100) -- **Throughput:** 125MiB/s - -### Supported infrastructures - -- **Operating System:** Linux-based -- **Architecture:** AMD64 - -#### Firecracker requirements - -To run Executors with Firecracker enabled requires the machine to support [Kernel-based Virtual Machine](https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine). -See [deploying Executors binary](/self-hosted/executors/deploy-executors-binary) for additional information on configuring Linux Machines. - -#### Cloud providers - -Machines on Cloud Providers have additional constraints for use with firecracker. - -- **AWS:** machine type must be - a [metal instance (`.metal`)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html) -- **GCP:** the instance - must [enable nested virtualization](https://cloud.google.com/compute/docs/instances/nested-virtualization/enabling) - -## Executor authentication - -Executors must be run separately from your Sourcegraph instance. - -Since they must still be able to reach the Sourcegraph instance in order to dequeue and perform work, requests between the Sourcegraph instance and the executors are authenticated via a shared secret. - -Before starting any executors, generate an arbitrary secret string (with at least 20 characters) and [set it as the `executors.accessToken` key in your Sourcegraph instance's site-config](/admin/config/site-config#view-and-edit-site-configuration). The `EXECUTOR_FRONTEND_PASSWORD` environment variable on the executor instance will need to be configured with the same secret string. - -## Executor installation - -Once the shared secret is set in Sourcegraph, you can start setting up executors that can use that access token to talk to the Sourcegraph instance. - -### Supported installation types - - - - - - - - - - -## Validating executor deployments - -If executor instances boot correctly and can authenticate with the Sourcegraph frontend, they will show up in the **Executors** page under **Site admin > Maintenance**. - -![Executor list in UI](https://storage.googleapis.com/sourcegraph-assets/docs/images/code-intelligence/sg-3.34/executor-ui-test.png) - -## Using private registries - -If you want to use docker images stored in a private registry that requires authentication, follow this section to configure it. - -Depending on the executor runtime that is being used, different options exist for provisioning access to private container registries: - -- Through a special secret called `DOCKER_AUTH_CONFIG`, set in [executor secrets](/admin/executors/executor-secrets) in Sourcegraph. -- Through the `EXECUTOR_DOCKER_AUTH_CONFIG` environment variable (also available as a variable in the terraform modules for executors). -- Through the [`config.json` file in `~/.docker`](https://docs.docker.com/engine/reference/commandline/login/). **If using executors with firecracker enabled (recommended) this option is not available.** - -When multiple of the above options are combined, executors will use them in the following order: - -- If a `DOCKER_AUTH_CONFIG` executor secret is configured, that will be preferred. That is so that users can overwrite the credentials being used in their user-settings. This is the only option available in Sourcegraph Cloud. -- If the `EXECUTOR_DOCKER_AUTH_CONFIG` environment variable is set, this will be used as the next option. -- Finally, if neither of the above are set, executors will fall back to the `config.json` file in the user home directory of the user that is owning the executor process. NOTE: This is not available in the firecracker runtime, as the rootfs is not shared with the host. - -The docker CLI supports three ways to use credentials: - -- Using static credentials -- Using [credential helpers](https://docs.docker.com/engine/reference/commandline/login/#credential-helpers) -- Using [credential stores](https://docs.docker.com/engine/reference/commandline/login/#credentials-store) - -Credential helpers and credential stores are only available for use with the `config.json` configuration option, as they require additional infrastructural changes. Thus, those options are not available on Sourcegraph Cloud. - -### Using static credentials - -The `EXECUTOR_DOCKER_AUTH_CONFIG` environment variable and the `DOCKER_AUTH_CONFIG` secret expect a docker config with only the necessary properties set for configuring authentication. -The format of this config supports multiple registries to be configured and looks like this: - -```json -{ - "auths": { - "myregistry.example.com[:port]": { - "auth": "base64(username:password)" - }, - "myregistry2.example.com[:port]": { - "auth": "base64(username:password)" - } - } -} -``` - -You can either create this config yourself by hand, or let docker do it for you by running: - -```bash -TMP_FILE="$(mktemp -d)" bash -c 'echo "" | docker --config "${TMP_FILE}" login --username "" --password-stdin "" && cat "${TMP_FILE}/config.json" && rm -rf "${TMP_FILE}"' -``` - -> NOTE: This doesn't work on Docker for Mac if "Securely store Docker logins in macOS keychain" is enabled, as it would store it in the credentials store instead. - -You can also run the following: - -```bash -echo -n "username:password" | base64 -``` - -and then paste the result of that into a JSON string like this: - -```json -{ - "auths": { - "myregistry.example.com[:port]": { - "auth": "" - } - } -} -``` - -For Google Container Registry, [follow this guide](https://cloud.google.com/container-registry/docs/advanced-authentication#json-key) for how to obtain long-lived static credentials. - -### Configuring the auth config for use in executors - -Now that the config has been obtained, it can be used for the `EXECUTOR_DOCKER_AUTH_CONFIG` environment variable (and terraform variable `docker_auth_config`) or you can create an [executor secret](/admin/executors/executor-secrets#creating-a-new-secret) called `DOCKER_AUTH_CONFIG`. Global executor secrets will be available to every execution, while user and organization level executor secrets will only be available to the namespaces executions. - -## Using custom certificates with executors - -By default, executors will search for certificates in the following files and directories: - -| Directory or file | Distribution | -| --------------------------------------------------- | ------------------------- | -| `/etc/ssl/certs/ca-certificates.crt` | Debian/Ubuntu/Gentoo etc. | -| `/etc/pki/tls/certs/ca-bundle.crt` | Fedora/RHEL 6 | -| `/etc/ssl/ca-bundle.pem` | OpenSUSE | -| `/etc/pki/tls/cacert.pem` | OpenELEC | -| `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` | CentOS/RHEL 7 | -| `/etc/ssl/cert.pem` | Alpine Linux | -| `/etc/ssl/certs` | SLES10/SLES11 | -| `/etc/pki/tls/certs` | Fedora/RHEL | -| `/system/etc/security/cacerts` | Android | - -If your environment makes use of custom certificates, you can add them to one of these locations in order for executors to pick them up. - -### Adding certificates to a binary deployment - -> NOTE: see the [troubleshooting guide](/self-hosted/executors/executors-troubleshooting#connecting-to-cloud-provider-executor-instances) for instructions on how to connect to cloud provider VMs. - -After successfully [deploying binaries](/self-hosted/executors/deploy-executors-binary), follow these steps: - -1. Copy your certificates to `/etc/ssl/certs`. -1. If you are using systemd, run `systemctl restart executor`. If not, proceed to the next step. -1. Run `executor run` on the VM in order to restart the executor service. - -#### Adding certificates with Firecracker - -When running executors with the [firecracker runtime](/self-hosted/executors/firecracker), custom certificates need to be added in -the container that is running within the Firecracker VM. To add custom certificates, you must create a new Docker image -that contains the certificates. For example, - -```dockerfile -FROM upstream:tag - -# Copy the certificates into the container -COPY customcert.crt /usr/local/share/ca-certificates/customcert.crt -# Update the certificate store -RUN chmod 644 /usr/local/share/ca-certificates/customcert.crt && update-ca-certificates -# ... -``` - -##### Code Intel - -Once the custom image is built, you can configure the executor to use it by setting -the `codeIntelAutoIndexing.indexerMap` to use the custom image. For example, - -```json -"codeIntelAutoIndexing.indexerMap": { - "go": "myregistry.company.com/scip-go:custom" -} -``` - -### Adding certificates to a Kubernetes deployment using manifests - -First, add the certificate data as a secret in your preferred namespace: - -```shell -SECRET_NAME=custom-certs -CERT_PATH=/path/to/cert.pem -kubectl create secret generic $SECRET_NAME --from-file=customcert.crt=$CERT_PATH -``` - -Or as a declarative manifest: - -```yaml -apiVersion: v1 -kind: Secret -metadata: - name: custom-certs -data: - customcert.crt: $(base64 -i /path/to/cert.pem) -type: Opaque -``` - -Next, mount the secret in the executor deployment. Add the following snippet to `spec.template.spec.volumes` of each executor deployment: - -```yaml -- name: custom-certs - secret: - secretName: custom-certs -``` - -Also add this snippet to `spec.template.spec.containers.volumeMounts` of each executor deployment (specifically, the executor container, in case you inject any sidecars): - -```yaml -- mountPath: /etc/ssl/certs - name: custom-certs - readOnly: true -``` - -Next, apply the updated YAML manifests. Once the executors have rolled out, they should be picking up your custom certificates. - -### Adding certificates to a Kubernetes deployment using Helm - -You may follow the same instructions for the manifest deployment to set custom certificates. - -### Adding certificates to a Docker Compose deployment - -First, ensure that the certificate file is present on the host machine. Next, add the volume to the [executor compose file](https://sourcegraph.com/github.com/sourcegraph/deploy-sourcegraph-docker/-/blob/docker-compose/executors/executor.docker-compose.yaml?L26-30): - -```yaml -- '/path/to/certs:/etc/ssl/certs' -``` - -Next, restart the deployment with `docker-compose down` and `docker-compose up -d`. diff --git a/docs/self-hosted/executors/executors-troubleshooting.mdx b/docs/self-hosted/executors/executors-troubleshooting.mdx index d6a0e790c..f7a46b1e2 100644 --- a/docs/self-hosted/executors/executors-troubleshooting.mdx +++ b/docs/self-hosted/executors/executors-troubleshooting.mdx @@ -188,7 +188,7 @@ This section lists some common mistakes with environment variables. Some of thes ## Verify Firecracker support -The VM instance must [support KVM](/self-hosted/executors/deploy-executors#firecracker-requirements). In effect, this means the instance must meet certain requirements depending on the Cloud provider in use. +The VM instance must [support KVM](/self-hosted/executors/firecracker#how-to-use). In effect, this means the instance must meet certain requirements depending on the cloud provider in use. ### GCP diff --git a/docs/self-hosted/executors/firecracker.mdx b/docs/self-hosted/executors/firecracker.mdx index db035d529..f43ae46ca 100644 --- a/docs/self-hosted/executors/firecracker.mdx +++ b/docs/self-hosted/executors/firecracker.mdx @@ -1,12 +1,23 @@ # Firecracker +## How it works + [Executors](/admin/executors/), by design, are services that run arbitrary code supplied by a user. The executor jobs produced by precise code intelligence [auto-indexing](/code-navigation/auto-indexing) and [server-side batch changes](/batch-changes/server-side) are built to invoke _templated_ execution plans, where some parts of execution may invoke code configured by a Sourcegraph administrator or user. Generating a precise index requires invoking an indexer for that language. Batch changes are configured to run arbitrary tooling over the contents of a repository. Because Sourcegraph has access to your code and credentials to external tools, we've designed executors to be able to run separately from the Sourcegraph instance (on a raw compute node) with the minimum API surface area and user code exposed to the job required to meet its objective. This effectively reduces the blast radius of a misconfiguration or insecure configuration. Jobs handled by the executor are defined as a series of Docker (or Docker-like) image invocations sharing the same filesystem workspace initialized with the contents of the repository being indexed or modified. When the executor process is running on a raw compute node (specifically, not using the Kubernetes runner in-cluster), the executor can create a [Firecracker "MicroVM"](https://firecracker-microvm.github.io/) for every job and run the containers in a _completely isolated_ manner. The Docker (or Docker-like) daemon running in a MicroVM is isolated from the host, hence each job is also isolated from one another. -See [this architecture diagram](/self-hosted/executors#firecracker) detailing Firecracker isolation. +![Executor architecture with Firecracker isolation](https://storage.googleapis.com/sourcegraph-assets/executor_firecracker_arch.png) + +The executor still polls Sourcegraph, clones one repository into a job workspace, and runs the job steps sequentially. For each job, Ignite starts a dedicated Firecracker MicroVM and attaches the workspace to it. Docker containers run inside that MicroVM rather than directly on the executor host, and the executor streams their logs and final status back to Sourcegraph. + +**On this page** + +- [When to use Firecracker](#when-to-use) +- [How to use Firecracker](#how-to-use) +- [Known caveats](#known-caveats) +- [Related configuration](#related-configuration) ## When to use @@ -42,3 +53,8 @@ Executor can be fined tuned with the following environment variables, ## Known caveats We configure iptables to prevent Firecracker-isolated processes from talking on [Private IPv4 Addresses](https://en.wikipedia.org/wiki/Private_network#Private_IPv4_addresses) (providing network-level isolation). They can talk to DNS and Sourcegraph only, which prevents users from talking to a 10.x.x.x, 172.x.x.x, or 192.168.x.x range IP. + +## Related configuration + +- [Add custom certificates with Firecracker](/self-hosted/executors/custom-certificates#add-certificates-with-firecracker) +- [Use private container registries](/self-hosted/executors/private-registries) diff --git a/docs/self-hosted/executors/index.mdx b/docs/self-hosted/executors/index.mdx index 63a7d0572..2ced1350a 100644 --- a/docs/self-hosted/executors/index.mdx +++ b/docs/self-hosted/executors/index.mdx @@ -1,181 +1,122 @@ -# Executors +# Deploy executors - - Supported on [Enterprise](/pricing/plans/enterprise) plans. - Available via the Web app. - +Executors run resource-intensive or untrusted jobs for precise code navigation and Batch Changes. This guide helps self-hosted operators choose a deployment method and prepare the infrastructure that executors require. -Executors are Sourcegraph's solution for running untrusted code in a secure and controllable way. Executors provide a sandbox that can run resource-intensive or untrusted tasks on behalf of the Sourcegraph instance, such as: +For feature concepts, Sourcegraph configuration, connected instance management, and executor secrets, see [Administer executors](/admin/executors). -- [Automatically indexing a repository for precise code navigation](/code-navigation/auto-indexing) -- [Running batch changes](/batch-changes/server-side) +## Prepare to deploy executors -## Installation +Complete these prerequisites before deploying executor infrastructure: -To deploy executors for your Sourcegraph instance, follow our [executor deployment guide](executors/deploy-executors). +- **Confirm version compatibility.** Run the same major and minor version of the executor as your Sourcegraph Enterprise instance. +- **Configure authentication to your Sourcegraph instance.** Complete [Setting up executors](/admin/executors#setting-up-executors) to create the shared authentication secret. The guide for your deployment method explains how to provide that secret to the executor. +- **Confirm network connectivity.** Executor infrastructure must be able to reach the Sourcegraph frontend. +- **Provide a supported runtime.** Current executor deployments require a Linux AMD64 (`x86_64`) environment. +- **Plan executor capacity.** Size CPU, memory, and disk for the number and type of concurrent jobs. See [Plan executor capacity](/self-hosted/executors/resource-sizing). -## Why use executors? +## Choose a deployment method -Running untrusted code is a core requirement of features such as precise code navigation [auto-indexing](/code-navigation/auto-indexing), and [running batch changes server-side](/batch-changes/server-side). +The primary decision is whether jobs require a virtual-machine isolation boundary. -Auto-indexing jobs, in particular, require the invocation of arbitrary and untrusted code to support the resolution of project dependencies. Invocation of post-install hooks, use of insecure [package management tools](https://github.com/golang/go/issues/29230), and package manager proxy attacks can create opportunities in which an adversary can gain unlimited use of compute or exfiltrate data. The latter outcome is particularly dangerous for on-premise installations of Sourcegraph, which is the chosen option for companies wanting to maintain strict privacy of their code property. +- **Use Firecracker for the strongest supported job isolation.** Choose a Terraform deployment on AWS or Google Cloud, or install the Linux binary on compatible infrastructure. Firecracker requires KVM: AWS uses bare-metal instances, while Google Cloud supports nested virtualization. +- **Use Docker when KVM is unavailable and privileged container execution is acceptable.** Install the Linux binary in Docker mode or use Docker Compose. +- **For Kubernetes deployments, prefer Docker-in-Docker.** Docker-in-Docker is the preferred option when running executors in an existing Kubernetes cluster. It remains in beta and does not provide the same MicroVM isolation boundary as Firecracker. Native Kubernetes executors are being deprecated. -Instead of performing this work within the Sourcegraph instance, where code is available on disk and unprotected internal services are available over the local network, we move untrusted compute into a sandboxed environment, the _executor_, that has access only to the clone of a single repository on disk (its _workspace_) and to the public internet. - -## Sandboxing Model - -Executors can be deployed with [Firecracker](https://sourcegraph.com/github.com/firecracker-microvm/firecracker) isolation in accordance with our [sandboxing model](/admin/executors/#how-it-works) to isolate jobs from each other and the host. -This requires executors to be run on machines capable of running Linux KVM extensions. On the most popular cloud providers, this either means running executors on bare-metal machines (AWS) or machines capable of nested virtualization (GCP). - -Optionally, executors can be run without using KVM-based isolation, which is less secure but might be easier to run on common machines. - -## Deciding which executor deployment method to use - -Deciding how to deploy the executor depends on your use case. For users that wish to process their untrusted compute in the most secure manner, we recommend leveraging the [Firecracker](./executors/firecracker) isolation method. For users that have constraints around running nested virtualization, the following flowchart can help you decide which deployment option is best for your environment: - -![Executor Deployment Flowchart](https://storage.googleapis.com/sourcegraph-assets/executor_deployment_tree.png) - -> **Note:** The flowchart above shows Kubernetes-based deployment options. Both Native Kubernetes -> and Docker-in-Docker Kubernetes executors are currently in beta with known production reliability -> limitations. For production workloads, Terraform or Linux binary deployments are recommended -> even when nested virtualization is not available. - -## How it works - -Executor instances are capable of being deployed in a variety of ways. Each runtime varies in how jobs are executed. - -### Locally with src-cli - -![Executors architecture - local with src-cli](https://storage.googleapis.com/sourcegraph-assets/executor_src_local_arch.png) - -1. User runs the `src` (e.g. `src batch`) command from the command line. -2. `src` calls the Sourcegraph API to clone a repository. - 1. The repositories are written to a directory. -3. A Docker Container is created for each "step." - 1. The directory containing the repository is mounted to the container. - 2. "Steps" are ran in sequential order. -4. The container run a defined command against the repository. -5. Logs from the container are sent back to `src`. -6. At the end of processing all repositories, the result is sent to a Sourcegraph API. - 1. e.g. Batch Changes sends a `git diff` to a Sourcegraph API (and invokes other APIs). - -### Binary - -![Executors architecture - binary](https://storage.googleapis.com/sourcegraph-assets/executor_binary_arch.png) - -1. The executor binary is installed to a machine. - 1. Additional executables (e.g. Docker, `src`) are installed as well -2. The executor instances pulls for available Jobs from a Sourcegraph API -3. A user initiates a process that creates executor Jobs. -4. The executor instance "dequeues" a Job. -5. Executor calls the Sourcegraph API to clone a repository. - 1. The repositories are written to a directory. -6. A Docker Container is created for each "step." - 1. If the Job is `batches` (non-native execution), `src` is invoked - 2. Docker is invoked directly for other Jobs (`codeintel` and native execution `batches`) - 3. The directory containing the repository is mounted to the container. - 4. "Steps" are ran in sequential order. -7. The container run a defined command against the repository. -8. Logs from the container are sent back to the executor. -9. Logs are streamed from the executor to a Sourcegraph API -10. The executor calls a Sourcegraph API to that "complete" the Job. - -### Firecracker - -> NOTE: [What the heck is firecracker, anyway](/self-hosted/executors/firecracker)?? - -![Executors architecture - firecracker](https://storage.googleapis.com/sourcegraph-assets/executor_firecracker_arch.png) - -1. The executor binary is installed to a machine. - 1. Additional executables (e.g. Docker, `src`) are installed as well -2. The executor instances pulls for available Jobs from a Sourcegraph API -3. A user initiates a process that creates executor Jobs. -4. The executor instance "dequeues" a Job. -5. Executor calls the Sourcegraph API to clone a repository. - 1. The repositories are written to a directory. -6. `ignite` starts up a Docker container that spawns a single Firecracker VM within the Docker container. - 1. The directory containing the repository is mounted to the VM. -7. Docker Container is created in the Firecracker VM for each "step." - 1. If the Job is `batches` (non-native execution), `src` is invoked - 2. Docker is invoked directly for other Jobs (`codeintel` and native execution `batches`) - 3. "Steps" are ran in sequential order. -8. Within each Firecracker VM a single Docker container is created -9. The container run a defined command against the repository. -10. Logs from the container are sent back to the executor. -11. Logs are streamed from the executor to a Sourcegraph API -12. The executor calls a Sourcegraph API to that "complete" the Job. - -### Docker - -![Executors architecture - docker](https://storage.googleapis.com/sourcegraph-assets/executor_docker_arch.png) - -1. The executor image is started as a Docker container on a machine -2. The executor pulls for available Jobs from a Sourcegraph API -3. A user initiates a process that creates executor Jobs. -4. The executor instance "dequeues" a Job. -5. Executor calls the Sourcegraph API to clone a repository. - 1. The repositories are written to a directory. -6. A Docker Container is created for each "step." - 1. If the Job is `batches` (non-native execution), `src` is invoked - 2. Docker is invoked directly for other Jobs (`codeintel` and native execution `batches`) - 3. The directory containing the repository is mounted to the container. - 4. "Steps" are ran in sequential order. -7. The container run a defined command against the repository. -8. Logs from the container are sent back to the executor. -9. Logs are streamed from the executor to a Sourcegraph API -10. The executor calls a Sourcegraph API to that "complete" the Job. - -### Native Kubernetes - - - Native Kubernetes executors are in beta and are not recommended for production use. - See the [deployment guide](/self-hosted/executors/deploy-executors-kubernetes) for details - on known limitations. - - -![Executors architecture - native kubernetes](https://storage.googleapis.com/sourcegraph-assets/executor_kubernetes_native_arch.png) - -1. The executor image is started as a pod in a Kubernetes node -2. The executor pulls for available Jobs from a Sourcegraph API -3. A user initiates a process that creates executor Jobs. -4. The executor instance "dequeues" a Job. -5. Executor calls the Sourcegraph API to clone a repository. - 1. The repositories are written to a directory. -6. A Kubernetes Job is created for each "step." - 1. The directory containing the repository is mounted to the container. - 2. "Steps" are ran in sequential order. -7. The container run a defined command against the repository. -8. Logs from the container are sent back to the executor. -9. Logs are streamed from the executor to a Sourcegraph API -10. The executor calls a Sourcegraph API to that "complete" the Job. - -### Docker-in-Docker Kubernetes +![Flowchart for choosing an executor deployment method](https://storage.googleapis.com/sourcegraph-assets/executor_deployment_tree.png) - Docker-in-Docker Kubernetes executors are in beta and are not recommended for production use. - See the [deployment guide](/self-hosted/executors/deploy-executors-dind) for details on known - limitations. + Executors run untrusted repository code and build tooling. Deployment + methods without Firecracker do not provide a MicroVM boundary. Isolate + executor hosts or nodes from sensitive workloads, restrict access to private + networks and cloud metadata services, and use disposable compute where + possible. -![Executors architecture - docker in docker kubernetes](https://storage.googleapis.com/sourcegraph-assets/executor_kubernetes_dind_arch.png) - -1. The executor image is started as a container in Kubernetes Pod - 1. The dind image is started as a sidecar container in the same Kubernetes Pod -2. The executor pulls for available Jobs from a Sourcegraph API -3. A user initiates a process that creates executor Jobs. -4. The executor instance "dequeues" a Job. -5. Executor calls the Sourcegraph API to clone a repository. - 1. The repositories are written to a directory. -6. A Docker Container is created for each "step." - 1. If the Job is `batches` (non-native execution), `src` is invoked - 2. Docker is invoked directly for other Jobs (`codeintel` and native execution `batches`) - 3. The directory containing the repository is mounted to the container. - 4. "Steps" are ran in sequential order. -7. The container run a defined command against the repository. -8. Logs from the container are sent back to the executor. -9. Logs are streamed from the executor to a Sourcegraph API -10. The executor calls a Sourcegraph API to that "complete" the Job. - -## Troubleshooting - -Refer to the [Troubleshooting Executors](/self-hosted/executors/executors-troubleshooting) document for common debugging operations. +### Recommended deployment methods + + + + + + + + + +### Kubernetes deployment methods + + + + + + +## Validating executor deployments + +Follow the validation steps in the guide for your selected deployment method. After the executor starts, confirm that it is active and reporting the expected version under **Site admin > Maintenance > Executor instances**. + +![Executor instances listed in the Sourcegraph site administration UI](https://storage.googleapis.com/sourcegraph-assets/docs/images/code-intelligence/sg-3.34/executor-ui-test.png) + +## Configure and operate executors + + + + + + + + +For details about the strongest supported job-isolation boundary, see [Firecracker isolation and caveats](/self-hosted/executors/firecracker). diff --git a/docs/self-hosted/executors/private-registries.mdx b/docs/self-hosted/executors/private-registries.mdx new file mode 100644 index 000000000..74ae67a6b --- /dev/null +++ b/docs/self-hosted/executors/private-registries.mdx @@ -0,0 +1,73 @@ +# Use private container registries + +If you want to use docker images stored in a private registry that requires authentication, follow this section to configure it. + +Depending on the executor runtime that is being used, different options exist for provisioning access to private container registries: + +- Through a special secret called `DOCKER_AUTH_CONFIG`, set in [executor secrets](/admin/executors/executor-secrets) in Sourcegraph. +- Through the `EXECUTOR_DOCKER_AUTH_CONFIG` environment variable (also available as a variable in the terraform modules for executors). +- Through the [`config.json` file in `~/.docker`](https://docs.docker.com/engine/reference/commandline/login/). **If using executors with firecracker enabled (recommended) this option is not available.** + +When multiple of the above options are combined, executors will use them in the following order: + +- If a `DOCKER_AUTH_CONFIG` executor secret is configured, that will be preferred. That is so that users can overwrite the credentials being used in their user-settings. This is the only option available in Sourcegraph Cloud. +- If the `EXECUTOR_DOCKER_AUTH_CONFIG` environment variable is set, this will be used as the next option. +- Finally, if neither of the above are set, executors will fall back to the `config.json` file in the user home directory of the user that is owning the executor process. NOTE: This is not available in the firecracker runtime, as the rootfs is not shared with the host. + +The docker CLI supports three ways to use credentials: + +- Using static credentials +- Using [credential helpers](https://docs.docker.com/engine/reference/commandline/login/#credential-helpers) +- Using [credential stores](https://docs.docker.com/engine/reference/commandline/login/#credentials-store) + +Credential helpers and credential stores are only available for use with the `config.json` configuration option, as they require additional infrastructural changes. Thus, those options are not available on Sourcegraph Cloud. + +## Use static credentials + +The `EXECUTOR_DOCKER_AUTH_CONFIG` environment variable and the `DOCKER_AUTH_CONFIG` secret expect a docker config with only the necessary properties set for configuring authentication. +The format of this config supports multiple registries to be configured and looks like this: + +```json +{ + "auths": { + "myregistry.example.com[:port]": { + "auth": "base64(username:password)" + }, + "myregistry2.example.com[:port]": { + "auth": "base64(username:password)" + } + } +} +``` + +You can either create this config yourself by hand, or let docker do it for you by running: + +```bash +TMP_FILE="$(mktemp -d)" bash -c 'echo "" | docker --config "${TMP_FILE}" login --username "" --password-stdin "" && cat "${TMP_FILE}/config.json" && rm -rf "${TMP_FILE}"' +``` + +> NOTE: This doesn't work on Docker for Mac if "Securely store Docker logins in macOS keychain" is enabled, as it would store it in the credentials store instead. + +You can also run the following: + +```bash +echo -n "username:password" | base64 +``` + +and then paste the result of that into a JSON string like this: + +```json +{ + "auths": { + "myregistry.example.com[:port]": { + "auth": "" + } + } +} +``` + +For Google Container Registry, [follow this guide](https://cloud.google.com/container-registry/docs/advanced-authentication#json-key) for how to obtain long-lived static credentials. + +## Configure registry authentication for executors + +Now that the config has been obtained, it can be used for the `EXECUTOR_DOCKER_AUTH_CONFIG` environment variable (and terraform variable `docker_auth_config`) or you can create an [executor secret](/admin/executors/executor-secrets#creating-a-new-secret) called `DOCKER_AUTH_CONFIG`. Global executor secrets will be available to every execution, while user and organization level executor secrets will only be available to the namespaces executions. diff --git a/docs/self-hosted/executors/resource-sizing.mdx b/docs/self-hosted/executors/resource-sizing.mdx new file mode 100644 index 000000000..ff9ae180a --- /dev/null +++ b/docs/self-hosted/executors/resource-sizing.mdx @@ -0,0 +1,38 @@ +# Plan executor capacity + +Executor capacity depends on the number of jobs each executor processes concurrently and the resources allocated to each job. Workload size and the selected deployment method can increase the total infrastructure required. + +## Start with the default job resources + +The default executor configuration allocates the following resources to each job: + +| Resource | Default per job | +| -------- | --------------: | +| CPU | 4 cores | +| Memory | 12 GB | +| Disk | 20 GB | + +Use these defaults as a starting point, then account for the executor host, container runtime, operating system, and any deployment-specific overhead. Monitor real workloads and adjust capacity when jobs are resource constrained or executor infrastructure is consistently underused. + +## Calculate capacity for concurrent jobs + +`EXECUTOR_MAXIMUM_NUM_JOBS` controls the maximum number of jobs an executor can process concurrently. As a starting estimate, multiply the per-job resources by this value. + +For example, an executor processing four jobs concurrently with the defaults needs approximately 16 CPU cores, 48 GB of memory, and 80 GB of job disk, plus resources for the host and runtime. + +You can adjust the calculation with these settings: + +- `EXECUTOR_JOB_NUM_CPUS` controls CPU allocated to each job. +- `EXECUTOR_JOB_MEMORY` controls memory allocated to each job. +- `EXECUTOR_FIRECRACKER_DISK_SPACE` controls disk allocated to each Firecracker virtual machine. + +See the [executor configuration reference](/self-hosted/executors/executors-config) for the defaults and behavior of each setting. + +## Account for the deployment method + +- **Firecracker:** The host must have enough capacity for each concurrent MicroVM in addition to host overhead. Infrastructure must also support KVM. See [Firecracker isolation](/self-hosted/executors/firecracker). +- **Docker:** Job limits apply to containers, but the Docker daemon and executor also consume host resources. +- **Kubernetes:** Include executor pods, sidecars, job pods, and cluster scheduling constraints when sizing nodes or node pools. +- **Terraform:** Choose an instance type that supports the selected isolation method, then tune executor concurrency and autoscaling for the workload. See the [AWS](/self-hosted/executors/deploy-executors-terraform-aws#auto-scaling) or [Google Cloud](/self-hosted/executors/deploy-executors-terraform-gcp#auto-scaling) deployment guide. + +For AWS deployments, provision disk throughput and IOPS for the expected concurrent workload. The Terraform module defaults to 125 MiB/s throughput; increase it when workload measurements show that disk performance is a bottleneck. diff --git a/docs/self-hosted/index.mdx b/docs/self-hosted/index.mdx index 7d95e1784..75878f207 100644 --- a/docs/self-hosted/index.mdx +++ b/docs/self-hosted/index.mdx @@ -58,7 +58,7 @@ Get started running Sourcegraph on-prem. ## [Executors](/self-hosted/executors/) - [Executors overview](/self-hosted/executors/) -- [Deploy executors](/self-hosted/executors/deploy-executors) +- [Deploy executors](/self-hosted/executors) - [Kubernetes deployment](/self-hosted/executors/deploy-executors-kubernetes) *(Beta)* - Terraform deployment: [AWS](/self-hosted/executors/deploy-executors-terraform-aws) | [Google Cloud](/self-hosted/executors/deploy-executors-terraform-gcp) - [Docker deployment](/self-hosted/executors/deploy-executors-docker) diff --git a/docs/technical-changelog.mdx b/docs/technical-changelog.mdx index 9b41994ed..9169574f3 100644 --- a/docs/technical-changelog.mdx +++ b/docs/technical-changelog.mdx @@ -12459,7 +12459,7 @@ The following PRs were merged onto the previous release branch but could not be - [search.largeF[iles](/admin/config/site-config#search-largeFiles) accepts an optional prefix `!` to negate a pattern. The o)rder of the patterns within search.largeFiles is honored such that the last pattern matching overrides preceding patterns. For patterns that begin with a literal `!` prefix with a backslash, for example, `\!fileNameStartsWithExcl!.txt`. Previously indexed files that become excluded due to this change will remain in the index until the next reindex [#45318](https://github.com/sourcegraph/sourcegraph-public-snapshot/pull/45318) - [Webh[ooks](/admin/webhooks/incoming) have been overhauled completely and can now be found under **Site admin >) Repositories > Incoming webhooks**. Webhooks that were added via code host configuration are [deprec[ated](/admin/webhooks/incoming#deprecation-notice) and will be removed in 5.1.0. - Added support fo)r receiving webhook `push` events from GitHub which will trigger Sourcegraph to fetch the latest commit rather than relying on polling. -- Added support for private container registries in Sourcegraph executors. [Using private registries](/self-hosted/executors/deploy-executors#using-private-registries) +- Added support for private container registries in Sourcegraph executors. [Using private registries](/self-hosted/executors/private-registries) ### Changed diff --git a/src/data/navigation.ts b/src/data/navigation.ts index 2744ec424..723ba5758 100644 --- a/src/data/navigation.ts +++ b/src/data/navigation.ts @@ -531,7 +531,20 @@ export const navigation: NavigationItem[] = [ href: '/admin/permissions' }, {title: 'OAuth apps', href: '/admin/oauth-apps'}, - {title: 'Executors', href: '/admin/executors'}, + { + title: 'Executors', + href: '/admin/executors', + subsections: [ + { + title: 'Setting up executors', + href: '/admin/executors#setting-up-executors' + }, + { + title: 'Executor secrets', + href: '/admin/executors/executor-secrets' + } + ] + }, {title: 'FAQs', href: '/admin/faq'}, {title: 'How-to guides', href: '/admin/how-to'}, { @@ -586,7 +599,57 @@ export const navigation: NavigationItem[] = [ }, { title: 'Executors', - href: '/self-hosted/executors' + href: '/self-hosted/executors', + subsections: [ + { + title: 'Private container registries', + href: '/self-hosted/executors/private-registries' + }, + { + title: 'Custom certificates', + href: '/self-hosted/executors/custom-certificates' + }, + { + title: 'Plan executor capacity', + href: '/self-hosted/executors/resource-sizing' + }, + { + title: 'Terraform on AWS', + href: '/self-hosted/executors/deploy-executors-terraform-aws' + }, + { + title: 'Terraform on Google Cloud', + href: '/self-hosted/executors/deploy-executors-terraform-gcp' + }, + { + title: 'Linux binary', + href: '/self-hosted/executors/deploy-executors-binary' + }, + { + title: 'Docker Compose', + href: '/self-hosted/executors/deploy-executors-docker' + }, + { + title: 'Docker-in-Docker on Kubernetes', + href: '/self-hosted/executors/deploy-executors-dind' + }, + { + title: 'Native Kubernetes', + href: '/self-hosted/executors/deploy-executors-kubernetes' + }, + { + title: 'Configuration reference', + href: '/self-hosted/executors/executors-config' + }, + { + title: 'Firecracker isolation', + href: '/self-hosted/executors/firecracker' + }, + { + title: 'Troubleshooting', + href: '/self-hosted/executors/executors-troubleshooting' + } + ] }, { title: 'Observability', diff --git a/src/data/redirects.ts b/src/data/redirects.ts index 38f856d75..133ab937f 100644 --- a/src/data/redirects.ts +++ b/src/data/redirects.ts @@ -6066,7 +6066,7 @@ const redirectsData = [ }, { source: '/admin/executors/deploy_executors', - destination: '/self-hosted/executors/deploy_executors', + destination: '/self-hosted/executors', permanent: true }, { @@ -6974,7 +6974,12 @@ const redirectsData = [ }, { source: '/self-hosted/executors/deploy_executors', - destination: '/self-hosted/executors/deploy-executors', + destination: '/self-hosted/executors', + permanent: true + }, + { + source: '/self-hosted/executors/deploy-executors', + destination: '/self-hosted/executors', permanent: true }, {