-
Notifications
You must be signed in to change notification settings - Fork 100
docs: reorganize executor administration and deployment guidance #1818
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
DaedalusG
wants to merge
6
commits into
main
Choose a base branch
from
wg/plat/executors-docs-overhaul
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
c899ccf
first draft nav overhaul
DaedalusG 1d4178d
admin/executors improvements
DaedalusG a3e3200
admin/executors sidebar nav
DaedalusG 8c4601d
callout cloud offering vs self hosted with regards to executors
DaedalusG 8e651ed
major self hosted docs overhaul, break up big list pages, and seperat…
DaedalusG 329cb99
tighten up Before you deploy instructions and seperate resourcing con…
DaedalusG File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -34,7 +34,7 @@ Before creating an Agentic Batch Change, make sure your Sourcegraph instance can | |||||
|
|
||||||
| - [Configure code host connections](/admin/code-hosts) for the repositories you want the agent to search and modify. | ||||||
| - [Configure Batch Changes credentials](/batch-changes/configuring-credentials) for each code host where the agent should publish pull requests or merge requests. | ||||||
| - [Deploy Sourcegraph executors](/self-hosted/executors/deploy-executors) so Agentic Batch Changes can run code-modification jobs in isolated workspaces. Executors are available with no additional setup on [Sourcegraph Cloud](/cloud). | ||||||
| - If you use self-hosted Sourcegraph, [deploy Sourcegraph executors](/self-hosted/executors) so Agentic Batch Changes can run code-modification jobs in isolated workspaces. [Sourcegraph Cloud](/cloud) includes Sourcegraph-managed executors by default, so Cloud customers using them do not need to deploy executor infrastructure. | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You can be in Cloud and have self-hosted executors
Suggested change
|
||||||
|
|
||||||
| ### Prompting | ||||||
|
|
||||||
|
|
||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -11,7 +11,7 @@ | |
|
|
||
| By default, Batch Changes uses a command line interface in your local environment to [compute diffs](/batch-changes/how-src-executes-a-batch-spec) and create changesets. This can be impractical for creating batch changes affecting hundreds or thousands of repositories, with large numbers of workspaces, or if the batch change steps require CPU, memory, or disk resources that are unavailable locally. | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this |
||
|
|
||
| Instead of computing Batch Changes locally using `src-cli`, you can offload this task to one or many remote server called an [executor](/self-hosted/executors/deploy-executors). Executors are also required to enable code navigation [auto-indexing](/code-navigation/auto-indexing). | ||
| Instead of computing Batch Changes locally using `src-cli`, you can offload this task to one or many remote servers called [executors](/admin/executors). Executors are also required to enable code navigation [auto-indexing](/code-navigation/auto-indexing). | ||
|
|
||
| This allows to: | ||
|
|
||
|
|
@@ -21,16 +21,15 @@ This allows to: | |
|
|
||
| ## Setup | ||
|
|
||
| This is a one-time process. Once a site-admin of the Sourcegraph instance sets up executors and enables running batch changes server-side, all users of the Sourcegraph instance can get started with no additional setup required. | ||
| Sourcegraph Cloud includes Sourcegraph-managed executors by default, so Cloud customers using them do not need to deploy executor infrastructure. | ||
|
|
||
| Make sure that [executors are deployed and are online](/self-hosted/executors/deploy-executors). | ||
| If you operate a self-hosted Sourcegraph instance, [deploy executors](/self-hosted/executors) and confirm that they are online. This is a one-time process. Once a site administrator sets up executors and enables running Batch Changes server-side, users can get started without additional executor setup. | ||
|
|
||
| ## Limitations | ||
|
|
||
| - Running batch changes server-side requires setting up executors. Executors are configured ready-to-use on Sourcegraph Cloud | ||
| - Self-hosted Sourcegraph instances require executor setup. Executors are ready to use on Sourcegraph Cloud. | ||
| - Running batch changes server-side is limited to user namespaces | ||
| - The newly introduced APIs for server-side are still experimental and will likely change | ||
| - Executors can only be deployed using Terraform (AWS or GCP) or using pre-built binaries (see [deploying executors](/self-hosted/executors/deploy-executors)). | ||
|
|
||
| Running batch changes server-side has been tested to run a simple **45K changeset batch change**. Actual performance and setup requirements depend on the complexity of the batch change. | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,108 @@ | ||
| # Use custom certificates with executors | ||
|
|
||
| By default, executors will search for certificates in the following files and directories: | ||
|
|
||
| | Directory or file | Distribution | | ||
| | --------------------------------------------------- | ------------------------- | | ||
| | `/etc/ssl/certs/ca-certificates.crt` | Debian/Ubuntu/Gentoo etc. | | ||
| | `/etc/pki/tls/certs/ca-bundle.crt` | Fedora/RHEL 6 | | ||
| | `/etc/ssl/ca-bundle.pem` | OpenSUSE | | ||
| | `/etc/pki/tls/cacert.pem` | OpenELEC | | ||
| | `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem` | CentOS/RHEL 7 | | ||
| | `/etc/ssl/cert.pem` | Alpine Linux | | ||
| | `/etc/ssl/certs` | SLES10/SLES11 | | ||
| | `/etc/pki/tls/certs` | Fedora/RHEL | | ||
| | `/system/etc/security/cacerts` | Android | | ||
|
|
||
| If your environment makes use of custom certificates, you can add them to one of these locations in order for executors to pick them up. | ||
|
|
||
| ## Add certificates to a binary deployment | ||
|
|
||
| > NOTE: see the [troubleshooting guide](/self-hosted/executors/executors-troubleshooting#connecting-to-cloud-provider-executor-instances) for instructions on how to connect to cloud provider VMs. | ||
|
|
||
| After successfully [deploying binaries](/self-hosted/executors/deploy-executors-binary), follow these steps: | ||
|
|
||
| 1. Copy your certificates to `/etc/ssl/certs`. | ||
| 1. If you are using systemd, run `systemctl restart executor`. If not, proceed to the next step. | ||
| 1. Run `executor run` on the VM in order to restart the executor service. | ||
|
|
||
| ### Add certificates with Firecracker | ||
|
|
||
| When running executors with the [firecracker runtime](/self-hosted/executors/firecracker), custom certificates need to be added in | ||
| the container that is running within the Firecracker VM. To add custom certificates, you must create a new Docker image | ||
| that contains the certificates. For example, | ||
|
|
||
| ```dockerfile | ||
| FROM upstream:tag | ||
|
|
||
| # Copy the certificates into the container | ||
| COPY customcert.crt /usr/local/share/ca-certificates/customcert.crt | ||
| # Update the certificate store | ||
| RUN chmod 644 /usr/local/share/ca-certificates/customcert.crt && update-ca-certificates | ||
| # ... | ||
| ``` | ||
|
|
||
| #### Code navigation | ||
|
|
||
| Once the custom image is built, you can configure the executor to use it by setting | ||
| the `codeIntelAutoIndexing.indexerMap` to use the custom image. For example, | ||
|
|
||
| ```json | ||
| "codeIntelAutoIndexing.indexerMap": { | ||
| "go": "myregistry.company.com/scip-go:custom" | ||
| } | ||
| ``` | ||
|
|
||
| ## Add certificates to a Kubernetes deployment using manifests | ||
|
|
||
| First, add the certificate data as a secret in your preferred namespace: | ||
|
|
||
| ```shell | ||
| SECRET_NAME=custom-certs | ||
| CERT_PATH=/path/to/cert.pem | ||
| kubectl create secret generic $SECRET_NAME --from-file=customcert.crt=$CERT_PATH | ||
| ``` | ||
|
|
||
| Or as a declarative manifest: | ||
|
|
||
| ```yaml | ||
| apiVersion: v1 | ||
| kind: Secret | ||
| metadata: | ||
| name: custom-certs | ||
| data: | ||
| customcert.crt: $(base64 -i /path/to/cert.pem) | ||
| type: Opaque | ||
| ``` | ||
|
|
||
| Next, mount the secret in the executor deployment. Add the following snippet to `spec.template.spec.volumes` of each executor deployment: | ||
|
|
||
| ```yaml | ||
| - name: custom-certs | ||
| secret: | ||
| secretName: custom-certs | ||
| ``` | ||
|
|
||
| Also add this snippet to `spec.template.spec.containers.volumeMounts` of each executor deployment (specifically, the executor container, in case you inject any sidecars): | ||
|
|
||
| ```yaml | ||
| - mountPath: /etc/ssl/certs | ||
| name: custom-certs | ||
| readOnly: true | ||
| ``` | ||
|
|
||
| Next, apply the updated YAML manifests. Once the executors have rolled out, they should be picking up your custom certificates. | ||
|
|
||
| ## Add certificates to a Kubernetes deployment using Helm | ||
|
|
||
| You may follow the same instructions for the manifest deployment to set custom certificates. | ||
|
|
||
| ## Add certificates to a Docker Compose deployment | ||
|
|
||
| First, ensure that the certificate file is present on the host machine. Next, add the volume to the [executor compose file](https://sourcegraph.com/github.com/sourcegraph/deploy-sourcegraph-docker/-/blob/docker-compose/executors/executor.docker-compose.yaml?L26-30): | ||
|
|
||
| ```yaml | ||
| - '/path/to/certs:/etc/ssl/certs' | ||
| ``` | ||
|
|
||
| Next, restart the deployment with `docker-compose down` and `docker-compose up -d`. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.