Skip to content

Commit e567403

Browse files
vikrantpuppalavikrantpuppala
committed
ci: add kernel-e2e workflow + KERNEL_REV pin
Wires up CI coverage for use_kernel=True. The kernel is a private repo with no published wheel, so we pin a kernel SHA in KERNEL_REV and build the wheel inline via maturin develop using the existing INTEGRATION_TEST_APP GitHub App (extended to include databricks/databricks-sql-kernel in its repo allowlist). Gate semantics mirror trigger-integration-tests.yml: - Plain PR events post a synthetic-success Kernel E2E check so the required check doesn't block PRs that don't touch kernel code. - The kernel-e2e label triggers a preview run on the PR and is auto-removed on synchronize for the same security reason as the integration-test label. - merge_group is the real gate — runs when kernel-relevant files change (src/databricks/sql/backend/kernel/, test_kernel_backend.py, KERNEL_REV, etc.), auto-passes otherwise. Unit tests are unchanged: tests/unit/test_kernel_*.py already runs in every code-quality-checks.yml matrix combo against a fake databricks_sql_kernel module injected at sys.modules import time. Required follow-up before this merges: 1. Extend the INTEGRATION_TEST_APP allowlist to include databricks/databricks-sql-kernel. 2. Create the kernel-e2e label in this repo. 3. Add Kernel E2E as a required check on main once a green run lands. Co-authored-by: Isaac Signed-off-by: Vikrant Puppala <vikrant.puppala@databricks.com>
1 parent 085cb56 commit e567403

2 files changed

Lines changed: 355 additions & 0 deletions

File tree

‎.github/workflows/kernel-e2e.yml‎

Lines changed: 354 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,354 @@
1+
name: Kernel E2E Tests
2+
3+
# Runs tests/e2e/test_kernel_backend.py against a real Databricks
4+
# warehouse with a freshly-built databricks-sql-kernel wheel.
5+
#
6+
# The kernel is a private repo with no published artifact. We pin a
7+
# kernel SHA in the `KERNEL_REV` file at the repo root, check the
8+
# kernel out via a GitHub App token, and run `maturin develop` to
9+
# install the wheel into the same venv as the connector. Bumping
10+
# `KERNEL_REV` is the only way to pick up a new kernel version —
11+
# this keeps the connector ↔ kernel pair bisectable.
12+
#
13+
# Gate semantics mirror trigger-integration-tests.yml:
14+
# - Plain PR events post a synthetic-success check so the required
15+
# "Kernel E2E" check doesn't block PRs that don't touch the kernel
16+
# path. Real tests run in the merge queue.
17+
# - `kernel-e2e` label triggers a preview run on the PR. The label
18+
# is auto-removed on `synchronize` for the same security reason
19+
# trigger-integration-tests.yml does it.
20+
# - merge_group fires the real gate — dispatches when kernel-relevant
21+
# files changed, auto-passes otherwise.
22+
#
23+
# Required external setup:
24+
# 1. `kernel-e2e` label exists in this repo.
25+
# 2. `INTEGRATION_TEST_APP_ID` / `INTEGRATION_TEST_PRIVATE_KEY`
26+
# secrets exist (already installed for the proxy-tests workflow).
27+
# The GitHub App's repo allowlist must include
28+
# `databricks/databricks-sql-kernel` — extend the existing App
29+
# config; do not create a new App.
30+
# 3. `KERNEL_REV` file at the repo root containing a 40-char kernel
31+
# commit SHA.
32+
# 4. `azure-prod` environment exposes DATABRICKS_HOST /
33+
# TEST_PECO_WAREHOUSE_HTTP_PATH / DATABRICKS_TOKEN
34+
# (already configured for code-coverage.yml).
35+
36+
on:
37+
pull_request:
38+
types: [opened, synchronize, reopened, labeled]
39+
merge_group:
40+
41+
permissions:
42+
contents: read
43+
44+
# Cancel in-flight kernel-e2e runs on PR pushes — the warehouse state
45+
# is shared with code-coverage.yml so we already pay this cost there.
46+
# Don't cancel on main / merge_group; each commit needs its own signal.
47+
concurrency:
48+
group: kernel-e2e-${{ github.workflow }}-${{ github.ref }}
49+
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
50+
51+
jobs:
52+
# ───────────────────────────────────────────────────────────────
53+
# Security: auto-remove `kernel-e2e` label on new commits, same as
54+
# trigger-integration-tests.yml.
55+
# ───────────────────────────────────────────────────────────────
56+
remove-label-on-new-commit:
57+
if: github.event_name == 'pull_request' && github.event.action == 'synchronize'
58+
runs-on:
59+
group: databricks-protected-runner-group
60+
labels: linux-ubuntu-latest
61+
permissions:
62+
pull-requests: write
63+
issues: write
64+
steps:
65+
- name: Remove kernel-e2e label
66+
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7.1.0
67+
with:
68+
script: |
69+
const labels = context.payload.pull_request.labels.map(l => l.name);
70+
if (!labels.includes('kernel-e2e')) {
71+
console.log('Label not present, nothing to remove.');
72+
return;
73+
}
74+
try {
75+
await github.rest.issues.removeLabel({
76+
owner: context.repo.owner,
77+
repo: context.repo.repo,
78+
issue_number: context.issue.number,
79+
name: 'kernel-e2e'
80+
});
81+
console.log('Removed kernel-e2e label.');
82+
} catch (error) {
83+
if (error.status !== 404) throw error;
84+
}
85+
86+
# ───────────────────────────────────────────────────────────────
87+
# Synthetic success on every non-label PR event so the required
88+
# "Kernel E2E" check doesn't permablock PRs that don't touch kernel
89+
# code. Real run happens in the merge queue (or via explicit label).
90+
# ───────────────────────────────────────────────────────────────
91+
skip-kernel-e2e-pr:
92+
if: github.event_name == 'pull_request' && github.event.action != 'labeled'
93+
runs-on:
94+
group: databricks-protected-runner-group
95+
labels: linux-ubuntu-latest
96+
permissions:
97+
checks: write
98+
steps:
99+
- name: Post synthetic-success check
100+
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7.1.0
101+
with:
102+
github-token: ${{ github.token }}
103+
script: |
104+
await github.rest.checks.create({
105+
owner: context.repo.owner,
106+
repo: context.repo.repo,
107+
name: 'Kernel E2E',
108+
head_sha: context.payload.pull_request.head.sha,
109+
status: 'completed',
110+
conclusion: 'success',
111+
completed_at: new Date().toISOString(),
112+
output: {
113+
title: 'Skipped on PR — runs in merge queue',
114+
summary: 'Kernel E2E is skipped on PRs and runs as a required gate in the merge queue. Add the `kernel-e2e` label to preview on this PR.'
115+
}
116+
});
117+
118+
# ───────────────────────────────────────────────────────────────
119+
# Detect whether kernel-relevant files changed. Used by both the
120+
# labelled PR path and the merge-queue path to decide between
121+
# "really run the suite" and "auto-pass the check".
122+
# ───────────────────────────────────────────────────────────────
123+
detect-changes:
124+
if: |
125+
github.event_name == 'merge_group' ||
126+
(github.event_name == 'pull_request' &&
127+
github.event.action == 'labeled' &&
128+
contains(github.event.pull_request.labels.*.name, 'kernel-e2e'))
129+
runs-on:
130+
group: databricks-protected-runner-group
131+
labels: linux-ubuntu-latest
132+
outputs:
133+
run_tests: ${{ steps.changed.outputs.run_tests }}
134+
head_sha: ${{ steps.refs.outputs.head_sha }}
135+
pr_number: ${{ steps.refs.outputs.pr_number }}
136+
steps:
137+
- name: Resolve head SHA + PR number
138+
id: refs
139+
env:
140+
MERGE_QUEUE_REF: ${{ github.event.merge_group.head_ref }}
141+
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7.1.0
142+
with:
143+
script: |
144+
if (context.eventName === 'pull_request') {
145+
core.setOutput('head_sha', context.payload.pull_request.head.sha);
146+
core.setOutput('pr_number', String(context.payload.pull_request.number));
147+
return;
148+
}
149+
// merge_group — extract PR # from gh-readonly-queue/<base>/pr-<N>-<sha>
150+
const ref = process.env.MERGE_QUEUE_REF || '';
151+
const m = ref.match(/pr-(\d+)/);
152+
if (!m) core.setFailed(`could not extract pr number from ${ref}`);
153+
core.setOutput('head_sha', context.payload.merge_group.head_sha);
154+
core.setOutput('pr_number', m ? m[1] : '');
155+
156+
- name: Check out repo at head SHA
157+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
158+
with:
159+
ref: ${{ steps.refs.outputs.head_sha }}
160+
fetch-depth: 0
161+
162+
- name: Detect kernel-relevant changes
163+
id: changed
164+
env:
165+
HEAD_SHA: ${{ steps.refs.outputs.head_sha }}
166+
BASE_SHA: ${{ github.event_name == 'merge_group' && github.event.merge_group.base_sha || github.event.pull_request.base.sha }}
167+
run: |
168+
CHANGED=$(git diff --name-only "$BASE_SHA" "$HEAD_SHA")
169+
echo "Changed files:"
170+
echo "$CHANGED"
171+
# Run when the connector kernel backend, kernel e2e tests,
172+
# this workflow, the kernel revision pin, or core deps move.
173+
if echo "$CHANGED" | grep -qE "^(src/databricks/sql/backend/kernel/|tests/e2e/test_kernel_backend\.py|tests/unit/test_kernel_|\.github/workflows/kernel-e2e\.yml|KERNEL_REV|pyproject\.toml|poetry\.lock)"; then
174+
echo "run_tests=true" >> "$GITHUB_OUTPUT"
175+
else
176+
echo "run_tests=false" >> "$GITHUB_OUTPUT"
177+
fi
178+
179+
# ───────────────────────────────────────────────────────────────
180+
# Real test job. Builds the kernel wheel from the pinned SHA and
181+
# runs the connector's kernel e2e suite against the dogfood
182+
# warehouse.
183+
# ───────────────────────────────────────────────────────────────
184+
run-kernel-e2e:
185+
needs: detect-changes
186+
if: needs.detect-changes.outputs.run_tests == 'true'
187+
runs-on:
188+
group: databricks-protected-runner-group
189+
labels: linux-ubuntu-latest
190+
# azure-prod holds the warehouse secrets. Fork PRs are paused at
191+
# "approval required" — same model as code-coverage.yml.
192+
environment: azure-prod
193+
permissions:
194+
contents: read
195+
checks: write
196+
env:
197+
DATABRICKS_SERVER_HOSTNAME: ${{ secrets.DATABRICKS_HOST }}
198+
DATABRICKS_HTTP_PATH: ${{ secrets.TEST_PECO_WAREHOUSE_HTTP_PATH }}
199+
DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}
200+
steps:
201+
- name: Check out connector
202+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
203+
with:
204+
ref: ${{ needs.detect-changes.outputs.head_sha }}
205+
206+
- name: Read pinned kernel SHA
207+
id: kernel-rev
208+
run: |
209+
if [[ ! -f KERNEL_REV ]]; then
210+
echo "::error::KERNEL_REV file missing"
211+
exit 1
212+
fi
213+
REV=$(tr -d '[:space:]' < KERNEL_REV)
214+
if [[ ! "$REV" =~ ^[0-9a-f]{40}$ ]]; then
215+
echo "::error::KERNEL_REV must be a 40-char commit SHA, got: $REV"
216+
exit 1
217+
fi
218+
echo "rev=$REV" >> "$GITHUB_OUTPUT"
219+
echo "Pinned kernel SHA: $REV"
220+
221+
- name: Generate GitHub App token (kernel repo read access)
222+
id: app-token
223+
uses: actions/create-github-app-token@f8d387b68d61c58ab83c6c016672934102569859 # v3.0.0
224+
with:
225+
app-id: ${{ secrets.INTEGRATION_TEST_APP_ID }}
226+
private-key: ${{ secrets.INTEGRATION_TEST_PRIVATE_KEY }}
227+
owner: databricks
228+
repositories: databricks-sql-kernel
229+
230+
- name: Check out kernel at pinned SHA
231+
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
232+
with:
233+
repository: databricks/databricks-sql-kernel
234+
ref: ${{ steps.kernel-rev.outputs.rev }}
235+
token: ${{ steps.app-token.outputs.token }}
236+
path: databricks-sql-kernel
237+
238+
- name: Set up Python 3.10
239+
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
240+
with:
241+
python-version: "3.10"
242+
243+
- name: Set up Rust toolchain
244+
uses: actions-rust-lang/setup-rust-toolchain@1780873c7b576612439a134613cc4cc74ce5538c # v1.15.2
245+
246+
- name: Cache cargo build artifacts (keyed on kernel SHA)
247+
uses: Swatinem/rust-cache@98c8021b550208e191a6a3145459bfc9fb29c4c0 # v2.8.0
248+
with:
249+
workspaces: databricks-sql-kernel
250+
# Keying on the kernel SHA means each pinned version gets a
251+
# warm cache; bumping KERNEL_REV invalidates and rewarms.
252+
key: kernel-${{ steps.kernel-rev.outputs.rev }}
253+
254+
- name: Install Kerberos system deps
255+
run: |
256+
sudo apt-get update
257+
sudo apt-get install -y libkrb5-dev
258+
259+
- name: Setup Poetry + connector deps
260+
uses: ./.github/actions/setup-poetry
261+
with:
262+
python-version: "3.10"
263+
install-args: "--all-extras"
264+
cache-suffix: "kernel-e2e-"
265+
266+
- name: Install maturin into the poetry venv
267+
run: poetry run pip install 'maturin>=1.5,<2.0'
268+
269+
- name: Build + install kernel wheel into poetry venv
270+
working-directory: databricks-sql-kernel/pyo3
271+
# `maturin develop` builds the extension and installs it into
272+
# whichever Python is on PATH. `poetry run` resolves to the
273+
# connector's .venv, so the wheel lands where pytest will
274+
# import it.
275+
run: poetry run maturin develop --release
276+
277+
- name: Smoke-check kernel import
278+
run: |
279+
poetry run python -c "import databricks_sql_kernel as k; assert k.__file__, 'kernel module has no __file__ — wheel install failed'; print('kernel ok:', k.__file__)"
280+
281+
- name: Run kernel e2e tests
282+
run: poetry run pytest tests/e2e/test_kernel_backend.py -v
283+
284+
- name: Post Kernel E2E check (success)
285+
if: success() && github.event_name == 'merge_group'
286+
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7.1.0
287+
with:
288+
github-token: ${{ github.token }}
289+
script: |
290+
await github.rest.checks.create({
291+
owner: context.repo.owner,
292+
repo: context.repo.repo,
293+
name: 'Kernel E2E',
294+
head_sha: '${{ needs.detect-changes.outputs.head_sha }}',
295+
status: 'completed',
296+
conclusion: 'success',
297+
completed_at: new Date().toISOString(),
298+
output: {
299+
title: 'Kernel E2E passed',
300+
summary: 'tests/e2e/test_kernel_backend.py ran green against the pinned kernel SHA.'
301+
}
302+
});
303+
304+
- name: Post Kernel E2E check (failure)
305+
if: failure() && github.event_name == 'merge_group'
306+
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7.1.0
307+
with:
308+
github-token: ${{ github.token }}
309+
script: |
310+
await github.rest.checks.create({
311+
owner: context.repo.owner,
312+
repo: context.repo.repo,
313+
name: 'Kernel E2E',
314+
head_sha: '${{ needs.detect-changes.outputs.head_sha }}',
315+
status: 'completed',
316+
conclusion: 'failure',
317+
completed_at: new Date().toISOString(),
318+
output: {
319+
title: 'Kernel E2E failed',
320+
summary: 'See workflow logs for details.'
321+
}
322+
});
323+
324+
# ───────────────────────────────────────────────────────────────
325+
# Auto-pass the Kernel E2E check in the merge queue when no kernel-
326+
# relevant files changed.
327+
# ───────────────────────────────────────────────────────────────
328+
auto-pass-merge-queue:
329+
needs: detect-changes
330+
if: github.event_name == 'merge_group' && needs.detect-changes.outputs.run_tests != 'true'
331+
runs-on:
332+
group: databricks-protected-runner-group
333+
labels: linux-ubuntu-latest
334+
permissions:
335+
checks: write
336+
steps:
337+
- name: Auto-pass
338+
uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7.1.0
339+
with:
340+
github-token: ${{ github.token }}
341+
script: |
342+
await github.rest.checks.create({
343+
owner: context.repo.owner,
344+
repo: context.repo.repo,
345+
name: 'Kernel E2E',
346+
head_sha: '${{ github.event.merge_group.head_sha }}',
347+
status: 'completed',
348+
conclusion: 'success',
349+
completed_at: new Date().toISOString(),
350+
output: {
351+
title: 'Skipped — no kernel-relevant changes',
352+
summary: 'No files under src/databricks/sql/backend/kernel/, tests/e2e/test_kernel_backend.py, KERNEL_REV, pyproject.toml, or poetry.lock changed.'
353+
}
354+
});

‎KERNEL_REV‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
aed2efbed8087171d61848f5ad98c7e171827698

0 commit comments

Comments
 (0)