feat(dataset-mount): mount datasets onto a computing unit#6896
Open
aicam wants to merge 2 commits into
Open
Conversation
Perform the FUSE mount for dataset repositories outside the (unprivileged) computing-unit pod. A per-node privileged `texera-mounter` DaemonSet runs GeeseFS on the pod's behalf and the read-only mount is exposed back into the pod via mount propagation, scoped to that computing unit. A JWT-authenticated S3 proxy in file-service fronts the LakeFS S3 gateway: it verifies the pod's JWT, checks the user's read access, and re-signs to LakeFS with credentials held only server-side, so no global credential ever enters the pod. - `texera-mounter` DaemonSet: mounter.py (+ tests), dockerfile, helm daemonset/rbac/values. - Unprivileged CU pod + mount-propagation wiring (KubernetesClient) and mounter config/env. - File-service JWT S3 proxy (S3ProxyServlet). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H27sHgS29u2JcLYPPR6Hdx
Add per-computing-unit dataset mounting on top of the mount
infrastructure. A user can mount / list / unmount a dataset version on a
running computing unit; the platform asks that CU's node mounter to
perform the mount and reports the in-pod path back.
- ComputingUnitManagingResource: GET/POST/DELETE
/computing-unit/{cuid}/mounts, backed by a credential-free MounterClient
that talks to the node mounter over HTTP.
- FileResolver: resolve a dataset path to (repo, versionHash) and back,
so mounts can be shown to the user as /owner/dataset/version.
- "Mount datasets into computing unit" UI on the computing-unit selector.
At this point datasets can be mounted onto a CU but are not yet consumed
by any operator; that follows in the Python-UDF integration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H27sHgS29u2JcLYPPR6Hdx
github-actions
Bot
added
feature
frontend
Contributor
Automated Reviewer SuggestionsBased on the
|
This was referenced Jul 25, 2026
Codecov Report❌ Patch coverage is ❌ Your patch status has failed because the patch coverage (18.81%) is below the target coverage (60.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #6896 +/- ##
============================================
+ Coverage 77.30% 82.54% +5.23%
+ Complexity 3524 2354 -1170
============================================
Files 1161 896 -265
Lines 45922 37881 -8041
Branches 5101 3992 -1109
============================================
- Hits 35501 31268 -4233
+ Misses 8840 5473 -3367
+ Partials 1581 1140 -441
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 390 | 0.238 | 24,720/36,464/36,464 us | 🔴 +9.7% / 🔴 +119.7% |
| 🟢 | bs=100 sw=10 sl=64 | 815 | 0.497 | 121,565/149,291/149,291 us | 🟢 -6.2% / 🔴 +36.8% |
| ⚪ | bs=1000 sw=10 sl=64 | 919 | 0.561 | 1,083,463/1,139,503/1,139,503 us | ⚪ within ±5% / 🔴 +8.9% |
Baseline details
Latest main 57d4dba from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 390 tuples/sec | 416 tuples/sec | 754.55 tuples/sec | -6.3% | -48.3% |
| bs=10 sw=10 sl=64 | MB/s | 0.238 MB/s | 0.254 MB/s | 0.461 MB/s | -6.3% | -48.3% |
| bs=10 sw=10 sl=64 | p50 | 24,720 us | 22,530 us | 12,816 us | +9.7% | +92.9% |
| bs=10 sw=10 sl=64 | p95 | 36,464 us | 34,386 us | 16,594 us | +6.0% | +119.7% |
| bs=10 sw=10 sl=64 | p99 | 36,464 us | 34,386 us | 19,806 us | +6.0% | +84.1% |
| bs=100 sw=10 sl=64 | throughput | 815 tuples/sec | 811 tuples/sec | 969.38 tuples/sec | +0.5% | -15.9% |
| bs=100 sw=10 sl=64 | MB/s | 0.497 MB/s | 0.495 MB/s | 0.592 MB/s | +0.4% | -16.0% |
| bs=100 sw=10 sl=64 | p50 | 121,565 us | 117,400 us | 103,584 us | +3.5% | +17.4% |
| bs=100 sw=10 sl=64 | p95 | 149,291 us | 159,118 us | 109,097 us | -6.2% | +36.8% |
| bs=100 sw=10 sl=64 | p99 | 149,291 us | 159,118 us | 117,304 us | -6.2% | +27.3% |
| bs=1000 sw=10 sl=64 | throughput | 919 tuples/sec | 937 tuples/sec | 1,004 tuples/sec | -1.9% | -8.4% |
| bs=1000 sw=10 sl=64 | MB/s | 0.561 MB/s | 0.572 MB/s | 0.613 MB/s | -1.9% | -8.4% |
| bs=1000 sw=10 sl=64 | p50 | 1,083,463 us | 1,066,430 us | 1,002,357 us | +1.6% | +8.1% |
| bs=1000 sw=10 sl=64 | p95 | 1,139,503 us | 1,124,801 us | 1,046,463 us | +1.3% | +8.9% |
| bs=1000 sw=10 sl=64 | p99 | 1,139,503 us | 1,124,801 us | 1,073,661 us | +1.3% | +6.1% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,512.93,200,128000,390,0.238,24719.60,36464.37,36464.37
1,100,10,64,20,2454.36,2000,1280000,815,0.497,121564.67,149290.87,149290.87
2,1000,10,64,20,21753.03,20000,12800000,919,0.561,1083462.81,1139502.72,1139502.72
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Add per-computing-unit dataset mounting on top of the mount infrastructure (#6866) — the platform step of the dataset-mounting feature (#6606).
A user can mount / list / unmount a dataset version on a running computing unit. The platform asks that CU's node mounter to perform the mount and reports the in-pod path back; nothing consumes the mount yet — that is the follow-up Python-UDF PR.
GET/POST/DELETE /computing-unit/{cuid}/mounts(ComputingUnitManagingResource), backed by a credential-freeMounterClientthat talks to the CU's node mounter over HTTP. Authorization stays in file-service via the pod's JWT — no credential enters the platform path.FileResolver— resolve a dataset path to(repo, versionHash)and back, so a mount is shown to the user as/owner/dataset/version.Any related issues, documentation, discussions?
Closes #6863 · part of #6606 · builds on #6866.
How was this PR tested?
sbt ComputingUnitManagingService/Test/compilegreen (main + test).MounterClientSpecexercises mount / list / unmount against a stub node-mounter HTTP server, asserting the request shape and non-2xx error handling.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8