feat(dataset-mount): use mounted datasets in Python UDFs#6897
Open
aicam wants to merge 3 commits into
Open
Conversation
Perform the FUSE mount for dataset repositories outside the (unprivileged) computing-unit pod. A per-node privileged `texera-mounter` DaemonSet runs GeeseFS on the pod's behalf and the read-only mount is exposed back into the pod via mount propagation, scoped to that computing unit. A JWT-authenticated S3 proxy in file-service fronts the LakeFS S3 gateway: it verifies the pod's JWT, checks the user's read access, and re-signs to LakeFS with credentials held only server-side, so no global credential ever enters the pod. - `texera-mounter` DaemonSet: mounter.py (+ tests), dockerfile, helm daemonset/rbac/values. - Unprivileged CU pod + mount-propagation wiring (KubernetesClient) and mounter config/env. - File-service JWT S3 proxy (S3ProxyServlet). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H27sHgS29u2JcLYPPR6Hdx
Add per-computing-unit dataset mounting on top of the mount
infrastructure. A user can mount / list / unmount a dataset version on a
running computing unit; the platform asks that CU's node mounter to
perform the mount and reports the in-pod path back.
- ComputingUnitManagingResource: GET/POST/DELETE
/computing-unit/{cuid}/mounts, backed by a credential-free MounterClient
that talks to the node mounter over HTTP.
- FileResolver: resolve a dataset path to (repo, versionHash) and back,
so mounts can be shown to the user as /owner/dataset/version.
- "Mount datasets into computing unit" UI on the computing-unit selector.
At this point datasets can be mounted onto a CU but are not yet consumed
by any operator; that follows in the Python-UDF integration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H27sHgS29u2JcLYPPR6Hdx
Let a Python UDF bind mounted datasets to variables. Each binding maps a dataset version to a Python variable that, at runtime, holds the local filesystem path of the mounted dataset. - PythonUDFOpDescV2: a list of (variableName -> dataset) bindings, validated as Python identifiers and resolved via FileResolver; carried through PhysicalOp and WorkerConfig. - DatasetMountManager: on the executor, ensure each bound dataset is mounted on the computing unit and hand its path to the Python worker, which injects it as a module-level variable (MOUNTED_DATASETS). - "Mounted dataset variables" property editor for the Python UDF. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01H27sHgS29u2JcLYPPR6Hdx
github-actions
Bot
added
feature
engine
pyamber
frontend
Contributor
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #6897 +/- ##
============================================
+ Coverage 77.30% 82.71% +5.40%
+ Complexity 3524 2353 -1171
============================================
Files 1161 894 -267
Lines 45922 37899 -8023
Branches 5101 3980 -1121
============================================
- Hits 35501 31348 -4153
+ Misses 8840 5412 -3428
+ Partials 1581 1139 -442
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 644 | 0.393 | 15,660/18,388/18,388 us | 🟢 -20.4% / 🔴 +22.2% |
| 🔴 | bs=100 sw=10 sl=64 | 1,315 | 0.803 | 75,582/99,481/99,481 us | 🟢 -14.9% / 🟢 +35.7% |
| 🟢 | bs=1000 sw=10 sl=64 | 1,599 | 0.976 | 624,123/659,629/659,629 us | 🟢 -5.8% / 🟢 +59.3% |
Baseline details
Latest main 57d4dba from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 644 tuples/sec | 683 tuples/sec | 754.55 tuples/sec | -5.7% | -14.7% |
| bs=10 sw=10 sl=64 | MB/s | 0.393 MB/s | 0.417 MB/s | 0.461 MB/s | -5.8% | -14.7% |
| bs=10 sw=10 sl=64 | p50 | 15,660 us | 14,754 us | 12,816 us | +6.1% | +22.2% |
| bs=10 sw=10 sl=64 | p95 | 18,388 us | 23,087 us | 16,594 us | -20.4% | +10.8% |
| bs=10 sw=10 sl=64 | p99 | 18,388 us | 23,087 us | 19,806 us | -20.4% | -7.2% |
| bs=100 sw=10 sl=64 | throughput | 1,315 tuples/sec | 1,366 tuples/sec | 969.38 tuples/sec | -3.7% | +35.7% |
| bs=100 sw=10 sl=64 | MB/s | 0.803 MB/s | 0.834 MB/s | 0.592 MB/s | -3.7% | +35.7% |
| bs=100 sw=10 sl=64 | p50 | 75,582 us | 68,800 us | 103,584 us | +9.9% | -27.0% |
| bs=100 sw=10 sl=64 | p95 | 99,481 us | 116,853 us | 109,097 us | -14.9% | -8.8% |
| bs=100 sw=10 sl=64 | p99 | 99,481 us | 116,853 us | 117,304 us | -14.9% | -15.2% |
| bs=1000 sw=10 sl=64 | throughput | 1,599 tuples/sec | 1,613 tuples/sec | 1,004 tuples/sec | -0.9% | +59.3% |
| bs=1000 sw=10 sl=64 | MB/s | 0.976 MB/s | 0.985 MB/s | 0.613 MB/s | -0.9% | +59.3% |
| bs=1000 sw=10 sl=64 | p50 | 624,123 us | 614,996 us | 1,002,357 us | +1.5% | -37.7% |
| bs=1000 sw=10 sl=64 | p95 | 659,629 us | 699,890 us | 1,046,463 us | -5.8% | -37.0% |
| bs=1000 sw=10 sl=64 | p99 | 659,629 us | 699,890 us | 1,073,661 us | -5.8% | -38.6% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,310.69,200,128000,644,0.393,15660.20,18388.07,18388.07
1,100,10,64,20,1520.35,2000,1280000,1315,0.803,75581.87,99481.33,99481.33
2,1000,10,64,20,12506.48,20000,12800000,1599,0.976,624122.59,659629.12,659629.12
Contributor
|
@aicam Please make the PR description more readable. Also this PR changed 44 files. Can we divide it? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this PR?
Let a Python UDF consume mounted datasets — the final step of the dataset-mounting feature (#6606), stacked on the per-computing-unit mounting PR (#6896).
Each UDF binding maps a dataset version to a Python variable that, at runtime, holds the local filesystem path of the mounted dataset, e.g.
open(f"{A}/file.csv").PythonUDFOpDescV2— a list of(variableName -> dataset)bindings, validated as Python identifiers and resolved viaFileResolver; the resolved locators flow throughPhysicalOpandWorkerConfig.DatasetMountManager— on the executor, ensures each bound dataset is mounted on the computing unit and hands its path to the Python worker, which injects it as a module-level variable (MOUNTED_DATASETS).Any related issues, documentation, discussions?
Closes #6895 · part of #6606 · builds on #6896.
How was this PR tested?
PythonUDFOpDescV2Spec(binding validation — Python identifiers, blank rows, empty dataset),DatasetVariableMappingSpec,DatasetMountManagerSpec(locator/identity validation),PythonWorkflowWorkerStartupConfigSpec, andtest_executor_manager/test_run_python_worker(theMOUNTED_DATASETSvariable injection).torch.load-ed it from the propagated path with bit-exact output.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 4.8