feat(dataset-mount): drive dataset mounts from the platform

[aicam]

What changes were proposed in this PR?

Build on the mount infrastructure (#6862) to let the platform and users mount datasets and use them in Python UDFs.

• Engine mount client — DatasetMountManager asks the node mounter to mount a dataset for the computing unit; the locator flows through PhysicalOp/WorkerConfig to the Python worker, which exposes the resolved local path.
• Per-computing-unit mount API — GET/POST/DELETE /computing-unit/{cuid}/mounts (ComputingUnitManagingResource + MounterClient), a stateless proxy to that CU's node mounter; FileResolver resolves dataset paths both ways.
• "Mount datasets into computing unit" UI — a per-CU modal to mount / list / unmount datasets.
• Python UDF dataset-variable bindings — bind each mounted dataset to a variable holding its local path, e.g. open(f"{A}/file.csv").

Any related issues, documentation, discussions?

Closes #6863 · part of #6606. Stacks on #6862 (mount infrastructure); this PR's diff becomes clean once #6862 merges.

How was this PR tested?

• sbt WorkflowOperator/WorkflowExecutionService/ComputingUnitManagingService/FileService compile + production frontend build green.
• Unit tests for the pure-logic paths: MounterClientSpec (mount / list / unmount + error handling against a stub mounter HTTP server), DatasetMountManagerSpec (locator + identity validation), PythonUDFOpDescV2Spec (dataset-variable validation), DatasetVariableMappingSpec, PythonWorkflowWorkerStartupConfigSpec, test_run_python_worker.py, and test_executor_manager.py (the MOUNTED_DATASETS variable injection).
• Validated end-to-end on minikube: mounted a dataset onto a Kubernetes CU via the API, bound it to a variable in a Python UDF, and read a ~2 GB sharded PyTorch model from the variable's path via torch.load with bit-exact output (model_output_sum == expected_sum); the mount API's mount → list → unmount cycle was exercised directly.

Note on patch coverage: the remaining uncovered lines are integration/IO glue — DatasetMountManager's HTTP call to the node mounter, the /computing-unit/{cuid}/mounts endpoints (k8s + DB + mounter), and the S3 proxy forwarding carried over from #6862 — which aren't meaningfully unit-testable without standing up a live mounter + LakeFS. That path is covered by the end-to-end validation above rather than by codecov-visible unit tests. We'd appreciate reviewers weighing the patch-coverage signal accordingly.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Opus 4.8

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• Contributors with relevant context: @Ma77Ball, @yangzhang75, @Yicong-Huang
  You can notify them by mentioning @Ma77Ball, @yangzhang75, @Yicong-Huang in a comment.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 20.77922% with 244 lines in your changes missing coverage. Please review.
✅ Project coverage is 76.92%. Comparing base (f02dd2f) to head (a378a50).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
...xera/amber/engine/common/DatasetMountManager.scala	0.00%	70 Missing ⚠️
...rg/apache/texera/service/util/S3ProxyServlet.scala	5.79%	65 Missing ⚠️
...rvice/resource/ComputingUnitManagingResource.scala	0.00%	57 Missing ⚠️
.../apache/texera/service/util/KubernetesClient.scala	4.00%	24 Missing ⚠️
...pache/texera/amber/core/storage/FileResolver.scala	26.66%	11 Missing ⚠️
...chitecture/pythonworker/PythonWorkflowWorker.scala	33.33%	4 Missing ⚠️
.../amber/operator/udf/python/PythonUDFOpDescV2.scala	55.55%	3 Missing and 1 partial ⚠️
...org/apache/texera/service/util/MounterClient.scala	87.87%	0 Missing and 4 partials ⚠️
...r/operator/udf/python/DatasetVariableMapping.scala	0.00%	3 Missing ⚠️
...apache/texera/amber/core/workflow/PhysicalOp.scala	66.66%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #6867      +/-   ##
============================================
- Coverage     77.30%   76.92%   -0.39%     
+ Complexity     3524     3522       -2     
============================================
  Files          1161     1165       +4     
  Lines         45922    46219     +297     
  Branches       5101     5137      +36     
============================================
+ Hits          35501    35552      +51     
- Misses         8840     9077     +237     
- Partials       1581     1590       +9     

Flag	Coverage Δ		*Carryforward flag
access-control-service	70.00% <ø> (ø)
agent-service	76.76% <ø> (ø)		Carriedforward from c16ed44
amber	68.72% <19.13%> (-0.38%)	⬇️	Carriedforward from c16ed44
computing-unit-managing-service	21.82% <26.08%> (+1.32%)	⬆️
config-service	66.66% <ø> (ø)
file-service	59.49% <5.71%> (-7.72%)	⬇️
frontend	82.58% <ø> (ø)		Carriedforward from c16ed44
notebook-migration-service	78.94% <ø> (ø)
pyamber	92.16% <100.00%> (+0.01%)	⬆️
workflow-compiling-service	55.14% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 4 better · 🔴 2 worse · ⚪ 9 noise (<±5%) · 0 without baseline

Compared against main f02dd2f benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	387	0.236	25,748/29,557/29,557 us	🟢 -29.7% / 🔴 +100.9%
🔴	bs=100 sw=10 sl=64	789	0.482	126,942/147,000/147,000 us	🟢 -11.0% / 🔴 +34.7%
⚪	bs=1000 sw=10 sl=64	911	0.556	1,099,075/1,138,991/1,138,991 us	⚪ within ±5% / 🔴 +9.6%

Baseline details

Latest main f02dd2f from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	387 tuples/sec	371 tuples/sec	754.55 tuples/sec	+4.3%	-48.7%
bs=10 sw=10 sl=64	MB/s	0.236 MB/s	0.226 MB/s	0.461 MB/s	+4.4%	-48.8%
bs=10 sw=10 sl=64	p50	25,748 us	23,025 us	12,816 us	+11.8%	+100.9%
bs=10 sw=10 sl=64	p95	29,557 us	42,018 us	16,594 us	-29.7%	+78.1%
bs=10 sw=10 sl=64	p99	29,557 us	42,018 us	19,806 us	-29.7%	+49.2%
bs=100 sw=10 sl=64	throughput	789 tuples/sec	806 tuples/sec	969.38 tuples/sec	-2.1%	-18.6%
bs=100 sw=10 sl=64	MB/s	0.482 MB/s	0.492 MB/s	0.592 MB/s	-2.0%	-18.5%
bs=100 sw=10 sl=64	p50	126,942 us	117,384 us	103,584 us	+8.1%	+22.5%
bs=100 sw=10 sl=64	p95	147,000 us	165,173 us	109,097 us	-11.0%	+34.7%
bs=100 sw=10 sl=64	p99	147,000 us	165,173 us	117,304 us	-11.0%	+25.3%
bs=1000 sw=10 sl=64	throughput	911 tuples/sec	909 tuples/sec	1,004 tuples/sec	+0.2%	-9.2%
bs=1000 sw=10 sl=64	MB/s	0.556 MB/s	0.555 MB/s	0.613 MB/s	+0.2%	-9.2%
bs=1000 sw=10 sl=64	p50	1,099,075 us	1,099,493 us	1,002,357 us	-0.0%	+9.6%
bs=1000 sw=10 sl=64	p95	1,138,991 us	1,159,845 us	1,046,463 us	-1.8%	+8.8%
bs=1000 sw=10 sl=64	p99	1,138,991 us	1,159,845 us	1,073,661 us	-1.8%	+6.1%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,517.27,200,128000,387,0.236,25747.79,29556.60,29556.60
1,100,10,64,20,2535.16,2000,1280000,789,0.482,126941.56,146999.69,146999.69
2,1000,10,64,20,21953.36,20000,12800000,911,0.556,1099074.85,1138990.92,1138990.92