Skip to content

README.md

Latest commit

 

History

History
116 lines (93 loc) · 5.19 KB

README.md

File metadata and controls

116 lines (93 loc) · 5.19 KB
Raw

SHARP Image2Splat service

FastAPI wrapper around the single-frame SHARP depth-injection path: RGB image + metric depth EXR → 3D Gaussian Splat .ply. Runs on a CUDA GPU host (built/tested on an RTX 4090). Any HTTP client can call it (e.g. via a SHARP_BACKEND_URL your app reads).

Setup (GPU host)

powershell -ExecutionPolicy Bypass -File service\install.ps1   # idempotent, ~few min first run
powershell -ExecutionPolicy Bypass -File service\run.ps1       # starts the server

install.ps1 clones apple/ml-sharp (pinned cdb4ddc6) as a sibling, builds a Python 3.13 venv at <repo>/.venv, installs everything, forces the CUDA torch build (ml-sharp's requirements otherwise pull CPU-only torch), and verifies torch.cuda.is_available() + import sharp, sharp_ext.

Auto-start on boot (headless)

Registered as a Windows Scheduled Task that starts at boot with no login and restarts on crash. Because the venv runs on the per-user Microsoft Store Python (which SYSTEM cannot launch), the task runs as the user account "whether logged on or not" — Task Scheduler stores the account password encrypted.

Install (elevated; prompts for the account password in a secure dialog):

powershell -ExecutionPolicy Bypass -File service\install-autostart.ps1

This registers task SHARP Image2Splat (AtStartup, RunLevel Limited, restart every 1 min on failure). _autostart.ps1 is the boot entry point — it runs serve.py with stdout/stderr appended to service\logs\service.log.

Manage it:

Start-ScheduledTask  -TaskName "SHARP Image2Splat"   # start now
Stop-ScheduledTask   -TaskName "SHARP Image2Splat"   # stop
Get-ScheduledTask    -TaskName "SHARP Image2Splat" | Get-ScheduledTaskInfo
Unregister-ScheduledTask -TaskName "SHARP Image2Splat"   # remove auto-start

Redeploy after a code change (e.g. pulling a new revision) — you must Stop then Start:

Stop-ScheduledTask  -TaskName "SHARP Image2Splat"
Start-ScheduledTask -TaskName "SHARP Image2Splat"

The task runs in a non-interactive session, so Stop-Process/Task-Manager from your logged-in session can't see it, and Start-ScheduledTask alone is a no-op while it's already running (MultipleInstances=IgnoreNew). Use Stop-ScheduledTask to terminate it.

Note: the task and run.ps1 both bind port 8765 — don't run both at once.

Config (env)

var default meaning
SHARP_HOST 0.0.0.0 bind address
SHARP_PORT 8765 bind port
SHARP_DEVICE cuda torch device (startup fails loud if cuda requested but unavailable)

Single worker by design — one model instance holds ~6 GB VRAM and is reused across requests.

Endpoints

GET /health

{ "ok": true, "cuda": true, "device": "cuda", "model_loaded": true }

POST /inspect — multipart

Field depth (EXR). Returns depth stats (shape, min/max/mean, sky/far %, finiteness) — use to sanity-check a depth pass before generating.

POST /generate — multipart

field type notes
image file (PNG/JPG) RGB; alpha stripped
depth file (EXR), optional camera-space Z, meters, +forward. Required for exr_pixel/exr_grade; ignored for sharp
albedo file (PNG/JPG), optional Houdini albedo AOV; required only for exr_grade + grade_source=region
f_px float (optional) focal length in px; OR provide the next two
focal_mm + aperture_mm float (optional) f_px = focal_mm/aperture_mm * image_width
depth_method str (default exr_pixel) sharp | exr_pixel | exr_grade (see below)
grade_curve str (default affine) exr_grade only: affine | polynomial | histogram
grade_source str (default percentile) exr_grade only: percentile | region
grade_min_slope float (default 0.0) exr_grade only: floor on the grade-curve slope to stop flicker/popping under camera motion. 0.3–0.6 recommended if you see it
blend_alpha float (default 0.4) exr_pixel only: 0=trust depth, 1=vanilla SHARP
output_path str (optional) if set, write the PLY server-side and return JSON instead of bytes

depth_method:

  • sharp — plain SHARP, no depth injection (no depth needed).
  • exr_pixel — per-pixel inverse-depth blend of the EXR (current behaviour).
  • exr_grade — globally remap SHARP's own predicted depth to match the EXR's distribution (colour-grade style; immune to RGB↔EXR drift). grade_source=percentile matches value percentiles; grade_source=region matches per-region medians using the albedo AOV for segmentation (requires the albedo upload).

Response header X-Depth-Method echoes the method used.

Response: the .ply bytes (application/octet-stream, headers X-Num-Gaussians, X-F-Px), or JSON {ok, path, num_gaussians, f_px, image_size} when output_path is given.

Example:

curl -s http://<gpu-host>:8765/generate \
  -F image=@beauty.png -F depth=@depth.exr \
  -F focal_mm=24 -F aperture_mm=35 -F blend_alpha=0.4 \
  -o splat.ply

Wiring a client

Point your client at the service — e.g. an env var it reads:

SHARP_BACKEND_URL=http://<gpu-host>:8765