Skip to content

Add StarRocks Arrow Flight SQL plugin with direct backend fetch#120

Open
Nitin Bhakar (nitinbhakar) wants to merge 5 commits into
mainfrom
feature/starrocks-direct-be-fetch
Open

Add StarRocks Arrow Flight SQL plugin with direct backend fetch#120
Nitin Bhakar (nitinbhakar) wants to merge 5 commits into
mainfrom
feature/starrocks-direct-be-fetch

Conversation

@nitinbhakar

@nitinbhakar Nitin Bhakar (nitinbhakar) commented Jul 14, 2026

Copy link
Copy Markdown

Summary

  • Adds a new StarRocks plugin (plugins/starrocks) that connects via Arrow Flight SQL and executes queries/statements against a configured cluster.
  • When the FE's GetFlightInfo response includes endpoints with distinct BE locations (rather than LocationReuseConnection/the FE's own address), the plugin dials each BE directly and redeems tickets in parallel via DoGet, instead of funneling every row back through the FE. Falls back to the FE connection when no distinct BE location is present (e.g. StarRocks' own arrow_flight_proxy feature is enabled).
  • Maps Arrow types to Heimdall's result column types, including ARRAY/MAP/STRUCT (StarRocks LIST/MAP/STRUCT), which are JSON-encoded rather than stringified.
  • Adds request/success/latency telemetry counters on the Execute entry point.

Dependency updates (Wiz findings)

  • Bumped google.golang.org/grpc (pulled in by the Arrow Flight SQL client) to v1.79.3, fixing CVE-2026-33186 (Critical). Compatible with Go 1.24.
  • Bumped go.opentelemetry.io/otel to v1.41.0, fixing CVE-2026-29181 (High). Compatible with Go 1.24.
  • Left golang.org/x/crypto and golang.org/x/net at their natural minimal-version-selection floor (v0.46.0, v0.48.0) rather than bumping to the fully-fixed versions (v0.52.0, v0.55.0), and kept go.mod's go directive at 1.24.6. Both packages' fixed versions require Go >=1.25 upstream — there's no version that's both fixed and Go-1.24-compatible. Per team decision, staying on Go 1.24 took priority over fixing these two specific findings in this PR. They are pre-existing on main today at even older vulnerable versions (v0.42.0, v0.44.0 respectively), so this isn't a new regression introduced here — genuinely fixing them requires a repo-wide Go 1.25 upgrade, tracked as separate follow-up work.
  • Wiz will likely re-flag x/crypto and x/net as Critical vulnerabilities on this PR; replying with #wiz_ignore + this justification on those specific findings.

Test plan

  • go build/go vet on internal/pkg/object/command/starrocks and the plugins/starrocks plugin (including -buildmode=plugin) against main
  • Verified end-to-end against a real StarRocks cluster via local Heimdall (SHOW TABLES, COUNT(*) over 500M+ rows, and confirmed direct-BE ticket redemption engages when the FE advertises a distinct BE location)
  • Wiz SAST/Secret/IaC/Data/Software Management scanners pass; Vulnerability Scanner passes for grpc/otel, x/crypto and x/net findings acknowledged/ignored per above
  • Direct-BE-fetch path requires StarRocks' arrow_flight_proxy disabled and BE/CN nodes network-reachable from Heimdall's runtime — not yet enabled in any cluster config (tracked as follow-up infra work); with the proxy enabled (current default), this plugin behaves identically to always routing through the FE

🤖 Generated with Claude Code

Connects to StarRocks via Arrow Flight SQL and, when the FE returns
distinct per-backend endpoint locations, redeems each ticket by
dialing the BE directly and fanning the fetches out in parallel
instead of funneling every row back through the FE.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@wiz-55ccc8b716

wiz-55ccc8b716 Bot commented Jul 14, 2026

Copy link
Copy Markdown

Wiz Scan Summary

Scanner Findings
Vulnerability Finding Vulnerabilities -
Data Finding Sensitive Data -
Secret Finding Secrets -
IaC Misconfiguration IaC Misconfigurations -
SAST Finding SAST Findings 1 Low
Software Management Finding Software Management Findings -
Total 1 Low

View scan details in Wiz

To detect these findings earlier in the dev lifecycle, try using Wiz Code VS Code Extension.

…rsion

- google.golang.org/grpc was pulled in below v1.79.3, affected by CVE-2026-33186
- dialFlightClient's TLS config lacked an explicit MinVersion (CWE-327)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Comment thread go.mod
module github.com/patterninc/heimdall

go 1.24.6
go 1.25.0

@prasadlohakpure prasadlohakpure Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If go upgrade is not required, can we hold this for now and handle it separately? It will require some thorough testing. It will also require changes in custom plugins build besides the standard one provided as part of heimdall.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reverted — go.mod is back to 1.24.6. Traced the actual driver: only grpc (Critical CVE-2026-33186) and otel (High CVE-2026-29181) needed bumping and both are Go 1.24-compatible, so those stayed fixed. x/crypto and x/net's fixed versions require Go >=1.25 upstream (no version satisfies both), so those two are left at their natural resolved versions for now — same as main today, just not a regression from this PR. Wiz will still flag those two; replying to those findings with the ignore justification and tracking the real fix as separate follow-up work.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wanted to lay out the full chain here since it wasn't obvious why this touched go.mod at all:

  1. Wiz flagged a Critical CVE (CVE-2026-33186) in google.golang.org/grpc, pulled in transitively by the new Arrow Flight SQL client this PR adds. Fix: bump to v1.79.3.
  2. Bumping grpc (+ go mod tidy) pulled its own transitive floor up for a few other indirect deps — go.opentelemetry.io/otel, golang.org/x/crypto, and golang.org/x/net all got nudged to newer minimums to satisfy grpc's own requirements.
  3. Those newer (but not newest) versions were themselves still below Wiz's fix thresholds, so Wiz flagged 3 more findings: otel <1.41.0 (High), x/crypto <0.52.0 (Critical, 13 CVEs), x/net <0.55.0 (Critical, 2 CVEs).
  4. otel's fix (v1.41.0) only requires Go 1.24 — no conflict, fixed cleanly.
  5. But x/crypto's fix (>=v0.50.0) and x/net's fix (v0.55.0) both require Go >=1.25 in their own go.mod. I checked every published version of both modules — neither project backports security fixes to older release lines, they only ship forward sequentially. So there's no version of either package that's both fixed and Go-1.24-compatible; the go.mod bump wasn't a choice, it was the unavoidable consequence of chasing those two fixes all the way through.

Given your comment, we've reverted: go.mod is back to 1.24.6, grpc + otel stay fixed (both compatible), and x/crypto/x/net are left at their natural resolved versions (v0.46.0/v0.48.0) rather than the fully-fixed ones. Wiz will keep flagging those two as a result — but they're pre-existing on main today at even older versions (v0.42.0/v0.44.0), so this PR isn't introducing a new regression, just not fixing something that already exists repo-wide. Actually fixing those two for real needs a deliberate, separate Go 1.25 upgrade — happy to help scope that as its own piece of work if useful.

go.opentelemetry.io/otel < 1.41.0 (CVE-2026-29181), golang.org/x/crypto
< 0.52.0 (13 CVEs), and golang.org/x/net < 0.55.0 (CVE-2026-25680,
CVE-2026-39821) were pulled in transitively by the grpc bump. The
crypto fix version requires go >= 1.25, hence the go.mod bump;
GOTOOLCHAIN=auto (unpinned in this repo) lets CI's Go 1.24.5 fetch
1.25 automatically rather than needing a workflow change.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@nitinbhakar Nitin Bhakar (nitinbhakar) force-pushed the feature/starrocks-direct-be-fetch branch from 5f7f9f9 to cfa0a26 Compare July 15, 2026 10:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants