Skip to content

Forward telemetry parameters to the kernel binding - #925

Merged
jay-xiao446 merged 17 commits into
mainfrom
jay/disable-python-telemetry-kernel
Aug 28, 2026
Merged

Forward telemetry parameters to the kernel binding#925
jay-xiao446 merged 17 commits into
mainfrom
jay/disable-python-telemetry-kernel

Conversation

@jay-xiao446

@jay-xiao446 jay-xiao446 commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Forward kernel-owned telemetry configuration into databricks_sql_kernel.Session: driver/runtime identity, telemetry_enabled, and telemetry_batch_size.
  • Set kernel telemetry_enabled from enable_telemetry; keep force_enable_telemetry as a Python wrapper-only feature-flag bypass concept.
  • Disable Python-side connector telemetry for use_kernel=True connections so wrapper telemetry does not duplicate kernel-owned telemetry.
  • Add coverage that Python wrapper telemetry is disabled on the kernel path while the kernel-bound telemetry parameter follows enable_telemetry.

Tests

  • .venv/bin/python -m pytest tests/unit/test_telemetry.py -q
  • .venv/bin/python -m pytest tests/unit/test_session.py -q
  • .venv/bin/python -m pytest tests/unit/test_kernel_client.py -q

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ No issues identified by the review bot.

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — the core gate (is_telemetry_enabled returns False for kernel connections, via a safe getattr) is correct and well-tested, and session.use_kernel is guaranteed set before it's read. One low-severity gap: the connection-failure telemetry path (connection_failure_log) is not gated on use_kernel, so a failed kernel connect still emits a Python-side failure log.

Other findings

  • 🔵 Low — This PR disables Python-side telemetry for kernel connections by short-circuiting is_telemetry_enabled (telemetry_client.py:125). However, the connection-failure telemetry path is a separate code path that bypasses is_telemetry_enabled entirely: when session.open() raises, connection_failure_log is invoked and gated only on enable_telemetry (defaulting to True), never on use_kernel. So a kernel connection that fails to open will still emit a Python-side failure log — inconsistent with the PR's stated goal of disabling Python telemetry for kernel connections (and potentially duplicating what the kernel reports). If failure-before-kernel-open telemetry is intentionally still desired on the Python side, this is fine as-is; otherwise consider passing use_kernel through and skipping connection_failure_log on that path.

@jay-xiao446
jay-xiao446 force-pushed the jay/disable-python-telemetry-kernel branch from ae59e2c to 449d175 Compare August 21, 2026 20:43
@jay-xiao446 jay-xiao446 changed the title Disable Python telemetry for kernel connections Disable Python wrapper telemetry for kernel connections Aug 21, 2026
@jay-xiao446 jay-xiao446 changed the title Disable Python wrapper telemetry for kernel connections Forward telemetry parameters to the kernel binding Aug 21, 2026

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 2 Low

Looks good overall — the change cleanly disables Python-side telemetry on kernel connections and threads phase-7 identity/telemetry kwargs into the kernel Session, with solid regression tests (including the force-enabled case). Two low-severity notes: an identity-vs-truthiness mismatch between the telemetry bypass and backend routing, and a potential wheel-version compat gap since the new kwargs are passed unconditionally against a ^0.2.0 pin.

Comment thread src/databricks/sql/telemetry/telemetry_client.py Outdated
Comment thread src/databricks/sql/backend/kernel/client.py

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 2 Low

Looks good overall — the kernel-path telemetry forwarding and the is_telemetry_enabled short-circuit are correct, defaults line up between connect()/session.py/_kernel_telemetry_kwargs, and the new tests are meaningful. Two low-severity notes: (1) the connection-failure telemetry path isn't gated on use_kernel, so the Python wrapper still emits failure telemetry on the kernel path; (2) the phase-7 Session kwargs are forwarded unconditionally while the kernel dependency floor stays ^0.2.0, a potential connect-time break if an older wheel is installed.

Other findings

  • 🔵 Low — The kernel-path guard in is_telemetry_enabled disables Python-side telemetry only for the successful connection path. When a use_kernel=True connection fails during session.open(), this connection_failure_log(...) call still fires with enable_telemetry=kwargs.get("enable_telemetry", True) — it is not gated on use_kernel. So the Python wrapper still emits a connection-failure telemetry event on the kernel path, which is the kind of wrapper-owned telemetry this PR set out to suppress for use_kernel. It may be intentional (the kernel session never opened, so it couldn't emit its own failure telemetry), but if the goal is "no Python-side connector telemetry on the kernel path," this branch is an unguarded exception worth confirming.

Comment thread src/databricks/sql/backend/kernel/client.py

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — a clean, well-tested change. Python-side telemetry is correctly disabled for kernel connections (session is assigned before the is_telemetry_enabled check), and kernel telemetry_enabled follows enable_telemetry as intended. One low note: the kernel path's default telemetry_batch_size differs from the Thrift/SEA path when the caller leaves it unset.

Comment thread src/databricks/sql/session.py Outdated
Addresses:
  - #3833615986 at src/databricks/sql/telemetry/telemetry_client.py:125
  - #3833615990 at src/databricks/sql/backend/kernel/client.py:176
  - #3833661667 at src/databricks/sql/backend/kernel/client.py:419
  - #3858839800 at src/databricks/sql/session.py:228
  - #3876277610 at src/databricks/sql/session.py:260

Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — the telemetry wiring is correct. Session.__init__ sets use_kernel before Connection calls is_telemetry_enabled, so the kernel-path telemetry suppression and the forwarded kernel kwargs (identity, telemetry_enabled, batch size, circuit breaker) are consistent with the described design and well covered by tests. One low-severity cleanup: two leftover probe files were committed.

Comment thread tests/unit/test_zz_kernel_sig_probe.py Outdated
Signed-off-by: Jay Xiao <jay.xiao@databricks.com>
Signed-off-by: Jay Xiao <jay.xiao@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — the telemetry parameter forwarding is coherent and well covered by tests. The use_kernel opt-out in is_telemetry_enabled (telemetry_client.py:125) uses getattr(..., False), which correctly handles both real sessions (attribute set in _create_backend before the check at client.py:432) and the MagicMock-based tests. One minor default-parity inconsistency noted inline (Low). Separately, note that the connection_failure_log path (client.py:404-419) is not gated on use_kernel and would still emit Python-side failure telemetry for a failed kernel connection — a small gap in the "no duplicate kernel-path telemetry" goal, though arguably acceptable since it's the failure path.

Comment thread src/databricks/sql/session.py
Addresses:
  - #3876849398 at src/databricks/sql/session.py:276

Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — clean, well-tested change forwarding telemetry config to the kernel Session and disabling duplicate wrapper telemetry on the kernel path. Option threading, defaults (enable_telemetry→False, batch_size/circuit_breaker parity), force_enable_telemetry semantics, and the getattr guard on connection.session all check out, and session creation precedes telemetry init so the new check is safe. One low-severity completeness note: the connect-time connection_failure_log path is not gated by the new kernel opt-out.

Comment thread src/databricks/sql/telemetry/telemetry_client.py
Addresses:
  - #3876884856 at src/databricks/sql/telemetry/telemetry_client.py:125

Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — clean, well-commented, and well-tested telemetry forwarding into the kernel Session. The use_kernel gate in is_telemetry_enabled is safe (session.use_kernel is set in _create_backend before the check at client.py:436), key names line up across session.py/client.py/kernel/client.py, and the intentional default divergences (enable_telemetry False on kernel, force_enable_telemetry as wrapper-only) are documented and covered by tests. One low-severity observability note on the failure-path suppression is filed inline; note also that force_enable_telemetry=True has no effect on the kernel path (wrapper off + kernel follows only enable_telemetry), which is intentional but may surprise users.

Comment thread src/databricks/sql/client.py
Signed-off-by: Jay Xiao <jay.xiao@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — well-scoped and thoroughly tested; the kwarg names, defaults, and the auth/common.py:55 citation all check out, and the session→client ordering means session.use_kernel is set before is_telemetry_enabled reads it. One low note: the kernel path forwards enable_telemetry as None when unset (deferring to a kernel default) while the wrapper path gates the same case behind a server feature flag, which may diverge in the never-opted-in case.

Comment thread src/databricks/sql/session.py
Addresses:
  - #3877217286 at src/databricks/sql/session.py:263

Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks good — clean, well-tested change forwarding telemetry config to the kernel binding and suppressing wrapper-side telemetry on the use_kernel path. Parity/default claims in the comments verify against the code, and there's no circular-import issue. One low-severity note about the explicit enable_telemetry=True case relying on an unverifiable kernel-side feature-flag gate.

Comment thread src/databricks/sql/session.py
Addresses:
  - #3877271142 at src/databricks/sql/session.py:275

Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Solid, well-tested change — telemetry forwarding, kernel-path suppression of wrapper telemetry, and the is_telemetry_enabled short-circuit all look correct, and self.session is assigned before is_telemetry_enabled(self) reads session.use_kernel. One medium concern: the kernel telemetry_circuit_breaker_enabled default of True diverges from the wrapper's effective default of False, contradicting the parity rationale in the comment.

Comment thread src/databricks/sql/session.py
Addresses:
  - #3877391550 at src/databricks/sql/session.py:290

Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium

Mostly solid — the telemetry short-circuit (is_telemetry_enabled → False for use_kernel), the failure-log suppression at client.py:418, and the enable_telemetry/telemetry_batch_size/circuit-breaker mapping all look correct and are well-tested. One medium concern: the new phase-7 Session identity kwargs are forwarded unconditionally to the real PyO3 constructor while the kernel wheel pin stays at ^0.2.0 and every test mocks Session, so nothing proves the installed wheel accepts them.

Comment thread src/databricks/sql/backend/kernel/client.py
Addresses:
  - #3877431848 at src/databricks/sql/backend/kernel/client.py:180

Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Medium · 1 Low

Mostly solid, well-tested telemetry-forwarding change — 1 medium concern worth addressing. The kwarg-gating helper falls open on introspection failure, which could break every use_kernel connection against a non-introspectable PyO3 binding (the test only covers a Python-function fake, so this path is unverified). Also an empty _scratch_ test file looks accidentally committed.

Other findings

  • 🔵 Low — This appears to be accidentally-committed scratch debris: a brand-new, empty file named _scratch_kernel_kwargs_probe.py under tests/unit/. It contains no code and its _scratch_ name signals a throwaway probe. pytest won't collect it (name doesn't match test_*), so it's harmless functionally, but it shouldn't land in the repo. Please delete it before merge.

Comment thread src/databricks/sql/backend/kernel/client.py Outdated
Addresses:
  - #3877505593 at src/databricks/sql/backend/kernel/client.py:194

Signed-off-by: peco-engineer-bot[bot] <peco-engineer-bot[bot]@users.noreply.github.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Low

Looks solid — the telemetry-forwarding wiring is coherent and the kwarg-gating (_kernel_session_accepts_kwarg falling closed on introspection failure), the use_kernel short-circuit in is_telemetry_enabled, and the failure-log suppression in client.py are all consistent with each other and well covered by new tests. One cleanup: an empty _scratch_ file was committed. Minor nit (not filed inline): _kernel_telemetry_kwargs re-runs inspect.signature(_kernel.Session) once per candidate kwarg (~14x per open_session); caching the parameter set once would be marginally cleaner, though the cost is negligible since it only runs at session open.

Other findings

  • 🔵 Low — This appears to be a leftover debugging artifact: the filename is prefixed with _scratch_ and the file is committed empty (0 bytes). It adds no test coverage and pollutes tests/unit/. Please drop it from the PR (git rm) — the actual kwarg-gating coverage already lives in test_kernel_client.py and test_telemetry.py.

Signed-off-by: Jay Xiao <jay.xiao@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: 1 Nit

Looks good — the telemetry-forwarding wiring is correct and unusually well-tested. The intentional divergences from the Thrift/SEA paths (unset enable_telemetry/circuit-breaker deferring to the kernel, batch-size defaulting to DEFAULT_BATCH_SIZE, wrapper telemetry suppressed on the kernel path) are documented and verified by unit tests, and the fixed-signature/non-introspectable PyO3 gate is exercised directly. Only one minor efficiency nit (repeated inspect.signature calls).

Other findings

  • ⚪ Nit — _kernel_telemetry_kwargs calls _kernel_session_accepts_kwarg(name) once per candidate (up to 14 times), and each call re-runs inspect.signature(_kernel.Session). Since the signature is identical across all candidates, it could be introspected once (e.g. resolve the parameter set / VAR_KEYWORD presence a single time, then filter). This runs only once per session-open so the cost is negligible — flagging as a preference, not a defect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engineer-bot Maintainer-applied gate: triggers engineer-bot (bug-fix on issue / take-over on PR).

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants