Add KQL string matching and join/lookup rules to the coding guidelines

[RolandKrummenacher]

🛠️ Description

The coding guidelines cover general practice, content, and the changelog, but say nothing about KQL — the primary language of the hub. docs-wiki/Coding-guidelines.md, AGENTS.md, and CONTRIBUTING.md don't mention KQL or Kusto at all today. That's part of why the redundant tolower() comparisons and word-search contains usages accumulated across the ingestion scripts until #2213 / #2220 — and why bare joins relying on the innerunique default accumulated until #2225. Nothing in the project's written standards contradicted them.

This PR writes those rules down where both humans and AI assistants will find them.

docs-wiki/Coding-guidelines.md — a new ⚡ KQL section with two rule sets:

String matching (from #2213 / #2220):

• String comparison: why tolower() in comparison position is always wrong (every KQL string operator is already case-insensitive; the _cs forms are the case-sensitive ones), with an avoid/prefer table for the common rewrites — tolower(x) contains → has, tolower(x) == → =~, tolower(a) != tolower(b) → !~, indexof(...) >= 0 → has, and dynamic operands not needing a tostring() wrapper.
• When contains is required: framed as a semantic distinction (whole term vs. arbitrary substring) rather than a performance one, with the legitimate cases — needles fused inside a larger token (ConsumedUnit contains 'MB' also matching Mbps), word-stem matching, and fragments that never form a whole term. Also documents the term-index limits for punctuation and sub-3-character needles.
• Verify before you swap: per-row cross-tabulation on real data rather than aggregate counts (which can hide offsetting false positives and negatives), plus a pointer to the executable equivalence harness added in Use case-insensitive KQL operators instead of tolower() comparisons #2220.

Joins and lookups (from #2225):

• The innerunique trap: a bare | join deduplicates the left side on the join key — the root cause of real silent data loss fixed in Fix KQL join pitfalls and adopt lookup for dimension enrichment #2225 (savings plan recommendations collapsing to one row per subscription). Explicit kind= is mandatory.
• Canonical forms by intent: lookup (+ summarize take_any(...) by key dedup — distinct is not a per-key dedup) for dimension enrichment, leftsemi/inner for filtering, leftanti for exclusion (never leftouter + isempty(), which duplicates before it filters), fullouter with key coalescing for period comparisons, and shuffle hints for large-to-large joins.
• ARG is not ADX: live-probing every flavor gave a matrix that differs from the ARG docs in both directions — supported: inner, innerunique, leftouter, rightouter, fullouter; rejected: lookup, all semi/anti flavors, and in/!in with a subquery. Since leftanti doesn't exist there, the leftouter + isempty() emulation is documented as the one acceptable exclusion form in ARG (with a key-unique right side).

AGENTS.md — both rule sets are also stated inline under Coding Standards. That file already listed Bicep, PowerShell, markdown, and commit conventions but not KQL, and it's what both Claude Code (via CLAUDE.md) and Copilot (via .github/copilot-instructions.md) load. Stating it there means an assistant sees the rule without having to follow the link.

Merge order: the join/lookup rules reference KqlJoinKinds.Tests.ps1, which lands in #2225 — merge #2225 first so the enforcement pointer resolves (the text marks it as "added in #2225" either way).

No behavior change — documentation only. The guidance is already enforced in CI: string rules by HubsKqlOperators.Tests.ps1 (from #2220), join rules by KqlJoinKinds.Tests.ps1 (from #2225). This PR documents the reasoning so contributors know why before the test tells them no.

📋 Checklist

🔬 How did you test this change?

• 🤏 Lint tests — markdownlint-cli2 with the repo config: 0 issues

📦 Deploy to test?

Not applicable — documentation only.

🙋‍♀️ Do any of the following that apply?

• 🚨 This is a breaking change.

📑 Did you update docs/changelog?

• N/A — docs-wiki is contributor documentation, not part of the release changelog. It syncs to the GitHub wiki automatically on merge to dev.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Documents project-specific KQL string-matching rules in the contributor coding standards so recurring anti-patterns (e.g., tolower() in comparisons and overuse of contains) are prevented via clear guidance alongside existing CI enforcement.

Changes:

• Adds a new ⚡ KQL section to docs-wiki/Coding-guidelines.md covering case-insensitive operators, has vs contains, and validation guidance.
• Adds a concise KQL rule summary to AGENTS.md under Coding Standards so assistants and contributors see it inline.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
docs-wiki/Coding-guidelines.md	Introduces detailed KQL string matching guidance (operators, has vs contains, and verification steps).
AGENTS.md	Adds an inline KQL standards summary referencing the coding guidelines and existing CI enforcement.

Suppressed comments (1)

docs-wiki/Coding-guidelines.md:65

• This row states that =~ accepts a dynamic operand directly. That can be true for scalar dynamic values, but it is not universally safe if the field can be an object/array (or otherwise not string-like), and may produce confusing comparisons. Suggest qualifying the guidance to avoid implying tostring() is never needed for dynamic fields.

| `tostring(Dyn.Field) =~ 'true'` | `Dyn.Field =~ 'true'` | These operators accept a dynamic operand directly |

[RolandKrummenacher]

Good catch on the dynamic-operand row — it was too absolute. Qualified in 150c52d.

Verified the actual behavior on a live cluster first, and the result is a bit different from what the comment assumed: tostring() is not the missing piece. Raw and tostring()-wrapped comparisons return identical results for every payload shape, objects and arrays included:

d.v	tostring(d.v)	d.v =~ 'true'	tostring(d.v) =~ 'true'	d.v has 'true'	tostring(d.v) has 'true'
"true" (string)	true	true	true	true	true
true (bool)	true	true	true	true	true
1 (number)	1	false	false	false	false
{"nested":"true"} (object)	{"nested":"true"}	false	false	true	true
["true"] (array)	["true"]	false	false	true	true
missing key	(empty)	false	false	false	false

So the real hazard isn't a missing cast — it's that a non-scalar gets compared against its JSON serialization, where has matches a value nested anywhere in the text ({"nested":"true"} has 'true' → true, while =~ → false). Adding tostring() produces the same JSON text and the same surprising result, so recommending it would give false confidence.

The table row is now scoped to scalar dynamic values, with an IMPORTANT note covering the JSON-serialization behavior, why tostring() doesn't help, and what to use instead — extract the member you mean (Dyn.Field.nested), or array_index_of() / set_has_element() for arrays. Every claim in that note was executed against a cluster.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (3)

docs-wiki/Coding-guidelines.md:113

• The doc references src/powershell/Tests/Lint/KqlJoinKinds.Tests.ps1 / KqlJoinKinds.Tests.ps1, but that file does not exist in the repository. This makes the “enforced by …” guidance untraceable for contributors; either add the referenced test or update the text to point at the correct enforcement location (or remove the specific path).

> **Azure Resource Graph is not ADX.** ARG queries (workbooks, recommendation queries, the alerts logic app) support no `lookup`, no join hints, and have a documented limit of 3 joins per query. Live-probing every flavor (2026-08) gave a matrix that differs from the [ARG docs](https://learn.microsoft.com/azure/governance/resource-graph/concepts/query-language) in both directions: **supported** – `inner`, `innerunique`, `leftouter`, `rightouter`, `fullouter`; **rejected with `InvalidQuery`** – `lookup`, `leftsemi`, `leftanti`, `rightsemi`, `rightanti`, and `in`/`!in` with a subquery. Two consequences: the explicit-`kind` rule is the only defense against the `innerunique` default (verified live: a 40-row left side with one distinct key returns 1 row from a bare join, 40 from `kind=inner`), and **exclusions in ARG must use the `leftouter` + `where isempty(...)` emulation** – keep the right side key-unique (`distinct <key>` only) so it cannot fan out. `KqlJoinKinds.Tests.ps1` fails the build if an ARG surface uses `lookup` or a semi/anti flavor.

**Verify before you swap.** Converting `join` to `lookup` drops the duplicated key columns (`Key1`) from the output – confirm nothing downstream references them. Adding a dimension dedup changes which row wins for duplicate keys – confirm the surviving values are equivalent (or pick deterministically with `arg_max()`). For conversions on hub transforms, run the old and new pipeline over the same data and compare row counts *and* values, not just execution success.

These rules are enforced on every pull request by `src/powershell/Tests/Lint/KqlJoinKinds.Tests.ps1`: any `join` without an explicit `kind=` fails the build, across every KQL-carrying surface (hub scripts, query catalog, workbooks, recommendation queries, the alerts logic app, optimization engine, and published docs). Pre-existing bare joins are baselined per file as a ratchet – counts can only go down, and lowering the baseline is enforced when a file is cleaned up (#2228 tracks the backlog).

docs-wiki/Coding-guidelines.md:105

• This join guidance uses escaped pipes (\|) inside inline code in the table. When someone copies from the raw markdown (or if the renderer doesn’t strip the escape), they’ll paste \| join ..., which is invalid KQL. Consider formatting these examples so the text is copy/paste-safe without backslashes (for example, avoid leading pipes in the snippet, or use a pipe representation that doesn’t require escapes in tables).

| `\| join (T) on Key` | `\| join kind=... (T) on Key` | Bare join = `innerunique`: left side deduplicated per key, rows silently dropped. State the intent, always |
| `\| join kind=leftouter (SmallDim) on Key` | `\| lookup kind=leftouter (SmallDim) on Key` | Fact-to-dimension enrichment is what `lookup` is for: broadcast, no duplicated key column. (ADX / Log Analytics only – ARG has no `lookup`) |
| `lookup (Dim \| distinct Key, Col1, Col2) on Key` | `lookup (Dim \| summarize take_any(Col1), take_any(Col2) by Key) on Key` | `distinct` over multiple columns still yields >1 row per key when the other columns differ – fact rows multiply |
| `join kind=leftouter (X) on K \| where isempty(K1)` | `join kind=leftanti (X) on K` | `leftanti` is duplicate-proof and never materializes right-side columns |
| `summarize Total=... \| join ... on 1 == 1` | `let Total = toscalar(...)` | `on 1 == 1` is not valid KQL (verified: fails with `General_BadRequest`); `toscalar()` also avoids a second full-table scan |

AGENTS.md:216

• This bullet references src/powershell/Tests/Lint/KqlJoinKinds.Tests.ps1, but that file does not exist in the repository. Either add the referenced test or update the bullet to point to the actual enforcement location (or remove the specific path) so contributors can verify the rule.

- KQL joins: Never write a bare `| join` — always state `kind=` explicitly (the `innerunique` default deduplicates the left side and silently drops rows; enforced by `src/powershell/Tests/Lint/KqlJoinKinds.Tests.ps1`). Prefer `lookup` over `join` for enriching a fact table from a small dimension (ADX/Log Analytics only — not available in Azure Resource Graph), dedupe the dimension side with `summarize take_any(...) by <key>` (not `distinct`), and use `kind=leftanti` for exclusions instead of `kind=leftouter` + `where isempty(...)` — except in ARG, which rejects `lookup` and all semi/anti flavors (verified live), so exclusions there use the `leftouter` + `isempty()` emulation with a key-unique right side. See the "Joins and lookups" section of `docs-wiki/Coding-guidelines.md`

[RolandKrummenacher]

Addressed the three suppressed review comments in 3c658f4:

• KqlJoinKinds.Tests.ps1 does not exist in this repo (Coding-guidelines.md:113, AGENTS.md:216) — correct: the test lands in Fix KQL join pitfalls and adopt lookup for dimension enrichment #2225 (src/powershell/Tests/Lint/KqlJoinKinds.Tests.ps1, 316 passing tests on that branch). Both references now say "added in Fix KQL join pitfalls and adopt lookup for dimension enrichment #2225", the ARG callout points at "the join lint (below)" instead of a bare filename, and the PR description calls out the merge order (Fix KQL join pitfalls and adopt lookup for dimension enrichment #2225 first).
• Escaped pipes in the avoid/prefer table (Coding-guidelines.md:105) — reworded every snippet so no pipe is needed at all (e.g. "distinct Key, Col1, Col2 as the right side of a lookup/join" instead of an inline piped query). The table is now copy-paste-safe from raw markdown and reads better rendered, too.

🤖 Generated with Claude Code