Fix KQL join pitfalls and adopt lookup for dimension enrichment

[RolandKrummenacher]

📝 Changes

Follow-up to a repo-wide review of KQL join vs lookup best practices (~880 join/lookup usages examined across hub database scripts, workbooks, ARG recommendation queries, the query catalog, and docs). This PR fixes the high-severity correctness bugs, adopts lookup where it is the documented best practice, and adds a lint rule so bare joins cannot come back. One commit per surface:

1. Workbooks (SavingsPlan, AHB ×2) — correctness fixes:
   • SavingsPlan summary/details: the final resourcecontainers join had no kind and defaulted to innerunique, deduplicating results by subscription and silently dropping all but one savings plan recommendation per subscription. Now kind=inner.
   • AHB "VM Latest Change Last 7 days": joined the resourcechanges record id against resources.id, which never match, so the tile was always empty. Now joins properties.targetResourceId (lowercased both sides) with kind=inner so mv-expanded license changes are preserved.
   • Get-SQL-AHB-Disabled/Enabled: bare join on VMName dropped SQL VMs with duplicate names (innerunique) and never matched uppercase names (left original case vs right tolower). Now joins properties.virtualMachineResourceId against the VM resource id.
2. ARG recommendation queries + finops-alerts logic app — every bare | join ( made an explicit kind=inner; Recommendations-Microsoft-SQLVMsWithoutAHB gets the same virtualMachineResourceId join-key fix as above. docs/deploy/finops-alerts-*.json are generated and intentionally untouched; the next release build picks up the bicep change.
3. Hub ingestion scripts — the v1_0/v1_2 EA transforms enriched cost rows from the open-data dimension tables (PricingUnits, Regions, ResourceTypes, Services) with join kind=leftouter; converted to lookup kind=leftouter (broadcasts the small dimension, no duplicated key columns; no downstream references to the suffixed columns existed). Also guards the Services enrichment against row fan-out with summarize take_any(...) by x_ResourceType — this was an active bug: Services.csv has 30 duplicate resource-type keys (up to ×31 for microsoft.sql/locations), so cost rows for those types were being multiplied.
4. Query catalog — tagging-policy-compliance, storage-tier-distribution, macc-consumption-vs-commitment: fact-to-small-dimension joins converted to lookup.
5. Docs — compute.md AHB examples (bare joins + the same VMName case bug, published as copy-paste guidance), commitment-coverage examples now teach lookup, invalid join ... on 1 == 1 percent-of-total examples rewritten with toscalar(), fullouter examples now coalesce their join keys so baseline-only rows keep their dimension values, and networking.md examples fixed (bare joins + an invalid resource table name that made the idle public IP example fail outright).
6. Lint rule — new Tests/Lint/KqlJoinKinds.Tests.ps1 scans every KQL-carrying surface and fails on any bare | join without an explicit kind=. The 48 remaining pre-existing bare joins (4 workbook files, all with unique left keys today) are baselined per file as a ratchet: counts can only go down (tracked in Workbooks: make remaining implicit and innerunique ARG join kinds explicit #2228). The SQL DB optimization runbook and networking.md were brought to zero in this commit.

✅ Validation

All changes that can run against live services were validated — three rounds documented in the comments:

• Round 1: ARG innerunique default proven with a controlled probe; Services fan-out confirmed from open data; ingestion lookup chain executed against a hub ADX cluster; old-vs-new catalog queries equivalent on 1.16M cost rows.
• Round 2: logic app queries, all edited doc examples (old on 1 == 1 versions fail live with General_BadRequest; new toscalar() versions return correct totals), fullouter fixes eliminate the empty-key rows, and all 8 edited transform/HubSetup functions compile-checked against real schemas.
• Round 3: two production-scale hub deployments (36.6M / 28.3M rows) — the pre-fix Services join is deployed in their live transforms; A/B on populated dimension tables shows 1 row → 5/4/2 rows with the old join vs 1:1 with the new lookup; no existing data corruption found (both ingest via a FOCUS path predating the enrichment). Lint suite: 161/161 passing.

⚠️ Known follow-ups (tracked as issues)

• Optimization Engine: KQL join patterns that can inflate counts and double-count cost #2226 — Optimization Engine join-correctness findings (anti-join emulation, missing dimension dedup, replication fan-out). Its two bare joins are fixed here so the lint starts clean.
• SavingsPlan workbook queries exceed ARG's documented 3-join limit #2227 — SavingsPlan workbook queries have 4 joins, above ARG''s documented 3-join guidance (pre-existing; ARG accepted them in testing).
• Workbooks: make remaining implicit and innerunique ARG join kinds explicit #2228 — 48 baselined bare joins + ~75 explicit kind=innerunique tag-filter semi-joins across workbooks (style only; left keys unique today).
• open-data README region refresh query could use lookup (maintenance-only query, intentionally left as is; not issue-worthy).

🤖 Generated with Claude Code

[RolandKrummenacher]

✅ Validation results (FTK test tenant + ftk-dev ADX cluster)

ARG semantics probe — confirmed ARG bare join defaults to innerunique: a 40-row left side with one distinct key returns 1 row bare vs 40 rows with kind=inner. The bug class this PR fixes is real.

Services open-data fan-out (commit 3) — confirmed against src/open-data/Services.csv: 30 resource types have duplicate rows, e.g. microsoft.apimanagement/service ×4 and microsoft.sql/locations ×31. Before this PR, the v1_0/v1_2 EA ingestion transforms multiplied every cost row for those resource types by that factor. The take_any dedup eliminates this.

Hub ingestion lookup chain (commit 3) — the exact edited fragment (PricingUnits → Regions → ResourceTypes → Services ×2) executed against the ftk-dev Ingestion database: compiles, returns the expected schema, no duplicated key columns, non-key conflicts suffixed identically to join (so downstream projections are unaffected).

Catalog queries (commit 4) — old vs new run back-to-back on ftk-dev Hub DB (1,162,170 Costs() rows):

• macc-consumption-vs-commitment: byte-identical
• storage-tier-distribution: identical rows (order differs; query has no order by)
• tagging-policy-compliance: identical to 9 decimal places (float summation order)

Recommendation queries + AHB/SavingsPlan workbook queries (commits 1-2) — all executed via the ARG REST API with parameters substituted: parse and run cleanly, incl. the 4-join SavingsPlan queries (ARG accepted 4 joins). Old vs new SavingsPlan counts match in this tenant (1 recommendation per subscription, so innerunique happens to coincide — the probe above shows the general case). resourcechanges.properties.targetResourceId verified populated and resource-id-shaped on 376/376 rows, confirming the new VM Latest Change join key; the old key (resourcechanges record id) can never match a resource id.

Not validated: row-level semantics of the SQL VM / public IP / app gateway queries (test tenant has no IaaS resources — all returned 0 rows, syntax-only), and the logic app queries (validated indirectly via identical shapes + bicep build).

🤖 Generated with Claude Code

[RolandKrummenacher]

✅ Validation round 2 — full coverage of everything testable

Completing the earlier validation, every remaining testable change has now been run against live services (ARG REST API on the FTK test tenant; ftk-dev ADX cluster for hub KQL):

ARG (now all executed):

• finops-alerts logicApp.bicep: both edited queries extracted from the bicep, variables('resourcesTable') substituted, executed — PASS.
• compute.md Windows-AHB + SQL-AHB examples — PASS.
• networking.md backendless app gateway + idle public IP examples — the idle-IP query failed on first run and exposed a pre-existing bug: the published example referenced table resource instead of resources (DisallowedLogicalTableName), so it never ran at all. Fixed in a follow-up commit; now PASS.

Hub ADX cluster (old vs new, Hub DB, 1.16M cost rows):

• compute.md commitment-coverage queries: join→lookup versions return identical row counts (3 / 100).
• on 1 == 1 rewrites (finops-hub-database-guide.md, ftk-database-query.md): old versions fail with General_BadRequest (confirming the invalid predicate), new toscalar() versions pass with real data (6 / 44 rows).
• fullouter key-coalesce fixes: same row counts old vs new, but old had 1 (cost-spike) and 3 (service-deep-dive) rows with an empty key column; new versions have 0 — exactly the fix intent.

Transform functions (compile-checked against real schemas):

• All 6 edited ingestion transforms (ActualCosts/AmortizedCosts/Costs_transform_v1_0/v1_2) extracted from the scripts and executed with | take 0 against the Ingestion DB (raw tables present) — all compile.
• Both edited HubSetup_v1_2 functions (CommitmentDiscountUsage_v1_2, Costs_v1_2) compile against the Hub DB.

Mirrors: the three edited queries in Compute/AHB.workbook verified byte-identical to the live-tested AHB/AHB.workbook copies.

Remaining untestable in this environment: row-level semantics of the SQL VM / public IP / app gateway ARG queries (test tenant has no IaaS — all validated as executing with 0 rows).

🤖 Generated with Claude Code

[RolandKrummenacher]

✅ Validation round 3 — production-scale hub deployments

Additional read-only validation against two production FinOps hub deployments (36.6M and 28.3M cost rows; identities withheld). Key findings:

The Services fan-out fix (commit 3) is directly relevant to deployed hubs:

• Both hubs have the Services open-data table populated with the same 30 duplicate resource-type keys as src/open-data/Services.csv.
• Both hubs have the pre-fix join kind=leftouter (Services | ... project ...) deployed in their live ActualCosts/AmortizedCosts_transform_v1_0/v1_2 functions.
• Live A/B on one hub's populated dimension tables: the old join turns 1 input row into 5 rows (microsoft.security/pricings), 4 rows (microsoft.web/sites), 2 rows (microsoft.storage/storageaccounts); the new lookup+take_any chain returns exactly 1:1 with correct enrichment values.
• On one hub, 66% of the last 30 days' cost rows (585,710 of 880,341) carry a duplicate-key resource type — the blast radius if the legacy EA path were used.
• Two keys (microsoft.security/pricings ×5, microsoft.web/sites ×2) have genuinely different value tuples, so they would fan out even through the newer lookup (Services | distinct ...) variant — validating that this PR also converts those distinct-based lookups to take_any ... by x_ResourceType.

No existing data corruption found in either hub: both currently ingest via the FOCUS path, and their deployed Costs_transform_v1_2 predates the Services enrichment entirely (probe: 0 charges with service-variant duplicates in stored data on both hubs). The buggy join is deployed but not currently exercised — this PR defuses it before it is.

Equivalence re-confirmed on production data: storage-tier-distribution old (join) vs new (lookup) on real multi-tier storage data — identical rows, 0 value mismatches.

🤖 Generated with Claude Code

[Copilot]

🟡 Human review recommended

The PR changes KQL behavior across multiple production-impacting surfaces (ingestion, recommendations, workbooks) and also introduces linting coverage gaps that should be resolved before relying on the new guardrail.

Pull request overview

This PR addresses correctness and performance pitfalls in KQL usage across the FinOps Toolkit by making join semantics explicit (avoiding ARG/ADX defaults like innerunique) and adopting lookup for small-dimension enrichment where appropriate. It also introduces a lint rule intended to prevent reintroducing bare join operators.

Changes:

• Fixed workbook and ARG recommendation queries where implicit/bad joins caused dropped rows or empty results, and made join kinds explicit.
• Converted several ingestion/catalog “fact-to-dimension” enrichments from join to lookup, including deduping Services mappings to prevent fan-out.
• Added a Pester lint test to detect bare | join usages (missing kind=) with a per-file baseline/ratchet.

File summaries

File	Description
src/workbooks/optimization/SavingsPlan/SavingsPlan.workbook	Makes the final subscription enrichment join explicit to prevent innerunique row loss.
src/workbooks/optimization/Compute/AHB.workbook	Fixes join keys/kinds for AHB tiles and SQL VM enrichment to avoid mismatches and implicit dedupe.
src/workbooks/optimization/AHB/AHB.workbook	Mirrors AHB workbook join-key/kind fixes for correctness and consistency.
src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Recommendations/queries/Recommendations-Microsoft-VMsWithoutAHB.json	Makes the subscription join explicit (kind=inner) to avoid implicit innerunique.
src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Recommendations/queries/Recommendations-Microsoft-UnattachedPublicIPs.json	Makes the PIP enrichment join explicit (kind=inner).
src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Recommendations/queries/Recommendations-Microsoft-SQLVMsWithoutAHB.json	Fixes SQLVM↔VM join key (resourceId) and makes join kinds explicit.
src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Recommendations/queries/Recommendations-Microsoft-BackendlessAppGateways.json	Makes join kind explicit for backend pool summarization/enrichment.
src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_v1_2.kql	Uses lookup for dimension enrichment and dedupes Services mapping to prevent fan-out.
src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_v1_0.kql	Same lookup + Services dedupe improvements for v1_0 transforms.
src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/HubSetup_v1_2.kql	Dedupes Services lookup to avoid x_ResourceType fan-out.
src/templates/finops-alerts/modules/logicApp.bicep	Makes embedded ARG query join kinds explicit in the logic app definition.
src/templates/claude-plugin/agents/ftk-database-query.md	Rewrites percent-of-total example to avoid invalid join on 1 == 1, using toscalar().
src/templates/agent-skills/finops-toolkit/references/service-cost-deep-dive.md	Improves fullouter join examples by coalescing join keys and projecting away duplicates.
src/templates/agent-skills/finops-toolkit/references/cost-spike-investigation.md	Same fullouter coalesce pattern for baseline/spike comparison example.
src/queries/finops-hub-database-guide.md	Rewrites percent-of-total example using toscalar() instead of join on 1 == 1.
src/queries/catalog/tagging-policy-compliance.kql	Converts a leftouter enrichment join to lookup for a small dimension.
src/queries/catalog/storage-tier-distribution.kql	Converts an inner join to lookup for currency totals enrichment.
src/queries/catalog/macc-consumption-vs-commitment.kql	Converts joins to lookup for small, aggregated right sides.
src/powershell/Tests/Lint/KqlJoinKinds.Tests.ps1	Adds a lint test to detect bare `
src/optimization-engine/runbooks/recommendations/Recommend-SqlDbOptimizationsToBlobStorage.ps1	Makes join kind=inner explicit in embedded KQL.
docs-mslearn/toolkit/changelog.md	Updates changelog and ms.date to reflect join/lookup fixes.
docs-mslearn/best-practices/networking.md	Updates examples to use explicit join kinds and corrects a table name typo.
docs-mslearn/best-practices/compute.md	Updates examples to use lookup and fixes SQL VM join guidance to use resource IDs.

Review details

Suppressed comments (1)

src/powershell/Tests/Lint/KqlJoinKinds.Tests.ps1:70

• If docs-mslearn/toolkit is added to the scan targets, the current baseline needs an entry for the existing bare joins in docs-mslearn/toolkit/workbooks/customize-workbooks.md (currently 2 occurrences of | join () so the ratchet works as intended.

            'src/workbooks/optimization/AHB/AHB.workbook'         = 24
            'src/workbooks/optimization/Compute/AHB.workbook'     = 20
            'src/workbooks/optimization/Networking/Networking.workbook' = 3
            'src/workbooks/governance/workbook.json'              = 1
        }

• Files reviewed: 23/23 changed files
• Comments generated: 1
• Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.