Semperis Lightning 4.1.0: merge CCF into existing offer (Function App + CCF)

[MitchellGulledge3]

Tested 4.1.0 package for Semperis Lightning.

What this is

Update to the existing Content Hub offer azure-sentinel-solution-semperislightning (Netskope/SailPoint pattern):

• Azure Functions connector → *_CL
• CCF connector → *V2_CL

Portal Connect succeeded in workspace centralus after treating the zone dropdown as an array (parameters('zone')[0]) and bumping the CCF content version to 4.1.0.

Copilot review on Azure#14949

Addressed here:

• MULTI-ACCOUNT.md is shippable (no “Do not merge”)
• ReleaseNotes.md table alignment
• createUiDefinition.json workspace filter uses /resourceGroups/<name>/
• testParameters.json “set up”

Reference tree: Solutions/SemperisLightningCCF

Partner PR to apply these onto: Azure#14949

[MitchellGulledge3]

Two deployment blockers found in live testing, now fixed (854e727)

Both surfaced deploying into real Sentinel workspaces. They are independent.

Defect B — blocked every deployment, including greenfield

bool is not a valid column type. Portal rejected the DCR:

InvalidPayload: Data collection rule is invalid,
[{"code":"InvalidStreamDeclaration","message":"'Type' must have one of the
allowed values: string,int,long,real,boolean,datetime,dynamic."}]

LightningAttackPaths_CL.Blowout was bool. Now boolean everywhere.

The two vocabularies are not identical:

Resource	Allowed
workspaces/tables	string, int, long, real, boolean, datetime, dynamic, guid
DCR streamDeclarations	string, int, long, real, boolean, datetime, dynamic (no guid)

bool is in neither.

Defect A — blocked workspaces with a prior 3.1.0 deploy

Log Analytics custom tables are additive only. You can add columns, never drop or retype them. Dropping ConnectorName failed against existing tables:

Invalid output table schema: The following columns which exist in the current
schema do not exist in the new schema or have different types

ConnectorName is restored in all 7 tables and populated via ConnectorName=tostring(SemperisInstanceName), so existing customer queries keep working.

Only conditional on a prior CCF deploy. The legacy Function App schema is fully additively compatible — 0 conflicts.

Root enabler

mainTemplate.json was hand-maintained. It is the artifact that actually deploys, but tests only validated the standalone files. So it drifted and shipped invalid while the suite passed.

It is now generated from the same constants, and tests scan it directly. Type validation also runs at generation time, so an invalid type fails the build rather than the deployment.

Suite: 8/14 → 14/14. Every new guard was mutation-tested.

Still open

Not yet validated against a live workspace. Both fixes are verified against the ARM/DCR schema contracts and the portal's own error text. The next redeploy is the real test.

Auth remains the blocking question for Semperis — unchanged.

[MitchellGulledge3]

4.0.0 — all seven tables renamed to V2

Pushed as d077544dca.

What changed

Every custom table now ends in V2_CL:

Legacy (unchanged, still written by the Function App)	This solution now writes
LightningTier0Nodes_CL	LightningTier0NodesV2_CL
LightningAttackPaths_CL	LightningAttackPathsV2_CL
LightningAttackPathLinks_CL	LightningAttackPathLinksV2_CL
LightningTier0Attackers_CL	LightningTier0AttackersV2_CL
LightningIndicatorExecutions_CL	LightningIndicatorExecutionsV2_CL
LightningIOEsMetadata_CL	LightningIOEsMetadataV2_CL
LightningIOEResults_CL	LightningIOEResultsV2_CL

Form matches the shipped SailPoint convention (SailPointIDN_EventsV2_CL).

Why

Zero overlap with the legacy Solutions/SemperisLightning Function App solution, which keeps writing the v1 tables. The two connectors now share no schema, so they can run side by side in one workspace with no interaction — no additive-only conflicts, no column fights, no ambiguity about which connector produced a row.

Scope of the rename

Table resources and filenames, DCR streamDeclarations keys, dataFlows[].streams[] and outputStream, poller dcrConfig.streamName and dataType, the connector definition graphQueries / sampleQueries / dataTypes / lastDataReceivedQuery, mainTemplate.json, and the package zip. A test fails the build if any v1 name survives anywhere, including inside the zip.

ConnectorName dropped

It only existed to satisfy the Log Analytics additive-only rule against tables a prior 3.1.0 deploy had created, and it duplicated SemperisInstanceName. V2 tables are brand new, so there is no constraint and no legacy consumer. Use SemperisInstanceName. The schema baseline was re-based to the V2 schemas rather than deleted, so once 4.0.0 ships a later column drop or retype is still caught.

Version

4.0.0, not 3.2.0 — renaming every table breaks every existing query, workbook and rule. ARM contentVersion deliberately stays 1.0.0.0: ARM validates it against a four-part pattern and rejects a three-part semver.

Validation

pwsh .script/local-validation/build-and-validate.ps1 -SolutionName SemperisLightningCCF -SkipPackaging → BUILD SUCCESSFUL, 11 passed / 0 failed / 6 skipped, ARM-TTK 48/48. Candidate suite 17/17. Skipped validators need .NET 3.1/8.0 and trufflehog, which are not installed locally.

⚠️ Still unresolved — auth

Unchanged and unverified. The Semperis token endpoint takes a single apiKey field. CCF JwtToken models a userName/password pair. The config still pads with an invented ccfCompatibility field. If Semperis rejects unknown JSON fields, token acquisition fails and no stream ingests. This needs live verification against real credentials — it is the first thing to check on deploy.

Still DO NOT MERGE — reference only.

[MitchellGulledge3]

Addendum — three verified findings (bf9af483c6)

Documented in MULTI-ACCOUNT.md. None of them change the shipped config.

1. Auth — the ccfCompatibility pad has no documented basis. Microsoft Learn states JwtToken requires username/password token acquisition and does not support API-key token requests. Two documented alternatives to raise with Semperis, deliberately not applied here:

• APIKey auth with IsApiKeyInPostPayload, if the polling endpoints accept the key directly and no JWT exchange is needed.
• OAuth2 client credentials, if the token service supports it.

Source: data-connector-connection-rules-reference#authentication-configuration

2. Upgrading over an existing 3.1.1 connection. Poller resource names are unchanged (SemperisLightning<Stream>-<uniqueString(connectionName, zone)>), so installing 4.0.0 updates the six existing pollers in place and repoints them at the V2 tables. The v1 tables are left behind with their data and are not deleted; the Function App keeps writing them. Expect v1 CCF ingestion to simply stop at the upgrade timestamp.

3. A transform projecting a column the table does not declare is silently discarded — accepted without error, billed, not stored. Not an ingestion failure. That silent-data-loss mode is why test_every_transform_output_column_exists_in_its_destination_table was added.

Source: data-collection-transformations-create

Correction to my earlier comment: I said ARM "rejects" a three-part contentVersion. More precisely — the ARM JSON schema pattern requires four parts and ARM-TTK enforces it, but Learn says "you can provide any value", so runtime rejection is unproven. 1.0.0.0 is kept regardless: it matches every shipped solution and passes ARM-TTK 48/48.

[MitchellGulledge3]

Pushed 02dbfb9c6d after a successful portal Connect in workspace centralus.

• Solution / CCF content version: 4.1.0
• Existing offer: azure-sentinel-solution-semperislightning
• Both connectors present: Function App + CCF
• Zone dropdown treated as an array (parameters('zone')[0])

Reference tree: https://github.com/MitchellGulledge3/Azure-Sentinel/tree/semperis-lightning-ccf-multi-account-reference/Solutions/SemperisLightningCCF

[MitchellGulledge3]

Addressed Copilot nits from Azure#14949 in 92ab765d1f. PR title no longer says DO NOT MERGE — this is the merge-ready 4.1.0 Function App + CCF package.