Add Purview plugin#102
Conversation
- Add dataSourceName field to Scans on Data Source stream, read directly from the API response rather than parsed from the scan id - Fix lastRunResult fields in dataSourceScans and scansForDataSource: status was scanRunStatus, times were nested under discoveryExecutionDetails, and runType was scanLevelType — all previously broken due to incorrect field name assumptions Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Fix classification summary (count from assets instead of missing facet count field) - Add Classified Assets stream (all classifications, one row per asset-classification pair) - Add Assets with Custom Classifications stream (non-Microsoft classifications only) - Add Custom Classifications stream (account-wide, available on Collection and Data Source) - Update Scans on Data Source stream with dataSourceName field and postRequestScript - Add Duration (s) numeric column to Scan History for average queries - Restrict Scan History to single object selection - Remove Collection and Data Source out-of-the-box dashboards Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resize Classification Coverage and Metadata Policies tiles to 2x2 and stack them on the right side of the Asset Count by Entity Type tile. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add Governance Domains stream (Unified Catalog API, account-wide) - Add Glossary Terms stream scoped to Purview Governance Domain objects - Index Governance Domains as a new object type - Strip HTML from description fields in both streams - Update README with Unified Catalog permission requirements - Add Purview Governance Domain to objectTypes and What gets indexed Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
📝 WalkthroughWalkthroughThe Microsoft Purview v1 plugin is added with OAuth configuration, validation, indexed collections, data sources, and governance domains. It also adds on-demand asset, lineage, classification, policy, glossary, and scan streams, normalized response scripts, an Overview dashboard, scopes, and documentation. ChangesMicrosoft Purview plugin foundation
Indexed entities and operational streams
On-demand asset analytics
Dashboard and scope wiring
Sequence Diagram(s)sequenceDiagram
participant PluginConfiguration
participant ConfigValidation
participant PurviewAPI
participant IndexDefinition
participant Dashboard
PluginConfiguration->>ConfigValidation: validate configured credentials
ConfigValidation->>PurviewAPI: request collections and data sources
PurviewAPI-->>ConfigValidation: return validation results
IndexDefinition->>PurviewAPI: retrieve indexed Purview entities
PurviewAPI-->>IndexDefinition: return collections, data sources, and domains
Dashboard->>PurviewAPI: request on-demand asset analytics
PurviewAPI-->>Dashboard: return transformed stream data
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 15
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/MicrosoftPurview/v1/dataStreams/assetLineage.json`:
- Around line 28-32: Constrain the depth field in the asset lineage
configuration to valid Purview lineage hop counts: require numeric integer
input, enforce a minimum of 0, and add an appropriate maximum if supported by
the API. Keep the existing defaultValue of 3 and help text unchanged.
- Around line 7-14: Update the assetGuid field validation in the UI
configuration to require a valid GUID format before the value is used in the
lineage endpoint path. Keep the field required and ensure values containing path
characters such as /, ?, or # are rejected.
In `@plugins/MicrosoftPurview/v1/dataStreams/assetsByEntityTypeGlobal.json`:
- Around line 13-17: Configure token-based pagination in the postRequest
configuration for assetsByEntityTypeGlobal.json (lines 13-17) and
classificationCoverage.json (lines 13-17), matching the contract used by
assetsByCollection.json. Ensure all Purview response pages are combined before
assetsByEntityTypeGlobal.js and classificationCoverage.js aggregate and count
results.
In `@plugins/MicrosoftPurview/v1/dataStreams/classificationTypes.json`:
- Around line 10-13: Update the getArgs entry in classificationTypes.json to use
the documented uppercase CLASSIFICATION value for the type filter, while leaving
the classificationTypes.js postRequestScript unchanged.
In `@plugins/MicrosoftPurview/v1/dataStreams/collections.json`:
- Around line 28-68: Replace the mustache-style valueExpression transformations
in plugins/MicrosoftPurview/v1/dataStreams/collections.json lines 28-68 with map
transformations for collectionName, displayName, parentCollectionName, and
sourceType, preserving alias, fallback, nested-field, and constant-value
behavior. Apply the same conversion in
plugins/MicrosoftPurview/v1/dataStreams/listDataSources.json lines 24-78 for the
nested collection, resource group, subscription, portal URL, and sourceType
fields; both sites require direct changes.
In `@plugins/MicrosoftPurview/v1/dataStreams/dataSourceScans.json`:
- Around line 29-34: Update the normalization script in dataSourceScans.js to
emit collection from props.collection?.referenceName, then change the collection
entry in dataSourceScans.json to a normal metadata field without computed or
valueExpression properties. Remove the dotted implementation-field derivation
and preserve the existing display name.
In `@plugins/MicrosoftPurview/v1/dataStreams/scansForDataSource.json`:
- Around line 2-6: Add the mandatory tags property to the scansForDataSource
stream definition, reusing the existing Data Sources tag used by scanHistory.
Keep the current metadata and hidden visibility unchanged.
In `@plugins/MicrosoftPurview/v1/dataStreams/scripts/accountAssetCount.js`:
- Around line 1-2: Update the asset-count result construction around
`@search.count` to detect `data?.['`@search.count.approximate`'] === true` and
avoid presenting the estimated value as the exact “Total Assets” KPI, using an
appropriate status field/label or equivalent non-exact representation. Increment
the plugin `version` in `metadata.json` for this change.
In `@plugins/MicrosoftPurview/v1/dataStreams/scripts/assetLineage.js`:
- Around line 7-16: Update the lineage processing around the relations loop and
role assignment to traverse the graph from baseGuid, collecting every reachable
upstream and downstream node across multiple hops rather than only direct
neighbors. Preserve Base for baseGuid and assign roles from the complete
traversal sets, then increment the version in metadata.json to reflect the
behavior change.
In `@plugins/MicrosoftPurview/v1/dataStreams/scripts/customClassifications.js`:
- Around line 3-12: The custom-classification output must expose category and
typeVersion. In
plugins/MicrosoftPurview/v1/dataStreams/scripts/customClassifications.js lines
3-12, update the normalized rows produced by the defs filter/map to include both
fields; in plugins/MicrosoftPurview/v1/dataStreams/customClassifications.json
lines 16-22, declare metadata for those same fields so Category and Version are
available to consumers.
In `@plugins/MicrosoftPurview/v1/dataStreams/searchAssets.json`:
- Around line 9-13: Update the keywords text field in searchAssets.json so its
placeholder includes an example search keyword, while retaining that the field
is optional.
- Around line 39-42: Update the postBody limit expression in the Purview search
request to coerce the input, constrain it to the valid 1–1000 range, and forward
at most 1000 while preserving the existing default behavior. Also constrain the
associated UI number field, if present in the diff, with minimum 1 and maximum
1000.
In `@plugins/MicrosoftPurview/v1/indexDefinitions/default.json`:
- Around line 2-50: Implement scan indexing by adding a scan-stream import step
with a stable ID, scan label, and “Purview Scan” source type in
plugins/MicrosoftPurview/v1/indexDefinitions/default.json lines 2-50; add
“Purview Scan” to objectTypes in plugins/MicrosoftPurview/v1/metadata.json lines
24-28, and retain the corresponding scan-indexing and object-type documentation
in plugins/MicrosoftPurview/v1/docs/README.md lines 7-7 and 63-70.
In `@plugins/MicrosoftPurview/v1/metadata.json`:
- Around line 45-52: The Microsoft Purview metadata configuration currently
routes all streams through the account-scoped baseUrl. Add a separate
authenticated base configuration for https://api.purview-service.microsoft.com/
using the existing OAuth2 settings, then update the governanceDomains and
glossaryTerms stream definitions to reference that base while leaving other
streams on the account-scoped endpoint.
In `@plugins/MicrosoftPurview/v1/ui.json`:
- Around line 12-37: Update the tenantId, clientId, and clientSecret fields in
the UI configuration with example GUID-based placeholders for the IDs and
concise help text describing where to obtain each credential. Preserve the
existing labels, field types, and required validation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI (base), Organization UI (inherited)
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 97e34756-0fb3-4a75-a5c5-73ed08e038f5
⛔ Files ignored due to path filters (1)
plugins/MicrosoftPurview/v1/icon.svgis excluded by!**/*.svg
📒 Files selected for processing (47)
plugins/MicrosoftPurview/v1/configValidation.jsonplugins/MicrosoftPurview/v1/custom_types.jsonplugins/MicrosoftPurview/v1/dataStreams/accountAssetCount.jsonplugins/MicrosoftPurview/v1/dataStreams/assetCountByEntityType.jsonplugins/MicrosoftPurview/v1/dataStreams/assetLineage.jsonplugins/MicrosoftPurview/v1/dataStreams/assetsByCollection.jsonplugins/MicrosoftPurview/v1/dataStreams/assetsByEntityTypeGlobal.jsonplugins/MicrosoftPurview/v1/dataStreams/classificationCoverage.jsonplugins/MicrosoftPurview/v1/dataStreams/classificationTypes.jsonplugins/MicrosoftPurview/v1/dataStreams/collectionClassificationSummary.jsonplugins/MicrosoftPurview/v1/dataStreams/collectionClassifiedAssets.jsonplugins/MicrosoftPurview/v1/dataStreams/collectionCustomClassifiedAssets.jsonplugins/MicrosoftPurview/v1/dataStreams/collectionPolicyCoverage.jsonplugins/MicrosoftPurview/v1/dataStreams/collections.jsonplugins/MicrosoftPurview/v1/dataStreams/customClassifications.jsonplugins/MicrosoftPurview/v1/dataStreams/dataSourceScans.jsonplugins/MicrosoftPurview/v1/dataStreams/glossaryTerms.jsonplugins/MicrosoftPurview/v1/dataStreams/governanceDomains.jsonplugins/MicrosoftPurview/v1/dataStreams/listDataSources.jsonplugins/MicrosoftPurview/v1/dataStreams/metadataPolicies.jsonplugins/MicrosoftPurview/v1/dataStreams/scanHistory.jsonplugins/MicrosoftPurview/v1/dataStreams/scansForDataSource.jsonplugins/MicrosoftPurview/v1/dataStreams/scripts/accountAssetCount.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/assetCountByEntityType.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/assetLineage.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/assetsByCollection.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/assetsByEntityTypeGlobal.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/classificationCoverage.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/classificationTypes.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/collectionClassificationSummary.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/collectionClassifiedAssets.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/collectionCustomClassifiedAssets.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/collectionPolicyCoverage.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/customClassifications.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/dataSourceScans.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/glossaryTerms.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/governanceDomains.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/metadataPolicies.jsplugins/MicrosoftPurview/v1/dataStreams/scripts/searchAssets.jsplugins/MicrosoftPurview/v1/dataStreams/searchAssets.jsonplugins/MicrosoftPurview/v1/defaultContent/manifest.jsonplugins/MicrosoftPurview/v1/defaultContent/overview.dash.jsonplugins/MicrosoftPurview/v1/defaultContent/scopes.jsonplugins/MicrosoftPurview/v1/docs/README.mdplugins/MicrosoftPurview/v1/indexDefinitions/default.jsonplugins/MicrosoftPurview/v1/metadata.jsonplugins/MicrosoftPurview/v1/ui.json
| "ui": [ | ||
| { | ||
| "type": "text", | ||
| "name": "assetGuid", | ||
| "label": "Asset GUID", | ||
| "placeholder": "Asset GUID from the catalog", | ||
| "validation": { "required": true }, | ||
| "help": "The Purview asset's GUID. Copy it from the asset details page in the Purview portal, or from the **Asset GUID** column of the Search Assets stream." |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files matching assetLineage.json:"
fd -a 'assetLineage\.json$' . | sed 's#^\./##'
file="$(fd 'assetLineage\.json$' . | grep 'plugins/MicrosoftPurview/v1/dataStreams/assetLineage.json' || true)"
if [ -n "$file" ]; then
echo
echo "assetLineage.json content:"
cat -n "$file"
fi
echo
echo "Search for plugin renderer/path interpolation and assetGuid usages:"
rg -n '"assetGuid"|endpointPath|getLineage|assetLineage|assetGuid' plugins/MicrosoftPurview -SRepository: squaredup/plugins
Length of output: 6232
🌐 Web query:
Microsoft Purview Data Map lineage REST API get lineage assetGuid asset guid path
💡 Result:
To retrieve lineage information for a specific asset in the Microsoft Purview Data Map via the REST API, you primarily use the lineage endpoint identified by the asset's GUID [1][2]. The API endpoint to get lineage by asset GUID is: GET {endpoint}/datamap/api/atlas/v2/lineage/{guid}?direction={direction}&depth={depth} [2] Key parameters for this request include: - {endpoint}: The base URL for your Purview Data Map data plane [2]. - {guid}: The globally unique identifier (GUID) of the asset for which you want to retrieve lineage [2]. - direction: Required query parameter specifying the lineage flow (INPUT, OUTPUT, or BOTH) [2]. - depth: Optional query parameter defining the number of hops (levels) in the lineage graph to retrieve [2]. If you do not have the asset GUID but possess unique attributes (such as a qualifiedName), you can use the Get By Unique Attribute endpoint: GET {endpoint}/datamap/api/atlas/v2/lineage/uniqueAttribute/type/{typeName}?attr:{attrName}={value}&direction={direction}&depth={depth} [3] For scenarios involving large lineage graphs or the need to traverse beyond the initial depth, the API supports pagination via the lineage next page endpoint: GET {endpoint}/datamap/api/atlas/v2/lineage/{guid}/next/ [4] Note that Microsoft Purview lineage is built upon the Apache Atlas API structure [3][5]. Lineage data returned by these endpoints includes a mapping of GUIDs to entity details (guidEntityMap) and a list of relations (relations) that define the connection between assets [3][2]. If you are performing tasks such as refreshing lineage, it is often more efficient to identify and manipulate the Process entities associated with the lineage rather than individual relationship records [4].
Citations:
- 1: https://learn.microsoft.com/en-us/rest/api/purview/datamapdataplane/lineage?view=rest-purview-datamapdataplane-2023-09-01
- 2: https://learn.microsoft.com/en-us/rest/api/purview/datamapdataplane/lineage/get?view=rest-purview-datamapdataplane-2023-09-01
- 3: https://learn.microsoft.com/en-us/rest/api/purview/datamapdataplane/lineage/get-by-unique-attribute?view=rest-purview-datamapdataplane-2023-09-01
- 4: https://techcommunity.microsoft.com/discussions/azurepurview/data-system-wide-lineage-via-api-request/4520156
- 5: https://learn.microsoft.com/en-us/purview/data-gov-api-create-lineage-relationships
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Find plugin package metadata and related MicrosoftPurview dataStreams:"
fd -a . plugins/MicrosoftPurview/v1 | sed 's#^\./##'
echo
echo "MicrosoftPurview package files and relevant stream configs:"
for f in $(fd '{metadata\.json|package\.json|plugin\.json|settings.*\.json}$' plugins/MicrosoftPurview/v1); do
echo "--- $f"
wc -l "$f"
done
echo
echo "Search for endpointPath substitution implementation:"
rg -n "endpointPath|{{.*}}|url\\.|URL|encodeURIComponent|encodeURI|replace\\(" -S --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' . | head -n 200Repository: squaredup/plugins
Length of output: 5133
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "MicrosoftPurview plugin metadata and relevant files:"
cat -n plugins/MicrosoftPurview/v1/metadata.json
echo
echo "Other endpointPath examples:"
rg -n '"endpointPath"' plugins/MicrosoftPurview/v1/dataStreams -S
echo
echo "Global search for endpointPath template substitution:"
rg -n 'endpointPath|{{[A-Za-z0-9_.]+}}|encodeURIComponent|encodeURI' -S --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' . | head -n 200
echo
echo "HTTP request integration references:"
rg -n 'httpRequest|baseDataSourceName|endpointPath|dataStreams|render|request' -S --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' . | head -n 300Repository: squaredup/plugins
Length of output: 5853
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Global search for endpointPath template substitution:"
rg -n 'endpointPath|{{[A-Za-z0-9_.]*}}|encodeURIComponent|encodeURI' -S --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' . | head -n 300
echo
echo "HTTP request integration references:"
rg -n 'httpRequest|baseDataSource|endpointPath|dataStreams|renders?|request' -S --glob '!**/node_modules/**' --glob '!**/dist/**' --glob '!**/build/**' . | head -n 500Repository: squaredup/plugins
Length of output: 358
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Global search for endpointPath template substitution:"
rg -n 'endpointPath|{{[A-Za-z0-9_.]*}}|encodeURIComponent|encodeURI' -S . | head -n 300 || true
echo
echo "HTTP request integration references:"
rg -n 'httpRequest|baseDataSource|endpointPath|dataStreams|renders?|request' -S . | head -n 500 || trueRepository: squaredup/plugins
Length of output: 50373
🌐 Web query:
Azure Data Explorer Kusto Azure Monitor query parameter encoded in URL slash allowed documentation
💡 Result:
When passing parameters in URLs for Azure Data Explorer (ADX) or Azure Monitor (such as in deep links or API requests), standard percent-encoding (HTML URI encoding) rules apply [1][2]. Regarding the use of forward slashes (/), they are reserved characters in URLs. If a forward slash is part of the data being passed (and not intended as a URL path delimiter), it must be percent-encoded as %2F to ensure the URL is interpreted correctly by the receiving service [3][4][5][6]. For Kusto (ADX) UI deep links specifically, the query parameter must be encoded according to standard URI query string rules [1][2]. If you are passing a Kusto query, the official documentation recommends: 1. Standard HTTP query parameter encoding [1][2]. 2. Alternatively, for complex or long queries, using a gzip-compressed and then Base64-encoded string to avoid URI length limitations and character encoding issues [1][2][7][8]. If you are dealing with resource IDs or other path-like parameters in Azure portal URLs (such as those for Azure Monitor Workbooks or Log Analytics), these often require forward slashes to be encoded as %2F to prevent the portal from misinterpreting the URL structure [4][5]. In Kusto Query Language (KQL) itself, if you need to generate these encoded strings programmatically, you can use built-in functions such as url_encode or url_encode_component [3][6]. Note that url_encode follows application/x-www-form-urlencoded rules (encoding spaces as '+'), while url_encode_component encodes spaces as '%20' [3][6].
Citations:
- 1: https://learn.microsoft.com/en-us/azure/data-explorer/kusto/api/rest/deeplink
- 2: https://learn.microsoft.com/en-us/kusto/api/rest/deeplink?view=azure-data-explorer
- 3: https://learn.microsoft.com/en-us/kusto/query/url-encode-function?view=microsoft-fabric
- 4: https://exchangetuts.com/how-to-pass-workbook-parameters-in-the-url-to-an-azure-monitor-workbook-1767084902550683
- 5: https://zimmergren.net/deep-linking-azure-log-analytics-and-app-insight-queries/
- 6: https://learn.microsoft.com/en-us/kusto/query/url-encode-component-function?view=microsoft-fabric
- 7: https://stackoverflow.com/questions/76354971/encoding-used-by-kusto-web-explorer-to-generate-shareable-links
- 8: https://stackoverflow.com/questions/77098554/pass-query-to-adx-ui-thru-url-params
🌐 Web query:
Python urllib.parse quote slash / encoded / encoded %2F default safe
💡 Result:
In Python's urllib.parse.quote function, the slash character ('/') is included in the default set of safe characters [1][2]. Because the 'safe' parameter defaults to '/' [1][2], the quote function will not encode slashes by default [1][3]. This behavior is designed to allow developers to quote URL paths while preserving the directory structure [3]. If you wish to encode slashes as '%2F', you must explicitly set the 'safe' parameter to an empty string (or any string that does not include '/') when calling the function [1][4]: urllib.parse.quote('a/b', safe='') # Result: 'a%2Fb' In contrast, the related function urllib.parse.quote_plus has a default 'safe' value of an empty string ('') [1][2]. Consequently, quote_plus will encode slashes as '%2F' by default [1][4].
Citations:
- 1: https://docs.python.org/3/library/urllib.parse.html
- 2: https://github.com/python/cpython/blob/main/Doc/library/urllib.parse.rst
- 3: https://github.com/python/cpython/blob/main/Lib/urllib/parse.py
- 4: https://docs.python.org/3.10/library/urllib.parse.html
Validate assetGuid before interpolating it into the path.
assetGuid accepts path characters (/, ?, #) while Line 37 inserts the raw value into datamap/api/atlas/v2/lineage/{{assetGuid}}. Since the API expects a single GUID path segment, validate against an appropriate GUID format before interpolation.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/MicrosoftPurview/v1/dataStreams/assetLineage.json` around lines 7 -
14, Update the assetGuid field validation in the UI configuration to require a
valid GUID format before the value is used in the lineage endpoint path. Keep
the field required and ensure values containing path characters such as /, ?, or
# are rejected.
| "type": "number", | ||
| "name": "depth", | ||
| "label": "Depth", | ||
| "defaultValue": 3, | ||
| "help": "Number of hops to traverse in each direction." |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Locate assetLineage and nearby metadata =="
git ls-files | rg '(^|/)dataStreams/assetLineage\.json$|(^|/)metadata\.json$' | sed -n '1,80p'
echo
echo "== assetLineage relevant lines =="
sed -n '1,120p' plugins/MicrosoftPurview/v1/dataStreams/assetLineage.json || true
echo
echo "== Existing number constraints patterns in dataStreams =="
python3 - <<'PY'
import json, pathlib
for p in pathlib.Path('dataStreams').glob('*.json'):
try:
data=json.loads(p.read_text())
except Exception:
continue
if 'parameters' not in data:
continue
for pm in data.get('parameters', []):
if isinstance(pm, dict) and pm.get('type')=='number' and {'minimum','maximum','step','multipleOf','multipleOfInteger','min','max'}.intersection(pm.keys()):
print(p, json.dumps(pm, ensure_ascii=False, separators=(',',':'))[:300])
PY
echo
echo "== Search for Purview depth calls/docs references =="
rg -n "depth|lineage|assetLineage|minimum|multipleOf" plugins/MicrosoftPurview dataStreams -S || trueRepository: squaredup/plugins
Length of output: 5455
🌐 Web query:
Microsoft Purview lineage graph depth parameter maximum integer Get lineage REST API
💡 Result:
The Microsoft Purview Lineage "Get" REST API defines the depth parameter as an integer (int32) [1][2]. While the API documentation specifies the data type, it does not explicitly document a hard maximum integer value for the depth parameter beyond the constraints of a 32-bit signed integer [1][2]. In practical application, Microsoft Purview does not recommend using high depth values for extensive lineage traversal due to performance constraints [3]. For large or complex lineage graphs, it is recommended to use the paginated traversal endpoint (e.g., GET /datamap/api/atlas/v2/lineage/{guid}/next/) instead of attempting to retrieve the entire graph in a single request by increasing the depth parameter [3]. Increasing the depth to very high values can lead to request timeouts or performance degradation [3].
Citations:
- 1: https://learn.microsoft.com/en-us/rest/api/purview/datamapdataplane/lineage/get?view=rest-purview-datamapdataplane-2023-09-01
- 2: https://learn.microsoft.com/en-us/rest/api/purview/datamapdataplane/lineage/get-by-unique-attribute?view=rest-purview-datamapdataplane-2023-09-01
- 3: https://techcommunity.microsoft.com/discussions/azurepurview/data-system-wide-lineage-via-api-request/4520156
Constrain depth to the values supported by the Purview lineage API.
depth is passed directly to the lineage endpoint and Purview treats it as an int32; a bare number UI field can accept fractions, negatives, or non-numeric strings that do not represent a valid lineage hop count. Add integer/minimum constraints (for example, minimum: 0) and, if useful, an upper bound on hops.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/MicrosoftPurview/v1/dataStreams/assetLineage.json` around lines 28 -
32, Constrain the depth field in the asset lineage configuration to valid
Purview lineage hop counts: require numeric integer input, enforce a minimum of
0, and add an appropriate maximum if supported by the API. Keep the existing
defaultValue of 3 and help text unchanged.
| "postBody": { | ||
| "keywords": null, | ||
| "limit": 1000 | ||
| }, | ||
| "postRequestScript": "assetsByEntityTypeGlobal.js" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Paginate both account-wide aggregations before counting.
Both streams cap Purview search responses at 1,000 rows but omit continuation-token paging. Their scripts aggregate only data.value, so larger accounts silently return incomplete entity-type counts and classification coverage. assetsByCollection.json demonstrates the required token-paging contract.
plugins/MicrosoftPurview/v1/dataStreams/assetsByEntityTypeGlobal.json#L13-L17: add token paging so all pages are included beforeassetsByEntityTypeGlobal.jsaggregates counts.plugins/MicrosoftPurview/v1/dataStreams/classificationCoverage.json#L13-L17: add the same token paging soclassificationCoverage.jscomputes coverage from the complete result set.
📍 Affects 2 files
plugins/MicrosoftPurview/v1/dataStreams/assetsByEntityTypeGlobal.json#L13-L17(this comment)plugins/MicrosoftPurview/v1/dataStreams/classificationCoverage.json#L13-L17
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/MicrosoftPurview/v1/dataStreams/assetsByEntityTypeGlobal.json` around
lines 13 - 17, Configure token-based pagination in the postRequest configuration
for assetsByEntityTypeGlobal.json (lines 13-17) and classificationCoverage.json
(lines 13-17), matching the contract used by assetsByCollection.json. Ensure all
Purview response pages are combined before assetsByEntityTypeGlobal.js and
classificationCoverage.js aggregate and count results.
| "getArgs": [ | ||
| { "key": "type", "value": "classification" } | ||
| ], | ||
| "postRequestScript": "classificationTypes.js" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== files =="
git ls-files | rg 'plugins/MicrosoftPurview/v1/dataStreams/(classificationTypes\.(json|js)|metadata\.json)|plugins/MicrosoftPurview/.*/metadata.json' || true
echo
echo "== classificationTypes.json =="
sed -n '1,220p' plugins/MicrosoftPurview/v1/dataStreams/classificationTypes.json
echo
echo "== classificationTypes.js =="
sed -n '1,260p' plugins/MicrosoftPurview/v1/dataStreams/classificationTypes.js
echo
echo "== metadata snippets =="
for f in plugins/MicrosoftPurview/v1/metadata.json plugins/MicrosoftPurview/metadata.json; do
if [ -f "$f" ]; then echo "--- $f"; sed -n '1,220p' "$f"; fi
done
echo
echo "== Purview type filter occurrences =="
rg -n '"type"|TypeCategory|classificationTypes|typeCategory|classifications|classification' plugins/MicrosoftPurview . --glob '!node_modules' --glob '!dist' --glob '!build' | head -200Repository: squaredup/plugins
Length of output: 1568
🌐 Web query:
Microsoft Purview REST API list TypeCategory enum value CLASSIFICATION type query parameter
💡 Result:
In the Microsoft Purview Data Map REST API (based on Apache Atlas), CLASSIFICATION is a valid value for the TypeCategory enum [1][2][3]. This enum is used to categorize type definitions (typedefs) within the metadata system [1][4][2]. When listing type definitions via the REST API—such as using the endpoint {endpoint}/datamap/api/atlas/v2/types/typedefs—you can use the type parameter to filter by these categories [1]. For example, by setting the query parameter type=CLASSIFICATION, you can retrieve only the classification type definitions [1][3]. The supported categories generally include: - PRIMITIVE - OBJECT_ID_TYPE - ENUM - STRUCT - CLASSIFICATION - ENTITY - RELATIONSHIP - BUSINESS_METADATA [1][2] If you are looking for specific assets that have been classified, you typically use different endpoints, such as retrieving classifications for a specific entity GUID [5] or using search/query APIs [6], as classification labels are properties of an entity rather than distinct assets themselves in the hierarchy [7][8].
Citations:
- 1: https://learn.microsoft.com/en-us/rest/api/purview/datamapdataplane/type/list?view=rest-purview-datamapdataplane-2023-09-01
- 2: https://learn.microsoft.com/en-us/purview/data-gov-api-custom-types
- 3: https://learn.microsoft.com/en-us/rest/api/purview/datamapdataplane/type/list-headers?view=rest-purview-datamapdataplane-2023-09-01
- 4: https://learn.microsoft.com/en-us/rest/api/purview/datamapdataplane/type/get-entity-def-by-name?view=rest-purview-datamapdataplane-2023-09-01
- 5: https://learn.microsoft.com/en-us/rest/api/purview/datamapdataplane/entity/get-classifications?view=rest-purview-datamapdataplane-2023-09-01
- 6: https://learn.microsoft.com/zh-cn/rest/api/purview/purview-unified-catalog/data-assets/query?view=rest-purview-purview-unified-catalog-2026-03-20-preview
- 7: https://learn.microsoft.com/en-us/rest/api/purview/datamapdataplane/entity/list-by-unique-attributes?view=rest-purview-datamapdataplane-2023-09-01
- 8: https://learn.microsoft.com/en-us/rest/api/purview/purview-unified-catalog/data-assets/create?view=rest-purview-purview-unified-catalog-2026-03-20-preview
🌐 Web query:
github squaredup plugins MicrosoftPurview v1 dataStreams classificationTypes.js
💡 Result:
In the SquaredUp plugins architecture, data streams are defined as independent JSON files located within a plugin's dataStreams directory [1]. Based on the official squaredup/plugins repository structure, a plugin version is organized as follows: plugins/<plugin-name>/v1/dataStreams/ [2][1]. There is no standard or required file named classificationTypes.js within the dataStreams folder or the broader plugin structure [2][1]. SquaredUp plugin data streams typically consist of a .json file defining the stream's configuration and, optionally, a script file (often named handler.js or similar) [2][1]. If you are encountering a reference to classificationTypes.js in a Microsoft Purview-related plugin or context, it is likely either: 1. A custom, non-standard component specific to a particular community-authored plugin implementation [3]. 2. A misunderstanding of the SquaredUp plugin structure, where standard files are metadata.json, ui.json, custom_types.json, and data_streams.json [2]. You can verify the contents of any specific plugin by exploring the squaredup/plugins repository on GitHub [3][4]. The repository documentation and the SKILL.md file provide the authoritative guidance for plugin development and structure [1].
Citations:
- 1: https://github.com/squaredup/plugins/blob/main/.claude/skills/build-plugin/SKILL.md
- 2: https://github.com/squaredup/squaredup-plugin-public
- 3: https://github.com/squaredup/plugins
- 4: https://github.com/squaredup/plugins/tree/main/plugins/DattoRMM/v1/dataStreams
Use the documented CLASSIFICATION filter value.
getArgs sends type=classification, but Microsoft Purview’s TypeCategory filter for this typedefs endpoint uses CLASSIFICATION. Use the documented enum value so the stream is sent the requested classification-type filter.
Proposed fix
"getArgs": [
- { "key": "type", "value": "classification" }
+ { "key": "type", "value": "CLASSIFICATION" }
],📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "getArgs": [ | |
| { "key": "type", "value": "classification" } | |
| ], | |
| "postRequestScript": "classificationTypes.js" | |
| "getArgs": [ | |
| { "key": "type", "value": "CLASSIFICATION" } | |
| ], | |
| "postRequestScript": "classificationTypes.js" |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/MicrosoftPurview/v1/dataStreams/classificationTypes.json` around
lines 10 - 13, Update the getArgs entry in classificationTypes.json to use the
documented uppercase CLASSIFICATION value for the type filter, while leaving the
classificationTypes.js postRequestScript unchanged.
| "name": "collectionName", | ||
| "displayName": "Collection", | ||
| "computed": true, | ||
| "valueExpression": "{{ $['name'] }}", | ||
| "visible": false | ||
| }, | ||
| { | ||
| "name": "friendlyName", | ||
| "displayName": "Friendly Name", | ||
| "visible": false | ||
| }, | ||
| { | ||
| "name": "displayName", | ||
| "displayName": "Name", | ||
| "computed": true, | ||
| "valueExpression": "{{ $['friendlyName'] || $['name'] }}", | ||
| "role": "label" | ||
| }, | ||
| { | ||
| "name": "description", | ||
| "displayName": "Description" | ||
| }, | ||
| { | ||
| "name": "parentCollection", | ||
| "visible": false | ||
| }, | ||
| { | ||
| "name": "parentCollectionName", | ||
| "computed": true, | ||
| "valueExpression": "{{ $['parentCollection'] && $['parentCollection'].referenceName ? $['parentCollection'].referenceName : '' }}", | ||
| "displayName": "Parent Collection" | ||
| }, | ||
| { | ||
| "name": "collectionProvisioningState", | ||
| "displayName": "State" | ||
| }, | ||
| { | ||
| "name": "sourceType", | ||
| "computed": true, | ||
| "valueExpression": "Purview Collection", | ||
| "visible": false |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Use map for the repeated field transformations.
Both streams use mustache-style valueExpression transformations for aliases, fallbacks, and nested-field extraction. Replace these with map transformations to follow the data-stream guideline.
plugins/MicrosoftPurview/v1/dataStreams/collections.json#L28-L68: map collection aliases, parent name, display name, and source type.plugins/MicrosoftPurview/v1/dataStreams/listDataSources.json#L24-L78: map nested collection, resource group, subscription, portal URL, and source type fields.
📍 Affects 2 files
plugins/MicrosoftPurview/v1/dataStreams/collections.json#L28-L68(this comment)plugins/MicrosoftPurview/v1/dataStreams/listDataSources.json#L24-L78
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/MicrosoftPurview/v1/dataStreams/collections.json` around lines 28 -
68, Replace the mustache-style valueExpression transformations in
plugins/MicrosoftPurview/v1/dataStreams/collections.json lines 28-68 with map
transformations for collectionName, displayName, parentCollectionName, and
sourceType, preserving alias, fallback, nested-field, and constant-value
behavior. Apply the same conversion in
plugins/MicrosoftPurview/v1/dataStreams/listDataSources.json lines 24-78 for the
nested collection, resource group, subscription, portal URL, and sourceType
fields; both sites require direct changes.
Source: Coding guidelines
| "type": "text", | ||
| "name": "keywords", | ||
| "label": "Keywords", | ||
| "placeholder": "Search keywords (optional)" | ||
| }, |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Use an example keyword placeholder.
The text-field placeholder describes the field but does not provide the required example.
Proposed fix
- "placeholder": "Search keywords (optional)"
+ "placeholder": "e.g. customer"📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "type": "text", | |
| "name": "keywords", | |
| "label": "Keywords", | |
| "placeholder": "Search keywords (optional)" | |
| }, | |
| "type": "text", | |
| "name": "keywords", | |
| "label": "Keywords", | |
| "placeholder": "e.g. customer" | |
| }, |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/MicrosoftPurview/v1/dataStreams/searchAssets.json` around lines 9 -
13, Update the keywords text field in searchAssets.json so its placeholder
includes an example search keyword, while retaining that the field is optional.
Source: Coding guidelines
| "postBody": { | ||
| "keywords": "{{ keywords || null }}", | ||
| "limit": "{{ Number(limit) || 100 }}", | ||
| "filter": "{{ (entityType || classification) ? { and: [entityType ? { entityType: entityType } : null, classification ? { classification: classification } : null].filter(Boolean) } : null }}" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "## Candidate file"
fd -a 'searchAssets\.json$' . | sed 's#^\./##'
echo
echo "## data stream relevant contents"
file="$(fd 'searchAssets\.json$' . | head -n1)"
if [ -n "${file:-}" ]; then
nl -ba "$file" | sed -n '1,140p'
fi
echo
echo "## metadata version references"
fd 'metadata\.json$' plugins | sed -n '1,50p' | while read -r f; do
echo "--- $f"
python3 - <<'PY' "$f"
import json,sys
p=sys.argv[1]
try:
data=json.load(open(p))
except Exception as e:
print(e); raise SystemExit
print({"path":p, "version": data.get("version"), "displayName": data.get("displayName")})
PY
done
echo
echo "## Limit-related occurrences in Purview plugin/data streams"
rg -n 'limit|query|queryAssets|queryAssetsWith|search/assets|search/assets' plugins/MicrosoftPurview || true
echo
echo "## Read-only behavioral probe of current template expression vs proposed expression"
python3 - <<'PY'
def current(limit):
return int(limit) or 100
def proposed(limit):
return min(1000, max(1, int(float(limit) if limit != '' else 100.0)))
samples = [None, '', -1, 0, 0.9, 1.2, 100, 999, 1000, 1001, 2000, float('inf')]
for s in samples:
try:
print(f"limit={s!r:15} current={current(s)!r:10} proposed={proposed(s)!r:10}")
except Exception as e:
print(f"limit={s!r:15} exception={type(e).__name__}: {e}")
PYRepository: squaredup/plugins
Length of output: 325
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file="/home/jailuser/git/plugins/MicrosoftPurview/v1/dataStreams/searchAssets.json"
echo "## Target file lines 1-140"
awk '{printf "%6d\t%s\n", NR, $0}' "$file" | sed -n '1,140p'
echo
echo "## Plugin metadata version"
python3 - <<'PY'
import json
from pathlib import Path
root = Path("/home/jailuser/git/plugins/MicrosoftPurview/v1")
for p in sorted(root.rglob("metadata.json")):
try:
data=json.loads(p.read_text())
except Exception as e:
print(f"{p}: read/error {e}")
continue
print(f"{p}: version={data.get('version')!r} displayName={data.get('displayName')!r}")
PY
echo
echo "## Limit-related occurrences in Purview plugin/data streams"
grep -RInE 'limit|queryAssets|queryAssetsWith|search/assets|assets/query?|postBody' /home/jailuser/git/plugins/MicrosoftPurview || true
echo
echo "## Read-only behavioral probe of current template expression vs proposed expression"
python3 - <<'PY'
import json
def current_value(limit):
return int(limit) or 100
def proposed_value(limit):
return min(1000, max(1, int(abs(float(limit)) if limit != '' else 100.0)))
samples = [None, "", -1, 0, 0.9, 1.2, 100, 999, 1000, 1001, 2000, -2000, float("inf")]
print("limit" + "".join(f"\t{m!r:>14}" for m in ("current", "proposed")))
for s in samples:
try:
print("current" + "".join(f"\t{current_value(s)!r:>14}" for m in ("current",)))
print("proposed" + "".join(f"\t{proposed_value(s)!r:>14}" for m in ("proposed",)))
except Exception as e:
print(f"limit={s!r:12} exception={type(e).__name__}: {e}")
PY
echo
echo "## JSON mustache expression extraction"
python3 - <<'PY'
import json
from pathlib import Path
p = Path("/home/jailuser/git/plugins/MicrosoftPurview/v1/dataStreams/searchAssets.json")
text = p.read_text()
print(text)
PYRepository: squaredup/plugins
Length of output: 8770
Clamp limit to the Purview search query maximum.
This UI field can send negative, fractional, or oversized values to datamap/api/search/query, which returns up to 1,000 per page. Clamp the coerced number to 1000 before forwarding it, and consider constraining the UI number field between 1 and 1000.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/MicrosoftPurview/v1/dataStreams/searchAssets.json` around lines 39 -
42, Update the postBody limit expression in the Purview search request to coerce
the input, constrain it to the valid 1–1000 range, and forward at most 1000
while preserving the existing default behavior. Also constrain the associated UI
number field, if present in the diff, with minimum 1 and maximum 1000.
| "steps": [ | ||
| { | ||
| "name": "collections", | ||
| "dataStream": { "name": "collections" }, | ||
| "timeframe": "none", | ||
| "objectMapping": { | ||
| "id": "name", | ||
| "name": "displayName", | ||
| "type": "sourceType", | ||
| "properties": [ | ||
| "collectionName", | ||
| { "parentCollection": "parentCollectionName" }, | ||
| "description" | ||
| ] | ||
| } | ||
| }, | ||
| { | ||
| "name": "dataSources", | ||
| "dataStream": { "name": "listDataSources" }, | ||
| "timeframe": "none", | ||
| "objectMapping": { | ||
| "id": "name", | ||
| "name": "name", | ||
| "type": "sourceType", | ||
| "properties": [ | ||
| "kind", | ||
| "collection", | ||
| "endpoint", | ||
| "resourceGroup", | ||
| "subscriptionId" | ||
| ] | ||
| } | ||
| }, | ||
| { | ||
| "name": "governanceDomains", | ||
| "dataStream": { "name": "governanceDomains" }, | ||
| "timeframe": "none", | ||
| "objectMapping": { | ||
| "id": "id", | ||
| "name": "name", | ||
| "type": "sourceType", | ||
| "properties": [ | ||
| "description", | ||
| "type", | ||
| "status" | ||
| ] | ||
| } | ||
| } | ||
| ] |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Implement scan indexing before advertising scan objects.
The index definition imports only collections, data sources, and governance domains; therefore no scan object can enter the graph despite the README claiming scans are indexed.
plugins/MicrosoftPurview/v1/indexDefinitions/default.json#L2-L50: add an import step for the scan stream with stable ID, label, andPurview Scansource type.plugins/MicrosoftPurview/v1/metadata.json#L24-L28: addPurview ScantoobjectTypes.plugins/MicrosoftPurview/v1/docs/README.md#L7-L7: retain the scan-indexing claim only after the import is implemented.plugins/MicrosoftPurview/v1/docs/README.md#L63-L70: retain thePurview Scanobject-type entry only after the import is implemented.
📍 Affects 3 files
plugins/MicrosoftPurview/v1/indexDefinitions/default.json#L2-L50(this comment)plugins/MicrosoftPurview/v1/metadata.json#L24-L28plugins/MicrosoftPurview/v1/docs/README.md#L7-L7plugins/MicrosoftPurview/v1/docs/README.md#L63-L70
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/MicrosoftPurview/v1/indexDefinitions/default.json` around lines 2 -
50, Implement scan indexing by adding a scan-stream import step with a stable
ID, scan label, and “Purview Scan” source type in
plugins/MicrosoftPurview/v1/indexDefinitions/default.json lines 2-50; add
“Purview Scan” to objectTypes in plugins/MicrosoftPurview/v1/metadata.json lines
24-28, and retain the corresponding scan-indexing and object-type documentation
in plugins/MicrosoftPurview/v1/docs/README.md lines 7-7 and 63-70.
| "baseUrl": "https://{{accountName}}.purview.azure.com/", | ||
| "authMode": "oauth2", | ||
| "oauth2GrantType": "clientCredentials", | ||
| "oauth2TokenUrl": "https://login.microsoftonline.com/{{tenantId}}/oauth2/v2.0/token", | ||
| "oauth2ClientId": "{{clientId}}", | ||
| "oauth2ClientSecret": "{{clientSecret}}", | ||
| "oauth2ClientSecretLocationDuringAuth": "body", | ||
| "oauth2Scope": "https://purview.azure.net/.default", |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files:"
git ls-files | rg 'plugins/MicrosoftPurview/v1/(metadata\.json|governanceDomains\.json)|metadata\.json$|governanceDomains\.json$' || true
echo
echo "Relevant MicrosoftPurview v1 metadata:"
sed -n '1,120p' plugins/MicrosoftPurview/v1/metadata.json
echo
echo "Relevant governanceDomains references:"
rg -n "governanceDomains|businessDomain|unified catalog|api\.purview-service|oauth2|baseUrl|objectTypes" plugins/MicrosoftPurview/v1 -S
echo
echo "GovernanceDomains JSON outline/content if present:"
if [ -f plugins/MicrosoftPurview/v1/governanceDomains.json ]; then
wc -l plugins/MicrosoftPurview/v1/governanceDomains.json
sed -n '1,240p' plugins/MicrosoftGivernanceDomains.json 2>/dev/null || sed -n '1,240p' plugins/MicrosoftPurview/v1/governanceDomains.json
fiRepository: squaredup/plugins
Length of output: 5085
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files:"
git ls-files | rg 'plugins/MicrosoftPurview/v1/(metadata\.json|governanceDomains\.json)|metadata\.json$|governanceDomains\.json$' || true
echo
echo "Relevant MicrosoftPurview v1 metadata:"
sed -n '1,120p' plugins/MicrosoftPurview/v1/metadata.json
echo
echo "Relevant governanceDomains references:"
rg -n "governanceDomains|businessDomain|unified catalog|api\.purview-service|oauth2|baseUrl|objectTypes" plugins/MicrosoftPurview/v1 -S
echo
echo "GovernanceDomains JSON outline/content if present:"
if [ -f plugins/MicrosoftPurview/v1/governanceDomains.json ]; then
wc -l plugins/MicrosoftPurview/v1/governanceDomains.json
sed -n '1,240p' plugins/MicrosoftPurview/v1/governanceDomains.json
fiRepository: squaredup/plugins
Length of output: 5085
🌐 Web query:
Microsoft Purview Unified Catalog enumerate business domains REST endpoint api.purview-service.microsoft.com
💡 Result:
The REST API endpoint to enumerate business domains in the Microsoft Purview Unified Catalog is: GET {endpoint}/datagovernance/catalog/businessdomains?api-version={api-version} The base {endpoint} is typically https://api.purview-service.microsoft.com/ [1]. Key details for this operation include: - HTTP Method: GET [1] - Query Parameters: - api-version (Required): The version of the API to use (e.g., 2026-03-20-preview) [2][1]. - $skipToken (Optional): Used for pagination [1]. - writeOnly (Optional): A boolean parameter used for write-obligation filtering [1]. Authentication for this endpoint requires an OAuth2 bearer token obtained from Microsoft Entra ID with the appropriate scope (typically https://purview.azure.net/.default) [1][3]. You must include this token in the authorization header of your request [4]. For further implementation details, refer to the official Microsoft documentation for the Purview Unified Catalog API [2][5].
Citations:
- 1: https://learn.microsoft.com/mt-mt/rest/api/purview/purview-unified-catalog/business-domain/enumerate?view=rest-purview-purview-unified-catalog-2025-09-15-preview
- 2: https://learn.microsoft.com/en-us/rest/api/purview/purview-unified-catalog/business-domain?view=rest-purview-purview-unified-catalog-2026-03-20-preview
- 3: https://learn.microsoft.com/en-us/purview/data-gov-api-rest-data-plane
- 4: https://learn.microsoft.com/en-us/rest/api/purview/
- 5: https://learn.microsoft.com/en-us/rest/api/purview/unified-catalog-api-overview
🌐 Web query:
Microsoft Purview REST API authenticate token scope https://purview.azure.net/.default client_credentials
💡 Result:
To authenticate to the Microsoft Purview REST API using the OAuth 2.0 client credentials flow, you must request an access token from the Microsoft identity platform [1]. The appropriate scope to use for this request is https://purview.azure.net/.default [2][3][4]. When using the client credentials grant type, the.default scope is required to refer generically to the Microsoft Purview resource, ensuring all configured application permissions are included in the token [3]. Key details for the authentication request: - Grant Type: client_credentials [5][6][1] - Scope: https://purview.azure.net/.default [2][3][4] - Endpoint: The token endpoint is https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token [2][1]. Older documentation or legacy implementations may refer to the resource parameter (e.g., resource=https://purview.azure.net) instead of the scope parameter [5][6][7]. However, when using the v2.0 endpoint and modern authentication practices, the scope parameter set to https://purview.azure.net/.default is the standard approach [2][3]. For development, if you are using the Azure SDK for Python, the library handles the underlying token acquisition automatically when you use the ClientSecretCredential class [8].
Citations:
- 1: https://learn.microsoft.com/en-us/entra/identity-platform/v2-oauth2-client-creds-grant-flow
- 2: https://docs.azure.cn/en-us/purview/data-gov-api-rest-data-plane
- 3: https://learn.microsoft.com/en-us/entra/identity-platform/scopes-oidc
- 4: https://learn.microsoft.com/en-us/rest/api/purview/purviewdataquality/get-schedules/get-schedules?view=rest-purview-purviewdataquality-2026-01-12-preview
- 5: https://learn.microsoft.com/en-us/purview/data-gov-api-rest-data-plane
- 6: https://hermanteng19.github.io/2024/04/09/An-Application-of-Microsoft-Purview-Authentication-Workflow-for-API-and-Data-Sources/
- 7: https://learn.microsoft.com/en-us/purview/data-gov-api-create-assets
- 8: https://learn.microsoft.com/en-us/purview/data-gov-python-sdk
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "All MicrosoftPurview v1 files:"
git ls-files plugins/MicrosoftPurview/v1 | sed -n '1,160p'
echo
echo "dataStreams governance domains content and JS:"
sed -n '1,180p' plugins/MicrosoftPurview/v1/dataStreams/governanceDomains.json
echo
if [ -f plugins/MicrosoftPurview/v1/dataStreams/governanceDomains.js ]; then
wc -l plugins/MicrosoftPurview/v1/dataStreams/governanceDomains.js
sed -n '1,220p' plugins/MicrosoftPurview/v1/dataStreams/governanceDomains.js
else
echo "governanceDomains.js not present"
fi
echo
echo "All relative endpoint definitions:"
rm -rf /tmp/purview-ends
git ls-files plugins/MicrosoftPurview/v1 | while read -r f; do
case "$f" in
*.json) rg -n '"url"' "$f" /tmp/purview-ends 2>/dev/null || true ;;
*.js) rg -n "fetch\\(|/get|/search|url:|url =|apiVersion|endpoint" "$f" || true ;;
esac
done
echo
echo "Metadata/base configs across Purview plugin:"
python3 - <<'PY'
import json, pathlib
for p in sorted(pathlib.Path('plugins/MicrosoftPurview/v1').rglob('metadata.json')):
print(f'--- {p} ---')
with p.open() as fh:
data=json.load(fh)
base=data.get('base')
print(json.dumps(base, indent=2, default=str))
PYRepository: squaredup/plugins
Length of output: 6125
🌐 Web query:
Azure REST API base query format /?api-version not included in URL
💡 Result:
The api-version query parameter is a mandatory requirement for Azure REST API operations [1][2][3]. According to official Microsoft Azure REST API guidelines, every request sent to an Azure service must include this parameter in the URL [2][3]. If a request omits the api-version parameter, the service will return an HTTP 400 Bad Request error with the error code MissingApiVersionParameter and the message: The api-version query parameter (?api-version=) is required for all requests [2][3][4]. Because this parameter is a fundamental part of the API contract, it cannot be excluded from the URL [2][3]. Azure services use this versioning scheme (typically in YYYY-MM-DD format) to ensure that clients interact with a stable and expected version of the service [1][2][3]. When using tools or SDKs, this parameter is generally appended to the request automatically [1]. If you are constructing manual REST calls, you must explicitly include it, for example: https://management.azure.com/subscriptions?api-version=2020-01-01 [1]
Citations:
- 1: https://learn.microsoft.com/en-us/azure/developer/intro/azure-service-sdk-tool-versioning
- 2: https://github.com/Microsoft/api-guidelines/blob/HEAD/azure/Guidelines.md
- 3: https://github.com/microsoft/api-guidelines/blob/refs/heads/vNext/azure/Guidelines.md
- 4: https://stackoverflow.com/questions/67728988/azure-pipeline-gate-invoke-rest-api-error-the-api-version-query-parameter
🌐 Web query:
site:learn.microsoft.com/en-us/rest/api/purview REST UriFormat BaseUriQueryApiVersion Azure REST API guidelines
💡 Result:
According to the Azure REST API guidelines, REST API URI structures are standardized to ensure consistency and predictability across services [1]. Base URI and Path Structure The base URI generally consists of a service endpoint (often a hostname or a combined subscription/resource group path) followed by the resource hierarchy [2][3][4]. Resource paths should be nouns representing the resource, and they should be hierarchical in nature [3][4]. API Versioning The api-version parameter is a mandatory query parameter in Azure REST APIs [2][5][3][4][1]. It is used to specify the version of the API contract the client wishes to interact with, ensuring backward compatibility and allowing for non-breaking API evolution [2][5][6]. Format Requirements 1. URI Format: URIs must be consistently formatted, typically using forward slashes to separate resource segments. Path parameters are enclosed in curly braces (e.g., {resourceName}) to indicate they are dynamic [2][3][4]. 2. Query Parameters: Parameters that modify the request, such as api-version, are appended to the path following a question mark (?) [2][5][6]. Multiple query parameters are separated by an ampersand (&) [6]. 3. Consistency: Azure guidelines emphasize using standard HTTP methods (GET, POST, PUT, DELETE, PATCH) correctly and maintaining consistent casing and naming conventions throughout the API surface [1]. For detailed implementation and best practices, Microsoft maintains the Azure REST API guidelines which serve as the authoritative reference for service design [1].
Citations:
- 1: https://learn.microsoft.com/en-us/rest/api/purview/purview-unified-catalog/data-assets/create?view=rest-purview-purview-unified-catalog-2026-03-20-preview
- 2: https://learn.microsoft.com/en-us/rest/api/purview/sharedataplane/received-shares/get?tabs=HTTP&view=rest-purview-sharedataplane-2023-02-15-preview
- 3: https://learn.microsoft.com/en-us/rest/api/purview/scanningdataplane/data-sources/get?view=rest-purview-scanningdataplane-2023-09-01
- 4: https://learn.microsoft.com/en-us/rest/api/purview/scanningdataplane/scans/create-or-replace?view=rest-purview-scanningdataplane-2023-09-01
- 5: https://learn.microsoft.com/en-us/rest/api/purview/purview-unified-catalog/terms/query?view=rest-purview-purview-unified-catalog-2026-03-20-preview
- 6: https://learn.microsoft.com/en-us/rest/api/purview/datamapdataplane/entity/create-or-update?view=rest-purview-datamapdataplane-2023-09-01
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Data stream definitions that use relative endpoint paths:"
python3 - <<'PY'
import json, pathlib
fields={'endpointPath','baseUrl','baseDataSourceName'}
for p in sorted(pathlib.Path('plugins/MicrosoftPurview/v1/dataStreams').glob('*.json')):
try:
data=json.loads(p.read_text())
except Exception as e:
print(f"--- parse error {p}: {e} ---")
continue
vals={k:data.get(k) for k in fields}
if any(vals.values()):
print(f"--- {p.name} ---")
for k,v in vals.items():
print(f" {k}: {v}")
PY
echo
echo "All fetch/endpoint construction references in Purview scripts:"
rg -n "fetch\\(|new URL|URL\\(|requestPath|body\\[\\]|\\?api-version|datagovernance|businessdomains|glossary|terms|api.purview-service|purview.azure.com|purview.azure.net" plugins/MicrosoftPurview/v1/dataStreams/scripts plugins/MicrosoftPurview/v1/dataStreams -SRepository: squaredup/plugins
Length of output: 3654
Route Unified Catalog streams through their required endpoint.
governanceDomains.json and glossaryTerms.json define relative Unified Catalog paths, but the plugin base URL points at the account-scoped endpoint. Add a separate authenticated base configuration for https://api.purview-service.microsoft.com/ and switch those streams to it.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/MicrosoftPurview/v1/metadata.json` around lines 45 - 52, The
Microsoft Purview metadata configuration currently routes all streams through
the account-scoped baseUrl. Add a separate authenticated base configuration for
https://api.purview-service.microsoft.com/ using the existing OAuth2 settings,
then update the governanceDomains and glossaryTerms stream definitions to
reference that base while leaving other streams on the account-scoped endpoint.
| { | ||
| "type": "text", | ||
| "name": "tenantId", | ||
| "label": "Directory (tenant) ID", | ||
| "placeholder": "Enter a directory (tenant) ID", | ||
| "validation": { | ||
| "required": true | ||
| } | ||
| }, | ||
| { | ||
| "type": "text", | ||
| "name": "clientId", | ||
| "label": "Application (client) ID", | ||
| "placeholder": "Enter an application (client) ID", | ||
| "validation": { | ||
| "required": true | ||
| } | ||
| }, | ||
| { | ||
| "type": "password", | ||
| "name": "clientSecret", | ||
| "label": "Client secret", | ||
| "placeholder": "Enter a client secret", | ||
| "validation": { | ||
| "required": true | ||
| } |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Add example placeholders and credential help text.
Replace the generic ID placeholders with example GUIDs and add concise help explaining where to obtain the tenant ID, client ID, and secret value. This makes the setup form self-contained.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@plugins/MicrosoftPurview/v1/ui.json` around lines 12 - 37, Update the
tenantId, clientId, and clientSecret fields in the UI configuration with example
GUID-based placeholders for the IDs and concise help text describing where to
obtain each credential. Preserve the existing labels, field types, and required
validation.
Sources: Coding guidelines, Learnings
1 participant
🔌 Plugin overview
🖼️ Plugin screenshots
Plugin configuration
Default dashboards
🧪 Testing
I have carried out functional testing to ensure that:
This is a first iteration of the plugin and I have encountered some data hierarchy issues which mean that data joins do not work as elegantly as I would like. For example, it is not possible to see all Scans at once - they can only be viewed per data source.
For the purpose of this exercise I have created a free Purview instance. This means that I do not have access to Enterprise features so these are not included in the data source
Some Purview functionality is in preview mode only and is not yet supported by the API
I have only been able to test against a small dataset. I don't now how the data source will perform under large loads
In a future iteration I would like to:
📚 Checklist
Summary by CodeRabbit