feat: [SG-43935/43934/43936/43950] Flow AM new data model with new component schemas and asset hierarchy by chenm1adsk · Pull Request #1114 · shotgunsoftware/tk-core

chenm1adsk · 2026-07-08T03:54:34Z

SG-43950 - Support component schemas in schema provisioning

Schema provisioning was limited to kind="type" schemas only. This extends the
provisioning system to handle all schema kinds (component, property, type) correctly,
which is a prerequisite for the new data model schemas introduced in the other tickets.

SG-43935 - Data model derivative update

Updates the DCC asset hierarchy and derivative workflow to align with the new Flow AM
data model. Workfile and derivative assets are now structured as a two-level hierarchy
under a root container asset, replacing the previous task-folder approach. Introduces
component.variantSet and component.source schemas to properly represent variant
groupings and derivation lineage.

SG-43934 - Introduce Layer component for asset hierarchy

Introduces a component.layer schema that tracks each pipeline step as a named
compositional relationship on the root container asset, eliminating the task-folder
level from the hierarchy. This enables the Asset Viewer to correctly display the asset
tree.

SG-43936 - Add Reference components on publish

Adds support for recording scene dependencies as component.reference entries on a
published revision. When a publish includes resolved asset dependencies, each one is
stored as a reference component pointing to the target asset version, enabling
dependency tracking in the Flow AM data model.

Files Changed

File	Changes
`python/tank/flowam/config.json`	Add component.layer, component.variantSet, component.source, component.reference schemas; bump version to 1.0.3
`python/tank/flowam/create.py`	Update hierarchy: root container + LayerComponent per pipeline step, remove task-folder level
`python/tank/flowam/utils.py`	Add `deps` parameter to `create_components_for_publish()`, add ReferenceComponentSpec support
`python/tank_vendor/flow_integration_sdk/globals.py`	Add BASE_COMPONENT_TYPE_ID, BASE_PROPERTY_TYPE_ID, KIND_BASE_TYPE_ID, LAYER_TYPE, VARIANT_SET_TYPE, DER_SOURCE_TYPE, REFERENCE_TYPE
`python/tank_vendor/flow_integration_sdk/objects.py`	Add get_derivatives(), get_layers(), get_sources(), get_variant_sets(), get_references(); fix reference property handling in FlowComponent
`python/tank_vendor/flow_integration_sdk/publish.py`	Add LayerComponentSpec, ReferenceComponentSpec, VariantSetComponentSpec; add components_action param; fix reference-2.0.0 serialization
`python/tank_vendor/flow_integration_sdk/schema.py`	Fix per-kind schema detection loop; fix cache_schema_config kind-aware base type
`python/tank_vendor/flow_integration_sdk/schema_builder.py`	Fix _schema_tree inheritance; add kind validation; formatting

codecov · 2026-07-08T04:01:13Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.10%. Comparing base (024121f) to head (c609362).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1114      +/-   ##
==========================================
- Coverage   80.11%   80.10%   -0.02%     
==========================================
  Files         203      203              
  Lines       19540    19540              
==========================================
- Hits        15654    15652       -2     
- Misses       3886     3888       +2

Flag	Coverage Δ
Linux	`79.54% <ø> (ø)`
Python-3.10	`79.91% <ø> (ø)`
Python-3.11	`79.81% <ø> (-0.02%)`	⬇️
Python-3.13	`79.81% <ø> (-0.02%)`	⬇️
Python-3.9	`79.88% <ø> (-0.02%)`	⬇️
Windows	`79.57% <ø> (-0.02%)`	⬇️
macOS	`79.51% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

yungsiow

Looks good, thanks Ming!

Issue ---- 1. Schema provisioning failed with HTTP 415 errors when querying existing schemas via `schemas_by_super_type`, which triggered cross-org-group enumeration on the backend. 2. Previous implementation of schema module only support `kind="type"` schemas, blocking creation of `kind="component"` schemas. Root Cause: ---- - `schemas_by_super_type` enumerated all org groups in the collection, causing HTTP 415 from unrelated org groups. - `_schema_tree` hardcoded component type IDs (BINARY, COMMENT, DER_SOURCE) as root nodes instead of inheriting from BASE_COMPONENT_TYPE_ID. - `cache_schema_config` always appended BASE_TYPE_ID as parent regardless of schema kind, making component schemas resolve ancestry incorrectly. - Schema creation raised an error if a schema already existed in the collection but not in our library (e.g. on first provisioning run). Fix Applied: ---- - Replace `schemas_by_super_type` with `schemas_by_library_id` scoped to `FLOW_TOOLKIT_LIBRARY_ID`, avoiding cross-org enumeration. - Handle library-not-found (permission error or NOT_FOUND) as empty set so first-run provisioning works correctly. - Add `BASE_COMPONENT_TYPE_ID`, `BASE_PROPERTY_TYPE_ID`, and `KIND_BASE_TYPE_ID` constants to `globals.py`. - Fix `_schema_tree` so component types correctly inherit from BASE_COMPONENT_TYPE_ID`. - Update `cache_schema_config` to use kind-appropriate base type and validate the `kind` field. - Add `kind` as a mandatory key in `SchemaBuilder` validation. - Catch "already exists" error in `build()` and treat it as success, allowing the version field to be updated and skipping re-provisioning on subsequent runs.

…emas_by_library_id

* update dcc workfile and derivative create/publish workflow - add new "variantSet" and "source" components to schema config - removed unnecessary calls to ensure_unique_name() when creating asset hierarchy - replace Task folder in asset hierarchy with a root asset which is of type "container" for now - necessary in order for the Asset Viewer to display its children - move generic assets directly under pipeline step in asset hierarchy - added FlowAsset.get_derivatives() function - switched existing derivative workflow to use new "source" component instead of previous temporary component for designating derivative source - use get_schema_id() within ComponentSpecs to avoid needing to pass in type_id parameter for components using custom schemas - added components_action parameter to publish_new_revision() so that user can choose to append components rather than replace them (which is default behaviour) - formatting for schema_builder.py * formatting * fix: correct reference-2.0.0 property serialization in component specs Properties typed as autodesk.me:reference-2.0.0 are nested schema objects, not primitives. The server expects {"objectId": {"id": "<urn>"}} rather than a plain string, matching the autodesk.data:reference-2.0.0 schema structure. - Add build_reference_value() utility to utils.py to centralize this format - Fix DerivativeSourceComponentSpec.targetVersion to use build_reference_value() - Fix VariantSetComponentSpec.targetAsset to use build_reference_value() * fix: update derivative query filter to match reference-2.0.0 schema structure The targetVersion property is typed as autodesk.me:reference-2.0.0, whose objectId field is autodesk.data:identifier-1.0.0 (a nested object with an id field), not a plain string. After fixing the serialization in DerivativeSourceComponentSpec, the RSQL filter path must reflect the new structure: data.targetVersion.objectId.id instead of data.targetVersion. Without this fix, get_derivatives() always returned empty because the old filter path never matched the nested object, causing a new derivative asset to be created on every publish instead of versioning the existing one. * Move build_reference_value from utils.py to ComponentSpec class * added component utilities - added get_sources() and get_variant_sets() utilities to ComponentMixin - handle reference properties in FlowComponent so that they're more useable. --------- Co-authored-by: chenm1 <ming.chen@autodesk.com>

…ierarchy (#1121) * feat: [SG-43934] Introduce Layer component for asset(pipeline step) hierarchy Part of the new Flow AM data model implementation. The pipeline step hierarchy no longer uses a task-folder level between the pipeline step and the workfile. Instead, a LayerComponent on the root asset container tracks each pipeline step as a named compositional relationship. Changes: - config.json: add component.layer schema (v1.0.3) with name (String) and targetAsset (reference-2.0.0) properties - globals.py: add LAYER_TYPE and LAYER_COMP constants - publish.py: add ComponentSpec.build_reference_value() static method for building reference-2.0.0 property values; add LayerComponentSpec; add components_action parameter to publish_new_revision() to support ListAction.ADD when appending layer components without replacing existing ones - objects.py: add get_layers() to FlowAsset and FlowRevision to read LayerComponents and return referenced layer folder assets - create.py: update get_or_create_workfile_parent() to create pipeline steps as FOLDER_TYPE_ID, publish a LayerComponent onto the root asset container with ListAction.ADD, and remove the task-folder level

* [Data Model] Add Reference components on publish * update dcc workfile and derivative create/publish workflow - add new "variantSet" and "source" components to schema config - removed unnecessary calls to ensure_unique_name() when creating asset hierarchy - replace Task folder in asset hierarchy with a root asset which is of type "container" for now - necessary in order for the Asset Viewer to display its children - move generic assets directly under pipeline step in asset hierarchy - added FlowAsset.get_derivatives() function - switched existing derivative workflow to use new "source" component instead of previous temporary component for designating derivative source - use get_schema_id() within ComponentSpecs to avoid needing to pass in type_id parameter for components using custom schemas - added components_action parameter to publish_new_revision() so that user can choose to append components rather than replace them (which is default behaviour) - formatting for schema_builder.py * formatting * fix: correct reference-2.0.0 property serialization in component specs Properties typed as autodesk.me:reference-2.0.0 are nested schema objects, not primitives. The server expects {"objectId": {"id": "<urn>"}} rather than a plain string, matching the autodesk.data:reference-2.0.0 schema structure. - Add build_reference_value() utility to utils.py to centralize this format - Fix DerivativeSourceComponentSpec.targetVersion to use build_reference_value() - Fix VariantSetComponentSpec.targetAsset to use build_reference_value() * fix: update derivative query filter to match reference-2.0.0 schema structure The targetVersion property is typed as autodesk.me:reference-2.0.0, whose objectId field is autodesk.data:identifier-1.0.0 (a nested object with an id field), not a plain string. After fixing the serialization in DerivativeSourceComponentSpec, the RSQL filter path must reflect the new structure: data.targetVersion.objectId.id instead of data.targetVersion. Without this fix, get_derivatives() always returned empty because the old filter path never matched the nested object, causing a new derivative asset to be created on every publish instead of versioning the existing one. * Move build_reference_value from utils.py to ComponentSpec class * added component utilities - added get_sources() and get_variant_sets() utilities to ComponentMixin - handle reference properties in FlowComponent so that they're more useable. * Fix import * Code review feedback * Code review feedback * More feedback * Refactor `get_references` * Update message --------- Co-authored-by: Yungsiow Yang <yungsiow.yang@autodesk.com> Co-authored-by: chenm1 <ming.chen@autodesk.com>

carlos-villavicencio-adsk

Leave the merge to me