Skip to content

Fixes 30929: Add Data Product as a top-level Explore filter - #31940

Merged
harsh-vador merged 2 commits into
mainfrom
fix-30929-explore-data-product-filter
Aug 25, 2026
Merged

Fixes 30929: Add Data Product as a top-level Explore filter#31940
harsh-vador merged 2 commits into
mainfrom
fix-30929-explore-data-product-filter

Conversation

@harsh-vador

@harsh-vador harsh-vador commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Describe your changes:

Fixes #30929

Data Product could only be filtered from Advanced Search. I added it as a top-level Explore quick filter so it sits next to Domain, Owner, Tag and Tier.

  • COMMON_DROPDOWN_ITEMS and DATA_ASSET_DROPDOWN_ITEMS now carry a label.data-product-plural entry keyed on EntityFields.DATA_PRODUCT (dataProducts.displayName.keyword).
  • SUPPORTED_EMPTY_FILTER_FIELDS gains DATA_PRODUCT, so the dropdown also offers the "No Data Products" (must_not exists) option like Domain/Tier/Tag do.
  • The field is keyword with a lowercase_normalizer in the index mappings, so the item declares sourceFields: 'dataProducts.displayName' — exactly what the Domain filter does — and getOptionsFromAggregationBucket recovers the properly cased display name from the aggregation top hit instead of showing the lowercased bucket key.

No backend change is needed: dataProducts.displayName.keyword is not in the global aggregation list, and ExploreQuickFilters already falls back to /api/v1/search/aggregate?field=<key> for any facet the page aggregations don't carry (which is the path every sourceFields filter takes anyway, Domain included). Filter application is generic — getExploreQueryFilterMust builds the term / must_not exists clauses from the field key.

Screen.Recording.2026-08-24.at.1.41.33.PM.mov
Screen.Recording.2026-08-24.at.8.51.26.PM.mov

Type of change:

  • New feature

High-level design:

N/A — small change.

Tests:

Use cases covered

  • On Explore, a user opens the Data Products dropdown and sees the data products present in
    the current result scope, with proper casing and counts.
  • Selecting a data product narrows results to the assets in it.
  • Selecting No Data Products returns assets that have no data product assigned.

Unit tests

  • I added unit tests for the new/changed logic.
  • Files added/updated: openmetadata-ui/src/main/resources/ui/src/utils/SearchClassBase.test.ts
    (asserts the Data Product filter is exposed for the table and dataAsset indices)

Backend integration tests

  • Not applicable (no backend API changes).

Ingestion integration tests

  • Not applicable (no ingestion changes).

Playwright (UI) tests

  • I added Playwright E2E tests for UI changes.
  • Files added/updated: openmetadata-ui/src/main/resources/ui/playwright/e2e/Features/ExploreQuickFilters.spec.ts
    • new should filter assets by data product test (creates a data product in the fixture domain,
      adds the fixture table to it, filters Explore by it and asserts the asset card)
    • should search for empty or null filters now also covers Data Products

Manual testing performed

Not yet run against a local stack — reviewer-facing verification below is what I ran:

  • npx jest src/utils/SearchClassBase.test.ts src/components/Explore src/components/ExploreV1 → 290 passed, 4 skipped
  • npx jest src/utils/AdvancedSearchUtils src/components/Glossary/GlossaryTerms/tabs src/components/SearchDropdown → 141 passed
  • ESLint + Prettier clean on all changed files; tsc --noEmit reports no errors in the changed files

UI screen recording / screenshots:

Pending — will attach a recording of the Data Products dropdown on Explore.

Checklist:

  • I have read the CONTRIBUTING document.
  • My PR is linked to a GitHub issue via Fixes #30929 above.
  • I have commented on my code, particularly in hard-to-understand areas.
  • For JSON Schema changes: not applicable.
  • For UI changes: I attached a screen recording and/or screenshots above.
  • I have added tests (unit + Playwright) and listed them above.

🤖 Generated with Claude Code

Greptile Summary

Adds Data Product as a top-level Explore quick filter, including support for filtering assets without an assigned data product.

  • Adds the Data Products facet to common and aggregate asset dropdown definitions.
  • Enables the empty-value option for the Data Product field.
  • Adds unit and Playwright coverage for facet exposure, selection, and empty filtering.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
openmetadata-ui/src/main/resources/ui/src/constants/AdvancedSearch.constants.ts Adds the Data Products facet and its display-name source field to common and data-asset quick-filter definitions.
openmetadata-ui/src/main/resources/ui/src/constants/explore.constants.ts Enables the generic empty-value filtering behavior for Data Products.
openmetadata-ui/src/main/resources/ui/src/utils/SearchClassBase.test.ts Verifies that table and aggregate data-asset scopes expose the new quick filter.
openmetadata-ui/src/main/resources/ui/playwright/e2e/Features/ExploreQuickFilters.spec.ts Adds fixture setup, indexing synchronization, and end-to-end coverage for Data Product filtering.

Reviews (2): Last reviewed commit: "test(ui): wait for the data product link..." | Re-trigger Greptile

Data Product could only be filtered through Advanced Search. Add it to the
common and data-asset quick-filter dropdowns so it sits alongside Domain,
Owner, Tag and Tier on Explore, and allow the "No Data Products" empty-value
option.

`dataProducts.displayName.keyword` is normalized to lowercase in the index, so
the filter carries `sourceFields: 'dataProducts.displayName'` (same as Domain)
to recover the properly cased label from the aggregation top hit.

Closes #30929

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added the UI UI specific issues label Aug 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows
will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

✅ Playwright Results — workflow succeeded

Validated commit fd74c01af60a754abc26c604fddbb8e57d6b8974 in Playwright run 32736937385, attempt 1.

✅ 569 passed · ❌ 0 failed · 🟡 1 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 54m 58s

⏱️ Max setup 6m 40s · max shard execution 17m 27s · max shard-job elapsed before upload 23m 21s · reporting 5s

🌐 213.60 requests/attempt · 2.78 app boots/UI scenario · 22.73% common-shard skew

Optimization targets still in progress:

  • Common shard skew was 22.73% (convergence target: at most 15%).
  • Browser traffic was 213.6 requests per attempt (convergence target: fewer than 200).
  • Application boot ratio was 2.78 per UI scenario (1643 boots / 592 scenarios; convergence target: at most 1).
Shard Passed Failed Flaky Skipped Lifecycle failed Lifecycle flaky
✅ Shard chromium-01 148 0 0 0 0 0
🟡 Shard chromium-02 141 0 1 0 0 0
✅ Shard chromium-03 126 0 0 0 0 0
✅ Shard data-asset-rules-01 61 0 0 0 0 0
✅ Shard domain-isolation-01 16 0 0 0 0 0
✅ Shard global-state-01 34 0 0 0 0 0
✅ Shard ingestion-01 1 0 0 0 0 0
✅ Shard reindex-01 2 0 0 0 0 0
✅ Shard search-01 11 0 0 0 0 0
✅ Shard search-rbac-01 29 0 0 0 0 0
🟡 1 flaky test(s) (passed on retry)
  • Pages/Entity.spec.tsAnnouncement create, edit & delete (shard chromium-02, 1 retry)

📦 Download artifacts

How to debug locally
# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

@harsh-vador harsh-vador self-assigned this Aug 24, 2026
@harsh-vador harsh-vador added the safe to test Add this label to run secure Github workflows on PRs label Aug 24, 2026
@github-actions

Copy link
Copy Markdown
Contributor

⚠️ UI Checkstyle passed — lint findings in changed files

🔍 ESLint findings in this PR's files — 0 error(s), 18 warning(s)

Errors block the build. Warnings do not yet — they are rules whose backlog is still
being worked down, listed so this PR does not add to it. See docs/ui-code-quality-gate.md.

0 error(s), 18 warning(s) across 2 changed file(s).

Count Rule
18 sonarjs/no-duplicate-string
All findings
Location Rule Message
🟡 src/constants/AdvancedSearch.constants.ts:22:12 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 7 times.
🟡 src/constants/AdvancedSearch.constants.ts:24:19 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 7 times.
🟡 src/constants/AdvancedSearch.constants.ts:32:12 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 8 times.
🟡 src/constants/AdvancedSearch.constants.ts:39:19 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 7 times.
🟡 src/constants/AdvancedSearch.constants.ts:42:12 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 5 times.
🟡 src/constants/AdvancedSearch.constants.ts:44:19 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 5 times.
🟡 src/constants/AdvancedSearch.constants.ts:47:12 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 5 times.
🟡 src/constants/AdvancedSearch.constants.ts:49:19 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 5 times.
🟡 src/constants/AdvancedSearch.constants.ts:52:12 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 5 times.
🟡 src/constants/AdvancedSearch.constants.ts:107:12 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 3 times.
🟡 src/constants/AdvancedSearch.constants.ts:112:12 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 3 times.
🟡 src/constants/AdvancedSearch.constants.ts:117:12 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 4 times.
🟡 src/constants/AdvancedSearch.constants.ts:282:12 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 3 times.
🟡 src/constants/AdvancedSearch.constants.ts:284:15 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 3 times.
🟡 src/constants/explore.constants.ts:20:40 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 5 times.
🟡 src/constants/explore.constants.ts:46:11 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 4 times.
🟡 src/constants/explore.constants.ts:53:11 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 4 times.
🟡 src/constants/explore.constants.ts:55:11 sonarjs/no-duplicate-string Define a constant instead of duplicating this literal 4 times.

Fix locally (fast - only checks files changed in this branch):

make ui-checkstyle-changed

@github-actions

github-actions Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Jest test Coverage

UI tests summary

Lines Statements Branches Functions
Coverage: 67%
67.06% (80441/119952) 51.47% (49252/95678) 52.45% (14705/28032)

shah-harshit
shah-harshit previously approved these changes Aug 24, 2026
Adding an asset to a data product updates the asset's search document
asynchronously, and the Data Products dropdown builds its options from that
aggregation — so opening the dropdown right after the bulk add could miss the
option. Poll the table's search doc until it carries the data product.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gitar-bot

gitar-bot Bot commented Aug 24, 2026

Copy link
Copy Markdown
Code Review ✅ Approved

Adds Data Product as a top-level Explore quick filter alongside Domain and Tier, complete with proper aggregation casing and empty-filter support. No issues found.

Options

Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Compact
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source

@sonarqubecloud

Copy link
Copy Markdown

@harsh-vador
harsh-vador added this pull request to the merge queue Aug 25, 2026
Merged via the queue into main with commit a40588c Aug 25, 2026
93 checks passed
@harsh-vador
harsh-vador deleted the fix-30929-explore-data-product-filter branch August 25, 2026 06:46
harsh-vador added a commit that referenced this pull request Aug 25, 2026
* feat(ui): add Data Product as a top-level Explore filter

Data Product could only be filtered through Advanced Search. Add it to the
common and data-asset quick-filter dropdowns so it sits alongside Domain,
Owner, Tag and Tier on Explore, and allow the "No Data Products" empty-value
option.

`dataProducts.displayName.keyword` is normalized to lowercase in the index, so
the filter carries `sourceFields: 'dataProducts.displayName'` (same as Domain)
to recover the properly cased label from the aggregation top hit.

Closes #30929

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(ui): wait for the data product link to be indexed before filtering

Adding an asset to a data product updates the asset's search document
asynchronously, and the Data Products dropdown builds its options from that
aggregation — so opening the dropdown right after the bulk add could miss the
option. Poll the table's search doc until it carries the data product.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit a40588c)
harsh-vador added a commit that referenced this pull request Aug 25, 2026
* feat(ui): add Data Product as a top-level Explore filter

Data Product could only be filtered through Advanced Search. Add it to the
common and data-asset quick-filter dropdowns so it sits alongside Domain,
Owner, Tag and Tier on Explore, and allow the "No Data Products" empty-value
option.

`dataProducts.displayName.keyword` is normalized to lowercase in the index, so
the filter carries `sourceFields: 'dataProducts.displayName'` (same as Domain)
to recover the properly cased label from the aggregation top hit.

Closes #30929

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(ui): wait for the data product link to be indexed before filtering

Adding an asset to a data product updates the asset's search document
asynchronously, and the Data Products dropdown builds its options from that
aggregation — so opening the dropdown right after the bulk add could miss the
option. Poll the table's search doc until it carries the data product.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit a40588c)
Rohit0301 added a commit that referenced this pull request Aug 26, 2026
…32015)

* Fixes 30929: Add Data Product as a top-level Explore filter (#31940)

* feat(ui): add Data Product as a top-level Explore filter

Data Product could only be filtered through Advanced Search. Add it to the
common and data-asset quick-filter dropdowns so it sits alongside Domain,
Owner, Tag and Tier on Explore, and allow the "No Data Products" empty-value
option.

`dataProducts.displayName.keyword` is normalized to lowercase in the index, so
the filter carries `sourceFields: 'dataProducts.displayName'` (same as Domain)
to recover the properly cased label from the aggregation top hit.

Closes #30929

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(ui): wait for the data product link to be indexed before filtering

Adding an asset to a data product updates the asset's search document
asynchronously, and the Data Products dropdown builds its options from that
aggregation — so opening the dropdown right after the bulk add could miss the
option. Poll the table's search doc until it carries the data product.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit a40588c)

* fix spec

* fix(ui): Fix lineage canvas scrolling when the filters panel is expanded [1.13] (#32077)

* Fixes #29940: render the tab that is on screen when the URL has no tab segment (#31448) (#32029)

* Fixes #29940 - Render the tab that is on screen when the URL has no tab

`GenericProvider` read the active tab only from the `:tab` route param and,
when it was absent, fell back to the first tab of the persona-customized page.
Entity landing URLs carry no tab segment (`/domain/:fqn`), and the domain tree
view keeps the tab in local state and never writes it to the URL at all, so the
provider resolved a different tab than the one the page was rendering.

With a persona customization that reorders the tabs, the Documentation tab
rendered the first customized tab's layout instead of its own. Every
non-documentation domain tab persists an empty layout, so the pane came up
blank. Clicking another tab and back recovered it (the URL then carried a tab);
in the tree view it never recovered.

Give `GenericProvider` an explicit `activeTab` prop, defaulting to the route
param, and pass the rendered tab from the pages that default the tab bar to a
fixed tab rather than letting it land on the customized first tab. The
`filteredKeys` reset now keys off the same value, so it also fires on an
inline tab switch in the tree view.



* fix persona level issue

* fix the tree level persona issue

* chore: drop activeTab pass-through from entity pages that don't need it

These 13 entity-detail pages (APIEndpoint, Chart, Dashboard, DriveService/*,
Metric, Topic, DatabaseSchema, SearchIndex, StoredProcedure, TagPage) always
carry a :tab URL segment, so GenericProvider's existing routeTab fallback is
sufficient. Passing activeTab={activeTab} here is a no-op and adds needless
churn to the fix for #29940.

The actual fix lives in GenericProvider, DomainDetails, DomainTreeView,
DataProductsDetailsPage, getRenderedActiveTab, and DomainClassBase — the six
files that address the real root cause.



* fix lint checks

* addressed comments

---------



(cherry picked from commit 489eb6e)

Co-authored-by: Sid <30566406+siddhant1@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>

* Fixes #31980: handle null persona customization pages (#31981) (#32037)

Backport of #31981 to the 1.13 branch.

The 1.13 branch predates the react-query docStore refactor, so the fix is
adapted to 1.13's data-fetching path rather than cherry-picked verbatim:

- Add PersonaPage.utils.ts (getPersonaPage / normalizePersonaDocument /
  updatePersonaDocumentPage), which safely filters null/invalid page entries
  before they reach the UI. Ported unchanged from the source PR since it only
  depends on generated types.
- useCustomPages, CustomizeStore, CustomizablePage, MyDataPage, and
  DataMarketplacePage now use these helpers instead of raw
  data.pages.find(...) / manual page mutation.
- MyDataPage guards against a non-array customized layout and falls back to
  the default layout.

Files that do not exist on 1.13 were omitted:
- rest/queries/docStoreQuery.ts (react-query layer not present)
- DataMarketplacePage.component.test.tsx (page has no test on 1.13)

The react-query-specific test additions to useCustomPages.test.ts were also
omitted; the equivalent MyDataPage tests (null legacy page + invalid layout)
were ported and adapted to 1.13's getDocumentByFQN-based test harness.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(mlflow): use single-quoted filter for UC model version search (#31387)

#30856 built the version-search filter with double quotes, which the
Databricks endpoint rejects:

  INVALID_PARAMETER_VALUE: Unsupported filter query : `name="..."`.
  Please specify your filter parameter in the format `name = 'model_name'`.

So UC ingestion cleared the original TypeError and then failed one step
later, recording every model as "Version not found".

Unity Catalog does not parse this filter locally -- UcModelRegistryStore
forwards the raw string to the Databricks REST endpoint. The double-quoted
form was validated against MLflow's client-side SearchModelUtils parser,
which is more permissive and is never invoked on the UC path, so that check
proved nothing about the code path that runs.

Revert to name='<model_name>', the form the reporter verified against their
own registry. Replace the round-trip test, whose premise this falsifies,
with one pinning the required syntax against the real failing name.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* fix(ingestion): detect missing Lake Formation grants in Athena test connection (#29515)

Backported to 1.13, which still exposes the Athena handler as module-level
functions rather than main's BaseConnection subclass. The GetTables/GetViews
behaviour is identical; only the plumbing differs:

- `_get_targeted_schemas` becomes a module-level `get_targeted_schemas`
  taking the service connection explicitly instead of reading `self`.
- The executors close over the `engine` and `service_connection` parameters
  instead of `self.client` / `self.service_connection`.
- Ruff's `# noqa: UP006/UP035` markers are dropped (1.13 lints with
  black/isort/pylint, not ruff) and the code is black-formatted at 88 cols.

Tests are ported minus the cases that only assert main's class refactor
(BaseConnection subclassing, engine disposal, classmethod URL builder). The
URL tests are omitted as duplicates - 1.13 already covers them in
tests/unit/test_source_connection.py. `get_connection_url`, `get_connection`
and `get_lake_formation_client` are left byte-identical to 1.13.

(cherry picked from commit 10aefa4)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* feat(bigquery): ingest table last-modified timestamp into lifeCycle.updated (#30316)

Backported to 1.13. The execution behaviour matches main: the mixin parses an
optional UPDATED_AT column and populates lifeCycle.updated when present, and
BigQuery uses the region-scoped INFORMATION_SCHEMA query (joining TABLE_STORAGE
for storage_last_modified_time) whenever the dataset region resolves, falling
back to the dataset-scoped created-only query otherwise.

Adapted for 1.13, which has diverged from main in two ways:

- BigquerySource caches dataset objects in a per-schema LRUCache
  (_dataset_obj_cache), where main still uses a single-slot
  _current_dataset_obj. The ported get_life_cycle_query tests reset the keyed
  cache instead, and 1.13's TestBigqueryPerSchemaCaching / _EvictedOnReadCache
  coverage is preserved untouched.
- 1.13 lints with black/isort/pylint rather than ruff, so the `# noqa:
  UP006/UP045` markers are dropped and the code is black-formatted at 88 cols.
  The `# pyright: ignore` comments are kept (1.13 does gate on basedpyright)
  and repositioned onto the lines black's rewrap moved them off, so the new
  code adds no type errors.

The two life-cycle query-shape tests from main are ported as-is; main's
stored-procedure query imports are omitted because 1.13's stored-procedure
tests assert on query text rather than the constants.

Redshift and Snowflake are unaffected: neither life-cycle query selects an
updated_at column, so lifeCycle.updated stays unset for them, and
get_life_cycle_query() receives exactly the arguments the inlined format did.

(cherry picked from commit 4381871)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(db2): adapt sqlalchemy-ibmi dialect to SQLAlchemy 2.0 (#30731)

The ibmi (DB2 for i) scheme failed its connection test at the first
step with "Not an executable object: 'VALUES CURRENT_SCHEMA'".

sqlalchemy-ibmi 0.9.3 pins sqlalchemy<2 but is installed with --no-deps,
so its SA-1.x call sites survive into our SA 2.0 runtime and fail on
first use. The dialect's initialize() passes raw strings to
Connection.execute(), which SA 2.0 rejects, and its reflection queries
use the removed select([cols], whereclause) form.

Patch the dialect at runtime before building the ibmi engine: wrap both
raw-string statements in text(), and reassign the module-level select to
a compat shim. The shim covers all 11 legacy reflection call sites at
once because the dialect resolves select as a global on each call, which
avoids reimplementing ~250 lines of vendor SQL that would drift from
upstream. The shim carries order_by over, which get_foreign_keys and
get_indexes rely on to group composite constraints in column order, and
patch_ibmi_dialect resolves the dialect via getattr so a partially
initialised module falls back instead of raising AttributeError.

(cherry picked from commit 1ab7634)

Adapted for 1.13: the patch call is placed in the ibmi branch of
get_connection rather than Db2IbmiStrategy.build(), since 1.13 predates
the connection Strategy refactor. Only the tests this commit introduced
are ported; the pre-existing url/args/dispatch tests belong to that
refactor and are left out.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(ingestion): prevent Airflow metadata loss on API failures (#31371)

Cherry-picked from cdff317 on main.

* fix(ingestion): preserve Airflow metadata on API failures
* fix(ingestion): satisfy Airflow source type check
* fix(ingestion): harden Airflow pagination failures
* test(ingestion): complete Airflow task instance page

(cherry picked from commit cdff317)

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Fixes 31401: Handle unnamed and quoted ROW fields in Trino reflection (#31402)

* fix(trino): handle unnamed and quoted ROW fields during reflection

Two defects in the Trino ROW/struct type parser, both verified against a
live Trino 483 instance.

1. Tables with unnamed ROW fields lost every column. Trino allows the field
   name to be omitted -- `CREATE TABLE t AS SELECT ROW(1,'x') AS s` reports
   `row(integer, varchar(1))` -- but the parser unpacked each field as
   `name type` and raised ValueError. sql_column_handler swallows that and
   falls back to an empty column list, so the table was published with zero
   columns behind a WARNING while the pipeline still reported success.
   The driver's own `datatype.parse_sqltype` has the same defect and runs
   first, so it is guarded too; the OpenMetadata data type comes from
   `system_data_type` for complex columns, so NULLTYPE there is harmless.
   Unnamed fields are named positionally to match Trino's 1-based field
   access, so `s[1]` reads as `field1`.

2. Struct child columns were named with literal quotes. Trino renders named
   fields quoted (`row("a" bigint)`, escaping an embedded quote as `""`), so
   children were `"a"` instead of `a`, breaking nested column FQNs and
   column-level lineage matching. The type string is also no longer
   lower-cased wholesale, which preserves the case of quoted field names in
   dataTypeDisplay.

A leading quote is what distinguishes a named field from an unnamed one
whose type contains spaces (`timestamp(3) with time zone`,
`interval day to second`), both of which are now covered.

Tests use verbatim `SHOW COLUMNS` output captured from Trino 483; the
existing fixtures used an unquoted form that Trino never emits, which is
why this went unnoticed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(trino): keep positional ROW field names from shadowing explicit ones

A ROW may mix an unnamed field with one named literally `fieldN`:
`CAST(ROW(1,'x') AS row(bigint, "field1" varchar))` is accepted by Trino 483
and reported as `row(bigint, "field1" varchar)`. Naming the unnamed field
positionally then produced two struct children called `field1`, which makes
the nested column FQN ambiguous and breaks column-level lineage that assumes
unique child names. Verified end-to-end: both children arrived as `field1`
with no validation error anywhere in the ingestion path.

Field naming moves out of split_row_field, which now reports an unnamed field
as None (mirroring how the driver's own ROW type models it), into
resolve_field_names, which sees the whole ROW and suffixes a positional name
until it is unique.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit 0048e2f)

* Fixes #30597: Parse array-root REST API schemas (#31698)

* fix(ingestion): parse array-root REST schemas

* fix(ingestion): avoid duplicate schema warnings

(cherry picked from commit afcf248)

* fix(ui): Fix lineage canvas scrolling when the filters panel is expanded [1.13]

Replaces explicit height/scroll calculations on .ant-card-body with a
flex-column layout so the canvas occupies available space when the filters
panel opens, preventing bottom control buttons from scrolling out of view.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(ui): adjust fullscreen lineage height after linter update [1.13]

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Anujkumar Yadav <anujf0510@gmail.com>
Co-authored-by: Sid <30566406+siddhant1@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Harsh Soni <64592571+harshsoni2024@users.noreply.github.com>
Co-authored-by: harshsoni2024 <harshsoni2024@gmail.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Rohit Jain <60229265+Rohit0301@users.noreply.github.com>
Co-authored-by: Anujkumar Yadav <anujf0510@gmail.com>
Co-authored-by: Sid <30566406+siddhant1@users.noreply.github.com>
Co-authored-by: Karan Hotchandani <33024356+karanh37@users.noreply.github.com>
Co-authored-by: Harsh Soni <64592571+harshsoni2024@users.noreply.github.com>
Co-authored-by: harshsoni2024 <harshsoni2024@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

safe to test Add this label to run secure Github workflows on PRs UI UI specific issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Data Product as a Top-Level Filter in Explore Page

4 participants