fix(security): upgrade Apache Airflow to 3.2.1

[akashverma0786]

…resolve CVEs

Describe your changes:

Fixes #

I worked on ... because ...

Type of change:

• Bug fix
• Improvement
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

High-level design:

N/A — small change.

Tests:

Use cases covered

Unit tests

Backend integration tests

Ingestion integration tests

Playwright (UI) tests

Manual testing performed

UI screen recording / screenshots:

Not applicable.

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• My PR is linked to a GitHub issue via Fixes #<issue-number> above.
• I have commented on my code, particularly in hard-to-understand areas.
• For JSON Schema changes: I updated the migration scripts or explained why it is not needed.
• For UI changes: I attached a screen recording and/or screenshots above.
• I have added tests (unit / integration / Playwright as applicable) and listed them above.

---

Summary by Gitar

• Airflow Compatibility:
  • Added support for Airflow 3.x compressed DAG data by implementing zlib decompression in _resolve_dag_data.
  • Updated get_pipelines_list to fetch and handle the _data_compressed column from SerializedDagModel.
• Dependency Updates:
  • Upgraded sqlalchemy-pytds to ~=1.0 in setup.py to fix compatibility issues with python-tds 1.x.
• Bug Fixes:
  • Refactored _test_task_detail_access to fall back to a dag_id query when _data is NULL, ensuring stability with enabled compression.

_{This will update automatically on new commits.}

[github-actions]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[github-actions]

🔴 Playwright Results — 1 failure(s), 12 flaky

✅ 4139 passed · ❌ 1 failed · 🟡 12 flaky · ⏭️ 86 skipped

Shard	Passed	Failed	Flaky	Skipped
✅ Shard 1	299	0	0	4
🟡 Shard 2	777	0	4	8
🔴 Shard 3	776	1	1	7
🟡 Shard 4	781	0	1	12
🟡 Shard 5	772	0	1	47
🟡 Shard 6	734	0	5	8

Genuine Failures (failed on all attempts)

❌ Features/RTL.spec.ts › Verify Following widget functionality (shard 3)

Error: page.goto: net::ERR_ABORTED at http://localhost:8585/table/pw-database-service-0ee3dc73.pw-database-c5f9fa09.pw-database-schema-49af75ae.pw-table-32ad1580-5d13-473a-adc3-0360196a480f
Call log:
�[2m  - navigating to "http://localhost:8585/table/pw-database-service-0ee3dc73.pw-database-c5f9fa09.pw-database-schema-49af75ae.pw-table-32ad1580-5d13-473a-adc3-0360196a480f", waiting until "domcontentloaded"�[22m

🟡 12 flaky test(s) (passed on retry)

• Features/BulkEditEntity.spec.ts › Glossary (shard 2, 1 retry)
• Features/BulkImport.spec.ts › Keyboard Delete selection (shard 2, 1 retry)
• Features/Glossary/GlossaryWorkflow.spec.ts › should display correct status badge color and icon (shard 2, 1 retry)
• Features/KnowledgeCenter.spec.ts › Article mentions in description should working for Knowledge Center (shard 2, 1 retry)
• Features/TestSuitePipelineRedeploy.spec.ts › Re-deploy all test-suite ingestion pipelines (shard 3, 1 retry)
• Pages/CustomProperties.spec.ts › Set & Update all CP types on apiCollection (shard 4, 1 retry)
• Pages/EntityDataConsumer.spec.ts › Tier Add, Update and Remove (shard 5, 1 retry)
• Pages/Glossary.spec.ts › Change glossary term hierarchy using menu options across glossary (shard 6, 1 retry)
• Pages/InputOutputPorts.spec.ts › Lineage section is collapsed by default (shard 6, 1 retry)
• Pages/Lineage/LineageFilters.spec.ts › Verify lineage schema filter selection (shard 6, 1 retry)
• Pages/Lineage/LineageRightPanel.spec.ts › Verify custom properties tab IS visible for supported type: searchIndex (shard 6, 1 retry)
• Pages/Teams.spec.ts › Delete a user from the table (shard 6, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[gitar-bot]

Code Review 👍 Approved with suggestions 4 resolved / 5 findings

Upgrades Apache Airflow to 3.2.1 and adds support for compressed DAG data, while addressing several dependency and code quality issues. Consider also catching json.JSONDecodeError in the _resolve_dag_data error handling block to ensure robustness.

💡 Edge Case: _resolve_dag_data catches zlib.error but not json.JSONDecodeError

📄 ingestion/src/metadata/ingestion/source/pipeline/airflow/metadata.py:487-494

The _resolve_dag_data method at line 488 calls json.loads(zlib.decompress(compressed_data)) but only catches zlib.error. If the data decompresses successfully but is not valid JSON, json.JSONDecodeError will bubble up. While the outer loop in get_pipelines_list (line 639) has a broad except Exception, the error message will be generic ('Wild error yielding dag...') rather than pointing the user to the decompression/serialization issue. Catching both exceptions here would provide a much better diagnostic message.

Catch both zlib.error and json.JSONDecodeError to provide a helpful diagnostic message for either failure mode.

try:
    return json.loads(zlib.decompress(compressed_data))
except (zlib.error, json.JSONDecodeError) as exc:
    logger.warning(
        f"Failed to decompress/parse serialized DAG data for '{dag_id}'. "
        f"Ensure COMPRESS_SERIALIZED_DAGS uses zlib compression (the Airflow default): {exc}"
    )
    return None

✅ 4 resolved

✅ Quality: upath workaround lacks comment and ticket reference

📄 ingestion/Dockerfile.ci:140
Line 140 in Dockerfile.ci adds RUN pip uninstall upath -y && pip install "universal-pathlib==0.3.10" without any comment explaining why this workaround is needed. Per project guidelines, workarounds should include a comment with a ticket reference so the team knows when it can be removed. Additionally, this workaround is absent from ingestion/Dockerfile, which may cause inconsistency between the two images.

✅ Edge Case: Unbounded version constraints for gcsfs and s3fs

📄 ingestion/setup.py:31 📄 ingestion/setup.py:73
The version specifiers for gcsfs and s3fs were changed from ~=2023.12.1 (which constrains to >=2023.12.1, ==2023.12.*) to >=2023.12.1 (no upper bound). Both libraries use calendar versioning and may introduce breaking API changes in future major releases. This risks build or runtime failures when a new incompatible version is published.

If the goal is to allow newer versions needed by Airflow 3.2.1, consider using a bounded range (e.g., >=2023.12.1,<2026.0.0) or a compatible-release operator at the year level (~=2023.12) to allow patch/minor updates while preventing major breakage.

✅ Bug: Duplicate imports shadow each other in three files

📄 ingestion/src/metadata/ingestion/source/database/snowflake/models.py:18 📄 ingestion/src/metadata/ingestion/source/database/snowflake/models.py:21 📄 ingestion/src/metadata/ingestion/ometa/utils.py:20 📄 ingestion/src/metadata/ingestion/ometa/utils.py:23 📄 ingestion/src/metadata/utils/entity_link.py:18 📄 ingestion/src/metadata/utils/entity_link.py:24
The PR adds from urllib.parse import ... lines but leaves existing imports of the same name from requests.utils / requests.compat intact. In Python, the later import shadows the earlier one, making the added import dead code. While functionally harmless today (both resolve to the same stdlib function), this creates confusion and linting noise:

• models.py: from urllib.parse import quote (line 18) is immediately shadowed by from requests.utils import quote (line 21)
• utils.py: from urllib.parse import quote as url_quote (line 20) is shadowed by from requests.utils import quote as url_quote (line 23)
• entity_link.py: from urllib.parse import unquote_plus (line 18) is shadowed by from requests.compat import unquote_plus (line 24)

If the intent is to migrate away from requests.utils/requests.compat re-exports, the old imports should be removed. If the intent is a fallback, only one should remain.

✅ Quality: Test file entirely commented out instead of deleted or fixed

📄 ingestion/tests/unit/topology/dashboard/test_lookml_bitbucket_reader.py:1-15
The entire test file test_lookml_bitbucket_reader.py has been commented out (59 lines). Per project guidelines, commented-out code is not allowed — code should either be deleted or fixed. If the test is failing due to the Airflow upgrade and needs to be temporarily disabled, it should be deleted with a TODO referencing a ticket for re-enabling it, or the test should be updated to work with the new dependencies.

🤖 Prompt for agents

Code Review: Upgrades Apache Airflow to 3.2.1 and adds support for compressed DAG data, while addressing several dependency and code quality issues. Consider also catching json.JSONDecodeError in the _resolve_dag_data error handling block to ensure robustness.

1. 💡 Edge Case: _resolve_dag_data catches zlib.error but not json.JSONDecodeError
   Files: ingestion/src/metadata/ingestion/source/pipeline/airflow/metadata.py:487-494

   The `_resolve_dag_data` method at line 488 calls `json.loads(zlib.decompress(compressed_data))` but only catches `zlib.error`. If the data decompresses successfully but is not valid JSON, `json.JSONDecodeError` will bubble up. While the outer loop in `get_pipelines_list` (line 639) has a broad `except Exception`, the error message will be generic ('Wild error yielding dag...') rather than pointing the user to the decompression/serialization issue. Catching both exceptions here would provide a much better diagnostic message.

   Fix (Catch both zlib.error and json.JSONDecodeError to provide a helpful diagnostic message for either failure mode.):
   try:
       return json.loads(zlib.decompress(compressed_data))
   except (zlib.error, json.JSONDecodeError) as exc:
       logger.warning(
           f"Failed to decompress/parse serialized DAG data for '{dag_id}'. "
           f"Ensure COMPRESS_SERIALIZED_DAGS uses zlib compression (the Airflow default): {exc}"
       )
       return None

Options

Display: compact → Showing less information.

Comment with these commands to change:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Gitar}

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
20.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud