Skip to content

Normalize bare entity IDs in test parameter preparation - #122

Open
Oluwajuwon-O wants to merge 3 commits into
datakind:mainfrom
Oluwajuwon-O:feature/bare_entity_ids
Open

Normalize bare entity IDs in test parameter preparation#122
Oluwajuwon-O wants to merge 3 commits into
datakind:mainfrom
Oluwajuwon-O:feature/bare_entity_ids

Conversation

@Oluwajuwon-O

@Oluwajuwon-O Oluwajuwon-O commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR addresses the entity identifier and parameter normalization issues described in the technical documentation .

Hardcoded Execution Artifacts in Test Parameters

Previously, test parameters stored in the database required users to hardcode internal execution prefixes (e.g., dot_model__) or dbt macro syntax (e.g., ref('dot_model__...')). This leaked DOT's internal execution implementation details into the user-facing configuration, making the test metadata brittle and domain-specific.

Resolution

  • Bare Entity ID Normalization: Introduced to_bare_entity_id, to_dbt_ref, and to_dbt_table in utils/configuration_utils.py. This allows the database to store clean, domain-neutral entity strings (e.g., all_airports_data).
  • Dynamic Artifact Generation: Updated generate_tests_from_db to pass parameters through a new prepare_test_parameters function. This dynamically wraps bare entity IDs into the correct dbt or Great Expectations syntax at runtime.
  • Backward Compatibility & Aliasing: The normalization layer safely parses legacy ref() strings and translates deprecated parameter keys (e.g., reference to to, form_name to data_table) to ensure older project configurations do not break.
  • Unit Coverage: Added test_entity_id_normalize.py to assert expected behavior across bare IDs, prefixed tables, quoted dbt references, and legacy parameter aliases.

In the Technical Documentation, this is addressed in Section 6.

Asana Task

Deployment Readiness*

Testing

Describe or check:

  • Created or updated unit, feature, and/or integration tests
  • Typical manual testing in the local env browser, dev pipeline, etc.

Deployment Notes

Describe or check:

  • No special deployment steps required

Rollback Plan

Describe or check:

  • Standard revert is sufficient (git revert)

Reviewer Guidance / Questions*

Screenshots / Testing Evidence*

SOC 2 Change Management Checklist

  • None of the below are true in this code
  • New roles/permissions are introduced without review and approval by the product manager
  • Hardcoded credentials, secrets, or API keys are present in this code
  • Secrets are being managed outside of the approved secrets management process (e.g., GitHub Secrets, environment variables)
  • PII or sensitive data handling is introduced or changed without being reviewed against our data classification policy
  • Sensitive data is written to logs
  • Input validation and sanitization is missing
  • An unnecessary attack surface has been introduced (e.g., unused endpoints, open ports, debug modes left enabled)
  • Common vulnerabilities have been introduced in the code (inc. any dependencies added or updated)
  • No review for common vulnerabilities has been conducted
  • Not tested in a non-production environment
  • Breaking changes to existing APIs or integrations with downstream consumers being notified
  • Performance impact has not been considered or acceptable
  • Appropriate audit logging is missing for any security-relevant actions introduced by this change
  • Log entries contain sensitive or PII data
  • All existing tests do not pass locally (./vendor/bin/pest)

Provide justification if you are submitting a PR with any boxes checked other than the first.


Reminder for Reviewers: By approving this PR you are confirming that you have reviewed the code for correctness, security, and compliance with our engineering and SOC 2 standards. Do not approve PRs where SOC 2 checklist items are checked without documented justification.

*Optional


@Oluwajuwon-O Oluwajuwon-O changed the title Bare entity IDs and parameter preparation Normalize bare entity IDs in test parameter preparation Aug 5, 2026
CI lints only changed Python files; the new unit test module fell below
the pylint threshold because pytest-style methods lack method docstrings.
@Oluwajuwon-O
Oluwajuwon-O force-pushed the feature/bare_entity_ids branch from f52bef6 to 8d16ad9 Compare August 5, 2026 12:09
Oluwajuwon-O added a commit to Oluwajuwon-O/Data-Observation-Toolkit that referenced this pull request Aug 5, 2026
Same CI pylint threshold issue as datakind#122; this stacked branch still carried
the pre-fix test module and failed lint at 6.79 (< 7.0).
@Oluwajuwon-O
Oluwajuwon-O marked this pull request as ready for review August 6, 2026 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant