Refactor domain-neutral duplicate detection parameters

[Oluwajuwon-O]

Description

This PR addresses the domain-neutral duplicate detection issues described in the technical documentation:

Healthcare-Specific Duplicate Detection

Previously, the dbt macro used to flag duplicate records (test_possible_duplicate_forms.sql) was heavily coupled to healthcare terminology. It expected parameters like table_specific_reported_date and table_specific_patient_uuid. This hardcoded vocabulary limited the reusability of the test for other domains (e.g., education or finance), where terms like "patient" or "form" are contextually inaccurate.

Resolution

• Domain-Neutral Macro: Refactored test_possible_duplicate_forms.sql to use generic parameter names: date_column, group_column, id_column, and period.
• Database & Documentation Updates: Updated 2-upload_static_data.sql, 4-upload_sample_dot_data.sql, and README.md to reflect the new parameter interface.
• Backward Compatibility: Added a translation layer in utils/configuration_utils.py (prepare_test_parameters). If DOT encounters legacy healthcare keys in a user's database, it automatically maps them to the new domain-neutral keys at runtime.
• Unit Coverage: Added tests in test_entity_id_normalize.py to guarantee that legacy parameter translation works flawlessly and prevents breaking changes for existing deployments.
• Integration Baselines: Updated test_results.csv and test_results_summary.csv to align with the new test parameter signatures.

In the Technical Documentation, this is addressed in Section 7.

Asana Task

Deployment Readiness*

Testing

Describe or check:

• Created or updated unit, feature, and/or integration tests
• Typical manual testing in the local env browser, dev pipeline, etc.

Deployment Notes

Describe or check:

• No special deployment steps required

Rollback Plan

Describe or check:

• Standard revert is sufficient (git revert)

Reviewer Guidance / Questions*

Screenshots / Testing Evidence*

SOC 2 Change Management Checklist

• None of the below are true in this code
• New roles/permissions are introduced without review and approval by the product manager
• Hardcoded credentials, secrets, or API keys are present in this code
• Secrets are being managed outside of the approved secrets management process (e.g., GitHub Secrets, environment variables)
• PII or sensitive data handling is introduced or changed without being reviewed against our data classification policy
• Sensitive data is written to logs
• Input validation and sanitization is missing
• An unnecessary attack surface has been introduced (e.g., unused endpoints, open ports, debug modes left enabled)
• Common vulnerabilities have been introduced in the code (inc. any dependencies added or updated)
• No review for common vulnerabilities has been conducted
• Not tested in a non-production environment
• Breaking changes to existing APIs or integrations with downstream consumers being notified
• Performance impact has not been considered or acceptable
• Appropriate audit logging is missing for any security-relevant actions introduced by this change
• Log entries contain sensitive or PII data
• All existing tests do not pass locally (./vendor/bin/pest)

Provide justification if you are submitting a PR with any boxes checked other than the first.

---

Reminder for Reviewers: By approving this PR you are confirming that you have reviewed the code for correctness, security, and compliance with our engineering and SOC 2 standards. Do not approve PRs where SOC 2 checklist items are checked without documented justification.

*Optional

---

• To see the specific tasks where the Asana app for GitHub is being used, see below:
  • https://app.asana.com/0/0/1217286136124896