Skip to content

Add education demo seed and EduProject metadata - #127

Open
Oluwajuwon-O wants to merge 7 commits into
datakind:mainfrom
Oluwajuwon-O:feature/education_demo_seed_data
Open

Add education demo seed and EduProject metadata#127
Oluwajuwon-O wants to merge 7 commits into
datakind:mainfrom
Oluwajuwon-O:feature/education_demo_seed_data

Conversation

@Oluwajuwon-O

@Oluwajuwon-O Oluwajuwon-O commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Description

This PR addresses the database bootstrap and demo data enhancements described in the technical documentation:

Lack of Domain-Agnostic Demonstration

Historically, DOT only included a single demo project (ScanProject1) which was tightly coupled to healthcare terminology and data structures. This made it difficult to demonstrate DOT's capabilities as a generic, domain-agnostic data quality framework.

Resolution

  • Education Schema Creation: Added db/dot/5-education_demo_data.sql to initialize a new education schema in the source database and populate it with synthetic tables (students, courses, enrollments, excel_grades).
  • EduProject Metadata: Added db/dot/6-eduproject_metadata.sql to register the new EduProject in the DOT metadata tables. This script configures a suite of data quality tests against the new education entities.
  • Validation of Multi-Tenant Architecture: This new project serves as a live demonstration of the multi-project schema isolation and domain-neutral duplicate detection features introduced in previous PRs. It proves DOT can simultaneously manage healthcare and education pipelines on the same database connection without collisions.
    Note: This PR depends on Isolate target schemas by source schema #125 and Incremental append loads with ID-based overlap deletion #126.

In the Technical Documentation, this supports the EduProject / education-schema examples in Sections 2.2 and 3.2 (bootstrap/demo data context in Section 9).

In the Technical Documentation, this is addressed in Sections 9.

Asana Task

Deployment Readiness*

Testing

Describe or check:

  • Created or updated unit, feature, and/or integration tests
  • Typical manual testing in the local env browser, dev pipeline, etc.

Deployment Notes

Describe or check:

  • No special deployment steps required

Rollback Plan

Describe or check:

  • Standard revert is sufficient (git revert)

Reviewer Guidance / Questions*

Screenshots / Testing Evidence*

SOC 2 Change Management Checklist

  • None of the below are true in this code
  • New roles/permissions are introduced without review and approval by the product manager
  • Hardcoded credentials, secrets, or API keys are present in this code
  • Secrets are being managed outside of the approved secrets management process (e.g., GitHub Secrets, environment variables)
  • PII or sensitive data handling is introduced or changed without being reviewed against our data classification policy
  • Sensitive data is written to logs
  • Input validation and sanitization is missing
  • An unnecessary attack surface has been introduced (e.g., unused endpoints, open ports, debug modes left enabled)
  • Common vulnerabilities have been introduced in the code (inc. any dependencies added or updated)
  • No review for common vulnerabilities has been conducted
  • Not tested in a non-production environment
  • Breaking changes to existing APIs or integrations with downstream consumers being notified
  • Performance impact has not been considered or acceptable
  • Appropriate audit logging is missing for any security-relevant actions introduced by this change
  • Log entries contain sensitive or PII data
  • All existing tests do not pass locally (./vendor/bin/pest)

Provide justification if you are submitting a PR with any boxes checked other than the first.


Reminder for Reviewers: By approving this PR you are confirming that you have reviewed the code for correctness, security, and compliance with our engineering and SOC 2 standards. Do not approve PRs where SOC 2 checklist items are checked without documented justification.

*Optional


@Oluwajuwon-O
Oluwajuwon-O marked this pull request as ready for review August 5, 2026 13:10
@Oluwajuwon-O
Oluwajuwon-O marked this pull request as draft August 5, 2026 13:10
information_schema.tables includes both object types; issuing only
DROP VIEW fails on base tables. Clear views first, then BASE TABLEs.
Pass source_schema through the Airflow sync path, namespace target and
test schemas as data_<conn>_<schema>, and add EduProject config pointing
at data_dot_data_education alongside ScanProject1's data_dot_data_public.
When date_field is set, save_object appends instead of replacing and
deletes overlapping target rows by id_field (chunked) before write.
@Oluwajuwon-O
Oluwajuwon-O force-pushed the feature/education_demo_seed_data branch from 7637754 to 9699d19 Compare August 8, 2026 07:45
@Oluwajuwon-O
Oluwajuwon-O marked this pull request as ready for review August 8, 2026 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant