feat(AF-623): automated sensitive-data discovery & classification scans - #657
Merged
Merged
Conversation
A new discovery module samples column data through the existing engine sampling path (AF-443), detects sensitive values with local regex + checksum detectors (email, PAN+Luhn, SSN, IBAN, phone) and an optional fail-safe AI pass (redacted samples only), and proposes classification tags an admin confirms (AF-447 derivation applies masking) or dismisses (permanent suppression). Includes the scheduled DiscoveryScanJob, an on-demand scan endpoint, a Discovery tab with bulk confirm/dismiss on DatasourceSettingsPage, audit rows, docs, website, and e2e coverage. Closes #623
Contributor
Frontend Test Results1 303 tests 1 303 ✅ 3m 53s ⏱️ Results for commit 9f585bf. |
Contributor
Coverage Report for Frontend Coverage (frontend)
File Coverage
|
||||||||||||||||||||||||||||||||||||||
Contributor
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #623
What
A new
discoverySpring Modulith module that closes the AF-447 loop: instead of admins hand-tagging hundreds of columns, an opt-in scanner finds the sensitive columns and proposes the classification tags.DiscoveryScanServiceenumerates tables viaintrospectSchemaForSystem, reads a bounded raw sample per table through the existing AF-443QueryExecutor.sampleTablepath (works for JDBC + all 10 engine plugins), and runs local regex + checksum detectors: email→PII, credit-card PAN with Luhn→PCI, US SSN→PII, IBAN with mod-97→FINANCIAL, phone→PII. First-match-wins ordering, ≥5 samples and ≥30% match ratio required. Raw values live only on the scan stack — findings persist a redacted sample only.ai.api.DataDiscoveryAiServiceclassifies remaining columns via the org's boundai_config; the provider only ever sees column names, types, and format-preserving-redacted samples.PENDINGrows (discovery_finding, unique per column×classification×detector); rescans refresh PENDING rows and never resurrect CONFIRMED/DISMISSED ones. Confirming applies the tag throughDataClassificationAdminService(auto-deriving masking); dismissing suppresses permanently. Bulk decisions are per-row independent (attestation pattern) withTAG_CONFLICThandling.DiscoveryScanJob(@SchedulerLock discoveryScanJob, poll defaultPT15M) drains enableddiscovery_scan_configrows past their per-datasourcescan_interval_hours; plus an on-demandPOST /datasources/{id}/discovery/scan(202/409) so the feature is demoable and e2e-testable.DatasourceSettingsPage: settings card (enable, sample size 10–1000, interval 1–720 h, AI toggle, Scan now, last-scan status) + paginated findings worklist with sticky bulk Confirm/Dismiss bar and partial-success row feedback; tab badge shows the PENDING count. i18n across all 7 locales.DISCOVERY_SCAN_COMPLETED,DISCOVERY_FINDING_CONFIRMED,DISCOVERY_FINDING_DISMISSED(+ resourcediscovery_finding). No new NotificationEventType (issue asks for audit only).PERM_DATA_CLASSIFICATION_MANAGE, documented in the API spec first): GET/PUT/datasources/{id}/discovery/config, GET/datasources/{id}/discovery/findings, POST…/findings/bulk-decision, POST…/scan.Docs & website updated
docs/03-data-model.md(new tables + enums + audit actions),docs/04-api-spec.md(5 endpoints),docs/05-backend.md(module section + job table row),docs/06-frontend.md(Discovery tab),docs/09-deployment.md+CLAUDE.md(ACCESSFLOW_DISCOVERY_*env vars + module tree),docs/12-roadmap.md(v2.2 item)README.mdfeature list;website/index.html(feature blurb + roadmap item),website/docs/index.html(Automated discovery section + env vars),website/README.mdcontent-source mapTesting
mvn verify -Pcoveragegreen — 5 detector unit suites,DiscoveryScanServiceTest(caps/deadline/upsert/suppression/AI fail-safe), config/finding/trigger/job/audit-writer/web-model tests,DefaultDataDiscoveryAiServiceTest(strict-JSON lenient parse), andDiscoveryControllerIntegrationTest(Testcontainers; bulk confirm creates the real tag and the derived masking policy).ApplicationModulesTest+ApiPackageDependencyTestpass.test:coverage(91.8% stmts / 83.6% branches), build — all green. NewDiscoveryTab.test.tsx(7 tests incl. partial-success bulk path).e2e/tests/discovery.spec.tsruns the whole loop against the real stack (seed a table with emails via the query workflow → enable + Scan now → finding appears → bulk confirm → tag on Classification tab + policy on Masking tab). Ran locally along with the six adjacent datasource-settings specs — 19/19 passed.Known limitations (v1)