Skip to content

PII: spaCy does not detect uppercased names as sensible #31991

Description

@pmetras

Affected module

Governance (Glossary / Classification / Domains)

Describe the bug

A French autoclassification agent will tag correctly as PII.Sensible a column containing the French first-names Geneviève, François, Michael... but won't be able to do so when the column contains uppercased first-names like SERGE, THÉODORE, HÉLÈNE, JEAN-MARC...

To Reproduce

Create a table with a column containing capitalized names and check that an auto-classification agent detects the column as sensible PII.
Create another table with the same column name, but now containing upper-cased names and run the same auto-classification agent: this columns is not detected as sensible PII.

Expected behavior

Casing of text should not degrade PII detection. This a real problem with databases of legacy systems where upper-case names are frequent.

OS

Kubernetes

Python version

3.10.19

OpenMetadata version

1.12.14

OpenMetadata Ingestion package version

1.12.14.2

Additional context

No response

Pre-submission checklist

  • I searched for duplicate issues.
  • I removed credentials, hostnames, emails, and other sensitive data from logs and config.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions