Add ignore_accents option to by_name() - #103
Open
gefleury wants to merge 4 commits into
Open
Conversation
by_name() gains an ignore_accents parameter (default False). When True, accents and other diacritical marks are stripped before matching, The generated modules import unicodedata for this.
Open
apdavison
requested changes
Aug 18, 2026
| cls._instance_lookup[key] = [instance] | ||
|
|
||
| def remove_accents(s): | ||
| nfd_form = unicodedata.normalize("NFD", s) |
Member
There was a problem hiding this comment.
This doesn't handle Ł ø đ æ ß ə ı
For example:
>>> import unicodedata
>>> s = "Łódź"
>>> unicodedata.normalize("NFD", s)
'Łódź'
Author
There was a problem hiding this comment.
You're right, I missed it.
- "NFKD" (instead of "NFD") doesn't help either
- unidecode handles those letters
from unidecode import unidecode
for s in ["Ł", "ø", "đ", "æ", "ß", "ə", "ı", "µA", "m³", "Ελλάδα"]:
print(f"{s} -> {unidecode(s)}")
### Output
Ł -> L
ø -> o
đ -> d
æ -> ae
ß -> ss
ə -> @
ı -> i
µA -> uA
m³ -> m3
Ελλάδα -> Ellada
but it adds a runtime dependency and transliterates too aggressively (e.g. it maps ə to @, which is wrong for eg Azerbaijan's synonym "Azərbaycan").
- My preferred option: keep NFD for accents and add a small explicit map (no dependency, full control):
_SPECIAL = str.maketrans({
"Ł": "L", "ł": "l",
"Ø": "O", "ø": "o",
"Đ": "D", "đ": "d",
"Æ": "AE", "æ": "ae",
"ß": "ss",
"Ə": "E", "ə": "e",
"ı": "i",
})
def remove_accents(s):
s = s.translate(_SPECIAL)
nfd_form = unicodedata.normalize("NFD", s)
return "".join(c for c in nfd_fo
Happy to implement it this way if you agree
Member
There was a problem hiding this comment.
yes, I'm happy with that approach
Author
There was a problem hiding this comment.
OK. It's now implemented in 21e0397 with an updated test.
| } | ||
| extra_imports = set() | ||
| if has_instances: | ||
| extra_imports.add("import unicodedata") |
Member
There was a problem hiding this comment.
Since unicodedata is used in only one specific place in the code, it might be better to import unicodedata inside the remove_accents() function.
remove_accents() only stripped accents that Unicode can split off from their base letter (NFD normalization), so letters with no such decomposition (ß, œ, æ, ø, ł, đ, etc.) passed through unchanged. Add an explicit translation map, applied after the NFD strip so composed forms (e.g. "ǿ") are also handled correctly. Updated test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds accent-insensitive matching to
by_name(), a follow-up to thecase_sensitiveandmatchoptions from #100.ignore_accentsparameter (defaultFalse, so existing behaviour is unchanged).True, accents and other diacritical marks are stripped before matching (e.g."Republique francaise"matches"République française"). Combine withcase_sensitive=Falseto absorb case differences as well.like
m³→m3) is deliberately not applied here. That's a separate concern that couldbe handled later if needed.
import unicodedata(added to the preamble intranslator.pyonly for modules that have
by_name).test_regressions.py