[datasets] fix zero-dimension synthetic image for blank glyphs (#2016) - #2123
[datasets] fix zero-dimension synthetic image for blank glyphs (#2016)#2123ousamabenyounes wants to merge 1 commit into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2123 +/- ##
==========================================
- Coverage 97.00% 96.99% -0.01%
==========================================
Files 169 169
Lines 9611 9613 +2
==========================================
+ Hits 9323 9324 +1
- Misses 288 289 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
felixdittrich92
left a comment
felixdittrich92
left a comment
There was a problem hiding this comment.
Hi @ousamabenyounes 👋,
Thanks for the PR! However, this is a false positive.
- Whitespace characters should not be included in the vocabulary
- Instead of silently overriding the character when there is no font that can render it, we should explicitly raise an error. In this case, it’s a user error that should be fixed by providing font(s) capable of rendering all characters
Maybe a pre-check would be useful to ensure that all characters in the vocabulary can be rendered correctly by the provided font(s)?
|
Thanks for the review — agreed on the clamp, I've dropped it. Turning a degenerate canvas into a 1px strip trades a crash for silent label noise, which is worse. Digging into the mechanism though, I don't think the "user error" framing covers the whole case.
from PIL import ImageFont
f = ImageFont.truetype("LiberationSerif-Regular.ttf", 32)
f.getbbox("฿") # (0, 29, 25, 29) -> 25px advance, 0px height
f.getmask("฿").getbbox() # None -> nothing drawn
from doctr.datasets.generator.base import synthesize_text_img
synthesize_text_img("฿฿", font_family="LiberationSerif-Regular.ttf").size
# (55, 0) -> the zero-dimension image behind #20166 of the 12 Liberation faces installed here are affected — all four So This is probably not the reporter's exact trigger: their On the mechanism itself: it isn't about the font lacking the glyph. Most families draw a tofu box for What I pushedI've implemented your pre-check suggestion in a separate commit (
Two decisions I'd like from you1. Whitespace. I exempted it from the pre-check, because 2. Existing setups. This starts raising for configurations that currently "work": One detail that shaped the placement: only ValidationRED on unmodified GREEN with the commit, and no baseline regression: Happy to reshape any of it. |
felixdittrich92
left a comment
felixdittrich92
left a comment
There was a problem hiding this comment.
Hi @ousamabenyounes 👋,
First the latex VOCABS entry can be ignored atm it's not used and more like an experimential addition for further work (formula recognition)
About the ฿ char - it should work out of the box with a fallback to system fonts besides the given ones.
I would suggest the following:
fonts.py:
# Copyright (C) 2021-2026, Mindee.
# This program is licensed under the Apache License 2.0.
# See LICENSE or go to <https://opensource.org/licenses/Apache-2.0> for full license details.
import logging
import platform
from functools import lru_cache
from PIL import ImageFont
__all__ = ["get_font", "get_font_candidates"]
_FONT_CANDIDATES: dict[str, tuple[str, ...]] = {
"Linux": (
"DejaVuSans.ttf",
"NotoSans-Regular.ttf",
"LiberationSans-Regular.ttf",
"FreeSans.ttf",
"FreeMono.ttf", # legacy default
),
"Darwin": (
"Arial Unicode.ttf",
"Helvetica.ttc",
"Arial.ttf", # legacy default
),
"Windows": (
"arial.ttf", # legacy default
"segoeui.ttf",
"tahoma.ttf",
),
}
def get_font_candidates() -> tuple[str, ...]:
"""Lists the recommended fonts which are installed on this system
Returns:
the available font families, by order of preference
"""
candidates = _FONT_CANDIDATES.get(platform.system(), _FONT_CANDIDATES["Linux"])
return tuple(family for family in candidates if _is_available(family))
def _is_available(font_family: str) -> bool:
try:
_ = ImageFont.truetype(font_family, 10)
except OSError:
return False
return True
@lru_cache(maxsize=1)
def _resolve_default_font_family() -> str | None:
"""Find the first available candidate font for this platform."""
candidates = get_font_candidates()
return candidates[0] if candidates else None
def get_font(font_family: str | None = None, font_size: int = 13) -> ImageFont.FreeTypeFont | ImageFont.ImageFont:
"""Resolves a compatible ImageFont for the system
Args:
font_family: the font family (or path to a font file) to use. If None,
the best available system font is picked automatically.
font_size: the size of the font upon rendering
Returns:
the Pillow font
"""
if font_family is not None:
return ImageFont.truetype(font_family, font_size)
default_family = _resolve_default_font_family()
if default_family is not None:
return ImageFont.truetype(default_family, font_size)
# Last resort: Pillow's built-in font.
try:
return ImageFont.load_default(size=font_size)
except TypeError: # pragma: no cover
logging.warning(
"Unable to load any recommended font family. Loading default PIL font, "
"font size issues may be expected. "
"To prevent this, it is recommended to specify the value of 'font_family'."
)
return ImageFont.load_default()base.py:
# Copyright (C) 2021-2026, Mindee.
# This program is licensed under the Apache License 2.0.
# See LICENSE or go to <https://opensource.org/licenses/Apache-2.0> for full license details.
import random
from collections.abc import Callable, Sequence
from functools import lru_cache
from typing import Any
from PIL import Image, ImageDraw, ImageFont
from doctr.io.image import tensor_from_pil
from doctr.utils import Sample
from doctr.utils.fonts import get_font, get_font_candidates
from ..datasets import AbstractDataset
@lru_cache(maxsize=32)
def _load_font(font_family: str | None, font_size: int) -> ImageFont.FreeTypeFont | ImageFont.ImageFont:
return get_font(font_family, font_size)
def _glyph_ink(font: ImageFont.FreeTypeFont | ImageFont.ImageFont, char: str) -> tuple[tuple[int, int], bytes] | None:
"""Rasterized glyph of a character, None if it is drawn without ink."""
mask = font.getmask(char, mode="L")
return (mask.size, bytes(mask)) if mask.getbbox() is not None else None
@lru_cache(maxsize=32)
def _notdef_ink(font_family: str | None, font_size: int) -> frozenset[tuple[tuple[int, int], bytes]]:
"""Glyph a font falls back to for characters it does not cover (the ".notdef" box)."""
font = _load_font(font_family, font_size)
# Unicode non-characters are never mapped, so they always yield ".notdef"
return frozenset(ink for char in ("\ufffe", "\U0010ffff") if (ink := _glyph_ink(font, char)) is not None)
@lru_cache(maxsize=8192)
def _renders_char(char: str, font_family: str | None, font_size: int) -> bool:
"""Whether a font draws the actual glyph of a character, rather than nothing or a box."""
ink = _glyph_ink(_load_font(font_family, font_size), char)
return ink is not None and ink not in _notdef_ink(font_family, font_size)
def _resolve_fonts(font_family: str | list[str] | None) -> list[str | None]:
font_families: list[str | None] = [*font_family] if isinstance(font_family, list) else [font_family]
for font in font_families:
try:
_ = _load_font(font, 32)
except OSError:
raise ValueError(f"unable to locate font: {font}")
return font_families
def _fonts_per_char(
vocab: str, font_families: Sequence[str | None], font_size: int = 32
) -> dict[str, tuple[str | None, ...]]:
"""Maps each character of the vocab to the fonts able to render it.
Characters none of the given fonts can render fall back to the system fonts.
Args:
vocab: the characters to render
font_families: the font families to pick from
font_size: the size the characters are rendered at
Returns:
the fonts able to render each character
Raises:
ValueError: if a character is rendered by none of the fonts
"""
mapping = {
char: tuple(font for font in font_families if _renders_char(char, font, font_size))
for char in dict.fromkeys(vocab)
}
missing = [char for char, fonts in mapping.items() if not fonts]
if missing:
fallbacks = [font for font in get_font_candidates() if font not in font_families]
mapping.update(
{char: tuple(font for font in fallbacks if _renders_char(char, font, font_size)) for char in missing}
)
unrenderable = [char for char in missing if not mapping[char]]
if unrenderable:
raise ValueError(
f"the following characters cannot be rendered, neither by the given fonts {list(font_families)} "
f"nor by the system fonts {fallbacks}: "
f"{', '.join(f'{char!r} (U+{ord(char):04X})' for char in unrenderable)}. "
"They are drawn blank or as a '.notdef' box, please provide font(s) covering the whole vocab."
)
return mapping
def synthesize_text_img(
text: str,
font_size: int = 32,
font_family: str | None = None,
background_color: tuple[int, int, int] | None = None,
text_color: tuple[int, int, int] | None = None,
) -> Image.Image:
"""Generate a synthetic text image
Args:
text: the text to render as an image
font_size: the size of the font
font_family: the font family (has to be installed on your system)
background_color: background color of the final image
text_color: text color on the final image
Returns:
PIL image of the text
Raises:
ValueError: if the text is empty or rendered without ink
"""
if not text:
raise ValueError("unable to synthesize an image from an empty text")
background_color = (0, 0, 0) if background_color is None else background_color
text_color = (255, 255, 255) if text_color is None else text_color
font = _load_font(font_family, font_size)
left, top, right, bottom = font.getbbox(text)
text_w, text_h = right - left, bottom - top
if text_w <= 0 or text_h <= 0:
raise ValueError(f"font {font_family!r} draws no ink for {text!r}, resulting in a zero-dimension image")
h, w = int(round(1.3 * text_h)), int(round(1.1 * text_w))
# If single letter, make the image square, otherwise expand to meet the text size
img_size = (h, w) if len(text) > 1 else (max(h, w), max(h, w))
img = Image.new("RGB", img_size[::-1], color=background_color)
d = ImageDraw.Draw(img)
# Offset so that the text is centered
text_pos = (int(round((img_size[1] - text_w) / 2)), int(round((img_size[0] - text_h) / 2)))
# Draw the text
d.text(text_pos, text, font=font, fill=text_color)
return img
class _CharacterGenerator(AbstractDataset):
def __init__(
self,
vocab: str,
num_samples: int,
cache_samples: bool = False,
font_family: str | list[str] | None = None,
img_transforms: Callable[[Any], Any] | None = None,
sample_transforms: Callable[[Sample], Sample] | None = None,
) -> None:
self.vocab = vocab
self._num_samples = num_samples
self.font_family = _resolve_fonts(font_family)
self._fonts_per_char = _fonts_per_char(self.vocab, self.font_family)
self.img_transforms = img_transforms
self.sample_transforms = sample_transforms
self._data: list[Image.Image] = []
if cache_samples:
self._data = [
(synthesize_text_img(char, font_family=font), idx) # type: ignore[misc]
for idx, char in enumerate(self.vocab)
for font in self._fonts_per_char[char]
]
def __len__(self) -> int:
return self._num_samples
def _read_sample(self, index: int) -> tuple[Any, int]:
# Samples are already cached
if len(self._data) > 0:
idx = index % len(self._data)
pil_img, target = self._data[idx] # type: ignore[misc]
else:
target = index % len(self.vocab)
char = self.vocab[target]
pil_img = synthesize_text_img(char, font_family=random.choice(self._fonts_per_char[char]))
img = tensor_from_pil(pil_img)
return img, target
class _WordGenerator(AbstractDataset):
def __init__(
self,
vocab: str,
min_chars: int,
max_chars: int,
num_samples: int,
cache_samples: bool = False,
font_family: str | list[str] | None = None,
img_transforms: Callable[[Any], Any] | None = None,
sample_transforms: Callable[[Sample], Sample] | None = None,
) -> None:
self.vocab = vocab
self.wordlen_range = (min_chars, max_chars)
self._num_samples = num_samples
self.font_family = _resolve_fonts(font_family)
fonts_per_char = _fonts_per_char(self.vocab, self.font_family)
# A word is drawn with a single font, so the font is picked first and the word is
# built from the characters this font can render, weighted by its coverage
self._vocab_per_font = {
font: "".join(char for char in self.vocab if font in fonts_per_char[char])
for font in dict.fromkeys(font for fonts in fonts_per_char.values() for font in fonts)
}
self._fonts = list(self._vocab_per_font)
self._font_weights = [len(vocab_) for vocab_ in self._vocab_per_font.values()]
self.img_transforms = img_transforms
self.sample_transforms = sample_transforms
self._data: list[Image.Image] = []
if cache_samples:
self._data = [self._synthesize_sample() for _ in range(num_samples)] # type: ignore[misc]
def _generate_string(self, min_chars: int, max_chars: int, vocab: str | None = None) -> str:
num_chars = random.randint(min_chars, max_chars)
return "".join(random.choice(self.vocab if vocab is None else vocab) for _ in range(num_chars))
def _synthesize_sample(self) -> tuple[Image.Image, str]:
font = random.choices(self._fonts, weights=self._font_weights)[0]
text = self._generate_string(*self.wordlen_range, vocab=self._vocab_per_font[font])
return synthesize_text_img(text, font_family=font), text
def __len__(self) -> int:
return self._num_samples
def _read_sample(self, index: int) -> tuple[Any, str]:
# Samples are already cached
if len(self._data) > 0:
pil_img, target = self._data[index] # type: ignore[misc]
else:
pil_img, target = self._synthesize_sample()
img = tensor_from_pil(pil_img)
return img, targetIt adds the prev check that all chars can be rendered additional it caches the fonts instead of reloading at each char
a0c6e41 to
d287360
Compare
mindee#2016) A font may map a codepoint to an empty outline: the glyph carries a horizontal advance but no ink, so a word made only of such characters yields a zero-dimension image and an opaque resize failure inside a DataLoader worker. Rather than refusing the vocabulary, characters the given fonts cannot draw now fall back to the system fonts, and only a character no font at all can render raises. Detection compares each glyph to the font's own ".notdef" box, obtained by rendering Unicode non-characters, so a tofu box counts as unrenderable too. Whitespace is exempt: it is inkless in every font by design, and judging it by ink would reject any vocabulary containing a space. Words are drawn with a single font, so the font is picked first and the word is built from the characters that font covers, weighted by its coverage. Fonts and per-character results are cached instead of being reloaded for every glyph. synthesize_text_img now rejects empty text and text drawn without ink.
d287360 to
f47c4f5
Compare
|
Applied your version, rebased onto current One thing to flag before this is reviewable: as written, the check rejects any vocabulary containing a space. _fonts_per_char("hello world", [None])
# ValueError: the following characters cannot be rendered ... ' ' (U+0020)A space is inkless in every font by design, so no font can be said to "cover" it. I exempted whitespace — that also un-breaks The fallback itself does the job: with I kept the capability tests independent of installed fonts, though. Asserting that Liberation Serif draws
|
felixdittrich92
left a comment
felixdittrich92
left a comment
There was a problem hiding this comment.
Two small things left 👍
There was a problem hiding this comment.
Let's remove all the comments under the functions here - the test names are clear enough
| # produced blank glyphs and 0-dimension images (#2016). | ||
| from doctr.datasets.generator import base | ||
|
|
||
| monkeypatch.setattr(base, "get_font_candidates", lambda: ()) |
There was a problem hiding this comment.
this code can be moved in the existing test_wordgenerator function with a small comment and the import should be moved on top of the file where the other imports are
def test_wordgenerator(monkeypatch):
... existing code
# each word must be drawn with a single font that can render all its chars (#2016)
monkeypatch.setattr(base, "get_font_candidates", lambda: ())
monkeypatch.setattr(base, "_load_font", lambda family, size: None)
# "b" is drawn by the second font only, "a" by both
monkeypatch.setattr(base, "_renders_char", lambda char, font, size: char == "a" or font == "second.ttf")
monkeypatch.setattr(base, "synthesize_text_img", lambda text, font_family=None: Image.new("RGB", (8, 8)))
ds = datasets.WordGenerator(
vocab="ab",
min_chars=4,
max_chars=4,
num_samples=16,
cache_samples=True,
font_family=["first.ttf", "second.ttf"],
)
assert ds._vocab_per_font == {"first.ttf": "a", "second.ttf": "ab"}
for _, target in ds._data:
assert len(target) == 42 participants
Summary
Closes #2016
Recognition training with the synthetic
WordGeneratorintermittently dies in a DataLoader worker with:Root cause.
synthesize_text_imgsizes the canvas fromfont.getbbox(text). A font may map a codepoint to an empty outline: the glyph carries a horizontal advance but draws no ink, sogetbboxreturns a zero-height box,hbecomes0, and the resulting 0-dimension image is rejected byF.resize.This is reachable from shipped components alone.
_BASE_VOCABS["currency"] = "£€¥¢฿"flows intoVOCABS["english"]and from there into 73 of the 215 shipped vocabs, includingpolish— and several Liberation faces draw฿(U+0E3F) blank:Fix. Characters the given fonts cannot draw fall back to the system fonts; only a character no font at all can render raises. Detection compares each glyph to the font's own
.notdefbox, obtained by rendering Unicode non-characters, so a tofu box counts as unrenderable too — a font merely lacking the codepoint is caught just as well as one mapping it to an empty outline.Words are drawn with a single font, so the font is picked first and the word is built from the characters that font covers, weighted by its coverage. Fonts and per-character results are cached rather than reloaded per glyph.
synthesize_text_imgnow rejects empty text and text drawn without ink.Validation
RED on unmodified
main:GREEN with this commit, and no baseline regression:
Files changed
doctr/utils/fonts.pyget_font_candidates()— the recommended families actually installeddoctr/datasets/generator/base.py.notdefdetection, caching, per-font word synthesistests/common/test_utils_fonts.pytests/common/test_datasets.py.notdefrejection, unrenderable error, empty/inkless texttests/pytorch/test_datasets_pt.py