Enhance image type handling and add embedding converter - #837
Merged
Conversation
Converters and explorers advertised allowed_types with the Python class name, so DashAIImage reached the frontend as DashAIImage while columns report their type as Image. Nothing matched and an image dataset was rejected naming a type no column can have. Types now carry their own DISPLAY_NAME and expose display_name(), which to_string() also reads, so one constant per class drives both directions and a new type with the same mismatch needs no change outside itself.
Turns a DashAIImage column into plain Float columns named
emb_{column}_{i} using a HuggingFace vision encoder, so image data can
be consumed by the existing tabular tasks and models with no new task
and no new model.
Unlike the text embedding converter it is modelled on, the source column
is retained by default through keep_source_column, so a later stage can
still reach the original pixels.
The round trip test documents that dashai_types metadata is lost through
a pandas round trip, which is why a converter rebuilding a dataset must
pass types explicitly.
… enum model_name was free text, which let a user name any repository including weights whose licence is not compatible with this project. It is now an enum over IMAGE_ENCODERS, three permissively licensed vision encoders, with the friendly name resolved to a repository id at construction. An unrecognised value still passes through so an API payload naming a repository directly keeps working. device now reuses DEVICE_ENUM, DEVICE_PLACEHOLDER and DEVICE_TO_IDX from the models package, so the form lists the machine's actual GPUs instead of a hardcoded cpu and cuda pair, and the label is mapped to a torch device string.
ResNet returns a spatially pooled pooler_output of shape (batch, channels, 1, 1) while a vision transformer such as DINOv2 returns (batch, hidden). The column builder indexed the second axis and handed pyarrow a nested list, failing with 'Could not convert [[0.057...]] with type list: tried to convert to float32'. Collapse every dimension after the batch axis, and reject a non two dimensional result with a message naming the column, so an injected encoder of the wrong shape does not surface as an opaque pyarrow error.
The switch to display_name() assumed every entry in inputs_types, outputs_types and allowed_types is a DashAIDataType. They are not: the component API tests register tasks whose types are HuggingFace dataset features, and a plugin can legitimately declare its own classes. The previous t.__name__ worked on any class, so this raised AttributeError for ClassLabel and errored 34 tests in CI. Ask the type for its own name when it can answer and fall back to the class name otherwise.
cristian-tamblay
force-pushed
the
feat/image-embedding-converter
branch
from
August 25, 2026 14:06
114deb1 to
37e35b3
Compare
cristian-tamblay
approved these changes
Aug 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an image embedding converter that turns a
DashAIImagecolumn into plainFloatcolumns using a pretrained vision encoder. This lets image data reach the existing tabular tasks and models with no new task and no new model.It also fixes a naming mismatch that made any image typed component unusable: converters and explorers advertised their accepted types with the Python class name, so
DashAIImagereached the frontend asDashAIImagewhile dataset columns report their type asImage. Nothing ever matched, and selecting an image dataset failed with "This dataset does not have any columns with the required types (DashAIImage)". No existing converter hit this because every other one usesText,FloatorCategorical, where the class name and the display name happen to be identical.Type of Change
Changes (by file)
DashAI/back/types/dashai_data_type.py: addsDISPLAY_NAMEanddisplay_name(), so a type whose class name differs from the name users see declares it itself instead of a caller translating it.DashAI/back/types/dashai_image.py: declaresDISPLAY_NAME = "Image", andto_string()now reads it, so one constant per class drives both directions.DashAI/back/converters/base_converter.py: serialisesallowed_typesthroughdisplay_name()instead of__name__.DashAI/back/exploration/base_explorer.py: same fix, same defect.DashAI/back/tasks/base_task.py: drops its private_CLASS_NAME_TO_DISPLAYdict in favour of the type's owndisplay_name().DashAI/back/converters/hugging_face/image_embedding.py: the new converter.model_nameis an enum over three permissively licensed encoders,devicereusesDEVICE_ENUMfrom the models package so the form lists real GPUs, andkeep_source_columndefaults to true so a later stage can still reach the original pixels.DashAI/back/initial_components.py: registers the converter.tests/back/converters/test_image_column_roundtrip.py: documents thatdashai_typesmetadata is lost through a pandas round trip, which is why a converter that rebuilds a dataset must passtypes=explicitly.Testing
DINOv2 smalland verify it returns the embedding columns.Notes
Encoder choices are
DINOv2 base,DINOv2 smallandResNet 50, all Apache 2.0, which is compatible with this project's MIT licence.