Add SAM 3 segmentation converter - #838
Open
Irozuku wants to merge 4 commits into
Open
Conversation
Detects every instance of a text prompt in an image column and appends segment_1..k plus seg_score_1..k, where k is max_masks, leaving the row count unchanged. Instances are ranked by descending confidence and an image with fewer detections leaves the remaining ranks null. SAM 3 is a gated HuggingFace model, so the converter declares REQUIRED_CREDENTIALS and inherits HFPretrainedDownloadMixin for its weights, following the Stable Diffusion 3 pattern. Sam3Model needs transformers 5, hence the dependency floor. BaseSegmenter is held by composition rather than registered, because the component registry rejects a class with two Base ancestors declaring a TYPE. Tests cover the abstraction and the mask rendering; the converter and the segmenter themselves are left untested here since they pull a 3.4 GB gated model.
The segmenter already accepted a device and auto-picked CUDA when available, but the converter never exposed the choice, so a user could not force CPU or select a GPU. It now carries the same DEVICE_ENUM field the image embedding converter uses, listing the machine's real GPUs, and resolves the label to a torch device string before handing it to the segmenter.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a promptable segmentation converter backed by SAM 3. Given a text prompt such as "cow", it detects every matching instance in an image column and appends
segment_1..kandseg_score_1..k, where k ismax_masks, leaving the row count unchanged. Instances are ranked by descending confidence, and an image with fewer detections leaves the remaining ranks null.Type of Change
Changes (by file)
DashAI/back/segmenters/base_segmenter.py:SegmentInstance, the mask, score and bbox contract, plus the segmenter interface. Held by composition rather than registered, becauseComponentRegistry._get_base_typerejects a class with twoBaseancestors declaring aTYPE.DashAI/back/segmenters/rendering.py: renders one detected object as a masked image, withcrop_to_bboxand a black, white or blurred background.DashAI/back/segmenters/sam3_segmenter.py: the SAM 3 adapter overSam3ModelandSam3Processor. Lazily loaded, so constructing the converter never pulls the weights into memory.DashAI/back/converters/segmentation/sam3_segment_converter.py: the converter. InheritsHFPretrainedDownloadMixinfor its weights and declaresREQUIRED_CREDENTIALS = ["HuggingFaceCredential"], following the Stable Diffusion 3 pattern, sincefacebook/sam3is gated.DashAI/back/initial_components.py: registers the converter.pyproject.toml: floorstransformersat 5.0.0, which is whereSam3Modellanded.tests/back/segmenters/test_rendering.py,tests/back/segmenters/test_base_segmenter.py: cover the mask rendering and the data contract.Testing
image_columnto its index within that scope.devicedefaults to the first GPU when CUDA is present. SAM 3 is a 0.9B model, so CPU is the slow path.max_masksabove 1 on images with a varying number of objects, which is what produces null ranks.Notes
The converter and the segmenter themselves are not unit tested here, since exercising them pulls a 3.4 GB gated model. Only the rendering and the data contract are covered.