Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
09ca5af
add model scaffold
zarmeen2 Mar 27, 2026
b297410
add implementation
zarmeen2 Mar 31, 2026
4ef2d1d
Integrate VQA-RAD dataset task with MedFlamingo scaffold
zDoda Mar 31, 2026
2778133
Merge remote-tracking branch 'origin/feat/model-scaffold' into feat/v…
zDoda Mar 31, 2026
aba7bde
Merge pull request #1 from zDoda/feat/model-scaffold
zDoda Mar 31, 2026
212ab6e
Merge branch 'sunlabuiuc:master' into master
zDoda Mar 31, 2026
ca223b9
add MedFlamingo to models.rst
zarmeen2 Apr 1, 2026
6d2bcfe
Merge branch 'sunlabuiuc:master' into master
zDoda Apr 6, 2026
5b3363b
Merge pull request #3 from zDoda/docs/add-medflamingo-to-models
zDoda Apr 6, 2026
f59f266
still failing a test, but got a prototype
zDoda Apr 6, 2026
61d3def
fix path error
zDoda Apr 6, 2026
5db61af
fixed dataset loader to match PR standards
zDoda Apr 6, 2026
b6f4083
Merge branch 'master' into feat/docs-example-repo-clean-up-and-testin…
zDoda Apr 6, 2026
57fde46
feat: complete MedFlamingo full pipeline (Dataset + Task + Model)
zarmeen2 Apr 7, 2026
2f7a379
lock file
zarmeen2 Apr 7, 2026
f954aae
Merge pull request #5 from zarmeen2/feat/medflamingo-full-pipeline
zDoda Apr 7, 2026
492b49a
Merge pull request #4 from zDoda/feat/docs-example-repo-clean-up-and-…
zarmeen2 Apr 7, 2026
ae39272
test fixes
zDoda Apr 8, 2026
c12bfb5
Merge branch 'master' into feat/docs-example-repo-clean-up-and-testin…
zDoda Apr 8, 2026
370b494
Merge pull request #6 from zDoda/feat/docs-example-repo-clean-up-and-…
zarmeen2 Apr 9, 2026
81c3ba0
Fix MedFlamingo device and embedding handling
zDoda Apr 13, 2026
e5baaf9
Merge branch 'master' into feat/vqarad-medflamingo-integration
zDoda Apr 13, 2026
df21714
Merge pull request #7 from zDoda/feat/vqarad-medflamingo-integration
zDoda Apr 13, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/api/datasets.rst
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,7 @@ Available Datasets
datasets/pyhealth.datasets.BMDHSDataset
datasets/pyhealth.datasets.COVID19CXRDataset
datasets/pyhealth.datasets.ChestXray14Dataset
datasets/pyhealth.datasets.VQARADDataset
datasets/pyhealth.datasets.TUABDataset
datasets/pyhealth.datasets.TUEVDataset
datasets/pyhealth.datasets.ClinVarDataset
Expand Down
11 changes: 11 additions & 0 deletions docs/api/datasets/pyhealth.datasets.VQARADDataset.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
pyhealth.datasets.VQARADDataset
===================================

The VQA-RAD dataset for medical visual question answering. The dataset loader
converts the public JSON annotations into a flat metadata CSV that PyHealth can
ingest, and its default task is :class:`~pyhealth.tasks.MedicalVQATask`.

.. autoclass:: pyhealth.datasets.VQARADDataset
:members:
:undoc-members:
:show-inheritance:
4 changes: 4 additions & 0 deletions docs/api/models.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@ routes each feature type automatically.
* - :doc:`models/pyhealth.models.GraphCare`
- You want to augment EHR codes with a medical knowledge graph
- Combines code sequences with a :class:`~pyhealth.graph.KnowledgeGraph`
* - :doc:`models/pyhealth.models.MedFlamingo`
- You are solving multimodal medical tasks with images plus text prompts (for example, VQA-style radiology QA)
- Flamingo-style architecture with a frozen vision encoder + frozen language model connected by gated cross-attention layers

How BaseModel Works
--------------------
Expand Down Expand Up @@ -194,6 +197,7 @@ API Reference
models/pyhealth.models.ConCare
models/pyhealth.models.Agent
models/pyhealth.models.GRASP
models/pyhealth.models.MedFlamingo
models/pyhealth.models.MedLink
models/pyhealth.models.TCN
models/pyhealth.models.TFMTokenizer
Expand Down
24 changes: 24 additions & 0 deletions docs/api/models/pyhealth.models.MedFlamingo.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
pyhealth.models.MedFlamingo
===================================

MedFlamingo: multimodal medical few-shot learner.

The separate callable MedFlamingoLayer (gated cross-attention dense block)
and the complete MedFlamingo model.

**Paper:** Moor et al. "Med-Flamingo: a Multimodal Medical Few-shot Learner" ML4H 2023.

.. note::

This is a stub implementation. The class structure and signatures are
in place, but ``forward()`` and ``generate()`` raise ``NotImplementedError``.

.. autoclass:: pyhealth.models.MedFlamingoLayer
:members:
:undoc-members:
:show-inheritance:

.. autoclass:: pyhealth.models.MedFlamingo
:members:
:undoc-members:
:show-inheritance:
1 change: 1 addition & 0 deletions docs/api/tasks.rst
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,7 @@ Available Tasks
DKA Prediction (MIMIC-IV) <tasks/pyhealth.tasks.dka>
Drug Recommendation <tasks/pyhealth.tasks.drug_recommendation>
Length of Stay Prediction <tasks/pyhealth.tasks.length_of_stay_prediction>
Medical VQA <tasks/pyhealth.tasks.MedicalVQATask>
Medical Transcriptions Classification <tasks/pyhealth.tasks.MedicalTranscriptionsClassification>
Mortality Prediction (Next Visit) <tasks/pyhealth.tasks.mortality_prediction>
Mortality Prediction (StageNet MIMIC-IV) <tasks/pyhealth.tasks.mortality_prediction_stagenet_mimic4>
Expand Down
12 changes: 12 additions & 0 deletions docs/api/tasks/pyhealth.tasks.MedicalVQATask.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
pyhealth.tasks.MedicalVQATask
===================================

Medical visual question answering task for paired radiology images and
questions. This task treats VQA-RAD answers as a multiclass prediction target
so the resulting ``SampleDataset`` can be trained with the standard PyHealth
trainer loop.

.. autoclass:: pyhealth.tasks.MedicalVQATask
:members:
:undoc-members:
:show-inheritance:
Loading