Skip to content

[DRAFT][STACKED on #206] Add Dep marker for explicit container dependencies in @Flow.model#239

Draft
timkpaine wants to merge 5 commits into
nk/auto_deps_auto_callable_modelfrom
nk/explicit-deps-containers
Draft

[DRAFT][STACKED on #206] Add Dep marker for explicit container dependencies in @Flow.model#239
timkpaine wants to merge 5 commits into
nk/auto_deps_auto_callable_modelfrom
nk/explicit-deps-containers

Conversation

@timkpaine

Copy link
Copy Markdown
Member

PR Summary: nk/explicit-deps-containers

Stacked PR Note

This builds on the @Flow.model PR: #206.
That PR should be reviewed/merged first. This PR has to target main
because GitHub cannot target the fork-only parent branch, please review this as
the follow-on Dep[...] container-dependency layer rather than re-reviewing
the base @Flow.model work.

TL;DR

This lets a flow consume a dynamic collection of upstream model outputs without
hard-coding every branch into the model signature. A fan-out step can produce
many sibling models, and a fan-in step can accept them as list[Dep[T]],
tuple[...], or dict[..., Dep[T]] while still receiving plain resolved
values inside the function. The marker keeps that power explicit: normal
containers stay literal unless the exact nested dependency slot is marked.

Tiny example:

@Flow.model
def load_one(shard: str, date: FromContext[str]) -> int:
    ...


@Flow.model
def combine(rows: list[Dep[int]]) -> int:
    return sum(rows)


def build_pipeline(shards: list[str]):
    # Fan-out: build one upstream model per shard.
    branch_models = [load_one(shard=shard) for shard in shards]

    # Fan-in: pass the branch models as one regular container input.
    return combine(rows=branch_models)

Without Dep, combine(rows=branch_models) would be rejected because regular
containers stay literal by default. With list[Dep[int]], each list item may be
either a literal int or a model producing an int, and combine() still sees
plain list[int] at execution time.

What Changed

This PR adds Dep[T], a narrow @Flow.model annotation marker for explicit
dependency leaves inside regular container inputs.

Regular parameters already accepted either a literal value or a direct
CallableModel supplying the whole parameter. That behavior is unchanged.
Dep[T] only adds marked nested slots inside literal containers:

from ccflow import Dep, Flow, FromContext


@Flow.model
def source(value: FromContext[int], offset: int) -> int:
    return value + offset


@Flow.model
def total(values: list[Dep[int]]) -> int:
    return sum(values)


model = total(values=[source(offset=1), 2, 3])
assert model.flow.compute(value=10).value == 16

The function body receives list[int]; generated @Flow.model code resolves
the marked model leaves before calling the function.

Semantics

  • list[int] still accepts a literal list[int] or a direct model returning
    list[int].
  • list[Dep[int]] additionally accepts model leaves inside the literal list.
  • Dep[T] means "literal T or CallableModel whose unwrapped result
    validates as T" at that exact annotation position.
  • Dep[...] is interpreted only by generated @Flow.model regular parameters.
  • Dep[...] is supported inside list, tuple, and dict values.
  • Top-level Dep[...] is rejected because direct whole-parameter dependencies
    already cover that case.
  • Union annotations may appear inside the marked slot, such as
    list[Dep[int | None]].
  • Union annotations may not wrap a Dep marker, including optional container
    forms like list[Dep[int]] | None.
  • Dict keys cannot be Dep[...].
  • Nested Dep[...] markers are rejected.
  • Dep[...] does not add automatic behavior to handwritten CallableModel
    fields; there it behaves like ordinary Annotated metadata.

Why This Shape

This keeps automatic dependency discovery simple and explicit. We do not scan
every container for models. Instead, users mark the exact nested positions where
model leaves are allowed. That preserves existing literal-container behavior
while supporting generic fan-in patterns such as lists of branch results.

Implementation Notes

  • Adds Dep as an Annotated[T, _DepMarker()] marker.
  • Extends generated model validation, execution, __deps__, and effective
    identity to walk only marked container positions.
  • Keeps registry alias fallback and serialized-model fallback available at
    marked dependency leaves.
  • Preserves other Annotated metadata when removing the Dep marker, so
    constraints such as Field(gt=0) still apply to literals and dependency
    results.
  • Updates Flow model examples to show explicit Dep container inputs.

Tests

Covered by ccflow/tests/test_flow_model.py:

  • marked model leaves inside list[Dep[int]] and list[Dep[list[int]]];
  • existing whole-parameter dependency behavior;
  • registry aliases and serialized model references at marked leaves;
  • pickle/cloudpickle and JSON/BaseModel roundtrips for models with marked
    dependency leaves;
  • graph dependency discovery and effective cache identity;
  • preservation of non-Dep Annotated metadata;
  • rejection of unmarked nested models, top-level Dep, nested Dep, dict-key
    Dep, and set usage;
  • handwritten CallableModel fields ignoring Dep as normal annotation
    metadata.

NeejWeej added 4 commits June 9, 2026 19:11
… @Flow.model

Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
Signed-off-by: Nijat K <nijat.khanbabayev@gmail.com>
@github-actions

github-actions Bot commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Test Results

    1 files  ± 0      1 suites  ±0   3m 8s ⏱️ +12s
1 203 tests +28  1 201 ✅ +28  2 💤 ±0  0 ❌ ±0 
1 209 runs  +28  1 207 ✅ +28  2 💤 ±0  0 ❌ ±0 

Results for commit 2db2cd9. ± Comparison against base commit 88c5667.

♻️ This comment has been updated with latest results.

@codecov

codecov Bot commented Jun 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.69404% with 64 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.01%. Comparing base (88c5667) to head (2db2cd9).

Files with missing lines Patch % Lines
ccflow/tests/test_flow_model.py 93.00% 24 Missing ⚠️
ccflow/examples/flow_model/flow_model_example.py 0.00% 13 Missing ⚠️
ccflow/flow_model.py 92.44% 7 Missing and 6 partials ⚠️
...amples/flow_model/flow_model_hydra_builder_demo.py 0.00% 11 Missing ⚠️
ccflow/_flow_model_binding.py 95.94% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@                         Coverage Diff                          @@
##           nk/auto_deps_auto_callable_model     #239      +/-   ##
====================================================================
- Coverage                             93.04%   93.01%   -0.04%     
====================================================================
  Files                                   162      163       +1     
  Lines                                 17925    18528     +603     
  Branches                               1162     1228      +66     
====================================================================
+ Hits                                  16679    17233     +554     
- Misses                                 1020     1065      +45     
- Partials                                226      230       +4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

`_pop_dep_marker` rebuilt the non-Dep Annotated metadata via
`Annotated.__class_getitem__((base, *metadata))`. Python 3.14 removed that
attribute (typing.Annotated.__getattr__ now raises AttributeError for
`__class_getitem__`), which broke every 3.14 CI build with
`AttributeError: __class_getitem__` in test_dep_marker_*.

Build the tuple first and subscript: `Annotated[(base, *metadata)]`. This is
equivalent, avoids the removed dunder, and stays portable (the original
3.11+-only concern was the star-in-subscript spelling, not a pre-built tuple).

Signed-off-by: Tim Paine <3105306+timkpaine@users.noreply.github.com>

@timkpaine timkpaine left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to self: prefer Depends

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants