Replace runtime-type quotation with splice-into-source typechecking by datvo06 · Pull Request #706 · BasisResearch/effectful

datvo06 · 2026-06-30T13:31:54Z

Closes #696. Addresses #577, #576.

Currently, for synthesis, we are reconstructing a Template's lexical context as synthetic type stubs. This is brittle and likely unsound.

This PR splices the generated code into the Template module, and type checks the whole file. This come with a few changes

mypy_type_check now parameterized with generated (code) and anchor (the Template). The anchor is used to recover its module source,

Note, in practice there is a few more tweaks:

Mypy runs on the whole module, but only diagnostics inside the spliced span are raised, so an unrelated pre-existing error elsewhere in the module can't fail synthesis.
Non-nestable constructs (from x import *, __future__) are caught by an explicit AST pre-scan, since mypy silently accepts a nested star-import; a fatal mypy error (exit ≥ 2) is surfaced rather than silently passed; unrecoverable source (no file, no linecache) is skipped with a logged reason.

Name collisions (Synthesized Callable Name shouldn't matter #542) are now handled structurally: the generated def is a function-body local that shadows a colliding context name rather than redefining it at module level, so the AST-rename pass is gone. The whole quotation layer is deleted — type_to_ast, signature_to_ast, collect_variable_declarations, collect_runtime_type_stubs, collect_imports, and _RenameTransformer/_generate_unique_name (net +510 / −1991).
Behavior change. Runtime-only / local types, and names with no lexical presence at the splice point, now raise instead of being (partially, fakily) stubbed. The check is sound on annotated generated code; unannotated bodies are checked at their signature only (--check-untyped-defs is deliberately off — it rejects runnable heterogeneous-container code).

Tests.

Deleted the quotation-internals unit tests; kept the security/REPL tests
Added behavior tests asserting raise-or-not (not mypy text): Synthesized Callable Name shouldn't matter #542 collision shadowing, the gate (unrelated module error doesn't block a correct synthesis), closures (enclosing locals in scope), method/static/class templates, Enum + dataclass context (Decoding runtime-only Pydantic models in Callables fails #576), injected-name, non-nestable, linecache-backed recovery (the REPL tool code should be typechecked when possible #690 path), annotation-collision, and end-to-end decode through Encodable[Callable]. mypy effectful/handlers/llm/ clean, ruff clean, full LLM suite green.

datvo06 · 2026-06-30T14:33:14Z

My intuition is mypy failed with stress test. Checking..

eb8680

Thanks, this seems like a big improvement. There are several heuristics and silent failures that I worry will undermine its correctness, though.

eb8680 · 2026-06-30T18:36:14Z

+
+    # Splice in place: replace the body with the generated body, binding the
+    # target against the (source) return annotation via `return`. Strip the
+    # app-level decorators (`@Template.define`, …) but KEEP `@staticmethod`/


Why do we need to strip the decorators at all, if the code being spliced is entirely confined to the function body? The less we do to the surrounding code, the more robust this splicing procedure will be.

Earlier Claude convinced me with some example that if we keep the annotator, then mypy would fail. But checking again carefully, it's not the case.
I'm removing the strip decorator step.

eb8680 · 2026-06-30T18:38:09Z

+    try:
+        module_ast = ast.parse(module_source)
+    except SyntaxError:
+        logger.warning("skipping type check: cannot parse module source for %r", fn)


Why would this fail, and why would we want that failure to be silent? Shouldn't we have confirmed by this point in the computation that the code is syntactically valid?

It would fail if we parsed the file while the writing was not done. I think I encountered it during one test, but I can't reproduce that since then. Will remove this.

eb8680 · 2026-06-30T18:39:34Z

+    # (unparse reassigns line numbers), so diagnostics can be scoped to it.
+    checked_ast = ast.parse(checked_source)
+    spliced = _find_def_by_qualname(checked_ast, fn.__qualname__)
+    if spliced is None:  # pragma: no cover - the splice is always re-locatable


What does this comment mean? When would spliced be None?

It can't be None if _find_def_by_qualname does not return None. I'm removing this check

eb8680 · 2026-06-30T18:54:45Z

+
+    # Locate the spliced def's line span in the *checked* source's coordinates
+    # (unparse reassigns line numbers), so diagnostics can be scoped to it.
+    checked_ast = ast.parse(checked_source)


Why are we unparseing and immediately parseing again here? Does this lose information? Is this doing something on top of what ast.fix_missing_locations is doing to module_ast?

It needed to do it to remove the comments, otherwise we would have to do comments re-indenting.

This is the example:

def count_a(s): return s.count('a')

but the template body is indented (inside def the_template(...):, maybe inside a class too). To splice it in, you must add leading spaces to every generated line. That's fine for code, but it corrupts multi-line string literals, because the leading spaces land inside the string:

def gen(): """line one line two""" # the string is "line one\nline two" indent by 4 to place it in the body → def gen(): """line one line two""" # the string is now "line one\n line two" ← changed!

line two gained 4 spaces inside the docstring. The string's value changed. And LLM-generated functions very often have docstrings, so this isn't a corner case.

eb8680 · 2026-06-30T18:59:37Z

+    template_def = _find_def_by_qualname(module_ast, fn.__qualname__)
+    if template_def is None:
+        logger.warning(
+            "skipping type check: cannot locate %r in its module source",


If we are able to recover the module's source, shouldn't fn have definition site line numbers associated with it that we can use to identify it directly, instead of this potentially failure-prone name matching heuristic _find_def_by_qualname? Under what conditions would we expect to recover module_ast but not be able to locate fn?

Yes, I now use template_def = _find_def_at_lineno(module_ast, fn.__code__.co_firstlineno), it's the first line comment, so it's both the comment and the function body, we wouldnt have to worry about aligning comment offset.

eb8680 · 2026-06-30T19:32:45Z

-    "__spec__",
-    "_frozen_importlib",
-)
+def _unwrap(fn: Any) -> Any:


Is this really necessary, or can it be removed if we outsource more of the introspection of fn to inspect? The fewer of these heuristics we have the more robust our type-checking step will be.

eb8680 · 2026-06-30T19:35:23Z

+    return "".join(lines) if lines else None
+
+
+def _find_def_by_qualname(


As mentioned in another comment, it would be better to remove this heuristic in favor of exact lookup using line numbers that guarantee correctness in all cases.

eb8680 · 2026-06-30T19:40:51Z

+    return region
+
+
+def mypy_type_check(generated: ast.Module, anchor: Any) -> None:


I think we probably want to split this into two functions, one that handles splicing and constructing the modified source code and one that just applies mypy to whatever source it's given, whether it came from the splicing function or somewhere else.

eb8680 · 2026-06-30T19:44:37Z

-                original_error=e,
-                raw_message=raw_message,
-            ) from e
+    # Tool-argument Callables have no source anchor (out of scope for the splice


What does this comment mean? When would this apply? Surely if we are synthesizing code to pass to a Tool we have access to its source, and we know it's happening lexically in the body of the Template definition?

eb8680 · 2026-06-30T20:07:58Z

+        with handler(
+            {
+                _get_history: lambda: history_copy,
+                type_check_anchor: lambda: template.__default__,


Rather than managing this synthesis context variable with an operation and handler that are hard to disentangle from Template.__apply__, it would probably be better to route it through Pydantic's context argument during decoding, as we already do with the lexical environment.

Type-check LLM-generated Callables by splicing them into the Template's real module source -- in place at the template def's position -- and running mypy on the whole module, raising only on diagnostics inside the spliced region. This removes the brittle quotation layer (type_to_ast, signature_to_ast, collect_*, _RenameTransformer) and the runtime/local-type stubbing it depended on, which mishandled common types such as Enum and runtime-created Pydantic models. - mypy_type_check now takes (generated, anchor); the anchor (Template.__default__) is provided by the type_check_anchor effect op, bound per call by Template.__apply__ (so nested synthesis sees the right one) and rebound to None around tool-argument decoding, so the argument path skips the check. - Region-scoped diagnostics: unrelated pre-existing errors elsewhere in the module never block synthesis (no spurious retry burn). - Leaves the template def's decorators untouched (mypy checks the body against its declared return type regardless of decorators); explicit pre-scan for non-nestable constructs (star-import, __future__); source recovery via getsourcelines/file/ linecache (REPL-defined templates); fatal mypy errors are surfaced not swallowed. - Name collisions (#542) are handled structurally by nesting/shadowing, so the AST-rename pass is deleted. - Delete the quotation emitters + their unit tests; add behavior tests covering collision shadowing, the gate, closures, method/static/class templates, Enum/dataclass context, injected-name, non-nestable, and linecache recovery.

The scan is provider-independent AST validation, not type-checking; lift it into the decode as a precondition. Keep it gated on the anchor -- the constructs it rejects are illegal only once nested by the splice, and legal on the argument path's module-level exec.

datvo06 force-pushed the dn-696-splice-typecheck branch 2 times, most recently from ba6761e to c4059f2 Compare June 30, 2026 14:24

datvo06 force-pushed the dn-696-splice-typecheck branch from c4059f2 to 0375180 Compare June 30, 2026 14:38

datvo06 marked this pull request as ready for review June 30, 2026 15:14

datvo06 requested a review from eb8680 June 30, 2026 15:14

eb8680 reviewed Jun 30, 2026

View reviewed changes

datvo06 force-pushed the dn-696-splice-typecheck branch 2 times, most recently from 0a36f8a to 9dd1252 Compare July 1, 2026 01:31

datvo06 force-pushed the dn-696-splice-typecheck branch from 9dd1252 to b3ec9c3 Compare July 1, 2026 17:11

		return "".join(lines) if lines else None


		def _find_def_by_qualname(

		return region


		def mypy_type_check(generated: ast.Module, anchor: Any) -> None:

Uh oh!

Conversation

datvo06 commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datvo06 commented Jun 30, 2026

Uh oh!

eb8680 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

datvo06 Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

datvo06 commented Jun 30, 2026 •

edited

Loading

datvo06 Jul 1, 2026 •

edited

Loading