Skip to content

feat(formats): implement Express and Elemental InferenceFormats #1982

Open
gspencergoog wants to merge 6 commits into
mainfrom
unified-inference-formats-strategies
Open

feat(formats): implement Express and Elemental InferenceFormats #1982
gspencergoog wants to merge 6 commits into
mainfrom
unified-inference-formats-strategies

Conversation

@gspencergoog

@gspencergoog gspencergoog commented Jul 10, 2026

Copy link
Copy Markdown
Collaborator

Summary

Refactors the structured inference SDK API and evaluation harness in Python to unify the handling of different inference formats (standard JSON transport, experimental Express DSL, and experimental Elemental HTML). It does this by restructuring the formats as polymorphic implementations of the InferenceFormat interface and using these abstractions to consolidate system prompt generation and response parsing logic.

Changes

  • SDK Package & Interface Unification:
    • Restructured experimental Express and Elemental formats under agent_sdks/python/a2ui_agent/src/a2ui/inference_formats/experimental/.
    • Defined polymorphic methods on the base InferenceFormat class (decompile, transform_examples, format_description, catalog_description) and implemented them across TransportFormat, ElementalFormat, and ExpressFormat.
    • Updated prompt generators (ExpressPromptGenerator, ElementalPromptGenerator) to align with the new structure.
  • Evaluation Solvers Consolidation:
    • Replaced the separate format-specific solvers (direct.py, express.py, elemental.py) under eval/a2ui_eval/strategies/ with a unified format.py solver.
    • The format solver dynamically loads the correct format implementation (TransportFormat, ExpressFormat, or ElementalFormat) based on config, allowing polymorphic prompt formatting and output compilation/validation.
  • Robustness & General Fixes:
    • Fixed case-insensitive action property resolution in ElementalCompiler when multiple action properties are present.
    • Ensured prompt generator helper/catalog/decompiler are initialized in catalog_description when called prior to generate().
    • Moved regular expression compilation in ElementalParser to the module level for efficiency.
    • Cleaned up redundant imports and optimized invariant dataset metadata accesses in eval/ to fail fast.
  • Tests & Conformance:
    • Renamed the conformance suite schema_manager.yaml to inference_format.yaml.
    • Updated all Python SDK test suites and conformance tests to reflect the new layout and verified all 621 tests pass.

Impact & Risks

  • This is a breaking refactoring for internal SDK layout, but encapsulates formatting details cleanly behind the unified InferenceFormat API and preserves external integration endpoints.

Testing

Verify the changes by running all SDK and evaluation unit tests:

uv run pytest

gemini-code-assist[bot]

This comment was marked as resolved.

@gspencergoog gspencergoog force-pushed the unified-inference-formats-strategies branch 3 times, most recently from 6238957 to 82bd501 Compare July 10, 2026 16:56
@gspencergoog gspencergoog force-pushed the unified-inference-formats-foundation branch from 89a547a to 1134c08 Compare July 10, 2026 16:59
@gspencergoog gspencergoog force-pushed the unified-inference-formats-strategies branch 10 times, most recently from 5266cef to 77a52dd Compare July 13, 2026 21:12
@gspencergoog gspencergoog force-pushed the unified-inference-formats-foundation branch from dd30576 to 914f571 Compare July 13, 2026 21:44
@gspencergoog gspencergoog force-pushed the unified-inference-formats-strategies branch 2 times, most recently from c0e7429 to 33fc7c2 Compare July 13, 2026 21:49
@gspencergoog gspencergoog force-pushed the unified-inference-formats-strategies branch 8 times, most recently from 556b974 to 280f9a9 Compare July 14, 2026 21:41
@gspencergoog gspencergoog force-pushed the unified-inference-formats-foundation branch from bc7c981 to a586f1f Compare July 14, 2026 22:52
@gspencergoog gspencergoog force-pushed the unified-inference-formats-strategies branch 7 times, most recently from fca8e9a to e910b3c Compare July 15, 2026 00:56
Base automatically changed from unified-inference-formats-foundation to main July 15, 2026 21:30
@gspencergoog gspencergoog force-pushed the unified-inference-formats-strategies branch from e910b3c to 090a859 Compare July 15, 2026 21:43
…r API

style(express): remove dead empty TYPE_CHECKING block in express prompt_generator.py

feat(sdk): remove remaining A2UI_EXPRESS_ENABLED environment variable references

feat(sdk): support complete flag on has_format_content for Express and Elemental

feat(sdk): implement has_format_content for Express and Elemental parsers

refactor(sdk): rename prompts directory to prompt (singular)

refactor(sdk): hide Express and Elemental prompt generators from package exports

refactor(sdk): expose prompt_generator on Express/Elemental formats and move to prompts/generator.py

Refactor the Express and Elemental inference strategies to the new configuration.
@gspencergoog gspencergoog force-pushed the unified-inference-formats-strategies branch from 0bbda91 to d495ff0 Compare July 15, 2026 21:52
@gspencergoog gspencergoog changed the title feat(formats): implement Express and Elemental inference format strategies feat(formats): implement Express and Elemental InferenceFormats Jul 15, 2026
gemini-code-assist[bot]

This comment was marked as resolved.

…plate duplication

- Promotes decompile, transform_examples, format_description, and catalog_description as abstract methods on the InferenceFormat base class.
- Standardizes parameter signatures for format_description and catalog_description across Express, Elemental, and Transport formats.
- Removes duplicated prompt templates and translation/compilation logic from ExpressPromptGenerator and ElementalPromptGenerator, delegating directly to the format classes.
- Standardizes backwards-compatible prompt generator helpers to fallback to default format instances if initialized with raw catalog models.
- Resolves all Python SDK unit tests and Kotlin conformance tests.
@gspencergoog gspencergoog force-pushed the unified-inference-formats-strategies branch from c7fd39d to 1cc6a6e Compare July 15, 2026 23:05
gemini-code-assist[bot]

This comment was marked as resolved.

@gspencergoog gspencergoog requested a review from nan-yu July 15, 2026 23:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant