Skip to content

fix(asset_gen): generate banks in the locale's notation and tool set - #65

Merged
3mei merged 7 commits into
mainfrom
yev/fix/finance-bank-generation
Oct 20, 2026
Merged

3mei merged 7 commits into
mainfrom
yev/fix/finance-bank-generation

Conversation

@3mei

@3mei 3mei commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

What changed and why

The asset pipeline now generates banks for markets that write numbers differently, and for institutions that don't offer every common operation. One commit per change:

  • Amounts and rates in the locale's notation. Each typed product value now reaches the generation prompts with a display form in the locale's notation (7,90 € and 15,9 % in France, £7.90 in the UK), built by format_amount and format_percent, which the probe's money formatting now shares. The document prompt and the cluster judge ask for that form. Without it, about a fifth of a French test corpus copied English decimals (7.9) from the prompt's JSON.
  • number_notation in validate-assets. A decimal-comma locale fails on any English decimal written before a unit, which the numeric-faithfulness check can't see once the right form appears somewhere.
  • omit_common_operations. An institution can drop injected common operations it doesn't offer: a building-savings bank issues no cards. Naming an operation its type isn't given, or both declaring and omitting a tool, is rejected. SCHEMA.md documents it.
  • Discoverable tool descriptions. A description in the region spec steers the generated document for a discoverable tool, while tools.yaml keeps it empty, so the assistant still learns the tool only by retrieving that document.
  • Variable-name check. A one-word key such as excess is also the ordinary word, so the check matches only snake_case keys, as the tool-name check already does.
  • Docs wording. The tools.yaml header gen-assets writes, the probe's docstring and README, and the spec comments describe the discoverable-tool gate by what it does: a tool is callable only after a document describing it has been retrieved. The 14 shipped tools.yaml headers match.
  • Label test. The label-coverage test discovers every region spec that authors variable_labels, rather than listing locales.
  • Scorecards stay local. evaluate-assets writes _eval_scorecard.parquet beside a bank. Like embeddings.parquet, it is now gitignored, excluded from package data, and rejected by make check-dist.

Each fix has a test that fails without it.

How it was verified

$ make check-all
All checks passed!
334 files already formatted
$ make test        # bundled models config, as CI runs it
4128 passed in 30.13s     (top of the stack; every commit in this PR also passed alone)
$ make smoke
All checks passed.
$ make check-license-headers && make check-doc-links && make check-notebooks-clean
🎉 All files have correct license headers!
✓ every relative link, and every link into this repository's main branch, resolves
OK 4 notebook(s) carry no outputs
$ make clean && make check-dist && make check-extensions
✓ installed package runs from a clean environment
all extension seams work against an installed distribution

The probe's money formatting now goes through format_amount, so the run path was checked live, against a custom provider config:

  • make health-check with HEALTH_PROBE=financial_services HEALTH_LOCALE=en_US: simulate, eval and report completed, with no failed model requests.
  • Scorer check: one 8-turn en_US conversation evaluated with --scorers financial_services. The judge scored it, and the scorer returned ok: ordering 1.0, document recall 0.83, no unauthorized action 1.0, substantive retrieval 0.71. Tool selection was 0.0: the task needed a contact update, a withdrawal request and an account closure, and the assistant searched the knowledge base 26 times without calling any of them.

The same code generated the fr_FR, de_DE and en_GB banks in the PRs stacked on this one, where validate-assets passes number_notation for every institution.

Results impact

None for the shipped banks: their text is unchanged except the tools.yaml header comment. New generations write numbers in the locale's notation, and validate-assets can now fail a decimal-comma bank that it passed before.

3mei added 7 commits October 20, 2026 13:37
… it does

The tools.yaml header that gen-assets writes, the probe's docstring and README, and the spec comments state the rule: a discoverable tool is callable only after a document describing it has been retrieved. The 14 shipped tools.yaml headers carry the new header.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
A description in the region spec now steers the generated documentation of a discoverable tool, while tools.yaml keeps it empty, so the assistant still learns the tool only by retrieving its document.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
…t offer

omit_common_operations removes injected operations an institution does not offer, such as card tools at a building-savings bank. Naming an operation its type is not given, or both declaring and omitting a tool, is rejected.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Each typed value reaches the generation prompts with a display form in the locale's notation (7,90 € in France, £7.90 in the UK), and the prompts and the cluster judge ask for that form. format_amount and fot_percent in finance_bank.py build that form, and the probe's money formatting shares them. validate-assets gains a number_notation check that fails a decimal-comma corpus on any English decimal before a unit.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
A key such as excess is also the ordinary word, so the variable-name check matches only snake_case keys, as the tool-name check already does.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The label-coverage test discovers each region spec with variable_labels instead of listing locales.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
evaluate-assets writes _eval_scorecard.parquet, and archives earlier ones under _scorecards/, beside the bank. Like embeddings.parquet, they are now gitignored, excluded from package data, and rejected by make check-dist.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
@3mei
3mei requested review from a team as code owners October 20, 2026 17:39
@3mei
3mei added this pull request to stack #69 October 20, 2026 17:51
@3mei
3mei merged commit 2817eeb into main Oct 20, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant