Repository navigation
fix(asset_gen): generate banks in the locale's notation and tool set - #65
Merged
Merged
Conversation
… it does The tools.yaml header that gen-assets writes, the probe's docstring and README, and the spec comments state the rule: a discoverable tool is callable only after a document describing it has been retrieved. The 14 shipped tools.yaml headers carry the new header. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
A description in the region spec now steers the generated documentation of a discoverable tool, while tools.yaml keeps it empty, so the assistant still learns the tool only by retrieving its document. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
…t offer omit_common_operations removes injected operations an institution does not offer, such as card tools at a building-savings bank. Naming an operation its type is not given, or both declaring and omitting a tool, is rejected. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
Each typed value reaches the generation prompts with a display form in the locale's notation (7,90 € in France, £7.90 in the UK), and the prompts and the cluster judge ask for that form. format_amount and fot_percent in finance_bank.py build that form, and the probe's money formatting shares them. validate-assets gains a number_notation check that fails a decimal-comma corpus on any English decimal before a unit. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
A key such as excess is also the ordinary word, so the variable-name check matches only snake_case keys, as the tool-name check already does. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
The label-coverage test discovers each region spec with variable_labels instead of listing locales. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
evaluate-assets writes _eval_scorecard.parquet, and archives earlier ones under _scorecards/, beside the bank. Like embeddings.parquet, they are now gitignored, excluded from package data, and rejected by make check-dist. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed and why
The asset pipeline now generates banks for markets that write numbers differently, and for institutions that don't offer every common operation. One commit per change:
displayform in the locale's notation (7,90 €and15,9 %in France,£7.90in the UK), built byformat_amountandformat_percent, which the probe's money formatting now shares. The document prompt and the cluster judge ask for that form. Without it, about a fifth of a French test corpus copied English decimals (7.9) from the prompt's JSON.number_notationinvalidate-assets. A decimal-comma locale fails on any English decimal written before a unit, which the numeric-faithfulness check can't see once the right form appears somewhere.omit_common_operations. An institution can drop injected common operations it doesn't offer: a building-savings bank issues no cards. Naming an operation its type isn't given, or both declaring and omitting a tool, is rejected. SCHEMA.md documents it.tools.yamlkeeps it empty, so the assistant still learns the tool only by retrieving that document.excessis also the ordinary word, so the check matches only snake_case keys, as the tool-name check already does.tools.yamlheadergen-assetswrites, the probe's docstring and README, and the spec comments describe the discoverable-tool gate by what it does: a tool is callable only after a document describing it has been retrieved. The 14 shippedtools.yamlheaders match.variable_labels, rather than listing locales.evaluate-assetswrites_eval_scorecard.parquetbeside a bank. Likeembeddings.parquet, it is now gitignored, excluded from package data, and rejected bymake check-dist.Each fix has a test that fails without it.
How it was verified
The probe's money formatting now goes through
format_amount, so the run path was checked live, against a custom provider config:make health-checkwithHEALTH_PROBE=financial_services HEALTH_LOCALE=en_US: simulate, eval and report completed, with no failed model requests.en_USconversation evaluated with--scorers financial_services. The judge scored it, and the scorer returnedok: ordering 1.0, document recall 0.83, no unauthorized action 1.0, substantive retrieval 0.71. Tool selection was 0.0: the task needed a contact update, a withdrawal request and an account closure, and the assistant searched the knowledge base 26 times without calling any of them.The same code generated the
fr_FR,de_DEanden_GBbanks in the PRs stacked on this one, wherevalidate-assetspassesnumber_notationfor every institution.Results impact
None for the shipped banks: their text is unchanged except the
tools.yamlheader comment. New generations write numbers in the locale's notation, andvalidate-assetscan now fail a decimal-comma bank that it passed before.