feat(ipa): Enrich IPA-100 Language with Guideline components#66
Conversation
Children of <Guideline> render with Docusaurus' markdown spacing; the .markdown > * rules only match direct children of a markdown container. CLOUDP-399880
|
There is an existing patch(es) for this commit SHA: Please note that the status that is posted is not in the context of this PR but rather the (latest) existing patch and that may affect some tests that may depend on the particular PR. If your tests do not rely on any PR-specific values (like base or head branch name) then your tests will report the same status. If you would like a patch to run in the context of this PR and abort the other(s), comment 'evergreen retry'. |
andmatei
changed the title
andmatei
requested review from
andreaangiolillo,
drinkbird,
lovisaberggren,
matt-condon,
saisundar,
wtrocki and
yelizhenden-mdb
There was a problem hiding this comment.
Q: Do we have a guideline or documentation on how to define guideline metadata in this repo? If not, we should definitely create one. It would be incredibly helpful for future IPA authors to see example cases showing how to define new guidelines.
This doesn't need to be part of this PR—a separate PR dedicated to this would be great. I think adding it to contributing.md makes the most sense, but feel free to suggest another location.
|
|
||
| <Guidelines> | ||
|
|
||
| <Guideline id="IPA-0100-must-use-american-english" given="spec" lintable> |
There was a problem hiding this comment.
Q: We validate American English using the json-schema-spell-checker on the MMS side (but not on the validation framework side, I assume). Should we flag this to the agent somehow? Or do we want them to perform a separate check anyway?
There was a problem hiding this comment.
I marked this as lintable since this should not have an agent running for this - what do you think about running that check as part of the validation framework too instead of just on mms side?
There was a problem hiding this comment.
@andmatei Good idea to make it lintable.
We should prefer deterministic (non agentic) checks when possible.
Guidelines help us to archive accuracy but also less token spend.
There was a problem hiding this comment.
It would be a good piece of pre-work before the SOA migration, as it would enable the validation of specs coming from other repos. However, I'm not sure if it's strictly needed right now or if we can just log it as a future improvement. For now, as far as I know, all of the FOAS is already generated on the MMS side, so I don't think we need to tackle this right away.
Second, correct me if I'm wrong, but doesn't a lintable tag without a corresponding lint rule tag imply that the agent should check for it? That was the main reason for my concern
There was a problem hiding this comment.
It would be a good piece of pre-work before the SOA migration, as it would enable the validation of specs coming from other repos. However, I'm not sure if it's strictly needed right now or if we can just log it as a future improvement. For now, as far as I know, all of the FOAS is already generated on the MMS side, so I don't think we need to tackle this right away.
Sounds good to me, can file a ticket to track that as a future improvement.
Second, correct me if I'm wrong, but doesn't a lintable tag without a corresponding lint rule tag imply that the agent should check for it? That was the main reason for my concern
So if it is marked as lintable, the agent should not check it, regardless of the rule tag
There was a problem hiding this comment.
Looking at the technical design, the lintable tag indicates whether there is a Spectral rule enforcing the guideline. So, when we see a lintable tag in the guideline component (like here), it means that it is set to true, and it flags to the agent that the rule is already enforced, meaning the agent doesn't need to check for it. Is it correct?
There was a problem hiding this comment.
Yes that is correct - we have a gap here that Spectral doesn't validate this, but we have other validation in place. So I think it would be a good idea to close that gap on Spectral side, but still tag this as lintable so the agent knows to skip it
@yelizhenden-mdb Filed this ticket to address this |
3 participants
Ticket: CLOUDP-399880
What
Migrates the IPA-100 (Language) principle to use the
<Guideline>component family, enriching each guideline with extra structured metadata (given,effort,lintable,informational,dependsOn) and Correct/Incorrect examples.Visual