Skip to content

feat: add public StandardsCatalog for listing and validating academic standards - #149

Open
adnanrhussain wants to merge 1 commit into
ahussain/sdk-kg-hardeningfrom
ahussain/sdk-standards-api
Open

feat: add public StandardsCatalog for listing and validating academic standards#149
adnanrhussain wants to merge 1 commit into
ahussain/sdk-kg-hardeningfrom
ahussain/sdk-standards-api

Conversation

@adnanrhussain

@adnanrhussain adnanrhussain commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #148

Why

Nothing public can list, validate, or resolve standards (src/index.ts exports only Jurisdiction). Batch can't check a statement_codes column before spending LLM calls on it, and consumers reimplement the REST calls — demos/typescript/server/kg.ts says so in a comment, and carried the same pagination bug #148 fixes.

What

StandardsCataloglistStandards, getStandard, resolveStandard, validateCodes. KnowledgeGraphClient stays internal so its cache and concurrency semantics don't become API.

The resolver, and why the classification has five states

A statement code does not reliably identify one standard. From a read-only scan of the production KG, Mathematics has 1,950 of 43,019 (jurisdiction, code) pairs matching multiple standards — worst case 40. Utah F.IF.7.b matches five, differing only by course (Secondary Math I/II/III, Honors, Precalculus) with 3/5/0/0/10 learning components. All five share identical isCurrent, adoptionStatus, normalizedStatementType and gradeLevel, and the course lives on an ancestor node that search cannot filter on. So no query fixes this; it has to be resolved client-side.

resolveStandard fetches all candidates, discards those with no learning components, and classifies what remains:

Status Meaning Math codes
resolved One candidate has learning components — or several share an identical set, so any is equivalent 732 + 245
no-learning-components The code is real but nothing evaluable is authored against it. An evaluation would score 0 of 0 — not a failure and not non-alignment 891
ambiguous Candidates carry different learning components; candidates[] is returned so the caller picks a uuid 82
not-found No such code in this jurisdiction
unchecked The lookup itself failed (network, 5xx). Says nothing about the code

That takes genuine guessing from 1,950 cases to 82 — and makes 0/0 mean one specific thing instead of two.

Interchangeability is decided on learning-component sets, not descriptions. 13 Mathematics codes have candidates whose descriptions read identically but whose components differ — Georgia A.PAR.4.1 has 14 versus 2. Comparing text would have called those safe.

Notes for review

  • validateCodes returns five states, not a boolean. A boolean forces a network blip into "the code is bad". Only AuthenticationError/ConfigurationError throw, since those doom every other lookup identically; the rest yield unchecked and the remaining codes still resolve. Promise.all here would have reintroduced the all-or-nothing failure the sibling PR removes from evaluateItems.

  • StandardNotFoundError carries STANDARD_NOT_FOUND, not the inherited KNOWLEDGE_GRAPH_ERROR, so consumers recording error.code can separate a typo from an outage.

  • truncated is set when the candidate list hit the search limit — 5 (jurisdiction, code) pairs across all subjects exceed 50, max 87, and search takes no cursor.

  • academicSubject has no default — an unfiltered grade listing mixes math and ELA, and defaulting to 'Mathematics' would bake the current single family into a general API.

  • Empty and over-long (>50 char) codes are rejected locally. Observed max real length is 45, so that bound rejects nothing valid.

  • validateCodes aborts early on a fatal error. A bad key fails every code identically, so queued lookups short-circuit rather than issuing N doomed requests. The check sits behind a pLimit deliberately — in front of it, every lookup is already in flight before the first rejection arrives.

  • Constructor validates its inputs: non-integer or sub-1 concurrency throws ConfigurationError rather than crashing inside p-limit, and a whitespace-only academicSubject is treated as absent instead of being sent as a filter that matches nothing.

  • Cost: for the 95.5% of codes with one candidate this is one extra learning-component fetch that the evaluator would make anyway. Ambiguous codes cost one fetch per candidate, cached, so repeats across CSV rows are free.

  • A standard is never called unauthored when its components merely lack descriptions. The KG
    documents description as optional and alignment is judged from that text, so such components are
    unevaluable — but reporting "no learning components are authored" for them is simply false. The status
    stays no-learning-components (nothing evaluable either way) while the message reports what was
    actually observed, and undescribedComponentCount is set so a report can separate a data-quality gap
    from a genuinely empty standard. Depends on getLearningComponentSet from fix: harden Knowledge Graph client pagination, code normalization, and ambiguity detection #148.

Not included

Verification

npm run lint (0 errors), typecheck, test:unit — 377 passing, 34 in the catalog suite. build confirms resolveStandard, CodeResolutionStatus and StandardCandidate reach dist/index.d.ts.

Mutation tested with Stryker: 88% on covered code, with the resolver's classification branches, the early-abort check, and both constructor validations all confirmed killed.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Introduces a new public TypeScript SDK surface for listing and validating academic standards via a StandardsCatalog, while hardening the underlying Knowledge Graph client’s normalization, ambiguity detection, and pagination behavior.

Changes:

  • Adds a public StandardsCatalog with listStandards, getStandard, and validateCodes, plus related exported types.
  • Enhances KnowledgeGraphClient with statement-code normalization, ambiguity flagging (limit=2), and shared cursor-pagination handling.
  • Updates math standards alignment evaluator/tests to reflect the KG client option changes (removal of caller-provided limit).

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
sdks/typescript/src/knowledge-graph/standards-catalog.ts New public catalog wrapper over KG lookups, including bulk code validation behavior.
sdks/typescript/src/knowledge-graph/client.ts Adds normalization helper, ambiguity detection, cursor pagination utility, and updated return shapes.
sdks/typescript/src/knowledge-graph/types.ts Extends StandardInfo with required statementCode and normalizedCode, plus ambiguous.
sdks/typescript/src/knowledge-graph/index.ts Re-exports catalog and normalization symbols/types from the KG module barrel.
sdks/typescript/src/index.ts Exposes StandardsCatalog, normalizeStatementCode, and related types/errors from the package entrypoint.
sdks/typescript/src/errors.ts Adds StandardNotFoundError and allows KnowledgeGraphError to carry distinct error codes.
sdks/typescript/src/evaluators/math/standards-alignment.ts Removes deprecated KG limit option usage and adapts to updated KG client behavior.
sdks/typescript/tests/unit/knowledge-graph/standards-catalog.test.ts New unit coverage for catalog listing and bulk validation semantics.
sdks/typescript/tests/unit/knowledge-graph/client.test.ts Updates/extends KG client unit tests for ambiguity, normalization, and pagination behavior.
sdks/typescript/tests/unit/evaluators/math/standards-alignment.test.ts Updates expected KG call options after limit removal.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread sdks/typescript/src/knowledge-graph/standards-catalog.ts Outdated
Comment thread sdks/typescript/src/knowledge-graph/standards-catalog.ts
@adnanrhussain
adnanrhussain force-pushed the ahussain/sdk-standards-api branch from 83295a4 to d7f8c92 Compare August 7, 2026 04:20
@codecov

codecov Bot commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.87234% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...ypescript/src/knowledge-graph/standards-catalog.ts 97.72% 2 Missing ⚠️

📢 Thoughts on this report? Let us know!

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

@adnanrhussain

Copy link
Copy Markdown
Collaborator Author

Five of six fixed in 4db75376; one declined with reasoning.

Fixed

  • getStandard guard asymmetry — both guards now live in a shared localCodeProblem used by getStandard and resolveStandard. getStandard throws ValidationError, and is now async so it rejects rather than throwing synchronously (a Promise-returning method that sometimes throws sync forces callers into both try/catch and .catch()).
  • Ambiguous message/candidates — both now report one representative per distinct component set, so interchangeable candidates no longer inflate the count or the list.
  • Truncated + all-zero components — returns unchecked with a message saying the list was capped, instead of a definitive no-learning-components it cannot support.
  • academicSubject doc — corrected: omitting it makes the result subject-dependent (Multi-State mixes subjects; another jurisdiction resolves to whichever per-subject framework comes first), not simply mixed.
  • platformApiKey now trimmed, matching academicSubject.

Declined: 400 as fatal. After the local guards, the plausible global cause is an invalid academicSubject, but a 400 can equally be code-specific — and treating it as fatal would abort the whole run for one malformed code, which is the failure mode this method was twice changed to avoid. unchecked is the honest answer for a code we could not check. Happy to revisit if you would rather validate academicSubject against the enum at construction, which fixes the global case without the abort risk.

Tests extended rather than added where possible; zero surviving mutants in every changed region, catalog now at 88.7% on covered code.

@adnanrhussain
adnanrhussain force-pushed the ahussain/sdk-standards-api branch from 4db7537 to b1b5550 Compare August 7, 2026 05:37
@adnanrhussain

Copy link
Copy Markdown
Collaborator Author

Both addressed in b1b55508.

unchecked doc drift — you're right, that was introduced by my truncation fix. The union member now reads: no verdict could be reached, either because the lookup failed (network, 429, 5xx) or it succeeded but its answer isn't supportable (candidate list capped at the search limit with no visible candidate carrying components). Check error for which.

Finding 2 — the misleading-answer half is fixed: a 400 now returns unchecked with a message naming academicSubject and jurisdiction as the likely cause, rather than a bare "couldn't check". So the user is pointed at the real problem instead of hunting the code.

Still not fixed, deliberately: it issues N requests before saying so, and academicSubject is not validated against the enum. I avoided both available fixes because each has a worse failure mode — making 400 fatal discards every other result for what might be a code-specific 400, and hardcoding the subject list would reject a valid subject the SDK hasn't been updated for.

The clean fix is a third option neither of us proposed: reuse validateCodes' existing fatal short-circuit to stop scheduling after the first 400, but degrade the remainder to unchecked instead of throwing. One request, N honest results, no abort. Happy to do that now if you want it in this PR — otherwise it's a small follow-up.

@adnanrhussain
adnanrhussain marked this pull request as ready for review August 7, 2026 05:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants