diff --git a/doc/changes/unreleased.md b/doc/changes/unreleased.md index bb2e66c07..f8092cb65 100644 --- a/doc/changes/unreleased.md +++ b/doc/changes/unreleased.md @@ -2,7 +2,7 @@ ## Features -- #940: Added shared validation for packaged agent skills and the `skills:check` Nox session. -- #938: Added the `skills:install` Nox session for installing the packaged PTB agent skill. - -## Summary +* #940: Added shared validation for packaged agent skills and the `skills:check` Nox session. +* #938: Added the `skills:install` Nox session for installing the packaged PTB agent skill. +* #942: Added api-contract-audit skill for identifying mismatches between type annotations, docstrings, and runtime + behavior \ No newline at end of file diff --git a/exasol/toolbox/skills/api-contract-audit/SKILL.md b/exasol/toolbox/skills/api-contract-audit/SKILL.md new file mode 100644 index 000000000..de5e6f77e --- /dev/null +++ b/exasol/toolbox/skills/api-contract-audit/SKILL.md @@ -0,0 +1,90 @@ +--- +name: api-contract-audit +description: Audit a Python library's public API for inconsistencies between type annotations, docstrings, user-facing documentation/examples, and actual runtime behavior. Use when reviewing API changes, checking whether public methods accept undocumented parameter shapes, or validating that docs and type hints match enforcement in code. +--- + +# API Contract Audit + +Use this skill when the task is to review a Python package's public API contract rather than implement features. + +Focus on externally visible behavior: +- public functions and methods +- exported classes +- user-facing docs and examples +- runtime validation and coercion + +Do not assume the annotation is the source of truth. The goal is to find drift between multiple sources of truth. + +## Inputs To Compare + +For each relevant public API entrypoint, compare: +- signature and type annotations +- docstring parameter and return descriptions +- examples in docs, README, and example scripts +- runtime behavior in the implementation path + +Treat these as separate claims. Report when they disagree. + +## What To Look For + +Prioritize these mismatch patterns: +- annotation says `str`, but implementation accepts or requires tuple-like schema-qualified identifiers +- annotation says one scalar type, but runtime hard-checks another with `isinstance(...)` +- docstring says a parameter or return type that does not match the signature +- docs/examples call the API with arguments that disagree with the annotation or actual signature +- implementation silently accepts more forms than the public docs mention +- wrappers expose narrower types than the lower-level public method they forward to +- runtime coercion like `int(val)` or `str(val)` that makes the public contract broader than the annotation suggests + +Typical search signals: +- `isinstance(` +- `type(` +- `raise ValueError` +- identifier-formatting helpers +- tuple-specific branches +- wrapper methods that pass through parameters unchanged + +## Workflow + +1. Enumerate the public API surface relevant to the request. +2. Read the implementation of each public method and the immediate downstream code it calls. +3. Trace parameter handling until the real runtime constraint is clear. +4. Cross-check docstrings and user-facing docs/examples. +5. Report only concrete inconsistencies or clearly label residual uncertainty. + +Prefer `rg` for discovery. Good starter patterns: + +```bash +rg -n "^class |^ def " package_dir +rg -n "isinstance\\(|type\\(|raise ValueError|raise TypeError" package_dir +rg -n "function_name\\(" README.md doc examples test +``` + +## Output Format + +Present findings first, ordered by severity. + +For each finding include: +- severity: High, Medium, or Low +- affected API +- what the annotation/doc claims +- what the implementation really does +- file references for both sides of the mismatch + +After findings, optionally include: +- open questions where intended behavior is unclear +- a short summary of recurring patterns + +If no findings are discovered, say that explicitly and mention any coverage limits. + +## Severity Guidance + +- High: likely to mislead callers, break type-checked usage, or document the wrong accepted input shape +- Medium: accepted behavior is real but under-documented, or docs/examples contradict each other +- Low: naming, docstring argument labels, stale prose, or smaller clarity issues + +## Boundaries + +- Do not rewrite the API contract on your own. If code, docs, and examples disagree, report the disagreement. +- Do not stop at the first example. Check for the same pattern across sibling APIs. +- Do not treat private helper inconsistencies as findings unless they affect public behavior. \ No newline at end of file diff --git a/test/resources/skills/api-contract-audit/eval_cases.yml b/test/resources/skills/api-contract-audit/eval_cases.yml new file mode 100644 index 000000000..285ae7ea8 --- /dev/null +++ b/test/resources/skills/api-contract-audit/eval_cases.yml @@ -0,0 +1,101 @@ +version: 1 +skill: "api-contract-audit" +cases: + - id: "audit-public-api-contract" + category: "audit" + prompt: > + Audit the target Python library's public API for concrete inconsistencies between its + type annotations, docstrings, user-facing documentation or examples, + and actual runtime behavior. Report findings first, ordered by severity. + For each finding, identify the affected API, the claimed contract, the + observed behavior, and relevant source file references. Do not modify + the code or rewrite the API contract. + expected: + must_include: + - "severity" + - "API" + - "annotation" + - "runtime" + - "file" + must_not_include: + - "rewrite the API" + - "fix the code" + + - id: "interpret-explicit-type-check" + category: "runtime-validation" + prompt: > + A public API implementation contains an isinstance() check. Audit + whether that check is consistent with the public annotation and + documentation. Explain what the check actually accepts or rejects and + whether the available evidence establishes a contract mismatch. Do not + treat the presence of isinstance() alone as proof of a defect. + expected: + must_include: + - "isinstance" + - "annotation" + - "documentation" + - "runtime" + - "evidence" + must_not_include: + - "isinstance() is automatically a mismatch" + - "fix the code" + + - id: "interpret-coercion-and-errors" + category: "runtime-validation" + prompt: > + Audit a public API whose implementation coerces or rejects values at + runtime, for example through int(), str(), ValueError, or TypeError. + Compare the accepted and rejected inputs with the public annotations and + documentation. Report any concrete contract mismatch, its severity, and + the relevant source file reference. + expected: + must_include: + - "coercion" + - "runtime" + - "annotation" + - "documentation" + - "severity" + - "file" + must_not_include: + - "rewrite the API" + - "fix the code" + + - id: "compare-docstrings-and-signatures" + category: "documentation" + prompt: > + Check whether the target Python library's public method signatures, type annotations, and + docstrings describe the same contract. Report concrete inconsistencies + with the affected API, severity, and source file references. If no + finding can be established, state the coverage limitation instead of + guessing. + expected: + must_include: + - "signature" + - "docstring" + - "annotation" + - "severity" + - "file" + must_not_include: + - "rewrite the API" + - "guess" + + - id: "compare-user-facing-examples" + category: "documentation" + prompt: > + Check whether the target Python library's user-facing examples in documentation, including + RST files, README content, and example scripts, agree with the public + API signature, annotations, and runtime behavior. Report concrete + inconsistencies with the affected API, severity, and source file + references. Do not assume an example is authoritative without checking + the implementation. + expected: + must_include: + - "example" + - "documentation" + - "API" + - "runtime" + - "severity" + - "file" + must_not_include: + - "rewrite the API" + - "assume the example is correct" \ No newline at end of file