From 5c148d4b839e4efad9cba875a3af3af5d29fa970 Mon Sep 17 00:00:00 2001 From: Rimsha Masood Date: Tue, 22 Sep 2026 07:00:12 +0200 Subject: [PATCH 1/5] Add api-contract-audit skill for type hint checking --- doc/changes/unreleased.md | 4 + .../skills/api-contract-audit/SKILL.md | 90 +++++++++++++++++++ .../skills/api-contract-audit/eval_cases.yml | 25 ++++++ 3 files changed, 119 insertions(+) create mode 100644 exasol/toolbox/skills/api-contract-audit/SKILL.md create mode 100644 test/resources/skills/api-contract-audit/eval_cases.yml diff --git a/doc/changes/unreleased.md b/doc/changes/unreleased.md index fac471d90..5d6d2e573 100644 --- a/doc/changes/unreleased.md +++ b/doc/changes/unreleased.md @@ -14,3 +14,7 @@ from the developer guide. ## Refactoring * #934: Removed unused, experimental Nox session `lint:import` + +## Feature + +* #942: Added api-contract-audit skill for identifying mismatches between type annotations, docstrings, and runtime behavior \ No newline at end of file diff --git a/exasol/toolbox/skills/api-contract-audit/SKILL.md b/exasol/toolbox/skills/api-contract-audit/SKILL.md new file mode 100644 index 000000000..de5e6f77e --- /dev/null +++ b/exasol/toolbox/skills/api-contract-audit/SKILL.md @@ -0,0 +1,90 @@ +--- +name: api-contract-audit +description: Audit a Python library's public API for inconsistencies between type annotations, docstrings, user-facing documentation/examples, and actual runtime behavior. Use when reviewing API changes, checking whether public methods accept undocumented parameter shapes, or validating that docs and type hints match enforcement in code. +--- + +# API Contract Audit + +Use this skill when the task is to review a Python package's public API contract rather than implement features. + +Focus on externally visible behavior: +- public functions and methods +- exported classes +- user-facing docs and examples +- runtime validation and coercion + +Do not assume the annotation is the source of truth. The goal is to find drift between multiple sources of truth. + +## Inputs To Compare + +For each relevant public API entrypoint, compare: +- signature and type annotations +- docstring parameter and return descriptions +- examples in docs, README, and example scripts +- runtime behavior in the implementation path + +Treat these as separate claims. Report when they disagree. + +## What To Look For + +Prioritize these mismatch patterns: +- annotation says `str`, but implementation accepts or requires tuple-like schema-qualified identifiers +- annotation says one scalar type, but runtime hard-checks another with `isinstance(...)` +- docstring says a parameter or return type that does not match the signature +- docs/examples call the API with arguments that disagree with the annotation or actual signature +- implementation silently accepts more forms than the public docs mention +- wrappers expose narrower types than the lower-level public method they forward to +- runtime coercion like `int(val)` or `str(val)` that makes the public contract broader than the annotation suggests + +Typical search signals: +- `isinstance(` +- `type(` +- `raise ValueError` +- identifier-formatting helpers +- tuple-specific branches +- wrapper methods that pass through parameters unchanged + +## Workflow + +1. Enumerate the public API surface relevant to the request. +2. Read the implementation of each public method and the immediate downstream code it calls. +3. Trace parameter handling until the real runtime constraint is clear. +4. Cross-check docstrings and user-facing docs/examples. +5. Report only concrete inconsistencies or clearly label residual uncertainty. + +Prefer `rg` for discovery. Good starter patterns: + +```bash +rg -n "^class |^ def " package_dir +rg -n "isinstance\\(|type\\(|raise ValueError|raise TypeError" package_dir +rg -n "function_name\\(" README.md doc examples test +``` + +## Output Format + +Present findings first, ordered by severity. + +For each finding include: +- severity: High, Medium, or Low +- affected API +- what the annotation/doc claims +- what the implementation really does +- file references for both sides of the mismatch + +After findings, optionally include: +- open questions where intended behavior is unclear +- a short summary of recurring patterns + +If no findings are discovered, say that explicitly and mention any coverage limits. + +## Severity Guidance + +- High: likely to mislead callers, break type-checked usage, or document the wrong accepted input shape +- Medium: accepted behavior is real but under-documented, or docs/examples contradict each other +- Low: naming, docstring argument labels, stale prose, or smaller clarity issues + +## Boundaries + +- Do not rewrite the API contract on your own. If code, docs, and examples disagree, report the disagreement. +- Do not stop at the first example. Check for the same pattern across sibling APIs. +- Do not treat private helper inconsistencies as findings unless they affect public behavior. \ No newline at end of file diff --git a/test/resources/skills/api-contract-audit/eval_cases.yml b/test/resources/skills/api-contract-audit/eval_cases.yml new file mode 100644 index 000000000..7c727260c --- /dev/null +++ b/test/resources/skills/api-contract-audit/eval_cases.yml @@ -0,0 +1,25 @@ +version: 1 +skill: "api-contract-audit" +cases: + - id: "find-type-annotation-mismatch" + category: "audit" + prompt: "Audit the PyExasol public API for mismatches between type annotations and runtime behavior." + expected: + must_include: + - "isinstance" + - "annotation" + - "severity" + must_not_include: + - "rewrite the API" + - "fix the code" + + - id: "find-docstring-mismatch" + category: "audit" + prompt: "Check if PyExasol docstrings match the actual function signatures and runtime behavior." + expected: + must_include: + - "docstring" + - "signature" + - "mismatch" + must_not_include: + - "rewrite the API" \ No newline at end of file From 9b5766b21fcadf95dc69386c862045aa75244bd0 Mon Sep 17 00:00:00 2001 From: Rimsha Masood Date: Tue, 22 Sep 2026 10:08:26 +0200 Subject: [PATCH 2/5] Fix unreleased.md content --- doc/changes/unreleased.md | 21 ++++----------------- 1 file changed, 4 insertions(+), 17 deletions(-) diff --git a/doc/changes/unreleased.md b/doc/changes/unreleased.md index 5d6d2e573..bda748ff4 100644 --- a/doc/changes/unreleased.md +++ b/doc/changes/unreleased.md @@ -1,20 +1,7 @@ # Unreleased -## Summary +## Features -This release documents how to discover and use PTB's Nox sessions in the user -guide, adds an agent skill for PTB work, and removes the unused Modules section -from the developer guide. - -## Documentation - -* #456: Documented how to discover PTB nox sessions in the user guide -* #933: Added an agent skill for PTB work - -## Refactoring - -* #934: Removed unused, experimental Nox session `lint:import` - -## Feature - -* #942: Added api-contract-audit skill for identifying mismatches between type annotations, docstrings, and runtime behavior \ No newline at end of file +* #940: Added shared validation for packaged agent skills and the `skills:check` Nox session. +* #942: Added api-contract-audit skill for identifying mismatches between type annotations, docstrings, and runtime + behavior \ No newline at end of file From c4b668437bf42da02f75f5c18404841f13e70c3a Mon Sep 17 00:00:00 2001 From: Rimsha Masood <144063231+Rimsha2535@users.noreply.github.com> Date: Thu, 24 Sep 2026 16:11:16 +0200 Subject: [PATCH 3/5] Update test/resources/skills/api-contract-audit/eval_cases.yml Co-authored-by: Ariel Schulz <43442541+ArBridgeman@users.noreply.github.com> --- test/resources/skills/api-contract-audit/eval_cases.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/resources/skills/api-contract-audit/eval_cases.yml b/test/resources/skills/api-contract-audit/eval_cases.yml index b5e312f6c..90dccf246 100644 --- a/test/resources/skills/api-contract-audit/eval_cases.yml +++ b/test/resources/skills/api-contract-audit/eval_cases.yml @@ -63,7 +63,7 @@ cases: - id: "compare-docstrings-and-signatures" category: "documentation" prompt: > - Check whether PyExasol public method signatures, type annotations, and + Check whether the target Python library's public method signatures, type annotations, and docstrings describe the same contract. Report concrete inconsistencies with the affected API, severity, and source file references. If no finding can be established, state the coverage limitation instead of From 14b2061ac6c22f207b81976e33aa253ea3d0c15a Mon Sep 17 00:00:00 2001 From: Rimsha Masood <144063231+Rimsha2535@users.noreply.github.com> Date: Thu, 24 Sep 2026 16:12:49 +0200 Subject: [PATCH 4/5] Update test/resources/skills/api-contract-audit/eval_cases.yml Co-authored-by: Ariel Schulz <43442541+ArBridgeman@users.noreply.github.com> --- test/resources/skills/api-contract-audit/eval_cases.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/resources/skills/api-contract-audit/eval_cases.yml b/test/resources/skills/api-contract-audit/eval_cases.yml index 90dccf246..6324be4bb 100644 --- a/test/resources/skills/api-contract-audit/eval_cases.yml +++ b/test/resources/skills/api-contract-audit/eval_cases.yml @@ -4,7 +4,7 @@ cases: - id: "audit-public-api-contract" category: "audit" prompt: > - Audit the PyExasol public API for concrete inconsistencies between its + Audit the target Python library's public API for concrete inconsistencies between its type annotations, docstrings, user-facing documentation or examples, and actual runtime behavior. Report findings first, ordered by severity. For each finding, identify the affected API, the claimed contract, the From 873dd365caca5817b01fe272def0688dc014053f Mon Sep 17 00:00:00 2001 From: Rimsha Masood <144063231+Rimsha2535@users.noreply.github.com> Date: Thu, 24 Sep 2026 16:12:59 +0200 Subject: [PATCH 5/5] Update test/resources/skills/api-contract-audit/eval_cases.yml Co-authored-by: Ariel Schulz <43442541+ArBridgeman@users.noreply.github.com> --- test/resources/skills/api-contract-audit/eval_cases.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/resources/skills/api-contract-audit/eval_cases.yml b/test/resources/skills/api-contract-audit/eval_cases.yml index 6324be4bb..285ae7ea8 100644 --- a/test/resources/skills/api-contract-audit/eval_cases.yml +++ b/test/resources/skills/api-contract-audit/eval_cases.yml @@ -82,7 +82,7 @@ cases: - id: "compare-user-facing-examples" category: "documentation" prompt: > - Check whether PyExasol user-facing examples in documentation, including + Check whether the target Python library's user-facing examples in documentation, including RST files, README content, and example scripts, agree with the public API signature, annotations, and runtime behavior. Report concrete inconsistencies with the affected API, severity, and source file