Skip to content

Update evals docs: Add tool-usage argument matching feature#704

Draft
claude[bot] wants to merge 1 commit into
mainfrom
docs/TSP-1328
Draft

Update evals docs: Add tool-usage argument matching feature#704
claude[bot] wants to merge 1 commit into
mainfrom
docs/TSP-1328

Conversation

@claude

@claude claude Bot commented Jul 1, 2026

Copy link
Copy Markdown

Summary

  • Documents the new argument conditions capability for the Tool Usage check type in evals
  • Adds a new "Argument conditions" subsection inside the Tool Usage accordion, covering four match types: exact match, contains, regex, and not empty
  • Notes that conditions compose with existing count and position checks, and that a plain-English preview is shown in the form
  • Includes a callout that the feature is currently behind the evals-toolusage-argument-matcher feature flag and works for both single-agent and workforce (node-scoped) evals

Linear issue

https://linear.app/relevance/issue/TSP-1328/

🤖 Generated with Claude Code

Documents the new argument conditions capability for Tool Usage evals checks,
including the four match types (exact, contains, regex, not empty) and how to
configure them in the eval rule form.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@claude claude Bot added the docs-drafter Documentation drafted by Claude label Jul 1, 2026
@linear

linear Bot commented Jul 1, 2026

Copy link
Copy Markdown

TSP-1328

@mintlify

mintlify Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
relevanceai 🟢 Ready View Preview Jul 1, 2026, 5:34 AM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

🎯 Vibe check

Reviewed: 1 file (1 with issues, 0 clean)

Scores

Dimension Score What's holding it back
🟡 Consistency 8/10 "workforce" not capitalized (product noun); bold label inside <Info> callout violates CLAUDE.md rule
🟢 Technical clarity 9/10 Feature flag mentioned on line 108 with no guidance on how to enable it
🟢 Non-technical clarity 9/10 No issues found beyond minor accordion content style
🟡 Structure 7/10 No closing CTA; sequential numbered lists should use <Steps>; example scenario accordions use label-value format instead of flowing sentences

Score key: 🟢 9–10, 🟡 6–8, 🔴 1–5.

Overall vibe: This is a comprehensive, well-structured reference for a complex feature — the table-heavy field descriptions, inline demo videos, and example scenarios are all genuinely useful. Three small issues (a callout rule violation, one missing capitalization, and accordion content style) are easy fixes before merging.

🔧 Issues (3)
  • build/agents/build-your-agent/evals.mdx:8<Info> callout starts with a bold label **Rollout Status**:. CLAUDE.md prohibits bold labels inside callouts — callouts must be a single plain paragraph. Remove the bold prefix and rewrite as prose: "Evals is rolling out progressively, starting with Enterprise customers. If you don't see this feature in your account yet, reach out to your account manager to discuss access."
  • build/agents/build-your-agent/evals.mdx:108 — "workforce (node-scoped) evals" — workforce should be Workforce. This refers to the Relevance AI Workforce product feature, which is a proper noun per CLAUDE.md.
  • build/agents/build-your-agent/evals.mdx:108 — The sentence "This feature is currently behind the evals-toolusage-argument-matcher feature flag" gives technical readers no indication of how to enable it. Add a brief note: e.g., "Contact your account manager to have this flag enabled." or link to wherever feature flags are managed.
🧩 Component suggestions (2)
  • build/agents/build-your-agent/evals.mdx:118–123, 139–168, 233–235, 283–289 — Four sequential procedures use plain numbered markdown lists ("Follow these steps…", "To create a Check from the Checks tab:"). CLAUDE.md says to use <Steps> for sequential procedures to add visual progress indicators. Note: the sibling page prompt.mdx also uses numbered lists, so this may be a repo-wide convention to align rather than a single-file fix.

  • build/agents/build-your-agent/evals.mdx:189–221 — The three example scenario <Accordion> components contain bold key-value labels (**Scenario name**:, **Description**:, **Up to**:, **Check**:, *Evaluation Prompt*:). CLAUDE.md says accordion content should be flowing sentences, not label-value formats. Options: rewrite as narrative prose describing each example ("This scenario simulates a frustrated customer who was double-charged…"), or pull the fields into a compact table that mirrors the scenario creation form.

🏗️ Page structure (1)
  • build/agents/build-your-agent/evals.mdx — No closing CTA. This is a concept + how-to page that introduces an entire feature; readers finishing it have no suggested next step. Consider adding a ## What's next? section. Natural candidates from the repo: /build/agents/build-your-agent/agent-settings (for publishing configuration that intersects with the Publish section), /build/agents/build-your-agent/triggers (for understanding what fires Agent tasks that Monitor evaluates), or a Workforce overview page if one exists (since the page references workforce evals).
⚠️ Contradictions (0)

No contradictions found between evals.mdx and the context pages read (prompt.mdx).

🔋 Credit usage
Item Count
Files reviewed 1
Context pages read 1
Total lines processed ~628

Files read: build/agents/build-your-agent/evals.mdx (407 lines), build/agents/build-your-agent/prompt.mdx (221 lines)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-drafter Documentation drafted by Claude

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant