Problem
Users who need AND (all must pass) or OR (any must pass) assertion logic have no examples showing how to achieve this with existing primitives. The composite evaluator and SDK inline assertions both support these patterns, but they are undocumented.
Issue #949 was closed because AND/OR logic is already expressible — but only if users know how. We need examples.
Examples to Add
AND logic (all must pass) — YAML
assertions:
- name: all_must_pass
type: composite
aggregator:
type: threshold
threshold: 1.0 # 100% of children must pass
assertions:
- type: contains
value: "capital"
- type: contains
value: "France"
OR logic (any must pass) — YAML
assertions:
- name: any_match
type: composite
aggregator:
type: weighted_average # any non-zero child raises the average above 0
assertions:
- type: contains
value: "Paris"
- type: contains
value: "paris"
- type: icontains
value: "the capital of france is paris"
Note: weighted_average is not a true OR — if only 1 of 3 passes, score = 0.33. For true OR, use a code-grader aggregator or an inline assertion.
OR logic (any must pass) — SDK inline assertion
await evaluate({
tests: [{
id: "capital",
input: "What is the capital of France?",
assert: [
({ output }) => ({
name: "mentions-paris",
score: /paris/i.test(output) || output.includes("the capital of France") ? 1 : 0,
}),
],
}],
task: myAgent,
});
Safety gate (AND with early fail) — YAML
assertions:
- name: safety_then_quality
type: composite
aggregator:
type: threshold
threshold: 1.0
assertions:
- type: llm-grader
prompt: ./safety.md
required: true # gates aggregate if safety fails
- type: llm-grader
prompt: ./quality.md
Where to add
examples/features/composite/README.md — add AND/OR patterns section
examples/features/composite/evals/ — add and-or-patterns.eval.yaml example file
- Main
README.md or docs — mention AND/OR is achievable via composite evaluator
Acceptance Signals
- AND pattern example exists in
examples/features/composite/ with working YAML
- OR pattern example exists (both YAML composite and SDK inline)
- Safety gate pattern documented
- README updated to reference these patterns
Problem
Users who need AND (all must pass) or OR (any must pass) assertion logic have no examples showing how to achieve this with existing primitives. The composite evaluator and SDK inline assertions both support these patterns, but they are undocumented.
Issue #949 was closed because AND/OR logic is already expressible — but only if users know how. We need examples.
Examples to Add
AND logic (all must pass) — YAML
OR logic (any must pass) — YAML
OR logic (any must pass) — SDK inline assertion
Safety gate (AND with early fail) — YAML
Where to add
examples/features/composite/README.md— add AND/OR patterns sectionexamples/features/composite/evals/— addand-or-patterns.eval.yamlexample fileREADME.mdor docs — mention AND/OR is achievable via composite evaluatorAcceptance Signals
examples/features/composite/with working YAML