Skip to content

feat(scorers): create LLM scorers and classifiers - #307

Merged
Cedric / ViaDézo1er (viadezo1er) merged 8 commits into
mainfrom
cedric/scorers-create
Aug 19, 2026
Merged

feat(scorers): create LLM scorers and classifiers#307
Cedric / ViaDézo1er (viadezo1er) merged 8 commits into
mainfrom
cedric/scorers-create

Conversation

@viadezo1er

@viadezo1er Cedric / ViaDézo1er (viadezo1er) commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Add bt scorers create for prompt-based LLM scorers and classifiers.

Parameter validation is a follow-up partly done in #308, and partly done in https://github.com/braintrustdata/braintrust/pull/18155 (and bt will need tweaking once the braintrust pr is merged).
At the moment it is possible to pass incorrect parameters to the scorers, and sometimes the web ui will even show them:

bt scorers create "Demo answer correctness" \
  --slug demo-answer-correctness \
  --description "Synthetic scorer used to demonstrate bt scorers create" \
  --model gpt-5.4-nano \
  --messages @/tmp/bt-scorer-messages.json \
  --choice-scores '{"PASS":1,"FAIL":0}' \
  --use-cot=false \
  --pass-threshold 0.5 \
  --metadata 'purpose: synthetic-demo' \
  --if-exists replace --temperature 1000 --reasoning-effort none
Screenshot 2026-08-14 at 16 16 22

bt-scorer-messages.json:

[
  {
    "role": "system",
    "content": "You are a strict evaluator. Compare the candidate answer with the expected answer. Return exactly PASS or FAIL."
  },
  {
    "role": "user",
    "content": "Question: {{input.question}}\nExpected answer: {{expected}}\nCandidate answer: {{output}}"
  }
]
❯ bt scorers create "Demo answer correctness" \
  --slug demo-answer-correctness \
  --description "Synthetic scorer used to demonstrate bt scorers create" \
  --model gpt-5.4-nano \
  --messages @/tmp/bt-scorer-messages.json \
  --choice-scores '{"PASS":1,"FAIL":0}' \
  --use-cot=false \
  --pass-threshold 0.5 \
  --metadata 'purpose: synthetic-demo' \
  --if-exists replace --json|jq
{
  "found_existing": true,
  "id": "6681c294-0ab3-489b-b979-33351add071d",
  "ignored": false,
  "project_id": "b667bd4c-7e55-49e0-a370-f6526e6bbccd",
  "slug": "demo-answer-correctness",
  "version": "1000197688638740786"
}

@github-actions

github-actions Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Latest downloadable build artifacts for this PR commit 7fed7130c346:

Available artifact names
  • artifacts-build-global
  • artifacts-build-local-aarch64-pc-windows-msvc
  • artifacts-build-local-x86_64-pc-windows-msvc
  • artifacts-build-local-x86_64-apple-darwin
  • artifacts-build-local-x86_64-unknown-linux-musl
  • artifacts-build-local-x86_64-unknown-linux-gnu
  • artifacts-build-local-aarch64-apple-darwin
  • artifacts-build-local-aarch64-unknown-linux-gnu
  • artifacts-plan-dist-manifest
  • cargo-dist-cache

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9963a33d87

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "Codex (@codex) review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "Codex (@codex) address that feedback".

Comment thread src/functions/create.rs

@Qard Stephen Belanger (Qard) left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still overall LGTM, though CI still seems to be failing, and looks like it needs a rebase.

@viadezo1er
Cedric / ViaDézo1er (viadezo1er) force-pushed the cedric/scorers-create branch 2 times, most recently from a8f8b84 to e737d2b Compare August 18, 2026 20:57
@viadezo1er

Copy link
Copy Markdown
Contributor Author

Rebase done

@viadezo1er
Cedric / ViaDézo1er (viadezo1er) merged commit 45b92d3 into main Aug 19, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants