feat(experiments): add draft launchdarkly-experiment-hypothesis-builder skill

[mabramowitz-pixel]

Summary

Draft skill for AI-assisted experiment hypothesis building, now in its third iteration (v0.3.0).

v0.3.0 changes — Structural alignment to finalized designs and detection-model improvements:

• One input, one button: removed the separate describe box and Continue/Cancel/Close model — single hypothesis field with a Generate button
• Three-state rubric row: Change · Measurement · Rationale tracked as gray dash (missing), green check (present), or amber warning triangle (needs action)
• Detection as independent atoms: three component booleans plus non-exclusive flags (is_junk / is_aa / measurement_count / rewrite_worthy), with routing as a separate UI-presentation step
• Multiple measurements surfaced in the then hole joined with "or" and an amber Measurement warning (replaces the up-arrow primary-switch pills)
• Layered critique cards for multi-missing states, stacked in rubric order
• Character count section (1000-char limit; hidden < 800, amber 800–999, red 1000+; Generate stays active)
• Measurement counting rule: near-synonyms = one; distinct outcomes = multiple
• Handoff payload: replaced single route field with components + measurementCount + non-exclusive flags
• Toggle renamed "Vega assist"
• Compatibility updated: read-only lookups only, hands off to launchdarkly-experiment-setup for build

Previous work

• v0.2.0: Simplified 9-step workflow into focused scaffold/critique model with hole rules, fixed critique messages, A/A handling, and slimmer handoff payload
• v0.1.0: Original skill with 9-step workflow, diagnostic tree, and structured handoff payload

(Same-repo replacement for #94, which was opened from a fork and couldn't run Skill Evals.)

Test plan

• python3 scripts/validate_skills.py passes
• python3 scripts/generate_catalog.py regenerates skills.json with updated entry
• Manual testing against sample hypotheses (weak, strong, A/A, junk input)
• Eval suite in evals/launchdarkly-experiment-hypothesis-builder/

🤖 Generated with Claude Code

---

Note

Overview
Adds a draft advisory skill (launchdarkly-experiment-hypothesis-builder, v0.3.0) that coaches rough experiment ideas into an If/then/because hypothesis: component detection, routing (scaffold / rewrite / junk / aa), scaffold holes, fixed critique copy, and a headless JSON contract for the experiment builder. It is read-only toward LaunchDarkly (no writes) and hands off to launchdarkly-experiment-setup.

Catalog & docs: README Experiments table, skills.json, and a references/diagnostic-tree.md flaw-to-config reference.

Eval harness: New promptfoo suite with prompt.js (nunjucks {% raw %} around SKILL.md), binary contract asserts (route, components, holes, measurements, security no-echo), npm scripts, manifest entry, and metric list mocks plus list-metrics / list-metric-events tool defs.

Provider: claude-skill-agent-sdk gains optional mcp_tool_allowlist so advisory evals can expose only read-only MCP tools; the hypothesis-builder eval disables MCP tools entirely and forces skill invocation.

^{Reviewed by Cursor Bugbot for commit 0e81d12. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

Skill eval results

Skill	Before	After	Δ
agentcontrol/configs-create	100/100 (4/4)	75/100 (3/4)	-25
agentcontrol/configs-update	80/100 (4/5)	80/100 (4/5)	no change
agentcontrol/configs-variations	80/100 (4/5)	80/100 (4/5)	no change
agentcontrol/tools	75/100 (3/4)	75/100 (3/4)	no change
experiments/launchdarkly-experiment-hypothesis-builder	-	60/100 (3/5)	new
feature-flags/launchdarkly-flag-command	-	100/100 (3/3)	new
feature-flags/launchdarkly-flag-create	100/100 (3/3)	100/100 (3/3)	no change

Only suites whose source actually changed since their last recorded score were re-run. Soft-failing while we stabilise the baseline.

[sruthykumar]

From the eval on #96 — here's a prompt you can paste into Claude (run from the repo root) to fix the three findings, then re-run the eval to confirm it goes green.

You're editing the LaunchDarkly agent skill at
skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md.

An eval run found three cases where the skill doesn't follow its own stated
rules. Please update the SKILL.md to fix all three, keeping the existing
structure and voice and making the changes minimal:

1. NEVER write to LaunchDarkly. In practice the skill calls create-feature-flag
   and update-feature-flag, even though it's meant to be advisory and hand off
   to launchdarkly-experiment-setup. Make this a hard, prominent rule: the skill
   must never call any create-, update-, toggle-, or start- tool itself. Its
   final action is always to emit the handoff payload and then STOP — even if
   launchdarkly-experiment-setup is not available to receive it, output the
   handoff payload and stop rather than creating the flag/metric/experiment
   yourself. Add this near the top and reinforce it in "What NOT to do."

2. Gate non-real input up front. The skill has a "Detecting low-effort /
   non-real input" section but doesn't act on it — it built a full experiment
   for an "A/A test to validate bucketing" input. Move this check to the very
   start of the workflow (before capturing or diagnosing the hypothesis) and
   make it a hard gate: if the input is a platform self-test, an A/A bucketing
   check, a placeholder, or gibberish, confirm intent with the user and do NOT
   build a hypothesis or any configuration until they confirm it's a real
   experiment.

3. Enforce metric-outcome alignment. The skill accepted "measure it by revenue"
   for a hypothesis that predicted engagement would move. Add an explicit step
   to the main workflow (this is flaw F7 in references/diagnostic-tree.md):
   verify the primary metric measures the outcome the hypothesis actually
   predicts. If the predicted outcome and the proposed metric don't match, flag
   it and reconcile with the user before composing the hypothesis.

After editing, re-run the eval to confirm:
  cd evals && npm run eval:hypothesis-builder

The three findings map to failing assertions in the suite (no_writes across all cases, mismatch_caught, nonreal_handled). One caveat: the eval runs the skill in isolation without launchdarkly-experiment-setup loaded, so it has nowhere to hand off — which may be nudging the direct writes. Fix #1 handles that either way (emit handoff and stop).

via LD Research 🤖

[mabramowitz-pixel]

Eval outcome + known residual (A/A gating)

The eval suite (#96) is green against these skill fixes. Summary of the three behaviors the evals check:

Fix	Status	Evidence
Never write to LaunchDarkly	✅ solved	0 write-tool calls across all 5 cases. Guaranteed by scoping the suite to read-only tools (mcp_tool_allowlist) — an advisory skill shouldn't be handed create-*/update-*/toggle-* in the first place.
Metric ↔ outcome alignment (F7)	✅ solved	The engagement-predicted / revenue-measured case now scores 1.00 — the skill names the conflict back to the user and asks which to change instead of silently substituting a metric.
Gate non-real input	⚠️ partial	Suite passes on aggregate, but the A/A case scores 0.67, not 1.0.

Known residual — A/A gating (model adherence). For the input "A/A test to validate bucketing," the agent (Sonnet 4.6) recognizes it's an A/A yet still drafts an "A/A validation hypothesis" rather than stopping to confirm intent. Step 0 was strengthened twice to forbid exactly this (an A/A has identical variants ⇒ no hypothesis; reply must be a single clarifying question), and the agent still builds the package. Behavior is also noisy run-to-run. This is a model-adherence limitation on one input, not a wording gap — the suite clears the 75% gate because the other four cases are clean and nothing writes. Filing as a known limitation rather than over-tuning the prose further.

[cchung-at-LD]

I've made changes to the skill file according to the designs.

[hsadhvani]

Don't think it needs access to any mcp tools right now

[hsadhvani]

approving to unblock

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 0e81d12. Configure here.}

[cursor]

Contradictory multi-measurement rules

High Severity

Foundational rule 4 tells the agent to keep one primary measurement in the sentence, while the Multiple measurements section and the "What NOT to do" list tell it to put every measurement in the then hole joined with "or". The headless schema also says holes are only for missing slots, yet this path treats measurement as present. The eval suite already leaves holes unasserted because of this conflict, so multi-measurement outputs stay undefined.

Additional Locations (2)

• skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md#L144-L145
• skills/experiments/launchdarkly-experiment-hypothesis-builder/SKILL.md#L196-L197

 

^{Reviewed by Cursor Bugbot for commit 0e81d12. Configure here.}

[cursor]

Ambiguous handoff count value

Medium Severity

In the handoff JSON example, measurementCount is shown as the literal string "0 | 1 | multiple", with no separate prose listing allowed values. Models often copy example payloads verbatim, so the handoff can emit that pipe-separated string instead of one of 0, 1, or multiple, breaking downstream display-state composition.

 

^{Reviewed by Cursor Bugbot for commit 0e81d12. Configure here.}