feat(onboarding): Patchy developer persona quiz step#6143
Draft
idoshamun wants to merge 15 commits into
Draft
Conversation
Bake the new 15-persona, 25-question quiz model into the Patchy quiz. Data (data.ts): - 15 personas, role-named (no "The" prefix), spanning mainstream stacks and the data-grounded niche personas the original 12 missed: Generalist · Full-Stack Web · Frontend Specialist · AI App Builder · AI Specialist · Engineering Leader · Backend · Architect · Systems Programmer · DevOps · PHP · Security · .NET · Game · Mobile. - 25 questions, including five new locks (Q15 AI is your work, Q17 PHP, Q18 .NET, Q24 games, Q25 security) and Q11/Q12 retuned for the modern AI dev tooling era. - Likelihood matrix derived from 93,539 active daily.dev users over 90 days. Mobile cluster hand-crafted from mobile-topic-heavy users. - Prior switched to log-shaped distribution so niche personas can compete with the much larger Generalist cluster without sacrificing the headline 75% top-1 accuracy. - New triplebreakFloor threshold (0.3) and renamed generalist fallback to match the new persona id. Engine / hook (usePersonaQuiz.ts): - New 'triplebreak' phase for the I'm-torn-between-three case (Security and Architect win it most often; both have weak top-2 signal because their content overlaps with Backend/Infra). - finish() chooses reveal / triplebreak / tiebreak / fallback by walking the same thresholds in decreasing confidence order. - Tiebreak buffer now stores up to three indices and the memoised selectors slice them down based on the current phase. UI (FunnelPersonaQuiz.tsx): - Render the new triplebreak phase with a three-column grid of persona cards and an "None of these. Let me pick." escape hatch that drops back into the manual picker. - Extracted a small PersonaCard component so the tiebreak and triplebreak screens stay visually consistent without duplicating markup. Headline metrics from the offline simulator on the v5 model: Top-1 75% · Top-2 88% · Top-3 92% · feed quality 96% · all 15 personas reachable via yes/not-sure/no answer paths · median 6 questions to lock.
User-reported issues from internal playtest:
1. Q2 'Your code runs on servers, not in a browser' was confusing
for mobile and game devs — their code does not run on servers
but is also not in a browser, leaving them without a clean
answer.
2. Q22 'You could tell who Stripe, OpenAI or Anthropic hired last
week' read as random trivia and not a real signal.
3. Answering yes to 'Security is your primary job' still landed
on Generalist or Backend. Same for 'You build games' — the
quiz kept asking unrelated AI questions before revealing.
Fixes:
Question rewording (data.ts):
- Q2: 'Your work is mostly backend or infrastructure, not
frontend or mobile.' Mobile and game devs now have a clean
no instead of a forced compromise.
- Q22: 'You actively follow tech industry news (deals, hiring,
leadership).' Same intent, less trivia-flavoured.
Hard-lock questions (data + engine):
- PersonaQuestion gained an optional lockPersonaId. When the
user answers yes to a lock question, the engine reveals that
persona immediately, regardless of accumulated Bayesian belief.
Six questions now lock:
Q6 iPhone or Android -> Mobile Developer
Q15 AI is what you build -> AI Specialist
Q17 main language is PHP -> PHP Developer
Q18 main stack is .NET -> .NET Developer
Q24 you build games -> Game Developer
Q25 security is your job -> Security Engineer
These are self-identification questions: if the user agrees,
there is no signal stronger than self-report. Trust it over
the matrix.
Instant-lock fallback (engine):
- New instantLockThreshold (0.85) and instantLockMargin (0.5)
in PERSONA_ENGINE_CONFIG. When belief becomes overwhelming
(one persona above 0.85 and at least 0.5 ahead of the runner
up), the quiz now reveals immediately without waiting for
minQuestions. Prevents the 'I told you I build games, why
are you still asking?' problem when the hard lock did not
trigger but the persona is otherwise clearly identified.
Matrix and prior regenerated (data.ts):
- Tightened the behavioural proxies behind Q17, Q18, Q24 and
Q25 to require a topic-share signal (>= 0.25-0.30) instead
of also accepting raw tag counts. This collapses the
Generalist false-positive rate on those columns from 12-15%
down to roughly 0%, so the matrix backs the hard locks
instead of fighting them. Lock-question precision stays
high (PHP 88%, .NET 83%, Games 76%, Security 68% true-
positive on the proxy).
User feedback from playtest: the quiz would ask 'Your main language is TypeScript or JavaScript', then a few questions later ask 'Your main language is Python' (or Go/Rust/PHP/.NET). Saying yes to one should rule out the others — they describe the same single attribute. Adds an exclusiveGroup label to PersonaQuestion. Every 'main language' question now belongs to the same 'main-language' group: Q8 TypeScript / JavaScript Q9 Python Q10 Go / Rust / C / C++ Q17 PHP Q18 C# / .NET The hook keeps an excludedGroupsRef set. When a question with an exclusiveGroup is answered yes, the group is closed out for the rest of the session, and pickNextQuestion in engine.ts skips any question that belongs to a closed group. PHP and .NET still lock immediately on yes via lockPersonaId, so this mostly affects the JS/Python/Go trio — those three no longer get asked in sequence once one is confirmed. restart() and start() also reset the group set so a replay does not inherit the previous run's exclusions.
Q1 ("You ship the things users see and click on") and Q2 ("Your work
is mostly backend or infrastructure, not frontend or mobile") sit on
opposite halves of the same axis. The explicit "not frontend or
mobile" in Q2 makes them mutually exclusive: a yes on either implies
no on the other.
Engine info-gain usually skips the second one after the first is
answered yes, but if belief stays diffuse the user could still see
both, which reads as contradictory. Tagging both with the same
exclusiveGroup ('primary-domain') guarantees only one is ever asked.
Audited the rest of the bank — every other near-conflict either:
- already short-circuits via lockPersonaId (mobile, AI, PHP, .NET,
game, security),
- is a hierarchy not an exclusion (Q11/Q15, Q12/Q16), or
- is genuinely compatible (Q4/Q5, Q24/Q6, etc).
Old wording 'AI tools are critical to your daily work, not just autocomplete' was vague: 'critical' is subjective and the negation read as a parenthetical afterthought. New wording names the actual tools and frames the threshold as active use vs passive autocomplete: 'You actively use AI tools (Cursor, Claude, Copilot) to do real work, not just for autocomplete.' This separates 'Copilot pushes me suggestions and I accept them' (everyone) from 'I direct AI tools to do meaningful work' (AI App Builder, AI Specialist, indie hackers).
Previous wording named Cursor, Claude, and Copilot together, but Copilot is fundamentally an autocomplete product — listing it next to agentic tools made the 'not just autocomplete' caveat read as a contradiction (everyone who uses Copilot would honestly say yes). The signal we actually want is whether the user lets AI do meaningful chunks of their work, not whether they happen to use a specific tool. Reframed around scope: 'You let AI write whole functions or features for you, not just autocomplete suggestions.' This separates 'AI completes the line I was going to write' from 'I give AI a task and use what it produces', which is the actual behavioural divide between casual Copilot users and AI App Builders / AI Specialists.
Previous wording ('You let AI write whole functions or features
for you, not just autocomplete suggestions') is no longer a real
discriminator: in 2026 essentially every developer lets AI write
functions for them via Cursor, Claude chat, or ChatGPT.
The signal we actually want to isolate is the leap from chat/
autocomplete to agentic AI work — letting an AI agent run for
minutes on a multi-step task and reviewing the result. That is
the behavioural divide that separates AI App Builders and AI
Specialists from the broad population that 'uses AI for code'.
New wording:
'You use agentic AI tools (Claude Code, Cursor Agent) that run
on tasks autonomously, not just chat or autocomplete.'
…tegist and modifier checkbox screen Persona model overhaul based on playtest feedback. AI tooling kept turning out to be a dimension layered on top of an underlying tech persona rather than a persona itself — everyone uses AI now, so it does not separate developers any better than 'has a code editor' would. Same goes for the Engineering Leader signal. Changes: Persona model (15 -> 14): - DROP AI App Builder: it was really 'Full-Stack Web Dev + heavy AI'. Those users now land in Full-Stack Web Dev with the AI Heavy modifier set instead. - DROP Engineering Leader as a primary persona: leading is a layer on top of an engineering identity, not the identity itself. Now surfaced via the Engineering Leader modifier instead. - ADD Tech Strategist: covers product managers, designers, tech execs and other tech-adjacent roles that don't write code as their day-to-day job. Detected via a hard-lock question and the Tech Strategist persona id. Quiz changes (25 -> 19 questions): - DROP Q3 'read more than code these days' (was Eng Leader signal) - DROP Q11 'agentic AI' (dimension) - DROP Q12 'shipped LLM API code' (dimension) - DROP Q13 'dotfiles weekend' (orphaned) - DROP Q14 'fine-tuned ML model' (dimension) - DROP Q16 'web apps with AI built in' (AI App Builder discriminator) - DROP Q22 'follow industry news' (Eng Leader signal) - ADD Q3 new 'You don't write code as part of your day-to-day job.' with lockPersonaId 'tech-strategist'. Modifiers (new): - New PersonaModifier type and MODIFIERS export with three entries: ai-heavy, indie-hacker, engineering-leader. Each has an id, label, emoji and description. - Modifiers are chosen by the user on a single checkbox screen after the persona is determined and before the final reveal. The selection ships out via onTransition details under a new modifiers field (string[]). Hook + UI: - usePersonaQuiz gains 'modifiers' phase, selectedModifierIds and toggleModifier, confirmModifiers. The phase flow is now playing -> tiebreak/triplebreak/picker -> modifiers -> reveal. - FunnelPersonaQuiz renders the modifiers checkbox screen with the Patchy mascot + three opt-in cards. Selected modifiers are shown as small chips on the reveal screen so the user can see what got applied. - The demo page surfaces the modifiers list in the completion card alongside persona / confidence / questions. Matrix and prior regenerated: - Drop AI App Builder and Engineering Leader rows from the likelihood matrix. - Drop the seven dropped question columns. - Insert a hand-crafted Tech Strategist row (Q21 'boxes and arrows' high since PMs/designers diagram a lot; everything else low). - Insert a hand-crafted column for the new Tech Strategist lock question (95% YES for Tech Strategist, <=8% for everyone else). - Redistribute prior mass: AI App Builder -> Full-Stack, Engineering Leader -> Software Architect, Tech Strategist seeded at ~2.5%. - Renormalised so prior sums to 1.
The v7 commit dropped two personas and seven questions but kept the existing matrix rows for the surviving personas. The Tech Strategist row and the new lock column were hand-crafted. That left the prior and the dropped users redistribution as a guess rather than data. This commit re-runs the full clustering pipeline against 93,345 active daily.dev users (90d engagement) using the v7 persona model and the trimmed 19-question set. The thirteen engineer-persona rows are recomputed from real cluster membership. Tech Strategist stays hand-crafted since non-engineers don't appear in the engagement clustering. Key calibration outcomes from the simulator on the recalibrated matrix: Top-1 65% / Top-2 85% / Top-3 93% / median 6 questions / 58% lock within 6 questions. Per-persona top-3 (the relevant metric when triple-tie UX is available): >95% for Generalist, Full-Stack, Frontend, Architect; 85-90% for Backend, Systems, DevOps, PHP, Mobile, .NET; 60-85% for AI Specialist, Game, Security. Tech Strategist guaranteed via its lock question (Q3) regardless of behaviour. Two structural notes: - Security Engineer cluster didn't surface as its own K-means centroid in this run (cluster shapes shifted slightly versus the v6 pull). Hand-crafted via topic-share filter (>=20% security engagement) to recover ~3,400 users into the cluster. - Engineering Leader cluster (high culture_career) is folded into Software Architect now that Engineering Leader is a modifier rather than a persona. The likelihood matrix and prior in data.ts are the canonical output of this run. Pipeline script in brain repo to be updated in a follow-up so it produces this exact matrix on next refresh.
Q4 'main output is a web app in a browser', Q6 'you build apps for iPhone or Android' and Q7 'your day involves Jupyter notebooks, datasets, or training runs' all describe the same axis: the user's primary build platform. A yes on one implies no on the others — your main output can be web OR mobile OR notebooks, not several. Putting all three in a 'primary-platform' exclusiveGroup. Once any one is answered yes, the engine stops asking the other two. Same mechanism we use for main-language (Q8/Q9/Q10/Q12/Q13) and primary-domain (Q1/Q2). Q6 (mobile) is also a lock, so the lock already short-circuits any follow-ups when it fires; the group label is the defensive belt for the case where Q4 or Q7 is answered first.
…rsona lock, answer debug log
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
|
Two fixes from a playtest where the user answered backend on Q2 and
then was asked Q4 ('main output is a web app') anyway, and ended up
classified as DevOps Engineer despite a clear senior-backend
answer pattern.
1. Cross-group implications
The exclusiveGroup mechanism only closes questions within the same
group. But Q1 ('you ship UI') and Q2 ('you're backend, not frontend
or mobile') are not just opposites of each other; they also imply
answers across the primary-platform group. A yes on Q2 rules out Q4
(web app) and Q6 (mobile app), but the engine had no way to express
that, so it asked Q4 later anyway when info gain was middling.
Added two optional fields to PersonaQuestion:
closesOnYes?: string[] // groups to close when this answer is yes
closesOnNo?: string[] // groups to close when this answer is no
Configured:
Q1 closesOnNo: ['primary-platform'] — not UI => no UI platform
Q2 closesOnYes: ['primary-platform'] — backend => no UI platform
usePersonaQuiz honors both when applying an answer.
2. DevOps lock too broad
Q14 was 'You've been the one paged at 3am when production went
down'. Every senior backend dev with on-call rotation answers yes,
which is why the playtest user (backend with Go/Rust and SQL and
diagrams) landed on DevOps Engineer instead of Backend Developer:
the DevOps row has 94 percent yes on Q14 vs 4 percent for Backend,
so yes on Q14 swamps the rest.
Reworded to identify DevOps as the role, not the duty:
'DevOps, SRE, or platform engineering is your job, not just a
side responsibility.'
A backend dev who is on rotation will answer no; a real DevOps,
SRE, or platform engineer will answer yes.
…uestion
Three fixes for a playtest where the user answered backend (Q2 yes),
got asked three of the five language questions (JS/TS, Go/Rust, .NET),
none of which was their language, then ended up classified as Backend
Developer without ever being asked about Python, PHP, or Java.
1. Active-group preference in pickNextQuestion
Until now the engine picked questions purely by expected information
gain. Inside the main-language group, once Q1 = no (not UI) was
answered, PHP belief collapsed to near zero (because PHP devs in the
data overwhelmingly answer Q1 = yes). PHP's expected info gain
followed, even though its conditional info gain on a yes is huge
(it's a lock). The engine reasonably decided PHP couldn't be the
answer and moved on, never asking the question.
The new rule: once any question in an open exclusiveGroup has been
asked, prefer the remaining group members before moving on. The
language group is treated as a single decision the user gets to
finish, not a single question the engine can drop after one no.
Same applies to primary-platform and primary-domain.
2. DevOps question is now a hard lock
'DevOps, SRE, or platform engineering is your job, not just a side
responsibility' is a self-identification question. Yes on it is
direct evidence the user IS DevOps, the same shape as Q12 (PHP),
Q13 (.NET), Q18 (game), Q19 (security). Added lockPersonaId:
'devops-engineer' so a yes locks the persona instead of relying on
the Bayesian update.
3. Added Java / Kotlin question
Java / Kotlin / JVM doesn't have its own persona (the JVM enterprise
cluster is too small to surface at K=16), so we never asked about
it. Users coming from Java land on Backend Developer or Software
Architect by inference. Added a new entry in the main-language
group ('Your main language is Java or Kotlin.') with no specific
lock; yes closes the language group and feeds the Bayesian
classifier with the right signal. Hand-crafted column in the
likelihood matrix: high yes rate for Backend Developer (35%),
Software Architect (30%) and Mobile Developer (30%, for Android
Kotlin), low everywhere else.
Bumped maxQuestions from 10 to 12 to make room for the longer
language pass. Median session length should still sit around 6-7
once a lock fires; only sessions that say no to every lock will
approach the new cap.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Akinator-style "developer persona" onboarding step (Patchy) that replaces the tag selection step in the flexible onboarding funnel, plus a demo page for iteration.
What's in it
FunnelStepType.PersonaQuizwired into the funnel types, step registry, andFunnelStepper.steps/persona/(personas, questions, likelihood matrix, prior, config) kept fully separate from logic, with fail-fast validation of matrix dimensions./onboarding-persona-demo(noindex) renders the step standalone and echoes theonTransitionpayload.Notes
{ persona, modifiers, confidence, questions, manual }.Demo: https://app.local.fylla.dev:5002/onboarding-persona-demo
Preview domain
https://feat-onboarding-persona-quiz.preview.app.daily.dev