Description
Competency Questions (CQs) are a single artifact that serves both directions of ontology work: they scope the ontology at design time (requirements) AND validate it at eval time (test suite). This is the classic Grüninger & Fox (1995) framing — the questions you write to define an ontology are the same questions you use to verify it.
CB today does not use CQs. evals.py generates eval questions from four sources — seed, doc, persona, work-item — tagged with layer hints. That's source-driven coverage testing, related to but distinct from ontology-competency-driven testing.
Opportunity: make CQs a first-class per-block artifact used for both (a) scoping/generating the ontology and (b) measuring coverage — where coverage = % of the block's competency questions the agent can answer. One artifact = design tool + eval suite + the concrete definition of the "context coverage" thesis.
Background / references (so the lineage isn't lost)
- Grüninger & Fox (1995) — origin of competency questions; CQs as both requirements and tests.
- METHONTOLOGY, On-To-Knowledge, NeOn methodology — established ontology engineering methodologies that formalize CQ-driven design.
- Ontology learning (Maedche & Staab) — automatic ontology induction from text; the technique behind "AI generates the ontology when the user can't define one."
- Competency-question-driven ontology learning — the hybrid:
seed → competency questions → ontology (more rigorous + publishable than seed → ontology).
- CQs as ontology unit tests — using the CQ set as a regression/coverage test.
- LLM-generated competency questions — emerging (2023-24); VERIFY current state before citing.
Acceptance Criteria
Out of Scope
- Implementation of AI ontology generation itself (separate concern)
- The cache/router retrieval mechanics (separate concern)
- Choosing/citing specific recent LLM-CQ papers (research pass, do later)
Description
Competency Questions (CQs) are a single artifact that serves both directions of ontology work: they scope the ontology at design time (requirements) AND validate it at eval time (test suite). This is the classic Grüninger & Fox (1995) framing — the questions you write to define an ontology are the same questions you use to verify it.
CB today does not use CQs.
evals.pygenerates eval questions from four sources —seed,doc,persona,work-item— tagged with layer hints. That's source-driven coverage testing, related to but distinct from ontology-competency-driven testing.Opportunity: make CQs a first-class per-block artifact used for both (a) scoping/generating the ontology and (b) measuring coverage — where coverage = % of the block's competency questions the agent can answer. One artifact = design tool + eval suite + the concrete definition of the "context coverage" thesis.
Background / references (so the lineage isn't lost)
seed → competency questions → ontology(more rigorous + publishable thanseed → ontology).Acceptance Criteria
competency-questions.yaml)evals.pyOut of Scope