Skip to content

feat: unify competency questions across ontology design and coverage evaluation #73

Description

@rajnavakoti

Description

Competency Questions (CQs) are a single artifact that serves both directions of ontology work: they scope the ontology at design time (requirements) AND validate it at eval time (test suite). This is the classic Grüninger & Fox (1995) framing — the questions you write to define an ontology are the same questions you use to verify it.

CB today does not use CQs. evals.py generates eval questions from four sourcesseed, doc, persona, work-item — tagged with layer hints. That's source-driven coverage testing, related to but distinct from ontology-competency-driven testing.

Opportunity: make CQs a first-class per-block artifact used for both (a) scoping/generating the ontology and (b) measuring coverage — where coverage = % of the block's competency questions the agent can answer. One artifact = design tool + eval suite + the concrete definition of the "context coverage" thesis.

Background / references (so the lineage isn't lost)

  • Grüninger & Fox (1995) — origin of competency questions; CQs as both requirements and tests.
  • METHONTOLOGY, On-To-Knowledge, NeOn methodology — established ontology engineering methodologies that formalize CQ-driven design.
  • Ontology learning (Maedche & Staab) — automatic ontology induction from text; the technique behind "AI generates the ontology when the user can't define one."
  • Competency-question-driven ontology learning — the hybrid: seed → competency questions → ontology (more rigorous + publishable than seed → ontology).
  • CQs as ontology unit tests — using the CQ set as a regression/coverage test.
  • LLM-generated competency questions — emerging (2023-24); VERIFY current state before citing.

Acceptance Criteria

  • Decide whether CQs become a first-class per-block artifact (e.g. competency-questions.yaml)
  • Spec CQ authoring: human-defined AND/OR AI-generated from seed context (competency-question-driven ontology learning)
  • Spec how the CQ set feeds ontology scoping at design time
  • Spec how the CQ set feeds coverage eval (coverage = % answerable), reconciled with the current 4-source eval in evals.py
  • Document the design↔eval unification in the architecture docs

Out of Scope

  • Implementation of AI ontology generation itself (separate concern)
  • The cache/router retrieval mechanics (separate concern)
  • Choosing/citing specific recent LLM-CQ papers (research pass, do later)

Metadata

Metadata

Assignees

No one assigned

    Labels

    frameworkScientific frameworks

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions