Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .devcontainer/devcontainer-lock.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{
"features": {
"ghcr.io/anthropics/devcontainer-features/claude-code:1.0": {
"version": "1.0.5",
"resolved": "ghcr.io/anthropics/devcontainer-features/claude-code@sha256:cfc2e7d3e9fd3b9b01f8d5cb158508a884c8c0ede2e23ed10f32dea5d4ffe69a",
"integrity": "sha256:cfc2e7d3e9fd3b9b01f8d5cb158508a884c8c0ede2e23ed10f32dea5d4ffe69a"
},
"ghcr.io/devcontainers/features/github-cli:1": {
"version": "1.1.0",
"resolved": "ghcr.io/devcontainers/features/github-cli@sha256:d22f50b70ed75339b4eed1ba9ecde3a1791f90e88d37936517e3bace0bbad671",
"integrity": "sha256:d22f50b70ed75339b4eed1ba9ecde3a1791f90e88d37936517e3bace0bbad671"
},
"ghcr.io/julialang/devcontainer-features/julia:1": {
"version": "1.2.0",
"resolved": "ghcr.io/julialang/devcontainer-features/julia@sha256:748ff7272e392e0c2ac7c837dbdc6c8cfa1dd3f81819fd29a111c718e3c4199d",
"integrity": "sha256:748ff7272e392e0c2ac7c837dbdc6c8cfa1dd3f81819fd29a111c718e3c4199d"
}
}
}
23 changes: 23 additions & 0 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
{
"name": "ComputerAdaptiveTesting.jl",
"image": "mcr.microsoft.com/devcontainers/base:ubuntu-24.04",
"features": {
"ghcr.io/julialang/devcontainer-features/julia:1": {
"channel": "1.12"
},
"ghcr.io/devcontainers/features/github-cli:1": {},
"ghcr.io/anthropics/devcontainer-features/claude-code:1.0": {}
},
"mounts": [
"source=claude-code-config-${devcontainerId},target=/home/vscode/.claude,type=volume"
],
"postCreateCommand": "sudo chown vscode:vscode /home/vscode/.claude && julia --project=. -e 'using Pkg; Pkg.instantiate(); Pkg.precompile()'",
"remoteEnv": {
"JULIA_NUM_THREADS": "auto"
},
"customizations": {
"vscode": {
"extensions": ["julialang.language-julia"]
}
}
}
57 changes: 57 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# AGENTS.md

ComputerAdaptiveTesting.jl implements Computer Adaptive Testing (CAT) in
Julia: composable building blocks for item selection, ability estimation,
termination rules, and simulation. See the [README](README.md) for motivation
and installation, `docs/src/` for tutorials, and docstrings/block comments in
`src/` for specifics — keep those up to date as you change code; they are the
primary documentation for humans too.

## Layout

- `src/ComputerAdaptiveTesting.jl` includes each submodule: `Aggregators`
(ability estimation), `NextItemRules` (item selection), `TerminationConditions`,
`Sim` (simulation loop), `Stateful` (step-wise interface), `DecisionTree`,
`Compat`/`Comparison` (R catR/mirtCAT shims and cross-checks).
- Workspaces: `test/` and `docs/` have their own `Project.toml` (Julia 1.12
workspace feature).

## Style — DRY and reuse first

Follow the patterns already in the codebase rather than inventing new ones:

- **Config composition**: user-facing constructors take a varargs "bag of
config bits" resolved with `@requiresome`/`@returnsome`/`find1_instance`
from `PsychometricsBazaarBase.ConfigTools` (see
`src/Aggregators/ability_estimator.jl`, `src/NextItemRules/`). Extend this
machinery; don't hand-roll keyword plumbing.
- **Dispatch over branching**: use trait/marker types (`DomainType`,
`IntValue`) and multiple dispatch instead of `if`/`isa` chains.
- **Composable wrappers**: prefer wrapper types (e.g.
`GuardedAbilityEstimator`) over conditionals embedded at call sites.
Configs are functors — callable strategy objects rooted at `CatConfigBase`.
- **Sister packages**: item banks come from FittedItemBanks.jl; integrators,
optimizers, and config machinery from PsychometricsBazaarBase.jl. Never
duplicate their functionality here.
- **Docstrings**: use DocStringExtensions (`$(TYPEDEF)`, `$(SIGNATURES)`);
see `src/NextItemRules/prelude/abstract.jl` for house style. New public API
must appear in `docs/src/api.md`.
Comment on lines +36 to +38

## Build, test, docs

- Julia 1.12 (`Project.toml` compat; CI tests only 1.12).
- Test: `julia --project=test test/runtests.jl` (includes Aqua and JET
checks). `Pkg.test()` currently does not work — see issue #52.
- Docs: `cd docs && ./build.sh` (needs R via CondaPkg; CI uses
`JULIA_CONDAPKG_BACKEND=System` with conda).

## Workflow

Work in a git worktree per task, never directly on `main`:

```sh
git worktree add ../ComputerAdaptiveTesting.jl-<task> -b <task>
```

Commit there, push with `git push -u origin <task>`, and open a pull request
with `gh pr create`. Clean up with `git worktree remove` after merge.
13 changes: 11 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,15 @@ also contains a number of examples.

## Repo organisation

* `/src`, Source code for the main ComputerAdaptiveTesting.jl package. The
main submodules are `Aggregators` (ability estimation), `NextItemRules`
(item selection), `TerminationConditions`, `Sim` (simulation loop),
`Stateful` (step-wise interface for interactive use), `DecisionTree`, and
`Compat`/`Comparison` (interoperability with and cross-checking against
the R packages catR and mirtCAT)
* `/docs`, Documentation source code, built with Documenter.jl via
`docs/build.sh`
* `/docs/examples`, Example code
* `/docs`, Documentation source code, build with Documenter.jl
* `/src`, Source code for the main ComputerAdaptiveTesting.jl package
* `/test`, Test suite, including Aqua.jl and JET.jl quality checks
* `/AGENTS.md`, Development conventions (style, testing, workflow) for
coding agents and human contributors alike
2 changes: 1 addition & 1 deletion docs/make.jl
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ makedocs(;
],
"Contributing" => "contributing.md",
],
warnonly = [:missing_docs],
checkdocs_ignored_modules = [ComputerAdaptiveTesting.Compat],
linkcheck = true)

if build_demos
Expand Down
2 changes: 1 addition & 1 deletion docs/src/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,5 +8,5 @@ CurrentModule = ComputerAdaptiveTesting
```

```@autodocs
Modules = [ComputerAdaptiveTesting, ComputerAdaptiveTesting.Aggregators, ComputerAdaptiveTesting.Responses, ComputerAdaptiveTesting.Sim, ComputerAdaptiveTesting.TerminationConditions, ComputerAdaptiveTesting.NextItemRules, ComputerAdaptiveTesting.Rules]
Modules = [ComputerAdaptiveTesting, ComputerAdaptiveTesting.Aggregators, ComputerAdaptiveTesting.Responses, ComputerAdaptiveTesting.Sim, ComputerAdaptiveTesting.TerminationConditions, ComputerAdaptiveTesting.NextItemRules, ComputerAdaptiveTesting.Rules, ComputerAdaptiveTesting.Stateful, ComputerAdaptiveTesting.DecisionTree, ComputerAdaptiveTesting.Compat, ComputerAdaptiveTesting.Comparison, ComputerAdaptiveTesting.ItemBanks, ComputerAdaptiveTesting.ConfigBase]
```
5 changes: 5 additions & 0 deletions docs/src/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@
Contributions are welcome. Please discuss any larger changes in the issues
before making a pull request to avoid wasted work.

Development conventions (code style, testing, git workflow) are documented in
`AGENTS.md`
at the repository root. A devcontainer is provided in `.devcontainer/` for a
ready-made development environment.

## Running tests

You can run tests locally like so:
Expand Down
8 changes: 4 additions & 4 deletions docs/src/stateful.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@
CurrentModule = ComputerAdaptiveTesting
```

```@docs
```@docs; canonical=false
Stateful.Stateful
```

## Interface

```@docs
```@docs; canonical=false
Stateful.StatefulCat
Stateful.next_item
Stateful.ranked_items
Expand All @@ -27,10 +27,10 @@ Stateful.get_ability

There is an implementation in terms of [CatRules](@ref):

```@docs
```@docs; canonical=false
Stateful.StatefulCatRules
```

## Usage

Just as [CatLoop](@ref) can wrap [CatRules](@ref), you can also use it with any implementor of [Stateful.StatefulCat](@ref), and run using [Sim.run_cat](@ref).
Just as [CatLoop](@ref) can wrap [CatRules](@ref), you can also use it with any implementor of [Stateful.StatefulCat](@ref), and run using [Sim.run_cat](@ref).
10 changes: 10 additions & 0 deletions src/Aggregators/Aggregators.jl
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ using PsychometricsBazaarBase.Integrators: Integrators,
using PsychometricsBazaarBase.Optimizers: OneDimOptimOptimizer, Optimizer, Optimizers
using PsychometricsBazaarBase.ConstDistributions: std_normal, std_mv_normal
using PsychometricsBazaarBase.IndentWrappers: indent
using DocStringExtensions
import Distributions: pdf
import Base: show

Expand Down Expand Up @@ -164,6 +165,15 @@ function AbilityOptimizer(bits...; ability_estimator = nothing)
@returnsome Optimizer(bits...) optimizer->FunctionOptimizer(optimizer)
end

"""
$(TYPEDEF)

Responses to items in `item_bank` (as [`BareResponses`](@ref)), together with
an `ability_tracker` that maintains an incrementally-updated ability estimate
(or distribution) as responses are added. This is the object threaded through
a CAT run and passed to next item rules, termination conditions and ability
estimators.
"""
@with_kw struct TrackedResponses{
BareResponsesT <: BareResponses,
ItemBankT <: AbstractItemBank,
Expand Down
63 changes: 63 additions & 0 deletions src/Aggregators/ability_estimator.jl
Original file line number Diff line number Diff line change
@@ -1,3 +1,23 @@
# Ability estimators come in two flavours:
# * `DistributionAbilityEstimator`s produce a (possibly unnormalized)
# density over ability, via `pdf(est, tracked_responses)`, e.g. the raw
# likelihood (`LikelihoodAbilityEstimator`) or a Bayesian posterior
# (`PosteriorAbilityEstimator`).
# * `PointAbilityEstimator`s reduce such a distribution to a single ability
# value when called as `est(tracked_responses)`, either by optimization
# (`ModeAbilityEstimator`, i.e. MAP/MLE) or by integration
# (`MeanAbilityEstimator`, i.e. EAP).
# `AbilityTracker`s (see ability_tracker.jl) wrap an estimator to maintain an
# incrementally-updated estimate as responses are added, avoiding
# recomputation from scratch after every response.
#
# Constructors here follow the package-wide "bag of config bits" convention:
# `XAbilityEstimator(bits...)` scans the heterogeneous `bits` arguments (which
# may include other estimators, integrators, optimizers, priors, item banks,
# etc.) via `find1_instance`/`find1_type` and assembles a suitable estimator,
# falling back to sensible defaults (e.g. a standard normal prior) when
# nothing more specific is found.

function Integrators.normdenom(integrator::AbilityIntegrator,
est::DistributionAbilityEstimator,
tracked_responses::TrackedResponses)
Expand All @@ -19,6 +39,13 @@ function pdf(ability_est::DistributionAbilityEstimator,
pdf(ability_est, tracked_responses)(x)
end

"""
$(TYPEDEF)

The ability likelihood distribution.

$(FUNCTIONNAME)()
"""
Comment thread
Copilot marked this conversation as resolved.
struct LikelihoodAbilityEstimator <: DistributionAbilityEstimator end

function pdf(::LikelihoodAbilityEstimator,
Expand All @@ -30,6 +57,17 @@ function power_summary(io::IO, ::LikelihoodAbilityEstimator)
println(io, "Ability likelihood distribution")
end

"""
$(TYPEDEF)

Ability posterior distribution: the response likelihood times a `prior`
distribution over ability (a standard normal by default).

$(FUNCTIONNAME)(; ncomp=0)

Constructs with a standard normal prior (`ncomp=0`) or a `ncomp`-dimensional
standard multivariate normal prior.
"""
struct PosteriorAbilityEstimator{PriorT <: Distribution} <: DistributionAbilityEstimator
prior::PriorT
end
Expand Down Expand Up @@ -209,6 +247,19 @@ function covariance_matrix(
)
end

"""
$(TYPEDEF)

Point ability estimate given by the mode of `dist_est` (e.g. MLE for a
[`LikelihoodAbilityEstimator`](@ref) or MAP for a
[`PosteriorAbilityEstimator`](@ref)), found using `optim`.

$(FUNCTIONNAME)(bits...)

Bag-of-config-bits constructor: uses any given `DistributionAbilityEstimator`
and `AbilityOptimizer` found in `bits`, or builds default ones from the rest
of `bits`.
"""
struct ModeAbilityEstimator{
DistEst <: DistributionAbilityEstimator,
OptimizerT <: AbilityOptimizer
Expand All @@ -231,6 +282,18 @@ function power_summary(io::IO, ability_estimator::ModeAbilityEstimator)
power_summary(indent_io, ability_estimator.optim)
end

"""
$(TYPEDEF)

Point ability estimate given by the mean (EAP) of `dist_est`, computed using
`integrator`.

$(FUNCTIONNAME)(bits...)

Bag-of-config-bits constructor: uses any given `DistributionAbilityEstimator`
and `AbilityIntegrator` found in `bits`, or builds default ones from the rest
of `bits`.
"""
struct MeanAbilityEstimator{
DistEst <: DistributionAbilityEstimator,
IntegratorT <: AbilityIntegrator
Expand Down
Loading
Loading