Ontology2SQL

Ontology-grounded agentic Text-to-SQL for BIRD Mini-Dev.

Results

Backend	Simple	Moderate	Challenging	Overall EX
SQLite	85.81%	68.00%	52.94%	351/500 (70.20%)
PostgreSQL	83.11%	63.60%	46.08%	329/500 (65.80%)

The SQLite score uses BIRD execution accuracy. The PostgreSQL score replays the same frozen predictions through deterministic dialect adaptation against the native PostgreSQL Mini-Dev databases.

See RESULTS.md for the frozen configuration, hashes, and score breakdown.

Frozen artifacts

• artifacts/sqlite/predict_dev.json — BIRD Mini-Dev SQLite submission
• artifacts/sqlite/ontology2sql-bird-mini-dev-70p2.zip — packaged SQLite submission
• artifacts/postgresql/predict_dev.json — PostgreSQL-adapted predictions
• artifacts/postgresql/ontology2sql-bird-mini-dev-postgresql-65p8.zip — packaged PostgreSQL predictions

SHA-256 checksums are listed in CHECKSUMS.sha256.

Setup

Ontology2SQL requires Python 3.12. Install the minimal runtime and test dependencies:

python -m venv .venv
.venv/Scripts/python -m pip install -r requirements.txt -r requirements-dev.txt

On Linux or macOS, activate or invoke the virtual environment using the usual .venv/bin/ paths.

Download BIRD Mini-Dev from the official BIRD distribution and arrange it as:

backend/data/benchmarks/bird-mini-dev/
├── mini_dev_sqlite.json
└── dataset/dev_20240627/dev_databases/
    └── <database_id>/
        ├── <database_id>.sqlite
        └── database_description/

BIRD data and gold SQL are intentionally not redistributed by this repository.

Build semantic mappings

python backend/scripts/build_text2sql_bird_mappings.py \
  --approve --reviewer "<reviewer>"

The command builds reviewed, schema-bound mappings under backend/data/benchmarks/bird-mini-dev/semantic_mappings/.

Run with the official DeepSeek API

Store the key in an environment variable; never add it to a config file or command-line argument.

PowerShell:

$env:DEEPSEEK_API_KEY = "<your-key>"
py -3.12 backend/scripts/run_text2sql_bird_parallel.py `
  --mode ontology-ir `
  --provider openai-compatible `
  --api-base https://api.deepseek.com/v1 `
  --api-key-env DEEPSEEK_API_KEY `
  --model deepseek-v4-flash `
  --reasoning-effort none `
  --max-output-tokens 16384 `
  --max-steps 12 `
  --stepwise-calculation `
  --fast-path `
  --workers 8 `
  --output backend/data/benchmarks/bird-mini-dev/runs/ontology2sql-full500

Bash:

export DEEPSEEK_API_KEY="<your-key>"
python backend/scripts/run_text2sql_bird_parallel.py \
  --mode ontology-ir \
  --provider openai-compatible \
  --api-base https://api.deepseek.com/v1 \
  --api-key-env DEEPSEEK_API_KEY \
  --model deepseek-v4-flash \
  --reasoning-effort none \
  --max-output-tokens 16384 \
  --max-steps 12 \
  --stepwise-calculation \
  --fast-path \
  --workers 8 \
  --output backend/data/benchmarks/bird-mini-dev/runs/ontology2sql-full500

The runner uses every question by default, executes independent resumable workers, and merges outputs by question ID with validation. For DeepSeek Chat Completions, --reasoning-effort none sends {"thinking":{"type":"disabled"}} as documented by DeepSeek.

PostgreSQL evaluation

After loading the official Mini-Dev PostgreSQL dump, evaluate the frozen SQLite predictions with:

python backend/scripts/evaluate_text2sql_bird_postgres.py \
  --source-predictions artifacts/sqlite/predict_dev.json \
  --questions backend/data/benchmarks/bird-mini-dev/mini_dev_sqlite.json \
  --dsn "postgresql://USER:PASSWORD@HOST:PORT/DATABASE" \
  --output postgresql-evaluation.json \
  --submission-output postgresql-predict_dev.json

Tests

python -m pytest -c backend/pytest.ini backend/tests

License

Apache License 2.0. BIRD dataset files remain subject to the BIRD dataset's own license and distribution terms.