Compile organizational policy into verifiable decision training data.
Every organization's judgment, turned into the data that makes AI agents trustworthy.
Why Decision Forge · See it in action · Quickstart · How it works · Documentation · Roadmap · Contributing · Changelog
English · 简体中文
For three hundred years, every organization was built from people. The next ones will be built from people and agents — and the agents will outnumber the people.
A second workforce has arrived. Agents now carry a task for hours without supervision, take on work that once went to a colleague, and call one another across tools and companies. For the first time, the supply of labor is no longer bounded by the number of people. Yet this abundance of intelligence has not become an abundance of capability. Agents wait in pilots because they are trusted with too little, or act in production while trusted with too much.
What is missing is trust. Over centuries, organizations learned to hand their work to thousands of strangers. People foresee what their actions will do, act within their authority, and answer for the results. Agents have been connected to every tool and to each other, but given none of this. The pipes are built. The trust layer is empty.
Trust begins with foresight. To trust an action is to know what it will do before it happens. Language models learned to speak from the world's text; world models are learning physics from its video. No public dataset records what happens inside an organization when an agent reassigns a client, shares a file, or approves a payment. That knowledge lives only within the organization, in three forms:
| What foresight requires | Where it lives | What Decision Forge does with it |
|---|---|---|
| Decision context: the state, roles, and options at the moment of action | Workflows, tickets, CRM records | Reconstructs each decision moment with explicit candidate actions, grouped into families |
| Judgment standard: what this organization allows, forbids, or escalates | Handbooks, SOPs, approval rules, permission tables | Compiles rules with verified source quotations, puts them through business review, and replays them to label every example |
| Execution receipts: what actually happened after the action | Approvals, outcomes, runtime logs | Ingests runtime events and reviewed outcomes as feedback for the next dataset version |
Every organization has its own sense of right. What one firm encourages, another forbids. That judgment cannot be downloaded from a general model. It has to be learned inside each organization, and it has to belong to that organization.
Decision Forge turns that judgment into data. It compiles an organization's own documents into verified decision training data: every label traces back to a rule, every rule back to its source, and the core runs inside the organization's own walls. It is the foundation of the stack we are building at Juneo Lab:
| Layer | Role | Status |
|---|---|---|
| AI-native organizations | People and agents allocate work, share information, act, and accept results together; their execution receipts flow back into the data | Built with partner organizations |
| Laplace | A causal-interventional model that foresees the consequence of an action before it runs, then lets it proceed, stops it, or hands it back to a person | In development |
| Decision Forge | An organization's documents, turned into verified decision training data | Open source, this repository |
The world we are building. Humans keep the right to decide. Agents earn the right to act, within limits that only narrow as work passes from hand to hand. Every action is foreseen before it happens, and every contribution is measured after it is done. Every organization owns the intelligence that carries its judgment.
We are building the trust layer that lets human organizations keep pace with machine intelligence. Read the full story in Why Decision Forge.
Organizational decisions depend on policies, thresholds, escalation rules, and acceptance criteria recorded in handbooks, SOPs, and workflow definitions. Decision Forge turns that knowledge into reproducible supervision for decision models.
The document compiler proposes a rule specification with source quotations for business review. The data pipeline evaluates scenarios against the approved rules and exports decision examples with verified labels, source provenance, and stable family assignments. The result is training data you can audit, reproduce, and hand to a trainer.
Note
Status: v0.3.2 alpha — organizational decision data pipeline.
Reviewed or authored specifications build into training data locally, without model credentials.
Document compilation and model-assisted synthesis use an explicitly selected model endpoint.
See project status for the current implementation and next milestones.
| Capability | What you get |
|---|---|
| Offline data path | Reviewed rules → scenarios → verified labels → exports, with no API keys or model downloads |
| Policy-verified labels | Recomputed from declared rules by a deterministic interpreter |
| Source provenance | Verified rule quotations and source hashes; corpus compilation adds spans and locators |
| Leak-resistant splits | Related variants share a family split; cross-split duplicates are reported and dropped |
| Training-ready exports | Four formats, including the candidate-readout decision contract |
| Broad ingestion | Text, Markdown, JSON/YAML, PDF, DOCX/XLSX/PPTX, HTML, CSV/TSV, and local OCR |
| Validated | 311 tests, 90.89% statement coverage in the full optional-adapter environment |
| Continuous checks | 10 CI jobs across Linux and macOS, including adapter, OCR, and Harbor controls; CodeQL and OpenSSF Scorecard scans |
Two fictional organizations, a speed-first startup and a clinic network, deploy agents that attempt the same actions. Each writes its own policy, and Decision Forge compiles each into verified decisions:
| Agent action | Lumen Labs | Halcyon Health |
|---|---|---|
| Refund 400 to a verified customer | proceed: “Agents may issue refunds of up to 1000 dollars to verified customers without approval.” | hand back: “Every other refund for a verified patient goes to a person in patient accounts.” |
| Send a health record to an internal team | send in full: “Agents may share full customer records with internal teams.” | send a summary: “Health information may reach internal teams only as a minimum-necessary summary.” |
| Delegate more authority than the agent holds | block: “An agent must never delegate more authority than it holds.” | block: “An agent must never delegate more authority than it holds.” |
Of 17 situations, 8 are decided differently, and every decision cites the sentence that governs it. Run the agent delegation example offline in one command.
compile-corpus drafts rules from organization sources through an explicitly selected model endpoint;
business review accepts them, or an authored specification skips both steps. Everything after review
runs offline: scenario families and quotas, label replay through the policy interpreter, deduplication
with family split isolation, and training exports.
The pipeline organizes workflow decision points into the four kinds of decisions an organization delegates to people and agents:
| Surface | Question the model learns to answer |
|---|---|
| Allocate work | Which person or agent should own this task? |
| Share information | Which recipient may receive which information? |
| Select actions | Should the workflow execute, adjust, or request authorization? |
| Evaluate outcomes | Does the deliverable meet the acceptance criteria? |
Requires Python 3.11+ and uv. No API keys, no model downloads. New here? The five-minute tutorial goes from a written policy to training data, and the notebook version runs in Colab.
git clone https://github.com/Juneo-Lab/decision-forge
cd decision-forge
uv sync --group dev
uv run decision-forge validate examples/customer_support/project.yaml
uv run decision-forge build examples/customer_support/project.yaml --output runs/demo
uv run decision-forge export runs/demo --output runs/demo-decision --format decisionThe fictional Northstar Support example is deliberately small (one handbook, ~1.2 KB of policy) and
produces a complete, verified dataset:
| Output | Value |
|---|---|
| Accepted episodes | 336 (requested 336, realized 336) |
| By surface | 96 allocate · 96 share · 96 act · 48 evaluate |
| By split | 196 train · 84 development · 56 test, assigned per family |
| Rejected for having no permitted choice | 0 |
| Model calls | 0 |
Artifacts in runs/demo/: episodes.jsonl, rejected.jsonl, distribution.json, dedup.json,
report.json, report.md, a hash-bound project.snapshot.json, and an integrity-checked
manifest.json. Rebuilding reuses content-addressed scenario caches; changing an input is detected
and refused rather than silently mixed into the old run.
The example's committed project.splits.lock.json records every family's split, so adding scenario
families later never moves an existing family between training and held-out data.
Each published scenario label is recomputed by the policy interpreter. Source checks and review records connect the declared rules to the organization's documents.
| Property | How it is enforced |
|---|---|
| Policy-verified labels | A deterministic interpreter applies priority, deny precedence, and set-valued ties to every candidate. Synthesis proposes states; the interpreter supplies their labels. |
| Source-grounded rules | Evidence quotations are validated against source text, allowing whitespace normalization. compile-corpus recomputes spans and source locators from extracted claims. |
| Content-bound review | Compiler output is a draft. Builds require accepted decisions for all current review items: rules, nodes, conflicts, and dataset:recipe. Authored specifications remain usable directly. |
| Family isolation | Related variants share a split, and a committed split lock keeps every family in its split across dataset versions. Duplicate detection keeps the held-out copy and reports cross-split overlap. Expansion imports validate current scenario/family/split lineage and re-derive every model-visible field. |
| Checkpoint recovery | Completed extraction chunks and accepted synthesis records are reused. Failed or interrupted calls retain their budget reservations; a response lost before checkpointing may require another call. |
| Bounded corpus compilation and synthesis | compile-corpus reserves budgets per HTTP attempt. expand reserves logical proposal calls and their input bytes; response-mode fallback may issue multiple HTTP attempts within one proposal call. Both retain budget journals and support explicit budget increases for resumption. |
| Visible distribution gaps | Requested and realized quotas, rejected proposals, and remaining shortfalls are reported. Only verified examples enter the dataset. |
Four export formats preserve the run's family split assignments. canonical retains set-valued
labels; the other formats omit rows with multiple valid choices, which can also omit entire families:
| Format | Shape | Use |
|---|---|---|
decision |
Explicit options + aligned target_probs, family grouping, provenance |
Candidate-readout training (recommended) |
canonical |
Input/target pair, set-valued labels preserved | Custom pipelines that keep ambiguity |
typed |
Choice question with per-candidate criteria | Constrained decoding / evaluation harnesses |
sft |
Chat messages with a single answer | Reference SFT baseline |
uv run decision-forge export runs/demo --output runs/train --format decision
uv run python scripts/finetune_evaluation.py prepare runs/demo --output runs/plan --budget-examples 2000The decision format emits objective: candidate_readout rows with unit weights and provenance.
decision_forge.training.encode_decision turns them into a native
non-thinking prefix: single-token candidate labels, target probabilities moved together with the
candidate order, no answer token appended, and truncation rejected rather than silently applied. A
trainer that only minimizes ordinary answer-token SFT loss cannot consume this encoding — the contract
states the required objective explicitly.
scripts/finetune_evaluation.py prepare writes selected training rows, their IDs and file digests,
and the required training objective. evaluate scores an explicitly selected endpoint on held-out
cases; compare requires the same dataset, case order, and prompt. Fixed-budget training and measured
decision-quality gains are the next evaluation milestone.
export FORGE_API_KEY="your-provider-key"
uv run decision-forge compile-corpus ./private/corpus.yaml --output runs/acme \
--base-url https://your-model-host.example/v1 --model your-model
uv run decision-forge review-status runs/acme/project.json
uv run decision-forge review-record runs/acme/project.json \
--item ITEM_ID_FROM_REVIEW_STATUS --decision accepted \
--reviewer you --reason "Explain the evidence supporting this specific item." \
--fingerprint FINGERPRINT_FROM_REVIEW_STATUS
# Review every rule, node, conflict, and dataset:recipe item using its returned ID and fingerprint.
# Build after review-status reports ready: true.
uv run decision-forge review-status runs/acme/project.json
uv run decision-forge build runs/acme/project.json --output runs/acme-dataCreate corpus.yaml using the corpus manifest format. Text,
Markdown, JSON, YAML, PDF, DOCX, XLSX, PPTX, HTML, CSV/TSV, and OCR image inputs carry version and
access metadata. Install documents for native office/PDF parsing and ocr-rapid for local
PP-OCR small models. Document formats and OCR covers automatic format
detection, Tesseract, and explicit vision OCR service configuration.
What happens during compilation
Extraction is chunked and resumable; assembly requires rule evidence to resolve to extracted claims. Invalid schemas and ambiguous quotations produce validation errors; policy conflicts enter the review queue for adjudication. Source snapshots and parsed generated objects are stored in private local artifacts, including caches and repair records. Provider credentials stay outside those artifacts.
An optional recipe planner turns a requested distribution into exact per-cell
quotas, measures what the deterministic pool can supply after deduplication, and separates work a
model could do from work only more source material can do. Verified synthesis (expand) then
publishes only proposals whose label is replayed from the declared policy.
Validation, benchmarks, and live model runs
Local validation (macOS ARM64, Python 3.12, all optional extras): 311 tests, 90.89% statement coverage, Ruff, strict mypy, strict MkDocs, wheel and sdist builds. CI runs 10 jobs: quality, six test jobs (Ubuntu and macOS × Python 3.11/3.12/3.13), pinned Data Designer and LangGraph adapter contracts, and Harbor task execution in Docker under oracle and no-op controls, plus real local OCR inference and seven-format ingestion. CodeQL and OpenSSF Scorecard scan the repository on pull requests and on a schedule.
Local OCR: the seven-format ingestion smoke test archives native file parsing and exact PNG/scanned-PDF transcriptions using bundled PP-OCR small models. Tesseract and vision endpoints have contract tests; their model quality is evaluated separately.
Offline benchmark: cold and warm builds are measured on 25, 100, and 400 synthetic families. The benchmark report records measured throughput, host and dependency versions, and matching dataset digests; the original JSON is archived with the results.
Live model runs (archived in experiments/, fictional inputs):
| Experiment | Models | Result | Recorded cost |
|---|---|---|---|
| Document compilation | Claude Opus 5.5, GPT-6.1 Sol | 13/13 exact claims each; 31/32 decision agreement each | Successful runs: $0.293428 / $0.1505675 |
| Multi-step workflows | Same two models | 3/3 tasks, 11/11 permitted decisions each | $0.036412 total, 22 calls |
The compilation ledger also records earlier diagnostics and failed attempts: $1.08587197 in known costs across recorded attempts, plus five attempts with unknown cost. Its total billed cost is unknown. The reference cases are maintainer-authored fixtures. Archived results document behavior on these small fictional inputs; organizational training-quality evaluation remains a separate milestone.
| Supported | Next milestones |
|---|---|
|
|
The synthesis command expand operates within the approved rule specification and proposes
decision-time state. Policy extraction and revision belong to document compilation and review.
The roadmap tracks each remaining acceptance item against a milestone.
| Guide | What it covers |
|---|---|
| Tutorial | From a written policy to training data in five minutes |
| Why Decision Forge | The problem, the data it needs, and where this project fits |
| Project status | Implemented paths and remaining validation |
| Architecture | Component boundaries and data flow |
| Specification | Rule, scenario, and recipe semantics |
| Document formats and OCR | Format detection, parsers, and OCR backends |
| Organization compiler | Ingestion, extraction, review, resumption |
| Generation planning | Quotas, verified synthesis, deduplication |
| Decision training | Export contract and candidate-readout encoding |
| Platform workflows | Multi-step tasks, grading, feedback, updates |
| Data protocol | Evaluation protocol and reporting rules |
| Benchmarks | Offline throughput protocol and reference results |
| Release process | Tagging, attested artifacts, and verification |
| References | Prior art and integration candidates |
uv sync --group dev --group docs --extra documents
uv run ruff check . && uv run ruff format --check .
uv run mypy src
uv run pytest --cov=decision_forge --cov-fail-under=80
uv run mkdocs build --strict
uv buildRun uvx pre-commit install once to apply the same formatting and configuration checks before each
commit.
| Document | Purpose |
|---|---|
| Contributing | Review bar, label taxonomy, and definition of done |
| Governance | How decisions are made |
| Support | Where to ask questions |
| Security | Reporting vulnerabilities and verifying releases |
| Code of Conduct | Expected behavior in project spaces |
Code and technical documentation are written in English; this README also has a maintained Simplified Chinese translation.
If you use Decision Forge in published work, cite the version you used. GitHub's Cite this
repository button reads CITATION.cff; export formats are versioned, and
CHANGELOG.md records every change.
@software{decision_forge,
title = {Decision Forge: Organizational Decision Data Infrastructure},
author = {{Decision Forge contributors}},
version = {0.3.2},
license = {Apache-2.0},
url = {https://github.com/Juneo-Lab/decision-forge}
}Decision Forge is licensed under Apache-2.0. See NOTICE for attribution.