Provably-Optimal-RL

Code release for "Provably Optimal Reinforcement Learning under Safety Filtering" (Oh, Nguyen, Hu, Fisac — IASEAI 2026, arXiv:2510.18082).

The paper shows that with a sufficiently permissive safety filter, RL trained inside the filtered MDP is safe throughout training, still converges, and is asymptotically as optimal as the best safe policy — safety enforcement and performance optimization fully separate. This repo reproduces the experiments and the main figure.

What this repo contains

This is a thin reproduction layer on top of three pinned upstream codebases (git submodules under external/):

Submodule	Provides
safety-stable-baselines @feat/safety_gymnasium	SafetySAC (safety policy/critic) + the safety-gymnasium fork (margin envs)
↳ integrations/safety-gymnasium (nested)	the SafetyCarCircle2-v0 / SafetyCarGoal1-v0 margin envs + TerminateOnCollisionWrapper
Safe-Policy-Optimization	the CPO baseline

Everything specific to this paper — the parameterized training driver, the two safety filters, the configs, and the figure pipeline — lives in porl/ and analysis/.

Experiment matrix

Two environments × four methods. A SafetySAC safety policy is trained first per env; both filters consume it.

Paper name	env id	figure code	value-filter ε	rollout-filter horizon
Circle	SafetyCarCircle2-v0	circle	0.1	100
Goal	SafetyCarGoal1-v0	pillar	0.4	200

Methods: native SAC (no filter) · value filter (safety-critic shielding) · rollout filter (gameplay-style shielding) · CPO (baseline).

Install (Python 3.10, pinned by the safety-gymnasium fork)

git clone --recurse-submodules git@github.com:SafeRoboticsLab/Provably-Optimal-RL.git
cd Provably-Optimal-RL
conda create -n porl python=3.10 -y && conda activate porl

pip install -e external/safety-stable-baselines
pip install -e external/safety-stable-baselines/integrations/safety-gymnasium   # the fork, once
pip install -e external/Safe-Policy-Optimization
pip install -e .

Pinned commits: safety-stable-baselines d186057, safety-gymnasium 39c9e85, Safe-Policy-Optimization 8d0fa76.

Regenerate the figure (no GPU)

The paper's main figure regenerates instantly from committed W&B exports in data/results/:

make figures      # -> figures/algorithm_comparison.{pdf,svg,png}

This is a 3×2 grid — Train Return / Evaluation Return / Total Violations (rows) × Goal / Circle (columns) — overlaying the four methods across 5 seeds. The rollout filter attains zero training-time violations while matching or beating the baselines.

Reproduce from scratch (GPU)

make weights                                        # or train the safety policies below
make train-safety CFG=configs/safety_policy/circle.yaml
make train-task   CFG=configs/task/circle_rollout.yaml SEED=0
make cpo          ENV=SafetyCarCircle2-v0 SEED=0
make collect                                        # local runs -> data/results/*.csv
make figures
# ...or the whole matrix at once (slow):
make reproduce

make collect reconstructs the figure CSVs from local TensorBoard logs (SAC family) and SafePO progress.csv files (CPO). Note: the committed CSVs in data/results/ are the canonical figure source (frozen W&B exports, 2025-10-10); the reconstructed CPO violation counts use cumsum(EpCost) and may differ in absolute scale from the W&B definition.

Optional model zoo

Published SafetySAC safety-policy weights (needed by both filters) are hosted as GitHub Release assets, described in weights/MANIFEST.json:

make weights      # downloads + checksum-verifies into weights/safety_policy/<env>/model.zip

Layout

porl/        envs.py · filters.py · train.py · train_safety_policy.py · run_cpo.py · logging_callbacks.py
configs/     safety_policy/{circle,goal}.yaml · task/{circle,goal}_{naive,value,rollout}.yaml
analysis/    plot_figures.py · figure_spec.py · collect_results.py · capture_env_images.py
data/results/  committed W&B-export CSVs (instant figure rebuild)
external/     pinned submodules
slurm/       5-seed array wrappers

Citation

@inproceedings{oh2026provably,
  title={Provably Optimal Reinforcement Learning under Safety Filtering},
  author={Oh, Donggeon David and Nguyen, Duy P. and Hu, Haimin and Fisac, Jaime F.},
  booktitle={International Association for Safe \& Ethical AI (IASEAI)},
  year={2026},
  note={arXiv:2510.18082}
}