A comprehensive, interactive framework for validating machine learning models —
from data quality through explainability, robustness, and fairness.
The Federal Reserve's SR 26-2 (jointly issued with OCC and FDIC, April 2026) supersedes SR 11-7 as the primary US supervisory guidance on Model Risk Management. Three shifts define the new landscape:
1. Principles over checklist — defensibility is the new bar. SR 26-2 is explicitly principles-based. Non-compliance with the guidance alone no longer automatically triggers supervisory criticism — but that means the compliance cover of a completed checklist is also gone. What counts now is whether your validation discipline can defend itself on its own merits: did you test the right things, and can you show why?
2. Rigor must be calibrated to model risk — not uniform. Materiality now formally drives validation depth across four dimensions: inherent risk, exposure, purpose, and use. A capital adequacy model and a marketing segmenter no longer warrant the same treatment. Annual revalidation defaults are removed. High-impact models (credit, fraud, capital, pricing) demand structured, multi-dimensional validation.
3. Traditional ML is in scope. GenAI is not — yet. SR 26-2 explicitly carves out Generative AI and agentic AI as "novel and rapidly evolving," with formal guidance deferred. This carve-out does not extend to traditional ML — Random Forest, Gradient Boosting, LightGBM, neural networks, and all models used in credit, fraud, underwriting, and capital allocation remain squarely in scope. The guidance also notes that existing MRM principles are expected to apply to GenAI pending future rules.
This framework is a direct response to SR 26-2. It provides the structured, reproducible, examiner-ready evidence the new guidance demands — each step maps to a recognised validation dimension, and every diagnostic test targets the failure modes regulators now expect you to find. → Full SR 26-2 analysis and framework mapping
Training a model is the beginning, not the end. Regulatory frameworks (SR 26-2, SS1/23, TRIM), internal model risk governance, and responsible AI standards all require rigorous model validation before deployment — and ongoing monitoring thereafter.
This framework provides a structured, widget-driven validation workflow that covers every major dimension of model quality in one notebook. No configuration files, no boilerplate — select a dataset, follow the steps, and get a complete validation picture.
Ideal for:
- 🏦 Risk and compliance practitioners validating credit, fraud, or underwriting models
- 🔬 Data scientists conducting pre-deployment model audits
- 🎓 Researchers benchmarking and explaining ML models
- 👩💻 ML engineers building internal model governance tooling
No installation required. Click the badge and run all cells:
- Click Open in Colab above
- Runtime → Run all (
Ctrl+F9/Cmd+F9) - The setup cell installs dependencies and clones this repo automatically
- Select a built-in dataset from the dropdown — task type and target variable are pre-configured
- Follow each numbered step in order; every step builds on the previous one
Tip: Built-in datasets (Breast Cancer, Credit Risk, etc.) are ready to use without any uploads. To validate your own model, upload a CSV in Step 1.
git clone https://github.com/minw0607/ml_validation_framework.git
cd ml_validation_framework
pip install -r requirements.txt
jupyter notebook notebooks/ML_Model_Validation.ipynbRequirements: Python 3.8+, see requirements.txt for the full dependency list.
The notebook guides you through nine steps, each powered by interactive widgets. Steps must be run in order — each one stores state used by the next.
flowchart LR
subgraph data ["🗄️ Data & Feature Layer"]
direction TB
A["Step 1\nLoad Data"] --> B["Step 2\nData Summary"]
B --> C["Step 3\nPreprocess"]
C --> D["Step 4\nModel Setup"]
D --> E["Step 5\nFeature Select"]
end
subgraph model ["🤖 Model Layer"]
direction TB
F["Step 6\nTrain"]
G["Step 7\nRegister"]
F --> G
end
subgraph govern ["🔍 Governance Layer"]
direction TB
H["Step 8\nExplainability"]
I["Step 9\nDiagnostics"]
H --> I
end
E --> F
G --> H
style I fill:#1a73e8,color:#fff,stroke:#1a73e8
style H fill:#34a853,color:#fff,stroke:#34a853
| Step | Method | Description |
|---|---|---|
| 1 | data_loader() |
Select a built-in dataset or upload your own CSV |
| 2 | data_summary() |
Schema, distributions, correlation heatmap, missing-value analysis |
| 3 | data_preprocess() |
Imputation, categorical encoding, and normalization |
| 4 | model_prepare() |
Set target variable, task type, train/test split, and random seed |
| 5 | feature_select() |
Pearson & Spearman correlation filtering + permutation importance |
| 6 | model_training() |
Train and compare multiple algorithms with tunable hyperparameters |
| 7 | model_register() |
Register best model(s) for downstream diagnostics |
| 8 | model_explain() |
Global and local explainability — SHAP, LIME, PDP |
| 9 | model_diagnose() |
Full diagnostic suite: overfitting, weak spots, fairness, robustness, and more |
The data loader offers two modes:
- Built-in datasets — select from the dropdown; the framework pre-configures the task type and target variable automatically. No file upload needed.
- Upload CSV — use the file picker to upload any comma-separated file. The framework infers the delimiter and encoding.
All built-in datasets are sampled to ≤ 30,000 rows for interactive speed. The full datasets are available in the data/ folder.
| Dataset | Task | Rows | Features | Domain |
|---|---|---|---|---|
| Breast Cancer (sklearn) | Binary classification | 569 | 30 | Healthcare |
| Wine Quality (sklearn) | Multi-class classification | 178 | 13 | Food science |
| Iris (sklearn) | Multi-class classification | 150 | 4 | Botany |
| Diabetes (sklearn) | Regression | 442 | 10 | Healthcare |
| California Housing (sklearn) | Regression | 20,640 | 8 | Real estate |
| GMSC Credit Risk | Binary classification | 30,000* | 10 | Credit risk |
| Taiwan Credit Default | Binary classification | 30,000 | 23 | Credit risk |
* Sampled from 150k rows; full dataset in data/cs-training.csv.
Provides five expandable sections for exploring the loaded dataset:
| Section | What it shows |
|---|---|
| Schema | Column names, data types (int64, float64, object…), non-null count, null count, null % |
| Numerical Statistics | Mean, std, min, 25th/50th/75th percentile, max for every numeric column |
| Categorical Statistics | Count, unique values, most frequent category for object/category columns |
| Missing Value Analysis | Per-column flag showing which columns have any nulls |
| Correlation Analysis | Interactive histogram (univariate), scatter plot (bivariate), and Pearson heatmap |
At the bottom, a type-conversion control lets you cast any column to Numerical or Categorical if it was mis-detected.
Key concept — Pearson correlation: Measures the linear relationship between two variables, ranging from −1 (perfect negative) to +1 (perfect positive). Values close to 0 indicate no linear relationship. Strong correlations between predictor columns (|r| > 0.8) indicate multicollinearity — two features carrying almost the same information. Retaining both typically inflates variance without adding predictive value.
Three interactive sections for preparing the data before modelling:
Handles missing values column by column. Select a feature column and a strategy:
| Strategy | Best used when |
|---|---|
mean |
Numeric column, roughly symmetric distribution |
median |
Numeric column with skew or outliers |
most_frequent |
Categorical column, or numeric with few unique values |
constant |
You want to fill with a specific sentinel value |
Dropping rows (not currently in the UI) is only safe when missingness is truly random and < 5% of the dataset. Otherwise imputation preserves sample size and avoids selection bias.
Converts text/category columns to numeric values so models can consume them:
| Method | How it works | When to use |
|---|---|---|
| One-hot encoding | Creates one binary column per category; drops the original | Nominal features (no inherent order) — e.g., country, product type |
| Label encoding | Assigns an integer 0, 1, 2… to each category | Ordinal features (order matters) — e.g., Low/Medium/High; also fine for tree-based models that split on thresholds |
Important: Apply encoding before training. If you add new one-hot columns after registering a model, the column set will mismatch at inference time.
Scales numeric features so they share a comparable magnitude:
| Method | Formula | Best for |
|---|---|---|
| Standard (z-score) | (x − μ) / σ | Linear models, MLP, distance-based models |
| Min-Max | (x − min) / (max − min) → [0, 1] | When bounded output is required; sensitive to outliers |
Tree-based models (Random Forest, GBM, LightGBM) are scale-invariant — normalization makes no difference to their predictions. Scale for linear models, logistic regression, SVM, and MLPs.
Configure the experiment before training:
| Setting | Widget | Notes |
|---|---|---|
| Target variable | Dropdown | The column the model will predict; all other columns become features |
| Task type | Dropdown | Classification or Regression; auto-filled for built-in datasets |
| Train / test split | Slider (0.1–0.9) | Fraction held out for testing; default 0.2 (80/20 split) |
| Random seed | Slider (0–10) | Fixes data split and model initialisation for reproducibility |
Click Confirm to lock in the settings. All downstream steps use these values.
Key concept — train/test split: The test set must never be seen during training or hyperparameter tuning. It serves as a stand-in for real-world unseen data. A common split is 70/30 or 80/20. For small datasets (< 1,000 rows), consider cross-validation instead.
Two complementary methods for identifying which features are most useful:
Heatmaps showing pairwise correlation between all features and the target. Use these to:
- Spot features with very low target correlation (likely uninformative)
- Identify highly correlated predictor pairs (multicollinearity candidates)
Spearman vs Pearson: Pearson measures linear correlation; Spearman measures monotonic (rank-order) correlation and is more robust to outliers and non-linear relationships. Use both: if Pearson is low but Spearman is high, the relationship exists but is non-linear.
Fits a model (default: Random Forest), then for each feature, randomly shuffles its values and measures how much accuracy drops. A large drop means the feature is important; near-zero or negative means the feature adds no value (or introduces noise).
Controls:
- Model selector — choose the fitter used for importance scoring
- Importance threshold slider — features below the threshold are highlighted
- Remove Features — drops low-importance features from the dataset
- Revert — restores all original features
Why permutation importance? Unlike gain-based importance (built into tree models), permutation importance is model-agnostic and directly measures out-of-sample predictive contribution. It is less susceptible to bias from high-cardinality features.
Trains one or more candidate models on the preprocessed data and compares their performance side by side.
Supported algorithms:
| Model | Classification | Regression | Notes |
|---|---|---|---|
| Logistic Regression | ✅ | — | Linear baseline; fast, interpretable |
| Random Forest | ✅ | ✅ | Ensemble of decision trees; handles non-linearity |
| Gradient Boosting (GBM) | ✅ | ✅ | Sequential boosting; strong out-of-the-box |
| LightGBM | ✅ | ✅ | Fast gradient boosting; good for large datasets |
| MLP Neural Network | ✅ | ✅ | Feedforward network; requires scaling |
Metrics shown:
| Task | Metrics |
|---|---|
| Classification | AUC-ROC, Accuracy, Precision, Recall, F1 Score, Confusion Matrix, ROC Curve |
| Regression | R², MAE, RMSE |
Key concept — AUC-ROC: The area under the Receiver Operating Characteristic curve measures how well a classifier separates classes at all decision thresholds. AUC = 1.0 is perfect; AUC = 0.5 is no better than random. For imbalanced datasets, prefer AUC over Accuracy as the primary metric — Accuracy can be misleadingly high when one class dominates.
Serialises trained model(s) to disk (.pkl) and logs metadata:
- Algorithm name and hyperparameters
- Evaluation metrics (from Step 6)
- Timestamp
Registered models are loaded automatically by Steps 8 and 9. Give your model a descriptive name (e.g. rf_credit_2026) to distinguish multiple experiments.
Three techniques for understanding model predictions at global and local scope:
flowchart LR
subgraph global ["🌍 Global Scope (all predictions)"]
G1["SHAP Beeswarm\nFeature importance + direction\nacross full dataset"]
G2["PDP One-Way\nMarginal effect of\na single feature"]
G3["PDP Two-Way\nInteraction heatmap\nfor two features"]
end
subgraph local ["🔎 Local Scope (single prediction)"]
L1["SHAP Waterfall\nWhich features pushed\nthis score up or down"]
L2["LIME\nLinear surrogate in\nlocal neighbourhood"]
end
Q["Question:\nWhy did the model\nscore THIS case?"] --> L1
Q --> L2
P["Question:\nWhat did the model\nlearn overall?"] --> G1
P --> G2
P --> G3
Based on cooperative game theory, SHAP assigns each feature a contribution value (Shapley value) for every prediction. It is the only method that satisfies all four desirable axioms: efficiency, symmetry, dummy, and additivity.
- Global view — beeswarm plot showing distribution of SHAP values across all predictions; dots coloured by feature value reveal direction (high feature value → positive/negative impact)
- Local view — waterfall chart for a single prediction showing exactly which features pushed the score up or down
Use the global SHAP plot to understand what the model learned overall. Use local SHAP or LIME to explain a specific high-risk or borderline prediction to a stakeholder.
Shows the marginal effect of one or two features on the predicted outcome, averaging out the influence of all other features.
- One-Way PDP — how does the prediction change as a single feature varies?
- Two-Way PDP — heatmap of predicted outcome across a grid of two features; reveals interaction effects
PDP assumes features are independent. If two selected features are strongly correlated, the two-way PDP can produce unrealistic feature combinations. In that case, use SHAP interaction values instead.
Fits a simple linear surrogate model in the local neighbourhood of a single prediction. The surrogate's coefficients approximate which features most influenced that specific prediction.
LIME is best for explaining individual predictions to non-technical audiences. It is an approximation — the surrogate is not the actual model.
The diagnostic suite stress-tests the registered model across seven dimensions, each in its own tab:
Segment-level performance breakdown — computes metrics within slices of the data (by feature bins or categories) to reveal whether the model performs consistently across subgroups, or whether accuracy is concentrated in easy cases.
Evaluates performance drift across data slices ordered by time or another stratification variable. A model whose accuracy degrades steadily across slices may be overfitting to temporal patterns in the training window.
Rather than a single point prediction, conformal prediction produces a prediction set (classification) or prediction interval (regression) that is guaranteed to contain the true label with a user-specified coverage probability (e.g., 90%).
Key metrics:
- Coverage — fraction of test instances where the true label falls inside the prediction set. Should be ≥ the target confidence level (1 − α).
- Efficiency — average size of prediction sets. Smaller sets indicate a more decisive, confident model.
Conformal prediction provides distribution-free validity guarantees under the only assumption that calibration and test data are exchangeable. It requires no assumptions about the underlying model or data distribution.
Three complementary tests to detect and characterise overfitting:
Test 1 — Residual Error Test (Train/Test Gap) Computes the same metric (Accuracy/AUC/F1 for classification; R²/MAE/MSE for regression) on both training and test data. A gap > 0.05 is flagged as an overfitting signal.
A model that achieves 0.95 AUC on training data and 0.72 AUC on test data has a gap of 0.23 — strong evidence of overfitting. The model has memorised training patterns that do not generalise.
Test 2 — Learning Curve (Degree of Freedom Proxy) Trains the model on progressively larger fractions of the training data (10% to 100%) and plots both training score and cross-validation score with confidence bands.
| Pattern | Interpretation |
|---|---|
| Large, persistent gap between train and CV score | High variance / overfitting — more data or regularisation needed |
| Both curves converge at a low score | High bias / underfitting — model is too simple; add features or complexity |
| Curves converge at a high score | Well-fitted — performance is limited by data, not model capacity |
Test 3 — Robustness to Feature Perturbation Adds Gaussian noise (10% × feature std) to each numeric feature one at a time and measures the resulting performance drop. Features with a drop > 0.02 are flagged as high-sensitivity.
High sensitivity to a specific feature means the model's predictions are fragile around that input. In production, noisy sensor readings or data entry errors on that feature could cause large prediction swings.
Uses a decision tree to segment the test data into leaf nodes, then flags segments where model accuracy falls significantly below the overall baseline (threshold: 0.1).
- One-Way — segments by a single feature; shows a heatmap of accuracy by segment with frequency counts
- Two-Way — cross-tabulates two features; reveals interaction effects that degrade performance in specific combinations
Weak spots are often more actionable than aggregate metrics. A model with 90% overall accuracy but 55% accuracy on a high-value customer segment is not production-ready for that segment.
Evaluates model stability under input perturbations at scale — across multiple features and perturbation magnitudes. Complements the overfitting robustness test with broader coverage and multiple iterations.
Evaluates whether the model treats demographic or protected groups equitably. Three industry-standard metrics are computed for any user-specified sensitive attribute and protected class:
Adverse Impact Ratio (AIR)
AIR = (Positive prediction rate for protected group) / (Positive prediction rate for reference group)
The EEOC "4/5ths rule" (80% rule) flags AIR < 0.8 as potential disparate impact — the protected group receives favourable outcomes at less than 80% the rate of the reference group. Bars below 0.8 are highlighted in red.
Precision Ratio (PR)
PR = PPV_protected / PPV_reference, where PPV = TP / (TP + FP)
Measures predictive parity — whether positive predictions are equally accurate for both groups. PR = 1.0 means the model's precision is identical across groups. Values far from 1.0 indicate that the model's false positive burden falls disproportionately on one group.
Standardized Mean Difference (SMD)
SMD = (mean_protected − mean_reference) / pooled_std
A scale-free measure of the difference in predicted scores between groups. Used primarily for regression tasks. |SMD| < 0.2 is generally considered negligible; > 0.5 is moderate; > 0.8 is large.
How to use:
- Select a sensitive attribute (e.g.,
age,gender) from the dropdown - Select the protected class value (e.g., a specific age bin or category)
- Click Add Feature/Class — repeat for multiple comparisons
- Click Generate Disparity Plot to render all metrics at once
Regulatory context: Many jurisdictions require disparate impact analysis for models used in lending (ECOA, Fair Housing Act), hiring, or insurance. AIR < 0.8 is the most commonly cited threshold in US regulatory guidance (EEOC, CFPB).
mindmap
root((ML Validation\nFramework))
Data
Schema & types
Missing values
Distributions
Correlations
Features
Pearson / Spearman
Permutation importance
Threshold filtering
Performance
AUC · F1 · Accuracy
R² · MAE · RMSE
Confusion matrix
Explainability
SHAP global + local
LIME local surrogate
PDP one-way & two-way
Diagnostics
Accuracy by segment
Stability & drift
Conformal prediction
Overfitting tests
Weak spot heatmaps
Robustness
Fairness · AIR · PR
| Dimension | Tests / Techniques | Regulatory Relevance |
|---|---|---|
| Data Quality | Missing value analysis, distributions, type checking | SR 26-2, TRIM |
| Feature Selection | Pearson/Spearman correlation, permutation importance | SR 26-2 |
| Model Performance | AUC, Accuracy, F1, R², MAE, RMSE | SR 26-2, Universal |
| Calibration | Conformal prediction coverage and efficiency | SR 26-2, SS1/23, IFRS 9 |
| Overfitting | Train/test gap, learning curves, feature robustness | SR 26-2, TRIM |
| Explainability | SHAP (global + local), LIME, PDP | SR 26-2, EU AI Act |
| Weak Spot | Decision-tree segmentation, accuracy heatmaps | SR 26-2 |
| Robustness | Input perturbation sensitivity | SR 26-2, DFAST |
| Fairness | AIR (EEOC 4/5ths), Precision Ratio, SMD | SR 26-2, ECOA, Fair Housing Act |
→ Full SR 26-2 compliance mapping
ml_validation_framework/
├── notebooks/
│ └── ML_Model_Validation.ipynb # Lightweight notebook — one call per step
├── src/
│ ├── __init__.py
│ └── validation.py # ValidationFramework class — all logic lives here
├── data/
│ ├── README.md # Dataset descriptions and sources
│ ├── cs-training.csv # GMSC Credit Risk (Give Me Some Credit)
│ └── UCI_Credit_Card.csv # Taiwan Credit Default dataset
├── requirements.txt
├── LICENSE
└── README.md
The notebook is intentionally thin — it imports ValidationFramework from src/validation.py and calls one method per step. All widget logic, model code, and visualisation is encapsulated in the source class, making it easy to extend or integrate into other projects.
| Category | Libraries |
|---|---|
| Data | pandas, numpy |
| Models | scikit-learn, lightgbm |
| Explainability | shap, lime, eli5 |
| Visualisation | matplotlib, seaborn, graphviz |
| UI | ipywidgets |
| Statistics | scipy |
MIT — see LICENSE for details.