nnfs — neural networks from scratch in NumPy

A small deep learning library with no deep learning dependency. Every forward pass, every analytic gradient and the training loop are derived by hand and written against NumPy alone — no autograd, no PyTorch, no TensorFlow.

The point is not to compete with a framework. It is to have nothing hidden: when a gradient is wrong here, there is no library to blame, and the test suite says so in relative error.

What's in it

Layers	Linear, ReLU, Dropout (inverted), Flatten, Conv2d, MaxPool2d
Losses	HingeLoss, MSELoss, L2 penalty
Optimiser	SGD with momentum and weight decay
Verification	central-difference gradient checking for every layer

How it works

Gradients are accumulated, not returned. A Module owns its Parameters, caches what its backward pass needs during forward, and adds into Parameter.grad. backward takes dL/d(output) and returns dL/d(input), so composing layers is just passing that value backwards — Sequential is a loop over reversed(children) and nothing more.

Convolution loops over kernel positions, not over images. For a fixed offset (i, j), every output pixel reads the input at a fixed stride-spaced slice. So a 3×3 convolution is nine vectorised operations covering the whole batch, rather than N × OH × OW scalar ones — the same idea as im2col without materialising the patch matrix. Both gradients fall out of the same loop, and dL/dx scatters back into those slices, where overlapping windows simply add. That addition is the chain rule for a variable used more than once.

Dropout is inverted. Scaling by 1/(1-p) at training time keeps the expected activation unchanged, so inference needs no rescaling and eval() is a genuine no-op instead of a second code path.

The gradient check uses a random linear functional. It differentiates sum(output * c) for fixed random c, not a plain sum. A plain sum contracts the Jacobian with the all-ones vector, which hides errors that cancel across output components. Central differences rather than forward, because the error is O(h²) instead of O(h) — six digits of agreement instead of three, and three is not enough to catch a gradient that is wrong by a constant factor.

Results

Reproduce everything with python examples/benchmark.py.

Gradient checks — max relative error against central finite differences:

Layer	input	weights	bias
Linear(5, 3)	1.40e-09	2.38e-10	1.22e-09
ReLU()	2.40e-10	—	—
Dropout(0.5)	1.56e-10	—	—
Flatten()	2.69e-08	—	—
Conv2d(3, 4, k=3, pad=1)	1.42e-08	3.18e-09	5.24e-11
Conv2d(3, 4, k=3, stride=2)	3.47e-09	1.44e-09	1.88e-11
MaxPool2d(2)	1.14e-07	—	—

Worst case across the library: 1.14e-07, on max-pooling — expected, since max is piecewise linear and a finite difference taken near a kink picks up more truncation error than a smooth function does.

Training — both datasets are generated, not downloaded, so runs are deterministic and need no network:

Model	Task	Loss	Train	Test	Time
2-32-16-2 MLP	two interleaving moons, 800/200	0.754 → 0.057	0.974	0.955	0.1s
conv8 → pool → conv16 → pool → fc	12×12 noisy shapes, 1200/300	1.262 → 0.065	0.983	0.947	2.4s

Both tasks are chosen so the number means something. Two-moons is not linearly separable, so a linear model caps out near 88% and anything above that is the hidden layer working. In the shape task, class 2 is exactly the union of classes 0 and 1 — identical pixel statistics, different spatial arrangement — so bag-of-pixels features cannot separate them and the convolution has to carry it. The noise is set to 0.5 deliberately: at 0.15 the CNN saturates at 100% and the score stops being informative.

Run it

pip install -r requirements-dev.txt   # numpy, pytest, ruff
pytest                                # 32 tests, including every gradient check
python examples/benchmark.py          # reproduces the tables above

Works straight from a clone — the root conftest.py puts the package on sys.path, since pytest deliberately leaves rootdir out of it. pip install -e . does the same thing and is what CI uses, because it exercises the packaging too.

Runtime dependency is NumPy alone; requirements.txt has just that.

Use it

import numpy as np
from nnfs import Sequential, Linear, ReLU, HingeLoss, SGD, minibatches, to_pm_one

model = Sequential(Linear(2, 32), ReLU(), Linear(32, 2))
loss = HingeLoss()
opt = SGD(model.parameters(), lr=0.05, momentum=0.9)

targets = to_pm_one(y, 2)
for epoch in range(60):
    for xb, tb in minibatches(x, targets, batch_size=64):
        opt.zero_grad()
        loss.forward(model.forward(xb), tb)
        model.backward(loss.backward())
        opt.step()

model.eval()
predictions = model.forward(x_test).argmax(axis=1)

Checking your own gradients:

from nnfs import check_module_gradients, Conv2d

errors = check_module_gradients(Conv2d(3, 8, 3, padding=1), np.random.randn(2, 3, 8, 8))
# {'input': 1.4e-08, 'param_0': 3.2e-09, 'param_1': 5.2e-11}

Stack

Python 3.9+, NumPy. pytest for the suite. Nothing else.

Licence

MIT.