diff --git a/.github/agents/orchestrator.agent.md b/.github/agents/orchestrator.agent.md index 12faa6ad..431a9845 100644 --- a/.github/agents/orchestrator.agent.md +++ b/.github/agents/orchestrator.agent.md @@ -33,7 +33,7 @@ Triage requests and route to the right specialist. Do NOT implement or plan your ## Context to gather - Module: `analysis`, `windowing`, `control_analysis`, `controllers`, `dynamics`, - `estimators`, `filters`, `filters::passive`, `math`, `neural_network`, + `estimators`, `filters`, `filters::passive`, `math`, `optimization`, `regularization`, `solvers` - Existing patterns to follow? - Documentation update needed? diff --git a/AGENTS.md b/AGENTS.md index 825cdd6f..c97f4979 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -51,7 +51,7 @@ resource-constrained embedded systems. Real-time, deterministic, no heap. `analysis`, `windowing`, `control_analysis`, `controllers`, `estimators`, `filters` (active Kalman family — **not** `filters::active`), `filters::passive`, `math`, -`neural_network`, `optimization`, `regularization`, `solvers`, and new: +`optimization`, `regularization`, `solvers`, and new: `robust_control`, `nonlinear_control`. ## Math functions diff --git a/CMakePresets.json b/CMakePresets.json index b1b695bb..74aa6918 100644 --- a/CMakePresets.json +++ b/CMakePresets.json @@ -135,10 +135,6 @@ "numerical.filters.active_test", "numerical.filters.passive_test", "numerical.math_test", - "numerical.neural_network.activation_test", - "numerical.neural_network.layer_test", - "numerical.neural_network.losses_test", - "numerical.neural_network.model_test", "numerical.nonlinear_control_test", "numerical.optimization_test", "numerical.regularization_test", @@ -160,10 +156,6 @@ "numerical.filters.active_test", "numerical.filters.passive_test", "numerical.math_test", - "numerical.neural_network.activation_test", - "numerical.neural_network.layer_test", - "numerical.neural_network.losses_test", - "numerical.neural_network.model_test", "numerical.nonlinear_control_test", "numerical.optimization_test", "numerical.regularization_test", diff --git a/README.md b/README.md index 2ea91d70..d9e057e1 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,6 @@ Refer to the documentation to quickly integrate and utilize the library's signal | [Controllers](doc/controllers/README.md) | Bang-Bang/Hysteresis, PID, LQR, LQI (Integral/Servo State Feedback), MPC, Saturation, Rate Limiter, Slew-Limited Saturation, Feedforward/2-DOF, Gain-Scheduled Controller, Lead-Lag Compensator, Luenberger Observer | | [Estimators](doc/estimators/README.md) | Linear Regression, Polynomial Fitting, Total Least Squares, Yule-Walker (offline), Recursive Least Squares, LMS / NLMS Adaptive Filter (online), Consistency Metrics / NEES / NIS | | [Filters](doc/filters/README.md) | Kalman, Extended Kalman, Unscented Kalman, Square-Root Kalman, Alpha-Beta/Alpha-Beta-Gamma, FIR, IIR, Exponential Moving Average, Moving Average, Complementary, Median Filter, CIC (Cascaded Integrator-Comb), Notch/Comb Filter, Savitzky-Golay Filter, Biquad/Second-Order-Section Cascade, IIR Filter Design (Butterworth/Chebyshev-I + Bilinear Transform), Madgwick/Mahony AHRS | -| [Neural Network](doc/neural_network/README.md) | Layers, activations, losses, model | | [Optimization](doc/optimization/README.md) | Gradient Descent | | [Regularization](doc/regularization/README.md) | L1 (Lasso), L2 (Ridge) | | [Math](doc/math/README.md) | CORDIC, Quaternion, MatrixNorms, Step Response Metrics, MatrixExponential | diff --git a/ROADMAP.md b/ROADMAP.md index a5d3f6ec..7dab252d 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -43,7 +43,7 @@ magnitude/phase. The items below are the missing pieces. All are **float-only**, on bounded `math::Vector`/`math::Matrix` inputs; tests are `TEST_F` on `float`. > **Test-only helpers (not production components).** A ULP/relative-error comparator and a -> finite-difference **gradient check** (for `neural_network/` and `optimization/`) are pure test +> finite-difference **gradient check** (for `optimization/`) are pure test > utilities — add them to the `numerical.math_test_helper` INTERFACE library > ([`numerical/math/test_doubles/`](numerical/math/test_doubles/)), not to `numerical/` production code. diff --git a/TESTING.md b/TESTING.md index 9b61f179..22d2d96b 100644 --- a/TESTING.md +++ b/TESTING.md @@ -61,7 +61,6 @@ Canonical rules still apply ([AGENTS.md](AGENTS.md), [testing.instructions.md](. | Optimization | ● | | | | ● | ● | | | | | Regularization | ● | | | | | ● | ● | | | | Solvers (linear / ODE / roots) | ● | | ○ | ● | ● | ● | ● | | ● | -| Neural network | ● | | | | ○ | ● | ● | | | | Math foundation | ● | | | | | ● | ● | | ● | ● primary ○ situational @@ -89,8 +88,6 @@ implementation against its own prior output. - **Transforms / filters** — known pairs (δ, sinusoid, DC); Butterworth/Chebyshev vs `scipy.signal`. - **Estimators / regression** — simulated known linear-Gaussian systems, AR processes with known coefficients, data on a known line/polynomial. Assert RMSE vs truth, NEES/NIS in χ² band, R². -- **Neural network** — hand-computed forward/backward passes; finite-difference gradient checks. - ### R2 — Unhappy flow (no exceptions here — errors are `std::optional`/status enums) Force **every** failure return and assert the *failure* value, not just success: non-SPD Cholesky, singular/rank-deficient solve, non-convergence at `maxIterations`, rank-deficient @@ -257,18 +254,7 @@ final value. within the requested tolerance. - **M7 conservation** — energy drift bounded for a conservative system over many steps. -### 10. Neural network — `neural_network/` -`activation/*`, `layer/Dense`, `losses/*`, `model/Model`. - -- **M1 activation values** — reference points: `sigmoid(0)=0.5`, `tanh(0)=0`, `relu(−x)=0`, - `leaky_relu` slope; output range bounds; monotonicity where expected. -- **M7 softmax** — outputs sum to 1 and are non-negative; shift-invariance. -- **M1 loss values & gradients** — MSE of identical vectors = 0; loss non-negative; analytic gradient - matches finite-difference (M1 gradient check). -- **M1 dense layer** — `output = W·x + b`; back-prop gradient check. -- **Model (M6)** — forward pass is deterministic and equals the manual layer composition. - -### 11. Math foundation — `math/` +### 10. Math foundation — `math/` `Matrix`, `ComplexNumber`, `Quaternion`, `Cordic`, `TrigonometricFunctions`, `HyperbolicFunctions`, `AdvancedFunctions`, `Statistics`, `LinearTimeInvariant`, `Toeplitz`, `QNumber`. diff --git a/doc/estimators/LinearRegression.md b/doc/estimators/LinearRegression.md index de0d547a..47c5576c 100644 --- a/doc/estimators/LinearRegression.md +++ b/doc/estimators/LinearRegression.md @@ -115,7 +115,7 @@ graph LR |-----------------------------------------------------------|--------------------------------------------------------------------------------------------| | [Gaussian Elimination](../solvers/GaussianElimination.md) | Used to solve the normal equation system | | [Yule-Walker](YuleWalker.md) | Structurally similar — also solves a linear system derived from correlations | -| [Neural Network](../neural_network/NeuralNetwork.md) | A single-layer neural network with no activation and MSE loss reduces to linear regression | +| Neural Network (neural-network-toobox-cpp) | A single-layer neural network with no activation and MSE loss reduces to linear regression | ## References & Further Reading diff --git a/doc/neural_network/NeuralNetwork.md b/doc/neural_network/NeuralNetwork.md deleted file mode 100644 index b8792e08..00000000 --- a/doc/neural_network/NeuralNetwork.md +++ /dev/null @@ -1,163 +0,0 @@ -# Neural Networks - -## Overview & Motivation - -A neural network is a parameterized function $f_\theta: \mathbb{R}^n \to \mathbb{R}^m$ built by composing simple, differentiable transformations called **layers**. Each layer applies an affine map followed by a non-linear activation, and the whole composition is trained end-to-end by gradient-based optimization. - -Neural networks are powerful because of the **universal approximation theorem**: a single hidden layer with enough neurons can approximate any continuous function on a compact set to arbitrary accuracy. In practice, *depth* (many layers) is more parameter-efficient than width for learning hierarchical features. - -This library provides a minimal, statically-sized neural network framework designed for **embedded inference and on-device training** — no heap allocation, no dynamic shapes, full compile-time dimension checking. - -## Mathematical Theory - -### Forward Propagation - -Given $L$ layers, the network computes: - -$$a_0 = x$$ -$$z_\ell = W_\ell \, a_{\ell-1} + b_\ell, \quad \ell = 1, \ldots, L$$ -$$a_\ell = f_\ell(z_\ell)$$ -$$\hat{y} = a_L$$ - -where $W_\ell \in \mathbb{R}^{n_\ell \times n_{\ell-1}}$ are weights, $b_\ell \in \mathbb{R}^{n_\ell}$ are biases, and $f_\ell$ is the activation function for layer $\ell$. - -### Loss Function - -Training minimizes a scalar loss $\mathcal{L}(\hat{y}, y)$ that measures how far the prediction $\hat{y}$ is from the target $y$. Common choices: - -| Loss | Formula | Use Case | -|------|--------------------------------------------------------|----------------------------| -| MSE | $\frac{1}{m}\sum(\hat{y}_i - y_i)^2$ | Regression | -| BCE | $-\sum[y_i \log \hat{y}_i + (1-y_i)\log(1-\hat{y}_i)]$ | Binary classification | -| CCE | $-\sum y_i \log \hat{y}_i$ | Multi-class classification | - -### Backpropagation - -Backpropagation efficiently computes $\nabla_\theta \mathcal{L}$ via the chain rule, working from the output layer backward: - -$$\delta_L = \nabla_{a_L}\mathcal{L} \odot f_L'(z_L)$$ -$$\delta_\ell = (W_{\ell+1}^T \delta_{\ell+1}) \odot f_\ell'(z_\ell)$$ - -The gradients with respect to parameters are: - -$$\frac{\partial \mathcal{L}}{\partial W_\ell} = \delta_\ell \, a_{\ell-1}^T, \qquad \frac{\partial \mathcal{L}}{\partial b_\ell} = \delta_\ell$$ - -### Parameter Update - -An optimizer uses the gradients to update the parameter vector $\theta$: - -$$\theta_{t+1} = \theta_t - \eta \, \nabla_\theta \mathcal{L}$$ - -where $\eta$ is the learning rate. More sophisticated optimizers (momentum, Adam) modify this basic rule. - -## Complexity Analysis - -| Phase | Time | Space | -|------------------|-----------------------------------------------------------|-------------------------------------------| -| Forward pass | $O\!\left(\sum_{\ell=1}^L n_\ell \cdot n_{\ell-1}\right)$ | $O\!\left(\sum n_\ell\right)$ activations | -| Backward pass | Same as forward | Same + gradient storage | -| Parameter update | $O(P)$ | $O(P)$ optimizer state | - -where $P = \sum_\ell (n_\ell \cdot n_{\ell-1} + n_\ell)$ is the total parameter count. For small embedded networks ($P < 10{,}000$), both passes complete in microseconds. - -## Step-by-Step Walkthrough - -**Network:** 2 inputs → 2 hidden (ReLU) → 1 output (Sigmoid). Learning XOR. - -**Architecture:** - -```mermaid -graph LR - x1((x₁)) --> h1((h₁)) - x1 --> h2((h₂)) - x2((x₂)) --> h1 - x2 --> h2 - h1 --> y((ŷ)) - h2 --> y -``` - -**Epoch 0 — Forward pass** with input $x = [1, 0]^T$, target $y = 1$: - -| Step | Computation | Result | -|-----------------------|--------------------------------------------------------|-----------------| -| Hidden pre-activation | $z_1 = W_1 x + b_1$ | $[0.3, -0.1]^T$ | -| Hidden activation | $a_1 = \text{ReLU}(z_1)$ | $[0.3, 0.0]^T$ | -| Output pre-activation | $z_2 = W_2 a_1 + b_2$ | $[0.15]$ | -| Output activation | $\hat{y} = \sigma(z_2)$ | $[0.537]$ | -| Loss | $\mathcal{L} = -(y\log\hat{y} + (1-y)\log(1-\hat{y}))$ | $0.621$ | - -**Epoch 0 — Backward pass:** - -| Step | Computation | Result | -|-----------------|-----------------------------------------------------|-------------------------| -| Output gradient | $\delta_2 = \hat{y} - y$ | $[-0.463]$ | -| $\nabla W_2$ | $\delta_2 \cdot a_1^T$ | $[-0.139, 0]$ | -| Hidden gradient | $\delta_1 = W_2^T \delta_2 \odot \text{ReLU}'(z_1)$ | $[-0.231, 0]^T$ | -| $\nabla W_1$ | $\delta_1 \cdot x^T$ | $[[-0.231, 0], [0, 0]]$ | - -**Update:** $W \leftarrow W - 0.1 \cdot \nabla W$. After ~500 epochs, the network correctly classifies all four XOR inputs. - -## Pitfalls & Edge Cases - -- **Vanishing gradients.** Deep networks with Sigmoid or Tanh activations suffer exponential gradient decay. Prefer ReLU-family activations in hidden layers. -- **Exploding gradients.** Large weights amplify gradients exponentially. Use proper weight initialization (Xavier/He) and gradient clipping. -- **Dead neurons.** ReLU neurons that receive only negative inputs output zero forever. LeakyReLU mitigates this. -- **Fixed-point saturation.** In Q15/Q31, activations and gradients must stay within $[-1, 1)$. Scale inputs and learning rates accordingly. -- **Learning rate sensitivity.** Too high → divergence; too low → no progress. Start with $\eta = 0.01$ and adjust. -- **Overfitting.** Small embedded datasets are easily memorized. Apply [regularization](../regularization/Regularization.md) (L2 weight decay). - -## Variants & Generalizations - -| Variant | Key Difference | -|----------------------------------------|-------------------------------------------------------------------------| -| **Convolutional Neural Network (CNN)** | Layers share weights spatially; efficient for image/signal data | -| **Recurrent Neural Network (RNN)** | Layers share weights across time steps; models sequences | -| **Residual Network (ResNet)** | Skip connections mitigate vanishing gradients in very deep networks | -| **Transformer** | Attention-based; no recurrence; state-of-the-art for sequences | -| **Quantized Neural Network** | Weights and activations in low-bit integers; optimal for MCU deployment | - -## Applications - -- **Function approximation** — Learning arbitrary input-output mappings from data. -- **Classification** — Mapping inputs to discrete categories (fault detection, gesture recognition). -- **Regression** — Predicting continuous values (sensor calibration, system identification). -- **Control** — Neural network policies for model-free or model-predictive control on embedded targets. -- **Signal processing** — Learned filters replacing hand-designed FIR/IIR chains. - -## Connections to Other Algorithms - -```mermaid -graph TD - NN["Neural Network"] - Layer["Dense Layer"] - Act["Activation Functions"] - Loss["Loss Functions"] - Opt["Optimizer"] - Reg["Regularization"] - Model["Model"] - LR["Linear Regression"] - - Layer --> NN - Act --> NN - Loss --> NN - Opt --> NN - Reg --> NN - NN --> Model - NN -.->|"single-layer, linear activation, MSE loss"| LR -``` - -| Component | Relationship | -|--------------------------------------------------------|-----------------------------------------------------------------| -| [Dense Layer](layer/Layer.md) | The fundamental building block; computes affine transformations | -| [Activation Functions](activation/Activation.md) | Introduce non-linearity after each layer | -| [Loss Functions](losses/Loss.md) | Define the training objective | -| [Optimizer](../optimization/Optimizer.md) | Drives parameter updates via gradient descent | -| [Regularization](../regularization/Regularization.md) | Penalizes complexity to prevent overfitting | -| [Model](model/Model.md) | Composes layers into a trainable pipeline | -| [Linear Regression](../estimators/LinearRegression.md) | Special case: single layer, identity activation, MSE loss | - -## References & Further Reading - -- Goodfellow, I., Bengio, Y., and Courville, A., *Deep Learning*, MIT Press, 2016 — Chapters 6–8. -- He, K. et al., "Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification", *ICCV*, 2015 — He initialization. -- Rumelhart, D.E., Hinton, G.E., and Williams, R.J., "Learning representations by back-propagating errors", *Nature*, 323, 1986. diff --git a/doc/neural_network/NeuralNetworkArchitecture.png b/doc/neural_network/NeuralNetworkArchitecture.png deleted file mode 100644 index c3817e9b..00000000 Binary files a/doc/neural_network/NeuralNetworkArchitecture.png and /dev/null differ diff --git a/doc/neural_network/README.md b/doc/neural_network/README.md deleted file mode 100644 index 0604db4c..00000000 --- a/doc/neural_network/README.md +++ /dev/null @@ -1,15 +0,0 @@ -# Neural Network - -Components for building, training, and evaluating feed-forward neural networks entirely on-device with static memory allocation. - -## Components - -| Component | Description | -|----------------------------------------------------|-----------------------------------------------------------------------------------------| -| [Neural Network Overview](NeuralNetwork.md) | Architecture, training loop, and how the components fit together | -| [Layers](layer/Layer.md) | Dense (fully connected) layer with forward and backward pass | -| [Activation Functions](activation/Activation.md) | Non-linear element-wise transforms: ReLU, LeakyReLU, Sigmoid, Tanh, Softmax | -| [Loss Functions](losses/Loss.md) | Objective functions: MSE, MAE, Binary Cross-Entropy, Categorical Cross-Entropy | -| [Model](model/Model.md) | Variadic-template model composing layers, optimizer, and loss into a trainable pipeline | - -> **See also:** [Optimization](../optimization/README.md) for optimizers (Gradient Descent) and [Regularization](../regularization/README.md) for weight penalties (L1, L2) — these are general-purpose domains used by neural networks and other algorithms. diff --git a/doc/neural_network/activation/Activation.md b/doc/neural_network/activation/Activation.md deleted file mode 100644 index 6bd43ae8..00000000 --- a/doc/neural_network/activation/Activation.md +++ /dev/null @@ -1,149 +0,0 @@ -# Activation Functions - -## Overview & Motivation - -An activation function $f$ is a non-linear, element-wise transformation applied after the affine map in each neural network layer: - -$$a = f(z) = f(W x + b)$$ - -Without activation functions, stacking layers would collapse into a single affine transformation — the network could only represent linear mappings regardless of depth. Activation functions are what give neural networks their expressive power. - -The choice of activation function controls **gradient flow** during back-propagation, **output range**, and **computational cost** — all critical on resource-constrained embedded targets. - -## Mathematical Theory - -### Forward and Backward - -Every activation function exposes two operations: - -| Operation | Definition | Purpose | -|--------------|------------|---------------------------------------------------| -| **Forward** | $a = f(z)$ | Transform the pre-activation | -| **Backward** | $f'(z)$ | Provide the local derivative for back-propagation | - -The chain rule connects them during training: - -$$\frac{\partial \mathcal{L}}{\partial z} = \frac{\partial \mathcal{L}}{\partial a} \cdot f'(z)$$ - -### Catalogue - -#### ReLU (Rectified Linear Unit) - -$$f(z) = \max(0, z), \qquad f'(z) = \begin{cases} 1 & z > 0 \\ 0 & z \le 0 \end{cases}$$ - -- Computationally cheapest (single comparison). -- Sparse activations accelerate training. -- **Risk:** neurons with $z \le 0$ for all inputs are permanently dead. - -#### Leaky ReLU - -$$f(z) = \begin{cases} z & z > 0 \\ \alpha z & z \le 0 \end{cases}, \qquad f'(z) = \begin{cases} 1 & z > 0 \\ \alpha & z \le 0 \end{cases}$$ - -where $\alpha$ is a small positive constant (typically $0.01$). Prevents dead neurons by allowing a small gradient for $z < 0$. - -#### Sigmoid - -$$f(z) = \frac{1}{1 + e^{-z}}, \qquad f'(z) = f(z)(1 - f(z))$$ - -- Output range $(0, 1)$ — natural for probabilities. -- **Risk:** saturates for $|z| \gg 0$, causing vanishing gradients. - -#### Tanh (Hyperbolic Tangent) - -$$f(z) = \tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}, \qquad f'(z) = 1 - f(z)^2$$ - -- Output range $(-1, 1)$ — zero-centered, unlike Sigmoid. -- Same saturation problem as Sigmoid at extremes. - -#### Softmax - -For a vector $\mathbf{z} \in \mathbb{R}^k$: - -$$f(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{k} e^{z_j}}$$ - -- Outputs form a probability distribution ($\sum f(z_i) = 1$). -- Used exclusively at the output layer for multi-class classification. -- The Jacobian is a full $k \times k$ matrix — more expensive than element-wise activations. - -## Complexity Analysis - -| Activation | Forward (per element) | Backward (per element) | Notes | -|------------|--------------------------------|-------------------------------|------------------------------| -| ReLU | $O(1)$ — comparison | $O(1)$ — comparison | Fastest | -| Leaky ReLU | $O(1)$ — comparison + multiply | $O(1)$ | Negligible overhead vs ReLU | -| Sigmoid | $O(1)$ — exp + divide | $O(1)$ — reuse forward result | Requires `exp()` | -| Tanh | $O(1)$ — exp (twice) | $O(1)$ — reuse forward result | Requires `exp()` | -| Softmax | $O(k)$ — vector exp + sum | $O(k^2)$ — full Jacobian | Significantly more expensive | - -## Step-by-Step Walkthrough - -**Scenario:** Forward and backward pass through a 3-neuron hidden layer with ReLU, given pre-activations $z = [-0.5, \; 1.2, \; 0.0]$. - -**Forward:** - -| Neuron | $z$ | $\text{ReLU}(z)$ | -|--------|--------|------------------| -| 1 | $-0.5$ | $0.0$ | -| 2 | $1.2$ | $1.2$ | -| 3 | $0.0$ | $0.0$ | - -**Backward** with incoming gradient $\frac{\partial \mathcal{L}}{\partial a} = [0.3, \; -0.7, \; 0.1]$: - -| Neuron | $f'(z)$ | $\frac{\partial \mathcal{L}}{\partial z}$ | -|--------|-------------------|-------------------------------------------| -| 1 | $0$ (dead) | $0.3 \times 0 = 0$ | -| 2 | $1$ | $-0.7 \times 1 = -0.7$ | -| 3 | $0$ (at boundary) | $0.1 \times 0 = 0$ | - -Neuron 1 is *dead* — its gradient is zero and its weights will not update. If this persists across all training samples, the neuron is permanently inactive. - -## Pitfalls & Edge Cases - -- **Vanishing gradients with Sigmoid/Tanh.** For deep networks, gradients shrink exponentially through saturated activations. Use ReLU or Leaky ReLU in hidden layers. -- **Dead ReLU neurons.** If a neuron's bias drifts negative enough that no input ever produces $z > 0$, it stops learning. Leaky ReLU or careful initialization (He init) prevents this. -- **Softmax numerical instability.** Computing $e^{z_i}$ directly overflows for large $z_i$. Subtract $\max(\mathbf{z})$ before exponentiation: $f(z_i) = e^{z_i - z_{\max}} / \sum e^{z_j - z_{\max}}$. -- **Fixed-point range.** Sigmoid outputs $(0, 1)$ and Tanh outputs $(-1, 1)$ both fit in Q15/Q31, but the exponential intermediate values do not. Compute in floating-point and convert. -- **Non-differentiable points.** ReLU is technically non-differentiable at $z = 0$. In practice, assigning $f'(0) = 0$ works fine. - -## Variants & Generalizations - -| Variant | Key Difference | -|------------------------------|---------------------------------------------------------| -| **PReLU (Parametric ReLU)** | $\alpha$ is a learnable parameter per channel | -| **ELU (Exponential LU)** | $\alpha(e^z - 1)$ for $z < 0$; smooth and zero-centered | -| **GELU (Gaussian Error LU)** | $z \cdot \Phi(z)$; used in Transformers | -| **Swish / SiLU** | $z \cdot \sigma(z)$; smooth, non-monotonic | -| **Hard Sigmoid / Hard Tanh** | Piece-wise linear approximations; no transcendentals | - -## Applications - -- **Hidden layers** — ReLU (or Leaky ReLU) is the default choice for feed-forward and convolutional hidden layers. -- **Binary classification output** — Sigmoid maps the output to a probability. -- **Multi-class classification output** — Softmax produces a probability distribution. -- **Recurrent networks** — Tanh is traditionally used in LSTM/GRU gates. -- **Embedded inference** — Hard Sigmoid/Tanh avoid expensive `exp()` calls on MCUs without FPU. - -## Connections to Other Algorithms - -```mermaid -graph TD - Act["Activation Functions"] - Layer["Dense Layer"] - NN["Neural Network"] - Loss["Loss Functions"] - Act --> Layer - Layer --> NN - NN --> Loss -``` - -| Component | Relationship | -|---------------------------------------|------------------------------------------------------------------------------------------| -| [Dense Layer](../layer/Layer.md) | Applies the activation function after the affine transformation | -| [Neural Network](../NeuralNetwork.md) | Activations enable the non-linear function approximation that makes deep networks useful | -| [Loss Functions](../losses/Loss.md) | The output activation must match the loss: Sigmoid + BCE, Softmax + CCE | - -## References & Further Reading - -- Nair, V. and Hinton, G.E., "Rectified linear units improve restricted Boltzmann machines", *ICML*, 2010. -- Glorot, X., Bordes, A., and Bengio, Y., "Deep sparse rectifier neural networks", *AISTATS*, 2011. -- Clevert, D.-A., Unterthiner, T., and Hochreiter, S., "Fast and accurate deep network learning by exponential linear units (ELUs)", *ICLR*, 2016. diff --git a/doc/neural_network/activation/ActivationFunctions.svg b/doc/neural_network/activation/ActivationFunctions.svg deleted file mode 100644 index 0aa1c7d8..00000000 --- a/doc/neural_network/activation/ActivationFunctions.svg +++ /dev/null @@ -1,117 +0,0 @@ - - - - - - Activation Functions - - - - - - x - - - - f(x) - - - - -2 - - - -1 - - - 0 - - - 1 - - - 2 - - - - 1 - - - -1 - - - - - - - ReLU - - - - LeakyReLU - - - - Sigmoid - - - - Tanh - - - - - - - - - z - - - - f(z) - - - - a - - - - - - Next Layer - - - - - - - - - - - - - - - - - - - - - - Properties - - - Simple - - - No dead neurons - - - Bounded [0,1] - - - Zero-centered - - \ No newline at end of file diff --git a/doc/neural_network/layer/Layer.md b/doc/neural_network/layer/Layer.md deleted file mode 100644 index 6ffe46ec..00000000 --- a/doc/neural_network/layer/Layer.md +++ /dev/null @@ -1,127 +0,0 @@ -# Dense Layer - -## Overview & Motivation - -A **dense** (fully-connected) layer is the most fundamental building block of a neural network. It maps an input vector $a_{\text{in}} \in \mathbb{R}^n$ to an output vector $a_{\text{out}} \in \mathbb{R}^m$ through a learnable affine transformation followed by a non-linear activation: - -$$a_{\text{out}} = f(W \, a_{\text{in}} + b)$$ - -Every input neuron is connected to every output neuron — hence "fully connected." The layer's **parameters** are the weight matrix $W$ and bias vector $b$; training adjusts these to minimize the loss. - -In this library, input size, output size, and parameter count are all **compile-time constants**, enabling stack allocation and dimension checking with zero runtime overhead. - -## Mathematical Theory - -### Forward Pass - -$$z = W \, a_{\text{in}} + b, \qquad a_{\text{out}} = f(z)$$ - -where $W \in \mathbb{R}^{m \times n}$, $b \in \mathbb{R}^m$, and $f$ is the [activation function](../activation/Activation.md). - -### Backward Pass - -Given the gradient of the loss with respect to the output $\frac{\partial \mathcal{L}}{\partial a_{\text{out}}}$: - -1. **Pre-activation gradient:** -$$\delta = \frac{\partial \mathcal{L}}{\partial a_{\text{out}}} \odot f'(z)$$ - -2. **Weight gradient:** -$$\frac{\partial \mathcal{L}}{\partial W} = \delta \, a_{\text{in}}^T$$ - -3. **Bias gradient:** -$$\frac{\partial \mathcal{L}}{\partial b} = \delta$$ - -4. **Input gradient** (propagated to the previous layer): -$$\frac{\partial \mathcal{L}}{\partial a_{\text{in}}} = W^T \delta$$ - -### Parameter Count - -$$P = m \times n + m = m(n + 1)$$ - -For a layer with 128 inputs and 64 outputs: $P = 64 \times 129 = 8{,}256$ parameters. - -## Complexity Analysis - -| Operation | Time | Space | -|---------------------------------------------|----------------|-----------------------------------| -| Forward ($W a + b$) | $O(m \cdot n)$ | $O(m)$ output + $O(m)$ cached $z$ | -| Backward ($\delta$, $\nabla W$, $\nabla b$) | $O(m \cdot n)$ | $O(m \cdot n)$ weight gradient | -| Total parameters | — | $O(m \cdot n + m)$ | - -The matrix-vector product dominates both passes. For embedded networks (e.g. $n = 32, m = 16$), a single forward pass takes ~512 multiply-accumulate operations. - -## Step-by-Step Walkthrough - -**Layer:** 3 inputs → 2 outputs, ReLU activation. - -$$W = \begin{bmatrix} 0.5 & -0.3 & 0.8 \\ 0.1 & 0.7 & -0.2 \end{bmatrix}, \quad b = \begin{bmatrix} 0.1 \\ -0.1 \end{bmatrix}, \quad a_{\text{in}} = \begin{bmatrix} 1.0 \\ 0.5 \\ -1.0 \end{bmatrix}$$ - -**Forward:** - -| Step | Computation | Result | -|-----------------------------------|------------------------------------------------------|---------------------| -| $z = W a_{\text{in}} + b$ | $[0.5 - 0.15 - 0.8 + 0.1,\; 0.1 + 0.35 + 0.2 - 0.1]$ | $[-0.35,\; 0.55]^T$ | -| $a_{\text{out}} = \text{ReLU}(z)$ | $[\max(0, -0.35),\; \max(0, 0.55)]$ | $[0.0,\; 0.55]^T$ | - -**Backward** with $\frac{\partial \mathcal{L}}{\partial a_{\text{out}}} = [0.2,\; -0.4]^T$: - -| Step | Computation | Result | -|--------------------------------------------------------|-----------------------------------------------------|------------------------------------------------------------| -| $\delta = \nabla a_{\text{out}} \odot \text{ReLU}'(z)$ | $[0.2 \cdot 0,\; -0.4 \cdot 1]$ | $[0,\; -0.4]^T$ | -| $\nabla W = \delta \, a_{\text{in}}^T$ | row 1: all zeros; row 2: $-0.4 \times [1, 0.5, -1]$ | $\begin{bmatrix}0 & 0 & 0\\-0.4 & -0.2 & 0.4\end{bmatrix}$ | -| $\nabla b = \delta$ | — | $[0,\; -0.4]^T$ | -| $\nabla a_{\text{in}} = W^T \delta$ | $W^T [0, -0.4]^T$ | $[-0.04,\; -0.28,\; 0.08]^T$ | - -## Pitfalls & Edge Cases - -- **Dimension mismatch.** In a multi-layer network, the output size of layer $\ell$ must equal the input size of layer $\ell+1$. This library enforces this at compile time. -- **Weight initialization.** Zero-initialized weights cause all neurons to compute the same thing (symmetry problem). Use He initialization for ReLU: $W_{ij} \sim \mathcal{N}(0, \sqrt{2/n})$. -- **Gradient accumulation.** When processing mini-batches, accumulate $\nabla W$ across samples before updating — do not update per-sample. -- **Fixed-point overflow in $W a$.** The dot product of $n$ terms can exceed Q15/Q31 range. Use a wider accumulator (Q31 for Q15 data) or scale weights. -- **Large layers exhaust stack.** A 256×256 weight matrix of `float` consumes 256 KB. Size layers to fit the target's stack budget. - -## Variants & Generalizations - -| Variant | Key Difference | -|-------------------------------|-----------------------------------------------------------------------------------------------------------| -| **Convolutional layer** | Weight sharing across spatial positions; $O(k^2 \cdot c)$ parameters per filter instead of $O(n \cdot m)$ | -| **Recurrent layer** | Shares weights across time steps; adds a hidden state feedback connection | -| **Batch normalization layer** | Normalizes activations to zero mean and unit variance; accelerates training | -| **Dropout layer** | Randomly zeros activations during training; regularization effect | -| **Sparse layer** | Only a subset of connections exist; reduces parameter count and computation | - -## Applications - -- **Hidden layers** — One or more dense layers form the core of feed-forward networks for regression and classification. -- **Output layer** — A final dense layer maps to the target dimensionality (1 for regression, $k$ for $k$-class classification). -- **Embedding projection** — Dense layers project high-dimensional sparse inputs to low-dimensional dense representations. -- **Controller networks** — In neural network-based control, small dense layers map state vectors to actuator commands. - -## Connections to Other Algorithms - -```mermaid -graph TD - Layer["Dense Layer"] - Act["Activation Functions"] - Model["Model"] - Opt["Optimizer"] - LR["Linear Regression"] - - Act --> Layer - Layer --> Model - Model --> Opt - Layer -.->|"no activation, MSE loss"| LR -``` - -| Component | Relationship | -|-----------------------------------------------------------|----------------------------------------------------------------------------------------| -| [Activation Functions](../activation/Activation.md) | Applied element-wise after the affine transformation | -| [Model](../model/Model.md) | Chains multiple dense layers into a network | -| [Optimizer](../../optimization/Optimizer.md) | Updates $W$ and $b$ using the computed gradients | -| [Linear Regression](../../estimators/LinearRegression.md) | A dense layer with identity activation and MSE loss is equivalent to linear regression | - -## References & Further Reading - -- Goodfellow, I., Bengio, Y., and Courville, A., *Deep Learning*, MIT Press, 2016 — Chapter 6. -- He, K. et al., "Delving deep into rectifiers", *ICCV*, 2015 — He weight initialization. -- Glorot, X. and Bengio, Y., "Understanding the difficulty of training deep feedforward neural networks", *AISTATS*, 2010 — Xavier initialization. diff --git a/doc/neural_network/layer/Layers.svg b/doc/neural_network/layer/Layers.svg deleted file mode 100644 index ffceb246..00000000 --- a/doc/neural_network/layer/Layers.svg +++ /dev/null @@ -1,191 +0,0 @@ - - - - - - Neural Network Layers - - - - - - - Dense Layer Architecture - - - - Input - - - - - x₁...xₙ - - - - - - - - - - - - - - - - - - - - - - - - - - Weights - W₁₁...Wᵢⱼ - - - - - Pre-activation - - - - - z = Wx + b - - - - - - - - - Biases - - - - - Activation - - f(z) - (ReLU, Sigmoid, etc.) - - - - - - - - - - - Output - - - - - a = f(z) - - - - - - - - - - - Next Layer - - - - - - - - - - - - - - - Forward Pass - - - - - - - - Backpropagation Through Layer - - - - - - ∂L/∂a - - - - ∂L/∂z = ∂L/∂a ⊙ f'(z) - - - - - - - - - - - - - - - - - - - - ∂L/∂W = ∂L/∂z · x^T - - - - ∂L/∂b = ∂L/∂z - - - - ∂L/∂x = W^T · ∂L/∂z - - - - - For Optimizer - - - - To Previous Layer - - - - - - Backward Pass - - - - - - - - - - - - \ No newline at end of file diff --git a/doc/neural_network/losses/Loss.md b/doc/neural_network/losses/Loss.md deleted file mode 100644 index 8b938641..00000000 --- a/doc/neural_network/losses/Loss.md +++ /dev/null @@ -1,139 +0,0 @@ -# Loss Functions - -## Overview & Motivation - -A loss function $\mathcal{L}(\hat{y}, y)$ quantifies how far a model's prediction $\hat{y}$ is from the true target $y$. Training a neural network means finding parameters $\theta$ that minimize the expected loss over the training data: - -$$\theta^* = \arg\min_\theta \; \mathbb{E}[\mathcal{L}(f_\theta(x), y)]$$ - -The loss function defines the **entire learning objective** — different losses lead to different optimal models even on the same data. It must also provide a **gradient** $\nabla_{\hat{y}} \mathcal{L}$ for back-propagation. - -## Mathematical Theory - -### Mean Squared Error (MSE) - -$$\mathcal{L}_{\text{MSE}} = \frac{1}{m} \sum_{i=1}^{m} (\hat{y}_i - y_i)^2$$ - -$$\frac{\partial \mathcal{L}}{\partial \hat{y}_i} = \frac{2}{m}(\hat{y}_i - y_i)$$ - -- Penalizes large errors quadratically → sensitive to outliers. -- Natural choice for **regression** tasks. -- Optimal when the error distribution is Gaussian. - -### Mean Absolute Error (MAE) - -$$\mathcal{L}_{\text{MAE}} = \frac{1}{m} \sum_{i=1}^{m} |\hat{y}_i - y_i|$$ - -$$\frac{\partial \mathcal{L}}{\partial \hat{y}_i} = \frac{1}{m} \mathrm{sign}(\hat{y}_i - y_i)$$ - -- Linear penalty → robust to outliers. -- Non-differentiable at $\hat{y}_i = y_i$ (use sub-gradient in practice). - -### Binary Cross-Entropy (BCE) - -$$\mathcal{L}_{\text{BCE}} = -\frac{1}{m}\sum_{i=1}^{m} \left[ y_i \log \hat{y}_i + (1 - y_i) \log(1 - \hat{y}_i) \right]$$ - -$$\frac{\partial \mathcal{L}}{\partial \hat{y}_i} = -\frac{1}{m}\left(\frac{y_i}{\hat{y}_i} - \frac{1 - y_i}{1 - \hat{y}_i}\right)$$ - -- Assumes $\hat{y}_i \in (0, 1)$ — pair with **Sigmoid** output activation. -- Maximum likelihood estimator for Bernoulli-distributed targets. - -### Categorical Cross-Entropy (CCE) - -$$\mathcal{L}_{\text{CCE}} = -\sum_{i=1}^{k} y_i \log \hat{y}_i$$ - -$$\frac{\partial \mathcal{L}}{\partial \hat{y}_i} = -\frac{y_i}{\hat{y}_i}$$ - -- Assumes $\hat{y}$ is a probability distribution ($\sum \hat{y}_i = 1$) — pair with **Softmax** output activation. -- Standard loss for **multi-class classification**. - -## Complexity Analysis - -| Loss | Forward | Backward | Notes | -|------|---------|----------|------------------------------| -| MSE | $O(m)$ | $O(m)$ | Cheapest; no transcendentals | -| MAE | $O(m)$ | $O(m)$ | Requires `sign()` | -| BCE | $O(m)$ | $O(m)$ | Requires `log()` | -| CCE | $O(k)$ | $O(k)$ | Requires `log()` | - -All losses are $O(m)$ where $m$ is the output dimension. The computational cost is negligible compared to the dense layer matrix products. - -## Step-by-Step Walkthrough - -**Scenario:** 3-class classification. Target $y = [0, 1, 0]$ (class 2). Softmax output $\hat{y} = [0.1, 0.7, 0.2]$. - -**CCE Forward:** - -$$\mathcal{L} = -(0 \cdot \log 0.1 + 1 \cdot \log 0.7 + 0 \cdot \log 0.2) = -\log(0.7) \approx 0.357$$ - -**CCE Backward:** - -| $i$ | $y_i$ | $\hat{y}_i$ | $\partial \mathcal{L}/\partial \hat{y}_i = -y_i / \hat{y}_i$ | -|-----|-------|-------------|--------------------------------------------------------------| -| 1 | 0 | 0.1 | $0$ | -| 2 | 1 | 0.7 | $-1.429$ | -| 3 | 0 | 0.2 | $0$ | - -The gradient is non-zero only for the true class, and its magnitude $1/\hat{y}_2$ grows as the prediction *worsens* — providing a strong corrective signal. - -**For comparison — MSE on the same example:** - -$$\mathcal{L}_{\text{MSE}} = \frac{1}{3}[(0.1)^2 + (0.7-1)^2 + (0.2)^2] = \frac{1}{3}[0.01 + 0.09 + 0.04] = 0.047$$ - -MSE gives a much weaker signal and does not account for the probabilistic nature of the output. - -## Pitfalls & Edge Cases - -- **$\log(0)$ is $-\infty$.** Clamp predictions to $[\varepsilon, 1 - \varepsilon]$ before computing BCE or CCE. A typical $\varepsilon = 10^{-7}$. -- **Mismatched activation and loss.** Sigmoid + CCE or Softmax + BCE produce incorrect gradients. Always pair: Sigmoid ↔ BCE, Softmax ↔ CCE. -- **MSE for classification.** MSE can train a classifier but converges slower than cross-entropy because its gradient does not account for the log-likelihood structure. -- **Label encoding.** CCE expects one-hot encoded targets. Integer labels must be converted first. -- **Fixed-point log.** The `log()` function needed for cross-entropy is expensive and ill-conditioned near zero in fixed-point. Evaluate in floating-point. - -## Variants & Generalizations - -| Variant | Key Difference | -|----------------------|----------------------------------------------------------------------------| -| **Huber loss** | Quadratic for small errors, linear for large; robust regression | -| **Focal loss** | Down-weights well-classified examples; addresses class imbalance | -| **KL divergence** | Measures distance between two distributions; used in variational inference | -| **Hinge loss** | Margin-based; used in SVMs and some neural classifiers | -| **Contrastive loss** | Learns similarity metrics; used in Siamese networks | - -## Applications - -- **Regression** — MSE for Gaussian noise, MAE for Laplacian noise or when outlier robustness is needed. -- **Binary classification** — BCE for yes/no decisions (fault detection, anomaly flagging). -- **Multi-class classification** — CCE for mutually exclusive categories (gesture recognition, signal type identification). -- **Probabilistic output** — Cross-entropy losses calibrate prediction confidence, not just accuracy. - -## Connections to Other Algorithms - -```mermaid -graph TD - Loss["Loss Functions"] - Act["Activation Functions"] - Opt["Optimizer"] - Reg["Regularization"] - Model["Model"] - LR["Linear Regression"] - - Act -->|"output activation must match loss"| Loss - Loss --> Opt - Reg -->|"added to loss"| Loss - Loss --> Model - Loss -.->|"MSE + normal equation"| LR -``` - -| Component | Relationship | -|-----------------------------------------------------------|------------------------------------------------------------------------------------------------------| -| [Activation Functions](../activation/Activation.md) | Output activation must match: Sigmoid ↔ BCE, Softmax ↔ CCE, identity ↔ MSE | -| [Optimizer](../../optimization/Optimizer.md) | Uses $\nabla \mathcal{L}$ to update parameters | -| [Regularization](../../regularization/Regularization.md) | Adds a penalty term to the loss: $\mathcal{L}_{\text{total}} = \mathcal{L} + \lambda \Omega(\theta)$ | -| [Linear Regression](../../estimators/LinearRegression.md) | Solved analytically when the loss is MSE and the model is linear | - -## References & Further Reading - -- Goodfellow, I., Bengio, Y., and Courville, A., *Deep Learning*, MIT Press, 2016 — Chapter 6.2 (cost functions). -- Bishop, C.M., *Pattern Recognition and Machine Learning*, Springer, 2006 — Chapter 4.3 (cross-entropy). -- Lin, T.-Y. et al., "Focal loss for dense object detection", *ICCV*, 2017. diff --git a/doc/neural_network/losses/LossFunctions.svg b/doc/neural_network/losses/LossFunctions.svg deleted file mode 100644 index 128c1dfe..00000000 --- a/doc/neural_network/losses/LossFunctions.svg +++ /dev/null @@ -1,141 +0,0 @@ - - - - - - Loss Functions - - - - - - Prediction - - - - Loss - - - - Target - - - - - - - MSE: (ŷ-y)² - - - - MAE: |ŷ-y| - - - - BCE: -y·log(ŷ)-(1-y)·log(1-ŷ) - - - - CCE: -Σy·log(ŷ) - - - - - - - - - - - - - Input - - - - - - - Hidden - - - - - - Output ŷ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Loss L(ŷ,y) - - - - Target y - - - - L - - - - Gradient ∇L - - - - - - - - - - - - Backpropagation - - - - - - - - - - \ No newline at end of file diff --git a/doc/neural_network/model/Model.md b/doc/neural_network/model/Model.md deleted file mode 100644 index f7602cb8..00000000 --- a/doc/neural_network/model/Model.md +++ /dev/null @@ -1,154 +0,0 @@ -# Model (Neural Network Composition) - -## Overview & Motivation - -A **Model** composes a sequence of [dense layers](../layer/Layer.md) into a single trainable function $f: \mathbb{R}^n \to \mathbb{R}^m$. It is the orchestrator that: - -1. Chains layers so the output of each feeds into the next (**forward pass**). -2. Propagates gradients backward through the chain (**backward pass**). -3. Flattens all layer parameters into a single vector for the [optimizer](../../optimization/Optimizer.md). -4. Verifies dimensional compatibility **at compile time** using variadic templates. - -In this library, the Model is fully statically typed — layer dimensions, parameter counts, and memory footprints are all known at compile time, enabling zero-overhead abstraction on embedded targets. - -## Mathematical Theory - -### Composition - -For $L$ layers with transformations $f_1, f_2, \ldots, f_L$: - -$$\hat{y} = (f_L \circ f_{L-1} \circ \cdots \circ f_1)(x) = f_L(f_{L-1}(\ldots f_1(x) \ldots))$$ - -Each $f_\ell$ is a dense layer: $f_\ell(a) = \sigma_\ell(W_\ell a + b_\ell)$. - -### Parameter Vector - -All weights and biases are concatenated into a single vector: - -$$\theta = [\text{vec}(W_1), b_1, \text{vec}(W_2), b_2, \ldots, \text{vec}(W_L), b_L] \in \mathbb{R}^P$$ - -where $P = \sum_{\ell=1}^L m_\ell(n_\ell + 1)$. - -### Forward Pass (Chained Evaluation) - -```mermaid -graph LR - X["x ∈ ℝⁿ"] --> L1["Layer 1"] --> L2["Layer 2"] --> Ldots["⋯"] --> LL["Layer L"] --> Y["ŷ ∈ ℝᵐ"] -``` - -### Backward Pass (Reverse Chain Rule) - -$$\frac{\partial \mathcal{L}}{\partial \theta_\ell} = \frac{\partial \mathcal{L}}{\partial a_L} \cdot \frac{\partial a_L}{\partial a_{L-1}} \cdots \frac{\partial a_{\ell+1}}{\partial a_\ell} \cdot \frac{\partial a_\ell}{\partial \theta_\ell}$$ - -Each layer stores its input $a_{\ell-1}$ during the forward pass so it can compute $\nabla W_\ell$ and $\nabla b_\ell$ during the backward pass. - -### Training Loop - -```mermaid -graph TD - FP["Forward pass: ŷ = Model(x)"] - LC["Loss: ℒ(ŷ, y)"] - BP["Backward pass: ∇θ ℒ"] - UP["Update: θ ← θ − η ∇θ ℒ"] - FP --> LC --> BP --> UP --> FP -``` - -## Complexity Analysis - -| Operation | Time | Space | -|-------------------|--------|-------------------------------------| -| Forward pass | $O(P)$ | $O(\sum n_\ell)$ cached activations | -| Backward pass | $O(P)$ | $O(P)$ gradients | -| `GetParameters()` | $O(P)$ | $O(P)$ flat vector | -| `SetParameters()` | $O(P)$ | — | - -All operations scale linearly with the total parameter count $P$. - -## Step-by-Step Walkthrough - -**Model:** 2 → 3 → 1 (two layers). - -**Compile-time verification chain:** - -| Check | Condition | Status | -|------------------------------------------|------------------|--------| -| Layer 1 input size = Model input size | $2 = 2$ | ✓ | -| Layer 1 output size = Layer 2 input size | $3 = 3$ | ✓ | -| Layer 2 output size = Model output size | $1 = 1$ | ✓ | -| All types derive from `Layer` | type trait check | ✓ | - -**Parameter layout** ($P = 3(2 + 1) + 1(3 + 1) = 9 + 4 = 13$): - -| Index | Parameter | -|-------|--------------------------| -| 0–5 | $W_1$ (3×2 = 6 elements) | -| 6–8 | $b_1$ (3 elements) | -| 9–11 | $W_2$ (1×3 = 3 elements) | -| 12 | $b_2$ (1 element) | - -**Forward pass** with $x = [1.0, 0.5]^T$: - -1. Layer 1: $a_1 = \text{ReLU}(W_1 x + b_1) = [0.0, 0.8, 0.3]^T$ -2. Layer 2: $\hat{y} = \sigma(W_2 a_1 + b_2) = [0.62]$ - -**Backward pass** with loss gradient $\delta_{\text{out}} = [0.12]$: - -1. Layer 2 backward → produces $\nabla W_2$, $\nabla b_2$, and $\delta_1 = W_2^T \delta_2 \odot \text{ReLU}'(z_1)$ -2. Layer 1 backward → produces $\nabla W_1$, $\nabla b_1$ - -**Optimizer** receives the full $\nabla \theta \in \mathbb{R}^{13}$ and updates $\theta$. - -## Pitfalls & Edge Cases - -- **Dimension mismatch caught at compile time.** If layer $\ell$ outputs $m$ but layer $\ell+1$ expects $n \ne m$, a `static_assert` fires during compilation. -- **Empty model.** The variadic template requires at least one layer. `static_assert(sizeof...(Layers) > 0)`. -- **Parameter ordering.** `GetParameters()` and `SetParameters()` must use the same concatenation order. The implementation iterates layers via `std::index_sequence` to guarantee consistency. -- **Training with wrong optimizer size.** The optimizer's parameter count template argument must equal `Model::TotalParameters`. A mismatch is also caught at compile time. -- **Large parameter vectors.** All parameters live on the stack. A model with $P > 10{,}000$ floats consumes 40 KB — verify this fits the target's stack. - -## Variants & Generalizations - -| Variant | Key Difference | -|--------------------------------|------------------------------------------------------------------------------------| -| **Sequential model (dynamic)** | Layers stored in a container; dimension checked at runtime instead of compile time | -| **Functional API** | Supports branching and merging (DAG topology instead of linear chain) | -| **Residual model** | Adds skip connections: $a_{\ell+2} = f_{\ell+1}(a_\ell) + a_\ell$ | -| **Recurrent model** | Unrolls the same layer across time steps | - -## Applications - -- **Embedded inference** — A pre-trained model's parameters are loaded via `SetParameters()` and only `Forward()` is called at runtime. -- **On-device training** — The full forward → loss → backward → update loop runs on the MCU for online learning/adaptation. -- **System identification** — A small model (2–3 layers) learns the plant dynamics from input-output data. -- **Sensor fusion** — Multiple sensor inputs are mapped to a unified state estimate through a trained model. - -## Connections to Other Algorithms - -```mermaid -graph TD - Model["Model"] - Layer["Dense Layer"] - Loss["Loss Functions"] - Opt["Optimizer"] - Reg["Regularization"] - NN["Neural Network"] - - Layer --> Model - Model --> Opt - Model --> Loss - Reg --> Loss - Model --> NN -``` - -| Component | Relationship | -|-------------------------------------------------------|--------------------------------------------------------------------------------------------| -| [Dense Layer](../layer/Layer.md) | The Model is a sequence of layers stored in a `std::tuple` | -| [Loss Functions](../losses/Loss.md) | Measures prediction error; the Model delegates loss computation to a Loss object | -| [Optimizer](../../optimization/Optimizer.md) | Receives the flat parameter/gradient vectors from the Model and returns updated parameters | -| [Regularization](../../regularization/Regularization.md) | Added to the loss before optimization | - -## References & Further Reading - -- Goodfellow, I., Bengio, Y., and Courville, A., *Deep Learning*, MIT Press, 2016 — Chapter 6 (deep feedforward networks). -- Paszke, A. et al., "PyTorch: An imperative style, high-performance deep learning library", *NeurIPS*, 2019 — inspiration for the sequential/functional model API. -- Abadi, M. et al., "TensorFlow: A system for large-scale machine learning", *OSDI*, 2016. diff --git a/doc/neural_network/model/Model.svg b/doc/neural_network/model/Model.svg deleted file mode 100644 index eb0406ed..00000000 --- a/doc/neural_network/model/Model.svg +++ /dev/null @@ -1,174 +0,0 @@ - - - - - - Neural Network Model - - - - - - - Model Architecture - - - - Input - - - - - - - - - - Layer 1 - - - - - - - - Dense - - - - - - Layer 2 - - - - - - - - Dense - - - - - - Layer N - - - - - - - - Dense - - - - - Output - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Forward Propagation - - - - - - - - Model Training - - - - - - Parameters - GetParameters() - SetParameters() - - - - Loss Function - Cost() - Gradient() - - - - Optimizer - Minimize() - UpdateParameters() - - - - model.Train() - - - - - - - - - - - - - - - - - - - - - - - Features: - • Variadic template layers - • Compile-time dimension checking - • Parameter management - • Type safety - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/numerical/CMakeLists.txt b/numerical/CMakeLists.txt index 2f392cd0..0166abbd 100644 --- a/numerical/CMakeLists.txt +++ b/numerical/CMakeLists.txt @@ -4,7 +4,6 @@ add_subdirectory(controllers) add_subdirectory(estimators) add_subdirectory(filters) add_subdirectory(math) -add_subdirectory(neural_network) add_subdirectory(optimization) add_subdirectory(regularization) add_subdirectory(nonlinear_control) diff --git a/numerical/neural_network/CMakeLists.txt b/numerical/neural_network/CMakeLists.txt deleted file mode 100644 index 9e3b3676..00000000 --- a/numerical/neural_network/CMakeLists.txt +++ /dev/null @@ -1,4 +0,0 @@ -add_subdirectory(activation) -add_subdirectory(layer) -add_subdirectory(losses) -add_subdirectory(model) diff --git a/numerical/neural_network/activation/ActivationFunction.cpp b/numerical/neural_network/activation/ActivationFunction.cpp deleted file mode 100644 index f4333bf6..00000000 --- a/numerical/neural_network/activation/ActivationFunction.cpp +++ /dev/null @@ -1,8 +0,0 @@ -#include "numerical/neural_network/activation/ActivationFunction.hpp" - -namespace neural_network -{ - template class ActivationFunction; - template class ActivationFunction; - template class ActivationFunction; -} diff --git a/numerical/neural_network/activation/ActivationFunction.hpp b/numerical/neural_network/activation/ActivationFunction.hpp deleted file mode 100644 index d67c24fa..00000000 --- a/numerical/neural_network/activation/ActivationFunction.hpp +++ /dev/null @@ -1,46 +0,0 @@ -#pragma once - -#if defined(__GNUC__) || defined(__clang__) -#pragma GCC optimize("O3", "fast-math") -#endif - -#include "numerical/math/CompilerOptimizations.hpp" -#include "numerical/math/QNumber.hpp" -#include - -namespace neural_network -{ - template - class ActivationFunction - { - static_assert(math::is_qnumber_v || std::is_floating_point_v, - "ActivationFunction can only be instantiated with math::QNumber types."); - - public: - virtual QNumberType Forward(QNumberType x) const = 0; - virtual QNumberType Backward(QNumberType x) const = 0; - - virtual void ForwardVector(std::span output, std::span input) const; - virtual void BackwardVector(std::span result, std::span preActivation, std::span output, std::span outputGradient) const; - }; - - template - OPTIMIZE_FOR_SPEED void ActivationFunction::ForwardVector(std::span output, std::span input) const - { - for (std::size_t i = 0; i < output.size(); ++i) - output[i] = Forward(input[i]); - } - - template - OPTIMIZE_FOR_SPEED void ActivationFunction::BackwardVector(std::span result, std::span preActivation, std::span /*output*/, std::span outputGradient) const - { - for (std::size_t i = 0; i < result.size(); ++i) - result[i] = outputGradient[i] * Backward(preActivation[i]); - } - -#ifdef NUMERICAL_TOOLBOX_COVERAGE_BUILD - extern template class ActivationFunction; - extern template class ActivationFunction; - extern template class ActivationFunction; -#endif -} diff --git a/numerical/neural_network/activation/CMakeLists.txt b/numerical/neural_network/activation/CMakeLists.txt deleted file mode 100644 index f20d4221..00000000 --- a/numerical/neural_network/activation/CMakeLists.txt +++ /dev/null @@ -1,31 +0,0 @@ -numerical_add_header_library(numerical.neural_network.activation) - -target_include_directories(numerical.neural_network.activation ${NUMERICAL_VISIBILITY} - "$" - "$" -) - -target_link_libraries(numerical.neural_network.activation ${NUMERICAL_VISIBILITY} - infra.util - numerical.math -) - -target_sources(numerical.neural_network.activation PRIVATE - ActivationFunction.hpp - LeakyReLU.hpp - ReLU.hpp - Sigmoid.hpp - Softmax.hpp - Tanh.hpp -) - -numerical_add_coverage_sources(numerical.neural_network.activation - ActivationFunction.cpp - ReLU.cpp - LeakyReLU.cpp - Sigmoid.cpp - Tanh.cpp - Softmax.cpp -) - -add_subdirectory(test) diff --git a/numerical/neural_network/activation/LeakyReLU.cpp b/numerical/neural_network/activation/LeakyReLU.cpp deleted file mode 100644 index a5e7fa00..00000000 --- a/numerical/neural_network/activation/LeakyReLU.cpp +++ /dev/null @@ -1,8 +0,0 @@ -#include "numerical/neural_network/activation/LeakyReLU.hpp" - -namespace neural_network -{ - template class LeakyReLU; - template class LeakyReLU; - template class LeakyReLU; -} diff --git a/numerical/neural_network/activation/LeakyReLU.hpp b/numerical/neural_network/activation/LeakyReLU.hpp deleted file mode 100644 index db1fb39b..00000000 --- a/numerical/neural_network/activation/LeakyReLU.hpp +++ /dev/null @@ -1,54 +0,0 @@ -#pragma once - -#if defined(__GNUC__) || defined(__clang__) -#pragma GCC optimize("O3", "fast-math") -#endif - -#include "numerical/math/CompilerOptimizations.hpp" -#include "numerical/neural_network/activation/ActivationFunction.hpp" - -namespace neural_network -{ - template - class LeakyReLU - : public ActivationFunction - { - public: - explicit LeakyReLU(QNumberType alpha = QNumberType(0.01f)); - - QNumberType Forward(QNumberType x) const override; - QNumberType Backward(QNumberType x) const override; - - private: - QNumberType alpha; - }; - - // Implementation - - template - LeakyReLU::LeakyReLU(QNumberType alpha) - : alpha(alpha) - {} - - template - OPTIMIZE_FOR_SPEED - QNumberType - LeakyReLU::Forward(QNumberType x) const - { - return x > QNumberType(0.0f) ? x : alpha * x; - } - - template - OPTIMIZE_FOR_SPEED - QNumberType - LeakyReLU::Backward(QNumberType x) const - { - return x > QNumberType(0.0f) ? QNumberType(0.9999f) : alpha; - } - -#ifdef NUMERICAL_TOOLBOX_COVERAGE_BUILD - extern template class LeakyReLU; - extern template class LeakyReLU; - extern template class LeakyReLU; -#endif -} diff --git a/numerical/neural_network/activation/ReLU.cpp b/numerical/neural_network/activation/ReLU.cpp deleted file mode 100644 index e7de7a9e..00000000 --- a/numerical/neural_network/activation/ReLU.cpp +++ /dev/null @@ -1,8 +0,0 @@ -#include "numerical/neural_network/activation/ReLU.hpp" - -namespace neural_network -{ - template class ReLU; - template class ReLU; - template class ReLU; -} diff --git a/numerical/neural_network/activation/ReLU.hpp b/numerical/neural_network/activation/ReLU.hpp deleted file mode 100644 index b9fec8c8..00000000 --- a/numerical/neural_network/activation/ReLU.hpp +++ /dev/null @@ -1,44 +0,0 @@ -#pragma once - -#if defined(__GNUC__) || defined(__clang__) -#pragma GCC optimize("O3", "fast-math") -#endif - -#include "numerical/math/CompilerOptimizations.hpp" -#include "numerical/neural_network/activation/ActivationFunction.hpp" - -namespace neural_network -{ - template - class ReLU - : public ActivationFunction - { - public: - QNumberType Forward(QNumberType x) const override; - QNumberType Backward(QNumberType x) const override; - }; - - // Implementation - - template - OPTIMIZE_FOR_SPEED - QNumberType - ReLU::Forward(QNumberType x) const - { - return x > QNumberType(0.0f) ? x : QNumberType(0.0f); - } - - template - OPTIMIZE_FOR_SPEED - QNumberType - ReLU::Backward(QNumberType x) const - { - return x > QNumberType(0.0f) ? QNumberType(0.9999f) : QNumberType(0.0f); - } - -#ifdef NUMERICAL_TOOLBOX_COVERAGE_BUILD - extern template class ReLU; - extern template class ReLU; - extern template class ReLU; -#endif -} diff --git a/numerical/neural_network/activation/Sigmoid.cpp b/numerical/neural_network/activation/Sigmoid.cpp deleted file mode 100644 index 775d1618..00000000 --- a/numerical/neural_network/activation/Sigmoid.cpp +++ /dev/null @@ -1,8 +0,0 @@ -#include "numerical/neural_network/activation/Sigmoid.hpp" - -namespace neural_network -{ - template class Sigmoid; - template class Sigmoid; - template class Sigmoid; -} diff --git a/numerical/neural_network/activation/Sigmoid.hpp b/numerical/neural_network/activation/Sigmoid.hpp deleted file mode 100644 index cb2166c3..00000000 --- a/numerical/neural_network/activation/Sigmoid.hpp +++ /dev/null @@ -1,46 +0,0 @@ -#pragma once - -#if defined(__GNUC__) || defined(__clang__) -#pragma GCC optimize("O3", "fast-math") -#endif - -#include "numerical/math/CompilerOptimizations.hpp" -#include "numerical/math/Math.hpp" -#include "numerical/neural_network/activation/ActivationFunction.hpp" - -namespace neural_network -{ - template - class Sigmoid - : public ActivationFunction - { - public: - QNumberType Forward(QNumberType x) const override; - QNumberType Backward(QNumberType x) const override; - }; - - // Implementation - - template - OPTIMIZE_FOR_SPEED - QNumberType - Sigmoid::Forward(QNumberType x) const - { - return QNumberType(1.0f / (1.0f + math::Exp(-math::ToFloat(x)))); - } - - template - OPTIMIZE_FOR_SPEED - QNumberType - Sigmoid::Backward(QNumberType x) const - { - QNumberType y = Forward(x); - return y * (QNumberType(0.9999f) - y); - } - -#ifdef NUMERICAL_TOOLBOX_COVERAGE_BUILD - extern template class Sigmoid; - extern template class Sigmoid; - extern template class Sigmoid; -#endif -} diff --git a/numerical/neural_network/activation/Softmax.cpp b/numerical/neural_network/activation/Softmax.cpp deleted file mode 100644 index b3890d9c..00000000 --- a/numerical/neural_network/activation/Softmax.cpp +++ /dev/null @@ -1,8 +0,0 @@ -#include "numerical/neural_network/activation/Softmax.hpp" - -namespace neural_network -{ - template class Softmax; - template class Softmax; - template class Softmax; -} diff --git a/numerical/neural_network/activation/Softmax.hpp b/numerical/neural_network/activation/Softmax.hpp deleted file mode 100644 index afa9af20..00000000 --- a/numerical/neural_network/activation/Softmax.hpp +++ /dev/null @@ -1,77 +0,0 @@ -#pragma once - -#if defined(__GNUC__) || defined(__clang__) -#pragma GCC optimize("O3", "fast-math") -#endif - -#include "numerical/math/CompilerOptimizations.hpp" -#include "numerical/math/Math.hpp" -#include "numerical/neural_network/activation/ActivationFunction.hpp" -#include - -namespace neural_network -{ - template - class Softmax - : public ActivationFunction - { - public: - QNumberType Forward(QNumberType x) const override; - QNumberType Backward(QNumberType x) const override; - void ForwardVector(std::span output, std::span input) const override; - void BackwardVector(std::span result, std::span preActivation, std::span output, std::span outputGradient) const override; - }; - - template - OPTIMIZE_FOR_SPEED - QNumberType - Softmax::Forward(QNumberType x) const - { - return QNumberType(math::Exp(math::ToFloat(x))); - } - - template - OPTIMIZE_FOR_SPEED - QNumberType - Softmax::Backward(QNumberType x) const - { - QNumberType y = Forward(x); - return y * (QNumberType(0.9999f) - y); - } - - template - OPTIMIZE_FOR_SPEED void Softmax::ForwardVector(std::span output, std::span input) const - { - QNumberType maxVal = input[0]; - for (std::size_t i = 1; i < input.size(); ++i) - if (input[i] > maxVal) - maxVal = input[i]; - - QNumberType sum = QNumberType(0.0f); - for (std::size_t i = 0; i < output.size(); ++i) - { - output[i] = QNumberType(math::Exp(math::ToFloat(input[i]) - math::ToFloat(maxVal))); - sum += output[i]; - } - - for (std::size_t i = 0; i < output.size(); ++i) - output[i] = std::max(std::min(output[i] / sum, QNumberType(0.9999f)), QNumberType(0.0001f)); - } - - template - OPTIMIZE_FOR_SPEED void Softmax::BackwardVector(std::span result, std::span /*preActivation*/, std::span output, std::span outputGradient) const - { - QNumberType dot = QNumberType(0.0f); - for (std::size_t i = 0; i < result.size(); ++i) - dot += outputGradient[i] * output[i]; - - for (std::size_t i = 0; i < result.size(); ++i) - result[i] = output[i] * (outputGradient[i] - dot); - } - -#ifdef NUMERICAL_TOOLBOX_COVERAGE_BUILD - extern template class Softmax; - extern template class Softmax; - extern template class Softmax; -#endif -} diff --git a/numerical/neural_network/activation/Tanh.cpp b/numerical/neural_network/activation/Tanh.cpp deleted file mode 100644 index 145ad563..00000000 --- a/numerical/neural_network/activation/Tanh.cpp +++ /dev/null @@ -1,8 +0,0 @@ -#include "numerical/neural_network/activation/Tanh.hpp" - -namespace neural_network -{ - template class Tanh; - template class Tanh; - template class Tanh; -} diff --git a/numerical/neural_network/activation/Tanh.hpp b/numerical/neural_network/activation/Tanh.hpp deleted file mode 100644 index b869ad9e..00000000 --- a/numerical/neural_network/activation/Tanh.hpp +++ /dev/null @@ -1,46 +0,0 @@ -#pragma once - -#if defined(__GNUC__) || defined(__clang__) -#pragma GCC optimize("O3", "fast-math") -#endif - -#include "numerical/math/CompilerOptimizations.hpp" -#include "numerical/math/Math.hpp" -#include "numerical/neural_network/activation/ActivationFunction.hpp" - -namespace neural_network -{ - template - class Tanh - : public ActivationFunction - { - public: - QNumberType Forward(QNumberType x) const override; - QNumberType Backward(QNumberType x) const override; - }; - - // Implementation - - template - OPTIMIZE_FOR_SPEED - QNumberType - Tanh::Forward(QNumberType x) const - { - return QNumberType(math::Tanh(math::ToFloat(x))); - } - - template - OPTIMIZE_FOR_SPEED - QNumberType - Tanh::Backward(QNumberType x) const - { - QNumberType y = Forward(x); - return QNumberType(0.9999f) - y * y; - } - -#ifdef NUMERICAL_TOOLBOX_COVERAGE_BUILD - extern template class Tanh; - extern template class Tanh; - extern template class Tanh; -#endif -} diff --git a/numerical/neural_network/activation/test/CMakeLists.txt b/numerical/neural_network/activation/test/CMakeLists.txt deleted file mode 100644 index 41779e06..00000000 --- a/numerical/neural_network/activation/test/CMakeLists.txt +++ /dev/null @@ -1,18 +0,0 @@ -add_executable(numerical.neural_network.activation_test) -emil_build_for(numerical.neural_network.activation_test BOOL NUMERICAL_TOOLBOX_BUILD_TESTS) -emil_add_test(numerical.neural_network.activation_test) - -target_link_libraries(numerical.neural_network.activation_test PUBLIC - gmock_main - numerical.neural_network.activation -) - -numerical_add_qemu_test(numerical.neural_network.activation_test) - -target_sources(numerical.neural_network.activation_test PRIVATE - TestLeakyReLU.cpp - TestReLU.cpp - TestSigmoid.cpp - TestSoftmax.cpp - TestTanh.cpp -) diff --git a/numerical/neural_network/activation/test/TestLeakyReLU.cpp b/numerical/neural_network/activation/test/TestLeakyReLU.cpp deleted file mode 100644 index e0e87fb9..00000000 --- a/numerical/neural_network/activation/test/TestLeakyReLU.cpp +++ /dev/null @@ -1,58 +0,0 @@ -#include "numerical/neural_network/activation/LeakyReLU.hpp" -#include "gmock/gmock.h" - -namespace -{ - template - class TestLeakyReLU - : public ::testing::Test - { - public: - neural_network::LeakyReLU activation; - }; - - using TestedTypes = ::testing::Types; - TYPED_TEST_SUITE(TestLeakyReLU, TestedTypes); -} - -TYPED_TEST(TestLeakyReLU, ForwardPositiveInput) -{ - EXPECT_EQ(this->activation.Forward(TypeParam(0.5f)), TypeParam(0.5f)); - EXPECT_EQ(this->activation.Forward(TypeParam(0.999f)), TypeParam(0.999f)); -} - -TYPED_TEST(TestLeakyReLU, ForwardNegativeInput) -{ - const auto alpha = TypeParam(0.01f); - neural_network::LeakyReLU activation(alpha); - - EXPECT_EQ(activation.Forward(TypeParam(-0.5f)), TypeParam(-0.005f)); -} - -TYPED_TEST(TestLeakyReLU, ForwardZeroInput) -{ - EXPECT_EQ(this->activation.Forward(TypeParam(0.0f)), TypeParam(0.0f)); -} - -TYPED_TEST(TestLeakyReLU, BackwardPositiveInput) -{ - EXPECT_EQ(this->activation.Backward(TypeParam(0.5f)), TypeParam(0.9999f)); - EXPECT_EQ(this->activation.Backward(TypeParam(0.999f)), TypeParam(0.9999f)); -} - -TYPED_TEST(TestLeakyReLU, BackwardNegativeInput) -{ - const auto alpha = TypeParam(0.01f); - neural_network::LeakyReLU activation(alpha); - - EXPECT_EQ(activation.Backward(TypeParam(-0.5f)), alpha); - EXPECT_EQ(activation.Backward(TypeParam(-0.999f)), alpha); -} - -TYPED_TEST(TestLeakyReLU, BackwardZeroInput) -{ - const auto alpha = TypeParam(0.01f); - neural_network::LeakyReLU activation(alpha); - - EXPECT_EQ(activation.Backward(TypeParam(0.0f)), alpha); -} diff --git a/numerical/neural_network/activation/test/TestReLU.cpp b/numerical/neural_network/activation/test/TestReLU.cpp deleted file mode 100644 index 6117eb28..00000000 --- a/numerical/neural_network/activation/test/TestReLU.cpp +++ /dev/null @@ -1,50 +0,0 @@ -#include "numerical/neural_network/activation/ReLU.hpp" -#include "gmock/gmock.h" - -namespace -{ - template - class TestReLU - : public ::testing::Test - { - public: - neural_network::ReLU activation; - }; - - using TestedTypes = ::testing::Types; - TYPED_TEST_SUITE(TestReLU, TestedTypes); -} - -TYPED_TEST(TestReLU, ForwardPositiveInput) -{ - EXPECT_EQ(this->activation.Forward(TypeParam(0.5f)), TypeParam(0.5f)); - EXPECT_EQ(this->activation.Forward(TypeParam(0.999f)), TypeParam(0.999f)); -} - -TYPED_TEST(TestReLU, ForwardNegativeInput) -{ - EXPECT_EQ(this->activation.Forward(TypeParam(-0.5f)), TypeParam(0.0f)); - EXPECT_EQ(this->activation.Forward(TypeParam(-0.999f)), TypeParam(0.0f)); -} - -TYPED_TEST(TestReLU, ForwardZeroInput) -{ - EXPECT_EQ(this->activation.Forward(TypeParam(0.0f)), TypeParam(0.0f)); -} - -TYPED_TEST(TestReLU, BackwardPositiveInput) -{ - EXPECT_EQ(this->activation.Backward(TypeParam(0.5f)), TypeParam(0.9999f)); - EXPECT_EQ(this->activation.Backward(TypeParam(0.999f)), TypeParam(0.9999f)); -} - -TYPED_TEST(TestReLU, BackwardNegativeInput) -{ - EXPECT_EQ(this->activation.Backward(TypeParam(-0.5f)), TypeParam(0.0f)); - EXPECT_EQ(this->activation.Backward(TypeParam(-0.999f)), TypeParam(0.0f)); -} - -TYPED_TEST(TestReLU, BackwardZeroInput) -{ - EXPECT_EQ(this->activation.Backward(TypeParam(0.0f)), TypeParam(0.0f)); -} diff --git a/numerical/neural_network/activation/test/TestSigmoid.cpp b/numerical/neural_network/activation/test/TestSigmoid.cpp deleted file mode 100644 index 2cd1e21f..00000000 --- a/numerical/neural_network/activation/test/TestSigmoid.cpp +++ /dev/null @@ -1,50 +0,0 @@ -#include "numerical/neural_network/activation/Sigmoid.hpp" -#include "gmock/gmock.h" - -namespace -{ - template - class TestSigmoid - : public ::testing::Test - { - public: - neural_network::Sigmoid activation; - }; - - using TestedTypes = ::testing::Types; - TYPED_TEST_SUITE(TestSigmoid, TestedTypes); -} - -TYPED_TEST(TestSigmoid, ForwardZeroInput) -{ - EXPECT_NEAR(math::ToFloat(this->activation.Forward(TypeParam(0.0f))), 0.5f, 0.001f); -} - -TYPED_TEST(TestSigmoid, ForwardPositiveInput) -{ - EXPECT_NEAR(math::ToFloat(this->activation.Forward(TypeParam(0.5f))), 0.622f, 0.001f); - EXPECT_NEAR(math::ToFloat(this->activation.Forward(TypeParam(0.999f))), 0.731f, 0.001f); -} - -TYPED_TEST(TestSigmoid, ForwardNegativeInput) -{ - EXPECT_NEAR(math::ToFloat(this->activation.Forward(TypeParam(-0.5f))), 0.378f, 0.001f); - EXPECT_NEAR(math::ToFloat(this->activation.Forward(TypeParam(-0.999f))), 0.269f, 0.001f); -} - -TYPED_TEST(TestSigmoid, BackwardZeroInput) -{ - EXPECT_NEAR(math::ToFloat(this->activation.Backward(TypeParam(0.0f))), 0.25f, 0.001f); -} - -TYPED_TEST(TestSigmoid, BackwardPositiveInput) -{ - EXPECT_NEAR(math::ToFloat(this->activation.Backward(TypeParam(0.5f))), 0.235f, 0.001f); - EXPECT_NEAR(math::ToFloat(this->activation.Backward(TypeParam(0.999f))), 0.197f, 0.001f); -} - -TYPED_TEST(TestSigmoid, BackwardNegativeInput) -{ - EXPECT_NEAR(math::ToFloat(this->activation.Backward(TypeParam(-0.5f))), 0.235f, 0.001f); - EXPECT_NEAR(math::ToFloat(this->activation.Backward(TypeParam(-0.999f))), 0.197f, 0.001f); -} diff --git a/numerical/neural_network/activation/test/TestSoftmax.cpp b/numerical/neural_network/activation/test/TestSoftmax.cpp deleted file mode 100644 index 7c069693..00000000 --- a/numerical/neural_network/activation/test/TestSoftmax.cpp +++ /dev/null @@ -1,118 +0,0 @@ -#include "numerical/neural_network/activation/Softmax.hpp" -#include "gmock/gmock.h" - -namespace -{ - template - class TestSoftmax - : public ::testing::Test - { - public: - neural_network::Softmax activation; - }; - - using TestedTypes = ::testing::Types; - TYPED_TEST_SUITE(TestSoftmax, TestedTypes); -} - -TYPED_TEST(TestSoftmax, ForwardZeroInput) -{ - EXPECT_NEAR(math::ToFloat(this->activation.Forward(TypeParam(-0.1f))), 0.905f, 0.001f); -} - -TYPED_TEST(TestSoftmax, ForwardNegativeInput) -{ - EXPECT_NEAR(math::ToFloat(this->activation.Forward(TypeParam(-0.5f))), 0.607f, 0.001f); - EXPECT_NEAR(math::ToFloat(this->activation.Forward(TypeParam(-0.9f))), 0.407f, 0.001f); -} - -TYPED_TEST(TestSoftmax, BackwardTinyValues) -{ - TypeParam x = TypeParam(-0.7f); - TypeParam y = this->activation.Forward(x); - - EXPECT_NEAR(math::ToFloat(this->activation.Backward(x)), - math::ToFloat(y * (TypeParam(0.9999f) - y)), 0.001f); -} - -TYPED_TEST(TestSoftmax, BackwardNegativeInputs) -{ - for (float x = -0.9f; x <= -0.5f; x += 0.1f) - { - TypeParam input = TypeParam(x); - TypeParam output = this->activation.Forward(input); - TypeParam derivative = this->activation.Backward(input); - TypeParam expected = output * (TypeParam(0.9999f) - output); - - EXPECT_NEAR(math::ToFloat(derivative), math::ToFloat(expected), 0.001f); - } -} - -TYPED_TEST(TestSoftmax, ComparativeTest) -{ - TypeParam x1 = TypeParam(-0.8f); - TypeParam x2 = TypeParam(-0.7f); - TypeParam y1 = this->activation.Forward(x1); - TypeParam y2 = this->activation.Forward(x2); - - EXPECT_LT(math::ToFloat(y1), math::ToFloat(y2)); -} - -TYPED_TEST(TestSoftmax, ForwardVectorNormalizesToOne) -{ - if constexpr (!std::is_floating_point_v) - GTEST_SKIP() << "Softmax vector ops exceed Q-number range"; - - constexpr std::size_t Size = 3; - TypeParam input[Size] = { TypeParam(-0.3f), TypeParam(-0.5f), TypeParam(-0.2f) }; - TypeParam output[Size] = {}; - - this->activation.ForwardVector(output, input); - - float sum = 0.0f; - for (std::size_t i = 0; i < Size; ++i) - sum += math::ToFloat(output[i]); - - EXPECT_NEAR(sum, 1.0f, 0.01f); -} - -TYPED_TEST(TestSoftmax, ForwardVectorPreservesOrdering) -{ - if constexpr (!std::is_floating_point_v) - GTEST_SKIP() << "Softmax vector ops exceed Q-number range"; - - constexpr std::size_t Size = 3; - TypeParam input[Size] = { TypeParam(-0.5f), TypeParam(-0.2f), TypeParam(-0.8f) }; - TypeParam output[Size] = {}; - - this->activation.ForwardVector(output, input); - - EXPECT_GT(math::ToFloat(output[1]), math::ToFloat(output[0])); - EXPECT_GT(math::ToFloat(output[0]), math::ToFloat(output[2])); -} - -TYPED_TEST(TestSoftmax, BackwardVectorProducesCorrectGradient) -{ - if constexpr (!std::is_floating_point_v) - GTEST_SKIP() << "Softmax vector ops exceed Q-number range"; - - constexpr std::size_t Size = 3; - TypeParam input[Size] = { TypeParam(-0.3f), TypeParam(-0.5f), TypeParam(-0.2f) }; - TypeParam output[Size] = {}; - this->activation.ForwardVector(output, input); - - TypeParam outputGradient[Size] = { TypeParam(0.9999f), TypeParam(0.0f), TypeParam(0.0f) }; - TypeParam result[Size] = {}; - - this->activation.BackwardVector(result, input, output, outputGradient); - - float dot = 0.0f; - for (std::size_t i = 0; i < Size; ++i) - dot += math::ToFloat(outputGradient[i]) * math::ToFloat(output[i]); - - for (std::size_t i = 0; i < Size; ++i) - { - float expected = math::ToFloat(output[i]) * (math::ToFloat(outputGradient[i]) - dot); - EXPECT_NEAR(math::ToFloat(result[i]), expected, 0.01f); - } -} diff --git a/numerical/neural_network/activation/test/TestTanh.cpp b/numerical/neural_network/activation/test/TestTanh.cpp deleted file mode 100644 index e9cef7ae..00000000 --- a/numerical/neural_network/activation/test/TestTanh.cpp +++ /dev/null @@ -1,76 +0,0 @@ -#include "numerical/neural_network/activation/Tanh.hpp" -#include "gmock/gmock.h" - -namespace -{ - template - class TestTanh - : public ::testing::Test - { - public: - neural_network::Tanh activation; - }; - - using TestedTypes = ::testing::Types; - TYPED_TEST_SUITE(TestTanh, TestedTypes); -} - -TYPED_TEST(TestTanh, ForwardZeroInput) -{ - EXPECT_NEAR(math::ToFloat(this->activation.Forward(TypeParam(0.0f))), 0.0f, 0.001f); -} - -TYPED_TEST(TestTanh, ForwardSmallPositiveInput) -{ - EXPECT_NEAR(math::ToFloat(this->activation.Forward(TypeParam(0.5f))), 0.462f, 0.001f); - EXPECT_NEAR(math::ToFloat(this->activation.Forward(TypeParam(0.9f))), 0.716f, 0.001f); -} - -TYPED_TEST(TestTanh, ForwardSmallNegativeInput) -{ - EXPECT_NEAR(math::ToFloat(this->activation.Forward(TypeParam(-0.5f))), -0.462f, 0.001f); - EXPECT_NEAR(math::ToFloat(this->activation.Forward(TypeParam(-0.9f))), -0.716f, 0.001f); -} - -TYPED_TEST(TestTanh, BackwardZeroInput) -{ - EXPECT_NEAR(math::ToFloat(this->activation.Backward(TypeParam(0.0f))), 1.0f, 0.001f); -} - -TYPED_TEST(TestTanh, BackwardSmallPositiveInput) -{ - TypeParam x1 = TypeParam(0.5f); - TypeParam y1 = this->activation.Forward(x1); - EXPECT_NEAR(math::ToFloat(this->activation.Backward(x1)), - 1.0f - math::ToFloat(y1) * math::ToFloat(y1), 0.001f); - - TypeParam x2 = TypeParam(0.8f); - TypeParam y2 = this->activation.Forward(x2); - EXPECT_NEAR(math::ToFloat(this->activation.Backward(x2)), - 1.0f - math::ToFloat(y2) * math::ToFloat(y2), 0.001f); -} - -TYPED_TEST(TestTanh, BackwardSmallNegativeInput) -{ - TypeParam x1 = TypeParam(-0.5f); - TypeParam y1 = this->activation.Forward(x1); - EXPECT_NEAR(math::ToFloat(this->activation.Backward(x1)), - 1.0f - math::ToFloat(y1) * math::ToFloat(y1), 0.001f); - - TypeParam x2 = TypeParam(-0.8f); - TypeParam y2 = this->activation.Forward(x2); - EXPECT_NEAR(math::ToFloat(this->activation.Backward(x2)), - 1.0f - math::ToFloat(y2) * math::ToFloat(y2), 0.001f); -} - -TYPED_TEST(TestTanh, SymmetryProperty) -{ - for (float x = 0.1f; x <= 0.9f; x += 0.2f) - { - TypeParam posInput = TypeParam(x); - TypeParam negInput = TypeParam(-x); - - EXPECT_NEAR(math::ToFloat(this->activation.Forward(posInput)), - -math::ToFloat(this->activation.Forward(negInput)), 0.001f); - } -} diff --git a/numerical/neural_network/layer/CMakeLists.txt b/numerical/neural_network/layer/CMakeLists.txt deleted file mode 100644 index 170a3448..00000000 --- a/numerical/neural_network/layer/CMakeLists.txt +++ /dev/null @@ -1,22 +0,0 @@ -numerical_add_header_library(numerical.neural_network.layer) - -target_include_directories(numerical.neural_network.layer ${NUMERICAL_VISIBILITY} - "$" - "$" -) - -target_link_libraries(numerical.neural_network.layer ${NUMERICAL_VISIBILITY} - infra.util - numerical.math -) - -target_sources(numerical.neural_network.layer PRIVATE - Dense.hpp - Layer.hpp -) - -numerical_add_coverage_sources(numerical.neural_network.layer - Dense.cpp -) - -add_subdirectory(test) diff --git a/numerical/neural_network/layer/Dense.cpp b/numerical/neural_network/layer/Dense.cpp deleted file mode 100644 index 73550169..00000000 --- a/numerical/neural_network/layer/Dense.cpp +++ /dev/null @@ -1,8 +0,0 @@ -#include "numerical/neural_network/layer/Dense.hpp" - -namespace neural_network -{ - template class Dense; - template class Dense; - template class Dense; -} diff --git a/numerical/neural_network/layer/Dense.hpp b/numerical/neural_network/layer/Dense.hpp deleted file mode 100644 index c2a88224..00000000 --- a/numerical/neural_network/layer/Dense.hpp +++ /dev/null @@ -1,150 +0,0 @@ -#pragma once - -#if defined(__GNUC__) || defined(__clang__) -#pragma GCC optimize("O3", "fast-math") -#endif - -#include "numerical/math/CompilerOptimizations.hpp" -#include "numerical/neural_network/activation/ActivationFunction.hpp" -#include "numerical/neural_network/layer/Layer.hpp" -#include - -namespace neural_network -{ - template - class Dense - : public Layer - { - public: - using BaseLayer = Layer; - using InputVector = typename BaseLayer::InputVector; - using OutputVector = typename BaseLayer::OutputVector; - using ParameterVector = typename BaseLayer::ParameterVector; - - Dense(const math::Matrix& initialWeights, ActivationFunction& activation); - - void Forward(const InputVector& input) override; - InputVector& Backward(const OutputVector& output_gradient) override; - const OutputVector& Output() const override; - ParameterVector& Parameters() const override; - void SetParameters(const ParameterVector& parameters) override; - - private: - ActivationFunction& activation; - - math::Matrix weights; - math::Vector biases; - - InputVector input; - OutputVector preActivation; - OutputVector output; - - math::Matrix weightGradients; - math::Vector biasGradients; - - mutable ParameterVector parameters; - InputVector inputGradient; - }; - - // Implementation // - - template - Dense::Dense(const math::Matrix& initialWeights, ActivationFunction& activation) - : weights(initialWeights) - , activation(activation) - {} - - template - OPTIMIZE_FOR_SPEED void Dense::Forward(const InputVector& input) - { - this->input = input; - - for (std::size_t i = 0; i < OutputSize; ++i) - { - preActivation[i] = biases[i]; - for (std::size_t j = 0; j < InputSize; ++j) - preActivation[i] += weights.at(i, j) * input[j]; - } - - activation.ForwardVector(output, preActivation); - } - - template - OPTIMIZE_FOR_SPEED - typename Dense::InputVector& - Dense::Backward(const OutputVector& output_gradient) - { - OutputVector preActivationGradient; - - activation.BackwardVector(preActivationGradient, preActivation, output, output_gradient); - - for (std::size_t j = 0; j < InputSize; ++j) - { - inputGradient[j] = QNumberType(0.0f); - for (std::size_t i = 0; i < OutputSize; ++i) - inputGradient[j] += weights.at(i, j) * preActivationGradient[i]; - } - - for (std::size_t i = 0; i < OutputSize; ++i) - { - for (std::size_t j = 0; j < InputSize; ++j) - weightGradients.at(i, j) = preActivationGradient[i] * input[j]; - - biasGradients[i] = preActivationGradient[i]; - } - - return inputGradient; - } - - template - const typename Dense::OutputVector& Dense::Output() const - { - return output; - } - - template - typename Dense::ParameterVector& Dense::Parameters() const - { - std::size_t idx = 0; - - for (std::size_t i = 0; i < OutputSize; ++i) - for (std::size_t j = 0; j < InputSize; ++j) - { - parameters[idx] = weights.at(i, j); - ++idx; - } - - for (std::size_t i = 0; i < OutputSize; ++i) - { - parameters[idx] = biases[i]; - ++idx; - } - - return parameters; - } - - template - void Dense::SetParameters(const ParameterVector& parameters) - { - std::size_t idx = 0; - - for (std::size_t i = 0; i < OutputSize; ++i) - for (std::size_t j = 0; j < InputSize; ++j) - { - weights.at(i, j) = parameters[idx]; - ++idx; - } - - for (std::size_t i = 0; i < OutputSize; ++i) - { - biases[i] = parameters[idx]; - ++idx; - } - } - -#ifdef NUMERICAL_TOOLBOX_COVERAGE_BUILD - extern template class Dense; - extern template class Dense; - extern template class Dense; -#endif -} diff --git a/numerical/neural_network/layer/Layer.hpp b/numerical/neural_network/layer/Layer.hpp deleted file mode 100644 index 1a864e29..00000000 --- a/numerical/neural_network/layer/Layer.hpp +++ /dev/null @@ -1,26 +0,0 @@ -#pragma once - -#include "numerical/math/Matrix.hpp" - -namespace neural_network -{ - template - class Layer - { - public: - using QNumberType_ = QNumberType; - using InputVector = math::Matrix; - using OutputVector = math::Matrix; - using ParameterVector = math::Matrix; - - static constexpr std::size_t InputSize = InputSize_; - static constexpr std::size_t OutputSize = OutputSize_; - static constexpr std::size_t ParameterSize = ParameterSize_; - - virtual void Forward(const InputVector& input) = 0; - virtual InputVector& Backward(const OutputVector& output_gradient) = 0; - virtual const OutputVector& Output() const = 0; - virtual ParameterVector& Parameters() const = 0; - virtual void SetParameters(const ParameterVector& parameters) = 0; - }; -} diff --git a/numerical/neural_network/layer/test/CMakeLists.txt b/numerical/neural_network/layer/test/CMakeLists.txt deleted file mode 100644 index fe92f8e3..00000000 --- a/numerical/neural_network/layer/test/CMakeLists.txt +++ /dev/null @@ -1,15 +0,0 @@ -add_executable(numerical.neural_network.layer_test) -emil_build_for(numerical.neural_network.layer_test BOOL NUMERICAL_TOOLBOX_BUILD_TESTS) -emil_add_test(numerical.neural_network.layer_test) - -target_link_libraries(numerical.neural_network.layer_test PUBLIC - gmock_main - numerical.neural_network.activation - numerical.neural_network.layer -) - -numerical_add_qemu_test(numerical.neural_network.layer_test) - -target_sources(numerical.neural_network.layer_test PRIVATE - TestDense.cpp -) diff --git a/numerical/neural_network/layer/test/TestDense.cpp b/numerical/neural_network/layer/test/TestDense.cpp deleted file mode 100644 index 12fba78d..00000000 --- a/numerical/neural_network/layer/test/TestDense.cpp +++ /dev/null @@ -1,296 +0,0 @@ -#include "numerical/neural_network/activation/ActivationFunction.hpp" -#include "numerical/neural_network/layer/Dense.hpp" -#include "numerical/neural_network/losses/Loss.hpp" -#include "numerical/optimization/Optimizer.hpp" -#include "gmock/gmock.h" - -namespace -{ - template - class ActivationMock - : public neural_network::ActivationFunction - { - public: - MOCK_METHOD(T, Forward, (T x), (const, override)); - MOCK_METHOD(T, Backward, (T x), (const, override)); - }; - - template - class MockLoss - : public neural_network::Loss - { - public: - using Vector = typename neural_network::Loss::Vector; - - MOCK_METHOD(T, Cost, (const Vector& parameters), (override)); - MOCK_METHOD(Vector, Gradient, (const Vector& parameters), (override)); - }; - - template - class MockOptimizer - : public optimization::Optimizer - { - public: - using Result = typename optimization::Optimizer::Result; - using Vector = typename optimization::Optimizer::Vector; - - const Result& Minimize(const Vector& initialGuess, optimization::ObjectiveFunction& objective) override - { - return result; - } - - Result result{ Vector{}, T(0), 0 }; - }; - - template - class TestDense - : public ::testing::Test - { - public: - static constexpr std::size_t InputSize = 3; - static constexpr std::size_t OutputSize = 2; - using WeightMatrix = math::Matrix; - using InputVector = math::Vector; - using OutputVector = math::Vector; - using ParameterVector = math::Vector; - - void SetUp() override - { - for (std::size_t i = 0; i < OutputSize; ++i) - for (std::size_t j = 0; j < InputSize; ++j) - initialWeight.at(i, j) = T(0.1f); - } - - WeightMatrix initialWeight; - ActivationMock activation; - }; - - using TestedTypes = ::testing::Types; - TYPED_TEST_SUITE(TestDense, TestedTypes); -} - -TYPED_TEST(TestDense, Construction) -{ - using T = TypeParam; - neural_network::Dense::InputSize, TestDense::OutputSize> - dense(this->initialWeight, this->activation); - - auto params = dense.Parameters(); - - std::size_t idx = 0; - for (std::size_t i = 0; i < TestDense::OutputSize; ++i) - { - for (std::size_t j = 0; j < TestDense::InputSize; ++j) - { - EXPECT_EQ(params[idx++], this->initialWeight.at(i, j)); - } - } - - for (std::size_t i = 0; i < TestDense::OutputSize; ++i) - { - EXPECT_EQ(params[idx++], T(0.0f)); - } -} - -TYPED_TEST(TestDense, ForwardPropagation) -{ - using T = TypeParam; - using InputVector = typename TestDense::InputVector; - - neural_network::Dense::InputSize, TestDense::OutputSize> - dense(this->initialWeight, this->activation); - - InputVector input; - input[0] = T(1.0f); - input[1] = T(2.0f); - input[2] = T(3.0f); - - T expectedPreActivation = T(0.6f); - - EXPECT_CALL(this->activation, Forward(expectedPreActivation)) - .Times(TestDense::OutputSize) - .WillRepeatedly(testing::Return(T(0.5f))); - - dense.Forward(input); -} - -TYPED_TEST(TestDense, BackwardPropagation) -{ - using T = TypeParam; - using InputVector = typename TestDense::InputVector; - using OutputVector = typename TestDense::OutputVector; - - neural_network::Dense::InputSize, TestDense::OutputSize> - dense(this->initialWeight, this->activation); - - InputVector input; - input[0] = T(1.0f); - input[1] = T(2.0f); - input[2] = T(3.0f); - - OutputVector outputGradient; - outputGradient[0] = T(0.5f); - outputGradient[1] = T(0.7f); - - T expectedPreActivation = T(0.6f); - EXPECT_CALL(this->activation, Forward(expectedPreActivation)) - .Times(TestDense::OutputSize) - .WillRepeatedly(testing::Return(T(0.5f))); - - dense.Forward(input); - - T expectedPreActivationBackward = T(0.6f); - EXPECT_CALL(this->activation, Backward(expectedPreActivationBackward)) - .Times(TestDense::OutputSize) - .WillRepeatedly(testing::Return(T(0.4f))); - - dense.Backward(outputGradient); -} - -TYPED_TEST(TestDense, ParametersExtractionAndSetting) -{ - using T = TypeParam; - using ParameterVector = typename TestDense::ParameterVector; - - neural_network::Dense::InputSize, TestDense::OutputSize> - dense(this->initialWeight, this->activation); - - auto params = dense.Parameters(); - - ParameterVector newParams; - for (std::size_t i = 0; i < ParameterVector::size; ++i) - { - newParams[i] = T(0.2f); - } - - dense.SetParameters(newParams); - - auto updatedParams = dense.Parameters(); - for (std::size_t i = 0; i < ParameterVector::size; ++i) - { - EXPECT_EQ(updatedParams[i], newParams[i]); - } -} - -TYPED_TEST(TestDense, OptimizationWithMockedOptimizer) -{ - using T = TypeParam; - using ParameterVector = typename TestDense::ParameterVector; - - constexpr size_t TotalParams = (TestDense::InputSize * TestDense::OutputSize) + TestDense::OutputSize; - - neural_network::Dense::InputSize, TestDense::OutputSize> - dense(this->initialWeight, this->activation); - - MockOptimizer optimizer; - - auto initialParams = dense.Parameters(); - - ParameterVector optimizedParams; - for (std::size_t i = 0; i < TotalParams; ++i) - optimizedParams[i] = T(0.05f); - - optimizer.result = typename optimization::Optimizer::Result( - optimizedParams, T(0.4f), 3); - - MockLoss loss; - auto& result = optimizer.Minimize(initialParams, loss); - - dense.SetParameters(result.parameters); - - auto updatedParams = dense.Parameters(); - for (std::size_t i = 0; i < TotalParams; ++i) - EXPECT_EQ(updatedParams[i], optimizedParams[i]); - - EXPECT_EQ(result.iterations, 3u); -} - -TYPED_TEST(TestDense, FullLayerSequence) -{ - using T = TypeParam; - using InputVector = typename TestDense::InputVector; - using OutputVector = typename TestDense::OutputVector; - - neural_network::Dense::InputSize, TestDense::OutputSize> - dense(this->initialWeight, this->activation); - - InputVector input; - input[0] = T(1.0f); - input[1] = T(2.0f); - input[2] = T(3.0f); - - OutputVector outputGradient; - outputGradient[0] = T(0.5f); - outputGradient[1] = T(0.7f); - - { - ::testing::InSequence seq; - - EXPECT_CALL(this->activation, Forward(testing::_)) - .WillOnce(testing::Return(T(0.5f))); - EXPECT_CALL(this->activation, Forward(testing::_)) - .WillOnce(testing::Return(T(0.6f))); - - EXPECT_CALL(this->activation, Backward(testing::_)) - .WillOnce(testing::Return(T(0.3f))); - EXPECT_CALL(this->activation, Backward(testing::_)) - .WillOnce(testing::Return(T(0.4f))); - } - - dense.Forward(input); - dense.Backward(outputGradient); -} - -TYPED_TEST(TestDense, MultipleForwardBackwardPasses) -{ - using T = TypeParam; - using InputVector = typename TestDense::InputVector; - using OutputVector = typename TestDense::OutputVector; - - neural_network::Dense::InputSize, TestDense::OutputSize> - dense(this->initialWeight, this->activation); - - InputVector input1, input2; - input1[0] = T(1.0f); - input1[1] = T(2.0f); - input1[2] = T(3.0f); - input2[0] = T(4.0f); - input2[1] = T(5.0f); - input2[2] = T(6.0f); - - OutputVector outputGradient1, outputGradient2; - outputGradient1[0] = T(0.1f); - outputGradient1[1] = T(0.2f); - outputGradient2[0] = T(0.3f); - outputGradient2[1] = T(0.4f); - - { - ::testing::InSequence seq; - - EXPECT_CALL(this->activation, Forward(testing::_)) - .WillOnce(testing::Return(T(0.5f))); - EXPECT_CALL(this->activation, Forward(testing::_)) - .WillOnce(testing::Return(T(0.6f))); - - EXPECT_CALL(this->activation, Backward(testing::_)) - .WillOnce(testing::Return(T(0.3f))); - EXPECT_CALL(this->activation, Backward(testing::_)) - .WillOnce(testing::Return(T(0.4f))); - - EXPECT_CALL(this->activation, Forward(testing::_)) - .WillOnce(testing::Return(T(0.7f))); - EXPECT_CALL(this->activation, Forward(testing::_)) - .WillOnce(testing::Return(T(0.8f))); - - EXPECT_CALL(this->activation, Backward(testing::_)) - .WillOnce(testing::Return(T(0.5f))); - EXPECT_CALL(this->activation, Backward(testing::_)) - .WillOnce(testing::Return(T(0.6f))); - } - - dense.Forward(input1); - dense.Backward(outputGradient1); - - dense.Forward(input2); - dense.Backward(outputGradient2); -} diff --git a/numerical/neural_network/losses/BinaryCrossEntropy.cpp b/numerical/neural_network/losses/BinaryCrossEntropy.cpp deleted file mode 100644 index 9b119e04..00000000 --- a/numerical/neural_network/losses/BinaryCrossEntropy.cpp +++ /dev/null @@ -1,6 +0,0 @@ -#include "numerical/neural_network/losses/BinaryCrossEntropy.hpp" - -namespace neural_network -{ - template class BinaryCrossEntropy; -} diff --git a/numerical/neural_network/losses/BinaryCrossEntropy.hpp b/numerical/neural_network/losses/BinaryCrossEntropy.hpp deleted file mode 100644 index b0f3438d..00000000 --- a/numerical/neural_network/losses/BinaryCrossEntropy.hpp +++ /dev/null @@ -1,79 +0,0 @@ -#pragma once - -#if defined(__GNUC__) || defined(__clang__) -#pragma GCC optimize("O3", "fast-math") -#endif - -#include "numerical/math/CompilerOptimizations.hpp" -#include "numerical/math/Math.hpp" -#include "numerical/neural_network/losses/Loss.hpp" -#include "numerical/regularization/Regularization.hpp" - -namespace neural_network -{ - template - class BinaryCrossEntropy - : public Loss - { - public: - using Vector = typename Loss::Vector; - - BinaryCrossEntropy(const Vector& target, regularization::Regularization& regularization); - QNumberType Cost(const Vector& parameters) override; - Vector Gradient(const Vector& parameters) override; - - private: - Vector target; - regularization::Regularization& regularization; - }; - - // Implementation // - - template - BinaryCrossEntropy::BinaryCrossEntropy( - const Vector& target, - regularization::Regularization& regularization) - : target(target) - , regularization(regularization) - {} - - template - OPTIMIZE_FOR_SPEED - QNumberType - BinaryCrossEntropy::Cost(const Vector& parameters) - { - QNumberType cost = QNumberType(0.0f); - - for (std::size_t i = 0; i < NumberOfFeatures; ++i) - { - QNumberType pred = std::max(std::min(parameters[i], QNumberType(0.9999f)), QNumberType(0.0001f)); - - cost += -(target[i] * math::Log(math::ToFloat(pred)) + - (QNumberType(0.9999f) - target[i]) * math::Log(math::ToFloat(QNumberType(0.9999f) - pred))); - } - - return cost + regularization.Calculate(parameters); - } - - template - typename BinaryCrossEntropy::Vector - OPTIMIZE_FOR_SPEED - BinaryCrossEntropy::Gradient(const Vector& parameters) - { - Vector gradient; - auto regGradient = regularization.Gradient(parameters); - - for (std::size_t i = 0; i < NumberOfFeatures; ++i) - { - QNumberType pred = std::max(std::min(parameters[i], QNumberType(0.9999f)), QNumberType(0.0001f)); - - gradient[i] = (pred - target[i]) / (pred * (QNumberType(0.9999f) - pred)) + regGradient[i]; - } - - return gradient; - } - -#ifdef NUMERICAL_TOOLBOX_COVERAGE_BUILD - extern template class BinaryCrossEntropy; -#endif -} diff --git a/numerical/neural_network/losses/CMakeLists.txt b/numerical/neural_network/losses/CMakeLists.txt deleted file mode 100644 index dd90150e..00000000 --- a/numerical/neural_network/losses/CMakeLists.txt +++ /dev/null @@ -1,31 +0,0 @@ -numerical_add_header_library(numerical.neural_network.losses) - -target_include_directories(numerical.neural_network.losses ${NUMERICAL_VISIBILITY} - "$" - "$" -) - -target_link_libraries(numerical.neural_network.losses ${NUMERICAL_VISIBILITY} - infra.util - numerical.math - numerical.neural_network.activation - numerical.optimization - numerical.regularization -) - -target_sources(numerical.neural_network.losses PRIVATE - BinaryCrossEntropy.hpp - CategoricalCrossEntropy.hpp - Loss.hpp - MeanAbsoluteError.hpp - MeanSquaredError.hpp -) - -numerical_add_coverage_sources(numerical.neural_network.losses - MeanSquaredError.cpp - MeanAbsoluteError.cpp - BinaryCrossEntropy.cpp - CategoricalCrossEntropy.cpp -) - -add_subdirectory(test) diff --git a/numerical/neural_network/losses/CategoricalCrossEntropy.cpp b/numerical/neural_network/losses/CategoricalCrossEntropy.cpp deleted file mode 100644 index df87fde3..00000000 --- a/numerical/neural_network/losses/CategoricalCrossEntropy.cpp +++ /dev/null @@ -1,6 +0,0 @@ -#include "numerical/neural_network/losses/CategoricalCrossEntropy.hpp" - -namespace neural_network -{ - template class CategoricalCrossEntropy; -} diff --git a/numerical/neural_network/losses/CategoricalCrossEntropy.hpp b/numerical/neural_network/losses/CategoricalCrossEntropy.hpp deleted file mode 100644 index 2a2876f3..00000000 --- a/numerical/neural_network/losses/CategoricalCrossEntropy.hpp +++ /dev/null @@ -1,99 +0,0 @@ -#pragma once - -#if defined(__GNUC__) || defined(__clang__) -#pragma GCC optimize("O3", "fast-math") -#endif - -#include "numerical/math/CompilerOptimizations.hpp" -#include "numerical/math/Math.hpp" -#include "numerical/neural_network/activation/Softmax.hpp" -#include "numerical/neural_network/losses/Loss.hpp" -#include "numerical/regularization/Regularization.hpp" - -namespace neural_network -{ - template - class CategoricalCrossEntropy - : public Loss - { - public: - using Vector = typename Loss::Vector; - - CategoricalCrossEntropy(const Vector& target, regularization::Regularization& regularization); - QNumberType Cost(const Vector& parameters) override; - Vector Gradient(const Vector& parameters) override; - - private: - Vector target; - regularization::Regularization& regularization; - Softmax softmax; - - Vector ComputeSoftmaxProbabilities(const Vector& x) const; - }; - - // Implementation // - - template - CategoricalCrossEntropy::CategoricalCrossEntropy( - const Vector& target, - regularization::Regularization& regularization) - : target(target) - , regularization(regularization) - {} - - template - typename CategoricalCrossEntropy::Vector - OPTIMIZE_FOR_SPEED - CategoricalCrossEntropy::ComputeSoftmaxProbabilities(const Vector& x) const - { - Vector output; - QNumberType sum = QNumberType(0.0f); - - for (std::size_t i = 0; i < NumberOfFeatures; ++i) - { - output[i] = softmax.Forward(x[i]); - sum += output[i]; - } - - for (std::size_t i = 0; i < NumberOfFeatures; ++i) - { - output[i] = output[i] / sum; - output[i] = std::max(std::min(output[i], QNumberType(0.9999f)), QNumberType(0.0001f)); - } - - return output; - } - - template - OPTIMIZE_FOR_SPEED - QNumberType - CategoricalCrossEntropy::Cost(const Vector& parameters) - { - Vector probabilities = ComputeSoftmaxProbabilities(parameters); - QNumberType cost = QNumberType(0.0f); - - for (std::size_t i = 0; i < NumberOfFeatures; ++i) - cost += -target[i] * math::Log(math::ToFloat(probabilities[i])); - - return cost + regularization.Calculate(parameters); - } - - template - typename CategoricalCrossEntropy::Vector - OPTIMIZE_FOR_SPEED - CategoricalCrossEntropy::Gradient(const Vector& parameters) - { - auto regGradient = regularization.Gradient(parameters); - Vector probabilities = ComputeSoftmaxProbabilities(parameters); - Vector gradient; - - for (std::size_t i = 0; i < NumberOfFeatures; ++i) - gradient[i] = probabilities[i] - target[i] + regGradient[i]; - - return gradient; - } - -#ifdef NUMERICAL_TOOLBOX_COVERAGE_BUILD - extern template class CategoricalCrossEntropy; -#endif -} diff --git a/numerical/neural_network/losses/Loss.hpp b/numerical/neural_network/losses/Loss.hpp deleted file mode 100644 index 64356ba1..00000000 --- a/numerical/neural_network/losses/Loss.hpp +++ /dev/null @@ -1,17 +0,0 @@ -#pragma once - -#include "numerical/optimization/ObjectiveFunction.hpp" - -namespace neural_network -{ - template - class Loss - : public optimization::ObjectiveFunction - { - static_assert(math::is_qnumber_v || std::is_floating_point_v, - "Loss can only be instantiated with math::QNumber types."); - - public: - using Vector = math::Vector; - }; -} diff --git a/numerical/neural_network/losses/MeanAbsoluteError.cpp b/numerical/neural_network/losses/MeanAbsoluteError.cpp deleted file mode 100644 index 7a5c6aa6..00000000 --- a/numerical/neural_network/losses/MeanAbsoluteError.cpp +++ /dev/null @@ -1,8 +0,0 @@ -#include "numerical/neural_network/losses/MeanAbsoluteError.hpp" - -namespace neural_network -{ - template class MeanAbsoluteError; - template class MeanAbsoluteError; - template class MeanAbsoluteError; -} diff --git a/numerical/neural_network/losses/MeanAbsoluteError.hpp b/numerical/neural_network/losses/MeanAbsoluteError.hpp deleted file mode 100644 index e92392ac..00000000 --- a/numerical/neural_network/losses/MeanAbsoluteError.hpp +++ /dev/null @@ -1,75 +0,0 @@ -#pragma once - -#if defined(__GNUC__) || defined(__clang__) -#pragma GCC optimize("O3", "fast-math") -#endif - -#include "numerical/math/CompilerOptimizations.hpp" -#include "numerical/neural_network/losses/Loss.hpp" -#include "numerical/regularization/Regularization.hpp" - -namespace neural_network -{ - template - class MeanAbsoluteError - : public Loss - { - public: - using Vector = typename Loss::Vector; - - MeanAbsoluteError(const Vector& target, regularization::Regularization& regularization); - QNumberType Cost(const Vector& parameters) override; - Vector Gradient(const Vector& parameters) override; - - private: - Vector target; - regularization::Regularization& regularization; - }; - - // Implementation // - - template - MeanAbsoluteError::MeanAbsoluteError(const Vector& target, regularization::Regularization& regularization) - : target(target) - , regularization(regularization) - {} - - template - OPTIMIZE_FOR_SPEED - QNumberType - MeanAbsoluteError::Cost(const Vector& parameters) - { - QNumberType cost = QNumberType(0.0f); - - for (std::size_t i = 0; i < NumberOfFeatures; ++i) - { - auto diff = parameters[i] - target[i]; - cost += diff < QNumberType(0.0f) ? -diff : diff; - } - - return cost + regularization.Calculate(parameters); - } - - template - OPTIMIZE_FOR_SPEED - typename MeanAbsoluteError::Vector - MeanAbsoluteError::Gradient(const Vector& parameters) - { - Vector gradient; - auto regGradient = regularization.Gradient(parameters); - - for (std::size_t i = 0; i < NumberOfFeatures; ++i) - { - auto diff = parameters[i] - target[i]; - gradient[i] = (diff > QNumberType(0.0f) ? QNumberType(0.9999f) : QNumberType(-0.9999f)) + regGradient[i]; - } - - return gradient; - } - -#ifdef NUMERICAL_TOOLBOX_COVERAGE_BUILD - extern template class MeanAbsoluteError; - extern template class MeanAbsoluteError; - extern template class MeanAbsoluteError; -#endif -} diff --git a/numerical/neural_network/losses/MeanSquaredError.cpp b/numerical/neural_network/losses/MeanSquaredError.cpp deleted file mode 100644 index d30ca40c..00000000 --- a/numerical/neural_network/losses/MeanSquaredError.cpp +++ /dev/null @@ -1,8 +0,0 @@ -#include "numerical/neural_network/losses/MeanSquaredError.hpp" - -namespace neural_network -{ - template class MeanSquaredError; - template class MeanSquaredError; - template class MeanSquaredError; -} diff --git a/numerical/neural_network/losses/MeanSquaredError.hpp b/numerical/neural_network/losses/MeanSquaredError.hpp deleted file mode 100644 index 33003ff7..00000000 --- a/numerical/neural_network/losses/MeanSquaredError.hpp +++ /dev/null @@ -1,73 +0,0 @@ -#pragma once - -#if defined(__GNUC__) || defined(__clang__) -#pragma GCC optimize("O3", "fast-math") -#endif - -#include "numerical/math/CompilerOptimizations.hpp" -#include "numerical/neural_network/losses/Loss.hpp" -#include "numerical/regularization/Regularization.hpp" - -namespace neural_network -{ - template - class MeanSquaredError - : public Loss - { - public: - using Vector = typename Loss::Vector; - - MeanSquaredError(const Vector& target, regularization::Regularization& regularization); - - QNumberType Cost(const Vector& parameters) override; - Vector Gradient(const Vector& parameters) override; - - private: - Vector target; - regularization::Regularization& regularization; - }; - - // Implementation // - - template - MeanSquaredError::MeanSquaredError(const Vector& target, regularization::Regularization& regularization) - : target(target) - , regularization(regularization) - {} - - template - OPTIMIZE_FOR_SPEED - QNumberType - MeanSquaredError::Cost(const Vector& parameters) - { - QNumberType cost = QNumberType(0.0f); - - for (std::size_t i = 0; i < NumberOfFeatures; ++i) - { - QNumberType diff = parameters[i] - target[i]; - cost += diff * diff; - } - - return QNumberType(math::ToFloat(cost) / 2.0f) + regularization.Calculate(parameters); - } - - template - OPTIMIZE_FOR_SPEED - typename MeanSquaredError::Vector - MeanSquaredError::Gradient(const Vector& parameters) - { - Vector gradient; - auto regGradient = regularization.Gradient(parameters); - - for (std::size_t i = 0; i < NumberOfFeatures; ++i) - gradient[i] = parameters[i] - target[i] + regGradient[i]; - - return gradient; - } - -#ifdef NUMERICAL_TOOLBOX_COVERAGE_BUILD - extern template class MeanSquaredError; - extern template class MeanSquaredError; - extern template class MeanSquaredError; -#endif -} diff --git a/numerical/neural_network/losses/test/CMakeLists.txt b/numerical/neural_network/losses/test/CMakeLists.txt deleted file mode 100644 index efc8684d..00000000 --- a/numerical/neural_network/losses/test/CMakeLists.txt +++ /dev/null @@ -1,17 +0,0 @@ -add_executable(numerical.neural_network.losses_test) -emil_build_for(numerical.neural_network.losses_test BOOL NUMERICAL_TOOLBOX_BUILD_TESTS) -emil_add_test(numerical.neural_network.losses_test) - -target_link_libraries(numerical.neural_network.losses_test PUBLIC - gmock_main - numerical.neural_network.losses -) - -numerical_add_qemu_test(numerical.neural_network.losses_test) - -target_sources(numerical.neural_network.losses_test PRIVATE - TestBinaryCrossEntropy.cpp - TestCategoricalCrossEntropy.cpp - TestMeanAbsoluteError.cpp - TestMeanSquaredError.cpp -) diff --git a/numerical/neural_network/losses/test/TestBinaryCrossEntropy.cpp b/numerical/neural_network/losses/test/TestBinaryCrossEntropy.cpp deleted file mode 100644 index 05a7515f..00000000 --- a/numerical/neural_network/losses/test/TestBinaryCrossEntropy.cpp +++ /dev/null @@ -1,143 +0,0 @@ -#include "numerical/neural_network/losses/BinaryCrossEntropy.hpp" -#include "gmock/gmock.h" - -namespace -{ - template - class MockRegularization - : public regularization::Regularization - { - public: - using Vector = typename regularization::Regularization::Vector; - MOCK_METHOD(QNumberType, Calculate, (const Vector& parameters), (const, override)); - MOCK_METHOD(Vector, Gradient, (const Vector& parameters), (const, override)); - }; - - template - class TestBinaryCrossEntropy - : public ::testing::Test - { - public: - static constexpr std::size_t NumberOfFeature = 2; - using Vector = math::Matrix; - - MockRegularization mockRegularization; - Vector target; - std::optional> loss; - - void SetUp() override - { - target = Vector{ T(0.8f), T(0.2f) }; - } - }; - - using TestedTypes = ::testing::Types; - TYPED_TEST_SUITE(TestBinaryCrossEntropy, TestedTypes); -} - -TYPED_TEST(TestBinaryCrossEntropy, CostCallsRegularization) -{ - typename TestBinaryCrossEntropy::Vector parameters{ TypeParam(0.7f), TypeParam(0.3f) }; - TypeParam regValue = TypeParam(0.1f); - - EXPECT_CALL(this->mockRegularization, Calculate(::testing::_)) - .WillOnce(::testing::Return(regValue)); - - this->loss.emplace(this->target, this->mockRegularization); - TypeParam cost = this->loss->Cost(parameters); - - TypeParam expectedCost = TypeParam(0.0f); - for (std::size_t i = 0; i < this->NumberOfFeature; ++i) - { - TypeParam pred = std::max(std::min(parameters[i], TypeParam(0.9999f)), TypeParam(0.0001f)); - expectedCost += -(this->target[i] * std::log(math::ToFloat(pred)) + - (TypeParam(0.9999f) - this->target[i]) * std::log(math::ToFloat(TypeParam(0.9999f) - pred))); - } - - EXPECT_NEAR(math::ToFloat(cost), math::ToFloat(expectedCost + regValue), 0.1f); -} - -TYPED_TEST(TestBinaryCrossEntropy, PerfectPrediction) -{ - typename TestBinaryCrossEntropy::Vector parameters = this->target; - TypeParam regValue = TypeParam(0.0f); - - EXPECT_CALL(this->mockRegularization, Calculate(::testing::_)) - .WillOnce(::testing::Return(regValue)); - - this->loss.emplace(this->target, this->mockRegularization); - TypeParam cost = this->loss->Cost(parameters); - - // Due to clamping, perfect prediction won't reach exactly zero - EXPECT_GT(math::ToFloat(cost), 0.0f); - EXPECT_LT(math::ToFloat(cost), 1.1f); -} - -TYPED_TEST(TestBinaryCrossEntropy, WorstPrediction) -{ - typename TestBinaryCrossEntropy::Vector parameters{ TypeParam(0.1f), TypeParam(0.9f) }; - TypeParam regValue = TypeParam(0.0f); - - EXPECT_CALL(this->mockRegularization, Calculate(::testing::_)) - .WillOnce(::testing::Return(regValue)); - - this->loss.emplace(this->target, this->mockRegularization); - TypeParam cost = this->loss->Cost(parameters); - - // Worst prediction should have significantly higher cost than best prediction - EXPECT_GT(math::ToFloat(cost), 1.0f); -} - -TYPED_TEST(TestBinaryCrossEntropy, GradientTest) -{ - typename TestBinaryCrossEntropy::Vector parameters{ TypeParam(0.6f), TypeParam(0.4f) }; - TypeParam regValue = TypeParam(0.1f); - - typename TestBinaryCrossEntropy::Vector regGradient; - for (std::size_t i = 0; i < this->NumberOfFeature; ++i) - regGradient[i] = regValue; - - EXPECT_CALL(this->mockRegularization, Gradient(::testing::_)) - .WillOnce(::testing::Return(regGradient)); - - this->loss.emplace(this->target, this->mockRegularization); - auto gradient = this->loss->Gradient(parameters); - - for (std::size_t i = 0; i < this->NumberOfFeature; ++i) - { - TypeParam pred = std::max(std::min(parameters[i], TypeParam(0.9999f)), TypeParam(0.0001f)); - TypeParam expectedGrad = (pred - this->target[i]) / (pred * (TypeParam(0.9999f) - pred)) + regValue; - EXPECT_NEAR(math::ToFloat(gradient[i]), math::ToFloat(expectedGrad), 0.15f); - } -} - -TYPED_TEST(TestBinaryCrossEntropy, MultipleParameters) -{ - std::array::Vector, 2> testParameters = { - typename TestBinaryCrossEntropy::Vector{ TypeParam(0.3f), TypeParam(0.7f) }, - typename TestBinaryCrossEntropy::Vector{ TypeParam(0.9f), TypeParam(0.1f) } - }; - - TypeParam regValue = TypeParam(0.2f); - - EXPECT_CALL(this->mockRegularization, Calculate(::testing::_)) - .Times(testParameters.size()) - .WillRepeatedly(::testing::Return(regValue)); - - this->loss.emplace(this->target, this->mockRegularization); - - for (const auto& params : testParameters) - { - TypeParam cost = this->loss->Cost(params); - - TypeParam expectedCost = TypeParam(0.0f); - for (std::size_t i = 0; i < this->NumberOfFeature; ++i) - { - TypeParam pred = std::max(std::min(params[i], TypeParam(0.9999f)), TypeParam(0.0001f)); - expectedCost += -(this->target[i] * std::log(math::ToFloat(pred)) + - (TypeParam(0.9999f) - this->target[i]) * std::log(math::ToFloat(TypeParam(0.9999f) - pred))); - } - - EXPECT_NEAR(math::ToFloat(cost), math::ToFloat(expectedCost + regValue), 0.1f); - } -} diff --git a/numerical/neural_network/losses/test/TestCategoricalCrossEntropy.cpp b/numerical/neural_network/losses/test/TestCategoricalCrossEntropy.cpp deleted file mode 100644 index fdf08319..00000000 --- a/numerical/neural_network/losses/test/TestCategoricalCrossEntropy.cpp +++ /dev/null @@ -1,156 +0,0 @@ -#include "numerical/math/QNumber.hpp" -#include "numerical/neural_network/losses/CategoricalCrossEntropy.hpp" -#include "gmock/gmock.h" - -namespace -{ - template - class MockRegularization - : public regularization::Regularization - { - public: - using Vector = typename regularization::Regularization::Vector; - MOCK_METHOD(QNumberType, Calculate, (const Vector& parameters), (const, override)); - MOCK_METHOD(Vector, Gradient, (const Vector& parameters), (const, override)); - }; - - template - class TestCategoricalCrossEntropy - : public ::testing::Test - { - public: - static constexpr std::size_t NumberOfFeature = 2; - using Vector = math::Matrix; - - MockRegularization mockRegularization; - Vector target; - std::optional> loss; - - void SetUp() override - { - target = Vector{ T(0.8f), T(0.2f) }; - } - }; - - using TestedTypes = ::testing::Types; - TYPED_TEST_SUITE(TestCategoricalCrossEntropy, TestedTypes); -} - -TYPED_TEST(TestCategoricalCrossEntropy, CostCallsRegularization) -{ - typename TestCategoricalCrossEntropy::Vector parameters{ TypeParam(0.7f), TypeParam(0.3f) }; - TypeParam regValue = TypeParam(0.1f); - - EXPECT_CALL(this->mockRegularization, Calculate(::testing::_)) - .WillOnce(::testing::Return(regValue)); - - this->loss.emplace(this->target, this->mockRegularization); - TypeParam cost = this->loss->Cost(parameters); - - typename TestCategoricalCrossEntropy::Vector probs; - TypeParam sum = TypeParam(0.0f); - - for (std::size_t i = 0; i < this->NumberOfFeature; ++i) - { - probs[i] = std::exp(math::ToFloat(parameters[i])); - sum += probs[i]; - } - - for (std::size_t i = 0; i < this->NumberOfFeature; ++i) - probs[i] = probs[i] / sum; - - TypeParam expectedCost = TypeParam(0.0f); - for (std::size_t i = 0; i < this->NumberOfFeature; ++i) - expectedCost += -this->target[i] * std::log(math::ToFloat(probs[i])); - - EXPECT_NEAR(math::ToFloat(cost), math::ToFloat(expectedCost + regValue), 0.01f); -} - -TYPED_TEST(TestCategoricalCrossEntropy, PerfectPrediction) -{ - typename TestCategoricalCrossEntropy::Vector parameters{ TypeParam(2.0f), TypeParam(-2.0f) }; - TypeParam regValue = TypeParam(0.0f); - - EXPECT_CALL(this->mockRegularization, Calculate(::testing::_)) - .WillOnce(::testing::Return(regValue)); - - this->loss.emplace(this->target, this->mockRegularization); - TypeParam cost = this->loss->Cost(parameters); - - EXPECT_GT(math::ToFloat(cost), 0.0f); - EXPECT_LT(math::ToFloat(cost), 1.0f); -} - -TYPED_TEST(TestCategoricalCrossEntropy, GradientTest) -{ - typename TestCategoricalCrossEntropy::Vector parameters{ TypeParam(0.2f), TypeParam(0.5f) }; - TypeParam regValue = TypeParam(0.1f); - - typename TestCategoricalCrossEntropy::Vector regGradient; - for (std::size_t i = 0; i < this->NumberOfFeature; ++i) - regGradient[i] = regValue; - - EXPECT_CALL(this->mockRegularization, Gradient(::testing::_)) - .WillOnce(::testing::Return(regGradient)); - - this->loss.emplace(this->target, this->mockRegularization); - auto gradient = this->loss->Gradient(parameters); - - typename TestCategoricalCrossEntropy::Vector probs; - TypeParam sum = TypeParam(0.0f); - - for (std::size_t i = 0; i < this->NumberOfFeature; ++i) - { - probs[i] = std::exp(math::ToFloat(parameters[i])); - sum += probs[i]; - } - - for (std::size_t i = 0; i < this->NumberOfFeature; ++i) - probs[i] = probs[i] / sum; - - // Account for limited precision in Q types - EXPECT_NEAR(math::ToFloat(gradient[0]), math::ToFloat(probs[0] - this->target[0] + regValue), 0.1f); - EXPECT_NEAR(math::ToFloat(gradient[1]), math::ToFloat(probs[1] - this->target[1] + regValue), 0.1f); -} - -TYPED_TEST(TestCategoricalCrossEntropy, MultipleParameters) -{ - std::array::Vector, 2> testParameters = { - typename TestCategoricalCrossEntropy::Vector{ TypeParam(0.3f), TypeParam(0.7f) }, - typename TestCategoricalCrossEntropy::Vector{ TypeParam(0.1f), TypeParam(0.9f) } - }; - - TypeParam regValue = TypeParam(0.2f); - - EXPECT_CALL(this->mockRegularization, Calculate(::testing::_)) - .Times(testParameters.size()) - .WillRepeatedly(::testing::Return(regValue)); - - this->loss.emplace(this->target, this->mockRegularization); - - for (const auto& params : testParameters) - { - TypeParam cost = this->loss->Cost(params); - - typename TestCategoricalCrossEntropy::Vector probs; - TypeParam sum = TypeParam(0.0f); - - for (std::size_t i = 0; i < this->NumberOfFeature; ++i) - { - probs[i] = TypeParam(std::exp(math::ToFloat(params[i]))); - sum += probs[i]; - } - - for (std::size_t i = 0; i < this->NumberOfFeature; ++i) - { - probs[i] = probs[i] / sum; - probs[i] = TypeParam(std::max(std::min(math::ToFloat(probs[i]), 0.8f), 0.0001f)); - } - - TypeParam expectedCost = TypeParam(0.0f); - for (std::size_t i = 0; i < this->NumberOfFeature; ++i) - expectedCost += -this->target[i] * std::log(math::ToFloat(probs[i])); - - EXPECT_NEAR(math::ToFloat(cost), math::ToFloat(expectedCost + regValue), 0.1f); - } -} diff --git a/numerical/neural_network/losses/test/TestMeanAbsoluteError.cpp b/numerical/neural_network/losses/test/TestMeanAbsoluteError.cpp deleted file mode 100644 index 2ea448b3..00000000 --- a/numerical/neural_network/losses/test/TestMeanAbsoluteError.cpp +++ /dev/null @@ -1,109 +0,0 @@ -#include "numerical/neural_network/losses/MeanAbsoluteError.hpp" -#include "gmock/gmock.h" - -namespace -{ - template - class MockRegularization - : public regularization::Regularization - { - public: - using Vector = typename regularization::Regularization::Vector; - MOCK_METHOD(QNumberType, Calculate, (const Vector& parameters), (const, override)); - MOCK_METHOD(Vector, Gradient, (const Vector& parameters), (const, override)); - }; - - template - class TestMeanAbsoluteError - : public ::testing::Test - { - public: - static constexpr std::size_t NumberOfFeature = 2; - using Vector = math::Matrix; - - MockRegularization mockRegularization; - Vector target; - std::optional> loss; - - void SetUp() override - { - target = Vector{ T(0.5f), T(-0.3f) }; - } - }; - - using TestedTypes = ::testing::Types; - TYPED_TEST_SUITE(TestMeanAbsoluteError, TestedTypes); -} - -TYPED_TEST(TestMeanAbsoluteError, CostCallsRegularization) -{ - typename TestMeanAbsoluteError::Vector parameters{ TypeParam(0.7f), TypeParam(-0.1f) }; - TypeParam regValue = TypeParam(0.1f); - - EXPECT_CALL(this->mockRegularization, Calculate(::testing::_)) - .WillOnce(::testing::Return(regValue)); - - this->loss.emplace(this->target, this->mockRegularization); - TypeParam cost = this->loss->Cost(parameters); - - TypeParam maeValue = TypeParam(0.0f); - TypeParam diff1 = parameters[0] - this->target[0]; - TypeParam diff2 = parameters[1] - this->target[1]; - maeValue += (diff1 < TypeParam(0.0f) ? -diff1 : diff1) + (diff2 < TypeParam(0.0f) ? -diff2 : diff2); - - EXPECT_NEAR(math::ToFloat(cost), math::ToFloat(maeValue + regValue), 0.001f); -} - -TYPED_TEST(TestMeanAbsoluteError, ZeroError) -{ - typename TestMeanAbsoluteError::Vector parameters = this->target; - TypeParam regValue = TypeParam(0.0f); - - EXPECT_CALL(this->mockRegularization, Calculate(::testing::_)) - .WillOnce(::testing::Return(regValue)); - - this->loss.emplace(this->target, this->mockRegularization); - TypeParam cost = this->loss->Cost(parameters); - - EXPECT_NEAR(math::ToFloat(cost), 0.0f, 0.001f); -} - -TYPED_TEST(TestMeanAbsoluteError, MultipleParameters) -{ - std::array::Vector, 2> testParameters = { - typename TestMeanAbsoluteError::Vector{ TypeParam(0.4f), TypeParam(-0.4f) }, - typename TestMeanAbsoluteError::Vector{ TypeParam(0.6f), TypeParam(-0.2f) } - }; - - TypeParam regValue = TypeParam(0.2f); - - EXPECT_CALL(this->mockRegularization, Calculate(::testing::_)) - .Times(testParameters.size()) - .WillRepeatedly(::testing::Return(regValue)); - - this->loss.emplace(this->target, this->mockRegularization); - - for (const auto& params : testParameters) - { - TypeParam cost = this->loss->Cost(params); - - TypeParam maeValue = TypeParam(0.0f); - TypeParam diff1 = params[0] - this->target[0]; - TypeParam diff2 = params[1] - this->target[1]; - maeValue += (diff1 < TypeParam(0.0f) ? -diff1 : diff1) + (diff2 < TypeParam(0.0f) ? -diff2 : diff2); - - EXPECT_NEAR(math::ToFloat(cost), math::ToFloat(maeValue + regValue), 0.001f); - } -} - -TYPED_TEST(TestMeanAbsoluteError, GradientTest) -{ - typename TestMeanAbsoluteError::Vector parameters{ TypeParam(0.7f), TypeParam(-0.6f) }; - TypeParam regValue = TypeParam(0.1f); - - this->loss.emplace(this->target, this->mockRegularization); - auto gradient = this->loss->Gradient(parameters); - - EXPECT_NEAR(math::ToFloat(gradient[0]), 0.9999f, 0.01f); - EXPECT_NEAR(math::ToFloat(gradient[1]), -0.9999f, 0.01f); -} diff --git a/numerical/neural_network/losses/test/TestMeanSquaredError.cpp b/numerical/neural_network/losses/test/TestMeanSquaredError.cpp deleted file mode 100644 index 830f51a1..00000000 --- a/numerical/neural_network/losses/test/TestMeanSquaredError.cpp +++ /dev/null @@ -1,117 +0,0 @@ -#include "numerical/neural_network/losses/MeanSquaredError.hpp" -#include "gmock/gmock.h" - -namespace -{ - template - class MockRegularization - : public regularization::Regularization - { - public: - using Vector = typename regularization::Regularization::Vector; - MOCK_METHOD(QNumberType, Calculate, (const Vector& parameters), (const, override)); - MOCK_METHOD(Vector, Gradient, (const Vector& parameters), (const, override)); - }; - - template - class TestMeanSquaredError - : public ::testing::Test - { - public: - static constexpr std::size_t NumberOfFeature = 2; - using Vector = math::Matrix; - - MockRegularization mockRegularization; - Vector target; - std::optional> loss; - - void SetUp() override - { - target = Vector{ T(0.5f), T(-0.3f) }; - } - }; - - using TestedTypes = ::testing::Types; - TYPED_TEST_SUITE(TestMeanSquaredError, TestedTypes); -} - -TYPED_TEST(TestMeanSquaredError, CostCallsRegularization) -{ - typename TestMeanSquaredError::Vector parameters{ TypeParam(0.7f), TypeParam(-0.1f) }; - TypeParam regValue = TypeParam(0.1f); - - EXPECT_CALL(this->mockRegularization, Calculate(::testing::_)) - .WillOnce(::testing::Return(regValue)); - - this->loss.emplace(this->target, this->mockRegularization); - TypeParam cost = this->loss->Cost(parameters); - - TypeParam mseValue = TypeParam(0.0f); - TypeParam diff1 = parameters[0] - this->target[0]; - TypeParam diff2 = parameters[1] - this->target[1]; - mseValue += diff1 * diff1 + diff2 * diff2; - mseValue = TypeParam(math::ToFloat(mseValue) / 2.0f); - - EXPECT_NEAR(math::ToFloat(cost), math::ToFloat(mseValue + regValue), 0.001f); -} - -TYPED_TEST(TestMeanSquaredError, ZeroError) -{ - typename TestMeanSquaredError::Vector parameters = this->target; - TypeParam regValue = TypeParam(0.0f); - - EXPECT_CALL(this->mockRegularization, Calculate(::testing::_)) - .WillOnce(::testing::Return(regValue)); - - this->loss.emplace(this->target, this->mockRegularization); - TypeParam cost = this->loss->Cost(parameters); - - EXPECT_NEAR(math::ToFloat(cost), 0.0f, 0.001f); -} - -TYPED_TEST(TestMeanSquaredError, MultipleParameters) -{ - std::array::Vector, 2> testParameters = { - typename TestMeanSquaredError::Vector{ TypeParam(0.4f), TypeParam(-0.4f) }, - typename TestMeanSquaredError::Vector{ TypeParam(0.6f), TypeParam(-0.2f) } - }; - - TypeParam regValue = TypeParam(0.2f); - - EXPECT_CALL(this->mockRegularization, Calculate(::testing::_)) - .Times(testParameters.size()) - .WillRepeatedly(::testing::Return(regValue)); - - this->loss.emplace(this->target, this->mockRegularization); - - for (const auto& params : testParameters) - { - TypeParam cost = this->loss->Cost(params); - - TypeParam mseValue = TypeParam(0.0f); - TypeParam diff1 = params[0] - this->target[0]; - TypeParam diff2 = params[1] - this->target[1]; - mseValue += diff1 * diff1 + diff2 * diff2; - mseValue = TypeParam(math::ToFloat(mseValue) / 2.0f); - - EXPECT_NEAR(math::ToFloat(cost), math::ToFloat(mseValue + regValue), 0.001f); - } -} - -TYPED_TEST(TestMeanSquaredError, GradientIncludesRegularization) -{ - typename TestMeanSquaredError::Vector parameters{ TypeParam(0.7f), TypeParam(-0.1f) }; - typename TestMeanSquaredError::Vector regGradient{ TypeParam(0.01f), TypeParam(-0.02f) }; - - EXPECT_CALL(this->mockRegularization, Gradient(::testing::_)) - .WillOnce(::testing::Return(regGradient)); - - this->loss.emplace(this->target, this->mockRegularization); - auto gradient = this->loss->Gradient(parameters); - - for (std::size_t i = 0; i < TestMeanSquaredError::NumberOfFeature; ++i) - { - float expected = math::ToFloat(parameters[i] - this->target[i] + regGradient[i]); - EXPECT_NEAR(math::ToFloat(gradient[i]), expected, 0.001f); - } -} diff --git a/numerical/neural_network/model/CMakeLists.txt b/numerical/neural_network/model/CMakeLists.txt deleted file mode 100644 index 35ea07ab..00000000 --- a/numerical/neural_network/model/CMakeLists.txt +++ /dev/null @@ -1,21 +0,0 @@ -numerical_add_header_library(numerical.neural_network.model) - -target_include_directories(numerical.neural_network.model ${NUMERICAL_VISIBILITY} - "$" - "$" -) - -target_link_libraries(numerical.neural_network.model ${NUMERICAL_VISIBILITY} - infra.util - numerical.math - numerical.neural_network.activation - numerical.neural_network.layer - numerical.neural_network.losses - numerical.optimization -) - -target_sources(numerical.neural_network.model PRIVATE - Model.hpp -) - -add_subdirectory(test) diff --git a/numerical/neural_network/model/Model.hpp b/numerical/neural_network/model/Model.hpp deleted file mode 100644 index 8b1d0145..00000000 --- a/numerical/neural_network/model/Model.hpp +++ /dev/null @@ -1,287 +0,0 @@ -#pragma once - -#if defined(__GNUC__) || defined(__clang__) -#pragma GCC optimize("O3", "fast-math") -#endif - -#include "infra/util/BoundedVector.hpp" -#include "numerical/math/CompilerOptimizations.hpp" -#include "numerical/neural_network/layer/Layer.hpp" -#include "numerical/neural_network/losses/Loss.hpp" -#include "numerical/optimization/Optimizer.hpp" -#include -#include -#include - -namespace neural_network -{ - template - auto make_layer(Args&&... args) - { - return [args = std::make_tuple(std::forward(args)...)]() mutable - { - return std::apply([](auto&&... params) - { - return Layer(std::forward(params)...); - }, - std::move(args)); - }; - } - - namespace detail - { - template - struct is_layer - { - static constexpr bool value = std::is_base_of_v< - Layer, - T>; - }; - - template - struct all_are_layers; - - template - struct all_are_layers : std::true_type - {}; - - template - struct all_are_layers - : std::integral_constant::value && - all_are_layers::value> - {}; - - template - struct verify_layer_sizes; - - template - struct verify_layer_sizes - { - static constexpr bool value = true; - }; - - template - struct verify_layer_sizes - { - using FirstLayer = Layer; - - static constexpr bool value = - (InputSize == FirstLayer::InputSize) && - verify_layer_sizes::value; - }; - - template - constexpr std::size_t calculate_total_parameters() - { - return (... + Layers::ParameterSize); - } - } - - template - class Model - { - static_assert(math::is_qnumber::value || std::is_floating_point::value, - "Model can only be instantiated with math::QNumber types or floating point types."); - - static_assert(detail::all_are_layers::value, - "All types in Layers must derive from Layer"); - - static_assert(sizeof...(Layers) > 0, "Model must have at least one layer"); - - static_assert(detail::verify_layer_sizes::value && - std::tuple_element_t>::OutputSize == OutputSize, - "Layer sizes do not match"); - - public: - using InputVector = math::Matrix; - using OutputVector = math::Matrix; - using ParameterVector = math::Vector()>; - static constexpr std::size_t TotalParameters = detail::calculate_total_parameters(); - - Model(); - - template> - Model(FactoryFuncs&&... factories) - : layers(std::invoke(std::forward(factories))...) - {} - - OutputVector Forward(const InputVector& input); - InputVector Backward(const OutputVector& output_gradient); - void Train(optimization::Optimizer& optimizer, Loss& loss, const ParameterVector& initialParameters); - void SetParameters(const ParameterVector& parameters); - ParameterVector GetParameters() const; - - private: - template - OutputVector ForwardImpl(const InputVector& input, std::index_sequence); - - template - void ForwardLayer(const InputType& layerInput); - - template - InputVector BackwardImpl(const OutputVector& output_gradient, std::index_sequence); - - template - InputVector BackwardLayer(const GradType& gradient); - - template - void SetParametersImpl(const ParameterVector& parameters, std::index_sequence); - - template - void SetLayerParameters(Layer& layer, const ParameterVector& parameters, std::size_t& offset); - - template - ParameterVector GetParametersImpl(std::index_sequence) const; - - template - void GetLayerParameters(const Layer& layer, ParameterVector& parameters, std::size_t& offset) const; - - std::tuple layers; - InputVector currentInput; - }; - - template - Model::Model() - : layers(std::make_tuple(Layers()...)) - {} - - template - typename Model::OutputVector - OPTIMIZE_FOR_SPEED - Model::Forward(const InputVector& input) - { - currentInput = input; - return ForwardImpl(input, std::make_index_sequence{}); - } - - template - typename Model::InputVector - OPTIMIZE_FOR_SPEED - Model::Backward(const OutputVector& output_gradient) - { - return BackwardImpl(output_gradient, std::make_index_sequence{}); - } - - template - void Model::Train( - optimization::Optimizer& optimizer, - Loss& loss, - const ParameterVector& initialParameters) - { - auto result = optimizer.Minimize(initialParameters, loss); - SetParameters(result.parameters); - } - - template - void Model::SetParameters(const ParameterVector& parameters) - { - SetParametersImpl(parameters, std::make_index_sequence{}); - } - - template - typename Model::ParameterVector - Model::GetParameters() const - { - return GetParametersImpl(std::make_index_sequence{}); - } - - template - template - typename Model::OutputVector - Model::ForwardImpl(const InputVector& input, std::index_sequence) - { - ForwardLayer<0>(input); - return std::get(layers).Output(); - } - - template - template - void Model::ForwardLayer(const InputType& layerInput) - { - std::get(layers).Forward(layerInput); - if constexpr (I + 1 < sizeof...(Layers)) - ForwardLayer(std::get(layers).Output()); - } - - template - template - typename Model::InputVector - Model::BackwardImpl(const OutputVector& output_gradient, std::index_sequence) - { - return BackwardLayer(output_gradient); - } - - template - template - typename Model::InputVector - Model::BackwardLayer(const GradType& gradient) - { - auto& inputGrad = std::get(layers).Backward(gradient); - if constexpr (I == 0) - return inputGrad; - else - return BackwardLayer(inputGrad); - } - - template - template - void Model::SetParametersImpl( - const ParameterVector& parameters, - std::index_sequence) - { - std::size_t offset = 0; - (SetLayerParameters(std::get(layers), parameters, offset), ...); - } - - template - template - void Model::SetLayerParameters( - Layer& layer, - const ParameterVector& parameters, - std::size_t& offset) - { - const std::size_t layerParameterSize = Layer::ParameterSize; - math::Vector layerParameters; - - for (std::size_t i = 0; i < layerParameterSize; ++i) - layerParameters[i] = parameters[offset + i]; - - layer.SetParameters(layerParameters); - offset += layerParameterSize; - } - - template - template - typename Model::ParameterVector - Model::GetParametersImpl( - std::index_sequence) const - { - ParameterVector parameters; - std::size_t offset = 0; - int dummy[] = { 0, (GetLayerParameters(std::get(layers), parameters, offset), 0)... }; - (void)dummy; - return parameters; - } - - template - template - void Model::GetLayerParameters( - const Layer& layer, - ParameterVector& parameters, - std::size_t& offset) const - { - const auto& layerParameters = layer.Parameters(); - for (std::size_t i = 0; i < Layer::ParameterSize; ++i) - parameters[offset + i] = layerParameters[i]; - offset += Layer::ParameterSize; - } -} diff --git a/numerical/neural_network/model/test/CMakeLists.txt b/numerical/neural_network/model/test/CMakeLists.txt deleted file mode 100644 index 1afc17cb..00000000 --- a/numerical/neural_network/model/test/CMakeLists.txt +++ /dev/null @@ -1,14 +0,0 @@ -add_executable(numerical.neural_network.model_test) -emil_build_for(numerical.neural_network.model_test BOOL NUMERICAL_TOOLBOX_BUILD_TESTS) -emil_add_test(numerical.neural_network.model_test) - -target_link_libraries(numerical.neural_network.model_test PUBLIC - gmock_main - numerical.neural_network.model -) - -numerical_add_qemu_test(numerical.neural_network.model_test) - -target_sources(numerical.neural_network.model_test PRIVATE - TestModel.cpp -) diff --git a/numerical/neural_network/model/test/TestModel.cpp b/numerical/neural_network/model/test/TestModel.cpp deleted file mode 100644 index 66491e07..00000000 --- a/numerical/neural_network/model/test/TestModel.cpp +++ /dev/null @@ -1,160 +0,0 @@ -#include "numerical/neural_network/layer/Layer.hpp" -#include "numerical/neural_network/model/Model.hpp" -#include "gmock/gmock.h" - -namespace -{ - template - class LayerMock - : public neural_network::Layer - { - public: - using BaseLayer = neural_network::Layer; - using InputVector = typename BaseLayer::InputVector; - using OutputVector = typename BaseLayer::OutputVector; - using ParameterVector = typename BaseLayer::ParameterVector; - using QNumberType = QNumberTypeMock; - static constexpr std::size_t InputSize = InputSizeMock; - static constexpr std::size_t OutputSize = OutputSizeMock; - static constexpr std::size_t ParameterSize = ParameterSizeMock; - - LayerMock(int mockParam = 0) - { - (void)mockParam; - setupDefaultBehavior(); - } - - LayerMock(const LayerMock& other) - : inputGradient(other.inputGradient) - , parameters(other.parameters) - { - setupDefaultBehavior(); - } - - LayerMock(LayerMock&& other) noexcept - : inputGradient(std::move(other.inputGradient)) - , parameters(std::move(other.parameters)) - { - setupDefaultBehavior(); - } - - LayerMock& operator=(const LayerMock&) = default; - LayerMock& operator=(LayerMock&&) = default; - - MOCK_METHOD(void, Forward, (const InputVector&), (override)); - MOCK_METHOD(InputVector&, Backward, (const OutputVector&), (override)); - MOCK_METHOD(const OutputVector&, Output, (), (const, override)); - MOCK_METHOD(ParameterVector&, Parameters, (), (const, override)); - MOCK_METHOD(void, SetParameters, (const ParameterVector&), (override)); - - private: - void setupDefaultBehavior() - { - ON_CALL(*this, Forward(testing::_)) - .WillByDefault([this](const InputVector& input) - { - (void)input; - }); - - ON_CALL(*this, Backward(testing::_)) - .WillByDefault([this](const OutputVector& output_gradient) -> InputVector& - { - (void)output_gradient; - return inputGradient; - }); - - ON_CALL(*this, Output()) - .WillByDefault([this]() -> const OutputVector& - { - return outputValue; - }); - - ON_CALL(*this, Parameters()) - .WillByDefault([this]() -> ParameterVector& - { - return parameters; - }); - - ON_CALL(*this, SetParameters(testing::_)) - .WillByDefault([this](const ParameterVector& p) - { - parameters = p; - }); - } - - InputVector inputGradient; - OutputVector outputValue; - ParameterVector parameters; - }; - - template - class TestModel - : public ::testing::Test - { - public: - static constexpr std::size_t InputSize = 1; - static constexpr std::size_t HiddenSize = 4; - static constexpr std::size_t OutputSize = 1; - static constexpr std::size_t ParameterSize = 2; - - using FirstLayer = LayerMock; - using SecondLayer = LayerMock; - using ModelType = neural_network::Model; - - TestModel() - : model(neural_network::make_layer(42), - neural_network::make_layer(43)) - {} - - ModelType model; - }; - - using TestedTypes = ::testing::Types; - TYPED_TEST_SUITE(TestModel, TestedTypes); -} - -TYPED_TEST(TestModel, LayerConstructionWithParameters) -{ - using FirstLayer = typename TestFixture::FirstLayer; - using SecondLayer = typename TestFixture::SecondLayer; - - auto firstLayerParam = 100; - auto secondLayerParam = 200; - - typename TestFixture::ModelType customModel( - neural_network::make_layer(firstLayerParam), - neural_network::make_layer(secondLayerParam)); -} - -TYPED_TEST(TestModel, SetParameters) -{ - using ParameterVector = math::Vector; - - ParameterVector parameters; - for (size_t i = 0; i < TestFixture::ModelType::TotalParameters; ++i) - parameters[i] = TypeParam(static_cast(i) * 0.1f); - - TestFixture::model.SetParameters(parameters); - - auto retrievedParams = TestFixture::model.GetParameters(); - - for (size_t i = 0; i < TestFixture::ModelType::TotalParameters; ++i) - EXPECT_EQ(math::ToFloat(retrievedParams[i]), math::ToFloat(parameters[i])); -} - -TYPED_TEST(TestModel, ConstructorVariants) -{ - typename TestFixture::ModelType defaultModel; - - typename TestFixture::ModelType factoryModel( - neural_network::make_layer(1), - neural_network::make_layer(2)); - - typename TestFixture::ModelType::InputVector input; - auto defaultOutput = defaultModel.Forward(input); - auto factoryOutput = factoryModel.Forward(input); - - typename TestFixture::ModelType::OutputVector outputGradient; - auto defaultGradient = defaultModel.Backward(outputGradient); - auto factoryGradient = factoryModel.Backward(outputGradient); -} diff --git a/simulator/CMakeLists.txt b/simulator/CMakeLists.txt index 24566f25..49cf9997 100644 --- a/simulator/CMakeLists.txt +++ b/simulator/CMakeLists.txt @@ -4,4 +4,3 @@ add_subdirectory(analysis) add_subdirectory(controllers) add_subdirectory(estimators) add_subdirectory(filters) -add_subdirectory(neural_network) diff --git a/simulator/neural_network/CMakeLists.txt b/simulator/neural_network/CMakeLists.txt deleted file mode 100644 index f53be19b..00000000 --- a/simulator/neural_network/CMakeLists.txt +++ /dev/null @@ -1 +0,0 @@ -add_subdirectory(NeuralNetwork) diff --git a/simulator/neural_network/NeuralNetwork/CMakeLists.txt b/simulator/neural_network/NeuralNetwork/CMakeLists.txt deleted file mode 100644 index d077418e..00000000 --- a/simulator/neural_network/NeuralNetwork/CMakeLists.txt +++ /dev/null @@ -1,24 +0,0 @@ -find_package(Qt6 REQUIRED COMPONENTS Widgets) - -add_subdirectory(application) -add_subdirectory(view) - -add_executable(numerical.simulator.neural_network.nn) -emil_build_for(numerical.simulator.neural_network.nn HOST All PREREQUISITE_BOOL NUMERICAL_TOOLBOX_BUILD_SIMULATOR) - -target_link_libraries(numerical.simulator.neural_network.nn PRIVATE - numerical.simulator.neural_network.nn.application - numerical.simulator.neural_network.nn.view -) - -target_sources(numerical.simulator.neural_network.nn PRIVATE - Main.cpp -) - -if(MSVC) - target_compile_options(numerical.simulator.neural_network.nn PRIVATE /W4 /wd4244 /wd4100) -else() - target_compile_options(numerical.simulator.neural_network.nn PRIVATE - -Wno-error - ) -endif() diff --git a/simulator/neural_network/NeuralNetwork/Main.cpp b/simulator/neural_network/NeuralNetwork/Main.cpp deleted file mode 100644 index 486713a4..00000000 --- a/simulator/neural_network/NeuralNetwork/Main.cpp +++ /dev/null @@ -1,12 +0,0 @@ -#include "simulator/neural_network/NeuralNetwork/view/NnMainWindow.hpp" -#include - -int main(int argc, char* argv[]) -{ - QApplication app(argc, argv); - - simulator::neural_network::nn::view::NnMainWindow window; - window.show(); - - return app.exec(); -} diff --git a/simulator/neural_network/NeuralNetwork/application/CMakeLists.txt b/simulator/neural_network/NeuralNetwork/application/CMakeLists.txt deleted file mode 100644 index 05411134..00000000 --- a/simulator/neural_network/NeuralNetwork/application/CMakeLists.txt +++ /dev/null @@ -1,21 +0,0 @@ -add_library(numerical.simulator.neural_network.nn.application STATIC) - -target_include_directories(numerical.simulator.neural_network.nn.application PUBLIC - "$" -) - -target_link_libraries(numerical.simulator.neural_network.nn.application PUBLIC - numerical.neural_network.model - numerical.regularization -) - -target_sources(numerical.simulator.neural_network.nn.application PRIVATE - NnSimulator.hpp - NnSimulator.cpp -) - -if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU" OR CMAKE_CXX_COMPILER_ID STREQUAL "Clang") - target_compile_options(numerical.simulator.neural_network.nn.application PRIVATE -Wno-error) -elseif(CMAKE_CXX_COMPILER_ID STREQUAL "MSVC") - target_compile_options(numerical.simulator.neural_network.nn.application PRIVATE /W4 /wd4244 /wd4100) -endif() diff --git a/simulator/neural_network/NeuralNetwork/application/NnSimulator.cpp b/simulator/neural_network/NeuralNetwork/application/NnSimulator.cpp deleted file mode 100644 index 2cea2d9a..00000000 --- a/simulator/neural_network/NeuralNetwork/application/NnSimulator.cpp +++ /dev/null @@ -1,210 +0,0 @@ -#include "simulator/neural_network/NeuralNetwork/application/NnSimulator.hpp" -#include -#include -#include -#include - -namespace simulator::neural_network::nn -{ - namespace - { - struct TrainingData - { - std::vector> inputs; - std::vector targets; - }; - - TrainingData GenerateXorData() - { - TrainingData data; - data.inputs = { { 0.0f, 0.0f }, { 0.0f, 1.0f }, { 1.0f, 0.0f }, { 1.0f, 1.0f } }; - data.targets = { 0.0f, 1.0f, 1.0f, 0.0f }; - return data; - } - - TrainingData GenerateSineData(std::size_t samples) - { - TrainingData data; - data.inputs.resize(samples); - data.targets.resize(samples); - - for (std::size_t i = 0; i < samples; ++i) - { - float x = static_cast(i) / static_cast(samples - 1) * 2.0f * std::numbers::pi_v; - data.inputs[i] = { x / (2.0f * std::numbers::pi_v)}; - data.targets[i] = (std::sin(x) + 1.0f) / 2.0f; - } - return data; - } - - float Sigmoid(float x) - { - return 1.0f / (1.0f + std::exp(-x)); - } - - float SigmoidDerivative(float output) - { - return output * (1.0f - output); - } - - float TanhActivation(float x) - { - return std::tanh(x); - } - - float TanhDerivative(float output) - { - return 1.0f - output * output; - } - - struct DenseLayer - { - std::vector> weights; - std::vector biases; - std::vector preActivation; - std::vector output; - std::vector input; - std::vector> weightGrad; - std::vector biasGrad; - - DenseLayer(std::size_t inputSize, std::size_t outputSize, std::mt19937& rng) - { - float scale = std::sqrt(2.0f / static_cast(inputSize)); - std::normal_distribution dist(0.0f, scale); - - weights.resize(outputSize, std::vector(inputSize)); - biases.resize(outputSize, 0.0f); - preActivation.resize(outputSize); - output.resize(outputSize); - weightGrad.resize(outputSize, std::vector(inputSize, 0.0f)); - biasGrad.resize(outputSize, 0.0f); - - for (auto& row : weights) - for (auto& w : row) - w = dist(rng); - } - - void Forward(const std::vector& x, bool useTanh) - { - input = x; - for (std::size_t i = 0; i < weights.size(); ++i) - { - preActivation[i] = biases[i]; - for (std::size_t j = 0; j < input.size(); ++j) - preActivation[i] += weights[i][j] * input[j]; - - output[i] = useTanh ? TanhActivation(preActivation[i]) : Sigmoid(preActivation[i]); - } - } - - std::vector Backward(const std::vector& outputGrad, bool useTanh) - { - std::vector inputGrad(input.size(), 0.0f); - - for (std::size_t i = 0; i < weights.size(); ++i) - { - float derivative = useTanh ? TanhDerivative(output[i]) : SigmoidDerivative(output[i]); - float delta = outputGrad[i] * derivative; - - for (std::size_t j = 0; j < input.size(); ++j) - { - weightGrad[i][j] += delta * input[j]; - inputGrad[j] += delta * weights[i][j]; - } - biasGrad[i] += delta; - } - return inputGrad; - } - - void ApplyGradients(float lr, std::size_t batchSize) - { - float scale = lr / static_cast(batchSize); - for (std::size_t i = 0; i < weights.size(); ++i) - { - for (std::size_t j = 0; j < weights[i].size(); ++j) - { - weights[i][j] -= scale * weightGrad[i][j]; - weightGrad[i][j] = 0.0f; - } - biases[i] -= scale * biasGrad[i]; - biasGrad[i] = 0.0f; - } - } - }; - - NnResult TrainNetwork(const NnConfig& config, const TrainingData& data) - { - std::mt19937 rng(42); - - std::size_t inputSize = data.inputs[0].size(); - std::size_t hiddenSize = config.hiddenSize; - - DenseLayer hidden(inputSize, hiddenSize, rng); - DenseLayer output(hiddenSize, 1, rng); - - NnResult result; - result.epochIndex.resize(config.epochs); - result.lossHistory.resize(config.epochs); - - for (std::size_t epoch = 0; epoch < config.epochs; ++epoch) - { - float epochLoss = 0.0f; - - for (std::size_t s = 0; s < data.inputs.size(); ++s) - { - hidden.Forward(data.inputs[s], true); - output.Forward(hidden.output, false); - - float error = output.output[0] - data.targets[s]; - epochLoss += error * error; - - std::vector outputGrad = { error }; - auto hiddenGrad = output.Backward(outputGrad, false); - hidden.Backward(hiddenGrad, true); - } - - hidden.ApplyGradients(config.learningRate, data.inputs.size()); - output.ApplyGradients(config.learningRate, data.inputs.size()); - - result.epochIndex[epoch] = static_cast(epoch); - result.lossHistory[epoch] = epochLoss / static_cast(data.inputs.size()); - } - - result.predictionInputs.resize(data.inputs.size()); - result.predictionTargets = data.targets; - result.predictionOutputs.resize(data.inputs.size()); - - for (std::size_t s = 0; s < data.inputs.size(); ++s) - { - result.predictionInputs[s] = data.inputs[s][0]; - hidden.Forward(data.inputs[s], true); - output.Forward(hidden.output, false); - result.predictionOutputs[s] = output.output[0]; - } - - return result; - } - } - - void NnSimulator::Configure(const Configuration& config) - { - configuration = config; - } - - NnResult NnSimulator::Run() - { - TrainingData data; - - switch (configuration.nn.demo) - { - case DemoType::Xor: - data = GenerateXorData(); - break; - case DemoType::SineApproximation: - data = GenerateSineData(configuration.nn.sineSamples); - break; - } - - return TrainNetwork(configuration.nn, data); - } -} diff --git a/simulator/neural_network/NeuralNetwork/application/NnSimulator.hpp b/simulator/neural_network/NeuralNetwork/application/NnSimulator.hpp deleted file mode 100644 index a35f0d07..00000000 --- a/simulator/neural_network/NeuralNetwork/application/NnSimulator.hpp +++ /dev/null @@ -1,46 +0,0 @@ -#pragma once - -#include -#include - -namespace simulator::neural_network::nn -{ - enum class DemoType - { - Xor, - SineApproximation - }; - - struct NnConfig - { - DemoType demo = DemoType::Xor; - std::size_t hiddenSize = 8; - float learningRate = 0.1f; - std::size_t epochs = 500; - std::size_t sineSamples = 50; - }; - - struct NnResult - { - std::vector epochIndex; - std::vector lossHistory; - std::vector predictionInputs; - std::vector predictionTargets; - std::vector predictionOutputs; - }; - - class NnSimulator - { - public: - struct Configuration - { - NnConfig nn; - }; - - void Configure(const Configuration& config); - NnResult Run(); - - private: - Configuration configuration; - }; -} diff --git a/simulator/neural_network/NeuralNetwork/view/CMakeLists.txt b/simulator/neural_network/NeuralNetwork/view/CMakeLists.txt deleted file mode 100644 index edb03739..00000000 --- a/simulator/neural_network/NeuralNetwork/view/CMakeLists.txt +++ /dev/null @@ -1,26 +0,0 @@ -add_library(numerical.simulator.neural_network.nn.view STATIC) - -set_target_properties(numerical.simulator.neural_network.nn.view PROPERTIES AUTOMOC ON) - -target_include_directories(numerical.simulator.neural_network.nn.view PUBLIC - "$" -) - -target_link_libraries(numerical.simulator.neural_network.nn.view PUBLIC - numerical.simulator.neural_network.nn.application - numerical.simulator.widgets - Qt6::Widgets -) - -target_sources(numerical.simulator.neural_network.nn.view PRIVATE - NnConfigurationPanel.hpp - NnConfigurationPanel.cpp - NnMainWindow.hpp - NnMainWindow.cpp -) - -if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU" OR CMAKE_CXX_COMPILER_ID STREQUAL "Clang") - target_compile_options(numerical.simulator.neural_network.nn.view PRIVATE -Wno-error) -elseif(CMAKE_CXX_COMPILER_ID STREQUAL "MSVC") - target_compile_options(numerical.simulator.neural_network.nn.view PRIVATE /W4 /wd4244 /wd4100) -endif() diff --git a/simulator/neural_network/NeuralNetwork/view/NnConfigurationPanel.cpp b/simulator/neural_network/NeuralNetwork/view/NnConfigurationPanel.cpp deleted file mode 100644 index 87d0aa95..00000000 --- a/simulator/neural_network/NeuralNetwork/view/NnConfigurationPanel.cpp +++ /dev/null @@ -1,77 +0,0 @@ -#include "simulator/neural_network/NeuralNetwork/view/NnConfigurationPanel.hpp" -#include -#include -#include -#include - -namespace simulator::neural_network::nn::view -{ - NnConfigurationPanel::NnConfigurationPanel(QWidget* parent) - : QWidget(parent) - { - auto* layout = new QVBoxLayout(this); - - auto* demoGroup = new QGroupBox("Demo Selection", this); - auto* demoLayout = new QVBoxLayout(demoGroup); - - demoLayout->addWidget(new QLabel("Problem:")); - demoCombo = new QComboBox(demoGroup); - demoCombo->addItem("XOR Classification", static_cast(DemoType::Xor)); - demoCombo->addItem("Sine Approximation", static_cast(DemoType::SineApproximation)); - demoLayout->addWidget(demoCombo); - - layout->addWidget(demoGroup); - - auto* networkGroup = new QGroupBox("Network Parameters", this); - auto* networkLayout = new QVBoxLayout(networkGroup); - - networkLayout->addWidget(new QLabel("Hidden Layer Size:")); - hiddenSizeSpinBox = new QSpinBox(networkGroup); - hiddenSizeSpinBox->setRange(2, 64); - hiddenSizeSpinBox->setSingleStep(2); - hiddenSizeSpinBox->setValue(8); - networkLayout->addWidget(hiddenSizeSpinBox); - - networkLayout->addWidget(new QLabel("Learning Rate:")); - learningRateSpinBox = new QDoubleSpinBox(networkGroup); - learningRateSpinBox->setRange(0.001, 10.0); - learningRateSpinBox->setDecimals(3); - learningRateSpinBox->setSingleStep(0.01); - learningRateSpinBox->setValue(0.5); - networkLayout->addWidget(learningRateSpinBox); - - networkLayout->addWidget(new QLabel("Epochs:")); - epochsSpinBox = new QSpinBox(networkGroup); - epochsSpinBox->setRange(10, 10000); - epochsSpinBox->setSingleStep(100); - epochsSpinBox->setValue(1000); - networkLayout->addWidget(epochsSpinBox); - - networkLayout->addWidget(new QLabel("Sine Samples:")); - sineSamplesSpinBox = new QSpinBox(networkGroup); - sineSamplesSpinBox->setRange(10, 500); - sineSamplesSpinBox->setSingleStep(10); - sineSamplesSpinBox->setValue(50); - networkLayout->addWidget(sineSamplesSpinBox); - - layout->addWidget(networkGroup); - - auto* computeButton = new QPushButton("Train", this); - layout->addWidget(computeButton); - - layout->addStretch(); - - connect(computeButton, &QPushButton::clicked, this, &NnConfigurationPanel::ComputeRequested); - } - - NnSimulator::Configuration NnConfigurationPanel::GetConfiguration() const - { - NnSimulator::Configuration config; - config.nn.demo = static_cast(demoCombo->currentData().toInt()); - config.nn.hiddenSize = static_cast(hiddenSizeSpinBox->value()); - config.nn.learningRate = static_cast(learningRateSpinBox->value()); - config.nn.epochs = static_cast(epochsSpinBox->value()); - config.nn.sineSamples = static_cast(sineSamplesSpinBox->value()); - return config; - } -} diff --git a/simulator/neural_network/NeuralNetwork/view/NnConfigurationPanel.hpp b/simulator/neural_network/NeuralNetwork/view/NnConfigurationPanel.hpp deleted file mode 100644 index 9e3e3ac2..00000000 --- a/simulator/neural_network/NeuralNetwork/view/NnConfigurationPanel.hpp +++ /dev/null @@ -1,31 +0,0 @@ -#pragma once - -#include "simulator/neural_network/NeuralNetwork/application/NnSimulator.hpp" -#include -#include -#include -#include - -namespace simulator::neural_network::nn::view -{ - class NnConfigurationPanel - : public QWidget - { - Q_OBJECT - - public: - explicit NnConfigurationPanel(QWidget* parent = nullptr); - - NnSimulator::Configuration GetConfiguration() const; - - signals: - void ComputeRequested(); - - private: - QComboBox* demoCombo; - QSpinBox* hiddenSizeSpinBox; - QDoubleSpinBox* learningRateSpinBox; - QSpinBox* epochsSpinBox; - QSpinBox* sineSamplesSpinBox; - }; -} diff --git a/simulator/neural_network/NeuralNetwork/view/NnMainWindow.cpp b/simulator/neural_network/NeuralNetwork/view/NnMainWindow.cpp deleted file mode 100644 index b67b7faf..00000000 --- a/simulator/neural_network/NeuralNetwork/view/NnMainWindow.cpp +++ /dev/null @@ -1,78 +0,0 @@ -#include "simulator/neural_network/NeuralNetwork/view/NnMainWindow.hpp" -#include "simulator/neural_network/NeuralNetwork/view/NnConfigurationPanel.hpp" -#include "simulator/widgets/TimeSeriesChartWidget.hpp" -#include -#include -#include - -namespace simulator::neural_network::nn::view -{ - NnMainWindow::NnMainWindow(QWidget* parent) - : QMainWindow(parent) - { - setWindowTitle("Neural Network Simulator"); - resize(1200, 700); - - auto* splitter = new QSplitter(Qt::Horizontal, this); - - configPanel = new NnConfigurationPanel(splitter); - configPanel->setMaximumWidth(350); - - tabWidget = new QTabWidget(splitter); - - lossChart = new widgets::TimeSeriesChartWidget(tabWidget); - predictionChart = new widgets::TimeSeriesChartWidget(tabWidget); - - tabWidget->addTab(lossChart, "Training Loss"); - tabWidget->addTab(predictionChart, "Predictions"); - - splitter->addWidget(configPanel); - splitter->addWidget(tabWidget); - splitter->setStretchFactor(0, 0); - splitter->setStretchFactor(1, 1); - - setCentralWidget(splitter); - statusBar()->showMessage("Configure network parameters and press Train"); - - connect(configPanel, &NnConfigurationPanel::ComputeRequested, this, &NnMainWindow::OnComputeRequested); - } - - void NnMainWindow::OnComputeRequested() - { - auto config = configPanel->GetConfiguration(); - - NnSimulator simulator; - simulator.Configure(config); - auto result = simulator.Run(); - - lossChart->SetTimeAxis(result.epochIndex); - lossChart->SetPanels({ - { - "Training Loss (MSE)", - "Loss", - { - { "MSE", QColor(231, 76, 60), result.lossHistory }, - }, - 1, - }, - }); - - predictionChart->SetTimeAxis(result.predictionInputs); - predictionChart->SetPanels({ - { - "Target vs Prediction", - "Value", - { - { "Target", QColor(41, 128, 185), result.predictionTargets }, - { "Prediction", QColor(231, 76, 60), result.predictionOutputs }, - }, - 1, - }, - }); - - auto finalLoss = result.lossHistory.empty() ? 0.0f : result.lossHistory.back(); - statusBar()->showMessage(QString("Training complete: %1 epochs, final loss = %2") - .arg(config.nn.epochs) - .arg(static_cast(finalLoss), 0, 'g', 6)); - } -} diff --git a/simulator/neural_network/NeuralNetwork/view/NnMainWindow.hpp b/simulator/neural_network/NeuralNetwork/view/NnMainWindow.hpp deleted file mode 100644 index f7a863f1..00000000 --- a/simulator/neural_network/NeuralNetwork/view/NnMainWindow.hpp +++ /dev/null @@ -1,35 +0,0 @@ -#pragma once - -#include -#include - -namespace simulator::neural_network::nn::view -{ - class NnConfigurationPanel; -} - -namespace simulator::widgets -{ - class TimeSeriesChartWidget; -} - -namespace simulator::neural_network::nn::view -{ - class NnMainWindow - : public QMainWindow - { - Q_OBJECT - - public: - explicit NnMainWindow(QWidget* parent = nullptr); - - private slots: - void OnComputeRequested(); - - private: - NnConfigurationPanel* configPanel; - QTabWidget* tabWidget; - widgets::TimeSeriesChartWidget* lossChart; - widgets::TimeSeriesChartWidget* predictionChart; - }; -}