Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"hash": "275de27a383338eb229b788a42e8ac50",
"result": {
"engine": "jupyter",
"markdown": "---\ntitle: manager.demo_metrics.calculate_metrics\n---\n\n\n\n```python\nmanager.demo_metrics.calculate_metrics(actual, predicted)\n```\n\nCalculate MAE and MSE for numeric evaluation.\n\nComputes Mean Absolute Error (MAE) and Mean Squared Error (MSE) between\nactual and predicted values. These metrics are essential for evaluating\nforecasting model performance in safety-critical applications.\n\n## Parameters {.doc-section .doc-section-parameters}\n\n| Name | Type | Description | Default |\n|-----------|------------------------------------------|---------------------------------------------------------------|------------|\n| actual | [pd](`pandas`).[Series](`pandas.Series`) | Series of actual observed values. | _required_ |\n| predicted | [pd](`pandas`).[Series](`pandas.Series`) | Series of predicted values (must have same length as actual). | _required_ |\n\n## Returns {.doc-section .doc-section-returns}\n\n| Name | Type | Description |\n|--------|---------------------------------------------------------|---------------------------------------------------------------------------------------------------|\n| | [Dict](`typing.Dict`)\\[[str](`str`), [float](`float`)\\] | Dict[str, float]: Dictionary containing: - 'MAE': Mean Absolute Error - 'MSE': Mean Squared Error |\n\n## Raises {.doc-section .doc-section-raises}\n\n| Name | Type | Description |\n|--------|----------------------------|---------------------------------------------------------|\n| | [ValueError](`ValueError`) | If series have different lengths or contain NaN values. |\n\n## Examples {.doc-section .doc-section-examples}\n\n\n::: {#51aa8de6 .cell execution_count=1}\n``` {.python .cell-code}\nimport pandas as pd\nfrom spotforecast2_safe.manager.demo_metrics import calculate_metrics\n\n# Perfect predictions: both MAE and MSE should be zero\nactual = pd.Series([1.0, 2.0, 3.0, 4.0, 5.0])\npredicted = pd.Series([1.0, 2.0, 3.0, 4.0, 5.0])\nmetrics = calculate_metrics(actual, predicted)\nprint(f\"MAE: {metrics['MAE']:.4f}\")\nprint(f\"MSE: {metrics['MSE']:.4f}\")\nassert metrics[\"MAE\"] == 0.0\nassert metrics[\"MSE\"] == 0.0\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nMAE: 0.0000\nMSE: 0.0000\n```\n:::\n:::\n\n\n::: {#9a1b7136 .cell execution_count=2}\n``` {.python .cell-code}\nimport pandas as pd\nfrom spotforecast2_safe.manager.demo_metrics import calculate_metrics\n\n# Small symmetric errors: MAE == MSE == 1.0\nactual = pd.Series([10.0, 20.0, 30.0, 40.0])\npredicted = pd.Series([11.0, 19.0, 31.0, 39.0])\nmetrics = calculate_metrics(actual, predicted)\nprint(f\"MAE: {metrics['MAE']:.4f}\")\nprint(f\"MSE: {metrics['MSE']:.4f}\")\nassert metrics[\"MAE\"] == 1.0\nassert metrics[\"MSE\"] == 1.0\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nMAE: 1.0000\nMSE: 1.0000\n```\n:::\n:::\n\n\n::: {#8439531d .cell execution_count=3}\n``` {.python .cell-code}\nimport pandas as pd\nfrom spotforecast2_safe.manager.demo_metrics import calculate_metrics\n\n# Larger asymmetric errors\nactual = pd.Series([100.0, 200.0, 300.0])\npredicted = pd.Series([95.0, 210.0, 290.0])\nmetrics = calculate_metrics(actual, predicted)\nprint(f\"MAE: {metrics['MAE']:.4f}\")\nprint(f\"MSE: {metrics['MSE']:.4f}\")\nassert abs(metrics[\"MAE\"] - 25 / 3) < 1e-9\nassert abs(metrics[\"MSE\"] - 75.0) < 1e-9\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nMAE: 8.3333\nMSE: 75.0000\n```\n:::\n:::\n\n\n::: {#4271bd79 .cell execution_count=4}\n``` {.python .cell-code}\nimport pandas as pd\nfrom spotforecast2_safe.manager.demo_metrics import calculate_metrics\n\n# Safety-critical: validate metrics stay within acceptable bounds\nactual = pd.Series([50.0, 55.0, 60.0, 65.0, 70.0])\npredicted = pd.Series([51.0, 54.0, 61.0, 64.0, 71.0])\nmetrics = calculate_metrics(actual, predicted)\nassert metrics[\"MAE\"] < 2.0, \"MAE exceeds safety threshold\"\nassert metrics[\"MSE\"] < 5.0, \"MSE exceeds safety threshold\"\nprint(\"Safety validation passed\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nSafety validation passed\n```\n:::\n:::\n\n\n::: {#cd27106b .cell execution_count=5}\n``` {.python .cell-code}\nimport pandas as pd\nfrom spotforecast2_safe.manager.demo_metrics import calculate_metrics\n\n# Time series with a datetime index\ndates = pd.date_range(\"2024-01-01\", periods=5, freq=\"D\")\nactual = pd.Series([10.5, 11.2, 10.8, 11.5, 12.0], index=dates)\npredicted = pd.Series([10.3, 11.4, 10.9, 11.3, 12.1], index=dates)\nmetrics = calculate_metrics(actual, predicted)\nprint(f\"MAE: {metrics['MAE']:.4f}\")\nprint(f\"MSE: {metrics['MSE']:.4f}\")\nassert abs(metrics[\"MAE\"] - 0.16) < 1e-9\nassert abs(metrics[\"MSE\"] - 0.028) < 1e-9\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nMAE: 0.1600\nMSE: 0.0280\n```\n:::\n:::\n\n\n::: {#b5316029 .cell execution_count=6}\n``` {.python .cell-code}\nimport pandas as pd\nfrom spotforecast2_safe.manager.demo_metrics import calculate_metrics\n\n# Compare two models: lower MAE wins\nactual = pd.Series([1.0, 2.0, 3.0, 4.0, 5.0])\npred_model_a = pd.Series([1.1, 2.1, 2.9, 4.2, 4.8])\npred_model_b = pd.Series([1.5, 2.5, 3.5, 4.5, 5.5])\nmetrics_a = calculate_metrics(actual, pred_model_a)\nmetrics_b = calculate_metrics(actual, pred_model_b)\nwinner = \"A\" if metrics_a[\"MAE\"] < metrics_b[\"MAE\"] else \"B\"\nprint(f\"Model A MAE: {metrics_a['MAE']:.4f}\")\nprint(f\"Model B MAE: {metrics_b['MAE']:.4f}\")\nprint(f\"Model {winner} has better MAE\")\nassert winner == \"A\"\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nModel A MAE: 0.1400\nModel B MAE: 0.5000\nModel A has better MAE\n```\n:::\n:::\n\n\n",
"supporting": [
"manager.demo_metrics.calculate_metrics_files"
],
"filters": [],
"includes": {}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"hash": "58c242eba1427e390e9e505d5a3555db",
"result": {
"engine": "jupyter",
"markdown": "---\ntitle: manager.predictor.get_model_prediction\n---\n\n\n\n```python\nmanager.predictor.get_model_prediction(\n model_name,\n model_dir=None,\n predict_size=None,\n)\n```\n\nGet the prediction package from the latest trained model.\n\nThis function retrieves the latest iteration of a specified model from the\ncache and calls its `package_prediction` method to obtain a comprehensive\nset of predictions and metrics.\n\n## Parameters {.doc-section .doc-section-parameters}\n\n| Name | Type | Description | Default |\n|--------------|--------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|------------|\n| model_name | [str](`str`) | Name of the model to use (e.g., 'lgbm', 'xgb'). | _required_ |\n| model_dir | [Optional](`typing.Optional`)\\[[Union](`typing.Union`)\\[[str](`str`), [Path](`pathlib.Path`)\\]\\] | Directory where models are stored. If None, defaults to the library's cache home. | `None` |\n| predict_size | [Optional](`typing.Optional`)\\[[int](`int`)\\] | Optional override for the prediction horizon. | `None` |\n\n## Returns {.doc-section .doc-section-returns}\n\n| Name | Type | Description |\n|--------|------------------------------------------------------------|-------------------------------------------------------------|\n| | [Dict](`typing.Dict`)\\[[str](`str`), [Any](`typing.Any`)\\] | A dictionary containing predictions and metrics produced by |\n| | [Dict](`typing.Dict`)\\[[str](`str`), [Any](`typing.Any`)\\] | `package_prediction()`. |\n\n## Raises {.doc-section .doc-section-raises}\n\n| Name | Type | Description |\n|--------|----------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|\n| | [FileNotFoundError](`FileNotFoundError`) | If no trained model is found for `model_name` in `model_dir`. |\n| | [AttributeError](`AttributeError`) | If the loaded model does not implement a `package_prediction` method. |\n| | [PredictionPackageError](`PredictionPackageError`) | If `package_prediction()` itself fails — see `forecaster.wrappers.model.ForecasterRecursiveModel.package_prediction`. |\n| | [OSError](`OSError`) | If the on-disk model file exists but cannot be deserialised (corrupt joblib). |\n\n## Notes {.doc-section .doc-section-notes}\n\n`predict_size` is accepted by `get_model_prediction()` but only has effect if the concrete model's `package_prediction()` accepts it.\nThe original `ForecasterRecursiveModel.package_prediction()` does not — so this parameter is currently forward-looking API design, not yet wired end-to-end.\n\n## Examples {.doc-section .doc-section-examples}\n\n\n::: {#bffee86b .cell execution_count=1}\n``` {.python .cell-code}\nimport tempfile\nfrom spotforecast2_safe.manager.predictor import get_model_prediction\n\n# When no model has been trained, get_model_prediction raises FileNotFoundError.\nwith tempfile.TemporaryDirectory() as tmpdir:\n try:\n get_model_prediction(\"lgbm\", model_dir=tmpdir)\n except FileNotFoundError as e:\n print(type(e).__name__)\n assert \"lgbm\" in str(e)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nFileNotFoundError\n```\n:::\n:::\n\n\n",
"supporting": [
"manager.predictor.get_model_prediction_files"
],
"filters": [],
"includes": {}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"hash": "97674d67521cfda53f1804f0ce987a1c",
"result": {
"engine": "jupyter",
"markdown": "---\ntitle: manager.trainer.get_last_model\n---\n\n\n\n```python\nmanager.trainer.get_last_model(model_name, model_dir=None)\n```\n\nGet the latest trained model from the cache.\n\n## Parameters {.doc-section .doc-section-parameters}\n\n| Name | Type | Description | Default |\n|------------|--------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|------------|\n| model_name | [str](`str`) | Name of the model (e.g., 'lgbm', 'xgb'). | _required_ |\n| model_dir | [Optional](`typing.Optional`)\\[[Union](`typing.Union`)\\[[str](`str`), [Path](`pathlib.Path`)\\]\\] | Directory where models are stored. If None, defaults to the library's cache home. | `None` |\n\n## Returns {.doc-section .doc-section-returns}\n\n| Name | Type | Description |\n|--------|-------------------------------------------------------|-------------------------------------------------------------------|\n| | [int](`int`) | A tuple (iteration, model_instance). If no model is found on disk |\n| | [Any](`typing.Any`) | (cache directory missing, no matching files, or no parseable |\n| | [tuple](`tuple`)\\[[int](`int`), [Any](`typing.Any`)\\] | iteration numbers), returns (-1, None) — a legitimate |\n| | [tuple](`tuple`)\\[[int](`int`), [Any](`typing.Any`)\\] | \"fresh install, no model yet\" state. |\n\n## Raises {.doc-section .doc-section-raises}\n\n| Name | Type | Description |\n|--------|----------------------|--------------------------------------------------------------------------------------------------------------|\n| | [OSError](`OSError`) | If the latest model file exists on disk but cannot be deserialised (corrupt joblib, version mismatch, etc.). |\n\n## Examples {.doc-section .doc-section-examples}\n\n\n::: {#cf7a6f88 .cell execution_count=1}\n``` {.python .cell-code}\nimport tempfile\nfrom spotforecast2_safe.manager.trainer import get_last_model\n\n# Empty directory — no model files yet.\nwith tempfile.TemporaryDirectory() as tmpdir:\n iteration, model = get_last_model(\"lgbm\", model_dir=tmpdir)\n print(iteration, model)\n assert iteration == -1\n assert model is None\n```\n\n::: {.cell-output .cell-output-stdout}\n```\n-1 None\n```\n:::\n:::\n\n\n",
"supporting": [
"manager.trainer.get_last_model_files"
],
"filters": [],
"includes": {}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"hash": "71b72677e4f2cc9c44b60c46bb2350c2",
"result": {
"engine": "jupyter",
"markdown": "---\ntitle: manager.trainer.get_path_model\n---\n\n\n\n```python\nmanager.trainer.get_path_model(name, iteration, model_dir=None)\n```\n\nYield the path to a model file for a given iteration and model name.\n\n## Parameters {.doc-section .doc-section-parameters}\n\n| Name | Type | Description | Default |\n|-----------|--------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------|------------|\n| name | [str](`str`) | Model name (e.g. ``\"lgbm\"``, ``\"xgb\"``). | _required_ |\n| iteration | [int](`int`) | Iteration of the model. | _required_ |\n| model_dir | [Optional](`typing.Optional`)\\[[Union](`typing.Union`)\\[[str](`str`), [Path](`pathlib.Path`)\\]\\] | Directory where models are stored. If *None*, defaults to `get_cache_home()`. | `None` |\n\n## Returns {.doc-section .doc-section-returns}\n\n| Name | Type | Description |\n|--------|------------------------|--------------------------------------------------|\n| Path | [Path](`pathlib.Path`) | Full path where the model file should be stored. |\n\n## Examples {.doc-section .doc-section-examples}\n\n\n::: {#dc5148a6 .cell execution_count=1}\n``` {.python .cell-code}\nimport tempfile\nfrom pathlib import Path\nfrom spotforecast2_safe.manager.trainer import get_path_model\n\nwith tempfile.TemporaryDirectory() as tmpdir:\n p = get_path_model(\"lgbm\", 3, model_dir=tmpdir)\n print(p.name)\n assert p.name == \"lgbm_forecaster_3.joblib\"\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nlgbm_forecaster_3.joblib\n```\n:::\n:::\n\n\n",
"supporting": [
"manager.trainer.get_path_model_files"
],
"filters": [],
"includes": {}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
{
"hash": "065499f6b41024b08be56dbce57779b4",
"result": {
"engine": "jupyter",
"markdown": "---\ntitle: manager.trainer.load_iteration\n---\n\n\n\n```python\nmanager.trainer.load_iteration(name, iteration, model_dir=None)\n```\n\nLoad a saved model at a given iteration.\n\n## Parameters {.doc-section .doc-section-parameters}\n\n| Name | Type | Description | Default |\n|-----------|--------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------|------------|\n| name | [str](`str`) | Model name (e.g. ``\"lgbm\"``). | _required_ |\n| iteration | [int](`int`) | Iteration of the model. | _required_ |\n| model_dir | [Optional](`typing.Optional`)\\[[Union](`typing.Union`)\\[[str](`str`), [Path](`pathlib.Path`)\\]\\] | Directory where models are stored. If *None*, defaults to `get_cache_home()`. | `None` |\n\n## Returns {.doc-section .doc-section-returns}\n\n| Name | Type | Description |\n|--------|------------------------------------------------------|------------------------------------------------------------------|\n| | [Optional](`typing.Optional`)\\[[Any](`typing.Any`)\\] | The loaded model instance, or *None* if the file does not exist. |\n\n## Raises {.doc-section .doc-section-raises}\n\n| Name | Type | Description |\n|--------|----------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|\n| | [OSError](`OSError`) | If the model file exists on disk but cannot be deserialised (corrupt joblib, version mismatch, etc.). A missing file returns `None` instead — that's a legitimate \"no model yet\" state. |\n\n## Examples {.doc-section .doc-section-examples}\n\n\n::: {#7151a213 .cell execution_count=1}\n``` {.python .cell-code}\nimport tempfile\nfrom spotforecast2_safe.manager.trainer import load_iteration\n\nwith tempfile.TemporaryDirectory() as tmpdir:\n result = load_iteration(\"lgbm\", 99, model_dir=tmpdir)\n print(result)\n assert result is None\n```\n\n::: {.cell-output .cell-output-stderr}\n```\nIteration 99 does not exist at /var/folders/dw/pvtj6mt91znd0hftcztqb0k00000gn/T/tmp8kneavsb/lgbm_forecaster_99.joblib!\n```\n:::\n\n::: {.cell-output .cell-output-stdout}\n```\nNone\n```\n:::\n:::\n\n\n",
"supporting": [
"manager.trainer.load_iteration_files"
],
"filters": [],
"includes": {}
}
}
Loading
Loading