Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,27 @@ updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
day: "monday"
interval: "monthly"
commit-message:
prefix: "chore(deps):"

- package-ecosystem: "pip"
directory: "/services/api"
schedule:
interval: "weekly"
interval: "monthly"
commit-message:
prefix: "chore(deps):"

- package-ecosystem: "pip"
directory: "/services/worker"
schedule:
interval: "weekly"
interval: "monthly"
commit-message:
prefix: "chore(deps):"

- package-ecosystem: "pip"
directory: "/services/jupyter"
schedule:
interval: "monthly"
commit-message:
prefix: "chore(deps):"
23 changes: 23 additions & 0 deletions .github/dependabot.yml.disabled
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
version: 2
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
day: "monday"
commit-message:
prefix: "chore(deps):"

- package-ecosystem: "pip"
directory: "/services/api"
schedule:
interval: "weekly"
commit-message:
prefix: "chore(deps):"

- package-ecosystem: "pip"
directory: "/services/worker"
schedule:
interval: "weekly"
commit-message:
prefix: "chore(deps):"
12 changes: 6 additions & 6 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
run: pip install ruff

- name: Run Lint
run: ruff check services/api/ services/worker/ --exclude __pycache__,*.apagar,*.pyc || true
run: ruff check services/api/ services/worker/ --exclude __pycache__,*.apagar,*.pyc


typecheck:
Expand All @@ -52,7 +52,7 @@ jobs:
- name: Run Mypy
run: |
# Focamos apenas na lógica da API e Worker da plataforma
mypy --ignore-missing-imports services/api/main.py services/worker/worker.py || true
PYTHONPATH=$PWD/services mypy --ignore-missing-imports services/api/main.py services/worker/worker.py


security:
Expand All @@ -70,7 +70,7 @@ jobs:
run: pip install bandit

- name: Run Bandit
run: bandit -r services/api/ services/worker/ -ll -ii -f txt -o bandit-report.txt || true
run: bandit -r services/api/ services/worker/ -ll -ii -f txt -o bandit-report.txt

- name: Upload Bandit Report
uses: actions/upload-artifact@v4
Expand Down Expand Up @@ -193,11 +193,11 @@ jobs:
cache-from: type=gha
cache-to: type=gha,mode=max

- name: Build Frontend
- name: Build Jupyter
uses: docker/build-push-action@v5
with:
context: ./services/frontend
file: ./services/frontend/Dockerfile
context: ./services/jupyter
file: ./services/jupyter/Dockerfile
push: false
cache-from: type=gha
cache-to: type=gha,mode=max
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# Python
site
configs
__pycache__/
*.py[cod]
Expand Down
45 changes: 45 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
# Changelog

All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]

## [0.1.0] - 2026-06-13

First public release of the DisSModel Platform MVP.

### Added

- **REST API** (`services/api`) — FastAPI gateway with `X-API-Key` authentication applied to all routes; endpoints for job submission (async and inline), status polling, reproduction, publishing, model listing, file upload/download, and admin sync.
- **Worker** (`services/worker`) — Redis-queue consumer that delegates execution to `dissmodel.executor.runner.execute_lifecycle`; publishes profiling metrics and `ExperimentRecord` JSON to MinIO.
- **JupyterLab** (`services/jupyter`) — Containerised notebook environment (port 8888) with `dissmodel`, `ipyleaflet`, `ipywidgets`, and `folium` pre-installed.
- **Streamlit CA Explorer** (`services/streamlit-ca`) — Interactive explorer for Cellular Automata models.
- **Streamlit SysDyn Explorer** (`services/streamlit-sysdyn`) — Interactive explorer for System Dynamics models.
- **Nginx reverse proxy** (`services/nginx`) — Routes `/dissmodel/jupyter`, `/dissmodel/api`, `/dissmodel/minio` in production.
- **Docker Compose** — Development (`docker-compose.yml`) and production (`docker-compose.prod.yml`) stacks with Redis, MinIO, config-sync sidecar, and all services.
- **CI pipeline** (`.github/workflows/ci.yml`) — Lint (ruff), type-check (mypy), security scan (bandit), API tests, worker executor validation, and Docker build jobs; all gates are enforced (no `|| true` bypasses).
- **Dependabot** — Automated dependency updates for `services/api`, `services/worker`, and `services/jupyter`.
- **Presigned URL generation** — Local HMAC signing for MinIO download links without extra network round-trips.
- **Config-sync sidecar** — Git-backed model registry auto-pulled into all services at runtime.
- **Executor contract validation** (`scripts/validate_executors.py`) — CI step that checks registered executors comply with the dissmodel interface before tests run.

### Fixed

- Moved mid-file imports (`hmac`, `hashlib`, `urllib.parse`, `datetime.timezone`) to module top in `services/api/main.py`.
- Removed unused imports (`json`, `timedelta`, `S3Error`, `start_sync_scheduler`, `reproduce_experiment`, `run_experiment`) from `services/api/main.py` and `services/worker/storage.py`.
- Added `# type: ignore[misc]` for redis-py sync/async stub ambiguity in `services/worker/worker.py`.
- Added `# nosec B104` and `# nosec B310` for intentional false positives in bandit scan.

### Changed

- Renamed `services/frontend/` → `services/jupyter/` to reflect that the service is JupyterLab, not a generic web frontend.
- Updated all repository URLs from `LambdaGeo/dissmodel-platform` → `DisSModel/dissmodel-platform` in `README.md` and `docs/deployment.md`.
- Updated core library link from `LambdaGeo/dissmodel` → `DisSModel/dissmodel`.
- Updated organisation name from `LambdaGeo / INPE` → `DisSModel / INPE` in `README.md` contact section.
- `typecheck` CI job now sets `PYTHONPATH=$PWD/services` so worker imports resolve correctly without stubs.

[Unreleased]: https://github.com/DisSModel/dissmodel-platform/compare/v0.1.0...HEAD
[0.1.0]: https://github.com/DisSModel/dissmodel-platform/releases/tag/v0.1.0
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ An integrated environment for developing and running geospatial models, featurin

```bash
# 1. Clone the repository
git clone https://github.com/LambdaGeo/dissmodel-platform.git
git clone https://github.com/DisSModel/dissmodel-platform.git
cd dissmodel-platform

# 2. Configure environment variables
Expand Down Expand Up @@ -175,13 +175,13 @@ MIT License — see [LICENSE](LICENSE)

## 🙏 Acknowledgements

- [DisSModel](https://github.com/LambdaGeo/dissmodel) — Core modelling library
- [DisSModel](https://github.com/DisSModel/dissmodel) — Core modelling library
- [Jupyter Project](https://jupyter.org/) — Development environment
- [MinIO](https://min.io/) — S3-compatible object storage
- [Pangeo](https://pangeo.io/) — Inspiration for cloud-native architecture

## 📞 Contact

- **Organisation:** LambdaGeo / INPE
- **Issues:** https://github.com/LambdaGeo/dissmodel-platform/issues
- **Discussions:** https://github.com/LambdaGeo/dissmodel-platform/discussions
- **Organisation:** DisSModel / INPE
- **Issues:** https://github.com/DisSModel/dissmodel-platform/issues
- **Discussions:** https://github.com/DisSModel/dissmodel-platform/discussions
2 changes: 1 addition & 1 deletion docker-compose.prod.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ services:
#sudo chmod -R 775 ./workspace
jupyter:
build:
context: ./services/frontend
context: ./services/jupyter
dockerfile: Dockerfile
container_name: dissmodel-jupyter
restart: unless-stopped
Expand Down
2 changes: 1 addition & 1 deletion docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ services:

jupyter:
build:
context: ./services/frontend
context: ./services/jupyter
dockerfile: Dockerfile
container_name: dissmodel-jupyter
restart: unless-stopped
Expand Down
2 changes: 1 addition & 1 deletion docs/deployment.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

```bash
# Clonar
git clone https://github.com/LambdaGeo/dissmodel-platform.git
git clone https://github.com/DisSModel/dissmodel-platform.git
cd dissmodel-platform

# Configurar
Expand Down
23 changes: 23 additions & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# DisSModel Platform Documentation

Welcome to the official documentation for the **DisSModel Platform**.

The DisSModel Platform is a robust service designed for the execution, tracking, and reproduction of Land Use and Cover Change (LUCC) simulations. It provides a standardized environment for scientific modeling, ensuring that experiments are audit-ready and fully reproducible.

## Key Features

* **Immutable Provenance:** Every simulation is recorded with a complete snapshot of its configuration and data integrity.
* **Model Registry:** Decouples model parameters from implementation via version-controlled TOML files.
* **Scalable Execution:** Leverages Dask and subprocess isolation for high-performance simulation runs.
* **Researcher-Centric:** Designed to facilitate the transition from local Jupyter exploration to large-scale production.

## Getting Started

To understand how the platform works, we recommend starting with the following sections:

1. [**Platform Overview**](platform/index.md) - Learn about the architecture and core components.
2. [**Researcher Guide**](platform/researcher-guide.md) - A step-by-step guide to developing and running models.
3. [**API Reference**](platform/api-reference.md) - Complete documentation of the available REST endpoints.

---
*This documentation was automatically generated from the source code.*
127 changes: 127 additions & 0 deletions docs/platform/api-reference.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# REST API Reference

The DisSModel Platform API is built with FastAPI. All endpoints (except `/health` and `/`) require the `X-API-Key` header for authentication.

**Base URL:** `http://localhost:8000`

## Job Submission

### `POST /submit_job`
Submit a simulation job using a model registered in the [Registry](registry.md).

**Body Schema:**
| Field | Type | Required | Description |
|---|---|---|---|
| `model_name` | `str` | Yes | Name of the registered model. |
| `input_dataset`| `str` | Yes | URI (`s3://`, `http://`, or local path). |
| `input_format` | `str` | Yes | `geotiff`, `vector`, or `zarr`. |
| `column_map` | `dict` | No | Mapping for vector columns. |
| `band_map` | `dict` | No | Mapping for raster bands. |
| `parameters` | `dict` | No | Overrides for TOML defaults. |
| `priority` | `str` | No | `high`, `normal` (default), or `low`. |

**Example:**
```bash
curl -X POST http://localhost:8000/submit_job \
-H "X-API-Key: dev-key" \
-H "Content-Type: application/json" \
-d '{
"model_name": "coastal-v1",
"input_dataset": "s3://dissmodel-inputs/data.tif",
"input_format": "geotiff",
"parameters": {"threshold": 0.8}
}'
```

---

### `POST /submit_job_inline`
Submit a job with an inline TOML spec. Used for development and exploration.

**Body Schema:**
| Field | Type | Required | Description |
|---|---|---|---|
| `model_spec_toml` | `str` | Yes | Full content of the TOML spec. |
| `input_dataset` | `str` | Yes | URI of the dataset. |
| `input_format` | `str` | Yes | `geotiff`, `vector`, or `zarr`. |
| `column_map` | `dict` | No | Mapping for vector columns. |
| `band_map` | `dict` | No | Mapping for raster bands. |
| `parameters` | `dict` | No | Overrides. |

---

## Status & Tracking

### `GET /job/{experiment_id}`
Return the current status and provenance record for a specific experiment.

**Response:** `JobResponse` object (see [ExperimentRecord](experiment-record.md)).

---

### `GET /jobs`
List simulations, optionally filtered by status.

**Query Parameters:**
| Parameter | Type | Default | Description |
|---|---|---|---|
| `limit` | `int` | `100` | Max number of records to return. |
| `status` | `str` | `null` | Filter by status (`completed`, `failed`, etc.). |

---

## Reproduce & Publish

### `POST /experiments/{experiment_id}/reproduce`
Re-run an experiment using its original `resolved_spec` snapshot, independent of the current registry state.

---

### `POST /experiments/{experiment_id}/publish`
Export a reproducibility package. Returns the full provenance as JSON.

---

## Model Registry

### `GET /models`
List all registered models and their basic metadata.

---

### `GET /models/{model_name}`
Return the full TOML spec for a specific model.

---

### `POST /admin/sync`
Force an immediate `git pull` of the `dissmodel-configs` repository.

---

## Data Management

### `POST /data/upload`
Upload a dataset to the `dissmodel-inputs` bucket.

**Body (Multipart Form):**
* `file`: The file to upload.
* `label`: A tag for the input directory (e.g., `baseline`).

**Response:**
```json
{
"uri": "s3://dissmodel-inputs/inputs/baseline/data.tif",
"checksum": "sha256...",
"size_mb": 45.2
}
```

---

### `GET /download`
Generate a presigned URL for downloading a file from S3.

**Query Parameters:**
* `uri`: The `s3://` URI.
* `expires_hours`: Validity period (default: `1`).
Loading
Loading