Feature/sw architecture

.gitignore

@@ -43,3 +43,7 @@ share/python-wheels/
 
 #envrc
 .envrc
+
+ingest_service/common/*
+rest_api/common/*
+processor_service/common/*

.pre-commit-config.yaml

@@ -21,8 +21,6 @@ repos:
         stages: [pre-commit]
       - id: detect-private-key
         stages: [pre-commit]
-      - id: check-added-large-files
-        stages: [pre-commit]
     # clang-format uses .clang-format to apply formatting
   - repo: https://github.com/pre-commit/mirrors-clang-format
     rev: v18.1.7

CHANGELOG.md

@@ -5,3 +5,11 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
 ## [Unreleased]
+
+### [Added]
+
+- Ingest Service
+- Processor Service
+- REST API
+- Mock Device into Ingest Service
+- Docker compose system architecture

Makefile

@@ -0,0 +1,15 @@
+# these will speed up builds, for docker-compose >= 1.25
+export COMPOSE_DOCKER_CLI_BUILD=1
+export DOCKER_BUILDKIT=1
+
+dev-up:
+	docker compose -f docker-compose-dev.yml -p ajdovscina-dev up --build
+
+dev-down:
+	docker compose -f docker-compose-dev.yml -p ajdovscina-dev down --remove-orphans
+
+prod-up:
+	docker compose -f docker-compose-prod.yml -p ajdovscina-prod up --build
+
+prod-down:
+	docker compose -f docker-compose-prod.yml -p ajdovscina-prod down --remove-orphans

README.md

@@ -1,63 +1,112 @@
-# Irnas's Projects template
+# Ajdovscina Cloud Software
 
-IRNAS template for a GitHub repository. It comes with a
-[basic group](https://github.com/IRNAS/irnas-workflows-software/tree/main/workflow-templates/basic)
-of CI workflows for release automation.
+This repository contains code related to the `ajdovscina` project, which is a system for sensor data
+ingestion from various endpoints and a REST API to expose this data.
 
-## Checklist
+## Architecture
 
-### General GitHub setup
+The system is designed with a microservices architecture, where each component is responsible for a
+specific functionality. The main components include:
 
-- [ ] Provide a concise and accurate description of your project in the GitHub "description" field.
-- [ ] Provide a concise and accurate description of your project in this `README.md` file, replace
-      the default title.
-- [ ] Ensure that your project follows [repository naming scheme].
-- [ ] Add a project logo to the `README.md` file. The logo should be placed in the `doc/images`
-      folder. Use the following syntax to include it:
+- **Data Ingestion Service**: Responsible for collecting data from various sensor endpoints.
+- **Data Processing Service**: Processes the ingested data, applying any necessary transformations
+  or aggregations.
+- **API Gateway**: Exposes a REST API for clients to access the processed data.
 
-  ```markdown
-  ![Project Logo](doc/images/logo.svg)
-  ```
+## Folder Structure
 
-[repository naming scheme]:
-  https://github.com/IRNAS/irnas-guidelines-docs/blob/main/docs/github_projects_guidelines.md#repository-naming-scheme-
+```text
+└── 📁ajdovscina-cloud-software
+    └── 📁alertmanager
+    └── 📁common
+        └── 📁logging
+    └── 📁dockerfiles
+    └── 📁docs
+        └── 📁architecture
+            ├── ajdovscina-architecture.drawio
+        └── 📁images
+    └── 📁grafana
+        └── 📁dashboards
+    └── 📁ingest_service
+        └── 📁models
+        └── 📁util
+        ├── Makefile
+    └── 📁init
+        ├── postgres.sql
+    └── 📁mock
+        └── 📁ingest_service
+    └── 📁processor_service
+        └── 📁src
+        └── 📁util
+        ├── Makefile
+    └── 📁prometheus
+    └── 📁rest_api
+        └── 📁models
+        └── 📁src
+        └── 📁util
+        ├── Makefile
+    ├── .gitignore
+    ├── .prettierrc
+    ├── committed.toml
+    ├── Makefile
+    ├── ruff.toml
+    └── typos.toml
+```
 
-### Tooling
+## Deployment
 
-- [ ] Turn on `pre-commit` tool by running `pre-commit install`. If you do not have it yet, follow
-      instructions
-      [here](https://github.com/IRNAS/irnas-guidelines-docs/tree/main/tools/pre-commit). **Note**:
-      This tool is optional for mechanical and electronic projects.
+Deployment is done via `docker`. The system can be run in development or production mode.
 
-### Non-software projects
+To start the system in development mode, use the following command:
 
-- [ ] If needed re-read [GitHub for non-software projects] section. It describes what files should
-      be committed to the repository and what files should be added to the release assets.
-- [ ] Create `assets` folder with files that will be part of the next release. The files in this
-      folder will be automatically compressed into a zip file and attached to the release assets
-      during the release process. The files should be added to the `assets` folder before creating
-      the first release. This step is **optional**, it depends on the type of your project.
+```bash
+make dev-up
+```
 
-[GitHub for non-software projects]:
-  https://github.com/IRNAS/irnas-guidelines-docs/blob/main/docs/github_projects_guidelines.md#github-for-non-software-projects-
+To stop the system in development mode, use:
 
-### Cleanup
+```bash
+make dev-down
+```
 
-- [ ] If you don't want to use `pre-commit` tool delete the following files:
-  - `.markdownlint.yaml`
-  - `.pre-commit-config.yaml`
-  - `.prettierrc.yaml`
-  - `committed.toml`
-  - `ruff.toml`
-  - `typos.toml`
-  - `.github/workflows/pre-commit.yaml`
-- [ ] As a final step delete this checklist and commit changes.
+For production mode, use:
 
-## Repository folder structure
+```bash
+make prod-up
+```
 
-- `docs/` - Project and development documentation.
-- `docs/images/` - Images used in the documentation, such as diagrams, screenshots, or logos.
-- `assets/` - Files that will be part of the next release process.
-- `CHANGELOG.md` - Document used for tracking changes in the project. It follows the
-  [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) format.
-- `.github` - GitHub Actions workflows and other GitHub related files.
+And to stop the production system:
+
+```bash
+make prod-down
+```
+
+> [!IMPORTANT] Each of the services can be deployed independently using their respective makefiles.
+
+### Configuration and Constants
+
+Before deploying the system, you need to configure certain environment variables and constants. This
+can be done by creating a `constants.py` file in specific directories.
+
+The following constants need to be defined in these folders:
+
+#### common/constants.py
+
+The constants found in the common/constants.py file are used throughout the application for
+consistent configuration.
+
+```python
+import logging
+
+LOG_LEVEL = logging.INFO
+
+DB_CONFIG = {
+    "dbname": "",  # Database name
+    "user": "",  # Database user
+    "password": "",  # Database password
+    "host": "",  # Internal database host
+    "port": ,  # Internal port
+}
+
+API_KEY = "your-secret-api-key"  # Replace with your actual key
+```

alertmanager/README.md

@@ -0,0 +1,52 @@
+# Alertmanager
+
+The `Alertmanager` is
+[part of Prometheus](https://prometheus.io/docs/alerting/latest/alertmanager/), used to handle
+alerts sent by Prometheus server or other clients.
+
+In our case, the `Alertmanager` is configured to send alerts to a dedicated `Slack` channel. It can
+be configured to send alerts to multiple `Slack` channels or other endpoints.
+
+## Slack integration
+
+A handy guide on `Slack` integration provided by Grafana is available
+[on this link](https://grafana.com/blog/2020/02/25/step-by-step-guide-to-setting-up-prometheus-alertmanager-with-slack-pagerduty-and-gmail/).
+
+Configuration of `Slack webhooks` is described in the official `Slack`
+[documentation](https://api.slack.com/messaging/webhooks).
+
+The rules we use to trigger alerts are defined in the `alertmanager.yml` file.
+
+Prometheus also needs to be configured properly, the following configuration in the `prometheus.yml`
+file is required:
+
+```yaml
+alerting:
+  # Alert manager instance
+  alertmanagers:
+    - static_configs:
+        - targets:
+            - alert-manager:9093
+```
+
+Furthermore, the `rules.yml` file contains the rules that trigger alerts:
+
+```yaml
+# Prometheus rules for alerting
+# Alerts trigger when the conditions defined in the rules are met
+
+groups:
+  - name: InstanceDown
+    rules:
+      - alert: InstanceDown
+        expr: up == 0
+        for: 1m
+        # Annotation - additional informational labels to store more information
+        annotations:
+          title: "Instance {{ $labels.instance }} down"
+          description:
+            "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 1 minute."
+        # Labels - additional labels to be attached to the alert
+        labels:
+          severity: "critical"
+```

alertmanager/alertmanager.yml

@@ -0,0 +1,38 @@
+# Alertmanager configuration
+# Defines the notification routes and the external systems they communicate with
+
+global:
+  resolve_timeout: 1m
+  slack_api_url: 'https://hooks.slack.com/services/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'  # TODO: Hide this in a secret
+
+route:
+  receiver: 'slack-notifications'
+
+receivers:
+- name: 'slack-notifications'
+  slack_configs:
+  - channel: '#telespor-alerts'
+    send_resolved: true
+    icon_url: https://avatars3.githubusercontent.com/u/3380462
+    title: |-
+     [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
+     {{- if gt (len .CommonLabels) (len .GroupLabels) -}}
+       {{" "}}(
+       {{- with .CommonLabels.Remove .GroupLabels.Names }}
+         {{- range $index, $label := .SortedPairs -}}
+           {{ if $index }}, {{ end }}
+           {{- $label.Name }}="{{ $label.Value -}}"
+         {{- end }}
+       {{- end -}}
+       )
+     {{- end }}
+    text: >-
+     {{ range .Alerts -}}
+     *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}
+
+     *Description:* {{ .Annotations.description }}
+
+     *Details:*
+       {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
+       {{ end }}
+     {{ end }}

common/authentication.py

@@ -0,0 +1,17 @@
+from fastapi import Depends, HTTPException, status
+from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
+
+from common.constants import API_KEY
+
+security = HTTPBearer()
+
+
+def verify_token(credentials: HTTPAuthorizationCredentials = Depends(security)):
+    """Verify the API key from the Authorization header."""
+    token = credentials.credentials
+    if token != API_KEY:
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="Invalid API key",
+        )
+    return token

common/constants.py

@@ -0,0 +1,17 @@
+import logging
+import os
+
+LOG_LEVEL = logging.INFO
+
+localhost = os.environ.get("LOCALHOST") == "true"
+
+# Localhost config
+DB_CONFIG = {
+    "dbname": "ajdovscina",
+    "user": "ajdovscina-geospatial-user",
+    "password": "password",
+    "host": "localhost" if localhost else "geospatial-data",
+    "port": 13328 if localhost else 5432,
+}
+
+API_KEY = "your-secret-api-key"  # Replace with your actual key