Threat Intel Database

Local threat intelligence database backed by MongoDB. Aggregates data from MISP feeds, vendor security blogs, open-source threat actor catalogs, and dark web monitoring into a single queryable store.

Data Sources

Source	What	Records
5 MISP feeds	IOCs: IPs, domains, URLs, hashes	~3,600
12 vendor RSS blogs	Structured threat reports (LLM-extracted)	growing
5 threat actor catalogs	Merged actor profiles with MITRE ATT&CK	~1,400
deepdarkCTI	Ransomware gangs, forums, markets, CVEs	~1,700

Prerequisites

• Linux (tested on Debian/Kali)
• just command runner
• Ansible (for automated setup)
• Python 3.10+ via micromamba
• ~2 GB disk for MongoDB data
• (Optional) GPU server with Ollama for threat report extraction

Installation

# Clone the repo
git clone <this-repo-url> && cd gizur-misp

# Install all prerequisites (MongoDB, Python packages, tools)
just setup

# Or dry-run to see what would change
just setup --check

The Ansible playbook installs MongoDB 8.0 from tarball, mongosh, Python dependencies (pymongo, requests, feedparser, trafilatura), and configures micromamba — all without root.

Configuration

Copy the example and edit as needed:

cp .env.example .env

All settings have sensible defaults that work out of the box. Available variables:

Variable	Default	Description
MONGO_URI	mongodb://127.0.0.1:27017	MongoDB connection string
MONGO_DB	misp	Database name
MONGO_PORT	27017	Port for local mongod
MONGOD_PATH	~/bin/mongod	Path to mongod binary
MONGOSH_PATH	~/bin/mongosh	Path to mongosh binary
MONGODUMP_PATH	~/bin/mongodump	Path to mongodump binary
PYTHON_PATH	~/micromamba-base/bin/python3	Path to Python interpreter
OLLAMA_URL	http://localhost:11434	Ollama API URL
OLLAMA_MODEL	qwen3:8b	Model for threat report extraction
OLLAMA_VIA_SSH	(unset)	SSH host where Ollama runs

Without a .env file, all defaults apply and the system works as-is.

Quick Start

just db start       # start local MongoDB
just misp sync      # download MISP feed events
just actors sync    # download & merge threat actor catalogs
just darkweb sync   # clone deepdarkCTI, parse dark web intel
just status         # dashboard: processes, DB stats, sync progress

To also pull vendor blog reports (requires Ollama):

just reports sync   # fetch RSS, extract threat intel via LLM

Or sync everything at once:

just sync                  # all sources (including reports)
just sync --skip-reports   # skip LLM-dependent report extraction

Commands

Status and Overview

just status                    # full dashboard
just db status                 # MongoDB database sizes

MISP Feeds

just misp sync                 # sync events (default: since 2023-01-01)
just misp sync --since 2024-01-01   # sync from specific date
just misp sync --dry-run       # preview without downloading
just misp status               # per-feed stats
just misp recent 7d            # events from last 7 days
just misp recent 2w            # events from last 2 weeks
just misp query 3              # view event #3 from recent listing
just misp query "APT28"        # search events by keyword

Threat Reports

just reports sync              # fetch new posts, extract via LLM
just reports sync --dry-run    # show new posts without processing
just reports status            # collection stats + top actors
just reports search "APT28"    # search by actor, TTP, malware, CVE
just reports list              # 20 most recent reports

Threat Actors

just actors sync               # download & merge from 5 sources
just actors status             # catalog stats
just actors search "APT28"     # search by name, alias, country
just actors show "APT28"       # full detail view
just actors list origin        # grouped by origin country
just actors list region DE     # actors targeting Germany
just actors list industry      # grouped by targeted industry
just actors profile SE         # full threat profile for Sweden

Dark Web Intel

just darkweb sync              # clone deepdarkCTI, parse, update DB
just darkweb status            # entity stats
just darkweb search "LockBit"  # search by name, URL, attack type
just darkweb gangs --online    # online ransomware gangs
just darkweb forums --online   # online forums/markets

Exploited CVEs

just cves status               # top vendors, vuln types
just cves search "Fortinet"    # search by CVE ID, vendor, type
just cves show CVE-2024-21762  # full detail with actor cross-refs

Database Management

just db start                  # start MongoDB
just db stop                   # stop MongoDB
just db shell misp             # open mongosh
just db export                 # gzip-compressed dump
just db restore <dir>          # restore from export

Database Schema

All data lives in a single MongoDB database (default: misp) with 6 collections:

Collection	Contents	Key fields
events	Full MISP event documents	Event.uuid, Event.date, _feed.name
attributes	Spillover attributes from events >16 MB	_event_uuid, type, value
threat_reports	Structured intel from vendor blogs	url, source, actors.name
threat_actors	Merged actor catalog from 5 open sources	name, aliases, country
darkweb_sites	Ransomware gangs, forums, markets, Telegram actors	name, type, status
cves	Commonly exploited CVEs with PoC links	cve_id, vendor, actor_names

License

This project aggregates data from publicly available open-source threat intelligence feeds. Respect the individual licenses of each data source.