MHBench: A system for generating multi-host environments for evaluating autonomous network attackers and defenders

MHBench is a system for generating multi-host environments to evaluate autonomous attackers against autonomous defenders. MHBench has 10 manually tuned environments based on real-world reports and 30 programmatically generated environments.

Research Papers:

On the Feasibility of Using LLMs to Execute Multistage Network Attacks

Perry: A High-level Framework for Accelerating Cyber Deception Experimentation

Requirements

• OpenStack project with API access and the ability to create networks, routers, floating IPs, and compute instances.
• Hardware requirements for local Openstack cluster: 64 vCPUs, 128 GB RAM, ~2 TB SSD.
• Python 3.13+ and uv for dependency management.

Setup

1. Install dependencies

   uv sync

2. Create a config file

   cp config/config_example.json config/config.json

   Fill in your OpenStack credentials (openstack_config), external IP, and any Elastic/C2 settings you use. The openstack_setup/ directory contains helper scripts for bootstrapping DevStack or Kolla if you need an OpenStack node.

Available environments

Pass one of these values to --type: EquifaxLarge, EquifaxMedium, EquifaxSmall, ICSEnvironment, ChainEnvironment, PEChainEnvironment, Star, StarPE, Dumbbell, DumbbellPE, EnterpriseA, EnterpriseB, Chain2Hosts, DevEnvironment, DevPrivTestEnvironment.

Usage

• Compile a topology: setup VMs, install dependencies, and save images.

  uv run python main.py --type EquifaxSmall compile

  Add --skip_network if you already built the network.

• Tear down all resources created by the range:

  uv run python main.py --type EquifaxSmall teardown

Run logs, Ansible artifacts, and Terraform outputs are stored under ./output/misc/<timestamp>/.

Tips

• Ensure your SSH key referenced in openstack_config.ssh_key_path exists and is registered in OpenStack.
• Snapshots are automatically managed during compile; use setup to restore from existing snapshots without rebuilding everything.