AutoResearch

Autonomous agentic framework for ML experimentation. A Planner-Executor-Critic loop powered by OpenAI GPT-4o and GPT-4o-mini that autonomously selects models, trains them, evaluates results, and decides when to stop — no human in the loop.

---

Architecture

main.py (CLI)
    |
loop.py (Orchestrator)
    |
    +-- Planner (gpt-4o)     Selects next model + hyperparams based on history
    +-- Executor (gpt-4o-mini) Calls tools to train and evaluate the model
    +-- Critic (gpt-4o)      Detects regressions, decides CONTINUE or STOP
            |
        Tool Layer (MCP-style)
            list_datasets | load_dataset | list_available_models
            train_model   | evaluate_model | compare_models | analyze_failure
            |
        Experiment Log (experiments.json)  — append-only run history

LLM vs VLM Routing

The router (router/model_router.py) automatically selects the model based on dataset modality:

Dataset type	Model used
Tabular (iris, wine, etc.)	gpt-4o-mini (fast, cost-efficient)
Vision (mnist_images)	gpt-4o (vision-capable)

---

Requirements

• Python 3.10+
• An OpenAI API key

---

Setup

1. Clone the repo

git clone <repo-url>
cd AutonomousAgent

2. Create a virtual environment (recommended)

python -m venv venv

# Windows
venv\Scripts\activate

# macOS / Linux
source venv/bin/activate

3. Install dependencies

pip install -r requirements.txt

4. Add your OpenAI API key

Copy the example env file and fill in your key:

cp .env.example .env

Edit .env:

OPENAI_API_KEY=sk-...

---

Usage

python main.py --dataset <dataset> --goal "<goal>" --max-iter <n>

Arguments

Argument	Default	Description
--dataset	iris	Dataset to use (see below)
--goal	maximize accuracy	Natural language optimization goal
--max-iter	10	Maximum agent loop iterations

Available Datasets

Name	Task	Features	Classes
iris	Classification	4	3
wine	Classification	13	3
breast_cancer	Classification	30	2
digits	Classification	64	10
diabetes	Regression	10	—

Examples

# Maximize accuracy on iris (classification)
python main.py --dataset iris --goal "maximize accuracy" --max-iter 5

# Maximize F1 score on breast cancer dataset
python main.py --dataset breast_cancer --goal "maximize f1 score" --max-iter 8

# Minimize MSE on diabetes (regression)
python main.py --dataset diabetes --goal "minimize mse" --max-iter 10

# Run with more iterations on a harder dataset
python main.py --dataset digits --goal "maximize accuracy" --max-iter 15

---

Output

Each iteration prints:

Iteration 1/5
Planner -> selecting next experiment...
+-- Plan -------------------------------------------------------+
| Model: RandomForest  Hyperparams: {n_estimators: 100, ...}    |
| Reasoning: ...                                                 |
+---------------------------------------------------------------+
Executor -> training and evaluating...
| Metric    | Value |
| accuracy  | 0.967 |
| f1        | 0.966 |
+-- Critic -----------------------------------------------------+
| Verdict: CONTINUE                                              |
| Best metric so far: 0.967                                      |
| Suggestions: Try GradientBoosting with higher n_estimators    |
+---------------------------------------------------------------+

At the end:

• Best model ID and metric printed to stdout
• Full run history saved to experiments.json

---

Project Structure

AutonomousAgent/
├── main.py                  # CLI entry point
├── loop.py                  # Planner -> Executor -> Critic orchestrator
├── config.py                # Model names, thresholds, API key loading
├── experiment_log.py        # Append-only JSON experiment log
├── requirements.txt
├── .env                     # Your API key (git-ignored)
├── .env.example             # Template
├── agents/
│   ├── planner.py           # gpt-4o: proposes next experiment as JSON
│   ├── executor.py          # gpt-4o-mini: calls tools, runs training/eval
│   └── critic.py            # gpt-4o: evaluates results, returns STOP/CONTINUE
├── tools/
│   ├── __init__.py          # TOOL_REGISTRY + TOOL_SCHEMAS for OpenAI function calling
│   ├── dataset_tools.py     # load_dataset, list_datasets
│   ├── training_tools.py    # train_model, list_available_models
│   ├── evaluation_tools.py  # evaluate_model, compare_models
│   └── analysis_tools.py    # analyze_failure (LLM-powered diagnosis)
└── router/
    └── model_router.py      # Routes tabular -> gpt-4o-mini, vision -> gpt-4o

---

Configuration

Edit config.py to change defaults:

PLANNER_MODEL = "gpt-4o"        # Model for planning and criticism
EXECUTOR_MODEL = "gpt-4o-mini"  # Model for tool-calling execution
MAX_ITERATIONS = 10             # Default loop limit
CONVERGENCE_THRESHOLD = 0.001   # Min improvement to continue

---

How It Works

1. Planner (gpt-4o) receives the goal and full experiment history, then returns the next experiment spec (model + hyperparams) as JSON.
2. Executor (gpt-4o-mini) receives the spec and uses OpenAI function calling to invoke tools: load_dataset → train_model → evaluate_model.
3. Critic (gpt-4o) receives the metrics and history. It detects convergence (improvement < 0.001 for 3 consecutive iterations) and returns CONTINUE or STOP with reasoning.
4. Results are appended to experiments.json after every iteration.
5. The loop terminates when the Critic says STOP or --max-iter is reached.

---

Extending

Add a new dataset:

In tools/dataset_tools.py, add an entry to AVAILABLE_DATASETS:

"my_dataset": ("classification", datasets.load_my_dataset),

Add a new model:

In tools/training_tools.py, add to CLASSIFIER_REGISTRY or REGRESSOR_REGISTRY:

"XGBoost": XGBClassifier,

Add a new tool:

1. Write the function + JSON schema in tools/
2. Register it in tools/__init__.py under TOOL_REGISTRY

---

License

MIT