Skip to content

Add MT5 optimization set support and terminal instance settings#4

Open
Marinski wants to merge 3 commits into
psyb0t:masterfrom
algotradingspace:feat/strategy-optimizer
Open

Add MT5 optimization set support and terminal instance settings#4
Marinski wants to merge 3 commits into
psyb0t:masterfrom
algotradingspace:feat/strategy-optimizer

Conversation

@Marinski
Copy link
Copy Markdown

@Marinski Marinski commented May 20, 2026

Check brief video explanation below:
MT5 HTTP API Backtest mode

Summary

This PR extends mt5-httpapi in two focused areas:

  1. terminal instance settings in the single-file config.yaml, so one broker/account can expose multiple MT5 terminal instances clearly through the existing startup pipeline (If instance is omitted or empty, the code treats it as default)
  2. MT5 optimization support in the backtest flow, including parsing optimization-style .set files and generating MT5-native .set content from structured JSON

The terminal-instance part is useful for bulk backtesting against the same broker, because it lets the config describe separate terminal clones cleanly while still flowing through the existing config.yaml -> config_helper.py -> start.bat -> api_runner.bat -> mt5api startup chain.

The branch lands as 2 commits on top of v4.3.1:

  1. feat(config): add terminal instance settings
  2. feat(backtest): add optimization set parsing

Motivation

There were two gaps in the current setup:

  1. the config model needed a cleaner way to represent multiple MT5 terminal instances for the same broker/account while keeping the current single-file YAML and startup/runtime flow intact
  2. the backtest API could already run tester jobs, but it did not yet support MT5 optimization-oriented .set workflows, which are required for parameter sweeps and optimization runs

This PR addresses both without changing the overall Docker + Windows VM deployment model or the existing /<broker>/<account>/... routing model.

What this PR adds

1. Terminal instance settings in the single-file config

A terminal entry in config.yaml can now carry an optional instance field so the same broker / account can be declared more than once on different ports.

This flows through:

  • config/config.yaml.example
  • mt5api/config.py
  • scripts/config_helper.py
  • scripts/start.bat
  • scripts/api_runner.bat
  • mt5api/main.py
  • mt5api/mt5client.py

The result is that one broker/account can expose multiple named/isolated terminal instances more clearly from the config layer, which is useful both for live trading and for backtesting/optimization against the same broker under separate terminal state.

Important routing note:

  • there is no queue field in config.yaml
  • instance selection is done by routing requests to a specific instance URL
  • explicit clones are addressed as /<broker>/<account>/<instance>/...
  • the legacy /<broker>/<account>/... alias remains only for the default instance

2. Optimization .set parsing and generation

The backtest flow now supports MT5 optimization-style parameter files.

New pieces added:

  • mt5api/backtest/optimization_parser.py
  • mt5api/backtest/set_builder.py

This adds support for MT5 UI-style optimization markers like:

Take_Profit=92||80||4||92||Y
Stop_Loss=0||0||1||10||N

Meaning:

  • current value
  • range start
  • step
  • range stop
  • optimize on/off (Y / N)

The API can now either:

  1. accept an MT5-saved optimization .set file directly
  2. generate MT5-native .set content from structured JSON

3. Backtest / optimization workflow support

The backtest handler and INI builder now support optimization-oriented execution in addition to plain tester runs.

That includes:

  • optimization mode on the generated INI
  • optimization criterion support
  • optimization-aware report behavior
  • parsed top-N optimization results from the MT5 XML report
  • support for returning optimization results in the job status payload

4. Documentation and support tooling

This PR also updates the docs and helper tooling around the new workflow, including:

  • README.md
  • config/config.yaml.example
  • scripts/compile-warmup-ea.bat

API behavior

Plain backtests and optimizations now share the same endpoint family, but differ by the INI and .set inputs.

Plain backtest

For a normal backtest, leave [Tester].Optimization=0 or omit it.

Optimization run

For an optimization, set [Tester].Optimization to one of:

  • 1 - slow complete algorithm
  • 2 - fast genetic algorithm
  • 3 - all symbols selected in Market Watch

Optimization runs require a .set file whose parameters contain MT5 optimization ranges.

Example requests

1. Build an optimization .set file from JSON

If the caller already has structured parameter metadata, the API can generate MT5-native .set content directly:

export URL=http://127.0.0.1:8888/darwinex/live/a
export TOK=changeme-mt5-httpapi-token

curl -sS -X POST "$URL/backtest/build-set" \
  -H "Authorization: Bearer $TOK" \
  -H "Content-Type: application/json" \
  -d '{
    "comments": [
      "saved on 2026.05.15 08:30:02",
      "this file contains input parameters for testing/optimizing MyEA"
    ],
    "parameters": [
      {"name": "_Properties_", "value": "------"},
      {
        "name": "Take_Profit",
        "value": 92,
        "start": 80,
        "step": 4,
        "stop": 92,
        "optimize": true
      },
      {
        "name": "Stop_Loss",
        "value": 0,
        "start": 0,
        "step": 1,
        "stop": 10,
        "optimize": false
      }
    ]
  }' > optimization.set

The response is plain-text MT5 .set content, ready to save or upload.

2. Build an optimization INI

The optimization itself still runs through the existing backtest endpoint family. The first step is to generate a tester INI with optimization enabled:

export URL=http://127.0.0.1:8888/darwinex/live/a
export TOK=changeme-mt5-httpapi-token

curl -sS -X POST "$URL/backtest/build-ini" \
  -H "Authorization: Bearer $TOK" \
  -H "Content-Type: application/json" \
  -d '{
    "symbol":"GBPCAD",
    "timeframe":"M15",
    "expert":"EA Studio GBPCAD M15 1615044595.ex5",
    "lastYears":1,
    "modelling":"open-prices",
    "expertParameters":"ea studio gbpcad m15 1615044595.take-profit-opt-80-92-step4.set",
    "optimization":1,
    "optimizationCriterion":0,
    "reportName":"gbpcad-m15-last1y-openprices-opt"
  }' > tester.ini

3. Submit the optimization job to POST /backtest

Once the INI is built, the request still goes through the same multipart POST /backtest endpoint:

JOB=$(curl -sS -X POST "$URL/backtest" \
  -H "Authorization: Bearer $TOK" \
  -F "ini=@tester.ini;filename=tester.ini" \
  -F "expert_name=EA Studio GBPCAD M15 1615044595.ex5" \
  -F "set_name=ea studio gbpcad m15 1615044595.take-profit-opt-80-92-step4.set" \
  -F "topPasses=20" \
  | jq -r .jobId)

echo "Submitted job: $JOB"

4. Poll the optimization job

while :; do
  STATUS_JSON=$(curl -sS -H "Authorization: Bearer $TOK" "$URL/backtest/$JOB")
  STATUS=$(printf '%s' "$STATUS_JSON" | jq -r '.status')
  echo "Status: $STATUS"
  [[ "$STATUS" == completed || "$STATUS" == failed ]] && break
  sleep 10
done

5. Inspect parsed optimization results

For optimization jobs, the status payload includes optimization metadata and parsed top-N rows from the MT5 XML report:

printf '%s\n' "$STATUS_JSON" | jq '{jobId,status,exitCode,durationSeconds,optimizationType,optimizationResults}'

6. Fetch the raw MT5 optimization report

Optimizations return the MT5 XML spreadsheet export at the report endpoint:

curl -sS -H "Authorization: Bearer $TOK" "$URL/backtest/$JOB/report" -o optimization-report.xml

So the intended optimization flow is:

  1. create or upload an optimization .set
  2. generate an INI with optimization enabled
  3. submit the multipart job to POST /backtest
  4. poll GET /backtest/<job_id>
  5. read parsed optimizationResults
  6. optionally fetch the raw XML report from GET /backtest/<job_id>/report

7. Route tests and optimizations to different terminal instances

Because instance selection happens in the URL, callers can direct work to different clones of the same broker/account explicitly.

Example:

  • send plain backtests to http://127.0.0.1:8888/darwinex/live/a/...
  • send optimization runs to http://127.0.0.1:8888/darwinex/live/b/...

That routing decision lives in the caller or manager layer, not in a config-level queue selector.

Compatibility

Fully additive:

  • the existing deployment model is unchanged
  • the existing route layout is unchanged
  • plain backtest flows continue to work
  • optimization support is opt-in through the INI and .set inputs
  • terminal-instance config extends the existing YAML/startup pipeline rather than replacing it

Validation

Focused test slice is green:

  • tests/test_terminal_instances.py
  • tests/test_backtest_handler.py
  • tests/test_backtest_ini_builder.py
  • tests/test_backtest_jobs.py
  • tests/test_backtest_set_builder.py
  • tests/test_optimization_parser.py

Result:

  • 67 passed

Review notes

The branch is intentionally split so the changes can be reviewed in two layers:

  1. terminal instance config / startup pipeline support
  2. optimization .set parsing and optimization execution support on top of the backtest API

The core idea is still narrow:

  • represent terminal instances more clearly in the single-file config
  • let MT5 optimization runs use native .set ranges and flow through the existing backtest endpoint family
  • keep instance selection explicit through URL routing rather than adding a separate config-level queue selector

@Marinski Marinski force-pushed the feat/strategy-optimizer branch from cf903e9 to fb5d2da Compare May 20, 2026 07:05
@Marinski Marinski marked this pull request as ready for review May 20, 2026 07:55
@psyb0t
Copy link
Copy Markdown
Owner

psyb0t commented May 21, 2026

Hey there @Marinski!
Nice stuff in here but about the instance being useful when live. Could you come up with a good reason? Why would we want 2 instances of MT5 for the same account when live?

@Marinski
Copy link
Copy Markdown
Author

Fair question.

The instance field exists primarily for config model cleanliness, not live trading strategy. But it also prevents key collisions when you need two separate processes pointing at the same broker/account pair - for example, running a live terminal alongside a backtest terminal on the same account. instance is a single optional field; the routing fallback alias handles the default case, and the per-instance directory isolation is a necessary consequence of how MT5 locks its data directory.

I believe that the current approach is the right tradeoff but I can make adjustments if you have something different in mind.

@Marinski
Copy link
Copy Markdown
Author

Also this is not related to the current PR but I wanted to discuss this with you. I wonder if you have an idea about a possible workaround for the locked history files when the Strategy Tester is actively using them.

The core problem is that while the Strategy Tester is running, it holds the relevant history-cache files (.hcc in bases<broker>\history<symbol>, plus the generated tick data under tester\bases) with Windows sharing modes that block outside writes and sometimes reads. Any external job that tries to sync, copy, or rotate those files mid-run will either fail outright or corrupt the cache and force a full re-download on next load.

When backtesting EURCAD for example on a USD account, the tester pulls EURCAD itself, plus USDCAD (or CADUSD) to convert CAD profit into USD. Depending on the symbol's margin calculation mode, it may also pull EURUSD to convert the EUR-denominated margin. And if a direct conversion pair doesn't exist in the broker's symbol set, the tester walks a two-hop bridge - e.g. EURGBP / GBPUSD - though that only happens with exotic crosses or thin symbol lists, not majors.
If we actively backtest on, say, 30 symbols × 8 years of full tick data, that's roughly 40–50GB of disk per terminal - so 5–10 instances each holding a physical copy would mean 250–500GB of pure duplication.

This is one potential solution I'm thinking about. We run one platform that acts as a master, storing all historical data from all brokers. A scheduled daily job keeps that master current - ideally by pulling history directly per symbol (via the MT5 Python module / httpapi bridge or a dedicated data-sync routine), with a dummy backtest as a fallback trigger where direct pulls aren't practical. The master is read-only and exists exactly once on disk.

The instances don't copy from it - they reference it. The overlay is built on the Linux host (where the master lives) and presented to each Windows MT5 VM as that terminal's history location, so every instance sees a complete bases directory while the bytes physically exist only once.

Before a run, a pre-run resolver takes the test request (symbol + account currency) and computes the conversion chain deterministically - EURCAD needs USDCAD for profit, EURUSD for margin, with bridge resolution if a direct pair is missing. Instead of copying, the resolver just verifies those symbols' tick data already exist and are current in the read-only master. If they do - and after the daily refresh they should - the test launches against its overlay view and reads them straight through. No copy, no per-instance duplication.

If the resolver misses a symbol, or the terminal needs to download something mid-run, the write simply lands in that instance's private overlay scratch layer - the test continues, the master is untouched, and other instances are unaffected. Anything that shows up in a scratch layer is also a signal that the resolver or the daily refresh missed something, which we can then promote into the master so it's shared next time.

@psyb0t
Copy link
Copy Markdown
Owner

psyb0t commented May 25, 2026

Fair question.

The instance field exists primarily for config model cleanliness, not live trading strategy. But it also prevents key collisions when you need two separate processes pointing at the same broker/account pair - for example, running a live terminal alongside a backtest terminal on the same account. instance is a single optional field; the routing fallback alias handles the default case, and the per-instance directory isolation is a necessary consequence of how MT5 locks its data directory.

I believe that the current approach is the right tradeoff but I can make adjustments if you have something different in mind.

How about for live, because it's singular, we just keep it under /account and when in backtest mode we do /account/backtest/instance1,2,3... This way the mental model is cleaner and also we avoid having unused stuff under /account. For example I do a lot of rate scraping and my live dirs now have 150-200gb historical data. And we just ignore or error out when instance is set but mode is live.

@psyb0t
Copy link
Copy Markdown
Owner

psyb0t commented May 25, 2026

Also this is not related to the current PR but I wanted to discuss this with you. I wonder if you have an idea about a possible workaround for the locked history files when the Strategy Tester is actively using them.

The core problem is that while the Strategy Tester is running, it holds the relevant history-cache files (.hcc in bases\history, plus the generated tick data under tester\bases) with Windows sharing modes that block outside writes and sometimes reads. Any external job that tries to sync, copy, or rotate those files mid-run will either fail outright or corrupt the cache and force a full re-download on next load.

When backtesting EURCAD for example on a USD account, the tester pulls EURCAD itself, plus USDCAD (or CADUSD) to convert CAD profit into USD. Depending on the symbol's margin calculation mode, it may also pull EURUSD to convert the EUR-denominated margin. And if a direct conversion pair doesn't exist in the broker's symbol set, the tester walks a two-hop bridge - e.g. EURGBP / GBPUSD - though that only happens with exotic crosses or thin symbol lists, not majors. If we actively backtest on, say, 30 symbols × 8 years of full tick data, that's roughly 40–50GB of disk per terminal - so 5–10 instances each holding a physical copy would mean 250–500GB of pure duplication.

This is one potential solution I'm thinking about. We run one platform that acts as a master, storing all historical data from all brokers. A scheduled daily job keeps that master current - ideally by pulling history directly per symbol (via the MT5 Python module / httpapi bridge or a dedicated data-sync routine), with a dummy backtest as a fallback trigger where direct pulls aren't practical. The master is read-only and exists exactly once on disk.

The instances don't copy from it - they reference it. The overlay is built on the Linux host (where the master lives) and presented to each Windows MT5 VM as that terminal's history location, so every instance sees a complete bases directory while the bytes physically exist only once.

Before a run, a pre-run resolver takes the test request (symbol + account currency) and computes the conversion chain deterministically - EURCAD needs USDCAD for profit, EURUSD for margin, with bridge resolution if a direct pair is missing. Instead of copying, the resolver just verifies those symbols' tick data already exist and are current in the read-only master. If they do - and after the daily refresh they should - the test launches against its overlay view and reads them straight through. No copy, no per-instance duplication.

If the resolver misses a symbol, or the terminal needs to download something mid-run, the write simply lands in that instance's private overlay scratch layer - the test continues, the master is untouched, and other instances are unaffected. Anything that shows up in a scratch layer is also a signal that the resolver or the daily refresh missed something, which we can then promote into the master so it's shared next time.

Yeah for this thing this is the way to go tho. Or junctions which is simpler but just a bit error prone. How bout we keep that out of this PR?

@Marinski
Copy link
Copy Markdown
Author

@psyb0t I added one more commit here to improve tester optimization support across all MT5 optimization modes, not just mode 3.

Background:
For normal backtests (Optimization=0), MT5 can generate a programmatically consumable HTML report. Optimization runs are different: MT5 does not reliably produce a usable programmatic XML report for all optimization modes, and in practice the real optimization pass data lives in Tester/cache/*.opt.

This change adds cache-based optimization parsing so the API can reconstruct optimization results from the underlying MT5 cache data instead of depending only on the exported report artifact.

What changed:

  • optimization modes 1, 2, and 3 now handle MT5 optimization artifacts more reliably instead of failing when the expected plain <report>.xml output is missing or insufficient
  • added parsing for Tester/cache/*.opt, which is where MT5 actually stores optimization pass results
  • mode 3 still gets special handling because MT5 writes <report>.symbols.xml and stores the real pass rows in cache rather than in the report file
  • for mode 3, pass-to-symbol recovery was hardened using agent logs so all-symbol runs return stable Pass + Symbol mappings
  • backtest job payloads now expose optimizationCache metadata to make cache-backed optimization runs easier to debug
  • README/docs were updated with mode-by-mode behavior and API examples for optimization modes 1, 2, and 3

Validation:

  • focused backtest/optimization test suite passes
  • live replay against a real server using mode 3 completed successfully and returned both optimizationCache and parsed optimization results for the tested Market Watch symbols

@Marinski
Copy link
Copy Markdown
Author

Fair question.
The instance field exists primarily for config model cleanliness, not live trading strategy. But it also prevents key collisions when you need two separate processes pointing at the same broker/account pair - for example, running a live terminal alongside a backtest terminal on the same account. instance is a single optional field; the routing fallback alias handles the default case, and the per-instance directory isolation is a necessary consequence of how MT5 locks its data directory.
I believe that the current approach is the right tradeoff but I can make adjustments if you have something different in mind.

How about for live, because it's singular, we just keep it under /account and when in backtest mode we do /account/backtest/instance1,2,3... This way the mental model is cleaner and also we avoid having unused stuff under /account. For example I do a lot of rate scraping and my live dirs now have 150-200gb historical data. And we just ignore or error out when instance is set but mode is live.

Let me go throught the codebase again and I'll implement a fix and will tag you for review once I commit it.

Comment thread README.md

- locates the first installed broker `base` terminal under
`C:\Users\Docker\Desktop\Shared\terminals\*\base`
- copies `C:\Users\Docker\Desktop\Assets\experts\MT5SystemWarmup.mq5`
Copy link
Copy Markdown
Owner

@psyb0t psyb0t May 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MT5SystemWarmup.mq5

the warmup script is missing from the repo. it should not be gitignored. but just the source, not also the compiled one

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solved that one within commit 9f394fc

@Marinski Marinski force-pushed the feat/strategy-optimizer branch from 1789188 to 9f394fc Compare May 25, 2026 12:32
@Marinski
Copy link
Copy Markdown
Author

Fair question.
The instance field exists primarily for config model cleanliness, not live trading strategy. But it also prevents key collisions when you need two separate processes pointing at the same broker/account pair - for example, running a live terminal alongside a backtest terminal on the same account. instance is a single optional field; the routing fallback alias handles the default case, and the per-instance directory isolation is a necessary consequence of how MT5 locks its data directory.
I believe that the current approach is the right tradeoff but I can make adjustments if you have something different in mind.

How about for live, because it's singular, we just keep it under /account and when in backtest mode we do /account/backtest/instance1,2,3... This way the mental model is cleaner and also we avoid having unused stuff under /account. For example I do a lot of rate scraping and my live dirs now have 150-200gb historical data. And we just ignore or error out when instance is set but mode is live.

Let me go throught the codebase again and I'll implement a fix and will tag you for review once I commit it.

@psyb0t

I replaced the flat /<broker>/<account>/<instance>/ scheme with a mode-aware one. Live terminals keep the clean /<broker>/<account>/ path. Backtest terminals move to /<broker>/<account>/backtest/<instance>/. An instance field on a mode: live terminal is a hard error. Everything folds into the existing commit history - no new commits.

)
_SYMBOL_PATTERNS = (
re.compile(r"Symbols\s+([^:]+):\s+symbol to be synchronized", re.IGNORECASE),
re.compile(r"symbol\s+([A-Z0-9]{3,10})\s+to be synchronized", re.IGNORECASE),
Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  ┌────────────┬────────┬───────────────────────────────────────────────────────────┐                                
  │   Symbol   │ Match? │                            Why                            │
  ├────────────┼────────┼───────────────────────────────────────────────────────────┤                                
  │ EURUSD     │ ✅     │ 6 uppercase, fits 3-10                                    │
  ├────────────┼────────┼───────────────────────────────────────────────────────────┤
  │ XAUUSD     │ ✅     │                                                           │
  ├────────────┼────────┼───────────────────────────────────────────────────────────┤                                
  │ GER40.cash │ ❌     │ contains ., lowercase cash                                │
  ├────────────┼────────┼───────────────────────────────────────────────────────────┤                                
  │ Cocoa.c    │ ❌     │ uppercase C is fine, but lowercase ocoa + . + lowercase c │
  ├────────────┼────────┼───────────────────────────────────────────────────────────┤                                
  │ US500.cash │ ❌     │ . + lowercase                                             │
  ├────────────┼────────┼───────────────────────────────────────────────────────────┤                                
  │ #AAPL      │ ❌     │ starts with #                                             │
  ├────────────┼────────┼───────────────────────────────────────────────────────────┤                                
  │ BTCUSD     │ ✅     │                                                           │
  └────────────┴────────┴───────────────────────────────────────────────────────────┘ 

this doesn't pass a couple of stuff that usually appears in brokers. like in roboforex procent, teledrade 1:1 and FTMO and i think ICMarkets too we have stuff like SYMBOL.cash or EURUSDpro or .r or stuff.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants