Add ONNX Runtime backend support (OpenVINO / DirectML / TensorRT / CPU, Windows & Linux)

[seniorfish]

updated

Add ONNX Runtime neural-net backend

Feedback welcome — see notes at the bottom.

Summary

This PR adds a new USE_BACKEND=ONNX option that runs KataGo's neural-net inference through ONNX Runtime, with pluggable execution providers.

It targets current master including the transformer trunk models, and reuses master's existing OnnxModelBuilder (shared with the TensorRT backend) instead of a parallel graph emitter — so the new backend is a single file plus wiring and touches no other backend.

Verified in this PR:

Provider	Where	Status
OpenVINO	Intel GPU & NPU (Arc B580 desktop, Arc 140T laptop)	verified end-to-end, benchmarked
DirectML	Windows (B580)	benchmarked
TensorRT	Ubuntu 24.04 NVIDIA RTX 4090	benchmarked
CPU	Windows	benchmarked

Other providers (cuda / migraphx / coreml) are wired in and should work but are unverified. Note: upstream verification status (per Compiling.md) is limited to OpenVINO on Windows + Intel GPU; DirectML / TensorRT / CPU are covered by this PR's CI and manual testing but are not continuously tested upstream.

What changed

• cpp/neuralnet/onnxbackend.cpp (new) — the full NeuralNet::* backend implementation behind #ifdef USE_ONNX_BACKEND. It serializes the model graph via the existing OnnxModelBuilder into an ONNX ModelProto and feeds it to an Ort::Session; input filling and output decode are copied line-for-line from trtbackend, so the IO protocol and post-processing are identical to TensorRT. onnxProvider selects among cpu / openvino / cuda / tensorrt / migraphx / coreml / directml.
• cpp/neuralnet/onnxmodelbuilder.{h,cpp} — graph inputs are now always declared in OpenVINO EP required order.
• DirectML EP wiring — __has_include-guarded dml_provider_factory.h plus DisableMemPattern + single-threaded execution, Windows only.
• OpenVINO EP options — per-thread device_type selection (onnxOpenVINODeviceType + onnxOpenVINODeviceTypeThread<N>, device indices map to a .N suffix), NumStreams/Precision/CacheDir/DeviceConfig_*, and SetIntraOpNumThreads(1) scoped to this provider.
• Scale-8 — applyScale8ToReduceActivations applied by default, with an onnxSkipScale8 opt-out (NPU models).
• cpp/CMakeLists.txt — new ONNX branch linking ORT + protobuf + zlib.
• cpp/program/setup.cpp — registers the onnx backend prefix; transformer trunks default to NHWC.
• cpp/main.cpp — version-info #elif USE_ONNX_BACKEND.
• cpp/program/gtpconfig.cpp — per-thread device config (onnxDeviceToUseThread<N> etc.).
• cpp/configs/gtp_example.cfg — ONNX settings example block.
• Compiling.md — per-EP build docs plus an execution-provider support matrix.
• CI — a single .github/workflows/onnx-backend.yml matrix workflow (CPU Windows/Linux, DirectML, OpenVINO, TensorRT in an NGC container) with runtests, backend verification, caches, and self-contained artifacts.

Extensibility:

The architecture is provider-agnostic: onnxProvider selects the EP and adding a future EP requires no code change — only config. New onnx* config keys are auto-recognized by the onnx prefix in setup.cpp, so provider-specific options can be added without touching the backend core.

How To Use

Prebuilt binaries:

• Windows x64 (OpenVINO EP + CPU, all runtime DLLs bundled): https://github.com/seniorfish/KataGo/releases/tag/onnx-backend-v1
• CI artifacts for each EP/platform: download from the ONNX backend build & test workflow (katago-<OS>-onnx-<ep>, e.g. katago-Linux-onnx-tensorrt).

The backend adds a family of onnx* keys to the config file (see configs/gtp_example.cfg for the full list). The essential ones:

Key	Default	Purpose
onnxProvider	cpu	Execution provider: cpu/openvino/cuda/tensorrt/migraphx/coreml/directml
onnxOpenVINODeviceType	GPU	OpenVINO device: GPU/CPU/NPU/AUTO:.../MULTI:...
onnxOpenVINODeviceTypeThread<N>	—	Per-thread override of onnxOpenVINODeviceType
onnxOpenVINOCacheDir	empty	Compile cache dir — speeds up cold starts
onnxOpenVINONumStreams	1	OpenVINO execution streams
onnxOpenVINOPrecision	empty	FP16/FP32/ACCURACY
onnxTransformerNHWC	true	NHWC trunk for transformer models
onnxSkipScale8	false	true = Skip applyScale8ToReduceActivations for speed

To use an Intel GPU you must set both:

onnxProvider = openvino
onnxOpenVINODeviceType = GPU

Tip:

• if you are on a discrete Intel GPU rather than an integrated one, setting onnxOpenVINONumStreams = 2 noticeably improves throughput.
• If you have both an Intel GPU and an NPU, you can use the NPU and iGPU together.

Benchmark

katago benchmark, -visits 60, default config otherwise noted. EloDiff is relative to the first row of each group.

Desktop — Intel Arc B580

OpenCL (baseline). Model b11c768h12nbt3tflrs-fson-silu.bin.gz (transformer, v17):

numSearchThreads	visits/s	nnEvals/s	nnBatches/s	avgBatchSize	time	EloDiff
2	14.61	14.11	14.06	1.00	41.7s	baseline
4	15.39	15.12	7.76	1.95	40.9s	−4
8	16.74	16.72	4.67	3.58	40.0s	−20

ONNX Runtime (OpenVINO EP). Model b11c768h12nbt3tflrs-fson-silu.bin.gz, onnxOpenVINONumStreams = 2, onnxTransformerNHWC = true:

numSearchThreads	visits/s	nnEvals/s	nnBatches/s	avgBatchSize	time	EloDiff
4	273.22	216.37	108.51	1.99	29.4s	baseline
8	362.81	290.80	73.26	3.97	22.2s	+87
12	394.78	321.04	54.35	5.91	20.5s	+102

Model kata1-zhizi-b40c768nbt-s11472M-d5982M.bin.gz (convnet, v15), onnxOpenVINONumStreams = 2:

numSearchThreads	visits/s	nnEvals/s	nnBatches/s	avgBatchSize	time	EloDiff
4	122.82	96.05	48.20	1.99	65.4s	baseline
8	156.49	125.23	31.55	3.97	51.6s	+62
12	169.99	134.65	22.75	5.92	47.7s	+66

Ablation (b11 transformer, numSearchThreads = 12), isolating NumStreams / transformerNHWC:

onnxOpenVINONumStreams	onnxTransformerNHWC	visits/s	nnEvals/s	avgBatchSize	time
1	true	126.72	103.04	5.91	63.9s
2	true	394.78	321.04	5.91	20.5s
2	false	185.39	150.18	5.91	43.7s

Both knobs matter: NumStreams = 2 ≈ 3× throughput vs 1, and transformerNHWC = true ≈ 2× vs false on this model.

Laptop — Intel Core Ultra 9 285H, Intel Arc 140T

ONNX Runtime (OpenVINO EP), numSearchThreads = 8:

numSearchThreads	visits/s	nnEvals/s	nnBatches/s	avgBatchSize	time	EloDiff
8	58.22	57.88	11.80	4.90	14.9s	baseline

More conditions and benchmark results are below.

[seniorfish]

Prebuilt binaries with ONNX Runtime + OpenVINO 2026.2.1 for Intel GPUs: https://github.com/seniorfish/KataGo/releases/tag/onnx-backend-v2 ~20% faster than v1 on transformer models (numStreams=2, 10 threads, 800 visits, 347.49 visits/s -> 416.52 visits/s).

[kaorahi]

Thank you for sharing your work! For some reason, it runs at only about 36% of the speed of #1171 in my environment (Intel Core Ultra 7 255U, Linux).

• OpenCL: 25.16 visits/s with kata1-b18c384nbt
• PR 1171 (iGPU): 76.33
• PR 1171 (NPU): 87.30
• PR 1171 + 3ed8922 (NPU + iGPU): 142.14
• PR 1222 (NPU): 31.52

Also, what do you think about supporting the combined use of the NPU and iGPU? Simply setting onnxOpenVINODeviceType = MULTI:NPU,GPU didn't make much difference. For a significant speedup, we may need "per-thread OpenVINO device and batch size configuration" like the patch linked above.

[kaorahi]

(1) katago benchmark fails with the older small model g170-b6c96-s175395328-d26788732.txt.gz when onnxOpenVINODeviceType = NPU. It works with CPU and GPU.

terminate called after throwing an instance of 'StringError'
  what():  Got nonfinite for policy sum

(2) It also fails with the model kata1-b18c384nbt-s9996604416-d4316597426.bin.gz when onnxOpenVINODeviceType = GPU. It works with CPU and NPU.

ERROR: Async bot thread failed: Got nonfinite for policy sum

(3) Removing modelDesc.applyScale8ToReduceActivations(); from cpp/neuralnet/onnxbackend.cpp makes NPU inference run at 4× the original speed (= 1.5× the speed of #1171).

• PR 1222 - Scale8 (NPU): 129.25 visits/s with kata1-b18c384nbt

[seniorfish]

@kaorahi, thank you. That's a very insightful observation. The patch has been committed. The new version gives me roughly a 4x performance improvement here as well (Core Ultra 9 285H reaching 54 visits/s, NPU).

Regarding your second issue, kata1-b18c384nbt-s9996604416-d4316597426 seems to run fine on my end, both on the B580 and the 285H, with onnxOpenVINODeviceType = GPU.
I'll test the first issue later.

As for your suggestion about supporting the combined use of the NPU and iGPU, I'll try to implement it.

[seniorfish]

@kaorahi, I referred to your implementation in 3ed8922 and implemented the combined use of the NPU and iGPU (even the CPU, but tests showed that its benefit was always negative).
The benchmark data is as follows.

for all
-model kata1-b18c384nbt-s9996604416-d4316597426.bin.gz, -t 4,6,8(or 6,8,10) -v 200(or 300)
Intel Core Ultra 285H(140T)
GPU only:
numSearchThreads =  4: 10 / 10 positions, visits/s = 109.65 nnEvals/s = 101.06 nnBatches/s = 50.34 avgBatchSize = 2.01 (18.5 secs) (EloDiff baseline)
numSearchThreads =  6: 10 / 10 positions, visits/s = 144.35 nnEvals/s = 132.80 nnBatches/s = 45.35 avgBatchSize = 2.93 (14.2 secs) (EloDiff +88)
numSearchThreads =  8: 10 / 10 positions, visits/s = 145.33 nnEvals/s = 135.35 nnBatches/s = 46.08 avgBatchSize = 2.94 (14.2 secs) (EloDiff +76)

NPU only:
numSearchThreads =  6: 10 / 10 positions, visits/s = 73.42 nnEvals/s = 67.51 nnBatches/s = 34.13 avgBatchSize = 1.98 (27.9 secs) (EloDiff baseline)
numSearchThreads =  8: 10 / 10 positions, visits/s = 80.82 nnEvals/s = 75.24 nnBatches/s = 37.95 avgBatchSize = 1.98 (25.6 secs) (EloDiff +18)
numSearchThreads = 10: 10 / 10 positions, visits/s = 79.81 nnEvals/s = 75.38 nnBatches/s = 38.07 avgBatchSize = 1.98 (26.2 secs) (EloDiff -4)

GPU + NPU with nnMaxBatchSize 6:2:
numSearchThreads =  4: 10 / 10 positions, visits/s = 139.54 nnEvals/s = 129.91 nnBatches/s = 113.74 avgBatchSize = 1.14 (14.5 secs) (EloDiff baseline)
numSearchThreads =  6: 10 / 10 positions, visits/s = 187.25 nnEvals/s = 170.89 nnBatches/s = 102.59 avgBatchSize = 1.67 (10.9 secs) (EloDiff +97)
numSearchThreads =  8: 10 / 10 positions, visits/s = 205.91 nnEvals/s = 194.57 nnBatches/s = 84.69 avgBatchSize = 2.30 (10.0 secs) (EloDiff +120)

GPU + NPU with nnMaxBatchSize 4:4:
numSearchThreads =  6: 10 / 10 positions, visits/s = 119.32 nnEvals/s = 107.19 nnBatches/s = 72.02 avgBatchSize = 1.49 (25.6 secs) (EloDiff baseline)
numSearchThreads =  8: 10 / 10 positions, visits/s = 237.83 nnEvals/s = 216.06 nnBatches/s = 86.79 avgBatchSize = 2.49 (12.9 secs) (EloDiff +249)
numSearchThreads = 10: 10 / 10 positions, visits/s = 259.90 nnEvals/s = 237.61 nnBatches/s = 73.09 avgBatchSize = 3.25 (11.9 secs) (EloDiff +272)

GPU + NPU + CPU 4:4:1:
numSearchThreads =  6: 10 / 10 positions, visits/s = 121.21 nnEvals/s = 109.64 nnBatches/s = 90.64 avgBatchSize = 1.21 (25.2 secs) (EloDiff baseline)
numSearchThreads =  8: 10 / 10 positions, visits/s = 128.33 nnEvals/s = 116.71 nnBatches/s = 64.17 avgBatchSize = 1.82 (23.9 secs) (EloDiff +6)
numSearchThreads = 10: 10 / 10 positions, visits/s = 181.46 nnEvals/s = 167.24 nnBatches/s = 69.23 avgBatchSize = 2.42 (17.0 secs) (EloDiff +126)

pre-builts: https://github.com/seniorfish/KataGo/releases/edit/onnx-backend-v4

[kaorahi]

I appreciate your onnx-backend-v4. It set a new record in my environment. katago benchmark recommended numSearchThreads = 5.

• PR 1222 v4 (NPU + iGPU): 189.89 visits/s with kata1-b18c384nbt

config:

numNNServerThreadsPerModel = 2
nnMaxBatchSizeThread0 = 1
onnxOpenVINODeviceTypeThread0 = NPU
onnxOpenVINODeviceTypeThread1 = GPU

By the way, I noticed a few things.

• Should the tag onnx-backend-v3 point to 4f86efa instead of 815378d ?
• Is this the correct URL for the prebuilt binaries (without edit) ? https://github.com/seniorfish/KataGo/releases/tag/onnx-backend-v4
• gpu_optimized.cfg and gtp_gpu_npu_example.cfg are missing from cpp/configs/. Is this intentional?

[seniorfish]

Fixed, it was just a tag pointing issue, the attachments are all correct. gpu_optimized.cfg and gtp_gpu_npu_example.cfg are merely two preset configurations for Intel hardware. To keep the commit clean, and since all newly added options are explained in gtp_example.cfg, I didn't include them in cpp/configs/.

[kaorahi]

This PR does not currently support HumanSL models, so I'm trying this patch to add support: d57c9bc. During analysis, I prefer to view a policy comparison between 5 kyu and 1 dan to learn from it.

[lightvector]

I added to the existing trt onnx builder support for HumanSL: 5ae42f8

Does that result in this having HumanSL support once it's rebased?

Starting to review a little bit! Regarding alignInputsToConsumptionOrder - why is this needed? If inputs in consumption order is now required, can we just simply document that it's required at the site that makes the inputs, and do the ordering unconditionally, rather than adding an extra argument?

Also, what is "consumption order" - for example if we enter the !requireExactNNLen branch (onnxmodelbuilder.cpp just after alignInputsToConsumptionOrder) will it mean that now the mask is actually referenced first in the graph and thus the new ordering will be bad again? Is this backend vendor doing something stupid and interface-violating that we need to work around Is there a way we can make this less fragile? Presumably it would be worth testing that all input order combinations work - when there is a mask, when there is no mask (require exact nn len), and with and without HumanSL's metadata encoder?

[seniorfish]

It has been confirmed that onnxOpenVINOEnableNPUFastCompile (enable_npu_fast_compile) has been deprecated by ONNX Runtime. This option has no effect, so it has been removed.

[lightvector]

Thanks for the work. Took a first initial pass at review, left comments.

[lightvector]

If we're going to do per-server-thread batch sizes, please verify that this is threaded everywhere needed? For example, maybeWarmupComputeHandle uses batch sizes too and may need to be run with the same batch size as post-warmup if a backend relies on it.

[lightvector]

Is it a bit weird that setting the number of threads also resets all batch sizes, even for the server threads that were configured to demand a particular batch size?

Might deserve at least a comment or some documentation?

[lightvector]

Why don't they benefit from the fp16 range workaround?

Would you be willing to try a selfplay game on large boards like 39x39 or 50x50 using the bigger board sizes compile flag (with low visits so that it actually runs in a reasonable time) and see if it works or if it blows up or something, on each of a few different nets, like a big transformer and a convnet? It will play terribly poorly of course but the point is to check what the extremal behavior actually is. The fp16 range reductions from scale 8 were addressing over-large activations that used to happen on b28 on large boards and even happened on Friday9i's large board finetuned net https://katagotraining.org/extra_networks/ (although less frequently) at large sizes, and that IIRC also extremely rarely happened on 19x19 or barely-larger than 19x19 sizes (23x23, etc) on existing nets. This was very rare and hard to see in testing because it required especially unusual positions, but when it happened and crashed katago it was of course very annoying for users. (Testing on a large board size, even if the net plays horribly, is one way probing how much "headroom" there is).

[lightvector]

What's this thread setting intended for? An ai-assisted pre-review pass for me flags this as potentially killing the performance of the CPU backend for onnx runtime (although I haven't verified this)

[lightvector]

This seems misleading. Can we wire it up so that if the user sets it to false it uses it as false?

Less ideally, if there is no easy way to wire it up, then it should perhaps fail loud if the user requests it as false. It shouldn't be the case that the user requests one thing and it silently gets ignored.

[lightvector]

Tossing over the wall a comment from an AI-assisted pre-review that I'm not an expert on but looks plausible to me that might need some consideration. If you think this is a false positive then feel free to refute/reject it.

Per the current OpenVINO EP docs, device_id isn't an accepted provider option any more — device selection is via device_type itself ("GPU.0", "GPU.1", etc.): https://onnxruntime.ai/docs/execution-providers/OpenVINO-ExecutionProvider.html. So on ORT builds that validate V2 provider options, any config that reaches this line (onnxOpenVINODeviceId, or onnxDeviceToUseThreadN with index > 0) would fail at session creation, and the retry at L411 wouldn't help since device_id isn't in the stripped-key set. Since this only triggers with multiple devices of the same type, it may never have been exercised in your testing. Suggest mapping the index into the device string instead — if threadDeviceType is a bare "GPU"/"NPU" without a .N suffix, emit device_type = "GPU." + idx — and dropping device_id (or keeping onnxOpenVINODeviceId as sugar that appends the suffix). Would be good to confirm with a two-GPU or GPU+iGPU config that per-thread selection actually lands on the intended devices.

[lightvector]

Double check whether protobuf is linked in a portable way here? See TensorRT's protobuf linking for a case that required a fix for windows.

[lightvector]

And see of course the earlier comment I left about alignInputsToConsumptionOrder both code-style-wise and in terms of the potential fragility, if there's a way we can work around it better for the provider that's mis-conforming.

[seniorfish]

@lightvector thanks for the review comments on alignInputsToConsumptionOrder. I ran a dedicated investigation (drafted by an agent, then carefully checked myself), and the full findings with reproduction scripts are at: https://github.com/seniorfish/KataGo/tree/research/onnx-input-order/research/onnx-input-order

In short: this is an input-binding bug inside the ONNX Runtime OpenVINO EP, present since ORT v1.23.0 (introduced by commit dfc27cd7c7e). KataGo feeds inputs by name through the ORT Run API, which is the conforming interface; the mismatch reproduces on our stack (self-built dev ORT 1.29 + OpenVINO 2026.2). So I'll keep the workaround for now, but will address your code-style point by making the ordering unconditional and documenting the requirement at the input-construction site, and I am also considering preparing an upstream bug report to microsoft/onnxruntime with our repro.

[seniorfish]

Implementing per-server-thread batch sizes is relatively complex and requires further verification. plan to treat it as an independent feature and may submit a separate PR for it.

[seniorfish]

@lightvector, Thanks for the initial review — all 7 comments are now addressed:

1. Per-thread max batch sizes — removed entirely to keep this PR focused.
2. scale8 — restored by default, with an opt-out onnxSkipScale8 (default off = scale8 applied). Verified on a 50×50 board with an M2 net, see c2e5552
3. SetIntraOpNumThreads(1) — now scoped to the OpenVINO EP only.
4. useFP16Mode — now fails loudly
5. device_id — dropped; the per-thread device index is now folded into device_type (e.g. "GPU.1"). Verified against the ORT/OpenVINO source (no dual-GPU hardware here to test live).
6. protobuf — linked portably via protobuf::libprotobuf + Protobuf_USE_STATIC_LIBS (same fix as the TensorRT path).

Also merged upstream master, the HumanSL model seems to work fine for me (513 visits/s on Arc B580), and added a CI workflow that builds the ONNX+OpenVINO backend on push and uploads a runnable artifact.

[seniorfish]

I've successfully added support for several execution-providers: OpenVINO, DirectML (~20 v/s on transformer models, outperforming OpenCL), and CPU on Windows, as well as TensorRT (though performance is currently underperforming at ~17 v/s on a 4090) and CPU on Linux.

All the binaries can be found in the artifacts. This demonstrates the great flexibility ONNX Runtime offers in integrating various backends. I'm currently working on adding Android support.

[seniorfish]

Added experimental WebAssembly (Emscripten) support to the ONNX backend to run KataGo's transformer network in the browser. Verified decent execution speeds using a local Go board web demo. Further development is ongoing in the feature branch: https://github.com/seniorfish/KataGo/tree/feature/onnx-wasm

[seniorfish]

I managed to get KataGo running in the browser by combining onnx-wasm with Sabaki. The code has been pushed to the feature/web branch. Please refer to README-web.md for build steps—feedback is welcome!

[lightvector]

@seniorfish thanks for this! This PR has now been merged into #1234 which will go right into master if it seems to be working well. I merged your PR directly and then made a few further refactors involving the onnx backend (mostly in eff9bd7 and 49d8d54), would you be able to take a look and to test that the tip of #1234 is working for you and performs the way you expect and so on?

[Coderchen-001]

Hi @lightvector, thanks for pushing the merge branch and asking me to test! I've been testing the ONNX backend code that's being merged in #1234 across 5 machines. Here's what I found.

Builds tested

• DirectML: self-built from the Add ONNX Runtime backend support (OpenVINO / DirectML / TensorRT / CPU, Windows & Linux) #1222 branch (ONNX backend, onnxProvider=directml, ONNX Runtime 1.24 with DirectML EP, useFP16=auto)
• OpenVINO: prebuilt release onnx-backend-v3 from seniorfish's repo (onnxruntime + OpenVINO 2026.3, onnxProvider=openvino, onnxOpenVINODeviceType=GPU, onnxOpenVINOPrecision=FP16, onnxOpenVINONumStreams=2, onnxTransformerNHWC=true)
• All benchmarks: 19x19 board, 800 visits, auto thread tuning

Machine matrix

Machine	OS	GPU	DirectML	OpenVINO	CPU provider
5080 rig (Ryzen 7 9800X3D)	Win11 25H2	RTX 5080	✅	n/a	—
AMD rig	Win11	RX 6900 XT	✅	n/a	—
i7-1160G7 laptop	Win11	Iris Xe (96EU)	✅ (slow)	✅	❌ hangs
1070 Ti PC (i7-8700K)	Win10 22H2	GTX 1070 Ti	❌ crash	n/a	✅ (slow, works)
1660 PC (i5-9400F)	Win10 22H2	GTX 1660	❌ crash	n/a	✅ (slow, works)

1. DirectML works on Win11 — NVIDIA / AMD / Intel all fine

Recommended threads and visits/s (Elo gain vs baseline at 5s search):

Model	Params	RTX 5080	RX 6900 XT	i7-1160G7
b10c384h6nbttflrs	10.5M transformer	80t, 477.9 (+515)	20t, 103.1 (+288)	20t, 7.35
b10c512h8nbt3tflrs-fson-silu-rsnh	28.5M transformer	64t, 232.7 (+309)	20t, 64.5 (+181)	—
b11c768h12nbt3tflrs-fson-silu	70.4M transformer	20t, 87.4 (+242)	20t, 45.1 (+153)	—
kata1-zhizi-b28c512nbt-muonfd2 (convnet)	72.8M	48t, 170.8 (+329)	24t, 102.6 (+247)	—

Observations:

• Perf ratio 5080:6900XT = 4.6x / 3.6x / 1.9x / 1.7x across the four models — the smaller the model, the bigger NVIDIA's lead (5080 scales up to 128 threads); the convnet gap is the smallest.
• DirectML under-utilizes the GPU: on the 5080 at low thread counts the whole card draws only ~50W with low utilization. Raising numSearchThreads helps, but peak visits/s still look far below what OpenCL achieves on the same hardware.
• On the 6900 XT the optimum is always 20-24 threads; on the 5080 it ranges 20-80. AMD cards seem to hit a thread-scaling ceiling much earlier under DML.
• On Iris Xe, DirectML is still improving at 20 threads (7.35 visits/s) and is ~4.5-6x slower than OpenVINO on the same iGPU.
• NHWC: transformerNHWC=true works well for the transformer models; the zhizi convnet ran with NHWC=false and completed fine.

2. DirectML crashes silently on Win10 22H2 (important)

On both Win10 machines — GTX 1070 Ti with a 2024 driver and GTX 1660 with a 2020 driver — katago exits silently during session creation, right after the ONNX graph is ready but before DirectML execution provider enabled is printed. Windows Event Viewer shows a 100% identical crash signature on both machines:

• Exception code 0xc0000409 (STATUS_STACK_BUFFER_OVERRUN — in practice a __fastfail/abort)
• Faulting module ucrtbase.dll 10.0.19041.3636, offset 0x7286e
• Same katago.exe timestamp on both

Since the exact same binary works on Win11 and the CPU provider works fine on these very same Win10 machines, packaging/model/config/driver are all excluded. It looks like ONNX Runtime's DirectML EP throws (or fast-fails) when D3D12 device creation fails on Win10, and the exception isn't caught → CRT abort with zero error output. Could we wrap DML EP initialization in try/catch and print a readable error (or fall back to another EP) instead of aborting? That would make Win10 compatibility issues much easier to diagnose and would be a nice robustness improvement regardless.

3. OpenVINO on Iris Xe (i7-1160G7) — works well, numStreams matters a lot

Both models completed (exitcode 0, device_type=GPU, FP16, num_streams=2):

Model	3t	5t	6t (rec)	8t	10t	12t
b10c384 (10.5M)	24.35 (base)	14.05 (-223)	31.65 (+63)	25.28 (-42)	30.03 (+0)	11.76 (-372)
zhizi (72.8M convnet)	10.41 (base)	10.83 (-9)	11.92 (+14)	11.62 (-20)	—	12.71 (-34)

Both models recommend 6 threads. Key finding: onnxOpenVINONumStreams=2 is a big win — 6 threads went from 6.60 visits/s (earlier test with numStreams=1) to 31.65, roughly 4.8x. OpenVINO is clearly the right backend on Intel iGPUs.

Minor issues on this machine: genconfig with OpenVINO is flaky (you have to explicitly set the device in the cfg; benchmark correctly depends on the cfg). Also the CPU provider (onnxProvider=cpu) still hangs at the "Testing" stage (0/3 positions, visits/s = -nan) — reproducible on both 10.5M and 70.4M models.

I'm happy to keep following the progress of these backends and provide more testing whenever there are new improvements to try. Just let me know if you'd like any raw logs or additional data.

[seniorfish]

@Coderchen-001 Thanks for the detailed report.

ORT's DirectML execution provider requires DirectML feature level 5.0, which requires DirectML 1.8.0 or above. I also reproduced this locally: running the same build with DirectML.dll 1.7.0 (feature level 4.1, one step below the requirement) dies with the exact same 0xc0000409 fastfail, and the log stops right after "ONNX graph ready", before "DirectML execution provider enabled" — matching your Win10 crash signature. Swapping in DirectML.dll 1.8.0 (feature level 5.0) makes it run fine, confirming 1.8.0 is the hard minimum.

Your deployment has no directml.dll next to onnxruntime.dll (the ort-dml package only ships onnxruntime.dll + onnxruntime_providers_shared.dll), so on Windows 10 the runtime falls back to the OS-inbox copy in C:\Windows\System32, and Windows 10's inbox DirectML is permanently stuck at 1.1.0 (feature level 2.0). So DMLCreateDevice1 returns DXGI_ERROR_UNSUPPORTED, ORT throws, the exception escapes the nn-server thread, and on Windows that becomes a silent fastfail. Windows 11 22H2+ ships DirectML 1.8.0 inbox, and that's why the same binary works on all your Win11 machines.

What to do on the Win10 machines: put a DirectML.dll of version 1.8.0 or above next to onnxruntime.dll (the same folder as katago.exe). The easiest way is to grab the Microsoft.AI.DirectML 1.15.4 package — the .nupkg is a plain zip (direct download) — and extract bin/x64-win/DirectML.dll from it. Windows loads the app-local copy in preference to the one in System32, so this cleanly overrides Windows 10's inbox 1.1.0. After copying it, katago should log DirectML execution provider enabled and run normally. If it still fails after that, the GPU driver is the next suspect: the DirectML feature level is also gated by the DML driver shipped in the GPU driver, and a 2020-era driver (as on your GTX 1660) likely can't reach feature level 5.0 either, so update the GPU driver as well.

I'll add a log line in a follow-up that reports the detected DirectML version and the exact fix, instead of crashing silently — thanks again for the detailed report and for testing across so many machines!

[Coderchen-001]

Hi @seniorfish,

Thanks for the root-cause analysis — it was exactly right. I placed DirectML.dll 1.15.4 next to onnxruntime.dll and now BOTH Win10 machines run DirectML successfully ("DirectML execution provider enabled, device_id=0"; previously both died with 0xc0000409 right after "ONNX graph ready").

Benchmarks (800 visits, 19x19, onnxProvider=directml):

GTX 1070 Ti (Win10, i7-8700K):

• b10c384h6nbttflrs (10.5M): 8T = 48.52 visits/s (recommended, +95 Elo); peak 55.46 @ 20T
• b10c512h8nbt3tflrs-fson-silu-rsnh (28.5M): 10T = 31.79 (+59); peak 31.61 @ 20T
• b11c768h12nbt3tflrs-fson-silu (70.4M): aborted early (15.22 @ 5T)

GTX 1660 Super (Win10, i5-9400F):

• b10c384h6nbttflrs (10.5M): 12T = 44.34 (+135); peak 55.35 @ 24T
• b10c512h8nbt3tflrs-fson-silu-rsnh (28.5M): 10T = 23.87 (+34); peak 28.10 @ 20T
• b11c768h12nbt3tflrs-fson-silu (70.4M): aborted early (14.58 @ 12T)

For reference, the same 10.5M model on the 1070 Ti via the CPU provider was 3.29 visits/s → DirectML is now ~15x faster.

Observations:

1. GPU power draw is low in every DirectML test (5080: ~50W @ 5 search threads; Win10 machines show ~0% utilization in Task Manager — likely the Compute-engine/burst-sampling artifact, since throughput looks right). Overall DirectML underperforms the vendor backends (TensorRT/CUDA, ROCm, OpenVINO) on every GPU I tried.
2. On the 5080, TensorRT 10.16.1 beat 10.9.0 by ~25%, mostly showing up as higher power draw — I suspect the newer TRT cut per-inference launch/scheduling overhead. Same "GPU starvation" as DirectML, just milder.
3. Question: will these new backends get an automatic tuning step (like OpenCL's tune) to optimize kernel params/thread counts on first run? The default-vs-tuned gap is huge on some cards.

Thanks again for the version-detection log line — that will save users a lot of debugging time.

[lightvector]

This PR is now merged to master via the merge of #1234. I was not able to follow the above discussion fully, as there is a lot of back-and-forth detail. If I missed something important, or if there are important further optimizations or tests, please feel free to open a new issue or PR and let me know what I missed.

[lightvector]

Note: I believe I did include seniorfish's directml fixes, so I think that means we should be good on windows 10 too?