Backends to merge

[lightvector]

Candidate branch to merge to master. Combines #1222 ONNX backend from @seniorfish and #1188 ROCm backend from @Looong01 along with a bunch of followup fixes and refactors I added on top of each backend from local review and testing and a few other features I also worked on recently.

Will go into master if everyone confirms that it's working well.

[seniorfish]

Looks good to me. I pulled the PR on Windows 11 and successfully built and verified the CPU(ONNX), DirectML, and OpenVINO targets.

To test correctness, I used a slightly modified runnonmanyposestest (adjusted so it could measure ONNX backends as well as human/metaEncoder models) across 254 positions. The test covered b11, humanv0, b18, b28, and b40.

The runnonmanyposestest results look good:

1. FP32 backends (CPU and DirectML) match the Eigen CPU down to machine precision, with policy squared error around 1e-10 (approx 1.5e-8 average error per point).
2. OpenVINO FP16 showed minimal noise with policy squared error between 1e-4 and 8e-4, which is noticeably better than previous OpenCL FP16 or MIGraphX FP16 baselines.

I also verified ownership correctness using a custom script calling the kata-raw-nn command across 50 random positions for all models and backends, and all ownership outputs passed well within tolerance(cpu/directml ~1e-6 and openvino-fp16 0.0014–0.0029).

I haven't tested Linux builds yet.

Thanks for working on these refactors!

[Looong01]

@lightvector Thanks for our working!

I just do a upgrade for v1.17.2. Just including a very little change: 86c7aaa
Maybe u can have a look!

And

I am sure everything is OK to merge! My test result & report as following:

Tested the tip of #1234 (66b12ac) on my machine. ROCm backend builds and runs.

Environment: RX 7900 XTX (gfx1100), ROCm 7.14, MIOpen 3.5.2, Composable Kernel 1.2.0, clang 23, Linux x86-64.

Build: Clean configure + full build (-DUSE_BACKEND=ROCM -DCMAKE_BUILD_TYPE=Release, 25 archs, make -j8). The ROCm prefix resolution correctly picks /opt/rocm/core-7.14, CK FMHA fused attention is enabled, and RPATH is set correctly. katago version reports v1.17.2 + 66b12ac with the ROCm backend.

Functionality: All four models load and play via GTP (genmove returns sane moves): b10c384h6nbttflrs, b10c512h8nbt3tflrs-fson-silu-rsnh, b11c768h12nbt3tflrs-fson-silu, kata1-zhizi-b40c768nbt-s11272M-d5935M.

Benchmark (peak visits/s, back-to-back on the same GPU, vs my AMD_GPU branch at v1.17.2 which is what dc7dd78 was squashed from):

Model	My branch	#1234 tip	Delta
b10c384h6nbttflrs	2490	2649	+6.4%
b10c512h8nbt3tflrs-fson-silu-rsnh	1396	1506	+7.9%
kata1-zhizi-b40c768nbt-s11272M-d5935M	336	354	+5.4%
b11c768h12nbt3tflrs-fson-silu	927	780	-15.8%

The first three are faster than before, which is great. However, the largest transformer (b11c768h12, also the flagship model of the three new ones) shows a reproducible ~15% regression (reproduced across two separate runs, so not thermal drift).

Some isolation I did on the b11 regression:

• rocmInputsUseNHWC=false gives the same 782 v/s, so the inputsUseNHWC default flip for ROCm in c002c97 is not the cause.
• With rocmDisableFusedAttention=true: my branch drops to 576, Backends to merge #1234 drops to 493. So the unified backend is ~15% slower for this model on BOTH the plain attention path and the CK FMHA path (my branch 576 -> 927 with CK on, Backends to merge #1234 493 -> 780 with CK on). The regression is not in the CK dispatch but somewhere in the shared trunk path (conv/matmul/buffer handling or MIOpen solution selection) that only hurts the 768-channel pure transformer trunk — the 384/512-channel transformers and the 768-channel hybrid b40 are all unaffected (and faster).

But even with this performance regression, overall it looks good to me and I think it's worth merging.

[seniorfish]

• seniorfish@aadfa4b logs DirectML init failure cause and fix guidance
• seniorfish@976b6e9 ships DirectML.dll in DirectML release artifacts

Should fix #1222 (comment) DirectML issues.

[foxrainowo]

Can you provide a compiled file for me to test? I have tested #1222 before. The NPU backend have some bugs, such as failing to load the b40 networks, missing some cfg files.

And, I have doubts about why setting the NPU_nnMaxBatchSizeThread more than 1 in gtp_gpu_npu_example.cfg?. As far as I know, NPU is only suitable in 1 thread. Also, it is too fancy that there are many precision can be adjust in gtp_gpu_npu_example.cfg. Can you make it more formal and consistent?

PS C:\Users\foxrain> C:\Users\foxrain\Downloads\katago-onnx-openvino-v4\katago.exe benchmark -model C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11272M-d5935M.bin.gz -config C:\Users\foxrain\Downloads\katago-onnx-openvino-v4\gtp_example.cfg -t 1
2026-08-12 12:29:39+0800: Running with following config:
allowResignation = true
lagBuffer = 1.0
logAllGTPCommunication = true
logDir = gtp_logs
logSearchInfo = true
logSearchInfoForChosenMove = false
logToStderr = false
maxTimePondering = 60.0
maxVisits = 500
numSearchThreads = 6
onnxOpenVINODeviceType = NPU
onnxProvider = openvino
ponderingEnabled = false
resignConsecTurns = 3
resignThreshold = -0.90
rules = tromp-taylor
searchFactorAfterOnePass = 0.50
searchFactorAfterTwoPass = 0.25
searchFactorWhenWinning = 0.40
searchFactorWhenWinningThreshold = 0.95

2026-08-12 12:29:39+0800: Loading model and initializing benchmark...
2026-08-12 12:29:39+0800: Testing with default positions for board size: 19
2026-08-12 12:29:39+0800: nnRandSeed0 = 4822533320517784866
2026-08-12 12:29:39+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11272M-d5935M.bin.gz useFP16 auto
2026-08-12 12:29:39+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-12 12:29:57+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-12 12:29:57+0800: ONNX backend thread 0: Model version 15
2026-08-12 12:29:57+0800: ONNX backend thread 0: Model name: kata1-zhizi-b40c768nbt-fdx6d-s11272M-d5935M (nbt convnet, 232532105 params)
2026-08-12 12:29:57+0800: ONNX backend thread 0: provider=openvino deviceIdx=NPU
2026-08-12 12:29:57+0800: ONNX backend: building ONNX graph from model weights...
2026-08-12 12:29:57+0800: Building internal onnx model, requireExactNNLen=true transformerNHWC=false
2026-08-12 12:29:59+0800: ONNX backend: ONNX graph built (930538868 bytes)
2026-08-12 12:29:59+0800: ONNX backend: OpenVINO EP enabled for thread 0, device_type=NPU
2026-08-12 12:30:04+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-12 12:30:04+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-12 12:30:04+0800: ONNX backend: session created, inputs=3 outputs=5
PS C:\Users\foxrain>

[seniorfish]

Thanks for the detailed feedback. I cannot reproduce the b40 load failure you hit, b11, b18, b28, b40 works on both GPU and NPU on this branch.
I admit the previous config file options were too confusing, so I streamlined them, removed unnecessary options, set more reasonable defaults, and now the NPU batch is fixed at 1. I also added a warmup phase to improve performance.

commits:

e01e4a1 — KataGo's search varies its NN batch size with the number of concurrent evaluations, and the OpenVINO EP compiled device code per batch size on first use, stalling the first searches by seconds. Each server thread now runs a throwaway forward at startup for every batch size it will use (NPU: 1; other devices: up to 4). On a Core Ultra 285H, 2-thread GPU throughput went from 16.9 to ~122 visits/s with no stalls.

ae7e731 — A batch of 2 on the NPU triggered a whole-graph recompile of 25-45 s on its first occurrence, since the EP forces static shapes per batch size there. NPU compute handle now split any batch above 1 into per-row batch-1 forwards, which costs nothing on this single-stream device and keeps the compiled graph stable. The transformer model at 2 threads went from 1.87 to 4.35 visits/s with no stalls (mixed GPU+NPU at 4 threads: 9.3 to 45.2).

c025a30 — The OpenVINO section had accumulated options with little value for a single-session synchronous backend, plus a per-device override layer that made the config hard to follow. onnxOpenVINONumOfThreads, onnxOpenVINOModelPriority, and the entire onnxOpenVINODeviceConfig_* layer are removed, and the bare CPU device is rejected in favor of onnxProvider = cpu. The OpenVINO config is now seven flat keys, with onnxSkipScale8 = true documented as a ~2x (GPU) / ~4x (NPU) lever.

Latest builds and config files for gpu, npu and both: https://github.com/seniorfish/KataGo/releases/tag/v1.17.2-onnx-openvino

[lightvector]

This PR is now merged. Thanks for all the help and testing and contributions everyone! Please submit any followup PRs or things I might have missed in the discussion as as PRs against the new tip.

@seniorfish I included most of your followup commits but I omitted the two involving warmup for the moment, and NPU splitting (which I think was codewise entangled with / conflicting if warmup was omitted) . Can you see to what degree this is possible using the same warmup mechanism as was implemented already for the CUDA backend, rather than writing a new warmup mechanism locally? For example maybe adding a NeuralNet::getWarmupBatchSizes(...,maxBatchSize) to nninterface.h so that the backend can report what warmup should be done at what batch sizes, returning empty if the backend doesn't need warmup. Additionally, does the NPU branch really have no benefit for batch size > 1? For example if we let it build its graph for batch size 2 as well, would we get better performance from splitting batches into 2s and then a 1 if odd, instead of splitting all into 1s?

[foxrainowo]

@seniorfish

On the NPU, for b11c768h12nbt3, onnxTransformerNHWC=false provides a 3× speedup; for b28c512nbt and b18c384nbt, onnxSkipScale8=true provides a 4× speedup.

On the GPU, for b11c768h12nbt3, onnxTransformerNHWC=true provides a 3× speedup; for b40c768nbt, b28c512nbt, and b18c384nbt, onnxSkipScale8=true provides a 2× speedup. The performance starts to decline with 15 threads or more, and the speed with 20 threads is below 5 v/s.

I used https://github.com/seniorfish/KataGo/releases/tag/v1.17.2-onnx-openvino to run the tests, but it still does not include the configuration file (match, human).

katago-onnx-openvino-c025a30	b11c768h12nbt3	b40c768nbt	b28c512nbt	b18c384nbt	b40c256
NPU_t1_onnxSkipScale8=false_onnxTransformerNHWC=false	34	---	32	61	276
NPU_t1_onnxSkipScale8=false_onnxTransformerNHWC=true	10	---	32	59	266
NPU_t1_onnxSkipScale8=true_onnxTransformerNHWC=false	35	---	141	253	249
NPU_t1_onnxSkipScale8=true_onnxTransformerNHWC=true	10	---	135	244	267
GPU_onnxSkipScale8=false_onnxTransformerNHWC=false	26	30	90	182	334
GPU_onnxSkipScale8=false_onnxTransformerNHWC=true	105	33	88	189	320
GPU_onnxSkipScale8=true_onnxTransformerNHWC=false	29	56	169	427	330
GPU_onnxSkipScale8=true_onnxTransformerNHWC=true	100	56	165	431	321

I tried adjusting the parameters, but b40c768nbt still does not work properly on the NPU backend. I’m guessing it’s because it’s too large? Has anyone figured this out?

PS C:\Users\foxrain> C:\Users\foxrain\Downloads\katago-onnx-openvino-c025a30\katago.exe benchmark -model C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11472M-d5982M.bin.gz -config C:\Users\foxrain\Downloads\katago-onnx-openvino-c025a30\gtp_example.cfg -t 1
2026-08-14 12:57:10+0800: Running with following config:
allowResignation = true
lagBuffer = 1.0
logAllGTPCommunication = true
logDir = gtp_logs
logSearchInfo = true
logSearchInfoForChosenMove = false
logToStderr = false
maxTimePondering = 60.0
maxVisits = 500
numSearchThreads = 1
onnxDeviceToUse = 0
onnxOpenVINOCacheDir = C://temp//katago_ov_cache
onnxOpenVINODeviceType = NPU
onnxOpenVINOPrecision = FP16
onnxProvider = openvino
onnxSkipScale8 = false
onnxTransformerNHWC = false
ponderingEnabled = false
resignConsecTurns = 3
resignThreshold = -0.90
rules = tromp-taylor
searchFactorAfterOnePass = 0.50
searchFactorAfterTwoPass = 0.25
searchFactorWhenWinning = 0.40
searchFactorWhenWinningThreshold = 0.95

2026-08-14 12:57:10+0800: Loading model and initializing benchmark...
2026-08-14 12:57:10+0800: Testing with default positions for board size: 19
2026-08-14 12:57:10+0800: nnRandSeed0 = 9550974731077147000
2026-08-14 12:57:10+0800: After dedups: nnModelFile0 = C:\Users\foxrain\Downloads\kata1-zhizi-b40c768nbt-s11472M-d5982M.bin.gz useFP16 auto
2026-08-14 12:57:10+0800: Initializing neural net buffer to be size 19 * 19 exactly
2026-08-14 12:57:24+0800: ONNX backend: creating compute context for 19x19 with provider 'openvino'
2026-08-14 12:57:24+0800: ONNX backend thread 0: Model version 15
2026-08-14 12:57:24+0800: ONNX backend thread 0: Model name: kata1-zhizi-b40c768nbt-s11472M-d5982M (nbt convnet, 232532105 params)
2026-08-14 12:57:24+0800: ONNX backend thread 0: provider=openvino deviceIdx=NPU
2026-08-14 12:57:24+0800: ONNX backend: building ONNX graph from model weights...
2026-08-14 12:57:24+0800: Building internal onnx model, requireExactNNLen=true transformerNHWC=false
2026-08-14 12:57:25+0800: ONNX backend: ONNX graph ready (930619286 bytes)
2026-08-14 12:57:25+0800: ONNX backend: OpenVINO execution provider enabled for thread 0, device_type=NPU, cache_dir=C://temp//katago_ov_cache, precision=FP16
2026-08-14 12:57:28+0800: ONNX backend: graph input order: [0]InputSpatial [1]InputGlobal [2]InputMask
2026-08-14 12:57:28+0800: ONNX backend: graph output order: [0]OutputPolicyPass [1]OutputPolicy [2]OutputValue [3]OutputScoreValue [4]OutputOwnership
2026-08-14 12:57:28+0800: ONNX backend: session created, inputs=3 outputs=5
2026-08-14 12:57:28+0800: ONNX backend: OpenVINO warmup for thread 0: precompiling batch sizes 1..1 on device_type=NPU (cache_dir=C://temp//katago_ov_cache, compiled graphs are reused by later runs)
PS C:\Users\foxrain>