ggml-hip: make -funsafe-math-optimizations opt-in (GGML_HIP_UNSAFE_MATH, default OFF)

[jimw567]

Overview

• Gate the HIP -funsafe-math-optimizations compile flag behind a new CMake option
  GGML_HIP_UNSAFE_MATH, defaulting OFF, so HIP builds are IEEE-conformant by
  default. Opt back into the fast-math speedup with -DGGML_HIP_UNSAFE_MATH=ON.
• Add a gfx1151-only CI check (gpu-rocm self-hosted job) asserting that MTP
  speculative decoding stays byte-identical to the non-speculative baseline at temp 0.

Motivation

Speculative decoding (incl. MTP) is a lossless technique: at temperature 0 its
non-associativity." -funsafe-math-optimizations enables -fassociative-math, which
reassociates FP reductions. On RDNA3.5 (gfx1151) this widens the numerical noise
enough to flip near-tie greedy argmax decisions, so MTP output diverges from a
baseline that was previously bit-identical (AIESW-40114).

For determinism-sensitive paths (MTP identity, reproducible greedy output) the
correct default is not to reorder FP ops. The flag stays available for users who want
the speed and accept reduced determinism. Note upstream #25495 already removed the
stronger -ffast-math; this addresses the remaining -funsafe-math-optimizations.

Evidence (gfx1151 / Qwen3.5-4B-Q4_0 MTP, temp 0)

Baseline --spec-type none vs --spec-type draft-mtp --spec-draft-n-max 3, compared
byte-for-byte:

Build	-funsafe-math	MTP vs baseline	CI check
stock upstream (unsafe-math on)	present	DIVERGED	would FAIL (exit 20)
this PR, default	absent	IDENTICAL	PASS (exit 0)
this PR, -DGGML_HIP_UNSAFE_MATH=ON	present	DIVERGED	FAIL (exit 20)

Both the fix and the test's negative control were validated on hardware: the test
goes green on the fixed build and red the moment fast math is re-enabled.

Changes

• ggml/CMakeLists.txt: add option(GGML_HIP_UNSAFE_MATH ... OFF).
• ggml/src/ggml-hip/CMakeLists.txt: wrap -funsafe-math-optimizations in
  if (GGML_HIP_UNSAFE_MATH).
• ci/run.sh: add gg_run_mtp_greedy, gated to GG_BUILD_ROCM. Downloads
  unsloth/Qwen3.5-4B-MTP (NextN head built in), drives baseline vs draft-mtp via
  llama-server at temp 0, and fails if the greedy outputs differ.

Test plan

• Default build: -funsafe-math-optimizations absent; MTP greedy == baseline (IDENTICAL)
• -DGGML_HIP_UNSAFE_MATH=ON: flag present; MTP greedy != baseline (DIVERGED) -> CI check exits nonzero
• gpu-rocm job: gg_run_mtp_greedy runs and passes on the gfx1151 runner

Additional information

#26611

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES. AI helped bisect recent commits and located the PR that introduced divergence. It also helped A/B tested the fix.

[ggerganov]

Avoid writing overly-verbose PR descriptions and comments with AI. This is against the AI policy of the project and does not leave a good impression.

Regarding the discussion about unsafe math - in theory, it is correct that spec decoding should give identical results to non-spec decoding. However, in practice this is never the case, regardless if unsafe math is used or not. The main source of discrepancy in this case is due to the tokens being processed in different batch sizes when spec decoding is on/off. This leads to different kernels being used, or different FP accumulation depending on how the data gets arranged in the memory.

The bottom line is that this test will sometimes fail even if unsafe math is disabled.

[ggerganov]

Avoid writing such long comments - they are not maintainable. Take a look at the rest of the script to see how comments should look like.

[jimw567]

Thanks for pointing this out. I will rewrite.

[jimw567]

I moved the test to test_speculative.py and shorten the comments.

[ggerganov]

This is a very big model - it is not suitable for CI.

[jimw567]

Ok. Let me see if I can find a smaller unit test for CI.

[jimw567]

This 4B model is the smallest MTP model that I can find to reliably guard this contract that my end users care about. I have put measures into the CI pipeline to help the CI runtime.

• Model download is cached. The first test on a fresh rocm runner will add 105s overhead with 78s to download the model. The subsequent PR runs will only incur 27s overhead.
• This test is only enabled for gpu-rocm. If the additional 27s overhead is too big, I can help add additional ROCm runners.

[ggerganov]

The ci/run.sh script does not contain server tests. The server tests should go in tools/server/tests.

[jimw567]

Ok. I will move it.

[jimw567]

I moved the test to test_speculative.py

[jimw567]

Avoid writing overly-verbose PR descriptions and comments with AI. This is against the AI policy of the project and does not leave a good impression.

Regarding the discussion about unsafe math - in theory, it is correct that spec decoding should give identical results to non-spec decoding. However, in practice this is never the case, regardless if unsafe math is used or not. The main source of discrepancy in this case is due to the tokens being processed in different batch sizes when spec decoding is on/off. This leads to different kernels being used, or different FP accumulation depending on how the data gets arranged in the memory.

The bottom line is that this test will sometimes fail even if unsafe math is disabled.

Points on AI usage are well taken. I will follow the project AI policy going forward.

You are right that "the main source of discrepancy in this case is due to the tokens being processed in different batch sizes when spec decoding is on/off.". However, for the models that we have been tracking (Qwen3.5 4B/9B Qwen3.6 27B/35B) in different modes, we hadn't seen any divergence in our daily regression for 2 months until #23962 and now #25495.

I will keep a close eye on this new test and will deal with failures as they show up.

[ggerganov]

This greedy sampling test failing is indeed a bit suspicious. Normally, for this case I would expect the top token to be quite ahead in terms of probability, so I don't see how unsafe math optimizations can make such a big perturbation to shift that and select another token. There might be something else going on beyond FP reduction order.

It would be nice to have a similar test in the server test suite, but the main issue that I am not sure how to address is that this is a big model. The current models that we use are intentionally tiny in order to keep the CI resources low:

llama.cpp/tools/server/tests/utils.py

Lines 505 to 637 in 9a31e1d

	class ServerPreset:
	@staticmethod
	def load_all() -> None:
	""" Load all server presets to ensure model files are cached. """
	servers: List[ServerProcess] = [
	method()
	for name, method in ServerPreset.__dict__.items()
	if callable(method) and name != "load_all"
	]
	for server in servers:
	server.offline = False
	server.start()
	server.stop()
	@staticmethod
	def tinyllama2() -> ServerProcess:
	server = ServerProcess()
	server.offline = True # will be downloaded by load_all()
	server.model_hf_repo = "ggml-org/test-model-stories260K"
	server.model_hf_file = None
	server.model_alias = "tinyllama-2"
	server.n_ctx = 512
	server.n_batch = 32
	server.n_slots = 2
	server.n_predict = 64
	server.seed = 42
	return server
	@staticmethod
	def bert_bge_small() -> ServerProcess:
	server = ServerProcess()
	server.offline = True # will be downloaded by load_all()
	server.model_hf_repo = "ggml-org/models"
	server.model_hf_file = "bert-bge-small/ggml-model-f16.gguf"
	server.model_alias = "bert-bge-small"
	server.n_ctx = 512
	server.n_batch = 128
	server.n_ubatch = 128
	server.n_slots = 2
	server.seed = 42
	server.server_embeddings = True
	return server
	@staticmethod
	def bert_bge_small_with_fa() -> ServerProcess:
	server = ServerProcess()
	server.offline = True # will be downloaded by load_all()
	server.model_hf_repo = "ggml-org/models"
	server.model_hf_file = "bert-bge-small/ggml-model-f16.gguf"
	server.model_alias = "bert-bge-small"
	server.n_ctx = 1024
	server.n_batch = 300
	server.n_ubatch = 300
	server.n_slots = 2
	server.fa = "on"
	server.seed = 42
	server.server_embeddings = True
	return server
	@staticmethod
	def tinyllama_infill() -> ServerProcess:
	server = ServerProcess()
	server.offline = True # will be downloaded by load_all()
	server.model_hf_repo = "ggml-org/test-model-stories260K-infill"
	server.model_hf_file = None
	server.model_alias = "tinyllama-infill"
	server.n_ctx = 2048
	server.n_batch = 1024
	server.n_slots = 1
	server.n_predict = 64
	server.temperature = 0.0
	server.seed = 42
	return server
	@staticmethod
	def stories15m_moe() -> ServerProcess:
	server = ServerProcess()
	server.offline = True # will be downloaded by load_all()
	server.model_hf_repo = "ggml-org/stories15M_MOE"
	server.model_hf_file = "stories15M_MOE-F16.gguf"
	server.model_alias = "stories15m-moe"
	server.n_ctx = 2048
	server.n_batch = 1024
	server.n_slots = 1
	server.n_predict = 64
	server.temperature = 0.0
	server.seed = 42
	return server
	@staticmethod
	def jina_reranker_tiny() -> ServerProcess:
	server = ServerProcess()
	server.offline = True # will be downloaded by load_all()
	server.model_hf_repo = "ggml-org/models"
	server.model_hf_file = "jina-reranker-v1-tiny-en/ggml-model-f16.gguf"
	server.model_alias = "jina-reranker"
	server.n_ctx = 512
	server.n_batch = 512
	server.n_slots = 1
	server.seed = 42
	server.server_reranking = True
	return server
	@staticmethod
	def tinygemma3() -> ServerProcess:
	server = ServerProcess()
	server.offline = True # will be downloaded by load_all()
	# mmproj is already provided by HF registry API
	server.model_hf_file = None
	server.model_hf_repo = "ggml-org/tinygemma3-GGUF:Q8_0"
	server.model_alias = "tinygemma3"
	server.n_ctx = 1024
	server.n_batch = 32
	server.n_slots = 2
	server.n_predict = 4
	server.seed = 42
	return server
	@staticmethod
	def router() -> ServerProcess:
	server = ServerProcess()
	server.offline = True # will be downloaded by load_all()
	# router server has no models
	server.model_file = None
	server.model_alias = None
	server.model_hf_repo = None
	server.model_hf_file = None
	server.n_ctx = 1024
	server.n_batch = 16
	server.n_slots = 1
	server.n_predict = 16
	server.seed = 42
	return server

In that category, a 4B model (even if quantized) is quite large. So we have to figure out some alternative.

[ggerganov]

Correction - we already have a speculative test here:

llama.cpp/tools/server/tests/unit/test_speculative.py

Lines 26 to 54 in 6de1b63

	def test_with_and_without_draft():
	global server
	server.model_draft = None # disable draft model
	server.spec_type = None
	server.start()
	res = server.make_request("POST", "/completion", data={
	"prompt": "I believe the meaning of life is",
	"temperature": 0.0,
	"top_k": 1,
	"n_predict": 16,
	})
	assert res.status_code == 200
	content_no_draft = res.body["content"]
	server.stop()
	# create new server with draft model
	create_server()
	server.start()
	res = server.make_request("POST", "/completion", data={
	"prompt": "I believe the meaning of life is",
	"temperature": 0.0,
	"top_k": 1,
	"n_predict": 16,
	})
	assert res.status_code == 200
	assert res.body["timings"]["draft_n"] > 0
	content_draft = res.body["content"]
	assert content_no_draft == content_draft

So you can try to start from it. If I am not mistaken, it's quite similar to the case that you want to add.

[IMbackK]

Since its causing correctness issues we should not offer the option at all until its fixed

[robegan21]

On my machine the effects of -funsafe-math-optmizations means that many Q8 model degenerate almost immediately. I agree that this option should be default Off and/or removed until stability can be proved.
I erroneously created this largely redundant (and now closed) PR #26870 which just removed that option from the ROCm build and also included a unit test to ensure no build will include unsafe operations which affect the quality of the response going forward.

[jimw567]

gpu-rocm CI failed: mtp_greedy

Automatically attached because the self-hosted ROCm runner cannot upload logs to Azure blob storage (egress firewalled).

• run: https://github.com/ggml-org/llama.cpp/actions/runs/31464300547
• log: mtp_greedy-pytest.log (tail)

error log tail

+ cd tools/server/tests
+ GG_MTP_GREEDY=1
+ LLAMA_SERVER_BIN_PATH=/scratch/actions-runner/_work/llama.cpp/llama.cpp/build-ci-release/bin/llama-server
+ N_GPU_LAYERS=99
+ python3 -m pytest -v -x unit/test_speculative.py::test_mtp_greedy_matches_baseline
============================= test session starts ==============================
platform linux -- Python 3.14.4, pytest-8.3.5, pluggy-1.6.0 -- /scratch/actions-runner/mnt/llama.cpp/venv/bin/python3
cachedir: .pytest_cache
rootdir: /scratch/actions-runner/_work/llama.cpp/llama.cpp/tools/server/tests
configfile: pytest.ini
plugins: anyio-4.14.2
collecting ... collected 1 item

unit/test_speculative.py::test_mtp_greedy_matches_baseline ERROR         [100%]

==================================== ERRORS ====================================
______________ ERROR at setup of test_mtp_greedy_matches_baseline ______________

    @pytest.fixture(autouse=True)
    def fixture_create_server():
>       return create_server()

unit/test_speculative.py:24: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
unit/test_speculative.py:15: in create_server
    server.model_draft = download_file(MODEL_DRAFT_FILE_URL)
utils.py:704: in download_file
    wget.download(url, out=output_file)
../../../../../../mnt/llama.cpp/venv/lib/python3.14/site-packages/wget.py:506: in download
    (fd, tmpfile) = tempfile.mkstemp(".tmp", prefix=prefix, dir=".")
/usr/lib/python3.14/tempfile.py:354: in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags, output_type)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

dir = '/scratch/actions-runner/_work/llama.cpp/llama.cpp/tools/server/tests'
pre = './tmp/stories15M-q4_0.gguf', suf = '.tmp', flags = 131266
output_type = <class 'str'>

    def _mkstemp_inner(dir, pre, suf, flags, output_type):
        """Code common to mkstemp, TemporaryFile, and NamedTemporaryFile."""
    
        dir = _os.path.abspath(dir)
        names = _get_candidate_names()
        if output_type is bytes:
            names = map(_os.fsencode, names)
    
        for seq in range(TMP_MAX):
            name = next(names)
            file = _os.path.join(dir, pre + name + suf)
            _sys.audit("tempfile.mkstemp", file)
            try:
>               fd = _os.open(file, flags, 0o600)
E               FileNotFoundError: [Errno 2] No such file or directory: '/scratch/actions-runner/_work/llama.cpp/llama.cpp/tools/server/tests/./tmp/stories15M-q4_0.ggufpxaj7u4g.tmp'

/usr/lib/python3.14/tempfile.py:255: FileNotFoundError
---------------------------- Captured stdout setup -----------------------------
Downloading https://huggingface.co/ggml-org/tiny-llamas/resolve/main/stories15M-q4_0.gguf to ./tmp/stories15M-q4_0.gguf
=========================== short test summary info ============================
ERROR unit/test_speculative.py::test_mtp_greedy_matches_baseline - FileNotFoundError: [Errno 2] No such file or directory: '/scratch/actions-runner/_work/llama.cpp/llama.cpp/tools/server/tests/./tmp/stories15M-q4_0.ggufpxaj7u4g.tmp'
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
=============================== 1 error in 0.03s ===============================

[jimw567]

@ggml-org/ci Can someone please help review this PR? Thanks!

[IMbackK]

Lets just remove the option and -funsafe-math-optimizations entirely until we are sure there is no real issue her, otherwise we will have users turning it back on and filing issues.

[IMbackK]

lets just remove the option for now (if we want the to spend the ci resources for the test is something @ggerganov or @CISC must decide)

I assigned myself #26982 to check what kernel is causing the difference.

[jimw567]

lets just remove the option for now (if we want the to spend the ci resources for the test is something @ggerganov or @CISC must decide)

I assigned myself #26982 to check what kernel is causing the difference.

@IMbackK Thanks for reviewing! Sure. Let me just remove the option entirely so the accuracy is preserved. I will roll the CI changes into a different PR.

[jimw567]

@IMbackK @ggerganov I reduced the PR to just one line removal. Please help review again.