[GSD-13038] Regression in 26.09+: dual-GPU workloads eagerly pin a host-RAM mirror of every large device USM allocation on the peer device, exhausting system memory

[blanelegant]

Summary

On a 2× Arc Pro B70 (Battlemage G31) system running PyTorch DDP (one rank per card,
xccl/oneCCL collectives), every large device-USM segment a rank allocates on its own
card is also registered on its file descriptor to the peer card as GTT (GPU-mapped
system memory). On 26.05 these peer registrations are bookkeeping only. On 26.09+
(observed on 26.22) they are eagerly pinned to physical host pages: ~17.7 GiB of
system RAM per rank for an ~18.5 GiB VRAM working set — 35.4 GiB total on a 64 GiB
machine. The kernel then hits order-0 page-allocation failures, the desktop swaps out,
and at production scale the workload dies. Downgrading only intel-compute-runtime
26.22 → 26.05 (nothing else changed) reduces pinned memory for the identical workload
from 25.9 GiB to 0.67 GiB with identical throughput.

Environment

component	version
intel-compute-runtime	26.22.38646.4 (bad) / 26.05.37020.3 (good)
GPUs	2× Intel Arc Pro B70 (Battlemage G31, 8086:e223), 32 GiB VRAM each, PCIe (no XeLink)
kernel / driver	Linux 7.1.3 (CachyOS), xe
level-zero-loader	1.28.6
intel-graphics-compiler	2.36.3
intel-gmmlib	22.10.0
workload stack	PyTorch 2.13.0+xpu, oneCCL 2022.0 (xccl backend), torchrun 2 ranks
system RAM	64 GiB

Reproducer

# gtt_repro.py — torchrun --standalone --nproc-per-node=2 gtt_repro.py
import os, time
import torch
import torch.distributed as dist

rank = int(os.environ["RANK"])
local = int(os.environ["LOCAL_RANK"])
dev = torch.device(f"xpu:{local}")
dist.init_process_group("xccl", device_id=dev)

# 8 GiB of device USM on this rank's card, in 1 GiB segments
held = [torch.empty(1024 ** 3 // 4, dtype=torch.float32, device=dev) for _ in range(8)]
# gradient-bucket-sized collective traffic
bucket = torch.ones(25_000_000, device=dev)
for _ in range(50):
    dist.all_reduce(bucket)
torch.xpu.synchronize(dev)

print(f"rank {rank} pid {os.getpid()} holding 8 GiB on {dev}; sleeping for inspection", flush=True)
time.sleep(120)
dist.barrier()
dist.destroy_process_group()

Inspect while it sleeps:

for pid in <rank pids>; do
  for fd in /proc/$pid/fdinfo/*; do grep -HE 'drm-resident-(gtt|vram0)' $fd; done
done
free -h   # or a kernel with pinned-GPU-memory accounting in /proc/meminfo

Observed per-fd accounting for the reproducer (each rank holds 8 GiB on its own card;
note the ~8.9 GiB GTT on the fd to the peer card, mirror-imaged between ranks):

pid 116400 (rank 0, computes on renderD128)
  fd4 /dev/dri/renderD129: drm-resident-gtt: 8913000 KiB  drm-resident-vram0:  191268 KiB
  fd5 /dev/dri/renderD128: drm-resident-gtt:  642792 KiB  drm-resident-vram0: 9355696 KiB
pid 116401 (rank 1, computes on renderD129)
  fd4 /dev/dri/renderD129: drm-resident-gtt:  638440 KiB  drm-resident-vram0: 9348024 KiB
  fd5 /dev/dri/renderD128: drm-resident-gtt: 9011816 KiB  drm-resident-vram0:  392456 KiB

(The capture above is from 26.05, where the registrations exist but stay unpinned.
On 26.22 the same registrations are backed by pinned host pages — see below.)

Bad vs good, identical real workload (PPO trainer, ~16 GiB VRAM working set per rank)

	26.22	26.05
peer-fd drm-resident-gtt per rank	12.8 GiB	13.3 GiB (bookkeeping only)
pinned host memory, both ranks (kernel GPUActive)	25.9 GiB	0.67 GiB
throughput	~3.0 s/wave	~3.0 s/wave

(GPUActive is the CachyOS kernel's pinned-GPU-page accounting in /proc/meminfo;
on a mainline kernel the same effect is visible as "used" memory in free that no
process RSS accounts for, disappearing when the workload exits.)

At full production scale on 26.22 the pinning reached 35.4 GiB (17.7 GiB per rank),
free showed ~7 GiB available of 64 GiB, the kernel logged order-0 allocation
failures in unrelated processes (kwin_wayland: page allocation failure: order:0),
and the training run died without OOM-killer involvement.

Growth is stepwise and exactly tracks the allocator's large segments: peer-fd GTT grew
in increments of 3,174,400 KiB — precisely the workload's 512×2048×795 fp32 buffer —
and plateaus at ~the VRAM working set once the caching allocator stops creating new
segments. Nothing in the workload allocates host or shared USM of this size; all
tensors are device-resident.

What I ruled out

• USM pooling knobs: NEOReadDebugKeys=1 EnableHostUsmAllocationPool=0 EnableDeviceUsmAllocationPool=0 EnableUsmAllocationPoolManager=0 on 26.22 → peer-fd
  GTT unchanged; additionally the workload then crashes in backward with
  UR_RESULT_ERROR_OUT_OF_RESOURCES (possibly its own issue).
• Workload-side caching: releasing unused allocator segments
  (torch.xpu.empty_cache()) after warmup reclaims only the warmup-transient portion
  (~2 GiB/rank); live segments remain mirrored and pinned.
• VRAM oversubscription/eviction: drm-resident-vram0 ≈ drm-total-vram0
  throughout; VRAM is never oversubscribed (≤ 19 of 32 GiB).

Expected behavior

Cross-device registrations for peer access (if that is what these are) should not
eagerly pin physical host pages proportional to the peer's entire VRAM working set —
26.05's lazy behavior, which performs identically for this workload, demonstrates the
pinning is unnecessary.

Possibly related

• [GSD-13036] 4x Arc Pro B60: multi-GPU DDP training fails with UR_RESULT_ERROR_OUT_OF_RESOURCES at first kernel (large models only; single-GPU and small-model DDP both work) #952 looks like this same defect presenting as a crash at 4-GPU scale. There:
  4× B60, 60 GB RAM, ~11.4 GiB reserved per rank, UR_RESULT_ERROR_OUT_OF_RESOURCES
  at the first large kernel — exactly what eager peer-pinning predicts (each rank's
  working set mirrored per peer exceeds host RAM at 4 ranks; our 2-GPU/64 GiB system
  survives at 35.4 GiB pinned but dies at the system level instead). Their
  discriminating evidence (single-GPU OK, independent concurrent jobs OK, small-model
  DDP OK, large-model DDP fails on 2/3/4 ranks) matches peer-mirror scaling, and their
  USM-pool debug-key attempts not helping matches ours.
• [GSD-12874] [BMG / Arc Pro B70] Is cross-root-port P2P expected? L0 dev-to-dev copy returns 0x70000003; kernel refuses at P2PDMA (host bridge not whitelisted) — and can the capability be made queryable? #935: on these cards (BMG, cross-root-port) L0 dev-to-dev P2P is refused by the
  kernel, so peer access is host-backed — presumably why these peer GTT registrations
  exist at all. The regression here is only that 26.09+ pins them eagerly.
• The 25.40→26.09 window matches the useUsmPoolManager / "l0 device usm growing
  pools" default flip referenced in [GSD-12641] USM device allocation fails with UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY when Level Zero context spans 2 discrete Arc A770 GPUs (regression from 25.40 to 26.09) #916 (also a dual-Arc, UR multi-device-context
  setup), though disabling those pools via debug keys did not change the behavior
  here, so the overlap may be coincidental.

[kgibala]

Hello @blanelegant,

Thank you for submitting this issue. We will move it internally for further isolation and investigation.

However, we'd like to kindly point out that this issue does not fully comply with our Issue Submission Guide. Please note that in the future, issues that do not follow the submission guidelines may need to be rejected - every issue should respect the guide to ensure efficient triage and resolution.

Could you please follow the Issue Submission Guide and, going forward, make sure new issues are created based on it?

We appreciate your understanding and cooperation.

[jwarchul]

Hi @blanelegant,

We are investigating this regression and suspect that "enable defer backing for bmg" might have caused it. Could you please check if running the test with the flags NEOReadDebugKeys=1 EnableDeferBacking=0 resolves the issue on your side?

Thanks for your help!