Enabling P2P reduces bandwidth on Arrow Lake / Z890?

[petertiedemann]

On my systems equipped with 4x 5060TI 16gb GPUs, i noticed that p2pBandwidthLatencyTest indicates that P2P bandwidth is lower than non-P2P. Is that expected, and if so, is it specific to the specific CPU, or determined by chipset etc?

In case it matters, performance is identical with iommu off completely as well as with intel_iommu=on iommu=pt.

Here is an example:

[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5060 Ti, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5060 Ti, pciBusID: 2, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5060 Ti, pciBusID: 4, pciDeviceID: 0, pciDomainID:0
Device: 3, NVIDIA GeForce RTX 5060 Ti, pciBusID: 5, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
Device=0 CAN Access Peer Device=3
Device=1 CAN Access Peer Device=0
Device=1 CAN Access Peer Device=2
Device=1 CAN Access Peer Device=3
Device=2 CAN Access Peer Device=0
Device=2 CAN Access Peer Device=1
Device=2 CAN Access Peer Device=3
Device=3 CAN Access Peer Device=0
Device=3 CAN Access Peer Device=1
Device=3 CAN Access Peer Device=2

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2     3
     0       1     1     1     1
     1       1     1     1     1
     2       1     1     1     1
     3       1     1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3
     0 470.29  12.24  12.13  11.80
     1  12.56 473.35  13.76  13.00
     2  12.49  14.25 473.05  12.94
     3  12.47  14.17  13.03 473.77
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2      3
     0 472.49   5.47   5.48   5.47
     1   5.47 475.08   5.48   5.47
     2   5.47   5.48 473.63   5.47
     3   5.48   5.47   5.48 473.49
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3
     0 470.56  15.23  15.35  15.39
     1  14.24 471.83  17.06  16.29
     2  14.19  19.06 471.26  16.33
     3  14.15  18.13  16.33 471.91
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3
     0 470.06  10.94  10.94  10.94
     1  10.94 472.12  10.94  10.95
     2  10.96  10.94 472.55  10.94
     3  10.94  10.92  10.95 471.83
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2      3
     0   1.49  14.72  14.81  14.72
     1  15.27   0.97  14.48  14.70
     2  14.77  14.41   0.97  14.40
     3  15.22  14.41  14.42   0.97

   CPU     0      1      2      3
     0   1.51   5.38   5.32   5.32
     1   5.23   1.50   5.28   5.25
     2   5.26   5.25   1.52   5.25
     3   5.17   5.24   5.26   1.52
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2      3
     0   0.96   0.49   0.49   0.41
     1   0.48   0.95   0.41   0.47
     2   0.41   0.41   0.97   0.40
     3   0.42   0.41   0.41   0.97

   CPU     0      1      2      3
     0   1.61   1.22   1.22   1.23
     1   1.33   1.51   1.24   1.24
     2   1.29   1.25   1.51   1.28
     3   1.30   1.27   1.27   1.53

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

## System Information Summary

┌────────────────┬─────────────────────────────────────────────────────────────────────────┐
│ Item           │ Detail                                                                  │
├────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ CPU            │ Intel Core Ultra 5 235 (Arrow Lake-HX), 14 cores, 1 socket              │
├────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ Motherboard    │ MSI MEG Z890 UNIFY-X (MS-7E20), BIOS 1.A90                              │
├────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ Chipset        │ Intel Z890 (Arrow Lake-HX / 800 Series PCH)                             │
├────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ Kernel         │ 7.0.0-22-generic (gcc 15.2.0, Ubuntu)                                   │
├────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ NVIDIA Driver  │ 610.43.02 (Open Kernel Module, source-built), CUDA 13.3                 │
├────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ Kernel cmdline │ intel_iommu=on iommu=pt  (also tried off entirely)                      │
├────────────────┼─────────────────────────────────────────────────────────────────────────┤
│ GPUs           │ 4x NVIDIA GeForce RTX 5060 Ti 16GB (GB206, rev a1, device ID 10de:2d04) │
└────────────────┴─────────────────────────────────────────────────────────────────────────┘

## PCIe Topology

CPU Host Bridge [8086:7d2a]
├── 00:01.0 [8086:7ecc] Meteor Lake-H PCIe Root Port #12  ── x4 @ 32GT/s
│   └── 01:00.0  GPU 0  ── x4 @ 32GT/s  (downgraded from x16 capable)
│
├── 00:06.0 [8086:ae4d] Arrow Lake-HX PCIe Root Port #13  ── x8 @ 32GT/s
│   └── 02:00.0  GPU 1  ── x8 @ 32GT/s  (downgraded from x16 capable)
│
├── 00:06.1 [8086:7eca] ── NVMe SSD (03:00.0)
│
├── 00:06.3 [8086:ae4e] Arrow Lake-HX Device             ── x4 @ 32GT/s
│   └── 04:00.0  GPU 2  ── x4 @ 32GT/s  (downgraded from x8 capable)
│
└── 00:06.4 [8086:ae4f] Arrow Lake-HX Device             ── x4 @ 32GT/s
    └── 05:00.0  GPU 3  ── x4 @ 32GT/s  (downgraded from x16 capable)

[badnikhil]

@petertiedemann Short answer: yes, that's expected on this kind of platform and it's not a bug.

With P2P off, cudaMemcpyPeer stages the copy through host RAM (GPU→RAM→GPU), which is a well-optimized, posted-write DMA path. With P2P on, the copy goes GPU→GPU directly, but on a consumer Z890 board that traffic has to route up through the CPU root complex (and across the DMI link to the chipset for the PCH-attached cards). Client root complexes are poor P2P forwarders NVIDIA's own GPUDirect docs note peer-to-peer read bandwidth is "severely limited" through the CPU. Add your reduced link widths (x4/x8) and the shared DMI uplink, and direct P2P ends up slower than host staging for bulk transfers. The latency improvement you saw is the real P2P benefit on this hardware; bandwidth isn't.

So: for throughput, you're better off leaving P2P disabled here. Good P2P bandwidth needs full x16 CPU-direct lanes (Threadripper Pro / EPYC) or a board with a PCIe switch.

[ikirsh]

I am experiencing the same issue with a dual RTX 5060 Ti 16GB setup on an MSI PRO Z890P motherboard paired with an Intel Core Ultra 265K (Arrow Lake) processor.

Below are the results from the connectivity and bandwidth matrix:

P2P Connectivity Matrix
     D\D      0      1
     0        1      1
     1        1      1

Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D      0      1 
     0 385.94  13.36 
     1  14.22 388.49 

Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D      0      1 
     0 354.63   5.47 
     1   5.47 388.30 

Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D      0      1 
     0 368.82  16.48 
     1  18.23 387.95 

Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D      0      1 
     0 369.13  10.92 
     1  10.94 387.09 

P2P=Disabled Latency Matrix (us)
   GPU      0      1 
     0   1.01  14.36 
     1  14.32   1.04 

   CPU      0      1 
     0   1.50   5.31 
     1   5.32   1.53 

P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU      0      1 
     0   0.99   0.47 
     1   0.49   1.04 

   CPU      0      1 
     0   1.54   1.25 
     1   1.27   1.52 

Given these metrics, I have two questions:

1. Does AMD offer better P2P bandwidth performance than Intel in this context?
2. Are there any cost-effective CPU and motherboard combinations that avoid these P2P bandwidth bottlenecks?