Error on P2P on 2 RTX 5090: mapping of buffer object failed or code=205(cudaErrorMapBufferObjectFailed) "cudaDeviceEnablePeerAccess(gpuid[1], 0)

[Panchovix]

NVIDIA Open GPU Kernel Modules Version

570.133, 570.144, 570.148, 570.153

Operating System and Version

Fedora 41

Hardware

RTX 5090x2+RTX4090x2+RTX3090x2+A6000

AMD Ryzen 7 7800X3D

192GB RAM

MSI Carbon X670E

Kernel Release

6.14.9

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Build Command

Terminal output/Build Log

More Info

Hi there. I'm trying to use P2P on two RTX 5090, but I get the next issues:

pancho@fedora:~/cuda-samples/build/Samples/5_Domain_Specific/p2pBandwidthLatencyTest$ export CUDA_VISIBLE_DEVICES=2,3
pancho@fedora:~/cuda-samples/build/Samples/5_Domain_Specific/p2pBandwidthLatencyTest$ ./p2pBandwidthLatencyTest 
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 5090, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 5090, pciBusID: 3, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1
     0       1     1
     1       1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 1728.49  24.63 
     1  24.70 1761.56 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
Cuda failure /home/pancho/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu:192: 'mapping of buffer object failed'
pancho@fedora:~/cuda-samples/build/Samples/0_Introduction/simpleP2P$ ./simpleP2P 
[./simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 2

Checking GPU(s) for support of peer to peer memory access...
> Peer access from NVIDIA GeForce RTX 5090 (GPU0) -> NVIDIA GeForce RTX 5090 (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU1) -> NVIDIA GeForce RTX 5090 (GPU0) : Yes
Enabling peer access between GPU0 and GPU1...
CUDA error at /home/pancho/cuda-samples/Samples/0_Introduction/simpleP2P/simpleP2P.cu:130 code=205(cudaErrorMapBufferObjectFailed) "cudaDeviceEnablePeerAccess(gpuid[1], 0)" 

I have other GPUs on my system (A6000, 3090s, 4090s) and P2P works fine on these cards.

I did try with the patch mentioned here #29 (comment), but as I mentioned on #29 (comment), it doesn't seem to work. I edited the files mentioned but there was no difference.

I have also tried https://github.com/tinygrad/open-gpu-kernel-modules/tree/570.148.08-p2p branch, but issue is still there.

I installed the driver -> then the patch with ./install.sh.

IOMMU and PCIe ACS are disabled.

Any help is appreciated.

[Panchovix]

A more complete log with the others GPUs alone and in the mix

pancho@fedora:~/cuda-samples/build/Samples/5_Domain_Specific/p2pBandwidthLatencyTest$ export CUDA_VISIBLE_DEVICES=0,1
pancho@fedora:~/cuda-samples/build/Samples/5_Domain_Specific/p2pBandwidthLatencyTest$ ./p2pBandwidthLatencyTest 
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 4090, pciBusID: 2, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 4090, pciBusID: 17, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1
     0       1     1
     1       1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 1008.06   6.34 
     1   6.34 926.75 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1 
     0 1015.27   6.60 
     1   6.60 945.63 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 1018.92   8.66 
     1   8.67 926.39 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 1016.92  12.88 
     1  12.88 924.56 
P2P=Disabled Latency Matrix (us)
   GPU     0      1 
     0   1.37  10.28 
     1  10.33   1.29 

   CPU     0      1 
     0   1.61   4.89 
     1   5.01   1.60 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1 
     0   1.36   0.88 
     1   0.90   1.28 

   CPU     0      1 
     0   1.61   1.35 
     1   1.35   1.58 

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
pancho@fedora:~/cuda-samples/build/Samples/5_Domain_Specific/p2pBandwidthLatencyTest$ export CUDA_VISIBLE_DEVICES=4,5,6
pancho@fedora:~/cuda-samples/build/Samples/5_Domain_Specific/p2pBandwidthLatencyTest$ ./p2pBandwidthLatencyTest 
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA RTX A6000, pciBusID: 12, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 3090, pciBusID: 6, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 3090, pciBusID: d, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
Device=1 CAN Access Peer Device=0
Device=1 CAN Access Peer Device=2
Device=2 CAN Access Peer Device=0
Device=2 CAN Access Peer Device=1

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2
     0       1     1     1
     1       1     1     1
     2       1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2 
     0 583.89   3.07   3.06 
     1   3.09 944.11   3.08 
     2   3.06   3.07 917.50 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2 
     0 584.77   2.32   6.60 
     1   2.08 946.97   2.08 
     2   6.60   2.32 918.58 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2 
     0 588.29   3.23   3.18 
     1   3.23 953.03   3.24 
     2   3.17   3.24 924.28 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2 
     0 588.40   2.32   6.60 
     1   2.31 950.43   2.32 
     2   6.60   2.32 923.46 
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2 
     0   1.89  19.50  24.48 
     1  19.39   1.61  19.34 
     2  24.44  19.56   1.67 

   CPU     0      1      2 
     0   1.67   5.16   4.99 
     1   4.83   1.70   4.73 
     2   4.79   6.08   1.61 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2 
     0   1.91   1.86   1.92 
     1   1.44   1.64   1.40 
     2   1.78   1.75   1.57 

   CPU     0      1      2 
     0   1.60   1.30   1.35 
     1   1.34   1.57   1.38 
     2   1.37   1.30   1.60 

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
pancho@fedora:~/cuda-samples/build/Samples/5_Domain_Specific/p2pBandwidthLatencyTest$ export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6
pancho@fedora:~/cuda-samples/build/Samples/5_Domain_Specific/p2pBandwidthLatencyTest$ ./p2pBandwidthLatencyTest 
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 4090, pciBusID: 2, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 4090, pciBusID: 17, pciDeviceID: 0, pciDomainID:0
Device: 2, NVIDIA GeForce RTX 5090, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 3, NVIDIA GeForce RTX 5090, pciBusID: 3, pciDeviceID: 0, pciDomainID:0
Device: 4, NVIDIA RTX A6000, pciBusID: 12, pciDeviceID: 0, pciDomainID:0
Device: 5, NVIDIA GeForce RTX 3090, pciBusID: 6, pciDeviceID: 0, pciDomainID:0
Device: 6, NVIDIA GeForce RTX 3090, pciBusID: d, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CANNOT Access Peer Device=2
Device=0 CANNOT Access Peer Device=3
Device=0 CANNOT Access Peer Device=4
Device=0 CANNOT Access Peer Device=5
Device=0 CANNOT Access Peer Device=6
Device=1 CAN Access Peer Device=0
Device=1 CANNOT Access Peer Device=2
Device=1 CANNOT Access Peer Device=3
Device=1 CANNOT Access Peer Device=4
Device=1 CANNOT Access Peer Device=5
Device=1 CANNOT Access Peer Device=6
Device=2 CANNOT Access Peer Device=0
Device=2 CANNOT Access Peer Device=1
Device=2 CAN Access Peer Device=3
Device=2 CANNOT Access Peer Device=4
Device=2 CANNOT Access Peer Device=5
Device=2 CANNOT Access Peer Device=6
Device=3 CANNOT Access Peer Device=0
Device=3 CANNOT Access Peer Device=1
Device=3 CAN Access Peer Device=2
Device=3 CANNOT Access Peer Device=4
Device=3 CANNOT Access Peer Device=5
Device=3 CANNOT Access Peer Device=6
Device=4 CANNOT Access Peer Device=0
Device=4 CANNOT Access Peer Device=1
Device=4 CANNOT Access Peer Device=2
Device=4 CANNOT Access Peer Device=3
Device=4 CAN Access Peer Device=5
Device=4 CAN Access Peer Device=6
Device=5 CANNOT Access Peer Device=0
Device=5 CANNOT Access Peer Device=1
Device=5 CANNOT Access Peer Device=2
Device=5 CANNOT Access Peer Device=3
Device=5 CAN Access Peer Device=4
Device=5 CAN Access Peer Device=6
Device=6 CANNOT Access Peer Device=0
Device=6 CANNOT Access Peer Device=1
Device=6 CANNOT Access Peer Device=2
Device=6 CANNOT Access Peer Device=3
Device=6 CAN Access Peer Device=4
Device=6 CAN Access Peer Device=5

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2     3     4     5     6
     0       1     1     0     0     0     0     0
     1       1     1     0     0     0     0     0
     2       0     0     1     1     0     0     0
     3       0     0     1     1     0     0     0
     4       0     0     0     0     1     1     1
     5       0     0     0     0     1     1     1
     6       0     0     0     0     1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6 
     0 1008.06   6.34   6.53   6.53   6.07   3.11   3.09 
     1   6.33 927.85   6.53   6.53   6.07   3.11   3.09 
     2   6.64   6.64 1749.78  24.69   6.23   4.92   4.90 
     3   6.64   6.64  24.75 1759.57   6.23   4.94   4.90 
     4   6.37   6.37   6.45   6.45 585.43   3.07   3.06 
     5   3.21   3.21   5.06   5.05   3.09 945.82   3.09 
     6   3.20   3.06   5.06   5.08   3.06   3.07 916.96 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
Cuda failure /home/pancho/cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest/p2pBandwidthLatencyTest.cu:192: 'mapping of buffer object failed'

pancho@fedora:~/cuda-samples/build/Samples/0_Introduction/simpleP2P$ export CUDA_VISIBLE_DEVICES=0,1
pancho@fedora:~/cuda-samples/build/Samples/0_Introduction/simpleP2P$ ./simpleP2P 
[./simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 2

Checking GPU(s) for support of peer to peer memory access...
> Peer access from NVIDIA GeForce RTX 4090 (GPU0) -> NVIDIA GeForce RTX 4090 (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 4090 (GPU1) -> NVIDIA GeForce RTX 4090 (GPU0) : Yes
Enabling peer access between GPU0 and GPU1...
Allocating buffers (64MB on GPU0, GPU1 and CPU Host)...
Creating event handles...
cudaMemcpyPeer / cudaMemcpy between GPU0 and GPU1: 6.13GB/s
Preparing host buffer and memcpy to GPU0...
Run kernel on GPU1, taking source data from GPU0 and writing to GPU1...
Run kernel on GPU0, taking source data from GPU1 and writing to GPU0...
Copy data back to host from GPU0 and verify results...
Disabling peer access...
Shutting down...
Test passed
pancho@fedora:~/cuda-samples/build/Samples/0_Introduction/simpleP2P$ export CUDA_VISIBLE_DEVICES=2,3
pancho@fedora:~/cuda-samples/build/Samples/0_Introduction/simpleP2P$ ./simpleP2P 
[./simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 2

Checking GPU(s) for support of peer to peer memory access...
> Peer access from NVIDIA GeForce RTX 5090 (GPU0) -> NVIDIA GeForce RTX 5090 (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 5090 (GPU1) -> NVIDIA GeForce RTX 5090 (GPU0) : Yes
Enabling peer access between GPU0 and GPU1...
CUDA error at /home/pancho/cuda-samples/Samples/0_Introduction/simpleP2P/simpleP2P.cu:130 code=205(cudaErrorMapBufferObjectFailed) "cudaDeviceEnablePeerAccess(gpuid[1], 0)" 

[changzhi1990]

I also met this problem.

[ChiikawaSama]

+1

[changzhi1990]

Please refer to #33 (comment). It can resolve my issue.

[ChiikawaSama]

#33 (comment)

thanks for your help, but my 5090s' bar size is already 32GB, still cannot resolve this problem. May you tell me what branch do you use to hack p2p?

[changzhi1990]

#33 (comment)

thanks for your help, but my 5090s' bar size is already 32GB, still cannot resolve this problem. May you tell me what branch do you use to hack p2p?

https://github.com/tinygrad/open-gpu-kernel-modules/tree/570.124.06-p2p

[ChiikawaSama]

#33 (comment)

thanks for your help, but my 5090s' bar size is already 32GB, still cannot resolve this problem. May you tell me what branch do you use to hack p2p?

https://github.com/tinygrad/open-gpu-kernel-modules/tree/570.124.06-p2p

thanks a lot! i'll try

[ChiikawaSama]

#33 (comment)

thanks for your help, but my 5090s' bar size is already 32GB, still cannot resolve this problem. May you tell me what branch do you use to hack p2p?

https://github.com/tinygrad/open-gpu-kernel-modules/tree/570.124.06-p2p

still not running, sad😂

[JzhJay]

Do you have any progress on this issue? I've been getting stuck at the same place...

[Panchovix]

No luck yet, still same issue sadly. Latest branch was also updated like 1 month ago so not much hopes for now :(.

[JzhJay]

No luck yet, still same issue sadly. Latest branch was also updated like 1 month ago so not much hopes for now :(.

@Panchovix Yea I'm also trying with the latest branch. I'm wondering what's your BIOS settings like? Have you also disabled IOMMU and enable resize BAR?

[Panchovix]

Yes, basically disabled PCIe ACS and IOMMU, but no luck with the 5090s, but it does work fine with some 4090s and 3090s I have on my PC. Probably it is something related to the code itself that doesn't support the 5090 for now.

[JzhJay]

Yes, basically disabled PCIe ACS and IOMMU, but no luck with the 5090s, but it does work fine with some 4090s and 3090s I have on my PC. Probably it is something related to the code itself that doesn't support the 5090 for now.

Yes, I guess there is some slight difference about the buffer manipulation between 4090 and 5090, I'm trying to figure it out