RuntimeError: CUDA error: invalid device ordinal

[KhambardeCal]

orch.cuda.set_device(args.local_rank)
File "/home/socialab/miniconda3/envs/IR_SimANS-env/lib/python3.10/site-packages/torch/cuda/init.py", line 314, in set_device
torch._C._cuda_setDevice(device)
RuntimeError: CUDA error: invalid device ordinal

[lx865712528]

Is there any clearer background on how you ran?