Regression in using NVCC as a CUDA compiler: redzone_allocator_kernel_cuda compilation fails #13460
Comments
|
Do we even support nvcc as a CUDA compiler? |
|
also CC @ddunl |
FWIW, NVCC is the default cuda compiler in configure.py. |
|
Sorry for the delay - I was out. I will look into this. Interestingly we don't see this on the CI. Which version of CUDA are you using? |
12.1.0 |
copybara-service bot
pushed a commit
that referenced
this issue
Jun 19, 2024
It used to be a `gpu_kernel_library` which means it gets compiled for device using NVCC in OSS. But the sources files don't contain any CUDA code, so there is no need for that. #13460 reports that this behaviour even fails with some versions of NVCC because NVCC can't deal with C++ template magic being pulled in by Abseil. PiperOrigin-RevId: 644658241
copybara-service bot
pushed a commit
to tensorflow/tensorflow
that referenced
this issue
Jun 19, 2024
It used to be a `gpu_kernel_library` which means it gets compiled for device using NVCC in OSS. But the sources files don't contain any CUDA code, so there is no need for that. openxla/xla#13460 reports that this behaviour even fails with some versions of NVCC because NVCC can't deal with C++ template magic being pulled in by Abseil. Reverts c0e79da PiperOrigin-RevId: 644658241
copybara-service bot
pushed a commit
that referenced
this issue
Jun 19, 2024
It used to be a `gpu_kernel_library` which means it gets compiled for device using NVCC in OSS. But the sources files don't contain any CUDA code, so there is no need for that. #13460 reports that this behaviour even fails with some versions of NVCC because NVCC can't deal with C++ template magic being pulled in by Abseil. PiperOrigin-RevId: 644658241
copybara-service bot
pushed a commit
to tensorflow/tensorflow
that referenced
this issue
Jun 19, 2024
It used to be a `gpu_kernel_library` which means it gets compiled for device using NVCC in OSS. But the sources files don't contain any CUDA code, so there is no need for that. openxla/xla#13460 reports that this behaviour even fails with some versions of NVCC because NVCC can't deal with C++ template magic being pulled in by Abseil. PiperOrigin-RevId: 644658241
copybara-service bot
pushed a commit
that referenced
this issue
Jun 19, 2024
It used to be a `gpu_kernel_library` which means it gets compiled for device using NVCC in OSS. But the sources files don't contain any CUDA code, so there is no need for that. #13460 reports that this behaviour even fails with some versions of NVCC because NVCC can't deal with C++ template magic being pulled in by Abseil. PiperOrigin-RevId: 644658241
copybara-service bot
pushed a commit
that referenced
this issue
Jun 19, 2024
It used to be a `gpu_kernel_library` which means it gets compiled for device using NVCC in OSS. But the sources files don't contain any CUDA code, so there is no need for that. #13460 reports that this behaviour even fails with some versions of NVCC because NVCC can't deal with C++ template magic being pulled in by Abseil. PiperOrigin-RevId: 644658241
copybara-service bot
pushed a commit
to tensorflow/tensorflow
that referenced
this issue
Jun 19, 2024
It used to be a `gpu_kernel_library` which means it gets compiled for device using NVCC in OSS. But the sources files don't contain any CUDA code, so there is no need for that. openxla/xla#13460 reports that this behaviour even fails with some versions of NVCC because NVCC can't deal with C++ template magic being pulled in by Abseil. PiperOrigin-RevId: 644658241
copybara-service bot
pushed a commit
that referenced
this issue
Jun 19, 2024
It used to be a `gpu_kernel_library` which means it gets compiled for device using NVCC in OSS. But the sources files don't contain any CUDA code, so there is no need for that. #13460 reports that this behaviour even fails with some versions of NVCC because NVCC can't deal with C++ template magic being pulled in by Abseil. PiperOrigin-RevId: 644685844
copybara-service bot
pushed a commit
to tensorflow/tensorflow
that referenced
this issue
Jun 19, 2024
It used to be a `gpu_kernel_library` which means it gets compiled for device using NVCC in OSS. But the sources files don't contain any CUDA code, so there is no need for that. openxla/xla#13460 reports that this behaviour even fails with some versions of NVCC because NVCC can't deal with C++ template magic being pulled in by Abseil. PiperOrigin-RevId: 644685844
|
Hey @pearu, I've pushed a (potential) fix. Would you be able to conform whether it actually fixes your issue? |
|
IIUC, the (potential) fix is equivalent to applying a patch containing: When I apply this to my local branch (I can try main later if needed), there is a progress: Just for the sake of experiment, making HTH |
|
Ok cool. That seems to be the same issue, but in a different file. I can fix that tomorrow. |
|
Hey, I won't be getting to this today. As a workaround you could use a later version of the CUDA toolkit. We use CUDA 12.3 in the CI, so I believe 12.3+ shouldn't have the bug anymore. Or - as you already discovered - you can use Clang as the CUDA compiler instead. |
Yes, it should be changed to Clang since that's the only thing we test in CI. |
|
I confirm that when using CUDA 12.3.2, this issue cannot be reproduced: bazel build is successful without applying the above mentioned patches. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
After commit d8f0c1a, building XLA fails for CUDA backend. Reproducer:
With
the XLA build is succesful.
Using:
CC: @beckerhe
The text was updated successfully, but these errors were encountered: