nodedev-detach hangs #22
Comments
|
Hey! I'm facing the same issue. I have a R9 270x for the host and a 1070ti for the guest. When I try to:
It just hangs. The only think I think might be creating an issue would be xorg locking the GPU somehow, but I haven't seen it mentioned anywhere else. I'll try to poke around some stuff and if I get around to solve it I'll post here. If you have any suggestions they are highly appreciated. |
|
So I've tried some things and tried to understand what's happening in each step. I noticed something. When I boot my PC I have the following: After I run the command As you can see, before, it was using the drivers nvidia, after running the command, there are no drivers in use. I guess it means that there is a problems loading the vfio-pci drivers, but I'm not sure. I'll try to look into it. |
|
Are you sure that after the detach command is run, that the nvidia driver is actually detached? Like can you run that by hand and make sure that the driver is actually detached? If not then I had a similar issue. |
|
Ok, so I ended up getting everything to work. A friend of mine on discord and I sat down to figure it out and we made some very interesting discoveries. Bear in mind, this is what the solution was in my case, but you will have to do your own testing to see if this works for you. where /dev/nvidia1 in my case was a reference to my second gpu. Since you only have one nvidia card, it would likely be nvidia0. Even after running the detach command, gnome shell is still using the gpu, aka, the file /dev/nvidia1 is still 'open', despite my xorg.conf telling it not to. For reference, here is my xorg.conf: The relevant parts there being the device section where I set my 1080 as the only GPU and never make a second device with the 1070, and the "AutoAddGPU" "off" option. But, gnome still uses it for some reason. The way I solved this was by setting up my system like any other gpu passthrough system before it. Load vfio drivers at boot and bind them to the second GPU, and then rebind the nvidia driver after boot. Here is a quick refresher, but I am doing this from memory so please correct me if I miss something: Step 1: This will load the drivers on boot. Step 2: Step 3: I had to reboot my system before running the update command, otherwise it got hung up on something, not really sure why. More details on regular vfio setups can be found here: https://forum.level1techs.com/t/vfio-in-2019-pop-os-how-to-general-guide-though-draft/142287 Once that is all done you would basically be ready to do GPU passthrough as normal..... BUT WE WANT TO USE THE GPU ON THE HOST NOW, RIGHT?! If you run the same lsof command as before, you will not see any reference to the second GPU, because it was never attached to gnome, and that's good. So now you can run But hold on, running the reattach command isn't enough. If you try to open something like blender and go to edit > preferences > system > cuda, your GPU wont be listed. Similarly, if you check At this point, you can use the GPU on the host, then run BUT WAIT! THERE'S MORE! Unfortunately, our testing uncovered another problem.... The audio driver... If you ever reattach the audio driver to the host, you will never be able to unbind it... at least... I couldn't. This is a problem, because by default when you close the VM that is using the device, it will release it's resource including the audio device, and the audio device will re-bind to snd_hda_intel. At that point you will have to actually reboot so that the vfio driver can re-bind to the audio device. I suspect that this is what causes the majority of the detach script hanging problem. I haven't found a very good solution to this other than to just blacklist that driver. Unfortunately for me, that also prevents my primary GPU from outputting audio over hdmi as well. It isnt a huge deal for me because I use headphones anyway, but it still isn't a very clean solution. A better solution would be to blacklist the driver on 'just' this one device at this specific pci address, but I haven't figured out how to do that. So in the end, I finished with was the following: Step 1: This will prevent the intel audio driver from ever loading. Run the update-initramfs command again. Step 2: After that, run This will auto bind the second GPU back to the host on a fresh boot. I then proceeded with the regular qemu hooks with the bind script being: And unbind: The reattach on the audio device really does nothing because there is no driver for it to attach to anymore. THIS WILL ONLY LET YOU USE THE SECOND GPU FOR CUDA ON HOST! THERE IS NO VIDEO OUTPUT WHEN REATTACHED! It would be cool if we could get that working but I don't think you could without starting a second xsession that could be killed when you want to switch. Also, if the second gpu is being used, and you try to start a VM, the VM won't start until the GPU is freed. For example, if I am running a render in Blender using that GPU, I have to close Blender before the VM will start. OK, I think that is everytihng. Please remember to replace my PCI ID |
|
Came across Moonlight63's comment here, it ended up being very helpful. I figured out that you can block In addition, to get this working with Dracut instead of initramfs-tools, you can create a directory at #!/bin/sh
check() {
return 0
}
depends() {
return 0
}
install() {
# Replace the filename here after $moddir/ with the name you gave the other script!
inst_hook pre-trigger 64 "$moddir/force-vfio-pci.sh"
}After that, you should be able to add it just like a standard Dracut module. |
(creating a new issue with the same body as my reply to the old one)
Originally posted by @Moonlight63 in #16 (comment)
Hello, Thank you for the guide. I have been running passthrough for a while, but just did a fresh install of Pop and thought it might be nice to be able to use my second gpu when not running VMs. I am having the same issues as others here.
Running
virsh nodedev-detach $VIRSH_GPU_VIDEOwhere, for me
causes a hang.
My gpus are on there own IOMMU groups
I have modified my xorg.conf so that it only uses the 1080 for host, and I have disabled AutoAddGPU, and I have a 3 monitor setup with all 3 plugged into the 3 displayports on the 1080, my full config is this:
And finally I have verified that the 1070 is not being used by anything with nvidia-smi:
As a side note, my CPU doesn't list the virtualization option as VT-d, but rather calls it by it's full name in dmesg:
Possibly because it's a xeon? Just thought I would mention it for others who come here.
Anyway, as far as I can tell, I've done everything mentioned and I can't find anything else that would be stopping the unload. Any ideas? Yes my scripts are executable, and I've been trying to just run the commands one by one in terminal to see if I can find an error exit, but since virsh nodedev-detach never completes and just hangs, no error is reported. Any help is greatly appreciated.
The text was updated successfully, but these errors were encountered: