Selecting a specific GPU

[gsuhm]

I have two 3090s, and want to specify which one Aphrodite should use - is that possible to specify which one? I searched around for that option but couldn't find it. Thanks!

[AlpinDale]

Hi. You can specify the GPU to use with the CUDA_VISIBLE_DEVICES env variable. For example, if you want to use the first GPU, you'll have to:

export CUDA_VISIBLE_DEVICES=0

1 for the second GPU, since it's 0-indexed.

If you want to use both GPUs, just pass -tp 2 when launching the engine -- it'll use the first 2 GPUs it detects.

[AlpinDale]

Since the tp arg uses the first set of GPUs it detects, you can use CUDA_VISIBLE_DEVICES to use the first and 4th GPUs, for example.

export CUDA_VISIBLE_DEVICES=0,3

[gsuhm]

worked perfectly, thank you!

[cosmojg]

How might one do this when using Aphrodite as a library? Running os.environ["CUDA_VISIBLE_DEVICES"] = 0,3 before initializing the LLM doesn't seem to work. (I want to dynamically select a GPU or set of GPUs at runtime.)