Just less than nvidia-smi?
NOTE: This works with NVIDIA Graphics Devices only, no AMD support as of now. Contributions are welcome!
Self-Promotion: A web interface of gpustat
is available (in alpha)! Check out gpustat-web.
Install from PyPI:
pip install gpustat
If you don't have root (sudo) privilege, please try installing gpustat
on user namespace: pip install --user gpustat
.
To install the latest version (master branch) via pip:
pip install git+https://github.com/wookayin/gpustat.git@master
Important
DO NOT: pip install pynvml
, nor include pynvml
as a dependency in your python project. This will not work.
Instead: pip install nvidia-ml-py
. nvidia-ml-py is NVIDIA's the official python binding for NVML.
- gpustat 1.2+: Requires
nvidia-ml-py >= 12.535.108
(#161) - gpustat 1.0+: Requires NVIDIA Driver 450.00 or higher and
nvidia-ml-py >= 11.450.129
. - If your NVIDIA driver is too old, you can use older
gpustat
versions (pip install gpustat<1.0
). See #107 for more details.
- gpustat<1.0: Compatible with python 2.7 and >=3.4
- gpustat 1.0: Python >= 3.4
- gpustat 1.1: Python >= 3.6
$ gpustat
Options (Please see gpustat --help
for more details):
--color
: Force colored output (even when stdout is not a tty)--no-color
: Suppress colored output-u
,--show-user
: Display username of the process owner-c
,--show-cmd
: Display the process name-f
,--show-full-cmd
: Display full command and cpu stats of running process-p
,--show-pid
: Display PID of the process-F
,--show-fan
: Display GPU fan speed-e
,--show-codec
: Display encoder and/or decoder utilization-P
,--show-power
: Display GPU power usage and/or limit (draw
ordraw,limit
)-a
,--show-all
: Display all gpu properties above--id
: Target and query specific GPUs only with the specified indices (e.g.--id 0,1,2
)--no-processes
: Do not display process information (user, memory) (#133)--watch
,-i
,--interval
: Run in watch mode (equivalent towatch gpustat
) if given. Denotes interval between updates.--json
: JSON Output (#10)--print-completion (bash|zsh|tcsh)
: Print a shell completion script. See #131 for usage.
- Try
gpustat --debug
if something goes wrong. - To periodically watch, try
gpustat --watch
orgpustat -i
(#41).- For older versions, one may use
watch --color -n1.0 gpustat --color
.
- For older versions, one may use
- Running
nvidia-smi daemon
(root privilege required) will make querying GPUs much faster and use less CPU (#54). - The GPU ID (index) shown by
gpustat
(andnvidia-smi
) is PCI BUS ID, while CUDA uses a different ordering (assigns the fastest GPU with the lowest ID) by default. Therefore, in order to ensure CUDA andgpustat
use same GPU index, configure theCUDA_DEVICE_ORDER
environment variable toPCI_BUS_ID
(before settingCUDA_VISIBLE_DEVICES
for your CUDA program):export CUDA_DEVICE_ORDER=PCI_BUS_ID
.
[0] GeForce GTX Titan X | 77°C, 96 % | 11848 / 12287 MB | python/52046(11821M)
[0]
: GPU index (starts from 0) as PCI_BUS_IDGeForce GTX Titan X
: GPU name77°C
: GPU Temperature (in Celsius)96 %
: GPU Utilization11848 / 12287 MB
: GPU Memory Usage (Used / Total)python/...
: Running processes on GPU, owner/cmdline/PID (and their GPU memory usage)
See CHANGELOG.md