Inferencing using YOLO with OpenVINO format

[Averen19]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

Does anyone know any resources or code that can do the inferencing using the openVINO toolkit with the converted IR files?
I would like to detect objects using the NCS 2 on the Raspberry Pi 4.

Additional

No response

[glenn-jocher]

@Averen19 I don't have NCS experience myself, but we've run OpenVINO on a variety of CPU backends with excellent results. It's usually one of the fastest export formats. See #6613 for details:

Run YOLOv5 benchmarks on a PyTorch model for all supported export formats. Currently operates on CPU, future updates will implement GPU inference.

Format	export.py --include	Model
PyTorch	-	yolov5s.pt
TorchScript	torchscript	yolov5s.torchscript
ONNX	onnx	yolov5s.onnx
OpenVINO	openvino	yolov5s_openvino_model/
TensorRT	engine	yolov5s.engine
CoreML	coreml	yolov5s.mlmodel
TensorFlow SavedModel	saved_model	yolov5s_saved_model/
TensorFlow GraphDef	pb	yolov5s.pb
TensorFlow Lite	tflite	yolov5s.tflite
TensorFlow Edge TPU	edgetpu	yolov5s_edgetpu.tflite
TensorFlow.js	tfjs	yolov5s_web_model/

"""
Requirements:
    $ pip install -r requirements.txt coremltools onnx onnx-simplifier onnxruntime openvino-dev tensorflow-cpu  # CPU
    $ pip install -r requirements.txt coremltools onnx onnx-simplifier onnxruntime-gpu openvino-dev tensorflow  # GPU

Usage:
    $ python utils/benchmarks.py --weights yolov5s.pt --img 640
"""

Colab++ V100 High-RAM CPU Results

benchmarks: weights=/content/yolov5/yolov5s.pt, imgsz=640, batch_size=1, data=/content/yolov5/data/coco128.yaml
Checking setup...
YOLOv5 🚀 v6.0-282-g8310a17 torch 1.10.0+cu111 CPU
Setup complete ✅ (8 CPUs, 51.0 GB RAM, 44.7/166.8 GB disk)

Benchmarks complete (637.94s)
                   Format  [email protected]:0.95  Inference time (ms)
0                 PyTorch      0.402908           111.652056
1             TorchScript      0.402908           142.402692
2                    ONNX      0.402908            64.537143
3                OpenVINO      0.402908            69.528472
4                TensorRT           NaN                  NaN
5                  CoreML           NaN                  NaN
6   TensorFlow SavedModel      0.402908           150.990861
7     TensorFlow GraphDef      0.402908           123.970838
8         TensorFlow Lite      0.402851           229.984051
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

Colab++ A100 High-RAM CPU Results

benchmarks: weights=/content/yolov5/yolov5s.pt, imgsz=640, batch_size=1, data=/content/yolov5/data/coco128.yaml
Checking setup...
YOLOv5 🚀 v6.0-267-g66e5d79 torch 1.10.0+cu111 CPU
Setup complete ✅ (12 CPUs, 83.5 GB RAM, 45.6/166.8 GB disk)

Benchmarks complete (437.85s)
                   Format  [email protected]:0.95  Inference time (ms)
0                 PyTorch      0.402908            91.519212
1             TorchScript      0.402908           116.423221
2                    ONNX      0.402908            46.936002
3                OpenVINO      0.402908            51.490320
4                TensorRT           NaN                  NaN
5                  CoreML           NaN                  NaN
6   TensorFlow SavedModel      0.402908            94.869515
7     TensorFlow GraphDef      0.402908            94.925985
8         TensorFlow Lite      0.402851           212.536491
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

MacOS Intel CPU Results (CoreML-capable)

benchmarks: weights=/Users/glennjocher/PycharmProjects/yolov5/yolov5s.pt, imgsz=640, batch_size=1, data=/Users/glennjocher/PycharmProjects/yolov5/data/coco128.yaml
Checking setup...
Setup complete ✅ (8 CPUs, 32.0 GB RAM, 750.5/931.6 GB disk)
YOLOv5 🚀 v6.0-282-g8310a17 torch 1.10.0 CPU

Benchmarks complete (671.88s)
                   Format  [email protected]:0.95  Inference time (ms)
0                 PyTorch      0.402908           283.422085
1             TorchScript      0.402908           291.616663
2                    ONNX      0.402908            94.682336
3                OpenVINO      0.402908            96.916517
4                TensorRT           NaN                  NaN
5                  CoreML      0.402076            73.937317
6   TensorFlow SavedModel      0.402908           172.185935
7     TensorFlow GraphDef      0.402908           145.169625
8         TensorFlow Lite      0.402851           208.791604
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

Ultralytics Hyperplane EPYC Milan AMD CPU Results

benchmarks: weights=/usr/src/app/yolov5s.pt, imgsz=640, batch_size=1, data=/usr/src/app/data/coco128.yaml
Checking setup...
/bin/sh: 1: git: not found
YOLOv5 🚀 2022-3-11 torch 1.11.0+cpu CPU
Setup complete ✅ (96 CPUs, 1007.7 GB RAM, 885.4/3519.3 GB disk)

Benchmarks complete (334.78s)
                   Format  [email protected]:0.95  Inference time (ms)
0                 PyTorch      0.462296            59.022402
1             TorchScript      0.462296            81.963500
2                    ONNX      0.462296            66.349735
3                OpenVINO      0.462296            28.817065
4                TensorRT           NaN                  NaN
5                  CoreML           NaN                  NaN
6   TensorFlow SavedModel      0.462296            91.322277
7     TensorFlow GraphDef      0.462296            96.317230
8         TensorFlow Lite      0.462334           160.701267
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

Resolves #6586

[github-actions]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs.

Access additional YOLOv5 🚀 resources:

• Wiki – https://github.com/ultralytics/yolov5/wiki
• Tutorials – https://docs.ultralytics.com/yolov5
• Docs – https://docs.ultralytics.com

Access additional Ultralytics ⚡ resources:

• Ultralytics HUB – https://ultralytics.com/hub
• Vision API – https://ultralytics.com/yolov5
• About Us – https://ultralytics.com/about
• Join Our Team – https://ultralytics.com/work
• Contact Us – https://ultralytics.com/contact

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLOv5 🚀 and Vision AI ⭐!