Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

YOLOv5 CPU Export Benchmarks #6613

Merged
merged 40 commits into from
Feb 12, 2022
Merged

YOLOv5 CPU Export Benchmarks #6613

merged 40 commits into from
Feb 12, 2022

Conversation

glenn-jocher
Copy link
Member

@glenn-jocher glenn-jocher commented Feb 11, 2022

Run YOLOv5 benchmarks on a PyTorch model for all supported export formats. Currently operates on CPU, future updates will implement GPU inference. For GPU benchmarks see #6963.

Format export.py --include Model
PyTorch - yolov5s.pt
TorchScript torchscript yolov5s.torchscript
ONNX onnx yolov5s.onnx
OpenVINO openvino yolov5s_openvino_model/
TensorRT engine yolov5s.engine
CoreML coreml yolov5s.mlmodel
TensorFlow SavedModel saved_model yolov5s_saved_model/
TensorFlow GraphDef pb yolov5s.pb
TensorFlow Lite tflite yolov5s.tflite
TensorFlow Edge TPU edgetpu yolov5s_edgetpu.tflite
TensorFlow.js tfjs yolov5s_web_model/
"""
Requirements:
    $ pip install -r requirements.txt coremltools onnx onnx-simplifier onnxruntime openvino-dev tensorflow-cpu  # CPU
    $ pip install -r requirements.txt coremltools onnx onnx-simplifier onnxruntime-gpu openvino-dev tensorflow  # GPU

Usage:
    $ python utils/benchmarks.py --weights yolov5s.pt --img 640
"""

Colab Pro+ V100 High-RAM CPU Results

benchmarks: weights=/content/yolov5/yolov5s.pt, imgsz=640, batch_size=1, data=/content/yolov5/data/coco128.yaml, device=cpu, half=False, test=False
Checking setup...
YOLOv5 🚀 v6.1-135-g7926afc torch 1.10.0+cu111 CPU
Setup complete ✅ (8 CPUs, 51.0 GB RAM, 41.5/166.8 GB disk)

Benchmarks complete (241.20s)
                   Format  [email protected]:0.95  Inference time (ms)
0                 PyTorch        0.4623               127.61
1             TorchScript        0.4623               131.23
2                    ONNX        0.4623                69.34
3                OpenVINO        0.4623                66.52
4                TensorRT           NaN                  NaN
5                  CoreML           NaN                  NaN
6   TensorFlow SavedModel        0.4623               123.79
7     TensorFlow GraphDef        0.4623               121.57
8         TensorFlow Lite        0.4623               316.61
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

MacOS Intel CPU Results (CoreML-capable)

iMac (Retina 5K, 27-inch, 2020) - 3.8 GHz 8-Core Intel Core i7

benchmarks: weights=yolov5s.pt, imgsz=640, batch_size=1, data=/Users/glennjocher/PycharmProjects/yolov5/data/coco128.yaml, device=, half=False, test=False, pt_only=False
Checking setup...
YOLOv5 🚀 v6.1-171-gb4f7fc5 torch 1.10.1 CPU
Setup complete ✅ (16 CPUs, 32.0 GB RAM, 213.4/465.6 GB disk)

Benchmarks complete (276.40s)
                   Format  [email protected]:0.95  Inference time (ms)
0                 PyTorch        0.4623               222.37
1             TorchScript        0.4623               231.01
2                    ONNX        0.4623                54.41
3                OpenVINO        0.4623                40.74
4                TensorRT           NaN                  NaN
5                  CoreML        0.4620                39.09
6   TensorFlow SavedModel        0.4623               153.32
7     TensorFlow GraphDef        0.4623               148.00
8         TensorFlow Lite        0.4623               162.60
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

iMac (Retina 5K, 27-inch, 2014)

benchmarks: weights=/Users/glennjocher/PycharmProjects/yolov5/yolov5s.pt, imgsz=640, batch_size=1, data=/Users/glennjocher/PycharmProjects/yolov5/data/coco128.yaml, device=, half=False, test=False
Checking setup...
YOLOv5 🚀 v6.1-135-g7926afc torch 1.11.0 CPU
Setup complete ✅ (8 CPUs, 32.0 GB RAM, 793.4/931.6 GB disk)

Benchmarks complete (288.68s)
                   Format  [email protected]:0.95  Inference time (ms)
0                 PyTorch        0.4623               281.47
1             TorchScript        0.4623               262.97
2                    ONNX        0.4623                77.30
3                OpenVINO        0.4623                74.12
4                TensorRT           NaN                  NaN
5                  CoreML        0.4620                69.36
6   TensorFlow SavedModel        0.4623               123.12
7     TensorFlow GraphDef        0.4623               120.82
8         TensorFlow Lite        0.4623               249.62
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

Ultralytics Hyperplane EPYC Milan AMD CPU Results

benchmarks: weights=yolov5s.pt, imgsz=640, batch_size=1, data=/usr/src/app/data/coco128.yaml, device=, half=False, test=False, pt_only=False
Checking setup...
YOLOv5 🚀 v6.1-174-gc4cb7c6 torch 1.11.0+cpu CPU
Setup complete ✅ (96 CPUs, 1007.7 GB RAM, 1948.4/3519.3 GB disk)

Benchmarks complete (128.39s)
                   Format  [email protected]:0.95  Inference time (ms)
0                 PyTorch        0.4623                56.52
1             TorchScript        0.4623                59.21
2                    ONNX        0.4623                60.95
3                OpenVINO        0.4623                27.44
4                TensorRT           NaN                  NaN
5                  CoreML           NaN                  NaN
6   TensorFlow SavedModel        0.4623                70.84
7     TensorFlow GraphDef        0.4623                72.44
8         TensorFlow Lite        0.4623               130.43
9     TensorFlow Edge TPU           NaN                  NaN
10          TensorFlow.js           NaN                  NaN

Resolves #6586

🛠️ PR Summary

Made with ❤️ by Ultralytics Actions

🌟 Summary

This PR introduces an enhanced export functionality, streamlines model type handling within the codebase, and adds a new benchmarking utility.

📊 Key Changes

  • Added pandas library to export.py for dataframe support.
  • Created export_formats function to list supported export formats in a dataframe.
  • Simplified suffix checking in models/common.py by utilizing the model_type method, which relies on the new export formats dataframe.
  • Ensured OpenVINO export handles directories correctly to find .xml files.
  • Introduced benchmarks.py, a new utility script to benchmark YOLOv5 across various supported export formats.
  • Updated val.py to support a 'benchmark' task, which affects image padding and inference shape handling.

🎯 Purpose & Impact

  • The addition of export_formats function and dataframe usage aims to centralize and organize the different supported export formats, making it easier to manage them.
  • Streamlining the model type determination process simplifies the way the code interprets and works with various model file types.
  • Introducing benchmarks for YOLOv5 models offers users a means to measure and compare performance across different formats, which can help in choosing the right format for deployment.
  • The option to alter image padding and the inference shape for benchmarks may provide more accurate and comparative speed measurements for the models.

Overall, these changes enhance the usability of the YOLOv5 export options and provide valuable insights into model performance, potentially benefiting developers and users focused on optimization and deployment of models in different environments. 🚀

This was referenced Nov 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
1 participant