-
Notifications
You must be signed in to change notification settings - Fork 2
Results
The results for the Default and Accuracy-aware quantization algorithms presented below are obtained using the defaultQuantization.ipynb and accuracyQuantization.ipynb notebooks, respectively. The quantized models can be found here.
The results of the default quantization method are presented on Table 1. The accuracy is calculated on the official CIFAR-10 test set.
| Model | Calibration Dataset | Accuracy | FPS |
|---|---|---|---|
| ResNet20 (PyTorch) | - | 0.926 | 186.7 |
| ResNet20 (IR) | - | 0.926 | 1277.52 |
| ResNet20 (8-bit quantized) | Fractal | 0.9158 | 1590.19 |
| ResNet20 (8-bit quantized) | CIFAR-10 | 0.9234 | 1565.45 |
| ResNet20 (8-bit quantized) | FakeCIFAR10 (StyleGAN2-ADA) | 0.922 | 1541.86 |
| ResNet20 (8-bit quantized) | FakeCIFAR10 (DiStyleGAN) | 0.9217 | 1602.92 |
| VGG16_bn (PyTorch) | - | 0.9416 | 54.85 |
| VGG16_bn (IR) | - | 0.9416 | 246.29 |
| VGG16_bn (8-bit quantized) | Fractal | 0.9351 | 654.41 |
| VGG16_bn (8-bit quantized) | CIFAR-10 | 0.9411 | 680.46 |
| VGG16_bn (8-bit quantized) | FakeCIFAR10 (StyleGAN2-ADA) | 0.9401 | 604.65 |
| VGG16_bn (8-bit quantized) | FakeCIFAR10 (DiStyleGAN) | 0.9409 | 623.14 |
| MobileNetV2_x1_4 (PyTorch) | - | 0.9421 | 37.5 |
| MobileNetV2_x1_4 (IR) | - | 0.9421 | 130.38 |
| MobileNetV2_x1_4 (8-bit quantized) | Fractal | 0.937 | 478.11 |
| MobileNetV2_x1_4 (8-bit quantized) | CIFAR-10 | 0.9414 | 425.89 |
| MobileNetV2_x1_4 (8-bit quantized) | FakeCIFAR10 (StyleGAN2-ADA) | 0.9416 | 447.95 |
| MobileNetV2_x1_4 (8-bit quantized) | FakeCIFAR10 (DiStyleGAN) | 0.9406 | 483.06 |
| ShuffleNetv2_x2_0 (PyTorch) | - | 0.9398 | 43.14 |
| ShuffleNetv2_x2_0 (IR) | - | 0.9398 | 228.89 |
| ShuffleNetv2_x2_0 (8-bit quantized) | Fractal | 0.1202 | 442.75 |
| ShuffleNetv2_x2_0 (8-bit quantized) | CIFAR-10 | 0.928 | 404.5 |
| ShuffleNetv2_x2_0 (8-bit quantized) | FakeCIFAR10 (StyleGAN2-ADA) | 0.8695 | 417.81 |
| ShuffleNetv2_x2_0 (8-bit quantized) | FakeCIFAR10 (DiStyleGAN) | 0.9258 | 443.99 |
| RepVGG_a2 (PyTorch) | - | 0.9527 | 11.43 |
| RepVGG_a2 (IR) | - | 0.9527 | 56.77 |
| RepVGG_a2 (8-bit quantized) | Fractal | 0.5551 | 154.89 |
| RepVGG_a2 (8-bit quantized) | CIFAR-10 | 0.5531 | 136.29 |
| RepVGG_a2 (8-bit quantized) | FakeCIFAR10 (StyleGAN2-ADA) | 0.5586 | 148.98 |
| RepVGG_a2 (8-bit quantized) | FakeCIFAR10 (DiStyleGAN) | 0.5445 | 135.94 |
Table 1: Default Quantization results
The results of the accuracy-control quantization method are presented on Table 2. We experimented with the two models, namely ShuffleNetV2 and RepVGG, that showcased accuracy degradation when we used the Default Quantization method. The accuracy is again calculated on the official CIFAR-10 test set. The inference speeds (FPS) are not reported, since they are similar to the ones presented on Table 1 for the corresponding models.
| Model \ Calibration Dataset | Fractal | CIFAR-10 | StyleGAN2-ADA | DiStyleGAN |
|---|---|---|---|---|
| ShuffleNetv2_x2_0 | 0.1202 | 0.928 | 0.9343 | 0.9388 |
| RepVGG_a2 | 0.551 | 0.9484 | 0.9481 | 0.9475 |
Table 2: Accuracy-control Quantization results (measure: accuracy)