7.1. Model Performance Benchmark¶

7.1.1. Descriptions¶

Test Conditions:
- Test Board: x3sdbx3-samsung2G-3200.
- Number of Test Core: latency single core, fps dual core.
- Frequency to obtain model performance data: Average of performance parameters over a 5-minute period.
Table Header Acronyms:
- C = Computation, in GOPs (i.e., billion operations per second), obtained by using the hb_perf tool.
- FPS = Frame(s) Per Second, obtained by running the fps.sh script with single thread of different models in the ai_benchmark sample package/script on the dev board. Post-processing included.
- ITC = Inference Time Consumption, in ms (millisecond), obtained by running the latency.sh script with single thread of different models in the ai_benchmark sample package/script on the dev board. Post-processing not included.
- TCPP = Postprocess Time Consumption, in ms (millisecond), obtained by running the latency.sh script with single thread of different models in the ai_benchmark sample package/script on the dev board.
- RV = Read Volume of single frame, in mb (Mbit), obtained by using the hb_perf tool.
- WV = Write Volume of single frame, in mb (Mbit), obtained by using the hb_perf tool.

7.1.2. Model-important Performance Data¶

MODEL NAME	INPUT SIZE	C(GOPs)	FPS	ITC(ms)	TCPP(ms)	ACCURACY	Dataset
MobileNetv1	1x224x224x3	1.14	661.12	3.200	0.086	Top1: 0.7061(FLOAT) 0.7034(INT8)	ImageNet
MobileNetv2	1x224x224x3	0.86	869.68	2.461	0.086	Top1: 0.7167(FLOAT) 0.7122(INT8)	ImageNet
GoogleNet	1x224x224x3	3.00	234.83	8.164	0.087	Top1: 0.7001(FLOAT) 0.6992(INT8)	ImageNet
Resnet18	1x224x224x3	3.65	228.31	8.754	0.086	Top1: 0.6836(FLOAT) 0.6830(INT8)	ImageNet
EfficientNet_Lite0	1x224x224x3	0.77	896.82	2.315	0.083	Top1: 0.7491(FLOAT) 0.7473(INT8)	ImageNet
EfficientNet_Lite1	1x240x240x3	1.20	624.36	3.196	0.081	Top1: 0.7647(FLOAT) 0.7625(INT8)	ImageNet
EfficientNet_Lite2	1x260x260x3	1.72	361.83	5.643	0.081	Top1: 0.7738(FLOAT) 0.7714(INT8)	ImageNet
EfficientNet_Lite3	1x280x280x3	2.77	229.13	8.550	0.080	Top1: 0.7922(FLOAT) 0.7901(INT8)	ImageNet
EfficientNet_Lite4	1x300x300x3	5.11	130.06	14.739	0.083	Top1: 0.8070(FLOAT) 0.8059(INT8)	ImageNet
YOLOv2_Darknet19	1x608x608x3	62.94	12.60	154.412	1.538	[IoU=0.50:0.95]= 0.2760(FLOAT) 0.2700(INT8)	COCO
YOLOv3_Darknet53	1x416x416x3	65.90	11.71	164.915	10.771	[IoU=0.50:0.95]= 0.3330(FLOAT) 0.3360(INT8)	COCO
YOLOv5s	1x672x672x3	19.18	32.62	65.179	20.329	[IoU=0.50:0.95]= 0.3520(FLOAT) 0.3420(INT8)	COCO
SSD_MobileNetv1	1x300x300x3	2.30	277.29	6.914	1.369	mAP: 0.7342(FLOAT) 0.7274(INT8)	VOC
EfficientDetd0	1x512x512x3	4.93	78.31	17.080	20.202	[IoU=0.50:0.95]= 0.3240(FLOAT) 0.3150(INT8)	COCO
CenterNet_Resnet50	1x512x512x3	51.75	18.03	110.484	26.496	[IoU=0.50:0.95]= 0.3180(FLOAT) 0.3130(INT8)	COCO
FCOS_efficientnetb0(QAT)	1x512x512x3	5.02	169.43	12.053	7.840	[IoU=0.50:0.95]= 0.3470(FLOAT) 0.3480(INT8)	COCO
Fcos_efficientnetb1	1x640x640x3	12.16	65.30	30.155	12.588	[IoU=0.50:0.95]= 0.4030(FLOAT) 0.4040(INT8)	COCO
FCOS_efficientnetb2	1x768x768x3	22.08	38.43	51.120	18.182	[IoU=0.50:0.95]= 0.4450(FLOAT) 0.4460(INT8)	COCO
FCOS_efficientnetb0(PTQ)	1x512x512x3	5.02	174.14	13.360	3.969	[IoU=0.50:0.95]= 0.3630(FLOAT) 0.3480(INT8)	COCO
UNet_mobilenet	1x1024x2048x3	7.37	57.16	27.690	21.501	mIoU: 0.6411(FLOAT) 0.6382(INT8)	Cityscapes
Deeplabv3plus_efficientnetb0	1x1024x2048x3	30.78	23.65	79.348	1.472	mIoU: 0.7630(FLOAT) 0.7567(INT8)	Cityscapes
Fastscnn_efficientnetb0	1x1024x2048x3	12.50	47.16	41.305	1.437	mIoU: 0.6997(FLOAT) 0.6927(INT8)	Cityscapes

7.1.3. Model Full Performance Data¶

7.1.3.1. MobileNetv1¶

INPUT SIZE: 1x224x224x3
C(GOPs): 1.14
FPS: 661.12
ITC(ms): 3.200
TCPP(ms): 0.086
RV(mb): 5.28
WV(mb): 0.83
Dataset: ImageNet
ACCURACY: Top1: 0.7061(FLOAT)/0.7034(INT8)
LINKS: https://github.com/shicai/MobileNet-Caffe

7.1.3.2. MobileNetv2¶

INPUT SIZE: 1x224x224x3
C(GOPs): 0.86
FPS: 869.68
ITC(ms): 2.461
TCPP(ms): 0.086
RV(mb): 3.99
WV(mb): 0.23
Dataset: ImageNet
ACCURACY: Top1: 0.7167(FLOAT)/0.7122(INT8)
LINKS: https://github.com/shicai/MobileNet-Caffe

7.1.3.3. GoogleNet¶

INPUT SIZE: 1x224x224x3
C(GOPs): 3.00
FPS: 234.83
ITC(ms): 8.164
TCPP(ms): 0.087
RV(mb): 10.31
WV(mb): 3.36
Dataset: ImageNet
ACCURACY: Top1: 0.7001(FLOAT)/0.6992(INT8)
LINKS: http://www.image-net.org

7.1.3.4. Resnet18¶

INPUT SIZE: 1x224x224x3
C(GOPs): 3.65
FPS: 228.31
ITC(ms): 8.754
TCPP(ms): 0.086
RV(mb): 13.26
WV(mb): 0.78
Dataset: ImageNet
ACCURACY: Top1: 0.6836(FLOAT)/0.6830(INT8)
LINKS: https://github.com/HolmesShuan/ResNet-18-Caffemodel-on-ImageNet

7.1.3.5. EfficientNet_Lite0¶

INPUT SIZE: 1x224x224x3
C(GOPs): 0.77
FPS: 896.82
ITC(ms): 2.315
TCPP(ms): 0.083
RV(mb): 5.31
WV(mb): 0.36
Dataset: ImageNet
ACCURACY: Top1: 0.7491(FLOAT)/0.7473(INT8)
LINKS: https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/lite

7.1.3.6. EfficientNet_Lite1¶

INPUT SIZE: 1x240x240x3
C(GOPs): 1.20
FPS: 624.36
ITC(ms): 3.196
TCPP(ms): 0.081
RV(mb): 6.37
WV(mb): 0.60
Dataset: ImageNet
ACCURACY: Top1: 0.7647(FLOAT)/0.7625(INT8)
LINKS: https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/lite

7.1.3.7. EfficientNet_Lite2¶

INPUT SIZE: 1x260x260x3
C(GOPs): 1.72
FPS: 361.83
ITC(ms): 5.643
TCPP(ms): 0.081
RV(mb): 6.61
WV(mb): 0.14
Dataset: ImageNet
ACCURACY: Top1: 0.7738(FLOAT)/0.7714(INT8)
LINKS: https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/lite

7.1.3.8. EfficientNet_Lite3¶

INPUT SIZE: 1x280x280x3
C(GOPs): 2.77
FPS: 229.13
ITC(ms): 8.550
TCPP(ms): 0.080
RV(mb): 9.16
WV(mb): 0.42
Dataset: ImageNet
ACCURACY: Top1: 0.7922(FLOAT)/0.7901(INT8)
LINKS: https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/lite

7.1.3.9. EfficientNet_Lite4¶

INPUT SIZE: 1x300x300x3
C(GOPs): 5.11
FPS: 130.06
ITC(ms): 14.739
TCPP(ms): 0.083
RV(mb): 14.96
WV(mb): 1.36
Dataset: ImageNet
ACCURACY: Top1: 0.8070(FLOAT)/0.8059(INT8)
LINKS: https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/lite

7.1.3.10. YOLOv2_Darknet19¶

INPUT SIZE: 1x608x608x3
C(GOPs): 62.94
FPS: 12.60
ITC(ms): 154.412
TCPP(ms): 1.538
RV(mb): 67.25
WV(mb): 15.25
Dataset: COCO
ACCURACY: [IoU=0.50:0.95]= 0.2760(FLOAT)/0.2700(INT8)
LINKS: https://pjreddie.com/darknet/yolo

7.1.3.11. YOLOv3_Darknet53¶

INPUT SIZE: 1x416x416x3
C(GOPs): 65.90
FPS: 11.71
ITC(ms): 164.915
TCPP(ms): 10.771
RV(mb): 103.57
WV(mb): 33.26
Dataset: COCO
ACCURACY: [IoU=0.50:0.95]= 0.3330(FLOAT)/0.3360(INT8)
LINKS: https://github.com/ChenYingpeng/caffe-yolov3

7.1.3.12. YOLOv5s¶

INPUT SIZE: 1x672x672x3
C(GOPs): 19.18
FPS: 32.62
ITC(ms): 65.179
TCPP(ms): 20.329
RV(mb): 42.32
WV(mb): 40.07
Dataset: COCO
ACCURACY: [IoU=0.50:0.95]= 0.3520(FLOAT)/0.3420(INT8)
LINKS: https://github.com/ultralytics/yolov5/releases/tag/v2.0

7.1.3.13. SSD_MobileNetv1¶

INPUT SIZE: 1x300x300x3
C(GOPs): 2.30
FPS: 277.29
ITC(ms): 6.914
TCPP(ms): 1.369
RV(mb): 9.23
WV(mb): 2.96
Dataset: VOC
ACCURACY: mAP: 0.7342(FLOAT)/0.7274(INT8)
LINKS: https://github.com/chuanqi305/MobileNet-SSD

7.1.3.14. EfficientDetd0¶

INPUT SIZE: 1x512x512x3
C(GOPs): 4.93
FPS: 78.31
ITC(ms): 17.080
TCPP(ms): 20.202
RV(mb): 7.62
WV(mb): 19.09
Dataset: COCO
ACCURACY: [IoU=0.50:0.95]= 0.3240(FLOAT)/0.3150(INT8)
LINKS: Download OE Package

7.1.3.15. CenterNet_Resnet50¶

INPUT SIZE: 1x512x512x3
C(GOPs): 51.75
FPS: 18.03
ITC(ms): 110.484
TCPP(ms): 26.496
RV(mb): 66.66
WV(mb): 33.88
Dataset: COCO
ACCURACY: [IoU=0.50:0.95]= 0.3180(FLOAT)/0.3130(INT8)
LINKS: Download OE Package

7.1.3.16. FCOS_efficientnetb0(QAT)¶

INPUT SIZE: 1x512x512x3
C(GOPs): 5.02
FPS: 169.43
ITC(ms): 12.053
TCPP(ms): 7.840
RV(mb): 7.82
WV(mb): 4.77
Dataset: COCO
ACCURACY: [IoU=0.50:0.95]= 0.3470(FLOAT)/0.3480(INT8)
LINKS: Download OE Package

7.1.3.17. Fcos_efficientnetb1¶

INPUT SIZE: 1x640x640x3
C(GOPs): 12.16
FPS: 65.30
ITC(ms): 30.155
TCPP(ms): 12.588
RV(mb): 29.70
WV(mb): 22.13
Dataset: COCO
ACCURACY: [IoU=0.50:0.95]= 0.4030(FLOAT)/0.4040(INT8)
LINKS: Download OE Package

7.1.3.18. FCOS_efficientnetb2¶

INPUT SIZE: 1x768x768x3
C(GOPs): 22.08
FPS: 38.43
ITC(ms): 51.120
TCPP(ms): 18.182
RV(mb): 51.87
WV(mb): 39.74
Dataset: COCO
ACCURACY: [IoU=0.50:0.95]= 0.4450(FLOAT)/0.4460(INT8)
LINKS: Download OE Package

7.1.3.19. FCOS_efficientnetb0(PTQ)¶

INPUT SIZE: 1x512x512x3
C(GOPs): 5.02
FPS: 174.14
ITC(ms): 13.360
TCPP(ms): 3.969
RV(mb): 8.06
WV(mb): 4.78
Dataset: COCO
ACCURACY: [IoU=0.50:0.95]= 0.3630(FLOAT)/0.3480(INT8)
LINKS: Download OE Package

7.1.3.20. UNet_mobilenet¶

INPUT SIZE: 1x1024x2048x3
C(GOPs): 7.37
FPS: 57.16
ITC(ms): 27.690
TCPP(ms): 21.501
RV(mb): 48.51
WV(mb): 49.91
Dataset: Cityscapes
ACCURACY: mIoU: 0.6411(FLOAT)/0.6382(INT8)
LINKS: Download OE Package

7.1.3.21. Deeplabv3plus_efficientnetb0¶

INPUT SIZE: 1x1024x2048x3
C(GOPs): 30.78
FPS: 23.65
ITC(ms): 79.348
TCPP(ms): 1.472
RV(mb): 84.31
WV(mb): 45.85
Dataset: Cityscapes
ACCURACY: mIoU: 0.7630(FLOAT)/0.7567(INT8)
LINKS: Download OE Package

7.1.3.22. Fastscnn_efficientnetb0¶

INPUT SIZE: 1x1024x2048x3
C(GOPs): 12.50
FPS: 47.16
ITC(ms): 41.305
TCPP(ms): 1.437
RV(mb): 47.00
WV(mb): 43.20
Dataset: Cityscapes
ACCURACY: mIoU: 0.6997(FLOAT)/0.6927(INT8)
LINKS: Download OE Package