10. Benchmark of Model Performance

10.1. Descriptions

  • Test Conditions:

    • Test Board: X5-DVB.

    • Number of Test Cores: Single core.

    • Frequency to obtain model performance data: Average of performance parameters over a 5-minute period.

    • Python version: Python 3.10.

  • Table Header Acronyms:

    • C = Computation, in GOPs (i.e., billion operations per second), obtained by using the hb_perf tool.

    • FPS = Frame(s) Per Second, obtained by running the fps.sh script with single thread of different models in the ai_benchmark sample package/script on the dev board. Post-processing included.

    • ITC = Inference Time Consumption, in ms (millisecond), obtained by running the latency.sh script with single thread of different models in the ai_benchmark sample package/script on the dev board. Post-processing not included.

    • TCPP = Postprocess Time Consumption, in ms (millisecond), obtained by running the latency.sh script with single thread of different models in the ai_benchmark sample package/script on the dev board.

    • RV = Read Volume of single frame, in mb (Mbit), obtained by using the hb_perf tool.

    • WV = Write Volume of single frame, in mb (Mbit), obtained by using the hb_perf tool.

10.2. Important Model Performance Data

MODEL NAME

INPUT SIZE

C(GOPs)

FPS

ITC(ms)

TCPP(ms)

Dataset

MobileNetv1

1x224x224x3

1.14

1365.00

1.015

0.051

ImageNet

MobileNetv2

1x224x224x3

0.63

1492.78

0.892

0.068

ImageNet

GoogleNet

1x224x224x3

3.00

631.55

1.851

0.052

ImageNet

Resnet18

1x224x224x3

3.65

449.30

2.523

0.052

ImageNet

EfficientNet_Lite0

1x224x224x3

0.77

1014.45

1.272

0.052

ImageNet

EfficientNet_Lite1

1x240x240x3

1.20

828.00

1.508

0.052

ImageNet

EfficientNet_Lite2

1x260x260x3

1.72

627.00

1.887

0.052

ImageNet

EfficientNet_Lite3

1x280x280x3

2.77

455.70

2.490

0.051

ImageNet

EfficientNet_Lite4

1x300x300x3

5.11

285.15

3.804

0.052

ImageNet

Vargconvnet

1x224x224x3

9.06

310.87

3.488

0.052

ImageNet

Efficientnasnet_m

1x300x300x3

4.53

291.47

3.493

0.052

ImageNet

Efficientnasnet_s

1x280x280x3

1.44

777.49

1.559

0.052

ImageNet

YOLOv2_Darknet19

1x608x608x3

62.94

38.33

26.448

1.288

COCO

YOLOv3_Darknet53

1x416x416x3

65.90

31.28

32.377

7.582

COCO

YOLOv5x_v2.0

1x672x672x3

243.91

10.37

96.864

29.262

COCO

Ssd_mobilenetv1

1x300x300x3

2.30

618.90

1.896

0.916

VOC

Centernet_resnet101

1x512x512x3

90.54

26.38

38.314

3.764

COCO

YOLOv3_VargDarknet

1x416x416x3

42.82

51.73

19.791

7.567

COCO

Deeplabv3plus_efficientnetb0

1x1024x2048x3

30.78

35.84

28.170

0.900

Cityscapes

Fastscnn_efficientnetb0

1x1024x2048x3

12.49

70.02

14.460

0.822

Cityscapes

Deeplabv3plus_efficientnetm1

1x1024x2048x3

77.05

18.91

53.123

0.881

Cityscapes

Deeplabv3plus_efficientnetm2

1x1024x2048x3

124.16

12.48

80.466

0.840

Cityscapes

Resnet50

1x224x224x3

7.72

209.85

4.994

0.068

ImageNet

VargNetV2

1x224x224x3

0.72

1521.44

1.010

0.068

ImageNet

Swint

1x224x224x3

8.98

44.53

22.725

0.069

ImageNet

MixVarGENet

1x224x224x3

2.07

1278.23

1.010

0.068

ImageNet

Fcos_efficientnetb0

1x512x512x3

5.02

342.34

3.177

0.217

COCO

Fcos_efficientnetb2

1x768x768x3

22.08

69.26

14.980

5.738

COCO

Fcos_efficientnetb3

1x896x896x3

41.45

37.42

27.320

7.753

COCO

Pointpillars_kitti_car

1x1x150000x4

66.82

26.15

77.429

2.296

Kitti3d

RetinaNet_vargnetv2_fpn

1x1024x1024x3

301.27

9.83

102.106

4.858

COCO

Yolov3_mobilenetv1

1x416x416x3

20.58

95.67

10.878

1.275

VOC

Ganet_mixvargenet

1x320x800x3

10.74

290.82

3.687

0.795

CuLane

DETR_resnet50

1x800x1333x3

202.99

7.60

131.894

1.391

MS COCO

DETR_efficientnetb3

1x800x1333x3

67.31

10.69

393.779

1.399

MS COCO

FCOS3D_efficientnetb0

1x512x896x3

19.94

597.53

3.921

8.714

nuscenes

Centerpoint_pointpillar

300000x5

127.73

102.36

24.822

52.530

nuscenes

Keypoint_efficientnetb0

1x128x128x3

0.45

1345.10

0.968

0.296

carfusion

Unet_mobilenetv1

1x1024x2048x3

7.36

175.54

5.943

0.474

Cityscapes

Pwcnet_pwcnetneck

1x384x512x6

81.71

24.32

41.366

0.244

flyingchairs

Motr_efficientnetb3

image:

1x800x1422x3

track_query:

1x2x128x156

ref_points:

1x2x128x4

mask_query:

1x1x256x1

64.43

12.87

76.466

21.386

Mot17

Bev_lss_efficientnetb0_multitask

image:

6x256x704x3

points(0&1):

10x128x128x2

2.41

31.27

32.997

14.508

nuscenes

Bev_gkt_mixvargenet_multitask

image:

6x512x960x3

points(0-8):

6x64x64x2

34.49

10.76

94.288

14.636

nuscenes

Bev_ipm_efficientnetb0_multitask

image:

6x512x960x3

points:

6x128x128x2

8.83

25.15

40.900

14.543

nuscenes

Bev_ipm_4d_efficientnetb0_multitask

image:

6x512x960x3

points:

6x128x128x2

prev_feat:

1x128x128x64

prev_point:

1x128x128x2

8.93

23.12

45.353

14.780

nuscenes

Detr3d_efficientnetb3_nuscenes

coords(0-3):

6x4x256x2

image:

6x512x1408x3

masks:

1x4x256x24

37.55

4.87

206.104

2.009

nuscenes

Centerpoint_mixvargnet_multitask

300000x5

51.45

104.42

23.771

50.498

nuscenes

Stereonetplus_mixvargenet

2x544x960x3

24.29

253.98

6.477

15.395

SceneFlow

Densetnt_vectornet

goals_2d:

30x1x2048x2

goals_2d_mask:

30x1x2048x1

instance_mask:

30x1x96x1

lane_feat:

30x9x64x11

traj_feat:

30x19x32x9

0.42

86.81

26.482

10.476

Argoverse 1

10.3. Model Full Performance Data

10.3.1. MobileNetv1

10.3.2. MobileNetv2

  • INPUT SIZE: 1x224x224x3

  • C(GOPs): 0.63

  • FPS: 1492.78

  • ITC(ms): 0.892

  • TCPP(ms): 0.068

  • RV(mb): 3.03

  • WV(mb): 0.12

  • Dataset: ImageNet

10.3.3. GoogleNet

10.3.4. Resnet18

10.3.5. EfficientNet_Lite0

10.3.6. EfficientNet_Lite1

10.3.7. EfficientNet_Lite2

10.3.8. EfficientNet_Lite3

10.3.9. EfficientNet_Lite4

10.3.10. Vargconvnet

10.3.11. Efficientnasnet_m

10.3.12. Efficientnasnet_s

10.3.13. YOLOv2_Darknet19

10.3.14. YOLOv3_Darknet53

10.3.15. YOLOv5x_v2.0

10.3.16. Ssd_mobilenetv1

10.3.17. Centernet_resnet101

10.3.18. YOLOv3_VargDarknet

10.3.19. Deeplabv3plus_efficientnetb0

10.3.20. Fastscnn_efficientnetb0

10.3.21. Deeplabv3plus_efficientnetm1

10.3.22. Deeplabv3plus_efficientnetm2

10.3.23. Resnet50

  • INPUT SIZE: 1x224x224x3

  • C(GOPs): 7.72

  • FPS: 209.85

  • ITC(ms): 4.994

  • TCPP(ms): 0.068

  • RV(mb): 25.30

  • WV(mb): 1.77

  • Dataset: ImageNet

10.3.24. VargNetV2

  • INPUT SIZE: 1x224x224x3

  • C(GOPs): 0.72

  • FPS: 1521.44

  • ITC(ms): 1.010

  • TCPP(ms): 0.068

  • RV(mb): 3.66

  • WV(mb): 0.02

  • Dataset: ImageNet

10.3.25. Swint

  • INPUT SIZE: 1x224x224x3

  • C(GOPs): 8.98

  • FPS: 44.53

  • ITC(ms): 22.725

  • TCPP(ms): 0.069

  • RV(mb): 46.90

  • WV(mb): 11.00

  • Dataset: ImageNet

10.3.26. MixVarGENet

  • INPUT SIZE: 1x224x224x3

  • C(GOPs): 2.07

  • FPS: 1278.23

  • ITC(ms): 1.010

  • TCPP(ms): 0.068

  • RV(mb): 2.37

  • WV(mb): 0.12

  • Dataset: ImageNet

10.3.27. Fcos_efficientnetb0

  • INPUT SIZE: 1x512x512x3

  • C(GOPs): 5.02

  • FPS: 342.34

  • ITC(ms): 3.177

  • TCPP(ms): 0.217

  • RV(mb): 5.85

  • WV(mb): 1.28

  • Dataset: COCO

10.3.28. Fcos_efficientnetb2

10.3.29. Fcos_efficientnetb3

10.3.30. Pointpillars_kitti_car

  • INPUT SIZE: 1x1x150000x4

  • C(GOPs): 66.82

  • FPS: 26.15

  • ITC(ms): 77.429

  • TCPP(ms): 2.296

  • RV(mb): 72.90

  • WV(mb): 40.00

  • Dataset: Kitti3d

10.3.31. RetinaNet_vargnetv2_fpn

  • INPUT SIZE: 1x1024x1024x3

  • C(GOPs): 301.27

  • FPS: 9.83

  • ITC(ms): 102.106

  • TCPP(ms): 4.858

  • RV(mb): 145.00

  • WV(mb): 87.30

  • Dataset: COCO

10.3.32. Yolov3_mobilenetv1

  • INPUT SIZE: 1x416x416x3

  • C(GOPs): 20.58

  • FPS: 95.67

  • ITC(ms): 10.878

  • TCPP(ms): 1.275

  • RV(mb): 29.30

  • WV(mb): 4.82

  • Dataset: VOC

10.3.33. Ganet_mixvargenet

  • INPUT SIZE: 1x320x800x3

  • C(GOPs): 10.74

  • FPS: 290.82

  • ITC(ms): 3.687

  • TCPP(ms): 0.795

  • RV(mb): 5.23

  • WV(mb): 3.71

  • Dataset: CuLane

10.3.34. DETR_resnet50

  • INPUT SIZE: 1x800x1333x3

  • C(GOPs): 202.99

  • FPS: 7.60

  • ITC(ms): 131.894

  • TCPP(ms): 1.391

  • RV(mb): 384.00

  • WV(mb): 264.00

  • Dataset: MS COCO

10.3.35. DETR_efficientnetb3

  • INPUT SIZE: 1x800x1333x3

  • C(GOPs): 67.31

  • FPS: 10.69

  • ITC(ms): 93.779

  • TCPP(ms): 1.399

  • RV(mb): 250.00

  • WV(mb): 187.00

  • Dataset: MS COCO

10.3.36. FCOS3D_efficientnetb0

  • INPUT SIZE: 1x512x896x3

  • C(GOPs): 19.94

  • FPS: 597.53

  • ITC(ms): 3.921

  • TCPP(ms): 8.714

  • RV(mb): 11.55

  • WV(mb): 5.10

  • Dataset: nuscenes

10.3.37. Centerpoint_pointpillar

  • INPUT SIZE: 300000x5

  • C(GOPs): 127.73

  • FPS: 102.36

  • ITC(ms): 24.822

  • TCPP(ms): 52.530

  • RV(mb): 39.37

  • WV(mb): 19.04

  • Dataset: nuscenes

10.3.38. Keypoint_efficientnetb0

  • INPUT SIZE: 1x128x128x3

  • C(GOPs): 0.45

  • FPS: 1345.10

  • ITC(ms): 0.968

  • TCPP(ms): 0.296

  • RV(mb): 4.38

  • WV(mb): 0.01

  • Dataset: carfusion

10.3.39. Unet_mobilenetv1

  • INPUT SIZE: 1x1024x2048x3

  • C(GOPs): 7.36

  • FPS: 175.54

  • ITC(ms): 5.943

  • TCPP(ms): 0.474

  • RV(mb): 19.60

  • WV(mb): 14.70

  • Dataset: Cityscapes

10.3.40. Pwcnet_pwcnetneck

  • INPUT SIZE: 1x384x512x6

  • C(GOPs): 81.71

  • FPS: 24.32

  • ITC(ms): 41.366

  • TCPP(ms): 0.244

  • RV(mb): 66.60

  • WV(mb): 44.90

  • Dataset: flyingchairs

10.3.41. Motr_efficientnetb3

  • INPUT SIZE: image: 1x800x1422x3 track_query: 1x2x128x156 ref_points: 1x2x128x4 mask_query: 1x1x256x1

  • C(GOPs): 64.43

  • FPS: 12.87

  • ITC(ms): 76.466

  • TCPP(ms): 21.386

  • RV(mb): 157.00

  • WV(mb): 103.00

  • Dataset: Mot17

10.3.42. Bev_lss_efficientnetb0_multitask

  • INPUT SIZE: image: 6x256x704x3 points(0&1): 10x128x128x2

  • C(GOPs): 2.41

  • FPS: 31.27

  • ITC(ms): 32.997

  • TCPP(ms): 14.508

  • RV(mb): 6.03

  • WV(mb): 4.28

  • Dataset: nuscenes

10.3.43. Bev_gkt_mixvargenet_multitask

  • INPUT SIZE: image: 6x512x960x3 points(0-8): 6x64x64x2

  • C(GOPs): 34.49

  • FPS: 10.76

  • ITC(ms): 94.288

  • TCPP(ms): 14.636

  • RV(mb): 22.70

  • WV(mb): 12.50

  • Dataset: nuscenes

10.3.44. Bev_ipm_efficientnetb0_multitask

  • INPUT SIZE: image: 6x512x960x3 points: 6x128x128x2

  • C(GOPs): 8.83

  • FPS: 25.15

  • ITC(ms): 40.900

  • TCPP(ms): 14.543

  • RV(mb): 8.33

  • WV(mb): 6.45

  • Dataset: nuscenes

10.3.45. Bev_ipm_4d_efficientnetb0_multitask

  • INPUT SIZE: image: 6x512x960x3 points: 6x128x128x2 prev_feat: 1x128x128x64 prev_point: 1x128x128x2

  • C(GOPs): 8.93

  • FPS: 23.12

  • ITC(ms): 45.353

  • TCPP(ms): 14.780

  • RV(mb): 9.81

  • WV(mb): 7.70

  • Dataset: nuscenes

10.3.46. Detr3d_efficientnetb3_nuscenes

  • INPUT SIZE: coords(0-3): 6x4x256x2 image: 6x512x1408x3 masks: 1x4x256x24

  • C(GOPs): 37.55

  • FPS: 4.87

  • ITC(ms): 206.104

  • TCPP(ms): 2.009

  • RV(mb): 103.00

  • WV(mb): 77.20

  • Dataset: nuscenes

10.3.47. Centerpoint_mixvargnet_multitask

  • INPUT SIZE: 300000x5

  • C(GOPs): 51.45

  • FPS: 104.42

  • ITC(ms): 23.771

  • TCPP(ms): 50.498

  • RV(mb): 33.84

  • WV(mb): 14.45

  • Dataset: nuscenes

10.3.48. Stereonetplus_mixvargenet

  • INPUT SIZE: 2x544x960x3

  • C(GOPs): 24.29

  • FPS: 253.98

  • ITC(ms): 6.477

  • TCPP(ms): 15.395

  • RV(mb): 15.40

  • WV(mb): 10.90

  • Dataset: SceneFlow

10.3.49. Densetnt_vectornet

  • INPUT SIZE: goals_2d: 30x1x2048x2 goals_2d_mask: 30x1x2048x1 instance_mask: 30x1x96x1 lane_feat: 30x9x64x11 traj_feat: 30x19x32x9

  • C(GOPs): 0.42

  • FPS: 86.81

  • ITC(ms): 26.482

  • TCPP(ms): 10.476

  • RV(mb): 3.29

  • WV(mb): 2.92

  • Dataset: Argoverse 1