6.4.2. Common Algorithm Model Samples¶

6.4.2.1. Sample Location¶

The common algorithmic model samples are located at 02_preq_examples/, 03_classification/, 04_detection/ and 07_segmentation/ folders in the horizon_model_convert_sample path.

6.4.2.2. Preparing Datasets¶

6.4.2.2.1. Dataset Download Address¶

The dataset can be downloaded from the following address.

Dataset	Address
ImageNet	https://www.image-net.org/download.php
COCO	https://cocodataset.org/
VOC	http://host.robots.ox.ac.uk/pascal/VOC/ (need to download both versions 2007 and 2012)
Cityscapes	https://github.com/mcordts/cityscapesScripts
CIFAR-10	http://www.cs.toronto.edu/~kriz/cifar.html

6.4.2.2.2. Dataset Reference Structure¶

To facilitate your subsequent steps, after the dataset has been downloaded, you need to process the evaluation dataset according to the structure suggested by D-Robotics below.

6.4.2.2.2.1. ImageNet¶

imagenet/
├── calibration_data
│   ├── ILSVRC2012_val_00000001.JPEG
│   ├── ...
│   └── ILSVRC2012_val_00000100.JPEG
├── ILSVRC2017_val.txt
├── val
│   ├── ILSVRC2012_val_00000001.JPEG
│   ├── ...
│   └── ILSVRC2012_val_00050000.JPEG
└── val.txt

6.4.2.2.2.2. COCO¶

coco/
├── calibration_data
│   ├── COCO_val2014_000000181007.jpg
│   ├── ...
│   └── COCO_val2014_000000181739.jpg
└── coco_val2017
    ├── annotations
    │   ├── instances_train2017.json
    │   └── instances_val2017.json
    └── images
        ├── 000000000139.jpg
        ├── 000000000285.jpg
        ├── ...
        ├── 000000581615.jpg
        └── 000000581781.jpg

6.4.2.2.2.3. VOC¶

Attention

Please note that the VOC2012 directory currently stores two datasets, VOC2007 and VOC2012, so please follow the directory structure below for the evaluation dataset.

VOCdevkit/
└── VOC2012
    ├── Annotations
    │   ├── 2007_000027.xml
    │   ├── ...
    │   └── 2012_004331.xml
    ├── ImageSets
    │   ├── Action
    │   │   ├── jumping_train.txt
    │   │   ├── jumping_trainval.txt
    │   │   ├── jumping_val.txt
    │   │   ├── ...
    │   │   ├── val.txt
    │   │   ├── walking_train.txt
    │   │   ├── walking_trainval.txt
    │   │   └── walking_val.txt
    │   ├── Layout
    │   │   ├── train.txt
    │   │   ├── trainval.txt
    │   │   └── val.txt
    │   ├── Main
    │   │   ├── aeroplane_train.txt
    │   │   ├── aeroplane_trainval.txt
    │   │   ├── aeroplane_val.txt
    │   │   ├── ...
    │   │   ├── train.txt
    │   │   ├── train_val.txt
    │   │   ├── trainval.txt
    │   │   ├── tvmonitor_train.txt
    │   │   ├── tvmonitor_trainval.txt
    │   │   ├── tvmonitor_val.txt
    │   │   └── val.txt
    │   └── Segmentation
    │       ├── train.txt
    │       ├── trainval.txt
    │       └── val.txt
    ├── JPEGImages
    │   ├── 2007_000027.jpg
    │   ├── ...
    │   └── 2012_004331.jpg
    ├── SegmentationClass
    │   ├── 2007_000032.png
    │   ├── ...
    │   └── 2011_003271.png
    ├── SegmentationObject
    │   ├── 2007_000032.png
    │   ├── ...
    │   └── 2011_003271.png
    └── train.txt

6.4.2.2.2.4. Cityscapes¶

cityscapes/
├── cityscapes_calibration_data
│   ├── aachen_000000_000019_leftImg8bit.png
│   ├── ...
│   └── aachen_000099_000019_leftImg8bit.png
├── gtFine
│   ├── test
│   │   ├── berlin
│   │   ├── ...
│   │   └── munich
│   ├── train
│   │   ├── aachen
│   │   ├── ...
│   │   └── zurich
│   └── val
│       ├── frankfurt
│       ├── lindau
│       └── munster
├── leftImg8bit
│   ├── test
│   │   ├── berlin
│   │   ├── ...
│   │   └── munich
│   ├── train
│   │   ├── aachen
│   │   ├── ...
│   │   └── zurich
│   └── val
│       ├── frankfurt
│       ├── lindau
│       └── munster
├── license.txt
└── README

6.4.2.2.2.5. CIFAR-10¶

cifar-10/
├── batches.meta
├── cifar10_val.txt
├── data_batch_1
├── data_batch_2
├── data_batch_3
├── data_batch_4
├── data_batch_5
├── readme.html
└── test_batch

6.4.2.3. Preparing Models¶

When using the model conversion sample package, please prepare the corresponding floating-point model first.

Note

If you need to do this process in the sample folder, you need to execute the 00_init.sh script in the folder first to get the corresponding original model and dataset.

Sources and modifications (if any) of the original models, please refer to below subsections.

6.4.2.3.1. Fcos_efficientnetb0¶

Attention

This model is a trained model using the QAT method.
To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Dequantize node from the bin model.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
be2fe17530bc366b038f5309199bf712	fcos_eff_b0.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.348(FLOAT)/0.349(INT8)

6.4.2.3.2. Fcos_efficientnetb2¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Dequantize node from the bin model.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
58ffc007a2ab9a053a559945fb27fac8	fcos_eff_b2.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.447(FLOAT)/0.447(INT8)

6.4.2.3.3. Fcos_efficientnetb3¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Dequantize node from the bin model.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
da37796ea8f2a4a54684a2520fdd6148	fcos_eff_b3.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.472(FLOAT)/0.474(INT8)

6.4.2.3.4. MobileNetv1/v2¶

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
3fd6889ec48bda46451d67274144e2a8	mobilenet.caffemodel
8922f90f629d428fecf866e798ac7c08	mobilenet_deploy.prototxt
54aab8425ea068d472e8e4015f22360c	mobilenet_v2.caffemodel
13101ee86ab6d217d5fd6ed46f7a4faa	mobilenet_v2_deploy.prototxt

3.Model accuracy:

MobileNetv1：0.7061(FLOAT)/0.7026(INT8)
MobileNetv2：0.7165(FLOAT)/0.7120(INT8)

6.4.2.3.5. GoogleNet¶

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
f107ae6806ea1016afbc718210b7a617	googlenet.onnx

3.Model accuracy: 0.7001(FLOAT)/0.6985(INT8)

6.4.2.3.6. ResNet18¶

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
0904d601fc930d4f0c62a2a95b3c3b93	resnet18.caffemodel
ee8ac82cd693a0fe55af42cca3fc52e5	resnet18_deploy.prototxt

3.Model accuracy: 0.6837(FLOAT)/0.6831(INT8)

6.4.2.3.7. EfficientNet_Lite0/1/2/3/4¶

Attention

To quickly start running samples and avoid the risks caused by third party tools, we strongly recommend using the off-the-shelf ONNX model in the model_zoo/mapper/ directory in D-Robotics model release package. If you are interested in reproducing the model conversion process of tflite2onnx, you can still try using the below 3rd-party tool, however, we do not guarantee the quality and success of the conversion.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum of the converted ONNX models in D-Robotics model_zoo:

md5sum	File
001a329bd367fbec22b415c7a33d7bdb	efficientnet_lite0_fp32.onnx
1205e95aea66650c71292bde236d55a9	efficientnet_lite1_fp32.onnx
474741c15494b79a89fe51d89e0c43c7	efficientnet_lite2_fp32.onnx
550455b41848d333f8359279c89a6bae	efficientnet_lite3_fp32.onnx
bde7fe57eadb4a30ef76f68da622dcd5	efficientnet_lite4_fp32.onnx

3.Download and get .tflite from the TAR package, and then convert it to an ONNX model using the tflite2onnx tool (https://pypi.org/project/tflite2onnx/).

Note that model layouts may vary by tflite2onnx version. If the input layout of the converted ONNX model is NHWC, when building, the configure input_layout_train of the EfficientNet_Lite0/1/2/3/4 should be NHWC.

4.Model accuracy:

EfficientNet_Lite0: 0.7490(FLOAT)/0.7469(INT8)
EfficientNet_Lite1: 0.7648(FLOAT)/0.7624(INT8)
EfficientNet_Lite2: 0.7738(FLOAT)/0.7715(INT8)
EfficientNet_Lite3: 0.7922(FLOAT)/0.7902(INT8)
EfficientNet_Lite4: 0.8069(FLOAT)/0.8058(INT8)

6.4.2.3.8. Vargconvnet¶

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
e21b8db17916f9046253bbe0bb8de3ef	vargconvnet.onnx

3.Model accuracy: 0.7790(FLOAT)/0.7785(INT8)

6.4.2.3.9. Efficientnasnet_m¶

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
fc36c052c6f034c0b64a6197b91b0c62	efficientnasnet-m.onnx

3.Model accuracy: 0.7973(FLOAT)/0.7916(INT8)

6.4.2.3.10. Efficientnasnet_s¶

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
e2744bd748f4265f4488676835a6ca24	efficientnasnet-s.onnx

3.Model accuracy: 0.7578(FLOAT)/0.7518(INT8)

6.4.2.3.11. YOLOv2_Darknet19¶

Attention

To quickly start running samples and avoid the risks caused by third party tools, we strongly recommend using the off-the-shelf Caffe model in the model_zoo/mapper/ directory in D-Robotics model release package. If you are interested in reproducing the model conversion process of darknet2caffe, you can still try using the below 3rd-party tool, however, we do not guarantee the quality and success of the conversion.
To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Dequantize node from the bin model.

1.YOLOv2_Darknet19 model: Obtain from model_zoo in the OE package.

Note

As this conversion tool is a simplified version, you should modify the the .prototxt file generated by this tool before the conversion, change the 'Reshape' layer to 'Passthrough' layer, and then add an NCHW2NHWC Permute operation to the output node.

For details about the parameters of the modified Passthrough layer, refer to the yolov2.prototxt sample.

2.MD5sum code:

md5sum	File
7aa7a6764401cebf58e73e72fcbd2a45	yolov2.caffemodel
72e9a51c1e284e4b66e69f72ca9214c8	yolov2_transposed.prototxt

3.Model accuracy:

[IoU=0.50:0.95]: 0.276(FLOAT)/0.273(INT8)

6.4.2.3.12. YOLOv3_Darknet53¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Dequantize node from the bin model.

1.YOLOv3_Darknet53 model: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
935af6e1530af5c0017b3674adce95e9	yolov3_transposed.prototxt
9a0f09c850656913ec27a6da06d9f9cc	yolov3.caffemodel

3.Model accuracy:

[IoU=0.50:0.95]: 0.333(FLOAT)/0.335(INT8)

6.4.2.3.13. YOLOv5x¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Dequantize node from the bin model.

1.YOLOv5x model: Obtain from model_zoo in the OE package.

Important

When cloning the source code, be sure to use v2.0 tags, otherwise the conversion may fail.

2.MD5sum code:

md5sum	File
2e296b5e31bf1e1b6b8ea4bf36153ea5	yolov5l.pt
16150e35f707a2f07e7528b89c032308	yolov5m.pt
42c681cf466c549ff5ecfe86bcc491a0	yolov5s.pt
069a6baa2a741dec8a2d44a9083b6d6e	yolov5x.pt

To better adapt to post-processing code, before exporting the ONNX model, we modified the code at Github as follows (for more code details see: https://github.com/ultralytics/yolov5/blob/v2.0/models/yolo.py):

  def forward(self, x):
      # x = x.copy()  # for profiling
      z = []  # inference output
      self.training |= self.export
      for i in range(self.nl):
          x[i] = self.m[i](x[i])  # conv
          bs, _, ny, nx = x[i].shape  # x(bs,255,20,20) to x(bs,3,20,20,85)
          #  x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()
          x[i] = x[i].permute(0, 2, 3, 1).contiguous()

Note

Remove the reshape from 4D to 5D at the end of each output branch (i.e., not splitting the channel from 255 to 3x85), then convert the layout from NHWC to NCHW before dumping.

The bottom left image displays the visualization of a certain output node before modifying the model, while the bottom right image displays the visualization of the corresponding output node after the modification.

After download, perform the pt to ONNX file conversion using the script https://github.com/ultralytics/yolov5/blob/v2.0/models/export.py.

Attention

When using the export.py script:

Because D-Robotics algorithm toolchain only supports ONNX opset 10 and 11, please modify the opset_version parameter in the torch.onnx.export as per the version you use.
Change the default input name parameter in the torch.onnx.export from 'image' into 'data' to keep it consistent with that of in the YOLOv5s sample in the model conversion sample package.
Change the default data input size in the parser.add_argument from 640x640 to 672x672 as in the YOLOv5x sample in the model conversion sample package.

3.Model accuracy:

[IoU=0.50:0.95]: 0.480(FLOAT)/0.466(INT8)

6.4.2.3.14. SSD_MobileNetv1¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Dequantize node from the bin model.

1.SSD_MobileNetv1 model: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
bbcb3b6a0afe1ec89e1288096b5b8c66	mobilenet_iter_73000.caffemodel
3c230e4415195a50c6248be80c49882d	MobileNetSSD_deploy.prototxt

3.Model accuracy (mAP): 0.7342(FLOAT)/0.7277(INT8)

6.4.2.3.15. Efficientdetd0¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Dequantize node from the bin model.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
ec4129c4b300cd04f1e8f71e0fe54ca5	efficientdet_nhwc.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.324(FLOAT)/0.315(INT8)

6.4.2.3.16. CenterNet_Resnet101¶

Attention

To achieve optimal on-board performance:

We put the maxpool and sigmoid nodes into the model and specify them to be compiled as BPU nodes for reducing the amount of computation during post-processing.
We modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Dequantize node from the bin model.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
db195ff784792f475e573c5126401d2a	centernet_resnet101_coco_modify.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.342(FLOAT)/0.335(INT8)

6.4.2.3.17. Fcos_efficientnetb0¶

Attention

This model is a trained model using the PTQ method.
To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Dequantize node from the bin model.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
9f9a1fe8508e2bd068e70146eb559b4f	fcos_efficientnetb0.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.358(FLOAT)/0.351(INT8)

6.4.2.3.18. Yolov4¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Dequantize node from the bin model.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
aaa3c3e5e4c4c1d4830b6501b1720e4d	yolov4_efficientnetb0.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.351(FLOAT)/0.334(INT8)

6.4.2.3.19. YOLOv3_VargDarknet¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Dequantize node from the bin model.

1.YOLOv3_VargDarknet model: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
fd4e46bc7c9798b51778d3aa09c5053a	yolov3_vargdarknet53.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.335(FLOAT)/0.327(INT8)

6.4.2.3.20. Fcos_resnet50¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Dequantize node from the bin model.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
0218942777615fac2f54cefdac4fbfa7	fcos_resnet50.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.426(FLOAT)/0.424(INT8)

6.4.2.3.21. Fcos_resnext101¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Dequantize node from the bin model.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
4b80efd22448021721ac5a860909c59f	fcos_resnext101.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.453(FLOAT)/0.450(INT8)

6.4.2.3.22. Unet_mobilenet¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Dequantize node from the bin model.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
21c6c645ebca92befbebc8c39d385c1e	tf_unet_trained.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.6411(FLOAT)/0.6379(INT8)

6.4.2.3.23. DeeplabV3plus_efficientnetb0¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Reshape and Cast node from the bin model.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
cf3a683f31b4b0ebe090647729f869d9	deeplabv3plus_efficientnetb0.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.7630(FLOAT)/0.7568(INT8)

6.4.2.3.24. Fastscnn_efficientnetb0¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Reshape and Cast node from the bin model.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
c1ace8f08a9c7b9c91509fa68327d0c8	fastscnn_efficientnetb0.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.6997(FLOAT)/0.6928(INT8)

6.4.2.3.25. Deeplabv3plus_dilation1248¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Transpose node from the bin model.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
ad002e572cbb49e1e99d893aac69f3e3	deeplabv3_cityscapes_dila1248_permute.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.7462(FLOAT)/0.7452(INT8)

6.4.2.3.26. Deeplabv3plus_efficientnetm1¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Reshape, Cast, and Transpose node from the bin model.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
0a1dfd01e173c68630d9e8dc1a6036fe	deeplabv3plus_efficientnetm1.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.7794(FLOAT)/0.7740(INT8)

6.4.2.3.27. Deeplabv3plus_efficientnetm2¶

Attention

To achieve optimal on-board performance, we modified the remove_node_type parameter in the YAML file used for bin model compilation and removed the Reshape, Cast, and Transpose node from the bin model.

1.Model source: Obtain from model_zoo in the OE package.

2.MD5sum code:

md5sum	File
c11a2673c4b3cf6e5d7bf1a051925d38	deeplabv3plus_efficientnetm2.onnx

3.Model accuracy:

[IoU=0.50:0.95]: 0.7882(FLOAT)/0.7856(INT8)

6.4.2.4. Demo of Algorithm Model Samples¶

Taking the GoogleNet model as an example, This section illustrates the steps of floating-point to fixed-point model conversion by using the scripts in 03_classification/02_googlenet/mapper/.

6.4.2.4.1. Docker Container Prep¶

First complete the docker installation and configuration and enter the docker container accoridng to Docker Container Deployment.

6.4.2.4.2. Obtain the Original Model and Calibration Dataset¶

Execute 00_init.sh in the 03_classification/02_googlenet/mapper/ folder to obtain the model and calibration dataset needed for the current sample.

# 1. Enter the folder the demo script resides
cd samples/ai_toolchain/horizon_model_convert_sample/03_classification/02_googlenet/mapper
# 2. Execute the script to obtain the original model and calibration dataset
sh 00_init.sh

6.4.2.4.3. Check If the Model is Executable¶

1.Run the script as follows:

# Execute model checking
sh 01_check.sh

2.Output the model inspection result:

The above script uses the hb_mapper checker tool to check whether the model can be supported by D-Robotics ASIC. An OP list will also be dumped to show whether an OP is processed by the BPU or the CPU.

======================================================================
Node                     ON   Subgraph  Type
-----------------------------------------------------------------------
Conv_0                   BPU  id(0)     HzSQuantizedConv
MaxPool_3                BPU  id(0)     HzQuantizedMaxPool
Conv_4                   BPU  id(0)     HzSQuantizedConv
Conv_7                   BPU  id(0)     HzSQuantizedConv
MaxPool_10               BPU  id(0)     HzQuantizedMaxPool
Conv_11                  BPU  id(0)     HzSQuantizedConv
Conv_14                  BPU  id(0)     HzSQuantizedConv
Conv_17                  BPU  id(0)     HzSQuantizedConv
Conv_20                  BPU  id(0)     HzSQuantizedConv
Conv_23                  BPU  id(0)     HzSQuantizedConv
MaxPool_26               BPU  id(0)     HzQuantizedMaxPool
Conv_27                  BPU  id(0)     HzSQuantizedConv
Concat_30                BPU  id(0)     Concat
Conv_31                  BPU  id(0)     HzSQuantizedConv
Conv_34                  BPU  id(0)     HzSQuantizedConv
Conv_37                  BPU  id(0)     HzSQuantizedConv
Conv_40                  BPU  id(0)     HzSQuantizedConv
Conv_43                  BPU  id(0)     HzSQuantizedConv
MaxPool_46               BPU  id(0)     HzQuantizedMaxPool
Conv_47                  BPU  id(0)     HzSQuantizedConv
Concat_50                BPU  id(0)     Concat
MaxPool_51               BPU  id(0)     HzQuantizedMaxPool
Conv_52                  BPU  id(0)     HzSQuantizedConv
Conv_55                  BPU  id(0)     HzSQuantizedConv
Conv_58                  BPU  id(0)     HzSQuantizedConv
Conv_61                  BPU  id(0)     HzSQuantizedConv
Conv_64                  BPU  id(0)     HzSQuantizedConv
MaxPool_67               BPU  id(0)     HzQuantizedMaxPool
Conv_68                  BPU  id(0)     HzSQuantizedConv
Concat_71                BPU  id(0)     Concat
Conv_72                  BPU  id(0)     HzSQuantizedConv
Conv_75                  BPU  id(0)     HzSQuantizedConv
Conv_78                  BPU  id(0)     HzSQuantizedConv
Conv_81                  BPU  id(0)     HzSQuantizedConv
Conv_84                  BPU  id(0)     HzSQuantizedConv
MaxPool_87               BPU  id(0)     HzQuantizedMaxPool
Conv_88                  BPU  id(0)     HzSQuantizedConv
Concat_91                BPU  id(0)     Concat
Conv_92                  BPU  id(0)     HzSQuantizedConv
Conv_95                  BPU  id(0)     HzSQuantizedConv
Conv_98                  BPU  id(0)     HzSQuantizedConv
Conv_101                 BPU  id(0)     HzSQuantizedConv
Conv_104                 BPU  id(0)     HzSQuantizedConv
MaxPool_107              BPU  id(0)     HzQuantizedMaxPool
Conv_108                 BPU  id(0)     HzSQuantizedConv
Concat_111               BPU  id(0)     Concat
Conv_112                 BPU  id(0)     HzSQuantizedConv
Conv_115                 BPU  id(0)     HzSQuantizedConv
Conv_118                 BPU  id(0)     HzSQuantizedConv
Conv_121                 BPU  id(0)     HzSQuantizedConv
Conv_124                 BPU  id(0)     HzSQuantizedConv
MaxPool_127              BPU  id(0)     HzQuantizedMaxPool
Conv_128                 BPU  id(0)     HzSQuantizedConv
Concat_131               BPU  id(0)     Concat
Conv_132                 BPU  id(0)     HzSQuantizedConv
Conv_135                 BPU  id(0)     HzSQuantizedConv
Conv_138                 BPU  id(0)     HzSQuantizedConv
Conv_141                 BPU  id(0)     HzSQuantizedConv
Conv_144                 BPU  id(0)     HzSQuantizedConv
MaxPool_147              BPU  id(0)     HzQuantizedMaxPool
Conv_148                 BPU  id(0)     HzSQuantizedConv
Concat_151               BPU  id(0)     Concat
MaxPool_152              BPU  id(0)     HzQuantizedMaxPool
Conv_153                 BPU  id(0)     HzSQuantizedConv
Conv_156                 BPU  id(0)     HzSQuantizedConv
Conv_159                 BPU  id(0)     HzSQuantizedConv
Conv_162                 BPU  id(0)     HzSQuantizedConv
Conv_165                 BPU  id(0)     HzSQuantizedConv
MaxPool_168              BPU  id(0)     HzQuantizedMaxPool
Conv_169                 BPU  id(0)     HzSQuantizedConv
Concat_172               BPU  id(0)     Concat
Conv_173                 BPU  id(0)     HzSQuantizedConv
Conv_176                 BPU  id(0)     HzSQuantizedConv
Conv_179                 BPU  id(0)     HzSQuantizedConv
Conv_182                 BPU  id(0)     HzSQuantizedConv
Conv_185                 BPU  id(0)     HzSQuantizedConv
MaxPool_188              BPU  id(0)     HzQuantizedMaxPool
Conv_189                 BPU  id(0)     HzSQuantizedConv
Concat_192               BPU  id(0)     Concat
GlobalAveragePool_193    BPU  id(0)     HzSQuantizedGlobalAveragePool
Gemm_195                 BPU  id(0)     HzSQuantizedConv
Gemm_195_reshape_output  CPU  --        Reshape

6.4.2.4.4. Prepare Calibration Dataset¶

Run the 02_preprocess.sh script in the same directory, as follows:

# Convert the images in 01_common/data/imagenet/calibration_data
# to: ./calibration_data_rgb_f32
sh 02_preprocess.sh

Note

We extracted 100 images from the ImageNet dataset and use them as a calibration dataset. Before calibration, we pre-processed the data and convert short size resize/crop size/NHWC to NCHW/to rgb.
The hb_mapper tool will load data from the converted binary data, the format of the pre-processed binary data file is c-order matrix storage, and data type of each matrix value is uint8.

6.4.2.4.5. Building Heterogeneous Models¶

Run the 03_build.sh script in the same directory, as follows:

sh 03_build.sh

Note

The above script uses the hb_mapper tool to convert the model. The most important thing to focus on is the conversion configuration file, refer to Model Compiling Tool (hb_mapper makertbin) section.

The output of the above script is as follows:

ls model_output | cat
cache.json
googlenet_224x224_nv12.bin
googlenet_224x224_nv12_calibrated_model.onnx
googlenet_224x224_nv12_optimized_float_model.onnx
googlenet_224x224_nv12_original_float_model.onnx
googlenet_224x224_nv12_quantized_model.onnx
torch-jit-export_subgraph_0.html
torch-jit-export_subgraph_0.json

Note

For now you ONLY need to focus on the googlenet_224x224_nv12.bin file.

6.4.2.4.6. Single Image Inference¶

Run the 04_inference.sh script to inference a single image, as follows:

sh 04_inference.sh

Note

As image pre-processing, model data post-processing are required in the image inference, we offered a Python sample script, please refer to sh 04_inference.sh.
This script is to perform the inference against a single image and verify whether the inference result meets the expectations. If you want to perform accuracy evaluation, refer to the script 05_evaluate.sh.

6.4.2.4.7. Accuracy Evaluation¶

Run the 05_evaluate.sh script to evaluate the accuracy, as follows:

export PARALLEL_PROCESS_NUM=${parallel_process_num}
sh 05_evaluate.sh

Note

As image pre-processing, model data post-processing are required in accuracy evaluation, awe offered a Python sample script, please refer to sh 05_evaluate.sh.
To accelerate the evaluation, you can adjust the number of concurrent processes by using the -p option; meanwhile, pay attention to the memory usage.

When the value of the -p option is set to 0 or left unfilled, the fixed-point models in the CPU environment will be processed by 10 processes, while other scenarios using 1 process.

6.4.2.5. FAQ¶

6.4.2.5.1. How to align the ONNX original floating-point model obtained using open-source frameworks with the *_original_float_model.onnx model from hb_mapper makertbin?¶

Note

The difference between the original floating-point model and the converted ONNX floating-point model is guaranteed by the toolchain product, hb_mapper, which is not a required standard process.

1. Understand the concepts of these 2 models

Let’s clarify the concepts of these two models.

The former refers to the original ONNX floating-point models trained by developers using opensource frameworks, e.g., TensorFlow、PyTorch、MXNet, etc.

The latter refers to the *_original_float_model.onnx intermediate model converted by using either the hb_mapper makertbin tool or the 03_classification/${modelname}/mapper/03_build.sh script in D-Robotics model conversion sample package (i.e., the released horizon_model_convert_sample), where, * denotes the name of a specific model (e.g., MobileNetv1, GoogleNet, etc.).

2. Understand the distinctions between these 2 models

Both *_original_float_model.onnx model and the original floating-point model share the same computational accuracy, except that the former added some data pre-processing computation.

Typically, you don’t really need to use this model unless there is abnormality in conversion results, which may help you solve the issue faster if you can provide this model to our technical support.

3. Write your own script to align these 2 models

Note

The following contents take the googlenet model in horizon_model_convert_sample package as an example to describe how to align the inference results of these 2 models.

To do this, you need to write your own scripts, where you need to note the following:

Attention

The logic between the following two should be consistent to avoid inconsistent inference results caused by different image data processing logic:

The image data processing logic in your own script
The logic in the mapper/preprocess.py script in the sample package

Note that code logic may vary by sample package version. In such cases, please refer to the image data preprocessing script or contact our technical staff.

See the following code block for each preprocessing transformer method:

import sys
sys.path.append("../../../01_common/python/data/")
from transformer import *
from dataloader import *

# image calibration transformer
def calibration_transformers():
  """
  step:
      1、short size resize to 256
      2、crop size 224 * 224 from center
      3、NHWC to NCHW
      4、bgr to rgb
  """
  transformers = [
      ShortSideResizeTransformer(short_size=256),
      CenterCropTransformer(crop_size=224),
      HWC2CHWTransformer(),
      BGR2RGBTransformer()
  ]
  return transformers

# image inference transformer
def infer_transformers(input_layout="NHWC"):
  """
  step:
      1、PIL resize to 256
      2、crop size 224*224 from PIL center
      3、bgr to nv12
      4、nv12 to yuv444
  :param input_layout: input layout
  """
    transformers = [
        PILResizeTransformer(size=256),
        PILCenterCropTransformer(size=224),
        BGR2NV12Transformer(data_format="HWC"),
        NV12ToYUV444Transformer((224, 224)),
                                yuv444_output_layout=input_layout[1:]),
    ]
    return transformers

You can align data pre-processing logic as follows:

def ShortSideResizeTransformer(data, short_size):
    image = data
    height, width, _ = image.shape
    if height < width:
        off = width / height
        image = cv2.resize(image,
                            (int(short_size * off), short_size))
    else:
        off = height / width
        image = cv2.resize(image,
                            (short_size, int(short_size * off)))
    data = image
    data = data.astype(np.float32)
    return data

def CenterCropTransformer(data, crop_size):
    image = data
    resize_height, resize_width, _ = image.shape
    resize_up = resize_height // 2 - crop_size // 2
    resize_left = resize_width // 2 - crop_size // 2
    data = image[resize_up:resize_up +
                    crop_size, resize_left:resize_left +
                    crop_size, :]
    data = data.astype(np.float32)
    return data

def preprocess(data):
    data = ShortSideResizeTransformer(data, short_size=256)  # ShortSideResize
    data = CenterCropTransformer(data, crop_size=224)  # CenterCrop
    data = np.transpose(data, (2, 0, 1))   # HWC2CHW
    data = data * 255  # (0, 1) --> (0, 255)

Attention

As shown in below figure, an HzPreprocess operator is added into the model googlenet_224x224_nv12_original_float_model.onnx, in order to implement the data_mean_and_scale operation in googlenet_config.yaml.

Therefore, you need to implement the data normalization based on the mean_value and scale_value parameters in googlenet_config.yaml, as follows:

# Normalize
data = data.astype(np.float32)
mean = np.array([127.5, 127.5, 127.5])
scale = np.array([0.0078431, 0.0078431, 0.0078431])
norm_data = np.zeros(data.shape).astype(np.float32)
for i in range(data.shape[0]):
    norm_data[i,:,:] = (data[i,:,:] - mean[i]) * scale[i]
norm_data = norm_data.reshape(1, 3, 224, 224).astype(np.float32)

Attention

By default, the mapper/04_inference.sh script of all models performs the fixed-point model inference. Therefore, when validating the inference result of a floating-point model, you should run sh 04_inference.sh origin to perform the floating-point model inference.

Note that code logic may vary by sample package version. In such cases, please refer to the comments in the script 03_classification/02_googlenet/mapper/04_inference.sh script.

After implementing the above steps, you can align the results of the original floating-point model with the googlenet_224x224_nv12_original_float_model.onnx model.

6.4.2.5.2. Why is the reproduced accuracy slightly different from the indicators in the documentation?¶

There are two possible reasons:

There may be minor differences in calculation methods when in different server environments, which can cause small data fluctuations in the accuracy of the compiled fixed-point ONNX models in different server environments when compared with the documentation.
The user side uses different versions of third-party libraries such as OpenCV and NumPy, which may produce different results after pre-processing, and this may also lead to slight data fluctuations in reproduced accuracy when compared with the documentation.

No need to worry much about this situation, the records provided in the documentation is only for reference, and it is ok that your reproduced accuracy is slightly different from those in documents.

6.4.2.5.3. Why is the fixed-point model accuracy not aligned with the on-board accuracy of the bin file in the ai_benchmark example?¶

In the standard delivery, when adding the example, we have already aligned the fixed-point model accuracy with the bin on-board accuracy in the ai_benchmark example.

If you find any unaligned accuracy, we recommend you first checking the model input for consistency.

When executing the fixed-point model evaluation script, you use the dataset of image type, while for the bin model used on board, you use the binary dataset converted by the hb_eval_preprocess tool.

Based on this, if the dataset you used on-board is not generated by using the above methods, we recommend that you first use our data preprocessing tool (i.e., hb_eval_preprocess) to regenerate the dataset needed for on-board running on the same server that you run the fixed-point model accuracy and rerun the on-board accuracy to ensure the model inputs are consistent.

Attention

Make sure to use the same environment to generate the dataset by using the hb_eval_preprocess tool and to run the fixed-point model accuracy.